WO2024252695A1 - Program, information processing method, and information processing system - Google Patents

Abstract

The present invention provides a variety of information to a user in a virtual space without degrading usability. A user device is caused to function as: a first output control unit that outputs, to a display visually recognized by a user, a virtual space image of a three-dimensional virtual space viewed from a virtual camera; a second output control unit that outputs, to the inside of the virtual space image, a projection screen that can be operated by a user; an input coordinate acquisition unit that acquires input coordinates onto an input device of the user; and an operation determination unit that, on the basis of the acquired input coordinates, determines whether the input operation of the user is an operation for an input region corresponding to the projection screen, wherein when the operation determination unit determines that the input operation is not for the input region, the first output control unit performs first display control processing relating to the three-dimensional virtual space other than control on the projection screen, and when the operation determination unit determines that the input operation is for the input region, the second output control unit performs second display control processing on the projection screen in accordance with the input coordinates.

Description

PROGRAM, INFORMATION PROCESSING METHOD AND INFORMATION PROCESSING SYSTEM

This disclosure relates to a program, an information processing method, and an information processing system.

A service is already being provided that displays a space, such as an event venue, created in a virtual space on a user device. For example, in this virtual space, the user can gather information by viewing exhibits (see, for example, Patent Document 1).

JP 2001-160065 A

Depending on the virtual space, the user device can accept instructions for exhibits based on user input operations, such as selecting an exhibit or enlarging the display. On the other hand, the user device must also accept user instructions for things other than exhibits that are related to the virtual space. The user device must distinguish between these instructions depending on the input operation. Increasing the number of acceptable instructions makes the input operation more complicated. This may impair usability, such as ease of operation. Note that similar issues exist not only in virtual spaces at event venues where exhibits are placed, but also in other virtual spaces where objects are placed that display information for users to view.

The program for solving the above problem causes one or more computers to function as a first output control unit that outputs a virtual space image of a three-dimensional virtual space seen from a virtual camera to a display viewed by a user, a second output control unit that outputs a projection screen that can be operated by the user within the virtual space image, an input coordinate acquisition unit that acquires input coordinates to an input device of the user, and an operation determination unit that determines whether or not an input operation by the user is an operation on an input area corresponding to the projection screen based on the acquired input coordinates, and when the operation determination unit determines that the input operation is not an operation on the input area, the first output control unit performs a first display control process for the three-dimensional virtual space other than control of the projection screen, and when the operation determination unit determines that the input operation is an operation on the input area, the second output control unit performs a second display control process on the projection screen according to the input coordinates.

The information processing method for solving the above problem includes a first output control process in which one or more computers output a virtual space image of a three-dimensional virtual space seen from a virtual camera to a display viewed by a user, a second output control process in which a projection screen that can be operated by the user is output within the virtual space image, an input coordinate acquisition process in which input coordinates are acquired for the user's input device, and an operation determination process in which, based on the acquired input coordinates, it is determined whether the user's input operation is an operation on an input area corresponding to the projection screen, and if it is determined in the operation determination process that the input operation is not an operation on the input area, a first display control process is performed for the three-dimensional virtual space other than control over the projection screen, and if it is determined in the operation determination process that the input operation is an operation on the input area, a second display control process is performed on the projection screen according to the input coordinates.

The information processing system that solves the above problem includes a first output control unit that outputs a virtual space image of a three-dimensional virtual space viewed from a virtual camera to a display viewed by a user, a second output control unit that outputs a projection screen that can be operated by the user within the virtual space image, an input coordinate acquisition unit that acquires input coordinates to an input device of the user, and an operation determination unit that determines whether or not the input operation of the user is an operation on an input area corresponding to the projection screen based on the acquired input coordinates, and when the operation determination unit determines that the input operation is not an operation on the input area, the first output control unit performs a first display control process for the three-dimensional virtual space other than control of the projection screen, and when the operation determination unit determines that the input operation is an operation on the input area, the second output control unit performs a second display control process on the projection screen according to the input coordinates.

According to this disclosure, it is possible to provide users with a wide variety of information in a virtual space without compromising usability.

1 is a schematic diagram illustrating an example of an information processing system according to a first embodiment. FIG. 2 is a diagram illustrating hardware of a device such as a user device in the first embodiment. FIG. 2 is a block diagram illustrating functions of a user device according to the first embodiment. FIG. 2 is a block diagram illustrating functions of a management server according to the first embodiment. FIG. 2 is a diagram illustrating user management information according to the first embodiment. FIG. 2 is a diagram illustrating view tally information according to the first embodiment. FIG. 4 is a diagram illustrating performance information according to the first embodiment. FIG. 2 is a diagram illustrating a virtual space according to the first embodiment. 4A to 4E are diagrams illustrating examples of input operations received by the user device of the first embodiment. 5 is a flowchart showing a processing procedure of a user device according to the first embodiment; 5 is a flowchart showing a processing procedure of a user device according to the first embodiment; FIG. 2 is a diagram illustrating a virtual space screen displayed on a user device according to the first embodiment. FIG. 2 is a diagram illustrating a virtual space screen displayed on a user device according to the first embodiment. FIG. 2 is a diagram illustrating a virtual space screen displayed on a user device according to the first embodiment. FIG. 2 is a diagram illustrating a virtual space screen displayed on a user device according to the first embodiment. 13A to 13C are diagrams illustrating transmission and reception of input operation data in a second embodiment. FIG. 11 is a diagram illustrating a virtual space screen displayed on a user device according to a second embodiment.

[First embodiment]
Hereinafter, a first embodiment of a program, an information processing method, and an information processing system will be described.

<Information Processing System>
An information processing system 11 will be described with reference to Fig. 1. The information processing system 11 includes a management server 12 and a user device 20.

The information processing system 11 provides a service of outputting an image of a three-dimensional virtual space (hereinafter referred to as virtual space) to a user device 20. In the virtual space, the virtual position of the user 10 is movable. The virtual space is also called a world or a field. The information processing system 11 is a system that distributes a video of the virtual space displayed by one user 10 to other users 10, or a system in which the user device 20 outputs an image of the three-dimensional virtual space without distributing such a video. In this embodiment, the information processing system 11 is described as a system that distributes videos.

In one embodiment, the management server 12 is managed by a PF provider that provides a platform (PF) for distributing and viewing videos in a virtual space. The web server 13 is managed by a publisher. The publisher uses the platform to provide resources, which are information on products or services, to users 10. When the publisher uses the platform, the resources are associated in advance with the virtual space in which the resources are published. When one publisher is associated with one virtual space (world), the resources published in a specified area of the virtual space are associated with this publisher. When multiple publishers are associated with one virtual space (world), the resources published in a specified area of the virtual space are associated with the multiple publishers. The publisher has the advantage of being able to advertise its own products or services by providing its own information to users 10. The PF provider may receive compensation from the publisher for making the platform available to the publisher. Alternatively, the PF provider may be the publisher.

The management server 12 comprises one or more devices. In other words, the management server 12 may be a group of servers. The management server 12 is connected to user devices 20 via a network such as the Internet. The user devices 20 are devices on which a program for distributing and viewing videos is installed (implemented), or devices used by users 10 who have registered with a video providing service for distributing and viewing videos. The management server 12 is responsible for relaying data related to video distribution between the user devices 20.

1 shows two users 10, namely, users 10A and 10B. User 10A is a user who distributes videos using user device 20A, i.e., a distribution user, and user 10B is a user who watches the distributed videos on user device 20B, i.e., a viewing user. One user 10 can be a distribution user 10A at one time and a viewing user 10B at another time. That is, when user 10 distributes videos, he is a distribution user 10A, and when he watches videos, he is a viewing user 10B. In addition, user device 20A that distributes videos is called a distribution user device, and user device 20 that outputs videos for viewing by users is called a viewing user device 20B. User device 20 can be switched between a mode for distributing videos, i.e., a distribution mode, and a mode for outputting videos for viewing, i.e., a viewing mode. In this embodiment, when describing without distinguishing between distribution user 10A and viewing user 10B, they will simply be called user 10. In addition, when describing the broadcasting user device 20A and the viewing user device 20B without distinguishing between them, they will simply be referred to as user devices 20.

<Hardware>
The hardware of the user device 20 will be described with reference to FIG. 2. The user device 20 is a smartphone (multi-function phone terminal), a tablet terminal, a personal computer, a console game machine, a wearable computer, or any other information processing device capable of playing videos other than these devices. The wearable computer may have a screen on which the user can watch videos, and may be, for example, a head-mounted display worn on the user's head, or a glasses-type wearable terminal. The user device 20 may be a device that operates independently, or may be a set of multiple devices that are connected so that various data can be transmitted and received between each other. An example of the latter user device 20 is, for example, a system that performs tracking using an outside-in method.

The user device 20 includes a processing circuit 100, a storage 104, a sensor unit 112, a display 109, an input device 110, and peripheral devices 111. The user device 20 illustrated in FIG. 2 includes a sensor unit 112 that detects the movement of the broadcast user 10A. Note that the sensor unit 112 may be a separate device from the user device 20 and may be communicably connected to the user device 20.

The processing circuit 100 includes a CPU 101, which is a processor that executes one or more of the control processes disclosed in this specification, and a memory 102. Processing data and instructions may be stored in the memory 102. In addition, these processing data and instructions may be stored in a storage 104, which is a storage medium disk such as a hard drive (HDD) or a portable storage medium, or may be stored in a storage medium provided separately from the processing circuit 100. The storage 104 is connected to the CPU 101 via a storage controller 103 and a bus 108. In this embodiment, an application is stored in the storage 104. The application includes a program for distributing and viewing videos, and various data necessary for executing the program.

Each function disclosed herein may be implemented using circuitry that may include, but is not limited to, a CPU 101, general-purpose processors, special-purpose processors, integrated circuits, ASICs (Application Specific Integrated Circuits), other conventional circuits, and/or combinations thereof, configured or programmed to perform the disclosed functions. A processor is a processing circuit or circuitry because it includes transistors and other circuits therein. A processor may be a programmed processor that executes a program stored in a memory. In this disclosure, a unit such as a processing circuit, or means, is hardware that performs or is programmed to perform the functions mentioned. The hardware may be any hardware disclosed herein or otherwise known that is programmed or configured to perform the functions mentioned.

Furthermore, the processor is not limited by the form of computer-readable medium on which the instructions of the process are stored. For example, the instructions may be stored on a CD, DVD, FLASH memory, RAM, ROM, PROM, EPROM, EEPROM, hard disk or other non-transitory computer-readable medium of an information processing device with which the processing circuit 100 communicates, such as a server or computer. The process may also be stored in network-based storage, cloud-based storage, or other mobile-accessible storage and executable by the processing circuit 100.

The descriptions or blocks in the flowcharts disclosed herein may be understood to represent modules, segments, or portions of code that include one or more executable instructions for implementing a particular logical function or step in a process. In addition, the descriptions or blocks in the flowcharts disclosed herein may represent two or more functions or steps that may be performed in a different order than that shown or described in the specification, such as substantially simultaneously or in reverse order.

The hardware elements for implementing the processing circuit 100 may be implemented by various circuit elements. Furthermore, each function disclosed in this specification may be implemented by a circuit including one or more processing circuits.

The processing circuit 100 also includes a network controller 106 for connecting to the network NW. The network NW may be a public network such as the Internet, or a private network such as a local area network (LAN) or a wide area network (WAN), or any combination thereof, and may also include a public switched telephone network (PSTN) or an integrated services digital network (ISDN, registered trademark) or a sub-network. The network NW may also be a wired network such as an Ethernet network, a universal serial bus (USB) cable, or a wireless network such as a cellular network including 3G, 4G and 5G wireless cellular systems. The wireless network may also be Wi-Fi, a wireless LAN, Bluetooth, or other known forms of wireless communication. Furthermore, the network controller 106 may be compliant with other direct communication standards such as Bluetooth, near field communication (NFC), infrared, etc.

The processing circuit 100 further includes a display controller 105 and an input/output interface 107. The display controller 105, the network controller 106, and the input/output interface 107 are connected to a bus 108. The display controller 105 is connected to a display 109. The input/output interface 107 is connected to an input device 110 and a peripheral device 111.

The sensor unit 112 will be described. The sensor unit 112 is one or more sensors that detect data indicating the movement of the user 10. The movement of the user 10 includes face motion indicating a change in the user's facial expression and body motion indicating a change in the relative position of the user's body with respect to the sensor unit 112. Face motion includes movements such as blinking and opening and closing of the mouth. A publicly known sensor unit can be used as the sensor unit 112. The sensor unit 112 has a sensor that measures the distance to the measurement object by irradiating the measurement object with light. As such a sensor, for example, a True Depth sensor or a "LIDAR" (Light Detection and Ranging, or Laser Imaging Detection and Ranging) sensor can be used. For example, the light-emitting section of the sensor unit 112 projects a dot pattern consisting of tens of thousands of invisible dots (points) onto the user's face, etc., using a dot projector. The sensor unit 112 detects and analyzes the reflected light of the dot pattern to form a depth map of the face and capture an infrared image of the face, etc. This allows the sensor unit 112 to capture accurate face data. The calculation processing unit of the sensor unit 112 generates various information based on the depth map and the infrared image, and compares the information with registered reference data to calculate the depth of each point on the face (the distance between each point and the near-infrared camera) and positional deviations other than in the depth direction. In another example, the sensor unit 112 includes a ToF sensor that measures the time of flight (time of flight) from when light is irradiated toward a measurement target such as a user's face until it is reflected back, or the phase difference between the irradiated light and the reflected light, using a distance measurement method other than the above-mentioned method. Alternatively or in addition to this, the sensor unit 112 may include a camera that captures the user's face and an image processing unit that performs image processing on the data captured by the camera. The ToF sensor calculates the distance to the measurement target based on the time of flight or phase difference.

The sensor unit 112 outputs the facial motion and body motion to the processing circuit 100 as tracking data. The tracking data is included in the motion data. Motion data refers to data in general for moving an avatar object corresponding to the user 10. For example, the motion data includes emote data in addition to tracking data. Emo data is data for making the avatar object perform a pre-registered specific movement such as "clap". Below, this pre-registered movement of the avatar object is referred to as an "emote".

The sensor unit 112 may also have a function of tracking not only the user's face but also their hands (hand tracking). The sensor unit 112 may also include a sensor that detects the position or orientation of a human body part other than the hand. The sensor unit 112 may further include a sensor that detects speed or acceleration, a sensor that detects direction or orientation (such as a gyro sensor), etc. The sensor unit 112 may have a spatial mapping function that recognizes objects in the real space where the user exists based on the detection results of the various sensors described above, and maps the recognized objects onto a spatial map, i.e., creates a map of the positions of the recognized objects in the real space.

The input device 110 may be a touch panel, a keyboard, a mouse, a controller operated by the user's hands (e.g., a game controller such as a game pad or joystick), or an operation button provided on the housing of the user device 20. The controller may have various known sensors built in, such as an acceleration sensor, an inertial measurement sensor (IMU: Inertial Measurement Unit) such as a gyroscope, etc. In another example, the input device 110 may be a tracking device that identifies the user's hand movements, eye movements, head movements, gaze direction, etc. In this aspect, for example, the user's instructions can be determined based on the user's hand movements, and various operations can be performed, such as starting or ending video distribution, rating messages and videos, and displaying specified objects.

The peripheral device 111 includes a microphone that collects sounds in the real world, such as the voice of the user 10, and a speaker that outputs sound effects and the voices of other users 10, etc.
A program included in the application stored in the storage 104 will be described. The program includes a native part implemented in the user device 20. The native part runs on an OS (Operating System) of the user device 20, and can access devices (physical resources) such as the storage 104 and a camera of the user device 20. The native part displays a video on the display 109 based on video data received from the management server 12.

In this embodiment, the program includes an in-app browser. The in-app browser is a component, such as WebView, that is incorporated into a program to realize a browser function. The WebView is, for example, an API (Application Programming Interface) such as SFSafariView or WKWebView. The in-app browser includes an analysis unit and an output control unit. The in-app browser acquires resources. The resources are, for example, web data. The web server 13 provides resources associated with the virtual space to the user device 20. The in-app browser is capable of acquiring resources from the web server 13 via the network. The web data is data necessary for displaying a browser screen on the display 109. The web data may include, for example, data written in HTML (Hyper Text Markup Language), data written in JavaScript (registered trademark) or the like, a CSS (Cascading Style Sheets) file, etc. Like an independent web browser, the in-app browser's analysis unit interprets HTML, CSS, etc., and the in-app browser's output control unit executes JavaScript (registered trademark).

In another aspect, the program may launch an independent web browser (i.e., a web browser installed on the user device 20 separately from the in-app browser of the application) when a predetermined user operation is received. In this embodiment, the program is described as having an in-app browser.

The management server 12 has a processing circuit 100 and storage 104 similar to those of the user device 20. The management server 12 can omit the input device 110, the peripheral device 111, the sensor unit 112, the display 109, etc.

<User Device>
The functions of the user device 20 will be described with reference to Fig. 3. The user device 20 functions as an input coordinate acquisition unit 21, an operation determination unit 22, a first output control unit 23, and a second output control unit 24 by the processing circuit 100 executing a program recorded in the storage 104.

The first output control unit 23 outputs a virtual space image of the virtual space viewed from the virtual camera to a display 109 which is visually recognized by the user.
The second output control unit 24 outputs a projection screen that can be operated by the user within the virtual space image. The resources obtained from the web server 13 are displayed on the projection screen.

The input coordinate acquisition unit 21 acquires input coordinates, which are the coordinates of an input made to the input device 110 by a user. When the input device 110 is a touch panel, the input coordinate acquisition unit 21 acquires a series of input coordinates corresponding to the trajectory of a touch operation on the touch panel. When the input device 110 is a mouse, the input coordinate acquisition unit 21 acquires a series of input coordinates corresponding to the trajectory of a drag of the mouse and input coordinates corresponding to a click.

The operation determination unit 22 determines whether or not the input operation by the user 10 is an operation on an input area corresponding to the projection screen, based on the acquired input coordinates. The input area is a predetermined area within the virtual space screen displayed on the display 109.
When the operation determination unit 22 determines that the input operation is not an operation on the input area, the first output control unit 23 performs a first display control process regarding the virtual space.

When the operation determination unit 22 determines that the input operation is an operation on the input area, the second output control unit 24 performs a second display control process on the projection screen according to the acquired coordinates.
Furthermore, when the operation determination unit 22 determines that the acquired series of coordinates changes continuously along the first axis, i.e., that the input operation corresponds to a first scroll operation, the second output control unit 24 scrolls the projection screen in the direction of the first scroll operation.

If the operation determination unit 22 determines that the input operation is a combination of a predetermined designation operation and a second scroll operation, the second output control unit 24 scrolls the projection screen in the direction of the second scroll operation. The second scroll operation is an input operation in which the acquired series of coordinates changes continuously along the second axis.

When the operation determination unit 22 determines that the input operation includes only the second scroll operation, the second output control unit 24 performs a first display control process.
When the operation determination unit 22 determines that the input operation is a selection operation for selecting an arbitrary position on the projection screen, the second output control unit 24 executes a process according to the position of the selection operation.

<Management Server>
The management server 12 will be described with reference to Fig. 4. The management server 12 functions as a distribution management unit 30, a browsing history information receiving unit 32, and a performance recording unit 33 by the processing circuit 100 executing a program recorded in the storage 104. The management server 12 also includes a user management information storage unit 34, a browsing tally information storage unit 36, and a performance information storage unit 37.

The distribution management unit 30 transmits video data to the user device 20 for outputting a virtual space screen including a virtual space image to the display 109. The distribution management unit 30 transmits asset data to the user device 20 for drawing objects in the virtual space. The distribution management unit 30 receives data for drawing avatar objects, including motion data, from the distribution user device 20A, and transmits the received data to other user devices 20.

The browsing history information receiving unit 32 receives browsing history information from the user device 20, which indicates that the user 10 has viewed a resource associated with the virtual space. The browsing history information receiving unit 32 associates the browsing history information with the user 10 and records it in the browsing summary information storage unit 36.

The result recording unit 33 calculates the number of times each resource has been viewed based on the viewing history information, and records the number of times each resource has been viewed in the result information storage unit 37 as the result information of the resource.
<Data structure>
Each piece of information recorded in the management server 12 will be described with reference to FIGS.

5 shows an example of user management information 40 recorded in the user management information storage unit 34. The user management information 40 is recorded for each user. The user management information 40 includes a user ID (identifier), which is user identification information, accumulated points, and coins. The accumulated points are points that are awarded for activities on an application or a platform that provides the application, such as distribution and viewing of videos, and are accumulated for each user 10. The accumulated points can be used as a consideration for a lottery (gacha) that can be executed on the platform, a consideration for a gift object, or a consideration for an item. A gift object is an object that a viewing user 10B provides to a distribution user 10A, etc. Items include items that are worn by an avatar object and items used in a game provided by the platform. The accumulated points may also be exchangeable for currency used in the real world. Coins differ from the accumulated points in that the user 10 can purchase them. Coins may be awarded according to the activity (distribution status or viewing status) of the user 10. The coins may also be used as a consideration for a lottery, a consideration for a gift object, etc., like the accumulated points.

FIG. 6 shows an example of the viewing summary information 42 recorded in the viewing summary information storage unit 36. In this embodiment, the viewing summary information 42 is updated based on the viewing history information transmitted from the user device 20. The viewing summary information 42 includes a user ID, a URL (Uniform Resource Locator), a room ID, a viewing start time, a viewing end time, and a behavior history. The viewing start time is the time when the user device 20 connects to the web server 13 or the time when the user device 20 displays a resource associated with the virtual space. The viewing end time is the time when the user device 20 disconnects from the web server 13 or the time when the user device 20 ends displaying the resource. The behavior history indicates the behavior performed by the user 10 on the displayed resource. Examples of the behavior history include requesting materials on a resource, purchasing a product or the like on a resource, etc.

FIG. 7 shows an example of performance information 43 recorded in performance information storage unit 37. Performance information 43 is recorded for each publisher. Publishers publish resources. Multiple resources may be associated with one publisher. Publishers can advertise products and services to users through resources displayed in the virtual space.

The performance information 43 includes the publisher identifier, URL, user stay time, questionnaire response information, and number of views. The publisher identifier is an identifier assigned to the publisher. The user stay (view) time indicates the time each user 10 viewed the resource, or statistical information on those viewing times. This time can be calculated from the viewing start time and viewing end time in the viewing summary information 42. The user stay time may be recorded as the time that the user viewed the resource individually. Alternatively, the user stay time may be the average stay time of the user, that is, the average value of those individually recorded times. The questionnaire response information indicates the content of the questionnaire that the user 10 answered in the virtual space, that is, the user 10's response to the questionnaire in the virtual space. The number of views is the number obtained by aggregating the viewing history information and is the total number of times the resource was displayed.

<Video distribution and viewing method>
The management server 12 receives a video distribution request from the user device 20 of the user 10. At this time, the user 10 can specify a virtual space. When the management server 12 receives the distribution request and identification information of the specified virtual space, it generates a room (distribution slot) for the virtual space specified by the user 10. In addition, the management server 12 transmits data (asset data) for displaying the virtual space to the user device 20.

The distribution slots in which videos are being distributed are displayed on a list screen displayed on each user device 20. The viewing user 10B can start watching the video by selecting the distribution slot they wish to view from the list screen.

The video can be delivered and viewed using one of the following methods: client rendering, browser rendering, video delivery, and server delivery. In this embodiment, the client rendering method is used.

In the client rendering method, each user device 20 renders objects by executing a program. The user device 20 stores three-dimensional model data of avatar objects and objects other than avatar objects (hereinafter referred to as physical objects). The three-dimensional model data of avatar objects includes video data of the main body parts that make up the avatar object itself and the attachable parts that can be attached to the avatar object. The data of the main body parts includes polygon data, skeletal data (bones) for expressing the movement of the avatar object, texture data, etc. The attached parts include texture data, etc. The broadcasting user 10A can set the attached parts of the avatar object corresponding to him/her according to his/her preferences. The model data of physical objects other than avatar objects includes polygon data, texture data, etc.

The first output control unit 23 performs drawing using video data including motion data and identification information (parts IDs) indicating parts of each object. The first output control unit 23 also transmits video data including motion data and coordinates of the avatar object to the management server 12. The first output control unit 23 also transmits part IDs indicating each part of the avatar object to the management server 12 at a predetermined timing (e.g., when distribution starts). The management server 12 transmits the video data to the viewing user device 20B, etc. The viewing user device 20B receives the data from the management server 12. The viewing user device 20B then uses the received data to draw images of the avatar object and physical object of the broadcasting user 10A.

The first output control unit 23 also transmits audio data based on the sound picked up by the microphone to the management server 12. The management server 12 transmits the audio data to the viewing user device 20B. A timestamp is attached to the motion data, audio data, etc.

The first output control unit 23 of the viewing user device 20B receives video data and audio data for displaying the video from the management server 12. The first output control unit 23 then uses the video data to draw an image including an avatar object corresponding to the broadcast user 10A.

The first output control unit 23 of the broadcasting user device 20A and the viewing user device 20B also receives data such as messages posted by other users from the management server 12. The first output control unit 23 generates video data by combining rendered images of avatar objects and other objects with posted messages, notifications, etc., and outputs the generated video data to the display 109. The first output control unit 23 also synchronizes audio data with the video data based on a timestamp and outputs it from the speaker.

In this embodiment, each user device 20 generates video using a client rendering method. Alternatively, the broadcasting user device 20A and the viewing user device 20B may display video using different methods. For example, the broadcasting user device 20A may display video using a client rendering method, while the viewing user device 20B may display video using a browser rendering method. In addition, the user may be able to select the method for displaying video. For example, some of the viewing user devices 20B may display video using a client rendering method, while other devices may display video using a browser rendering method.

<Virtual space>
A virtual space 50 in this embodiment will be described with reference to Fig. 8. The virtual space 50 includes a plurality of objects 51. The objects 51 located in the virtual space 50 include an avatar object 52 and other physical objects 53.

In the virtual space 50, a world coordinate system 90 is set, which indicates a position in the virtual space. The world coordinate system is a coordinate system having an X-axis, a Y-axis, and a Z-axis. The world coordinate system 90 is also called a global coordinate system. The position of the object 51 can be specified by the coordinates of the world coordinate system 90. The coordinates specified in the world coordinate system 90 include coordinates (X, Y, Z) expressed in a Cartesian coordinate system. These coordinates may also include rotation coordinates (roll, pitch, yaw) for each coordinate axis. Each object 51 has its own object coordinate system 91 set. The object coordinate system 91 is a coordinate system having an X-axis, a Y-axis, and a Z-axis. The orientation, movement, etc. of the avatar object 52 can be specified by the coordinates of the Cartesian coordinate system or the rotation coordinates.

In the virtual space 50, a virtual camera 54 associated with each broadcast user 10A is positioned. A camera coordinate system 92 is set for the virtual camera 54. For example, the camera coordinate system 92 includes coordinates (X, Y, Z) specified in a three-dimensional Cartesian coordinate system, as well as rotational coordinates (Xθ, Yθ, Zθ) centered on each coordinate axis.

The first output control unit 23 performs rendering based on the coordinates of objects included in a predetermined field of view (for example, a field of view cone having an approximately pyramidal shape) centered on the virtual optical axis of the virtual camera 54. The first output control unit 23 then outputs a two-dimensional image generated by rendering to the display 109. The rendering here refers to a drawing process that includes obtaining the position of the virtual camera 54, perspective projection, and hidden surface removal based on depth information associated with each object. The rendering may be at least one of these processes, or may further include processes such as shading and texture mapping.

The position of the virtual camera 54 can be changed by at least one of the broadcasting user 10A and the viewing user 10B. The broadcasting user 10A can select a first-person perspective (avatar position) or a third-person perspective as a setting for the position of the virtual camera 54. In the first-person perspective, the first output control unit 23 performs rendering with the position of the avatar's eyes coinciding or nearly coinciding with the position of the virtual camera 54. In this case, the line of sight extending from the front of the avatar object 52 nearly coincides with the optical axis of the virtual camera 54. As a result, an image is displayed on the display 109 as if the virtual space 50 is being viewed from the avatar object 52, i.e., through the eyes of the avatar. In the third-person perspective, the first output control unit 23 sets the virtual camera 54 at a position around the avatar object 52. When the virtual camera 54 is set in the third-person perspective, an image of the avatar object 52 can be displayed on the screen. The first output control unit 23 can change the position of the virtual camera 54 in the third-person perspective. Alternatively, an image may be generated using a multi-view camera in which one (or more) of virtual cameras 54 positioned at three or more positions, including a first-person viewpoint position, a third-person viewpoint position, and one or more other positions, is selected.

The camera coordinate system 92 determines the camera orientation and operations such as panning and tilting. In the case of a third-person perspective, the first output control unit 23 changes at least one of the position and direction of the virtual camera 54 based on the operation of the user 10. In the case of a first-person perspective, the first output control unit 23 changes at least one of the position and direction of the virtual camera 54 based on the operation of the user 10 to move the avatar object 52.

In this embodiment, in the case of a third-person perspective, the virtual camera 54 may be associated with the avatar object 52 so as to maintain a positional relationship with respect to the avatar object 52. For example, the first output control unit 23 causes the virtual camera 54 to follow the movement of the avatar object 52 during a period in which no camera operation is being performed. For example, the first output control unit 23 acquires the movement direction indicated by the distribution user 10A as a first direction, converts the first direction into a second direction as a movement direction in the world coordinate system 90, and moves the avatar object 52 in the second direction. In addition, the first output control unit 23 moves the virtual camera 54 in the second direction while keeping the relative distance between the avatar object 52 and the virtual camera 54 constant. This prevents the avatar object 52 from moving away from the virtual camera 54, thereby preventing the avatar object 52 from leaving the screen or being displayed so small that it is difficult to see. Even when the virtual camera 54 is made to follow the avatar object 52, the user 10 can change at least one of the position and direction of the virtual camera 54 by an input operation. In this embodiment, the first output control unit 23 changes the position of the virtual camera 54 based on a slide operation by the user 10.

A browser screen 57 is output as a projection screen in the virtual space 50. The browser screen 57 is a screen projected onto a display object 58. The display object 58 is an object that has coordinates in the world coordinate system in the virtual space 50. Only one display object 58 may be provided in the virtual space 50, or multiple display objects 58 may be provided.

For example, when an exhibition is held in the virtual space 50, a display object 58 may be provided for each of a number of exhibition booths. The display object 58 is an object for displaying resources made public by the exhibition booth owner. The display object 58 is, for example, a two-dimensional object with no thickness, or a three-dimensional object with a display surface. Note that an exhibition object 51 other than the display object 58 may be set as an exhibit in the exhibition booth.

The second output control unit 24 reads web data identified by a specific URL using an in-app browser, and outputs a browser screen 57 based on the web data to a display object 58 using a technique such as texture mapping. The browser screen 57 may include images such as still images in JPEG, GIF, PNG, etc., and videos based on video data such as MPEG4. The browser screen 57 is displayed in the same manner as if it were displayed by a web browser separate from the in-app browser of the application. Note that details of the browser screen 57 are omitted in Figure 8.

The first output control unit 23 converts coordinates P1 in the display area 97 displayed on the screen output to the display 109 into coordinates P2 on the browser screen 57 (see arrow 95 in Figure 8). An example of this coordinate conversion process will be described. An input coordinate system 93 is set in the virtual space 50. The input coordinate system 93 is a two-dimensional Cartesian coordinate system that specifies coordinates on the screen of the display 109. The Y axis of the input coordinate system 93 corresponds to the first axis. The X axis corresponds to the second axis.

In the virtual space 50, a UI area 55 and a browser area 56 are set. The browser area 56 corresponds to the input area. The coordinates on these areas are specified by the input coordinate system 93. The first output control unit 23 or the second output control unit 24 detects the position and direction of the input operation by acquiring the coordinates of the input operation by the user 10 for these areas. In addition, the first output control unit 23 or the second output control unit 24 determines the content of the user instruction corresponding to the input operation according to at least one of the position and direction of the input operation, and performs control according to the user instruction.

The UI area 55 is an area (layer) that accepts operations on GUI (Graphical User Interface) elements such as buttons, text boxes, and icons. FIG. 8 shows UI areas 55A to 55D as the UI area 55. Although the UI area 55 itself is hidden, GUI elements (not shown in FIG. 8) are displayed within the UI area 55. That is, while the GUI elements within the UI area 55 are displayed on the virtual space screen, the UI area 55 itself is not displayed. In the example of FIG. 8, a GUI element such as a setting button is displayed within the UI area 55A. In the UI areas 55B and 55C, GUI elements such as a button for moving the avatar object 52 and a button for jumping are displayed, respectively. In the UI area 55D, a GUI element such as a text box in which a message can be input is displayed. Note that the UI areas 55A to 55D in FIG. 8 are simplified views of the actual screens.

The browser area 56 is an area (layer) that accepts input operations for the browser screen 57. The coordinates of the browser area 56 can be specified by the input coordinate system 93. The first output control unit 23 sets the browser area 56 when the display conditions for the browser screen 57 are satisfied. At this time, the first output control unit 23 sets the browser area 56 so that the browser area 56 overlaps with the browser screen 57 when viewed from the virtual camera 54.

Furthermore, when at least one of the position and orientation of the virtual camera 54 is changed, the first output control unit 23 sets the browser area 56 to match the shape and size of the browser screen 57 as seen by the virtual camera 54. For example, when the browser screen 57 becomes larger on the screen as a result of the virtual camera 54 approaching the display object 58 in response to the operation of the user 10, the first output control unit 23 enlarges the browser area 56 to match the size of the browser screen 57. Similarly, when the browser screen 57 becomes smaller on the screen, the first output control unit 23 reduces the browser area 56 to match the size of the browser screen 57. Furthermore, when the relative angle between the virtual camera 54 and the browser screen 57 changes in response to the operation of the user 10, the first output control unit 23 deforms the browser area 56 to match the shape of the browser screen 57 on the screen.

The operation determination unit 22 identifies the coordinates of the input to the browser area 56. In other words, the operation determination unit 22 identifies an input trajectory that corresponds to the trajectory of continuous input in the browser area 56. The operation determination unit 22 also converts the identified coordinates into coordinates on the browser screen 57. The coordinates on the browser screen 57 are identified in the browser screen coordinate system 94.

The second output control unit 24 receives a request for display control based on input coordinates, which are coordinates in the browser screen coordinate system 94, from the operation determination unit 22. The second output control unit 24 performs second output control processing based on user instructions for controlling the browser screen 57. For example, when a series of input coordinates on the browser screen 57 are consecutive coordinates along the Y axis (positive Y direction and negative Y direction) of the browser screen coordinate system 94, the second output control unit 24 determines that an input operation for scrolling the browser screen 57 has been performed. Then, the second output control unit 24 scrolls the browser screen 57 in the same direction as the direction of the input operation.

When the operation determination unit 22 acquires coordinates in the input coordinate system 93 other than the UI area 55 and the browser area 56 as input coordinates, it requests the first output control unit 23 to perform display control based on the acquired coordinates. The first output control unit 23 performs display control processing related to the virtual space 50.

<Input operation>
Next, referring to FIG. 9, an input operation by the user 10 and display control by the first output control unit 23 or the second output control unit 24 in response to the input operation will be described. Here, a case will be described in which the input device 110 is a touch panel. An input operation on the touch panel is performed by contact with a finger or a stylus or the like. An input operation on the browser area 56 only needs to be executable by at least one of the broadcasting user 10A and the viewing user 10B. In this embodiment, when the user device 20 is in a broadcasting mode for broadcasting a video,
The operation determination unit 22 of that user device 20 accepts an input operation into the browser area 56. Furthermore, when the user device 20 is in a viewing mode for watching videos, the operation determination unit 22 of that user device 20 does not accept an input operation into the browser area 56. In other words, only the broadcast user 10A can perform an input operation into the browser area 56.

9(a) to (e) each show a virtual space screen 60 including a browser screen 57. The virtual space screen 60 includes a virtual space image 62 including other objects 51 (not shown in FIG. 9), and GUI elements (not shown in FIG. 9). Hereinafter, for the virtual space screen 60, the direction parallel to the X axis of the input coordinate system 93 (positive X direction and negative X direction) is referred to as the horizontal direction. The positive X direction is the direction in which the X coordinate increases, and the negative X direction is the direction in which the X coordinate decreases. Additionally, the direction parallel to the Y axis of the input coordinate system 93 (positive Y direction and negative Y direction) and perpendicular to the X axis is referred to as the vertical direction. The positive Y direction is the direction in which the Y coordinate increases, and the negative Y direction is the direction in which the Y coordinate decreases.

The first output control unit 23 determines the moving direction of the position of the virtual camera 54 based on a predetermined condition. The determination of the direction indicated by the user 10 may be performed using a known method, and is not particularly limited. For example, when the user 10 performs an input operation that is diagonal to the X-axis and Y-axis, the first output control unit 23 compares the amount of change in the X-coordinate and the amount of change in the Y-coordinate of the input trajectory (vector). The first output control unit 23 then determines that the input operation is an input operation along the one of the two coordinate axes with the greater amount of change. For example, if the amount of change in the X-coordinate is greater than the amount of change in the Y-coordinate, the first output control unit 23 determines that the input operation is an input operation along the X-axis. Alternatively, the first output control unit 23 may determine the direction of the input operation based on the angle of the vector of the input trajectory.

As shown in FIG. 9(a), the user 10 can specify an input start position PA outside the UI area (not shown in FIG. 9) and outside the browser area 56 by a touch operation. At this time, the operation determination unit 22 acquires the input coordinates of the input start position PA.

9B, the user 10 may slide the touch position in the X-positive direction from the input start position PA to the input end position PB. The operation determination unit 22 acquires a series of input coordinates that change continuously from the input start position PA to the input end position PB in response to this sliding operation. In this case, the operation determination unit 22 requests the first output control unit 23 to perform display control based on the acquired input coordinates. The first output control unit 23 performs a first display control process. In the first display control process, the position of the virtual camera 54 is changed. More specifically, the first output control unit 23 moves the position of the virtual camera 54 in a direction from the input start position PA toward the input end position PB (the X-positive direction in FIG. 9). The position of the virtual camera 54 may be moved parallel to the X-axis of the camera coordinate system 92, or may be moved in an arc so as to maintain a constant relative distance to a predetermined object 51 (e.g., an avatar object 52).

In addition, when the first output control unit 23 acquires a series of input coordinates that continuously change in the positive Y direction (or negative Y direction) from the input start position PA, the position of the virtual camera 54 may be moved in the positive Y direction (or negative Y direction).
9A, the user 10 can specify an input start position PC in the browser area 56. At this time, the operation determination unit 22 acquires the input coordinates of the input start position PC.

9(c), the user 10 may slide the touch position in the negative Y direction from the input start position PC to the input end position PD. At this time, the operation determination unit 22 acquires a series of input coordinates that change continuously from the input start position PC to the input end position PD in response to this slide operation. In this case, the operation determination unit 22 requests the second output control unit 24 to perform display control based on the acquired input coordinates. The second output control unit 24 scrolls the browser screen 57 in the negative Y direction of the browser screen coordinate system 94 (the same direction as the negative Y direction of the input coordinate system 93). In other words, the second output control unit 24 displays a range that is not displayed on the browser screen 57 shown in FIG. 9(a) and is located above the range that was displayed before the slide operation started. Note that the first output control unit 23 may determine that the operation is a vertical scroll operation on the browser screen 57 even if the input end position PD is outside the browser area 56, as long as part of the input trajectory of the vertical slide operation is included in the browser area 56.

As shown in FIG. 9(d), after the user 10 performs a long press on the input start position PC, i.e., continues the touch operation on the input start position PC for a predetermined period of time, the user 10 may perform a slide operation in the positive X direction to the input end position PE without releasing the touch operation. In other words, the operation determination unit 22 may determine that the input operation is a combination of a designation operation, which is a long press operation, and a horizontal slide operation. At this time, the operation determination unit 22 requests the second output control unit 24 to perform display control based on the acquired input coordinates.

The second output control unit 24 determines whether the input start position PC is included in the horizontally scrollable display range 61. The horizontally scrollable display range is the range of the screen in which the display content can be switched by moving it left and right, and is also called a carousel. When the second output control unit 24 determines that the input start position PC is included in the horizontally scrollable display range 61, it scrolls that range of the browser screen 57 in the direction from the input start position PC toward the input end position PE.

As shown in FIG. 9(e), the user 10 may perform a slide operation in the positive X direction from the input start position PC to the input end position PF without performing a long press operation. At this time, the operation determination unit 22 acquires a series of input coordinates that change continuously from the input start position PC to the input end position PF in response to this slide operation. In this case, the operation determination unit 22 requests the first output control unit 23 to perform display control based on the acquired input coordinates.

The first output control unit 23 moves the position of the virtual camera 54 in the positive X direction. In other words, the user 10 can move the position of the virtual camera 54 by performing a horizontal sliding operation outside the UI area 55 and outside the browser area 56, or a sliding operation that does not include a long press operation within the browser area 56.

Furthermore, suppose that the user 10 taps a predetermined area of the browser screen 57 (for example, the input start position PC shown in FIG. 9(a)). In this case, the operation determination unit 22 outputs a determination result including the input coordinates to the second output control unit 24. The second output control unit 24 performs display control according to the input coordinates acquired from the operation determination unit 22. For example, when the input coordinates are included in a video display area, the second output control unit 24 plays a stopped video or stops a video being played. The video display area is an area where thumbnails (small-sized images) of videos are displayed. Alternatively, when the input coordinates are included in an area that accepts a link selection operation, the second output control unit 24 acquires a resource specified by the link from the web server 13. The second output control unit 24 then displays the acquired link on the browser screen 57.

Furthermore, suppose that the user 10 performs either a pinch in or pinch out operation (hereinafter, pinch operation) on the virtual space screen 60. In this case, the operation determination unit 22 determines that it is a pinch operation, and outputs the determination result including the input coordinates to the first output control unit 23. The first output control unit 23 reduces or enlarges the entire virtual space image 62, regardless of whether the acquired input coordinates are included in the browser area 56. In other words, even if the user 10 performs a pinch out operation within the browser area 56, the virtual space image 62 is enlarged. As a result, the browser screen 57 included in the virtual space 50 is also enlarged.

When the broadcast user device 20A performs display control in response to the input operation of the broadcast user 10A in this way, the broadcast user device 20A transmits input operation data indicating the content of the input operation to the viewing user device 20B via the management server 12. The input operation data includes an operation instruction for the browser screen 57, identification information of the display object 58 to be operated, the operation amount, the position of the virtual camera 54, etc. The operation instruction is vertical scrolling, horizontal scrolling, link selection, stopping or playing a video, etc. The operation amount is the operation amount of scrolling, etc. The viewing user device 20B performs display control of the browser screen 57 and controls the virtual camera 54 based on the input operation data. As a result, the screen output to the broadcast user device 20A and the screen output to the viewing user device 20B are synchronized. Alternatively, the broadcast user device 20A may transmit data indicating the display position of the browser screen 57 to the viewing user device 20B in order to synchronize the browser screens 57 between the broadcast user device 20A and the viewing user device 20B.

<Operation>
The processing of the user device 20 will be described with reference to Figs. 10 and 11. Here, it is assumed that the broadcast user device 20A displays a virtual space screen 60 including a browser screen 57 on the display 109. The display condition of the browser screen 57 is that the position of the avatar object 52 is within a predetermined area set for the display object 58, or that the broadcast user 10A has performed an operation to display the browser screen 57. When the second output control unit 24 determines that the display condition is satisfied, it requests a resource specified by the URL from the web server 13 based on the URL associated with the display object 58. When the second output control unit 24 receives the resource from the web server 13, it launches an in-app browser and outputs the browser screen 57 based on the resource to the display surface of the display object 58 in the virtual space 50.

The input coordinate acquisition unit 21 acquires input coordinates, which are the coordinates of an input made by the broadcast user 10A to the input device 110 (step S10). The operation determination unit 22 determines which area of the input coordinate system 93 the input coordinates are an operation for. For example, the operation determination unit 22 determines whether the acquired input coordinates are included in the UI area 55 (step S11). If the operation determination unit 22 determines that the input coordinates are included in the UI area 55 (step S11: YES), it requests the first output control unit 23 to perform display control processing corresponding to that UI area 55. The first output control unit 23 performs display control processing corresponding to that UI area 55 (step S12). For example, if the first output control unit 23 determines that a GUI element for moving the avatar object 52 has been operated, it moves the avatar object 52 in the direction indicated by the broadcast user 10A.

When the operation determination unit 22 determines that the input coordinates are not included within the UI area 55 (step S11: NO), it determines whether or not the input coordinates are included within the browser area 56 (step S13). When the operation determination unit 22 determines that the input coordinates are included within the browser area 56 (step S13: YES), it requests the second output control unit 24 to perform display control based on the input coordinates. The second output control unit 24 performs operation determination processing based on the input coordinates (step S14).

When the operation determination unit 22 determines that the input coordinates are not included in either the UI area 55 or the browser area 56 (step S13: NO), it requests the first output control unit 23 to perform display control based on the input coordinates. The first output control unit 23 performs a first display control process for the virtual space 50 (step S15). Here, the first display control process is the control of the virtual camera 54.

Next, the operation determination process (step S14) will be described. The second output control unit 24 converts the input coordinates into coordinates on the browser screen 57. The second output control unit 24 then determines the input operation that corresponds to the converted coordinates.

If the second output control unit 24 determines that the input operation is a pinch-in operation, it reduces the virtual space image 62. If the second output control unit 24 determines that the input operation is a pinch-out operation, it enlarges the virtual space image 62. If the second output control unit 24 determines that the input operation is a selection operation, it transitions the screen, or stops or plays the video, depending on the input coordinates.

If the converted coordinates change continuously, the second output control unit 24 determines whether the user instruction is for vertical scrolling, horizontal scrolling, or control of the virtual camera 54.

As shown in FIG. 11, the second output control unit 24 determines whether the slide operation is a vertical operation (step S20). If the second output control unit 24 determines that the slide operation is a vertical operation (step S20: YES), it scrolls the browser screen 57 vertically (step S21). If the second output control unit 24 determines that the slide operation is a horizontal operation rather than a vertical operation (step S20: YES), it determines whether the input operation includes a designation operation (step S22).

If the second output control unit 24 determines that the input operation includes a designation operation (step S22: YES), it scrolls the horizontally scrollable range of the browser screen 57 in the direction in which the converted coordinates change (step S23). If the second output control unit 24 determines that the input operation does not include a designation operation (step S22: NO), it controls the virtual camera 54 (camera control) (step S24).

Next, the process of collecting view tally information by the management server 12 will be described.
When the broadcast user device 20A outputs the browser screen 57, it transmits browsing history information to the management server 12. The browsing history information includes the broadcast user's 10A user ID, URL, room ID, and browsing start time. Furthermore, the broadcast user 10A may transmit browsing information including the input operation as a behavior history to the management server 12 each time an input operation is performed. Furthermore, when the broadcast user device 20A no longer outputs the browser screen 57, it transmits browsing history information including the viewing end time to the management server 12. The browsing history information receiving unit 32 uses the browsing history information to record the viewing tally information 42 in the viewing tally information storage unit 36.

The performance recording unit 33 generates performance information 43 using the viewing count information 42. At this time, the number of views may be calculated based on the number of viewing users 10B. After generating the performance information 43, the performance recording unit 33 records it in the performance information storage unit 37. The administrator of the management server 12 (e.g., the PF provider) can provide the performance information 43 to the publisher. The publisher can check the user response by checking the frequency with which the information they published was viewed and the responses to questionnaires, etc. For example, the administrator can receive compensation from the publisher according to the performance information.

<Screen display>
The virtual space screen 60 (distribution screen) displayed by the broadcasting user device 20A will be described with reference to Figures 12 to 15. Note that Figures 12 to 15 show a screen when the position of the virtual camera 54 is a third-person perspective. Also, the virtual space screen (viewing screen) displayed by the viewing user device 20B is almost the same as the virtual space screen 60 displayed by the broadcasting user device 20A, except for the difference in GUI elements. However, in this embodiment, the viewing user 10B cannot operate the browser screen 57 as described above.

As shown in FIG. 12, an avatar object 52 is displayed on the virtual space screen 60. For example, when the avatar object 52 moves to the vicinity of a display object 58 so that the display object 58 is included in the field of view of the virtual camera 54, the first output control unit 23 includes the display object 58 in the virtual space screen 60. When the first output control unit 23 determines that the display conditions are met, it outputs a browser screen 57. The browser screen 57 displays a screen of a website identified by a URL associated with the display object 58. For example, if an exhibition is being held in the virtual space 50, a screen of a website related to the contents of the exhibition is displayed. The website is a site that displays information about the exhibitor, details of a product or service, etc.

The virtual space screen 60 also displays a plurality of GUI elements 65 and icons. The virtual space screen 60 also displays a movement button 66 and a jump button 67, which are GUI elements. When the user 10 operates the movement button 66, the first output control unit 23 moves the avatar object 52. When the user 10 operates the jump button 67, the first output control unit 23 makes the avatar object 52 jump. This makes it possible to move the avatar object 52 to any exhibition booth, for example.

When the broadcast user 10A performs a vertical slide operation on the browser screen 57 (see arrow 96 in FIG. 12), the second output control unit 24 scrolls the browser screen 57 in the direction of the operation. On the website, the broadcast user 10A can perform actions similar to those performed on an independent browser. For example, the broadcast user 10A can view information about products, purchase products, request documents, etc.

In this way, when the broadcasting user 10A views information about a product or the like on the browser screen 57, the viewing user 10B who watches the broadcasted video will also see that information. For this reason, the publisher can advertise the product or the like by exhibiting in the virtual space 50 and having the broadcasting user 10A broadcast a video. Furthermore, if the broadcasting user 10A is able to operate the browser screen 57 within the virtual space 50, more information can be provided to the broadcasting user 10A and the viewing user 10B compared to when a non-operable image is displayed in the display object 58. For this reason, the publisher can further increase the advertising effect of the information they publish.

FIG. 13 shows a virtual space screen 60 in which the position of the virtual camera 54 has been moved. When the broadcasting user 10A performs an input operation to move the virtual camera 54, the first output control unit 23 changes the position 54A of the virtual camera 54 to position 54B based on the direction of the input operation. At this time, as illustrated in FIG. 13, the first output control unit 23 may move the virtual camera 54 a distance according to the amount of the input operation while keeping the virtual optical axis of the virtual camera 54 facing the avatar object 52. The first output control unit 23 outputs to the display 109 a virtual space image 62 included in the field of view Z1 of the virtual camera 54 at position 54B.

As shown in FIG. 14, when the broadcast user 10A performs a designation operation and a horizontal sliding operation successively in the horizontally scrollable display range 99 in the browser area 56, the second output control unit 24 scrolls the display range 99 horizontally (see arrow 98 in FIG. 14).

FIG. 15 shows the virtual space screen 60 when the broadcast user 10A pinches out in the browser area 56. The second output control unit 24 enlarges only the virtual space image 62 on the virtual space screen 60. As a result, the browser screen 57 is also enlarged and displayed.

The effects of the first embodiment will be described.
(1-1) A browser screen 57 that allows input operations by the user 10 is displayed on the virtual space screen 60. The user 10 can efficiently collect information that interests him or her while exploring the virtual space 50. In addition, the publisher can provide information to users 10 who tend to behave differently from customers in the real world as potential customers through the browser screen 57. In addition, the operation determination unit 22 distinguishes between operations for controlling the display of the browser screen 57 and operations for controlling the virtual camera 54 according to the input coordinates. Therefore, the user 10 can intuitively operate the browser screen 57 without performing complicated operations.

(1-2) If the operation determination unit 22 determines that the input coordinates are a scroll operation that changes continuously along the first axis (vertical direction), the second output control unit 24 scrolls the browser screen 57 in the direction of the operation. This allows the user 10 to scroll the browser screen 57 along the first axis with an intuitive operation.

(1-3) If the operation determination unit 22 determines that the input operation is a combination of a predetermined designation operation and a scroll operation in which the input coordinates change continuously along the second axis (horizontal direction), the second output control unit 24 scrolls the horizontally scrollable display area 97 of the browser screen 57 in the direction of the operation. Also, if the operation determination unit 22 determines that the input operation includes only a scroll operation in which the input coordinates change continuously along the second axis, the first output control unit 23 changes the position of the virtual camera 54 in the direction of the operation. This allows the user device 20 to distinguish between a slide operation that changes the position of the virtual camera 54 and an operation for scrolling along the second axis.

(1-4) If the operation determination unit 22 determines that the input operation is not an input operation within the UI area 55 or the browser area 56, the first output control unit 23 changes the position of the virtual camera 54 in the direction of the operation. This allows the user 10 to control the virtual camera 54 by intuitive operations.

(1-5) If the operation determination unit 22 determines that the input operation is a selection operation in the browser area 56, the second output control unit 24 executes processing according to the position of the selection operation. This allows the user 10 to control the virtual camera 54 with an intuitive operation.

(1-6) The first output control unit 23 outputs an avatar object 52 corresponding to the user 10 into the virtual space. As a result, through the avatar object 52, it is possible to stimulate at least one of the interactions between the broadcast user 10A and the viewing users 10B, and interactions between the broadcast users 10A.

(1-7) The second output control unit 24 accepts input operations into the browser area 56 in the distribution mode, but does not accept input operations into the browser area 56 in the viewing mode. This allows the viewing user 10B to view the browser screen 57 being viewed by the distribution user 10A on the viewing user device 20B, making it easier for them to interact with the distribution user 10A.

[Second embodiment]
The second embodiment of the program, the information processing method, the server, the information processing method of the server, and the information processing system will be described below with reference to FIG.

In the first embodiment, only the broadcast user 10A performs input operations in the browser area 56. In the second embodiment, in addition to the broadcast user 10A, a viewing user 10B who has authority can operate the browser screen 57. Also, in the second embodiment, in addition to each broadcast user device 20 performing broadcasting independently, multiple broadcast users 10A perform collaborative broadcasting in which multiple broadcast user devices 20 broadcast together. Even in collaborative broadcasting, each broadcast user 10A can independently perform input operations in the browser area 56. Hereinafter, parts similar to those in the first embodiment are given the same reference numerals, and detailed explanations thereof will be omitted.

Now, we will explain collaboration distribution. Collaboration distribution refers to multiple users distributing videos in the same virtual space 50. Hereinafter, the distribution user 10A who requests the creation of a room will be referred to as the host user, and the distribution user 10A who participates in the room created by the host user will be referred to as the guest user. There may be multiple guest users.

User 10 can participate in the collaboration distribution as a guest user by accepting the participation request from the host user, or by having the participation request sent by user 10 approved by the host user.

The broadcasting user device 20A used by the host user is referred to as the host user device 20H, and the broadcasting user device 20A used by the guest user is referred to as the guest user device 20G. The host user device 20H outputs a virtual space screen 60 seen from a virtual camera 54 set by the host user. The guest user device 20G outputs a virtual space screen 60 seen from a virtual camera 54 set by the guest user. In other words, the host user device 20H and the guest user device 20G broadcast within the same virtual space 50, but the display range of each virtual space screen 60 may differ. Alternatively, if the position of the virtual camera 54 is fixed, each broadcasting user 10A may output a virtual image of the same display range.

The second output control units 24 of the host user device 20H and the guest user device 20G each control the display of the browser screen 57 based on the input operation of the broadcast user 10A. The input operation of the browser screen 57 here includes operations that control the display state of the browser screen 57, such as operations for scrolling vertically, scrolling horizontally, and selection operations, as in the first embodiment. Note that when multiple display objects 58 are located within the virtual space 50, the host user and the guest user may each operate the browser screen 57 displayed on a different display object 58.

FIG. 16 shows the state in which data is transmitted and received between a broadcasting user device 20A and a viewing user 20B. Note that the management server 12 that relays the transmission and reception of data between the user devices 20 is not shown. Also, video data, gift display requests, message data, position information of the virtual camera 54, etc. transmitted between the user devices 20 are not shown.

One or more of the viewing users 10B have the authority to perform input operations on the browser screen 57, while the remaining viewing users 10B do not have the authority to perform input operations. Viewing users 10B who meet a certain condition have the authority to perform input operations. The certain condition may be, for example, that the viewing user 10B is associated with the broadcasting user 10A in advance (following or being a friend) before watching the video, that the viewing user 10B has made a request to the broadcasting user 10A to display a gift object (tipping), or that the viewing user device 20B of the viewing user 10B who meets the certain condition accepts input operations on the browser screen 57.

In this embodiment, the authority of distributor user 10A and viewing user 10B to operate browser screen 57, and reflection of the operated browser screen 57 on other devices are as follows.
The host user device 20H accepts input operations on the browser screen 57, and transmits data for reflecting the display state of the operated browser screen 57 on other devices.
The guest user device 20G accepts input operations on the browser screen 57, and transmits data for reflecting the display state of the operated browser screen 57 on other devices.
The viewing user device 20B of the authorized viewing user 10B accepts input operations on the browser screen 57, but does not transmit data for reflecting the display state of the operated browser screen 57 on another device.

When the host user device 20H receives an input operation on the browser screen 57 of the host user, it transmits input operation data D1 including an input operation instruction, identification information of the display object 58 to be operated, and a URL to the guest user device 20G and the viewing user device 20B. The input operation instruction includes a scroll direction and scroll amount. Based on the input operation data D1, the guest user device 20G and the viewing user device 20B output the browser screen 57 to the display object 58 to be operated by the host user. If the input operation data D1 does not include a URL, the guest user device 20G and the viewing user device 20B obtain a URL associated with the display object 58 to be displayed from the storage 104 or the management server 12 of their own device. Then, they obtain a resource identified by the URL from the web server 13. Based on the input operation data D1, the guest user device 20G and the viewing user device 20B scroll the browser screen 57. In other words, the guest user device 20G and the viewing user device 20B control the display of the browser screen 57 in response to the input operation of the host user. This causes the browser screens 57 to be synchronized between the users 10.

Similarly, when the guest user device 20G accepts an input operation on the guest user's browser screen 57, it transmits input operation data D1 to the host user device 20H and the viewing user device 20B. Based on the input operation data D1, the host user device 20H and the viewing user device 20B output the browser screen 57 to the display object 58 that is the target of the guest user's operation.

On the other hand, when the viewing user device 20B of the authorized viewing user 10B accepts an input operation on the viewing user's browser screen 57, it controls the display of the browser screen 57 based on the input operation. However, this viewing user device 20B does not transmit the input operation data D1 to the broadcasting user device 20A and other viewing user devices 20B. Furthermore, the viewing user device 20B of the authorized viewing user 10B accepts the input operation when it has not received the input operation data D1 from the broadcasting user device 20A. In other words, when this viewing user device 20B receives the input operation data D1 from the broadcasting user device 20A, it controls the browser screen 57 by prioritizing the received input operation data D1. When the viewing user 10B and the broadcasting user 10A operate the browser screens 57 output to different display objects 58, the display of each browser screen 57 is controlled by the input operation of the viewing user 10B and the input operation of the broadcasting user 10A, respectively.

As described above, the viewing user device 20B of the unauthorized viewing user 10B does not accept input operations on the viewing user's browser screen 57. This viewing user device 20B makes a resource transmission request D2 to the web server 13 based on the input operation data of each distribution user device 20A, and receives D3 the resources in response to the transmission request.

When one display object 58 is simultaneously displayed on both the display 109 of the host user device 20H and the guest user device 20G, and both the host user and the guest user perform input operations, the input operation that is received first may be given priority. That is, for example, if the host user device 20H receives an input operation by the host user immediately after receiving input operation data D1 transmitted from the guest user device 20G, the input operation by the guest user is given priority. If the host user device 20H receives input operation data transmitted from the guest user device 20G immediately after receiving an input operation by the host user, the input operation by the host user is given priority. The same applies to the guest user device 20G. When the input operation on one browser screen 57 is completed, the other distribution user 10A and viewing user 10B can also perform input operations.

Also, for example, when different display objects 58 are operated simultaneously by different broadcast users 10A, a browser screen 57 corresponding to each input operation is output to each display object 58.

FIG. 17 is an example of a virtual space screen 60 output to the display 109 of the host user device 20H in collaboration distribution. The virtual space screen 60 includes an avatar object 52A corresponding to the host user. If an avatar object 52B of a guest user is included in the field of view of the virtual camera 54, the avatar object 52B is drawn. The virtual space screen 60 also includes a mini-avatar 52C, which is a simplified avatar object 52 corresponding to an authorized viewing user 10B. The mini-avatar 52C has a smaller processing load for drawing than the avatar object 52 of the distribution user 10A. For example, the mini-avatar 52C is a two-dimensional image.

According to the second embodiment, in addition to the effects (1-1) to (1-7) of the first embodiment, the following effects can be obtained.
(2-1) In the second embodiment, in addition to the distribution user 10A who performs collaboration distribution, the authorized viewing user 10B can also perform input operations to control the browser screen 57. This makes it possible to provide the display contents of the browser screen 57 to a larger number of users, thereby improving the advertising effect.

The above-described embodiments may be modified as follows: The above-described embodiments and the following modifications may be combined with each other to the extent that no technical contradiction occurs.
<First output control unit>
In each of the above embodiments, the first output control unit 23 of the user device 20 distributes and views the video using the client rendering method. Alternatively, the video may be distributed and viewed using any of the above methods other than the client rendering method. The browser rendering method is a method in which the broadcasting user device 20A and the viewing user device 20B display the video using independent web browsers stored in the storage 104. At this time, the broadcasting user device 20A transmits video data, etc. to the management server 12. The user device 20 that displays the video receives web data from the management server 12 that is written in a markup language such as HTML (Hyper Text Markup Language) and uses CSS (Cascading Style Sheets), JavaScript (registered trademark), etc. The web browser program started on the user device 20 draws avatar objects and other objects using JavaScript running on the browser. In this method, model data of the avatar object, etc. is not stored in the user device 20. In the case of the browser rendering method, an application does not need to be installed on the user device 20. Alternatively, an application including an in-app browser may be installed on the user device 20 .

In the browser rendering method, at least one of the well-known methods such as static rendering (Static Server-Side Rendering), server-side rendering (Server-Side Rendering), client-side rendering (Client-Side Rendering), pre-rendering (CSR with Prerendering), and hydration rendering (SSR with (Re)Hydration) can be used as a method for generating a file to display the virtual space screen 60 by rendering. Rendering here refers to interpreting data written in a markup language or the like and calculating the arrangement of image elements and characters to be displayed on the virtual space screen 60.

In static rendering, the web server 13 stores a static HTML file in advance. The user device 20 downloads the static HTML file from the web server 13. The user device 20 outputs the rendered HTML file to the display 109 to display the virtual space screen 60. In server-side rendering (SSR), the web server 13 performs rendering etc. in response to a request from the user device 20, and transmits the rendered HTML file to the user device 20. The user device 20 outputs the downloaded HTML file to the display 109 to display the virtual space screen 60. The web server 13 includes an application server. In client-side rendering (CSR), the user device 20 performs rendering by executing JavaScript (registered trademark). At this time, the user device 20 downloads a simple HTML file from the web server 13, and also downloads the necessary JavaScript (registered trademark) and CSS files, etc. In pre-rendering, a type of CSR, the user device 20 generates and records some HTML files in advance, and downloads and renders the necessary content from the web server 13 in response to a request. In hydration rendering, the web server 13 generates and records HTML files that have not been rendered. The user device 20 downloads HTML files from the web server 13, and also downloads and renders the necessary data based on the HTML files. In each of the above methods, the user device 20 may obtain the necessary data from the management server 12, rather than from the web server 13.

The video distribution method is a method in which the broadcasting user device 20A generates video data. Specifically, the broadcasting user device 20A uses motion data and the like to draw objects and generate video data. The broadcasting user device 20A also transmits the encoded video data and audio data to the management server 12. The management server 12 transmits the video data and audio data to the viewing user device 20B. The viewing user device 20B displays video on the display 109 based on the video data received from the broadcasting user device 20A.

The server distribution method is a method in which the management server 12 generates video data. The management server 12 receives motion data and audio data from the broadcasting user device 20A. The management server 12 then generates output data for outputting the video based on that data. The management server 12 transmits the generated output data to the broadcasting user device 20A and the viewing user device 20B. The broadcasting user device 20A and the viewing user device 20B output the video to the display 109 based on the received output data.

In this way, either the user device 20 or the management server 12 is responsible for generating data for outputting a video. The data to be generated differs depending on the distribution method. The device responsible for generating the data can be changed depending on the distribution method.
In each of the above embodiments, the virtual space 50 in which an exhibition is held is exemplified, but the present invention is not limited to this virtual space 50. The virtual space 50 may be one in which a projection screen is displayed, and may be one in which events such as live performances and matches are held. Alternatively, the virtual space 50 may be a virtual store or an entertainment facility such as an amusement park. Alternatively, the virtual space 50 may not hold any events, and may simply allow users 10 to explore or communicate with each other. There may not be any objects corresponding to exhibits in the virtual space 50.

<First display control process>
In the above embodiment, the first display control process is a process of controlling the position of the virtual camera 54. Alternatively or in addition to this, the first display control process may be a process of controlling the direction of the virtual camera 54 without changing its position. For example, when the operation direction is the X-positive direction, the orientation of the virtual camera 54 is moved in the X-positive direction while maintaining the position of the virtual camera 54. The first display control process may also be a process of moving an object 51, such as the avatar object 52, in a direction indicated by the user 10.

<Second Output Control Unit>
In each of the above embodiments, the first output control unit 23 outputs the browser screen 57 as the projection screen. Instead of or in addition to this, the projection screen may be a screen on which information recorded in the storage 104 of the user device 20 or information acquired from an external server is output. This information is, for example, image data including still images and videos, an image list, a document file, etc. Or, this information is video data acquired from an external server and playable on the user device 20, call data for conversation through a video of another user, etc. Or, it may be the top page of an application that plays videos. Specifically, it is image data such as photos or videos recorded by the album function of the storage 104, books such as comics, data created by presentation software such as "Microsoft PowerPoint (registered trademark)", etc. The user 10 can vertically scroll, horizontally scroll, transition between screens, and play or stop videos on this projection screen as well, in the same way as the browser screen 57.

- When the operation determination unit 22 determines that the input operation includes a combination of a designation operation and a horizontal sliding operation, the second output control unit 24 scrolls the display area 97 of the browser screen 57 horizontally. Furthermore, when the operation determination unit 22 determines that the input operation does not include a designation operation and includes only a horizontal sliding operation, the second output control unit 24 controls the virtual camera 54. The second output control unit 24 may perform the opposite process depending on these input operations. In other words, when the input operation includes a designation operation and a horizontal sliding operation, the second output control unit 24 may control the virtual camera 54, and when the input operation does not include a designation operation and includes only a horizontal sliding operation, the second output control unit 24 may perform display control of the browser screen 57.

- When the operation determination unit 22 determines that the input operation includes a lock operation, the second output control unit 24 may determine that the input operation during the period between the input of the lock operation and the input of the unlock operation is an operation on the browser screen 57. For example, the second output control unit 24 determines that all coordinates input between a long press operation on the input device 110 and a long press operation for unlocking are operations on the browser screen 57. The second output control unit 24 performs processing on the browser screen 57 based on the input operation during the period from the lock operation to the input of the unlock operation for unlocking the lock operation. Therefore, the user 10 can scroll the display area 97 of the browser screen 57 horizontally by simply performing a horizontal slide operation in the browser area 56, without performing a designation operation for designating horizontal scrolling of the browser screen 57.

The second output control unit 24 synchronizes the projection screens such as the browser screen 57 by transmitting input operation data to the other user device 20. Alternatively, the second output control unit 24 may transmit data (image data) capturing the browser screen 57 to the other user device 20. The user device 20 that receives this data projects the received data onto the display object 58.

<Second display control process>
In each of the above embodiments, the second output control unit 24 performs, as the second display process, vertical scrolling, horizontal scrolling, link selection, and video playback or stopping of the browser screen 57 in response to an input operation of the user 10. The second output control unit 24 may perform at least one of these processes. Alternatively, the second output control unit 24 may perform a process other than the above-mentioned processes as long as the process is a process for a projection screen including the browser screen 57. For example, the second output control unit 24 may close or open the browser screen 57 in response to an input operation of the user 10. The second output control unit 24 may also operate a GUI element, such as selecting one of the items displayed in a pull-down menu, in response to an input operation of the user 10.

<Input coordinate acquisition section>
In each of the above embodiments, the input coordinate acquisition unit 21 acquires coordinates where a touch of a finger or a stylus is detected on a touch panel that is the input device 110. The input coordinate acquisition unit 21 acquires input coordinates according to the input device 110. When the input device 110 is a mouse or a controller, the input coordinate acquisition unit 21 acquires coordinates on the screen that the user 10 specifies using the mouse or controller. When the input device 110 includes a tracking sensor that detects the position of the pupil, the input coordinate acquisition unit 21 detects the direction of the user 10's gaze and detects the position on the screen that the user 10 specifies. When the input device 110 includes a tracking sensor that detects the position of a part of the user 10's body, such as a finger, the input coordinate acquisition unit 21 detects the direction in which the user 10 points the finger and detects the position on the screen that the user 10 specifies.

<Operation determination unit>
In each of the above embodiments, the operation determination unit 22 determines whether or not the input operation is an operation on the browser screen 57. Alternatively, the second output control unit 24 may determine whether or not the input operation is an operation on the browser screen 57 based on the input coordinates acquired from the operation determination unit 22.

<Management Server>
The management server 12 may be a single server device equipped with multiple virtual servers, or may be a set of multiple physical servers. For example, the management server 12 may be a set of a combination of a server having a distribution management unit 30, a server having a browsing history information receiving unit 32, and a server having a performance recording unit 33.

<Browser History Information Receiving Unit>
The browsing history information receiving unit 32 may receive browsing information from the web server 13. The web server 13 records an access log of a resource specified by a URL that the user 10 can register. The access log includes the connection date and time (browsing start time), the disconnection date and time (browsing end time), the identification information of the connected user device 20, the operation content, etc. The web server 13 transmits the access log to the management server 12. The browsing history information receiving unit 32 of the management server 12 records the received access log in the browsing tally information storage unit 36 as browsing tally information 42. The performance recording unit 33 records performance information 43 based on the access log in the performance information storage unit 37.

<Permission to operate the projection screen>
In the first embodiment, only the distribution user 10A can operate the projection screen. In the second embodiment, the distribution user 10A including the host user and the guest user and the viewing user 10B having authority can operate the projection screen, and only the operation of the distribution user 10A is reflected (synchronized) in the video of the other users 10. The attributes of the user 10 who can operate the projection screen and the attributes of the user whose input operation is synchronized with the other user devices 20 can be changed as appropriate. In other words, for each attribute of each user 10, whether or not the browser screen 57 can be operated is set, and if the browser screen 57 can be operated, whether or not the browser screen 57 can be synchronized with the other devices is set. Note that the browser screen 57 may be synchronized with all other user devices 20 other than the own device, or the browser screen 57 may be synchronized with only some of the other user devices 20.

For example, the attributes of these users 10 may be in the following forms:
[First aspect]
(A) The host user device 20H accepts an input operation on the browser screen 57, and transmits data for reflecting the operated browser screen 57 on another device.
(B) The guest user device 20G and the viewing user device 20B of the authorized viewing user 10B accept input operations on the browser screen 57, but do not transmit data for reflecting the operated browser screen 57 on another device.
[Second aspect]
(C) The host user device 20H accepts an input operation on the browser screen 57 and transmits data for reflecting the operated browser screen 57 on another device.
(D) The guest user device 20G and the viewing user device 20B of an authorized viewing user 10B accept input operations on the browser screen 57. When a synchronization instruction, which is an operation different from the input operation on the browser screen 57, is input to the user device 20 by these users 10, the user device 20 transmits data for reflecting the browser screen 57 on another device. The user device 20 does not transmit data for reflecting the browser screen 57 on another device unless a synchronization instruction is input. The synchronization instruction is input by operating a synchronization button or the like displayed on the virtual space screen 60.

The data to be reflected in the other device is data generated by the user device 20 or the management server 12, and it is sufficient that the user device 20 receiving this data can determine the operation contents of the user 10, who is the operator of the resource and the browser screen 57. For example, the data to be reflected in the other device may include a URL that identifies the resource. Furthermore, the data to be reflected in the other device may include data indicating the operation contents of the user 10, which is the first user device 20 used by the user 10, who is the operator. Upon detecting the operation of the user 10, the first user device 20 transmits data indicating the operation contents to the management server 12. The management server 12 generates an identifier as needed based on the data indicating the operation contents. The management server 12 then transmits the identifier to the other device. The second user device 20, which is the other device that receives the identifier, determines the operation contents based on the identifier and displays the browser screen 57 according to the operation contents.

If the input operations of the host user and the user 10 of the own device overlap, the input operation of the host user takes priority. Alternatively, the user device 20 may take priority over the input operation of the browser screen 57 of the user 10 of the own device.

In another embodiment, all users 10 may operate the projected screens, and synchronization of the projected screens may not be performed.
In addition, the browser screen 57 that the user 10 has started (displayed) or the browser screen 57 that the user 10 has operated once may be operated by another user 10. That is, in one aspect, only the host user may be able to both operate the projected screen and synchronize the operation, and the user 10 other than the host user may be able to operate the projected screen. For example, after the first user 10 starts the browser screen 57, if a predetermined condition is satisfied, the second user 10 may be able to operate the browser screen 57. In this aspect, the user device 20 of the second user 10 who has performed the operation halfway through transmits data for synchronizing the browser screen 57 to the other device if the second user 10 has the authority to synchronize the browser screen 57. If the second user 10 does not have the authority to synchronize the browser screen 57, the user device 20 of the second user 10 does not transmit data for synchronizing the browser screen 57 to the other device. The predetermined condition is that the host user is not performing the operation. Or, the predetermined condition is that the user device 20 has accepted the operation of the user 10.

<Avatar Object>
In each of the above embodiments, the user device 20 draws the avatar object 52 corresponding to the broadcast user 10A using tracking data or emote data that detects the movement of the broadcast user 10A. This allows the movement of the broadcast user 10A or the movement selected by the broadcast user 10A to be reflected in the avatar object 52. Alternatively or in addition to this, the user device 20 may draw the avatar object 52 temporarily or continuously without using tracking data and emote data. The first output control unit 23 controls the avatar object 52 based on the input operation of the user 10. The avatar object 52 performs an action in response to the input operation of the user 10. In this aspect, the load of the drawing process on the user device 20 can be reduced when drawing a large number of avatar objects 52 in the virtual space 50.

In each of the above embodiments, the user device 20 renders an avatar object 52 corresponding to the broadcast user 10A. Alternatively, the user device 20 may render an image of the virtual space 50 from a first-person perspective, and may not need to render the avatar object 52.

<Availability of video streaming>
In each of the above embodiments, it is assumed that the broadcast user device 20A broadcasts videos. Alternatively or in addition to this, the application implemented by the user device 20 may be an application that allows the user 10 to move around the virtual space 50 without broadcasting videos. In other words, the management server 12 does not perform processing for broadcasting videos, but provides asset data of the virtual space 50, etc., in response to a request from the user device 20. The user device 20 outputs a virtual space screen 60. However, if an avatar object 52 corresponding to another user 10 is located in the same virtual space 50, the avatar object 52 may be displayed on the virtual space screen 60. In this aspect, the user 10 launches an application and selects a desired virtual space 50. The user device 20 displays a virtual space screen of the virtual space 50 as seen from each virtual camera 54. In this aspect, the user 10 can operate the projection screen in the same manner as the broadcast user 10A in the first embodiment.

<User>
In the above embodiments, the broadcasting user 10A refers to a user who transmits at least one of information related to images (video) and information related to audio. For example, the broadcasting user 10A is a user who broadcasts videos alone (first embodiment), or a user who performs collaboration broadcasting in which multiple people can participate (second embodiment). Alternatively, the broadcasting user 10A may be a user who hosts or organizes a video chat or voice chat in which multiple people can participate and/or watch, or an event (party, etc.) in a virtual space in which multiple people can participate and/or watch. In other words, the broadcasting user 10A can also be called a host user, organizer user, or organizer user.

On the other hand, viewing user 10B refers to a user who receives at least one of video-related information and audio-related information. However, viewing user 10B not only receives the above information, but can also react to the information. For example, viewing user 10B is a user who watches a video broadcast or collaboration broadcast. Or viewing user 10B may be a user who participates in and/or watches video chat, voice chat, or an event. Therefore, viewing user 10B can also be referred to as a guest user, participating user, listener, viewing user, supporting user, etc.

<Information Processing System>
In each of the above embodiments, the management server 12 is managed by the PF provider, and the web server 13 is managed by the publisher. Alternatively or in addition to this, the user 10 may display an arbitrary web page on the browser screen 57, instead of a resource previously associated with the virtual space 50 as in each of the above embodiments. For example, when the user 10 finds a display object 58 in the virtual space 50, the user 10 may input a search word or the like in a predetermined area and execute a search, thereby outputting an arbitrary browser screen 57. The predetermined area is, for example, a text box displayed in the UI area 55.

In each of the above embodiments, the user device 20 is an information processing device such as a smartphone, a mobile phone, a tablet terminal, a personal computer, a console game machine, or a wearable computer such as a head-mounted display. Alternatively, the information processing system may be a system provided in a studio for video distribution. The information processing system has an information processing device, a server, a sensor unit worn on the body of the distribution user 10A, a tracking system that detects the position of the sensor unit, an operation unit, a speaker, a display, etc. The tracking system may be equipped with a multi-axis laser emitter that emits pulsed laser light for synchronization. The sensor unit has a sensor that detects laser light, and detects its own position and orientation while synchronizing with a synchronization pulse.

- By executing an application, the user device 20 functions as an input coordinate acquisition unit 21, an operation determination unit 22, a first output control unit 23, and a second output control unit 24. At least one of these functions may be executed by the management server 12 or another device in the information processing system 11. Furthermore, the management server 12 functions as a distribution management unit 30, a browsing history information receiving unit 32, and a performance recording unit 33 by executing a program. At least one of these functions may be executed by the user device 20 or another device in the information processing system 11. Furthermore, the management server 12 includes a user management information storage unit 34, a browsing aggregation information storage unit 36, and a performance information storage unit 37. At least one of these functions may be included in the user device 20 or another device in the information processing system 11.

Claims (12)

One or more computers,
a first output control unit that outputs a virtual space image of the three-dimensional virtual space viewed from a virtual camera to a display that is visually recognized by a user;
a second output control unit that outputs a projection screen that can be operated by the user within the virtual space image;
an input coordinate acquisition unit that acquires input coordinates to an input device of the user;
functioning as an operation determination unit that determines whether or not an input operation by the user is an operation on an input area corresponding to the projection screen, based on the acquired input coordinates;
when the operation determination unit determines that the input operation is not an operation on the input area, the first output control unit performs a first display control process regarding the three-dimensional virtual space other than a control for the projection screen;
A program in which, when the operation determination unit determines that the input operation is an operation on the input area, the second output control unit performs a second display control process on the projection screen according to the input coordinates. When the operation determination unit determines that the input operation is a scroll operation in which the input coordinates continuously change along a first axis,
The program according to claim 1 , wherein the second output control unit scrolls the projection screen in a direction of the scroll operation. When the operation determination unit determines that the input operation is a combination of a predetermined designation operation and a scroll operation in which the input coordinates continuously change along a second axis,
The program according to claim 1 , wherein the second output control unit scrolls the projection screen in a direction of the scroll operation. When the operation determination unit determines that the input operation includes only a scroll operation in which the input coordinates continuously change along a second axis,
The program according to claim 1 , wherein the first output control unit performs the first display control process. The program according to any one of claims 1 to 4, wherein the first output control unit changes at least one of the position and the direction of the virtual camera based on the input coordinates as the first display control process. The program according to any one of claims 1 to 5, wherein when the operation determination unit determines that the input operation is a selection operation for selecting an arbitrary position on the projection screen, the second output control unit executes processing according to the position of the selection operation. When the operation determination unit determines that the input operation in the input area includes a lock operation that specifies that the input operation is an operation on the projection screen,
The program according to any one of claims 1 to 6, wherein the second output control unit performs processing on the projection screen based on the input operation within a period between the input of the lock operation and the input of an unlock operation to unlock the lock operation. The program according to any one of claims 1 to 7, wherein the second output control unit outputs a browser screen displayed using a browser as the projection screen. the first output control unit outputs an avatar object corresponding to the user to the three-dimensional virtual space;
The program according to any one of claims 1 to 8, further comprising transmitting data for outputting a video including the avatar object and the projection screen to a user device used by another user. The program according to any one of claims 1 to 9, wherein the second output control unit accepts input operations into the input area when the user is in a distribution mode in which the user distributes a video, and does not accept input operations into the input area when the user is in a viewing mode in which a video is output based on video data transmitted by another user. One or more computers
a first output control process for outputting a virtual space image of the three-dimensional virtual space viewed from a virtual camera onto a display visually recognized by a user;
a second output control process for outputting a projection screen operable by the user within the virtual space image;
an input coordinate acquisition process for acquiring input coordinates to an input device of the user;
an operation determination process for determining whether or not an input operation by the user is an operation within an input area corresponding to the projection screen based on the acquired input coordinates;
performing a first display control process for the three-dimensional virtual space other than a control process for the projection screen when it is determined in the operation determination process that the input operation is not an operation on the input area;
an information processing method comprising: performing a second display control process on the projection screen according to the input coordinates when it is determined in the operation determination process that the input operation is an operation on the input area; a first output control unit that outputs a virtual space image of the three-dimensional virtual space viewed from a virtual camera to a display that is visually recognized by a user;
a second output control unit that outputs a projection screen that can be operated by the user within the virtual space image;
an input coordinate acquisition unit that acquires input coordinates to an input device of the user;
an operation determination unit that determines whether or not an input operation by the user is an operation on an input area corresponding to the projection screen based on the acquired input coordinates,
when the operation determination unit determines that the input operation is not an operation on the input area, the first output control unit performs a first display control process regarding the three-dimensional virtual space other than a control for the projection screen;
An information processing system in which, when the operation determination unit determines that the input operation is an operation on the input area, the second output control unit performs a second display control process on the projection screen according to the input coordinates.

Images