JP7506135B2 - Information processing device, program, and display method - Google Patents

Description

The present invention relates to an information processing device, a program, and a display method.

Patent Literature 1 describes AR glasses equipped with a display unit that can display images such as text and figures while remaining transparent or semi-transparent, thereby superimposing AR (Augmented Reality) virtual objects on the scenery in real space.
[Prior Art Literature]
[Patent Documents]
[Patent Document 1] JP 2019-114078 A

According to one embodiment of the present invention, an information processing device is provided. The information processing device may include a storage unit that stores correspondence information in which a type of location is associated with each of a plurality of applications registered by a user of a glasses-type device. The information processing device may include a location identification unit that identifies a type of location included in an imaging range captured by a camera of the glasses-type device. The information processing device may include a display control unit that refers to the correspondence information to identify an application corresponding to the type of location identified by the location identification unit, and displays content of the identified application at a position on a display unit of the glasses-type device that corresponds to the identified location.

In the information processing device, the location type may include at least one of the ground, sky, building, and space.

In any of the information processing devices, the display control unit may not display the content of an application executed on the glasses-type device if a location of a type corresponding to the application is not included in the imaging range of the glasses-type device, and may display the content of the application at a position on the display unit of the glasses-type device that corresponds to the identified location if the location is included.

In any of the information processing devices, the type of location may include space, the correspondence information may associate a height range of the space with the application, and when a space within the height range is included in the imaging range of the glasses-type device, the display control unit may display the content of the application corresponding to the height range of the space at a position on the display unit of the glasses-type device that corresponds to the height range of the space.

Any of the information processing devices may include a condition presentation unit that presents a condition of a type of place specified by a provider of the application to the user of the glasses-type device, and a registration unit that, when a type of place that satisfies the condition of the type of place presented to the user by the condition presentation unit is specified for the application, registers the application and the type of place in the correspondence information. The condition presentation unit may present a range of heights of a space specified by the provider of the application to the user of the glasses-type device, and the registration unit may register the application and the specified range of heights of a space in the correspondence information when a range of heights of a space within the range of heights of the space presented to the user by the condition presentation unit is specified for the application.

Any of the information processing devices may communicate with the glasses-type device via a network.

Any of the information processing devices may be mounted on the glasses-type device.

According to one embodiment of the present invention, a program is provided for causing a computer to function as the information processing device.

According to one embodiment of the present invention, a display method executed by a computer is provided. The display method may include a storage step of storing correspondence information in which a type of location is associated with each of a plurality of applications registered by a user of a glasses-type device. The display method may include a location identification step of identifying a type of location included in an imaging range captured by a camera of the glasses-type device. The display method may include a display control step of referring to the correspondence information to identify an application corresponding to the type of location identified by the location identification unit, and displaying content of the identified application at a position on the display unit of the glasses-type device corresponding to the identified location.

Note that the above summary of the invention does not list all of the necessary features of the present invention. Also, subcombinations of these features may also be inventions.

1 illustrates a schematic diagram of an example of a system 10. 1 shows an example of the configuration of a glasses-type device 100. 4 illustrates an example of correspondence information 410. 4 illustrates an example of correspondence information 420. 2 shows a schematic diagram of a display example by the glasses-type device 100. 2 shows a schematic diagram of a display example by the glasses-type device 100. 2 shows a schematic diagram of a display example by the glasses-type device 100. 2 illustrates an example of a functional configuration of the server 200. 13 shows an example of a functional configuration of the control device 300 in a case where the glasses-type device 100 stores correspondence information and the glasses-type device 100 takes the lead in controlling the display of content on the glasses-type device 100. An example of the hardware configuration of a computer 1200 functioning as the server 200 or the control device 300 is shown in schematic form.

For example, in a glasses-type device such as AR glasses, when CG (Computer Graphics) content is superimposed on a structure such as a building, the owner of the building can set the content to be displayed. However, various contents are scattered in unowned spaces such as roads and the sky, and when they are displayed all at once, too much content is displayed in the field of view of the glasses-type device wearer, resulting in poor visibility. If the content to be displayed is limited, only that content is displayed (when using a navigation app, content other than navigation (game characters, weather forecasts, etc.) will not be displayed). In contrast, the glasses-type device 100 and server 200 according to this embodiment make it possible to display multiple pieces of information and content by, for example, setting the content (apps/services) to be displayed for each location.

The present invention will be described below through embodiments of the invention, but the following embodiments do not limit the invention according to the claims. Furthermore, not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention.

FIG. 1 shows a schematic diagram of an example of a system 10. The system 10 includes a plurality of glasses-type devices 100 and a server 200. The server 200 may be an example of an information processing device.

The glasses-type devices 100 and the server 200 communicate with each other via a network 20. The network 20 includes a cloud. The network 20 may include a mobile communication network. The mobile communication network may conform to any of the following communication methods: 5G (5th Generation) communication method, LTE (Long Term Evolution) communication method, 3G (3rd Generation) communication method, 6G (6th Generation) communication method and later. The network 20 may support a Wi-Fi (registered trademark) communication method. The network 20 may be accessible via a Wi-Fi access point. The network 20 may be accessible using other wireless communication methods in general. The server 200 may be connected to the network 20 by wire or wirelessly. The glasses-type device 100 may be connected to the network 20 by mobile communication, Wi-Fi communication, or other wireless communication methods in general.

The glasses-type device 100 may be any glasses-type device that has wireless communication capabilities and a display capability for displaying content. The glasses-type device 100 may be, for example, AR glasses.

The glasses-type device 100 is capable of executing multiple applications. The glasses-type device 100 executes an application to display the content of the application. For example, the glasses-type device 100 executes a route guidance application, a weather forecast application, a guide application, a game application, etc.

The content of an application may be content generated in the application. For example, the content of a weather forecast application is content showing a weather forecast. For example, the content of a route guidance application is various types of content such as arrows for guiding the user along the way. For example, the content of a guide application is content for guiding the user, and in the case of providing guidance regarding a store, the content is content showing store information, etc. For example, the content of a game application is content representing game characters, items, etc.

The glasses-type device 100 may also execute a learning app, a business app, a shopping app, an image management app, etc. These are just examples, and the glasses-type device 100 may execute any app.

The content displayed by the glasses-type device 100 may be in any data format. For example, the content is 3D CG data. For example, the content is a still image. Also, for example, the content is a video. For example, the content is text. The content may include a combination of these.

The glasses-type device 100 may be a device capable of giving the user 102 the sensation that the content is located in real space by displaying the content on transparent or semi-transparent glass. In other words, the glasses-type device 100 may be a so-called transmissive (see-through) device. The glasses-type device 100 may also be a video pass-through device. In this case, the glasses-type device 100 may display a captured image capturing a range corresponding to the field of view of the user 102 on a non-transmissive display, and superimpose the content.

The server 200 manages each of the multiple glasses-type devices 100. The server 200 may manage content to be distributed to the glasses-type devices 100. The server 200 may cause the glasses-type devices 100 to display content by coordinating with an application running on the glasses-type devices 100.

In this embodiment, the server 200 may store correspondence information for each of the multiple glasses-type devices 100, in which a location type is associated with each of the multiple applications registered by the user 102 of the glasses-type device 100. The server 200 may use the correspondence information to control the display of content on each of the multiple glasses-type devices 100.

For example, if a location type corresponding to an application being executed on the glasses-type device 100 is not included in the imaging range captured by the camera of the glasses-type device 100, the server 200 does not display the content of the application, and if it is included, displays the content of the application at a position on the display unit of the glasses-type device 100 that corresponds to the identified location. This makes it possible for the user 102 to control the type of location in which content is displayed on an application-by-application basis.

In this way, in the system 10 according to this embodiment, the server 200 may store the correspondence information, and the server 200 may mainly control the display of content by the glasses-type device 100, but this is not limited to the above. The glasses-type device 100 may store the correspondence information, and the glasses-type device 100 may mainly control the display of content on the glasses-type device 100.

In this case, the glasses-type device 100 stores correspondence information in which a type of location is associated with each of the multiple applications registered by the user 102. The glasses-type device 100 may use the correspondence information to control the display of content.

For example, if a location type corresponding to a running application is not included in the imaging range captured by the camera of the glasses-type device 100, the glasses-type device 100 does not display the content of the application, and if it is included, displays the content of the application at a position on the display unit of the glasses-type device 100 that corresponds to the identified location.

Figure 2 shows an example of the configuration of the glasses-type device 100. The glasses-type device 100 includes a frame 110 having a rim 112 and temples 118. The glasses-type device 100 includes a control device 300 that executes various controls in the glasses-type device 100.

The location of the control device 300 in FIG. 2 is an example, and the control device 300 may be located in another position on the frame 110. For example, the control device 300 may be located outside the frame 110. Also, for example, the control device 300 may be located inside the temple 118. As a specific example, the control device 300 may be located at the tip of the temple 118, i.e., at the temple. The control device 300 may include a battery. The control device 300 and the battery may be separate and located in another position on the frame 110.

The glasses-type device 100 may include a camera 120. The imaging range of the camera 120 may be a range corresponding to the field of view of the user 102 wearing the glasses-type device 100. The arrangement of the camera 120 in FIG. 2 is an example, and the camera 120 may be arranged in another position.

The glasses-type device 100 may include a sensor 122. The sensor 122 may be a sensor for identifying the environment around the glasses-type device 100. The sensor 122 may include a LiDAR (Light Detection and Ranging). The sensor 122 may include a distance measurement sensor. FIG. 2 illustrates an example in which the sensor 122 is disposed on the bridge 114, but the arrangement of the sensor 122 in FIG. 2 is an example, and the sensor 122 may be disposed in another position.

The glasses-type device 100 may include a sensor 124. The sensor 124 detects, for example, the line of sight of the user 102 who is wearing the glasses-type device 100. The sensor 124 detects the line of sight of the user 102, for example, by capturing an image of the eyes of the user 102 and monitoring them. The arrangement of the sensor 124 in FIG. 2 is an example, and the sensor 124 may be arranged in another position.

The glasses-type device 100 may include a microphone 126. The microphone 126 is placed at a position where it is possible to acquire the speech of the user 102 who is the wearer of the glasses-type device 100. The placement of the microphone 126 in FIG. 2 is an example, and the microphone 126 may be placed at another position.

The control device 300 may accept various inputs. The control device 300 may accept various inputs from the user 102 via an operation unit arranged at any position on the glasses-type device 100. The control device 300 accepts various inputs from the user 102, for example, via an input unit such as a button arranged at any position on the frame 110. The control device 300 may accept various inputs from the user 102, for example, via a mobile terminal such as a smartphone or a wearable terminal, a tablet terminal, a PC, etc. The glasses-type device 100 may establish a wireless connection with these devices using a short-range wireless communication method such as Wi-Fi, Bluetooth (registered trademark), and Zigbee (registered trademark).

The control device 300 may include various sensors. The control device 300 may include, for example, a positioning sensor. For example, the control device 300 may include a GNSS (Global Navigation Satellite System) sensor such as a GPS (Global Positioning System) sensor. The control device 300 may include an acceleration sensor. The control device 300 may include a gyro sensor. The control device 300 may include other sensors.

The control device 300 may generate imaging range information indicating the imaging range of the camera 120. The control device 300 may generate the imaging range information based on the position, orientation, and posture of the glasses-type device 100. The control device 300 may identify the position, orientation, and posture of the glasses-type device 100 based on the output of various sensors.

The control device 300 may have a display function on the lens 116. The lens 116 may be an example of a display unit of the glasses-type device 100. The lens 116 may be a transparent or semi-transparent display, and the control device 300 may cause the lens 116 to perform various displays. In other words, the glasses-type device 100 may be a transparent or semi-transparent display-type device. The control device 300 may also have a function of projecting a display onto the lens 116. In other words, the glasses-type device 100 may be a projection-type device.

The control device 300 may have a function of performing communication via the network 20. The control device 300 may access the network 20 by at least one of mobile communication, Wi-Fi communication, and other general wireless communication. The control device 300 may communicate with the server 200 via the network 20.

Figure 3 shows an example of correspondence information 410 for a certain glasses-type device 100. In the correspondence information 410 shown in Figure 3, the ground is associated with a route guidance app, the sky is associated with a weather forecast app, and space is associated with a game app. Here, it is assumed that the glasses-type device 100 is running all of the route guidance app, weather forecast app, and game app.

When the server 200 is the subject, if the imaging range of the glasses-type device 100 includes the ground, the server 200 causes the glasses-type device 100 to display the content of a route guidance app at a position on the lens 116 corresponding to the ground. If the imaging range of the glasses-type device 100 does not include the ground, the server 200 does not cause the glasses-type device 100 to display the content of a route guidance app. If the imaging range of the glasses-type device 100 includes the sky, the server 200 causes the glasses-type device 100 to display the content of a weather forecast app at a position on the lens 116 corresponding to the sky. If the imaging range of the glasses-type device 100 does not include the sky, the server 200 does not cause the glasses-type device 100 to display the content of a weather forecast app. The server 200 causes the glasses-type device 100 to display the content of a game app at a position on the lens 116 corresponding to the space.

When the glasses-type device 100 is the subject, if the imaging range includes the ground, the glasses-type device 100 displays the content of a route guidance app at a position on the lens 116 corresponding to the ground. If the imaging range does not include the ground, the glasses-type device 100 does not display the content of a route guidance app. If the imaging range includes the sky, the glasses-type device 100 displays the content of a weather forecast app at a position on the lens 116 corresponding to the sky. If the imaging range does not include the sky, the glasses-type device 100 does not display the content of a weather forecast app. The glasses-type device 100 displays the content of a game app at a position on the lens 116 corresponding to the space.

When content from many applications is displayed on the lens 116, the visibility of the user 102 is reduced. On the other hand, when the applications that display content are limited, only the content from those applications is displayed. In contrast, according to the system 10 of this embodiment, by associating a location type with each of the multiple applications, it is possible to appropriately limit the number of contents displayed on the lens 116, and to display the content for each of the multiple applications in a location where the user 102 desires to display the content. This makes it possible to provide the user 102 with an environment in which the content of a weather forecast application, which has a strong relationship with the sky, is displayed in the sky, and the content of a route guidance application, which has a strong relationship with the ground, such as a road, is displayed on the ground.

FIG. 4 shows a schematic diagram of an example of the correspondence information 420. In the correspondence information 420 shown in FIG. 4, the ground is associated with a route guidance app, a space 2 m above the ground is associated with a game app, a space 2 to 50 m high is associated with a guide app, and a space 50 m or higher is associated with a weather forecast app. Here, it is assumed that a route guidance app, a game app, a guide app, and a weather forecast app are all running on a single glasses-type device 100.

When the server 200 is the subject, if the imaging range of the glasses-type device 100 includes the ground, the server 200 causes the glasses-type device 100 to display the content of the route guidance app at a position on the lens 116 that corresponds to the ground. If the imaging range of the glasses-type device 100 does not include the ground, the server 200 does not cause the glasses-type device 100 to display the content of the route guidance app.

If the imaging range of the glasses-type device 100 includes a space at a height of 2 to 50 m, the server 200 causes the glasses-type device 100 to display the content of the guide app at a position on the lens 116 that corresponds to the space at a height of 2 to 50 m. If the imaging range of the glasses-type device 100 does not include a space at a height of 2 to 50 m, the server 200 does not cause the glasses-type device 100 to display the content of the guide app.

When the imaging range of the glasses-type device 100 includes a space of 50 m or more, the server 200 causes the glasses-type device 100 to display the contents of a weather forecast app at a position on the lens 116 that corresponds to the space of 50 m or more. When the imaging range of the glasses-type device 100 does not include a space of 50 m or more, the server 200 does not cause the glasses-type device 100 to display the contents of a weather forecast app.

When the glasses-type device 100 is the subject, if the imaging range of the glasses-type device 100 includes the ground, the glasses-type device 100 displays the content of the route guidance app at a position on the lens 116 that corresponds to the ground. If the imaging range of the glasses-type device 100 does not include the ground, the glasses-type device 100 does not display the content of the route guidance app.

When the imaging range of the glasses-type device 100 includes a space 2 m above the ground, the glasses-type device 100 displays the game app content at a position on the lens 116 that corresponds to the space 2 m above the ground. When the imaging range of the glasses-type device 100 does not include a space 2 m above the ground, the glasses-type device 100 does not display the game app content.

When the imaging range of the glasses-type device 100 includes a space at a height of 2 m to 50 m, the glasses-type device 100 displays the content of the guide app at a position on the lens 116 that corresponds to the space at a height of 2 m to 50 m. When the imaging range of the glasses-type device 100 does not include a space at a height of 2 m to 50 m, the glasses-type device 100 does not display the content of the guide app.

When the imaging range of the glasses-type device 100 includes a space with a height of 50 m or more, the glasses-type device 100 displays the contents of a weather forecast app at a position on the lens 116 that corresponds to the space with a height of 50 m or more. When the imaging range of the glasses-type device 100 does not include a space with a height of 50 m or more, the glasses-type device 100 does not display the contents of a weather forecast app.

In this way, by being able to set a range of heights in space, the position where content is displayed can be controlled in more detail.

Figure 5 shows a schematic example of a display on the glasses-type device 100. Here, we explain a case where the sky is associated with a weather forecast app, and the ground is associated with a route guidance app.

In the example shown in FIG. 5, the imaging range of the glasses-type device 100 includes the sky and the ground, so that the content 510 of a weather forecast app is displayed at a position on the lens 116 of the glasses-type device 100 that corresponds to the sky, and the content 520 of a route guidance app is displayed at a position on the lens 116 of the glasses-type device 100 that corresponds to the ground.

Figure 6 shows a schematic example of a display on the glasses-type device 100. Here, differences from Figure 5 will be mainly described. In this example, in the correspondence information of the user 102 of the glasses-type device 100, a space 2 m above the ground is associated with a route guidance app. Since the imaging range of the glasses-type device 100 includes the sky and a space 2 m above the ground, content 510 of the weather forecast app is displayed at a position on the lens 116 of the glasses-type device 100 corresponding to the sky, and content 520 of the route guidance app is displayed at a position on the lens 116 of the glasses-type device 100 corresponding to the space 2 m above the ground.

As illustrated in Figures 5 and 6, in a certain application, the content registered at a certain real-world position may be the same regardless of the type of space, but may also differ depending on the type of space. For example, in a certain application, content with different expressions or sizes may be registered at a certain real-world position depending on the type of space. As a specific example, planar content may be registered for the ground, and three-dimensional content may be registered for the space.

Figure 7 shows a schematic example of a display on the glasses-type device 100. Here, differences from Figure 6 will be mainly described. In this example, a space 2 m above the ground is associated with the navigation app, but planar content 520 is registered with respect to the ground in the navigation app, and three-dimensional content 522 is registered with respect to the space 2 m above the ground. In the correspondence information of the user 102 of the glasses-type device 100, a space 2 m above the ground is associated with the navigation app. Since the imaging range of the glasses-type device 100 includes the sky and a space 2 m above the ground, a weather forecast app content 510 is displayed at a position on the lens 116 of the glasses-type device 100 corresponding to the sky, and a navigation app content 522 is displayed at a position on the lens 116 of the glasses-type device 100 corresponding to the space 2 m above the ground.

Figure 8 shows an example of the functional configuration of the server 200. The server 200 includes a memory unit 202, a registration unit 204, an external communication unit 206, a condition presentation unit 208, a location identification unit 210, and a display control unit 212. Note that it is not essential that the server 200 includes all of these.

The storage unit 202 stores various information. The registration unit 204 registers various information. The registration unit 204 stores the registered information in the storage unit 202. The registration unit 204 registers various information, for example, by input from a user of the server 200.

The external communication unit 206 performs communication with the outside. The external communication unit 206 may communicate with other devices via the network 20. The registration unit 204 may register information received from other devices via the external communication unit 206.

The external communication unit 206 communicates with, for example, the glasses-type device 100. For example, the external communication unit 206 continuously receives imaging range information from the glasses-type device 100 that is the subject of management. The glasses-type device 100 may transmit the imaging range information to the server 200 periodically or irregularly. In addition, the glasses-type device 100 may transmit the imaging range information to the server 200 each time the imaging range changes.

The registration unit 204 may register multiple pieces of content. The registration unit 204 may register multiple pieces of content for each of multiple applications.

The registration unit 204 may, for example, receive and register application content from a communication device used by the application provider via the external communication unit 206. The registration unit 204 may receive multiple pieces of content and multiple real-world locations corresponding to the multiple pieces of content from the communication device used by the application provider, and register the multiple pieces of content and the multiple real-world locations in association with each other. The registration unit 204 may further receive and register conditions for the types of locations in which the application content can be displayed from the communication device used by the application provider.

The location type condition may indicate the type of location where the content can be displayed. The location type condition may be, for example, a condition that specifies the ground as a location where the content can be displayed. The location type condition may be, for example, a condition that specifies the sky as a location where the content can be displayed. The location type condition may be, for example, a condition that specifies space as a location where the content can be displayed. The location type condition may be, for example, a condition that specifies a height range of space as a location where the content can be displayed.

The location type condition may indicate the type of location where the content is not to be displayed. The location type condition may be, for example, a condition that specifies the ground as a location where the content is not to be displayed. In this case, when the glasses-type device 100 is executing an application corresponding to the content, if the imaging range of the glasses-type device 100 includes only the ground, the content is not displayed, and if the imaging range does not include the ground but includes other locations, or if the imaging range includes both the ground and other locations, the content is displayed.

The location type condition may be, for example, a condition that specifies the sky as a location where the content is not to be displayed. In this case, when the glasses-type device 100 is executing an application corresponding to the content, if the imaging range of the glasses-type device 100 includes only the sky, the content is not displayed, and if the imaging range does not include the sky but includes other locations, or if the imaging range includes both the sky and other locations, the content is displayed.

The location type condition may be, for example, a condition that specifies a space as a location where the content is not to be displayed. In this case, when the glasses-type device 100 is executing an application corresponding to the content, if only the space is included in the imaging range of the glasses-type device 100, the content is not displayed, and if the space is not included but other locations are included, or if both the space and other locations are included, the content is displayed.

The location type condition may be, for example, a condition that specifies a height range of space as a location where the content is not displayed. In this case, when the glasses-type device 100 is executing an application corresponding to the content, if the imaging range of the glasses-type device 100 includes only the height range of the space, the content is not displayed, and if the height range of the space is not included but other locations are included, or if both the height range of the space and other locations are included, the content is displayed.

The registration unit 204 may register various information by accepting input from the user 102 of the glasses-type device 100 via the external communication unit 206. For example, the registration unit 204 registers correspondence information in which a type of location is associated with each of a plurality of applications.

The location type may include ground. The location type may include sky. The location type may include a structure. The location type may include space. The location type may include a floor. The location type may include a mountain. The location type may include a pond, a lake, a swamp, a river, an ocean, etc. The location type may include other types. When the location type is space, the correspondence information may associate a height range of the space with an application.

The condition presenting unit 208 presents the location type conditions specified by the application provider to the user 102 of the glasses-type device 100. The condition presenting unit 208 may present the location type conditions, which the registration unit 204 has registered for the application and which are received from a communication device used by the application provider, to the glasses-type device 100 via the external communication unit 206, to the user 102 of the glasses-type device 100.

When the user 102 of the glasses-type device 100 registers a location type corresponding to an application, the user 102 registers the location type so that it satisfies the conditions of the presented location type. When a location type that satisfies the conditions of the location type presented to the user 102 by the condition presenting unit 208 is specified for the application, the registration unit 204 registers the application and the location type in the correspondence information. When a location type that does not satisfy the conditions of the location type presented to the user 102 by the condition presenting unit 208 is specified for the application, the registration unit 204 may reject the registration.

The condition presentation unit 208 may present the range of height of the space specified by the application provider to the user 102 of the glasses-type device 100. When a range of height of the space is specified for the application within the range of height of the space presented to the user 102 by the condition presentation unit 208, the registration unit 204 registers the application and the specified range of height of the space in the correspondence information. When a range of height of the space is specified for the application outside the range of height of the space presented to the user 102 by the condition presentation unit 208, the registration unit 204 may reject the registration. For example, when a space with a height of 30 m or more is presented to the user 102 by the condition presentation unit 208, and the user 102 specifies a range that satisfies the condition, such as a height of 30 m or more or a height of 50 m or more, the registration unit 204 executes the registration, and when a range that does not satisfy the condition, such as a range lower than 30 m in height, is specified, the registration unit 204 rejects the registration.

The registration unit 204 may register multiple contents for each type of location for each of the multiple applications. For example, the registration unit 204 registers multiple contents for display on the ground, multiple contents for display in space, and multiple contents for display in the sky for the application in accordance with a registration instruction from a communication terminal used by the application provider.

This allows, for example, content A to be registered for display on the ground, content B for display in space, and content C for display in the sky for a certain position in the real world. Content A, content B, and content C may, for example, indicate the same content, but differ in content size or content expression (whether the content is two-dimensional or three-dimensional, etc.). Note that content A, content B, and content C may also indicate different content.

If the user 102 of the glasses-type device 100 registers "ground" as a type of location for the application, content A is displayed when the ground at that real-world location is included in the imaging range of the glasses-type device 100. If the user 102 of the glasses-type device 100 registers "space" as a type of location for the application, content B is displayed when the space at that real-world location is included in the imaging range of the glasses-type device 100. If the user 102 of the glasses-type device 100 registers "sky" as a type of location for the application, content C is displayed when the sky at that real-world location is included in the imaging range of the glasses-type device 100.

As a specific example, a provider of a navigation app registers planar content A as content to be displayed on the ground, registers content B of a smaller size than the default as content to be displayed in a space between 2m and 50m high, and registers content C of the default size as content to be displayed in a space of 50m or higher high.

The condition presenting unit 208 may present the type of location in which content is registered for each of multiple applications to the user 102 of the glasses-type device 100. For example, if content to be displayed on the ground, content to be displayed in the space, and content to be displayed in the sky are registered for a certain application, the condition presenting unit 208 presents the ground, space, and sky for that application to the user 102. For example, if content to be displayed on the ground and content to be displayed in the space are registered for another application, the condition presenting unit 208 presents the ground and space to the user 102.

As an example of a specific flow, first, the user 102 selects an application on the glasses-type device 100. In response to the selection, the condition presenting unit 208 identifies the type of location for which content is registered for the selected application, and presents the identified type of location to the user 102. The user 102 selects at least one of the presented types of location. The registration unit 204 registers the selected type of location for the selected application.

As a specific example, when a road guidance app is selected by the user 102, the condition presenting unit 208 presents to the user 102 the types of places in which content is registered in the road guidance app. In this example, content A is registered on the ground, content B is registered in a space with a height of 2 to 50 m, and content C is registered in a space with a height of 50 m or more, so the condition presenting unit 208 presents to the user 102 the ground, a space with a height of 2 to 50 m, and a space with a height of 50 m or more. The user 102 selects at least one of the three presented. For example, when the ground is selected, the ground is registered in the correspondence information in association with the road guidance app. This allows the user 102 to understand the types of places in which content is registered in the road guidance app, and supports the user 102 in registering.

For example, in the glasses-type device 100, the user 102 may first select a location type, and the condition presentation unit 208 may present the user 102 with a list of applications that have registered content to be displayed in the selected location type.

As an example of a specific flow, first, the user 102 selects a type of location on the glasses-type device 100. In response to the selection, the condition presentation unit 208 identifies applications in which content to be displayed for the selected type of location is registered, and presents a list of the identified applications to the user 102. The user 102 selects at least one of the presented applications. The registration unit 204 registers the selected application for the selected type of location.

As a specific example, when the user 102 selects the ground, the condition presenting unit 208 presents to the user 102 applications in which content to be displayed on the ground is registered. Here, an example is given in which content to be displayed on the ground is registered for a navigation app and a game app. The user 102 selects at least one of the presented navigation app and game app. For example, when a navigation app is selected, the navigation app is registered in the correspondence information with the ground associated with it. This allows the user 102 to know which applications have content to be displayed on the ground registered, and assists the user 102 in registering the applications.

The location identification unit 210 identifies the type of location included in the imaging range captured by the camera 120 of the glasses-type device 100. The location identification unit 210 may identify the type of location included in the imaging range based on imaging range information that the external communication unit 206 continuously receives from the glasses-type device 100.

The location identification unit 210, for example, stores map data of various locations in advance, and identifies which range in the map data is included in the imaging range of the glasses-type device 100 from the position, orientation, and posture included in the imaging range information received from the glasses-type device 100, thereby identifying the type of location included in the imaging range of the glasses-type device 100. The map data is, for example, three-dimensional map data including information on the three-dimensional shape of objects included in the coverage range. The map data may be two-dimensional map data that separately includes information on the three-dimensional shape of objects included in the coverage range. The map data may be two-dimensional images such as Geospatial API, or GPS-based map data. The location identification unit 210 may identify the type of location included in the imaging range of the glasses-type device 100 using any other method.

The location identification process may be performed by the glasses-type device 100, rather than by the server 200. In this case, the location identification unit 210 may acquire information on the type of location included in the imaging range from the glasses-type device 100, along with the imaging range information. The glasses-type device 100 may identify the type of location included in the imaging range from the image captured by the camera 120 and the detection result by the sensor 122. The glasses-type device 100 may identify the type of location included in the imaging range using any method, such as by cooperating with a VPS (Visual Positioning System) server.

The display control unit 212 controls the display of content by the glasses-type device 100. The display control unit 212 controls the display of content of an application being executed by the glasses-type device 100. The display control unit 212 controls the display of content on the glasses-type device 100 based on the imaging range information received by the external communication unit 206.

For example, the display control unit 212 refers to the correspondence information of the glasses-type device 100 stored in the storage unit 202, identifies an application corresponding to the type of location identified by the location identification unit 210, and displays the content of the identified application at a position on the lens 116 of the glasses-type device 100 corresponding to the identified location. The display control unit 212 identifies an application corresponding to the type of location identified by the location identification unit 210, and if there is content of the application whose corresponding real-world location is included in the imaging range of the glasses-type device 100, the display control unit 212 may display the content at a position on the lens 116 of the glasses-type device 100 corresponding to the identified location.

When a location type corresponding to an application being executed on the glasses-type device 100 is not included in the imaging range of the glasses-type device 100, the display control unit 212 may not display the content of the application, and when the location type corresponding to the application is included, the display control unit 212 may display the content of the application at a position corresponding to the identified location on the lens 116 of the glasses-type device 100.

When the location identification unit 210 identifies a spatial height range, such as 2 m from the ground, the display control unit 212 displays the content of the application corresponding to the spatial height range at a position on the lens 116 of the glasses-type device 100 that corresponds to the spatial height range.

When the registration unit 204 has registered content for each type of location for each of a plurality of applications, the display control unit 212 may refer to the correspondence information of the glasses-type device 100 stored in the storage unit 202, identify an application corresponding to the type of location identified by the location identification unit 210, and display the content of the identified application corresponding to the type of location at a position on the lens 116 of the glasses-type device 100 corresponding to the identified location. For example, when the ground is identified by the location identification unit 210, the display control unit 212 may identify an application corresponding to the ground, and display the content of the identified application corresponding to the ground at a position on the lens 116 of the glasses-type device 100 corresponding to the ground.

Figure 9 shows an example of the functional configuration of the control device 300 in a case where the glasses-type device 100 stores the correspondence information and the glasses-type device 100 mainly controls the display of content on the glasses-type device 100. The control device 300 may be an example of an information processing device.

The control device 300 includes a memory unit 302, an information acquisition unit 304, a server communication unit 306, a registration unit 308, a condition presentation unit 310, a location identification unit 312, and a display control unit 314. It is not essential that the control device 300 includes all of these.

The storage unit 302 stores various information. The information acquisition unit 304 acquires various information. The information acquisition unit 304 may store the acquired information in the storage unit 302.

The information acquisition unit 304, for example, acquires an image captured by the camera 120 from the camera 120. The information acquisition unit 304 acquires, for example, information output by various sensors. The information acquisition unit 304 may acquire information output by the sensor 122. The information acquisition unit 304 may acquire information output by the sensor 124. The information acquisition unit 304 may acquire information output by a positioning sensor. The information acquisition unit 304 may acquire information output by an acceleration sensor. The information acquisition unit 304 may acquire information output by a gyro sensor. The information acquisition unit 304 may acquire information output by other sensors. The information acquisition unit 304, for example, acquires the speech of the user 102 acquired by the microphone 126. The information acquisition unit 304 may acquire the content of the speech by performing voice recognition on the speech of the user 102 acquired by the microphone 126.

The information acquisition unit 304 may acquire various inputs from the user 102. The information acquisition unit 304 may acquire input information input by the user 102 via an operation unit or a mobile terminal. The information acquisition unit 304 may acquire the contents of the voice input by the user 102. The information acquisition unit 304 may acquire input information based on the hand movements of the user 102.

The information acquisition unit 304 may generate imaging range information indicating the imaging range of the camera 120. The information acquisition unit 304 may generate the imaging range information based on the position, orientation, and posture of the glasses-type device 100. The information acquisition unit 304 may identify the position, orientation, and posture of the glasses-type device 100 based on the output of various sensors.

The information acquisition unit 304 acquires the execution status of applications in the glasses-type device 100. The information acquisition unit 304 may acquire an application execution status indicating an application that is being executed among multiple applications installed in the glasses-type device 100.

The server communication unit 306 communicates with the server 200 via the network 20. For example, the server communication unit 306 transmits various information acquired by the information acquisition unit 304 to the server 200.

The registration unit 308 registers various pieces of information. The registration unit 308 may register various pieces of information in accordance with various inputs from the user 102 acquired by the information acquisition unit 304. The registration unit 308 stores the registered information in the storage unit 302.

The registration unit 308 registers correspondence information in which a type of location is associated with each of the multiple applications. The user 102 may consider and register the type of location in which the user wants to display content for each of the multiple applications.

The condition presentation unit 310 presents the location type conditions specified by the application provider to the user 102. The condition presentation unit 310 may receive the location type conditions specified by the application provider from the server 200 and present them to the user 102. The condition presentation unit 310 may, for example, cause the lens 116 to display the location type conditions.

The location identification unit 312 identifies the type of location included in the imaging range captured by the camera 120. For example, the location identification unit 312 stores map data for various locations in advance, and identifies which ranges in the data map are included in the imaging range of the camera 120 from the position, direction, and attitude included in the imaging range information acquired by the information acquisition unit 304, thereby identifying the type of location included in the imaging range. The location identification unit 312 may identify the type of location included in the imaging range using any method, such as by working with a VPS server.

The display control unit 314 controls the display of the lens 116. The display control unit 314 may cause the lens 116 to display the content received by the server communication unit 306 from the server 200. The display control unit 314 may control the display of the content based on the execution status of the application in the glasses-type device 100 and the type of location identified by the location identification unit 312.

For example, first, the server communication unit 306 transmits the imaging range information acquired by the information acquisition unit 304 to the server 200, receives the content to be displayed from the server 200, and stores it in the storage unit 302. The server communication unit 306 may receive all the content whose corresponding real-world position is included in the imaging range indicated by the imaging range information from the server 200. The display control unit 314 refers to the correspondence information stored in the storage unit 302 and identifies an application corresponding to the type of location identified by the location identification unit 312. Then, when the content received by the server communication unit 306 includes the content of the identified application that is an application being executed, the display control unit 314 displays the content at a position corresponding to the identified location on the lens 116. Even if the application is being executed in the glasses-type device 100, if the type of location corresponding to the application is not included in the imaging range of the camera 120, the display control unit 314 does not display the content of the application.

When content is registered for each type of location for each of a plurality of applications, the server communication unit 306 may receive from the server 200 all content whose corresponding real-world location is included in the imaging range indicated by the imaging range information, regardless of the type of the corresponding location. The display control unit 314 refers to the correspondence information stored in the storage unit 302 to identify an application corresponding to the type of location identified by the location identification unit 312. Then, when the content received by the server communication unit 306 includes content that is the identified application and corresponds to the identified type of location of the application being executed, the content may be displayed at a position corresponding to the identified location on the lens 116. For example, for a navigation app, content is registered for each of the ground, space, and sky, the navigation app and the ground are associated in the correspondence information, and the ground is identified by the location identification unit 312. In this case, the display control unit 314 identifies the navigation app corresponding to the ground, and displays the content of the navigation app corresponding to the ground at a position corresponding to the ground on the lens 116.

Also, for example, first, the server communication unit 306 transmits to the server 200 the imaging range information and application execution status acquired by the information acquisition unit 304, and the type of location identified by the location identification unit 312. The server 200 refers to the correspondence information stored in the storage unit 202, and identifies, from among the applications being executed in the glasses-type device 100, an application whose corresponding location type matches the type of location received from the glasses-type device 100, and transmits content of the identified application to the glasses-type device 100. The server communication unit 306 receives the content transmitted by the glasses-type device 100, and displays the content at a position on the lens 116 corresponding to the location identified by the location identification unit 312.

When content is registered for each type of location for each of multiple applications, the server 200, which has received the imaging range information, application execution status, and type of location, refers to the correspondence information stored in the memory unit 202, identifies, from among the applications running on the glasses-type device 100, an application whose corresponding type of location matches the type of location received from the glasses-type device 100, and transmits content for the identified application corresponding to the type of location to the glasses-type device 100.

Figure 10 shows an example of the hardware configuration of the server 200 or the control device 300. The computer 1200 according to this embodiment includes a CPU 1212, a RAM 1214, and a graphics controller 1216, which are connected to each other by a host controller 1210. The computer 1200 also includes input/output units such as a communication interface 1222, a storage device 1224, a DVD drive, and an IC card drive, which are connected to the host controller 1210 via the input/output controller 1220. The DVD drive may be a DVD-ROM drive, a DVD-RAM drive, or the like. The storage device 1224 may be a hard disk drive, a solid state drive, or the like. The computer 1200 also includes a legacy input/output unit such as a ROM 1230 and a keyboard, which are connected to the input/output controller 1220 via an input/output chip 1240.

The CPU 1212 operates according to the programs stored in the ROM 1230 and the RAM 1214, thereby controlling each unit. The graphics controller 1216 acquires image data generated by the CPU 1212 into a frame buffer or the like provided in the RAM 1214 or into itself, and causes the image data to be displayed on the display device 1218.

The communication interface 1222 communicates with other electronic devices via a network. The storage device 1224 stores programs and data used by the CPU 1212 in the computer 1200. The DVD drive reads programs or data from a DVD-ROM or the like and provides them to the storage device 1224. The IC card drive reads programs and data from an IC card and/or writes programs and data to an IC card.

ROM 1230 stores therein a boot program or the like executed by computer 1200 upon activation, and/or a program that depends on the hardware of computer 1200. I/O chip 1240 may also connect various I/O units to I/O controller 1220 via USB ports, parallel ports, serial ports, keyboard ports, mouse ports, etc.

The programs are provided by a computer-readable storage medium such as a DVD-ROM or an IC card. The programs are read from the computer-readable storage medium, installed in storage device 1224, RAM 1214, or ROM 1230, which are also examples of computer-readable storage media, and executed by CPU 1212. The information processing described in these programs is read by computer 1200, and brings about cooperation between the programs and the various types of hardware resources described above. An apparatus or method may be constructed by realizing the operation or processing of information according to the use of computer 1200.

For example, when communication is performed between computer 1200 and an external device, CPU 1212 may execute a communication program loaded into RAM 1214 and instruct communication interface 1222 to perform communication processing based on the processing described in the communication program. Under the control of CPU 1212, communication interface 1222 reads transmission data stored in a transmission buffer area provided in RAM 1214, storage device 1224, a DVD-ROM, or a recording medium such as an IC card, and transmits the read transmission data to the network, or writes received data received from the network to a reception buffer area or the like provided on the recording medium.

The CPU 1212 may also cause all or a necessary portion of a file or database stored in an external recording medium such as the storage device 1224, a DVD drive (DVD-ROM), an IC card, etc. to be read into the RAM 1214, and perform various types of processing on the data on the RAM 1214. The CPU 1212 may then write back the processed data to the external recording medium.

Various types of information, such as various types of programs, data, tables, and databases, may be stored in the recording medium and may undergo information processing. The CPU 1212 may perform various types of processing on the data read from the RAM 1214, including various types of operations, information processing, conditional judgment, conditional branching, unconditional branching, information search/replacement, etc., as described throughout this disclosure and specified by the instruction sequence of the program, and writes back the results to the RAM 1214. The CPU 1212 may also search for information in a file, database, etc. in the recording medium. For example, when a plurality of entries each having an attribute value of a first attribute associated with an attribute value of a second attribute are stored in the recording medium, the CPU 1212 may search for an entry whose attribute value of the first attribute matches a specified condition from among the plurality of entries, read the attribute value of the second attribute stored in the entry, and thereby obtain the attribute value of the second attribute associated with the first attribute that satisfies a predetermined condition.

The above-described program or software module may be stored in a computer-readable storage medium on the computer 1200 or in the vicinity of the computer 1200. In addition, a recording medium such as a hard disk or RAM provided in a server system connected to a dedicated communication network or the Internet can be used as a computer-readable storage medium, thereby providing the program to the computer 1200 via the network.

The blocks in the flowcharts and block diagrams in this embodiment may represent stages of a process in which an operation is performed or "parts" of a device responsible for performing the operation. Particular stages and "parts" may be implemented by dedicated circuitry, programmable circuitry provided with computer-readable instructions stored on a computer-readable storage medium, and/or a processor provided with computer-readable instructions stored on a computer-readable storage medium. The dedicated circuitry may include digital and/or analog hardware circuits and may include integrated circuits (ICs) and/or discrete circuits. The programmable circuitry may include reconfigurable hardware circuits including AND, OR, XOR, NAND, NOR, and other logical operations, flip-flops, registers, and memory elements, such as, for example, field programmable gate arrays (FPGAs) and programmable logic arrays (PLAs).

A computer-readable storage medium may include any tangible device capable of storing instructions that are executed by a suitable device, such that a computer-readable storage medium having instructions stored thereon comprises an article of manufacture that includes instructions that can be executed to create means for performing the operations specified in the flowchart or block diagram. Examples of computer-readable storage media may include electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, and the like. More specific examples of computer-readable storage media may include floppy disks, diskettes, hard disks, random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), electrically erasable programmable read-only memories (EEPROMs), static random access memories (SRAMs), compact disk read-only memories (CD-ROMs), digital versatile disks (DVDs), Blu-ray disks, memory sticks, integrated circuit cards, and the like.

The computer readable instructions may include either assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages such as Smalltalk (registered trademark), JAVA (registered trademark), C++, etc., and conventional procedural programming languages such as the "C" programming language or similar programming languages.

The computer-readable instructions may be provided to a processor of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, or a programmable circuit, either locally or over a local area network (LAN), a wide area network (WAN), such as the Internet, to cause the processor of the general-purpose computer, special-purpose computer, or other programmable data processing apparatus, or a programmable circuit, to execute the computer-readable instructions to generate means for performing the operations specified in the flowcharts or block diagrams. Examples of processors include computer processors, processing units, microprocessors, digital signal processors, controllers, microcontrollers, etc.

The present invention has been described above using an embodiment, but the technical scope of the present invention is not limited to the scope described in the above embodiment. It is clear to those skilled in the art that various modifications and improvements can be made to the above embodiment. It is clear from the claims that forms with such modifications or improvements can also be included in the technical scope of the present invention.

The order of execution of each process, such as operations, procedures, steps, and stages, in the devices, systems, programs, and methods shown in the claims, specifications, and drawings is not specifically stated as "before" or "prior to," and it should be noted that the processes may be performed in any order, unless the output of a previous process is used in a later process. Even if the operational flow in the claims, specifications, and drawings is explained using "first," "next," etc. for convenience, it does not mean that it is necessary to perform the processes in that order.

10 System, 100 Glasses-type device, 102 User, 110 Frame, 112 Rim, 114 Bridge, 116 Lens, 118 Temple, 120 Camera, 122 Sensor, 124 Sensor, 126 Microphone, 200 Server, 202 Memory unit, 204 Registration unit, 206 External communication unit, 208 Condition presentation unit, 210 Location identification unit, 212 Display control unit, 300 Control device, 302 Memory unit, 304 Information acquisition unit, 306 Server communication unit, 308 Registration unit, 310 Condition presentation unit, 312 Location identification unit, 314 Display control unit, 410 Correspondence information, 420 Correspondence information, 510, 520, 522 Content, 1200 Computer, 1210 Host controller, 1212 CPU, 1214 RAM, 1216 Graphics controller, 1218 display device, 1220 input/output controller, 1222 communication interface, 1224 storage device, 1230 ROM, 1240 input/output chip

Claims (12)

a storage unit that stores correspondence information in which a type of place is associated with each of a plurality of applications registered by a user of the glasses-type device;
a place identification unit that identifies a type of place included in an imaging range captured by a camera of the glasses-type device;
a display control unit that refers to the correspondence information to identify an application corresponding to the type of the location identified by the location identification unit, and displays content of the identified application at a position on a display unit of the glasses-type device that corresponds to the identified location,
a condition presenting unit that presents a location type condition specified by a provider of the application to the user of the glasses-type device;
a registration unit that, when a type of place that satisfies the condition of the type of place presented to the user by the condition presentation unit is designated for the application, registers the application and the type of place in the correspondence information;
The information processing device further comprises : 2 . The information processing device according to claim 1 , wherein when a location type is specified for the application that does not satisfy the location type conditions presented to the user by the condition presentation unit, the registration unit does not register the application and the location type in the correspondence information. The information processing device according to claim 1, wherein the location type includes at least one of the ground, sky, building, and space. The information processing device according to claim 1, wherein the display control unit does not display the content of an application executed on the glasses-type device if a location of a type corresponding to the application is not included in the imaging range of the glasses-type device, and displays the content of the application at a position corresponding to the identified location on the display unit of the glasses-type device if the location is included in the imaging range of the glasses-type device. The type of location includes space;
The correspondence information corresponds a range of heights in a space to the application;
The information processing device according to claim 1, wherein when a space within the height range is included in the imaging range of the glasses-type device, the display control unit displays content of the application corresponding to the height range of the space at a position corresponding to the height range of the space on the display unit of the glasses-type device. the condition presenting unit presents a range of heights of a space specified by a provider of the application to the user of the glasses-type device;
2. The information processing device according to claim 1, wherein when a range of heights of a space is specified for the application within the range of heights of the space presented to the user by the condition presentation unit , the registration unit registers the application and the specified range of heights of the space in the correspondence information. a storage unit that stores correspondence information in which a type of place is associated with each of a plurality of applications registered by a user of the glasses-type device;
a place identification unit that identifies a type of place included in an imaging range captured by a camera of the glasses-type device;
a display control unit that refers to the correspondence information to identify an application corresponding to the type of the location identified by the location identification unit, and displays content of the identified application at a position on a display unit of the glasses-type device that corresponds to the identified location;
Equipped with
The type of location includes space;
The correspondence information corresponds a range of heights in a space to the application;
The display control unit of the information processing device displays content of the application corresponding to the height range of the space at a position corresponding to the height range of the space on the display unit of the glasses-type device when a space within the height range is included in the imaging range of the glasses-type device . The information processing apparatus according to claim 1 , wherein the information processing apparatus communicates with the glasses-type device via a network. The information processing device according to claim 1 , wherein the information processing device is mounted on the glasses-type device. A program for causing a computer to function as the information processing device according to any one of claims 1 to 7 . 1. A computer-implemented display method comprising:
a storage step of storing correspondence information in which a type of place is associated with each of a plurality of applications registered by a user of the glasses-type device;
A place identification step of identifying a type of place included in an imaging range captured by a camera of the glasses-type device;
a display control step of identifying an application corresponding to the type of the place identified in the place identification step by referring to the correspondence information, and displaying content of the identified application at a position on a display unit of the glasses-type device corresponding to the identified place ,
a condition presentation step of presenting a condition of a type of place specified by a provider of the application to the user of the glasses-type device;
a registration step of registering the application and the type of place in the correspondence information when the type of place is specified to the application and satisfies the condition of the type of place presented to the user in the condition presentation step;
The display method further comprises : 1. A computer-implemented display method comprising:
a storage step of storing correspondence information in which a type of place is associated with each of a plurality of applications registered by a user of the glasses-type device;
A place identification step of identifying a type of place included in an imaging range captured by a camera of the glasses-type device;
a display control step of identifying an application corresponding to the type of the place identified in the place identification step by referring to the correspondence information, and displaying content of the identified application at a position on a display unit of the glasses-type device corresponding to the identified place;
Equipped with
The type of location includes space;
The correspondence information corresponds a range of heights in a space to the application;
The display control step is a display method in which, when a space within the height range is included in the imaging range of the glasses-type device, content of the application corresponding to the height range of the space is displayed at a position corresponding to the height range of the space on the display unit of the glasses-type device.

Images