WO2025224986A1 - Wearable terminal and information processing method - Google Patents

Abstract

Provided is a wearable terminal worn by a user, said wearable terminal comprising: a display which displays, on a display screen, a virtual space which may include a virtual object, an image of a real space and a virtual object, or only an image of a real space; a wired connection interface to which a wired cable is connected and which is used for a wired connection with a device; and a processor. When the processor identifies that the wired connection with the device has been established and identifies that the position of the device is at least a prescribed distance away, the processor provides the user with a notification encouraging the user to pay attention to the use of the wearable terminal.

Description

Wearable device and information processing method

The present invention relates to a wearable device and an information processing method.

Technology for experiencing virtual reality (VR) by generating a virtual space and displaying it on a display is known. Augmented reality (AR) technology is also known, which superimposes virtual objects created using CG (Computer Graphics) or similar on images of real space. These technologies are widely used, and wearable devices worn by users are becoming increasingly common. One example is a head-mounted display (HMD), which is worn by the user on the head. HMDs can be used, for example, for gaming experiences and simulations.

Patent Document 1 discloses technology related to security when transmitting power wirelessly, and discloses technology in which, if a smartwatch determines that it is a predetermined distance away from a charging device, it determines that a third party has taken the charging device and issues a warning.

JP 2016-110342 A

A variety of devices, such as controllers, chargers, and earphones, may be connected to a wearable device via a wired cable. As an example, while the wearable device is in use (for example, while the user is moving around using the wearable device), the wired cable may be pulled unintentionally by the user, which may interfere with the use of the wearable device.

Therefore, there is a need to provide technology that enables good use of wearable devices when devices are connected to the wearable device via a wired connection and the wearable device is used by a user.

According to a first aspect of the present invention, the following wearable terminal is provided. This wearable terminal comprises a display, a wired connection interface, and a processor. The display displays a virtual space that may include virtual objects, an image of real space and virtual objects, or only an image of real space on a display screen. A wired cable is connected to the wired connection interface, which is used for a wired connection with a device. The processor then identifies the establishment of a wired connection with the device and, when it identifies that the processor is more than a predetermined distance away from the device's location, issues a notification to the user advising them to be careful about using the wearable terminal.

According to a second aspect of the present invention, the following information processing method is provided. This information processing method uses a wearable device that includes a display that displays on a display screen a virtual space that may include virtual objects, an image of real space and virtual objects, or only an image of real space; a wired connection interface that is connected to a wired cable and is used for a wired connection with a device; and a processor. This method includes the steps of: determining, by the processor, the establishment of a wired connection with the device; acquiring, by the processor, the distance from the device; and, if the processor identifies the establishment of a wired connection with the device and identifies that the wearable device is located a predetermined distance or more away from the device, issuing a notification to the user of the wearable device to warn them about use.

The present invention provides technology that enables effective use of a wearable device when a device is wired to the wearable device and the wearable device is used by a user. Note that other issues, configurations, and advantages will become clear from the description of the embodiments of the invention below.

1A and 1B are diagrams for explaining the external appearance of an example of a wearable terminal and an information processing system according to an embodiment of the present invention; 1 is a diagram illustrating an example of the positional relationship between a wearable terminal, a device connected via a wired connection, and a wired connection medium in a wearable terminal and an information processing system according to this embodiment. FIG. 10 is a diagram illustrating an example of a display screen in use assistance control for a wired connection. FIG. 10 is a diagram illustrating an example of a display screen in use assistance control for a wired connection. FIG. 10 is a diagram illustrating an example of a display screen in use assistance control for a wired connection. FIG. 10 is a diagram illustrating an example of a display screen in use assistance control for a wired connection. FIG. 10 is a diagram illustrating an example of a display screen in use assistance control for a wired connection. FIG. 10 is a diagram illustrating an example of a display screen in use assistance control for a wired connection. FIG. 10 is a diagram illustrating an example of a display screen in use assistance control for a wired connection. FIG. 2 is a diagram illustrating an example of a hardware configuration of a wearable terminal. 10A and 10B are diagrams illustrating an example of information data and an example of the functional configuration of a wearable terminal. FIG. 10 is a diagram illustrating an example of a display screen in use assistance control for a wired connection. FIG. 10 is a diagram illustrating an example of a display screen in use assistance control for a wired connection. FIG. 10 is a diagram illustrating an example of a display screen in use assistance control for a wired connection. FIG. 10 is a diagram illustrating an example of a display screen in use assistance control for a wired connection. FIG. 10 is a diagram illustrating an example of a display screen in use assistance control for a wired connection. FIG. 10 is a diagram illustrating an example of a display screen in use assistance control for a wired connection. FIG. 10 is a diagram illustrating an example of a display screen when a user is located inside a third distance. 10 is a flowchart illustrating an example of a processing flow of a wearable terminal or an information system. FIG. 10 is a diagram illustrating an example of a condition for starting use assistance control for a wired connection. FIG. 10 is a diagram illustrating an example of auxiliary control for use of a wired connection. FIG. 10 is a diagram illustrating an example of a condition for ending the use assistance control of a wired connection.

Embodiments of the present invention will be described below with reference to the drawings. The embodiments are examples for explaining the present invention, and have been omitted or simplified as appropriate for clarity of explanation. The present invention can also be implemented in various other forms. Unless otherwise specified, each component may be singular or plural.

In order to facilitate understanding of the invention, the position, size, shape, range, etc. of each component shown in the drawings may not represent the actual position, size, shape, range, etc. Therefore, the present invention is not necessarily limited to the position, size, shape, range, etc. disclosed in the drawings.

Although various types of information may be described using expressions such as "table," "list," and "queue," these types of information may also be expressed using other data structures. For example, various types of information such as "XX table," "XX list," and "XX queue" may also be expressed as "XX information." When describing identification information, expressions such as "identification information," "identifier," "name," "ID," and "number" are used, but these are interchangeable.

When there are multiple components with the same or similar functions, they may be described using the same reference numeral with different subscripts. Also, when there is no need to distinguish between these multiple components, the subscripts may be omitted.

In the embodiments, there may be described processes performed by executing a program. Here, a computer executes a program using a processor (e.g., a CPU or GPU), and performs the processes defined in the program using storage resources (e.g., memory) and interface devices (e.g., communication ports). Therefore, the entity that performs the processes by executing a program may be the processor. Similarly, the entity that performs the processes by executing a program may be a controller, device, system, computer, or node that has a processor. The entity that performs the processes by executing a program may be any computing unit, and may include a dedicated circuit that performs specific processing. Here, a dedicated circuit is, for example, an FPGA (Field Programmable Gate Array), ASIC (Application Specific Integrated Circuit), CPLD (Complex Programmable Logic Device), etc.

A program may be installed on a computer from a program source. The program source may be, for example, a program distribution server or a computer-readable storage medium. If the program source is a program distribution server, the program distribution server may include a processor and storage resources for storing the program to be distributed, and the processor of the program distribution server may distribute the program to be distributed to other computers. Furthermore, in an embodiment, two or more programs may be realized as one program, or one program may be realized as two or more programs.

In this embodiment, we describe a technology that enables good use of a wearable device when a device is wired to the wearable device and the user uses the wearable device by performing auxiliary control for wired connections.

First, an example of a wearable device and an information system will be described with reference to Figure 1. As shown in Figure 1, in this embodiment, the wearable device 100 is an immersive HMD that is worn on the head of the user 10 so as to cover the eyes of the user 10. The wearable device 100 (in this example, the HMD) includes, for example, gaze sensors (101, 102), a camera 103, a distance sensor 104, a positioning sensor 105, a microphone 106, a speaker 107, and a vibrator 108. Each component will be described in detail later.

The wearable device 100 can perform display operations based on, for example, a VR display mode in which a virtual space that may include virtual objects is displayed on the display screen, an AR display mode in which both an image of real space captured by the camera 103 and virtual objects are displayed on the display screen, or a captured image display mode in which only an image captured by the camera 103 is displayed on the display screen.

In the example of FIG. 1, a character 151, which is an example of a virtual object, is displayed on the display screen 150a. The wearable terminal 100 can also display various types of information. In this example, status information regarding the wired connection of the wearable terminal 100 is displayed by the wired connection usage assistance control. For example, a message 152 indicating that the wearable terminal 100 is currently connected to the external device 120 via a wired connection, and a graphic 153 (an icon in this example) are displayed.

Note that the type of information to be displayed is not limited to this example. Furthermore, the display format is not particularly limited. For example, the wearable terminal 100 may display information based on symbols and colors in addition to the message 152 and the graphic 153. Furthermore, the wearable terminal 100 may display information based on a combination of multiple of the message 152, graphic 153, symbols, colors, etc.

Also provided is an information system including a wearable terminal 100 and a device 120 that is wired to the wearable terminal 100. In this example, the wearable terminal 100 includes a connection unit 109 to which a wired cable 130 is connected. As an example, this connection unit 109 can be configured to include a wired connection input/output interface, which will be described later. The device 120 also includes a connection unit 121 to which the wired cable 130 is connected, forming an input/output interface. The wearable terminal 100 and the device 120 are connected via the wired cable 130 (wired connection medium).

As an example, device 120 is a monitor that outputs video processed by wearable terminal 100. However, any device that is wired to wearable terminal 100 may be used, and other types of devices may also be used in the information system. Device 120 may also be a charging device for wearable terminal 100, an audio input device connected to wearable terminal 100 that inputs external audio, an audio output device that outputs audio processed by wearable terminal 100, a camera separate from camera 103 of wearable terminal 100, a personal computer, etc.

Also, as an example, the wired device 120 is placed on the table 122. However, the device 120 may be placed in a location other than on the table 122 in real space. Note that the distance 140 shown in the figure is the distance between the device 120 and the user 10 in real space.

Next, an example of the usage status of a wearable device will be described with reference to Figure 2A. Also, an example of a display will be described with reference to Figures 2B-2F. Figure 2A shows an example of the relative positions of a wearable device, a device connected to the wearable device via a wired connection, and a wired cable.

As shown in FIG. 2A, a user 10 wearing an HMD 100 uses the HMD 100, for example, at a predetermined distance (i.e., the distance corresponding to the distance 140 described above) from a device 120 connected via a wired cable 130. Furthermore, before performing use assistance control of the wired connection, for example, a first distance 201, a second distance 203, and a third distance 205 are input by the user 10 to the HMD 100, and the HMD 100 sets these distances (201, 203, 205). These distances (201, 203, 205) indicate the predetermined distance from the device 120, based on the position of the device 120.

The first distance 201 is, for example, a distance set by the user 10 based on the judgment that a distance greater than this could cause inconvenience, such as being pulled by the connected wired cable 130. The second distance 203 is, for example, a distance longer than the first distance 201 but shorter than the maximum length to which the wired cable 130 can be extended. The third distance 205 is, for example, a distance shorter than the first distance 201 and closer to the device 120.

Therefore, the area 202 between the first distance 201 and the third distance 205 is an area where the user 10 is unlikely to feel annoyed by being pulled by the wired cable 130 when moving. In contrast, the area 204 between the first distance 201 and the second distance 203 is an area where the user 10 is likely to feel annoyed by being pulled by the wired cable 130 when moving. The area inside the third distance 205 is the area near the device 120 (in this example, the monitor).

As an example, when the HMD 100 identifies that the distance to the device 120 is equal to or greater than the first distance 201 (i.e., when the HMD 100 identifies that the distance 140 is equal to or greater than the first distance 201), the HMD 100 may perform use assistance control for a wired connection. Here, for example, when the user 10 faces the device 120, the HMD 100, in use assistance control for a wired connection, can display a wired connection area 211 in the field of view 206 of the camera of the HMD 100 when facing the device 120. Note that the wired connection area 211 relates, by way of example, to the area 204 between the first distance 201 and the second distance 203. Also, by way of example, it relates to the area within the first distance 201. Also, by way of example, it relates to the area within the second distance 203.

FIG. 2B shows an example of a display screen when displaying a wired connection area. In the example of FIG. 2B, a character 151 is displayed on the display screen 210b. Then, by executing the use assistance control for the wired connection, the HMD 100 displays, as an example, a wired connection area 211 within the field of view 206 on the display screen 210b. Note that the HMD 100 may also display an image of real space corresponding to the wired connection area 211 acquired using the camera 103. The HMD 100 may also generate and display a virtual object indicating the wired connection area 211.

On the other hand, for example, when the user 10 faces away from the device 120, the HMD 100, in the use assistance control of the wired connection, can display a wired connection area 221 in the field of view 207 of the camera of the HMD 100 when the user faces away from the device 120. Note that, like the wired connection area 211, the wired connection area 221 relates to the area 204 between the first distance 201 and the second distance 203, for example. Also, for example, it relates to the area within the second distance 203.

Figure 2C shows an example of a display screen when displaying a wired connection area. In the example of Figure 2C, a character 151 is displayed on the display screen 210c. Then, by executing the use assistance control for the wired connection, the HMD 100 displays, as an example, a wired connection area 221 within the field of view 207 on the display screen 210c. Note that the HMD 100 may also display an image of real space corresponding to the wired connection area 221 acquired using the camera 103. The HMD 100 may also generate and display a virtual object indicating the wired connection area 221.

Therefore, by making the user 10 aware of the wired connection area (211, 221) that takes into account the length of the connected wired cable 130, it is possible to help prevent annoyances that the user 10 may experience, such as the user 10 being pulled by the wired cable 130 (wired connection medium).

As an example, when the HMD 100 identifies that it is a predetermined distance or more from the device 120, it may provide assistance by notification as a control to assist in the use of a wired connection. The HMD 100 can display, for example, information on the display screen regarding the direction toward the connection unit 121 of the device 120, the position of the connection unit 121 of the device 120, the distance between the HMD 100 or the user 10 and the device 120, and the length of the wired cable 130.

Figure 2D shows an example of a display screen including notification information. In this example, a character 151 is displayed on the display screen 210d. The HMD 100 then displays, as auxiliary controls for use of the wired connection, a direction mark 231 indicating the direction toward the monitor's connection point, a position mark 232 indicating the position of the monitor's connection point, a message 233 indicating the distance between the HMD 100 or user 10 and the monitor, and a message 234 indicating the length of the wired cable 130, "wired cable length 200 cm." Note that real-space objects such as the monitor, table, and wired cable, shown with thin dashed lines, do not need to actually be displayed on the display screen 210d. In Figure 2D, the real-space objects are illustrated to clarify their positions on the display screen.

By allowing the user 10 to recognize notification information such as the direction of the wired connected device 120, the position of the device 120, the distance between the device 120 and the HMD 100 or the user 10, and the length of the wired cable 130, it is possible to help prevent events that the user 10 finds annoying, such as the user 10 being pulled by the wired cable 130.

As an example, when the HMD 100 determines that it is at a certain distance from the device 120, it may provide assistance in notifying the user of the real-space situation as a control to assist in using a wired connection. The HMD 100 can, for example, display on the display screen an object in the real space around the HMD 100 that has been photographed by the camera 103, or a virtual object of this object.

Figure 2E shows an example of a display screen. In this example, a character 151 is displayed on the display screen 210e. In use assistance control for a wired connection, the HMD 100 may also display an object in real space captured by the camera 103, or a virtual object of this object. In this example, in use assistance control for a wired connection, the HMD 100 displays a table 122, a monitor which is an example of a device 120, and a wired cable 130.

By allowing the user 10 to recognize the situation in real space, it is possible to help prevent events that the user 10 finds annoying, such as the user 10 being pulled by the wired cable 130.

As another example, the HMD 100 may provide assistance in notifying the user of the status of the wired cable 130 as a means of assisting in the use of a wired connection. The HMD 100 may, for example, display the wired cable 130 on a display screen.

Figure 2F shows an example of a display screen. In this example, a character 151 is displayed on the display screen 210f. In use assistance control for a wired connection, the HMD 100 may display an image of the wired cable 130 captured by the camera 103, or a virtual object 241 of this wired cable 130. Note that real-space objects such as the monitor and table shown with thin dashed lines do not need to be actually displayed on the display screen 210f. In Figure 2F, the monitor and table are illustrated to clarify their positions on the display screen.

As shown in the figure, the HMD 100 may display the wired cable 130 or the virtual object 241 of the wired cable 130 on the display screen 210f so that it corresponds to its position in real space. However, the HMD 100 may also display the wired cable 130 or the virtual object 241 of the wired cable 130 on the display screen 210f so that it does not correspond to its position in real space. The HMD 100 may, for example, display the wired cable 130 or the virtual object 241 of the wired cable 130 on the edge of the display screen 210f so as not to interfere with the user 10's viewing. Also, for example, an area on the display screen 210f for displaying the wired cable 130 or the virtual object 241 of the wired cable 130 may be set in advance, and the HMD 100 may adjust the size of the displayed wired cable 130 or the virtual object 241 of the wired cable 130 so that it fits within this area.

Therefore, by allowing the user 10 to recognize the state of the wired cable 130 (for example, the shape of the wired cable, whether the wired cable is bent, whether the wired cable is tangled, etc.), it is possible to help prevent events that the user 10 finds annoying, such as being pulled by the wired cable 130. Alternatively, it is possible to help the user 10 immediately recognize and deal with events that become annoying, such as being pulled by the wired cable 130.

Furthermore, in regard to notification of the state of the wired cable 130, the HMD 100 may display, for example, as shown in Figures 2G-2H. Figure 2G shows an example of a display screen. In this example, a character 151 is displayed on the display screen 210g. The HMD 100 may also display information indicating that the wired cable 130 is taut and information indicating that pressure is being applied to the connection terminal in the use assistance control of the wired connection. In this example, the HMD 100 displays, on the display screen 210g, a figure 261 (icon) and a message 262 indicating that the wired cable 130 is taut, and a figure 263 (icon) and a message 264 indicating that pressure is being applied to the connection terminal.

Therefore, by allowing the user 10 to recognize that the wired cable 130 is taut and that pressure is acting on the connection terminal, it is possible to help prevent events that the user 10 finds annoying, such as being pulled by the wired cable 130. Alternatively, it is possible to help the user 10 immediately recognize and take action when an annoying event occurs, such as being pulled by the wired cable 130.

FIG. 2H shows an example of a display screen. In this example, a character 151 is displayed on the display screen 210h. The HMD 100 may also display information indicating that the wired cable 130 has been detached during use assistance control for a wired connection. In this example, a figure 271 (icon) and a message 272 indicating that the wired cable 130 has been detached are displayed on the display screen 210h.

Therefore, by making the user 10 aware that the wired cable 130 has been disconnected, it is possible to assist the user 10 in using the HMD 100.

Next, an example of the configuration of a wearable device will be described with reference to Figure 3A. As described above, the wearable device 100 (e.g., HMD) includes gaze sensors (101, 102), a camera 103, a distance sensor 104, a positioning sensor 105, a microphone 106, a speaker 107, and a vibrator 108.

The gaze sensors (101, 102) include, for example, a left eye gaze sensor 101 and a right eye gaze sensor 102 for the right eye. The left eye gaze sensor 101 detects the left eye of the user 10 and can capture the viewpoint of the left eye at the gaze point by detecting the movement of the left eye, movement in the gaze direction, etc. The right eye gaze sensor 102 detects the right eye of the user 10 and can capture the viewpoint of the right eye at the gaze point by detecting the movement of the right eye, movement in the gaze direction, etc.

The wearable terminal 100 may perform predetermined processing using gaze information acquired from the gaze sensors (101, 102). The wearable terminal 100 may, for example, display an input operation screen such as an operation menu on the display 308. The wearable terminal 100 may then acquire the position of an icon, button, etc. on the input operation screen at which the user 10 is looking as input operation information, and perform predetermined processing.

The process for detecting eye movement can utilize well-known technology commonly used for eye tracking. For example, in a method using corneal reflex, infrared light is directed at the face from an infrared LED (Light Emitting Diode), and an image is taken with an infrared camera, an example of an eye gaze sensor. Then, using the light reflection point on the cornea as a reference point, eye movement (eye direction) can be estimated from changes in eye position and other geometric features.

The camera 103 captures a captured image by converting light incident from the lens into an electrical signal using an imaging element such as a CCD image sensor or CMOS image sensor. The camera 103 can be positioned so that it can capture images of the area around the wearable device 100. Based on the captured images captured by the camera 103, the wearable device 100 can acquire information such as the position of the wired connected device 120, the position of the connection part 121 of the wired cable 130 in the wired connected device 120, and the position and status of the wired cable 130. Note that there are no particular restrictions on the placement, orientation, number, viewing angle, etc. of the cameras 103 as long as appropriate processing can be performed.

The ranging sensor 104 is a sensor that can measure the distance from the wearable device 100 to an object, the object's position, and capture the three-dimensional shape of the object. Examples of ranging sensors that can be used include LiDAR (Light Detection and Ranging), which irradiates an object with laser light such as infrared light and measures the scattered light that bounces back; TOF (Time of Flight) sensors, which measure distance by measuring the reflection time of pulsed light irradiated on the subject for each pixel; and millimeter-wave radar, which emits millimeter-wave radio waves and captures the reflected waves. The ranging sensor 104 may also be a sensor that performs measurements based on the angle at which the light reflected by the object is received. In other words, the ranging sensor 104 may be a triangulation sensor. The ranging sensor 104 may also be a stereo camera that performs measurements based on parallax images. The wearable device 100 processes the measurement results from the distance measurement sensor 104 and can obtain information such as the position of the wired connection destination device 120, the position of the connection part 121 of the wired cable 130 on the wired connection destination device 120, and the position and status of the wired cable 130.

The positioning sensor 105 is, for example, a GPS (Global Positioning System) receiver. In this case, by receiving satellite radio waves from GPS satellites and performing positioning calculations, it is possible to determine, for example, the coordinates on the Earth at which the wearable device 100 is located. The positioning calculations may be performed by the GPS receiver, or by other components (such as the processor 320 of the wearable device 100 or an external device). The wearable device 100 may also acquire location information using other methods. For example, the wearable device may determine its current location by analyzing images captured by the camera 103 and/or three-dimensional point cloud data measured by the ranging sensor 104; techniques such as SLAM (Simultaneous Localization and Mapping) are known. If location information is acquired without using the positioning sensor 105, the positioning sensor 105 may be omitted.

Furthermore, technologies such as VPS (Visual Positioning System) are known that identify location and the like by analyzing camera images. The wearable device 100 may acquire location information by VPS using images captured by the camera 103. The wearable device 100 may also transmit the images captured by the camera 103 to an external device. The external device may then use the captured images to acquire location information based on the VPS and distribute the acquired location information to the wearable device 100. Note that if location information is acquired without using the positioning sensor 105, the positioning sensor 105 may be omitted.

The wearable device 100 may also acquire location information by Pedestrian Dead Reckoning (PDR), for example, using a sensor described below. If location information is acquired without using the positioning sensor 105, the positioning sensor 105 may be omitted.

The microphone 106 collects the user's 10 own speech, external sounds, and other sounds, and converts them into audio data. The user 10 may speak to give instructions such as input operations, and the wearable device 100 may acquire the audio data collected by the microphone 106 as instruction information for input operations, etc., and execute predetermined processing.

The speaker 107 outputs audio based on the audio data, and can, for example, notify the user 10 of notification information by audio. The wearable device 100 can, for example, issue a warning message such as "Connected via wire!" by audio from the speaker 107 to notify the user 10. By providing audio notifications, usability can be improved.

The vibrator 108 is a device that generates vibrations under the control of the processor 320, and converts notification information sent by the wearable device 100 to the user 10 into vibrations. The vibrator 108 transmits vibrations to the user 10 wearing the wearable device 100, allowing the user 10 to be notified of notification information such as a warning message such as "Wired connection in progress!" or the display operation of the wired connection area, thereby improving usability.

The wearable terminal 100 also includes an acceleration sensor 304, a gyro sensor 305, and a geomagnetic sensor 306. The acceleration sensor 304 can detect, for example, the tilt and direction of the wearable terminal 100. The gyro sensor 305 is a sensor that detects angular velocity or angular acceleration in the rotational direction, and can capture the posture of an object. Therefore, the acceleration sensor 304 and gyro sensor 305 can be used to detect the posture, such as the tilt and direction, of the wearable terminal 100. The geomagnetic sensor 306 is a sensor that detects the Earth's magnetic force, and detects the direction in which the wearable terminal 100 is facing. It is also possible to detect the movement of the wearable terminal 100 by using a three-axis type that detects geomagnetic fields in the up and down directions in addition to the front-back and left-right directions, and by capturing changes in geomagnetic fields in response to the movement of the wearable terminal 100.

The wearable device 100 can use these sensors to acquire information about the posture and movements of the user 10. The wearable device 100 can then use the acquired information to understand the state of the user 10, and as a result, can analyze with greater precision the position and state of the wired connected device 120, the position and state of the wired cable 130, and so on.

The wearable terminal 100 also includes an operation input interface 307 and a display 308. The operation input interface 307 is an interface that realizes input means, such as gaze, hands, a pointer, or even operating devices separate from the wearable terminal 100, such as a keyboard, key buttons, or touch keys, and the user 10 inputs the information they wish to input via the operation input interface 307.

The wearable terminal 100 may display an input operation screen such as an operation menu on the display screen of the display 308, take in the position on the input operation screen at which the user 10 is looking as input operation information, and perform an operation according to the input operation information. The wearable terminal 100 may also display a cursor such as a hand or pointer on the input operation screen, take in operations performed by the user 10 using the cursor as input operation information, and perform an operation according to the input operation information. The wearable terminal 100 may also take in gestures made by the user 10 as input operation information, and perform an operation according to the input operation information. In addition, when input is made using an input operation device such as a keyboard, key buttons, or touch keys, the input operation device may be connected to the wearable terminal 100 main body using a wired cable 130, or may be connected wirelessly.

Examples of user 10 input operations include setting the first, second, and third distances (201, 203, 205) related to assistive control of wired connections, initial settings for warning messages (152, 234, 262, 264, 272), and settings to disable control when connected via wire. Other examples of user 10 input operations include application operations, volume adjustments, and various other setting operations.

The display 308 can be configured using a display such as an LCD display, a touch panel display, or an organic EL display, and displays virtual objects, camera-captured images, etc. For example, if the wearable device 100 is an immersive HMD, the display 308 can be configured using a LCD panel, a display that displays holographic images, etc., and displays virtual objects, camera-captured images, etc.

The wearable terminal 100 also includes a processor 320, memory 330, a wired connection input/output interface 345, a short-range wireless communication interface 346, and a network communication interface 347. The processor 320 is the main body that executes predetermined processes and is configured using a CPU (Central Processing Unit) and the like. As shown in the figure, each component of the wearable terminal 100 is connected to the processor 320 via a bus 350. The processor 320 executes an operating system (OS) 333 and application programs 334 for operation control and the like stored in the memory 330, thereby controlling each component of the wearable terminal 100 and realizing the functions of the OS, middleware, applications, etc.

Memory 330 is composed of a main memory device, a non-volatile memory device, etc., and stores various programs 331 and information data 332 handled by processor 320, etc. An example of information data 332 will be described later. Memory 330 stores, as programs 331, for example, an operating system 333, and various application programs 334 for controlling operations used by processor 320, etc. Memory 330 may also store various settings, settings made by user 10, etc.

The wired connection input/output interface 345 (wired connection interface) is configured for wired connection and is connected to the wired connection destination device 120 via the wired cable 130. In the same figure, the wired connection destination device 120 is, as an example, an information processing device 360 (wired connection destination device), and the wired connection input/output interface 345 is an input/output interface that performs wired communication of information with the wired connection destination information processing device 360 via the wired cable 130. Note that a charging device or the like may be connected as the wired connection destination device 120, and the wired connection input/output interface 345 may be an input/output interface that inputs or outputs power via the wired cable 130. The wired connection input/output interface 345 may also be configured to be connected via Power of Ethernet (PoE; Ethernet is a registered trademark).

Devices 120 that can be connected via a wired connection using the wired connection input/output interface 345 include display devices such as monitors, information processing devices such as personal computers, and charging devices, as well as audio devices such as earphones or loudspeakers, and input operation devices such as keyboards, key buttons, and touch keys, as shown in FIG. 1.

The wired cable 130 is a connection cable to the device 120 that is the wired connection destination. More specifically, the wired cable 130 is a connection cable such as a charging cable used to connect to a charging device, a twisted pair cable used with POE, a cable for wired earphones, or an audio cable. Furthermore, if the information processing device 360 is a personal computer, the wired cable 130 is, for example, a LAN cable.

The short-range wireless communication interface 346 is a communication interface that performs short-range wireless communication with a device (in this example, the information processing device 360) within the range where short-range wireless communication is possible. Short-range wireless communication is performed using, for example, an electronic tag, but is not limited to this. If the wearable terminal 100 is located near a device that is at least capable of wireless communication, short-range wireless communication may be performed using Bluetooth (registered trademark), IrDA (Infrared Data Association, registered trademark), Zigbee (registered trademark), HomeRF (Home Radio Frequency, registered trademark), or wireless LAN (IEEE802.11a, IEEE802.11b, IEEE802.11g).

The network communication interface 347 is a communication interface that performs wireless communication with an external information server 380 or the like via an external network 381 using base station communication or the like. The network communication interface 347 can receive various information stored in the information server 380. Note that long-distance wireless communication such as W-CDMA (Wideband Code Division Multiple Access) or GSM (Registered Trademark) (Global System for Mobile communications) can be used for base station communication.

Next, an example of information data and an example of the functional configuration of a wearable device will be described with reference to Figure 3B.

As shown in FIG. 3B, the memory 330 can store, as information data 332, image information 335, ranging information 336, location information 337, wired connection area information 338, virtual space information 339, wired connection medium status information 340, wired connection status information 341, and video information 342, for example.

Image information 335 is information relating to images acquired by the wearable device 100, and includes, for example, information relating to images of real space acquired by the camera 103 and information relating to images acquired by the distance measurement sensor 104. Note that image information 335 may also include information relating to images used to acquire position information of the wearable device 100.

The ranging information 336 is information relating to the distance to an object around the wearable device 100, and includes, for example, information relating to the distance to the object measured by the ranging sensor 104. The ranging information 336 may also include information relating to the position of the object obtained using the ranging sensor 104. The ranging information 336 may also include information relating to the shape of the object obtained using the ranging sensor 104.

Location information 337 is information about the location of the wired connected device 120 and information about the location of the wearable device 100 (i.e., the location of the user 10). The location information 337 is acquired by the wired connection destination device/wearable terminal location information processing unit 321, which will be described later.

Wired connection area information 338 is information about the wired connection area, which is an area where a connected wired cable 130 may be present. As an example, as described in the example display screen above, wired connection area information 338 may include information about the area between first distance 201 and second distance 203, which is an area where the wired cable 130 is likely to be an annoyance. Also, as an example, wired connection area information may include information about the area within first distance 201. Also, as an example, it may include information about the area within second distance 203. Also, as an example, it may include information about the area between first distance 201 and third distance 205. Wired connection area information 338 is acquired by wired connection area setting processing unit 323, which will be described later.

The virtual space information 339 is information about the virtual space and virtual objects displayed by the wearable terminal 100. The virtual space information 339 may be generated by the processor 320 of the wearable terminal 100 executing a program, application software, or the like. The virtual space information 339 may also be information generated by an external device (for example, an information server 380) and acquired by the wearable terminal 100 via communication.

Wired connection medium status information 340 is information regarding the status of the wired cable 130. Wired connection medium status information 340 is acquired by the wired connection medium status identification processing unit 326, which will be described later.

Wired connection status information 341 is information indicating whether the wired cable 130 is connected to the device 120. In other words, wired connection status information 341 indicates whether the wired cable 130 is disconnected. Wired connection status information 341 is acquired by the wired connection status identification processing unit 327, which will be described later.

Video information 342 is information relating to video captured using camera 103. Video information 342 includes video captured by camera 103 of connection part 121 of wired device 120. This video is video showing the disconnection of wired cable 130, and includes, for example, a scene in which wired cable 130 is disconnected, a scene when connected wired cable 130 is disconnected, etc.

The timing for starting to acquire video information may be, for example, when the wired cable 130 is detached. That is, the wearable device 100 may start capturing this video (playback video) when the wired cable 130 is unplugged. The timing for acquiring video information may also be, for example, when the wired cable 130 reaches a predetermined state. That is, the wearable device 100 may start capturing this video (playback video) when, for example, it acquires information that a predetermined amount of pressure or more has been applied to the wired cable 130, the connection terminal of the wiredly connected device 120, and/or the connection terminal of the wearable device 100. Note that the wearable device 100 may stop recording a predetermined time after the wired cable 130 is detached from the connection terminal.

As shown in FIG. 3B, the wearable terminal 100 includes, as an example, a wired connection destination device/wearable terminal position information processing unit 321, an icon/message notification processing unit 322, a wired connection area setting processing unit 323, a captured image display processing unit 324, a virtual space generation and display processing unit 325, a wired connection medium status identification processing unit 326, a wired connection status identification processing unit 327, and a video generation and display processing unit 328. These components are realized by the processor 320 storing data such as programs in the main storage device of the memory 330 and processing the data. Note that other components (sensors, interfaces, etc.) may also be used to realize these components.

The wired connection destination device/wearable terminal location information processing unit 321 acquires the position of the wired connection destination device 120 and the position of the wearable terminal 100. The wired connection destination device/wearable terminal location information processing unit 321 acquires the position of the wired connection destination device 120, for example, using image information 335 and/or distance measurement information 336. The wired connection destination device/wearable terminal location information processing unit 321 may also acquire the position of the wired connection destination device 120, for example, by communicating with the wired connection destination device 120.

The wired connection destination device/wearable terminal location information processing unit 321 may, for example, obtain the position of the wearable terminal 100 using the positioning sensor 105. The wired connection destination device/wearable terminal location information processing unit 321 may also obtain the position of the wearable terminal 100 using, for example, technology related to self-position estimation, VPS technology, etc.

The wired connection destination device/wearable terminal location information processing unit 321 can store information regarding the location of the wired connection destination device 120 and the location of the wearable terminal 100 in the memory 330 as location information 337.

For example, in assistive use control for a wired connection, the icon/message notification processing unit 322 generates an icon and/or message for notification information and outputs it to the display 308 to notify the user 10. The icon/message notification processing unit 322 may also notify the user 10 of the content based on the notification information by generating audio data related to the message and outputting the audio data. The icon/message notification processing unit 322 may also notify the user 10 of the content based on the notification information by generating vibration data corresponding to the content of the message and vibrating the vibrator 108 based on the generated vibration data.

The wired connection area setting processing unit 323 sets a wired connection area based on the information about the set distances (201, 203, 205) and stores the set details in memory 330 as wired connection area information 338. The wired connection area setting processing unit 323 sets the area within the second distance 203 as wired connection area information 338 and stores the wired connection area information 338 in memory 330.

When displaying virtual objects or the like in an AR display format, the captured image display processing unit 324 uses image information 335 to generate an image captured by camera 103 as a background and displays it on display 308. Furthermore, when performing a captured image display format in which only images captured by camera 103 are displayed on the display screen, the captured image display processing unit 324 uses image information 335 to display the image captured by camera 103 on display 308.

In addition, the captured image display processing unit 324 uses image information 335 to display images of surrounding objects (such as the wired cable 130) on the display 308 during auxiliary control of wired connection use.

When displaying virtual objects, etc. in an AR display format, the virtual space generation and display processing unit 325 uses virtual space information 339 to generate the virtual objects to be displayed and displays them on the display 308. When displaying virtual spaces, virtual objects, etc. in a VR display format, the virtual space generation and display processing unit 325 uses virtual space information 339 to generate the virtual spaces, virtual objects, etc. to be displayed and displays them on the display 308.

Furthermore, in assistive control of wired connections, the virtual space generation and display processing unit 325 uses image information 335 and the like to generate virtual objects of surrounding objects (such as the wired cable 130) and displays the virtual objects on the display 308. Furthermore, in assistive control of wired connections, the virtual space generation and display processing unit 325 generates virtual objects such as direction marks and position marks and displays the virtual objects on the display 308.

The wired connection medium state identification processing unit 326 acquires wired connection medium state information 340 regarding the state of the wired cable 130 and stores it in memory 330. The wired connection medium state identification processing unit 326 uses, for example, image information 335, distance measurement information 336, etc. to identify the shape of the wired cable 130, and detects whether the wired cable 130 is taut, whether the wired cable 130 is slack, whether the wired cable 130 is tangled, etc., to acquire the wired connection medium state information 340. The wired connection medium state identification processing unit 326 may also compare the acquired information regarding the length of the wired cable 130 with information regarding the distance between the wired connection destination device 120 and the wearable terminal 100 or user 10 to detect whether the wired cable 130 is taut, whether the wired cable 130 is slack, etc., and acquire the wired connection medium state information 340.

The wired connection medium status identification processing unit 326 may also acquire information regarding whether pressure is being applied to the wired cable 130, the connection terminal of the wired connected device 120, and/or the connection terminal of the wearable terminal 100, and store this information in the memory 330.

As an example, the pressure at each connection terminal is detected by a piezoelectric element incorporated in each connection terminal. The wearable terminal 100 acquires the pressure at the connection terminal on the device 120 side, for example, via the wired cable 130. The wearable terminal 100 also acquires the pressure at the connection terminal on the device 120 side, for example, by communicating with the device 120 (wired communication or wireless communication). The wearable terminal 100 acquires the pressure on the wearable terminal 100 side from the connection terminal of the wearable terminal 100. The pressure on the wired cable 130 can be, for example, the same as the pressure applied to the connection terminal. A piezoelectric element may be provided in the connection terminal of the wired cable 130, and the wearable terminal 100 may acquire the pressure applied to the connection terminal of the wired cable 130 from this piezoelectric element.

The wired connection status identification processing unit 327 acquires wired connection status information 341 related to the attachment and detachment of the wired cable 130 and stores it in the memory 330. The wired connection status identification processing unit 327 may, for example, identify the presence or absence of input/output of data, etc. via the wired connection input/output interface 345, and acquire the wired connection status information 341. In other words, the wired connection status identification processing unit 327 may, for example, acquire wired connection status information 341 indicating that the wired connection has been disconnected when input/output of data, signals, power, etc. is not possible between the wearable terminal 100 and the device 120 via the wired connection input/output interface 345.

The wired connection status identification processing unit 327 may also use image information 335, distance measurement information 336 indicating the shape of objects, and the like, around the connection unit 121 to identify whether the wired cable 130 is connected and acquire wired connection status information 341. The wired connection status identification processing unit 327 may also acquire information regarding whether a wired connection has been established from the device 120 via communication.

The video generation and display processing unit 328 performs processing to output a video based on the video information 342. For example, in assistive control of wired connection use, the video generation and display processing unit 328 uses the video information 342 to generate a video including, for example, a scene in which the wired cable 130 is disconnected and/or a scene when the connected wired cable 130 is disconnected, and outputs the video to the display 308.

Next, we will explain display examples for various usage situations with reference to Figures 4 to 7A. Figures 4 to 7A show examples of display screens for use assistance control for wired connections.

In the example of Figure 4, the portion of the wired cable 130 enclosed by the encircling line 401 is in an area outside the first distance 201, a virtual object of the character 151 is displayed on the display screen 150b, and the user 10 is using the HMD 100 in the area 202 between the first distance 201 and the third distance 205. Even when the user 10 is located in the area 202 between the first distance 201 and the third distance 205, a portion of the wired cable 130 is located outside the first distance 201. Therefore, in the use assistance control for the wired connection, the HMD 100 displays a message and/or an icon indicating that a wired connection is in progress on the display screen 150b, for example, by executing the icon/message notification processing unit 322. For example, the display screen 150b displays or re-displays an icon 153 indicating that a wired connection is in progress. Note that the icon 153 may be highlighted based on factors such as color or size. Additionally, for example, a warning message 152 saying "Wired connection in progress!" is displayed on the display screen 150b.

This prevents the user 10 from feeling annoyed, for example, by being pulled by the wired cable 130 that extends beyond the first distance 201.

In the example of Figure 5, the user 10 is facing away from the wired device 120 (in this example, a monitor), and a character virtual object 151 is displayed on the display screen 210i. In use assistance control for a wired connection, the HMD 100 may display the wired cable 130 or a virtual object of the wired cable 130 on the display screen so as to correspond to a position in real space; however, as shown in Figure 5, when the user 10 faces away from the device 120, the wired cable 130 is not within the field of view 501 of the user 10, and it may not be possible to display the wired cable 130 or a virtual object of the wired cable 130 on the display screen so as to correspond to a position in real space.

5, the HMD 100 may not display the wired cable 130 or a virtual object of the wired cable 130 to correspond to a position in real space during use assistance control for a wired connection. In such a situation, the HMD 100 may generate a virtual object 502 of the wired cable 130 by executing the virtual space generation and display processing unit 325, and display the virtual object 502 of the wired cable 130 at an appropriate position on the display screen 210i. Note that the display position may be, for example, on the edge of the display screen 210i so as not to interfere with the user 10's viewing of content, etc. Furthermore, the HMD 100 may generate and display a virtual object of the entire wired cable 130, or may generate a virtual object 502 of a portion of the wired cable 130 (for example, a portion nearby that is outside the field of view) and display it on the display screen 210i.

As a result, even if the actual wired cable 130 does not appear on the display screen within the user's field of view, the user 10 can recognize the presence of the wired cable 130 as a virtual object. This allows the user 10 to sense any annoying events, such as being pulled by the wired cable 130, and can prevent such events from occurring.

In the example of FIG. 6A, the user 10 is using the HMD 100 with the distance between the device 120 (in this example, a monitor) and the HMD 100 large and the wired cable 130 taut. In use assistance control for a wired connection, the HMD 100 can display the wired cable 130 or a virtual object of the wired cable 130 on the display screen so as to correspond to a position in real space. In this example, the HMD 100 generates a virtual object 603 of the wired cable 130 by executing the virtual space generation and display processing unit 325 and displays it on the display screen 210j.

Furthermore, the HMD 100 executes the virtual space generation and display processing unit 325 to generate and display virtual objects: a direction mark 601 indicating the direction toward the connection unit 121 of the device 120, and a position mark 602 indicating the position of the connection unit 121 of the device 120. Furthermore, the HMD 100 executes the icon/message notification processing unit 322 to display on the display screen 210j a message 262 and/or an icon 261 indicating that the wired cable 130 is taut. For example, the display screen 210j displays or re-displays the icon 261 indicating that the wired cable 130 is taut. Note that the icon 261 may be highlighted based on factors such as color and size. For example, the display screen 210j displays a warning message 262 stating "A wired cable is taut!"

This allows the user 10 to recognize that the wired cable 130 is taut. This helps the user 10 to immediately notice and deal with any annoying events, such as being pulled by the wired cable 130. Note that the table and equipment (in this example, a monitor) on the display screen 210j shown in the figure are shown to clarify their positions on the display screen 210j, and do not actually need to be displayed.

In the example of FIG. 6B, the user 10 is using the HMD 100 with the wired cable 130 caught on the object 610 and pressure acting on the connection terminal. In the use assistance control of the wired connection, the HMD 100 can display the wired cable 130 or a virtual object of the wired cable 130 on the display screen so as to correspond to a position in real space. In this example, the HMD 100 executes the virtual space generation and display processing unit 325 to generate a virtual object 604 of the wired cable 130 caught on the object 610 and displays it on the display screen 210k.

Furthermore, as in the case of FIG. 6A, the HMD 100 executes the virtual space generation and display processing unit 325 to display a direction mark 601 pointing toward the device 120 and a position mark 602 indicating the connection part 121 of the device 120, and executes the icon/message notification processing unit 322 to display an icon and/or message. For example, an icon 263 indicating that pressure is being applied to the connection terminal is displayed or re-displayed on the display screen 210k. Note that the icon 263 may be highlighted based on color, size, etc. Also, for example, a warning message 264 stating "Pressure is being applied to the connection terminal!" is displayed on the display screen 210k.

This allows the user 10 to recognize that pressure is being applied to the connection terminal. This allows the user 10 to sense an annoying event, such as being pulled by the wired cable 130, and can prevent this from happening. Alternatively, it can help the user 10 to immediately sense an annoying event, such as being pulled by the wired cable 130, and take action. Note that the table, equipment (in this example, a monitor), and objects on the display screen shown in the same figure are illustrated to clarify their positions on the display screen, and do not actually need to be displayed.

6C, the user 10 is using the HMD 100 with a tangled portion 605 in the wired cable 130. In the use assistance control of a wired connection, the HMD 100 can display the wired cable 130 or a virtual object of the wired cable 130 on the display screen so as to correspond to a position in real space. In this example, the HMD 100 generates a virtual object 607 of the wired cable 130 by executing the virtual space generation and display processing unit 325, and displays it on the display screen 210l. Note that this virtual object 607 includes a tangled portion 606 that corresponds to the tangled portion 605 in real space. The HMD 100 may also display an icon and/or a message indicating that the wired cable 130 is tangled by executing, for example, the icon/message notification processing unit 322.

This allows the user 10 to recognize that the wired cable 130 has become tangled. The user 10 can then sense an annoying event, such as being pulled by the wired cable 130, and can prevent this from happening. Note that the table and equipment (in this example, a monitor) on the display screen shown in the figure are shown to clarify their positions on the display screen, and do not actually need to be displayed.

In a situation such as that shown in Figure 6C, the wearable device 100 (e.g., HMD) may identify the tangled portion 605 of the wired cable 130, and the wired connection medium state identification processing unit 327 may identify the state of the wired cable 130 excluding that portion. For example, if the wired cable 130 is tangled and taut, the HMD 100 may identify the tangled portion 605 of the wired cable 130, and the wired connection medium state identification processing unit 327 may identify the state of the wired cable 130 as being straight.

FIG. 7A shows an example of a situation in which the wired cable 130 connected to the device 120 becomes disconnected when the user 10 is using the HMD 100 at a position greater than the third distance 205. In the use assistance control of the wired connection, the HMD 100 can display the wired cable 130 or a virtual object of the wired cable 130 on the display screen so as to correspond to a position in real space. In this example, the HMD 100 generates a virtual object 703 of the wired cable 130 by executing the virtual space generation and display processing unit 325 and displays it on the display screen 210m.

Furthermore, the HMD 100 executes the virtual space generation and display processing unit 325 to display a direction mark 701 pointing toward the device 120 and a position mark 702 indicating the connection part 121 of the device 120, and executes the icon/message notification processing unit 322 to display an icon and/or message. For example, an icon 271 indicating that the wired cable 130 has been disconnected is displayed or re-displayed on the display screen 210m. Note that the icon 271 may be highlighted based on color, size, etc. Also, for example, a warning message 272 stating "The wired cable has been disconnected!" is displayed on the display screen 210m.

This allows the user 10 to recognize that the wired cable 130 has been disconnected. Note that the table and equipment (in this example, a monitor) on the display screen 210m shown in the figure are shown to clarify their positions on the display screen 210m, and do not actually need to be displayed.

Furthermore, the HMD 100 may display a video (playback video) showing the detachment of the wired cable 130 on the display screen. In the use assistance control of the wired connection, the HMD 100 can execute the video generation and display processing unit 328 to display a video 704 including the situation in which the wired cable 130 has been detached on the display screen 210n. By viewing the video 704, the user 10 can recognize the situation in which the wired cable 130 has been detached.

Note that the user 10 may set the video playback time, which is the time from the start to the end of the video, and the video playback display processing unit 328 may display the video 704 showing the situation in which the wired cable 130 has been disconnected, for the video playback time set by the user 10. Alternatively, the video playback time may be determined automatically, and the video playback display processing unit 328 may display the video 704 showing this situation for the automatically determined video playback time. In the example of FIG. 7A, the video generation and display processing unit 328 generates a 15-second video 704 including the situation in which the wired cable 130 has been disconnected, based on the video playback time. Note that the video 704 displays objects around the connection unit 121 of the device 120, and in the example of the display screen 210n in FIG. 7A, a table, the device 120 (in this example, a monitor), and the wired cable 130 are displayed.

Furthermore, the display method is not particularly limited, but for example, the user 10 may select a display related to display screen 210m or a display related to display screen 210n, input the selection, and the wearable terminal 100 (in this example, an HMD) may display display screen 210m or display screen 210n based on the user's selection. On the other hand, the wearable terminal may automatically transition between display screen 210m and display screen 210n. Note that the wearable terminal 100 may simultaneously display display screen 210m and display screen 210n on the display 308.

With these display screens (210m, 210n), if a problem occurs with a wired connection, for example, the user 10 can accurately identify the problem and receive assistance in easily resolving the problem.

Next, with reference to Figure 7B, an example of processing when the user 10 is located inside the third distance 205 will be described. Figure 7B shows an example of a display screen when the user 10 is located inside the third distance 205. If the wired cable 130 becomes unplugged while the user 10 is located within the third distance 205, it is assumed that the user 10 intentionally unplugged the wired cable 130 from the device 120. Therefore, the wearable terminal 100 (in this example, the HMD) does not display information indicating that the wired cable 130 has become unplugged on the display screen 210o. The third distance 205 is a criterion for determining that the user 10 intentionally unplugged the wired cable 130.

Next, an example of the processing flow of the wearable terminal 100 or the information system will be described with reference to Figure 8. The processor 320 of the wearable terminal 100 confirms the establishment of a wired connection between the wearable terminal 100 and the device 120 (S801). Then, while the user 10 is using the wearable terminal 100, the processor 320 determines whether the conditions for executing usage assistance control for a wired connection (usage assistance control during a wired connection) are met (S802). Then, if the processor 320 determines that the conditions for executing usage assistance control for a wired connection are met, it executes usage assistance control for a wired connection (S803).

Here, with reference to FIG. 9A, an example of a determination item related to S802 will be described. The processor 320 performs wired connection usage assistance control on the condition that, for example, the distance between the wearable terminal 100 and the device 120 (apparatus) to which it is wired connected, or the connection unit 121 (wired connection point) of the device 120 (i.e., the distance 140 between the device 120 and the user 10) is equal to or greater than the first distance 201 (condition 1).

The processor 320 performs auxiliary control of the wired connection, for example, on the condition that at least a portion of the wired cable 130 (wired connection medium) is located at a position equal to or greater than the first distance 201 from the device 120 to which the wired connection is connected or the connection section 121 of the device 120 (Condition 2). For example, if a portion of the wired cable 130 is located outside the first distance 201, the processor 320 performs auxiliary control of the wired connection.

The processor 320 performs auxiliary control of the use of the wired connection, for example, on the condition that the second distance 205 is greater than the length of the wired cable 130 (condition 3).

The processor 320 performs auxiliary control of the wired connection, for example, on the condition that the pressure applied to the connection terminal of the wired cable 130 is equal to or greater than a first pressure amount (condition 4). Here, the first pressure amount is an amount of pressure that is set appropriately.

The processor 320 performs auxiliary control of the wired connection, for example, if the pressure applied to the connection terminal of the wired device 120 is equal to or greater than a second pressure amount (condition 5). Here, the second pressure amount is an amount of pressure that is set as appropriate.

The processor 320 performs auxiliary control of wired connection use, for example, on the condition that the pressure applied to the connection terminal of the wearable device 100 is equal to or greater than a third pressure amount (condition 6). Here, the third pressure amount is an amount of pressure that is set as appropriate.

The processor 320 performs auxiliary control of the wired connection, for example, on the condition that the wired cable 130 is in a straight line (condition 7). Here, the processor 320 can determine the state of the wired cable 130, for example, using the wired connection medium information 340.

The processor 320 performs assistance control for use of the wired connection, for example, when the distance between the wearable terminal 100 and the device 120 to which it is wired connected, or the connection unit 121 of the device 120, is equal to or greater than the third distance 205, and the wired connection has been disconnected (condition 8). In other words, the processor 320 performs assistance control for use of the wired connection, when the distance 140 between the device 120 and the user 10 is equal to or greater than the third distance 205, and the wired connection has been disconnected. Here, the processor 320 can determine whether the wired connection has been disconnected, for example, using the wired connection status information 341.

Furthermore, with reference to FIG. 9B, an example of the control content related to S803 will be described. That is, an example of the control content for controlling assistance in using a wired connection will be described. Note that the processor 320 may execute a combination of multiple control contents.

The processor 320 issues a notification indicating that a wired connection is in progress (control 1). Here, the processor 320 confirms that a wired connection is in progress, for example, based on the wired connection status information 341. The manner of notification is not particularly limited, and the processor 320 may issue a notification to that effect by displaying an icon, a message, a virtual object, outputting sound, outputting vibration, etc.

The processor 320 displays, for example, the device 120 connected via a wired connection and/or the area within the second distance 203 from the device 120 (control 2). Here, the processor 320 generates and displays a virtual object indicating the wired connection area, for example, based on the wired connection area information 338. Note that the processor 320 may also display, for example, an image of real space corresponding to the wired connection area acquired using the camera 103.

The processor 320 may, for example, issue a warning that the set second distance 203 is longer than the length of the wired cable 130 (control 3). Note that the length of the wired cable 130 used to connect to the device 120 may, for example, be input in advance by the user 10. The processor 320 may then store the length of the wired cable 130 set by the user 10 as information data 332 in the memory 330 and use it for processing.

The processor 320 displays, for example, a direction mark indicating the direction from the wearable device 100 to the device 120 to which it is wired connected, or the connection unit 121 of that device 120 (control 4). Here, the processor generates and displays a virtual object of the direction mark using the virtual space generation and display processing unit 325.

The processor 320 displays, for example, the wired device 120 or the connection part 121 of the device 120 (control 5). The processor 320 displays, for example, the wired device 120 or the connection part 121 of the device 120 captured by the camera 103. The processor 320 generates and displays, for example, a virtual object of a mark indicating the wired device 120 or the connection part 121 of the device 120, using the virtual space generation and display processing unit 325.

The processor 320 displays, for example, the distance from the wearable terminal 100 to the device 120 to which it is wired connected, or to the connection unit 121 of the device 120 (control 6). Here, the processor 320 displays the distance based on, for example, the display of an icon, a message, or a virtual object. The processor 320 may also, for example, provide an audio notification regarding the distance.

The processor 320, for example, displays the length of the connected wired cable 130 (control 7). Here, similar to control 3 above, the processor 320 may store the length of the wired cable 130 set by the user 10 as information data 332 in the memory 330 and use it for processing.

The processor 320 issues a notification, for example, indicating that the wired cable 130 is straight (control 8). That is, the processor 320 issues a notification, for example, indicating that the wired cable 130 is taut. Here, the processor 320 checks the shape of the wired cable 130, for example, based on the wired connection medium status information 340. The manner of notification is not particularly limited, and the processor 320 issues a notification by displaying an icon, displaying a message, displaying a virtual object, outputting sound, outputting vibration, etc.

The processor 320 issues a notification indicating that pressure is being applied to, for example, the wired cable, the wearable terminal, or at least one connection terminal of the wired connected device (control 9). Here, the processor 320 acquires information indicating that pressure is being applied, for example, based on the wired connection medium status information 340. The manner of notification is not particularly limited, and the processor 320 issues the notification by displaying an icon, displaying a message, displaying a virtual object, outputting audio, outputting vibration, etc.

The processor 320, for example, displays an image of real space (control 10). Here, the processor 320 may, for example, confirm that the wired cable 130 is within the field of view of the camera 103, and display an image of real space including the wired cable 130. The processor 320 may also, for example, confirm that the wired device 120 is within the field of view of the camera 103, and display an image of real space including the wired device 120.

The processor 320, for example, generates and displays a virtual object corresponding to the connected wired cable 130 (control 11). Note that, for example, if the wired cable 130 is tangled, a virtual object of the wired cable 130 including the tangle may be displayed. The processor 320 may also display an image of the wired cable 130 acquired by the camera 103.

The processor 320, for example, displays the position of the disconnected wired cable 130 (control 12). The processor 320, for example, generates and displays a virtual object indicating the position of the connection part 121 of the device 120 to which the wired cable 130 was connected. The processor 320 may also generate and display a virtual object indicating the position of the end of the disconnected wired cable 130. The processor 320 may also issue a notification indicating that the cable has been disconnected. Here, the manner of notification is not particularly limited, and the processor 320 may issue a notification indicating that the cable has been disconnected by displaying an icon, a message, a virtual object, outputting sound, outputting vibration, or the like.

The processor 320, for example, displays a playback video (control 13). That is, the processor 320 displays a video of a situation in which the wired cable 130 has become disconnected, for example.

Returning to Figure 8, we will continue to explain the processing flow. The processor 320 of the wearable terminal 100 determines whether the conditions for canceling use assistance control for wired connections are met (S804). If the conditions for canceling use assistance control for wired connections are met, the processor 320 cancels use assistance control for wired connections (S805).

An example of the determination items related to S804 will be described with reference to Figure 9C. In the example in Figure 9C, conditions 1 to 8 relate to conditions 1 to 8 described above using Figure 9A. In other words, if conditions 1 to 8 in Figure 9A are no longer met, the processor 320 releases the wired connection usage assistance control (conditions 1 to 8).

Furthermore, the processor 320 releases the use assistance control of the wired connection, for example, if the wired connection is released when the distance between the wearable terminal 100 and the wired device 120 or the connection unit 121 of the device 120 is less than the third distance 205 (Condition 9). In other words, since it is considered that the wired connection has been intentionally released by the user 10, the use assistance control of the wired connection is terminated.

Furthermore, if user 10 makes an input to cancel the use assistance control of the wired connection, processor 320 acquires the input from user 10 and cancels the use assistance control of the wired connection (condition 10).

Returning to Figure 8, we will continue to explain the processing flow. The user 10 disconnects the wired connection within the third distance 205 (S806). Then, when the user 10 ends use of the wearable terminal 100, the processing ends. On the other hand, when the user 10 continues use of the wearable terminal 100 and makes a new wired connection, the same processing as above is performed (S807).

Depending on how a wearable device 100 such as an HMD is used, there are situations where the user 10 is unable to grasp the situation in real space. If a warning is not issued, there is an issue where the user 10 may feel annoyed by the wired connection medium, for example, by being pulled by the connected wired connection medium. Furthermore, the user 10 may become entangled in the wired connection medium, or the wired connection medium may wrap around the user's body, causing annoyance.

In contrast, according to this embodiment, the use assistance control for wired connections issues notifications urging caution regarding use of the wearable terminal 100, allowing the user 10 to be aware of the situation, thereby enabling better use of the wearable terminal 100. For example, even when a wired connection is made to another peripheral device, the inconvenience caused by the wired connection medium is reduced, allowing for comfortable and easy operation.

In the above explanation, an immersive HMD that is worn on the user's head and displays virtual space, real space, etc. on the display unit has been used as a specific example of the wearable device 100. However, even in the case of an optically transparent HMD that displays virtual objects on the display unit while allowing the user to see physical objects in front of them, it is clear that when it is determined that the device is more than a predetermined distance away from the location of the device to which it is wired connected, it can notify the user 10 by displaying an icon or message indicating the wired connection status, thereby providing the effect of reducing the inconvenience caused by the wired cable 130.

The present invention is not limited to the above-described embodiments, and includes various modifications. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and are not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace part of the configuration of each embodiment with other configurations.

Furthermore, the numerical values, messages, etc. appearing in the text and figures are merely examples, and the effects of the present invention will not be diminished even if different ones are used.

Furthermore, some or all of the above-mentioned configurations, functions, processing units, acquisition sensors, etc. may be realized in hardware, for example by designing them as integrated circuits. Furthermore, the above-mentioned configurations, functions, etc. may be realized in software by a processor interpreting and executing a program that realizes each function. The processor includes transistors and other circuits, and may be considered circuitry or processing circuitry. Information such as programs, tables, and files that realize each function may be stored in memory, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card, SD card, or DVD, or may be stored in a device on a communications network.

Furthermore, the control and information lines shown are those considered necessary for the explanation, and do not necessarily represent all control and information lines on the product. In reality, it is safe to assume that almost all components are interconnected.

The HMD 100 may also be configured by connecting a device such as a smartphone or tablet. For example, the HMD 100 may be configured to use the display of the attached smartphone or tablet as the display 308.

As described above, the first distance 201, the second distance 203, and the third distance 205 may be determined, input, and set by the user 10. Alternatively, for example, the first distance 201, the second distance 203, and the third distance 205 may be automatically determined by the user 10 inputting the length of the wired cable 130. Here, the length of the wired cable 130 may be input in a format in which the user selects from predetermined lengths, such as 3 m, 5 m, and 7 m, for example. The wearable terminal 100 may then store data (such as a table) in which the first distance 201, the second distance 203, and the third distance 205 are associated with the length of the wired cable 130, and set these distances by reading out the corresponding first distance 201, second distance 203, and third distance 205 from the input length of the wired cable 130. Note that the first distance 201, second distance 203, and third distance 205 are pre-registered in this data as appropriate distances.

100 Wearable terminal 101 Left eye gaze sensor 102 Right eye gaze sensor 103 Camera 104 Distance measurement sensor 105 Positioning sensor 106 Microphone 107 Speaker 108 Vibrator 130 Wired cable 304 Acceleration sensor 305 Gyro sensor 306 Geomagnetic sensor 307 Operation input interface 308 Display 320 Processor 330 Memory 345 Wired connection input/output interface 346 Near field wireless communication interface 347 Network communication interface

Claims (20)

A wearable terminal worn by a user,
a display that displays a virtual space that may include virtual objects, an image of real space and virtual objects, or only an image of real space on a display screen;
a wired connection interface to which a wired cable is connected and which is used for wired connection with a device;
a processor;
Equipped with
The processor:
When the establishment of a wired connection with the device is identified and the device is located at a predetermined distance or more from the location of the device, a notification is sent to the user to warn them about using the wearable device.
A wearable device characterized by: The wearable terminal according to claim 1,
The processor:
As a notification to call attention to the use of the wearable device, at least one of the following is executed: displaying a graphic indicating that a wired connection is in progress on the display; redisplaying the graphic that was once hidden; and displaying a message indicating that a wired connection is in progress.
A wearable device characterized by: The wearable terminal according to claim 1,
The processor:
When it is determined that the wearable terminal is located at a distance equal to or greater than a first distance set as the predetermined distance from the device, a wired connection area relating to an area where the wired cable connected to the device may be present, which is based on the first distance and/or a second distance longer than the first distance, is displayed on a display as a notice to alert the user to the use of the wearable terminal.
A wearable device characterized by: The wearable terminal according to claim 1,
The processor:
as a notification to call attention to the use of the wearable device, a notification is executed regarding at least one of a direction toward the device, a position of the device, a distance from the device, and a length of the wired cable;
A wearable device characterized by: The wearable terminal according to claim 1,
The wearable terminal includes:
Equipped with a camera that captures the outside,
The processor:
displaying an image of the real space captured by the camera on the display as a notice to call attention to use of the wearable device;
A wearable device characterized by: The wearable terminal according to claim 1,
The wearable terminal includes:
Equipped with a camera that captures the outside,
The processor:
generating a virtual object corresponding to the wired cable captured by the camera as a notice to call attention to use of the wearable device, and displaying the generated virtual object on the display;
A wearable device characterized by: The wearable terminal according to claim 1,
The wearable terminal includes:
A camera for capturing an image of the outside and/or a distance measuring sensor for measuring the shape of an external object are provided,
The processor:
When the wired cable is identified as being straight, a notification indicating that the wired cable is straight is executed as a notification to call attention to the use of the wearable device.
A wearable device characterized by: The wearable terminal according to claim 1,
The processor:
information about the pressure acting on the connection terminal is acquired from at least one of a piezoelectric element provided on the connection terminal of the wired cable, a piezoelectric element provided on the connection terminal of the wearable terminal to which the wired cable is connected, and a piezoelectric element provided on the connection terminal of the device to which the wired cable is connected, and when pressure equal to or greater than a predetermined amount is acquired, a notification is issued to warn users about the pressure being applied to the connection terminal.
A wearable device characterized by: The wearable terminal according to claim 1,
The wearable terminal includes:
Equipped with a camera that captures the outside,
a first distance is set as the predetermined distance,
The processor:
When the wearable device is located at a distance equal to or greater than a third distance that is shorter than the first distance from the device and when it is determined that the wired connection has been disconnected, at least one of a notification indicating that the wired connection has been disconnected and a video display indicating that the wired connection has been disconnected is executed as a notification to alert the user to the use of the wearable device.
A wearable device characterized by: The wearable terminal according to claim 9,
The processor:
When the wearable device is located within the third distance from the device and the wired connection is disconnected, no notification is issued to warn about use of the wearable device.
A wearable device characterized by: An information processing method using a wearable terminal including: a display that displays, on a display screen, a virtual space that may include virtual objects, an image of real space and virtual objects, or only an image of real space; a wired connection interface that is connected to a wired cable and is used for wired connection with a device; and a processor,
the processor determining the establishment of a wired connection with the device;
the processor obtaining a distance from the device;
a step of notifying a user of the wearable terminal to be careful about use when the processor identifies that a wired connection with the device has been established and that the wearable terminal is located at a predetermined distance or more from the device;
An information processing method comprising: 12. The information processing method according to claim 11,
The processor:
As a notification to call attention to the use of the wearable device, at least one of the following is executed: displaying a graphic indicating that a wired connection is in progress on the display; redisplaying the graphic that was once hidden; and displaying a message indicating that a wired connection is in progress.
1. An information processing method comprising: 12. The information processing method according to claim 11,
The processor:
When it is determined that the wearable terminal is located at a distance equal to or greater than a first distance set as the predetermined distance from the device, a wired connection area relating to an area where the wired cable connected to the device may be present, which is based on the first distance and/or a second distance longer than the first distance, is displayed on a display as a notice to alert the user to the use of the wearable terminal.
1. An information processing method comprising: 12. The information processing method according to claim 11,
The processor:
as a notification to call attention to the use of the wearable device, a notification is executed regarding at least one of a direction toward the device, a position of the device, a distance from the device, and a length of the wired cable;
1. An information processing method comprising: 12. The information processing method according to claim 11,
The wearable terminal includes:
Equipped with a camera that captures the outside,
The processor:
displaying an image of the real space captured by the camera on the display as a notice to call attention to use of the wearable device;
1. An information processing method comprising: 12. The information processing method according to claim 11,
The wearable terminal includes:
Equipped with a camera that captures the outside,
The processor:
generating a virtual object corresponding to the wired cable captured by the camera as a notice to call attention to use of the wearable device, and displaying the generated virtual object on the display;
1. An information processing method comprising: 12. The information processing method according to claim 11,
The wearable terminal includes:
A camera for capturing an image of the outside and/or a distance measuring sensor for measuring the shape of an external object are provided,
The processor:
When the wired cable is identified as being straight, a notification indicating that the wired cable is straight is executed as a notification to call attention to the use of the wearable device.
1. An information processing method comprising: 12. The information processing method according to claim 11,
The processor:
information about the pressure acting on the connection terminal is acquired from at least one of a piezoelectric element provided on the connection terminal of the wired cable, a piezoelectric element provided on the connection terminal of the wearable terminal to which the wired cable is connected, and a piezoelectric element provided on the connection terminal of the device to which the wired cable is connected, and when pressure equal to or greater than a predetermined amount is acquired, a notification is issued to warn users about the pressure being applied to the connection terminal.
1. An information processing method comprising: 12. The information processing method according to claim 11,
The wearable terminal includes:
Equipped with a camera that captures the outside,
a first distance is set as the predetermined distance,
The processor:
When the wearable device is located at a distance equal to or greater than a third distance that is shorter than the first distance from the device and when it is determined that the wired connection has been disconnected, at least one of a notification indicating that the wired connection has been disconnected and a video display indicating that the wired connection has been disconnected is executed as a notification to alert the user to the use of the wearable device.
1. An information processing method comprising: 20. The information processing method according to claim 19,
The processor:
When the wearable device is located within the third distance from the device and the wired connection is disconnected, no notification is issued to warn about use of the wearable device.
1. An information processing method comprising: