WO2025181507A1 - Information processing method, information processing device, and program - Google Patents

Abstract

The present invention comprises: a determination process for determining, on the basis of a linear movement path (first path) from a first position to a second position and content displayed on a display unit, whether or not the visibility of specific content is impaired if an agent image, which is displayed at the first position on the display unit, moves along the first path when the agent image moves to the second position on the basis of the state of an occupant; and a control process for executing a first movement process or the like for moving the agent image along a movement path (second path) which is different from the first path and which does not impair the visibility of the specific content if the visibility of the specific content is impaired.

Description

Information processing method, information processing device, and program

The present invention relates to an information processing method, information processing device, and program for controlling an agent capable of communicating with a user.

Conventionally, technologies for conveying various types of information to a user in a vehicle have been proposed. For example, JP2020-055348A proposes a technology in which, when an image to be provided to a passenger is displayed on a display unit, the display position of an agent image is moved to the display position of the provided image in order to guide the passenger's gaze to the display position of the provided image.
Summary of the Invention

In the conventional technology described above, when the display position of an agent image is moved to the display position of the image provided to the occupant, there is a risk that the content displayed on the display unit may be obscured by the moving agent image. For example, if content related to the vehicle's driving route or vehicle driving control is displayed on the display unit, there is a possibility that the moving agent image may obscure that content. Therefore, it is important to ensure that the visibility of specific content is not obstructed by the moving agent image.

The present invention aims to appropriately move an agent image on a display unit so as not to obstruct the visibility of specific content.

One aspect of the present invention is an information processing method for displaying an agent image on a display unit that communicates with a vehicle occupant, and controlling the display state of the agent image based on the state of the occupant. This information processing method includes, when an agent image displayed at a first position on the display unit moves to a second position based on the state of the occupant, a determination process for determining whether movement of the agent image along the first path will obstruct the visibility of specific content, based on a linear movement path (first path) from the first position to the second position and the content displayed on the display unit; if the visibility of the specific content will not be obstructed, moving the agent image along the first path in a first display mode; and, if the visibility of the specific content will be obstructed, a movement path different from the first path that does not obstruct the visibility of the specific content (second path). ), a second movement process of moving the agent image from a first position to a second position in a second display mode that is a display mode different from the first display mode and does not impair the visibility of specific content, and a third movement process of moving the agent image from the first position to the second position at a second movement timing that is different from the first movement timing in the first display mode and does not impair the visibility of specific content, or at a second movement speed that is different from the first movement speed in the first display mode and does not impair the visibility of specific content.

FIG. 1 is a simplified diagram showing an example of the configuration of the interior of a vehicle. FIG. 2 is a simplified diagram showing an example of the configuration of the interior of a vehicle. FIG. 3 is a simplified diagram showing an example of the configuration of the interior of a vehicle. FIG. 4 is a diagram showing an example of a transition when an agent image is moved. FIG. 5 is a block diagram showing an example of the system configuration of the information processing system. FIG. 6 is a diagram showing an example of the transition of the agent image. FIG. 7 is a diagram showing an example of the transition of the agent image. FIG. 8 is a diagram showing an example of a transition of an agent image. FIG. 9 is a diagram showing an example of a transition of an agent image. FIG. 10 is a flowchart showing an example of an agent migration process. FIG. 11 is a flowchart showing an example of an agent migration process.

Embodiments of the present invention will be described below with reference to the accompanying drawings.

[Display unit installation example]
Fig. 1 is a simplified diagram showing an example of the configuration of the interior of a vehicle C1. Fig. 1 shows the interior of the vehicle C1 in front of the driver's seat and passenger seat (not shown) as viewed from the rear in the longitudinal direction of the vehicle C1. To facilitate explanation, Fig. 1 omits illustrations of components other than a dashboard 2, a steering wheel 3, a windshield 4, a rearview mirror 5, and an output device 200.

The output device 200 is an output device installed inside the vehicle C1, and performs various output operations based on instructions from the information processing device 110 (see Figure 5). The output device 200 is an apparatus installed on the dashboard 2, and includes a display unit 201, a sound output unit 202, and a reception unit 203 (see Figure 5). In Figure 1, an output device that is long in the left-right direction of the vehicle C1 is shown as an example of the output device 200. The output device 200 is an apparatus whose display area extends, for example, from near the center of the dashboard 2 to the driver's seat.

For example, the output device 200 is an in-vehicle system consisting of one or more devices capable of providing various types of information. The output device 200 can be, for example, at least one of a navigation device, an audio device, a DVD device, a TV tuner, an IVI (In-Vehicle Infotainment), etc. The images displayed on each output device 200 can also be displayed using a HUD (Head Up Display) implemented on the windshield 4, or other display device. Alternatively, a portable device that can be carried by the driver D1 (or a device that can be installed in the vehicle C1), such as a smartphone, tablet, or portable personal computer, can be used.

In addition, the output device 200 executes output operations that enable communication with users (including the driver D1) aboard the vehicle C1. For example, an agent image AG1 (see Figures 2 and 3) that enables communication with the occupants is displayed on the display unit 201 of the output device 200, and various effects are executed using the agent image AG1. Note that an agent image that enables communication with the occupants may also be displayed on the windshield 4 (HUD display area) or another display device, and various effects may be executed using the agent image.

[Example of agent image display]
2 and 3 are simplified diagrams showing examples of the configuration of the interior of the vehicle C1. Note that since the configuration examples shown in Fig. 2 and 3 are similar to the configuration example shown in Fig. 1, detailed description thereof will be omitted here.

Figures 2 and 3 show an example in which content CT1 and agent image AG1 are displayed on the display unit 201 of the output device 200. Content CT1 is content that displays map information including a current location indicator PL1 indicating the current location of vehicle C1 and route information RI1 indicating the route traveled by vehicle C1. For example, route information RI1 is displayed in a display format different from that of a map (for example, arrows colored blue, red, etc.). Although not shown, it is also possible to display map information with a different viewpoint as appropriate. For example, content CT1 is displayed based on a navigation function.

Agent image AG1 is an image representing an object capable of communicating with the user. In this embodiment, an example is shown in which an image that resembles a human face is used as agent image AG1. However, agent image AG1 is not limited to this, and for example, an image that resembles the entire body or part of a human (for example, the upper body) may be used as agent image AG1, or an image that resembles an animal such as a rabbit or pig (or the face of that animal), or an image that resembles a virtual creature (for example, the face of an anime character) (or that face). In this way, a mimicked image can be used as agent image AG1.

The output device 200 performs various operations related to driving assistance (e.g., operations related to content CT1 and agent image AG1) based on instructions from the information processing device 110 (see Figure 5). For example, the output device 200 outputs various information related to driving assistance when the driver D1 performs driving operations, surrounding facilities, etc., based on the control of the information processing device 110.

Note that driving assistance using the agent image AG1 is expected to include, for example, alerting the driver of moving objects ahead or behind. For example, to alert the driver of a moving object ahead, a voice output such as "Be careful, there is a railroad crossing ahead" or "There is a person ahead" can be output. At the time of each of these outputs, various actions can be performed using the agent image AG1. In other words, driving assistance can be performed using the agent image AG1. Furthermore, for example, the output device 200 performs various processes using the agent image AG1 based on the control of the information processing device 110. For example, the output device 200 performs communication processing to exchange various types of information with the driver D1 using the agent image AG1. For example, the output device 200 uses the agent image AG1 to exchange various types of conversations with the driver D1 or to provide information preferred by the driver D1.

For example, as shown in Figure 2, it is conceivable that in response to the agent image AG1 uttering voice information S1 "Turn right at the next intersection," driver D1 utters voice S2 "Is there a highway beyond that?". Then, in response to the voice S2 uttered by driver D1, agent image AG1 utters voice information S3 "We'll get on the highway next" (see Figure 3). Note that these exchanges are simplified and are not limited to these.

In this way, when an occupant of vehicle C1 makes some kind of user input, it is conceivable that the agent image AG1 can be moved near the occupant who made the user input, making it easier to provide assistance to that occupant. For example, suppose that when agent image AG1 is displayed in the position shown in Figure 2, driver D1 utters voice S2, "Is that a highway up ahead?" In this case, it is conceivable that the agent image AG1 can be moved near driver D1 (for example, in the direction of driver D1's line of sight), as shown in Figure 3. In other words, it is conceivable to move agent image AG1 to the position where the user is operating it, or to the position where the user wants to see it. The method of moving agent image AG1 in this case will be explained with reference to Figure 4.

[Example of agent image movement]
FIG. 4 is a diagram showing an example of a transition when the agent image AG1 is moved on the display unit 201 of the output device 200. In FIG.

In Figure 4(A), when moving agent image AG1 on the display unit 201, the source agent image is shown as agent image AG1a, and the destination agent image is shown as agent image AG1e.

In Figure 4(B), dotted lines MT1 and MT2 indicate the range of the movement trajectory of agent image AG1 when moving linearly from agent image AG1a to agent image AG1e on the display unit 201. Also in Figure 4(B), part of the movement trajectory of agent image AG1 is schematically indicated by dotted circles AG1a to AG1e. In this way, Figure 4(B) shows the relationship between the first movement trajectory AG1a to AG1e of agent image AG1 from the first position AG1a to the second position AG1e and the currently displayed content CT1.

Here, an example is shown in which a linear movement path is set from the initial position (first position AG1a) of the agent image AG1 to a response position (second position AG1e), which is the position to which the agent image AG1 is moved in response to user input (e.g., manual input, voice input).

Note that each position on the display surface of the display unit 201 can be managed, for example, as coordinate information in the agent management DB 132 (see FIG. 5). For example, the output content determination unit 126 (see FIG. 5) can obtain the current position of the agent image AG1 by sequentially storing the positions of each image (content image, agent image AG1) displayed on the display surface of the display unit 201 in the agent management DB 132. Furthermore, the movement mode determination unit 127 can calculate the range of the movement trajectory of the agent image AG1 as it moves on the display surface of the display unit 201 based on the display area (display area) of the agent image AG1, the current position of the agent image AG1, and the destination position of the agent image AG1. A known calculation method can be used to calculate this range.

For example, it is conceivable to move the agent image AG1 based on the line of sight of the occupant of vehicle C1, the position of the occupant's hands, etc. For example, when the occupant performs a desired operation on the display unit 201, it is possible to move the agent image AG1 to a position that is easy for the occupant to operate based on the occupant's actions. For example, when the occupant operates with their hands, it is conceivable to move the agent image AG1 to a position close to their hands. For example, it is conceivable to move in a straight line from the source to the destination, as shown by arrow AW1.

In this way, by having the agent image AG1 follow the user's operations and moving the agent image AG1 continuously, it is possible to show that the agent image AG1 is responding to the user's operations. Furthermore, by looking at the agent image AG1, the user can easily visually grasp the information that the agent image AG1 is trying to convey. For this reason, it is conceivable that the user may feel uneasy if the agent image AG1 disappears even temporarily. However, there is a possibility that the agent image AG1 may overlap with information that should not be hidden while moving in response to user input.

For example, if the agent image AG1 is displayed in the foreground and the vehicle moves along the path indicated by the arrow AW1, the agent image AG1 will overlap the front side of the content CT1 (i.e., the display surface side of the display unit 201) as it moves. In this case, the occupant will not be able to see part of the content CT1 while the agent image AG1 is moving. In other words, the movement of the agent image AG1 will impede the visibility of the content CT1.

Therefore, in this embodiment, if the movement of the agent image AG1 impairs the visibility of specific content, the movement path of the agent image AG1 is changed, the display mode of the agent image AG1 is changed, or the movement timing or movement speed of the agent image AG1 is changed so as not to impair the visibility of the specific content.

[Configuration example of information processing system]
FIG. 5 is a block diagram showing an example of the system configuration of the information processing system 100 installed in the vehicle C1.

The information processing system 100 includes a sound acquisition unit 101, a driver image acquisition unit 102, a vehicle interior image acquisition unit 103, a vehicle exterior image acquisition unit 104, an information processing device 110, and an output device 200. The information processing device 110 is an example of a device that controls the output device 200, which is capable of communicating with the occupants of the vehicle C1 (including the driver D1).

The information processing device 110 and the output device 200 are connected via a communication method using wired communication or wireless communication. The information processing device 110 is also connected to the network 20 via a communication method using wireless communication. The network 20 is a network such as a public line network or the Internet. The output device 200 may also be connected to the network 20 via a communication method using wireless communication. While Figure 5 shows an example in which the information processing device 110 and the output device 200 are configured as separate entities, the information processing device 110 and the output device 200 may also be configured as an integrated device.

The sound acquisition unit 101 is provided inside the vehicle C1, acquires sounds inside the vehicle C1, and outputs sound information related to the acquired sounds to the information processing device 110. The sound acquisition unit 101 can be, for example, one or more microphones or sound acquisition sensors.

The driver image acquisition unit 102 captures an image of the driver D1 in the vehicle C1 and generates an image (image data), and outputs image information related to the generated image to the information processing device 110. The driver image acquisition unit 102 is provided at least inside the vehicle C1 and captures an image of the driver D1 in the vehicle C1 and generates an image (image data). The driver image acquisition unit 102 is configured, for example, with one or more camera devices or image sensors capable of capturing an image of the driver D1. For example, one driver image acquisition unit 102 can be provided in the front of the interior of the vehicle C1 (e.g., on the ceiling), and can capture an image of the driver D1 from in front of the vehicle C1 and generate an image (image data). For example, the driver image acquisition unit 102 can be provided above the windshield 4, i.e., above the rearview mirror 5. The driver image acquisition unit 102 and the vehicle interior image acquisition unit 103 may be the same device or different devices.

The vehicle interior image acquisition unit 103 captures images of subjects inside the vehicle C1 to generate images (image data), and outputs image information related to the generated images to the information processing device 110. The vehicle interior image acquisition unit 103 is provided at least inside the vehicle C1 (e.g., the ceiling) and captures images of subjects inside the vehicle C1 to generate images (image data). The vehicle interior image acquisition unit 103 is composed of, for example, one or more camera devices or image sensors capable of capturing images of subjects. For example, one vehicle interior image acquisition unit 103 may be provided at the front of the vehicle C1 to capture images of subjects from the front to generate images (image data), and another vehicle interior image acquisition unit 103 may be provided at the rear of the vehicle C1 to capture images of subjects from the rear to generate images (image data).

The outside-vehicle image acquisition unit 104 captures images of subjects outside the vehicle C1 to generate images (image data), and outputs image information related to the generated images to the information processing device 110. Two or more outside-vehicle image acquisition units 104 may be provided, and all or some of the images from these outside-vehicle image acquisition units 104 may be used. For example, one outside-vehicle image acquisition unit 104 may be provided in front of the vehicle C1 to capture images of subjects from in front of the vehicle C1 to generate images (image data), and another outside-vehicle image acquisition unit 104 may be provided behind the vehicle C1 to capture images of subjects behind the vehicle C1 to generate images (image data). Furthermore, one or more devices capable of capturing subjects in all directions from the vehicle C1 and subjects inside the vehicle C1, such as a 360-degree camera, may be used.

The driver image acquisition unit 102, the vehicle interior image acquisition unit 103, and the vehicle exterior image acquisition unit 104 are each composed of, for example, an image sensor that receives light from a subject collected by a lens, and an image processing unit that performs predetermined image processing on the image data generated by the image sensor. For example, a CCD (Charge Coupled Device) type or CMOS (Complementary Metal Oxide Semiconductor) type image sensor can be used as the image sensor.

The information processing device 110 includes a control unit 120, a storage unit 130, and a communication unit 140. The communication unit 140 exchanges various types of information with other devices using wired or wireless communication under the control of the control unit 120. For example, when the communication unit 140 receives driving assistance information, operation information of the agent image AG1, etc. from an external device (e.g., a server), it outputs each piece of information to the control unit 120.

The control unit 120 controls each unit based on various programs stored in the memory unit 130. The control unit 120 is realized by a processing device such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit). The vehicle ECU (Electronic Control Unit) of the vehicle C1 may also be used as the control unit 120, or a processing device different from the vehicle ECU may be provided as the control unit 120.

The control unit 120 executes various controls based on the information output from the sound acquisition unit 101, driver image acquisition unit 102, vehicle interior image acquisition unit 103, vehicle exterior image acquisition unit 104, communication unit 140, etc., and the information acquired by the vehicle information acquisition unit 125. For example, the control unit 120 executes control processing to control the operating state of the output device 200. Specifically, the control unit 120 includes an utterance acquisition unit 121, a driver status acquisition unit 122, a vehicle interior status acquisition unit 123, a vehicle exterior status acquisition unit 124, a vehicle information acquisition unit 125, an output content determination unit 126, a movement mode determination unit 127, and an output control unit 128.

The speech acquisition unit 121 performs a predetermined sound analysis process on the sound information output from the sound acquisition unit 101 to acquire speech information related to the speech of each user (including driver D1) contained in the sound information, and outputs the speech information to the output content determination unit 126. This sound analysis process can be performed using known sound analysis processes.

The driver status acquisition unit 122 performs a predetermined image analysis process on the image information output from the driver image acquisition unit 102 to acquire various pieces of information about driver D1 contained in the image information, and outputs each piece of information to the output content determination unit 126. This image analysis process can be performed using known image analysis processes. As various pieces of information about driver D1, it is possible to acquire various actions of driver D1 related to, for example, driver D1's facial expression, line of sight, hand, face, body movements, etc. From this, it is possible to detect, for example, whether driver D1 has gotten in and out of vehicle C1, whether each seat in vehicle C1 is occupied, and hand movements of driver D1. In other words, driver status information related to the state of driver D1 is acquired by the driver status acquisition unit 122.

The cabin status acquisition unit 123 acquires various pieces of information about the interior of the vehicle C1 contained in the image information by performing a predetermined image analysis process on the image information output from the cabin image acquisition unit 103, and outputs the information to the output content determination unit 126. This image analysis process can be performed using known image analysis processes. As various pieces of information about the interior of the vehicle C1, it is possible to acquire, for example, each user's facial expression, each user's gaze, and each user's actions related to the movements of their hands, face, body, etc. In other words, the cabin status acquisition unit 123 acquires user status information about the status of each user aboard the vehicle C1, and vehicle status information about the status of the vehicle C1.

The vehicle exterior situation acquisition unit 124 acquires various pieces of information about the outside of the vehicle C1 contained in the image information by performing a predetermined image analysis process on the image information output from the vehicle exterior image acquisition unit 104, and outputs each piece of information to the output content determination unit 126. This image analysis process can use known image analysis processes. As various pieces of information about the outside of the vehicle C1, for example, it is possible to detect whether the vehicle C1 is traveling on a road, whether the vehicle C1 is stopped on a road, and traffic lights, signs, etc. that are present in front of the vehicle C1. In other words, the vehicle exterior situation acquisition unit 124 acquires vehicle state information about the state of the vehicle C1.

The vehicle information acquisition unit 125 acquires information (vehicle state information) relating to various vehicle states of the vehicle C1 and outputs the acquired vehicle state information to the output content determination unit 126. The vehicle state information can be acquired, for example, from a CAN (Controller Area Network) signal. The vehicle state information includes, for example, vehicle speed, acceleration, shift lever position (e.g., P range, D range), accelerator pedal depression amount, brake pedal depression amount, position information, and abnormality occurrence information. For example, it is possible to determine whether the vehicle C1 is stopped or moving based on the vehicle speed, acceleration, etc. Furthermore, various warning information can be displayed based on the abnormality occurrence information relating to the vehicle C1.

The vehicle information acquisition unit 125 may also acquire sensor detection information output from various sensors installed in the vehicle C1. Examples of such sensors include LiDAR (Light Detection and Ranging), RADAR (Radio Detection and Ranging), sonar, vehicle speed sensor, acceleration sensor, steering sensor, accelerator position sensor, position information acquisition sensor (position information acquisition unit), seat occupancy sensor, and seat belt sensor. Publicly known sensors can be used for each of these sensors. LiDAR, RADAR, sonar, and the like are examples of sensors that detect the conditions around the vehicle C1. Vehicle speed sensors, acceleration sensors, steering sensors, accelerator position sensors, and the like are examples of sensors that detect the driving operation conditions of the driver D1. These are just examples, and other sensors may also be used. Furthermore, only some of these sensors may be used.

The location information acquisition unit acquires location information regarding the location where vehicle C1 is located. For example, it can be realized by a GNSS receiver that acquires location information using the GNSS (Global Navigation Satellite System). This location information includes position-related data such as latitude, longitude, and altitude at the time the GNSS signal is received. Location information may also be acquired using other location information acquisition methods. For example, location information may be derived using information from nearby access points or base stations. Location information may also be acquired using beacons.

The seating sensor (or seat sensor) is a sensor that detects whether or not an occupant is seated in each seat of the vehicle C1. The seat belt sensor is a sensor that detects whether or not an occupant is wearing a seat belt in each seat of the vehicle C1. For example, whether or not the driver D1 sitting in the driver's seat is wearing a seat belt can be detected using a seating sensor, seat belt sensor, etc.

The output content determination unit 126 determines the content of each piece of output information to be output from the output device 200 based on the information output from the speech acquisition unit 121, driver status acquisition unit 122, cabin status acquisition unit 123, exterior status acquisition unit 124, vehicle information acquisition unit 125, reception unit 203, etc. The output content determination unit 126 then outputs the determined content of each piece of output information and the information used in determining this content to the movement mode determination unit 127 and output control unit 128. For example, the output content determination unit 126 acquires at least one of user status information regarding the status of each occupant (including driver D1) in vehicle C1 and vehicle status information regarding the status of vehicle C1. The output content determination unit 126 then determines the content of each piece of information to be output from the output device 200 based on at least one of the acquired user status information and vehicle status information. For example, if the output content determination unit 126 detects, based on the user status information or vehicle status information, that assistance information to be communicated to driver D1, it determines to output the assistance information in order to communicate the assistance information to driver D1. Furthermore, if the output content determination unit 126 detects, based on the user status information or vehicle status information, that a predetermined communication (e.g., conversation) is to be performed with any of the occupants of vehicle C1, it determines the output information for performing that communication. Furthermore, the output content determination unit 126 determines, for example, the display mode, display position, etc. of agent image AG1 based on the user status information or operation information received by the reception unit 203.

The movement mode determination unit 127 determines the movement mode when moving the agent image AG1 based on the content of each output information determined by the output content determination unit 126, the state of the driver D1 (e.g., driving load, line of sight direction), the state of the vehicle C1 (e.g., whether the surrounding environment is high driving load), etc. The movement mode determination unit 127 then outputs the determined movement mode of the agent image AG1 to the output control unit 128. For example, the movement mode determination unit 127 checks the relationship between the movement trajectory of the agent image AG1 moving from the source to the destination and one or more pieces of content displayed on the display unit 201. The movement mode determination unit 127 then determines whether there is content currently being displayed between the source and the destination, and if there is content currently being displayed between the source and the destination, determines whether the content overlaps with the movement trajectory of the agent image AG1. For example, if there is content currently being displayed between the source and destination and that content overlaps with the movement trajectory of the agent image AG1, the movement mode determination unit 127 determines to execute a movement effect that moves the agent image AG1 so as not to impair the visibility of that content. This movement effect will be described in detail with reference to Figures 6 to 9. Furthermore, of the content currently being displayed, only specific content may be subject to the determination of whether or not there is an overlap.

[Regarding specific content]
In this embodiment, when there is a possibility that the visibility of content (particularly specific content) is impaired by the movement of the agent image AG1, as described above, the movement path of the agent image AG1 is changed, the display mode of the agent image AG1 is changed, the movement timing or movement speed of the agent image AG1 is changed, etc. Here, the specific content will be described.

For example, of the content displayed on the display unit 201, content containing particularly important information can be designated as specific content. This important information is, for example, information that is required by law to be continuously displayed (i.e., information that is required by law to be continuously displayed, information that is required by law to be concealed). This information is, for example, various indicator lights and warning lights. Examples include battery level warning lights, maximum speed signs, battery level gauges, cruising range indicators, speedometers, position indicators (shift lever positions), odometers (total distance meters), trip meters (segment distance meters), warning lights indicating vehicle malfunctions, etc. Note that these are only examples of representative information, and other information (information that is required by law to be continuously displayed) may also be designated as important information.

Furthermore, for example, important information is information of high urgency. This high urgency information is, for example, information related to driving assistance that is presented to driver D1 while driving. Examples of such information include information notifying the driver D1 of the direction of travel of vehicle C1, and information notifying the driver of objects (e.g., railroad crossings, accident vehicles) that exist in the direction of travel of vehicle C1.

Furthermore, for example, important information may include information related to driving operations, driving control information related to driving control, and abnormality occurrence information regarding the occurrence of an abnormality in vehicle C1.

Information related to driving operations is, for example, map information (map content) displayed to guide the vehicle C1 in the direction of travel, and in particular map information including a current location indicator indicating the current location of the vehicle C1 and a route indicator indicating the route the vehicle C1 will take (e.g., the route from the current location to the destination). For example, the route the vehicle C1 will take in the future is more important to the driver D1 than the route the vehicle C1 has taken so it is preferable to treat the route the vehicle C1 will take in the future as important information. Furthermore, for example, when autonomous driving is being performed in the vehicle C1, it is expected that the autonomous driving route will be displayed in the map content. Furthermore, when autonomous driving is being performed in the vehicle C1, it is expected that the driver D1 will often check the driving route or perform operations other than driving. In such cases, it is expected that the agent image AG1 will move more frequently in response to manual operations by the driver D1. Furthermore, guidance information regarding the direction the vehicle C1 will take can also be specified as specific content as information related to driving operations.

Furthermore, for example, in non-autonomous driving situations, when highly urgent information is displayed, it is expected that the passenger in the passenger seat will often perform the operation. In such cases, it is expected that there will be more opportunities for the agent image AG1 to move in response to manual operation by the passenger in the passenger seat. In this way, if the passenger in the passenger seat performs manual operation while driver D1 is driving, it is possible that the agent image AG1 displayed on the driver D1 side will move in response to the operation of the passenger in the passenger seat. In this case, it is expected that the movement of the agent image AG1 will cause driver D1 to lose sight of important information (specific content).

Furthermore, driving control information related to driving control is, for example, information regarding the driving of vehicle C1, turning of vehicle C1, stopping of vehicle C1, etc.

Furthermore, the abnormality occurrence information content regarding the occurrence of an abnormality in vehicle C1 is, for example, content that includes information regarding malfunctions in various parts of vehicle C1. For example, it is content that includes information corresponding to the various warning lights that are generally displayed on meters installed in front of the driver's seat.

The above shows an example in which specific content is set in advance, but specific content may also be set based on the state of the occupant looking at the display unit 201. For example, if there is an occupant looking at the display unit 201, content that is in the occupant's line of sight (i.e., line of sight) may be set as specific content. In this case, content may be set as specific content on the condition that the same content is in the occupant's line of sight (i.e., line of sight) for a predetermined period of time or more.

In addition, entertainment information (entertainment content) that displays network information such as SNS (social networking service), music information, and entertainment video information can be set to not be specified as specific content.

[Configuration example of information processing system]
The output control unit 128 controls the operating state of the output device 200 based on the content of the output information determined by the output content determination unit 126 and the movement mode of the agent image AG1 determined by the movement mode determination unit 127. Each of these operations will be described in detail with reference to Figs. 6 to 11 etc.

The memory unit 130 is a storage medium that stores various types of information. For example, the memory unit 130 stores various types of information (e.g., control programs, agent information DB 131, agent management DB 132, content management DB 133, map information DB) required by the control unit 120 to perform various processes. The memory unit 130 also stores various types of information acquired via the communication unit 140. The memory unit 130 can be, for example, ROM (Read Only Memory), RAM (Random Access Memory), SRAM (Static Random Access Memory), HDD (Hard Disk Drive), SSD (Solid State Drive), or a combination of these.

Agent information DB131 stores various types of information required to realize the various operations of agent image AG1 displayed on output device 200. For example, image information related to agent image AG1 to be displayed on display unit 201 of output device 200 and audio information related to the audio of agent image AG1 to be output from audio output unit 202 are stored in agent information DB131. In addition, for example, operation information for operating agent image AG1 on output device 200 when various communications are performed is stored in agent information DB131.

The agent management DB 132 stores agent management information (coordinate information) for managing the display position of the agent image AG1 displayed on the display unit 201. For example, information regarding the position of the agent image AG1 on the display surface of the display unit 201 and the display area of the agent image AG1 is managed as agent management information.

Content management DB 133 stores content management information (coordinate information) for managing the display position of each piece of content (e.g., map content, music content) displayed on display unit 201. For example, information regarding the position of each piece of content on the display surface of display unit 201, the display area of the content, and the type of content is managed as content management information.

The output device 200 is a device that can display the agent image AG1 based on instructions from the information processing device 110 and convey various information to the driver D1, etc. using the agent image AG1.

The output device 200 includes a display unit 201, a sound output unit 202, and a reception unit 203. The display unit 201, the sound output unit 202, and the reception unit 203 are controlled based on a control unit (not shown) included in the output device 200.

The display unit 201 is a display unit that displays various images based on instructions from the information processing device 110.

The sound output unit 202 outputs various sounds based on instructions from the information processing device 110. For example, one or more speakers can be used as the sound output unit 202.

The reception unit 203 receives user input from the occupants of the vehicle C1 and outputs the received input to the control unit 120. The reception unit 203 may be, for example, a touch panel or various operating members. The display unit 201 and reception unit 203 may be configured as a touch panel that allows the user to perform operation input by touching or bringing their finger close to the display surface, or may be configured as a separate user interface. The display unit 201, sound output unit 202, and reception unit 203 are examples of user interfaces, and some of them may be omitted, or other user interfaces may be used.

[Example of driving load detection]
The degree of driving load can be determined based on whether the vehicle C1 is stopped or moving. For example, when the vehicle C1 is stopped, it can be determined that the driving load is low (e.g., the driving load is less than a threshold value). On the other hand, when the vehicle C1 is moving, it can be determined that the driving load is high (e.g., the driving load is equal to or greater than a threshold value). Whether the vehicle C1 is stopped or moving can be determined based on vehicle information (e.g., vehicle speed, acceleration, position of the shift lever (e.g., P range, D range), accelerator pedal depression amount, brake pedal depression amount) acquired by the vehicle information acquisition unit 125.

Furthermore, the driving load can be determined based on the traffic light ahead of vehicle C1. For example, if the traffic light ahead of vehicle C1 is red and vehicle C1 is stopped, it can be determined that the driving load is low (for example, the driving load is below a threshold). On the other hand, if vehicle C1 is stopped but the traffic light ahead of vehicle C1 turns green, it can be determined that the driving load is high (for example, the driving load is above a threshold). Note that determinations other than these can be made similar to those based on whether vehicle C1 is stopped or moving. Note that when vehicle C1 is in autonomous driving mode, it can be determined that the driving load is low (for example, the driving load is below a threshold) even if vehicle C1 is moving.

In addition, the driving load of driver D1 may be determined based on the driving operation of driver D1, the surrounding conditions of vehicle C1, etc.

For example, the operating status of driver D1, the status around vehicle C1, etc. can be acquired by the driver status acquisition unit 122, the vehicle exterior status acquisition unit 124, the vehicle information acquisition unit 125, etc. For example, if there are many accelerator operations and many steering operations within a predetermined time, it can be assumed that driver D1's driving load is high. In addition, the driving load of driver D1 can be estimated based on the status around vehicle C1, etc. For example, if vehicle C1 is traveling on a winding road, a narrow road, a highly congested road, a road with many people, etc., it can be assumed that driver D1's driving load is high. On the other hand, for example, if vehicle C1 is traveling on a long, straight road, it can be assumed that driver D1's driving load is low. In addition, the driving load of driver D1 can be estimated based on the user's facial expression, the voice of driver D1, etc.

For example, steering entropy can be used to measure the driving operation of driver D1. The steering entropy method is a measurement method that measures and estimates the driver's load based on the smoothness of the driver's steering angle, and known calculation methods can be used. Furthermore, this steering entropy method makes it possible to use digitized measurement results (measured values) as information entropy values calculated based on time-series steering angle data.

Furthermore, for example, the number of traffic participants around vehicle C1, the weather around vehicle C1, the darkness around vehicle C1, the shape of the road around vehicle C1, etc. can be used as the surrounding conditions of vehicle C1.

The driving load of driver D1 can be determined using the values described above. For example, a predetermined calculation (e.g., addition) can be performed on the values described above, and the results of this calculation can be used to determine the driving load of driver D1. Note that the driving load of driver D1 may also be determined using at least one of the values described above. Other well-known methods for determining driving load can also be used.

[Example of agent image movement]
6 and 7 are diagrams showing an example of the transition of the agent image AG1 when the agent image AG1 is moved from the first position PP1 to the second position SP1 on the display unit 201 of the output device 200. FIG.

Figure 6(A) shows an example of a transition when agent image AG1 is moved from first position PP1 to second position SP1 when content CT2, which is not specific content, is displayed on display unit 201. Content CT2 is music content that is displayed to allow various operations when listening to music, for example. Also, in Figure 6(A), the movement trajectory of agent image AG1 moving from first position PP1 to second position SP1 is schematically shown by agent images AG1a to AG1e.

As shown in FIG. 6(A), when the agent image AG1 moves along a linear path from the first position PP1 to the second position SP1, it is possible that the moving agent image AG1 and the displayed content CT2 may overlap. For example, the moving agent images AG1b to AG1d may overlap with the content CT2. In this case, the moving agent images AG1b to AG1d are displayed so as to overlap the front side of the content CT2 (i.e., the display surface side of the display unit 201), preventing the occupant from seeing all or part of the content CT2. As mentioned above, in the case of content other than specific content, even if the occupant is temporarily unable to see it, it is expected that this will not interfere with driving, etc., and will not cause the occupant any unpleasant impression. Furthermore, by continuing to display the moving agent image AG1, it becomes easier to recognize the identity of the agent image AG1, which is thought to increase trust and attachment to the agent image AG1. Therefore, when the content CT2 being displayed is not a specific content, the output control unit 128 executes display control to move the agent image AG1 along a linear movement path from the first position PP1 to the second position SP1.

For example, since it is possible to detect the line of sight of each occupant of vehicle C1, it is possible to determine whether any occupant is looking at content CT2. For example, if any occupant of vehicle C1 is looking at content CT2, continuing to display the moving agent image AG1 is thought to make it easier to recognize the identity of the agent image AG1. However, if none of the occupants of vehicle C1 are looking at content CT2, it is assumed that no occupant will look at the agent image AG1 that overlaps with content CT2 while moving, even if the moving agent image AG1 continues to be displayed. Therefore, if none of the occupants of vehicle C1 are looking at content CT2, the display of the moving agent image AG1 may be stopped. In other words, after erasing the agent image AG1 from the first position PP1, the agent image AG1 may be displayed at the second position SP1. Note that if it is detected that any occupant is looking at content CT2, the agent image AG1 may be moved without overlapping with content CT2. An example of this is shown in Figure 11.

Figures 6(B) (C) and 7(A) to (C) show an example of a transition when an agent image AG1 is moved from a first position PP1 to a second position SP1 when a specific piece of content CT1 is displayed on the display unit 201. Figure 6(B) shows an example of moving the agent image AG1 behind the content CT1 (i.e., the opposite side of the display surface of the display unit 201).

As shown in Figure 6 (B), when the agent image AG1 moves along a linear path from the first position PP1 to the second position SP1, the moving agent image AG1 overlaps with the displayed content CT1. In this case, if the displayed content CT1 is specific content, it is important not to impede the visibility of the content CT1. Therefore, when the displayed content CT1 is specific content, the output control unit 128 executes display control such that the agent image AG1 moves behind the content CT1 (i.e., on the opposite side of the display surface of the display unit 201) along the linear path from the first position PP1 to the second position SP1. This makes it possible to maintain the identity of the moving agent image AG1 while not impeding the visibility of the content CT1.

Figure 6(C) shows an example in which a transparent or semi-transparent agent image AG1 is moved across the front side of content CT1 (i.e., the display surface side of the display unit 201).

As shown in FIG. 6(C), when the displayed content CT1 is specific content, the output control unit 128 executes display control to display a transparent or semi-transparent agent image AG1 moving on the front side of the content CT1 (i.e., the display surface side of the display unit 201) along a linear movement path from the first position PP1 to the second position SP1. This makes it possible to maintain the identity of the moving agent image AG1 while not impeding the visibility of the content CT1. The transparency of the agent image AG1 can be set according to the importance of the overlapping content. For example, if the content is specific and cannot be hidden by law, the importance is determined to be high, and the transparency of the agent image AG1 is set high. In other words, the agent image AG1 is set to be completely transparent. On the other hand, if the content is specific and it is considered acceptable for some of it to be hidden, the importance is determined to be low, and the transparency of the agent image AG1 is set low.

Figures 7(A) and (B) show an example of moving agent image AG1 so as to bypass content CT1.

As shown in Figures 7(A) and (B), when the content CT1 being displayed is specific content, the output control unit 128 executes display control to display the agent image AG1 moving along a detour route that bypasses the content CT1 as a movement route from the first position PP1 to the second position SP1. In Figures 7(A) and (B), the moving agent image AG1 is shown as agent images AG1a to AG1f.

Here, as shown in Figures 7(A) and (B), when moving agent image AG1 along a detour route, it is possible that the range of the detour route is narrower than the display area of agent image AG1. In this case, it is possible to reduce the size of agent image AG1 while it is moving, or to display part of agent image AG1 while it is moving behind content CT1 (i.e., the opposite side of the display surface of display unit 201), or to display it outside the display surface of display unit 201.

In the example shown in Figure 7 (A), the size of the moving agent image AG1 is reduced, and a portion of the moving agent image AG1 is displayed outside the display surface of the display unit 201.

In the example shown in Figure 7 (B), the size of the moving agent image AG1 is reduced, and a display mode is shown in which part of the moving agent image AG1 is displayed on the back side of the content CT1 (i.e., the opposite side of the display surface of the display unit 201) and on the outside of the display surface of the display unit 201.

Figure 7(C) shows an example of moving the agent image AG1 so as to bypass specific content CT3.

As shown in Figure 7(C), when the content CT3 being displayed is specific content, the output control unit 128 executes display control to display the agent image AG1 moving along a detour route that bypasses the content CT3 as a movement route from the first position PP1 to the second position SP1. In Figure 7(C), the moving agent image AG1 is shown by agent images AG1a to AG1d.

Here, as shown in Figure 7(C), when moving the agent image AG1 along a detour route, it is possible that the range of the detour route will be larger than the display area of the agent image AG1. In this case, it is possible to maintain the size of the agent image AG1 while it is moving.

It is also possible that there are areas on the detour route where the agent image AG1 and the specific content do not overlap, but are close to each other. For example, in the example shown in Figure 7(A), the lower parts of the moving agent images AG1b to AG1e are close to the upper part of the specific content CT1. In such cases, it is also possible that the proximity of the agent image AG1 to the specific content will reduce the visibility of the content. Therefore, if there is a close area on the detour route where the agent image AG1 and the specific content are close to each other, the agent image AG1 in that close area may be displayed transparent or semi-transparent. The distance for proximity determination can be set appropriately based on experiments, simulations, etc. For example, a distance of several millimeters to several centimeters can be used as the criterion for determining proximity.

As shown in Figures 6 and 7, when moving the agent image AG1, the movement trajectory of the agent image AG1 may be expressed using animation processing, etc.

This makes it possible to maintain the identity of the moving agent image AG1 while not impeding the visibility of the content CT1. The detour route and display mode of the agent image AG1 for specific content can be set appropriately based on experiments, simulations, etc.

Here, there are cases where it is possible to set multiple detour routes that bypass specific content. In such cases, it is possible to set detour routes based on preset criteria. For example, it is possible to set detour routes based on the line of sight of the occupants. For example, the occupants who are viewing specific content are identified based on the line of sight of each occupant. Then, based on the line of sight of the occupant viewing the specific content, the position on the display surface of the display unit 201 that the occupant is most likely to be looking at (focus position) is identified. Then, from among the multiple detour routes that bypass the specific content, the detour route closest to the focus position is determined. This allows the occupants to easily recognize the movement of the agent image AG1 while it is moving.

[Example of movement when multiple specific contents exist on the movement route]
The above has shown an example of the movement of the agent image AG1 when one specific content currently being displayed overlaps with the moving agent image AG1. However, it is also possible that when multiple specific contents are displayed on the display unit 201, two or more of these specific contents overlap with the moving agent image AG1. In this case, too, it is possible to move the agent image AG1 in the same way as in the examples shown in Figures 6(B) and (C) and Figures 7(A) to (C). An example of this is shown in Figure 8.

Figure 8 shows an example of the transition of the agent image AG1 when the agent image AG1 is moved from the first position PP1 to the second position SP1 when multiple specific contents CT3 and CT4 are displayed on the display unit 201 of the output device 200.

As shown in Figure 8, when the agent image AG1 moves along a linear path from the first position PP1 to the second position SP1, the moving agent image AG1 overlaps with the displayed content CT3 and CT4. In this case, too, it is possible to move the agent image AG1, similar to the examples shown in Figures 6(B)(C) and 7(A) to (C). Figure 8 shows a modified example of the example shown in Figure 7(A), in which the size of the moving agent image AG1 is maintained, and a portion of the moving agent image AG1 is displayed outside the display surface of the display unit 201.

[Example of movement when an agent image moves on a detour route across multiple displays]
The above has shown an example of movement of the agent image AG1 when a specific content being displayed on the display unit 201 overlaps with the moving agent image AG1. Here, it is also possible that when one or more specific contents are displayed on multiple display units (including the display unit 201), the specific content and the moving agent image AG1 overlap. In this case, too, it is possible to move the agent image AG1 in the same way as in the examples shown in Figures 6(B) and (C), 7(A) to (C), and 8. An example of this is shown in Figure 9.

Figure 9 shows an example of the transition of the agent image AG1 when the agent image AG1 is moved from the first position PP1 to the second position SP1 when a specific content CT5 is displayed on the display unit 201 of the output device 200 and a specific content CT6 is displayed in the HUD display area of the windshield 4.

Here, when displaying various images in the HUD display area of the windshield 4, a HUD display device for realizing the HUD is provided at the top of the dashboard 2, near the boundary with the windshield 4.

The HUD display device is a display device, such as a projector or optical system, that projects light onto the display area 4a of the windshield 4 and uses the reflected light to create a HUD display that shows a virtual image to the driver D1. That is, the light projected from the HUD display device onto the display area 4a of the windshield 4 is reflected by the windshield 4, and the reflected light heads toward the eyes of the driver D1. The reflected light that is projected onto the display area 4a and enters the eyes of the driver D1 is then displayed together with actual objects visible through the windshield 4, superimposed on those objects. In this way, the HUD display device creates a HUD display by using the windshield 4 to display a virtual image.

For example, the HUD display device displays the agent image AG1 in the display area 4a based on the control of the information processing device 110 (see Figure 5). The windshield 4 also functions as a display medium for the HUD of the vehicle C1.

As shown in FIG. 9, when the agent image AG1 moves along a linear path (a straight line in three-dimensional space or a straight line in two-dimensional space based on the occupant's visual field) from the first position PP1 to the second position SP1, the moving agent image AG1 overlaps with the displayed content CT5 and CT6. In this case, the agent image AG1 can be moved in the same way as in the examples shown in FIGS. 6(B)(C), 7(A) to (C), and 8. Similarly to the example shown in FIG. 8, FIG. 9 shows an example of a display mode in which the size of the moving agent image AG1 is maintained and a portion of the moving agent image AG1 is displayed outside the display surface of the display unit 201. Here, when moving the agent image AG1 between multiple physically separated display units, it is preferable to set a detour route or the like within the range that the occupant viewing the agent image AG1 can create. For example, if a detour route would make the movement difficult to see, the agent image AG1 can be displayed transparently or semi-transparently.

Note that while Figure 9 shows an example in which specific content is displayed on the display unit 201 and the HUD display area on the windshield 4, this is not limiting. For example, if a display unit is provided on the steering wheel 3, it is also possible for specific content to be displayed on that display unit. In this case, it is possible to perform a movement process for the agent image AG1 based on the overlap state between the specific content displayed on at least one of the display unit 201, the HUD display area on the windshield 4, and the display unit of the steering wheel 3 and the moving agent image AG1.

In this way, this embodiment can be understood as a display system comprising at least one or more display units that display the agent image AG1, and a control unit that controls the position at which the agent image AG1 is displayed on any of the display units.

The above example shows how the movement processing of the agent image AG1 is changed when the agent image AG1 moving along a linear path overlaps with specific content. Here, for example, if the agent image AG1 moving along a linear path does not overlap with the specific content but is close to it, it is possible that a passenger looking at the specific content may find the visibility of the specific content reduced due to the influence of the nearby agent image AG1. Therefore, the movement processing of the agent image AG1 may also be changed in a similar manner when the agent image AG1 moving along a linear path is close to the specific content. For example, a determination is made as to whether the linear path passes through an area (proximity area) within a predetermined distance of the specific content. If it is determined that the linear path passes through the proximity area, the agent image AG1 is moved along a path (detour route) that passes through a position farther away than the predetermined distance. The display mode of the proximity area may also be changed to move the agent image AG1. The movement processing in these cases is the same as the movement processing shown in Figures 6 to 9, etc.

[Example of operation of information processing system]
10 and 11 are flowcharts showing an example of agent migration processing in the information processing system 100. This agent migration processing is executed by the control unit 120 (see FIG. 5) based on a program stored in the storage unit 130 (see FIG. 5). This agent migration processing is constantly executed for each control cycle. This agent migration processing will be explained with appropriate reference to FIGS. 1 to 9.

In step S501, the output content determination unit 126 determines whether or not user input has been received. For example, if a manual operation by any of the occupants is received by the reception unit 203, if a voice operation by any of the occupants is received by the speech acquisition unit 121, or if a gesture operation by any of the occupants is received by the driver status acquisition unit 122 or the vehicle interior status acquisition unit 123, it is determined that user input has been received. If user input has been received, the process proceeds to step S502. On the other hand, if user input has not been received, monitoring continues.

In step S502, the output content determination unit 126 determines whether the user input received in step S501 is a voice input. That is, if the speech acquisition unit 121 receives a voice operation from any of the occupants, the user input is determined to be a voice input. If the user input is a voice input, the process proceeds to step S503. On the other hand, if the user input is not a voice input, the process proceeds to step S510 (see FIG. 11).

In step S503, the output content determination unit 126 acquires the current display position (first position) of the agent image AG1 and determines the destination display position (second position) of the agent image AG1. The first position can be acquired based on the display position of the agent image AG1 stored in the agent management DB 132. For example, the position of the occupant who made the user input (e.g., hand position, eye position) can be used as a reference, and the position on the surface of the display unit 201 closest to the occupant's position can be determined as the second position. The occupant who made the user input can be determined based on the relationship between the timing at which the utterance was acquired by the speech acquisition unit 121 and the occupant's mouth movement acquired by the driver status acquisition unit 122 or the vehicle interior status acquisition unit 123. For example, if there is an occupant moving their mouth at the time the utterance was made, it can be assumed that the occupant is making a sound as user input. The presence or absence of an occupant in each seat of the vehicle C1 can be determined based on detection values from seat occupancy sensors, seat belt sensors, etc. Furthermore, the presence or absence of an occupant in each seat of the vehicle C1 may be determined based on images acquired by the driver status acquisition unit 122 or the vehicle interior status acquisition unit 123. Furthermore, if multiple microphones or the like are installed in the vehicle C1, the location where a sound is generated can be identified based on sounds acquired by those microphones. Therefore, the location of the occupant who made the user input may be identified using technology for identifying the location where these sounds are generated.

For example, if the user input is a voice input, it is not a manual operation, and therefore the second position can be determined based on the eye position of the occupant who made the user input, rather than the hand position. In this case, the position on the display surface of the display unit 201 that is closest to the eye position of the occupant who made the user input can be determined as the second position. Also, for example, if the user input is something other than a voice input (for example, a manual operation), the third position (shown in step S511) can be determined based on the hand position of the occupant who made the user input. In this case, the position on the display surface of the display unit 201 that is closest to the hand position of the occupant who made the user input can be determined as the third position (see step S511).

In step S504, the movement mode determination unit 127 determines whether the user input received in step S501 was made by driver D1. For example, when driver D1 is driving, manual operation is often not possible, so it is assumed that user input is often made by voice. The method for identifying the occupant who made the user input is the same as the method shown in step S503. If the user input was made by driver D1, the process proceeds to step S505. On the other hand, if the user input was made by an occupant other than the driver, the process proceeds to step S506.

In step S505, the movement mode determination unit 127 determines whether the driving load of driver D1 who provided user input is less than a threshold. The method for detecting the driving load is the same as the detection method described above. The threshold value shown here is determination information for determining whether the driving load of driver D1 is high or low. This threshold value can be set appropriately based on experiments, simulations, etc. Note that here, an example is shown in which it is determined whether the driving load of driver D1 is less than a threshold value, but if autonomous driving is not being performed, it may also be determined whether the vehicle C1 is moving or stopped. If the driving load of driver D1 is less than the threshold value, the process proceeds to step S506. On the other hand, if the driving load of driver D1 is equal to or greater than the threshold value, the process proceeds to step S510.

In step S506, the movement mode determination unit 127 checks the relationship between the first movement trajectory of the agent image AG1 moving from the first position to the second position determined in step S503 and one or more pieces of content displayed on the display unit 201. For example, the movement mode determination unit 127 determines whether or not there is content currently being displayed between the first position and the second position, and if there is content currently being displayed between the first position and the second position, determines whether or not that content overlaps with the first movement trajectory of the agent image AG1. Note that the method for determining whether or not the first movement trajectory of the agent image AG1 moving from the first position to the second position overlaps with the currently displayed content is the same as the determination method shown in Figure 4 (B).

In step S507, the movement mode determination unit 127 determines whether specific content exists among the displayed content that overlaps with the first movement trajectory of the agent image AG1 moving from the first position to the second position. If specific content exists that overlaps with the first movement trajectory, the process proceeds to step S508. On the other hand, if specific content does not exist that overlaps with the first movement trajectory, the process proceeds to step S509.

In step S508, the movement mode determination unit 127 determines a movement path and display mode for moving the agent image AG1 from the first position to the second position so as not to impair the visibility of the specific content determined in step S507 to overlap with the first movement trajectory of the agent image AG1. The output control unit 128 then moves the agent image AG1 from the first position to the second position in accordance with the determined movement path and display mode. For example, as shown in Figures 6(B) (C) and 7 to 9, it is possible to move the agent image AG1 so as not to impair the visibility of the specific content CT1.

In step S509, the movement mode determination unit 127 determines to move the agent image AG1 along the first movement trajectory. Then, the output control unit 128 moves the agent image AG1 from the first position to the second position along the first movement trajectory. In other words, the agent image AG1 is moved over the shortest distance. For example, as shown in FIG. 6(A), it is possible to move the agent image AG1 so that it overlaps with the front side of the currently displayed content CT2 (i.e., the display surface side of the display unit 201).

In step S510, the movement mode determination unit 127 determines to move only the position of the agent image AG1 without executing a movement effect that moves the agent image AG1. In other words, the movement mode determination unit 127 determines to erase the agent image AG1 displayed at the first position, and then display the agent image AG1 at the second position. Then, the output control unit 128 erases the agent image AG1 displayed at the first position, and then displays the agent image AG1 at the second position.

As mentioned above, when driver D1 is driving, manual operation is often not possible, so it is assumed that user input is often performed via voice. Furthermore, when driver D1's driving load is high (above a threshold), it is thought that driver D1 does not have the time to see the movement of agent image AG1, so even if agent image AG1 is temporarily erased, it is thought that this will not affect the presence of agent image AG1. Furthermore, when driver D1's driving load is high (above a threshold), it is thought that it is easier for driver D1 to recognize agent image AG1 if it is moved instantaneously from a first position to a second position rather than by executing a movement effect. Therefore, in step S510, only the display position of agent image AG1 is changed without executing a movement effect for agent image AG1.

In step S511, the output content determination unit 126 acquires the current display position (first position) of the agent image AG1 and determines the destination display position (third position) of the agent image AG1. Here, assuming that the user input is manual input rather than voice input, an example is shown in which the hand position of the occupant who made the user input is used as the reference position, and the position on the surface of the display unit 201 closest to the occupant's hand position is determined as the third position. The occupant who made the user input (manual input) can be determined based on the relationship between the timing at which the operation is accepted by the acceptance unit 203 and the occupant's hand movement acquired by the driver status acquisition unit 122 or the vehicle interior status acquisition unit 123. For example, if an occupant is moving their hand near the display unit 201 at the timing at which the acceptance unit 203 accepts the operation, it can be assumed that the occupant is performing a manual operation as user input. Furthermore, if a touch operation is performed on the surface of the display unit 201, the hand position can be determined to be the position at which the touch operation was performed.

For example, it is assumed that a selection operation to select an operation button related to content displayed on the display unit 201 is performed as a user input. In this case, in order to provide assistance with that operation, it is possible to move the agent image AG1 near the operation button that was selected.

The processes in steps S512 to S514 and S516 correspond to the processes in steps S506 to S509, so explanations will be omitted here.

In step S515, the movement mode determination unit 127 determines whether the occupant is looking at the displayed content (other than the specific content) that was determined to overlap with the first movement trajectory of the agent image AG1 in step S513. As described above, when driver D1 is driving, manual operation is often not possible. Therefore, it is assumed that the occupant targeted for each process in steps S511 to S516 is an occupant other than driver D1 (or a driver D1 whose driving load is below a threshold). It is also assumed that an occupant other than driver D1 (or a driver D1 whose driving load is below a threshold) is enjoying viewing the content displayed on the display unit 201. Therefore, if there is an occupant viewing the displayed content, it is important not to impede the visibility of that content, even if the content is not the specific content. Therefore, if the occupant is viewing the displayed content, a movement effect is executed that does not impede the visibility of that content. This movement effect can be the same as the movement effect for the specific content. That is, in step S514, the content the occupant is viewing is also subject to movement effect, along with the specific content.

Note that while the above describes an example in which the movement path or display mode of the agent image AG1 is changed so as not to impede the visibility of specific content, this embodiment is not limited to this. For example, the movement timing or movement speed of the agent image AG1 may be changed so as not to impede the visibility of specific content. For example, by moving the agent image AG1 at a speed faster than the normal movement speed, it is possible to avoid impeding the visibility of specific content. For example, even if there is an overlapping portion between the agent image AG1 and the specific content, it is possible to shorten the time during which the specific content is invisible by moving the agent image AG1 quickly along a movement path that includes the overlapping portion. Furthermore, for example, it is possible to determine whether any occupant of vehicle C1 is looking at specific content, and move the agent image AG1 (at a normal speed or a speed faster than normal) when the occupant is not looking at the specific content. For example, when moving the agent image AG1 along a travel route that includes an overlapping portion where the agent image AG1 and specific content overlap, it is possible to prevent the occupant from feeling that the specific content is not visible by moving the agent image AG1 at a time when the occupant is not looking at the specific content.

10 and 11 show an example in which the control content is changed based on whether the user input is a voice input, but this embodiment is not limited to this and may be realized using other control examples. For example, the agent movement process (steps S503 to S510 or steps S511 to S516) may be executed regardless of whether any user input is made. Also, FIG. 10 shows a control process in which the movement mode of the agent image AG1 is changed based on the process of determining whether the driver's driving load is high or low (step S505). However, this embodiment is not limited to this and may be realized using other control examples. For example, the process of determining whether the driver D1's driving load is high or low (steps S505, S510) may be omitted. Also, FIG. 11 shows a control process in which the movement mode of the agent image AG1 is changed based on the process of determining whether the occupant is viewing content (step S515). However, this embodiment is not limited to this and may be realized using other control examples. For example, the process of determining whether the occupant is viewing content (step S515) may be omitted.

Also, an example has been shown in which a control process is performed to change the movement pattern of the agent image AG1 based on whether or not the agent image AG1 overlaps with specific content. However, this control process may be performed using other methods. For example, the control process can be performed using artificial intelligence (AI). For example, various situations related to the driver D1, vehicle C1, etc., the movement pattern of the agent to be executed in response to those situations, and the overlapping state between the agent image AG1 and specific content can be learned in advance, and this learned data can be used in the control process. For example, the learned data for various situations that arise related to the driver D1, vehicle C1, etc., and the overlapping state between the agent image AG1 and specific content can be used to determine the movement pattern of the agent, and the agent image AG1 can be moved in that determined movement pattern.

[Example of effects of this embodiment]
As described above, in this embodiment, when the agent image AG1 is moved to a position corresponding to a user input in response to the user input, it is possible to move the agent image AG1 so as not to obstruct the visibility of specific content (important content). In other words, if specific content is displayed along the shortest movement route of the agent image AG1, the agent image AG1 is moved to avoid the specific content so as not to obstruct the visibility of the specific content. For example, it is possible to move the agent image AG1 along a detour route, change the shape of the agent image AG1, change the transparency of the agent image AG1, change the movement speed of the agent image AG1, or change the movement timing of the agent image AG1.

As a result, the occupant who has made a user input can visually confirm (or feel the presence of) the agent image AG1 that moves in response to the user input, and the visibility of specific content is not obstructed. This allows the agent image AG1 to provide appropriate operational assistance to the user. It is also possible to create an impression of intelligence in the agent image AG1 that moves without obscuring specific content. This makes it possible to increase trust and attachment to the agent image AG1, and to improve the ability of the agent image AG1 to convey information to each occupant. In this way, in this embodiment, the agent image AG1 can be moved appropriately on the display unit 201 so as not to obstruct the visibility of specific content.

[Example of executing processing in other devices or other systems]
Although the above describes an example in which the determination process, detection process, control process, etc. are executed in the information processing device 110 (or the information processing system 100), all or part of each of these processes may be executed in other devices. In this case, the information processing system is configured by the devices that execute part of each of these processes. For example, at least part of each process can be executed using various information processing devices and various electronic devices, such as in-vehicle devices, devices that can be used by the user (e.g., smartphones, tablet terminals, personal computers, car navigation devices, IVIs), and servers that can be connected via a predetermined network such as the Internet.

Furthermore, part (or all) of the information processing system capable of executing the functions of the information processing device 110 (or the information processing system 100) may be provided by an application that can be provided via a predetermined network such as the Internet. This application is, for example, SaaS (Software as a Service).

[Configuration example and effects of this embodiment]
The information processing method of this embodiment is an information processing method in which an agent image AG1 that communicates with an occupant riding in vehicle C1 is displayed on display unit 201 (including each display unit such as the HUD display area of the front window 4), and the display state of the agent image AG1 is controlled based on the state of the occupant. This information processing method includes, when an agent image AG1 displayed at a first position (initial position) on the display unit 201 moves to a second position (a position to which the agent image AG1 moves in response to a user input) based on the state of the occupant, a determination process for determining whether or not the visibility of a specific content is obstructed when the agent image AG1 moves along the first path based on a linear movement path (first path) from the first position to the second position and the content displayed on the display unit 201 (steps S506, S507, S512, S513), and if the visibility of the specific content is not obstructed, moving the agent image AG1 along the first path in a first display mode (steps S509, S516). If the visibility of the specific content is obstructed, moving the agent image AG1 along the first path in a first display mode (steps S509, S516). The control process includes control processes (steps S508, S509, S514, S516) for executing one of the following: a first movement process for moving the agent image AG1 along a movement path (second path) that does not obstruct the visibility of the specific content; a second movement process for moving the agent image AG1 from a first position to a second position in a display mode (second display mode) that is different from the first display mode and does not obstruct the visibility of the specific content; and a third movement process for moving the agent image AG1 from the first position to the second position at a second movement timing that is different from the first movement timing in the first display mode and does not obstruct the visibility of the specific content, or at a second movement speed that is different from the first movement speed in the first display mode and does not obstruct the visibility of the specific content (steps S508, S514). For example, the first movement process is performed by the movement processes shown in FIGS. 7 to 9. Furthermore, for example, the second movement process is performed by the movement processes shown in FIGS. 6B and 6C. The program according to this embodiment is a program that causes a computer to execute each of these processes. In other words, the program according to this embodiment is a program that causes a computer to realize each of the functions that can be executed by the information processing device 110.

With this configuration, the agent image AG1, which moves based on the occupant's state, can be visually confirmed (or its presence, if transparent), and the visibility of specific content is not obstructed. In other words, the agent image AG1 can be moved appropriately on the display unit 201 so as not to obstruct the visibility of specific content.

In the information processing method according to this embodiment, the first movement process moves the agent image AG1 along a second route that is a detour route that bypasses specific content. For example, the agent image AG1 can be moved using the movement processes shown in Figures 7 to 9.

With this configuration, the agent image AG1 that bypasses a specific piece of content can be visually confirmed near the specific content.

In the information processing method according to this embodiment, in the first movement process, if there is a portion of the detour route where the agent image AG1 overlaps with specific content, or if there is a portion of the detour route where the agent image AG1 is close to specific content, the display mode is such that the agent image AG1 in the overlapping or close portion is transparent or semi-transparent, or the agent image AG1 in the overlapping portion is displayed as if it were positioned behind the specific content (the farther depthwise of the display unit 201). For example, as shown in FIG. 7(B), if there is a portion of the detour route where the agent image AG1 overlaps with specific content CT1, it is possible to display the agent image AG1 in the overlapping portion as if it were positioned behind the specific content CT1.

With this configuration, if the agent image AG1 and specific content overlap or come close to each other on a detour route, it is possible to prevent the specific content from becoming difficult to see due to that overlap or proximity.

In the information processing method according to this embodiment, in the first movement process, if there is a portion of the agent image AG1 that extends beyond the display surface of the display unit 201 on the detour route, that portion of the agent image AG1 that extends beyond the display surface of the display unit 201 is not displayed. For example, as shown in FIG. 7(A), if there is a portion of the agent image AG1 that extends beyond the display surface of the display unit 201 on the detour route (the upper portions of agent images AG1b to AG1e), it is possible to prevent that portion of the agent image AG1 from being displayed.

With this configuration, even if the route traveled by the agent image AG1 on a detour is narrow, it is possible to display the agent image AG1 while maintaining its size, etc.

In the information processing method according to this embodiment, the determination process (steps S506, S507, S512, S513) may determine that the moving agent image AG1 obstructs the visibility of specific content when the agent image AG1 moving along the first path is adjacent to or overlaps with specific content. In this case, the second movement process may move the agent image AG1 along the first path, and display the overlapping portion of the agent image AG1 as if it were positioned behind the specific content, or may display the adjacent or overlapping portion of the agent image AG1 as if it were positioned behind the specific content, as the second display mode. For example, as shown in FIG. 6(B), the agent image AG1 may move along the first path, and display the overlapping portion of the agent image AG1 as if it were positioned behind the specific content. Furthermore, as shown in FIG. 6(C), the overlapping portion of the agent image AG1 may be displayed as if it were transparent or semi-transparent. Additionally, the agent image AG1 in the vicinity of specific content may be displayed transparently or semi-transparently.

With this configuration, it is possible to move the agent image AG1 along the first route, and for parts close to or overlapping with the specific content, it is possible to display the agent image AG1 as if it were positioned behind the specific content, or to display the agent image AG1 as transparent or semi-transparent. This makes it possible to move the agent image AG1 along the same movement route as for content other than the specific content, making it possible to perform movement effects that are easy for the occupant to predict.

In the information processing method according to this embodiment, in the third movement process, the agent image AG1 may be moved at a second movement speed that is faster than the first movement speed. However, if the specific content is information that is required by law to be continuously displayed, this movement control is not executed.

With this configuration, even if there is an overlapping portion between the agent image AG1 and specific content, it is possible to shorten the time that the specific content is not visible by quickly moving the agent image AG1 along a path that includes the overlapping portion. This makes it possible to appropriately move the agent image AG1 on the display unit 201 so as not to impede the visibility of the specific content.

The information processing method according to this embodiment may further include a detection process for detecting the gaze of an occupant based on an image including the occupant's eyes. For example, the movement mode determination unit 127 may detect the gaze of an occupant (including driver D1) based on images acquired by the driver state acquisition unit 122 and the vehicle interior state acquisition unit 123. Note that known gaze detection technology may be used for this gaze detection. Then, in the third movement process, it may be determined whether the occupant is looking at specific content based on the occupant's gaze, and the timing when the occupant is not looking at the specific content may be set as the second movement timing.

With this configuration, when moving the agent image AG1 along a travel route that includes an overlapping portion where the agent image AG1 and specific content overlap, it is possible to move the agent image AG1 at a time when the occupant is not looking at the specific content. This makes it possible to appropriately move the agent image AG1 on the display unit 201 so as not to obstruct the visibility of the specific content.

In the information processing method according to this embodiment, the specific content can be at least one of map content displayed to guide the direction of travel of vehicle C1, route information content showing the route from the current location of vehicle C1 to the destination in the map content, driving control information content related to driving control of vehicle C1, abnormality occurrence information content related to the occurrence of an abnormality in vehicle C1, and information that is required by law to be continuously displayed.

This configuration makes it possible to set important information as specific content.

The information processing method according to this embodiment may further include a driving load determination process (step S505) that determines the driving load of driver D1 based on at least one of environmental information related to the environment outside vehicle C1 and driving behavior information related to driver D1's driving behavior. In this case, the determination process (steps S506, S507) determines whether the visibility of specific content is impaired if the driving load is below a threshold. Then, in the control process, if the driving load is equal to or greater than the threshold, the agent image AG1 displayed at the first position is erased and then displayed at the second position (step S510). If the driving load is below the threshold, a movement process of agent image AG1 is executed (steps S508, S508) based on the determination result of the determination process (steps S506, S507).

For example, if driver D1 is under a high driving load, it is thought that driver D1 will not have time to see the movement of agent image AG1, and therefore temporarily erasing agent image AG1 will not affect the presence of agent image AG1. Furthermore, if driver D1 is under a high driving load, it is thought that it will be easier for driver D1 to recognize agent image AG1 if it is moved instantaneously from a first position to a second position rather than by executing a movement effect. In this way, it is possible to appropriately perform the movement effect of agent image AG1 based on driver D1's driving load.

The information processing device 110 is an information processing device that displays an agent image AG1, which communicates with a passenger in the vehicle C1, on the display unit 201 (including each display unit such as the HUD display area of the windshield 4), and controls the display state of the agent image AG1 based on the state of the passenger. When the agent image AG1, which is displayed at a first position (initial position) on the display unit 210, moves to a second position (a position to which the image moves in response to user input) based on the state of the passenger, the information processing device 110 includes a movement mode determination unit 127 (an example of a determination unit) that determines whether or not the visibility of specific content will be obstructed when the agent image AG1 moves along the first path based on a linear movement path (first path) from the first position to the second position and the content displayed on the display unit 201; if the visibility of the specific content is not obstructed, the information processing device moves the agent image AG1 along the first path in a first display mode; and if the visibility of the specific content is obstructed, the information processing device 110 moves the agent image AG1 along a movement path different from the first path that obstructs the visibility of the specific content. The information processing device 110 includes an output control unit 128 (an example of a control unit) that executes one of the following: a first movement process that moves the agent image AG1 along a movement path (second path) that does not impair visibility of the specific content; a second movement process that moves the agent image AG1 from a first position to a second position in a display mode (second display mode) different from the first display mode that does not impair visibility of the specific content; and a third movement process that moves the agent image from the first position to the second position at a second movement timing different from the first movement timing in the first display mode that does not impair visibility of the specific content, or at a second movement speed different from the first movement speed in the first display mode that does not impair visibility of the specific content. Note that the information processing device 110 may be a device built into the output device 200 or may be a device separate from the output device 200. Furthermore, instead of the information processing device 110, an information processing system configured with multiple devices capable of executing the processes realized by the information processing device 110 may be implemented.

With this configuration, the agent image AG1, which moves based on the occupant's state, can be visually confirmed (or its presence, if transparent), and the visibility of specific content is not obstructed. In other words, the agent image AG1 can be moved appropriately on the display unit 201 so as not to obstruct the visibility of specific content.

Note that the processing steps shown in this embodiment are merely examples for realizing this embodiment, and the order of some of the processing steps may be changed, some of the processing steps may be omitted, or other processing steps may be added, to the extent that this embodiment can be realized.

Note that each process in this embodiment is executed based on a program that causes a computer to execute various processing procedures. This embodiment can also be understood as an embodiment of a program that realizes the function of executing each process, and a recording medium that stores that program. For example, an update process for adding a new function to an information processing device can store the program in the storage device of the information processing device. This makes it possible to cause the updated information processing device to perform each process shown in this embodiment.

The above describes an embodiment of the present invention, but the above embodiment merely illustrates an application example of the present invention, and is not intended to limit the technical scope of the present invention to the specific configuration of the above embodiment.

Claims (11)

1. An information processing method for displaying an agent image that communicates with a vehicle occupant on a display unit, and controlling a display state of the agent image based on a state of the occupant, comprising:
a determination process for determining, when the agent image displayed at a first position on the display unit moves to a second position based on the state of the occupant, whether or not the visibility of specific content is obstructed when the agent image moves along a first route, which is a linear movement route from the first position to the second position, and based on content displayed on the display unit;
When the visibility of the specific content is not impaired, the agent image is moved along the first route in a first display mode;
If the visibility of the specific content is impaired,
a first movement process of moving the agent image along a second route, which is a movement route different from the first route and does not obstruct the visibility of the specific content;
a second movement process of moving the agent image from the first position to the second position in a second display mode that is a display mode different from the first display mode and does not impair visibility of the specific content;
a control process for executing either a second movement timing different from a first movement timing in the first display mode and not interfering with the visibility of the specific content, or a third movement process for moving the agent image from the first position to the second position at a second movement speed different from a first movement speed in the first display mode and not interfering with the visibility of the specific content,
Information processing methods. 2. The information processing method according to claim 1,
In the first movement process, the agent image is moved along a detour route that bypasses the specific content as the second route.
Information processing methods. 3. The information processing method according to claim 2,
In the first movement process, if there is a portion on the detour route where the agent image overlaps with the specific content, or if there is a portion on the detour route where the agent image is close to the specific content, the display mode is set to make the agent image in the overlapping portion or the close portion transparent or semi-transparent, or to display the agent image in the overlapping portion as if it were positioned behind the specific content.
Information processing methods. 3. The information processing method according to claim 2,
In the first movement process, if there is a portion of the agent image that protrudes from the display surface of the display unit on the detour route, the protruding portion of the agent image is not displayed.
Information processing methods. 2. The information processing method according to claim 1,
In the determination process, when the agent image moving along the first route is close to or overlaps with the specific content, it is determined that the moving agent image obstructs visibility of the specific content;
In the second movement process,
moving the agent image along the first path;
The second display mode is a display mode in which the agent image in the overlapping portion with the specific content is displayed as if it were positioned behind the specific content, or a display mode in which the agent image in the proximity or overlapping portion with the specific content is made transparent or semi-transparent.
Information processing methods. 2. The information processing method according to claim 1,
In the third movement process, the agent image is moved at a second movement speed that is faster than the first movement speed.
Information processing methods. 2. The information processing method according to claim 1,
a detection process for detecting a line of sight of the occupant based on an image including the eyes of the occupant,
In the third movement process, it is determined whether the occupant is watching the specific content based on a line of sight of the occupant, and a time when the occupant is not watching the specific content is set as the second movement timing.
Information processing methods. 8. An information processing method according to claim 1,
The specific content is at least one of map content displayed to guide the vehicle in the traveling direction, route information content indicating a route from the current location of the vehicle to the destination in the map content, driving control information content related to driving control of the vehicle, abnormality occurrence information content related to the occurrence of an abnormality in the vehicle, and information content that is required by law to be continuously displayed.
Information processing methods. 8. An information processing method according to claim 1,
a driving load determination process for determining a driving load of the driver based on at least one of environmental information relating to an environment outside the vehicle and driving behavior information relating to a driving behavior of the driver,
In the determination process, when the driving load is less than a threshold, it is determined whether visibility of the specific content is impaired;
In the control process,
When the driving load is equal to or greater than a threshold, the agent image displayed at the first position is erased, and then the agent image is displayed at the second position;
If the driving load is less than a threshold, a process of moving the agent image is executed based on the determination result of the determination process.
Information processing methods. 1. An information processing device that displays an agent image that communicates with a vehicle occupant on a display unit and controls a display state of the agent image based on a state of the occupant,
a determination unit that, when the agent image displayed at a first position on the display unit moves to a second position based on the state of the occupant, determines whether or not the visibility of specific content is impaired when the agent image moves along a first route, which is a linear movement route from the first position to the second position, based on the content displayed on the display unit;
When the visibility of the specific content is not impaired, the agent image is moved along the first route in a first display mode;
If the visibility of the specific content is impaired,
a first movement process of moving the agent image along a second route, which is a movement route different from the first route and does not obstruct the visibility of the specific content;
a second movement process of moving the agent image from the first position to the second position in a second display mode that is a display mode different from the first display mode and does not impair visibility of the specific content;
a control unit that executes either a second movement timing that is different from the first movement timing in the first display mode and does not impair visibility of the specific content, or a third movement process that moves the agent image from the first position to the second position at a second movement speed that is different from the first movement speed in the first display mode and does not impair visibility of the specific content,
Information processing device. A program executed by a computer to display an agent image that communicates with a vehicle occupant on a display unit and control a display state of the agent image based on a state of the occupant,
a determination process for determining, when the agent image displayed at a first position on the display unit moves to a second position based on the state of the occupant, whether or not the visibility of specific content is obstructed when the agent image moves along a first route, which is a linear movement route from the first position to the second position, and based on content displayed on the display unit;
When the visibility of the specific content is not impaired, the agent image is moved along the first route in a first display mode;
If the visibility of the specific content is impaired,
a first movement process of moving the agent image along a second route, which is a movement route different from the first route and does not obstruct the visibility of the specific content;
a second movement process of moving the agent image from the first position to the second position in a second display mode that is a display mode different from the first display mode and does not impair visibility of the specific content;
a control process for executing either a second movement timing different from a first movement timing in the first display mode and not interfering with the visibility of the specific content, or a third movement process for moving the agent image from the first position to the second position at a second movement speed different from a first movement speed in the first display mode and not interfering with the visibility of the specific content;
A program that causes a program to execute the following.