JP2021131472A - Information processor, information processing method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To realize a device and a method for generating and outputting a response utterance on behalf of a user to a device utterance output by an interactive device. SOLUTION: A device utterance from a user utterance partner device, which is a user's utterance partner, is input, and a user proxy utterance is generated and output on behalf of the user. It has a response necessity determination unit for determining the necessity of user proxy utterance, and a response generation unit for generating user proxy utterance when it is determined that user proxy utterance is necessary. The response generation unit generates and outputs a proxy utterance that reflects the user's intention by referring to, for example, the user behavior history information. The response necessity determination unit determines that the user proxy utterance is necessary when the user utterance is not performed within the predetermined threshold time from the utterance completion timing of the device utterance. [Selection diagram] FIG. 4

Description

The present disclosure relates to an information processing device, an information processing method, and a program. More specifically, the present invention relates to an information processing device that speaks on behalf of the user, an information processing method, and a program.

Recently, many voice dialogue systems have been put into practical use as hands-free user interfaces.
The voice dialogue system analyzes the user's utterance input through the microphone, and performs various processes and responses based on the analysis result.

As an example of such a system, for example, there is a car navigation (car navigation) device mounted on a vehicle.
The car navigation device analyzes user utterances such as a driver, for example, user utterances related to designation of a destination, displays a route to the destination, and makes an announcement (system utterance) for a driving guide.

A car navigation device that executes processing according to a user's utterance is described in, for example, Patent Document 1 (US Pat. No. 5,274,560).
However, the user needs to speak to a voice dialogue system such as a car navigation device, which may cause a mental burden on the user. For example, if the driver is nervous while driving on an unfamiliar road and makes the driver think about the content of the utterance to the car navigation system, the driver's concentration on driving may be reduced and the driver may be in a dangerous situation. ..

When using not only a car navigation system but also various other voice dialogue systems, it is not easy for the user to respond to inquiries and utterances output from the voice dialogue system in a timely manner. In particular, it is not easy for a user who is unfamiliar with conversation with the system, and it is often a burden on the user.

U.S. Pat. No. USP5,274,560

The present disclosure has been made in view of the above problems, for example, even if the user does not speak to a voice interactive device that analyzes and processes the user's utterance, such as a car navigation device. , An information processing device that speaks on behalf of a user, an information processing method, and a program.

The first aspect of the disclosure is
Input the device utterance output from the user utterance partner device, which is the user's utterance partner,
It has a data processing unit that generates and outputs a user substitute utterance in place of the user in response to the device utterance.
The data processing unit
A response necessity determination unit that determines the necessity of the user proxy utterance, and a response necessity determination unit.
The information processing device has a response generation unit that generates a user proxy utterance when the response necessity determination unit determines that a user proxy utterance is necessary.

Further, the second aspect of the present disclosure is
It is an information processing method that executes information processing in an information processing device.
The information processing device
Input the device utterance output from the user utterance partner device, which is the user's utterance partner,
It has a data processing unit that generates and outputs a user substitute utterance in place of the user in response to the device utterance.
The data processing unit
The response necessity determination process for determining the necessity of the user proxy utterance and the response necessity determination process
This is an information processing method for executing a response generation process for generating a user proxy utterance when it is determined in the response necessity determination process that a user proxy utterance is necessary.

Further, the third aspect of the present disclosure is
A program that executes information processing in an information processing device.
The information processing device
Input the device utterance output from the user utterance partner device, which is the user's utterance partner,
It has a data processing unit that generates and outputs a user substitute utterance in place of the user in response to the device utterance.
The program is installed in the data processing unit.
The response necessity determination process for determining the necessity of the user proxy utterance and the response necessity determination process
There is a program that executes a response generation process for generating a user proxy utterance when it is determined that a user proxy utterance is necessary in the response necessity determination process.

The program of the present disclosure is, for example, a program that can be provided by a storage medium or a communication medium that is provided in a computer-readable format to an information processing device or a computer system that can execute various program codes. By providing such a program in a computer-readable format, processing according to the program can be realized on an information processing device or a computer system.

Still other objectives, features and advantages of the present disclosure will become apparent by more detailed description based on the examples of the present disclosure and the accompanying drawings described below. In the present specification, the system is a logical set configuration of a plurality of devices, and the devices having each configuration are not limited to those in the same housing.

According to the configuration of one embodiment of the present disclosure, a device and a method for generating and outputting a response utterance on behalf of the user with respect to the device utterance output by the interactive device are realized.
Specifically, for example, the device utterance from the user utterance partner device, which is the user's utterance partner, is input, and the user substitute utterance is generated and output on behalf of the user. It has a response necessity determination unit that determines the necessity of a user proxy utterance, and a response generation unit that generates a user proxy utterance when it is determined that a user proxy utterance is necessary. The response generation unit generates and outputs a proxy utterance that reflects the user's intention by referring to, for example, the user action history information. The response necessity determination unit determines that the user proxy utterance is necessary when the user utterance is not performed within the predetermined threshold time from the utterance completion timing of the device utterance.
With this configuration, a device and a method for generating and outputting a response utterance on behalf of the user for the device utterance output by the interactive device are realized.
The effects described in the present specification are merely exemplary and not limited, and may have additional effects.

It is a figure explaining the use example and the problem of a voice dialogue system. It is a figure explaining the use example and the problem of a voice dialogue system. It is a figure explaining the use example and the problem of a voice dialogue system. It is a figure explaining the dialogue example which added the user substitute utterance device in addition to the user and a user utterance partner device. It is a figure explaining the dialogue example which added the user substitute utterance device in addition to the user and a user utterance partner device. It is a figure explaining the dialogue example which added the user substitute utterance device in addition to the user and a user utterance partner device. It is a figure explaining the example of the specific dialogue sequence accompanied by the utterance of a user substitute utterance apparatus. It is a figure explaining the example of the specific dialogue sequence accompanied by the utterance of a user substitute utterance apparatus. It is a figure explaining the example of the specific dialogue sequence accompanied by the utterance of a user substitute utterance apparatus. It is a figure explaining the configuration example of the user substitute utterance apparatus of this disclosure. It is a figure explaining the configuration example of the user substitute utterance apparatus of this disclosure. It is a figure which shows the flowchart explaining the sequence of the process executed by the response necessity determination part of the user substitute utterance apparatus of this disclosure. It is a figure which shows the flowchart explaining the sequence of the process executed by the response necessity determination part of the user substitute utterance apparatus of this disclosure. It is a figure which shows the flowchart explaining the sequence of the process executed by the response necessity determination part of the user substitute utterance apparatus of this disclosure. It is a figure which shows the flowchart explaining the sequence of the process executed by the response necessity determination part of the user substitute utterance apparatus of this disclosure. It is a figure explaining the specific example of the input utterance correspondence response database held by the user substitute utterance apparatus of this disclosure. It is a figure explaining the specific example of the user behavior history information held by the user substitute utterance apparatus of this disclosure. It is a figure explaining the structure and processing of the response generation part of the user substitute utterance apparatus of this disclosure. It is a figure which shows the flowchart explaining the sequence of processing executed by the response generation part of the user substitute utterance apparatus of this disclosure. It is a figure which shows the flowchart explaining the sequence of processing executed by the response generation part of the user substitute utterance apparatus of this disclosure. It is a figure explaining the hardware configuration example of an information processing apparatus.

Hereinafter, the details of the information processing apparatus, the information processing method, and the program of the present disclosure will be described with reference to the drawings. The explanation will be given according to the following items.
1. 1. Outline of processing executed by the information processing apparatus of the present disclosure 2. Regarding an example of a specific dialogue sequence accompanied by the utterance of the user proxy utterance device of the present disclosure. 4. Regarding the configuration example of the user proxy utterance device of the present disclosure. Details of the processing executed by each component of the user proxy utterance device 4- (1). Details of the processing executed by the utterance detection unit 101 4- (2). Details of the processing executed by the voice recognition unit 102 4- (3). Details of the process executed by the response necessity determination unit 103 4- (4). Details of the processing executed by the response generation unit 104 4- (5). Details of the process executed by the voice synthesizer 105 5. About hardware configuration example of information processing device 6. Summary of the structure of this disclosure

[1. Outline of processing executed by the information processing apparatus of the present disclosure]
First, the outline of the processing executed by the information processing apparatus of the present disclosure will be described with reference to FIGS. 1 and 1 and below.

As mentioned above, recently, voice dialogue systems have been used in various fields as hands-free user interfaces.
A voice dialogue system is a system that analyzes user utterances input through a microphone and performs various processes and responses based on the analysis results. As an example of such a system, for example, a car navigation system mounted on a vehicle ( There is a car navigation) device.

The car navigation device analyzes user utterances such as a driver, for example, user utterances related to designation of a destination, displays a route to the destination, and makes an announcement (system utterance) for a driving guide.
However, when the user speaks to a voice dialogue system such as a car navigation device, the user's mental burden may occur and the dialogue with the system may be interrupted.
In addition, if a driver who is driving on an unfamiliar road while being tense is made to think about the content of the utterance to the car navigation system, the driver's concentration on driving may be reduced and a dangerous situation may occur.

An example of using the voice dialogue system and a specific example of the problem will be described with reference to FIG. 1 and below.
FIG. 1 shows the car navigation device as the user utterance partner device 10.
User 1 is, for example, a vehicle driver.

First, the user 1 makes the following user utterance in step S11.
User utterance = "Tell me the route to Tokyo Tower"
In response to this user utterance, the user utterance partner device 10 which is a car navigation device utters the following device in step S12.
Device utterance = "There are three candidates for the route to Tokyo Tower. Which one do you want?"

When the user 1 is concentrating on driving, he / she may miss the utterance of this device. Also, even if you listen to it, you may not be able to judge it immediately.
In such a case, in S13, the user utterance is not performed from the user 1, and then the car navigation device cannot proceed with the process.

The user utterance partner device 10 is not limited to the car navigation device. Various interactive devices can be the user utterance partner device 10.
FIG. 2 shows an example in which the user speaking partner device 10 is used as an English conversation lesson device.

The user utterance partner device 10 which is an English conversation lesson device utters the following devices in step S21.
Device utterance = "Have you ever been to New York?"

If the user 1 cannot hear or understand the device utterance, the user utterance is not performed in step S22. After that, the dialogue with the user speaking partner device 10 which is the English conversation lesson device is interrupted, and the English conversation lesson device cannot proceed with the process.

FIG. 3 shows an example in which the user utterance partner device 10 is used as a character dialogue device. The character dialogue device is a device that enables various dialogues such as daily conversations between the user and the character.
The user utterance partner device 10, which is a character dialogue device, performs the following device utterance in step S31.
Device utterance = "Where did you go today?"

If the user 1 cannot hear the device utterance, is wondering what to answer, or forgets the name of the place, the user utterance is performed in step S32. No. After that, the dialogue with the user utterance partner device 10 which is the character dialogue device is interrupted.

In this way, there are many situations in which the user 1 cannot speak to the user utterance partner device 10 which is a voice dialogue system in a timely manner. In such a case, the dialogue between the user 1 and the user utterance partner device 10 is performed. It will be interrupted and the processing on the user utterance partner device 10 side will be delayed.

The present disclosure prevents the occurrence of such a situation. That is, a user proxy utterance device that executes an utterance on behalf of the user 1 is provided.
A specific example of the process executed by the user proxy utterance device of the present disclosure will be described with reference to FIG. 4 and the following.

FIG. 4 is a diagram showing the car navigation device described above with reference to FIG. 1 as the user utterance partner device 10.
FIG. 4 further shows the user proxy utterance device 20.
The smartphone (smartphone) is shown as the user proxy utterance device 20.

The user proxy utterance device 20 may be a device other than a smartphone, for example, a PC or a tablet terminal. Furthermore, it can be realized as another dedicated information processing device. Further, the configuration may be integrated with the user utterance partner device 10 such as a car navigation device.

For example, when a smartphone (smartphone) is used as the user proxy utterance device 20, a program (application) that executes the user proxy utterance process is installed and used on the smartphone.
A processing example of the user proxy utterance device 20 (smartphone) shown in FIG. 4 will be described.

The user 1 is, for example, a driver of a vehicle.
First, the user 1 makes the following user utterance in step S11.
User utterance = "Tell me the route to Tokyo Tower"
In response to this user utterance, the user utterance partner device 10 which is a car navigation device utters the following device in step S12.
Device utterance = "There are three candidates for the route to Tokyo Tower. Which one do you want?"

When the user 1 is concentrating on driving, he / she may miss the utterance of this device. Also, even if you listen to it, you may not be able to judge it immediately.
In such a case, the user proxy utterance device 20 (smartphone) shown in FIG. 4 performs the following "user proxy utterance" instead of the user 1 in step S13.
User proxy device utterance = "Select the fastest route"

The user proxy device utterance by the user proxy utterance device 20 (smartphone) is input to the user speech partner device 10, that is, the car navigation device.
The car navigation device interprets this user proxy device utterance as a user speech and executes processing according to the user proxy device utterance.
That is, in the example of FIG. 4, the process of selecting one route that arrives earliest from the three routes to Tokyo Tower is performed.

In this way, the user proxy utterance device 20 (smartphone) executes the user proxy device utterance instead of the user 1, so that the car navigation device can proceed with the process according to the user proxy device utterance, and the process can be performed. It can be executed without delay.

FIG. 5 is an example in which the user speaking partner device 10 described above with reference to FIG. 2 is used as an English conversation lesson device.

The user utterance partner device 10 which is an English conversation lesson device utters the following devices in step S21.
Device utterance = "Have you ever been to New York?"

In the example described above with reference to FIG. 2, the user 1 cannot hear or understand the device utterance, and cannot immediately make the user utterance.
In the example shown in FIG. 5, the user substitute utterance device 20 (smartphone) shown in FIG. 5 performs the following user substitute utterance device instead of the user 1 in step S22.
User proxy device utterance = "Yes, I want to New York last summer."

The user proxy device utterance by the user proxy utterance device 20 (smartphone) is input to the user speech partner device 10, that is, the English conversation lesson device.
The English conversation lesson device interprets this user proxy device utterance as a user speech and executes processing according to the user proxy device utterance.
That is, in the example of FIG. 5, it is possible to shift to the next device utterance.

FIG. 6 is an example in which the user utterance partner device 10 described above with reference to FIG. 3 is used as the character dialogue device.
The user utterance partner device 10, which is a character dialogue device, performs the following device utterance in step S31.
Device utterance = "Where did you go today?"

If the user 1 cannot hear the device utterance, is wondering what to answer, or forgets the name of the place, the user 1 cannot make the user utterance.

In the example shown in FIG. 6, the user substitute utterance device 20 (smartphone) shown in FIG. 6 performs the following user substitute utterance device instead of the user 1 in step S32.
User agency utterance = "I went to the museum"

The user proxy device utterance by the user proxy utterance device 20 (smartphone) is input to the user speech partner device 10, that is, the character dialogue device.
The character dialogue device interprets this user proxy device utterance as a user speech, and executes processing according to the user proxy device utterance.
That is, in the example of FIG. 6, it is possible to shift to the next device utterance.

[2. Regarding an example of a specific dialogue sequence involving utterance of the user proxy utterance device of the present disclosure]
Next, an example of a specific dialogue sequence accompanied by the utterance of the user substitute utterance device of the present disclosure will be described.

FIG. 7 shows the user 1 when the user utterance partner device 10 is a car navigation device, the user utterance partner device 10 (car navigation device), and the user proxy utterance device 20 (as described above with reference to FIG. 4). It is a figure which shows an example of the dialogue sequence which a smartphone) executes.
Utterance No. 1-No. The following utterance sequences up to 9 are shown.

Utterance No. 1 (User utterance) = Navigating from here to Odaiba, utterance No. 2 (User utterance partner device (car navigation device)) = I'm clever. Are you sure you want to go to Odaiba Station? Utterance No. 3 (User proxy utterance device (smartphone)) = Speaking No. at Odaiba Kaihin Koen Station. 4 (User utterance partner device (car navigation device)) = I'm clever. Set Odaiba Kaihin Koen Station as the destination. Utterance No. 5 (User proxy utterance device (smartphone)) = Yeah. OK
Utterance No. 6 (User utterance partner device (car navigation device)) = Do you use the expressway? Utterance No. 7 (User proxy utterance device (smartphone)) = Yeah. Use utterance No. 8 (User utterance partner device (car navigation device)) = I'm clever. Take the highway.
Utterance No. 9 (user utterance) = No, today using the lower road

Utterance No. 1 is a user utterance, and the user utters "navigate from here to Odaiba".
In response to this user utterance, the user utterance partner device (car navigation device) makes the following utterances.
Utterance No. 2 (User utterance partner device (car navigation device)) = I'm clever. Are you sure you want to go to Odaiba Station?

This utterance No. 2 (user utterance partner device (car navigation device)) is an utterance including a question from the car navigation device to the user.
When the user does not respond to this question and a predetermined time elapses, the user proxy utterance device (smartphone) changes to the user and makes the following response.
Utterance No. 3 (User proxy utterance device (smartphone)) = Odaiba Kaihin Koen Station

Hereinafter, from utterance N0.4 to utterance N0.8, a dialogue is performed between the user utterance partner device (car navigation device) and the user substitute utterance device (smartphone).
The user does not participate in the dialogue between the utterances N0.4 to N0.8, and simply listens to the dialogue between the user utterance partner device (car navigation device) and the user substitute utterance device (smartphone).

However, the utterance No. of the user utterance partner device (car navigation device). 8 utterances, i.e.
Utterance No. 8 (User utterance partner device (car navigation device)) = I'm clever. Take the highway.
The user who hears this utterance makes the following user utterance.
Utterance No. 9 (user utterance) = No, today using the lower road

As shown in these utterance sequences, the user listens to the dialogue between the user utterance partner device (car navigation device) and the user proxy utterance device (smartphone), and whenever the user does not agree with the intention of the user, the user listens to the dialogue. It is possible to participate in the dialogue and convey the user's intention to the user utterance partner device (car navigation device).

In this way, the user substitute utterance device 10 (smartphone) has a dialogue with the user utterance partner device 20 instead of the user 1.
In the storage unit of the user substitute utterance device 10 (smartphone), an input utterance correspondence response database that associates "sample input utterance" and "response utterance" that can be used for dialogue with the user utterance partner device 20 is stored. , The user substitute utterance device 10 (smartphone) selects and outputs the substitute utterance from the information registered in the input utterance correspondence response database.

Further, in the dialogue sequence shown in FIG. 7, the user proxy utterance device 10 (smartphone) includes a proxy utterance that conveys the user's intention to the user speech partner device 20 (car navigation device) instead of the user 1. Is done.
As described above, in order to perform the proxy utterance including the intention of the user 1, the user proxy utterance device 10 (smartphone) needs to perform a process of estimating the intention of the user 1.

For this user intention estimation process, the user proxy utterance device 10 (smartphone) refers to, for example, the user action history information stored in the storage unit in the user proxy utterance device 10 (smartphone).
A specific example of the user action history information stored in the storage unit in the user proxy utterance device 10 (smartphone) will be described later.

In the storage unit in the user proxy utterance device 10 (smartphone), history information of past actions of the user 1 is stored.
For example, information such as a place frequently visited by the user 1 and a road to be used is recorded.
The user proxy utterance device 10 (smartphone) refers to the user action history information, estimates the user's intention, determines the proxy utterance of the user 1, and outputs it.

The user proxy utterance device 10 (smartphone) estimates the user's intention by referring not only to the user behavior history information but also other information such as user profile information and information acquired from an external server such as an SNS server. Then, a process of determining the proxy utterance of the user 1 may be performed.
Specific examples of these processes will be described later.

In the example shown in FIG. 7, for example, the following utterance, that is,
Utterance No. 3 (User proxy utterance device (smartphone)) = Odaiba Kaihin Koen Station, and this utterance No. 3 is an estimation that when the user proxy utterance device 20 (smartphone) refers to the user behavior history information and the user 1 speaks "Odaiba", the user 1 intends to "Odaiba Kaihin Koen Station". This is a user proxy utterance.

In addition, the utterance No. Dialogue between 6-7 devices, i.e.
Utterance No. 6 (User utterance partner device (car navigation device)) = Do you use the expressway? Utterance No. 7 (User proxy utterance device (smartphone)) = Yeah. Use the utterance No. in this case. In 7, the user proxy utterance device 20 (smartphone) refers to the user behavior history information and confirms that the user 1 frequently uses the expressway, and under this confirmation, the user 1 determines. Judging that he wants to use the expressway, the above utterance No. This is the utterance on behalf of the user in 7.

However, at this time, it happened that the user did not meet his / her wishes, so the utterance No. In 9, the user himself corrected, "Today, use the lower road."

In this way, the user 1 can also listen to the dialogue performed between the user utterance partner device 10 (car navigation device) and the user substitute utterance device 20 (smartphone), and when the user's intention is not met. Can participate in the dialogue at any time and convey the user's intention to the user utterance partner device 10 (car navigation device).

FIG. 8 shows the user 1 when the user speaking partner device 10 is an English conversation lesson device, the user speaking partner device 10 (English conversation lesson device), and the user proxy speaking device, as described above with reference to FIG. It is a figure which shows an example of the dialogue sequence executed by 20 (smartphone).
Utterance No. 1-No. The following utterance sequences up to 4 are shown.

Utterance No. 1 (User utterance partner device (English conversation lesson device)) = How was your holiday?
Utterance No. 2 (user proxy utterance device (smartphone)) = I want to my homeown.
Utterance No. 3 (User utterance partner device (English conversation lesson device)) = That's great! Where is your homeown?
Utterance No. 4 (user utterance) = Hokkaido.

Utterance No. Reference numeral 1 denotes an utterance of the user's utterance partner device (English conversation lesson device), and the English conversation lesson device utters "How was your holiday?".
This utterance No. 1 (user utterance partner device (English conversation lesson device)) is an utterance including a question to the user from the English conversation lesson device.

Since the user was wondering how to answer this question and felt a mental burden, the user himself did not answer and left the response to the user proxy utterance device (smartphone).
When the user does not respond and a predetermined time elapses, the user proxy utterance device (smartphone) takes the place of the user and makes the following response.
Utterance No. 2 (user proxy utterance device (smartphone)) = I want to my homeown.

The user utterance partner device (English conversation lesson device) recognizes the utterance from the user substitute utterance device (smartphone) as a response from the user, and further utters the following in response to this response.
Utterance No. 3 (User utterance partner device (English conversation lesson device)) = That's great! Where is your homeown?
This utterance No. 3 (User utterance partner device (English conversation lesson device)) is also an utterance including a question to the user from the English conversation lesson device.

In response to this question, the user understood the question of the user's utterance partner device (English conversation lesson device), and immediately, the following utterance No. Speak 4.
Utterance No. 4 (user utterance) = Hokkaido.

In these utterance sequences as well, the user can participate in the dialogue at the timing when the user wants to speak while listening to the dialogue performed between the user utterance partner device (English conversation lesson device) and the user substitute utterance device (smartphone). Will be.

In the dialogue sequence shown in FIG. 8, the user substitute utterance device 10 (smartphone) transmits information on the past behavior of the user to the user utterance partner device 20 (car navigation device) instead of the user 1.
In order for the user substitute utterance device 10 (smartphone) to speak about the past actions of the user 1, the user substitute utterance device 10 (smartphone) needs to acquire information on the past actions of the user 1.

As described above, the user action history information is stored in the storage unit of the user substitute utterance device 10 (smartphone), and the data processing unit of the user substitute utterance device 10 (smartphone) stores the user action stored in the storage unit. Refer to the history information and execute the utterance according to the reference result.
A specific example of the user action history information stored in the storage unit in the user proxy utterance device 10 (smartphone) will be described later.

FIG. 9 shows the user 1 when the user utterance partner device 10 is a character dialogue device, the user utterance partner device 10 (character dialogue device), and the user substitute utterance device, as described above with reference to FIG. It is a figure which shows an example of the dialogue sequence executed by 20 (smartphone).
Utterance No. 1-No. The following utterance sequences up to 6 are shown.

Utterance No. 1 (User utterance partner device (character dialogue device)) = Hello. Let's talk with me Utterance No. 2 (User proxy utterance device (smartphone)) = Then, first tell me your name and utterance No. 3 (User utterance partner device (character dialogue device)) = Hanako. you?
Utterance No. 4 (User proxy utterance device (smartphone)) = Taro Speaking No. 5 (User utterance partner device (character dialogue device)) = Mr. Taro? I want to ask about my hobbies.
Utterance No. 6 (User utterance) = My hobby is watching movies. Sometimes I go to the cinema, but most of the time I watch it on video.

Utterance No. Reference numeral 1 denotes an utterance of the user's utterance partner device (character dialogue device), and the character dialogue device utters "Hello. Let's talk with me."
Since the user was wondering how to answer this utterance and felt a mental burden, the user himself did not answer and left the response to the user proxy utterance device (smartphone).
When the user does not respond and a predetermined time elapses, the user proxy utterance device (smartphone) takes the place of the user and makes the following utterances.
Utterance No. 2 (User proxy utterance device (smartphone)) = Then, first tell me your name

The user utterance partner device (character dialogue device) recognizes the utterance from the user substitute utterance device (smartphone) as a response from the user, and further utters the following in response to this response.
Utterance No. 3 (User utterance partner device (character dialogue device)) = Hanako. you?

This utterance No. 3 (user utterance partner device (character dialogue device)) is an utterance including a question from the character dialogue device to the user.

To this question, the user didn't find it interesting just to answer his name, so he left the response to the proxy dialogue system.
The user proxy utterance device (smartphone) makes the following utterances on behalf of the user.
Utterance No. 4 (user utterance) = Taro

The user utterance partner device (character dialogue device) recognizes the utterance from the user substitute utterance device (smartphone) as a response from the user, and further utters the following in response to this response.
Utterance No. 5 (User utterance partner device (character dialogue device)) = Mr. Taro? I want to ask about my hobbies.

This utterance No. 3 (user utterance partner device (character dialogue device)) is an utterance including a question from the character dialogue device to the user.

The user responded to this question because he found it interesting to be able to talk about his hobbies. That is, the following user utterances were made.
Utterance No. 6 (User utterance) = My hobby is watching movies. Sometimes I go to the cinema, but most of the time I watch it on video.

Also in these utterance sequences, the user can participate in the dialogue at the timing when the user wants to speak while listening to the dialogue performed between the user utterance partner device (character dialogue device) and the user substitute utterance device (smartphone). It becomes.

In the dialogue sequence shown in FIG. 9, the user proxy utterance device 10 (smartphone) has the utterance No. In 2, the user utterance partner device 20 (character dialogue device) is uttered by asking for a name.
In addition, the user proxy utterance device 10 (smartphone) has an utterance No. In 4, the user's name is answered instead of the user 1.

The user proxy utterance device 10 (smartphone) has the utterance No. In 2, in order to make a name-questioning utterance to the user utterance partner device 20 (character dialogue device), a function of uttering a response to the utterance from the user utterance partner device 20 (character dialogue device) is required. ..

As described above, the storage unit of the user substitute utterance device 10 (smartphone) is an input utterance correspondence response database in which "sample input utterance" and "response utterance" that can be used for dialogue with the user utterance partner device 20 are associated with each other. Is stored.
The user proxy utterance device 10 (smartphone) refers to the information registered in the input utterance correspondence response database, and refers to the utterance No. of FIG. The utterance in 2, that is, the utterance to the user utterance partner device 20 (character dialogue device) is performed.
A specific example of the input utterance correspondence response database will be described later.

In addition, the user proxy utterance device 10 (smartphone) has the utterance No. In 4, in order to speak the user's name to the user's utterance partner device 20 (character dialogue device), the user proxy utterance device 10 (smartphone) must know the user's name.
The data processing unit of the user substitute utterance device 10 (smartphone) acquires the user name from the user profile information stored in the storage unit of the user substitute utterance device 10 (smartphone), and obtains the user name from the user profile information, and the utterance No. In 4, the utterance explaining the user's name is given.

As described with reference to FIGS. 4 to 9, if the user proxy utterance device 10 of the present disclosure is used, most of the dialogue with the user utterance partner device 20 is left to the user proxy utterance device 10, and the user is positive. It is possible to speak to the user and participate in the dialogue at any time in the situation where he / she wants to participate in the dialogue.

Specifically, the situation in which the user wants to actively participate in the dialogue is, for example, a situation in which the utterance of the user substitute utterance device 10 needs to be corrected, or a situation in which the user 1 feels a mental burden to make the user's utterance by himself / herself. If not, there is a topic that user 1 wants to explain by himself.

[3. About the configuration example of the user substitute utterance device of this disclosure]
Next, a configuration example of the user proxy utterance device, which is the information processing device of the present disclosure, will be described.

FIG. 10 is a diagram showing a configuration example of the user proxy utterance device 20 which is the information processing device of the present disclosure.
As shown in FIG. 10, the user proxy utterance device 20 includes a voice input unit (mic) 21, a data processing unit 22, a voice output unit (speaker) 23, a communication unit 24, a storage unit 25, and an image input unit ( Has a camera).

The voice input unit (microphone) 21 inputs the user utterance voice 51 emitted from the user 1 and the user utterance partner device output voice 52 output from the user utterance partner device 10 such as a car navigation device.

The voice data input by the voice input unit (microphone) 21 is input to the data processing unit 22.
The data processing unit 22 analyzes the input voice, determines whether or not the user proxy utterance device 20 should speak, and if it determines that the utterance should be made, generates an utterance and generates a voice output unit (speaker). ) Output to 23.

The voice output unit (speaker) 23 outputs the utterance generated by the data processing unit 22 as the user utterance agent utterance 53.

The communication unit 24 performs communication for acquiring information necessary for determining whether or not the data processing unit 22 needs to execute an utterance and generating an utterance from an external server or a user utterance partner device 10.
The storage unit 25 records information necessary for determining whether or not the data processing unit 22 needs to execute an utterance and for generating an utterance. The user behavior history information described above, the input utterance correspondence response database, the user profile information, and the like are recorded.

The image input unit (camera) captures, for example, a face image or an eye image of the user 1. The captured image is used, for example, for the line-of-sight direction analysis of the user 1, and is used for determination processing of whether or not the user utterance is an utterance made toward the user utterance partner device 10.

Next, the detailed configuration of the data processing unit 22 and the storage unit 24 of the user proxy utterance device 20 will be described with reference to FIG.

FIG. 11 is a diagram showing a detailed configuration of the data processing unit 22 and the storage unit 24 of the user proxy utterance device 20.
As shown in FIG. 11, the data processing unit 22 of the user substitute utterance device 20 includes an utterance detection unit 101, a voice recognition unit 102, a response necessity determination unit 103, a response generation unit 104, and a voice synthesis unit 105.
Further, the storage unit 24 of the user substitute utterance device 20 stores the input utterance correspondence response database 121, the user action history information 122, and the user profile information 123.

Further, the communication unit 25 executes communication with the user utterance partner device 10 and the external server 150.
The external server 150 is, for example, a server that stores information that can be used by the user substitute utterance device 20 to understand the input utterance, information necessary for the user substitute utterance device 20 to make a utterance, and the like. For example, it is composed of various databases such as a knowledge database that stores general knowledge and the like, a scenario database that stores dialogue sequence information, and an SNS (Social Networking Service) server.

Further, when the user utterance partner device 10 is, for example, a car navigation device, the travel history information of the vehicle equipped with the car navigation device may be recorded in the storage unit of the car navigation device.
The data processing unit 22 can acquire the travel history information via the communication unit 25, analyze the user behavior, and use it as reference information at the time of utterance generation.

Next, the components of the data processing unit 22, that is, the utterance detection unit 101, the voice recognition unit 102, the response necessity determination unit 103, the response generation unit 104, the voice synthesis unit 105, and the processing executed by each of these components. explain.

The utterance detection unit 101 inputs the user utterance voice 51 emitted from the user 1 and the user utterance partner device output voice 52 output from the user utterance partner device 10 such as a car navigation device via the voice input unit (microphone) 21. do.
When the utterance detection unit 101 detects that the voice data has been input from the voice input unit (mic) 21, the spoke detection unit 101 outputs the input voice data to the voice recognition unit 102.

The voice recognition unit 102 executes a text (utterance text) generation process based on the voice data input from the speech detection unit 101.
The voice recognition unit 102 has, for example, an ASR (Automatic Speech Recognition) function, and converts voice data into text (utterance text) data composed of a plurality of words.
The utterance text generated by the voice recognition unit 102 is output to the response necessity determination unit 103.

In the voice recognition unit 102, the voice data input from the utterance detection unit 101 is
(A) Is it the user-spoken voice 51 emitted from the user 1?
(B) Whether the user utterance partner device output voice 52 is output from the user utterance partner device 10 such as a car navigation device.
It is determined which of these voices (a) and (b) is, and the utterance subject identifier (speaking subject identification tag) indicating the discrimination information is used as the attribute information associated with the utterance text, and it is necessary to respond to both the utterance texts. Output to the determination unit 103.
In addition, a utterance subject identification unit for identifying the utterance subject is provided as a configuration different from the voice recognition unit 102, and the utterance subject identifier (utterance subject identification tag) generated in the utterance subject identification unit is used in the response necessity determination unit 103 together with the utterance text. It may be configured to output.

The process of determining whether the voice data input from the utterance detection unit 101 is the user utterance voice 51 or the user utterance partner device output voice 52 is performed by analyzing the voice frequency included in the voice data. Since the output voice 52 of the user utterance partner device such as a car navigation device is the output voice from the speaker, it is composed of only the frequency data in a predetermined range according to the characteristics of the speaker, and is different from the frequency included in the human utterance.

The voice recognition unit 102 analyzes the frequency characteristics of the voice data input from the utterance detection unit 101, and determines whether the input voice data is the user utterance voice 51 or the user utterance partner device output voice 52. , The utterance subject identifier (speech subject identification tag) indicating the discrimination information is associated with the utterance text and output to the response necessity determination unit 103.

The response necessity determination unit 103 inputs the utterance text in which the utterance subject identifier (utterance subject identification tag) is set from the voice recognition unit 102, and the input utterance text is generated based on the user utterance partner device output voice 52. In the case of the utterance text, it is determined whether or not the user substitute utterance device 20 needs to make a response utterance to the utterance text, that is, the text based on the output utterance of the user utterance partner device output voice 52. ..

The response necessity determination unit 103 performs, for example, the following processing as a process of determining whether or not it is necessary to perform a response utterance.
If the utterance of the user 1 is not input within the predetermined threshold time (for example, 2 seconds) from the utterance completion timing of the user utterance partner device 10, it is determined that the user substitute utterance device 20 should respond instead of the user 1. ..

That is, when the elapsed time from the utterance completion timing of the user utterance partner device 10 is measured and there is no utterance from the user 1 within the predetermined threshold time (for example, 2 seconds), the user acts on behalf of the user 1. It is determined that the uttering device 20 should respond.

When the response necessity determination unit 103 determines that the user proxy utterance device 20 should respond, the response necessity determination unit 103 requests the next response generation unit 104 to generate a response.

When the response generation request is input from the response necessity determination unit 103, the response generation unit 104 generates an utterance text corresponding to the utterance to be output from the user proxy utterance device 20.
When generating this utterance text, various information stored in the storage unit 24, that is, the input utterance correspondence response database 121, the user action history information 122, and the user profile information 123 can be used. Further, it is also possible to use the information that can be acquired via the communication unit 25, that is, the information acquired from the external server 150 or the user utterance partner device 10.
Specific examples of these processes will be described later.

The utterance text generated by the response generation unit 104 is input to the speech synthesis unit 105.
The voice synthesis unit 105 generates a synthetic voice based on the utterance text generated by the response generation unit 104. That is, the voice synthesis process (TTS: Text To Speech) is executed, and the generated synthetic voice is output as the user speech agent output voice 53 shown in the figure via the voice output unit (speaker) 23.

In this embodiment, the user utterance partner device 10 and the user substitute utterance device 20 are configured as separate devices, but it is also possible to configure both as one device. For example, a user proxy utterance execution unit that performs a user's proxy utterance may be provided in the car navigation device.

[4. Details of the processing executed by each component of the user proxy utterance device]
Next, the components of the data processing unit 22 of the user substitute utterance device 20, that is, the utterance detection unit 101, the voice recognition unit 102, the response necessity determination unit 103, the response generation unit 104, the voice synthesis unit 105, and each of these configurations. The details of the specific processing of the elements will be described in order.

(4- (1). Details of the process executed by the utterance detection unit 101)
As described above, the utterance detection unit 101 outputs the user utterance voice 51 emitted from the user 1 and the user utterance partner device 10 output from the user utterance partner device 10 such as a car navigation device via the voice input unit (mic) 21. The voice 52 is input, and the input voice data is output to the voice recognition unit 102.

The input, output, and execution processes of the utterance detection unit 101 are as follows.
(A) Input The voice data input includes various noise sounds in addition to the utterance voice of the user utterance partner device 10 and the user 1.
(B) Output utterance voice (speech voice of user utterance partner device 10 or utterance voice of user 1)
The output is spoken voice data excluding various noise sounds.

(C) Processing The utterance voice data of the utterance voice of the user utterance partner device 10 or the user 1 is extracted from the voice data including various noise sounds to generate the utterance voice data as output data.
An existing voice section detection program can be used for the voice section detection process. For example, existing open source software that is allowed to be freely used or modified may be used.

(4- (2). Details of the process executed by the voice recognition unit 102)
The voice recognition unit 102 executes a text (utterance text) generation process based on the voice data input from the speech detection unit 101 as described above.
The voice recognition unit 102 has, for example, an ASR (Automatic Speech Recognition) function, and converts voice data into text (utterance text) data composed of one or a plurality of words.

Further, the voice recognition unit 102 receives the voice data input from the utterance detection unit 101.
(A) Is it the user-spoken voice 51 emitted from the user 1?
(B) Whether the user utterance partner device output voice 52 is output from the user utterance partner device 10 such as a car navigation device.
It is determined which of these voices (a) and (b) is, and the utterance subject identifier (speaking subject identification tag) indicating the discrimination information is used as the attribute information corresponding to the utterance text, and the response necessity determination unit is used for both the utterance texts. Output to 103.

The input, output, and execution processes of the voice recognition unit 102 are as follows.
(A) Input utterance voice (speech voice of user utterance partner device 10 or utterance voice of user 1)
(B) Output utterance text to which the utterance subject identifier (speaking subject identification tag) is added (the utterance text of the user utterance partner device 10 or the utterance text of the user himself / herself)

(C) Processing Automatically convert spoken voice to text. For this process (speech recognition), for example, a program that executes the above-mentioned ASR (Automatic Speech Recognition) function is used. Open source software may be used.

Further, the voice data input from the utterance detection unit 101 is
(A) Is it the user-spoken voice 51 emitted from the user 1?
(B) Whether the user utterance partner device output voice 52 is output from the user utterance partner device 10 such as a car navigation device.
It is determined which of these voices (a) and (b) is.

As described above, since the output voice 52 of the user utterance partner device such as the car navigation device is the output voice from the speaker, it is composed only of the frequency data in a predetermined range according to the characteristics of the speaker, and the frequency included in the human utterance. Is different.
The voice recognition unit 102 analyzes the frequency characteristics of the voice data input from the utterance detection unit 101, and determines whether the input voice data is the user utterance voice 51 or the user utterance partner device output voice 52. , The utterance subject identifier (speech subject identification tag) indicating the discrimination information is associated with the utterance text and output to the response necessity determination unit 103.

(4- (3). Details of the process executed by the response necessity determination unit 103)
The response necessity determination unit 103 inputs the utterance text in which the utterance subject identifier (utterance subject identification tag) generated by the voice recognition unit 102 is set as described above, and the tag set in the input utterance text is the user's utterance. In the case of the utterance text based on the output utterance of the remote device output voice 52, it is determined whether or not the user substitute utterance device 20 needs to make a response utterance.

Specifically, for example, when the utterance of the user 1 is not input within the predetermined threshold time (for example, 2 seconds) from the utterance completion timing of the user utterance partner device 10, the user substitute utterance device 20 is used instead of the user 1. Determines that should respond.
When the response necessity determination unit 103 determines that the user proxy utterance device 20 should respond, the response necessity determination unit 103 requests the next response generation unit 104 to generate a response.

The input, output, and execution processing of the response necessity determination unit 103 are as follows.
(A) Speaking text in which the input utterance subject identifier (speaking subject identification tag) is set (b) Output utterance necessity identification value indicating the necessity of response by the user substitute utterance device 20 (for example, necessary = 1, unnecessary = 0)

The response necessity identification value (for example, necessary = 1, unnecessary = 0), which is an output, is output to the response generation unit 104.
When the response necessity identification value (necessity = 1) indicating the response required by the user substitute utterance device 20 is output to the response generation unit 104, the utterance subject identifier (utterance subject identification tag) generated by the voice recognition unit 102 is used. The set utterance text is also output to the response generation unit 104.

(C) Processing When the utterance text in which the utterance subject identifier (speaker subject identification tag) generated by the voice recognition unit 102 is set is the utterance text based on the output utterance of the user utterance partner device output voice 52, the user substitute utterance device. 20 determines whether or not it is necessary to make a response utterance.

The response necessity determination unit 103 performs the following processing, for example.
When the utterance of the user 1 is not input within the predetermined threshold time (N seconds (for example, N = 2)) from the completion timing of the utterance of the user utterance partner device 10, the user substitute utterance device 20 is used instead of the user 1. Determine to respond.

The utterance completion timing of the user utterance partner device 10 and the user utterance detection timing, which are required in this determination process, are set from the voice recognition unit 102 to the response necessity determination unit 103 as the utterance subject identifier (utterance subject identification tag). Judgment is based on the time the utterance text is entered. Alternatively, the utterance detection time in which the utterance is detected by the utterance detection unit 101 may be used.

In addition, there are a plurality of different processing modes as follows as the processing mode of the response necessity determination processing for determining whether or not the user substitute utterance device 20 should respond instead of the user 1, and any one of them or It is possible to execute a combination of a plurality of processing examples.

(Processing Example 1) At each completion timing of each utterance of the user utterance partner device 10, it is measured whether or not the elapsed time without utterance of the user 1 exceeds the threshold time (N seconds (for example, N = 2)). Then, a process of determining whether or not a response is required of the user proxy utterance device 20 is performed.

(Processing Example 2) The determination process of the above (Processing Example 1), that is, for each utterance completion timing of the user utterance partner device 10, the elapsed time without utterance of the user 1 is the threshold time (N seconds (for example, N). = 2)) The process of determining whether or not the response is required by the user proxy utterance device 20 is performed by measuring whether or not the above is achieved. In addition to this processing, it is further determined whether or not the user utterance detected within the threshold time is an utterance made to the user utterance partner device 10, and whether or not the user proxy utterance device 20 needs to respond is determined. Performs the judgment process.
For example, even if the utterance of the user 1 is detected within the threshold time, if the utterance is not made to the user utterance partner device 10, the response of the user substitute utterance device 20 is determined to be necessary. ..

(Processing Example 3) If the user substitute utterance device 20 has not made the first utterance after the start of the dialogue sequence between the user 1 and the user utterance partner device 10, the threshold value is set from the utterance completion timing of the user utterance partner device 10. If there is no user utterance of user 1 within the time (N seconds (for example, N = 2)), it is determined that the response of the user substitute utterance device 20 is necessary. However, after the start of the dialogue sequence between the user 1 and the user utterance partner device 10 and the user substitute utterance device 20 making the first utterance, the elapsed time from the utterance completion timing of the user utterance partner device 10 to the user utterance detection is measured. Is not performed, and it is determined that the response of the user substitute utterance device 20 is necessary.

(Processing Example 4) If the user substitute utterance device 20 has not made the first utterance after the start of the dialogue sequence between the user 1 and the user utterance partner device 10, the threshold value is set from the utterance completion timing of the user utterance partner device 10. If there is no user utterance of user 1 within the time (N seconds (for example, N = 2)), it is determined that the response of the user substitute utterance device 20 is necessary. However, after the dialogue sequence between the user 1 and the user utterance partner device 10 is started and the user substitute utterance device 20 makes the first utterance, the threshold time is N seconds shorter than the first N (for example, N = 1). Is set to, and it is determined whether or not the user proxy utterance device 20 needs to respond.

(Processing Example 5) Whether or not to wait for one user to speak based on not only the elapsed time (N seconds) from the utterance completion timing of the user utterance partner device 10 but also other information, that is, the user's utterance on behalf of the user. It is determined whether or not the response of the device 20 is necessary.

For example, the face image of the user 1 is photographed by using the image input unit (camera) 26 provided in the user proxy utterance device 20, and the line-of-sight information of the user 1 is analyzed from the photographed face image.
As a result of this line-of-sight analysis, when it is determined that the user 1 is looking toward the utterance partner device 10, it is determined that the user 1 is willing to speak, and the user waits for the utterance of the user himself / herself to be input.

Alternatively, an image of the user 1 is taken, the gesture (signal) of the user 1 is analyzed from the taken image, and a "gesture indicating that the user himself / herself wants to speak" or a "gesture indicating that the user himself / herself does not want to speak" or "a gesture indicating that the user himself / herself does not want to speak". , And based on the gesture analysis result, it may be determined whether or not the response of the user proxy utterance device 20 is necessary.

A plurality of response necessity determination processing sequences executed by the response necessity determination unit 103 of the user proxy utterance device 20 will be described with reference to the flowcharts shown in FIGS. 12 to 15.

First, with reference to the flowchart shown in FIG. 12, the above-mentioned (processing example 1), that is,
(Processing Example 1) At each completion timing of each utterance of the user utterance partner device 10, it is measured whether or not the elapsed time without utterance of the user 1 exceeds the threshold time (N seconds (for example, N = 2)). Then, a process of determining whether or not a response is required of the user proxy utterance device 20 is performed.
The processing sequence of this (processing example 1) will be described.

(Step S101)
First, in step S101, the response necessity determination unit 103 of the user substitute utterance device 20 inputs the utterance text of the user utterance partner device 10 from the voice recognition unit 102.

As described above, the voice recognition unit 102 receives the voice data input from the utterance detection unit 101.
(A) Is it the user-spoken voice 51 emitted from the user 1?
(B) Whether the user utterance partner device output voice 52 is output from the user utterance partner device 10 such as a car navigation device.

The identification information of which of the voices (a) and (b) is used, that is, the utterance text in which the utterance subject identifier (speaking subject identification tag) is set as additional information is input to the response necessity determination unit 103.

When the response necessity determination unit 103 refers to the utterance subject identifier (utterance subject identification tag) and confirms that the input text from the voice recognition unit 102 is the utterance text of the user utterance partner device 10, the step The processing of S102 and the like is executed.

(Step S102)
Next, in step S102, the response necessity determination unit 103 determines whether or not the user utterance is detected within the specified threshold time (N seconds) from the utterance completion timing of the user utterance partner device 10.

The utterance completion timing of the user utterance partner device 10 and the user utterance detection timing, which are required in this determination process, are set from the voice recognition unit 102 to the response necessity determination unit 103 to set the utterance subject identifier (utterance subject identification tag). Judgment is made based on the time when the spoken text is input. Alternatively, the utterance detection time in the utterance detection unit 101 may be used.

If the user utterance is detected within the specified threshold time (N seconds) from the utterance completion timing of the user utterance partner device 10, the determination in step S102 is Yes, and the process proceeds to step S103.

That is, for example, the user utterance spoken by the user 1 from the voice recognition unit 102 within a specified threshold time (N seconds) from the utterance text input timing of the user utterance partner device 10 to the response necessity determination unit 103 from the voice recognition unit 102. When the text is input, the determination in step S102 is Yes, and the process proceeds to step S103.

On the other hand, if the user utterance is not detected within the specified threshold time (N seconds) from the utterance completion timing of the user utterance partner device 10, the determination in step S102 is Yes, and the process proceeds to step S103.

That is, for example, the user utterance spoken by the user 1 from the voice recognition unit 102 within a specified threshold time (N seconds) from the utterance text input timing of the user utterance partner device 10 to the response necessity determination unit 103 from the voice recognition unit 102. If no text is input, the determination in step S102 is No, and the process proceeds to step S104.

(Step S103)
In step S102, when the user utterance is detected within the specified threshold time (N seconds) from the utterance completion timing of the user utterance partner device 10, the process of step S103 is executed.

In this case, in step S103, the response necessity determination unit 103 is "unnecessary" for the response of the user substitute utterance device 20, that is, the utterance from the user substitute utterance device 20 for the output utterance from the user utterance partner device 10. The response necessity identification value (0) indicating the above is generated and output to the response generation unit 104.

In this case, the response generation unit 104 does not execute the response utterance generation process for outputting from the user proxy utterance device 20.

(Step S104)
On the other hand, in step S102, if the user utterance is not detected within the specified threshold time (N seconds) from the utterance completion timing of the user utterance partner device 10, the process of step S104 is executed.

In this case, in step S104, the response necessity determination unit 103 "needs" the response of the user substitute utterance device 20, that is, the utterance from the user substitute utterance device 20 to the output utterance from the user utterance partner device 10. The response necessity identification value (1) indicating the above is generated and output to the response generation unit 104.

In this case, the response generation unit 104 executes a response utterance generation process for outputting from the user proxy utterance device 20.

Next, referring to the flow shown in FIG. 13, when the user utterance is detected within the specified threshold time (N seconds) from the above (processing example 2), that is, the utterance completion timing of the user utterance partner device 10. In the present invention, an example will be described in which it is determined whether or not the user utterance is made to the user utterance partner device 10 such as a car navigation device, and the process is changed according to the determination result.

Specifically, it is said that the user utterance detected within the specified threshold time (N seconds) from the utterance completion timing of the user utterance partner device 10 is made to the user utterance partner device 10 such as a car navigation device. When the determination is made, a response necessity identification value (0) indicating that the utterance from the user substitute utterance device 20 for the output utterance from the user utterance partner device 10 is "unnecessary" is generated, and the response generation unit 104 generates a response necessity identification value (0). Output.

On the other hand, when it is determined that the user utterance is not made to the user utterance partner device 10 such as a car navigation device, the utterance from the user substitute utterance device 20 for the output utterance from the user utterance partner device 10 is "necessary". A response necessity identification value (1) indicating that there is is generated and output to the response generation unit 104.

It should be noted that the determination as to whether or not the user utterance is made to the user utterance partner device 10 such as the car navigation device can be executed based on, for example, the semantic analysis result of the user utterance. By the semantic analysis of the user utterance, it can be determined whether or not the user utterance is made as a response to the utterance of the immediately preceding user utterance partner device 10.
For example, it is possible to analyze the text of the user's utterance and determine whether or not it is an operation command to the car navigation device (a response to the utterance of the user's utterance partner device 10).

Alternatively, the line-of-sight direction of the user 1 is analyzed from the face image of the user 1 taken by the image input unit (camera) 26 mounted on the user substitute utterance device 20, and the line-of-sight of the user 1 is transferred to the user utterance partner device 10. If it is directed, it is determined that the user utterance is an utterance made to the user utterance partner device 10, and if the line of sight of the user 1 is not directed to the user utterance partner device 10, the user utterance is It may be determined that the utterance is not made to the user utterance partner device 10.

The processing of each step of the flow shown in FIG. 13 will be described.
(Steps S121-S122)
The processes of steps S121 to S122 are the same as the processes of steps S101 to S102 of the flow described with reference to FIG.

That is, first, the response necessity determination unit 103 of the user substitute utterance device 20 inputs the utterance text of the user utterance partner device 10 from the voice recognition unit 102 in step S121.
Enter the utterance text in which the utterance subject identifier (speaking subject identification tag) indicating whether the utterance text is the user utterance text of the user 1 or the utterance text of the output utterance of the user utterance partner device 10 is set as additional information. ..

When the response necessity determination unit 103 refers to the utterance subject identifier (utterance subject identification tag) and confirms that the input text from the voice recognition unit 102 is the utterance text of the user utterance partner device 10, step S122. Execute the following processing.

In step S122, the response necessity determination unit 103 determines whether or not the user utterance is detected within the specified threshold time (N seconds) from the utterance completion timing of the user utterance partner device 10.

If the user utterance text is input within the specified threshold time (N seconds) from the utterance completion timing of the user utterance partner device 10, the determination in step S122 is Yes, and the process proceeds to step S123.
On the other hand, if the user utterance text is not input within the specified threshold time (N seconds) from the utterance completion timing of the user utterance partner device 10, the determination in step S122 becomes No, and the process proceeds to step S125.

(Step S123)
In step S122, when the user utterance is detected within the specified threshold time (N seconds) from the utterance completion timing of the user utterance partner device 10, the process of step S123 is executed.

In this case, in step S123, the response necessity determination unit 103 detects the user utterance within the specified threshold time (N seconds) from the utterance completion timing of the user utterance partner device 10 to the user utterance partner device such as a car navigation device. It is determined whether or not it was made for 10.
For example, the determination is made using the semantic analysis result of the user's utterance and the analysis result of the line-of-sight information from the user's face image.

If it is determined that the detected user utterance is made to the user utterance partner device 10, the process proceeds to step S124.
On the other hand, if it is determined that the detected user utterance is not made to the user utterance partner device 10, the process proceeds to step S125.

(Step S124)
In step S122, it is determined that the user utterance is detected within the specified threshold time (N seconds) from the utterance completion timing of the user utterance partner device 10, and further, in step S123, the user utterance is the user utterance partner device 10. If it is determined that the item has been made for, the process proceeds to step S124.

In this case, in step S124, the response necessity determination unit 103 is "unnecessary" for the response of the user substitute utterance device 20, that is, the utterance from the user substitute utterance device 20 for the output utterance from the user utterance partner device 10. The response necessity identification value (0) indicating the above is generated and output to the response generation unit 104.

In this case, the response generation unit 104 does not execute the response utterance generation process for outputting from the user proxy utterance device 20.

(Step S125)
On the other hand, in step S122, when the user utterance is not detected within the specified threshold time (N seconds) from the utterance completion timing of the user utterance partner device 10, or in step S123, the utterance of the user utterance partner device 10 is completed. If it is determined that the user utterance detected within the specified threshold time (N seconds) from the timing is not made to the user utterance partner device 10, the process proceeds to step S125.

In this case, in step S125, the response necessity determination unit 103 "needs" the response of the user substitute utterance device 20, that is, the utterance from the user substitute utterance device 20 with respect to the output utterance from the user utterance partner device 10. The response necessity identification value (1) indicating the above is generated and output to the response generation unit 104.

In this case, the response generation unit 104 executes a response utterance generation process for outputting from the user proxy utterance device 20.

Next, a processing sequence according to the above (processing example 3) will be described with reference to the flow shown in FIG.

(Processing example 3) starts from the utterance completion timing of the user utterance partner device 10 when the user substitute utterance device 20 does not make the first utterance after the start of the dialogue sequence between the user 1 and the user utterance partner device 10. If there is no user utterance of user 1 within the threshold time (N seconds (for example, N = 2)), it is determined that the response of the user substitute utterance device 20 is necessary. However, after the start of the dialogue sequence between the user 1 and the user utterance partner device 10 and the user substitute utterance device 20 making the first utterance, the elapsed time from the utterance completion timing of the user utterance partner device 10 to the user utterance detection is measured. This is a processing example in which it is determined that the response of the user substitute utterance device 20 is necessary.
A processing sequence according to this processing example will be described with reference to FIG.

The processing of each step of the flow shown in FIG. 14 will be described.
(Step S141)
The process of step S141 is the same process as the process of step S101 of the flow described with reference to FIG.

That is, first, the response necessity determination unit 103 of the user substitute utterance device 20 inputs the utterance text of the user utterance partner device 10 from the voice recognition unit 102 in step S141.
Enter the utterance text in which the utterance subject identifier (speaking subject identification tag) indicating whether the utterance text is the user utterance text of the user 1 or the utterance text of the output utterance of the user utterance partner device 10 is set as additional information. ..
When the response necessity determination unit 103 confirms that the input text from the voice recognition unit 102 is the utterance text of the user utterance partner device 10 by referring to the utterance subject identifier (utterance subject identification tag), step S142 Execute the following processing.

(Step S142)
Next, in step S142, the response necessity determination unit 103 determines whether or not the user substitute utterance device 20 is speaking as the utterance immediately before the input utterance text of the user utterance partner device 10.

If this is done, the determination in step S142 is Yes, and the process proceeds to step S144.
If it has not been performed, the determination in step S142 becomes No, and the process proceeds to step S143.
In the case where the utterance is not performed, the utterance immediately before the input utterance text of the user utterance partner device 10 does not exist, and the utterance immediately before the utterance text of the input user utterance partner device 10 is the user 1. Two cases are included in the case of user utterance.

(Step S143)
If it is determined in step S142 that the utterance of the user substitute utterance device 20 has not been performed as the utterance immediately before the utterance text of the input user utterance partner device 10, the process proceeds to step S143.

In this case, the response necessity determination unit 103 determines in step S143 whether or not the user utterance is detected within the specified threshold time (N seconds) from the utterance completion timing of the user utterance partner device 10.

If the user utterance text is input within the specified threshold time (N seconds) from the utterance completion timing of the user utterance partner device 10, the determination in step S143 is Yes, and the process proceeds to step S145.
On the other hand, if the user utterance text is not input within the specified threshold time (N seconds) from the utterance completion timing of the user utterance partner device 10, the determination in step S143 becomes No, and the process proceeds to step S144.

(Step S144)
The process of step S144 is a process to be executed in any of the following cases.
(A) In step S142, when it is determined that the utterance of the user substitute utterance device 20 is being performed as the utterance immediately before the utterance text of the input user utterance partner device 10.
(B) In step S142, it is determined that the user substitute utterance device 20 has not spoken as the utterance immediately before the utterance text of the user utterance partner device 10 that has been input, and further, in step S143, the user utterance partner device 10 When it is determined that the user utterance is not detected within the specified threshold time (N seconds) from the utterance completion timing of

In the case of any of the above (a) and (b), in step S144, the response necessity determination unit 103 responds to the user proxy utterance device 20, that is, the user proxy utterance device for the output utterance from the user utterance partner device 10. A response necessity identification value (1) indicating that the utterance from 20 is "necessary" is generated and output to the response generation unit 104.

In this case, the response generation unit 104 executes a response utterance generation process for outputting from the user proxy utterance device 20.

(Step S145)
On the other hand, in step S142, it is determined that the user substitute utterance device 20 has not spoken as the utterance immediately before the utterance text of the user utterance partner device 10 that has been input, and further, in step S143, the user utterance partner device 10 When it is determined that the user utterance is detected within the specified threshold time (N seconds) from the utterance completion timing, the response necessity determination unit 103 executes the process of step S145.

In this case, in step S145, the response necessity determination unit 103 is "unnecessary" for the response of the user substitute utterance device 20, that is, the utterance from the user substitute utterance device 20 for the output utterance from the user utterance partner device 10. The response necessity identification value (0) indicating the above is generated and output to the response generation unit 104.

In this case, the response generation unit 104 does not execute the response utterance generation process for outputting from the user proxy utterance device 20.

Next, with respect to a processing example (processing example 2 + 3) in which (processing example 2) described with reference to FIG. 13 and (processing example 3) described with reference to FIG. 14 are executed together with reference to FIG. explain.

The processing of each step of the flow shown in FIG. 15 will be described.
(Step S151)
The process of step S151 is the same process as the process of step S101 of the flow described with reference to FIG.

That is, first, the response necessity determination unit 103 of the user substitute utterance device 20 inputs the utterance text of the user utterance partner device 10 from the voice recognition unit 102 in step S151.
Enter the utterance text in which the utterance subject identifier (speaking subject identification tag) indicating whether the utterance text is the user utterance text of the user 1 or the utterance text of the output utterance of the user utterance partner device 10 is set as additional information. ..
When the response necessity determination unit 103 refers to the utterance subject identifier (utterance subject identification tag) and confirms that the input text from the voice recognition unit 102 is the utterance text of the user utterance partner device 10, step S152. Execute the following processing.

(Step S152)
Next, in step S152, the response necessity determination unit 103 determines whether or not the user substitute utterance device 20 is speaking as the utterance immediately before the input utterance text of the user utterance partner device 10.

If this is done, the determination in step S152 is Yes, and the process proceeds to step S155.
If it has not been performed, the determination in step S152 becomes No, and the process proceeds to step S153.
In the case where the utterance is not performed, the utterance immediately before the input utterance text of the user utterance partner device 10 does not exist, and the utterance immediately before the utterance text of the input user utterance partner device 10 is the user 1. Two cases are included in the case of user utterance.

(Step S153)
If it is determined in step S152 that the utterance of the user substitute utterance device 20 has not been performed as the utterance immediately before the utterance text of the input user utterance partner device 10, the process proceeds to step S153.

In this case, the response necessity determination unit 103 determines in step S153 whether or not the user utterance is detected within the specified threshold time (N seconds) from the utterance completion timing of the user utterance partner device 10.

If the user utterance text is input within the specified threshold time (N seconds) from the utterance completion timing of the user utterance partner device 10, the determination in step S153 is Yes, and the process proceeds to step S154.
On the other hand, if the user utterance text is not input within the specified threshold time (N seconds) from the utterance completion timing of the user utterance partner device 10, the determination in step S153 becomes No, and the process proceeds to step S155.

(Step S154)
The process of step S154 is a process to be executed in the following cases.
In step S152, it is determined that the utterance of the user substitute utterance device 20 has not been performed as the utterance immediately before the utterance text of the user utterance partner device 10 input, and further, in step S153, the utterance of the user utterance partner device 10 is completed. When it is determined that the user utterance is detected within the specified threshold time (N seconds) from the timing.

In this case, in step S154, the response necessity determination unit 103 detects the user utterance within the specified threshold time (N seconds) from the utterance completion timing of the user utterance partner device 10 to the user utterance partner device such as a car navigation device. It is determined whether or not it was made for 10.
For example, the determination is made using the semantic analysis result of the user's utterance and the analysis result of the line-of-sight information from the user's face image.

If it is determined that the detected user utterance is made to the user utterance partner device 10, the process proceeds to step S156.
On the other hand, if it is determined that the detected user utterance is not made to the user utterance partner device 10, the process proceeds to step S155.

(Step S155)
The process of step S155 is a process to be executed in any of the following cases.
(A) In step S152, when it is determined that the utterance of the user substitute utterance device 20 is being performed as the utterance immediately before the utterance text of the input user utterance partner device 10.
(B) In step S152, it is determined that the user substitute utterance device 20 has not spoken as the utterance immediately before the utterance text of the user utterance partner device 10 that has been input, and further, in step S153, the user utterance partner device 10 When it is determined that the user utterance is not detected within the specified threshold time (N seconds) from the utterance completion timing of
(C) In step S152, it is determined that the user substitute utterance device 20 has not spoken as the utterance immediately before the utterance text of the user utterance partner device 10 that has been input, and further, in step S153, the user utterance partner device 10 It is determined that the user utterance is detected within the specified threshold time (N seconds) from the utterance completion timing of the above, and further, in step S154, the detected user utterance must be made to the user utterance partner device 10. When judged.

In any of the above cases (a), (b), and (c), in step S155, the response necessity determination unit 103 responds to the response of the user substitute utterance device 20, that is, the user to the output utterance from the user utterance partner device 10. A response necessity identification value (1) indicating that the utterance from the substitute utterance device 20 is “necessary” is generated and output to the response generation unit 104.

In this case, the response generation unit 104 executes a response utterance generation process for outputting from the user proxy utterance device 20.

(Step S156)
On the other hand, in step S152, it is determined that the user substitute utterance device 20 has not spoken as the utterance immediately before the utterance text of the user utterance partner device 10 that has been input, and further, in step S153, the user utterance partner device 10 It is determined that the user utterance is detected within the specified threshold time (N seconds) from the utterance completion timing, and further, in step S154, it is determined that the detected user utterance is made to the user utterance partner device 10. When the determination is made, the response necessity determination unit 103 executes the process of step S156.

In this case, in step S156, the response necessity determination unit 103 is "unnecessary" for the response of the user substitute utterance device 20, that is, the utterance from the user substitute utterance device 20 for the output utterance from the user utterance partner device 10. The response necessity identification value (0) indicating the above is generated and output to the response generation unit 104.

In this case, the response generation unit 104 does not execute the response utterance generation process for outputting from the user proxy utterance device 20.

As described above, a plurality of processing examples have been described with respect to the processing examples executed by the response necessity determination unit 103.
Further, for example, when the response necessity determination unit 103 determines that the response is "necessary" (= 1), the response necessity determination unit 103 outputs a signal for notifying the user 1 that the user substitute utterance device 20 is about to speak (response). It may be configured to be used.
Specifically, as a signal, for example, a blinking LED lamp, an output of a sound effect, or an utterance such as "I will answer" is output.

(4- (4). Details of the process executed by the response generator 104)
Next, the details of the process executed by the response generation unit 104 will be described.

When the response generation request is input from the response necessity determination unit 103 as described above, the response generation unit 104 generates an utterance text corresponding to the utterance to be output from the user proxy utterance device 20.
When generating this utterance text, various information stored in the storage unit 24, that is, the input utterance correspondence response database 121, the user action history information 122, and the user profile information 123 can be used. Further, it is also possible to use the information that can be acquired via the communication unit 25, that is, the information acquired from the external server 150 or the user utterance partner device 10.

The input, output, and execution processing of the response generation unit 104 are as follows.
(A) Response necessity identification value indicating whether or not the response is required by the input user proxy utterance device 20 (for example, required = 1, unnecessary = 0)
When the response necessity identification value (necessary = 1) indicating the response requirement by the user substitute utterance device 20 is input, the utterance text in which the utterance subject identifier (utterance subject identification tag) generated by the voice recognition unit 102 is also set. It is also entered.

(B) Output A utterance text indicating the utterance content output by the user proxy utterance device 20.

(C) Processing When a response generation request is input from the response necessity determination unit 103, the utterance text corresponding to the utterance to be output from the user proxy utterance device 20 is generated.

When the response generation unit 104 generates the utterance text of the utterance output from the user substitute utterance device 20, various information stored in the storage unit 24, that is, the input utterance correspondence response database 121, the user action history information 122, and the user The profile information 123 and the information acquired from the external server 150 and the user utterance partner device 10 are used.

A specific example of the input utterance correspondence response database 121 is shown in FIG.
As shown in FIG. 16, the input utterance correspondence response database 121 is
(A) Sample input utterance (B) Response utterance This is a database that stores a large number of corresponding data of (A) and (B).

(A) The sample input utterance is sample data of the utterance text assuming the utterance of the user utterance partner device 10.
(B) The response utterance is the utterance text data of the utterance output from the user substitute utterance device 20 for each sample data of the utterance text assuming the utterance of the user utterance partner device 10.

For example, the response generation unit 104 of the user substitute utterance device 20 first inputs the utterance text from the response necessity determination unit 103 together with the response necessity identification value (necessity = 1). That is, the utterance text of the user utterance partner device 10 is input.
Next, "(A) sample input utterance" having the highest degree of similarity to the input utterance text is selected from the input utterance correspondence response database 121.
Further, the registered "(B) response utterance" associated with the selected "(A) sample input utterance" is selected, and the utterance text of the selected "(B) response utterance" is used as the output utterance. ..

However, it may not be possible to determine the utterance text to be output from one user proxy utterance device 20 only by referring to the registered data of the input utterance correspondence response database 121.
In such a case, the user action history information 122, the user profile information 123, and the information acquired from the external server 150 and the user utterance partner device 10 are used.

A specific example of the user action history information 122 is shown in FIG.
As shown in FIG. 17, the user action history information 122 contains the past action history information of the user 1.
(A) Date and time (b) Behavior A large number of corresponding data of these (a) and (b) are stored.

When the response generation unit 104 of the user substitute utterance device 20 cannot determine the utterance text to be output from one user substitute utterance device 20 only by referring to the registration data of the input utterance correspondence response database 121, the response generation unit 104 registers the user action history information 122. By referring to the data and other information, the intention of the user 1 is estimated, and the proxy utterance on behalf of the user 1 is determined.

Other information other than the user action history information 122 includes, for example, information acquired from various external servers such as a user profile information 123, a knowledge database, a scenario database storing dialogue sequence information, and an SNS (Social Networking Service) server. Or, the accumulated data of the user talking partner device 10.
The response generation unit 104 acquires and refers to this information, estimates the intention of the user 1, and determines a substitute utterance on behalf of the user 1.

The user profile information 123 records the gender, age, hobbies, family structure, etc. of the user 1.
Further, the external server 150 includes various servers such as a knowledge database, a scenario database storing dialogue sequence information, and an SNS (Social Networking Service) server.
For example, in the SNS server, utterances, images, and the like made by the user 1 using the SNS are registered. The response generation unit 104 of the user substitute utterance device 20 performs a process of estimating the user's intention by analyzing the utterance or posted image of the user 1 registered in the SNS server, for example.

Hereinafter, a specific example of the processing executed by the response generation unit 104 will be described.
As an input from the response necessity determination unit 103,
When a response necessity identification value (necessary = 1) indicating the response required by the user substitute utterance device 20 is input, and an utterance text in which the utterance subject identifier (speaking subject identification tag) generated by the voice recognition unit 102 is input is input. The processing of is described.

It is assumed that the input utterance text, that is, the utterance text of the user utterance partner device 10 is the following utterance text.
The other party's utterance text = "I'm clever. Are you sure you want to go to Odaiba Station? "

When such an utterance text is input, the response generation unit 104 generates, for example, the following utterance text of the user substitute utterance device 20.
Proxy device utterance text = "Please make it Odaiba Kaihin Koen Station"

The response generation unit 104 uses the input utterance correspondence response database 121, the user action history information 122, and the user profile information 123 in order to generate an utterance suitable as a response to the utterance of the user utterance partner device 10 as described above. Further, if necessary, information that can be acquired via the communication unit 25, that is, information acquired from the external server 150 or the user utterance partner device 10 is used.

A specific response utterance generation processing sequence executed by the response generation unit 104 will be described.

First, the response generation unit 104 has the highest degree of similarity to the utterance text input from the response necessity determination unit 103 together with the response necessity identification value (necessity = 1), that is, the utterance text of the user speech partner device 10. (A) Sample input utterance ”is selected from the input utterance correspondence response database 121.

For example, "(A) sample input utterance" of the input utterance correspondence response database 121 having the highest degree of similarity to the utterance text of the user utterance partner device 10 is
"Understood. Are you sure you want to go to Odaiba Station? "
It is assumed that the above sample input utterance is used.

The "similarity" used as the determination index value when selecting "(A) sample input utterance" in the input utterance correspondence response database 121 having the highest degree of similarity to the utterance text of the user utterance partner device 10 is, for example, You can use the degree of duplication of words and phrases contained in the utterance. For example, morphological analysis can be performed to determine the degree of similarity based on the number of common morphemes.

As a specific example
User utterance The utterance of the other party device 10
The other party's utterance text = "I'm clever. Are you sure you want to go to Odaiba Station? "
The degree of similarity between this utterance and the two "(A) sample input utterances" of entry 122 and entry 201 shown in FIG. 16, which are the registration data of "(A) sample input utterance" in the input utterance correspondence response database 121. compare.
Sample input utterance of entry 121 = "I'm clever. May I decide on a destination? ”
Sample input utterance of entry 201 = "I'm clever. Are you sure you want to go to Odaiba Station? "
The degree of similarity with these two "(A) sample input utterances" is compared.

The other party's utterance text = "I'm clever. Are you sure you want to go to Odaiba Station? "
First, the utterance text of the other device is divided into words in morpheme units.
"Understood/. / Purpose / Location / Ha / Odaiba / Station / De / OK / Is / Ka ”
(13 morphemes (words))

On the other hand, the sample input utterance of entry 122 = "I'm clever. May I decide on a destination? ”
Sample input utterance of entry 201 = "I'm clever. Are you sure you want to go to Odaiba Station? "
These two "(A) sample input utterances" are also divided into words in morpheme units.

Entry 122 = "Skillful / better / ta /. / Purpose / Ground / Determine / Decide / te / mo / good / is / ka ”(morpheme (word) 13 words)
Entry 201 = "Skillful / better / ta /. / Destination / Ha / Odaiba / Station / De / OK / Is / ka ”(Morpheme (word) 12 words)

The number of common morphemes between the utterance text of the other device and the entry 122 is 8 (smart / better / ta /. / purpose / ground / is / ka).
The number of morphemes common to the other device's utterance text and entry 201 is 11 (Kashikomari / Mashi / Ta /.

As a result, the response generation unit 104
The other party's utterance text = "I'm clever. "Are you sure you want to go to Odaiba Station?"
Sample input utterance of entry 201 = "I'm clever. Are you sure you want to go to Odaiba Station? "
Judge that it is the above entry.

In the similarity determination process, the response generation unit 104 not only compares the numbers of the above common morphemes, words, phrases, etc., but also has different notations such as "destination" and "destination". You may also perform more flexible similarity calculation considering that there are words and phrases that are semantically close to each other. As a specific process in this case, for example, there is a process of performing similarity analysis using a morpheme (word) or a distributed expression of a phrase.

It should be noted that the registered data of "(A) sample input utterance" of the input utterance correspondence response database 121 shown in FIG. 16 is shown in FIG.
Sample input utterance of entry 201 = "I'm clever. Are you sure you want to go to Odaiba Station? "
Sample input utterance of entry 202 = "I'm clever. Are you sure you want to go to Odaiba Station? "
These are exactly the same sample input utterances.

That is, these two entries 201 and 202 registered in the input utterance correspondence response database 121 are
The other party's utterance text = "I'm clever. Are you sure you want to go to Odaiba Station? ”Will be selected as the entry with the highest degree of similarity.

In this way, when a plurality of entries in which the same sample input utterance having the highest similarity to the utterance text of the other device is registered are detected as the entries of "(A) sample input utterance" in the input utterance correspondence response database 121. , The response generation unit 104 executes any of the following processes.
(A) Randomly select one or the other.
(B) Other information, that is, the user action history information 122, the user profile information 123, the external server 150, and the information acquired from the user utterance partner device 10 are used for selection.

For example, when performing a selection process using the user behavior history information 122, it was confirmed that the user behavior history information 122 frequently visits "Odaiba Kaihin Koen Station" as the behavior history of the user 1. If the response utterance of entry 201, ie
Response utterance = "Please make it Odaiba Kaihin Koen Station"
Processing such as selecting this response utterance can be performed.

In addition to this, even if the process of deciding by referring to information such as user profile (name, age, gender), remarks on SNS (what kind of topic you are interested in and disseminating information) is included. good.

If the candidate cannot be narrowed down to one even by using the user action history information 122 or other information, a candidate randomly selected from the remaining candidates may be output, or the user may output the candidate. You may make an utterance requesting a response to one person.

Further, as an additional function of the response generation unit 104, the utterance text to be generated may be converted into a wording suitable for the user himself / herself. For example, a personal pronoun (for example, me, me, me), a sentence ending expression (for example, kashira, kana, dayo), or other expressions frequently used by the user 1 may be applied according to the user 1. The unique representation of this user is acquired by referring to, for example, the user profile information 123 and the registration data of the SNS server and the like constituting the external server 150.

FIG. 18 is a diagram showing a detailed configuration example of the response generation unit 104.
As shown in FIG. 18, the response generation unit 104 includes a substitute utterance selection unit 151 and a substitute utterance expression conversion unit 152.

The substitute utterance selection unit 151 inputs the “utterance text of the user utterance partner device 10” 201 shown in FIG. 18 together with the response generation request from the response necessity determination unit 103.
The substitute utterance selection unit 151 performs a selection process of the utterance text corresponding to the utterance to be output from the user substitute utterance device 20 with respect to the input text.

The utterance text selection process is selected from the input utterance correspondence response database 121 in the storage unit 24.
That is, it has been described above with reference to FIG.
(A) Sample input utterance (B) Response utterance Select from the input utterance correspondence response database 121 in which these correspondence data are registered.

The substitute utterance selection unit 151 selects "(A) sample input utterance" of the input utterance correspondence response database 121 having the highest degree of similarity to the utterance text of the user utterance partner device 10.
As mentioned above, for example, the multiplicity of words and phrases contained in an utterance can be used. For example, morphological analysis is performed to determine the degree of similarity based on the number of common morphemes.

If, as entries for "(A) sample input utterance" in the input utterance correspondence response database 121, a plurality of entries in which the same sample input utterance having the highest similarity to the other device's utterance text is registered are detected on behalf of the other device. The utterance selection unit 151 executes any of the following processes.
(A) Randomly select one or the other.
(B) Other information, that is, the user action history information 122, the user profile information 123, the external server 150, and the information acquired from the user utterance partner device 10 are used for selection.

The utterance selected by the substitute utterance selection unit 151 is input to the substitute utterance expression conversion unit 152.
The substitute utterance expression conversion unit 152 performs a process of converting the utterance selected by the substitute utterance selection unit 151 into the wording of the user himself / herself. For example, as described above, user 1's frequently used expressions such as personal pronouns (for example, me, me, me) and sentence ending expressions (for example, kashira, kana, dayo) are applied according to user 1. Change the utterance expression. The user's unique representation is acquired by referring to, for example, the user profile information 123 and the registration data of the SNS server and the like constituting the external server 150.

The utterance text generated by the substitute utterance expression conversion unit 152, that is, the “speech text of the user substitute utterance device” 202 shown in FIG. 18 is input to the voice synthesis unit 105.

Next, a sequence of processes executed by the response generation unit 104 will be described with reference to the flowchart shown in FIG.
The processing of each step of the flow shown in FIG. 19 will be sequentially described.

(Step S201)
First, in step S201, the response generation unit 104 inputs the “utterance text of the user utterance partner device 10” together with the response generation request from the response necessity determination unit 103.

(Step S202)
Next, in step S202, the response generation unit 104 accesses the input utterance correspondence response database 121, executes a search process based on the utterance text of the output utterance of the user utterance partner device 10, and inputs a sample registered in the database. From the utterances, select the entry in which the sample input utterance with the highest similarity is registered.

As mentioned above, for example, the multiplicity of words and phrases contained in an utterance can be used. For example, morphological analysis is performed to determine the degree of similarity based on the number of common morphemes.

(Step S203)
Next, in step S203, the response generation unit 104 determines whether or not an entry having a high degree of similarity is selected in the entry selection process in step S202.

If the selection of entries having a high degree of similarity fails, the determination in step S203 becomes No, and the process ends. In this case, no utterance is made from the user substitute utterance device 20.
On the other hand, if the selection of entries having a high degree of similarity is successful, the determination in step S203 is Yes, and the process proceeds to step S204.

(Step S204)
If the entry with high similarity is successfully selected from the input utterance correspondence response database 121 in step S202, the process of step S204 is executed.

In step S204, the response generation unit 104 can select one entry in which a response that can be determined to be optimal is registered based on the user behavior history registered in the user behavior history information 122 and other information. If it is possible to select it, select it.

The other information includes user profile information 123, a knowledge database, a scenario database storing dialogue sequence information, information acquired from various external servers such as an SNS (Social Networking Service) server, and a user speaking partner device 10. Accumulated data, etc.

(Step S205)
Next, in step S205, the response generation unit 104 determines whether or not one entry of the substitute utterance output by the user substitute utterance device has been successfully selected.

If one entry registered in the input utterance correspondence response database 121 is successfully selected, the process proceeds to step S207.
If the narrowing down of the entries registered in the input utterance correspondence response database 121 fails and a plurality of entries are selected, the process proceeds to step S206.

(Step S206)
If the narrowing down of the entries registered in the input utterance correspondence response database 121 fails in step S205 and a plurality of entries are selected, the response generation unit 104 selects one from the plurality of selected entries in step S206. Select randomly.

(Step S207)
Finally, in step S207, the response generation unit 104 selects the response utterance registered in one selection entry and outputs it to the speech synthesis unit 105.

If one entry registered in the input utterance correspondence response database 121 is successfully selected in step S205, the response utterance registered in the one entry is selected and output to the speech synthesis unit 105.

On the other hand, if the selection of one entry registered in the input utterance correspondence response database 121 fails in step S205, the response utterance registered in the entry randomly selected in step S206 is selected and the voice synthesis unit 105 selects the response utterance. Output.

In the flow shown in FIG. 19, when the response generation unit 104 fails to select one entry registered in the input utterance correspondence response database 121 in step S205, the response generation unit 104 starts from the plurality of entries selected in step S206. The configuration is such that one entry is randomly selected. For example, in such a case, the proxy utterance may be stopped and the user utterance may be waited for.

Alternatively, the configuration may be such that an utterance for requesting the user to speak is output.
FIG. 20 shows a processing sequence when executing a process of outputting an utterance for requesting an utterance to a user.

The flow shown in FIG. 20 is the processing of steps S201 to S205, and the processing of step S207 is the same processing as the processing of each step of the flow described with reference to FIG.

In the flow shown in FIG. 20, the process of step S206 of the flow shown in FIG. 19 is replaced with the process of step S221. The process of step S221 will be described.

(Step S221)
In step S205, when the narrowing down of the entries registered in the input utterance correspondence response database 121 fails and a plurality of entries are selected, the response generation unit 104 requests the user to speak in step S221. Generates and outputs the utterance of.
Such processing may be performed.

(4- (5). Details of the process executed by the speech synthesizer 105)
Next, the details of the processing executed by the voice synthesis unit 105 will be described.
As described above, the voice synthesis unit 105 generates a synthetic voice based on the utterance text generated by the response generation unit 104. That is, the voice synthesis process (TTS: Text To Speech) is executed, and the generated synthetic voice is output as the user speech agent output voice 53 shown in the figure via the voice output unit (speaker) 23.

The input, output, and execution processes of the voice synthesis unit 105 are as follows.
(A) Spoken text indicating the utterance content output by the user substitute utterance device 20 generated by the input response generation unit 104 (b) Synthetic voice generated based on the utterance text generated by the output response generation unit 104.

(C) Processing A synthetic voice is generated based on the utterance text generated by the response generation unit 104. That is, voice synthesis processing (TTS: Text To Speech) is executed to generate synthetic voice.
The generated synthetic voice is output via the voice output unit (speaker) 23.

For example, it is assumed that the utterance text indicating the utterance content output by the user substitute utterance device 20 generated by the response generation unit 104 is the following text.
"Please make it Odaiba Kaihin Koen Station"
The voice synthesis unit 105 converts this utterance text into voice. For this process (speech synthesis), a speech synthesis process (TTS: Text To Speech) execution program can be used. It may also be executed using open source software.

The voice synthesis unit 105 may change the voice synthesis model according to the user 1 so that the voice similar to the utterance of the user himself / herself is output during the synthetic voice generation process. For example, a method of selecting a model according to the attributes (age, gender, etc.) of the user 1 may be adopted, or a process of selecting a model similar to the voice quality of the user himself / herself from among a number of speech synthesis models may be adopted. May be.

Further, when the utterance voice from the user 1 is input during the voice output, it is determined that the user 1 intervenes in the utterance of the user substitute utterance device 20 and intends to stop the utterance, and the voice output is stopped. You may do so.

When the utterance voice from the user 1 is input during the voice output, it is determined whether or not the user utterance is directed to the user substitute utterance device 20, and the utterance is directed to the user substitute utterance device 20. The utterance of the user proxy utterance device 20 may be stopped only when it is determined that the utterance is a user.

[5. Information processing device hardware configuration example]
Next, a hardware configuration example of the information processing device will be described with reference to FIG.
The hardware described with reference to FIG. 21 is an example of the hardware configuration of the information processing device constituting the user proxy utterance device 20.

The CPU (Central Processing Unit) 301 functions as a control unit or a data processing unit that executes various processes according to a program stored in the ROM (Read Only Memory) 302 or the storage unit 308. For example, the process according to the sequence described in the above-described embodiment is executed. The RAM (Random Access Memory) 303 stores programs and data executed by the CPU 301. These CPU 301, ROM 302, and RAM 303 are connected to each other by a bus 304.

The CPU 301 is connected to the input / output interface 305 via the bus 304, and the input / output interface 305 is connected to an input unit 306 consisting of various switches, a keyboard, a mouse, a microphone, a sensor, etc., and an output unit 307 consisting of a display, a speaker, and the like. Has been done. The CPU 301 executes various processes in response to a command input from the input unit 306, and outputs the process results to, for example, the output unit 307.

The storage unit 308 connected to the input / output interface 305 is composed of, for example, a hard disk or the like, and stores a program executed by the CPU 301 and various data. The communication unit 309 functions as a transmission / reception unit for Wi-Fi communication, Bluetooth (registered trademark) (BT) communication, and other data communication via a network such as the Internet or a local area network, and communicates with an external device.

The drive 310 connected to the input / output interface 305 drives a removable medium 311 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory such as a memory card, and records or reads data.

[6. Summary of the structure of this disclosure]
As described above, the examples of the present disclosure have been described in detail with reference to the specific examples. However, it is self-evident that one of ordinary skill in the art can modify or substitute the examples without departing from the gist of the present disclosure. That is, the present invention has been disclosed in the form of an example, and should not be construed in a limited manner. In order to judge the gist of this disclosure, the column of claims should be taken into consideration.

The technology disclosed in the present specification can have the following configuration.
(1) Input the device utterance output from the user utterance partner device, which is the user's utterance partner, and input the device utterance.
It has a data processing unit that generates and outputs a user substitute utterance in place of the user in response to the device utterance.
The data processing unit
A response necessity determination unit that determines the necessity of the user proxy utterance, and a response necessity determination unit.
An information processing device having a response generation unit that generates a user proxy utterance when the response necessity determination unit determines that a user proxy utterance is necessary.

(2) The response necessity determination unit is
The information processing device according to (1), wherein it is determined that a user proxy utterance is necessary when the user utterance is not performed within a predetermined threshold time from the utterance completion timing of the device utterance.

(3) The response necessity determination unit is
The information processing device according to (1) or (2), wherein it is determined that the user proxy utterance is unnecessary when the user utterance is performed within a predetermined threshold time from the utterance completion timing of the device utterance.

(4) The response necessity determination unit is
When the user utterance is performed within the predetermined threshold time from the utterance completion timing of the device utterance.
It is determined whether the user utterance is an utterance made to the user utterance partner device, and the user utterance is determined.
If it is determined that the user utterance is not the utterance made to the user utterance partner device, it is determined that the user proxy utterance is necessary.
The information processing device according to any one of (1) to (3), wherein when it is determined that the user utterance is an utterance made to the user utterance partner device, it is determined that the user proxy utterance is unnecessary.

(5) The response necessity determination unit is
The information processing device according to (4), wherein the determination process of whether or not the user utterance is an utterance made to the user utterance partner device is determined by using the semantic analysis result of the user utterance.

(6) The response necessity determination unit is
The information processing device according to (4), wherein the determination process of whether or not the user's utterance is an utterance made to the user's utterance partner device is determined by using the user's line-of-sight direction analysis result.

(7) The response necessity determination unit is
In the case where the substitute utterance has never been executed after the start of the dialogue sequence with the user utterance partner device.
The information according to any one of (1) to (6), in which it is determined that the user substitute utterance is necessary when the user utterance is not performed within the predetermined threshold time from the utterance completion timing of the device utterance. Processing equipment.

(8) The response necessity determination unit is
When the proxy utterance is executed more than once after the start of the dialogue sequence with the user utterance partner device
The information processing device according to (7), wherein it is determined that all subsequent device utterances require user proxy utterances.

(9) The information processing device is
A voice recognition unit that identifies whether the input utterance to the information processing device is the user utterance uttered by the user or the device utterance output by the user utterance partner device, and generates the utterance subject identifier that is the identification result. Have and
The response necessity determination unit
When it is confirmed that the input utterance to the information processing device is the device utterance output by the user utterance partner device based on the utterance subject identifier input from the voice recognition unit.
The information processing device according to any one of (1) to (8), which determines the necessity of the user proxy utterance.

(10) The response generation unit
From the input utterance support database that stores a large number of entries that associate sample input utterances with response utterances
Select the entry for the sample input utterance that has the highest similarity to the device utterance.
The information processing device according to any one of (1) to (9), wherein the response utterance of the selected entry is set as the user substitute utterance.

(11) The response generation unit
When selecting the entry for the sample input utterance that has the highest similarity to the device utterance
The information processing apparatus according to (10), wherein the device utterance is compared with each morpheme, word, or phrase of each sample input utterance to determine the degree of similarity.

(12) The response generation unit
The information processing device according to any one of (1) to (11), which estimates the user's intention by referring to the user behavior history information and generates a user proxy utterance that reflects the user's intention.

(13) The response generation unit
The information processing apparatus according to any one of (1) to (12), wherein the user's intention is estimated by referring to the user profile information or the registration information of the external server, and the user's proxy utterance reflecting the user's intention is generated.

(14) The response generation unit
The information processing device according to any one of (1) to (13), which generates a user proxy utterance to which an expression often used by the user is applied.

(15) An information processing method for executing information processing in an information processing device.
The information processing device
Input the device utterance output from the user utterance partner device, which is the user's utterance partner,
It has a data processing unit that generates and outputs a user substitute utterance in place of the user in response to the device utterance.
The data processing unit
The response necessity determination process for determining the necessity of the user proxy utterance and the response necessity determination process
An information processing method for executing a response generation process for generating a user proxy utterance when it is determined in the response necessity determination process that a user proxy utterance is necessary.

(16) A program that executes information processing in an information processing device.
The information processing device
Input the device utterance output from the user utterance partner device, which is the user's utterance partner,
It has a data processing unit that generates and outputs a user substitute utterance in place of the user in response to the device utterance.
The program is installed in the data processing unit.
The response necessity determination process for determining the necessity of the user proxy utterance and the response necessity determination process
A program that executes a response generation process for generating a user proxy utterance when it is determined in the response necessity determination process that a user proxy utterance is necessary.

In addition, the series of processes described in the specification can be executed by hardware, software, or a composite configuration of both. When executing processing by software, install the program that records the processing sequence in the memory in the computer built in the dedicated hardware and execute it, or execute the program on a general-purpose computer that can execute various processing. It can be installed and run. For example, the program can be pre-recorded on a recording medium. In addition to installing on a computer from a recording medium, the program can be received via a network such as LAN (Local Area Network) or the Internet and installed on a recording medium such as a built-in hard disk.

The various processes described in the specification are not only executed in chronological order according to the description, but may also be executed in parallel or individually as required by the processing capacity of the device that executes the processes. Further, in the present specification, the system is a logical set configuration of a plurality of devices, and the devices having each configuration are not limited to those in the same housing.

As described above, according to the configuration of one embodiment of the present disclosure, a device and a method for generating and outputting a response utterance on behalf of the user with respect to the device utterance output by the interactive device are realized. ..
Specifically, for example, the device utterance from the user utterance partner device, which is the user's utterance partner, is input, and the user substitute utterance is generated and output on behalf of the user. It has a response necessity determination unit that determines the necessity of a user proxy utterance, and a response generation unit that generates a user proxy utterance when it is determined that a user proxy utterance is necessary. The response generation unit generates and outputs a proxy utterance that reflects the user's intention by referring to, for example, the user action history information. The response necessity determination unit determines that the user proxy utterance is necessary when the user utterance is not performed within the predetermined threshold time from the utterance completion timing of the device utterance.
With this configuration, a device and a method for generating and outputting a response utterance on behalf of the user for the device utterance output by the interactive device are realized.

10 User utterance partner device 20 User utterance device 21 Voice input unit (microphone)
22 Data processing unit 23 Audio output unit (speaker)
24 Communication unit 25 Storage unit 26 Image input unit (camera)
101 Utterance detection unit 102 Voice recognition unit 103 Response necessity determination unit 104 Response generation unit 105 Voice synthesis unit 121 Input utterance correspondence response database 122 User action history information 123 User profile information 150 External server 151 Proxy utterance selection section 152 Proxy utterance expression conversion Part 301 CPU
302 ROM
303 RAM
304 Bus 305 Input / output interface 306 Input unit 307 Output unit 308 Storage unit 309 Communication unit 310 Drive 311 Removable media

Claims (16)

Input the device utterance output from the user utterance partner device, which is the user's utterance partner,
It has a data processing unit that generates and outputs a user substitute utterance in place of the user in response to the device utterance.
The data processing unit
A response necessity determination unit that determines the necessity of the user proxy utterance, and a response necessity determination unit.
An information processing device having a response generation unit that generates a user proxy utterance when the response necessity determination unit determines that a user proxy utterance is necessary. The response necessity determination unit
The information processing device according to claim 1, wherein it is determined that a user proxy utterance is necessary when the user utterance is not performed within a predetermined threshold time from the utterance completion timing of the device utterance. The response necessity determination unit
The information processing device according to claim 1, wherein it is determined that the user proxy utterance is unnecessary when the user utterance is performed within a predetermined threshold time from the utterance completion timing of the device utterance. The response necessity determination unit
When the user utterance is performed within the predetermined threshold time from the utterance completion timing of the device utterance.
It is determined whether the user utterance is an utterance made to the user utterance partner device, and the user utterance is determined.
If it is determined that the user utterance is not the utterance made to the user utterance partner device, it is determined that the user proxy utterance is necessary.
The information processing device according to claim 1, wherein when it is determined that the user utterance is an utterance made to the user utterance partner device, it is determined that the user proxy utterance is unnecessary. The response necessity determination unit
The information processing device according to claim 4, wherein a determination process of whether or not the user utterance is an utterance made to the user utterance partner device is determined by using the semantic analysis result of the user utterance. The response necessity determination unit
The information processing device according to claim 4, wherein a determination process of whether or not the user's utterance is an utterance made to the user's utterance partner device is determined by using the user's line-of-sight direction analysis result. The response necessity determination unit
In the case where the substitute utterance has never been executed after the start of the dialogue sequence with the user utterance partner device.
The information processing device according to claim 1, wherein it is determined that a user proxy utterance is necessary when the user utterance is not performed within a predetermined threshold time from the utterance completion timing of the device utterance. The response necessity determination unit
When the proxy utterance is executed more than once after the start of the dialogue sequence with the user utterance partner device
The information processing device according to claim 7, wherein it is determined that all subsequent device utterances require user proxy utterances. The information processing device
A voice recognition unit that identifies whether the input utterance to the information processing device is the user utterance uttered by the user or the device utterance output by the user utterance partner device, and generates the utterance subject identifier that is the identification result. Have and
The response necessity determination unit
When it is confirmed that the input utterance to the information processing device is the device utterance output by the user utterance partner device based on the utterance subject identifier input from the voice recognition unit.
The information processing device according to claim 1, wherein the necessity of the user proxy utterance is determined. The response generator
From the input utterance support database that stores a large number of entries that associate sample input utterances with response utterances
Select the entry for the sample input utterance that has the highest similarity to the device utterance.
The information processing device according to claim 1, wherein the response utterance of the selected entry is set as the user proxy utterance. The response generator
When selecting the entry for the sample input utterance that has the highest similarity to the device utterance
The information processing device according to claim 10, wherein the device utterance is compared with each morpheme, word, or phrase of the sample input utterance to determine the similarity. The response generator
The information processing device according to claim 1, wherein the user's intention is estimated by referring to the user behavior history information, and the user's proxy utterance reflecting the user's intention is generated. The response generator
The information processing device according to claim 1, wherein the user's intention is estimated by referring to the user profile information or the registration information of the external server, and the user's proxy utterance reflecting the user's intention is generated. The response generator
The information processing device according to claim 1, wherein a user proxy utterance to which an expression often used by the user is applied is generated. It is an information processing method that executes information processing in an information processing device.
The information processing device
Input the device utterance output from the user utterance partner device, which is the user's utterance partner,
It has a data processing unit that generates and outputs a user substitute utterance in place of the user in response to the device utterance.
The data processing unit
The response necessity determination process for determining the necessity of the user proxy utterance and the response necessity determination process
An information processing method for executing a response generation process for generating a user proxy utterance when it is determined in the response necessity determination process that a user proxy utterance is necessary. A program that executes information processing in an information processing device.
The information processing device
Input the device utterance output from the user utterance partner device, which is the user's utterance partner,
It has a data processing unit that generates and outputs a user substitute utterance in place of the user in response to the device utterance.
The program is installed in the data processing unit.
The response necessity determination process for determining the necessity of the user proxy utterance and the response necessity determination process
A program that executes a response generation process for generating a user proxy utterance when it is determined in the response necessity determination process that a user proxy utterance is necessary.

Images