CN110136713A - User's dialogue method and system in multimodal interaction - Google Patents

Abstract

The present invention discloses a kind of dialogue method of user in multi-modal interaction, including, establish semantic corresponding relationship.Establish semantic instructions collection.Current session paragraph is obtained by interactive device, the interactive device is the interactive device that can be realized multi-modal information input.The semantic corresponding relationship is called according to current session paragraph, obtains current semantics information.The semantic instructions collection is called according to the current semantics information, obtains and currently locally executes instruction.Described currently locally execute is executed to instruct and show implementing result at user session end.What synchronous local cache data was solved to cloud service.After a wheel end-of-dialogue or user switches dialogue topic, for example from weather, is switched to music, and want to carry out the dialogue of weather associated topic, precisely due in above scheme, supports dialogue state is synchronized in cloud service by client to solve.

Description

User's dialogue method and system in multimodal interaction

technical field

The invention belongs to the technical field of user interaction and multimodality, realizes multimodal interaction experience such as voice, screen, camera, etc., and particularly relates to a user dialogue method and system in multimodal interaction.

Background technique

In the dialogue process of intelligent interaction and human-machine dialogue, the client performs multimodality, and the multimodality here includes voice interaction, text interaction and event interaction. The above-mentioned multi-modal interaction method requires network interaction with the dialogue server, but multi-modal operation interaction is a relatively simple action compared to voice interaction, and the user has an intuitive feeling. After the touch screen operation, I hope to get correct and more timely feedback, but in the case of a poor network environment, network interaction will inevitably be affected, especially in the navigation field, so the user experience will be greatly reduced.

When the prior art solves the above problems, in general, the multi-modal operation is first converted into text with the same meaning, and transmitted to the cloud network through the network, and the process of recognition, semantic analysis, and dialogue is performed. After that, the client needs to support the interworking of multi-modal operations and voice interaction processes; generally, competing companies will only support a certain type of action, such as smart speaker products, which only need to support voice interaction, and are used in scenarios where the network environment is relatively stable, such as at home. . Finally, the client needs to have offline semantic parsing.

SUMMARY OF THE INVENTION

Embodiments of the present invention provide a method and system for user dialogue in multimodal interaction, which are used to solve at least one of the above technical problems.

In a first aspect, an embodiment of the present invention provides a user dialogue method in multimodal interaction, including:

Step S101, establishing a semantic correspondence, where the semantic correspondence is a one-to-one correspondence between multiple dialogue segments and multiple semantic information;

Step S102, establishing a semantic instruction set, where the semantic instruction set refers to: establishing a corresponding association between the plurality of semantic information and a plurality of local execution instructions, and the set formed by the association;

Step S103, obtaining the current dialogue segment through an interactive device, where the interactive device is an interactive device capable of inputting multimodal information;

Step S104, calling the semantic correspondence according to the current dialogue segment to obtain current semantic information;

Step S105, calling the semantic instruction set according to the current semantic information to obtain the current local execution instruction;

Step S106: Execute the current local execution instruction and display the execution result on the user dialogue terminal; the user dialogue terminal is the user local terminal.

In another embodiment of the present invention, the step S103 includes:

Step S1031a, obtaining current voice information and current text information through the interactive device;

Step S1032a, converting the current voice information into a current text field by voice recognition;

Step S1033a, determining the current text information or the current text field as the current dialogue segment.

In another embodiment of the present invention, the interaction device includes a touch screen capable of serving as an input device;

The step S103 includes:

Establish an event correspondence relationship, where the event correspondence relationship is a one-to-one correspondence between multiple event operation information and multiple dialogue segments, and the multiple event operation information is the user's single-click, double-click, The information that the touch screen can obtain when sliding along the single-line trajectory and sliding along the dual-line trajectory;

Step S1031b, receiving current event operation information through the interactive device, where the current event operation information is included in the event operation information;

Step S1032b, calling the event correspondence according to the current event operation information to obtain the current dialogue segment.

In another embodiment of the present invention, the plurality of semantic information includes: a character field for receiving a retrieval character, a character field for calling a map, a character field for calling a system operation, a screen zooming in and a screen zooming out character field;

The multiple local execution instructions include: retrieving and displaying local data, retrieving and displaying a local map, displaying a local dialog box, executing a system call, a command for zooming in an image of a partial area of the screen, and an instruction for reducing an image of a partial area of the screen.

In another embodiment of the present invention, the plurality of locally executed instructions include calling a local

Data execution instructions and execution instructions for calling remote data;

The step S106 includes:

Step S1061, if the current local execution instruction is an execution instruction that calls local data, then call the current local data, execute the current local execution instruction according to the current local data and display the execution result on the user dialog; and

Step S1062, if the current local execution instruction is an execution instruction that calls remote data, then establish a remote data link, call the current remote data from a remote location, execute the current local execution instruction according to the current remote data, and display the execution result in the User dialog.

In another embodiment of the present invention, the step S1062 further includes, if the remote data link cannot be established, periodically detecting the remote network signal; if the remote network signal can be detected, establishing the remote data link.

In a second aspect, an embodiment of the present invention provides a user dialogue system in multimodal interaction, including:

The semantic correspondence unit is configured to establish the semantic correspondence as a one-to-one correspondence between a plurality of dialogue segments and a plurality of semantic information.

A semantic instruction set unit, which is configured to establish a corresponding association between the plurality of semantic information and a plurality of locally executed instructions, a set formed by the association.

The current dialogue segment acquiring unit is configured to acquire the current dialogue segment through an interaction device, the interaction device being an interaction device capable of inputting multimodal information.

The current semantic information obtaining unit is configured to call the semantic correspondence in the semantic correspondence unit according to the current dialogue segment to obtain the current semantic information.

The current local execution instruction acquiring unit is configured to call the semantic instruction set in the semantic instruction set unit according to the current semantic information to acquire the current local execution instruction.

A dialog unit, configured to execute the current local execution instruction and display the execution result on the user dialog end; the user dialog end is the user local end.

In another embodiment of the present invention, the current dialogue segment obtaining unit is further configured to: obtain current voice information and current text information through the interactive device; convert the current voice information into current text through voice recognition field; determining the current text information or the current text field as the current dialogue segment.

In another embodiment of the present invention, the interaction device includes a touch screen capable of serving as an input device;

The current dialogue segment acquisition unit is further configured to: establish an event corresponding relationship, the event corresponding relationship is a one-to-one correspondence between multiple event operation information and multiple dialogue segments, and the multiple event operation information is the user's corresponding relationship. The information that the touch screen can obtain when the touch screen performs single-click tapping, double-clicking, sliding along a single-line trajectory, and sliding along a double-line trajectory; and receiving current event operation information through the interactive device, the current event operation information Included in the event operation information; calling the event correspondence according to the current event operation information to obtain the current dialogue segment.

In another embodiment of the present invention, the plurality of local execution instructions include an execution instruction for calling local data and an execution instruction for calling remote data;

The dialogue unit is further configured to, if the current local execution instruction is an execution instruction that calls local data, call the current local data, execute the current local execution instruction according to the current local data, and display the execution result in the user dialogue. and if the current local execution instruction is an execution instruction that calls remote data, then a remote data link is established, the current remote data is called from a remote location, the current local execution instruction is executed according to the current remote data and the execution result is displayed in the User dialog.

In a third aspect, an electronic device is provided, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, The instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the method of any embodiment of the present invention.

In a fourth aspect, an embodiment of the present invention further provides a computer program product, the computer program product includes a computer program stored on a non-volatile computer-readable storage medium, the computer program includes program instructions, and when the program is The instructions, when executed by a computer, cause the computer to perform the steps of the method of any embodiment of the present invention.

In the method of user dialogue in multimodal interaction in this application, the multimodal information (voice interaction information, text interaction information and event interaction information) input by the user locally is analyzed by establishing a semantic instruction set locally, and the current user's current interaction information is obtained. the corresponding command. The corresponding instructions are executed on the user side, so that the user can perform multi-modal operations to meet the interaction requirements, and the system will not respond slowly due to network delays and other reasons, thereby reducing the user experience. The system responds quickly because the above solution provides local dialog customization and synchronization of locally cached data to cloud services. After a round of conversation ends, or the user switches the conversation topic, such as from weather to music, and wants to have a conversation on weather-related topics, this is because in the above solution, the client is supported to synchronize the conversation state to the cloud service to solve the problem. .

Description of drawings

In order to illustrate the technical solutions of the embodiments of the present invention more clearly, the following briefly introduces the accompanying drawings used in the description of the embodiments. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained from these drawings without any creative effort.

FIG. 1 is a flowchart of a method for user dialogue in multimodal interaction according to an embodiment of the present invention;

FIG. 2 is a flowchart of another method for user dialogue in multimodal interaction provided by an embodiment of the present invention;

FIG. 3 is a flowchart of another method for user dialogue in multimodal interaction provided by an embodiment of the present invention;

FIG. 4 is a flowchart of still another method for user dialogue in multimodal interaction provided by an embodiment of the present invention;

5 is a structural diagram of a user dialogue system in multimodal interaction provided in an embodiment of the present invention;

6 is a schematic diagram of the composition of a user dialogue system in multimodal interaction according to an embodiment of the present invention;

FIG. 7 is a schematic structural diagram of an electronic device provided by an embodiment of the present invention.

Detailed ways

In order to make the purposes, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments These are some embodiments of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

In an embodiment of the present invention, as shown in FIG. 1 , a method for user dialogue in multimodal interaction is disclosed, including the following steps:

Step S101, establishing a semantic correspondence.

In this step, a semantic correspondence is established, and the semantic correspondence is a one-to-one correspondence between a plurality of dialogue segments and a plurality of semantic information. The above-mentioned multiple dialogue segments are segments that the user may input.

Step S102, establishing a semantic instruction set.

In this step, a semantic instruction set is established, and the semantic instruction set refers to: establishing corresponding associations between a plurality of semantic information and a plurality of locally executed instructions, and a set formed by the associations.

Step S103, acquiring the current dialogue segment.

In this step, the current dialogue segment is acquired through an interactive device, which is an interactive device capable of inputting multimodal information. The plurality of semantic information includes a character field for receiving a retrieval character, a character field for invoking a map, a character field for invoking a system operation, and a character field for screen enlargement and screen reduction. The multiple local execution commands include: retrieving and displaying local data, retrieving and displaying local maps, displaying local dialog boxes, executing system calls, commands for zooming in on a local area of the screen, and commands for reducing images in a local area of the screen.

Step S104, obtaining current semantic information.

In this step, the semantic correspondence is invoked according to the current dialogue segment to obtain the current semantic information.

Step S105, obtaining the current local execution instruction.

In this step, the semantic instruction set is invoked according to the current semantic information to obtain the current local execution instruction.

Step S106, realizing interaction.

In this step, the current local execution instruction is executed and the execution result is displayed on the user dialog. The user dialog end is the user's local end.

In another preferred embodiment of a user dialogue method in multimodal interaction of the present invention, as shown in FIG. 2 , step S103 includes:

Step S1031a, acquiring voice information.

In this step, the current voice information and the current text information are acquired through the interactive device.

Step S1032a, performing text conversion.

In this step, the current voice information is converted into the current text field through voice recognition.

Step S1033a, determine the dialogue segment.

In this step, the current text information or text field is determined as the current dialogue segment.

In another preferred embodiment of a user dialogue method in multimodal interaction of the present invention, as shown in FIG. 3 , the interaction device includes a touch screen that can be used as an input device.

Step S103 includes:

Establish an event correspondence relationship. The event correspondence relationship is a one-to-one correspondence between multiple event operation information and multiple dialogue segments. Multiple event operation information is the user's single-tap, double-click, sliding along a single-line trajectory and along the touch screen. The information that can be obtained by the touch screen when the two-line track slides.

Step S1031b, receiving the current event operation.

In this step, the current event operation information is received through the interaction device, and the current event operation information is included in the event operation information.

Step S1032b, determining the dialogue segment corresponding to the event.

In this step, the current dialogue segment is obtained by invoking the event correspondence according to the current event operation information.

In another preferred embodiment of a user dialogue method in multimodal interaction of the present invention, as shown in FIG. 4 , the multiple local execution instructions include an execution instruction for calling local data and an execution instruction for calling remote data instruction.

Step S106 includes:

Step S1061, calling local data.

In this step, if the current local execution instruction is an execution instruction for calling local data, the current local data is called, the current local execution instruction is executed according to the current local data, and the execution result is displayed on the user dialog. and

Step S1062, calling remote data.

In this step, if the current local execution instruction is an execution instruction for calling remote data, a remote data link is established, the current remote data is called remotely, the current local execution instruction is executed according to the current remote data, and the execution result is displayed on the user dialog.

The above-mentioned S1062 also includes: if the remote data link cannot be established, periodically detecting the remote network signal. If a remote network signal can be detected, a remote data link is established.

In an embodiment of the present invention, as shown in FIG. 5 , a method for user dialogue in multimodal interaction is disclosed, including the following steps:

The full English name and Chinese definition of the English abbreviations appearing in the present invention are as follows:

1.Usr:user user.

2.Sys: system system.

3.AVS: Alexa Voice Service Amazon Smart Conversation Service.

4. Dueros: Baidu AI Open Platform.

step one:

When a user performs a multimodal operation, such as manually clicking on the screen to perform an operation, a local machine program is called to generate processable information, such as text.

Step 2:

The client scheduling module transmits the above text to the local semantic parsing module for offline processing to obtain semantic data that can be processed by the dialogue; processing refers to converting the valid value into semantic data that can be processed by the dialogue according to the relevant protocol (actual The above is a json format data) semantic data is json format data that satisfies a certain protocol.

Step 3:

The client-side scheduling module makes deep customization of the above semantic data, such as customizing the dialogue reply text, or executing commands locally.

Step 4:

The client scheduling module obtains the data returned by customization and executes it directly offline, and at the same time caches the data obtained in the above steps locally.

If necessary, the client can cycle through the above steps locally;

Step 5:

When you need to communicate with the cloud service, that is, when the local interaction cannot meet the user's needs, you need to use the cloud dialog. In order to ensure the consistency of the dialog, the cache of the previous rounds of dialog needs to be synchronized to the server.

The client will preferentially synchronize the above-mentioned data cached locally to the cloud dialogue service, and then conduct a normal voice dialogue. At this time, the cloud scheduling service will save the information of the entire dialogue process, although the previous dialogue has been separated from the cloud. . It can be synchronized in real time, or it can be set to synchronize in certain scenarios, such as: the network status is poor, the transmission fails, it can be cached locally, and then synchronized to the cloud.

In the normal voice interaction process, whether it is the first round or the second round of dialogue, if the client needs to perform multi-modal operations, such as the following usage scenarios:

usr:Voice input: "navigate to the central square of Suzhou".

sys: Navigate the interface, display the address list, and broadcast: "I found multiple addresses for you, please select", at the same time, the car enters the tunnel and other places where the network environment is unstable.

usr: Manually click on the screen "first" option.

sys: Display the first list, and broadcast "Navigate to Suzhou City Center Plaza for you, are you sure?" At the same time, the car switches to a place where the network environment is stable.

usr:Voice input: "OK".

sys: Navigate to Suzhou City Center Plaza for you.

As an embodiment of the present invention, the provided alternatives are:

Perform multi-modal operations locally, directly generate a "selection intent" locally, bypass the semantic parsing process, and then transmit the data to the cloud, where the cloud dialogue performs dialogue processing, and replies the data to the client for corresponding processing; select intent Refers to: according to the agreement, generate the relevant data stream. The advantage of the above solution is that the click response time is optimized to a certain extent. Cloud dialogue does not require additional compatibility processing, and reuses the process of cloud processing. But when it comes to network operations, when the network is unstable, the user experience is poor.

Another embodiment of the present invention is:

1. The client downloads the conversation data to the local.

2. The user performs a multi-modal operation, skips the semantic processing process, directly triggers a dialog intent, and hands it to the offline dialog for processing.

3. The client scheduling module obtains the offline dialogue processing data and performs corresponding operations.

The advantages of the above scheme are:

1. Under the existing technical conditions, it is immature to do offline voice-related process interaction. This solution can directly solve the defect of voice dialogue in the scenario of unstable network environment, and improve the user experience;

2. Provides solutions for multi-modal operations such as touch screen, gesture, and voice interaction to improve user experience.

Further advantages of this solution are:

1. It can support the needs of dialogue recovery, such as:

Usr: What's the weather like in Suzhou?

Sys: The weather in Suzhou is very good.

At this point, due to the dialogue between the user and the system, it has ended. The system will no longer turn on the microphone and wait for the user's voice input. After 3 minutes, the user goes on a business trip to Beijing and wakes up the system when he is ready to leave.

Usr: What about Beijing?

At this time, the client can cache the weather topics related to the previous rounds of conversations and synchronize them to the cloud service. At this time, the cloud service will have the information of the previous rounds of conversations, so that the conversation recovery function can be realized.

Sys: The weather in Beijing is cloudy.

At the same time, the present invention also provides a user dialogue system in multimodal interaction, as shown in Figure 6, including:

The semantic correspondence unit 101 is configured to establish a semantic correspondence as a one-to-one correspondence between a plurality of dialogue segments and a plurality of semantic information.

The semantic instruction set unit 201 is configured to establish a corresponding association between a plurality of semantic information and a plurality of locally executed instructions, a set formed by the association.

The current dialogue segment acquiring unit 301 is configured to acquire the current dialogue segment through an interactive device, which is an interactive device capable of inputting multimodal information.

The current semantic information obtaining unit 401 is configured to call the semantic correspondence in the semantic correspondence unit 101 according to the current dialogue segment to obtain the current semantic information.

The current local execution instruction acquiring unit 501 is configured to call the semantic instruction set in the semantic instruction set unit 201 according to the current semantic information to acquire the current local execution instruction.

The dialog unit 601 is configured to execute the current local execution instruction and display the execution result on the user dialog end. The user dialog end is the user's local end.

In a preferred embodiment of the present invention, the current dialogue segment obtaining unit 301 is further configured to obtain current voice information and current text information through an interactive device. Convert the current voice information into the current text field through speech recognition. Identify the current text message or text field as the current dialogue segment.

In a preferred embodiment of the present invention, the interaction device includes a touch screen that can be used as an input device.

The current dialogue segment acquiring unit 301 is further configured to: establish an event correspondence relationship, where the event correspondence relationship is a one-to-one correspondence between multiple event operation information and multiple dialogue segments, and multiple event operation information is for the user to perform a single point on the touch screen. The information that can be obtained by the touch screen when tapping, double-clicking, sliding along a single-line trajectory, and sliding along a double-line trajectory. The current event operation information is received through the interactive device, and the current event operation information is included in the event operation information. Call the event correspondence according to the current event operation information to obtain the current dialogue segment.

In a preferred embodiment of the present invention, the multiple local execution instructions include an execution instruction for calling local data and an execution instruction for calling remote data.

The dialog unit 601 is further configured to call the current local data if the current local execution instruction is an execution instruction calling local data, execute the current local execution instruction according to the current local data, and display the execution result on the user dialog. And if the current local execution instruction is an execution instruction for calling remote data, a remote data link is established, the current remote data is called from the remote, the current local execution instruction is executed according to the current remote data, and the execution result is displayed on the user dialog. .

In other embodiments, embodiments of the present invention further provide a non-volatile computer storage medium, where the computer storage medium stores computer-executable instructions, and the computer-executable instructions can execute the voice signal in any of the foregoing method embodiments methods of processing and use;

As an embodiment, the non-volatile computer storage medium of the present invention stores computer-executable instructions, and the computer-executable instructions are set to:

establishing a semantic correspondence, where the semantic correspondence is a one-to-one correspondence between multiple dialogue segments and multiple semantic information;

establishing a semantic instruction set, where the semantic instruction set refers to: establishing a corresponding association between the plurality of semantic information and a plurality of locally executed instructions, and a set formed by the association;

Obtain the current dialogue segment through an interactive device, the interactive device being an interactive device capable of inputting multimodal information;

Invoke the semantic correspondence according to the current dialogue segment to obtain the current semantic information;

Invoke the semantic instruction set according to the current semantic information to obtain the current local execution instruction;

The current local execution instruction is executed and the execution result is displayed on the user dialog end; the user dialog end is the user local end.

As a non-volatile computer-readable storage medium, it can be used to store non-volatile software programs, non-volatile computer-executable programs and modules, such as program instructions/program instructions corresponding to the voice signal processing method in the embodiments of the present invention. module. One or more program instructions are stored in a non-volatile computer-readable storage medium, and when executed by a processor, perform the speech signal processing method in any of the above method embodiments.

The non-volatile computer-readable storage medium may include a stored program area and a stored data area, wherein the stored program area may store an operating system and an application program required by at least one function; Use the data created by the dialog method in etc. In addition, the non-volatile computer-readable storage medium may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the non-transitory computer-readable storage medium may optionally include memory located remotely from the processor that can be connected to the user's method of dialogue in the multimodal interaction through a network. Examples of such networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

An embodiment of the present invention further provides a computer program product, the computer program product includes a computer program stored on a non-volatile computer-readable storage medium, the computer program includes program instructions, and when the program instructions are executed by a computer, the computer is made to execute the above Any speech signal processing method.

FIG. 7 is a schematic structural diagram of an electronic device provided by an embodiment of the present invention. As shown in FIG. 7 , the device includes: one or more processors 710 and a memory 720 . In FIG. 7 , one processor 710 is used as an example. The apparatus of the speech signal processing method may further include: an input device 730 and an output device 740 . The processor 710, the memory 720, the input device 730, and the output device 740 may be connected by a bus or in other ways, and the connection by a bus is taken as an example in FIG. 7 . The memory 720 is the aforementioned non-volatile computer-readable storage medium. The processor 710 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions, and modules stored in the memory 720, that is, to implement the user's dialogue method in the multimodal interaction of the above method embodiments. . The input device 730 may receive input numerical or character information, and generate key signal input related to user settings and function control of the information delivery device. The output device 740 may include a display device such as a display screen.

The above product can execute the method provided by the embodiment of the present invention, and has corresponding functional modules and beneficial effects for executing the method. For technical details not described in detail in this embodiment, reference may be made to the method provided by the embodiment of the present invention.

As an embodiment, the above electronic device can be applied to an intelligent voice dialogue platform, and includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a program that can be executed by the at least one processor. instructions that are executed by at least one processor to enable at least one processor to:

establishing a semantic correspondence, where the semantic correspondence is a one-to-one correspondence between multiple dialogue segments and multiple semantic information;

establishing a semantic instruction set, where the semantic instruction set refers to: establishing a corresponding association between the plurality of semantic information and a plurality of locally executed instructions, and a set formed by the association;

Obtain the current dialogue segment through an interactive device, the interactive device being an interactive device capable of inputting multimodal information;

Invoke the semantic correspondence according to the current dialogue segment to obtain the current semantic information;

Invoke the semantic instruction set according to the current semantic information to obtain the current local execution instruction;

The current local execution instruction is executed and the execution result is displayed on the user dialog end; the user dialog end is the user local end.

The electronic devices in the embodiments of the present application exist in various forms, including but not limited to:

(1) Mobile communication equipment: This type of equipment is characterized by having mobile communication functions, and its main goal is to provide voice and data communication. Such terminals include: smart phones (eg iPhone), multimedia phones, feature phones, and low-end phones.

(2) Ultra-mobile personal computer equipment: This type of equipment belongs to the category of personal computers, has computing and processing functions, and generally has the characteristics of mobile Internet access. Such terminals include: PDAs, MIDs, and UMPC devices, such as iPads.

(3) Portable entertainment equipment: This type of equipment can display and play multimedia content. Such devices include: audio and video players (eg iPod), handheld game consoles, e-books, as well as smart toys and portable car navigation devices.

(4) Server: A device that provides computing services. The composition of the server includes a processor, a hard disk, a memory, a system bus, etc. The server is similar to a general computer architecture, but due to the need to provide highly reliable services, the processing power, stability , reliability, security, scalability, manageability and other aspects of high requirements.

(5) Other electronic devices with data interaction function.

The device embodiments described above are only illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place , or distributed to multiple network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution in this embodiment. Those of ordinary skill in the art can understand and implement it without creative effort.

From the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and certainly can also be implemented by hardware. Based on this understanding, the above-mentioned technical solutions can be embodied in the form of software products in essence or the parts that make contributions to the prior art, and the computer software products can be stored in computer-readable storage media, such as ROM/RAM, magnetic Disks, optical discs, etc., include instructions for causing a computer device (which may be a personal computer, server, or network device, etc.) to perform the methods of various embodiments or portions of embodiments.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, but not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that it can still be The technical solutions described in the foregoing embodiments are modified, or some technical features thereof are equivalently replaced; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims (10)

1. The user's dialogue method in multimodal interaction, including: Step S101, establishing a semantic correspondence, where the semantic correspondence is a one-to-one correspondence between multiple dialogue segments and multiple semantic information; Step S102, establishing a semantic instruction set, where the semantic instruction set refers to a set consisting of the corresponding associations between the multiple semantic information and multiple local execution instructions; Step S103, obtaining the current dialogue segment through an interactive device, where the interactive device is an interactive device capable of inputting multimodal information; Step S104, calling the semantic correspondence according to the current dialogue segment to obtain current semantic information; Step S105, calling the semantic instruction set according to the current semantic information to obtain the current local execution instruction; Step S106: Execute the current local execution instruction and display the execution result on the user dialogue terminal; the user dialogue terminal is the user local terminal. 2. The dialogue method according to claim 1, wherein the step S103 comprises: Step S1031a, obtaining current voice information and current text information through the interactive device; Step S1032a, converting the current voice information into a current text field by voice recognition; Step S1033a, determining the current text information or the current text field as the current dialogue segment. 3. The dialogue method according to claim 1, wherein the interaction device comprises a touch screen capable of serving as an input device; The step S103 includes: Establish an event correspondence relationship, where the event correspondence relationship is a one-to-one correspondence between multiple event operation information and multiple dialogue segments, and the multiple event operation information is that the user performs a single tap or double click on the touch screen or The information that the touch screen can obtain when sliding along a single-line trajectory or sliding along a double-line trajectory; Step S1031b, receiving current event operation information through the interactive device, where the current event operation information is included in the event operation information; Step S1032b, calling the event correspondence according to the current event operation information to obtain the current dialogue segment. 4. The dialogue method according to claim 1, wherein the plurality of semantic information includes: a character field for receiving a retrieval character, a character field for calling a map, a character field for calling a system operation, a screen zooming in and a screen zooming out character field; The multiple local execution instructions include: retrieving and displaying local data, retrieving and displaying a local map, displaying a local dialog box, executing a system call, a command for zooming in an image of a partial area of the screen, and an instruction for reducing an image of a partial area of the screen. 5. The dialogue method according to claim 1, wherein the plurality of local execution instructions include an execution instruction for calling local data and an execution instruction for calling remote data; The step S106 includes: Step S1061, if the current local execution instruction is an execution instruction that calls local data, then call the current local data, execute the current local execution instruction according to the current local data and display the execution result on the user dialog; and Step S1062, if the current local execution instruction is an execution instruction that calls remote data, then establish a remote data link, call the current remote data from the remote, execute the current local execution instruction according to the current remote data and display the execution result in the User dialog. 6. The dialogue method according to claim 5, wherein the S1062 further comprises: If the remote data link cannot be established, the remote network signal is detected regularly; If a remote network signal can be detected, a remote data link is established. 7. User dialogue system in multimodal interaction, including: a semantic correspondence unit, configured to establish the semantic correspondence as a one-to-one correspondence between a plurality of dialogue segments and a plurality of semantic information; a semantic instruction set unit, which is configured to establish a corresponding association between the plurality of semantic information and a plurality of locally executed instructions, a set formed by the association; A current dialogue segment acquiring unit, configured to acquire the current dialogue segment through an interaction device, the interaction device being an interaction device capable of inputting multimodal information; A current semantic information obtaining unit, configured to call the semantic correspondence in the semantic correspondence unit according to the current dialogue segment, to obtain the current semantic information; A current local execution instruction acquisition unit, configured to call the semantic instruction set in the semantic instruction set unit according to the current semantic information, to acquire the current local execution instruction; A dialog unit, configured to execute the current local execution instruction and display the execution result on the user dialog end; the user dialog end is the user local end. 8. The dialogue system according to claim 7, wherein the current dialogue segment acquisition unit is further configured to: acquire current voice information and current text information through the interactive device; Converting to a current text field; determining the current text information or the current text field as the current dialogue segment. 9. The dialog system of claim 7, wherein the interaction device comprises a touch screen capable of serving as an input device; The current dialogue segment acquisition unit is further configured to: establish an event corresponding relationship, the event corresponding relationship is a one-to-one correspondence between multiple event operation information and multiple dialogue segments, and the multiple event operation information is the user's corresponding relationship. The information that the touch screen can obtain when the touch screen performs single-click tapping, double-clicking, sliding along a single-line trajectory, and sliding along a double-line trajectory; and receiving current event operation information through the interactive device, the current event operation information Included in the event operation information; calling the event correspondence according to the current event operation information to obtain the current dialogue segment. 10. The dialogue system according to claim 7, wherein the plurality of local execution instructions include an execution instruction for calling local data and an execution instruction for calling remote data; The dialogue unit is further configured to, if the current local execution instruction is an execution instruction that calls local data, call the current local data, execute the current local execution instruction according to the current local data, and display the execution result in the user dialogue. and if the current local execution instruction is an execution instruction that calls remote data, then establish a remote data link, call the current remote data from a remote location, execute the current local execution instruction according to the current remote data and display the execution result in the User dialog.