WO2024055299A1 - Real-time speech translation method, system, device, and storage medium - Google Patents

Abstract

The present invention provides a real-time speech translation method, system, and device, and a storage medium. The method comprises the steps of: performing speech recognition on first speech information inputted by a first user, and obtaining first text data; translating the first speech information, and obtaining second synthesized speech corresponding to a second language category; on the basis of the first text data, obtaining first synthesized speech corresponding to a first language category; and outputting the first synthesized speech to the first user, and synchronously outputting the second synthesized speech to a second user. In the present application, communication during cross-language exchange is smoother, and the communication efficiency of cross-language exchanging parties is increased.

Description

Real-time speech translation method, system, equipment and storage medium Technical field

The present invention relates to the technical field of online translation, and specifically to a real-time speech translation method, system, equipment and storage medium.

Background technique

With the increase in international exchanges, language communication in different languages is becoming more and more frequent. During the communication process, due to different languages, it is necessary to rely on the language skills of both parties or manual spoken translation to achieve communication, which brings inconvenience to both parties. Therefore, it will become the most convenient way to achieve real-time online translation by using translation devices or wearable devices with headphones and using robot voices.

During the call, the robot voice translator is responsible for translating the speaker's content into the listener's language for playback. However, the translation content may have certain errors with the speaker's original intention, and there will be a certain time difference between the spoken content and the real-time translated voice. The speaker does not understand the translated voice content, so the speaker cannot know in time that some of the content just said was heard by the listening party. The specific time and correctness of the information, that is, whether the correct one has been heard, makes the speaker unable to determine the timeliness and correctness of what he said. The speaker needs to pause consciously and wait for the speech translation to be completed before guessing the listener. Have you heard and understood correctly? Therefore, the efficiency and accuracy of communication between both parties cannot be guaranteed.

Contents of the invention

In view of the problems in the prior art, the purpose of the present invention is to provide a real-time speech translation method, system, equipment and storage medium, which improves the accuracy of cross-language translation and the communication efficiency of both parties in cross-language communication.

To achieve the above objectives, the present invention provides a real-time speech translation method, which method includes the following steps:

Perform speech recognition on the first voice information input by the first user to obtain first text data;

Translate the first voice information to obtain a second synthesized voice corresponding to the second language category;

Based on the first text data, obtain a first synthesized speech corresponding to the first language category;

The first synthesized voice is output to the first user, and the second synthesized voice is outputted to the second user simultaneously.

Optionally, the translating the first speech information to obtain a second synthesized speech corresponding to the second language category includes:

Translate the first text data obtained based on speech recognition of the first voice information to obtain second text data corresponding to the second language category;

Based on the second text data, a second synthesized speech corresponding to the second language category is obtained.

Optionally, obtaining the first synthesized speech corresponding to the first language category based on the first text data includes:

Calculate the estimated time required to complete the playback of the second synthesized speech to obtain the first estimated duration;

Based on the first estimated duration, obtain second timestamp information about the second synthesized speech;

Perform speech synthesis based on the first text data to obtain a first synthesized speech; and synchronize the first timestamp corresponding to the first synthesized speech with the second timestamp information.

Optionally, the method also includes:

After receiving the positive feedback information of the first synthesized voice from the first user, continue to collect the next first voice message input by the first user;

After receiving the negative feedback information of the first synthesized voice from the first user, collecting the first voice information re-inputted by the first user, and outputting the information re-inputted by the first user to the second user. Prompt information.

Optionally, the method also includes:

Extract respective interest tags and corresponding timestamp information based on the first text data and the second text data respectively;

According to the timestamp information, corresponding interest tags are marked on the first text data and the second text data respectively.

Optionally, before outputting the first synthesized voice to the first user and synchronously outputting the second synthesized voice to the second user, the method includes:

Collect the voiceprint information of the first user and the second user respectively;

Based on the voiceprint information, identify the identity information of the first user and the second user respectively;

Based on the identity information of the first user and the second user, determine the type of relationship between the first user and the second user;

Based on the relationship type between the first user and the second user and the preset identity relationship interest library, filter the interest tags and use the retained interest tags as the target interest tags;

Outputting the first synthesized voice to the first user and synchronously outputting the second synthesized voice to the second user include:

Provide voice prompts to the voice associated with the target interest tag among the first synthesized voice and the second synthesized voice.

Optionally, performing speech recognition on the first voice information input by the first user to obtain the first text data includes:

Obtain previous data about the first voice information;

Perform speech recognition on the first voice information input by the first user to obtain initial text data;

The initial text data is modified using the preceding data to obtain first text data.

Optionally, before outputting the first synthesized voice to the first user and synchronously outputting the second synthesized voice to the second user, the method includes:

When it is detected that a pair of earphones for outputting voice are worn by different users, the earphones are controlled to work in the first state; when it is detected that the first user and the second user each wear a pair of earphones, the earphones are controlled to work in the second state;

Among them, in the first state, the two earphones output different voices respectively, and one earphone worn by a user serves as the user's voice output channel; in the second state, two of the pair of earphones output the same voice, A pair of headphones worn by a user serves as the user's voice output channel.

Optionally, the pair of headphones is provided with a UWB communication module; before outputting the first synthesized voice to the first user and synchronously outputting the second synthesized voice to the second user, the method include:

The pair of headphones is based on a UWB communication module and detects the distance between the two headphones in real time;

When the distance between the two earphones is greater than the first preset threshold, the earphones are controlled to work in the first state; when the distance between the two earphones is less than the first preset threshold, the earphones are controlled to work in the second state. state.

Optionally, the method also includes:

Combine and form an audio adjustment library based on the unplayed audio frames in the first synthesized speech and the audio frames that have collected and generated the first speech information and have not been converted into the first synthesized speech;

Based on the number of audio frames in the audio adjustment library, adjust the playback speed of the first synthesized speech;

Wherein, the playback speech speed corresponding to the first synthesized speech increases as the number of audio frames in the audio adjustment library increases, and decreases as the number of audio frames in the audio adjustment library decreases. Small.

The present invention also provides a real-time speech translation system for realizing the above-mentioned real-time speech translation method. The system includes:

The first text data generation module performs speech recognition on the first voice information input by the first user to obtain the first text data;

A second synthesized speech generation module translates the first speech information to obtain a second synthesized speech corresponding to the second language category;

A first synthesized speech generation module, based on the first text data, obtains a first synthesized speech corresponding to the first language category;

The synthesized voice playing module outputs the first synthesized voice to the first user, and simultaneously outputs the second synthesized voice to the second user.

The invention also provides a real-time speech translation device, including:

processor;

A memory in which an executable program of the processor is stored;

Wherein, the processor is configured to execute the steps of any one of the above real-time speech translation methods by executing the executable program.

The present invention also provides a computer-readable storage medium for storing a program that implements any of the above steps of the real-time speech translation method when the program is executed by a processor.

Compared with the existing technology, the present invention has the following advantages and outstanding effects:

The real-time speech translation method, system, equipment and storage medium provided by the present invention synthesize the first synthesized speech based on the first text data obtained by speech recognition based on the speaking content of the speaker, and while playing the second synthesized speech to the listening party, The first synthesized speech is played synchronously to the speaker, so that errors in the translation content caused by errors in the speech recognition process can be known to the speaker in time, improving the accuracy of cross-language translation and the communication efficiency of both parties in cross-language communication; making Communicate more smoothly in cross-language communication.

Description of drawings

Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of the non-limiting embodiments with reference to the following drawings.

Figure 1 is a schematic diagram of a real-time speech translation method disclosed in an embodiment of the present invention;

Figure 2 is a schematic diagram of an application scenario involved in the real-time speech translation method disclosed in an embodiment of the present invention;

Figure 3 is a schematic diagram of another application scenario involved in the real-time speech translation method disclosed in an embodiment of the present invention;

Figure 4 is a schematic diagram of another application scenario involved in the real-time speech translation method disclosed in an embodiment of the present invention;

Figure 5 is a schematic diagram of a real-time speech translation method disclosed in another embodiment of the present invention;

Figure 6 is a schematic diagram of the synchronous working principle of a real-time speech translation method disclosed in an embodiment of the present invention;

Figure 7 is a schematic diagram of a real-time speech translation method disclosed in another embodiment of the present invention;

Figure 8 is a schematic diagram of a real-time speech translation method disclosed in another embodiment of the present invention;

Figure 9 is a schematic diagram of a real-time speech translation method disclosed in another embodiment of the present invention;

Figure 10 is a schematic structural diagram of a real-time speech translation system disclosed in an embodiment of the present invention;

Figure 11 is a schematic structural diagram of a real-time speech translation device disclosed in an embodiment of the present invention. .

Detailed ways

Example embodiments will now be described more fully with reference to the accompanying drawings. Example embodiments may, however, be embodied in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of example embodiments to those skilled in the art. The same reference numerals in the drawings represent the same or similar structures, and thus their repeated description will be omitted.

As shown in Figure 1, one embodiment of the present invention discloses a real-time speech translation method. This embodiment can be applied to the situation where two parties perform speech recognition and translation in real-time across languages. This method can be performed by online real-time speech translation. To execute, the device can be implemented by hardware and/or software, and the device can be configured in any terminal or network element. Through online real-time voice translation, translation services can be provided in real time for both users who speak different languages, allowing both parties to communicate smoothly. The method specifically includes the following steps:

S110: Perform speech recognition on the first voice information input by the first user to obtain first text data.

Referring to Figure 2, in the above operation, the first user 21 and the second user 22 conduct cross-language language communication. The first user 21 can use the first voice based on the first language to input the first voice information, and the online voice translation device can Perform speech recognition on it to obtain the first text data. Collecting the voice uttered by the first user 21 can be implemented through the headphones worn by the first user 21 .

This embodiment uses the first user 21 as the speaker at the beginning of the cross-language communication, and the second user 22 as the listener at the beginning of the communication. Therefore, at the beginning of the communication, this step collects the input of the first user 21 first voice information, and then recognize the user's voice information.

Since communication is two-way, during subsequent continuous communication, this step will also collect the second voice information input by the second user 22, and then recognize the second voice information. The above-mentioned online real-time speech translation device may be the translation device 23 in FIG. 2 or a server. This application does not limit this.

On the other hand, when the online real-time speech translation device is a translation device, it may not be limited to one translation device. For example, referring to FIG. 3 , the first user 21 and the second user 22 can each use a translation device. For example, the first user 21 uses the first translation device 24 and the second user 22 uses the second translation device 25 . Data transmission and communication are performed between the first translation device 24 and the second translation device 25. Text translation between different languages can be completed by the first translation device 24 or the second translation device 25. The first translation device 24 and the second translation device 25 are also used to complete speech synthesis for respective corresponding users.

It should be noted that the above-mentioned Figures 2 and 3 show face-to-face real-time translation scenarios. This embodiment can also be applied to long-distance online translation scenarios. Figure 4 shows a schematic diagram of a long-distance online translation scenario. In this illustration, the first user 21 and the second user 22 respectively perform online translation based on laptop computers.

S120: Translate the above-mentioned first speech information to obtain a second synthesized speech corresponding to the second language category. Specifically, online speech translation generally involves two links: the first link is speech recognition, that is, recognizing the first language speech input by the first user as text information; the second link is translating the text information based on the translation corpus , and then generate voice information or text information in the second language and provide it to the second user. Therefore, in some embodiments, step S120 includes:

S121. Translate the above-mentioned first text data to obtain second text data corresponding to the second language category.

S122. Based on the above-mentioned second text data, obtain a second synthesized speech corresponding to the second language category. The above steps are to translate the first text data in the first language into the second text data in the second language, and then perform speech synthesis based on the second text data.

S130: Based on the above-mentioned first text data, obtain the first synthesized speech corresponding to the first language category. Among them, the above-mentioned first synthesized speech is used to allow the first user who is the speaker at this time to know the playback progress of the translated speech through the user's robot, and to promptly detect errors in the speech recognition process, so that the speaker can Accurately obtain the correctness of the translated content and the time it was conveyed to the other party, so that communication with different languages can correct errors in time and improve interactivity.

In order to keep the speaking progress of the content played by the speaking robot and the listening robot, the timestamps of the first synthesized voice and the second synthesized voice need to be synchronized. Thus, in some embodiments, referring to Figure 5, step S130 includes:

S131. Calculate the estimated time required to complete the playback of the second synthesized voice, and obtain the first estimated duration.

S132: Obtain second timestamp information about the second synthesized speech based on the first estimated duration. as well as

S133. Perform speech synthesis based on the first text data to obtain a first synthesized speech; and synchronize the first timestamp corresponding to the first synthesized speech with the second timestamp information.

This embodiment uses the above steps to estimate the time required for speech synthesis of the second language, and performs timestamp synchronization based on this. During specific implementation, for example, each of the two synthesized voices can be divided into multiple audio paragraphs, and then the audio paragraph of the first synthesized voice and the audio paragraph corresponding to the second synthesized voice are timestamp synchronized, thereby realizing two Timestamp synchronization of synthesized speech.

In this way, this application can establish a synchronization mechanism between the synthesized speech of the own language content and the synthesized speech of the other party's language content, so that the playback content of the two parties can maintain the same progress.

S140: Output the first synthesized voice to the first user, and simultaneously output the second synthesized voice to the second user.

This embodiment plays two synthesized voices synchronously, which is beneficial to ensuring that the playback content maintains the same progress, allowing the speaker to accurately obtain the correctness of the translated content and the time it was conveyed to the other party, thereby allowing communication in different languages to correct errors in a timely manner and improve interactivity.

Moreover, during synchronous playback, the speaker can continue to speak, or can temporarily stop speaking and listen to the first synthesized speech. If the speaker continues to speak, then the playback volume of the first synthesized speech is set to be no higher than the fourth preset threshold of the speaker's pronunciation volume. This will not interfere with the speaker's normal expression and allow the speaker to hear it through the earphones. What is said.

Since real-time online translation is divided into two scenarios: on-site face-to-face and off-site long-distance translation, during specific implementation, in the on-site face-to-face translation scenario, two users can each wear one of a pair of headphones, that is, share a pair of headphones. You can also wear a pair of headphones separately. When each wears one earphone, the first synthesized voice can be output to the first user through the left channel earphone, and the second synthesized voice can be simultaneously output to the second user through the right channel earphone. In the off-site long-distance translation scenario, two users each wear a pair of headphones.

Figure 6 is a schematic diagram of the principle of synchronously playing synthesized speech. Referring to Figure 6, user A is the first user, and the voice person of A refers to the robot used to play the first synthesized voice to user A. User B is the second user, and B's voice person refers to the robot used to play the second synthesized voice to user B.

Sx (including S1, S2 and S3) is the original voice of the first user. S1, S2 and S3 can respectively represent the first, second and third pieces of speech sent by user A. Sx’ is the machine speech synthesized by the speaker based on the content of Sx recognized as the first language, that is, the synthesized speech after recognition. S1', S2' and S3' respectively represent the synthetic voice of the first language corresponding to the first segment of voice S1, the synthetic voice of the first language corresponding to the second segment of voice S2 and the synthetic voice of the third segment played to user A respectively. The first language synthetic speech corresponding to speech S3.

TSx’ is B’s voice, which is the synthesized machine voice after the person translates it into the text of the second language based on the Sx recognition content, that is, the translated voice. Then, TS1', TS2' and TS3' respectively represent the second language synthetic speech played to user B and respectively corresponding to the first speech S1, the second language synthetic speech corresponding to the second speech S2 and the second language synthetic speech corresponding to the second speech S2. The second language synthetic speech corresponding to the three-segment speech S3.

In another embodiment of the present application, another real-time speech translation method is disclosed. Referring to Figure 7, based on the above-mentioned corresponding embodiment of Figure 1, the method also includes steps:

S150: Determine whether positive feedback information from the first user on the first synthesized voice is received. If yes, step S160 is executed. Otherwise, execute step S170.

S160: After receiving the positive feedback information of the first synthesized voice from the first user, continue to collect the next first voice message input by the first user.

S170: Collect the first voice information re-inputted by the first user, and output prompt information about the re-input by the first user to the second user.

Specifically, that is, after the first user confirms that the translation is accurate, the next segment of the first voice message input by the first user continues to be collected. When the first user confirms the translation error, the first user is prompted to re-express the content and re-collect it, and the second user is informed that the second synthesized speech just played has an error and will be played again.

Regarding the implementation method of determining whether the translation is accurate, in other embodiments, the translation can also be determined by not receiving the preset utterance of the first user within a preset time period after the first synthesized voice completes playing. precise. Otherwise, it is determined that there is an error in the translation. This allows communication with different languages to correct errors in a timely manner and improve interactivity.

In some embodiments, based on the above-mentioned corresponding embodiment of Figure 1, step S110 includes:

Perform voiceprint detection on the first voice information, perform speech recognition only on the corresponding audio paragraphs uttered by the first user, and obtain the first text data. That is, speech recognition is not performed for the corresponding audio paragraphs that do not belong to the first user's utterance.

Step S120 includes: performing voiceprint detection on the first voice information, and translating only the corresponding audio paragraphs uttered by the first user to obtain a second synthesized voice. That is, during the translation process, when an audio segment that does not belong to the first user's utterance is detected, the translation is paused. When an audio segment belonging to the first user's utterance is detected, translation is continued until the translation is completed.

This helps ensure the accuracy of real-time translation during cross-language communication.

In some embodiments, based on the corresponding embodiment of Figure 1 above, before step S140, the method further includes:

The environmental voice data of the first user and the second user are respectively collected in real time.

Perform voiceprint detection on environmental voice data.

When it is detected that other people's voiceprint information corresponds to the first user's ambient voice data, first preset prompt information is sent to the first user. The first preset prompt message is used to remind the first user by voice that there are outsiders around and to pay attention to protecting privacy.

When it is detected that the voiceprint information of another person exists in the environmental voice data of the second user, the translation of the first voice information is suspended, and the second preset prompt information is sent to the second user.

Until the second user side detects no other person's voiceprint information, continue to translate the first voice information.

This is beneficial to protecting the privacy of the first user and the second user. For example, the above steps may be located between step S130 and step S140.

In some embodiments, based on the above-mentioned corresponding embodiment of Figure 1, before step S140, the method further includes the steps:

Based on the above-mentioned first text data and the above-mentioned second text data, respective interest tags and corresponding timestamp information are extracted.

According to the above timestamp information, corresponding interest tags are marked on the above first text data and the above second text data respectively.

For example, the above steps may be located between step S130 and step S140.

Step S140 includes: when playing the first synthesized voice and the second synthesized voice, providing a voice prompt for the voice associated with the interest tag.

Among them, the above-mentioned interest tags and timestamps have a one-to-one correspondence. Using this interest tag can facilitate users to quickly find key content when listening back to the voice. The above interest tags can be displayed in the form of text or voice.

This application actively activates the interest tag annotation function to record the time and corresponding content for repeatedly mentioned content or speech tone or semantic analysis content, sensitive privacy, or when the listener is temporarily unable to concentrate on listening but wants to review the content later. Interest tags and storage. When the user looks back at the interest tags, the points of interest of the call translation can be displayed, and the user can jump to view the content before and after the point of interest, or re-listen to the conversation history before and after the point of interest to improve the user experience. You can also rely on interest tags to make the two communicating parties pay attention to key content to avoid being misunderstood or ignored, which will help improve the cross-language communication experience.

In some embodiments, based on the above embodiments, before step S140, the method further includes the steps:

The voiceprint information of the first user and the second user is collected respectively.

Based on the voiceprint information, the identity information of the first user and the second user is respectively identified.

Based on the identity information of the first user and the second user, a relationship type between the first user and the second user is determined.

Based on the relationship type between the first user and the second user and the preset identity relationship interest library, the above-extracted interest tags are filtered, and the retained ones are used as target interest tags.

For example, the above steps may be located between step S130 and step S140.

Step S140 includes: when playing the first synthesized voice and the second synthesized voice, providing a voice prompt for the voice associated with the target interest tag.

The determination of the relationship type between the first user and the second user can be based on a preset identity relationship database.

In some embodiments, based on the above-mentioned corresponding embodiment of Figure 1, step S110 includes:

S111. Obtain previous data about the first voice information.

S112: Perform speech recognition on the first voice information input by the first user to obtain initial text data.

S113, use the previous data to modify the initial text data to obtain the first text data.

Specifically, obtaining other data uttered by the user before uttering the first voice message, and correcting the recognized text based on these data will help improve the accuracy of translation, thereby improving the smoothness of cross-language communication. For example, if the text data translates the same speech into a first word, and this step translates the speech into a second word, then the second word is corrected into the first word.

In some embodiments, based on the above-mentioned corresponding embodiment of Figure 1, with reference to Figure 8, before step S140, the steps are also included:

S180, when it is detected that a pair of earphones for outputting voice are worn by different users respectively, control the above-mentioned earphones to work in the first state; when it is detected that the first user and the second user each wear a pair of earphones, control the above-mentioned earphones to work in the second state.

In the first state, the two earphones output different voices respectively, and the earphone worn by the user serves as the user's voice output channel. In the second state, two of the pair of earphones output the same voice, and the pair of earphones worn by the user serves as the user's voice output channel.

For example, the above steps may be located between step S130 and step S140.

Step S140 is replaced with step S141: outputting the first synthesized voice to the first user based on the earphone worn by the first user, and simultaneously outputting the second synthesized voice to the second user based on the earphone worn by the second user.

Specifically, for example, when two earphones in a pair of earphones are worn by two users respectively, that is, the first user wears one and the second user wears the other, then the pair of earphones can be controlled to work in a face-to-face translation state. When each user wears a pair of headphones, the pair of headphones is controlled to work in a long-distance translation state. In the face-to-face translation state, the user's synthesized speech needs to be encoded into the audio channel corresponding to the headset worn by the user. For example, when the first user wears a left-channel earphone in a pair of earphones, the first synthesized speech is encoded as left-channel audio. Accordingly, the second synthesized speech is encoded as right channel audio.

The earphone worn by the first user is used as the first voice output channel, and the earphone worn by the second user is used as the second voice output channel. The first synthesized voice is output to the first user based on the first voice output channel, and the second synthesized voice is simultaneously output to the second user based on the second voice output channel.

In some embodiments, based on the above embodiments, the above-mentioned earphones are provided with a UWB (Ultra Wide Band, ultra-wideband wireless carrier communication) communication module. Step S180 includes:

The above pair of headphones are based on the UWB communication module and detect the distance between the two headphones in real time.

When the distance between the two earphones is greater than the first preset threshold, the earphones are controlled to operate in the first state; when the distance between the two earphones is less than the first preset threshold, the earphones are controlled to operate in the second state.

Specifically, when the distance between the two earphones is greater than the first preset threshold, it means that the two earphones in a pair of earphones are respectively worn by two users. When the distance between the two earphones is less than the first preset threshold, it indicates that one pair of earphones is worn by the same user.

For example, the above-mentioned first preset threshold may be 25cm. In other embodiments, the distance between the earphones can also be detected through the built-in ultrasonic signal module in the earphones, which is not limited in this application.

In some embodiments, based on the above-mentioned corresponding embodiment of Figure 1, with reference to Figure 9, the method includes the steps:

S190: Combine and form an audio adjustment library based on the unplayed audio frames in the first synthesized speech and the audio frames that have collected and generated the first speech information and have not been converted into the first synthesized speech.

S200: Adjust the playback speed of the first synthesized speech based on the number of audio frames in the audio adjustment library. Wherein, the playback speech speed corresponding to the first synthesized speech increases as the number of audio frames in the audio adjustment library increases, and decreases as the number of audio frames in the audio adjustment library decreases. In some embodiments, the playback speed of the second synthesized voice can also be adjusted synchronously.

This can avoid, for example, when the user has spoken to the S6th audio paragraph, the headset is still playing back the S3th audio paragraph, which will not be conducive to the speaker's timely discovery of errors in the speech recognition process, thereby allowing communication methods with different languages to Able to correct errors in a timely manner.

In some embodiments, based on the above-mentioned corresponding embodiment of Figure 1, before the above-mentioned step S110, the method further includes:

When it is detected that the first user is wearing only one pair of headphones, and the distance between the two headphones in the pair is greater than the first preset threshold, the environmental sound data of the environment where the first user is located is collected, and the environmental sound data is collected from the above-mentioned environmental sound data. Multiple voice data are extracted from the data.

The language category corresponding to the plurality of voice data is determined as the second language category.

Prompt information is played to the external environment based on the other pair of earphones; the above prompt information is used to prompt the second user to wear the other pair of earphones.

The above step S140 includes:

After detecting that the second user is wearing the second earphone, the second synthesized voice is output to the second user through the second earphone.

This can facilitate the rapid establishment of communication with the second user when the first user is in an unfamiliar language environment, such as when traveling to another country and needs face-to-face translation with the second user, which will help improve the smoothness of cross-language communication. Spend.

It should be noted that all the above-mentioned embodiments disclosed in this application can be freely combined, and the technical solution obtained after the combination is also within the protection scope of this application.

As shown in Figure 10, an embodiment of the present invention also discloses a real-time speech translation system 10. The system includes:

The first text data generating module 101 performs speech recognition on the first voice information input by the first user to obtain the first text data.

The second synthesized speech generation module 102 translates the first speech information to obtain a second synthesized speech corresponding to the second language category.

The first synthesized speech generation module 103 obtains a first synthesized speech corresponding to the first language category based on the first text data.

The synthesized speech playing module 104 outputs the first synthesized speech to the first user, and simultaneously outputs the second synthesized speech to the second user.

It can be understood that the real-time speech translation system of the present invention also includes other existing functional modules that support the operation of the real-time speech translation system. The real-time speech translation system shown in Figure 10 is only an example and should not impose any restrictions on the functions and scope of use of the embodiments of the present invention.

The real-time speech translation system in this embodiment is used to implement the above-mentioned real-time speech translation method. Therefore, for the specific implementation steps of the real-time speech translation system, reference can be made to the above description of the real-time speech translation method, which will not be described again here.

An embodiment of the present invention also discloses a real-time speech translation device, which includes a processor and a memory, wherein the memory stores an executable program of the processor; the processor is configured to execute the above real-time speech translation method by executing the executable program. steps in. Figure 11 is a schematic structural diagram of the real-time speech translation device disclosed in the present invention. An electronic device 600 according to this embodiment of the present invention is described below with reference to FIG. 11 . The electronic device 600 shown in FIG. 11 is only an example and should not impose any limitations on the functions and usage scope of the embodiments of the present invention.

As shown in Figure 11, electronic device 600 is embodied in the form of a general computing device. The components of the electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different platform components (including the storage unit 620 and the processing unit 610), a display unit 640, and the like.

Among them, the storage unit stores program code, and the program code can be executed by the processing unit 610, so that the processing unit 610 performs the steps according to various exemplary embodiments of the present invention described in the above-mentioned real-time speech translation method part of this specification. For example, processing unit 610 may perform steps as shown in FIG. 1 .

The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 6201 and/or a cache storage unit 6202, and may further include a read-only storage unit (ROM) 6203.

Storage unit 620 may also include a program/utility 6204 having a set of (at least one) program modules 6205 including, but not limited to: an operating system, one or more application programs, other program modules, and program data, Each of these examples, or some combination, may include the implementation of a network environment.

Bus 630 may be a local area representing one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, a graphics acceleration port, a processing unit, or using any of a variety of bus structures. bus.

Electronic device 600 may also communicate with one or more external devices 700 (e.g., keyboard, pointing device, Bluetooth device, etc.), may also communicate with one or more devices that enable a user to interact with electronic device 600, and/or with Any device (eg, router, modem, etc.) that enables the electronic device 600 to communicate with one or more other computing devices. This communication may occur through input/output (I/O) interface 650. Furthermore, the electronic device 600 may also communicate with one or more networks (eg, a local area network (LAN), a wide area network (WAN), and/or a public network, such as the Internet) through the network adapter 660. Network adapter 660 may communicate with other modules of electronic device 600 via bus 630. It should be understood that, although not shown in the figures, other hardware and/or software modules may be used in conjunction with electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives And data backup storage platform, etc.

The invention also discloses a computer-readable storage medium for storing a program. When the program is executed, the steps in the above real-time speech translation method are implemented. In some possible implementations, various aspects of the present invention can also be implemented in the form of a program product, which includes program code. When the program product is run on a terminal device, the program code is used to cause the terminal device to execute the above described instructions. The steps according to various exemplary embodiments of the present invention are described in the real-time speech translation method.

As shown above, when the program of the computer-readable storage medium of this embodiment is executed, the speaking content of the speaking party is synthesized based on the first text data obtained through speech recognition, and the second synthesized speech is played to the listening party. At the same time as the voice is being spoken, the first synthesized voice is simultaneously played to the speaker, so that errors in the translation content caused by errors in the speech recognition process can be known to the speaker in a timely manner, improving the accuracy of cross-language translation and improving the cross-language communication between both parties. Communication efficiency.

An embodiment of the invention discloses a computer-readable storage medium. The storage medium is a program product that implements the above method, which can be a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, a readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, apparatus or device.

The Program Product may take the form of one or more readable media in any combination. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: electrical connection with one or more conductors, portable disk, hard disk, random access memory (RAM), read only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above.

A computer-readable storage medium may include a data signal propagated in baseband or as part of a carrier wave carrying the readable program code therein. Such propagated data signals may take many forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A readable storage medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transport the program for use by or in connection with an instruction execution system, apparatus, or device. Program code contained on a readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above.

Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, etc., as well as conventional procedural programming. Language—such as "C" or a similar programming language. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server execute on. In situations involving remote computing devices, the remote computing device may be connected to the user computing device through any kind of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device, such as provided by an Internet service. (business comes via Internet connection).

The real-time speech translation method, system, device and storage medium provided by the embodiments of the present invention synthesize the first synthesized speech based on the first text data obtained by speech recognition based on the speaking content of the speaker, and play the second synthesized speech to the listening party. At the same time, the first synthesized voice is played synchronously to the speaker, so that errors in the translation content caused by errors in the speech recognition process can be known to the speaker in time, improving the accuracy of cross-language translation and the communication efficiency of both parties in cross-language communication. ; Makes communication smoother in cross-language communication.

The above content is a further detailed description of the present invention in combination with specific preferred embodiments, and it cannot be concluded that the specific implementation of the present invention is limited to these descriptions. For those of ordinary skill in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, and all of them should be regarded as belonging to the protection scope of the present invention.

Claims (13)

A real-time speech translation method, characterized by including the following steps: Perform speech recognition on the first voice information input by the first user to obtain first text data; Translate the first voice information to obtain a second synthesized voice corresponding to the second language category; Based on the first text data, obtain a first synthesized speech corresponding to the first language category; The first synthesized voice is output to the first user, and the second synthesized voice is outputted to the second user simultaneously. The real-time speech translation method according to claim 1, wherein said translating the first speech information to obtain a second synthesized speech corresponding to the second language category includes: Translate the first text data obtained based on speech recognition of the first voice information to obtain second text data corresponding to the second language category; Based on the second text data, a second synthesized speech corresponding to the second language category is obtained. The real-time speech translation method according to claim 2, characterized in that, based on the first text data, obtaining the first synthesized speech corresponding to the first language category includes: Calculate the estimated time required to complete the playback of the second synthesized speech to obtain the first estimated duration; Based on the first estimated duration, obtain second timestamp information about the second synthesized speech; Perform speech synthesis based on the first text data to obtain a first synthesized speech; and synchronize the first timestamp corresponding to the first synthesized speech with the second timestamp information. The real-time speech translation method according to claim 1, characterized in that the method further includes: After receiving the positive feedback information of the first synthesized voice from the first user, continue to collect the next first voice message input by the first user; After receiving the negative feedback information of the first synthesized voice from the first user, collecting the first voice information re-inputted by the first user, and outputting the information re-inputted by the first user to the second user. Prompt information. The real-time speech translation method according to claim 2, characterized in that the method further includes: Extract respective interest tags and corresponding timestamp information based on the first text data and the second text data respectively; According to the timestamp information, corresponding interest tags are marked on the first text data and the second text data respectively. The real-time speech translation method according to claim 5, characterized in that, before outputting the first synthesized speech to the first user and synchronously outputting the second synthesized speech to the second user, the Methods include: Collect the voiceprint information of the first user and the second user respectively; Based on the voiceprint information, identify the identity information of the first user and the second user respectively; Determine the relationship type between the first user and the second user based on the identity information of the first user and the second user; Based on the relationship type between the first user and the second user and the preset identity relationship interest library, filter the interest tags and use the retained interest tags as the target interest tags; Outputting the first synthesized voice to the first user and synchronously outputting the second synthesized voice to the second user include: Provide voice prompts to the voice associated with the target interest tag among the first synthesized voice and the second synthesized voice. The real-time speech translation method according to claim 1, wherein performing speech recognition on the first speech information input by the first user to obtain the first text data includes: Obtain previous data about the first voice information; Perform speech recognition on the first voice information input by the first user to obtain initial text data; The initial text data is modified using the preceding data to obtain first text data. The real-time speech translation method according to claim 1, characterized in that, before outputting the first synthesized speech to the first user and synchronously outputting the second synthesized speech to the second user, the Methods include: When it is detected that a pair of earphones for outputting voice are worn by different users, the earphones are controlled to work in the first state; when it is detected that the first user and the second user each wear a pair of earphones, the earphones are controlled to work in the second state; Among them, in the first state, the two earphones output different voices respectively, and one earphone worn by a user serves as the user's voice output channel; in the second state, two of the pair of earphones output the same voice, A pair of headphones worn by a user serves as the user's voice output channel. The real-time speech translation method of claim 8, wherein the pair of earphones is provided with a UWB communication module; during the step of outputting the first synthesized speech to the first user, and synchronizing the second Before outputting the synthesized speech to the second user, the method includes: The pair of headphones is based on a UWB communication module and detects the distance between the two headphones in real time; When the distance between the two earphones is greater than the first preset threshold, the earphones are controlled to work in the first state; when the distance between the two earphones is less than the first preset threshold, the earphones are controlled to work in the second state. state. The real-time speech translation method according to claim 1, characterized in that the method further includes: Combine and form an audio adjustment library based on the unplayed audio frames in the first synthesized speech and the audio frames that have collected and generated the first speech information and have not been converted into the first synthesized speech; Based on the number of audio frames in the audio adjustment library, adjust the playback speed of the first synthesized speech; Wherein, the playback speech speed corresponding to the first synthesized speech increases as the number of audio frames in the audio adjustment library increases, and decreases as the number of audio frames in the audio adjustment library decreases. Small. A real-time speech translation system, used to implement the real-time speech translation method as claimed in claim 1, characterized in that the system includes: The first text data generation module performs speech recognition on the first voice information input by the first user to obtain the first text data; A second synthesized speech generation module translates the first speech information to obtain a second synthesized speech corresponding to the second language category; A first synthesized speech generation module, based on the first text data, obtains a first synthesized speech corresponding to the first language category; The synthesized speech playing module outputs the first synthesized speech to the first user, and simultaneously outputs the second synthesized speech to the second user. A real-time voice translation device, characterized by including: processor; A memory in which an executable program of the processor is stored; Wherein, the processor is configured to execute the steps of the real-time speech translation method in any one of claims 1 to 10 by executing the executable program. A computer-readable storage medium used to store a program, characterized in that when the program is executed by a processor, the steps of the real-time speech translation method in any one of claims 1 to 10 are implemented.

Images