CN105898556A - Plug-in subtitle automatic synchronization method and device - Google Patents

Abstract

The invention relates to the technical field of video playback, and discloses an automatic synchronization method and device for external subtitles. In the present invention, the audio part of the video file is first extracted, and the audio part is decoded to obtain pulse code modulation data; then the pulse code modulation data is divided into audio segments, and the audio segments are classified, and then classified into voice audio Divide the segment into short sentences, and determine the start time and end time of the short sentences; search for a match in the external subtitle file based on the determined start time and end time of the short sentences; change the start time of the match to The playback timestamp PTS of the current video, and according to the playback timestamp, update the start time of each item in the external subtitle file whose start time is greater than the start time of the matching item. The invention makes the display time of the subtitle file consistent with the playing time of the audio and video, thereby realizing the automatic synchronization of the external subtitles and improving the viewing experience of users.

Description

A method and device for automatic synchronization of external subtitles

technical field

The invention relates to the technical field of video playback, in particular to an automatic synchronization method and device for external subtitles.

Background technique

Subtitles (subtitles of motion picture) refer to non-image content such as dialogues in TV, movies, and stage works in the form of text, and also generally refer to the text of post-processing of film and television works. When making video works such as movies, video files and subtitle files can be integrated together, so that the subtitles that cannot be changed or removed during playback are called embedded subtitles. In some works, the video file and the subtitle file exist independently, and the required version of the subtitle file can be imported when the video is played. This type of subtitle file is called an external subtitle. Compared with embedded subtitles, external subtitles have the advantages of being flexible, easy to import, and will not damage video quality.

External subtitles generally use special subtitle software for subtitle production. This production method first needs to listen to the complete lines manually, and input the subtitles of the complete lines into the electronic text according to the content described in each line. It uses special subtitle software to listen to the content of the subtitles while manually segmenting the lines. Determine the start time and length of each line of dialogue, the so-called "time axis". When all the subtitles are finished, the subtitle software will output one or more external subtitle files in one or more formats. When a playback system can recognize and support the playback method of external subtitles, these subtitle files can be loaded during video playback. However, due to the characteristics of the production of external subtitle files, the time stamp accuracy of external subtitle files is poor, resulting in poor synchronization with audio and video during playback, and it is very troublesome for users to manually adjust the playback time of subtitles, which seriously affects Users watch movies normally.

Contents of the invention

The purpose of the present invention is to provide an automatic synchronization method and device for external subtitles, so that the display time of subtitle files is consistent with the playback time of audio and video, thereby realizing automatic synchronization of external subtitles and improving the viewing experience of users.

In order to solve the above-mentioned technical problems, the embodiment of the present invention provides a kind of automatic synchronizing method of subtitles, comprising the following steps: extracting the audio part of the video file, and decoding the audio part to obtain pulse code modulation data; The coded modulation data is divided into audio segments, and the audio segments are classified; wherein, the categories of the classification include: silence, speech and non-speech; the audio segments classified as speech are divided into short sentences, and the determined The start time and the end time of the phrase; according to the start time and the end time of the determined phrase, a match is searched in the external subtitle file; the start time of the match is changed to the current video The playback time stamp PTS, and according to the playback time stamp, update the start time of each item in the external subtitle file whose start time is greater than the start time of the matching item.

Embodiments of the present invention also provide an automatic synchronization device for external subtitles, including: an extraction module, a segmentation module, a division module, a search module and an update module; the extraction module is used to extract the audio part of the video file, and The audio part is decoded to obtain pulse code modulation data; the segmentation module is used to segment the pulse code modulation data into audio segments, and classify the audio segments; wherein, the categories of the classification include: silence, Speech and non-speech; the division module is used to divide the audio segment classified as speech into short sentences, and determine the start time and end time of the short sentences; the search module is used to determine according to the The start time and the end time of the short sentence are searched for a match in the external subtitle file; the update module is used to change the start time of the match to the playback time stamp PTS of the current video, and play according to the Timestamp, updating the start time of each item in the external subtitle file whose start time is greater than the start time of the matching item.

Compared with the prior art, the embodiment of the present invention extracts the audio part of the video file, decodes the audio part to obtain pulse code modulation data, divides the pulse code modulation data into audio segments, and classifies the audio segments into speech, silence , non-speech, and then divide the audio clips classified as speech into short sentences, and determine the start time and end time of the short sentences, and then search in the external subtitle file according to the determined start time and end time of the short sentences A matching item, and change the start time of the matching item to the playback timestamp PTS of the current video, and update the start of each item in the external subtitle file whose starting time is greater than the starting time of the matching item according to the playback timestamp Time, so that the display time of the subtitle file's dialogue is automatically synchronized with the video playback, improving the user's viewing experience.

Preferably, in the step of searching for a matching item in the external subtitle file according to the start time and end time of the determined short sentence, the following sub-steps are included: within a preset duration before and after the start time , find the corresponding item in the external subtitle file; in the found corresponding item, find out all items whose dialogue duration with the phrase is within the allowable range of error; if the number of found items is more than one, A previous record of the determined phrase is compared with a previous record of the found item until the most similar item is found as a match. Thereby, the matching efficiency and accuracy of subtitles and audio and video are improved.

Preferably, in the step of dividing the audio segment into short sentences, the division is performed according to speech pauses; wherein, the speech pauses include at least a first preset number of audio segments. Thus, the efficiency of sentence division can be improved.

Preferably, the first preset number is 2. Thereby, the short accompanying sound information can be ignored, and the integrity of a sentence can be better protected.

Preferably, the phrase contains at least a second preset number of audio segments, and the second preset number is three. Therefore, the short-term invalid information in the audio can be filtered out, and the efficiency of sentence division can be improved.

Description of drawings

Fig. 1 is the flow chart of the automatic synchronization method of subtitles according to the first embodiment of the present invention;

Fig. 2 is a schematic diagram of a matching algorithm between phrases and subtitles according to the first embodiment of the present invention;

Fig. 3 is a structural block diagram of an automatic synchronization device for subtitles according to a second embodiment of the present invention.

detailed description

In order to make the object, technical solution and advantages of the present invention clearer, various embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings. However, those of ordinary skill in the art can understand that, in each implementation manner of the present invention, many technical details are provided for readers to better understand the present application. However, even without these technical details and various changes and modifications based on the following implementation modes, the technical solution claimed in each claim of the present application can be realized.

The first embodiment of the present invention relates to an automatic synchronization method for external subtitles, the specific process is shown in Figure 1, including the following steps:

Step 10: extract the audio part of the video file, and decode the audio part to obtain pulse code modulation data.

A video file is synthesized from a video stream and an audio stream. When playing a video online, the audio stream is first extracted from the video file. You can use the open source library ffmpeg to extract the audio part of the video file, and then decode the audio part into PCM (Pulse Coding Modulation, pulse code modulation, PCM for short) data through the corresponding decoder.

Step 11: Segment the PCM data into audio segments, and classify the audio segments.

In this embodiment, the extracted audio (PCM data) can be classified by using Marsyas software. For example, by Marsyas, the category of the audio data can be determined: silence, speech and non-speech. The audio frame length can be set to 32ms through the interface provided by Marsyas, and 5 audio frames are used as an audio segment, that is, the audio segment length is 0.16s. In the classification process, one classification can be performed in units of audio segments, so as to improve the efficiency of classification. In this embodiment, there is no specific limitation on the classification method of audio segments, as long as speech and non-speech can be distinguished. It can be seen that, through the classification in this step, the start time and end time of the voice segment in the audio segment can be obtained, laying a foundation for extracting the voice sentence from the audio segment.

Step 12: Divide the audio segment classified as speech into short sentences, and determine the start time and end time of the short sentences. Through the classification in step 11, the start time and end time of voice, non-voice, silence, etc. can be determined, and then the voice can be divided into short sentences according to the voice pause.

In this embodiment, detecting the beginning and end of a sentence is the key to short sentence division, because only when a high endpoint detection accuracy is achieved can it be targeted and control the length and number of sentences can be achieved. This step is based on the classification information obtained in step 11, and the speech unit (ie short sentence) can be intercepted from the audio by adopting a preset segmentation algorithm. Specifically, the following strategy can be used for audio segmentation: the time point of the silent segment or non-speech segment before entering the continuous speech segment is used as the start time of the sentence, and the time point of the last speech segment at the end of the continuous speech segment is used as the sentence end time of . In this way, after the audio is segmented, a "sentence-like" unit with a certain duration of speech pause as the segmentation boundary and relatively complete semantics can be obtained, that is, a short sentence in this embodiment.

However, detecting sentence endpoints through the above segmentation strategy may cause some extreme situations: for example, some extremely short sentences will be divided, the length of these sentences is only one or two audio segments, and such sentences usually only contain one or two words , does not even contain any valid speech information, so these sentences need to be filtered out and cannot be subtitled as valid sentences.

In order to improve the segmentation efficiency, it is set in the segmentation strategy that the speech pause includes at least a first preset number of audio segments, preferably, the first preset number of audio segments is, for example, 2 audio segments. By setting the minimum length of the speech pause, shorter audio information, such as the speaker's instantaneous breath change, can be ignored, thereby protecting the integrity of a sentence.

Further, the segmented phrase contains at least a second preset number of audio segments, preferably, the second preset number of audio segments can be, for example, 3 audio segments, that is, speech units with a total length of less than 0.48 seconds are ignored, By limiting the minimum sentence length, short-term invalid information in the audio, such as the speaker's light cough, can be filtered out.

It should be understood that this embodiment does not limit the specific values of the first preset data or the second preset number. In practical applications, adjustments can be made according to the characteristics of the language to determine the start time and end of the sentence unit more accurately and efficiently. time.

Through step 12, the extracted piece of audio is divided into individual sentences, and the start time and end time of the sentences are obtained, so that the playback duration of the sentences can be determined.

Step 13: Search for a matching item in the external subtitle file according to the determined start time and end time of the phrase.

Usually, the external subtitle file contains information such as start time, dialogue duration and so on. In this embodiment, during playback, the external subtitle file is obtained, and a data structure datastruct1 of <start time, dialogue duration> is created according to the external subtitle file, so that the start time and dialogue duration of each dialogue can be easily found. Then search for the matching item item in the data structure datastruct1 according to the start time and end time of the short sentence (ie, the dialogue in the video) divided in step 12.

Specifically, step 13 includes the following sub-steps:

Sub-step 130: Find the corresponding item in the external subtitle file within a preset time period before and after the start time.

Ideally, the start time and end time of each dialogue in the audio part (similar to the short sentences in this embodiment) and the start time and end time of the corresponding dialogue in the subtitle file (that is, the corresponding item in this embodiment) Time synchronization. Due to the characteristics of subtitle file production in the prior art, the start time and end time of corresponding items in the subtitle file deviate from the start time and end time of the dialogue in the audio part. Therefore, this step needs to find the corresponding item from the external subtitles within the preset duration (that is, the difference between the start time of the corresponding item in the possible subtitles and the start time of the audio dialogue), and the preset duration in this embodiment can be is 1 minute, that is, find the corresponding item in the external subtitles within 1 minute before and after the start time of the short sentence extracted from the video file. It should be understood that the preset duration can be set according to the actual characteristics of the subtitle file, and this embodiment does not limit the specific size of the preset duration.

Sub-step 131: Among the found corresponding items, find out all items whose dialogue duration with the short sentence is within the allowable range of error.

For example, within 1 minute before and after the start time of the short sentence, find all items in datastruct1 that have an error of 3 seconds from the dialogue duration of the short sentence. For example, the dialogue duration of a short sentence is 4 seconds, if within 1 minute, there are 3 subtitle items whose dialogue duration is between 2.5 seconds and 5.5 seconds, then the 3 corresponding items are extracted. It should be understood that the examples of specific numerical values in the tolerance range of errors in this embodiment are only for the convenience of understanding, and are not intended to limit the protection scope of the present invention.

Sub-step 132: Determine whether the number of found items is more than one. If the number of items found is one, then the corresponding item is considered to be the matching item of the corresponding audio, and proceed to step 14, if the number of items found is more than one, it is necessary to further filter the closest matching item, so continue to execute Step 133.

Sub-step 133: Compare the previous record of the determined phrase with the previous record of the found item until the most similar item is found as a matching item.

Now give an example and illustrate as follows: as shown in Figure 2, such as in step 131, phrase P finds 3 subtitle items (i.e. subtitle item A, subtitle item B, subtitle item C) in datastruct1, then continue to use phrase P's The previous record phrase P-1 is matched with the previous subtitle item A-1, subtitle item B-1, and subtitle item C-1 of subtitle item A, subtitle item B, and subtitle item C respectively, and the matching algorithm can be compared. Start time and dialogue duration, etc., if the phrase P-1 finds more than 2 subtitle items, then continue to record the last record phrase P-2 of the phrase P-1 with the last record of the multiple subtitle items found. One record is matched, and so on, until a subtitle entry matching the phrase is found.

Step 14: Change the starting time of the matching item to the playback time stamp PTS of the current video, and update the starting time of each item in the external subtitle file whose starting time is greater than the starting time of the matching item according to the playing time stamp.

Specifically, first change the starting time of the matching item to the playback time stamp PTS (Presentation time stamp, current time stamp, PTS for short) of the current video, and the starting time in the external subtitle file can be updated by the following formula to be greater than the matching item The start time of each entry:

Start time 2 = start time 1-(item.start time-video.pts)

Among them, item.start time is the start time of the current matching item, video.pts is the time of the current video frame, then (item.start time-video.pts) indicates the time difference between the current matching item and the audio and video. Start time 1 represents the start time of the uncorrected subtitle item in datastruct1, and start time 2 represents the start time of the corrected subtitle item in datastruct1.

This implementation mode can be embedded in the playback software. During the video playback process, this implementation mode is executed at the beginning of the video playback and within a predetermined time interval (such as 10 minutes) thereafter, that is, the audio data with a certain duration is acquired, and the Decode to obtain the pulse code modulation data, and then classify and segment the part of the audio data into short sentences, and find the matching item of the short sentence in the subtitle file, and then update the matching item and all subtitles whose playback time is after the matching item start time of . Or, it is also possible to match the starting time of all the dialogues in the audio data, so that the external subtitles are completely synchronized with the audio and video, achieving a better viewing effect.

Compared with the prior art, this embodiment extracts the audio part of the video file, and decodes the audio part to obtain the pulse code modulation data, so that the voice information in the audio can be analyzed, and then the pulse code modulation data is divided into audio segments , so that the audio clips can be classified into speech, silence, and non-speech through analysis, and further the audio clips classified as speech can be divided into short sentences, and the start time of the short sentences can be determined with the playback time stamp PTS of the current video frame and end time, and then search for a match in the external subtitle file according to the start time and end time of the determined phrase, so that the start time of the match can be changed to the playback timestamp PTS of the current video, and according to the playback Timestamp, update the start time of each item in the external subtitle file whose start time is greater than the start time of the matching item. Through the above steps, this embodiment can automatically correct the display time of the subtitle item according to the dialogue time, so that the subtitle display is consistent with the audio and video playback time, so that the external subtitles and audio and video are automatically synchronized to achieve a better viewing effect and improve user experience. .

The division of steps in the above methods is only for the sake of clarity of description. During implementation, they can be combined into one step or some steps can be split and decomposed into multiple steps. As long as they contain the same logical relationship, they are all within the scope of protection of this patent. ; Adding insignificant modifications or introducing insignificant designs to the algorithm or process, but not changing the core design of the algorithm and process are all within the scope of protection of this patent.

The second embodiment of the present invention relates to an automatic synchronization device for external subtitles. As shown in FIG. 3 , it includes: an extraction module, a segmentation module, a division module, a search module and an update module.

The extraction module is used to extract the audio part of the video file, and decode the audio part to obtain pulse code modulation data.

The segmentation module is used to segment the PCM data into audio segments and classify the audio segments, wherein the categories of classification include: silence, speech and non-speech.

The division module is used to divide the audio segment classified as speech into short sentences, and determine the start time and end time of the short sentences. Specifically, the division module is used to perform short sentence division according to the speech pause, the speech pause contains at least a first preset number of audio segments, and divides the audio segment into short sentences containing at least a second preset number of audio segments. Wherein, the duration of the first preset number of audio segments is , and the duration of the second preset number of audio segments is . It should be understood that the first preset number and the second preset number can be set according to the characteristics of the audio data and the subtitle file, and this embodiment does not limit their specific values.

The search module further includes: an initial matching submodule, a dialogue matching submodule and a comparison matching submodule. The start matching submodule is used to find a corresponding item in the external subtitle file within a preset time period before and after the start time, and is used to search for a matching item in the external subtitle file according to the determined start time and end time of the short sentence. The dialogue matching submodule is used to find out all items whose dialogue duration with the short sentence is within the allowable error range among the corresponding items found by the initial matching submodule. The comparison and matching submodule is used to compare the last record of the determined phrase with the last record of the found item when the number of items found by the dialogue matching submodule is more than one, until the most similar item is found as matches.

The update module is used to change the starting time of the matching item to the playback timestamp PTS of the current video, and update the starting time of each item in the external subtitle file whose starting time is greater than the starting time of the matching item according to the playback timestamp .

Compared with the prior art, this embodiment extracts the audio data in the video file, classifies the audio data, and divides the audio data into sentences, so as to obtain the precise start time and end time of the sentence, and then add them to the subtitle file accordingly. Find a matching item in , and modify the start time in the matching item accordingly to synchronize the subtitle file with the audio and video. Therefore, this embodiment does not require the user to manually adjust the external subtitles, and can automatically synchronize the external subtitles with the audio and video, thereby achieving a better viewing effect and improving user experience.

It is not difficult to find that this embodiment is a system embodiment corresponding to the first embodiment, and this embodiment can be implemented in cooperation with the first embodiment. The relevant technical details mentioned in the first embodiment are still valid in this embodiment, and will not be repeated here in order to reduce repetition. Correspondingly, the relevant technical details mentioned in this implementation manner can also be applied in the first implementation manner.

It is worth mentioning that all the modules involved in this embodiment are logical modules. In practical applications, a logical unit can be a physical unit, or a part of a physical unit, or multiple physical units. Combination of units. In addition, in order to highlight the innovative part of the present invention, units that are not closely related to solving the technical problems proposed by the present invention are not introduced in this embodiment, but this does not mean that there are no other units in this embodiment.

Those of ordinary skill in the art can understand that the above-mentioned embodiments are specific examples for realizing the present invention, and in practical applications, various changes can be made to it in form and details without departing from the spirit and spirit of the present invention. scope.

Claims (11)

1. the automatic synchronous method of plug-in captions, it is characterised in that comprise the steps of

Extract the audio-frequency unit of video file, and audio-frequency unit is decoded, it is thus achieved that pulse code modulation Data；

Described pulse code modulation data is cut into audio fragment, and described audio fragment is carried out point Class；Wherein, the classification of described classification comprises: quiet, voice and non-voice；

The described audio fragment being categorized as voice is divided into short sentence, and determines the initial time of described short sentence And the end time；

Initial time according to the described short sentence determined and end time, plug-in subtitle file is searched for one Individual occurrence；

The initial time of described occurrence is changed to the reproduction time stamp PTS of current video, and according to institute State reproduction time stamp, update initial time in plug-in subtitle file and be more than the initial time of described occurrence The initial time of each.

The automatic synchronous method of plug-in captions the most according to claim 1, it is characterised in that The described initial time according to the described short sentence determined and end time, plug-in subtitle file is searched for one In the step of individual occurrence, comprise following sub-step:

Before and after described initial time in preset duration, in described plug-in subtitle file, find corresponding entry；

In the described corresponding entry found, find out the dialogue duration with described short sentence in error allowed band All items；

If the item number found out is more than one, a upper record of the described short sentence determined is looked for described A upper record of the item gone out compares, until finding most like one as occurrence.

The automatic synchronous method of plug-in captions the most according to claim 1 and 2, it is characterised in that Described, described audio fragment is divided in the step of short sentence, divides according to speech pause；

Wherein, described speech pause is including at least the audio section of the first preset number.

The automatic synchronous method of plug-in captions the most according to claim 3, it is characterised in that institute Stating the first preset number is 2.

The automatic synchronous method of plug-in captions the most according to claim 3, it is characterised in that institute State the short sentence audio section including at least the second preset number.

The automatic synchronous method of plug-in captions the most according to claim 5, it is characterised in that institute Stating the second preset number is 3.

The automatic synchronous method of plug-in captions the most according to claim 1, it is characterised in that

In the described initial time determining described short sentence and the step of end time, to enter continuous speech Before Duan quiet section or the time point of non-speech segment are as the time started of sentence, to terminate continuous speech The time point of last voice segments during section is as the end time of sentence.

8. the automatic synchronizing apparatus of plug-in captions, it is characterised in that comprise: extraction module, cut Sub-module, division module, search module and more new module；

Described extraction module is for extracting the audio-frequency unit of video file, and is decoded audio-frequency unit, Obtain pulse code modulation data；

Described cutting module is used for being cut into described pulse code modulation data audio fragment, and to described Audio fragment is classified；Wherein, the classification of described classification comprises: quiet, voice and non-voice；

Described division module is for being divided into short sentence by the described audio fragment being categorized as voice, and determines institute State initial time and the end time of short sentence；

Described search module is for the initial time according to the described short sentence determined and end time, plug-in Subtitle file is searched for an occurrence；

When described more new module for changing to the broadcasting of current video by the initial time of described occurrence Between stab PTS, and stab according to described reproduction time, update in plug-in subtitle file initial time more than described The initial time of each of the initial time of occurrence.

The self-synchronous system of plug-in captions the most according to claim 8, it is characterised in that institute State search module to comprise: initial matched sub-block, dialogue matched sub-block and comparison match submodule；

Described initial matched sub-block is used for before and after described initial time in preset duration, described plug-in Subtitle file finds corresponding entry；

Described dialogue matched sub-block, in the corresponding entry that described initial matched sub-block finds, is found out With the dialogue duration of the described short sentence all items in error allowed band；

Described comparison match submodule is more than one for the item number found out in described dialogue matched sub-block Time individual, a upper record of a upper record of the described short sentence determined with the described item found out is compared Relatively, until finding most like one as occurrence.

The self-synchronous system of plug-in captions the most according to claim 8 or claim 9, it is characterised in that Described division module is additionally operable to divide according to speech pause；

Wherein, described speech pause is including at least the audio section of the first preset number.

The self-synchronous system of 11. plug-in captions according to claim 10, it is characterised in that Described division module is additionally operable to described audio fragment is divided into the audio frequency including at least the second preset number The short sentence of section.