JP2003114845A - Media conversion method and media conversion device - Google Patents

Abstract

(57) [Summary] [Problem] A conventional video distribution format cannot be directly applied to a case where the content is reproduced from the middle of the content or a stream which is encoded in real time. SOLUTION: In order to achieve the above-mentioned object, a conversion processing unit 56 of a distribution server performs processing of
Convert the three parts to moov71. According to the above means, the server can realize the stream started from the middle of the original content with a small processing amount, while the terminal side does not change the processing for receiving and reproducing the conventional storm. Instead, playback from the middle can be realized.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a video distribution server, and more particularly to a processing method for distributing a video file from the middle and for distributing real-time video through the distribution server.

[0002]

2. Description of the Related Art In response to a request from a terminal, video or audio (hereinafter, "audio" is used to mean voice or audio) and video are distributed from a server to a terminal via a transmission line. Has
A system layer is required to multiplex video data, audio data, and sync information as one data, and sync information for indicating the reproduction timing of each medium, that is, video and audio. Conventionally, the file format (hereinafter MP4 format) defined by ISO / IEC 14496-1 is used as the method for defining these system layers and synchronization information.
was there. As shown in FIG. 1, the MP4 format is composed of an incidental information part 11 called moov and encoded media data (video data or audio data part 12) called mdat. As shown in FIG. 2, the moov 11 further includes header information / media information (header information 13), storage position of each media, and reproduction time information (time stamp) portion (media access information 14).
Composed of. The header information 13 includes, for example, the number of videos included in the subsequent data, the image size, the encoding method,
The bit rate and the like are described. On the other hand, the media access information 14 includes a reproduction unit of video (or audio) data in mdat 12 (hereinafter, access unit: AU).
Storage location information for each and reproduction time information for each AU are stored.

In the case of an MP4 format file as shown in FIG. 1, the file is distributed via a transmission line, and the terminal which receives the file receives the file from the middle of receiving the file in parallel with the receiving operation.
Considering the operation of reproducing a video, it is necessary to read all the data of the moov 11 portion that is not used for reproducing the beginning portion of the file, which increases the delay time from the start of file reception to the start of reproduction. In order to reduce the delay time in such a case, as shown in FIG. 3, the contents are subdivided into short-time contents, and the media access information and the media data corresponding to the respective short-time contents are alternately arranged.
How to disperse and arrange in the file, that is, at the beginning
A method is known in which moov 21 and a plurality of moofs 23 and 25 are dispersed and arranged. moov, one moov and one or more mo
The structure of moov when distributed and arranged in of is as shown in FIG. 4, and since the information 32 regarding moof exists,
Hereafter, it shows when moof existed. The structure of moof is, as shown in Fig. 5, the serial number 41 of the corresponding moof and the md following the corresponding moof.
It consists of media access information (data position and time stamp) for each media included in at.

[0004]

The above-mentioned prior art cannot be directly applied to the case where the content is reproduced from the middle or the stream which is encoded in real time. SUMMARY OF THE INVENTION It is an object of the present invention to provide a stream conversion method that enables reproduction of a content from the middle or a permanent stream encoded in real time in a terminal compatible only with the conventional technique.

[0005]

[Means for Solving the Problems] In order to achieve the above-mentioned object, the moof part of the original stream is mo
convert to ov.

[0006]

BEST MODE FOR CARRYING OUT THE INVENTION A first embodiment according to the present invention is shown in FIG. FIG. 6 shows an original stream 53 in which the video 51 is once subjected to the encoding process 52 and once conformed to the MP4 format.
Is generated and once accumulated by the accumulation process 54, the reproduction start time 60 is received, the accumulated stream 55 is read out, and the conversion process 56 starts a converted stream 57 that starts from a time near a designated midpoint of the stream 53. Is generated, and the generated storm 57 is distributed to the terminal as a distribution stream 59 by the distribution process 58.

FIG. 7 shows the relationship between the original stream 53 and the converted stream 57 in the above conversion processing. In the original stream 53, moov 21 and mdat 22 following it are arranged at the beginning, and the following moof 23 / mdat
24, moof25 / mdat26, moof and corresponding md
The combination of at is repeated (in the following description, the pair of media data mdat and their corresponding moof are
"/" is used like moof / mdat). When the AU corresponding to the start time is included in mdat24, a new start point is set.
At the beginning of mdat24, in conversion processing 56, moov21
A new moov 70 is generated from the information of each medium (video and audio) described in 1. and the information of mdat 24 described in moof 23. Thereafter, the mdats of the original stream 53 are copied to the mdats 24 and 26, and the moof 25 is output as the moof 71 after the serial number is changed. The stream 55 is a stream of the stream 53, which is not necessary for generating the stream 57, such as mdat 22.

FIG. 8 is a flow chart showing the details of the above conversion process. In the conversion process 56, first,
The moov 21 of the original stream is read and the header information described therein is read. Next, the moof 23 which is the new stream start position is searched. In the moov output processing 80, the above header information and moof that is the stream start position
A new moov 70 is output using the information of. Loop process 8 that outputs mdat 24 corresponding to moof 23 after moov 70 and then outputs a predetermined number of moof / mdat combinations
Enter 1. In the loop processing, first, it is determined whether there is a subsequent moof / mdat combination, that is, the stream end determination 82. When the stream ends, the process moves to the process 83, the information such as the data size and the reproduction time of the content written in the moov 70 is updated, and the process ends. On the other hand, if the end determination processing 70 is not finished, the loop 81
Side processing is performed. That is, the next moof is read, the serial number is modified to a new value, and then the modified moof
, And then output the corresponding mdat. After performing these processes, the end determination process 70 is performed again. It should be noted that the conversion is terminated by (1) when the final data of the content is output, (2) when the termination request is received from the terminal,
(3) There is a case where the server voluntarily terminates due to a delivery error, a timeout of a response from the terminal, or the like.

FIG. 9 is a flow chart for explaining the details of the moov output process 80 of FIG. In the main processing of Moov, first, the header information of the already read moov 21 is output. Next, the synchronization correction information for correcting the reproduction time of each medium is output. Then, the media access information of the top moof 23 is output. The number of bytes of all of these output data is counted and described as the size of moov 70 at the head of moov.

FIG. 10 shows an output process 8 of the synchronization correction information of FIG.
It is a figure for demonstrating the content of 5. FIG. 10 shows the synchronization correction when an original stream composed of two media, audio and video, is converted into a stream that starts halfway. Generally, for audio and video within one content,
Since the sampling time of each AU is asynchronous,
When the stream is extracted from the middle of the original stream, the reproduction time of the first audio AU and the reproduction time of the first video AU do not match. That is, as shown in FIG. 10, when a point near the boundary between the video AU2 and the video AU3 is set as a new start point,
As shown in the figure, the video starts from AU3 and the audio starts from AU8, resulting in a time difference T. In the synchronization correction process, by describing the value of T in the conversion stream, it becomes possible to make the time relationship between the audio and video at the same time position in the original stream at the time of reproduction at the terminal. In FIG. 10, since the video is delayed, the information indicating “delay the start of playback of the video signal by T” is output. However, if audio is being sent, a message “delay the start of playback of the audio signal by a predetermined time” is output. Output information.
These playback times are all moov or
Since it is described in moof, judgment and calculation are performed based on the size of the time stamp value described in moov or moof of the original stream.

In the case of low rate coding, the reproduction time of the audio AU is constant and short, for example, at a period of 30 ms, whereas the reproduction time of the video AU is, for example, 10 frames / frame.
It is as long as seconds, that is, 100 ms, has a variable frame rate, and its cycle is often irregular. Therefore, on the terminal side, reproduction is often performed with the audio cycle as the reference cycle. That is, the terminal side is configured to perform the video output process only when the video reproduction process is required on the basis of the audio signal reproduction process. Therefore, the load on the terminal side is reduced by setting the reproduction start position so that the video is displayed at the same time as or after the reproduction of the reference audio is started, that is, "the start of the reproduction of the video signal is delayed". be able to. On the other hand, if audio playback is delayed, the video signal playback process must be started before the start of the standard process, and the process at the start becomes a control method different from the normal process, and additional processing is added to the terminal side. Requires software or hardware. As described above, according to the embodiment described with reference to FIG. 6 to FIG. 10, the server starts the generation of a new moov for the stream started from the middle of the original content.
It can be realized with a small amount of processing only by minor correction of of, while
The terminal side can realize the reproduction from the middle without any change from the conventional processing of receiving and reproducing the storm.

[0012] Especially when the configuration of Fig. 8 is combined with a video server, the same contents are delivered when the contents are delivered from the beginning of the contents and when delivered from the middle of the contents.
Moreover, it can be realized by holding only a single data, and the capacity of the storage device of the video server can be reduced,
Alternatively, there is an effect that more data can be held in a certain storage device.

FIG. 11 shows a second embodiment of the present invention. FIG. 11 shows a processing configuration in which an original stream (real-time video) encoded in real time is cut out from an arbitrary time point and delivered as a new converted stream. The input video 51 is encoded in real time by the encoding process 52, and the original stream 5
3 is generated. In the real-time data conversion processing 101, this original stream 53 is converted in real time into a converted stream 57 that starts from the time when there is an instruction,
It is distributed by the distribution process 58. This is for example shown in FIG.
As shown in 2, the surveillance camera or the like is used for the purpose of accessing the image taken at any time from a plurality of terminals.

FIG. 13 is a flow chart for explaining the details of the real-time data conversion processing of FIG. First, before the conversion is started, the header information of the stream 53 is acquired and the accumulation process 120 is executed. Next, it waits for a distribution request from the terminal. If there is no distribution request, the next stream 53
Search for moof or mdat.

When a delivery request is made, the first mo after the request is sent.
Search for of, generate moov from this moof information and the previous header information, and output it. Subsequent processing is the same as in FIG. Note that the end processing of the real-time processing includes the case of “not ending” in addition to the end processing of FIG. Therefore,
In the fields such as the data size and the playback time (content length) of the content described in the head moov, information indicating the meaning such as “undefined”, “infinite”, “real-time delivery” is described, and data such as the case of FIG. It is necessary to distinguish it from the case where the size is finite. Even in the case of FIG. 8, the leading mo
Modifying the data described in moov by entering information indicating the meaning such as “undefined”, “infinite”, “real-time delivery” in fields such as the data size and playback time described in ov.
It is possible to omit 3.

By temporarily storing the moof and the corresponding mdat in a buffer or the like, the delay time between the original stream and the distribution stream becomes long, and the real-time property is slightly impaired. However, the distribution request is issued from the terminal and the video distribution is started. By compensating for the time difference, the video can be distributed from the video at the distribution request time.

In the accumulating process 120 for acquiring the header information, the acquisition of the header information is set at the start of encoding, but the following process may be used. (1) The header information is periodically delivered in the stream 53 or by a channel provided in the stream 53. (2) The real-time conversion processing unit inquires of the encoder unit about the header information, and the encoder unit notifies the header information for each inquiry in the stream 53 or by the channel provided in the stream 53. (3) Header information is recorded in advance in the real-time conversion processing unit, and the encoder unit performs processing with the same parameters.

FIG. 14 shows a moov output process for real-time data. The content of the processing is the same as that of FIG. 9, but in order to secure the real-time property, the output of the generated moov is used as a temporary buffer, and the moov data is immediately processed by the data main processing 125 in the buffer after the moov generation is completed. To deliver.

FIG. 15 shows a third embodiment of the present invention.
In FIG. 15, a plurality of streams (three in the example of FIG. 16) having different bit rates are prepared for the same content, and a plurality of streams are switched in units of moof / mdat combinations in response to a request from the terminal. This enables variable bit rate transmission. FIG. 16 shows an example in which the bit rate is variable and is applied to a system in which the bit rate of the line to the terminal fluctuates, and it is possible to deliver at a bit rate adapted to the bit rate of the network.

In FIG. 16, there is a request to change the distribution initially started at 32 kbps to 48 kbps in the middle of time 4 due to the expansion of the bandwidth from around time 4, and the rate is changed from time 5 and thereafter time 11 The bit rate is changed from 64 kbps to 64 kbps and from time 13 to 32 kbps.

FIG. 17 is a flow chart for explaining the details of the real-time data conversion processing corresponding to the processing of FIG. Although the contents of the process are almost the same as those in FIG. 13,
Bitrate setting process 150, each moof / mda at the start of distribution
13 is different from FIG. 13 in that a bit rate change request presence / absence determination 151 is added before distribution and a bit rate change processing 152 is added when there is a bit rate change request. Also,
Each moof / mdat is read from the stream corresponding to the bit rate set at each time point. On the other hand, for bit rate streams that do not correspond to the above,
In steps 160 and 161, moof / mdat is skipped and always synchronized. The third embodiment of the present invention is the second embodiment.
Although the description has been given based on the first embodiment of the present invention, it is obvious that it can be combined with the first embodiment of the present invention.

By the processing of FIGS. 15 to 17, even in a system in which the line rate to the terminal fluctuates, it is possible to deliver at a rate suitable for the line rate at that time. Further, it is also effective when the line rate does not fluctuate during distribution but the actual line rate is determined by the surrounding environment or the like and is not decided in advance. The line rate is measured using the following environment information.

(1) The terminal notifies the reception bit rate measured by the terminal. (2) From the terminal, notify the information related to the line rate,
The distributor sets an appropriate bit rate based on the received information. For example, the maximum bit rate etc., the number of acquired lines in a communication path that bundles multiple lines, the strength of radio waves in a wireless communication path,
The error rate value is used. (3) In the case of a system in which the server and the terminal operate in synchronization with each other, that is, in a system in which a notification of data reception completion or a request for transmitting the next data is obtained from the terminal side, the server measures the transmission bit rate. (4) When the server and the terminal operate in synchronization, the transmission bit rate is estimated from the remaining amount of the transmission buffer. (5) The communication bit rate is notified from the network. (6) The network notifies the information related to the line rate. (7) A combination of the above.

FIG. 18 is a diagram for explaining the outline of the fourth embodiment of the present invention. The fourth embodiment is a modification of the first embodiment of FIG. 6, and in the first embodiment, the stream using the moof in the form of FIG. 8 is targeted, but in the fourth embodiment,
The target is a stream that does not use moof in FIG.

In the fourth embodiment, since the input original stream has the format shown in FIG. 1, the contents are as shown in the upper part of FIG. That is, there is only one moov, and among them,
It consists of header information 13 and media access information 14.
Here, the media access information 14 can be logically considered by dividing it into data positions and time stamps for each subdivided data, which are composed of a short time. When the reproduction start time is specified, the moov 70 of the converted stream is generated from the media access information 201 corresponding to the time and the header information 13. Also, the corresponding subdivision data 20
2 is output as mdat24. Subsequent da in mdat12
The ta and corresponding media access information in moov 11 are sequentially output.

FIG. 19 is a flow chart for explaining the processing of the fourth embodiment. Although the basic processing is the same as the processing of FIG. 8, in FIG. 19, the media access information 201 search processing 210 of the start data is performed instead of the moof 23 search of FIG. Further, in FIG. 8, the head mdat24 is output as it is, whereas in FIG. 19, the start data 202 is obtained based on the data position obtained in the process 210.
The search 211 is performed and the obtained mdat is output as the head mdat24.

Thereafter, in the loop 220, similarly, the medium access information reading 212 and the next data reading processing 214 in mdat are performed. In addition, in FIG.
Only the serial number of moof was corrected and output.
In the case of, in step 213, moof is generated from the corresponding moov data and is output.

As described above, the present invention can be applied to a stream that does not use moof. When applied to a stream that does not use moof, data in moov needs to be analyzed, and the processing amount is higher than when moof is used. On the other hand, if you are using moof,
Although the starting point was set only in units of, in the fourth embodiment, it is possible to start from any AU. However, the starting AU needs to be a randomly accessible AU.
Further, by applying the fourth embodiment and performing a process of analyzing the inside of moov or moof, it is possible to generate a stream having an AU in the middle of moof as a start point for a stream using moof. .

When the second to fourth embodiments are combined with the video server as in the first embodiment,
It has the following effects. In the second embodiment, real-time data can be delivered from any moof / mdat. Further, since the conversion processing amount is small, it is possible to simultaneously deliver to more terminals having different delivery start positions with a limited CPU.

In the third embodiment, a transcodec (conversion device combining a decoder and an encoder) is installed in order to convert the bit rate at the time of distribution from the video server, and uncompressed contents are prepared. In addition, it is possible to realize the distribution corresponding to the band fluctuation with a very small processing amount as compared with the distribution while performing the encoding process in real time for each terminal. Also,
Even if the fluctuation of the band for each terminal is different, it is possible to process with the same processing amount, so with a CPU with a constant processing amount,
The number of terminals that can be processed does not change.

[0031]

In the first embodiment, the stream started from the middle of the original content is displayed on the server side as mo.
It can be realized with a small amount of processing only by generating ov and finely adjusting moof. On the other hand, the terminal side can realize reproduction from the middle without any change from the processing of receiving and reproducing the previous storm. . In the second embodiment, the processing of cutting out the stream coded in real time from an arbitrary time point and delivering it as a new stream can be realized with a small processing amount, while the terminal side can use the conventional storm. Real-time video can be played back without any change to the process of receiving and playing back.

In the third embodiment, even in a system where the line rate to the terminal fluctuates, it is possible to deliver at a rate that matches the line rate at that time. Also,
Although the line rate does not fluctuate during distribution, it is also effective when the actual line rate is determined by the surrounding environment or the like and is not determined in advance.

[Brief description of drawings]

FIG. 1 is a diagram illustrating an MP4 file format.

FIG. 2 is a diagram illustrating details of moov11 in MP4 file format.

FIG. 3 is a diagram illustrating a file format using moof.

FIG. 4 is a diagram illustrating details of moov 21 when using moof.

FIG. 5 is a diagram illustrating details of moof 23.

FIG. 6 is a configuration diagram of a first embodiment of the present invention.

FIG. 7 is a diagram illustrating an outline of conversion processing according to the first embodiment of this invention.

FIG. 8 is a flowchart illustrating a detailed algorithm of the first exemplary embodiment of the present invention.

9 is a flowchart illustrating details of moov output processing in FIG.

FIG. 10 is a diagram for explaining synchronization correction between media.

FIG. 11 is a diagram illustrating an outline of conversion processing according to the second embodiment of this invention.

FIG. 12 is a diagram illustrating an application example of the second embodiment of the present invention.

FIG. 13 is a flowchart illustrating a detailed algorithm of the second exemplary embodiment of the present invention.

14 is a flowchart illustrating details of moov output processing in FIG.

FIG. 15 is a diagram illustrating an outline of conversion processing according to the third embodiment of this invention.

FIG. 16 is a diagram for explaining the outline of the operation of the third embodiment of the present invention.

FIG. 17 is a flowchart illustrating a detailed algorithm of a third exemplary embodiment of the present invention.

FIG. 18 is a diagram for explaining the outline of the operation of the fourth embodiment of the present invention.

FIG. 19 is a flowchart illustrating a detailed algorithm of a fourth exemplary embodiment of the present invention.

[Explanation of symbols]

11 moov 12 mdat 53 Hara Stream 56 Conversion process 57 Midway Play Stream 80 moov output processing 101 Real-time data conversion processing 130 Bit rate change request

Continued front page    (72) Inventor Toru Yokoyama             1-280, Higashikoigakubo, Kokubunji, Tokyo             Central Research Laboratory, Hitachi, Ltd. (72) Inventor Norihiro Suzuki             1-280, Higashikoigakubo, Kokubunji, Tokyo             Central Research Laboratory, Hitachi, Ltd. (72) Inventor Masahiro Wada             2-15-1 Ohara, Kamifukuoka City, Saitama Stock             Company CAD Research Institute (72) Inventor Yasuhiro Takishima             2-15-1 Ohara, Kamifukuoka City, Saitama Stock             Company CAD Research Institute (72) Inventor Shigeyuki Sazawa             2-15-1 Ohara, Kamifukuoka City, Saitama Stock             Company CAD Research Institute (72) Inventor Satoshi Miyaji             2-15-1 Ohara, Kamifukuoka City, Saitama Stock             Company CAD Research Institute F-term (reference) 5C064 BA01 BB05 BC10 BC16 BD02                       BD07

Claims (6)

[Claims] 1. A code including header information, media access information subdivided with respect to the header information, and media data corresponding to the media access information is input, and further reproduction start position information is input. Enter
New header information is generated from the header information and media access information corresponding to the reproduction start position information, and media access information and media data after the start position corresponding to the new header information and the reproduction start position information. A media conversion method characterized in that a new code is generated from and output. 2. The media conversion method according to claim 1, wherein the header information is accumulated, and a request to generate a code is regarded as an input of the reproduction start position information, and a new code is generated and output. 3. A plurality of codes having different characteristics composed of header information, media access information subdivided and arranged, and media data, and environment information for selectively judging one code from the plurality of codes. Is generated, selection information for selecting one piece of media access information and media data is generated based on the environment information, and one piece is created for each piece of media access information and media data that is subdivided using the selection information. The media access information and the media data are selected, and a series of media access information and the media data obtained by the selection and the header information, and the newly generated header information from the specified media access information after conversion are combined into one. A media conversion method characterized by outputting as a new code. 4. The media conversion method according to claim 3, wherein the plurality of codes having different characteristics are a plurality of codes having different bit rates. 5. The process of generating one piece of media access information and selection information for selecting media data based on the environment information is executed only once at the start of one media conversion. 3. The media conversion method according to claim 3 or claim 4. 6. An encoding processing unit for encoding an image and outputting an original stream, a storage processing unit for accumulating the original stream from the encoding processing unit, and inputting a reproduction start time to generate new header information. A media distribution apparatus comprising: a conversion processing unit that generates a conversion stream from a time near a specified point in the middle of the original stream stored in the storage processing unit, and a distribution processing unit that distributes the conversion stream. .