GB2426401A

GB2426401A - Handling scalable media data

Info

Publication number: GB2426401A
Application number: GB0608288A
Authority: GB
Inventors: Marco Boltz; Stephan Lietz; Lars Placke; Achim Dahlhoff; Ulrich-Lorenz Benzler
Original assignee: Robert Bosch GmbH
Current assignee: Robert Bosch GmbH
Priority date: 2005-05-20
Filing date: 2006-04-27
Publication date: 2006-11-22
Anticipated expiration: 2026-04-27
Also published as: GB2426401B; GB0608288D0; FR2886498B1; ITMI20060901A1; DE102005023355A1; FR2886498A1

Abstract

The invention relates to a method for handling scalable medial data 59 which exist in the form of sequences. In each case one sequence is allocated to at least one partial datum 63, 65, 67 under consideration of certain parameters. Partial data are managed in accordance with the parameters and at least one partial datum is transmitted to a receiver 69, 71, 73 in dependence upon at least one parameter.

Description

DESCRIPTION

METHOD OF HANDLING SCALABLE MEDIAL DATA

The invention relates to a method and system for handling scalable medial data and to a computer program and computer program product.

The storing, e.g. the digital recording, of TV programs can he effected by manual operation of appropriate devices such as video recorders or hard discs. The quality of transmitted TV programs is set by the transmission bandwidth provided by a provider. For legal reasons, when video streams are supplied via on-line providers over the internet, recording is usually totally prohibited or is at any rate of very poor quality.

Scalable encoding of multimedia data has grown considerably in importance owing to the standardisatjon effected within the ISO/IEC SC29WG 11 (MPEG), e.g. scalable AAC to BSAC for audio encoding and motion-compensated temporal filtering (MCTF) for video encoding.

These methods offer the possibility of adapting a corresponding data stream, without transcoding, in its transmission rate to the requirements of the transmission channel and/or the end device. Furthermore, the methods permit adaptation of the signals with respect to temporal sampling (audio: sampling rate, video: frame frequency), resolution (audio: bandwidth, video frame size) and quality (signal distortion corresponding to the transmission rate).

On this basis a method having the features of claim 1, a system. having the features of claim 8, a computer program having the features of claim 9 and a computer program product having the features of claim 10 are proposed.

In the method in accordance with the invention for handling scalable medial data which exist in the form of sequences one sequence is respectively allocated to at least one partial datum undcr consideration of certain parameters. In so doing partial data are managed in accordance with the parameters and at least one partial datum is transmitted to a receiver in dependence upon at least one parameter.

In the preferred embodiment of the method, provision is made that within the at least one partial datum to he transmitted, at least one first sequence and at least one second sequence, which follow each other, are transmitted. The at least one first sequence is directly encoded and transmitted at a first moment. By means of preferably temporal motion compensation of the at least one first sequence to the at least one second sequence a high pass sequence is generated.

By means of, in particular, temporal inverse motion compensation from the high pass sequence back to the first sequence a low pass sequence is generated. By carrying out a subtraction between the low pass sequence and the first sequence a delta low pass frequency is generated.

The high pass sequence and the delta low pass sequence are transmitted at a second moment after the first moment.

With this arrangement a method for scalable encoding and decoding of medial data on the basis of motion-compensated temporal filtering (MCTF) is provided which, with a short delay time, still leads to no significant deterioration in the encoding efficiency. An item of information appertaining to the low pass sequence is split into the at least one first sequence and the delta low pass sequence.

In so doing the at least one encoded first sequence can be considered to be an encoding unit comparable with the intrasequence known from the MPEG format. It is processed with no temporal reference to other sequences and therefore with a very short encoding delay. A similar method is already known by the term unrestricted MCTF" (UMCTF). However, in contrast to the UMCTF in the proposed method an additional item of information, namely the delta low pass sequence, is transmitted in addition to the high pass sequence, which delta low pass sequence contains the difference between the at least one first sequence and the low pass sequence required for efficient reconstruction of the at least one second sequence. In order to generate the delta low pass sequence both the items of information from the at least one first and also from the at least one second sequence are required. This can thus only be generated and transmitted with a delay in comparison to the at least one first sequence. 1-lowever, since during encoding only reconstruction of the at least one second sequence takes place, there is no increase in the overall delay during encoding and transmission.

Furthermore, provision can be made for the partial data to be managed via pointers which are chained in hierarchical lists, wherein the hierarchical lists are provided in accordance with the parameters. At least one list can consist of pointers to further lists, wherein in these further lists information for selection of specific groups of partial data for different parameters e.g. qualitative disassembly steps or resolutions is stored.

By these further features a method for efficient storage of scalable medial data is provided which permits easy access to the partial data relevant to the respective adaptation but n so doing brings about absolutely no limitation with respect to the combination of different parameters serving for the purpose of adaptation, e.g. temporal sampling, resolution or quality. At least one hierarchically chained list constitutes the core of this aspect of the method, pointers to specific data or sequences within an overall data stream being managed within this list. An uppermost level in the hierarchy is formed by a list of pointers to the temporal disassembly steps.

Respective lists are located therein having pointers to the associated local disassembly steps where lists with pointers to qualitative disassembly steps may in turn possibly be managed Ii provision is made for access to the corresponding data of the overall data slicain w ihce Ist for the purpose of the adaptation, only a subset must be selected from temporal. ocal and possibly qualitative disassembly steps, which is suitable for a respective end device or a transmission channel. An adaptation is carried out only through the use of the pointers from the hierarchically chained lists and is therefore possible at the running time without precise knowledge of the bit stream syntax or processing (parsing).

In a further embodiment of the method a program consisting of medial data and able to he 1 5 reconstructed from partial data is transmitted to the receiver or to a user, wherein, in dependence upon different qualitative disassembly steps, at least one first partial datum is transmitted freely to the user and at least one second partial datum is transmitted in encrypted form to thc uei.

wherein the at least one second encrypted partial datum is decrypted by the user using a key provided by the provider.

The system in accordance with the invention for handling scalable medial data in the form of sequences is designed in such a way as to respectively allocate a sequence to at least one partial datum under consideration of parameters, to manage partial data in accordance with the parameters and to transmit the at least one partial datum to a receiver in dependence upon at least one parameter.

The method in accordance with the invention or the system in accordance with the invention, which comprises, e.g. a scalable digital video recorder or a data processing device with suitable software, permit the user to record a program or the medial data in a non-encrypted basic quality fbr initial viewing free of charge or at least for a low price. If necessary, corresponding clearance or the additional information required for a higher quality can be ordered subsequently from a provider for payment of a charge. This system can include both the device, e.g. the receiver, as well as a service, which use the method in accordance with the invention for scalable encoding of video contents.

The provider of this service requires no additional storage space for provision of the film material in different quality levels. All desired quality levels can be extracted from a single transmitted data stream. For the user this method permits targeted and therefore inexpensive selection in a quality sufficient for him depending on his ideas of reusability or consumer interest in the material. The cost advantage between simply watching missed programs as opposed to quality viewing or saving of the video material provides for personally differentiated classification of the programs offered by television broadcasters or online video libraries The computer program in accordance with the invention having program encoding means is designed in such a way that all steps of the method in accordance the invention are to be carried out when the computer program is executed on a computer or a corresponding computing unit.

in particular in the system in accordance with the invention.

The computer program product in accordance with the invention having program encoding means is designed in such a way that all steps of the method in accordance with the invention are to be carried out when the computer program is executed on a computer or a corresponding computing unit, in particular in the system in accordance with the invention.

Further features and embodiments of the invention will become clear from the description and the enclosed drawing.

It will be understood that the above-mentioned features and those still to be explained hereinunder can not only be used in the combination mentioned in each case but also in other combinations or individually without exceeding the scope of the present invention.

The invention is illustrated schematically in the drawing with the aid of an exemplified embodiment and is described in detail hereinunder with reference to the drawing in which: Figure 1 is a diagram relating to the process of a known method for video encoding on the basis of the motioncompensated temporal filtering of pictures.

Figure 2 is a diagram relating to the temporal process of a preferred embodiment of one aspect of the method in accordance with the invention relating to the effecting of encoding by generation of motion-compensated information.

Figure 3 is a diagram relating to a preferred embodiment of a further aspect of the method in accordance with the invention concerning the management ot medial data h\ hierarchical construction of list elements, and Figure 4 is a diagram relating to a preferred embodiment of a further aspect of the method in accordance with the invention relating to the transmission of medial data.

Figure 1 shows a method for video encoding in accordance with the prior art on the basis of motion-compensated temporal filtering of pictures. In order to effect temporal filtering a plurality of successive individual pictures, in this case first sequences I and second sequences.

are combined to form a group (GOP). Within this GOP the two successive sequences 1. 3 are each motion-compensated. From these two sequences 1, 3 a temporal high pass sequence 7 is generated by conventional motion compensation 5 of the first sequence 1 to the second sequence 3. By means of an inverse motion compensation 9 of the high pass sequence 7 back to the first sequence I a low pass sequence 11 is produced. The combination of conventional motion compensation 5 and inverse motion compensation 9 corresponds to the known filtering in accordance with the so-called "lifting scheme" which permits perfect reconstruction of the filtered sequences 1, 3 in a receiver. In the case of transmission 13 the high pass sequence 7 and the low pass sequence 11 are each transmitted.

In further temporal filtering steps in each case two, or generally also several, successive low pass sequences 11 of the preceding step are now motion-compensated according to the same method, whereby tree-like binary temporal disassembly of the picture sequence takes place.

The temporally filtered sequences can then be subjected to local disassembly, e.g. by means of the DCI known from JPEG, the EBCOT encoding used in JPEG2000 or the EZBC method Fiowever, the combination of conventional and inverse motion compensation 5. 9 during the temporal filtering can cause an encoding delay to occur since all sequences of the GOP must be available before a transmission of encoded data starts. The encoding delay therefore amounts to at least the length of a GOP. This encoding delay can be avoided when the inverse motion compensation II is omitted, this manner of procedure is known as so-called "unrestricted MCTF". 1-lowever, this is associated with a definite deterioration in the encoding efficiency (comparison with a necessary transmission rate for the encoding with the same quality). A further possible way of reducing the encoding delay is to shorten the GOP, which, however, leads to a reduction in the encoding efficiency.

On the basis of this background Figure 2 shows a schematic diagram of an embodiment of a manner of proceeding to effect video encoding in accordance with the invention. In this case first sequences 15 (A-frames, subframes) and second sequences 17 (B-frames, subframes) follow in a group of pictures (GOP) in a temporally successive and alternating maimer.

A temporal motion compensation 19 (MC, also called "prediction step") is carried out in the proposed method according to the MCTF method known from the literature. By means of the motion compensation 19 from the first sequence 15 to the second sequence 17 a high pass sequence 21 (H- frame, also high pass frame) is produced which can therefore be allocated temporally to the second sequence 17. in contrast to the MCTF method, however, the first sequence iSis directly encoded without temporal reference to the second sequence 17, there is therefore no significant delay in a first transmission 23 arid reconstruction of the first sequence in a receiver or decoder, wherein this transmission 23 takes place in each case at a first moment.

However, in order to permit efficient reconstruction of the second sequence 17 in the receiver an inverse motion compensation 25 (IMC, also called "update step') from the second sequence 17 to the first sequence 15 is necessary. This information, which is implicitly transmitted in a low pass sequence 27 (L-frame or low pass frame) in the MCTF method, is produced separately in the proposed method by carrying out a subtraction between the low pass sequence 27 and the first sequence 15, wherein a socalled delta low pass sequence 29 (ö-L-frame) is provided. In each case at one particular moment a second transmission 31 is taking place during which the delta low pass sequence 29 together with the high pass sequence 1 5 and the motion information are transmitted to the receiver. Since the delta low pass sequence 29 in the receiver is not required for reconstruction of the first sequence 15 but only for reconstruction of the second sequence 17 there is no additional encoding delay as observed with the MC IF method shown iii Figure 1.

The process of the motion-compensated temporal filtering with delta low pass sequences 29 is shown in Figure 2 for a first stage of temporal disassembly. Further temporal disassembly steps are to be effected analogously to the first disassembly step. This method is suitable and is provided for encoding of any type of data which exist as sequences 15. 17. in particular 01 medial data such as video data, multimedia data and also audio data.

Figure 3 shows a diagram ofa hierarchical construction of list elements by means of which partial data are managed in the preferred embodiment of the proposed method. This diagram includes a data stream 33 illustrated in a sectional manner as a partial datum, which is also designated as a bit stream, having a so-called file header 35, a list 37 for a first access point and a list 39 for a second access point. Between these two lists 37, 39 for the access points all the encoded data 41, which also include sequences, of the first access point are provided within the data stream 33. As an enlargement of the list 37 for the first access point shows, this list includes a first pointer 43 to lists for a first temporal resolution and a second pointer 45 to lists for a second temporal resolution. Furthermore, the list 37 of the first access point has two blocks 47, 49, also shown enlarged. The first block 47 includes a third pointer 51 to encoded data of a first temporal and a first local resolution and a fourth pointer 53 to encoded data of a first temporal and second local resolution. These two pointers 51, 53 point, as shown by the broken arrows, to corresponding encoded data 41 of the first access point. The second block 49 includes a fiflh pointer 55 to encoded data of a second temporal and first local resolution and a sixth pointer 57 to encoded data of a second temporal and second local resolution. These two pointers 55, 57 also point, as shown by the broken arrows, to corresponding encoded data 41 of the first access point.

The diagram in Figure 3 describes the preferred embodiment of the method using the example of scalable video encoding (MCTF). For scalable audio encoding the embodiment of the method can be applied in a corresponding manner, wherein a frame frequency is replaced by the so-called "audio sample frequency" and a local resolution or local subbands is (are) replaced by an audio bandwidth or scale bands.

The method is based on the MP4 file format which is derived from the ISO Base Media File format. These formats divide the data stream 33 or the partial datum into so-called atoms or sequences which represent the smallest access units to the encoded data 41. For audio encoding these are audio frames, for video encoding they are the individual pictures. Using the pointers 43, 45, 51, 53, 55, 57 and by means of the hierarchically managed lists 37, 39 in the case of the present method a relative position is respectively allocated to individual atoms or data 41 within the data stream 33, whereby effective access is possible to the atoms for the respective temporal, local and qualitative resolution.

A first level of the hierarchy within the so-called access points constitutes the temporal scaling which permits addressing of encoded data 41 for reconstruction with a different frame frequency in the data stream 33. Depending on the managed setting of the MCTF in the encoder, grading of the frame frequency by whole number factors, e.g. two or three, is possible. In the case of the proposed method a pointer 43, 45, 51, 53, 55, 57 to the beginning of a further list is stored in the file format for each grading process of the temporal resolution, this further list containing information about the local resolution steps appertaining to this temporal resolution.

This list of local resolution steps contains pointers 43, 45, 51, 53, 55, 57 to the encoded data 41 which are required to transmit and decode the respective local resolution step. Alternatively this list can also consist of pointers 43, 45, 51, 53, 55,57 to further lists which store information for selection of specific data groups for different quality levels. This may be necessary since, for certain quality levels, other combinations of quantized coefficients of the local resolution steps lead to a subjectively higher quality ("rate distortion optimisation"). The information as to whether the pointers 43, 45, 51, 53, 55, 57 in the list of local resolution steps relate directly to encoded data or to further quality level lists, is stored as additional information in the file format.

An application which, with the aid of the proposed method, is intended to extract a partial data stream in accordance with a certain combination of local, temporal and qualitative resolution, must, in accordance with the selected combination, merely search the hierarchical lists 37, 39 for the access points and directly obtains the pointers 43, 45, 51, 53, 55, 57 to the relevant encoded data. In so doing it is not necessary to search through all the data.

Figure 4 shows a diagram relating to an embodinient of the method concerning transmission of medial data or a program 59 to a user in different quality levels. The program 59 can InLJUJe audiovisual data, multimedia data or audio data, in the latter case the method serves in particular to provide so-called partially encrypted pay-per-program radio or television in which, in paral!e with freely receivable programming, it is possible to receive programs or services for which a charge must he paid and which usually do riot include advertising.

The program 59 is divided by a provider with a scalable encoder 61 into three data streams 63, 65, 67 or partial data which are transmitted in a parallel or simultaneous manner on one channel.

It is proposed that a basic data stream 63 provides the program 59 free of charge and in low quality. A first receiver 69 for free services is designed only to receive this basic data stream 63. An encrypted first additional data stream 65 supplies further partial data to the program 59.

This first additional data stream 65 can be decrypted using a second receiver 71 for services of medium quality. This second receiver 71 can combine the basic data stream O3 and the hr additional data stream 65 and therefore reproduce the program 59 largely in a medium quality Partial data within a second, also encrypted, additional data stream 67 for high quuIii ate to)C reproduced only by a third receiver 73 for services of high quality in return for correspondlngl\ high charges. Since the third receiver 73 can also receive the basic data stream 63 and the tirsi additional data stream 65 it is possible using this third receiver 73 to provide or produce in audiovisual form the program 59 in its entirety.

If absent at a specific transmission time, an interested user can prepare a receiving device and effect storage of the program 59 as has generally been the case thus far, In so doing, e.g in the case of a rather low level of interest or where it is desired to view the material only once, he selects the basic data stream 63 in non-encrypted basic quality and accordingly does not pay for this service. If, after or even during viewing of the recording, the user becomes interested in storing it in a higher quality be orders - for an appropriate charge - the clearance of a higher quality level and thus the additional data streams 65, 67 from the provider so that complete high-quality reproduction of the program 59 is possible.

Storage of the transmitted program 59 may also be possible only after payment of a certain minimum charge. Below this amount ii is only possible to see or listen to the program live. The possibility of repeated viewing lapses after a certain time so that recordings of the program 5o) become invalid or unusable.

A further possibility is that of transmitting the additional data streams 65, 67 for the higher quality to the user at the same time but separately encrypted, and of permitting decoding and thus display of the higher quality only after payment of a charge. In this case it is feasible to combine all three receivers 69, 71, 73 in one device or in a video recorder. Clearance or decryption of the additional data streams 65, 67 can then be effected by a suitable key, e.g. a card or password, possibly even at short notice.

The procedure illustrated in Figure 4 is provided for scalable medial data which exist in the form of sequences. An additional outlay for separate encoding and an additional transmission bandwidth or additional storage space is thus avoided. Depending on the level of the charges which are paid by a user for the program 59 which requires registration or payment of a charge, the program 59 is provided in the respective quality by appropriate reconstructing combination of the data streams 63, 65, 67.

Claims

A method for handling scalable medial data which exist in the form of sequences, wherein one sequence is respectively allocated to at least one partial datum under consideration of certain parameters, wherein partial data are managed in accordance with the parameters and S at least one partial datum is transmitted to a receiver in dependence upon at least one parameter.
2 A method as claimed in claim 1, wherein provision is made that within the at least one partial datum to be transmitted at least one first sequence and at least one second sequence follow each other, - wherein the at least one first sequence is directly encoded and transmitted at a first moment, and - wherein by means of motion compensation of the at least one first sequence to the at least one second sequence a high pass sequence is generated, by means of inverse motion compensation from the high pass sequence back to the first sequence a low pass sequence is generated and by carrying out a subtraction between the low pass sequence and the first sequence a delta low pass frequency is generated, wherein the high pass sequence and the delta low pass sequence are transmitted at a second moment.
3 A method as claimed in claim 1 or 2, which is carried out for at least one temporal disassembly step as a parameter.
4 A method as claimed in any of claims I to 3, wherein the transmitted sequences are received by the receiver and the at least one transmitted partial datum is reconstructed by decoding.
5 A method as claimed in any of claims Ito 4. wherein the partial data are managed via pointers which are chained in hierarchical lists, wherein the hierarchical its ar p1 u \ IJL'd accordance with the parameters.
6 A method as claimed in claim 5, wherein at least one list consists of pointers to further lists, in which items of information are stored as parameters for selection of specific groups of' partial data for different qualitative disassembly steps.
7 A method as claimed in any of the preceding claims, by means of which a program which can be reconstructed from partial data is transmitted to the receiver, wherein, in dependence upon different qualitative disassembly steps, at least one first partial datum is provided freely and at least one second partial datum is provided in encrypted form to the receiver, wherein the at least one second encrypted partial datum is decrypted by the receiver using a key provided by a provider.
8 A system for handling scalable medial data which exist in the form of sequences, which is formed in such a way as to respectively allocate a sequence to at least one partial datum under consideration, of parameters, to manage partial data in accordance with the parameters and to transmit at least one partial datum to a receiver in dependence upon at least one parameter.
9 A computer program having program encoding means to carry out all steps of a method in accordance with any of claims 1 to 7, when the computer program is executed on a computer or corresponding computing unit, in particular a system as claimed in claim 8.
A computer program product having program encoding means which are stored on a computer-readable data support in order to carry out all steps of a method as claimed in any of claims I to 7, when the computer program is executed on a computer or corresponding computing unit, in particular a system as claimed in claim 8.
11 A method for handling scalable medial data which exist in the form of sequences, substantially as hereinbefore described, with reference to the Fogs. 2 to 4 of the accompanying drawings.
12 A system for handling scalable rneidca! data which exist in the form of sequences.

substantially as hereinbefore described with reference to and as illustrated in the accomapnying drawings.