CN101199015A - Recording medium, device for mixing audio data and method thereof - Google Patents

Abstract

Disclosed is a recording medium, an apparatus for mixing audio data, and a method thereof, thereby providing various sounds according to content variations provided by a content provider in a manner that dynamically mixes various audio streams provided by an external input signal and/or a high-density recording medium. The invention includes: step (a), decoding a main audio stream and an auxiliary audio stream, each of the main audio stream and the auxiliary audio stream comprising at least one or more channels; and step (b), mixing the decoded main audio stream and the auxiliary audio stream together, step (b) comprising: step (b1), applying level control to the decoded auxiliary audio stream using a command set defined according to each channel of the auxiliary audio stream; and step (b2), adding the level-controlled auxiliary audio stream to the decoded main audio stream.

Description

Recording medium, device for mixing audio data and method thereof

technical field

The present invention relates to audio data mixing, and more particularly to a recording medium, an apparatus for mixing audio data and a method thereof.

Background technique

In general, optical discs capable of recording large-scale data are widely used as recording media. Recently, in order to store high-quality video data and high-quality audio data for a long period of time, new high-density recording media such as Blu-ray Discs (hereinafter abbreviated as BD) have been developed.

A high-density recording medium, which is a next-generation recording medium technology, is a next-generation optical recording scheme that provides data significantly superior to conventional DVDs. In addition, many efforts have been made in researching and developing high-density recording media and other digital devices.

An optical recording/reproducing apparatus applying the specification of a high-density recording medium has begun to take shape. However, due to imperfect specifications of high-density recording media, there are many difficulties in the perfect development of optical recording/reproducing devices.

Specifically, in the process of reproducing audio signals contained in an external input signal or in a high-density recording medium, since a better method of mixing audio has not been proposed and formed, the high-density recording medium based on an optical recording/reproducing device There are many limiting factors to full-fledged development.

Contents of the invention

Accordingly, the present invention is directed to a recording medium, an apparatus for mixing audio data, and a method thereof that substantially obviate one or more problems due to limitations and disadvantages of the related art.

An object of the present invention is to provide a recording medium, an apparatus for mixing audio data, and a method thereof, whereby various audio data provided by an external input signal and/or a high-density recording medium are dynamically mixed according to changes in content provided by a content provider. Various sounds are provided in the form of audio streams.

Another object of the present invention is to provide a recording medium, an apparatus for mixing audio data and a method thereof, whereby the audio data is mixed using a command set stored in the recording medium.

Another object of the present invention is to provide a recording medium, an apparatus for mixing audio data, and a method thereof, whereby interactive audio is mixed using metadata stored in the recording medium.

A part of the other advantages, objectives and features of the present invention will be clarified in the following description, and another part of the content will become apparent to those skilled in the art by carefully reading the following content or learned from the practice of the present invention. The objectives and other advantages of the invention may be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.

To achieve these objects and other advantages and in accordance with the objects of the present invention, as represented and broadly described herein, a method of mixing audio data according to the present invention comprises: step (a), decoding the primary audio stream and the secondary audio stream, the primary Each of the audio stream and the auxiliary audio stream comprises at least one or more sound channels; and step (b) of mixing together the decoded main audio stream and the auxiliary audio stream, step (b) comprising step (b1) using the A command set defined per channel of the auxiliary audio stream applies level control to the decoded auxiliary audio stream and step (b2) adds the level controlled auxiliary audio stream to the decoded main audio stream.

Step (b-1) further comprises the step of applying pan control to the decoded auxiliary audio stream using the command set.

The method also includes: step (c), mixing the mixed primary audio stream and auxiliary audio stream with interactive audio including at least one or more channels. Additionally, step (c) includes the step (c1) of applying level control to the interactive audio using command sets defined per channel of the interactive audio and the step (c2) of combining the level-controlled interactive audio with the mixed primary audio stream Added to the auxiliary audio stream.

Step (c1) also includes the step of applying panning control to the interactive audio using the command set.

The method also includes the step (d) of mixing the mixed primary and secondary audio streams with interactive audio comprising at least one or more channels.

Additionally, step (d) includes: step (d1) applying pan control and/or level control to the interaction audio using mixing metadata defined in accordance with each channel of the interaction audio; and step (d2) converting the controlled interaction Audio is added to the mixed primary and secondary audio streams.

In another aspect of the present invention, a method of mixing audio data includes: step (a) decoding a primary audio stream and an auxiliary audio stream, each of the primary audio stream and the auxiliary audio stream comprising at least one or more sound channels; and step (b) mixing the decoded primary and secondary audio streams. Additionally, step (b) comprises the step (b1) of applying panning and/or level control to the decoded auxiliary audio stream using mixing metadata defined per channel of the auxiliary audio stream and the step (b2) of controlling The auxiliary audio stream and the decoded main audio stream are added. Mixed metadata is included in the auxiliary audio stream.

In another aspect of the present invention, an apparatus for mixing audio data includes: a main audio decoder for decoding a main audio stream including at least one or more channels; an auxiliary audio decoder for decoding a main audio stream including at least one or more channels an auxiliary audio stream of a plurality of channels; and a first audio mixer for applying level control to the auxiliary audio stream and level-controlled auxiliary audio using a command set defined in accordance with each channel of the auxiliary audio stream The decoded primary audio stream is mixed with the decoded secondary audio stream by adding the decoded primary audio stream to the decoded primary audio stream.

The first audio mixer can apply pan control to the auxiliary audio stream using the command set.

The apparatus still further includes a second audio mixer for applying level control to the interactive audio using a command set defined for each channel of the interactive audio and combining the controlled interactive audio with the mixed main Addition of the audio stream and the auxiliary audio stream mixes the mixed primary audio stream and the auxiliary audio stream with interactive audio comprising at least one or more channels.

The second audio mixer can use the command set to apply panning control to the interactive audio.

The apparatus also includes a second audio mixer to apply pan and/or level control to the interactive audio using mixing metadata defined for each channel of the interactive audio and to control the controlled interactive The mixed primary and secondary audio streams and interactive audio comprising at least one or more channels are mixed in a manner that audio is added to the mixed primary and secondary audio streams.

In another aspect of the present invention, an apparatus for mixing audio data includes: a main audio decoder for decoding a main audio stream including at least one or more channels; an auxiliary audio decoder for decoding a main audio stream including at least one or more channels an auxiliary audio stream of a plurality of channels; and a first audio mixer for applying panning and/or electric power to the auxiliary audio stream using mixing metadata defined according to each channel of the auxiliary audio stream The decoded primary audio stream and the decoded secondary audio stream are mixed in a format that controls the level and adds the controlled secondary audio stream and the decoded primary audio stream.

Mixing metadata can be included in the auxiliary audio stream.

In another aspect of the present invention, a recording medium includes: a primary audio stream containing at least one or more channels; an auxiliary audio stream containing at least one or more channels; and an auxiliary audio stream for mixing with the primary audio stream. A set of commands for level control of the audio stream, defined per channel of the auxiliary audio stream.

The recording medium also includes a command set for panning control of the secondary audio stream mixed with the primary audio stream. Also, the command set is defined per channel of the auxiliary audio stream.

The recording medium also includes interactive audio including at least one or more channels and mixing elements for panning and/or level control of an auxiliary audio stream mixed with the mixed primary and auxiliary audio streams data. In addition, mixing metadata is defined per channel of interactive audio.

In another aspect of the present invention, a recording medium includes: a primary audio stream containing at least one or more channels; an auxiliary audio stream containing at least one or more channels; and an auxiliary audio stream mixed with the primary audio stream Mixing metadata for stream panning and/or level control. In addition, mixing metadata is defined per channel of the auxiliary audio stream.

Mixing metadata is included in the auxiliary audio stream.

With the present invention, it is possible to dynamically mix audio data according to changes in content. Therefore, the present invention can provide a plurality of different sounds to the user. It is to be understood that both the foregoing general description and the following detailed description of the invention are exemplary and explanatory and are intended to provide further explanation of the invention as claimed.

Description of drawings

The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this application, illustrate embodiments of the invention and together with the description serve to explain the principles of the invention. In the attached picture:

Fig. 1 is the schematic diagram that facilitates the principle understanding of the present invention;

2 is a block diagram of an optical recording/reproducing apparatus according to the present invention;

Figure 3 is a schematic diagram of an audio mixing model according to the present invention;

Fig. 4 is the schematic diagram of the audio mixing of the first audio mixer according to the present invention;

Fig. 5 is the schematic diagram of the audio mixing of the second audio mixer according to the present invention;

Figure 6 is a block diagram of a mixer according to one embodiment of the invention;

Fig. 7 is a flowchart of an audio mixing method according to one embodiment of the present invention;

Fig. 8 is a flowchart of an audio mixing method according to another embodiment of the present invention.

Detailed ways

Reference will now be made in detail to the preferred embodiments of the invention, examples of which are illustrated in the accompanying drawings. Wherever possible, the same reference numbers are used throughout the drawings to refer to the same or like parts.

First, for ease of description, the present invention takes an optical disc, more specifically a "Blu-ray Disc (BD)" as an example of a recording medium. However, it is obvious that the technical idea of the present invention is also applicable to other recording media.

In the present invention, "memory" is a storage device provided in the optical recording/reproducing apparatus shown in FIG. 1 and means a unit in which a user can randomly store necessary information and data for use. In practice, currently commonly used memories include "hard disk", "system memory", "flash memory", etc. without limiting the scope of the present invention.

Specifically, "storage" is used as means for storing data associated with a recording medium such as a Blu-ray Disc. The data associated with the recording medium stored in the memory generally includes data downloaded from the outside.

Furthermore, it is obvious that part of the license data read directly from the recording medium or the system-generated data associated with the recording and reproduction of the recording medium (such as metadata, etc.) can be stored in the memory.

For convenience of description of the present invention, data recorded in a recording medium is referred to as "original data" and data associated with the recording medium among data stored in a memory is referred to as "additional data".

In the present invention, "panning" refers to positioning auxiliary audio signals or interactive audio in a multi-channel sound field of stereo or main audio. Main audio, auxiliary audio and interactive audio will be described in Figure 3 below.

FIG. 1 is a schematic diagram to facilitate the understanding of the principle of the present invention, in which the combined use between an optical recording/reproducing apparatus 10 and peripheral devices is exemplarily shown.

Referring to FIG. 1, an "optical recording/reproducing apparatus" 10 according to the present invention allows recording or reproducing an optical disc according to various specifications. In addition, the optical recording/reproducing apparatus 10 can also be designed to record/play back an optical disc of a specific specification (eg, BD or DVD).

In addition, it is obvious that the "optical recording/reproducing apparatus" 10 includes a "drive" that can be loaded into a computer or the like.

The optical recording/reproducing apparatus 10 according to the present invention has a function of recording/playing an optical disc 30 and a function of receiving an external input signal, performing signal processing on the received signal, and delivering a corresponding image to a user via another external display 20 . In this case, there is no restriction on the external input signal. In addition, a DMB (Digital Multimedia Broadcasting) signal, an Internet signal, etc. may be one of external input signal representations. In the case of the Internet as a medium for easy access, specific data on the Internet can be downloaded via the optical recording/reproducing apparatus 10 for use.

Also, entities that provide content are collectively referred to as "content providers (CP)".

In the present invention, the content configuring a title represents data provided by the author of the recording medium.

Raw data and additional data are detailed below. For example, if a multiplexed AV stream of a specific title is recorded in an optical disc as originally recorded data and if an audio stream (such as English) different from that of the original data (such as Korean) is given as additional data on the Internet is output, a request to download an audio stream (such as English) as additional data on the Internet for reproduction together with the AV stream of the original data or a request to download only an audio stream (such as English) as additional data on the Internet for reproduction is given out for the user to choose. To fulfill these requests, it is necessary to adjust the association between original data and additional data and to provide a systematic method of managing/reproducing data according to user requests.

In addition, in the case of reproducing an audio stream of original data and an audio stream of additional data together, it is necessary to establish an apparatus and method for mixing streams.

In the above explanation, for the convenience of explanation, the signal recorded in the disc is called original data and the signal existing outside the disc is called additional data, which is determined according to the method of obtaining each data, but this is not the original Data or additional data is limited to specific data.

FIG. 2 is a block diagram of an optical recording/reproducing apparatus 10 according to the present invention.

Referring to Fig. 2, the optical recording/reproducing apparatus 10 according to one embodiment of the present invention basically includes: pickup 11, is used for reproducing the management information that comprises the original data and reproduction management file information that are recorded in the optical disc; Server 14, is used for Control the action of pick-up 11; Signal processor 13, is used for restoring the reproduced signal received from pick-up 11 to specific signal value, and said signal processor 13 modulates the signal to be recorded into a signal that can be recorded on the optical disc, The signal processor 13 delivers the modulated signal; and the microprocessor 16 controls the overall operation.

Additional data existing in places other than the optical disk is downloaded to the controller 12 by a user command or the like. The controller 12 allows storing downloaded data into the memory 15 and reproducing additional data in the memory 15 or original data in the optical disk according to user requests.

The playback system finally decodes the data under the control of the controller 12 and then provides the decoded data to the user. Specifically, the playback system 17 includes a decoder that decodes AV signals and a player model that determines a playback direction by interpreting user commands input via the controller 12 .

In addition, in order to perform the function of recording the signal in the optical disc, the AV encoder 18 converts the input signal into a signal of a specific format such as MPEG2 transport stream under the control of the controller 12, and then supplies the converted signal to the signal processing Device 13.

Fig. 3 is a diagram of an audio mixing model according to the present invention.

Referring to FIG. 3 , 'audio mixing' means mixing a main audio stream with an auxiliary audio stream and/or interactive audio.

The audio mixing model shown in Fig. 3 combines the main audio stream, the auxiliary audio stream and the interactive audio stream according to the mixing coefficient.

The model includes a pair of audio decoders D1 and D2 and a pair of audio mixers M1 and M2 for decoding and mixing respectively.

The content provider controls the audio mixing process using audio mixing control parameters such as P1, P2 and P3.

Main Audio is the audio stream with a high bit rate of the Main Audio program supplied to the Main Audio Decoder (D1). The main audio generally includes the soundtrack of the movie contained in the recording medium. However, the master audio can be stored in the memory 15 by being downloaded from the network. The necessary main audio transport stream (TS) is output from the transport stream according to the PID (Packet Identifier) in the PID filter 1 and then supplied to the main audio decoder 320a via the first buffer (B1) 310a.

Secondary audio is a low bitrate audio stream that provides supplemental audio that is synchronized and mixed with the main audio. The auxiliary audio may be choreographic notes or supplementary audio designed to be mixed with the main audio and generally exists in the memory of the optical recording/reproducing device. Necessary auxiliary audio TS is selected by the PID filter 2 and then supplied to the auxiliary audio decoder 320b via the second buffer (B2) 310b.

Interactive audio is LPCM audio activated by the application. The interactive audio is provided to the second audio mixer 330b via the third buffer (B3) to be mixed with the main audio. In the case where auxiliary audio is provided as part of the audio effect, the main audio is mixed with the interaction audio after being mixed with the auxiliary audio. The interactive audio stream may reside on a recording medium or memory. In addition, interactive audio streams are used to provide dynamic sounds associated with interactive applications, typically as key press sounds.

"D1" indicates the main audio decoder 320a. "D1" decodes the main audio stream into LPCM (Linear Pulse Code Modulation). "D1 (320a)" is configured to decode all channels present in the main audio track, or to downmix the main audio to fewer channels.

"D2" refers to the auxiliary audio decoder 320b. "D2 (320b)" decodes the auxiliary audio stream to LPCM and extracts the mixing metadata contained in the auxiliary audio stream, translates the extracted metadata into a mixing matrix format, and then passes the mixing matrix to the first audio mixer 330a . Metadata is used to control the blending process. The auxiliary audio decoder 320b is configured to decode all channels present in the auxiliary audio track or a 2-channel downmix contained in the auxiliary audio bitstream. Each decoded channel output from the auxiliary audio decoder 320b may be mixed with one or more channels output from the main audio decoder 320a.

A mixing matrix is created in response to mixing parameters, audio mixing is the ability to add two gain-controlled (level-controlled) audio streams. Audio mixing can be controlled in each audio mixer of an audio mixing model by using a mixing matrix. The mixing matrix includes two sets of coefficients: coefficients for each channel of the main stream ("Primary Audio Stream" in M1 and "Output Stream of M1" in M2) and coefficients for each channel of the auxiliary stream ("Auxiliary Audio Stream" in M1 330a) Stream" and "Interactive Audio" in M2 330b) coefficients for each channel.

Mixing metadata from the auxiliary audio stream can be a source of mixing parameters available to content providers. Additionally, command sets may be a source of mixed parameters available to content providers.

The command set is a set of programs to use the functions of the application program executable in the optical recording/reproducing apparatus. Through the command set, the interface with the function of the optical recording/reproducing device is configured. Through the command set, various functions of the optical recording/reproducing device can be used. A representative example of the command set is an application programming interface (hereinafter simply referred to as API). In the description below, the API is used as an example of the command set.

In addition, the command set is stored in a recording medium to be provided to the optical recording/reproducing apparatus. However, the command set may also be loaded in the optical recording/reproducing device at the time of manufacturing the optical recording/reproducing device.

The audio mixing model according to the present invention provides content providers with mixing control parameters (mixing parameters). In addition, the mixing parameters include parameters for pan control of the auxiliary audio stream, parameters for level control of the main and auxiliary audio streams, and parameters for pan/level control of the interactive audio stream.

In the following description for FIG. 3 and the like, "P1", "P2", and "P3" refer to mixing parameters. Additionally, the present invention is not limited to parameter names. In addition, it is obvious that other parameters may exist by being unified or separated according to functions.

Meanwhile, metadata on/off API can turn on/off the processing of audio mixing metadata from auxiliary audio stream. When metadata control is turned on, the P1 mix parameter provides secondary audio stream metadata control that supports gain (level) control of both the main and secondary audio and pan control of the secondary audio stream. When metadata control is off, the P1 mix parameter provides panning control of the auxiliary audio stream. P1 mixing parameters are provided from API or D2 320b. When metadata control is on, the P1 blend parameters come from the D2 320b. When metadata control is off, the P1 mix parameter comes from the API.

The P2 mix parameter provides level control for the primary and secondary audio streams. When metadata control is on, level control using a command set (such as API) is combined with metadata level control.

Additionally, the P3 mix parameter provides panning and/or level control of the interactive audio stream.

In the present invention, the command set is used simultaneously to pan the auxiliary audio and to control the level of the auxiliary audio stream.

FIG. 4 is a diagram of audio mixing of the first audio mixer according to the present invention, wherein the audio mixer M1330a pans and mixes auxiliary audio.

Referring to FIG. 4, the first audio mixer M1 30a mixes the decoded primary audio stream and the decoded auxiliary audio stream according to the mixing matrix 401. The audio mixing process according to the present invention is controlled by a set of commands. Thus, the source of mixing parameters P1, P2 and P3 comprises the command set. Additionally, audio mixer M1 330a is controlled by mixing parameters P1 and P2.

As shown in FIG. 3, the P1 mix parameter is switched between the API and the metadata from the auxiliary audio decoder 320b using the metadata on/off API. Specifically, the P1 mixing parameters may be provided from the API or from the auxiliary audio decoder 320b. The P2 mix parameters are provided from the command set.

The level control provided by the P2 mix parameter is applied to the mix matrix input provided by P1. Therefore, when metadata control is turned on, both metadata and command sets are used to control hybrid processing.

In the embodiment shown in Figure 4, the auxiliary audio stream is mono and the first audio mixer M1 provides a 5.1 channel output. Additionally, auxiliary audio metadata is turned off.

The system software in the optical recording/reproducing apparatus according to the present invention translates the pan control API and the level control API into a mixing matrix 401, and supplies the mixing matrix 401 to the first audio mixer 330a.

For panning of the auxiliary audio stream, the auxiliary audio stream is converted to five channels and the corresponding gain is applied to each channel before being summed with the main audio.

Therefore, according to the mixing matrix 401, the auxiliary audio becomes "Gs1xM, Gs2xM, Gs3xM, Gs4xM, Gs5x5" (402) and the main audio becomes "Gp1xL, Gp2xR, Gp3 ×C, Gp4×LFE, Gp5×Ls, Gp×Rs” (403). Additionally, the primary audio and secondary audio are summed together (404). The output value 404 summed by the first audio mixer M1 330a is provided to the second audio mixer 330b for mixing with the interactive audio.

In the embodiment shown in FIG. 4, "Gs" represents the gain of the auxiliary audio stream and "Gp" represents the gain of the primary audio stream. "M" means mono, "L" means left channel, "R" means right channel, "C" means center, "S" means surround, "Ls" means left surround, "Rs" means right surround , and "LFE" indicates the low-frequency effects channel.

Furthermore, the mixing matrix 401 is controlled by mixing parameters P1 and P2, i.e. P1 and P2 both define the mixing matrix 401 which in turn controls the first audio mixer M1 330a.

5 is a schematic diagram of audio mixing of the second audio mixer according to an embodiment of the present invention, wherein the second audio mixer M2 330b uses panning interactive audio and combines the panning interactive audio from the first audio mixer M1 The output streams of 330a are summed to perform a blending process.

Referring to Figure 5, the interactive audio is mono. The second audio mixer M2 330b mixes the output stream of the first audio mixer M1 330a with the interactive sound according to the mixing matrix 501.

In the present invention, the mixing of interactive audio is realized by mixing metadata. The mixing metadata is stored in the recording medium to be provided to the audio mixer or downloaded to the memory 15 together with the interactive audio data from an external source of the recording medium. The mixing matrix 501 can be generated from API or mixing metadata by the optical recording/reproducing device.

API and/or mixing metadata perform panning and/or level control of interactive audio. And, the panned and/or level controlled interactive audio 502 is summed with the output stream 404 of the audio mixer M1 330a to await output (503).

Although the auxiliary audio and the interactive audio are monophonic in the description of Figs. 4 and 5, it is obvious that the auxiliary audio and the interactive audio may be stereo or multi-channel streams.

In the case of stereo or multi-channel, auxiliary and interactive audio can be mixed with the main audio only by level control without panning. However, for smoother sound, it is possible to control the level (gain) after panning each channel contained in the audio stream.

In a case where the auxiliary audio and the interactive audio are multi-channel, a command set for the mixing process of the present invention is defined according to each channel of the auxiliary audio and the interactive audio. For example, where the auxiliary audio includes a channel "L" and a channel "R", then sets of panning and/or level control commands are defined separately for the channel "L" and the channel "R".

Figure 6 is a block diagram of an audio mixer according to one embodiment of the present invention.

Referring to FIG. 6, an audio mixer according to one embodiment of the present invention is a 5.1-channel mixer including two mixing blocks.

The first mixing block, i.e. the first audio mixer M1 330a mixes the primary audio stream and the secondary audio stream according to the mixing metadata contained in the secondary audio stream and/or the command set.

Additionally, a second mixing block, the second audio mixer M2 330b mixes the output of the audio mixer M1 330a with the interactive audio. The mixing process of the second audio mixer M2 330b is controlled by mixing metadata and/or command sets.

The gain and pan data contained in the auxiliary audio stream is periodically provided to the audio mixer at a rate significantly lower than the audio sampling rate. In order to prevent auditory artifacts (e.g. whistling noise during fades or pans, clicks due to sudden gain changes between adjacent audio samples, etc.), the gain value passed via the mixing metadata is passed through the control smoothing filter 610a , 610b and 610c are smoothed to the audio sample rate.

The decoded auxiliary audio stream is panned via the mixing API and/or a mixing matrix that translates into a "gain/pan LUT (look-up table)". The output level of the panned secondary audio stream is controlled and then summed with the main audio stream. In this case, the main audio stream is decoded data whose output level is controlled according to the mixing matrix.

The output value of the first audio mixer M1 330a is provided to the second audio mixer M2 330b for summing with the interactive audio. The interactive audio is panned and its output level is then controlled according to the mixing matrix generated from the mixing API and/or metadata before being summed with the output value of the first audio mixer M1 330a. The mixing result of the second audio mixer M2 330b is then provided to the user as-is or down-mixed.

The mixing of the primary and secondary audio streams takes place at predetermined times and is controlled by predetermined mixing parameters generated during the disc manufacturing process.

By using these mixing parameters correctly, the primary and secondary audio streams can be prevented from overloading the audio mixer. However, the addition of interactive audio in the second audio mixer M2 330b does not occur at the scheduled time. Therefore, the output of the second audio mixer M2 330b may be overloaded. To prevent overloading, a peak limiter/overload protection 620 is applied to the output value of the audio mixer.

FIG. 7 is a flowchart of an audio mixing method according to an embodiment of the present invention. Fig. 8 is a flowchart of an audio mixing method according to another embodiment of the present invention.

In Figures 7 and 8, the audio mixing is controlled using command sets. In the embodiment shown in FIG. 7 pan control and output level control are performed, whereas in the embodiment shown in FIG. 8 only the output level is controlled.

Referring to FIG. 7 , an AV stream of a clip of a transport stream provided from a recording medium and or provided from a memory is provided to an audio mixing model. The necessary transport stream is selected according to the PID from the PID filter to provide it to the audio decoder.

The main audio stream and the auxiliary audio stream are respectively decoded by the main audio decoder 320a and the auxiliary audio decoder 320b (s710).

The auxiliary audio stream is panned by the first audio mixer 330a and its level (gain) is controlled (S720). Panning and level control are performed by using a command set defined according to each channel contained in the auxiliary audio stream, such as API (Application Programming Interface). Auxiliary audio can be mono or multi-channel.

Furthermore, panning and level control can be performed by using mixing metadata defined per channel contained in the auxiliary audio stream.

The panned and level-controlled auxiliary audio stream is mixed with the main audio stream in the first audio mixer 330a (S730). As described in the description of FIG. 4, the output level of the main audio stream can be controlled according to the mixing matrix before being added to the auxiliary audio stream.

The output value of the first audio mixer 330a is provided to the second audio mixer 330b. The second audio mixer 330b pans the interactive audio and controls the level (gain) of the interactive audio using a command set defined according to each channel (S740). The panned and level-controlled interactive audio is mixed with the output value of the first audio mixer 330a to be provided to the user (S750). Interaction audio can be mono or multi-channel.

In case the auxiliary audio stream is not provided, the output value of the first audio mixer 330a becomes a value generated by the control level of the main audio stream. Therefore, if no secondary audio stream is provided, the interactive audio is mixed with the level-controlled primary audio stream. Likewise, if an auxiliary audio stream is provided and no interactive audio is provided, the result of mixing the main audio and the auxiliary audio is provided to the user.

Fig. 8 is a flowchart of an audio mixing method according to another embodiment of the present invention.

Referring to FIG. 8 , a command set defined according to each channel contained in the auxiliary audio stream is used for the same audio mixing process as in FIG. 7 . However, the embodiment shown in FIG. 8 differs from the embodiment shown in FIG. 7 in that only gain control is performed without panning in a case where the auxiliary audio and/or interactive audio is multi-channel.

First, as in the previous embodiment shown in FIG. 7, the main audio stream and the auxiliary audio stream are decoded to be provided to the audio mixer (S810).

However, if the auxiliary audio includes a plurality of channels, the first audio mixer 330a controls the output level of the auxiliary audio (S820). The first audio mixer 330a then adds it to the main audio (S830). In other words, the multi-channel auxiliary audio is not panned in the mixing process. The output value of the first audio mixer 330a is provided to the second audio mixer 330b to wait to be mixed with interactive audio.

In the case that the interactive audio is multi-channel, the level of the interactive audio is controlled according to a command set defined by each channel included in the interactive audio (S840), and the controlled interactive audio is combined with the first audio mixer 330a The output values are added to be mixed (S850). That is, in the embodiment shown in FIG. 8 , no panning is performed on the multi-channel interactive audio during the mixing process.

Additionally, since the present invention uses metadata for mixing interactive audio, metadata can be used for panning and output level control in the embodiments shown in FIGS. 7 and 8 . Specifically, metadata is converted into a mixing matrix (501) with or without a command set for level (gain) and/or pan control of interactive audio.

Therefore, the present invention allows various audio streams provided from external input signals and/or high-density recording media to be dynamically mixed, thereby providing users with more variable sounds according to content changes provided by content providers.

Industrial applicability

It will be apparent to those skilled in the art that various modifications and changes can be made to the present invention without departing from the spirit or scope of the invention. Thus, it is intended that the present invention cover all the modifications and variations of this invention provided they come within the scope of the appended claims and their equivalents.

Claims (19)

1. the method for a mixing audio data comprises:

Step (a), decoded main audio stream and auxiliary audio stream, each of main audio stream and auxiliary audio stream comprises at least one or a plurality of sound channel; And

Step (b) mixes decoded main audio stream and auxiliary audio stream, and step (b) comprising:

Step (b1), use applies level control according to the command set of each sound channel definition of auxiliary audio stream to decoded auxiliary audio stream; And

Step (b2), the auxiliary audio stream that level is controlled and decoded main audio stream addition.

2. the method for claim 1 is characterized in that, step (b-1) comprises that also the decoded auxiliary audio stream of utility command set pair applies the step that moves control.

3. the method for claim 1 is characterized in that, also comprises:

Step (c) is mixed the main audio stream that mixed and auxiliary audio stream with the interactive audio that comprises at least one or a plurality of sound channels, step (c) comprising:

Step (c1), use applies level control according to the command set of each sound channel definition of described interactive audio to described interactive audio; And

Step (c2), the interactive audio that level is controlled and main audio stream that has mixed and auxiliary audio stream addition.

4. method as claimed in claim 3 is characterized in that, step (c1) comprises that also the described interactive audio of utility command set pair applies the step that moves control.

5. the method for claim 1 is characterized in that, also comprises:

Step (d) is mixed the main audio stream that mixed and auxiliary audio stream with the interactive audio that comprises at least one or a plurality of sound channels, step (d) comprising:

Step (d1) is used mixing metadata according to each sound channel definition of described interactive audio that described interactive audio is applied and is moved and/or level control; And

Step (d2) is with controlled interactive audio and main audio stream that has mixed and auxiliary audio stream addition.

6. the method for a mixing audio data comprises:

Step (a), decoded main audio stream and auxiliary audio stream, each in main audio stream and the auxiliary audio stream comprises at least one or a plurality of sound channel; And

Step (b) is mixed decoded main audio stream and auxiliary audio stream, and step (b) comprising:

Step (b1) is used mixing metadata according to each sound channel definition of described auxiliary audio stream to apply to decoded auxiliary audio stream and is moved and/or level control; And

Step (b2) is with controlled auxiliary audio stream and decoded main audio stream addition.

7. method as claimed in claim 6 is characterized in that, described mixing metadata is comprised in the described auxiliary audio stream.

8. device that is used for mixing audio data comprises:

Main audio decoder, being used for decoding comprises the main audio stream of at least one or a plurality of sound channels;

The auxiliary audio frequency demoder, being used for decoding comprises the auxiliary audio stream of at least one or a plurality of sound channels; And

First Audio mixer, be used for auxiliary audio stream being applied level control and the auxiliary audio stream that level is controlled and the mode of decoded main audio stream addition, decoded main audio stream and decoded auxiliary audio stream are mixed to use according to the command set of each sound channel definition of auxiliary audio stream.

9. device as claimed in claim 8 is characterized in that, the described first Audio mixer utility command set pair auxiliary audio stream applies and moves control.

10. device as claimed in claim 8 is characterized in that, also comprises:

Second Audio mixer, described second Audio mixer is to use command set according to each sound channel definition of the interactive audio that comprises at least one or a plurality of sound channels interactive audio is applied level control and with the mode of controlled interactive audio with main audio stream that has mixed and auxiliary audio stream addition, main audio stream and the auxiliary audio stream that has mixed to be mixed with described interactive audio.

11. device as claimed in claim 10 is characterized in that, the described second Audio mixer utility command set pair interactive audio applies and moves control.

12. device as claimed in claim 8 is characterized in that, also comprises:

Second Audio mixer, described second Audio mixer moves and/or level control and with the mode of controlled interactive audio with main audio stream that has mixed and auxiliary audio stream addition to use to apply to interactive audio according to the mixing metadata of each sound channel definition of the interactive audio that comprises at least one or a plurality of sound channels, and main audio stream and the auxiliary audio stream that has mixed mixed with described interactive audio.

13. the device of a mixing audio data comprises:

Main audio decoder, being used to decode comprises the main audio stream of at least one or a plurality of sound channels;

The auxiliary audio frequency demoder, being used to decode comprises the auxiliary audio stream of at least one or a plurality of sound channels; And

First Audio mixer, described first Audio mixer moves and/or level control and with the mode of controlled auxiliary audio stream and decoded main audio stream addition to use according to the mixing metadata of each sound channel definition of auxiliary audio stream auxiliary audio stream to be applied, and decoded main audio stream is mixed with decoded auxiliary audio stream.

14. device as claimed in claim 13 is characterized in that, described mixing metadata is comprised in the auxiliary audio stream.

15. a recording medium comprises:

The main audio stream that comprises at least one or a plurality of sound channels;

The auxiliary audio stream that comprises at least one or a plurality of sound channels; And

Be used for the command set of the level control of the auxiliary audio stream that mixes with main audio stream, described command set defines according to each sound channel of auxiliary audio stream.

16. recording medium as claimed in claim 15 is characterized in that, also comprises the command set that moves control that is used for the auxiliary audio stream that mixes with main audio stream, described command set is according to each sound channel definition of auxiliary audio stream.

17. recording medium as claimed in claim 15, it is characterized in that, also comprise interactive audio, described interactive audio comprises the mixing metadata of moving of interactive audio stream that at least one or a plurality of sound channel and the main audio stream that is used for having mixed and auxiliary audio stream are mixed and/or level control

Wherein, described mixing metadata defines according to each sound channel of interactive audio.

18. a recording medium comprises:

The main audio stream that comprises at least one or a plurality of sound channels;

The auxiliary audio stream that comprises at least one or a plurality of sound channels; And

Be used for moving and/or the mixing metadata of level control of the auxiliary audio stream that mixes with main audio stream, described mixing metadata defines according to each sound channel of auxiliary audio stream.

19. recording medium as claimed in claim 18 is characterized in that, described mixing metadata is comprised in the auxiliary audio stream.

Images