HK1094071B - Method and apparatus for modeling film grain patterns in the frequency domain - Google Patents

Description

Method and apparatus for modeling film grain patterns in the frequency domain

This application is based on 35U.S. c.119(e), claiming priority from U.S. provisional patent application 60/498,945, filed on 29/8/2003, the teachings of which are incorporated herein by reference.

Technical Field

The present invention relates to a technique for modeling film grain patterns in the frequency domain.

Background

Motion picture film typically includes signal dependent noise, often referred to as film grain, that results from the exposure and development processes of photographic film. This noise creates a characteristic quasi-random pattern or texture due to the physical granularity of the photographic emulsion. Alternatively, signal dependent noise may occur as a result of subsequent editing of the image. The grain pattern may be simulated for video compression purposes.

The ITU-T h.2641 MPEG-4 AVC video compression standard has accepted the contents of the film grain SEI (supplemental enhancement information) message in its "fidelity range Extensions" attribute. The film grain SEI message conveys a series of parameters that allow film grain simulation at the receiver. For the ITU-T H.2641 MPEG-4 AVC compression standard, the parameters in the SEI message can be specified according to two different models: an autoregressive model and a frequency filtering model. The two models represent film grain pattern (size and shape), intensity and color related characteristics through different sets of parameters for different brightness levels. In particular, the frequency filtering model characterizes film grain patterns by specifying a set of cut-off frequencies that define a 2D band-pass filter in the frequency domain. Note that ITU-T h.2641 MPEG-4 AVC only standardizes the syntax required for transmission of the cut-off frequency, but does not provide a method for calculating the cut-off frequency for video sequences with film grain.

Therefore, there is a need for the following techniques: allows automatic modeling of film grain patterns in the frequency domain, as specified by the frequency domain filtering model in the ITU-T h.2641 MPEG-4 AVC compression standard. The results of this technique can be used in an automatic film grain modeling application or as an initialization step for a film grain assisted modeling process.

Disclosure of Invention

Briefly, in accordance with a preferred embodiment, there is provided a method for modeling (i.e., characterizing) film grain patterns in the frequency domain. The method comprises the steps of (1) transforming a set of homogeneous film grain samples received as input to the process to the frequency domain, thereby producing a set of transform coefficients having a particular pattern; (2) analyzing a pattern created by the transform coefficients; and (3) estimating the cut-off frequency of the 2D frequency filter by filtering the random noise, the cut-off frequency being effective to emulate the pattern of transform coefficients. The cut-off frequency established with this method can be conveyed in SEI messages to enable film grain simulation and reinsertion at the decoder, according to the ITU-T h.2641mpeg-4 AVC standard.

Drawings

Fig. 1 illustrates, in flow chart form, the steps of a method for representing film grain patterns in accordance with the present principles; and

fig. 2 illustrates, in flow diagram form, a variation of the film grain characterization method of fig. 1.

Detailed Description

Fig. 1 illustrates, in flow chart form, the steps of a method in accordance with the present principles for modeling film grain patterns in the frequency domain when a series of film grain samples representing homogeneous film grain patterns are received. As described in more detail later, the method of the present principles parameterizes the pattern of the input sample by analyzing the size and shape of the structures forming the grain. These same type film grain samples are typically associated with similar luminance values measured on the film image, since grain can be formed differently depending on film exposure. The film grain samples at the processing input can be any set (or sets) of adjacent pixels that retain information about the size and shape of the film grain. In the illustrated embodiment, it is assumed for simplicity that the film grain samples are arranged in blocks of N × N pixels, with the particular transform being implemented based on the DCT of the N × N pixel blocks, although other transforms, such as fast fourier transforms, could be equally applicable.

The method of the present principles assumes that the pair of relationships occurs at I according to_grain[x，y，c]The film grain in (1) is modeled:

I_grain[x，y，c]＝I_{without grain}[x，y，c]+G[x，y，c] (1)

where G x, y, c represents the simulated grain at pixel coordinates (x, y) for color component c. Calculate G [ x, y, c ] as:

G[x，y，c]＝p* Q[x，y，c]+u* G[x，y，c-1] (2)

where parameter p is the standard deviation of the random noise and parameter u models the cross-color correlation between different color components. More specifically, the term Q [ c ] includes a two-dimensional random field generated by filtering a block b of N M random values, which is generated with a normalized Gaussian distribution N (0, 1). In a particular embodiment, the band-pass filtering of block b may be performed in the frequency domain by the following three steps:

step 1: transformation of

B＝DCT_N×M(b)

Step 2: frequency filtering

for(y＝0；y＜N；y++)

for(x＝0；x＜M；x++)

if((x＜LOW_HF && y＜LOW_VF)||

x＞HIGH_HF||y＞HIGH_VF)

B[x，y]＝0；

Where LOW _ HF and LOW _ VF are the LOW level and vertical cutoff frequencies, respectively, and HIGH _ HF and HIGH _ VF are the HIGH level and vertical cutoff frequencies, respectively. When mapping the film grain image in the frequency domain, the cut-off frequency defines the boundary between the retained and filtered coefficients and is used to characterize the size of the grain.

And step 3: inverse transformation

b’＝IDCT_N×M(B)

Finally, Q [ c ] is formed by incorporating the filtered block b' into the composite image. Low pass filtering of block transitions reduces the possible "blockiness". Although M and N may take any value, square blocks of 16 × 16, 8 × 8, or 4 × 4 are most suitable in practice. It should also be noted that in steps 1 and 3, other transforms, such as Fast Fourier Transforms (FFTs), may be substituted for the DCT process.

By these principles, modeling the film grain pattern is equivalent to extracting the cut-off frequencies LOW _ HF, LOW _ VF, HIGH _ HF, and HIGH _ VF that represent the characteristics of the bandpass filter in the frequency domain.

When step 101 is performed, the principle of the method starts, wherein each block of nxn pixels is processedThe discrete cosine transform, and then the resulting N x N coefficient array is stored during step 102. During step 103, a check is made to determine if there is a need for more blocks with film grain samples in order to get more coefficients for storage. Typically, all blocks of film grain samples available at the input are transformed. However, to reduce memory requirements or computational load, processing may be stopped after a certain number of blocks have been transformed. After a sufficient number of transformed blocks have been stored, step 104 is performed in which an average block (B) is calculated by averaging the coefficients from all the stored blocks_mean). Let K be the number of blocks stored, the position [ x, y ] is done as follows]The averaging process for the coefficients:

next, steps 105 and 106 are typically performed in parallel. During step 105, by pairing B according to the following relationship_meanAveraging the N frequency coefficients of each row to calculate a horizontal mean vector B_H：

In a particular embodiment, the effect of the DC coefficient may be avoided when averaging the first row according to the following relation:

during step 106, by pairing B according to the following relationship_meanAveraging the N frequency coefficients of each column, and calculating a vertical mean vector:

in certain embodiments, the effect of the DC coefficient may be avoided when averaging the first column according to the following relationship:

from the frequency vector, horizontal and vertical cutoff frequencies are selected during steps 107 and 108, respectively, to estimate the film grain size. As shown in fig. 1, steps 107 and 108 are typically performed in parallel. The horizontal cut-off frequency selection is performed during step 107 in the following manner. First, the components in the horizontal mean vector are low-pass filtered to avoid spurious peaks. In the illustrated embodiment, this low pass filtering of the horizontal mean vector is performed by convolving the mean vector with an impulse response h [ n ] according to the following relationship:

for example, one may have a coefficient w according to the following relationship₀、w₁And w₂The 3-tap linear filter of (2) is applied to each coefficient:

B’_H[n]＝w₀·B_H[n-1]+w₁·B_H[n]+w₂·B_H[n+1]，0≤n≤N-1(7)

it can be observed that in order to apply filtering to the edges of the mean vector B, it is necessary to fill (pad) the original mean vector in order to define the samples for n < 0 and n > 1.

Next, B 'is calculated by averaging its components according to the following relationship'_HAverage value of (d):

then, vector B 'is added'_HIs represented as a curve and is calculated to have a mean value of B'_HThe intersection point of (a). If a single crossover point is found, then B 'is selected'_HThe index n of the nearest component in (a) is taken as the value of the horizontal high cut-off frequency; the horizontal low cut-off frequency is assumed to be 0. If two intersections are found, the index of the closest component is found for each intersection. The lowest value corresponds to a low level cut-off frequency and the highest value corresponds to a high level cut-off frequency. If more than two intersections are obtained, no spatial correlation is detected. Assuming that the horizontal low cutoff frequency is 0 and assuming that the horizontal high cutoff frequency is N-1, this indicates that a film grain simulation function without frequency filtering is required to mimic the original grain.

During step 108, the same procedure as described for selecting the horizontal cut-off frequency is performed in order to utilize the vertical frequency vector B_vThe vertical cut-off frequency is selected. Upon completion of steps 107 and 108, the method of fig. 1 produces four cut-off frequencies (LOW _ HF, LOW _ VF, HIGH _ HF, and HIGH _ VF) that are characteristic of the size and elongation of the particle. Elongated particles appear when LOW _ HF ≠ LOW _ VF and/or HIGH _ HF ≠ HIGH _ VF.

FIG. 2 shows aAlternative particle modeling methods, in which the particles may be restricted to circular shapes. This implies that the horizontal and vertical cut-off frequencies remain the same. The method of fig. 2 includes a number of steps in common with the method of fig. 1. Therefore, similar reference numerals are used in fig. 2 as in fig. 1 to describe similar steps. The method of fig. 2 differs from the method of fig. 1 in that the vertical and horizontal frequency vectors (B) are aligned during step 109 of fig. 2_HAnd B_v) Averaging is performed to create a single frequency vector (B). Then, the same procedure is performed during steps 107 and 108 of fig. 2, as was performed in steps 107 and 108 of fig. 1, to estimate the low and high cut-off frequencies.

The foregoing describes a technique for modeling film grain patterns in the frequency domain.

Claims (14)

1. A method for modeling film grain, comprising the steps of:

transforming the film grain sample set to the frequency domain;

storing each set of coefficients obtained by such a transformation;

analyzing the transformation coefficient; and

the cut-off frequency of the filter that can simulate the distribution of the transform coefficients is estimated.

2. The method according to claim 1 wherein the film grain samples include at least one set of adjacent pixels that retain information about film grain shape and size.

3. The method of claim 1, further comprising the step of: at least one cut-off frequency is transmitted as supplementary information.

4. The method of claim 1, wherein the cutoff frequency is estimated by filtering random noise in the frequency domain.

5. The method of claim 1 wherein the film grain samples are processed in blocks of N x N pixels.

6. The method of claim 5, wherein the step of analyzing the pattern created by the transform coefficients further comprises the steps of:

calculating a mean block of nxn transform coefficients by averaging the transform coefficients from all the blocks stored after the transform of each nxn block of pixels;

defining horizontal and vertical mean vectors for each of the N components by averaging the mean blocks of the nxn coefficients along rows and columns, respectively;

representing the horizontal and vertical mean vectors as separate curves; and

the horizontal and vertical cut-off frequencies are generated from the curves represented by the horizontal and vertical mean vectors, respectively.

7. The method of claim 6, further comprising the step of: the at least one mean vector is low-pass filtered.

8. The method of claim 6, wherein one of the horizontal and vertical cutoff frequencies is generated based on an intersection in a curve representing a respective one of the mean horizontal and vertical vectors, respectively

9. The method of claim 6, wherein each of the low and high level and vertical cutoff frequencies are generated from first and second intersections in curves representing mean horizontal and vertical vectors, respectively.

10. The method of claim 5, wherein the step of analyzing the pattern created by the transform coefficients further comprises the steps of:

calculating a mean block of nxn transform coefficients by averaging the transform coefficients from all the stored blocks after the transform of each block of pixels;

defining horizontal and vertical mean vectors for each of the N components by averaging the mean blocks of the nxn transform coefficients along rows and columns, respectively;

averaging the horizontal and vertical mean vectors into a single mean vector;

representing the mean vector as a curve; and

horizontal and vertical cut-off frequencies are generated from the curve represented by the mean vector.

11. The method of claim 10, further comprising the step of: and low-pass filtering the average vector.

12. The method of claim 10, wherein one of the horizontal and vertical cutoff frequencies is generated based on an intersection in a curve representing a respective one of the mean horizontal and vertical vectors, respectively.

13. The method of claim 10, wherein each of the low and high levels and the vertical cutoff frequency are generated from first and second intersections in curves representing mean horizontal and vertical vectors, respectively.

14. The method of claim 1 further comprising the step of receiving a set of film grain samples; and in the method, the film grain samples include at least one set of neighboring pixels that retain information about film grain shape and size; the coefficients form a pattern and the formed pattern is analyzed; and estimating a cutoff frequency of a 2D band-pass filter capable of simulating a pattern of transform coefficients by filtering the random noise in the frequency domain.