TW201301199A - Edge-based video interpolation for video and image upsampling - Google Patents

Abstract

Edge-based interpolation for upsampling. One method may include determining an edge characteristic associated with an interpolation point, the edge characteristic having an edge magnitude and an edge angle; selecting an interpolation filter in response to the edge angle; and determining a pixel value at the interpolation point using the selected interpolation filter. Other embodiments include edge-based interpolation followed by an adaptive sharpening filter. The sharpening filter is controlled by the edge-based interpolation parameters that determine the pixels to be sharpened and the sharpening strength.

Description

Video and image magnification edge-based video interpolation

CROSS-REFERENCE TO RELATED APPLICATIONS RELATED APPLICATIONS [ s s s s s s s s s s s s s s s s s s s s s s s s s s s s s s s s s s s s s s s s s s s s s s s s s The priority of U.S. Provisional Patent Application Serial No. 61/535,353, filed on Sep. 15, 2011, which is incorporated herein by reference in its entirety, the entire entire entire entire entire entire entire entire entire entire entire entire entire entire entire entire entire entire entire entire entire entire entire entire all

In image and video processing, zooming in or zooming refers to the process of increasing the resolution of a digital image or video. In general, amplification can be found in most video players (such as VLC players, Windows Media Players, etc.). Online video hosting sites (such as YouTube, Hulu, DailyMotion, etc.) offer users the option to zoom in to full screen resolution. Many current TV and DVD players are beginning to be equipped with built-in amplification modules.
Amplification typically involves generating new image pixels from existing pixels. In the video frame, pixels are usually coherent with their neighboring pixels. This feature is used by most amplification methods to generate new pixels. In video or video transcoder pipelines, amplification is challenging because the input is low resolution video with coding artifacts.
The easiest way to zoom in is pixel copying, which simply involves copying pixels in columns and rows. However, this method can lead to severe blockiness. Blockiness and aliasing artifacts seen in pixel duplication can be mitigated by applying a low pass filter to the magnified frame. Some techniques for reducing blockiness include the application of low pass filters. Other techniques, commonly referred to as interpolation, may include averaging adjacent pixels to determine new pixel values. An interpolation technique utilizes "zero fill" followed by low pass filtering. This can be described as amplifying the size of the image by placing a zero-valued pixel at an intermediate position, and then applying a low-pass filter to the magnified image.
In addition, video that is typically found on video host sites is sometimes recompressed to a low bit rate and may therefore have low resolution. A common artifact in these videos is blurriness. The magnification process generally enhances the visibility of the blur intensity in the video. Therefore, a sharpening filter can be used to restore some details to the video.
Traditionally, sharpening involves high pass filtering the blurred image and adding the weighted high pass image to the original image. This is called unsharp masking. The high pass filter has several implementations, including a Laplacian filter and a Laplacian-Gaussian filter.

One embodiment described herein is a method for interpolating an image, the method comprising: determining an edge characteristic associated with the interpolated point, the edge characteristic having an edge size and an edge angle; selecting the interpolation in response to the edge angle a filter; and using the selected interpolation filter to determine a pixel value at the interpolation point. The edge characteristics may be based on determining a horizontal gradient and a vertical gradient of pixel values in adjacent regions associated with the interpolation points. The neighboring area associated with the interpolated point may be a horizontal rectangular area, a vertical rectangular area, or a square area. In one embodiment, the neighboring region is determined in response to the interpolated point being a column interpolated point, an interpolated point, or a central interpolated point. In some embodiments, the edge characteristics are determined by using a first step filter (which may be referred to as a "mask" because the convolution operation is not performed). One such first step filter is a modified Sobel operator.
The method can also include selecting an interpolation filter in response to the edge characteristic. The selection of the interpolation filter can be made in response to the edge size. That is, an image point whose edge size (or estimated size) is greater than a threshold may use a set of interpolation filters (eg, implemented as a hardware circuit or software running on a microprocessor or a combination thereof). Another set of interpolation filters can be used for image points whose edge size (or estimated size) is below the threshold. In some embodiments, the threshold may also be adjusted depending on the characteristics of the image and/or the edge characteristics.
The method may utilize an interpolation filter that applies a larger weight to pixels located in a direction along the edge angle and applies a smaller weight to a direction that is perpendicular to the edge angle Weights are applied to pixels that are located in a direction that is perpendicular to the edge angle. In one embodiment, the interpolation filter applies a greater weight to the nearest neighboring pixels, a medium weight to the pixels located in the direction along the edge angle, and a pixel located in a direction perpendicular to the edge angle Apply the smallest weight or no weight. In one embodiment, the nearest neighboring pixels are pixels that are in the same column or in the same row as the interpolated points.
The method described herein can be implemented on a processor. Thus, a computer readable medium can be used to store instructions that, when executed by a processor, cause the processor to: acquire a plurality of edge characteristics, each edge characteristic and a respective one of a plurality of interpolation points The interpolation points are associated, and each edge characteristic has an edge size and an edge angle; in response to the respective edge angles, an interpolation filter is selected for each of the plurality of interpolation points; using the corresponding selected ones Interpolating a filter to determine a pixel value for each of the plurality of interpolated points; and outputting the pixel value.
In some embodiments, the method is implemented in dedicated hardware or uses hardware to speed up the subset of calculations. In one embodiment, the interpolation device includes an edge characteristic calculator configured to determine edge characteristics of each of the plurality of interpolation points, and an interpolation filter selector configured to perform edge characteristics Interpolating and responsively generating an interpolation filter identifier for each of the plurality of interpolation points; and an interpolation filter circuit configured to apply the plurality of responses in response to the interpolation filter identifier Interpolating one of the interpolation filters and outputting the interpolated values for the plurality of interpolation points. One or more of these elements can be implemented by a processor for executing code stored in a computer readable storage medium.
In yet another embodiment, edge-based interpolation for amplification followed by an adaptive sharpening filter is described. In one embodiment, the sharpening filter is controlled by edge-based interpolation parameters that determine the pixels to be sharpened and the sharpening strength. In one embodiment, the method includes: determining gradient data of image pixels to be interpolated; determining adjacent pixel values using neighboring pixels according to gradient data; and identifying image pixels to be sharpened based on gradient data And use some of the filter strength determined based on the gradient data to sharpen the image pixels.
In an alternative embodiment, the edge based interpolation filter operation is combined with an adaptive sharpening filter into a single filter called a joint filter. In one embodiment, the method includes: determining gradient data of image pixels to be interpolated; selectively sharpening adjacent original pixels; selectively identifying in response to a pixel category of a center column or row of pixels The neighboring pixels to be interpolated; and the neighboring pixels are used to determine the interpolated and sharpened pixel values according to the joint sharpening and interpolation filters.

The interpolation methods and apparatus described herein can be used in wired or wireless networks. A device including a handheld device, a desktop computer, a laptop computer, or other computer can be used to perform the method. This includes cellular phones, PDAs, tablets and/or displays, as well as cable box boxes, televisions, and the like.
The interpolation or video amplification scheme described herein maintains edge fidelity and computational complexity lower than typical FIR filtering techniques. The system can implement any amplification factors N and M that are applied to the width and height of the video, respectively. For simplicity, it is described here that the image or video frame is magnified by a factor of two (N=2 and M=2) in these two dimensions.
Figure 1 shows a pixel grid containing the original pixel (represented by a square) and the estimated pixel or pixel value (indicated by a circle) estimated at the interpolation point. Pixel A has diagonal original pixels; Pixel B has two adjacent original pixels along the same row; and Pixel C has two adjacent original pixels along the same column. Therefore, the estimated pixels can be classified into three groups: (a) pixels having diagonal adjacent original pixels (labeled as "A" in FIG. 1), and estimated pixels located at these interpolation points can be called a central pixel; (b) a neighboring original row pixel as the nearest neighboring pixel (labeled "B" in Figure 1), the estimated pixel may be referred to as a row pixel; and (c) adjacent The original column pixel is the pixel of the nearest neighboring pixel (labeled "C" in Figure 1), which may be referred to as a column pixel.
In one embodiment, the method includes the following aspects: edge detection; edge angle determination; and pixel estimation. In another embodiment, the method can include determining an edge characteristic associated with the interpolation point, wherein the edge characteristic includes an edge size and an edge angle; selecting an interpolation filter in response to the edge angle; and using the selected An interpolation filter is used to determine the pixel value at the interpolation point.
Any software or hardware (collectively referred to as an operator) used to calculate the horizontal and vertical gradients of a pixel can be used for edge detection. In one embodiment, the modified Sobel operator is used because of its low computational complexity. The Sobel operator includes two square masks for calculating the horizontal and vertical gradients. These gradients are then used to calculate or otherwise obtain the angle of the gradient and the angle of the edge or its estimate. The standard Sobel operator is applied to the square pixel grid, and the modified Sobel operator can be applied to the rectangular pixel grid.
When the determined edge characteristics, estimates the interpolated pixel classification or classification points can be used to select different correction Sobel mask G _x and G _y. As shown in Figures 2A-2C, G _x and G _y can be used to multiply the image pixels point by point and sum the products to calculate the horizontal (Δx) and vertical (Δy) gradients (or their estimates), respectively. Masked.
In one embodiment, the magnitude G of the gradient (or edge) is determined by calculating G = |Δx| + |Δy|. If G is greater than the critical value T _edge , the edge may exist. In the case where G exceeds the critical value, the angle of the edge gradient can be determined as follows. The value of the _edge of the threshold T may depend on the resolution of the video, the pixel class, and other factors. Critical values can be determined empirically, and some threshold values that have been found to perform well are given in Table 1 below. The selection of the thresholds can be different: in some embodiments a fixed threshold can be selected, while in other embodiments, the threshold can be adjusted on a frame-by-frame basis or on a pixel block basis. The adaptive threshold can be automatically calculated based on pixel characteristics within the block. At lower resolutions, neighboring pixels are less likely to be coherent, resulting in a larger gradient. Using smaller thresholds will cause many pixels to be classified as edges. Therefore, in order to reduce inaccurate classification, a larger threshold can be used for smaller resolution video.

In various implementations, the interpolation filter can be selected in response to an edge (or gradient) size, an edge (or gradient) angle, or both. In one embodiment, an interpolation filter may be selected for the interpolated point where the corresponding edge size G is less than the T _edge . In other embodiments, an interpolation filter may be selected for an interpolation point where the corresponding edge has Δx or Δy equal to zero. In each case, there is no need to determine or provide an edge angle, and the interpolation filter used to insert new pixels at the interpolated point is as follows:
a. For the center pixel: (a0 + a1 + b0 + b1) / 4
b. For row pixels: (a1+b1)/2
c. For column pixels: (b0+b1)/2
Wherein a0, a1, b0 and b1 correspond to the pixels marked in the 2A-2C diagram. Note that Δy=0 and Δx=0 correspond to gradient angles of 0 and 90 degrees, respectively.
[Mu] gradient angle can be calculated as ^{μ = tan -1 (Δy / Δx} ). Since the edge angle is perpendicular to the gradient angle, the edge angle is Θ = μ + 90 ^o . Note that since the edge and edge gradients can be correlated by a simple 90 degree rotation, the two can be used interchangeably if this relationship is considered. It should also be noted that the range of edge (or gradient) angles can be reduced to a range of 0-180 degrees.
In some embodiments, the interpolation filters used at the respective interpolation points can be determined in response to pixel categories and edge angles. In one embodiment, the interpolation assumes that the edges are linear. For curved edges, other angles can be checked during pixel estimation. Alternatively, a Hough transform based method can be used to detect edge pixels belonging to a curve, and these pixels can be used to insert new pixels along the curved edge.
A pixel estimation process for interpolation points will now be described in which the edge size exceeds a critical value. In one embodiment, for the center pixel, the interpolation filter is selected based on whether the edge angle is approximately 45 degrees. One embodiment uses a center interpolation filter for a range of edge angles between 45 and 135 degrees. Alternative embodiments may utilize other angular ranges.
For central pixel estimation, interpolation can be performed as follows:
If θ is between 35 and 55 degrees, the new pixel = (a1 + b0)/2.
If θ is between 125 and 145 degrees, the new pixel = (a0 + b1)/2.
In other cases, the new pixel = (a0 + a1 + b0 + b1) / 4.
For line pixel estimation,
If θ is between 0 and 90 degrees, the new pixel = (b0 + 2 * (a1 + b1) + a2) / 6.
In other cases, the new pixel = (a0 + 2 * (a1 + b1) + b2) / 6.
For column pixel estimation,
If θ is between 0 and 90 degrees, the new pixel = (c0 + 2 *(b0+b1) + a1)/6.
In other cases, the new pixel = (a0 + 2 * (b0 + b1) + c1) / 6.
Thus, the method can utilize an interpolation filter that applies a larger weight to the pixels located in the direction along the edge angle and applies a smaller weight to the direction perpendicular to the edge angle The weight is not applied to the upper pixel. In one embodiment, the interpolation filter applies a larger weight to the nearest neighboring pixel, applies a medium weight to the pixel located in the direction along the edge angle, and applies the smallest weight to be located perpendicular to the edge angle Pixels in the direction or no weight applied to them. In one embodiment, the nearest neighboring pixels are pixels that are in the same column or in the same row as the interpolated points.
In one embodiment, the gradient measurements Δx and Δy can be used with a lookup table or LUT to determine a suitable interpolation filter. The LUT can store the desired filter impulse response, or can simply provide an interpolation filter identifier that can be used to determine and apply a suitable interpolation filter.
One embodiment described herein is a method for interpolating an image, the method comprising: determining an edge characteristic associated with the interpolated point, the edge characteristic including an edge size and an edge angle; selecting the interpolation in response to the edge angle a filter; and using the selected interpolation filter to determine a pixel value at the interpolation point. The edge characteristics may be based on determining a horizontal gradient and a vertical gradient of pixel values in adjacent regions associated with the interpolation points. The adjacent area associated with the interpolation point may be a horizontal rectangular area as shown in FIG. 2B, a vertical rectangular area as shown in FIG. 2C, or a square area as shown in FIG. 2A. In one embodiment, the neighboring region is determined in response to the interpolated point being a column interpolated point, an interpolated point, or a central interpolated point. In some embodiments, a step filter is used (which may be referred to as a "mask" when a convolution operation is not performed) to determine edge characteristics. One such step filter is a modified Sobel operator.
The performance comparison of edge-based interpolation with pentad filter techniques using the video listed in Table 2 is described below. (In comparison, only the quintile filter was used, but no sharpening filter was used.) Both methods were tested by applying these two methods to the transcoded video. The quality of the amplified video is measured using the peak signal to noise ratio (PSNR), video quality metric (VQM) score, and subjective comparison of the luma component. The transcoder's input video is used as a reference for quality measurements here.

A VQM score close to zero indicates no artifacts/damage, while a score close to one indicates severe damage. VQM only intercepted 15 seconds of video for comparison. The amplification scheme was tested on different video and transcoder bit rates, and the results are listed in Table 3. The items "2X" and "3X" in Table 3 refer to code-converted video that are being encoded at half and one-third of the original bit rate, respectively. For both "vid" video, both amplification methods produce low damage, but edge-based interpolation is slightly better than a 5-tap filter. It should be noted that edge-based interpolation looks significantly better when the viewer has two processed video. This is expected because most objective measurement solutions have limitations. Compared to 2X video, 3X video often produces a slightly higher VQM score because lowering the bit rate increases coding distortion.
For Avery's book video, edge-based interpolation has a small perceptual improvement over the 5-tap method compared to the 5-tap method. The VQM score of Avery's book is higher than the VQM score of the "vid" video because the downsampled video is encoded at a bit rate that is 50 times lower than the original video.
For "vid" video, the PSNR is calculated for video segments that do not have scene changes, and the results are reported in Table 4. For Avery's book, consider all frames with a PSNR of 50 dB, as all frames with a higher PSNR may be blank frames. For vid15 and vid16, a maximum PSNR improvement of 0.7 dB is achieved compared to a 5-tap filter. Figure 3 shows a snapshot of the PSNR and frame for the vid01_2X sequence. Obviously, even on a frame-by-frame basis, there is an improvement in PSNR when using an edge-based interpolation method.


In addition to the VQM score, the two amplification methods are also compared by viewing the amplified video. Experimental results obtained by subjecting the video frame to these two amplification schemes show that the edge-based interpolation method has sharper edges and more detail than the 5-tap filter method.
The complexity comparison is performed by running a magnification algorithm on each test sequence of the 1000 frame on a 2.53 GHz PC, and the average coding time of each frame is measured. The relative speed improvement is calculated by obtaining the ratio of the processing time of the 5-tap filter to the edge-based interpolation filter. The results are given in Table 5. We found that edge-based interpolation is up to 2.5 times faster than a 5-tap filter.

Apparatus and methods for interpolating and sharpening video signals are also described herein. In one embodiment, the method is performed by a two-stage filter method in which the interpolation filter amplifies the input video to a higher spatial resolution and then sharpens the filter to enhance the detail in the video. In another embodiment, the method is performed by a joint filter method in which a filter is applied to the input video signal to amplify the input video signal to a higher resolution while enhancing signal details. . The joint filter performs both interpolation and sharpening and can have associated processing gains. In some embodiments, memory access can also be reduced.
As shown in FIG. 4A, two separate filters, an edge-based interpolation filter and an adaptive sharpening filter, are combined to produce an enlarged and sharpened video frame from the original video frame source. An embodiment is depicted in FIG. 4B in which the YUV frame is first magnified twice in two dimensions by using an edge based interpolator. The edge based interpolator described above can be used. The edge information captured during the interpolation of the luminance component (Y) of the frame can be used to determine the pixel to be sharpened and the strength used to control the adaptive sharpening filter. As outlined in FIG. 9, edge information may be obtained from an edge detection filter selected in part at the position of the interpolated pixel as the center pixel A, the column pixel B, or the row pixel C. Sharpening can be performed on the luma component because the luma component usually contains more detail or edges. The sharpened luma component is then combined with the chrominance component to produce an enlarged sharpened frame. Edge information may also be extracted from one or more chrominance components in the chrominance component, and a weighted combination of edge information from Luma and one or more chrominance components from the chrominance components may be used. Additionally, one or more of the chrominance components may also undergo adaptive sharpening.
The block diagrams in Figures 5 and 6 show one algorithm for the edge-based interpolation and sharpening stages, respectively. Two buffers—the mapping [N][M] and the intensity [N][M]—store the sharpening map and edge intensity, respectively, where N and M are the width and height of the enlarged frame. . These two buffers are initialized to zero before interpolating the frame. The item "1" in the map [i][j] indicates that the pixel at (i, j) is to be sharpened, otherwise it is "0". For each pixel (i, j) to be interpolated, a modified Sobel operator is used to calculate the gradient (grad _ij ). If grad _ij ≧ sharpens the critical value (sharp_thresh), the mapping [i][j] element for the interpolated pixel and its neighboring pixels is set to one (1), where sharp_thresh is the sharpening threshold. Similarly, the intensity [i][j] for the interpolated pixel and its neighboring pixels is set to grad _ij . For example, if the gradient of the interpolated pixel at (i, j) is grad _ij ≧sharp_thresh, then:



This is illustrated in Figure 8, where the symbol "x" represents the original pixel and the "o" represents the interpolated pixel. Black dots are interpolated edge pixels that are labeled for sharpening, and other pixels in the bounding box are also labeled for sharpening because of their proximity to the edge pixels. In one embodiment, a sharpening threshold of 100 is appropriate, but the selection of the thresholds can be different. In other embodiments, the sharpening threshold may also be determined from local image characteristics and adjusted to different values throughout the image.
Apply Laplacian-Gaussian (LoG) filter to (i,j) mapping [i] after obtaining the interpolated frame and intensity [i][j] and mapping [i][j] [j]=1 pixels. The neighboring pixels of the edge pixels are sharpened to remove artifacts that may appear to be spots due to non-uniform variations in brightness. In the sharpening phase, the mapping [i][j] and intensity [i][j] information from the interpolation phase are used again. This leads to lower computational complexity.
The definition of a continuous form of LoG filter is as follows:

Where x and y are the distances from the center of LoG (ie (0,0)) and σ ² is the Gaussian variance. Discrete LoG filters of size KxL are defined as follows:

The parameter σ determines the filter strength. A larger σ (> 1) can be used for weak sharpening, while a smaller σ (≦ 1) can be used for stronger sharpening. The sharpening filter can also have various sizes of win_size. For example, for input video with small resolution (< 640x360), you can choose a LoG filter with win_size = 5x5; for higher resolution video (≧ 640x360), you can choose a LoG filter with win_size = 9x9. The filter size determines the number of neighboring pixels that are considered during sharpening. In other embodiments, different filter sizes can be selected based on video resolution and content. In addition, the filter size can be adjusted throughout the image frame. As an example, the operation of a 5x5 LoG filter can be expressed as follows:

Where C[i][j] and C _h [i][j] are the amplified pixels at (i, j) and their corresponding high frequency components, respectively. Figure 10 shows the span of pixels to which filter coefficients are applied, where the black dots represent the interpolated pixels (referred to as central pixels) to be sharpened using a 5x5 LoG filter. The LoG filter coefficients are applied to the pixels in the bounding box.
In one embodiment, σ _ij is selected based on the edge strengths described below:


Where ρ ₁ , . . . , ρ _S is the edge strength critical value, and α ₁ , . . . , α _S are different σ _ij values of S associated with different sharpening levels. In one embodiment, σ _ij is defined as follows:


In addition to the edge gradient, local image characteristics can also be used to determine σ _ij . For example, excessive sharpening of strong edges (ie, using small σ _ij values for strong edges) can result in ringing artifacts. To avoid this, a larger σ _ij value can be used when the edge intensity [i][j] is increased.
Finally, the sharpened pixel C _out [i][j] is obtained as follows:

λ=1 is used in one embodiment, but in other embodiments, different λ values may be used. In other embodiments, the sharpening filter can also be sensitive to noise by detecting noise before the interpolation phase and using this information to select a LoG filter having a larger σ(σ > 1). This will ensure that the noise detected as edges is not amplified. Similarly, in other embodiments, the sharpening process can be made sensitive to spurious edges, such as block artifacts. Block artifact detection algorithms can be used before interpolation. During the sharpening process, pixels classified as pseudo edges can be smoothed rather than sharpened.
Figure 7 depicts an embodiment of a joint edge-based interpolation and adaptive sharpening filter. In the previous two-stage filter method, the interpolated pixels were calculated in the filter window before the sharpening filter was applied. In the joint filter method, the dependency between interpolation and sharpening is reduced. In this embodiment, the edge based interpolation is combined with a sharpening filter to provide a joint filter. A flow chart of the combined filtering method is shown in Fig. 11. The modified Sobel operator is used to calculate the gradient grad _ij for the pixel located at (i, j). The use of two buffers - the buffer sharpening flag (sharp_flag) and the future gradient (future_grad) - is used to mark the sharpening of neighboring pixels at (k, n) that have not been interpolated. If the gradient grad _{ij is} greater than or equal to the predefined sharpening threshold sharp_thresh, the following two operations are performed on the adjacent 3x3 pixels, as shown in FIG. 12:
1. Use a LoG filter to sharpen adjacent original pixels, where equation (5) is used to select the σ of the LoG filter based on grad _ij . In Fig. 12, these pixels are indicated by "x".
2. Set the sharp_flag and future_grad of neighboring pixels that have not been interpolated as follows:
If the pixel (i, j) is the center pixel, then set:
.

If the pixel (i, j) is a column pixel, then set:
.

If the pixel (i, j) is a row pixel, then set:
.

In one embodiment, sharp_thresh = 100 and the pixel sharpening candidate is a neighboring 3x3 pixel. In other embodiments, a larger neighborhood is considered and a different sharp_thresh can be used; for example, the value of sharp_thresh can be determined based on local image characteristics. If grad _{ij is} smaller than sharp_thresh, it is checked whether the associated sharp_flag[i][j] is equal to 1. If sharp_flag[i][j] is equal to 1, the associated future_grad[i][j] value is used to select the appropriate joint filter, otherwise grad _{ij is} used to select the joint filter. By using future_grad [i] [j], using a gradient system before (ia, jb) at the adjacent pixels are interpolated, wherein grad _{ia jb} ≧ sharp_thresh and 0 ≦ a, b ≦ 1.
As depicted in Figure 12, the black dots are pixels that are being interpolated and sharpened, and have a gradient that is greater than the sharpening threshold. "x" is the original pixel to be sharpened, and the gray point is the pixel to be interpolated and labeled for sharpening. There are three possible scenarios: (a) center, (b) column, and (c) row pixel interpolation and sharpening.
Below are two implementations of the joint filter. In other embodiments, in addition to the original pixels, the joint filter operates in part on the pixels that have been interpolated.
In the two-stage filter method described above, the pixels within win_size are sharpened as shown by the block in FIG. In this embodiment, all of the original pixels for obtaining the interpolated pixels in the bounding box are identified as indicated by the bold X in Fig. 13. The sharpening filter operates on these raw pixels. Since these pixels are used to acquire the interpolated pixels in the bounding box, the high frequency components calculated from these raw pixels will be close to the high frequency components calculated from all the pixels in the bounding box.
As shown below, the joint filter can be written as the sum of two filters:
(7)

Where LOG is a sharpening filter applied only to the original pixels, and h _EI is an edge-based interpolator. In this embodiment, λ = 1, but in other embodiments different λ values may be selected. The LoG parameter σ is a function of edge strength, such as the function defined in equation (5). For an implementation where the input video resolution is less than 640 x 360, set win_size = 5x5, otherwise set it to 9x9.
The edge-based interpolator h _EI is a function of gradient, edge angle, filter size, and pixel class. As described above with reference to FIG. 9, for edge-based interpolation, the interpolated pixels are classified into a center pixel, a column pixel, and a row pixel based on the position of the interpolated pixel in the pixel grid. Therefore, the enlarged frame, including the original pixels, has four pixel categories. Each of the categories of interpolated pixels (eg, center, column, or row) has three filters, each corresponding to one of the three edge directions. For column and row pixels, the three edge directions are no edge, an edge angle between 0 and 90 degrees, and an edge angle between 90 and 180 degrees. For the center pixel, the three edge directions are no edge, an edge angle between 35 and 55 degrees, and an edge angle between 125 and 145 degrees. During sharpening, consider the original pixels used to retrieve the interpolated pixels within win_size. For example, in order to sharpen the center pixel in Fig. 13, filtering is performed on the original pixel of 4x4. Therefore, a 4x4 LOG filter is considered, resulting in a 4x4 joint filter in equation (7). The size of the joint filter is represented by the filter size (win_size), so for Figure 13, the filter size (win_size) = filter size (5x5) = 4x4. Similarly, as shown in Figures 14, 15, and 16, the filter sizes (5x5) for the column, row, and original pixels are 5x4, 4x5, and 5x5, respectively.
In Fig. 13, interpolation and sharpening of the center pixel indicated by black dots are shown. The interpolated pixels within win_size are taken from the original pixels represented as bold "x". A 4x4 LoG filter is applied to these original pixels. In Fig. 14, interpolation and sharpening of column pixels represented by black dots are shown. A 5x4 LoG filter is applied to the original pixel represented by the bold "x". In Fig. 15, interpolation and sharpening of line pixels represented by black dots are shown. A 4x5 LoG filter is applied to the original pixel represented by the bold "x". In Fig. 16, the sharpening of the original pixel labeled "A" is depicted. A 5x5 LoG filter is applied to the original pixel represented by the bold "x".
The following is an implementation of a joint filter for a central pixel of small screen video resolution (eg, 480x204). This program is similar to programs for columns, rows, and center pixels. Small modifications can be made to achieve a joint filter design for greater video resolution (eg, 1080p, 720p). For small screen video resolution (eg using win_size = 5x5), the resulting joint filter size is the filter size (win_size) = 4x4.
The following is an interpolation filter for generating a center pixel based on the edge direction:


Wherein, the edge threshold determines whether the interpolated pixel is an edge pixel.
An embodiment of the central pixel interpolation filter is given as follows:


In one embodiment, the edge threshold > sharp_thresh. A LoG filter having a standard deviation σ of size filter size (win_size) = KxL is as defined in equation (3).
The LoG filters for the four σ values in equation (5) (where α ₁ = ∞, α ₂ =1, α ₃ = 1.2, and α ₄ = 1.4) are as follows:


The joint filter for the center pixel is as follows:


Similarly,


When only interpolation is applied and no sharpening is applied, the _{center of the} filter h is used _{, ∞} . A joint filter for all other pixel classes can be obtained in a similar manner. In one embodiment, four different σ values and two different win_size values listed in equation (5) are used. Therefore, the number of joint filters for the interpolated pixels is:

The original pixels only need to be sharpened, so edge-based interpolation is not included during its filter design. The number of joint filters used for the original pixels is:
(12)
The total number of filters required is:
(13)
These filters can be stored in a lookup table and can be selected based on pixel class, gradient, edge angle, and win_size.
In an alternative embodiment, the joint filter is the same as equation (7), but it uses a different sharpening component. An example of an alternative joint filter is described for a central pixel of a small screen video resolution (eg, 480x204). The same design approach can be extended to columns, rows, and raw pixels. For this example, win_size = 5x5. Figure 17 shows the original pixels for sharpening and the estimated interpolated pixels, where all pixels within win_size are used for sharpening, and the interpolated pixels (grey dots) are from the original pixels ( Estimated by the bold "x").
In one embodiment, the edge-based interpolation is used to estimate the interpolated pixels (the gray point in Figure 17) when the gradient is less than the edge threshold. This corresponds to the following estimation filters for center, column and row pixels:
(14)
Figure 17 can be represented in the form of a table in Table 6 below, where C _ij is a pixel. The gray array represents the pixels in win_size = 5x5, and the bold entries represent the original pixels. Therefore, the non-bold pixels in the gray area are the estimated interpolated pixels.
Table 6



The 5x5 sharpening filter is expressed as follows.

A sharpening filter g that operates only on the original pixel but is still capable of providing the same high-frequency component C _h as in Equation (16) can be used according to the following formula:

To illustrate the design of g, the coefficients g ₀₀ and g _{01 are} derived. Each product term in equation (16) is represented by the original pixel as follows:



From equation (17), the sum of the coefficients corresponding to C ₀₀ is g ₀₀ , and the sum of the coefficients corresponding to C ₀₂ is g ₀₁ . Combine the terms in equation (18) to get:


_Center filter coefficient g for the center pixel is given as follows:
_Center g ([sigma] (gradient), the filter size (win_size)) = g _Center ([sigma] (gradient), 4x4) =


The filter coefficients g _columns , g _rows, and g _primitives for column, row, and original pixels, respectively, can be obtained in a similar manner.
The joint filter design 2 is defined as:


Where g(.) is the sharpening component. Continuing, the joint filter for the center pixel is: h _center (gradient, edge angle, 5x5) = h _{EI_ center} (gradient, edge angle, 4x4) + g _center (σ (gradient), 4x4), where h _{EI_ The center} is the same as defined in equation (8).
The total number of filters used in these two embodiments of the joint filter (Design 1 and Design 2) is the same as given in equation (13). The filter sizes in these two methods are listed in Table 7. Design 2 uses fewer filters, resulting in lower complexity. Design 2 can also provide better subjective quality than Design 1, and it has similar qualities to the two-stage filter approach. This is expected because Design 2 uses the estimated interpolated pixels during sharpening and is therefore mathematically close to the two-stage filter approach. It has been observed that sometimes Design 1 provides better subjective quality than Design 2. Thus, an adaptive method can be used to switch between the two joint filter designs; for example, the adaptive model can be based on local image characteristics.
Table 7 Size of the joint filter for win_size=5x5

The interpolation and sharpening filters described herein can be incorporated into any of a variety of terminals such as, but not limited to, digital televisions, wireless communication devices, wireless broadcast systems, personal digital assistants (PDAs), laptops or desktop computers, Mini notebook computers, digital cameras, digital recording devices, video game devices, video game consoles, cellular or satellite radio phones, digital media players, etc.
FIG. 18A illustrates an illustration of an exemplary communication system 100 in which one or more disclosed embodiments may be implemented. Communication system 100 may be a multiple access system that provides content, such as voice, material, video, messaging, broadcast, etc., to multiple wireless users. Communication system 100 is capable of enabling multiple wireless users to access such content via sharing of system resources, including wireless bandwidth. For example, communication system 100 can employ one or more channel access methods, such as code division multiple access (CDMA), time division multiple access (TDMA), frequency division multiple access (FDMA), orthogonal FDMA (OFDMA). Single carrier FDMA (SC-FDMA), etc.
As shown in FIG. 18A, communication system 100 can include wireless transmit/receive units (WTRUs) 102a, 102b, 102c, 102d, radio access network (RAN) 104, core network 106, public switched telephone network (PSTN). 108. Internet 110 and other networks 112, but it should be appreciated that the disclosed embodiments encompass any number of WTRUs, base stations, networks, and/or network elements. Each of the WTRUs 102a, 102b, 102c, 102d may be any type of device configured to operate and/or communicate in a wireless environment. By way of example, the WTRUs 102a, 102b, 102c, 102d may be configured to transmit and/or receive wireless signals, and may include user equipment (UE), mobile stations, fixed or mobile subscriber units, pagers, cellular phones, Personal digital assistants (PDAs), smart phones, laptops, mini-notebooks, personal computers, wireless sensors, consumer electronics, and more.
Communication system 100 can also include a base station 114a and a base station 114b. Each of the base stations 114a and 114b may be configured to wirelessly interface with at least one of the WTRUs 102a, 102b, 102c, 102d to facilitate access to one or more communication networks (such as a core network) 106. Any type of device of the Internet 110 and/or the network 112). By way of example, base stations 114a, 114b may be base transceiver stations (BTS), node B, eNodeB, home node B, home eNodeB, site controller, access point (AP), wireless router, etc. . While base stations 114a, 114b are each depicted as a single component, it should be appreciated that base stations 114a, 114b can include any number of interconnected base stations and/or network elements.
The base station 114a may be part of the RAN 104, which may also include other base stations and/or network elements (not shown), such as a base station controller (BSC), a radio network controller (RNC), Following the node and so on. Base station 114a and/or base station 114b may be configured to transmit and/or receive wireless signals within a particular geographic area (which may be a cell (not shown)). The cell can also be further divided into cell sectors. For example, a cell associated with base station 114a can be divided into three sectors. Thus, in one embodiment, base station 114a may include three transceivers, i.e., one transceiver per sector of the cell. In another embodiment, the base station 114a may employ multiple input multiple output (MIMO) technology, and thus, multiple transceivers may be used for each sector of the cell.
The base stations 114a, 114b can communicate with one or more of the WTRUs 102a, 102b, 102c, 102d via the air interface 116, where the air interface 116 can be any suitable wireless communication link (eg, radio frequency (RF), Microwave, infrared (IR), ultraviolet (UV), visible light, etc.). The air interface 116 can be established using any suitable radio access technology (RAT).
More specifically, as noted above, communication system 100 can be a multiple access system and can employ one or more channel access schemes such as CDMA, TDMA, FDMA, OFDMA, SC-FDMA, and the like. For example, base station 114a and WTRUs 102a, 102b, 102c in RAN 104 may implement a radio technology such as Universal Mobile Telecommunications System (UMTS) Terrestrial Radio Access (UTRA), which may use broadband CDMA (WCDMA) to establish air. Interface 116. WCDMA may include communication protocols such as High Speed Packet Access (HSPA) and/or Evolution HSPA (HSPA+). HSPA may include High Speed Downlink Packet Access (HSDPA) and/or High Speed Uplink Packet Access (HSUPA).
In another embodiment, base station 114a and WTRUs 102a, 102b, 102c may implement radio technologies such as Evolved UMTS Terrestrial Radio Access (E-UTRA), which may use Long Term Evolution (LTE) and/or LTE-Advanced (LTE). -A) to establish the air interface 116.
In other embodiments, base station 114a and WTRUs 102a, 102b, 102c may implement, for example, IEEE 802.16 (ie, Worldwide Interoperability for Microwave Access (WiMAX)), CDMA2000, CDMA2000 1X, CDMA2000 EV-DO, Provisional Standard 2000 (IS-2000) Radio Technologies such as Interim Standard 95 (IS-95), Interim Standard 856 (IS-856), Global System for Mobile Communications (GSM), Enhanced Data Rate GSM Evolution (EDGE), GSM EDGE (GERAN).
The base station 114b in FIG. 18A may be, for example, a wireless router, a home Node B, a home eNodeB, or an access point, and may be utilized to facilitate wireless within a local area (such as a business location, home, vehicle, campus, etc.) Any suitable RAT for connectivity. In one embodiment, base station 114b and WTRUs 102c, 102d may implement a radio technology such as IEEE 802.11 to establish a wireless local area network (WLAN). In another embodiment, base station 114b and WTRUs 102c, 102d may implement a radio technology such as IEEE 802.15 to establish a wireless personal area network (WPAN). In still another embodiment, base station 114b and WTRUs 102c, 102d may utilize a cellular based RAT (eg, WCDMA, CDMA2000, GSM, LTE, LTE-A, etc.) to establish picocells or femtocells. As shown in FIG. 18A, base station 114b may have a direct connection to internet 110. Thus, base station 114b may not need to access Internet 110 via core network 106.
The RAN 104 can communicate with a core network 106, which can be configured to provide voice, data, application, and/or internet protocol voice to one or more of the WTRUs 102a, 102b, 102c, 102d. Any type of network (VoIP) service. For example, core network 106 may provide call control, billing services, mobile location based services, prepaid calling, internet connectivity, video distribution, etc. and/or perform high level security functions such as user authentication. Although not shown in FIG. 18A, it should be appreciated that the RAN 104 and/or the core network 106 can communicate directly or indirectly with other RANs that can employ the same RAT or different RAT as the RAN 104 employs. RAT. For example, in addition to being connected to the RAN 104 that is utilizing the E-UTRA radio technology, the core network 106 can also communicate with another RAN (not shown) employing a GSM radio technology.
The core network 106 can also act as a gateway for the WTRUs 102a, 102b, 102c, 102d to access the PSTN 108, the Internet 110, and/or other networks 112. The PSTN 108 may include a circuit switched telephone network for providing Plain Old Telephone Service (POTS). The Internet 110 may include a globally interconnected computer network that uses public communication protocols such as Transmission Control Protocol (TCP), User Datagram Protocol (UDP), and Internet Protocol (IP) in the TCP/IP Internet Protocol Group. Road and device system. Network 112 may include a wired or wireless communication network that is owned and/or operated by other service providers. For example, network 112 may include a connection to one or more RANs (which are employed with RAN 104) Another core network of the same or different RATs of the RAT.
Some or all of the WTRUs 102a, 102b, 102c, 102d in the communication system 100 may include multi-mode capabilities, i.e., the WTRUs 102a, 102b, 102c, 102d may include communications for communicating with different wireless networks via different wireless links. Multiple transceivers. For example, the WTRU 102c shown in FIG. 18A can be configured to communicate with a base station 114a that can employ a cellular-based radio technology, and with a base station 114b that can employ an IEEE 802 radio technology.
Figure 18B is a system diagram of an exemplary WTRU 102. As shown in FIG. 18B, the WTRU 102 may include a processor 118, a transceiver 120, a transmit/receive element 122, a speaker/microphone 124, a keyboard 126, a display/touchpad 128, a non-removable memory 130, and a removable Memory 132, power source 134, global positioning system (GPS) chip set 136, and other peripheral devices 138. It should be appreciated that the WTRU 102 may include any sub-combination of the above-described elements while remaining consistent with the embodiments.
The processor 118 can be a general purpose processor, a special purpose processor, a conventional processor, a digital signal processor (DSP), a plurality of microprocessors, one or more microprocessors associated with the DSP core, a controller, a micro control , dedicated integrated circuit (ASIC), field programmable gate array (FPGA) circuit, any other type of integrated circuit (IC), state machine, etc. The processor 118 can perform signal coding, data processing, power control, input/output processing, and/or any other functionality that enables the WTRU 102 to operate in a wireless environment. The processor 118 can be coupled to a transceiver 120 that can be coupled to the transmit/receive element 122. Although FIG. 18 depicts processor 118 and transceiver as separate components, it should be appreciated that processor 118 and transceiver 120 can be integrated together into an electronic package or wafer.
The transmit/receive element 122 can be configured to transmit signals to a base station (e.g., base station 114a) via air interface 116 or to receive signals from a base station (e.g., base station 114a) via air interface 116. For example, in one embodiment, the transmit/receive element 122 can be an antenna configured to transmit and/or receive RF signals. In another embodiment, the transmit/receive element 122 can be an illuminant/detector configured to transmit and/or receive, for example, IR, UV, or visible light signals. In still another embodiment, the transmit/receive element 122 can be configured to transmit and receive both RF and optical signals. It should be appreciated that the transmit/receive element 122 can be configured to transmit and/or receive any combination of wireless signals.
Additionally, although transmission/reception element 122 is depicted as a single element in FIG. 18, WTRU 102 may include any number of transmission/reception elements 122. More specifically, the WTRU 102 may employ MIMO technology. Thus, in one embodiment, the WTRU 102 may include two or more transmission/reception elements 122 (e.g., multiple antennas) for transmitting and receiving wireless signals via the air interface 116.
The transceiver 120 can be configured to modulate signals to be transmitted by the transmission/reception element 122 and to demodulate signals received by the transmission/reception element 122. As noted above, the WTRU 102 may have multi-mode capabilities. Thus, transceiver 120 may include, for example, multiple transceivers for enabling WTRU 102 to communicate via multiple RATs, such as UTRA and IEEE 802.11.
The processor 118 of the WTRU 102 can be coupled to and can receive from the speaker/microphone 124, the keyboard 126, and/or the display/touchpad 128 (eg, a liquid crystal display (LCD) display unit or an organic light emitting diode (OLED) display unit) ) User input data. The processor 118 can also output user profiles to the speaker/microphone 124, the keyboard 126, and/or the display/touchpad 128. In addition, the processor 118 can access information from any type of suitable memory (such as the non-removable memory 130 and/or the removable memory 132) and in any type of appropriate memory (such as non-removable memory). The data is stored in the body 130 and/or the removable memory 132). The non-removable memory 130 may include random access memory (RAM), read only memory (ROM), hard disk, or any other type of memory storage device. The removable memory 132 can include a Subscriber Identity Module (SIM) card, a memory stick, a secure digital (SD) memory card, and the like. In other embodiments, the processor 118 may access information from memory (such as a server or home computer (not shown)) that is not physically located on the WTRU 102, and is not physically located on the WTRU 102. The data is stored in a body such as a server or a home computer (not shown).
The processor 118 can receive power from the power source 134 and can be configured to allocate and/or control power to other elements in the WTRU 102. Power source 134 may be any suitable device for powering WTRU 102. For example, the power source 134 may include one or more dry cells (eg, nickel-cadmium (NiCd), nickel-zinc (NiZn), nickel metal hydride (NiMH), lithium ion (Li-ion), etc.), solar cells, fuel Battery, etc.
The processor 118 may also be coupled to a set of GPS chips 136 that may be configured to provide location information (e.g., longitude and dimension) with respect to the current location of the WTRU 102. In addition to or in lieu of information from the GPS chipset 136, the WTRU 102 may receive location information from a base station (e.g., base station 114a, 114b) via air interface 116, and/or based on receiving from two or more neighboring base stations. The timing of the signal to determine its position. It should be appreciated that the WTRU 102 may obtain location information using any suitable location determination method while remaining consistent with the implementation.
The processor 118 can also be coupled to other peripheral devices 138, which can include one or more software and/or hardware modules for providing other features, functionality, and/or wired or wireless connectivity. For example, peripheral device 138 may include an accelerometer, an electronic compass, a satellite transceiver, a digital camera (for photo or video), a universal serial bus (USB) port, a vibrating device, a television transceiver, a hands-free headset, and a Bluetooth device. R module, FM radio unit, digital music player, media player, video game console module, internet browser, etc.
Figure 18C is a system diagram of RAN 104 and core network 106, in accordance with an embodiment. As noted above, the RAN 104 can employ UTRA radio technology to communicate with the WTRUs 102a, 102b, 102c via the air interface 116. The RAN 104 can also communicate with the core network 106. As shown in FIG. 18C, RAN 104 may include Node Bs 140a, 104b, 140c, wherein each of Node Bs 140a, 104b, 140c may include one or more for communicating with WTRUs 102a, 102b, 102c via air interface 116. Transceivers. Node Bs 140a, 104b, 140c may each be associated with a particular cell (not shown) in RAN 104. The RAN 104 may also include RNCs 142a, 142b. It should be appreciated that the RAN 104 may include any number of Node Bs and RNCs while remaining consistent with the implementation.
As shown in Figure 18C, Node Bs 140a, 140b can communicate with RNC 142a. Additionally, Node B 140c can communicate with RNC 142b. Node Bs 140a, 140b, 140c can communicate with respective RNCs 142a, 142b via an Iub interface. The RNCs 142a, 142b can communicate with each other via the Iur interface. Each of the RNCs 142a, 142b is configured to control the respective Node Bs 140a, 140b, 140c to which it is connected. Additionally, each of the RNCs 142a, 142b can be configured to perform or support other functions, such as outer loop power control, load control, admission control, packet scheduling, handover control, macro diversity, security functions, data encryption. Wait.
The core network 106 shown in FIG. 18C may include a media gateway (MGW) 144, a mobile switching center (MSC) 146, a serving GPRS support node (SGSN) 148, and/or a gateway GPRS support node (GGSN) 150. . While each element of the aforementioned elements is depicted as being part of core network 106, it should be appreciated that any of these elements can be owned and/or operated by entities other than the core network operator.
The RNC 142a in the RAN 104 can be connected to the MSC 146 in the core network 106 via an IuCS interface. The MSC 146 can be connected to the MGW 144. The MSC 146 and the MGW 144 may provide the WTRUs 102a, 102b, 102c with access to a circuit-switched network, such as the PSTN 108, to facilitate communication between the WTRUs 102a, 102b, 102c and conventional landline communication devices.
The RNC 142a in the RAN 104 can also be connected to the SGSN 148 in the core network 106 via an IuPS interface. The SGSN 148 can be connected to the GGSN 150. The SGSN 148 and GGSN 150 may provide the WTRUs 102a, 102b, 102c with access to a packet switched network, such as the Internet 110, to facilitate communication between the WTRUs 102a, 102b, 102c and IP-enabled devices.
As noted above, the core network 106 can also be connected to the network 112, which can include other wired or wireless networks that are owned and/or operated by other service providers.
Figure 18D is a system diagram of RAN 104 and core network 106, in accordance with an embodiment. As noted above, the RAN 104 can employ E-UTRA radio technology to communicate with the WTRUs 102a, 102b, 102c via the air interface 116. The RAN 104 can also communicate with the core network 106.
The RAN 104 may include eNodeBs 160a, 160b, 160c, but it should be appreciated that the RAN 104 may include any number of eNodeBs while maintaining consistency with the embodiments. Each of the eNodeBs 160a, 160b, 160c may include one or more transceivers for communicating with the WTRUs 102a, 102b, 120c via the air interface 116. In one embodiment, the eNodeB can implement MIMO technology. Thus, for example, the eNodeB 160a may use multiple antennas to transmit wireless signals to and receive wireless signals from the WTRU 102a.
Each eNodeB can be associated with a particular cell (not shown) and can be configured to handle radio resource management decisions, handover decisions, user scheduling in the uplink and/or downlink, and the like. As shown in Fig. 18D, the eNodeBs can communicate with each other via the X2 interface.
The core network 106 shown in FIG. 18D may include a mobility management gateway (MME) 162, a service gateway 164, and a packet data network (PDN) gateway 166. While each element of the aforementioned elements is depicted as being part of core network 106, it should be appreciated that any of these elements can be owned and/or operated by entities other than the core network operator.
The MME 162 may be connected to each of the eNodeBs 160a, 160b, 160c in the RAN 104 via an S1 interface. For example, MME 162 may be responsible for authenticating users of WTRUs 102a, 102b, 102c, bearer activation/deactivation, selecting a particular service gateway during initial connection of WTRUs 102a, 102b, 102c, and the like. The MME 162 may also provide control plane functionality for switching between the RAN 104 and a RAN (not shown) employing other radio technologies, such as GSM or WCDMA.
Service gateway 164 may be connected to each eNodeB in RAN 104 via an S1 interface. The service gateway 164 can generally route and forward user data packets to the WTRUs 102a, 102b, 102c, and route and forward user data packets from the WTRUs 102a, 102b, 102c. The service gateway 164 may also perform other functions, such as anchoring the user plane during handover between eNodeBs, triggering paging, managing and storing the WTRUs 102a, 102b, 102c when the downlink profiles are available to the WTRUs 102a, 102b, 102c. Context, etc.
The service gateway 164 can also be coupled to the PDN gateway 166 to facilitate communication between the WTRUs 102a, 102b, 102c and IP-enabled devices, wherein the PDN gateway 166 can provide the WTRUs 102a, 102b, 102c to the packet switched network. Access (such as the Internet 110).
The core network 106 can facilitate communication with other networks. For example, core network 106 may provide WTRUs 102a, 102b, 102c with access to a circuit-switched network, such as PSTN 108, to facilitate communication between WTRUs 102a, 102b, 102c and conventional landline communication devices. For example, core network 106 may include or may be in communication with an IP gateway (eg, an IP Multimedia Subsystem (IMS) server), where the IP gateway acts as an interface between core network 106 and PSTN 108. In addition, core network 106 can provide WTRUs 102a, 102b, 102c with access to network 112, which can include other wired or wireless networks that are owned and/or operated by other service providers.
Figure 18E is a system diagram of RAN 104 and core network 106, in accordance with an embodiment. The RAN 104 may be an Access Service Network (ASN) that communicates with the WTRUs 102a, 102b, 102c via the air interface 116 using IEEE 802.16 radio technology. As discussed further below, the communication links between the different functional entities of the WTRUs 102a, 102b, 102c, RAN 104, and core network 106 may be defined as reference points.
As shown in FIG. 18E, the RAN 104 may include base stations 170a, 170b, 170c and ASN gateway 172, but it should be appreciated that the RAN 104 may include any number of base stations and while maintaining consistency with the embodiments. ASN gateway. Base stations 170a, 170b, 170c may each be associated with a particular cell (not shown) in RAN 104 and may each include one or more transceivers for communicating with WTRUs 102a, 102b, 102c via air interface 116. Device. In one embodiment, base stations 170a, 170b, 170c may implement MIMO technology. Thus, for example, base station 170a can use multiple antennas to transmit wireless signals to, and receive wireless signals from, WTRU 102a. The base stations 170a, 170b, 170c may also provide mobility management functions such as handover triggering, tunnel establishment, radio resource management, traffic classification, quality of service (QoS) policy enforcement, and the like. The ASN gateway 172 can act as a traffic aggregation point and can be responsible for paging, user profile cache, routing to the core network 106, and the like.
The air interface 116 between the WTRUs 102a, 102b, 102c and the RAN 104 may be defined as an Rl reference point for implementing the IEEE 802.16 specification. In addition, each of the WTRUs 102a, 102b, 102c can establish a logical interface (not shown) with the core network 106. The logical interface between the WTRUs 102a, 102b, 102c and the core network 106 can be defined as an R2 reference point that can be used for authentication, authorization, IP host configuration management, and/or mobility management.
The communication link between each of the base stations 140a, 140b, 140c may be defined as an R8 reference point that includes protocols for facilitating data transfer between the WTRU handover and the base station. The communication link between the base station and the ASN gateway 172 can be defined as an R6 reference point. The R6 reference point may include an agreement to facilitate mobility management based on mobility events associated with each of the WTRUs 102a, 102b, 102c.
As shown in FIG. 18C, the RAN 104 can be connected to the core network 106. The communication link between the RAN 104 and the core network 106 can be defined as an R3 reference point that includes protocols for facilitating, for example, data transfer and mobility management capabilities. The core network 106 may include a Mobile IP Home Agent (MIP-HA) 144, an Authentication, Authorization, Accounting (AAA) server 176, and a gateway 178. While each element of the aforementioned elements is depicted as being part of core network 106, it should be appreciated that any of these elements can be owned and/or operated by entities other than the core network operator.
The MIP-HA may be responsible for IP address management and can enable the WTRUs 102a, 102b, 102c to roam between different ASNs and/or different core networks. The MIP-HA 174 may provide the WTRUs 102a, 102b, 102c with access to a packet switched network, such as the Internet 110, to facilitate communication between the WTRUs 102a, 102b, 102c and IP-enabled devices. The AAA server 146 can be responsible for user authentication and for supporting user services. Gateway 178 can facilitate interconnection with other networks. For example, gateway 178 can provide WTRUs 102a, 102b, 102c with access to a circuit-switched network, such as PSTN 108, to facilitate communication of WTRUs 102a, 102b, 102c with conventional landline communication devices. In addition, gateway 178 can provide access to network 112 to WTRUs 102a, 102b, 102c, which can include other wired or wireless networks that are owned and/or operated by other service providers.
Although not shown in Figure 18E, it should be appreciated that the RAN 104 can be connected to other ASNs and the core network 106 can be connected to other core networks. The communication link between the RAN 104 and other ASNs may be defined as an R4 reference point, which may include a protocol for coordinating the mobility of the WTRUs 102a, 102b, 102c between the RAN 104 and other ASNs. The communication link between core network 106 and other core networks may be defined as an R5 reference point, which may include protocols for facilitating interconnection between the local core network and the core network being accessed.
Although features and elements are described above in a particular combination, those of ordinary skill in the art will recognize that each feature or element can be used alone or in any combination with other features and elements. Additionally, the methods described herein can be implemented in a computer program, software or firmware embodied in a computer readable medium embodied by a computer or processor. Examples of computer readable media include computer readable storage media. Examples of computer readable storage media include, but are not limited to, read only memory (ROM), random access memory (RAM), scratchpad, cache memory, semiconductor storage devices, such as internal hard drives or removable disks. Magnetic media, magneto-optical media, and optical media such as CD-ROM discs and digital versatile discs (DVDs). A processor associated with the software can be used to implement a radio frequency transceiver for use in a WTRU, UE, terminal, base station, RNC, or any host computer.
Variations of the various methods, apparatus, and systems described above are possible without departing from the scope of the invention. In view of the various embodiments that can be applied, it is to be understood that the illustrated embodiments are only illustrative and are not to be construed as limiting the scope of the appended claims.
Moreover, in the embodiments described above, a processing platform, computing system, controller, and other devices including a processor are indicated. These devices may include at least one central processing unit ("CPU") and memory. The actions and symbolic representations of operations or instructions can be performed by various CPUs and memories, depending on the practice of the skilled person in the field of computer programming. Such actions and operations or instructions may be referred to as being "executed,""executed by a computer," or "executed by a CPU."
Those of ordinary skill in the art will appreciate that the actions or instructions of the acts and symbolic representations can include manipulation of electronic signals that are executed by the CPU. An electronic system represents a data bit that can cause an effective transformation or reduction of an electronic signal, and maintenance of a data bit at a memory location in a memory system, thereby reconfiguring or otherwise altering the operation of the CPU And other processing of the signal. The memory location in which the data bit is held is the physical location having a particular electrical, magnetic, optical or organic characteristic corresponding to the data bit or representing the data bit. It should be understood that the exemplary embodiments are not limited to the platforms or CPUs mentioned above, and that other platforms and CPUs may also support the methods described above.
The data bits can also be maintained on a computer readable medium, including a magnetic disk, a compact disc, and any other volatiles (eg, random access memory ("RAM") or non-volatile that can be read by the CPU. (eg, a read-only memory ("ROM") mass storage system. Computer-readable media can include cooperating or interconnected computer readable media, either exclusively on a processing system or in multiple interconnected processes The system (which may be local to the processing system or remote from the processing system) is distributed. It should be understood that the exemplary embodiments are not limited to the memory mentioned above, and other platforms and memories may also support the methods described above.
No element, act, or instruction used in the description of the present application should be construed as being critical or essential to the present invention unless specifically described in this manner. Also, as used herein, the word "a" is intended to include one or more items. The term "one" or a similar language is used when it is desired to refer only to one item. In addition, as used herein, any of a plurality of items and/or a plurality of categories of items is intended to include separate items and/or items of these categories or in combination with other items and/or other categories of items. Any one, any combination, any number, and/or any combination. Also, as used herein, the term "set" is intended to include any number of items, including zero. Also, as used herein, the term "amount" is intended to include any number, including zero.
Moreover, the scope of the claims should not be construed as being limited In addition, the term "means" in the scope of any patent application is intended to invoke 35 USC § 112, ¶ 6, and it is not intended that any patent application scope without the word "means" is also invoked.

100. . . Exemplary communication system

102a, 102b, 102c, 102d. . . Wireless transmit/receive unit (WTRU)

104. . . Radio access network (RAN)

106. . . Core network

108. . . Public Switched Telephone Network (PSTN)

110. . . Internet

112. . . Other network

114a, 114b, 170a, 170b, 170c. . . Base station

116. . . Air interface

118. . . processor

120. . . transceiver

122. . . Transmission/reception component

124. . . Speaker/microphone

126. . . keyboard

128. . . Display/trackpad

130. . . Non-removable memory

132. . . Removable memory

134. . . power supply

136. . . Global Positioning System (GPS) chipset

138. . . Peripheral device

140a, 104b, 140c. . . Node B

142a, 142b. . . Radio Network Controller (RNC)

144. . . Media Gateway (MGW)

146. . . Mobile Switching Center (MSC)

148. . . Serving GPRS Support Node (SGSN)

150. . . Gateway GPRS Support Node (GGSN)

160a, 160b, 160c. . . eNodeB

162. . . Mobility Management Gateway (MME)

164. . . Service gateway

166. . . Packet Data Network (PDN) gateway

172. . . Access Service Network (ASN) Gateway

174. . . Mobile IP Local Agent (MIP-HA)

176. . . Authentication, Authorization, Accounting (AAA) Server

178. . . Gateway

The invention may be understood in more detail from the following description, which is given by way of example, and
Figure 1 is a diagram of pixels in an image;
Figure 2A-2C is a diagram of the interpolation points of the center, row and column;
Figure 3 is a frame-level PSNR comparison between the edge-based interpolation and the five-tap filter method for the vid01_2X sequence; and Figure 4A is an edge-based interpolation using two separate filters. And a block diagram of the adaptive sharpening system;
Figure 4B is a more detailed block diagram of an edge-based interpolation and adaptive sharpening system using two separate filters;
Figure 5 is a flow chart of the interpolation process;
Figure 6 is a flow chart of the sharpening process;
Figure 7 is a block diagram of a joint edge-based interpolation and adaptive sharpening filter;
Figure 8 is a pixmap of the flagged pixel;
Figure 9 is a pixmap of the center, column and row pixels;
Figure 10 shows the span of pixels to which filter coefficients are applied;
Figure 11 is a flow diagram of one embodiment of a combined filtering method;
Figure 12 is a pixmap showing the allocation of sharpening parameters for adjacent pixels of (a) center, (b) column, and (c) row of pixels;
Figures 13-16 are pixmaps identifying the original pixels used to derive the interpolated pixels in the bounding box;
Figure 17 identifies the original pixels used for sharpening and the estimated interpolated pixels;
Figure 18A is a system diagram of an exemplary communication system in which one or more disclosed embodiments may be implemented;
Figure 18B is a system diagram of an exemplary wireless transmit/receive unit (WTRU) that can be used in the communication system shown in Figure 18A; and Figure 18C-18E is a diagram that can be used in the communication system shown in Figure 18A A system diagram of an exemplary radio access network and an exemplary core network.

Claims (31)

A method for interpolating an image, the method comprising:
Determining an edge characteristic associated with an interpolated point, the edge characteristic having an edge size and an edge angle;
Selecting an interpolation filter in response to the edge angle; and using the selected interpolation filter to determine a pixel value at the interpolation point. The method of claim 1, wherein the edge characteristic is based on determining a horizontal gradient and a vertical gradient of a pixel value in a neighboring region associated with the interpolation point. The method of claim 2, wherein the adjacent area associated with the interpolated point is one of a horizontal rectangular area, a vertical rectangular area, or a square area. The method of claim 2, wherein the neighboring region is determined in response to the interpolated point being a column of interpolated points, a row of interpolated points, or a central interpolated point. The method of claim 1, wherein the first step filter is used to determine the edge characteristic. The method of claim 1, wherein a modified Sobel operator is used to determine the edge characteristic. The method of claim 1, wherein the method further comprises: selecting an interpolation filter in response to the edge size. The method of claim 1, wherein the method further comprises: selecting an interpolation filter in response to the edge gradient. The method of claim 1, wherein the interpolation filter applies a larger weight to pixels located along a side of the edge angle and applies a pixel located at a gradient angle along the edge Small weights or no weights applied. The method of claim 1, wherein the interpolation filter applies a greater weight to a nearest neighbor pixel and a pixel located in a direction along the edge angle, and is located along the edge gradient The pixels on one side of the corner apply a smaller weight or do not apply a weight. The method of claim 10, wherein the nearest neighboring pixel is a pixel that is in the same column or in the same row as the interpolated point. A computer readable medium for storing instructions that, when executed by a processor, cause the processor to:
Obtaining a plurality of edge characteristics, each of the edge characteristics being associated with a respective one of the plurality of interpolation points, each edge characteristic having an edge size and an edge angle;
Selecting an interpolation filter for each of the plurality of interpolation points in response to respective edge angles;
Determining a pixel value of each of the plurality of interpolation points using a corresponding selected interpolation filter; and outputting the pixel value. An interpolation device, the interpolation device comprising:
An edge characteristic calculator configured to determine an edge characteristic for each of the plurality of interpolation points;
An interpolation filter selector configured to operate the edge characteristic and responsively generate an interpolation filter identifier for each of the plurality of interpolation points; and an interpolation a filter circuit configured to apply one of the plurality of interpolation filters in response to the interpolation filter identifier and output a plurality of interpolated values for the plurality of interpolation points . A method comprising:
Interpolating a pixel in an image frame in response to a gradient data; and in response to the gradient data, sharpening pixels in the image frame. The method of claim 14, wherein the interpolating comprises weighting a neighboring pixel based on the gradient data. The method of claim 14, wherein sharpening comprises determining a high frequency information associated with a single pixel. The method of claim 14, wherein the pixel sharpening is performed on a pixel having a gradient greater than a threshold. The method of claim 14, wherein the pixel sharpening is performed on a pixel having a gradient greater than a threshold and a corresponding neighboring pixel. The method of claim 14, wherein pixel sharpening is performed on a pixel according to a sharpening map. The method of claim 14, wherein the pixel sharpening is performed using an intensity parameter. The method of claim 20, wherein the intensity parameter is determined based on a gradient size. The method of claim 14, wherein a combined filter is used to simultaneously perform pixel interpolation and pixel sharpening. A device comprising:
An interpolation module configured to interpolate a pixel in the image frame in response to a gradient data; and a sharpening filter module configured to respond to the gradient data to the image The pixels in the box are sharpened. The device of claim 23, wherein the interpolation module and the sharpening filter module comprise a processor and a computer readable memory device for storing an instruction. The apparatus of claim 23, wherein the apparatus comprises one or more digital signal processing filters. A tangible computer readable storage medium having stored thereon an instruction that, when executed, causes a processing device to perform the method of claim 14. A method comprising:
Determining a gradient data for an image pixel to be interpolated;
Determining an interpolated pixel value using a neighboring pixel based on the gradient data;
The image pixels to be sharpened are identified based on the gradient data; and the image pixels are sharpened using a filter intensity determined based at least in part on the gradient data. The method of claim 27, wherein determining the interpolated pixel value is performed based in part on whether a pixel class is a center, a column, or a row. A method comprising:
Determining a gradient data for an image pixel to be interpolated;
Selectively sharpening a neighboring original pixel;
Responding to a pixel class of a center, a column or a row of pixels, selectively identifying a neighboring pixel to be interpolated; and using a joint sharpening and interpolation filter to determine an interpolated and sharp using neighboring pixels The pixel value. The method of claim 29, wherein the joint sharpening and interpolation filters perform interpolation based on an edge angle and perform sharpening based on an original pixel within a window. The method of claim 30, wherein the joint sharpening and interpolation filters perform sharpening based on an estimated interpolation point within the window.