JP2018098582A - Image processing apparatus and method, and imaging apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To perform a rewriting process in which noise is reduced while maintaining a sense of resolution of a portion to be rewritten as much as possible. SOLUTION: A smoothing processing unit (304) that performs smoothing processing on a first image signal of each pixel of an input image to generate a second image signal, and a second image for each pixel. A virtual light source addition processing unit (303) that generates a third image signal, which is a reflection component of virtual light emitted from a virtual light source, from the signal and adds the third image signal and the first image signal. ), And the smoothing process by the smoothing processing unit is controlled based on at least one of the signal level of the first image signal and the virtual light intensity. [Selection diagram] Fig. 3

Description

  The present invention relates to an image processing apparatus and method, and an imaging apparatus, and more particularly to an image processing apparatus and method for correcting the brightness of an input image, and an imaging apparatus.

  Conventionally, a technique for performing relighting by irradiating light from a virtual light source onto a subject in an image after photographing is known. This makes it possible to brighten dark areas such as shadows caused by ambient light and obtain a preferable image.

  Patent Document 1 discloses a lighting processing unit that performs pseudo lighting processing on a captured image. Specifically, the shadow of the subject can be suppressed by extracting a brightness area lower than the average brightness of the entire face area as a shadow area and increasing the brightness of the extracted shadow area.

JP 2010-135996 A

  In the method of Patent Document 1 described above, the shadow region is reduced by increasing the brightness by increasing the gain. However, when the brightness is increased by increasing the gain, the noise component is also amplified, and the S / N ratio is increased. It gets worse. In order to solve this problem, a noise removal filter is applied in Patent Document 1. However, if the noise removal process is simply performed, the resolution of the subject is also lowered by dropping noise more than necessary. there is a possibility.

  The present invention has been made in view of the above-described problems, and an object of the present invention is to perform relighting processing with reduced noise while maintaining as much as possible the resolution of the portion where relighting is performed.

  To achieve the above object, the image processing apparatus of the present invention performs smoothing processing on the first image signal of each pixel of the input image to generate a second image signal, and For each pixel, a third image signal that is a reflection component of virtual light emitted from a virtual light source is generated from the second image signal, and the third image signal and the first image signal are generated. And controlling the smoothing process by the smoothing unit based on at least one of the signal level of the first image signal and the virtual light intensity. It is characterized by.

  According to the present invention, it is possible to perform relighting processing with reduced noise while maintaining as much as possible the resolution of the location where relighting is performed.

1 is a block diagram illustrating a configuration of a digital camera according to an embodiment of the present invention. The block diagram which shows the structure of the image process part in embodiment. The block diagram which shows the structure of the relighting process part in embodiment. The flowchart which shows the relighting process in embodiment. The figure which shows the relationship between a to-be-photographed object and virtual light in embodiment. The figure explaining the parameter characteristic of noise reduction processing in an embodiment.

  DESCRIPTION OF EMBODIMENTS Hereinafter, embodiments for carrying out the present invention will be described in detail with reference to the accompanying drawings. In the present embodiment, an example in which the present invention is applied to a digital camera as an image processing apparatus will be described.

  FIG. 1 is a block diagram illustrating a configuration example of a digital camera according to an embodiment of the present invention. Hereinafter, the configuration and basic operation of the digital camera 100 will be described with reference to FIG. In the digital camera 100 shown in FIG. 1, light incident through a lens group 101 (imaging optical system) including a zoom lens and a focus lens and a shutter 102 having a diaphragm function is photoelectrically converted in an imaging unit 103. The imaging unit 103 is configured by a CCD, a CMOS element, or the like, and an electric signal obtained by photoelectric conversion is output to the A / D converter 104 as an image signal. The A / D converter 104 converts the analog image signal output from the imaging unit 103 into a digital image signal (image data) and outputs the digital image signal to the image processing unit 105.

  The image processing unit 105 performs color conversion processing such as white balance processing, gamma processing on image data from the A / D converter 104 or image data read from the image memory 106 via the memory control unit 107. Various image processing such as edge enhancement processing and color correction processing are performed. The image data output from the image processing unit 105 is written into the image memory 106 via the memory control unit 107. The image memory 106 stores image data output from the image processing unit 105 and image data to be displayed on the display unit 109.

  In the image processing unit 105, a predetermined evaluation value calculation process is performed using the captured image data, and the system control unit 50 performs exposure control and distance measurement control based on the obtained evaluation value. Thus, TTL (through-the-lens) AF (autofocus) processing, AE (automatic exposure) processing, AWB (auto white balance) processing, and the like are performed.

  The D / A converter 108 converts the display digital image data stored in the image memory 106 into an analog signal and supplies the analog signal to the display unit 109. The display unit 109 performs display according to an analog signal from the D / A converter 108 on a display such as an LCD.

  The codec unit 110 compresses and encodes the image data stored in the image memory 106 based on standards such as JPEG and MPEG. The system control unit 50 stores the encoded image data in a recording medium 112 such as a memory card or a hard disk via an interface (I / F) 111.

  Further, the image data read from the recording medium 112 via the I / F 111 is decoded and expanded by the codec unit 110 and stored in the image memory 106. Then, by displaying the image data stored in the image memory 106 on the display unit 109 via the memory control unit 107 and the D / A converter 108, the image can be reproduced and displayed.

  The relighting processing unit 113 performs relighting processing (relighting processing) on the captured image. The relighting process performed by the relighting processing unit 113 will be described later in detail.

  The system control unit 50 controls the entire system of the digital camera 100. The nonvolatile memory 121 is configured by a memory such as an EEPROM, and stores programs, parameters, and the like necessary for processing of the system control unit 50. The system memory 122 is used for developing constants and variables for operation of the system control unit 50, programs read from the nonvolatile memory 121, and the like. The operation unit 120 receives an operation by a user.

  In addition to the above basic operation, the system control unit 50 executes each program of the present embodiment described later by executing a program recorded in the nonvolatile memory 121. The program here is a program for executing various flowcharts described later in the present embodiment. At this time, constants, variables for operation of the system control unit 50, programs read from the nonvolatile memory 121, and the like are expanded in the system memory 122.

  FIG. 2 is a block diagram illustrating a configuration of the image processing unit 105. Hereinafter, the processing in the image processing unit 105 will be described with reference to FIG. In the present embodiment, it is assumed that the imaging unit 103 is covered with a Bayer array color filter. Accordingly, one of R, G, and B image signals is output from each pixel of the imaging unit 103.

  The image signal input from the A / D converter 104 of FIG. 1 to the image processing unit 105 is first input to the luminance / color signal generation unit 200 of FIG. The luminance / color signal generation unit 200 performs a synchronization process on the input R, G, and B image data, and generates a color signal RGB for each pixel. Also, the luminance / color signal generation unit 200 generates a luminance signal Y from the color signals RGB, and outputs the generated luminance signal Y to the contour enhancement processing unit 201 and the color signal RGB to the WB amplification unit 203.

  The contour enhancement processing unit 201 performs contour enhancement processing on the luminance signal Y and outputs the result to the luminance gamma processing unit 202. The luminance gamma processing unit 202 performs gamma correction on the luminance signal Y and outputs the luminance signal Y to the image memory 106 via the memory control unit 107.

  The WB amplification unit 203 adjusts the white balance by applying a gain to the color signal RGB of each pixel generated based on the white balance gain value calculated by the system control unit 50 through processing to be described later. The color conversion processing unit 204 performs, for example, matrix calculation on the color signal RGB that has been subjected to white balance adjustment, and converts it into a desired color balance. The color gamma processing unit 205 performs gamma correction on the color signals RGB. The color difference signal generation unit 206 generates color difference signals RY and BY from the RGB signals.

  The luminance signal Y and the color difference signals RY and BY (image data) generated in this way are temporarily stored in the image memory 106, compressed and encoded by the codec unit 110, and recorded on the recording medium 112. .

  Next, an operation in which the relighting processing unit 113 performs relighting processing on the image output from the image processing unit 105 will be described. Prior to the operation of the relighting processing unit 113, the system control unit 50 calculates control parameters to be set in the relighting processing unit 113 and sets the parameters in the relighting processing unit 113.

  FIG. 4A is a flowchart showing processing of the system control unit 50 during relighting processing. Hereinafter, the operation of the system control unit 50 during the relighting process will be described with reference to FIG.

  In step S <b> 401, an operation from the user for the operation unit 120 is received, and parameters necessary for the relighting process are set in the relighting process unit 113. Specifically, a relighting process is selected from a menu (not shown) by a user operation on the operation unit 120, and parameters of the relighting process are input.

  As an example of the parameters of the relighting process, the relighting area in the image, that is, the center position of the virtual light emitted from the virtual light source, and the virtual light source intensity (α) (virtual light intensity) , And virtual light color gain (Rv_Gain, Bv_Gain). The present invention is not limited to these parameters. For example, other parameters such as the distance from the virtual light source to the subject may be used additionally or in place of the above parameters. These parameters are input by the user and set in the relighting processing unit 113. As a parameter setting method, for example, a method in which an arbitrary value is input by a user or a method in which a parameter set is selected from a predetermined parameter set may be used.

  FIG. 5A shows an example of the center position of the relighting region and the virtual light irradiation range. In FIG. 5A, 501 indicates the center position of the relighting region, 502 indicates the virtual light irradiation range, and the subject in the irradiation range 502 is affected by the virtual light. . The virtual light irradiation range 502 is determined by the virtual light source intensity (α) input by the user, and a circular range is set as the irradiation range from the center position of the relighting area, and the virtual light source intensity (α) is stronger. , The irradiation range becomes wider.

Further, the virtual light color gains (Rv_Gain, Bv_Gain) are set in the relighting processing unit 113 by setting the color of the virtual light to be irradiated as a gain for the environmental light at the time of photographing. If the environment light source and virtual light source are the same,
Rv_Gain = 1
Bv_Gain = 1
It becomes.

  In S402, the relighting processing unit 113 is operated based on these set values.

  FIG. 3 is a block diagram illustrating a configuration of the relighting processing unit 113. Hereinafter, the configuration and operation of the relighting processing unit 113 will be described with reference to FIG.

  The relighting processing unit 113 reads out the luminance signal Y and the color difference signals BY and RY processed by the image processing unit 105 and recorded in the image memory 106 as inputs. The RGB signal conversion unit 301 converts the input luminance signal Y and color difference signals BY and RY into RGB signals and outputs them to the degamma processing unit 302.

  The de-gamma processing unit 302 performs a calculation (de-gamma processing) of characteristics opposite to the gamma characteristics of the gamma correction in the luminance gamma processing unit 202 and the color gamma processing unit 205 of the image processing unit 105, and converts them into linear data. The degamma processing unit 302 outputs the RGB signals (Rt, Gt, Bt) converted into linear data to the virtual light source addition processing unit 303 and the smoothing processing unit 304.

  The smoothing processing unit 304 performs a noise smoothing process using the RGB signals to be processed using peripheral pixels. The virtual light source addition processing unit 303 adds the lighting effect by the virtual light source using the RGB signals (Ra, Ga, Ba) generated by the smoothing processing unit 304 by the process described later, and generates the generated RGB signal (Rout). , Gout, Bout).

  In the smoothing processing unit 304, as the noise smoothing process, for example, nine pixels included in a peripheral 3 × 3 pixel region, for example, centering on the processing target pixel are used, and the color of the peripheral pixel is set for each color component. An average value process for replacing with an average value with the signal value is performed. Of course, the peripheral pixel region used for the average value processing is not limited to 3 × 3 pixels, and can be set as appropriate. In the present embodiment, when this average value processing is performed, a difference absolute value between the processing target pixel and each pixel of a plurality of surrounding pixels is calculated, and when the difference absolute value is equal to or less than the filtering threshold, it is used for the average value processing. Treat as a pixel. Further, pixels whose absolute difference value exceeds the filtering threshold are excluded and are not used for the average value processing. By performing average value processing using only pixels having values close to the pixel to be processed, noise can be reduced while maintaining a sense of resolution such as edges.

  Here, an example of a filtering threshold calculation method used in the smoothing process in the present embodiment will be described. First, as shown in FIG. 6A, a filtering threshold value corresponding to the signal level of the processing target pixel is set in advance, and the value is acquired according to the signal level. As the signal level increases, the amount of noise amplified by the relighting process using the virtual light source to be combined increases. Therefore, the filtering threshold value may be increased to increase the noise suppression effect.

  Alternatively, as shown in FIG. 6B, a filtering threshold value corresponding to the magnitude of the virtual light source intensity is set in advance, and the value is acquired according to the signal level. As the virtual light source intensity increases, the amount of noise amplified by the relighting process using the virtual light source increases. Therefore, the filtering threshold value is increased so that the noise suppression effect is enhanced.

  However, the present invention is not limited to this, and may have a characteristic that the filtering threshold is constant or small from a certain signal level, or a characteristic that the filtering threshold is constant or small from a certain virtual light source intensity.

  Further, a filtering threshold value obtained by combining a filtering threshold value as shown in FIG. 6A and a filtering threshold value as shown in FIG. 6B may be used, and at least one of the filtering threshold values may be used. .

  Further, the smoothing process is not limited to the above average value process, and may be performed by a filtering process (low-pass filter process) having an LPF characteristic as shown in FIGS. The filter characteristics of the LPF used in the low-pass filter processing performed here adaptively change according to the virtual light source intensity. As the virtual light source intensity increases, the amount of noise amplified by the relighting process using the virtual light source increases, so that the filter characteristics are controlled to be narrower and a stronger LPF effect is produced. For example, when processing is performed with a filter characteristic as shown in FIG. 6D at a certain virtual light source intensity, when the virtual light source intensity becomes larger than that, as shown in FIG. The filter characteristics are changed so that the output of the band with high f becomes small.

  Further, the noise smoothing process may be performed by an ε filter process that combines the control of the filter characteristic and the above-described filtering threshold value control.

  In this way, the RGB signals (Ra, Ga, Ba) whose noise has been smoothed are output to the virtual light source addition processing unit 303.

  The virtual light source addition processing unit 303 performs processing for adding the relighting effect by the virtual light source to the input image. Hereinafter, the operation of the virtual light source addition processing unit 303 will be described with reference to the flowchart of FIG. The virtual light source addition processing unit 303 processes the entire input RGB image in units of pixels.

  In S <b> 410, it is determined whether the pixel to be processed is within the virtual light irradiation range 502. If the processing target pixel 503 is within the irradiation range 502, the process proceeds to S411. If the processing target pixel 503 is not within the virtual light irradiation range 502, the light source addition process is not performed, and the process ends.

In S411, virtual light reflection color components (Rv, Gv, Bv) when virtual light is applied to the pixel to be processed are determined. This virtual light reflection color component (Rv, Gv, Bv) is a characteristic including a characteristic when it is photographed by a camera. The virtual light reflection color component (Rv, Gv, Bv) is the virtual light color gain (Rv_Gain) set in S301 with respect to the smoothed image (Ra, Ga, Ba) calculated by the smoothing processing unit 304. , Bv_Gain).
Rv = Rv_Gain × Ra
Gv = Ga
Bv = Bv_Gain × Ba

In S412, as shown in FIG. 5B, a distance D between the processing target pixel 503 and the center position 501 of the virtual light irradiation range 502 is calculated. The distance D is calculated from the coordinates (X, Y) on the image of the processing target pixel 503 and the coordinates (Xc, Yc) of the center position 501 by the following formula.
D = √ ((Xc−X) ² + (Yc−Y) ² )

In S413, based on the virtual light source intensity (α) and the distance D, the virtual light reflection color components (Rv, Gv, Bv) are added to the RGB signals (Rt, Gt, Bt) of the processing target pixel. That is, in this embodiment, output RGB signals (Rout, Gout, Bout) of the processing target pixel irradiated with virtual light are calculated by the following formula. However, the virtual light source intensity (α) is α <1.
Rout = Rt + α × (1 / (D ² )) × Rv
Gout = Gt + α × (1 / (D ² )) × Gv
Bout = Bt + α × (1 / (D ² )) × Bv

  Returning to FIG. 3, the RGB signals (Rout, Gout, Bout) output from the virtual light source addition processing unit 303 are input to the gamma processing unit 305. The gamma processing unit 305 performs gamma correction on the input RGB signals (Rout, Gout, Bout). The luminance / color difference signal conversion unit 306 converts the input RGB signal into a luminance signal Y and color difference signals BY and RY and outputs the converted signals. The above is the operation of the relighting processing unit 113.

  The system control unit 50 accumulates the luminance / color difference signals output from the relighting processing unit 113 in the image memory 106 under the control of the memory control unit 107, and then performs compression encoding in the codec unit 110. In addition, recording is performed on the recording medium 112 via the I / F 111.

  As described above, in this embodiment, the reflected color component obtained by reflecting the actual ambient light to the subject is acquired, and from the acquired reflected color component, the virtual light source intensity, and the virtual light color gain, The reflection components (Rv, Gv, Bv) from which the light is reflected are estimated. And it has the structure which performs a relighting process by adding the reflection component of the estimated virtual light-like light to a process target pixel (Rt, Gt, Bt).

  Furthermore, when calculating virtual light reflection components (Rv, Gv, Bv), the noise reduction process is adaptively controlled to generate a reflection component that maintains the sense of resolution while suppressing the noise component. To do. As a result, the relighting effect can be added to the RGB signals (Rt, Gt, Bt) of the pixel to be processed without raising the noise component as compared with the case of simply adding gain.

  Further, in the present embodiment, the virtual light irradiation range 502 has been described with a circle shown in FIG. 5 for the sake of simplification, but the virtual light influence range is whatever. I do not care. For example, by using a face detection process (not shown), a person's face area may be extracted, and the virtual light influence range may be used as a reference, the virtual light irradiation direction and the distance to the target subject. Depending on, the circle may be deformed. Alternatively, a distance sensor (not shown) may be used to calculate distance information in units of pixels of the captured image, and processing may be performed in which virtual light is applied only to those having a short distance.

  Further, the calculation method of the distance D between the center position of the relighting region and the processing target pixel is not limited to this embodiment, and any calculation method may be used. For example, a camera position and a subject position may be acquired as a three-dimensional position, and a three-dimensional distance may be calculated.

  Further, in the case of adding virtual light, the calculation is performed using an equation that is inversely proportional to the square of the distance D, but the amount of virtual light addition is not limited to that calculated by this method. For example, an expression that is inversely proportional to the distance D or an expression in which the irradiation range changes in a Gaussian distribution may be used.

<Other embodiments>
Note that the present invention may be configured such that an image obtained by an imaging apparatus is transmitted to an external image processing apparatus, and the above-described relighting process is performed on the transmitted image in the image processing apparatus.

  Further, the present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus execute the program. It can also be realized by a process of reading and executing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  50: System control unit, 103: Imaging unit, 105: Image processing unit, 111: Interface, 113: Relighting processing unit, 121: Non-volatile memory, 122: System memory, 120: Operation unit, 303: Virtual light source addition processing unit 304: Smoothing processing unit

Claims (13)

Smoothing means for performing a smoothing process on the first image signal of each pixel of the input image to generate a second image signal;
For each pixel, a third image signal that is a reflection component of virtual light emitted from a virtual light source is generated from the second image signal, and the third image signal and the first image signal are generated. And processing means for adding
An image processing apparatus, wherein smoothing processing by the smoothing means is controlled based on at least one of the signal level of the first image signal and the virtual light intensity.   The image processing apparatus according to claim 1, wherein the processing unit generates the third image signal by processing the second image signal based on a preset parameter.   The image processing apparatus according to claim 2, wherein the parameter includes at least a center position of the virtual light irradiation in the input image, and an intensity and color gain of the virtual light. The smoothing means includes
Find the absolute value of the difference in signal level between the target pixel to be smoothed and each of the peripheral pixels of the target pixel,
4. The average value of signal levels of peripheral pixels excluding peripheral pixels having an absolute value larger than a predetermined threshold among the plurality of peripheral pixels and the signal level of the target pixel is obtained. The image processing apparatus according to any one of the above.   The image processing apparatus according to claim 4, wherein the threshold value increases as the signal level of the target pixel increases.   The threshold value increases as the signal level increases until the signal level of the target pixel reaches a predetermined level, and the threshold value becomes constant or smaller when the signal level becomes higher than the predetermined level. 5. The image processing apparatus according to 4.   The image processing apparatus according to claim 4, wherein the threshold value increases as the virtual light intensity increases.   Until the virtual light intensity reaches a predetermined intensity, the threshold value increases as the virtual light intensity increases. When the virtual light intensity becomes higher, the threshold value becomes constant or smaller. The image processing apparatus according to claim 4, wherein:   The smoothing means performs low-pass filter processing, and the filter characteristic of the low-pass filter processing is narrower as the intensity of the virtual light is higher. The image processing apparatus according to item 1. Imaging means for outputting an image signal obtained by photoelectrically converting incident light; and
An image processing apparatus comprising: the image processing apparatus according to claim 1, wherein the image signal output from the image capturing unit is processed as the first image signal. A smoothing step in which the smoothing means performs a smoothing process on the first image signal of each pixel of the input image to generate a second image signal;
A generating step of generating, for each of the pixels, a third image signal that is a reflection component of virtual light emitted from a virtual light source, from the second image signal;
The processing means includes a processing step of adding the third image signal and the first image signal,
An image processing apparatus that controls a smoothing process in the smoothing step based on at least one of a signal level of the first image signal and an intensity of the virtual light.   The program for functioning a computer as each means of the image processing apparatus of any one of Claim 1 thru | or 9.   A computer-readable storage medium storing the program according to claim 12.

Images