JP7118585B2 - Image processing device, imaging device and control method - Google Patents

Description

The present invention relates to an image processing device, an imaging device, and a control method.

Panning has been proposed as a photographing technique for expressing a sense of speed of a subject (moving body). The purpose of the panning is to freeze the moving subject image and make the background image flow by moving the camera according to the movement of the subject.

During panning, if there is a difference between the moving speed of the subject and the speed of moving the camera, there is a problem that the image of the subject is blurred. Patent Literature 1 discloses an imaging device that absorbs the difference between the moving speed of a subject and the speed of moving a camera by moving a shift lens, thereby making it easier for the user to track the subject. Further, Japanese Patent Application Laid-Open No. 2002-200002 discloses an imaging device that performs tilt correction with high precision using outputs of an acceleration sensor and an angular velocity sensor.

JP-A-2006-317848 JP 2014-199964 A

Since panning is performed while moving the camera, it becomes difficult to keep the camera horizontal, resulting in a tilted image. Therefore, it is necessary to correct the tilt of the image. However, the imaging apparatus disclosed in Patent Document 1 does not have a function of correcting an image according to the inclination of the camera.

In addition, when the photographer keeps the camera horizontal and follows the subject, as shown in (A-1) of FIG. becomes as shown in (A-2). However, when the camera is slightly tilted from the horizontal, as shown in (B-1), the angle of view indicated by the dashed line in the figure is also slightly tilted from the horizontal, so the photographed image becomes (B-2). It tilts like It is conceivable to obtain a corrected image such as (B-3) by correcting the tilt of such an image so as to correspond to when the camera is kept horizontal.

However, in panning, it is not necessarily the photographer's intention to keep the camera horizontal and chase. For example, as shown in (C-1) and (D-1) of FIG. 3, it is assumed that the subject is moving diagonally with respect to the horizontal. In this case, as shown in (C-1), when the photographer keeps the camera horizontal and follows the subject, as shown in (C-2), an image expressing the inclination of the moving direction of the subject with respect to the horizontal is obtained. On the other hand, as shown in (D-1), when the photographer follows the subject by tilting the camera to the same extent as the inclination of the subject, the background is placed against the image as shown in (D-2). It is possible to flow sideways. However, if the technique disclosed in Patent Document 2 is applied to such a case, as shown in (D-3), an image that is not originally intended by the photographer is corrected. SUMMARY OF THE INVENTION It is an object of the present invention to provide an image processing apparatus that corrects the inclination of an image reflecting the intention of the photographer during panning.

An image processing apparatus according to an embodiment of the present invention includes calculation means for calculating a flowing direction of a background in a captured image obtained by panning; detection means for detecting acceleration; and a control means for correcting the tilt of the captured image based on a correction angle calculated according to the detected acceleration, wherein the control means corrects the calculated background flowing direction. correcting the inclination of the captured image when the angle with respect to one direction is smaller than a second threshold, and the angle of the calculated background flowing direction with respect to the first direction is greater than or equal to the second threshold; Correcting the tilt of the captured image when the tilt of the image processing device with respect to the horizontal according to the detected acceleration is less than a first threshold, and correcting the tilt of the captured image when the tilt is greater than or equal to the first threshold. characterized by not

According to the image processing apparatus of the present invention, it is possible to correct the tilt of an image reflecting the intention of the photographer during panning.

1 is a block diagram showing the basic configuration of an imaging device to which an image processing device is applied; FIG. 9 is a flowchart for explaining an example of tilt correction processing for a panning image; FIG. 10 is a diagram illustrating an example of tilt correction according to shooting conditions; FIG. 10 is a diagram illustrating tilt correction according to the tilt of the moving direction of the subject with respect to the horizontal and the tilt of the camera with respect to the horizontal; FIG. 11 is a diagram for explaining processing of the image processing apparatus of Example 4; FIG. 11 is a diagram for explaining processing of the image processing apparatus according to the fifth embodiment;

(Example 1)
FIG. 1 is a block diagram showing the basic configuration of an imaging device to which the image processing device of this embodiment is applied.
The imaging device 100 is, for example, a camera such as a digital camera or a digital video camera, or an electronic device having a camera function such as a camera-equipped mobile phone or a camera-equipped computer. An optical system 101 is an imaging optical system including a lens group, a shutter, an aperture, and the like. The lens group includes a correction lens for correcting camera shake and the like, a focus lens, and the like.

The optical system 101 forms an image of subject light on the image sensor 102 according to a control signal from the CPU 103 . The imaging element 102 is an imaging device such as a CCD (charge-coupled device) image sensor or a CMOS (complementary metal-oxide semiconductor) image sensor, and converts light formed through the optical system 101 into an image signal by photoelectric conversion.

The CPU 103 executes a program pre-stored in the memory, thereby controlling each unit forming the imaging apparatus 100 according to input signals and the like. The primary storage device 104 is a volatile device such as a RAM (Random Access Memory), stores temporary data, and is used as a work memory for the CPU 103 . Information stored in the primary storage device 104 is used by the image processing unit 105 and may be recorded on the recording medium 106 . The secondary storage device 107 is, for example, a non-volatile storage device such as EEPROM (Electrically Erasable Programmable Read-Only Memory). The secondary storage device 107 stores a program (firmware) for controlling the imaging device 100 and various setting information, and is used by the CPU 103 .

A recording medium 106 records captured image data and the like stored in the primary storage device 104 . Also, the recording medium 106 can be removed from the imaging device 100 like a semiconductor memory card, for example. The data recorded on the recording medium 106 can be read out by attaching it to an external device such as a PC (personal computer). In other words, the imaging apparatus 100 has an attachment/detachment mechanism and a read/write function for the recording medium 106 . A display unit 108 displays a viewfinder image at the time of shooting, a shot image, a GUI (Graphical User Interface) image for interactive operation, and the like. An operation unit 109 is an input device group that receives user operations and transmits input information to the CPU 103, and includes, for example, buttons, levers, and a touch panel. An input device using voice, line of sight, or the like can also be used for operation.

The imaging apparatus 100 has a plurality of image processing patterns that the image processing unit 105 applies to the captured image, and these patterns can be set by the operation unit 109 as imaging modes. The image processing unit 105 performs various processing such as color tone adjustment according to the shooting mode, including image processing called so-called development processing. Note that at least part of the functions of the image processing unit 105 may be realized by the CPU 103 through software processing.

FIG. 2 is a flowchart illustrating an example of tilt correction processing for a panning image.
First, in step S200, the CPU 103 acquires a frame. Specifically, the CPU 103 acquires an image before the actual panning shot. Since step S200 is executed multiple times, there are multiple frames.

Next, in step S201, the CPU 103 calculates the moving angle of the background based on the frame image obtained in step S200. That is, the CPU 103 functions as a calculation means for calculating the movement angle of the background in the captured image obtained by panning. Various proposals such as the technique disclosed in Patent Document 1 and the method of calculating a movement vector from a group of frame images can be used for detecting the movement angle of the background, so description thereof will be omitted.

Next, in step S202, the CPU 103 determines whether or not to perform the exposure operation for the main shooting of panning. Specifically, the CPU 103 determines whether the shutter button included in the operation unit 109 has been fully pressed (hereinafter referred to as “S2”). If S2 has not been performed, the CPU 103 repeats the operations from step S200 to step S201. If S2 has been performed, the process proceeds to step S203.

In step S203, the CPU 103 starts the panning exposure operation. Specifically, the CPU 103 controls the shutter included in the optical system 101 . Subsequently, in step S204, the CPU 103 determines whether a predetermined exposure time has elapsed. If the exposure time has not elapsed, the process returns to step S204. If the exposure time has elapsed, the process proceeds to step S205. In step S205, the CPU 103 terminates the panning exposure operation.

Next, in step S206, the CPU 103 functions as a control unit that calculates the correction angle of the captured image according to the background movement angle calculated in S201, and corrects the tilt of the captured image based on the correction angle. . In this example, the CPU 103 calculates the correction angle based on the inclination of the direction in which the background flows with respect to the lateral direction (first direction) of the captured image.

FIG. 3 is a diagram illustrating an example of tilt correction according to shooting conditions.
(B-3), (D-3), (E-3), and (F-3) are the inclinations when the inclination correction corresponding to when the camera is kept horizontal, which is disclosed in Patent Document 2 Fig. 4 shows a corrected image; (B-4), (C-4), (E-4), and (F-4) are tilt-corrected images when tilt correction is performed so as to align the moving direction of the camera and the subject in the present embodiment. indicates

For example, as shown in (A-1), it is assumed that the camera is kept horizontal and the moving direction of the subject is also horizontal. In this case, the horizontal direction of the photographed image and the direction in which the background flows are the same as shown in (A-2), so tilt correction is unnecessary. On the other hand, as shown in (B-1), it is assumed that the camera is tilted from the horizontal. In this case, as shown in (B-2), the direction in which the background flows is tilted with respect to the lateral direction of the captured image. For example, the CPU 103 sets the correction angle θ to an angle (θ=−Δ) that cancels the inclination Δ of the background flow direction with respect to the horizontal direction of the image. As a result, as shown in (B-4), the photographed image shown in (B-2) can be tilt-corrected so that the camera and the movement direction match.

Also, for example, as shown in (D-1), when the user intentionally tilts the camera at the same angle as the moving direction of the subject and takes a picture, the photographed image ((D-2)) is taken in the horizontal direction. Since the direction in which the background flows is the same, tilt correction is unnecessary (Δ=0). Therefore, it is not necessary to perform the correction contrary to the user's intention (tilt correction shown in (D-3)).

In addition, as shown in (E-1), in the case where the image is tilted too much, the photographed image ((E-2)) may be changed by the user as shown in (E-4). It is possible to make a correction that matches the intention.

According to the image processing apparatus of the first embodiment, when the user intentionally tilts the camera at the same angle as the moving direction of the subject in panning, the moving direction of the background becomes the horizontal direction with respect to the image. It is possible to make corrections as follows.

(Example 2)
The image processing apparatus according to the first embodiment performs tilt correction using the tilt in the direction in which the background flows with respect to the horizontal direction of the captured image. The image processing apparatus according to the second embodiment also performs tilt correction using information obtained from a spirit level that detects acceleration. This makes it possible to correct an image that is slightly tilted from the horizontal.

As shown in (B-1) of FIG. 3, even when the user intends to hold the camera horizontally, the camera may be tilted unintentionally. In this case, if the moving direction of the object is horizontal, the tilt correction as shown in (B-4) can be performed by the method of the first embodiment. However, as shown in (C-1), if the moving direction of the object is not horizontal, the tilt of the camera cannot be corrected. Therefore, in the second embodiment, the CPU 103 performs tilt correction based on the tilt of the background flowing direction with respect to the lateral direction of the image and the tilt φ of the camera relative to the horizontal determined according to the acceleration detected by an acceleration sensor such as a spirit level. I do.

In the second embodiment, in S206 of FIG. 2, the CPU 103 determines the correction angle .theta.
θ=−MIN(|φ|,|Δ|)
|x| indicates the absolute value of x, and MIN(x, y) indicates to select the smaller of x and y. In other words, the CPU 103 uses the smaller of the absolute value of the tilt of the camera with respect to the horizontal, which is determined according to the acceleration, and the absolute value of the angle of the flow direction of the background in the captured image with respect to the lateral direction of the captured image. , to calculate the correction angle.

FIG. 4 is a diagram for explaining tilt correction according to the tilt of the moving direction of the subject with respect to the horizontal and the tilt of the camera with respect to the horizontal.
In the first embodiment, when the inclination of the moving direction of the object with respect to the horizontal is different from the inclination of the camera with respect to the horizontal as shown in FIG. 4B, the CPU 103 Tilt correction is performed so that the tilt of the camera matches the tilt of the movement direction. In the second embodiment, the CPU 103 further performs tilt correction as shown in (C) based on the tilt of the camera with respect to the horizontal obtained from the information of an acceleration sensor such as a spirit level. In other words, the CPU 103 corrects the slightly tilted camera with respect to the horizontal so that it becomes equivalent to when the camera is horizontal. In this way, by using an acceleration sensor such as a spirit level, it is possible to correct an object that is tilted from the horizontal, and to correct an object that is intentionally tilted in the same direction as the moving direction of the object by the user. It is possible to do

(Example 3)
In the third embodiment, tilt correction is performed in response to the user intentionally tilting the flowing direction of the background with respect to the lateral direction of the image. (F-1) in FIG. 3 shows a photographing situation when the user tilts the camera more than the moving direction of the subject. In this case, as shown in (F-2), the user's aim is to take a panning shot so that the direction of the background flowing with respect to the horizontal direction of the image is an oblique direction (for example, the longitudinal direction of the image). The point is that it gives a sense of dynamism. Therefore, it is desirable not to perform tilt correction under certain conditions.

In the third embodiment, thresholds Th1 and Th2 regarding correction are provided. Th1 indicates the angle at which the camera is unintentionally tilted when trying to hold it horizontally. As Th1, for example, it is also possible to designate a certain angle. It is also possible for the CPU 103 to determine Th1 based on at least one of the moving speed of the background, the subject distance, and the focal length. For example, when the moving speed of the background is fast, the speed of swinging the camera is fast, so it is conceivable that the camera tends to tilt. Therefore, Th1 is increased in this case. Also, when the object distance or the focal length is long, Th1 may be decreased because the speed of swinging the camera is often slow.

Th2 indicates an angle that is insufficient or excessively tilted in the movement direction when attempting to match the tilt of the camera with the tilt in the movement direction of the subject. As for Th2, it is also possible to specify a constant angle or, like Th1, specify a large threshold because the faster the camera is shaken, the more difficult it is to adjust the angle. By providing Th1 and Th2 in this way, as shown in FIG. 4D, when there is a large difference between the tilt of the moving direction of the subject with respect to the horizontal and the tilt of the camera with respect to the horizontal, tilt correction is not performed. becomes possible. The CPU 103 obtains the correction angle θ at this time, for example, according to the following formula.
θ=−MIN(|φ|, |Δ|) (|φ|<Th1 &|Δ|<Th2)
θ=−φ (|φ|<Th1 & |Δ|≧Th2)
θ=−Δ (|φ|≧Th1 &|Δ|<Th2)
θ=0 (|φ|≧Th1 & |Δ|≧Th2)
As described above, in the third embodiment, the CPU 103 sets the threshold for correction. For example, when the angle φ indicating the tilt of the camera with respect to the horizontal is smaller than a threshold, the CPU 103 calculates the correction angle so as to cancel out the angle φ. Further, for example, the CPU 103 calculates a correction angle to cancel the angle Δ when the angle Δ of the direction in which the background in the captured image flows with respect to the lateral direction of the captured image is smaller than a threshold value. According to the image processing apparatus of the third embodiment, it is possible to perform suitable correction even when the user intentionally inclines the flow direction in order to give a dynamic feeling to the image.

(Example 4)
In the third embodiment, an example has been described in which correction is not performed (θ=0) when the correction angle exceeds the threshold. The image processing apparatus of Example 4 further stabilizes the amount of tilt when the user intentionally tilts the camera.

(F-1) in FIG. 3 shows a photographing situation in which the user intentionally tilts the camera. An image is obtained in which the angle at which the background flows slightly varies each time. Therefore, in the image processing apparatus of the fourth embodiment, the user stabilizes the angle at which the background flows in such a case.

FIG. 5 is a diagram for explaining processing of the image processing apparatus according to the fourth embodiment.
For example, as shown in (A) of FIG. 5, it is assumed that the user intentionally tilts and shoots, and the image is plotted at the position of (B). In order to achieve the same background flow direction as in the photographing situation shown in (A), subsequent photographing must be plotted in the range of (C). Therefore, for example, when photographing plotted at positions (D) and (E) is performed, the CPU 103 performs tilt correction so that the image is on the line (C). This makes it possible to shoot at an angle where the background flows, similar to the shooting in (A). In other words, when the inclination of the background flowing direction with respect to the horizontal direction of the image in the photographing of (A) is Δ1, and the same inclination in the photographing of (D) is Δ2, the correction angle θ is as follows.
θ=−|Δ1−Δ2|
That is, the CPU 103 calculates the correction angle so as to cancel out the difference between the tilt Δ1 regarding the first imaging and the tilt Δ2 regarding the second imaging.

It should be noted that, when photographing another scene, it is necessary to reset the correction based on the correction angle θ once. For example, the correction is reset when a certain period of time elapses or θ becomes equal to or greater than a certain threshold. According to this embodiment, by correcting so as to stabilize the angle at which the background flows, it is possible to provide an image in which the angle at which the background flows is stable when the user continuously photographs the same scene.

(Example 5)
FIG. 6 is a diagram for explaining processing of the image processing apparatus according to the fifth embodiment.
In tilt correction, the larger the correction, the more the image is cropped. For example, in FIG. 6, it is assumed that the image (A-1) is tilt-corrected to the angle (A-2). Since the area outside (A-1) does not have data, the correction result will be a clipped area like (A-3). At this time, in the case shown in FIG. 6A, the subject is cut off. Therefore, in such a case, the CPU 103 limits the correction angle so that the subject is not cut.

In limiting the correction angle, the CPU 103 calculates, for example, the maximum correctable angle. First, the CPU 103 sets an area that includes the moving body area and maintains the original aspect, as shown in (B-1). This area corresponds to the area including the moving object (A-4) area in FIG. 6(A). Let W be the width and H be the height of the region (B-1). In order to rotate by θ while including the area (B-1), it is necessary to secure the area shown by (B-2). At this time, the height H2 and width W2 of the area (B-2) are expressed as follows.
H2=sin(α+θ)×SQRT{(H÷2)̂2+(W÷2)̂2}×2
W2=H2×(W/H)
(SQRT{x} represents the square root of x.)
In order to cut out the area (B-2), the width and height indicated by (B-3) are required. At this time, the height H3 and width W3 of the area (B-3) are expressed as follows.
H3=cos(θ)×H2+sin(θ)×W2
W3=H3×(W/H)
The CPU 103 determines a correction angle θ such that H3 and W3 do not exceed the original image size.

As described above, in this embodiment, the CPU 103 limits the correction angle based on the information on the subject area so that the subject is not cut off by correcting the tilt of the captured image. This makes it possible to prevent the subject from being cut off due to tilt correction. Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist. For example, the above-described Embodiments 1 to 5 may be appropriately combined and applied.

(Other embodiments)
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by processing to It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

100 imaging device 103 CPU

Claims (9)

a calculation means for calculating the flowing direction of the background in the captured image obtained by panning;
detection means for detecting acceleration;
an image processing apparatus comprising: a control unit that corrects the tilt of the captured image based on the calculated direction of background flow and the correction angle calculated according to the detected acceleration,
The control means corrects the tilt of the captured image when the angle of the calculated background flowing direction with respect to the first direction is smaller than a second threshold value, and corrects the tilt of the captured image, is greater than or equal to the second threshold, and the tilt of the image processing device with respect to the horizontal corresponding to the detected acceleration is less than the first threshold, the tilt of the captured image is corrected, and the An image processing apparatus, wherein the tilt of the captured image is not corrected when the tilt is equal to or greater than a threshold value of 1 . The control means controls that the angle of the calculated background flowing direction with respect to the first direction is smaller than the second threshold value, and the tilt of the image processing device with respect to the horizontal corresponding to the detected acceleration is the first direction. 2. The image processing apparatus according to claim 1, wherein the correction angle is calculated according to an angle of the direction in which the background in the captured image flows with respect to the first direction, when the angle is equal to or greater than a threshold of . The image processing apparatus according to claim 2, wherein the first direction is a lateral direction of the captured image. The control means is configured such that the angle of the calculated background flowing direction with respect to the first direction is equal to or greater than the second threshold, and the inclination of the image processing device with respect to the horizontal corresponding to the detected acceleration is the second threshold. 3. The correction angle according to claim 1 or 2, wherein when the acceleration is less than a threshold of 1, the correction angle is calculated based on the tilt of the image processing device with respect to the horizontal corresponding to the detected acceleration. image processing device. The control means controls that the angle of the calculated background flowing direction with respect to the first direction is smaller than the second threshold value, and the tilt of the image processing device with respect to the horizontal corresponding to the detected acceleration is the first direction. of the absolute value of the tilt with respect to the horizontal of the image processing device according to the detected acceleration, and the absolute value of the angle of the flowing direction of the background in the captured image with respect to the first direction , if the threshold is less than 5. The image processing apparatus according to claim 4, wherein the correction angle is calculated using the smaller angle. 6. The controller according to any one of claims 1 to 5 , wherein the controller calculates the first threshold based on at least one of acceleration of the image processing device, subject distance, and focal length. 1. The image processing apparatus according to claim 1. 7. The apparatus according to any one of claims 1 to 6 , wherein the control means limits the correction angle so that the subject is not cut off by correcting the tilt of the captured image based on the information of the subject area. The described image processing device. An imaging apparatus comprising: an acquisition unit configured to photoelectrically convert subject light to acquire the captured image; and the image processing apparatus according to any one of claims 1 to 7 . a calculation step of calculating the flowing direction of the background in the captured image obtained by panning;
a detection step for detecting acceleration;
A control method for an image processing device, comprising a control step of correcting the tilt of the captured image based on the calculated direction of background flow and the correction angle calculated according to the detected acceleration,
The control step corrects the tilt of the captured image when the angle of the calculated background flowing direction with respect to the first direction is smaller than a second threshold value, and corrects the tilt of the captured image. is greater than or equal to the second threshold, and the tilt of the image processing device with respect to the horizontal corresponding to the detected acceleration is less than the first threshold, the tilt of the captured image is corrected, and the A control method for an image processing apparatus, wherein the tilt of the captured image is not corrected when the tilt is equal to or greater than a threshold value of 1 .

Images