[go: up one dir, main page]

WO2018168228A1 - Dispositif, procédé, et programme de traitement d'image - Google Patents

Dispositif, procédé, et programme de traitement d'image Download PDF

Info

Publication number
WO2018168228A1
WO2018168228A1 PCT/JP2018/002813 JP2018002813W WO2018168228A1 WO 2018168228 A1 WO2018168228 A1 WO 2018168228A1 JP 2018002813 W JP2018002813 W JP 2018002813W WO 2018168228 A1 WO2018168228 A1 WO 2018168228A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
lens
focal length
image processing
processing apparatus
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2018/002813
Other languages
English (en)
Japanese (ja)
Inventor
修一 麻島
明 徳世
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Sony Corp
Original Assignee
Sony Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Sony Corp filed Critical Sony Corp
Priority to JP2019505752A priority Critical patent/JP7131541B2/ja
Priority to US16/486,597 priority patent/US10944899B2/en
Publication of WO2018168228A1 publication Critical patent/WO2018168228A1/fr
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/64—Computer-aided capture of images, e.g. transfer from script file into camera, check of taken image quality, advice or proposal for image composition or decision on when to take image
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/45—Cameras or camera modules comprising electronic image sensors; Control thereof for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
    • G—PHYSICS
    • G03—PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
    • G03B—APPARATUS OR ARRANGEMENTS FOR TAKING PHOTOGRAPHS OR FOR PROJECTING OR VIEWING THEM; APPARATUS OR ARRANGEMENTS EMPLOYING ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ACCESSORIES THEREFOR
    • G03B17/00—Details of cameras or camera bodies; Accessories therefor
    • G03B17/02—Bodies
    • G03B17/12—Bodies with means for supporting objectives, supplementary lenses, filters, masks, or turrets
    • G03B17/14—Bodies with means for supporting objectives, supplementary lenses, filters, masks, or turrets interchangeably
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G06T7/55—Depth or shape recovery from multiple images
    • G06T7/571—Depth or shape recovery from multiple images from focus
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/66—Remote control of cameras or camera parts, e.g. by remote control devices
    • H04N23/663—Remote control of cameras or camera parts, e.g. by remote control devices for controlling interchangeable camera parts based on electronic image sensor signals
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/667—Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/68—Control of cameras or camera modules for stable pick-up of the scene, e.g. compensating for camera body vibrations
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/80—Camera processing pipelines; Components thereof
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10016—Video; Image sequence

Definitions

  • the present technology relates to an image processing apparatus, an image processing method, and an image processing program.
  • focal length information is recorded as tag information on the lens
  • the focal length information of the lens is transmitted to the camera body
  • the camera body uses various focal length information such as camera shake correction. Processing can be performed.
  • the user when shooting with a so-called old lens that does not have electronic tag information recorded or that does not have a communication function with the camera body, the user must attach the focal length of the old lens to the camera body. It is necessary to input information directly, and it is very troublesome to repeat this operation every time the lens is changed.
  • Patent Document 1 As a solution to this problem, there is a technique using two images taken consecutively (Patent Document 1). In this method, matching is performed between two images, and the focal length is calculated backward from the optical flow. (Patent Document 1).
  • the present technology has been made in view of such problems, and an object thereof is to provide an image processing device, an image processing method, and an image processing program that can obtain the focal length of a lens whose focal length is unknown. .
  • the first technique uses a first image photographed by a first lens having an unknown focal length and a second image photographed by a second lens having a known focal length.
  • the image processing apparatus calculates the focal length of the first lens.
  • the second technique uses the first image captured by the first lens with the unknown focal length and the second image captured by the second lens with the known focal length. This is an image processing method for calculating a distance.
  • the third technique uses the first image captured by the first lens with the unknown focal length and the second image captured by the second lens with the known focal length, to focus the first lens.
  • An image processing program for causing a computer to execute an image processing method for calculating a distance.
  • the focal length of a lens whose focal length is unknown it is possible to obtain the focal length of a lens whose focal length is unknown.
  • the effect described here is not necessarily limited, and may be any effect described in the specification.
  • FIG. 1 is a block diagram illustrating configurations of an image processing unit 400 and an imaging apparatus 500 as image processing apparatuses.
  • the imaging device 500 includes a first lens 100, a mount 150, a second lens 200, and an imaging device body 300.
  • the first lens 100 is a so-called interchangeable lens that can be attached to the imaging apparatus main body 300, and includes an optical imaging system 110 and a lens driving driver 120 in a lens barrel (not shown).
  • the first lens 100 is attached to the imaging apparatus main body 300 via the mount 150.
  • the first lens 100 functions as a main lens in the imaging apparatus 500.
  • the optical imaging system 110 includes a photographic lens for condensing light from a subject on an imaging device, a drive mechanism for moving the photographic lens to perform focusing and zooming, a shutter mechanism, an iris mechanism, and the like. . These are driven based on control signals from the control unit 301 and the lens driving driver 120 of the imaging apparatus main body 300.
  • the optical image of the subject obtained through the optical imaging system 110 is formed on the first imaging element 302 included in the imaging apparatus main body 300.
  • the lens driving driver 120 is configured by, for example, a microcomputer, and moves the photographing lens by a predetermined amount along the optical axis direction according to the control of the control unit 301 of the imaging apparatus main body 300 so as to focus on the target subject. Perform autofocus. Further, in accordance with control from the control unit 301, the operation of the drive mechanism, shutter mechanism, iris mechanism, etc. of the optical imaging system 110 is controlled. Thereby, adjustment of exposure time (shutter speed), aperture value (F value), etc. are made.
  • the first lens 100 has an unknown focal length. Further, the first lens 100 does not have a function of transmitting / receiving information to / from the imaging apparatus main body 300. Therefore, the first lens 100 is one in which the imaging apparatus main body 300 cannot obtain the focal length information of the first lens 100.
  • the focal length of the first lens 100 which is an interchangeable lens whose focal length is unknown, is calculated by image processing. The focal length is the distance from the lens to the image sensor when focusing. Some lenses have a fixed value, and others have a range such as “18 mm-55 mm” by moving the lens in a direction parallel to the optical axis. .
  • the second lens 200 is configured integrally with the imaging apparatus main body 300, and includes an optical imaging system 210 and a lens driving driver 220.
  • the configurations of the optical imaging system 210 and the lens driving driver 220 are the same as those included in the first lens 100.
  • the optical image of the subject obtained by the second lens 200 is formed on the second imaging element 303 provided in the imaging apparatus main body 300.
  • the second lens 200 functions as a sub lens in the imaging apparatus 500.
  • the condition is that the second lens 200 has a known focal length, and the imaging apparatus main body 300 can obtain the focal length information of the second lens 200.
  • the imaging apparatus main body 300 includes a control unit 301, a first imaging element 302, a second imaging element 303, a camera processing circuit 304, an image memory 305, a storage unit 306, a display unit 307, an input unit 308, a motion sensor 309, and an external input / output. 310 and an image processing unit 400.
  • the control unit 301 includes a CPU (Central Processing Unit), a RAM (Random Access Memory), a ROM (Read Only Memory), and the like.
  • the ROM stores a program that is read and operated by the CPU.
  • the RAM is used as a work memory for the CPU.
  • the CPU controls the entire imaging apparatus 500 by executing various processes in accordance with programs stored in the ROM and issuing commands.
  • the first image sensor 302 photoelectrically converts incident light from the subject into a charge amount, and outputs a pixel signal. Then, the first image sensor 302 outputs a pixel signal to the camera processing circuit 304.
  • a CCD Charge-Coupled Device
  • CMOS Complementary Metal-Oxide Semiconductor
  • the second image sensor 303 has the same configuration as the first image sensor 302.
  • the light image of the subject obtained by the first lens 100 is formed on the first image sensor 302, and the light image of the subject obtained by the second lens 200 is formed on the second image sensor 303.
  • an image captured by the first lens 100 is referred to as a first image
  • an image captured by the second lens 200 is referred to as a second image.
  • the first image and the second image are images taken at the same timing in response to a shooting instruction such as pressing the release button by the user.
  • the first image and the second image are images that are photographed so as to include the same subject.
  • a camera processing circuit 304 performs sample hold and AGC (Auto Gain Control) processing for keeping a good S / N (Signal / Noise) ratio by CDS (Correlated Double Sampling) processing on an image signal output from the image sensor.
  • a / D (Analog / Digital) conversion is performed to create an image signal.
  • the camera processing circuit 304 performs predetermined signal processing such as demosaic processing, white balance adjustment processing, color correction processing, gamma correction processing, Y / C conversion processing, AE (Auto-Exposure) processing, resolution conversion processing on an image signal. You may give to.
  • predetermined signal processing such as demosaic processing, white balance adjustment processing, color correction processing, gamma correction processing, Y / C conversion processing, AE (Auto-Exposure) processing, resolution conversion processing on an image signal. You may give to.
  • the image memory 305 is a volatile memory, for example, a buffer memory composed of a DRAM (Dynamic Random Access Memory).
  • the image memory 305 temporarily stores image data that has been subjected to predetermined processing by the camera processing circuit 304.
  • the storage unit 306 is a mass storage medium such as a hard disk or an SD memory card.
  • the image is stored in a compressed state based on a standard such as JPEG (Joint Photographic Experts Group).
  • EXIF Exchangeable Image File Format
  • the moving image is stored in a format such as MPEG2 (Moving / Pictures / Experts / Group2) or MPEG4.
  • focal length information of the first lens 100 calculated by the present technology can also be stored as EXIF information.
  • the display unit 307 is a display device configured by, for example, an LCD (Liquid Crystal Display), a PDP (Plasma Display Panel), an organic EL (Electro Luminescence) panel, or the like.
  • the display unit 307 displays a user interface of the imaging apparatus 500, a menu screen, a monitoring image being shot, a shot image recorded in the storage unit 306, a shot moving image, and the like.
  • the input unit 308 is used for inputting an imaging instruction, various settings, and the like to the imaging apparatus main body 300.
  • a control signal corresponding to the input is generated and output to the control unit 301.
  • the control unit 301 performs arithmetic processing corresponding to the control signal and controls the imaging apparatus 500.
  • the input unit 308 includes a release button, a shooting start instruction button, a power button for switching power on / off, hardware buttons such as a zoom adjustment operator, and a touch panel configured integrally with the display unit 307. There is.
  • the motion sensor 309 detects the movement of the imaging device 500 using, for example, an acceleration sensor, an angular velocity sensor, a gyro sensor, or the like with respect to a biaxial or triaxial direction.
  • the speed at which the rotation angle changes during rotation (angular speed) and the angular speed around the Y-axis direction are detected and output to the control unit 301 and the image processing unit 400.
  • the external input / output 310 is a communication terminal such as a USB (Universal Serial Bus) or a module for the imaging apparatus main body 300 and an external device (such as a personal computer, a tablet terminal, and a smartphone) to transmit and receive various data such as image data, a module, and the like. It is. Transmission / reception of data between the imaging apparatus main body 300 and the external device is not limited to wired communication, and may be wireless communication such as wireless LAN (Local Area Network) such as Wi-Fi, Bluetooth (registered trademark), and ZigBee.
  • wireless LAN Local Area Network
  • Wi-Fi Wireless Fidelity
  • Bluetooth registered trademark
  • ZigBee ZigBee
  • the image processing unit 400 is supplied with a first image taken by the first lens 100 and a second image taken by the second lens 200.
  • the image processing unit 400 calculates the focal length of the first lens 100 by performing predetermined processing on the first image and the second image. Details of the focal length calculation processing by the image processing unit 400 will be described later.
  • the image processing unit 400 is configured by a program, and the program may be installed in the imaging apparatus main body 300 in advance, or may be distributed by download, a storage medium, or the like and installed by the user.
  • the control unit 301 may function as the image processing unit 400 when the control unit 301 executes the program.
  • the image processing unit 400 is not only realized by a program, but may be realized by combining a dedicated device, a circuit, or the like using hardware having the function.
  • the image processing unit 400 and the imaging device 500 as the image processing device are configured as described above.
  • FIG. 2 is a flowchart showing the flow of the focal length calculation process. It is assumed that the first lens 100 is attached to the imaging apparatus main body 300 before the focal length calculation process. In order to perform this focal length calculation process, the user needs to perform photographing (referred to as provisional photographing) for calculating the focal length before photographing a desired subject (referred to as main photographing) with the imaging device 500. The focal length calculation process is performed in the provisional photographing.
  • provisional photographing photographing for calculating the focal length before photographing a desired subject (referred to as main photographing) with the imaging device 500.
  • the focal length calculation process is performed in the provisional photographing.
  • step S11 the imaging apparatus 500 transitions to the focal length calculation mode. This is performed, for example, in response to an input from the user instructing the input unit 308 to transition to the focal length calculation mode.
  • the imaging apparatus 500 enters a state of performing temporary shooting.
  • step S12 the imaging apparatus 500 acquires a captured image in response to a shooting instruction such as pressing the release button from the user.
  • a shooting instruction such as pressing the release button from the user.
  • both the first image obtained by the first lens 100 and the first image sensor 302 and the second image obtained by the second lens 200 and the second image sensor 303 are acquired.
  • the first image and the second image acquired by shooting are supplied to the image processing unit 400.
  • the image processing unit 400 creates a pyramid image.
  • the creation of a pyramid image creates an image group consisting of a series of enlarged images and reduced images in which the magnification (reduction rate and enlargement rate) of the first image is changed at a constant step size. It is processing to do.
  • the pyramid image is created by reducing or enlarging the vertical and horizontal directions at the same magnification with the center of the first image as the origin. Further, the size of the pyramid image is the same as that of the first image as the original image, and the blank area generated by the reduction is filled with an arbitrary value.
  • a plurality of pyramid images are created with the step size of the change in the reduction ratio and the enlargement ratio being 0.1.
  • step S14 the image processing unit 400 performs matching processing between each of the plurality of pyramid images and the second image.
  • this matching process associates each of the pyramid images created in step S13 with the second image, and calculates SAD (Sum of Absolute Difference) (sum of absolute differences) values for each of the pyramid images. Is to be calculated. Details of the matching process will be described later.
  • step S15 the image processing unit 400 determines whether the result of the matching process performed in step S14 has sufficient accuracy to calculate the focal length of the first lens 100. Details of the accuracy determination process will be described later.
  • step S15 when it is determined that the matching process result does not have sufficient accuracy for calculating the focal length, the process proceeds to step S12 (No in step S15). Then, upon receiving a shooting instruction such as pressing the release button from the user, the imaging apparatus 500 acquires the first image and the second image again. This processing is repeated until it is determined in step S15 that the matching processing result has sufficient accuracy for calculating the focal length.
  • step S16 the focal length of the first lens 100 is calculated.
  • the focal length of the first lens 100 is calculated using the following formula (1) from the magnification of the pyramid image with the smallest SAD value calculated in the matching and the focal length of the second camera.
  • the focal length of the second lens 200 is 20 mm
  • the SAD value is the smallest when the magnification of the pyramid image is 1.0 or less, usually when the first lens 100 is telephoto.
  • the SAD value is the smallest when the second lens 200 is telephoto.
  • step S21 a plurality of pyramid images created in step S13 in the flowchart of FIG. 2 are input as matching processing targets.
  • the matching process only the effective area (an area other than the blank area) of the pyramid image is used in the case of the pyramid image obtained by reducing the first image, and the entire image is used in the case of the pyramid image obtained by enlarging the first image. Is used.
  • the imager size is different between the first image sensor 302 and the second image sensor 303, the blurring feeling of the image acquired by photographing is also different, and the matching accuracy is lowered.
  • focus area determination is performed in step S22. This is to determine whether the focused area of each pyramid image is greater than or equal to a predetermined area. This is because when the in-focus area is equal to or smaller than the predetermined area, the in-focus area is small and the focal length cannot be calculated, or even if it is performed, the accuracy is low.
  • each of the pyramid images is divided into a number of areas. Then, it is determined whether each area is an in-focus area or an out-of-focus area. If the in-focus area is a predetermined number or more, it can be determined that the in-focus area is a predetermined area or more.
  • step S23 the reflection of the first lens 100 in the second image is determined.
  • the reflection of the first lens 100 in the second image is as shown in FIG. 6A. If the first lens 100 is reflected in the second image, the first lens 100 becomes a black region. This black area is used as a reflection area. Since the first lens 100 is not reflected in the first image, the focal length cannot be calculated if the area of the reflected region in the second image is too large, or the accuracy is low even if it is performed.
  • the reflection determination can be performed by the method shown in FIG. 6B.
  • the second image is divided into a plurality of strip-shaped areas with a certain width (the divided areas are referred to as strips).
  • the ratio of the reflection area in the strip is calculated for each strip. If the number of strips in which the ratio of the reflection area exceeds a predetermined amount is greater than or equal to the predetermined number, it is determined that the reflection of the first lens 100 in the second image is large and cannot be used for focal length calculation. In this case, the user is notified that another shooting should be performed.
  • step S24 SAD values are calculated for each of the plurality of pyramid images. As shown in FIG. 7, the calculation of the SAD value is performed using the following formula (2) by comparing each of the pyramid images and the second image in the effective pixel region in units of blocks.
  • step S25 the SAD value is normalized based on the number of effective pixels and the number of effective strips in the reflection determination.
  • step S26 it is determined whether or not the step size of the magnification in the pyramid image creation needs to be a smaller value. As the accuracy of matching between the pyramid image and the second image increases, the SAD value decreases. Therefore, to make the step size of the magnification in the pyramid image creation smaller, a more detailed SAD value is calculated and matching is performed. This is to increase the accuracy.
  • step size of the magnification needs to be a smaller value is determined by setting a predetermined threshold value for the SAD value and whether or not the SAD value is equal to or less than the threshold value.
  • the threshold value may be set in advance, or may be set by the user for adjustment of matching accuracy.
  • step S27 all the SAD values calculated for each pyramid image are sorted, and in step S28, information indicating the minimum SAD value and the magnification of the pyramid image corresponding to the minimum SAD value is output. .
  • This minimum SAD value and the magnification of the corresponding pyramid image are used for calculation of the focal length in step S16 of the flowchart of FIG.
  • step S29 a plurality of pyramid images with a smaller step size are created.
  • the pyramid image created in step S13 in the flowchart of FIG. 2 is a plurality of images obtained by enlarging or reducing the first image by changing the magnification by 0.1.
  • a plurality of pyramid images in which the magnification is changed by 0.01 are created.
  • An SAD value is calculated for each pyramid image, and within a range where the SAD value is equal to or less than a predetermined threshold, a plurality of pyramid images having a smaller step size are created to calculate the SAD value.
  • the range where the SAD value is less than or equal to the threshold value is the vicinity of the magnification with the highest matching degree between the pyramid image and the second image. Therefore, the accuracy of matching is improved by reducing the step size of the magnification of the pyramid image within that range. be able to.
  • the range in which the SAD value exceeds the threshold is a range of 0.6 to 0.4 magnification (hatching range), and therefore, in the range of 0.6 to 0.4 as shown in FIG.
  • a pyramid image is created with a scaling factor of 0.01.
  • the SAD value is also calculated for those pyramid images, and the focal length of the first lens 100 is calculated using the magnification of the pyramid image with the smallest SAD value.
  • step S15 of the flowchart of FIG. 2 has sufficient accuracy for calculating the focal length.
  • the determination can be made by confirming whether a predetermined condition is satisfied.
  • the following conditions can be used as the predetermined conditions.
  • the minimum value of the SAD value is not more than a predetermined value
  • the brightness of the first image is not less than the predetermined value
  • the ISO sensitivity of the first image is not more than the predetermined value (4)
  • Photographing (5)
  • the first lens 100 and the second lens 200 are facing in the same direction. (6)
  • the first lens 100 is reflected in the second image. No, or even if it is reflected, the amount of reflection is less than the predetermined amount (7)
  • the first image is in focus
  • the result of the matching process may not have sufficient accuracy to calculate the focal length, and each condition is weighted by priority, A determination of accuracy may be made. Moreover, it may not have sufficient accuracy when the predetermined number is not satisfied among these conditions.
  • the condition (1) can be determined by comparing the minimum value of the SAD value described with reference to FIGS. 8 and 9 with a predetermined threshold.
  • the conditions (2), (3), and (7) can be determined by analyzing the first image.
  • the condition (4) can be determined from the detection result of the motion sensor 309 included in the imaging apparatus main body 300.
  • the condition (5) can be determined by confirming whether the same subject is captured by comparing the first image captured by the first lens 100 and the second image captured by the second lens 200.
  • (6) can be determined by the reflection determination of the first lens 100 in the second image described above with reference to FIG.
  • condition (1) since the minimum value of the SAD value decreases in proportion to the accuracy of matching, if the minimum value of the SAD value is not less than or equal to the predetermined value, matching is not properly performed in the first place. Because there is a fear.
  • the condition (2) is because the matching cannot be performed accurately if the first image is too dark.
  • the condition (3) is because if the ISO sensitivity is too high, a problem such as noise occurs in the image and matching cannot be performed accurately.
  • condition (4) is because if the movement of the imaging apparatus 500 is too large, the subjects in the first image and the second image are blurred and matching cannot be performed accurately.
  • the condition (5) is because when the first lens 100 and the second lens 200 are in different directions, even if the image is captured, a different subject is captured and matching cannot be performed.
  • the condition of (6) is that, even if the first lens 100 of the second image is reflected, the first lens 100 is not reflected in the first image, and therefore matching cannot be performed accurately. is there.
  • the condition (7) is because when the first image is not in focus, the subject is blurred and cannot be accurately matched.
  • step S12 the imaging apparatus 500 captures again upon receiving a shooting instruction such as pressing the release button from the user.
  • a first image and a second image are acquired as images. This processing is repeated until it is determined in step S15 that the matching processing result has sufficient accuracy for calculating the focal length.
  • step S15 the user may be notified that re-shooting is necessary.
  • This notification can be realized, for example, by displaying a predetermined message or mark on the display unit 307 of the imaging apparatus 500.
  • this notification turns on the light provided in the imaging apparatus 500, outputs a predetermined message as a sound from the speaker provided in the imaging apparatus 500, vibrates the imaging apparatus 500 itself, and the like. It can also be realized by a technique. Notification may be performed by using these methods alone or in combination.
  • the focal length of the first lens 100 in the first embodiment is calculated. After calculating the focal length by provisional shooting, the user can perform the main shooting using the calculated focal length information.
  • the focal length of the first lens 100 can be calculated from the first image photographed by the first lens 100 and the second image photographed by the second lens 200.
  • processing using the focal length can be executed.
  • the processing of the imaging apparatus 500 that uses the focal length of the lens such as camera shake correction, can be executed.
  • the calculated focal length can be used for any processing of the imaging apparatus 500 that uses the focal length of the lens.
  • focal length information can be added as Exif information to an image photographed by the imaging apparatus 500. Furthermore, since two images simultaneously shot with different lenses are used, the subject in the temporary shooting for calculating the focal length is not limited to a stationary object.
  • Second Embodiment> [2-1. Focal length calculation process] Next, a second embodiment of the present technology will be described.
  • the focal length of the first lens 100 is calculated in moving image shooting. Since the configuration of the imaging apparatus 500 including the first lens 100, the second lens 200, the imaging apparatus main body 300, and the image processing unit 400 is the same as that of the first embodiment, the description thereof is omitted.
  • FIG. 10 is a flowchart showing the flow of focal length calculation processing in the second embodiment. Steps S11 to S16 are the same flow as those in the first embodiment, and a description thereof will be omitted.
  • the image acquired by the 1st lens 100 and the 2nd lens 200 in step S12 is a frame image which comprises a moving image. The processing from step S13 to step S16 is performed on the frame image.
  • the focal length is updated in step S31.
  • the focal lengths for a total of 10 frames from the first frame to the tenth frame constituting the moving image are stored, and the average is calculated. This average focal length is set as the focal length of the first lens 100.
  • step S32 it is determined whether or not the moving image shooting has been completed. Whether or not the moving image shooting has ended can be determined based on whether or not the user inputs a moving image shooting stop. If the moving image shooting has not been completed, the process proceeds to step S12, and frame images constituting the moving image are acquired (No in step S32).
  • step S16 when the focal length of the eleventh frame is calculated in step S16, the average focal length for the next 10 frames (from the second frame to the eleventh frame) is calculated again in step S31. This average focal length is taken as the focal length of the first lens 100.
  • step S16 when the focal length of the twelfth frame is calculated in step S16, the average focal length for the next 10 frames (from the third frame to the twelfth frame) is calculated again in step S31. This average focal length is taken as the focal length of the first lens 100.
  • step S32 If the moving image shooting has been completed in step S32, the process ends (Yes in step S32).
  • the focal length of the first lens 100 can always be updated to the latest information following the progress of moving image shooting.
  • the greatly different focal length may not be used for calculating the average focal length.
  • the average focal length may be calculated from a total of nine focal lengths excluding the greatly different focal lengths, or a total of ten new focal lengths are added by excluding the greatly different focal lengths.
  • the average focal length may be calculated from the focal length. Whether or not the calculated focal length of one frame is significantly different from the average focal length can be determined by setting a threshold for the average focal length and whether or not the threshold is exceeded.
  • the average focal length is calculated from the focal lengths of 10 frames.
  • the number is not limited to 10, and the average focal length may be calculated with 10 or more frames.
  • the average focal length may be calculated with 10 or less frames.
  • the focal length is calculated as described above.
  • the focal length of the first lens 100 is calculated even when moving image shooting is performed using a lens with an unknown focal length as the first lens 100, and further in accordance with the progress of moving image shooting.
  • the focal length can be updated.
  • the processing of the imaging apparatus 500 using the focal length of the lens such as camera shake correction can be executed.
  • the calculated focal length can be used for any processing of the imaging apparatus 500 that uses the focal length of the lens.
  • FIG. 11 is a diagram illustrating an example of a user interface.
  • a character string as an icon for instructing a transition to a mode for calculating a focal length is arranged on a menu screen displayed on the display unit 307 of the imaging apparatus 500.
  • the imaging apparatus 500 transitions to a provisional shooting mode for calculating the focal length.
  • the menu screen in FIG. 11 is merely an example, and the configuration of the menu screen is not limited to the example in FIG.
  • the imaging apparatus 500 may be provided with a hardware button for making a transition to the temporary shooting mode for calculating the focal length, and when the hardware button is pressed, the transition to the temporary shooting mode may be made.
  • a mechanical switch for detecting the mounting of the first lens 100 on the imaging apparatus main body 300 is provided in the imaging apparatus main body 300, and a temporary shooting mode for automatically calculating the focal length when the mounting of the first lens 100 is detected. You may make it transition to.
  • the focal length of the first lens 100 is reset when the user inputs a reset instruction (initially). ).
  • the focal length may be automatically reset when a predetermined time (for example, one day) elapses from the execution of the focal length calculation using the time counting function provided in the imaging apparatus 500.
  • the focal length calculated last time when the transition to the provisional photographing mode is performed next to calculate the focal length may be reset.
  • single-lens reflex cameras can be applied to digital video cameras, digital cameras, and the like as long as they can be mounted with a separate lens.
  • smartphones, mobile phones, portable game consoles, in-vehicle cameras, surveillance cameras, laptop computers, tablet devices, and watch-type wearables can be used as long as they have a shooting function and can be equipped with a separate lens.
  • the present invention can also be applied to terminals, glasses-type wearable terminals, and the like.
  • the first lens 100 is not limited to an interchangeable lens for a so-called single-lens reflex camera, and may be a so-called lens-style camera that includes a lens and an image sensor and transmits and receives image data by wireless communication with an external device.
  • the second lens 200 may also be a lens style camera.
  • the second lens 200 is not limited to the one built in the image pickup apparatus main body 300, but is a device that is mechanically attached to the image pickup apparatus main body 300 using a mount 150 or the like, or a communication method such as Bluetooth (registered trademark) or Wi-Fi.
  • the lens may be connected to the imaging apparatus main body 300 and capable of transmitting and receiving data to and from the imaging apparatus main body 300.
  • a pyramid image is created by performing both enlargement and reduction.
  • the first lens 100 and the second lens 200 are telephoto, if the first lens 100 is telephoto, the first image is reduced to create a pyramid image, and the second lens 200 is telephoto.
  • the pyramid image may be created by enlarging the first image.
  • the image signal acquired by the first lens 100 and the first image sensor 302 and the image signal acquired by the second lens 200 and the second image sensor 303 are processed by the single camera processing circuit 304. Processing may be performed by a separate camera processing circuit and supplied to the image processing unit 400 that performs focal length calculation.
  • the first lens 100 and the second lens 200 are not parallel, if the parallax is large, the angle of view of the first image and the second image may be too different, and the matching accuracy may be lowered.
  • the corresponding points are extracted from the two images, the first image and the second image, the projective transformation matrix is calculated, and the SAD value is again set after matching the other angle of view with one of the angle of view.
  • a technique for performing matching based on the above may be employed. Thereby, even if the angle of view of the two lenses is different, highly accurate matching can be performed.
  • step S15 if it is determined in step S15 that the result of the matching process does not have sufficient accuracy to calculate the focal length, the process proceeds to step S41 and is obtained by the first lens 100.
  • Two first images may be acquired in succession, and the focal length may be calculated from the continuous images (No in step S15).
  • methods described in Japanese Patent Application Nos. 2004-315927 and 2006-129173 can be employed.
  • the pyramid image is first created with a scale factor of 0.1 and further created with a scale factor of 0.01. However, it may be created with a scale factor of 0.01 from the beginning. Further, the step size of the magnification is not limited to 0.1 times, 0.01 times, etc., and may be any value such as 0.2 times, 0.05 times.
  • the first lens 100 may be an optical system having a plurality of lenses in which a converter lens (conversion lens) having an unknown magnification is mounted on a lens (master lens) having a known focal length.
  • a converter lens is a lens that is attached to a master lens for photographing with a camera, so that photographing can be performed on the wide-angle side or on the telephoto side relative to the focal length of the original optical system.
  • this corresponds to the case where a converter lens is attached to the tip of a lens provided integrally with the camera body, or the case where the converter lens is provided on a mount connecting the camera body and the interchangeable lens. Further, for example, this corresponds to a case where a converter lens is attached to one camera of a smartphone including two cameras.
  • the present technology can also have the following configurations.
  • a matching process is performed between a plurality of pyramid images obtained by enlarging and / or reducing the first image at a plurality of magnifications changed by a first step size, and the second image, and a matching degree among the plurality of pyramid images
  • an SAD value is calculated for each of the plurality of pyramid images from the second image, and a pyramid image having the smallest SAD value is set as an image having the highest matching degree.
  • the first image is enlarged at a plurality of magnifications that are changed at a second step size smaller than the first step size within a range where the SAD value is equal to or smaller than the predetermined value. And / or performing a matching process between the reduced plurality of pyramid images and the second image, and determining the first magnification from the magnification of the image having the highest matching degree among the plurality of pyramid images and the focal length of the second lens.
  • the image processing apparatus according to (4), wherein a focal length of one lens is calculated.
  • the predetermined condition is that there is no reflection of the first lens in the second image, or the reflection area of the first lens in the second image is not more than a predetermined size (6).
  • the image processing apparatus according to any one of 9 The image processing apparatus according to any one of 9.
  • the size of the reflection area of the first lens in the second image is equal to or smaller than a predetermined size, the matching is performed using a region in the second image where the reflection of the first lens is not performed ( The image processing apparatus according to 10).
  • (12) The image processing apparatus according to any one of (6) to (11), wherein the predetermined condition is that the first image is in focus.
  • the image processing apparatus according to any one of (1) to (16), wherein the first lens is a lens in which a converter with an unknown magnification is attached to a lens with a known focal length. (18) It has a mode for calculating the focal length of the first lens, and transitions to the mode triggered by input from a user or mounting of the first lens on an imaging device. Image processing apparatus. (19) An image processing method for calculating a focal length of the first lens using a first image photographed by a first lens having an unknown focal length and a second image photographed by a second lens having a known focal length. .

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Theoretical Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Studio Devices (AREA)
  • Structure And Mechanism Of Cameras (AREA)
  • Cameras In General (AREA)

Abstract

L'invention concerne un dispositif de traitement d'image qui calcule la distance focale d'une première lentille à l'aide d'une première image photographiée avec la première lentille ayant une distance focale inconnue, et une seconde image photographiée avec une seconde lentille ayant une distance focale connue.
PCT/JP2018/002813 2017-03-15 2018-01-30 Dispositif, procédé, et programme de traitement d'image Ceased WO2018168228A1 (fr)

Priority Applications (2)

Application Number Priority Date Filing Date Title
JP2019505752A JP7131541B2 (ja) 2017-03-15 2018-01-30 画像処理装置、画像処理方法および画像処理プログラム
US16/486,597 US10944899B2 (en) 2017-03-15 2018-01-30 Image processing device and image processing method

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2017-049721 2017-03-15
JP2017049721 2017-03-15

Publications (1)

Publication Number Publication Date
WO2018168228A1 true WO2018168228A1 (fr) 2018-09-20

Family

ID=63522904

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2018/002813 Ceased WO2018168228A1 (fr) 2017-03-15 2018-01-30 Dispositif, procédé, et programme de traitement d'image

Country Status (3)

Country Link
US (1) US10944899B2 (fr)
JP (1) JP7131541B2 (fr)
WO (1) WO2018168228A1 (fr)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11102412B1 (en) * 2020-02-13 2021-08-24 Panasonic Intellectual Property Management Co., Ltd. Imaging apparatus
JP7665400B2 (ja) * 2021-04-26 2025-04-21 キヤノン株式会社 電子機器、その制御方法、プログラムおよび記憶媒体

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2011047883A (ja) * 2009-08-28 2011-03-10 Victor Co Of Japan Ltd ３次元形状センサ
WO2011132552A1 (fr) * 2010-04-22 2011-10-27 コニカミノルタホールディングス株式会社 Dispositif de traitement d'informations, programme, procédé de traitement d'informations, et système de traitement d'informations
JP2012133194A (ja) * 2010-12-22 2012-07-12 Nikon Corp 撮像装置

Family Cites Families (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP4094473B2 (ja) 2003-04-18 2008-06-04 新日本製鐵株式会社 高温成形後硬化能に優れた熱間成形加工用鋼板およびその使用方法
JP4652271B2 (ja) 2006-05-08 2011-03-16 富士フイルム株式会社 画像処理装置、及び画像処理システム
JP5413625B2 (ja) 2012-03-09 2014-02-12 カシオ計算機株式会社 画像合成装置及びプログラム
JP6147172B2 (ja) 2013-11-20 2017-06-14 キヤノン株式会社 撮像装置、画像処理装置、画像処理方法、及びプログラム
CN105046676A (zh) 2015-08-27 2015-11-11 上海斐讯数据通信技术有限公司 一种基于智能终端的图像融合方法及设备
CN106339998B (zh) 2016-08-18 2019-11-15 南京理工大学 基于对比度金字塔变换的多聚焦图像融合方法

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2011047883A (ja) * 2009-08-28 2011-03-10 Victor Co Of Japan Ltd ３次元形状センサ
WO2011132552A1 (fr) * 2010-04-22 2011-10-27 コニカミノルタホールディングス株式会社 Dispositif de traitement d'informations, programme, procédé de traitement d'informations, et système de traitement d'informations
JP2012133194A (ja) * 2010-12-22 2012-07-12 Nikon Corp 撮像装置

Also Published As

Publication number Publication date
JP7131541B2 (ja) 2022-09-06
US10944899B2 (en) 2021-03-09
US20190394394A1 (en) 2019-12-26
JPWO2018168228A1 (ja) 2020-01-23

Similar Documents

Publication Publication Date Title
US11039059B2 (en) Imaging capturing device and imaging capturing method
JP4872797B2 (ja) 撮像装置、撮像方法および撮像プログラム
JP6512810B2 (ja) 撮像装置および制御方法とプログラム
US9258478B2 (en) Imaging apparatus, imaging method thereof, and computer readable recording medium
CN107850753A (zh) 检测设备、检测方法、检测程序和成像设备
JP2012090216A (ja) 撮像装置及び撮像装置の制御方法
JP7131541B2 (ja) 画像処理装置、画像処理方法および画像処理プログラム
JP6465705B2 (ja) 動きベクトル検出装置、動体角速度算出装置、撮像装置およびレンズ装置
JP5789725B2 (ja) 撮像装置及びその合焦方法と合焦制御プログラム
JP2015106851A (ja) デジタルカメラ
JP5800600B2 (ja) 撮像装置、撮像方法およびプログラム
WO2010131727A1 (fr) Appareil photo numérique
JP5196768B2 (ja) 撮像装置
WO2020044844A1 (fr) Dispositif de traitement de signal, procédé de traitement, programme de traitement de signal et dispositif de capture d'image
JP6025954B2 (ja) 撮像装置及びその制御方法
JP5182395B2 (ja) 撮像装置、撮像方法および撮像プログラム
JP5744527B2 (ja) 撮像装置及びその制御方法
JP6645614B2 (ja) 撮像装置及び電子機器
KR20130092213A (ko) 디지털 촬영 장치 및 이의 제어 방법
JP6645612B1 (ja) 撮像装置及び電子機器
JP2019216398A (ja) 撮像装置、撮像装置の制御方法、および、プログラム
JP2025119919A (ja) 画像処理装置、撮像装置、画像処理方法、及びプログラム
KR101825147B1 (ko) 디지털 촬영이미지의 흔들림 보정 촬영방법
WO2024048069A1 (fr) Dispositif de traitement d'informations, procédé de traitement d'informations et programme
JP2014215564A (ja) 撮像装置及びその制御方法

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 18768174

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2019505752

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 18768174

Country of ref document: EP

Kind code of ref document: A1