[go: up one dir, main page]

WO2011101036A1 - Traitement de données d'image à ouvertures multiples - Google Patents

Traitement de données d'image à ouvertures multiples Download PDF

Info

Publication number
WO2011101036A1
WO2011101036A1 PCT/EP2010/052154 EP2010052154W WO2011101036A1 WO 2011101036 A1 WO2011101036 A1 WO 2011101036A1 EP 2010052154 W EP2010052154 W EP 2010052154W WO 2011101036 A1 WO2011101036 A1 WO 2011101036A1
Authority
WO
WIPO (PCT)
Prior art keywords
image data
image
aperture
frequency
distance
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/EP2010/052154
Other languages
English (en)
Inventor
Andrew Augustine Wajs
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
IPLINK Ltd
Original Assignee
IPLINK Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by IPLINK Ltd filed Critical IPLINK Ltd
Priority to PCT/EP2010/052154 priority Critical patent/WO2011101036A1/fr
Priority to JP2012553197A priority patent/JP5670481B2/ja
Priority to EP10704373A priority patent/EP2537345A1/fr
Priority to US13/579,568 priority patent/US9495751B2/en
Priority to CN201080066095.7A priority patent/CN103229509B/zh
Publication of WO2011101036A1 publication Critical patent/WO2011101036A1/fr
Anticipated expiration legal-status Critical
Priority to US14/922,817 priority patent/US20160042522A1/en
Ceased legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G06T7/55—Depth or shape recovery from multiple images
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/67—Focus control based on electronic image sensor signals
    • H04N23/673—Focus control based on electronic image sensor signals based on contrast or high frequency components of image signals, e.g. hill climbing method
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/70—Circuitry for compensating brightness variation in the scene
    • H04N23/741—Circuitry for compensating brightness variation in the scene by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/95—Computational photography systems, e.g. light-field imaging systems
    • H04N23/958—Computational photography systems, e.g. light-field imaging systems for extended depth of field imaging
    • H04N23/959—Computational photography systems, e.g. light-field imaging systems for extended depth of field imaging by adjusting depth of field during image capture, e.g. maximising or setting range based on scene characteristics
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10048—Infrared image

Definitions

  • the invention relates to processing multi-aperture image data, and, in particular, though not exclusively, to a method and a system for processing multi-aperture image data, an image processing apparatus for use in such system and a computer program product using such method.
  • PCT applications with international patent application numbers PCT/EP2009/050502 and PCT/EP2009/060936, which are hereby incorporated by reference, describe ways to extend the depth of field of a fixed focus lens imaging system through use of an optical system which combines both color and infrared imaging techniques.
  • the combined use of an image sensor which is adapted for imaging both in the color and the infrared spectrum and a wavelength selective multi-aperture aperture allows extension of depth of field and increase of the ISO speed for digital cameras with a fixed focus lens in a simple and cost effective way. It requires minor adaptations to known digital imaging systems thereby making this process especially suitable for mass production.
  • multi-aperture imaging system provides substantial advantages over known digital imaging systems, such system may not yet provide same functionality as provided in single-lens reflex cameras.
  • the invention may relate to a method for processing multi-aperture image data, wherein the method may comprise:
  • the method allows generation of depth information, which relates objects in an image to an object to camera distance.
  • a depth map associated with a captured image may be generated. The distance information and the depth map allows implementation of image processing functions which may provide a fixed lens imaging system enhanced
  • said at least second and third apertures may be positioned with respect to each other such that high-frequency information in said second image data is displaced a function of the distance between an object and said imaging system.
  • the multi-aperture configuration introduces displacement information in the image data, which may be used for generating depth information.
  • the method may comprise:
  • the method may comprise: identifying a single peak associated with an edge of an imaged object that is in focus and/or identifying double or multiple peaks associated with an imaged object that is out-of-focus ; relating said single peaks and/or the distance between peaks in said double or multiple peaks to a distance between said imaging system and at least one of said objects by using a predetermined depth function.
  • said first part of the electromagnetic spectrum may be associated with at least part of the visible spectrum and/or said second part of the electromagnetic spectrum may be associated with at least part of the invisible spectrum, preferably the infrared spectrum.
  • the use of the infrared spectrum allows efficient use of the sensitivity of the image sensor thereby allowing significant improvement of the signal to noise ratio. Simultaneously capturing an color image and an infrared image using a wavelength-selective multi-aperture diaphragm allows the generation of color images which are enhanced with the sharpness information in the infrared image.
  • the method may comprise: determining said high-frequency second image data by
  • the method comprises:
  • a depth map associated with at least part of said captured image by associating displacement information in said second image data, preferably displacement information in an auto-correlation function of the high-frequency image data associated with said second image data, with a distance between said imaging system and at least one of said objects.
  • a depth map for a captured image may be generated. The depth map associates each pixel data or each groups of pixel data in an image to a distance value.
  • the method comprises: generating at least one image for use in stereoscopic viewing by shifting pixels in said first image data on the basis of said depth information.
  • images may be generated for stereoscopic viewing. These images may be generated on the basis of an image captured by the multi-aperture imaging system and its associated depth map. The captured image may be enhanced with high-frequency infrared information.
  • the method may comprise:
  • the depth information may thus provide control of the depth of field.
  • the method may comprise:
  • the depth information may thus provide control of the focus point.
  • the method may comprise:
  • the depth information may also be used in conventional image processing steps such as
  • the method may comprise: providing at least one threshold peak width and/or peak height
  • the method may comprise:
  • the invention may relate to a multi-aperture system, preferably a wavelength-selective multi-aperture system, more preferably a diaphragm comprising a wavelength-selective multi-aperture system, wherein said multi-aperture system may comprise: at least a first aperture for controlling exposure of an image sensor to at least a first part of the electromagnetic spectrum; at least a second and third aperture for controlling exposure of an image sensor in an imaging system to at least a second part of the
  • second image data associated with said second part of the electromagnetic spectrum, wherein said second and third apertures are positioned with respect to each other such that high-frequency information in said second image data is displaced as a function of the distance between an object and said imaging system.
  • the dimensions of said first aperture may be substantially larger than the dimensions of said second and third aperture.
  • said first aperture may be formed as an opening in an opaque thin-film on a transparent substrate or lens, said opaque thin-film blocking at least both first and second part of said electromagnetic spectrum
  • said at least second and third aperture may be formed as openings in a thin-film filter located within said first aperture, said thin-film filter blocking radiation in said second part of the electromagnetic spectrum and transmitting radiation is said first part of the electromagnetic spectrum.
  • said at least second and third multi apertures may be located as multiple small infrared apertures along the periphery of said first aperture.
  • the invention may relate to a multi-aperture imaging system, comprising: an image sensor; an optical lens system; a wavelength-selective multi-aperture configured for simultaneously exposing said image sensor to spectral energy associated with at least a first part of the electromagnetic spectrum using at least a first aperture and to spectral energy associated with at least a second part of the electromagnetic spectrum using at least a second and third aperture; a first processing module for generating first image data associated with said first part of the electromagnetic spectrum and second image data associated with said second part of the electromagnetic spectrum; and, a second processing module for generating depth information associated with said captured image on the basis displacement information in said second image data, preferably on the basis of displacement information in an auto-correlation function of the high- frequency image data associated with said second image data.
  • invention may related to a method of determining a depth function using multi-aperture image data, comprising: capturing one or more images of one or more objects at different predetermined object-to-camera distances, each image being captured by simultaneously exposing an image sensor in an imaging system to spectral energy associated with at least a first part of the
  • the invention may also relate to a signal processing module, comprising: an input for receiving first captured image data associated with said first part of the
  • electromagnetic spec trum and second captured image data associated with said second part of the electromagnetic spectrum at least one high-pass filter for generating high- frequency data associated with said first and/or second captured image data; an autocorrelation processor for
  • a memory comprising a depth function, said depth function relating displacement
  • a depth information processor for generating depth information on the basis said depth function and displacement information in said second image data, preferably displacement information in an autocorrelation function of the high-frequency image data
  • the invention may also relate to a digital camera, preferably digital camera for use in a mobile terminal, comprising a signal processing module as described above and/or a multi-aperture imaging system as described above.
  • the invention may also relate to a computer program product for processing image data, said computer program product comprising software code portions configured for, when run in the memory of a computer system, executing the method steps according to any of the method as described above.
  • Fig. 1 depicts a multi-aperture imaging system according to one embodiment of the invention.
  • Fig. 2 depicts color responses of a digital camera.
  • Fig. 3 depicts the response of a hot mirror filter and the response of Silicon.
  • Fig. 4 depicts a schematic optical system using a multi-aperture system.
  • Fig. 5 depicts an image processing method for use with a multi-aperture imaging system according to one
  • Fig. 6A depicts a method for determining of a depth function according to one embodiment of the invention.
  • Fig. 6B depicts a schematic of a depth function and graph depicting high-frequency color and infrared information as a function of distance.
  • Fig. 7 depicts a method for generating a depth map according to one embodiment of the invention.
  • Fig. 8 depicts a method for obtaining a stereoscopic view according to one embodiment of the invention.
  • Fig. 9 depicts a method for controlling the depth of field according to one embodiment of the invention.
  • Fig. 10 depicts a method for controlling the focus point according to one embodiment of the invention.
  • Fig. 11 depicts an optical system using a multi- aperture system according to another embodiment of the invention .
  • Fig. 12 depicts a method for determining a depth function according to another embodiment of the invention.
  • Fig. 13 depicts a method for controlling the depth o field according to another embodiment of the invention.
  • Fig. 14 depicts multi-aperture systems for use in multi-aperture imaging system.
  • Fig. 1 illustrates a multi-aperture imaging system 100 according to one embodiment of the invention.
  • the imaging system may be part of a digital camera or integrated in a mobile phone, a webcam, a biometric sensor, image scanner or any other multimedia device requiring image-capturing
  • the system depicted in Fig. 1 comprises an image sensor 102, a lens system 104 for focusing objects in a scene onto the imaging plane of the image sensor, a shutter 106 and an aperture system 108 comprising a predetermined number apertures for allowing light (electromagnetic radiation) of a first part, e.g. a visible part, and at least a second part of the EM spectrum, e.g. a non-visible part such as part of the infrared) of the electromagnetic (EM) spectrum to enter the imaging system in a controlled way.
  • a first part e.g. a visible part
  • EM spectrum e.g. a non-visible part such as part of the infrared
  • the multi-aperture system 108 which will be
  • the multi-aperture system may define at a least first aperture of a first size for exposing the image sensor with a first part of the EM spectrum and at least a second aperture of a second size for exposing the image sensor with a second part of the EM spectrum.
  • the first part of the EM spectrum may relate to the color spectrum and the second part to the infrared spectrum.
  • the multi-aperture system may comprise a predetermined number of apertures each designed to expose the image sensor to radiation within a predetermined range of the EM spectrum.
  • the exposure of the image sensor to EM radiation is controlled by the shutter 106 and the apertures of the multi- aperture system 108.
  • the aperture system controls the amount of light and the degree of
  • the shutter may be a mechanical shutter or, alternatively, the shutter may be an electronic shutter integrated in the image sensor.
  • the image sensor comprises rows and columns of photosensitive sites (pixels) forming a two dimensional pixel array.
  • the image sensor may be a CMOS (Complimentary Metal Oxide Semiconductor) active pixel sensor or a CCD (Charge
  • the image sensor may relate to other Si (e.g. a-Si) , III-V (e.g. GaAs) or conductive polymer based image sensor structures.
  • each pixel When the light is projected by the lens system onto the image sensor, each pixel produces an electrical signal, which is proportional to the electromagnetic radiation
  • a color filter array 120 is interposed between the lens and the image sensor.
  • the color filter array may be integrated with the image sensor such that each pixel of the image sensor has a corresponding pixel filter.
  • Each color filter is adapted to pass light of a predetermined color band into the pixel.
  • RGB red, green and blue
  • Each pixel of the exposed image sensor produces an electrical signal proportional to the electromagnetic
  • the array of pixels thus generates image data (a frame) representing the spatial distribution of the electromagnetic energy (radiation) passed through the color filter array.
  • the signals received from the pixels may be amplified using one or more on-chip amplifiers.
  • each color channel of the image sensor may be amplified using a separate
  • pixel signals may be sampled, quantized and transformed into words of a digital format using one or more Analog to Digital (A/D) converters 110, which may be
  • the digitized image data are processed by a digital signal processor 112
  • DSP coupled to the image sensor, which is configured to perform well known signal processing functions such as interpolation, filtering, white balance, brightness
  • the DSP is coupled to a central processor 114, storage memory 116 for storing captured images and a program memory 118 such as EEPROM or another type of
  • nonvolatile memory comprising one or more software programs used by the DSP for processing the image data or used by a central processor for managing the operation of the imaging system.
  • the DSP may comprise one or more signal processing functions 124 configured for obtaining depth information associated with an image captured by the multi- aperture imaging system. These signal processing functions may provide a fixed-lens multi-aperture imaging system with extended imaging functionality including variable DOF and focus control and stereoscopic 3D image viewing capabilities. The details and the advantages associated with these signal processing functions will be discussed hereunder in more detail.
  • the lens system may be configured to allow both visible light and infrared radiation or at least part of the infrared radiation to enter the imaging system.
  • Filters in front of lens system are configured to allow at least part of the infrared radiation entering the imaging system.
  • these filters do not comprise infrared blocking filters, usually referred to as hot-mirror filters, which are used in conventional color imaging cameras for blocking infrared radiation from entering the camera.
  • the EM radiation 122 entering the multi- aperture imaging system may thus comprise both radiation associated with the visible and the infrared parts of the EM spectrum thereby allowing extension of the photo-response of the image sensor to the infrared spectrum.
  • Fig. 2-3 The effect of (the absence of) an infrared blocking filter on a conventional CFA color image sensor is illustrated in Fig. 2-3.
  • curve 202 represents a typical color response of a digital camera without an infrared blocking filter (hot mirror filter) .
  • Graph A illustrates in more detail the effect of the use of a hot mirror filter.
  • the response of the hot mirror filter 210 limits the spectral response of the image sensor to the visible spectrum thereby substantially limiting the overall sensitivity of the image sensor. If the hot mirror filter is taken away, some of the infrared radiation will pass through the color pixel filters.
  • graph B illustrating the photo- responses of conventional color pixels comprising a blue pixel filter 204, a green pixel filter 206 and a red pixel filter 208.
  • the color pixel filters in particular the red pixel filter, may (partly) transmit infrared radiation so that a part of the pixel signal may be attributed to infrared
  • Fig. 3 depicts the response of the hot mirror filter 302 and the response of Silicon 304 (i.e. the main
  • sensitivity of a Silicon image sensor to infrared radiation is approximately four times higher than its sensitivity to visible light.
  • the image sensor 102 in the imaging system in Fig. 1 may be a conventional image sensor.
  • the infrared radiation is mainly sensed by the red pixels.
  • the DSP may process the red pixel signals in order to extract the low-noise infrared information therein. This process will be described hereunder in more detail.
  • the image sensor may be especially configured for imaging at least part of the infrared spectrum.
  • the image sensor may comprise for example one or more infrared (I) pixels in conjunction with color pixels thereby allowing the image sensor to produce a RGB color image and a relatively low-noise infrared image.
  • An infrared pixel may be realized by covering a photo-site with a filter material, which substantially blocks visible light and substantially transmits infrared radiation, preferably infrared radiation within the range of
  • the infrared transmissive pixel filter may be provided in an infrared/color filter array (ICFA) may be realized using well known filter materials having a high transmittance for wavelengths in the infrared band of the spectrum, for example a black polyimide material sold by Brewer Science under the trademark "DARC 400".
  • ICFA infrared/color filter array
  • An ICFA may contain blocks of pixels, e.g. 2 x 2 pixels, wherein each block comprises a red, green, blue and infrared pixel.
  • image ICFA color image sensor may produce a raw mosaic image comprising both RGB color information and infrared information.
  • a RGB color image and an infrared image may obtained.
  • the sensitivity of such ICFA image color sensor to infrared radiation may be increased by increasing the number of infrared pixels in a block.
  • the image sensor filter array may for example comprise blocks of sixteen pixels, comprising four color pixels RGGB and twelve infrared pixels.
  • the image sensor may relate to an array of photo- sites wherein each photo-site comprises a number of stacked photodiodes well known in the art.
  • each photo-site comprises a number of stacked photodiodes well known in the art.
  • such stacked photo-site comprises at least four stacked photodiodes
  • stacked photodiodes may be integrated into the Silicon substrate of the image sensor.
  • the multi-aperture system e.g. a multi-aperture diaphragm, may be used to improve the depth of field (DOF) of the camera.
  • DOF depth of field
  • the principle of such multi-aperture system 400 is illustrated in Fig. 4.
  • the DOF determines the range of distances from the camera that are in focus when the image is captured. Within this range the object is acceptable sharp.
  • DOF is determined by the focal length of the lens N, the f-number associated with the lens opening (the aperture) , and the ob ect-to-camera distance s. The wider the aperture (the more light received) the more limited the DOF.
  • Visible and infrared spectral energy may enter the imaging system via the multi-aperture system.
  • such multi-aperture system may comprise a filter- coated transparent substrate with a circular hole 402 of a predetermined diameter Dl .
  • the filter coating 404 may transmit visible radiation and reflect and/or absorb infrared
  • An opaque covering 406 may comprise a circular opening with a diameter D2, which is larger than the diameter Dl of the hole 402.
  • the cover may comprise a thin-film coating which reflects both infrared and visible radiation or, alternatively, the cover may be part of an opaque holder for holding and positioning the substrate in the optical system. This way the multi-aperture system comprises multiple
  • wavelength-selective apertures allowing controlled exposure of the image sensor to spectral energy of different parts of the EM spectrum. Visible and infrared spectral energy passing the aperture system is subsequently projected by the lens 412 onto the imaging plane 414 of an image sensor comprising pixels for obtaining image data associated with the visible spectral energy and pixels for obtaining image data associated with the non-visible (infrared) spectral energy.
  • the pixels of the image sensor may thus receive a first (relatively) wide-aperture image signal 416 associated with visible spectral energy having a limited DOF overlaying a second small-aperture image signal 418 associated with the infrared spectral energy having a large DOF.
  • Objects 420 close to the plane of focus N of the lens are projected onto the image plane with relatively small defocus blur by the visible radiation, while objects 422 further located from the plane of focus are projected onto the image plane with relatively small defocus blur by the infrared radiation.
  • a dual or a multiple aperture imaging system uses an aperture system comprising two or more apertures of different sizes for controlling the amount and the collimation of radiation in different bands of the spectrum exposing the image sensor.
  • the DSP may be configured to process the captured color and infrared signals.
  • Fig. 5 depicts typical image processing steps 500 for use with a multi-aperture imaging system.
  • the multi-aperture imaging system comprises a conventional color image sensor using e.g. a Bayer color filter array.
  • the red color pixel data of the captured image frame comprises both a high-amplitude visible red signal and a sharp, low-amplitude non-visible infrared signal.
  • the infrared component may be 8 to 16 times lower than the visible red component.
  • the red balance may be adjusted to compensate for the slight distortion created by the presence of infrared radiation.
  • the an RGBI image sensor may be used wherein the infrared image may be directly obtained by the I-pixels.
  • a first step 502 Bayer filtered raw image data are captured. Thereafter, the DSP may extract the red color image data, which also comprises the infrared information (step 504) . Thereafter, the DSP may extract the sharpness
  • One way of extracting the sharpness information in the spatial domain may be achieved by applying a high pass filter to the red image data.
  • a high-pass filter may retain the high frequency information (high frequency components) within the red image while reducing the low frequency
  • the kernel of the high pass filter may be designed to increase the brightness of the centre pixel relative to neighbouring pixels.
  • the kernel array usually contains a single positive value at its centre, which is completely surrounded by negative values.
  • a simple non- limiting example of a 3x3 kernel for a high-pass filter may look like:
  • the red image data are passed through a high-pass filter (step 506) in order to extract the high-frequency components (i.e. the sharpness information) associated with the infrared image signal.
  • filtered high-frequency components are amplified in proportion to the ratio of the visible light aperture relative to the infrared aperture (step 508).
  • the amplified high-frequency components derived from the infrared image signal are added to (blended with) each color component of the Bayer filtered raw image data (step 510) .
  • the combined image data may be transformed into a full RGB color image using a demosaicking algorithm well known in the art (step 512) .
  • the Bayer filtered raw image data are first demosaicked into a RGB color image and
  • the method depicted in Fig. 5 allows the multi- aperture imaging system to have a wide aperture for effective operation in lower light situations, while at the same time to have a greater DOF resulting in sharper pictures. Further, the method effectively increase the optical performance of lenses, reducing the cost of a lens required to achieve the same performance .
  • the multi-aperture imaging system thus allows a simple mobile phone camera with a typical f-number of 7 (e.g. focal length N of 7 mm and a diameter of 1 mm) to improve its DOF via a second aperture with a f-number varying e.g. between 14 for a diameter of 0,5 mm up to 70 or more for diameters equal to or less than 0,2 mm, wherein the f-number is defined by the ratio of the focal length f and the effective diameter of the aperture.
  • Preferable implementations include optical systems comprising an f-number for the visible radiation of approximately 2 to 4 for increasing the sharpness of near objects in combination with an f-number for the infrared aperture of approximately 16 to 22 for increasing the
  • the multi-aperture imaging system as described with reference to Fig. 1-5, may be used for generating depth information associated with a single captured image.
  • the DSP of the multi- aperture imaging system may comprise at least one depth function, which depends on the parameters of the optical system and which in one embodiment may be determined in advance by the manufacturer and stored in the memory of the camera for use in digital image processing functions.
  • An image may contain different objects located at different distances from the camera lens so that objects closer to the focal plane of the camera will be sharper than objects further away from the focal plane.
  • a depth function may relate sharpness information associated with objects imaged in different areas of the image to information relating to the distance from which these objects are removed from the camera.
  • a depth function R may involve determining the ratio of the sharpness of the color image components and the infrared image components for objects at different distances away from the camera lens.
  • a depth function D may involve autocorrelation analyses of the high-pass filtered infrared image.
  • a depth function R may be defined by the ratio of the sharpness information in the color image and the sharpness information in the infrared image.
  • the sharpness parameter may relate to the so-called circle of confusion, which corresponds to the blur spot diameter measured by the image sensor of an unsharply imaged point in object space.
  • the blur disk diameter representing the defocus blur is very small (zero) for points in the focus plane and progressively grows when moving away to the
  • the increase or decrease in sharpness of the RGB components of a color image relative to the sharpness of the IR components in the infrared image depends on the distance of the imaged object from the lens. For example, if the lens is focused at 3 meters, the sharpness of both the RGB components and the IR components may be the same. In contrast, due to the small aperture used for the infrared image for objects at a distance of 1 meter, the sharpness of the RGB components may be
  • This dependence may be used to estimate the distances of objects from the camera lens.
  • the camera may determine the points in an image where the color and the infrared components are equally sharp. These points in the image correspond to objects, which are located at a relatively large distance (typically the background) from the camera. For objects located away from the hyperfocal distance H, the relative difference in sharpness between the infrared components and the color components will increase as a
  • Fig. 6A depicts a flow diagram 600 associated with the determination of a depth function according to one embodiment of the invention.
  • a test object may be positioned at least at the hyperfocal distance H from the camera.
  • image data are captured using the multi- aperture imaging system. Then, sharpness information
  • step 606-608 associated with a color image and infrared information is extracted from the captured data.
  • the ratio between the sharpness information R(H) is subsequently stored in a memory (step 610) .
  • the test object is moved over a distance ⁇ away from the hyperfocal distance H and R is determined at this distance. This process is repeated until R is determined for all distances up to close to the camera lens (step 612) . These values may be stored into the memory.
  • Interpolation may be used in order to obtain a continuous depth function R(s) (step 614) .
  • R may be defined as the ratio between the absolute value of the high- frequency infrared components D ir and the absolute value of the high-frequency color components D co i measured at a particular spot in the image. .
  • the difference between the infrared and color components in a particular area may be calculated. The sum of the differences in this area may then be taken as a measure of the distance.
  • Fig. 6B depicts a plot of D co i and Di r as a function of distance (graph A) and a plot of as a function of distance (graph B) .
  • graph A it shown that around the focal distance N the high-frequency color components have the highest values and that away from the focal distance high- frequency color components rapidly decrease as a result of blurring effects. Further, as a result of the relatively small infrared aperture, the high-frequency infrared components will have relatively high values over a large distance away from the focal point N.
  • Graph B depicts the resulting depth function R defined as the ratio between D ⁇ r /D col , indicating that for distances substantially larger than the focal distance N the sharpness information is comprised in the high-frequency infrared image data.
  • the depth function R(s) may be obtained by the manufacturer in advance and may be stored in the memory of the camera, where it may be used by the DSP in one or more post-processing functions for processing an image captured by the multi-aperture imaging system.
  • one of the post-processing functions may relate to the generation of a depth map associated with a single image captured by the multi-aperture imaging system.
  • Fig. 7 depicts a schematic of a process for generating such depth map according to one
  • the DSP may separate the color and infrared pixel signals in the captured raw mosaic image using e.g. a known demosaicking algorithm (step 704) . Thereafter, the DSP may use a high-pass filter on the color image data (e.g. an RGB image) and the infrared image data in order to obtain the high frequency components of both image data (step 706).
  • a high-pass filter on the color image data (e.g. an RGB image) and the infrared image data in order to obtain the high frequency components of both image data (step 706).
  • the DSP may associate a distance to each pixel p(i,j) or a group of pixels.
  • the DSP may then associate the measured sharpness ratio R(i,j) at each pixel with a distance s(i, j) to the camera lens (step 710) . This process will generate a distance map wherein each distance value in the map is associated with a pixel in the image.
  • the thus generated map may be stored in a memory of the camera (step 712) .
  • edges in the image may be detected using a well known edge-detection algorithm. Thereafter, the areas around these edges may be used as sample areas for determining distances from the camera lens using the sharpness ration R in these areas. This variant provides the advantage that it requires less computation.
  • the digital imaging processer comprising the depth function may determine an associated depth map ⁇ s(i,j) ⁇ . For each pixel in the pixel frame the depth map comprises an associated distance value.
  • the depth map may be determined by calculating for each pixel p(i,j) an associated depth value s(i,j).
  • the depth map may be determined by associating a depth value with groups of pixels in an image.
  • the depth map may be stored in the memory of the camera together with the captured image in any suitable data format.
  • the process is not limited to the steps described with reference to Fig. 7.
  • the high-pass filtering may applied before the demosaicking step.
  • the high-frequency color image is obtained by demosaicking the high-pass filtered image data.
  • the sharpness information may also be analyzed in the frequency domain.
  • DFT Transform
  • the DFT may be used to calculate the Fourier coefficients of both the color image and the infrared image. Analysis of these coefficients, in particular the high- frequency coefficient, may provide an indication of distance.
  • the absolute difference between the high-frequency DFT coefficients associated with a particular area in the color image and the infrared image may be used as an indication for the distance.
  • the Fourier components may be used for analyzing the cutoff frequency associated with infrared and the color signals. For example if in a particular area of the image the cutoff frequency of the infrared image signals is larger than the cutoff frequency of the color image signal, then this difference may provide an indication of the
  • Fig. 8 depicts a scheme 800 for obtaining a stereoscopic view according to one embodiment of the invention.
  • the original camera position Co positioned at a distance s from an object P
  • two virtual camera positions Ci and C2 (one for the left eye and one for the right eye) may be defined.
  • Each of these virtual camera positions are symmetrically displaced over a distance -t/2 and +t/2 with respect to an original camera position.
  • the amount of pixel shifting required to generate the two shifted "virtual" images associated with the two virtual camera positions may be determined by the expressions:
  • the image processing function may calculate for each pixel po(i,j) in the original image, pixels pi(i,j) and p 2 (i,j) associated with the first and second virtual image (steps 802-806) . This way each pixel po(i,j) in the original image may be shifted in
  • Fig. 9 depicts a further image processing function 900 according to one embodiment. This function allows
  • the optical system delivers images with a fixed (improved) DOF of the optical system. In some circumstances however, it may be desired to have a variable DOF.
  • image data and an associated depth map may be generated. Thereafter, the function may allow selection of a particular distance s' (step 904) which may be used as a cut-off distance after which the sharpness
  • the DSP may identified first areas in an image, which are associated with at an ob ect-to-camera distance larger than the selected distance s' (step 906) and second areas, which are associated with an object-to-camera distance smaller than the selected distance s' . Thereafter, the DSP may retrieve the high- frequency infrared image and set the high-frequency infrared components in the identified first areas to a value according to a masking function (step 910). The thus modified high frequency infrared image may then be blended (step 912) with the RGB image in a similar way as depicted in Fig. 5. That way an RGB image may be obtained wherein the objects in the image which up to a distance s' away from the camera lens are enhanced with the sharpness information obtained from the high-frequency infrared components. This way, the DOF may be reduced in a controlled way.
  • a distance range [si, s2] may be selected by the user of the multi-aperture system. Objects in an image may be related to distances away form the camera. Thereafter, the DSP may determine which object areas are located within this range. These areas are subsequently enhanced by the sharpness information in the high-frequency components.
  • FIG. 10 Yet a further image processing function may relate to controlling the focus point of the camera. This function is schematically depicted in Fig. 10. In this embodiment, a
  • (virtual) focus distance N' may be selected (step 1004) .
  • the areas in the image associated with this selected focus distance may be determined (step 1006).
  • the DSP may generate a high-frequency infrared image (step 1008) and set all high-frequency components outside the identified areas to a value according to a masking function (step 1010) .
  • infrared image may be blended with the RGB image (step 1012), thereby only enhancing the sharpness in the areas in the image associated with the focus distance N' . This way, the focus point in the image may be varied in a controllable way.
  • controlling the focus distance may include selection of multiple focus distances N',N'', etc. For each of these elected distances the associated high- frequency components in the infrared image may be determined. Subsequent modification of the high-frequency infrared image and blending with the color image in a similar way as
  • the focus control as described with reference to Fig. 9 and 10 may be applied to one or more particular areas in an image.
  • a user or the DSP may select one or more particular areas in an image in which focus control is desired.
  • the distance function R(s) and/or depth map may be used for processing said captured image using a known image processing function (e.g. filtering, blending, balancing, ect . ) , wherein one or more image process function parameters associated with such function are
  • a known image processing function e.g. filtering, blending, balancing, ect .
  • the depth information may be used for controlling the cut-off frequency and/or the roll-off of the high-pass filter that is used for generating a high-frequency infrared image.
  • the sharpness information in the color image and the infrared image for a certain area of the image are used for controlling the sharpness information in the color image and the infrared image for a certain area of the image.
  • a high-pass filter having very high cut-off frequency may be used.
  • a high-pass filter having lower cut-off frequency may be used so that the blur in the color image may be compensated by the sharpness information in the infrared image. This way, throughout the image or in specific part of the image, the roll-off and/or the cut-off frequency of the high-pass filter may be adjusted according to the difference in the sharpness information in the color image and the infrared image.
  • Fig. 11 depicts a schematic of a multi-aperture imaging system 1100 for generating a depth information
  • the depth information is obtained through use of a modified multi- aperture configuration.
  • the multi-aperture 1101 in Fig. 11 comprises multiple, (i.e. two or more) small infrared apertures 1102,1104 at the edge (or along the
  • the multi-aperture comprising multiple small infrared apertures allows the formation of discrete, sharp images.
  • the use of multiple infrared apertures allows the use of smaller apertures thereby achieving further enhancement of the depth of field.
  • the displacement distance ⁇ between the two imaged infrared images is a function of the distance between the object and the camera lens and may be used for determining a depth function A(s) .
  • the depth function A(s) may be determined by imaging a test object at multiple distances from the camera lens and measuring ⁇ at those different distances.
  • A(s) may be stored in the memory of the camera, where it may be used by the DSP in one or more post-processing functions as discussed
  • one post-processing functions may relate to the generation of a depth information associated with a single image captured by the multi-aperture imaging system comprising a discrete multiple-aperture as described with reference to Fig. 11.
  • the DSP may separate the color and infrared pixel signals in the captured raw mosaic image using e.g. a known demosaicking algorithm.
  • the DSP may subsequently use a high pass filter on the infrared image data in order to obtain the high frequency components of infrared image data, which may comprise areas where objects are in focus and areas where objects are out-of- focus .
  • the DSP may derive depth information from the high-frequency infrared image data using an
  • a single spike 1206 will appear at the high-frequency edges of an imaged object 1208 that is in focus.
  • the autocorrelation function will generate a double spike 1210 at the high frequency edges of an imaged object 1212 that is out- of-focus.
  • the shift between the spikes represents the shift ⁇ between the two high-frequency infrared images, which is dependent on the distance s between the imaged object and the camera lens.
  • the auto-correlation function of (part of) the high-frequency infrared image will comprise double spikes at locations in the high-frequency infrared image where objects are out ⁇ of-focus and wherein the distance between the double spike provides a distance measure (i.e. a distance away from the focal distance) .
  • the auto-correlation function will comprise a single spike at locations in the image where objects are in focus.
  • the DSP may process the autocorrelation function by associating the distance between the double spikes to a distance using the predetermined depth function A(s) and transform the information therein into a depth map associated with "real distances".
  • control of DOF and focus point may be performed as described above with reference to Fig. 8-10.
  • A(s) or the depth map may be used to select high- frequency components in the infrared image which are
  • Certain image processing functions may be achieved by analyzing the autocorrelation function of the high-frequency infrared image.
  • Fig. 13 depicts for example a process 1300 wherein the DOF is reduced by comparing the width of peaks in the autocorrelation function with a certain threshold width.
  • a first step 1302 an image is captured using a multi- aperture imaging system as depicted in Fig. 11, color and infrared image data are extracted (step 1304) and a high- frequency infrared image data is generated (step 1306) .
  • an autocorrelation function of the high-frequency infrared image data is calculated (step 1308) . Further, a threshold width w is selected (step 1310) . If a peak in the autocorrelation function associated with a certain imaged object is narrower than the threshold width, the high- frequency infrared components associated with that peak in the autocorrelation function are selected for combining with the color image data. If peaks or the distance between two peaks in the autocorrelation function associated with an edge of certain imaged object are wider than the threshold width, the high-frequency components associated with that peak in the correlation function are set in accordance to a masking function (steps 1312-1314) .
  • the thus modified high-frequency infrared image is processed using standard image processing techniques in order to eliminate the shift ⁇ introduced by the multi-aperture so that it may be blended with the color image data (step 1316) .
  • a color image is formed a with reduced DOF is formed. This process allows control of the DOF by selecting a predetermined threshold width.
  • FIG. 14 depicts two non-limiting examples 1402,1410 of a multi-aperture for use in a multi-aperture imaging system as described above.
  • a first multi-aperture 1402 may comprise a transparent substrate with two different thin-film filters: a first circular thin-film filter 1404 in the center of the substrate forming a first aperture transmitting radiation in a first band of the EM spectrum and a second thin-film filter 1406 formed (e.g. in a concentric ring) around the first filter transmitting radiation in a second band of the EM spectrum.
  • the first filter may be configured to transmit both visible and infrared radiation and the second filter may be configured to reflect infrared radiation and to transmit visible radiation.
  • the outer diameter of the outer concentric ring may be defined by an opening in an opaque aperture holder 1408 or, alternatively, by the opening defined in an opaque thin film layer 1408 deposited on the substrate which both blocks infra-read and visible radiation. It is clear for the skilled person that the principle behind the formation of a thin-film multi-aperture may be easily extended to a multi- aperture comprising three or more apertures, wherein each aperture transmits radiation associated with a particular band in the EM spectrum.
  • the second thin-film filter may relate to a dichroic filter which reflects radiation in the infra-red spectrum and transmits radiation in the visible spectrum.
  • Dichroic filters also referred to as interference filters are well known in the art and typically comprise a number of thin-film dielectric layers of specific thicknesses which are configured to reflect infra-red radiation (e.g.
  • radiation having a wavelength between approximately 750 to 1250 nanometers) and to transmit radiation in the visible part of the spectrum.
  • a second multi-aperture 1410 may be used in a multi- aperture system as described with reference to Fig. 11.
  • the multi-aperture comprises a relatively large first aperture 1412 defined as an opening in an opaque
  • aperture holder 1414 or, alternatively, by the opening defined in an opaque thin film layer deposited on a transparent substrate, wherein the opaque thin-film both blocks infra-read and visible radiation.
  • the opaque thin-film both blocks infra-read and visible radiation.
  • multiple small infrared apertures 1416-1422 are defined as openings in a thin-film hot mirror filter 1424, which is formed within the first aperture.
  • multi apertures may be located as multiple small infrared apertures along the
  • Embodiments of the invention may be implemented as a program product for use with a computer system.
  • the program (s) of the program product define functions of the embodiments (including the methods described herein) and can be contained on a variety of computer-readable storage media.
  • Illustrative computer-readable storage media include, but are not limited to: (i) non-writable storage media (e.g., read-only memory devices within a computer such as CD-ROM disks readable by a CD-ROM drive, flash memory, ROM chips or any type of solid- state non-volatile semiconductor memory) on which information is permanently stored; and (ii) writable storage media (e.g., floppy disks within a diskette drive or hard-disk drive or any type of solid-state random-access semiconductor memory) on which alterable information is stored.
  • non-writable storage media e.g., read-only memory devices within a computer such as CD-ROM disks readable by a CD-ROM drive, flash memory,

Landscapes

  • Engineering & Computer Science (AREA)
  • Signal Processing (AREA)
  • Multimedia (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Computing Systems (AREA)
  • General Physics & Mathematics (AREA)
  • Studio Devices (AREA)
  • Color Television Image Signal Generators (AREA)
  • Measurement Of Optical Distance (AREA)
  • Stereoscopic And Panoramic Photography (AREA)
  • Image Processing (AREA)

Abstract

L'invention concerne un procédé et un système de traitement de données d'image à ouvertures multiples ledit procédé consistant à : capturer les données d'image associées à un ou plusieurs objets en exposant simultanément un capteur d'images d'un système d'imagerie à l'énergie spectrale associée à au moins une première partie du spectre électromagnétique en utilisant au moins une première ouverture, et à l'énergie spectrale associée à au moins une seconde partie du spectre électromagnétique en utilisant au moins une deuxième et une troisième ouverture; générer de premières données d'image associées à ladite première partie du spectre électromagnétique et de secondes données d'image associées à ladite seconde partie du spectre électromagnétique; et générer des informations de profondeur associées à ladite image capturée sur la base des informations de déplacement dans lesdites secondes données d'image, de préférence sur la base des informations de déplacement dans une fonction d'autocorrélation des données d'image haute fréquence associées auxdites secondes données d'image.
PCT/EP2010/052154 2010-02-19 2010-02-19 Traitement de données d'image à ouvertures multiples Ceased WO2011101036A1 (fr)

Priority Applications (6)

Application Number Priority Date Filing Date Title
PCT/EP2010/052154 WO2011101036A1 (fr) 2010-02-19 2010-02-19 Traitement de données d'image à ouvertures multiples
JP2012553197A JP5670481B2 (ja) 2010-02-19 2010-02-19 多開口画像データの処理
EP10704373A EP2537345A1 (fr) 2010-02-19 2010-02-19 Traitement de données d'image à ouvertures multiples
US13/579,568 US9495751B2 (en) 2010-02-19 2010-02-19 Processing multi-aperture image data
CN201080066095.7A CN103229509B (zh) 2010-02-19 2010-02-19 处理多孔径像数据
US14/922,817 US20160042522A1 (en) 2010-02-19 2015-10-26 Processing Multi-Aperture Image Data

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/EP2010/052154 WO2011101036A1 (fr) 2010-02-19 2010-02-19 Traitement de données d'image à ouvertures multiples

Related Parent Applications (2)

Application Number Title Priority Date Filing Date
US13/579,569 Continuation-In-Part US20130033579A1 (en) 2010-02-19 2010-02-19 Processing multi-aperture image data
PCT/EP2010/052151 Continuation-In-Part WO2011101035A1 (fr) 2010-02-19 2010-02-19 Traitement de données d'image à ouvertures multiples

Related Child Applications (2)

Application Number Title Priority Date Filing Date
US13/579,568 A-371-Of-International US9495751B2 (en) 2010-02-19 2010-02-19 Processing multi-aperture image data
US14/922,817 Continuation-In-Part US20160042522A1 (en) 2010-02-19 2015-10-26 Processing Multi-Aperture Image Data

Publications (1)

Publication Number Publication Date
WO2011101036A1 true WO2011101036A1 (fr) 2011-08-25

Family

ID=42458671

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/EP2010/052154 Ceased WO2011101036A1 (fr) 2010-02-19 2010-02-19 Traitement de données d'image à ouvertures multiples

Country Status (5)

Country Link
US (1) US9495751B2 (fr)
EP (1) EP2537345A1 (fr)
JP (1) JP5670481B2 (fr)
CN (1) CN103229509B (fr)
WO (1) WO2011101036A1 (fr)

Cited By (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2014008939A1 (fr) 2012-07-12 2014-01-16 Dual Aperture, Inc. Interface utilisateur basée sur des gestes
WO2014006514A3 (fr) * 2012-07-04 2014-05-22 Opera Imaging B.V. Traitement d'image dans une caméra à plusieurs canaux
US8917349B2 (en) 2010-07-16 2014-12-23 Dual Aperture, Inc. Flash system for multi-aperture imaging
US9459695B2 (en) 2012-08-27 2016-10-04 Pixart Imaging Inc. Gesture recognition system and method
EP3151542A1 (fr) * 2015-09-30 2017-04-05 Samsung Electronics Co., Ltd. Dispositif d'imagerie multispectrale pour estimation de distance
US9721357B2 (en) 2015-02-26 2017-08-01 Dual Aperture International Co. Ltd. Multi-aperture depth map using blur kernels and edges
US10146101B2 (en) 2016-12-28 2018-12-04 Axis Ab Method for sequential control of IR-filter, and an assembly performing such method
EP3422700A1 (fr) * 2017-06-29 2019-01-02 Axis AB Procédé d'amélioration d'images en couleurs et ensemble exécutant ledit procédé
US10386554B2 (en) 2016-12-28 2019-08-20 Axis Ab IR-filter arrangement
US10567713B2 (en) 2016-12-28 2020-02-18 Axis Ab Camera and method of producing color images
CN115128799A (zh) * 2020-02-18 2022-09-30 中国人民解放军陆军工程大学 多孔径单探测器光学成像系统

Families Citing this family (34)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8428342B2 (en) * 2010-08-12 2013-04-23 At&T Intellectual Property I, L.P. Apparatus and method for providing three dimensional media content
US9143749B2 (en) * 2011-10-11 2015-09-22 Sony Corporation Light sensitive, low height, and high dynamic range camera
JP5830348B2 (ja) * 2011-10-26 2015-12-09 オリンパス株式会社 撮像装置
CN104205827B (zh) * 2012-03-30 2016-03-16 富士胶片株式会社 图像处理装置及方法、以及摄像装置
JP5687803B2 (ja) 2012-05-09 2015-03-25 富士フイルム株式会社 画像処理装置及び方法並びに撮像装置
EP2959684A4 (fr) * 2013-02-20 2016-11-09 Intel Corp Conversion automatique en temps réel d'images ou d'une vidéo en deux dimensions pour obtenir des images stéréo ou une vidéo en trois dimensions
US20150374210A1 (en) * 2013-03-13 2015-12-31 Massachusetts Institute Of Technology Photometric stereo endoscopy
JP6088042B2 (ja) 2013-03-13 2017-03-01 富士通フロンテック株式会社 画像処理装置、画像処理方法、及びプログラム
CN109120823B (zh) * 2013-08-01 2020-07-14 核心光电有限公司 具有自动聚焦的纤薄多孔径成像系统及其使用方法
KR102066938B1 (ko) * 2013-08-20 2020-01-16 한화테크윈 주식회사 영상 처리 장치 및 방법
KR102112298B1 (ko) * 2013-09-30 2020-05-18 삼성전자주식회사 컬러 영상 및 깊이 영상을 생성하는 방법 및 장치
KR101591172B1 (ko) 2014-04-23 2016-02-03 주식회사 듀얼어퍼처인터네셔널 이미지 센서와 피사체 사이의 거리를 결정하는 방법 및 장치
FR3022377B1 (fr) * 2014-06-13 2017-08-25 Morpho Dispositif d'acquisition optique pour systemes biometriques
KR20170120567A (ko) * 2015-01-20 2017-10-31 재단법인 다차원 스마트 아이티 융합시스템 연구단 이미지로부터 심도 정보를 추출하기 위한 방법 및 장치
US20160255334A1 (en) * 2015-02-26 2016-09-01 Dual Aperture International Co. Ltd. Generating an improved depth map using a multi-aperture imaging system
WO2016137238A1 (fr) * 2015-02-26 2016-09-01 Dual Aperture International Co., Ltd. Traitement de données d'image à ouvertures multiples
US9674466B2 (en) * 2015-02-26 2017-06-06 Dual Aperture International Co., Ltd. Hybrid image correction for dual-aperture camera
US9712809B2 (en) * 2015-05-22 2017-07-18 Intel Corporation Integrated digital camera platform with NIR apodization filter for enhanced depth sensing and image processing
DE102015215837A1 (de) * 2015-08-19 2017-02-23 Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V. Multiaperturabbildungsvorrichtung, Verfahren zum Herstellen derselben und Abbildungssystem
US9456195B1 (en) 2015-10-08 2016-09-27 Dual Aperture International Co. Ltd. Application programming interface for multi-aperture imaging systems
TWI577971B (zh) * 2015-10-22 2017-04-11 原相科技股份有限公司 雙孔徑測距系統
US20170140221A1 (en) * 2015-11-13 2017-05-18 Intel Corporation Dual function camera for infrared and visible light with electrically-controlled filters
US10051263B2 (en) * 2015-11-23 2018-08-14 Center For Integrated Smart Sensors Foundation Multi-aperture camera system using scan line processing
KR101672669B1 (ko) * 2015-11-23 2016-11-03 재단법인 다차원 스마트 아이티 융합시스템 연구단 시차를 이용하는 멀티 애퍼처 카메라 시스템
KR101684194B1 (ko) * 2015-12-17 2016-12-07 재단법인 다차원 스마트 아이티 융합시스템 연구단 초점 맞는 피사체 거리 스캔(focusing distance scan)을 통하여 깊이 정확도를 향상시키는 멀티 애퍼처 카메라 시스템
CN105635548A (zh) * 2016-03-29 2016-06-01 联想(北京)有限公司 一种摄像模组
KR102635884B1 (ko) 2018-10-31 2024-02-14 삼성전자주식회사 조리개를 포함하는 카메라 모듈
DE102019204075B4 (de) * 2019-03-25 2022-03-31 Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V. Vorrichtung mit einer Multiaperturabbildungsvorrichtung zur Erzeugung einer Tiefenkarte
WO2021121540A1 (fr) * 2019-12-16 2021-06-24 Huawei Technologies Co., Ltd. Système d'estimation de profondeur
US11853845B2 (en) * 2020-09-02 2023-12-26 Cognex Corporation Machine vision system and method with multi-aperture optics assembly
US12022214B2 (en) 2020-09-04 2024-06-25 Qualcomm Incorporated Sensitivity-biased pixels
CN116529785A (zh) * 2020-11-23 2023-08-01 指纹卡安娜卡敦知识产权有限公司 包括滤光器的生物特征成像装置和使用生物特征成像装置进行成像的方法
CN115327551B (zh) * 2021-04-25 2025-11-25 上海禾赛科技有限公司 激光雷达
US11606517B1 (en) 2021-06-07 2023-03-14 Waymo Llc Enhanced depth of focus cameras using variable apertures and pixel binning

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2002059692A1 (fr) * 2000-12-22 2002-08-01 Afsenius Sven-Aake Camera combinant les parties les mieux focalisees de differentes expositions d'une image
US20080055400A1 (en) * 1999-02-25 2008-03-06 Visionsense Ltd. Optical device
US20090159799A1 (en) 2007-12-19 2009-06-25 Spectral Instruments, Inc. Color infrared light sensor, camera, and method for capturing images
WO2009097552A1 (fr) * 2008-02-01 2009-08-06 Omnivision Cdm Optics, Inc. Systemes et procedes de fusion de donnees d’image

Family Cites Families (39)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US2921509A (en) * 1955-10-24 1960-01-19 Freund Karl Multiple exposure camera
US4965840A (en) 1987-11-27 1990-10-23 State University Of New York Method and apparatus for determining the distances between surface-patches of a three-dimensional spatial scene and a camera system
JP3614898B2 (ja) * 1994-11-08 2005-01-26 富士写真フイルム株式会社 写真撮影装置、画像処理装置及び立体写真作成方法
US5814803A (en) * 1994-12-23 1998-09-29 Spectra-Physics Scanning Systems, Inc. Image reader with multi-focus lens
US7006132B2 (en) * 1998-02-25 2006-02-28 California Institute Of Technology Aperture coded camera for three dimensional imaging
US8248457B2 (en) * 1999-02-25 2012-08-21 Visionsense, Ltd. Optical device
JP4020527B2 (ja) * 1999-03-16 2007-12-12 オリンパス株式会社 電子カメラ
JP2002344999A (ja) * 2001-05-21 2002-11-29 Asahi Optical Co Ltd ステレオ画像撮像装置
US7224540B2 (en) * 2005-01-31 2007-05-29 Datalogic Scanning, Inc. Extended depth of field imaging system using chromatic aberration
US7577309B2 (en) * 2005-06-18 2009-08-18 Muralidhara Subbarao Direct vision sensor for 3D computer vision, digital imaging, and digital video
US20070102622A1 (en) * 2005-07-01 2007-05-10 Olsen Richard I Apparatus for multiple camera devices and method of operating same
US8384763B2 (en) 2005-07-26 2013-02-26 Her Majesty the Queen in right of Canada as represented by the Minster of Industry, Through the Communications Research Centre Canada Generating a depth map from a two-dimensional source image for stereoscopic and multiview imaging
JP2007139893A (ja) 2005-11-15 2007-06-07 Olympus Corp 合焦検出装置
CN101371568B (zh) * 2006-01-20 2010-06-30 松下电器产业株式会社 复眼方式的照相机模块及其制造方法
US7819591B2 (en) * 2006-02-13 2010-10-26 3M Innovative Properties Company Monocular three-dimensional imaging
US7372642B2 (en) * 2006-02-13 2008-05-13 3M Innovative Properties Company Three-channel camera systems with non-collinear apertures
US20070189750A1 (en) * 2006-02-16 2007-08-16 Sony Corporation Method of and apparatus for simultaneously capturing and generating multiple blurred images
WO2007096816A2 (fr) 2006-02-27 2007-08-30 Koninklijke Philips Electronics N.V. Rendu d'une image de sortie
JP5315574B2 (ja) * 2007-03-22 2013-10-16 富士フイルム株式会社 撮像装置
JP4757221B2 (ja) * 2007-03-30 2011-08-24 富士フイルム株式会社 撮像装置及び方法
US20080252882A1 (en) * 2007-04-13 2008-10-16 John Kesterson Filter assembly and image enhancement system for a surveillance camera and method of using the same
JP4359855B2 (ja) 2007-07-09 2009-11-11 Ｓｍｃ株式会社 電磁弁駆動回路及び電磁弁
US20090015681A1 (en) * 2007-07-12 2009-01-15 Sony Ericsson Mobile Communications Ab Multipoint autofocus for adjusting depth of field
US7956924B2 (en) * 2007-10-18 2011-06-07 Adobe Systems Incorporated Fast computational camera based on two arrays of lenses
JP2009232348A (ja) * 2008-03-25 2009-10-08 Nikon Corp 撮像装置、距離情報取得方法、画像処理方法及び光学系の駆動制御方法
US8958539B2 (en) * 2008-04-23 2015-02-17 Centurylink Intellectual Property Llc System and method for network based call transfers
EP3328048B1 (fr) * 2008-05-20 2021-04-21 FotoNation Limited Capture et traitement d'images au moyen d'un réseau de caméras monolithiques avec des imageurs hétérogènes
GB2463480A (en) * 2008-09-12 2010-03-17 Sharp Kk Camera Having Large Depth of Field
US8558940B2 (en) * 2008-11-27 2013-10-15 Nikon Corporation Image sensor and image-capturing device
US8830377B2 (en) * 2010-01-04 2014-09-09 Duke University Monocentric lens-based multi-scale optical systems and methods of use
US9077916B2 (en) 2009-01-16 2015-07-07 Dual Aperture International Co. Ltd. Improving the depth of field in an imaging system
US8228417B1 (en) * 2009-07-15 2012-07-24 Adobe Systems Incorporated Focused plenoptic camera employing different apertures or filtering at different microlenses
WO2011023224A1 (fr) 2009-08-25 2011-03-03 Iplink Limited Réduction de bruit dans une image couleur
JP5728673B2 (ja) 2010-02-19 2015-06-03 デュアル・アパーチャー・インターナショナル・カンパニー・リミテッド 多開口画像データの処理
US20140192238A1 (en) * 2010-10-24 2014-07-10 Linx Computational Imaging Ltd. System and Method for Imaging and Image Processing
US9247238B2 (en) * 2011-01-31 2016-01-26 Microsoft Technology Licensing, Llc Reducing interference between multiple infra-red depth cameras
US8749636B2 (en) * 2011-07-12 2014-06-10 Lockheed Martin Corporation Passive multi-band aperture filters and cameras therefrom
US9117281B2 (en) * 2011-11-02 2015-08-25 Microsoft Corporation Surface segmentation from RGB and depth images
US9264676B2 (en) * 2012-01-06 2016-02-16 Microsoft Technology Licensing, Llc Broadband imager

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20080055400A1 (en) * 1999-02-25 2008-03-06 Visionsense Ltd. Optical device
WO2002059692A1 (fr) * 2000-12-22 2002-08-01 Afsenius Sven-Aake Camera combinant les parties les mieux focalisees de differentes expositions d'une image
US20090159799A1 (en) 2007-12-19 2009-06-25 Spectral Instruments, Inc. Color infrared light sensor, camera, and method for capturing images
WO2009097552A1 (fr) * 2008-02-01 2009-08-06 Omnivision Cdm Optics, Inc. Systemes et procedes de fusion de donnees d’image

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
KAZUYA KODAMA ET AL: "All-in-Focus Image Generation by Merging Multiple Differently Focused Images in Three-Dimensional Frequency Domain", 1 January 2005, ADVANCES IN MULITMEDIA INFORMATION PROCESSING - PCM 2005 LECTURE NOTES IN COMPUTER SCIENCE;;LNCS, SPRINGER, BERLIN, DE, PAGE(S) 303 - 314, ISBN: 978-3-540-30027-4, XP019023969 *
See also references of EP2537345A1

Cited By (26)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8917349B2 (en) 2010-07-16 2014-12-23 Dual Aperture, Inc. Flash system for multi-aperture imaging
US9635275B2 (en) 2010-07-16 2017-04-25 Dual Aperture International Co. Ltd. Flash system for multi-aperture imaging
US20150124083A1 (en) * 2012-07-04 2015-05-07 Linx Computational Imaging Ltd. Image Processing in a Multi-Channel Camera
WO2014006514A3 (fr) * 2012-07-04 2014-05-22 Opera Imaging B.V. Traitement d'image dans une caméra à plusieurs canaux
US10109063B2 (en) 2012-07-04 2018-10-23 Apple Inc. Image processing in a multi-channel camera
US9733717B2 (en) 2012-07-12 2017-08-15 Dual Aperture International Co. Ltd. Gesture-based user interface
WO2014008939A1 (fr) 2012-07-12 2014-01-16 Dual Aperture, Inc. Interface utilisateur basée sur des gestes
US9459695B2 (en) 2012-08-27 2016-10-04 Pixart Imaging Inc. Gesture recognition system and method
US9721357B2 (en) 2015-02-26 2017-08-01 Dual Aperture International Co. Ltd. Multi-aperture depth map using blur kernels and edges
US9721344B2 (en) 2015-02-26 2017-08-01 Dual Aperture International Co., Ltd. Multi-aperture depth map using partial blurring
US10341585B2 (en) 2015-09-30 2019-07-02 Samsung Electronics Co., Ltd. Electronic device
EP3151542A1 (fr) * 2015-09-30 2017-04-05 Samsung Electronics Co., Ltd. Dispositif d'imagerie multispectrale pour estimation de distance
CN107018294A (zh) * 2015-09-30 2017-08-04 三星电子株式会社 电子装置
US10812738B2 (en) 2015-09-30 2020-10-20 Samsung Electronics Co., Ltd. Electronic device configured to reduce light loss of possible light image and obtain distance information of an object
CN107018294B (zh) * 2015-09-30 2020-07-14 三星电子株式会社 电子装置
US10146101B2 (en) 2016-12-28 2018-12-04 Axis Ab Method for sequential control of IR-filter, and an assembly performing such method
US10386554B2 (en) 2016-12-28 2019-08-20 Axis Ab IR-filter arrangement
US10567713B2 (en) 2016-12-28 2020-02-18 Axis Ab Camera and method of producing color images
US10531058B2 (en) 2017-06-29 2020-01-07 Axis Ab Method for enhancing color images, and an assembly performing such method
TWI684365B (zh) * 2017-06-29 2020-02-01 瑞典商安訊士有限公司 產生彩色影像的攝影機和方法
KR102113488B1 (ko) 2017-06-29 2020-05-22 엑시스 에이비 컬러 이미지를 향상시키기 위한 방법, 및 이러한 방법을 수행하는 어셈블리
KR20190002348A (ko) * 2017-06-29 2019-01-08 엑시스 에이비 컬러 이미지를 향상시키기 위한 방법, 및 이러한 방법을 수행하는 어셈블리
EP3422700A1 (fr) * 2017-06-29 2019-01-02 Axis AB Procédé d'amélioration d'images en couleurs et ensemble exécutant ledit procédé
TWI708510B (zh) * 2017-06-29 2020-10-21 瑞典商安訊士有限公司 用於強化彩色影像的方法以及執行此方法的組件
CN115128799A (zh) * 2020-02-18 2022-09-30 中国人民解放军陆军工程大学 多孔径单探测器光学成像系统
CN115128799B (zh) * 2020-02-18 2023-04-25 中国人民解放军陆军工程大学 多孔径单探测器光学成像系统

Also Published As

Publication number Publication date
US9495751B2 (en) 2016-11-15
CN103229509A (zh) 2013-07-31
CN103229509B (zh) 2016-08-17
JP2013520855A (ja) 2013-06-06
EP2537345A1 (fr) 2012-12-26
US20130033578A1 (en) 2013-02-07
JP5670481B2 (ja) 2015-02-18

Similar Documents

Publication Publication Date Title
US9495751B2 (en) Processing multi-aperture image data
US20130033579A1 (en) Processing multi-aperture image data
EP2594062B1 (fr) Système de flash pour la formation d'une image multi-ouverture
US20160042522A1 (en) Processing Multi-Aperture Image Data
US20160286199A1 (en) Processing Multi-Aperture Image Data for a Compound Imaging System
US9721357B2 (en) Multi-aperture depth map using blur kernels and edges
US20170034456A1 (en) Sensor assembly with selective infrared filter array
TWI496463B (zh) 形成全彩色影像之方法
US20170230638A1 (en) Depth Measurement Techniques for a Multi-Aperture Imaging System
EP3154251A1 (fr) Interface de programmation d'applications pour systèmes d'imagerie à ouvertures multiples
WO2012057622A1 (fr) Système et procédé d'imagerie utilisant un appareil photo à ouvertures multiples
US20160255334A1 (en) Generating an improved depth map using a multi-aperture imaging system
US20110018993A1 (en) Ranging apparatus using split complementary color filters
JP2020038319A (ja) 撮像装置及びその制御方法

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 10704373

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

REEP Request for entry into the european phase

Ref document number: 2010704373

Country of ref document: EP

WWE Wipo information: entry into national phase

Ref document number: 2012553197

Country of ref document: JP

Ref document number: 2010704373

Country of ref document: EP

WWE Wipo information: entry into national phase

Ref document number: 13579568

Country of ref document: US