WO2025115710A1 - Method used for image correction, and processing circuit for executing said method - Google Patents

Abstract

This method used for image correction includes: employing an image that is generated by imaging a first subject as a first image, employing an image that includes information pertaining to a plurality of wavelength bands and is generated by imaging a second subject as a second image, and acquiring the first image as a correction image for correcting the second image; and determining, on the basis of a pixel value for the first image, whether the first image satisfies a suitability condition as the correction image.

Description

METHOD FOR IMAGE CORRECTION AND PROCESSING CIRCUIT FOR CARRYING OUT THE METHOD - Patent application

The present disclosure relates to a method used to correct an image, and a processing circuit that executes the method.

By utilizing spectral information from a large number of narrow wavelength bands (hereafter simply referred to as "bands"), for example several dozen bands, it becomes possible to grasp the detailed physical properties of a subject, which was not possible with conventional RGB images that have information from three bands of red (R), green (G), and blue (B). A camera that captures images in such many wavelength bands is called a "hyperspectral camera." These cameras are used in a variety of fields, such as food inspection, biomedical testing, pharmaceutical development, and mineral composition analysis.

Patent Document 1 discloses an image analysis device for analyzing the distribution of substances in biological tissue. The image analysis device obtains multiple sample images by illuminating biological tissue with light in multiple wavelength bands selected from a predetermined wavelength range and photographing the tissue. Sample data based on the multiple sample images is compared with teacher data on the substances to generate distribution data of the substances in the tissue. Patent Document 1 discloses normalizing and correcting the intensity of light reflected by a sample based on the intensity of light reflected by a reference member such as a whiteboard.

Patent Document 2 discloses an example of a hyperspectral imaging device that uses compressed sensing technology. Compressive sensing technology is a technology that restores more data than the observed data by assuming that the data distribution of the observed object is sparse in a certain space (e.g., frequency space). The imaging device disclosed in Patent Document 2 is equipped with a coding mask, which is an array of multiple optical filters with different spectral transmittances, on the optical path connecting the subject and the image sensor. The imaging device can generate images of multiple wavelength bands in one shot by performing restoration calculations based on a compressed image acquired by imaging using the coding mask.

International Publication No. 2015/199067 U.S. Pat. No. 9,599,511 JP 2006-153498 A

In the correction disclosed in Patent Document 1, if the subject for correction, such as a reference member, is not appropriate, it is not easy to accurately obtain spectral information of the subject to be analyzed due to inappropriate correction. Therefore, there is a demand for more accurate acquisition of spectral information of the subject to be analyzed by preventing inappropriate correction.

A method used for correcting an image according to one aspect of the present disclosure includes: taking an image generated by photographing a first subject as a first image; taking an image generated by photographing a second subject as a second image, the first image being a correction image for correcting the second image; and determining whether the first image satisfies a suitability condition for the correction image based on pixel values of the first image.

A comprehensive or specific aspect of the present disclosure may be realized in a system, an apparatus, a method, an integrated circuit, a computer program, or a recording medium such as a computer-readable recording disk, or in any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium. A computer-readable recording medium may include a non-volatile recording medium such as a CD-ROM (Compact Disc-Read Only Memory). An apparatus may be composed of one or more devices. When an apparatus is composed of two or more devices, the two or more devices may be located within a single device, or may be located separately within two or more separate devices. In this specification and claims, "apparatus" may mean not only one device, but also a system consisting of multiple devices.

According to one aspect of the present disclosure, by preventing inappropriate corrections, it is possible to obtain more accurate spectral information of the subject being analyzed.

FIG. 1A is a diagram for explaining the relationship between a target wavelength range W and a plurality of wavelength bands W ₁ , W ₂ , . . . , _Wi included therein. FIG. 1B is a diagram illustrating a schematic example of a hyperspectral image. FIG. 2A is a block diagram illustrating a schematic configuration of an imaging system according to an exemplary embodiment of the present disclosure that captures an image of a correction subject. FIG. 2B is a block diagram illustrating a schematic configuration of an imaging system according to an exemplary embodiment of the present disclosure for imaging another subject for correction. FIG. 2C is a block diagram illustrating a schematic configuration of an imaging system according to an exemplary embodiment of the present disclosure that captures an image of a subject to be analyzed. FIG. 3 is a diagram illustrating a specific configuration of an imaging system according to an exemplary embodiment of the present disclosure. FIG. 4A is a diagram illustrating an example of white board correction. FIG. 4B is a diagram illustrating an example of black filtering. FIG. 5 is a flowchart illustrating an outline of a first example of a processing operation executed by a processing circuit in the imaging system according to the present embodiment. FIG. 6A is a diagram illustrating a first hyperspectral image and a reflectance spectrum when a correction subject is appropriate. FIG. 6B is a diagram illustrating a hyperspectral image and a reflectance spectrum when the correction subject is not appropriate. FIG. 7 is a diagram showing a first hyperspectral image and an edge image obtained by performing edge detection on the first hyperspectral image when the correction subject is appropriate and when it is inappropriate. FIG. 8A is a diagram showing a first hyperspectral image and a histogram of pixel values of the first hyperspectral image when another object for correction is appropriate. FIG. 8B is a diagram showing the first hyperspectral image and a histogram of pixel values of the first hyperspectral image when another subject for correction is not appropriate. FIG. 9 is a flowchart illustrating an example 2 of the processing operation executed by the processing circuit in the imaging system according to this embodiment. FIG. 10A is a diagram showing a first compressed image and a histogram of its pixel values when the subject for correction is appropriate. FIG. 10B is a diagram showing the first compressed image and a histogram of its pixel values when the correction subject is not appropriate. FIG. 11 is a flowchart illustrating an example 3 of the processing operation executed by the processing circuit in the imaging system according to the present embodiment. FIG. 12A is a diagram illustrating an example of a UI of the display device. FIG. 12B is a diagram illustrating another example of the UI of the display device. FIG. 12C is a diagram illustrating still another example of a UI of the display device. FIG. 13 is a flowchart outlining a summary of examples 1 to 3 of the processing operation executed by the processing circuit in the imaging system according to this embodiment. FIG. 14 is a diagram showing an example of an image generated by capturing a scene including a correction subject and an analysis target subject with a camera. FIG. 15 is a flowchart illustrating an outline of a fourth example of the processing operation executed by the processing circuit in the imaging system according to the present embodiment. FIG. 16A is a diagram illustrating a schematic configuration example of a compressed sensing type hyperspectral camera. FIG. 16B is a diagram illustrating a schematic configuration example of a compressed sensing type hyperspectral camera. FIG. 16C is a diagram illustrating a schematic diagram of yet another configuration example of a compressed sensing type hyperspectral camera. FIG. 16D is a diagram illustrating a schematic diagram of yet another configuration example of a compressed sensing type hyperspectral camera. FIG. 17A is a schematic diagram illustrating an example of a filter array. FIG. 17B is a diagram showing an example of the spatial distribution of the light transmittance of each of the wavelength bands W ₁ , W ₂ , . . . , W _N included in the target wavelength range. FIG. 17C is a diagram showing an example of the spectral transmittance of a certain region included in the filter array shown in FIG. 17A. FIG. 17D is a diagram showing an example of the spectral transmittance of another region included in the filter array shown in FIG. 17A. FIG. 18A is a diagram for explaining the characteristics of the spectral transmittance in a certain region of the filter array. FIG. 18B is a diagram showing the results of averaging the spectral transmittance shown in FIG. 18A for each of the wavelength bands W ₁ , W ₂ , . . . , W _N.

The embodiments described below are all comprehensive or specific examples. The numerical values, shapes, materials, components, the arrangement and connection forms of the components, steps, and the order of the steps shown in the following embodiments are merely examples and are not intended to limit the technology of the present disclosure. Among the components in the following embodiments, those not described in the independent claims showing the highest concept are described as optional components. Each figure is a schematic diagram and is not necessarily an exact illustration. Furthermore, in each figure, substantially the same or similar components are given the same reference numerals. Duplicate descriptions may be omitted or simplified.

In this disclosure, all or part of a circuit, unit, device, member or part, or all or part of a functional block in a block diagram may be implemented by one or more electronic circuits including, for example, a semiconductor device, a semiconductor integrated circuit (IC), or an LSI (large scale integration). The LSI or IC may be integrated into a single chip, or may be configured by combining multiple chips. For example, functional blocks other than memory elements may be integrated into a single chip. Although referred to as an LSI or IC here, the name may vary depending on the degree of integration, and may be called a system LSI, VLSI (very large scale integration), or ULSI (ultra large scale integration). A Field Programmable Gate Array (FPGA), which is programmed after the LSI is manufactured, or a reconfigurable logic device, which can reconfigure the junctions within the LSI or set up circuit partitions within the LSI, may also be used for the same purpose.

Furthermore, all or part of the functions or operations of a circuit, unit, device, member or part can be executed by software processing. In this case, the software is recorded on one or more non-transitory recording media such as ROM, optical disk, hard disk drive, etc., and when the software is executed by a processor, the functions specified in the software are executed by the processor and peripheral devices.

The system or apparatus may include one or more non-transitory recording media on which the software is recorded, a processor, and any necessary hardware devices, such as interfaces.

(Findings that formed the basis of this disclosure)
Before describing the embodiments of the present disclosure, the findings on which the present disclosure is based will be described.

First, referring to FIG. 1A and FIG. 1B, an example of a hyperspectral image generated by a hyperspectral camera will be briefly described. A hyperspectral image is image data having more wavelength information than a general RGB image. An RGB image has values for each of three bands, red (R), green (G), and blue (B), for each pixel. In contrast, a hyperspectral image has values for more bands for each pixel than the number of bands in an RGB image. In this specification, a "hyperspectral image" means image data including multiple images corresponding to each of four or more multiple bands included in a predetermined target wavelength range. In the following description, the value that each pixel has for each band is referred to as a "pixel value." The number of bands in a hyperspectral image is typically 10 or more, and in some cases may exceed 100. A "hyperspectral image" is also called a "hyperspectral data cube" or a "hyperspectral cube."

FIG. 1A is a diagram for explaining the relationship between the target wavelength range W and a plurality of wavelength bands W ₁ , W ₂ , ..., W _i included therein. The target wavelength range W can be set to various ranges depending on the application. The target wavelength range W can be, for example, a visible light wavelength range of about 400 nm to about 700 nm, a near-infrared wavelength range of about 700 nm to about 2500 nm, or a near-ultraviolet wavelength range of about 10 nm to about 400 nm. Alternatively, the target wavelength range W may be a mid-infrared or far-infrared wavelength range. In this manner, the wavelength range used is not limited to the visible light range. In this specification, not only visible light, but also electromagnetic waves of wavelengths not included in the visible light wavelength range, such as ultraviolet and near-infrared, are referred to as "light" for convenience.

In the example shown in FIG. 1A, N is an arbitrary integer equal to or greater than 4, and the wavelength bands obtained by dividing the target wavelength range W into N equal parts are designated as wavelength bands W ₁ , W ₂ , ..., W _N. However, this is not limited to such an example. The number of wavelength bands included in the target wavelength range W may be set arbitrarily. Each wavelength band may be a wavelength range having a predetermined width, such as 5 nm, 10 nm, 20 nm, or 50 nm. The widths of the multiple wavelength bands may be the same or different. If the number of wavelength bands is four or more, more information can be obtained from the hyperspectral image than from the RGB image.

FIG. 1B is a diagram showing a schematic example of a hyperspectral image. In the example shown in FIG. 1B, the subject is an apple. The hyperspectral image 36 includes images 36W ₁ , 36W ₂ , ..., 36W _N corresponding to wavelength bands W ₁ , W ₂ , ..., W _N , respectively. Each of these images includes a plurality of pixels arranged two-dimensionally. FIG. 1B shows an example of vertical and horizontal dashed lines indicating pixel divisions. The actual number of pixels per image can be a large value, for example, tens of thousands to tens of millions, but in FIG. 1B, for ease of understanding, the pixel divisions are shown as if the number of pixels is extremely small. Reflected light generated when the subject is irradiated with light is detected by a plurality of photodetection elements included in the image sensor. A signal indicating the amount of light detected by each photodetection element represents the pixel value of the pixel corresponding to that photodetection element. Each pixel in the hyperspectral image 36 has a pixel value for each wavelength band. Therefore, by acquiring the hyperspectral image 36, spectral information of the subject can be obtained. Based on the spectral information of the object, it becomes possible to accurately analyze the light characteristics of the object.

Next, a correction method that is performed when a hyperspectral image of an object is acquired by photographing the object with a hyperspectral camera will be described. The method may include, for example, the following operations (1) to (3).
(1) A hyperspectral image for correction is obtained by photographing a correction subject with a hyperspectral camera. The hyperspectral image for correction includes a plurality of correction images respectively corresponding to a plurality of wavelength bands.
(2) A hyperspectral image of the analysis target is obtained by photographing the analysis target object with a hyperspectral camera. The hyperspectral image of the analysis target includes a plurality of images of the analysis target corresponding to a plurality of wavelength bands.
(3) Correcting the hyperspectral image to be analyzed based on the hyperspectral image for correction.

In (3), an example of correcting a hyperspectral image of an analysis target based on a hyperspectral image for correction is normalizing each of a plurality of images of an analysis target by a corresponding correction image among a plurality of correction images in that the wavelength bands are the same, as in the correction disclosed in Patent Document 1. When the subject for correction is a white board, such correction is called "white board correction." White board correction is performed to reduce the effects of the spectral shape of the irradiated light, the illumination distribution during shooting, the peripheral light falloff of the lens, the uneven sensitivity of the image sensor, and the like.

In this specification, "normalizing image A by image B" means dividing the pixel value of each of the multiple pixels in image A by the pixel value of the corresponding pixel among the multiple pixels in image B, and multiplying the result by the maximum pixel value that the pixel can have. If the spatial distribution of the pixel values of image B is approximately constant, the pixel value of one representative pixel or the average of the pixel values of two or more representative pixels may be used instead of the pixel value of the corresponding pixel. The maximum pixel value that the pixel can have is 255 for 8 bits and 4095 for 12 bits.

"Normalizing image A by image B" may be interpreted as determining (pvmax x (pixel value pvA11 of pixel pA11 included in image A)/(pixel value pvB11 of pixel pB11 included in image B)), ..., (pvmax x (pixel value pvAmn of pixel pAmn included in image A)/(pixel value pvBmn of pixel pBmn included in image B)). Images A and B each contain m x n pixels, and the position of pixel pA11 in image A corresponds to pixel pB11 in image B, ..., and the position of pixel pAmn in image A corresponds to pixel pBmn in image B. The maximum value that each of pixel values pvA11, ..., pixel value pvAmn, pixel value pvB11, ..., pixel value pvBmn can take may be pvmax.

In this specification, pixel β in image B can be said to correspond to pixel α in image A if the position of pixel β is the same as the position of pixel α. Alternatively, pixel β in image B can be said to correspond to pixel α in image A if, in an image sensor that outputs an image signal, the position of the photodetection element that outputs the signal of pixel β is the same as the position of the photodetection element that outputs the signal of pixel α.

In addition, in (3), another example of correcting the hyperspectral image to be analyzed based on the hyperspectral image for correction is to subtract from each of the images to be analyzed a corresponding correction image from among multiple correction images in that the correction image has the same wavelength band. When the correction subject is a light-blocking lens cap attached to a hyperspectral camera, the pixel values of the correction image are nearly zero, and this type of correction is called "blackout subtraction." Blackout subtraction is performed to reduce the effects of image sensor dark current, bright spot pixels, fixed pattern noise, and fluctuations in sensor performance.

In this specification, "subtracting image B from image A" means subtracting the pixel value of a corresponding pixel among multiple pixels in image B from the pixel value of each of multiple pixels in image A. "Subtracting image B from image A" may be interpreted as finding ((pixel value pvA11 of pixel pA11 included in image A) - (pixel value pvB11 of pixel pB11 included in image B)), ..., ((pixel value pvAmn of pixel pAmn included in image A) - (pixel value pvBmn of pixel pBmn included in image B)). Images A and B each contain m x n pixels, and the position of pixel pA11 in image A corresponds to pixel pB11 in image B, ..., and the position of pixel pAmn in image A corresponds to pixel pBmn in image B. In black subtraction, correction image B is subtracted from the image to be analyzed A. If the spatial distribution of pixel values in image B is approximately constant, the pixel value of one representative pixel or the average of the pixel values of two or more representative pixels may be used instead of the pixel values of the corresponding pixels.

By performing whiteboard correction and/or black subtraction, it is possible to obtain spectral information of the subject being analyzed based on multiple corrected images corresponding to multiple wavelength bands.

However, in whiteboard correction, if the whiteboard used as the subject for correction is dirty, or if an object other than a whiteboard is mistakenly photographed as the subject for correction, the accuracy of the whiteboard correction decreases. In blackout correction, if the light-blocking lens cap used as the subject for correction is not properly attached to the hyperspectral camera, the accuracy of the blackout correction decreases.

The inventors have found the above problem and have come up with a method for correcting an image according to an embodiment of the present disclosure to solve the problem. A more specific embodiment of the present disclosure will be described below with reference to the drawings.

(Embodiment)
[1. Imaging System]
[1.1. Schematic configuration example of imaging system]
Below, with reference to FIG. 2A to FIG. 2C, a schematic configuration example of the imaging system according to the embodiment of the present disclosure will be described. FIG. 2A to FIG. 2C are block diagrams that show a schematic configuration of the imaging system according to the exemplary embodiment of the present disclosure. FIG. 2A illustrates a white board as the correction subject 10a1. The white board may be, for example, a standard white board with a uniform spatial distribution of reflectance and a small wavelength dispersion of reflectance. FIG. 2B illustrates a light-shielding lens cap that can realize imaging in a light-shielding state as the correction subject 10a2. FIG. 2C illustrates an apple as the subject 10b to be analyzed. The subject 10b is not limited to an apple and may be any object. In this specification, the correction subject 10a1 or the subject 10a2 is also referred to as a "first subject", and the analysis subject 10b is also referred to as a "second subject".

The imaging system 100 shown in Figures 2A to 2C includes a light source 20, a camera 30, a display device 40, and a processing device 50. The processing device 50 includes a processing circuit 52, a memory 54, and a storage device 56. The thick lines with arrows shown in Figures 2A to 2C indicate the flow of signals.

The camera 30 may be, for example, a line scan type or snapshot type hyperspectral camera, which will be described later. In this case, a hyperspectral image for correction is generated by photographing the subject 10a1 or subject 10a2 with the camera 30 as shown in FIG. 2A or FIG. 2B. The hyperspectral image for correction is generated when the user receives an instruction to photograph the subject 10a1 or subject 10a2. Similarly, a hyperspectral image to be analyzed is generated by photographing the subject 10b with the camera 30 as shown in FIG. 2C. The hyperspectral image to be analyzed is generated when the user receives an instruction to photograph the subject 10b.

Alternatively, the camera 30 may be a compressed sensing type hyperspectral camera, which will be described later. In that case, as shown in FIG. 2A or FIG. 2B, a compressed image for correction is generated by photographing the subject 10a1 or the subject 10a2 with the camera 30, and a hyperspectral image for correction is generated based on the compressed image. The compressed image for correction is generated when the user receives an instruction to photograph the subject 10a1 or the subject 10a2. Similarly, as shown in FIG. 2C, a compressed image to be analyzed is generated by photographing the subject 10b with the camera 30, and a hyperspectral image to be analyzed is generated based on the compressed image. The compressed image to be analyzed is generated when the user receives an instruction to photograph the subject 10b.

In this specification, the hyperspectral image for correction or the compressed image for correction is also referred to as the "first image," and the hyperspectral image to be analyzed or the compressed image to be analyzed is also referred to as the "second image." Since a hyperspectral image includes multiple images corresponding to multiple wavelength bands, and information of multiple wavelength bands is compressed in a compressed image, it can be said that the first and second images include information of multiple wavelength bands.

As will be explained in detail later, in the imaging system 100 according to this embodiment, the first image is acquired as a correction image for correcting the second image. When the first image is generated by imaging the subject 10a1 shown in FIG. 2A, the correction of the second image based on the first image is whiteboard correction. When the first image is generated by imaging the subject 10a2 shown in FIG. 2B, the correction of the second image based on the first image is blackout correction.

As mentioned above, in the case of whiteboard correction, if the whiteboard used as the subject 10a1 is dirty, or if an object other than a whiteboard is mistakenly used as the subject 10a1, the accuracy of the whiteboard correction decreases. In the case of blackout correction, if a light-blocking lens cap is not properly attached to the camera 30 as the subject 10a2, the accuracy of the blackout correction decreases.

In this embodiment, the imaging system 100 determines whether the first image satisfies the suitability conditions for an image to be corrected based on the pixel values of the first image. Therefore, even if the user believes that he or she has photographed the appropriate subject 10a1 or subject 10a2 in response to instructions, inappropriate correction can be prevented if the subject 10a1 or subject 10a2 is actually not appropriate. As a result, appropriate correction makes it possible to more accurately obtain spectral information for subject 10b.

In this specification, "spectral information" means information about a spectrum that indicates the wavelength dependency of light intensity. Spectral information may be, for example, information about the spectrum itself, or information from which a spectrum can be derived. The spectrum may be a reflection spectrum or a transmission spectrum.

As used herein, analysis can be, for example, determining characteristics of the subject 10b, such as sugar content and ripeness, and inspecting the subject 10b for defects and/or foreign matter. Such analysis includes human evaluation as well as machine processing.

The components of the imaging system 100 are described below.

<Light source 20>
The light source 20 emits illumination light for illuminating the subject 10a1 or the subject 10b. The illumination light includes light of a plurality of wavelength bands. The light source 20 may be, for example, an incandescent lamp, a halogen lamp, a mercury lamp, a fluorescent lamp, or an LED lamp that emits white light. The hollow arrows shown in Figures 2A and 2C represent the light emitted from the light source 20 and the light reflected from the subject 10a1 and the subject 10b.

Note that the light source 20 is not necessarily required. If the imaging system 100 does not include the light source 20, the subject 10a1 or the subject 10b may be illuminated with sunlight or indoor light.

<Camera 30>
The camera 30 captures an image of the subject 10a1 or 10a2 as shown in Fig. 2A or 2B. Similarly, the camera 30 captures an image of the subject 10b as shown in Fig. 2C.

The target wavelength range W shown in FIG. 1A is a wavelength range in which the camera 30 can detect light. When the camera 30 includes an optical system and an image sensor, the target wavelength range W can be determined, for example, based on the transmission range of the optical system and the sensitivity range of the image sensor. When the transmission range of the optical system includes the sensitivity range of the image sensor, the target wavelength range W is determined by the sensitivity range of the image sensor. When the camera 30 further includes a bandpass filter, the target wavelength range W can be determined based on the transmission range of the bandpass filter in addition to the transmission range of the optical system and the sensitivity range of the image sensor. When the transmission range of the optical system includes the sensitivity range of the image sensor and the sensitivity range of the image sensor includes the transmission range of the bandpass filter, the target wavelength range W is determined by the transmission range of the bandpass filter.

Examples of the camera 30 include line-scan, snapshot, and compressed sensing hyperspectral cameras. Below, we explain the representative components and operation of each hyperspectral camera.

Line-scan hyperspectral camera When the camera 30 is a line-scan hyperspectral camera, the camera 30 includes a prism or a diffraction grating, an image sensor, and a slide mechanism for sliding the object to be photographed in one direction. The light source 20 emits a line beam extending in a direction perpendicular to the one direction. The object to be photographed is illuminated with the line beam emitted from the light source 20. The light generated by the illumination is separated into wavelength bands via the prism or the diffraction grating and detected by the image sensor. In line scanning, such light detection is performed while the object to be photographed is moved in one direction by the slide mechanism.

Camera 30 generates and outputs a hyperspectral image for correction by line scanning subject 10a1 or subject 10a2. Similarly, camera 30 generates and outputs a hyperspectral image to be analyzed by line scanning subject 10b. Line-scan hyperspectral cameras have high spatial and wavelength resolution, but the imaging time is long due to the line scan.

Snapshot-type hyperspectral camera When the camera 30 is a snapshot-type hyperspectral camera, the camera 30 includes a plurality of light-transmitting regions corresponding to a plurality of wavelength bands, respectively, and an image sensor. Each of the plurality of light-transmitting regions transmits light of the corresponding wavelength band among the plurality of wavelength bands in the target wavelength range. Light from an object to be photographed is detected by the image sensor via the plurality of light-transmitting regions.

Camera 30 generates and outputs a hyperspectral image for correction by capturing an image of subject 10a1 or subject 10a2 in one shot. Similarly, camera 30 generates and outputs a hyperspectral image to be analyzed by capturing an image of subject 10b in one shot. This is similar to the principle of a color camera capturing an image of a target object in one shot through red, green, and blue color filters to generate and output red, green, and blue images. Although snapshot-type hyperspectral cameras are capable of capturing images in one shot, their sensitivity and spatial resolution are often insufficient.

Compressive sensing type hyperspectral camera When the camera 30 is a compressed sensing type hyperspectral camera as disclosed in Patent Document 2, the camera 30 includes an encoding mask including a plurality of regions having different transmission spectra, an image sensor, and an image processing device. Light from an object to be photographed is detected by the image sensor through the encoding mask, and a compressed image is generated in which information of a plurality of wavelength bands is compressed. The image processing device generates a hyperspectral image of the object based on the compressed image.

Camera 30 photographs subject 10a1 or subject 10a2 to generate a compressed image for correction, and generates and outputs a hyperspectral image for correction based on the compressed image. Similarly, camera 30 photographs subject 10b to generate a compressed image of the subject to be analyzed, and generates and outputs a hyperspectral image of the subject to be analyzed based on the compressed image. A compressed sensing type hyperspectral camera can generate a hyperspectral image for correction or analysis in one shot without reducing sensitivity and spatial resolution.

In a compressed sensing type hyperspectral camera, the camera 30 may be configured to include an encoding mask and an image sensor, but may not include an image processing device. In such a configuration, the camera 30 generates and outputs a compressed image, and the processing device 50 generates a hyperspectral image based on the compressed image.

Details about compressed sensing hyperspectral cameras will be provided later.

<Display device 40>
The display device 40 displays an input user interface (UI) 42 and a display UI 44. The input UI 42 is used for a user to input information. The information input by the user to the input UI 42 is received by the processing circuitry 52. The display UI 44 is used for displaying information generated by the processing circuitry 52.

The input UI 42 and the display UI 44 are displayed as a GUI (Graphical User Interface). It can be said that the information shown on the input UI 42 and the display UI 44 is displayed on the display device 40. The input UI 42 and the display UI 44 may be realized by a device capable of both input and output, such as a touch screen. In that case, the touch screen may function as the display device 40. When a keyboard and/or a mouse are used as the input UI 42, the input UI 42 is a device independent of the display device 40.

<Processing device 50>
The processing circuitry 52 included in the processing device 50 controls the operations of the light source 20, the camera 30, and the storage device 56. The processing circuitry 52 acquires the hyperspectral image for correction and the hyperspectral image to be analyzed generated by the camera 30, and performs processing based on these hyperspectral images. Alternatively, the processing circuitry 52 acquires the compressed image for correction and the compressed image to be analyzed generated by the camera 30, and performs processing based on these compressed images.

The memory 54 included in the processing device 50 stores a computer program executed by the processing circuit 52. The processing circuit 52 and the memory 54 may be integrated on a single circuit board or may be provided on separate circuit boards. The functions of the processing circuit 52 may also be distributed across multiple circuits. A part or the entirety of the processing circuit 52 may be installed in a remote location away from the light source 20, the camera 30, and the storage device 56, and may control the operation of these components via a wired or wireless communication network.

The storage device 56 included in the processing device 50 is a device that includes any storage medium, such as a semiconductor storage medium or a magnetic storage medium. The storage device 56 is connected to the processing circuit 52 and stores the processing results of the processing circuit 52.

[1.2. Specific configuration example of imaging system]
Next, a specific configuration example of the imaging system according to the embodiment of the present disclosure will be described with reference to FIG. 3. FIG. 3 is a diagram showing a schematic configuration of the imaging system according to the exemplary embodiment of the present disclosure. The imaging system 100 shown in FIG. 3 further includes a stage 60, a support 70, and an adjustment device 80 in addition to the above-mentioned light source 20, the camera 30, the display device 40, and the processing device 50. In the example shown in FIG. 3, the number of light sources 20 is two, but may be one, or may be three or more. The processing device 50 is connected to the light source 20, the camera 30, the display device 40, and the adjustment device 80 by wire or wirelessly. The processing circuit 52 shown in FIG. 2A to FIG. 2C included in the processing device 50 controls the operation of the adjustment device 80 in addition to the operation of the light source 20, the camera 30, and the display device 40.

The stage 60 has a flat support surface for placing the subject 10a1 shown in FIG. 2A and the subject 10b shown in FIG. 2C. The support 70 is fixed to the stage 60 and has a structure that extends in a direction perpendicular to the support surface of the stage 60, i.e., in the height direction. The support 70 supports the light source 20, the camera 30, and the adjustment device 80.

The adjustment device 80 includes a mechanism for independently moving the light source 20 and the camera 30 in a direction perpendicular to the support surface of the stage 60. The adjustment device 80 may include an actuator including one or more motors, such as a linear actuator. The actuator may be configured to change the distance between the light source 20 and the subject 10a1 or the distance between the light source 20 and the subject 10b, and the distance between the camera 30 and the subject 10a1 or the distance between the camera 30 and the subject 10b, using, for example, an electric motor, hydraulic pressure, or pneumatic pressure.

Because the adjustment device 80 can adjust the distance between the light source 20 and the subject 10a1 or the distance between the light source 20 and the subject 10b, it is possible to appropriately adjust the amount of light emitted from the light source 20, reflected by the subject 10a1 or the subject 10b, and incident on the camera 30. This reduces the possibility that the hyperspectral image or compressed image will be too bright or too dark. Furthermore, because the adjustment device 80 can adjust the distance between the camera 30 and the subject 10a1 or the distance between the camera 30 and the subject 10b, it becomes easier to focus the camera 30.

The adjustment device 80 further includes a measuring device that measures the distance between the stage 60 and the light source 20, and the distance between the stage 60 and the camera 30. The support 150 is provided with a scale that indicates the height from the support surface of the stage 190. The positions of the light source 20 and the camera 30 in the height direction can be known based on the scale.

[2. Issues with whiteboard correction and blackboard subtraction]
Problems in white board correction and black subtraction will be described below with reference to FIGS. 4A and 4B.

[2.1. White board correction]
In the whiteboard correction, the hyperspectral image to be analyzed is normalized by the hyperspectral image for correction. More specifically, each of the multiple images included in the hyperspectral image to be analyzed is normalized by a corresponding image among the multiple images included in the hyperspectral image for correction in that the wavelength bands are the same. In many cases, a suitable whiteboard has a certain or higher reflectance in the wavelength range included in the hyperspectral image. The whiteboard correction is performed to reduce the effects of the spectral shape of the irradiated light, the irradiance distribution during shooting, the peripheral light falloff of the lens, and the uneven sensitivity of the image sensor. However, if there is no need to reduce such effects, it is not necessary to perform the whiteboard correction.

FIG. 4A is a diagram showing a schematic example of white board correction. The top three diagrams in FIG. 4A show examples of correct white board correction, while the middle three diagrams and the bottom three diagrams in FIG. 4A show examples of incorrect white board correction. Of the three diagrams in each row, the diagram on the left shows a hyperspectral image containing subject 10b, the central diagram shows a hyperspectral image containing a white board, and the diagram on the right shows a hyperspectral image after white board correction. In FIG. 4A, an image corresponding to a certain wavelength band contained in the hyperspectral image is shown as an example of the hyperspectral image.

When an appropriate white board is used as the subject 10a1, an appropriate hyperspectral image for correction is generated, as shown in the center diagram in the top row. The hyperspectral image to be analyzed is corrected based on the appropriate hyperspectral image for correction, so that a corrected hyperspectral image after white board correction is correctly generated, as shown in the right diagram in the top row.

In contrast, if dirt adheres to the white board, the dirt will appear in the hyperspectral image for correction, as shown in the center figure in the middle row. If the hyperspectral image to be analyzed is corrected based on such an inappropriate hyperspectral image for correction, the hyperspectral image after white board correction will not be generated correctly, as shown in the right figure in the middle row. In the hyperspectral image for correction, the parts that correspond to the dirt will be darker than the other parts, so due to the normalization described above, the parts that correspond to the dirt will appear white in the hyperspectral image after white board correction.

Alternatively, if an object other than a white board is mistakenly used as the subject 10a1, the object will appear in the hyperspectral image for correction, as shown in the center diagram in the lower row. A fish is shown as an example of the object. If the hyperspectral image to be analyzed is corrected based on such an inappropriate hyperspectral image for correction, the hyperspectral image after white board correction will not be generated correctly, as shown in the right diagram in the lower row. In the hyperspectral image for correction, the part corresponding to the object will be darker than the other parts, and therefore, due to the above normalization, the part corresponding to the object will appear white in the hyperspectral image after white board correction.

Therefore, with incorrect white board correction, it is not easy to accurately obtain spectral information of the subject 10b based on the hyperspectral image after white board correction.

[2.2. Black Line]
In the blackout correction, a correction hyperspectral image is subtracted from the hyperspectral image to be analyzed. More specifically, from each of the images included in the hyperspectral image to be analyzed, a corresponding image in the same wavelength band is subtracted from the images included in the hyperspectral image to be corrected. Blackout correction is performed to reduce the effects of dark current, bright pixels, fixed pattern noise, and sensor performance fluctuations of the image sensor. However, if there is no need to reduce such effects, blackout correction is not necessarily required.

In the following description, the hyperspectral image to be analyzed may be read as the compressed image to be analyzed, and the hyperspectral image for correction may be read as the compressed image to be corrected.

FIG. 4B is a diagram showing a schematic example of black subtraction. The top three diagrams in FIG. 4B show examples of correct black subtraction, and the bottom three diagrams in FIG. 4B show examples of incorrect black subtraction. Of the three diagrams in each row, the diagram on the left shows a hyperspectral image containing subject 10b, the diagram in the center shows a hyperspectral image representing a light-shielded image, and the diagram on the right shows a hyperspectral image after black subtraction. In FIG. 4B, an image corresponding to a certain wavelength band contained in the hyperspectral image is shown as an example of the hyperspectral image.

When a light-blocking lens cap is properly attached to the camera 30 as the subject 10a2, i.e., when an image is taken in a light-blocking state, an appropriate hyperspectral image for correction is generated, as shown in the center diagram in the top row. The hyperspectral image for correction, which is a light-blocked image, contains noise caused by the image sensor. The hyperspectral image to be analyzed contains subject 10b with the noise superimposed thereon, as shown in the left diagram in the top row. Since the hyperspectral image to be analyzed is corrected based on the appropriate hyperspectral image for correction, a correctly generated hyperspectral image after blackout is generated, as shown in the right diagram in the top row. The hyperspectral image after blackout contains subject 10b with the noise removed.

In contrast, if the light-blocking lens cap is forgotten to be attached to the camera 30 and a white board is photographed, a hyperspectral image containing the white board, rather than a light-blocked image, is generated as the hyperspectral image for correction, as shown in the center diagram in the lower row. If the hyperspectral image to be analyzed is corrected based on such an inappropriate hyperspectral image for correction, a corrected hyperspectral image after black-out will not be generated correctly, as shown in the right diagram in the lower row.

Because the correction hyperspectral image, which is not a light-blocked image, is subtracted from the hyperspectral image to be analyzed, more pixel values than necessary are subtracted. This causes the incorrectly black-subtracted image to be darker than the correctly black-subtracted hyperspectral image. In the incorrectly black-subtracted image, pixel values may be negative. In such cases, pixel values may be output as zero. Therefore, the corrected hyperspectral image may be different from the image obtained by simply subtracting the correction hyperspectral image from the hyperspectral image to be analyzed.

Therefore, with incorrect black subtraction, it is not easy to accurately obtain spectral information of the subject 10b based on the corrected hyperspectral image.

As described above, if subject 10a1 or subject 10a2 is not suitable, it is not easy to accurately obtain spectral information of subject 10b. If subject 10a2 is not suitable, for example, a light-blocking lens cap may not be properly attached to camera 30, making it difficult to capture an image in a light-blocking state.

The inventors discovered this problem and came up with a method used to correct images in this embodiment that can solve the problem.

[3. Method 1 used for image correction]
3.1. Processing Operation
Next, an example of a method used for image correction according to the present embodiment when the camera 30 generates and outputs a hyperspectral image will be described with reference to Fig. 5. The image correction described below may be either whiteboard correction or black subtraction.

FIG. 5 is a flow chart showing an outline of example 1 of the processing operation performed by the processing circuit 52 in the imaging system according to this embodiment. The processing circuit 52 performs the operations of steps S101 to S108 shown in FIG. 5. The "HS image" shown in FIG. 5 represents a hyperspectral image.

<Step S101>
The processing circuitry 52 causes the display device 40 to display an instruction to the user to photograph a whiteboard or an instruction to the user to photograph in a light-shielded state. Alternatively, if the photographing system 100 includes a speaker, the processing circuitry 52 may cause the speaker to output the instruction by voice.

The user receives the instruction and places subject 10a1 in front of camera 30, or attaches subject 10a2 to camera 30. Processing circuit 52 receives input from the user via input UI 42 and causes camera 30 to capture subject 10a1 or subject 10a2 to generate a first hyperspectral image. In this way, the first hyperspectral image is generated when the user receives an instruction to capture subject 10a1 or subject 10a2.

Before photographing subject 10a1 or subject 10a2, processing circuitry 52 may receive input from a user via input UI 42 and adjust parameters related to camera 30. The parameters related to camera 30 may be, for example, exposure time, gain, number of integrations, and/or the distance between camera 30 and subject 10a1.

When photographing subject 10a1, processing circuit 52 receives input from the user via input UI 42 before step S101 and causes light source 20 to emit illumination light for illuminating subject 10a1. When photographing subject 10a2, processing circuit 52 does not need to cause light source 20 to emit illumination light before step S101.

Before causing the light source 20 to emit irradiation light, the processing circuit 52 may receive input from the user via the input UI 42 and adjust parameters related to the light source 20. The parameters related to the light source 20 may be, for example, the distance between the light source 20 and the subject 10a1, the current for driving the light source 20, the voltage, the duty ratio of a PWM (Pulse Width Modulation) signal, and/or the attenuation rate of an ND (Neutral Density) filter (not shown) disposed between the light source 20 and the subject 10a1.

<Step S102>
The processing circuitry 52 acquires a first hyperspectral image as a hyperspectral image for correction from the camera 30. The processing circuitry 52 may store the acquired first hyperspectral image in the storage device 56.

<Step S103>
The processing circuitry 52 causes the display device 40 to display an instruction to the user to photograph the subject to be analyzed. Alternatively, if the photographing system 100 includes a speaker, the processing circuitry 52 may cause the speaker to output the instruction as sound.

The user receives the instruction and places the subject 10b in front of the camera 30. The processing circuit 52 receives input from the user via the input UI 42 and causes the camera 30 to capture the subject 10b and generate a second hyperspectral image. In this way, the second hyperspectral image is generated when the user receives an instruction to capture the subject 10b.

When subject 10a1 is photographed in step S101, subject 10b is illuminated with the above-mentioned illumination light. When subject 10a2 is photographed in step S101, processing circuit 52 causes light source 20 to emit illumination light for illuminating subject 10b before step S103.

<Step S104>
The processing circuitry 52 acquires a second hyperspectral image from the camera 30 as the hyperspectral image to be analyzed. The processing circuitry 52 may store the acquired second hyperspectral image in the storage device 56.

In addition, the processing circuit 52 may execute the operations of steps S103 and S104 between steps S105 and S106.

<Step S105>
The processing circuit 52 determines whether the first hyperspectral image satisfies a suitability condition for use as an image for correction of the second hyperspectral image based on the pixel values of the first hyperspectral image. The suitability condition for use as an image for whiteboard correction and black subtraction will be described later.

If the determination is Yes, the processing circuit 52 executes the operation of step S106. If the determination is No, the processing circuit 52 executes the operation of step S108.

<Step S106>
The processing circuitry 52 performs whiteboard correction by normalizing the second hyperspectral image by the first hyperspectral image, or performs black subtraction by subtracting the first hyperspectral image from the second hyperspectral image.

In whiteboard correction, black subtraction may already have been performed on the second hyperspectral image.

<Step S107>
The processing circuitry 52 stores the corrected hyperspectral image in the memory device 56 .

<Step S108>
The processing circuit 52 causes the display device 40 to display an error indicating that there is an abnormality in the first hyperspectral image.

If part of the processing circuit 52 is an external server installed in a remote location, the external server may perform the operations of steps S102 and S105.

The above method used for image correction in this embodiment can prevent inappropriate whiteboard correction when a user receives instructions to photograph an appropriate subject 10a1 but the subject 10a1 is actually not suitable. Similarly, when a user receives instructions to photograph an appropriate subject 10a2 but the subject 10a2 is actually not suitable, it can prevent inappropriate blackout. As a result, it is possible to obtain spectral information of the subject 10b more accurately than when it is not determined whether the first hyperspectral image satisfies the suitability conditions.

[3.2. Suitability conditions for an image for whiteboard correction]
6A and 6B, examples of suitability conditions that the first hyperspectral image satisfies as an image for whiteboard correction will be described below. Here, whether or not the first hyperspectral image satisfies the suitability conditions as an image for whiteboard correction is determined based on the spectral information acquired from the first hyperspectral image.

6A is a diagram showing a first hyperspectral image and a reflectance spectrum when the subject 10a1 is appropriate. The first hyperspectral image is shown at the top of FIG. 6A. The first hyperspectral image includes five images corresponding to five wavelength bands. For the wavelength bands W ₁ to W ₅ , the smaller the subscript number, the shorter the central wavelength of the wavelength band. These images are smooth images without boundaries and structures. The whiter the image, the brighter it is, and the blacker the image, the darker it is.

The reflection spectrum of the subject 10a1 is shown at the bottom of FIG. 6A. The reflection spectrum was generated based on the first hyperspectral image. The reflection intensity in each wavelength band is calculated by averaging the pixel values of multiple pixels near the center of the image corresponding to the wavelength band. The appropriately selected subject 10a1 has the same reflectance in each of the wavelength bands W ₁ to W _5. The reflection spectrum of the subject 10a1 is almost the same as the spectrum of the light emitted from the light source 20. In other words, the spectrum obtained when the camera 30 detects the reflected light is almost the same as the spectrum obtained when the camera 30 directly detects the light emitted from the light source 20. The reflected light is light generated by the light emitted from the light source 20 being reflected by the subject 10a. In the example shown in FIG. 6A, the reflection intensity in the wavelength band W ₁ is the highest, the reflection intensity in the wavelength band W ₂ is the lowest, and the reflection intensity in the wavelength bands W ₃ to W ₅ is higher than the reflection intensity in the wavelength band W ₂ and lower than the reflection intensity in the wavelength band W ₁ . The reflection intensity in waveband _W4 is higher than the reflection intensity in waveband _W3 and waveband _W5 .

FIG. 6B is a diagram showing a schematic diagram of the first hyperspectral image and the reflectance spectrum when the subject 10a1 is not suitable. The diagrams shown in the upper and lower parts of FIG. 6B are as described above.

If the subject 10a1 is not suitable, the reflection spectrum of the subject 10a1 will differ from the spectrum of the light emitted from the light source 20. In the example shown in FIG. 6B, a blue plate was used as the subject 10a1. In the example shown in FIG. 6B, as the central wavelength of the wavelength band increases, the reflection intensity decreases and the image becomes darker.

Spectral information of the light from the light source 20 is pre-stored in the storage device 56. The processing circuit 52 acquires the spectral information of the light from the light source 20 from the storage device 56, and determines whether or not the first hyperspectral image satisfies the suitability conditions by comparing the spectral information acquired from the first hyperspectral image with the spectral information of the light from the light source 20. If the shapes of the two spectra are similar, the processing circuit 52 determines that the first hyperspectral image satisfies the suitability conditions.

The two pieces of spectral information can be compared using, for example, the Spectral Angular Mapping (SAM) method. When a spectrum contains N light intensities corresponding to N wavelength bands, the spectrum is expressed as an N-dimensional vector containing the N light intensities as components. If the vector representing the spectrum acquired from the first hyperspectral image is u and the vector representing the spectrum of the light from the light source 20 is v, the spectral angle formed by the vectors u and v can be calculated. The spectral angle is

where u·v represents the inner product of vector u and vector v, |u| represents the magnitude of vector u, and |v| represents the magnitude of vector v.

When the spectral angle is zero, vector u and vector v face in the same direction. In this case, the shapes of the two spectra are the same. Even if |u| is different from |v|, if vector u and vector v face in the same direction, it can be said that the shapes of the two spectra are the same. Therefore, in the SAM method, vector u and vector v can be compared regardless of the amount of light irradiated when generating the first hyperspectral image. If the spectral angle formed by vector u and vector v is, for example, 1° or less, 5° or less, or 10° or less, or any angle in the range of 1° to 10°, the first hyperspectral image may be determined to satisfy the suitability condition.

Next, referring to FIG. 7, another example of the suitability conditions for white board correction that the first hyperspectral image satisfies will be described. In reality, it is rare to mistakenly capture a board of a different color, such as blue, instead of a white board, as described above, and it is common to mistakenly capture a scene in which one or more objects are present. Therefore, in many cases, if the spatial distribution of pixel values in the first hyperspectral image is constant, it is safe to consider that subject 10a1 is suitable.

Here, the first hyperspectral image is subjected to edge detection, and based on the edge image obtained thereby, it is determined whether the first hyperspectral image satisfies the suitability conditions. For example, the Sobel method can be used for edge detection.

7 shows the first hyperspectral image when the subject 10a1 is appropriate and when it is not appropriate, and an edge image obtained by edge detection of the first hyperspectral image. The upper left side of FIG. 7 shows the first hyperspectral image when the subject 10a1 is appropriate, and the upper right side of FIG. 7 shows an edge image obtained by edge detection of the first hyperspectral image. The lower left side of FIG. 7 shows the first hyperspectral image when a white board is not appropriately used as the subject 10a1, and the lower right side of FIG. 7 shows an edge image obtained by edge detection of the first hyperspectral image. In FIG. 7, an image corresponding to a certain wavelength band included in the first hyperspectral image is illustrated as the first hyperspectral image. In the following description referring to FIG. 7, the first hyperspectral image means an image corresponding to a certain wavelength band included in the first hyperspectral image.

As shown in FIG. 7, when the subject 10a1 is appropriate, the first hyperspectral image is a smooth image without boundaries or structures. Even if edge detection is performed on the first hyperspectral image, zero pixels are detected as edges.

In contrast, if the subject 10a1 is not suitable, the first hyperspectral image is a non-smooth image having boundaries and structures. When edge detection is performed on the first hyperspectral image, there are pixels that are detected as edges. In the example shown in FIG. 7, the first hyperspectral image contains a number of vegetables. Of the 65,535 pixels contained in the first hyperspectral image, 2,109 pixels are detected as edges.

In view of the above, if the condition for determining that the spatial distribution of pixel values of the first hyperspectral image is constant is satisfied, the first hyperspectral image may be determined to satisfy the suitability condition. The condition may be determined, for example, based on the number of pixels detected as edges among multiple pixels in an image corresponding to a certain wavelength band included in the first hyperspectral image. Specifically, the condition may be, for example, that the number of pixels detected as edges is equal to or less than a predetermined ratio. The number of images for edge detection among the multiple images included in the first hyperspectral image may be one or more.

[3.3. Suitability conditions for images for black out]
An example of the suitability conditions for the first hyperspectral image to be used as an image for black subtraction that the first hyperspectral image satisfies will be described below with reference to Figures 8A and 8B. Here, it is determined whether the first hyperspectral image satisfies the suitability conditions for the image for black subtraction based on the pixel values of the first hyperspectral image.

FIG. 8A is a diagram showing a first hyperspectral image and a histogram of pixel values of the first hyperspectral image when the subject 10a2 is appropriate. The upper part of FIG. 8A shows a hyperspectral image, and the lower part of FIG. 8A shows a histogram of pixel values of the first hyperspectral image. FIG. 8A shows an example of an image corresponding to a certain wavelength band contained in the first hyperspectral image as the first hyperspectral image. In the following description referring to FIG. 8A, the first hyperspectral image means an image corresponding to a certain wavelength band contained in the first hyperspectral image.

As shown in FIG. 8A, the first hyperspectral image is a dark image. All pixels in the first hyperspectral image have pixel values near zero. In the example shown in FIG. 8A, the first hyperspectral image includes 256×256 pixels. In the histogram of pixel values of the first hyperspectral image, pixel values are normalized by the maximum pixel value that the pixel can have. In the histogram of pixel values of the first hyperspectral image, the pixels with the lowest pixel value are the most numerous, accounting for more than 97% of all pixels. The minimum normalized pixel value is 2.44×10 ⁻⁴ . Pixels with normalized pixel values of 1.0×10 ⁻³ or more account for 0.07% of all pixels.

FIG. 8B is a diagram showing the first hyperspectral image and a histogram of pixel values of the first hyperspectral image when the subject 10a2 is not suitable. The figures shown at the top and bottom of FIG. 8B are as described above. In the following description referring to FIG. 8B, the first hyperspectral image means an image corresponding to a certain wavelength band contained in the first hyperspectral image.

As shown in Fig. 8B, the first hyperspectral image is not a dark image but a landscape image. In the histogram of pixel values of the first hyperspectral image, the pixel with the lowest pixel value accounts for 0.006% of all pixels. The lowest normalized pixel value is 0.0273. The pixels with normalized pixel values of 1.0 x ^10-3 or more account for 100% of all pixels.

In view of the above, if the condition for determining that the pixel value of the first hyperspectral image is small is satisfied, the first hyperspectral image may be determined to satisfy the suitability condition. The condition may be determined, for example, based on the number of pixels having the lowest pixel value, or the number of pixels whose pixel values are greater than or equal to a threshold value or less than or equal to a threshold value, in an image corresponding to a certain wavelength band included in the first hyperspectral image. Specifically, the condition may be that the number of pixels having the lowest pixel value is greater than or equal to a predetermined percentage, that the number of pixels whose pixel values are greater than or equal to a threshold value is less than or equal to a predetermined percentage, or that the number of pixels whose pixel values are less than or equal to a threshold value is greater than or equal to a predetermined percentage. Of the multiple images included in the first hyperspectral image, the number of images whose pixel value histograms are checked may be one or more.

Alternatively, as described with reference to FIG. 7, if the conditions for determining that the spatial distribution of pixel values of the first hyperspectral image is constant are met, even with black subtraction, as with whiteboard correction, the first hyperspectral image may be determined to satisfy the suitability conditions.

[4. Method 2 used for image correction]
4.1. Processing Operation
An example of a method used for image correction according to the present embodiment when the camera 30 generates and outputs a compressed image in a compressed sensing type hyperspectral camera will be described below with reference to Fig. 9. The image correction described below is white board correction.

FIG. 9 is a flow chart that shows an outline of a second example of the processing operation performed by the processing circuit 52 in the imaging system according to this embodiment. The processing circuit 52 performs the operations of steps S201 to S210 shown in FIG. 9.

<Step S201>
The processing circuitry 52 causes the display device 40 to display an instruction to the user to photograph the whiteboard. Alternatively, if the imaging system 100 includes a speaker, the processing circuitry 52 may cause the speaker to output the instruction as voice.

The user receives the instruction and places the subject 10a1 in front of the camera 30. The processing circuit 52 receives input from the user via the input UI 42 and causes the camera 30 to capture the subject 10a1 and generate a first compressed image. In this way, the first compressed image is generated when the user receives an instruction to capture the subject 10a1.

Before step S201, the processing circuit 52 receives input from the user via the input UI 42 and causes the light source 20 to emit irradiation light for irradiating the subject 10a1.

<Step S202>
The processing circuitry 52 acquires a first compressed image from the camera 30. The processing circuitry 52 may store the acquired first compressed image in the storage device 56.

<Step S203>
The processing circuitry 52 generates a first hyperspectral image based on the first compressed image. The processing circuitry 52 may store the generated first hyperspectral image in the storage device 56.

<Step S204>
The processing circuitry 52 causes the display device 40 to display an instruction to the user to photograph the subject to be analyzed. Alternatively, if the photographing system 100 includes a speaker, the processing circuitry 52 may cause the speaker to output the instruction as sound.

The user receives the instruction and places the subject 10b in front of the camera 30. The processing circuit 52 receives input from the user via the input UI 42 and causes the camera 30 to capture the subject 10b and generate a second compressed image. In this way, the second compressed image is generated when the user receives an instruction to capture the subject 10b. The subject 10b is illuminated with the above-mentioned illumination light.

<Step S205>
The processing circuitry 52 acquires a second compressed image from the camera 30. The processing circuitry 52 may store the acquired second compressed image in the storage device 56.

<Step S206>
The processing circuitry 52 generates a second hyperspectral image based on the second compressed image. The processing circuitry 52 may store the generated second hyperspectral image in the storage device 56.

In addition, the processing circuit 52 may execute the operations of steps S203 to S206 between steps S207 and S208.

<Step S207>
The processing circuit 52 determines whether the first compressed image satisfies a suitability condition for an image for whiteboard correction based on the pixel values of the first compressed image. The suitability condition for an image for whiteboard correction will be described later.

If the determination is Yes, the processing circuit 52 executes the operation of step S208. If the determination is No, the processing circuit 52 executes the operation of step S210.

<Step S208>
The processing circuit 52 performs whiteboard correction by normalizing the second hyperspectral image by the first hyperspectral image. The second hyperspectral image may have already been subjected to black subtraction.

<Step S209>
The processing circuitry 52 stores the corrected hyperspectral image in the memory device 56 .

<Step S210>
The processing circuit 52 causes the display device 40 to display an error indicating that there is an abnormality in the first compressed image.

If part of the processing circuit 52 is an external server installed in a remote location, the external server may execute the operations of steps S202 and S207.

The above method used for image correction in this embodiment can prevent inappropriate whiteboard correction when the user believes that he or she has photographed an appropriate subject 10a1 in response to instructions, but the subject 10a1 is actually not appropriate. As a result, it is possible to obtain more accurate spectral information of the subject 10b than when it is not determined whether the first compressed image satisfies the suitability conditions for an image for whiteboard correction.

In addition, the processing circuit 52 may execute the operations of S105 to S108 shown in FIG. 5 instead of the operations of S207 to S210 shown in FIG. 9.

[4.2. Suitability conditions for an image for whiteboard correction]
10A and 10B, examples of suitability conditions that the first compressed image satisfies as an image for whiteboard correction will be described below. Here, whether or not the first compressed image satisfies the suitability conditions as an image for whiteboard correction is determined based on a histogram of pixel values of the first compressed image.

FIG. 10A is a diagram showing a first compressed image and a histogram of its pixel values when the subject 10a1 is appropriate. The upper part of FIG. 10A shows the first compressed image, and the lower part of FIG. 10A shows a histogram of the pixel values of the first compressed image.

As shown in FIG. 10A, the first compressed image has an irregular distribution of light and dark pixel values that reflects the spatial distribution of the transmittance of the encoding mask. A histogram of pixel values of the first compressed image roughly shows a single peak. When the subject 10a1 is a white board, the peak width is narrower than when it is not. In the example shown in FIG. 10A, if the average pixel value is μ and the standard deviation of the pixel values is σ, then σ/μ = 0.2615.

FIG. 10B is a diagram showing the first compressed image and its pixel value histogram when the subject 10a1 is not appropriate. The upper and lower diagrams in FIG. 10B are as described above.

As shown in FIG. 10B, the first compressed image has an irregular distribution of light and dark pixel values. To the naked eye, the first compressed image shown in FIG. 10B resembles the first compressed image shown in FIG. 10A. A histogram of pixel values of the first compressed image roughly shows a single peak. If the subject 10a1 is not a white board, the peak width will be wider. In the example shown in FIG. 10B, σ/μ=0.3016.

In view of the above, it may be possible to determine whether or not the first compressed image satisfies the suitability conditions based on a histogram of the pixel values of the first compressed image. For example, if the σ/μ of the histogram of the first compressed image is equal to or less than a predetermined value, it is determined that the first compressed image satisfies the suitability conditions.

[5. Method 3 used for image correction]
Another example of the method used for image correction according to the present embodiment when the camera 30 generates and outputs a compressed image in a compressed sensing type hyperspectral camera will be described below with reference to Fig. 11. The image correction described below is black subtraction.

FIG. 11 is a flow chart that shows an outline of example 3 of the processing operation performed by the processing circuit 52 in the imaging system according to this embodiment. The processing circuit 52 performs the operations of steps S301 to S308 shown in FIG. 11.

<Step S301>
The processing circuitry 52 causes the display device 40 to display an instruction to the user to take a picture in a light-shielded state. Alternatively, if the photographing system 100 includes a speaker, the processing circuitry 52 may cause the speaker to output the instruction as sound.

The user receives the instruction and attaches the subject 10a2 to the camera 30. The processing circuit 52 receives input from the user via the input UI 42 and causes the camera 30 to capture the subject 10a2 and generate a first compressed image. In this way, the first compressed image is generated when the user receives an instruction to capture the subject 10a2.

<Step S302>
The processing circuitry 52 acquires a first compressed image from the camera 30. The processing circuitry 52 may store the acquired first compressed image in the storage device 56.

<Step S303>
The processing circuitry 52 causes the display device 40 to display an instruction to the user to photograph the subject to be analyzed. Alternatively, if the photographing system 100 includes a speaker, the processing circuitry 52 may cause the speaker to output the instruction as sound.

The user receives the instruction and places the subject 10b in front of the camera 30. The processing circuit 52 receives input from the user via the input UI 42 and causes the camera 30 to capture the subject 10b and generate a second compressed image. In this way, the second compressed image is generated when the user receives an instruction to capture the subject 10b.

Before step S303, the processing circuit 52 receives input from the user via the input UI 42 and causes the light source 20 to emit irradiation light for irradiating the subject 10b.

<Step S304>
The processing circuitry 52 acquires a second compressed image from the camera 30. The processing circuitry 52 may store the acquired second compressed image in the storage device 56.

In addition, the processing circuit 52 may execute the operations of steps S303 and S304 between steps S305 and S306.

<Step S305>
The processing circuit 52 determines whether the first compressed image satisfies the suitability conditions for an image for black subtraction based on the pixel values of the first compressed image. The suitability conditions for an image for black subtraction correction that the first compressed image satisfies may be the same as the suitability conditions described with reference to Figures 8A and 8B. More specifically, the processing circuit 52 may determine whether the first compressed image satisfies the suitability conditions based on the number of pixels in the first compressed image that have the lowest pixel value or the number of pixels whose pixel values are equal to or greater than a predetermined value or less than a predetermined value.

Alternatively, as described with reference to FIG. 7, the first compressed image may be determined to satisfy the suitability conditions if the conditions for determining that the spatial distribution of pixel values of the first compressed image is constant are satisfied, even in the case of black subtraction, as in the case of whiteboard correction.

If the determination is Yes, the processing circuit 52 executes the operation of step S306. If the determination is No, the processing circuit 52 executes the operation of step S308.

<Step S306>
The processing circuit 52 performs black subtraction by subtracting the first compressed image from the second compressed image.

<Step S307>
The processing circuitry 52 causes the storage device 56 to store the corrected compressed image.

<Step S308>
The processing circuit 52 causes the display device 40 to display an error indicating that there is an abnormality in the first compressed image.

If part of the processing circuit 52 is an external server installed in a remote location, the external server may execute the operations of steps S302 and S305.

The above method used for image correction in this embodiment can prevent inappropriate black subtraction when the user believes that he or she has photographed an appropriate subject 10a2 in response to instructions, but the subject 10a2 is actually not appropriate. As a result, it is possible to obtain more accurate spectral information of the subject 10b than when it is not determined whether the first compressed image satisfies the suitability conditions for an image for black subtraction.

[6. Example of displaying an error on a display device]
Below, examples of errors displayed on the display device 40 will be described with reference to Figures 12A to 12C. Figures 12A to 12C are diagrams that diagrammatically show examples of UIs of the display device 40 when the camera 30 generates and outputs a compressed image in a compressed sensing type hyperspectral camera. The UI serves as both an input UI 42 and a display UI 44.

The upper left side of FIG. 12A shows a space for a compressed image of subject 10b. The lower side of FIG. 12A shows a space for a corrected hyperspectral image of subject 10b. The upper right side of FIG. 12A shows the shooting conditions, such as resolution, exposure time, gain, and number of integrations. The center right of FIG. 12A shows buttons for "whiteboard correction" and "black subtraction." When the user presses the "whiteboard correction" button, the processing circuit 52 executes the operation of the flowchart shown in FIG. 9. When the user presses the "black subtraction" button, the processing circuit 52 executes the operation of the flowchart shown in FIG. 11.

When the user presses the "Whiteboard Correction" button, if the first compressed image does not satisfy the suitability conditions, an error pop-up is displayed on the UI of the display device 40, as shown in FIG. 12B. The error pop-up states, "An abnormality has been detected in the compressed image of the whiteboard. Please check whether it is an appropriate whiteboard. Do you want to continue processing?" In this way, the error may prompt the user to check whether the whiteboard has been correctly photographed as the subject 10a1 for correction. Note that the compressed image of subject 10b generated during the processing is shown on the upper left side.

When the user presses the "black out" button, if the first compressed image does not satisfy the suitability conditions, an error pop-up is displayed on the UI of the display device 40, as shown in FIG. 12C. The error pop-up states, "An abnormality has been detected in the shading image. Please check the shading state. Do you want to continue processing?" In this way, the error may prompt the user to check the shading state when photographing the correction subject 10a2.

As shown in FIG. 12B and FIG. 12C, an error may be displayed and the process may be interrupted, or the process may be continued after confirmation from the user. Alternatively, the user may be requested to retake a compressed image of subject 10a1 or subject 10a2.

[7. Summary of methods 1 to 3 used for image correction]
Below, with reference to FIG. 13, a summary of methods 1 to 3 used for image correction according to this embodiment will be described, focusing on processing operations common to these methods 1 to 3.

FIG. 13 is a flowchart outlining a summary of examples 1 to 3 of the processing operations performed by the processing circuit 52 in the imaging system according to this embodiment. The processing circuit 52 performs the operations of steps S401 to S406 shown in FIG. 13.

<Step S401>
The processing circuit 52 acquires a first image from the camera 30 as an image for correction.

In method 1 used to correct an image, the first image is a first hyperspectral image. In methods 2 and 3 used to correct an image, the first image is a first compressed image.

<Step S402>
The processing circuitry 52 obtains a second image from the camera 30 .

In methods 1 and 2 used to correct an image, the second image is a second hyperspectral image. In method 2 used to correct an image, the processing circuit 52 generates and obtains a second hyperspectral image based on the second compressed image. In method 3 used to correct an image, the second image is a second compressed image.

The processing circuit 52 may also execute the operation of step S402 between steps S403 and S404.

<Step S403>
The processing circuit 52 determines whether the first image satisfies the suitability conditions for a correction image for correcting the second image based on the pixel values of the first image. If the determination is Yes, the processing circuit 52 executes the operation of step S404. If the determination is No, the processing circuit 52 executes the operation of step S406.

<Step S404>
The processing circuitry 52 corrects the second image based on the first image.

In method 1 used for image correction, the processing circuit 52 performs whiteboard correction by normalizing the second hyperspectral image by the first hyperspectral image. Alternatively, the processing circuit 52 performs black subtraction by subtracting the first hyperspectral image from the second hyperspectral image.

In method 2 used to correct an image, the processing circuit 52 performs whiteboard correction by generating a first hyperspectral image based on the first compressed image and normalizing the second hyperspectral image by the first hyperspectral image.

In method 3 used for image correction, the processing circuit 52 performs black removal by subtracting the first compressed image from the second compressed image.

<Step S405>
The processing circuitry 52 causes the storage device 56 to store the corrected image.

<Step S406>
The processing circuit 52 causes the display device 40 to display an error indicating that there is an abnormality in the first image.

If part of the processing circuit 52 is an external server installed in a remote location, the external server may execute the operations of steps S401 and S403.

The above method used for image correction in this embodiment can prevent inappropriate correction when the user believes that he or she has photographed the appropriate subject 10a1 or subject 10a2 in response to instructions, but the subject 10a1 or subject 10a2 is actually not appropriate. As a result, it is possible to obtain more accurate spectral information for the subject 10b than when it is not determined whether the first image satisfies the suitability conditions.

[8. Method 4 used for image correction]
Method 4 used for image correction according to the present embodiment will be described below with reference to Figures 14 and 15. Here, an image generated by capturing a scene including the subject 10a1 and the subject 10b by the camera 30 is used. The image correction described below is whiteboard correction.

FIG. 14 is a schematic diagram showing an example of an image generated by capturing a scene including subjects 10a1 and 10b with camera 30. Image 38 shown in FIG. 14 is a hyperspectral image or a compressed image. Subjects 10a1 and 10b are captured in image 38.

FIG. 15 is a flow chart that shows an outline of example 4 of the processing operation performed by the processing circuit 52 in the imaging system according to this embodiment. The processing circuit 52 performs the operations of steps S501 to S507 shown in FIG. 15.

<Step S501>
The processing circuitry 52 causes the display device 40 to display instructions to the user to photograph the subject for correction and the subject to be analyzed. Alternatively, if the photographing system 100 includes a speaker, the processing circuitry 52 may cause the speaker to output the instructions as sound.

The user receives instructions and places subjects 10a1 and 10b in front of camera 30. Processing circuit 52 receives input from the user via input UI 42 and causes a scene including subjects 10a1 and 10b to be photographed, generating image 38 in which subjects 10a1 and 10b appear. In this way, image 38 is generated when the user receives instructions to photograph subjects 10a1 and 10b.

<Step S502>
The processing circuitry 52 acquires the image 38 from the camera 30 .

<Step S503>
Processing circuitry 52 generates, based on image 38, a first sub-image corresponding to object 10a1 and a second sub-image corresponding to object 10a2.

If image 38 is a hyperspectral image, processing circuitry 52 extracts first and second sub-images from image 38, for example by edge detection. Thus, the first sub-image is one part of the hyperspectral image, and the second sub-image is another part of the hyperspectral image. The first sub-image may be an area defined by the contour of subject 10a1 appearing in image 38, or an area including subject 10a1 appearing in image 38 and having any shape, such as a rectangle or a circle. The same applies to the second sub-image. Note that processing circuitry 52 may execute an operation of extracting the second sub-image from the hyperspectral image between steps S504 and S505.

If image 38 is a compressed image, processing circuitry 52 generates a hyperspectral image based on the compressed image and extracts the first and second sub-images from the hyperspectral image. Thus, the first sub-image is one part of the hyperspectral image and the second sub-image is another part of the hyperspectral image. Note that processing circuitry 52 may perform the operation of extracting the second sub-image from the hyperspectral image between steps S504 and S505.

Alternatively, if image 38 is a compressed image, processing circuitry 52 extracts a first sub-image from the compressed image. Processing circuitry 52 further generates a hyperspectral image based on the compressed image and extracts a second sub-image from the hyperspectral image. Thus, the first sub-image is part of the compressed image and the second sub-image is part of the hyperspectral image. Note that processing circuitry 52 may perform the operations of generating a hyperspectral image based on the compressed image and extracting the second sub-image from the hyperspectral image between steps S504 and S505.

<Step S504>
The processing circuit 52 determines, based on the pixel values of the first sub-image, whether the first sub-image satisfies a suitability condition for use as a correction image for correcting the second sub-image.

When image 38 is a hyperspectral image, the suitability conditions for an image for whiteboard correction are as described with reference to Figures 6A to 7. The same applies when image 38 is a compressed image and the first sub-image is part of a hyperspectral image.

When image 38 is a compressed image and the first sub-image is part of the compressed image, the suitability conditions for the image to be used for whiteboard correction are as described with reference to Figures 10A and 10B.

If the determination is Yes, the processing circuit 52 executes the operation of step S505. If the determination is No, the processing circuit 52 executes the operation of step S507.

<Step S505>
The processing circuitry 52 corrects the second sub-image based on the first sub-image.

If image 38 is a hyperspectral image, processing circuitry 52 performs whiteboard correction by normalizing the second sub-image by the first sub-image. The same is true if image 38 is a compressed image, the first sub-image is part of the hyperspectral image, and the second sub-image is another part of the hyperspectral image.

If image 38 is a compressed image, the first sub-image is part of the compressed image, and the second sub-image is part of a hyperspectral image, processing circuit 52 performs whiteboard correction by extracting a sub-image corresponding to subject 10a1 from the hyperspectral image from which the second sub-image has been extracted, and normalizing the second sub-image by the extracted sub-image.

<Step S506>
The processing circuitry 52 causes the storage device 56 to store the corrected sub-image.

<Step S507>
The processing circuit 52 causes the display device 40 to display an error indicating that there is an abnormality in the first sub-image.

If part of the processing circuit 52 is an external server installed in a remote location, the external server may execute the operations of steps S502 to S504.

The above method used for image correction in this embodiment can prevent inappropriate whiteboard correction when the user believes that he or she has photographed an appropriate subject 10a1 in response to instructions, but the subject 10a1 is actually not appropriate. As a result, it is possible to obtain more accurate spectral information about the subject 10b compared to a case where it is not determined whether the first sub-image satisfies the suitability conditions. Furthermore, the above method makes it possible to perform whiteboard correction even in cases where it is not easy to capture a whiteboard over the entire image, such as when photographing outdoors.

[9. Compressive sensing type hyperspectral camera]
Hereinafter, a configuration example of a compressed sensing type hyperspectral camera will be described with reference to Figs. 16A to 18B. Fig. 16A is a diagram showing a schematic configuration example of a camera 30 which is a compressed sensing type hyperspectral camera. The camera 30 shown in Fig. 16A includes an optical system 31, a filter array 32, an image sensor 33, and an image processing device 34, similar to the configuration disclosed in Patent Document 2. The optical system 31 and the filter array 32 are disposed on the optical path of light incident from the subject 10b. In the example shown in Fig. 16A, the filter array 32 is disposed between the optical system 31 and the image sensor 33.

The image sensor 33 generates data of a compressed image 35 in which information of a plurality of wavelength bands is compressed as a two-dimensional monochrome image. The image processing device 34 generates data representing a plurality of images corresponding one-to-one to the plurality of wavelength bands included in the target wavelength range, based on the data of the compressed image 35 generated by the image sensor 33. Here, the number of wavelength bands included in the target wavelength range is set to N (N is an integer equal to or greater than 4). In the following description, the N images generated based on the compressed image 35 are referred to as restored images _36W1 , _36W2 , ..., _36WN , and these may be collectively referred to as "hyperspectral image 36."

The filter array 32 is an array of multiple light-transmitting filters arranged in rows and columns. The multiple filters include multiple types of filters that differ from one another in terms of spectral transmittance, i.e., the wavelength dependency of light transmittance. The filter array 32 modulates the intensity of the incident light for each wavelength and outputs it. This process performed by the filter array 32 is called "encoding," and the filter array 32 is also called an "encoding mask."

In the example shown in FIG. 16A, the filter array 32 is placed near or directly above the image sensor 33. Here, "near" means close enough that a relatively clear image of the light from the optical system 31 is formed on the surface of the filter array 32. "Directly above" means that the two are so close that there is almost no gap between them. The filter array 32 and the image sensor 33 may be integrated.

The optical system 31 includes at least one lens. In FIG. 16A, the optical system 31 is shown as one lens, but the optical system 31 may be a combination of multiple lenses. The optical system 31 forms an image on the imaging surface of the image sensor 33 via the filter array 32.

The filter array 32 may be disposed away from the image sensor 33. FIGS. 16B to 16D are diagrams showing configuration examples of the camera 30 in which the filter array 32 is disposed away from the image sensor 33. In the example shown in FIG. 16B, the filter array 32 is disposed between the optical system 31 and the image sensor 33 and at a position distant from the image sensor 33. In the example shown in FIG. 16C, the filter array 32 is disposed between the subject 10b and the optical system 31. In the example shown in FIG. 16D, the camera 30 has two optical systems 31A and 31B, and the filter array 32 is disposed between them. As in these examples, an optical system including one or more lenses may be disposed between the filter array 32 and the image sensor 33.

The image sensor 33 is a monochrome type light detection device having a plurality of light detection elements (also referred to as "pixels" in this specification) arranged two-dimensionally. The image sensor 33 may be, for example, a charge-coupled device (CCD), a complementary metal oxide semiconductor (CMOS) sensor, or an infrared array sensor. The light detection elements include, for example, photodiodes. The image sensor 33 does not necessarily have to be a monochrome type sensor. For example, a color type sensor may be used. The color type sensor may include, for example, a plurality of red (R) filters that transmit red light, a plurality of green (G) filters that transmit green light, and a plurality of blue (B) filters that transmit blue light. The color type sensor may further include a plurality of IR filters that transmit infrared light. The color type sensor may also include a plurality of transparent filters that transmit all red, green, and blue light. By using a color type sensor, the amount of information regarding wavelengths can be increased, and the accuracy of the reconstruction of the hyperspectral image 36 can be improved. The wavelength range to be acquired may be determined arbitrarily, and is not limited to the visible wavelength range, but may be ultraviolet, near infrared, mid-infrared, or far infrared.

The image processing device 34 may be a computer including one or more processors and one or more storage media such as a memory. The image processing device 34 generates data of restored images 36W ₁ , 36W ₂ , ..., 36W _N based on the compressed image 35 acquired by the image sensor 33.

FIG. 17A is a diagram showing a schematic example of a filter array 32. The filter array 32 has a number of regions arranged two-dimensionally. In this specification, these regions are sometimes referred to as "cells." In each region, an optical filter having an individually set spectral transmittance is arranged. The spectral transmittance is expressed as a function T(λ), where λ is the wavelength of the incident light. The spectral transmittance T(λ) can take a value between 0 and 1.

In the example shown in FIG. 17A, the filter array 32 has 48 rectangular regions arranged in 6 rows and 8 columns. This is merely an example, and in actual applications, more regions may be provided. The number may be approximately the same as the number of pixels in the image sensor 33, for example. The number of filters included in the filter array 32 is determined according to the application, ranging from several tens to tens of millions, for example.

Fig. 17B is a diagram showing an example of the spatial distribution of the light transmittance of each of the wavelength bands _W1 , _W2 , ..., _WN included in the target wavelength range. In the example shown in Fig. 17B, the difference in the shading of each region represents the difference in the transmittance. The lighter the region, the higher the transmittance, and the darker the region, the lower the transmittance. As shown in Fig. 17B, the spatial distribution of the light transmittance differs depending on the wavelength band.

17C and 17D are diagrams showing examples of the spectral transmittance of region A1 and region A2 included in the filter array 32 shown in FIG. 17A, respectively. The spectral transmittance of region A1 and the spectral transmittance of region A2 are different from each other. In this way, the spectral transmittance of the filter array 32 differs depending on the region. However, it is not necessary that the spectral transmittance of all regions is different. In the filter array 32, the spectral transmittance of at least some of the multiple regions is different from each other. The filter array 32 includes two or more filters having different spectral transmittances from each other. In one example, the number of spectral transmittance patterns of the multiple regions included in the filter array 32 may be the same as or greater than the number N of wavelength bands included in the target wavelength range. The filter array 32 may be designed so that the spectral transmittance of more than half of the regions is different.

FIG. 18A is a diagram for explaining the characteristics of the spectral transmittance in a certain region of the filter array 32. In the example shown in FIG. 18A, the spectral transmittance has multiple maximum values P1 to P5 and multiple minimum values for wavelengths in the target wavelength range W. In the example shown in FIG. 18A, the maximum value of the light transmittance in the target wavelength range W is normalized to 1 and the minimum value is 0. In the example shown in FIG. 18A, the spectral transmittance has maximum values in wavelength ranges such as wavelength band W ₂ and wavelength band W _N-1 . In this way, the spectral transmittance of each region can be designed to have maximum values in at least two wavelength ranges among the wavelength bands W ₁ , W ₂ , ..., W _N. In the example of FIG. 18A, the maximum values P1, P3, P4, and P5 are 0.5 or more.

In this way, the light transmittance of each region varies depending on the wavelength. Therefore, the filter array 32 transmits a large amount of components in a certain wavelength range among the incident light, and does not transmit components in other wavelength ranges very much. For example, the transmittance of light in k wavelength bands out of the N wavelength bands may be greater than 0.5, and the transmittance of light in the remaining N-k wavelength bands may be less than 0.5, where k is an integer satisfying 2≦k<N. If the incident light is white light that contains all visible light wavelength components evenly, the filter array 32 modulates the incident light into light having multiple discrete intensity peaks with respect to wavelength for each region, and outputs this multi-wavelength light by superimposing it.

FIG. 18B is a diagram showing, as an example, the result of averaging the spectral transmittance shown in FIG. 18A for each wavelength band W ₁ , W ₂ , ..., W _N. The averaged transmittance is obtained by integrating the spectral transmittance T(λ) for each wavelength band and dividing by the bandwidth of the wavelength band. In this specification, the transmittance value averaged for each wavelength band in this way is defined as the transmittance in that wavelength band. In this example, the transmittance is remarkably high in three wavelength ranges having maximum values P1, P3, and P5. In particular, the transmittance exceeds 0.8 in two wavelength ranges having maximum values P3 and P5.

In the examples shown in Figures 17A to 17D, a grayscale transmittance distribution is assumed in which the transmittance of each region can take any value between 0 and 1. However, it is not necessary to use a grayscale transmittance distribution. For example, a binary scale transmittance distribution may be used in which the transmittance of each region can take a value of either approximately 0 or approximately 1. In a binary scale transmittance distribution, each region transmits most of the light in at least two of the multiple wavelength ranges included in the target wavelength range, and does not transmit most of the light in the remaining wavelength ranges. Here, "most" refers to approximately 80% or more.

A part of all the cells, for example, half of the cells, may be replaced with a transparent region. Such a transparent region transmits the light of each of the wavelength bands W ₁ , W ₂ , ..., W _N included in the target wavelength range W with a similarly high transmittance, for example, a transmittance of 80% or more. In such a configuration, the multiple transparent regions may be arranged, for example, in a checkerboard pattern. That is, in two arrangement directions of the multiple regions in the filter array 32, regions whose light transmittance varies depending on the wavelength and transparent regions may be arranged alternately.

The data showing the spatial distribution of the spectral transmittance of the filter array 32 is acquired in advance based on design data or actual measurement calibration, and is stored in a storage medium provided in the image processing device 34. This data is used in the calculation processing described below.

The filter array 32 may be configured using, for example, a multilayer film, an organic material, a diffraction grating structure, a microstructure including a metal, or a metasurface. When a multilayer film is used, for example, a dielectric multilayer film or a multilayer film including a metal layer may be used. In this case, at least one of the thickness, material, and stacking order of each multilayer film is formed so that it differs for each cell. This allows different spectral characteristics to be realized for each cell. By using a multilayer film, a sharp rise and fall in the spectral transmittance can be realized. A configuration using an organic material can be realized by making the pigment or dye contained different for each cell, or by stacking different materials. A configuration using a diffraction grating structure can be realized by providing a diffraction structure with a different diffraction pitch or depth for each cell. A microstructure including a metal can be created by utilizing the spectrum due to the plasmon effect. A metasurface can be created by microfabricating a dielectric material in a size smaller than the wavelength of the incident light. In this structure, the refractive index for the incident light is spatially modulated. Alternatively, the incident light may be encoded by directly processing a plurality of pixels included in the image sensor 33 without using the filter array 32.

From the above, it can be said that the camera 30 has multiple light receiving areas with different optical response characteristics. When the camera 30 is equipped with a filter array 32 including multiple filters, and the multiple filters have optical transmission characteristics that are irregularly different from one another, the multiple light receiving areas can be realized by an image sensor 33 arranged adjacent to or directly above the filter array 32. In this case, the optical response characteristics of the multiple light receiving areas are determined based on the optical transmission characteristics of the multiple filters included in the filter array 32.

Alternatively, if the camera 30 does not include a filter array 32, the multiple light receiving regions can be realized by, for example, an image sensor 33 in which multiple pixels are directly processed so that their photoresponse characteristics are irregularly different from one another. In this case, the photoresponse characteristics of the multiple light receiving regions are determined based on the photoresponse characteristics of the multiple pixels included in the image sensor 33.

The above multilayer film, organic material, diffraction grating structure, microstructure including metal, or metasurface can encode incident light if it is configured such that the spectral transmittance is modulated to vary depending on the position in a two-dimensional plane. Therefore, the above multilayer film, organic material, diffraction grating structure, microstructure including metal, or metasurface does not need to be configured with multiple filters arranged in an array.

Next, an example of signal processing by the image processing device 34 will be described. The image processing device 34 reconstructs a multi-wavelength hyperspectral image 36 based on the compressed image 35 output from the image sensor 33 and the spatial distribution characteristics of the transmittance for each wavelength of the filter array 32. Here, multi-wavelength means more wavelength ranges than the wavelength ranges of the three colors RGB captured by a normal color camera, for example. The number of wavelength ranges can be, for example, about 4 to 100. This number of wavelength ranges is referred to as the "number of bands." Depending on the application, the number of bands may exceed 100.

The data to be obtained is the data of the hyperspectral image 36, and the data is denoted as f. If the number of bands is denoted as N, f is data obtained by integrating the data f ₁ , f ₂ , ..., f _N of N image bands. Here, the horizontal direction of the image is the x direction, and the vertical direction of the image is the y direction. If the number of pixels in the x direction of the image data to be obtained is denoted as m, and the number of pixels in the y direction is denoted as n, each of the image data f ₁ , f ₂ , ..., f _N has m x n pixel values. Therefore, the data f is data with the number of elements m x n x N. On the other hand, the data g of the compressed image 35 obtained by encoding and multiplexing by the filter array 32 is two-dimensional data including m x n pixel values corresponding to m x n pixels. The data g can be expressed by the following formula (1).

In formula (1), f represents the data of the hyperspectral image expressed as a one-dimensional vector. Each of _{f 1} , f ₂ , ..., f _N has m x n elements. Therefore, the vector on the right side is a one-dimensional vector of m x n x N rows and 1 column. In formula (1), the data g of the compressed image 35 is converted and expressed as a one-dimensional vector of m x n rows and 1 column. The matrix H represents a transformation in which each component f ₁ , f ₂ , ..., f _N of the vector f is encoded and intensity-modulated with different encoding information for each wavelength band, and then added. Therefore, H is a matrix of m x n rows and m x n x N columns. Formula (1) can also be expressed as follows.

g=(pg ₁₁ ...pg _1n ...pg _m1 ...pg _mn ) ^T =H( _f1 ... _fN ) ^T
Here, pg _ij represents the pixel value in the i-th row and j-th column of the compressed image 35 .

If vector g and matrix H are given, it seems possible to calculate f by solving the inverse problem of equation (1). However, because the number of elements m×n×N of the desired data f is greater than the number of elements m×n of the acquired data g, this problem is ill-posed and cannot be solved as is. Therefore, the image processing device 34 utilizes the sparsity of the image contained in the data f to find a solution using a compressed sensing technique. Specifically, the desired data f is estimated by solving the following equation (2).

Here, f' represents the estimated f data. The first term in the parentheses in the above equation represents the amount of deviation between the estimated result Hf and the acquired data g, the so-called residual term. Here, the sum of squares is used as the residual term, but the absolute value or the square root of the sum of squares, etc. may also be used as the residual term. The second term in the parentheses is a regularization term or stabilization term. Equation (2) means to find f that minimizes the sum of the first and second terms. The function in the parentheses in equation (2) is called the evaluation function. The image processing device 34 can converge the solution by recursive iterative calculations and calculate the f that minimizes the evaluation function as the final solution f'.

The first term in the parentheses in formula (2) means an operation to obtain the sum of squares of the difference between the acquired data g and Hf obtained by transforming f in the estimation process by the matrix H. The second term Φ(f) is a constraint condition in the regularization of f, and is a function reflecting the sparse information of the estimated data. This function has the effect of smoothing or stabilizing the estimated data. The regularization term can be expressed, for example, by the discrete cosine transform (DCT), wavelet transform, Fourier transform, or total variation (TV) of f. For example, when the total variation is used, stable estimated data that suppresses the influence of noise in the observed data g can be obtained. The sparsity of the subject 10b in the space of each regularization term differs depending on the texture of the subject 10b. A regularization term that makes the texture of the subject 10b sparser in the space of the regularization term may be selected. Alternatively, multiple regularization terms may be included in the operation. τ is a weighting coefficient. The larger the weighting factor τ, the more redundant data is reduced, and the higher the compression ratio. The smaller the weighting factor τ, the weaker the convergence to a solution. The weighting factor τ is set to an appropriate value that allows f to converge to a certain extent, but does not result in over-compression.

16B and 16C, the image encoded by the filter array 32 is acquired in a blurred state on the imaging surface of the image sensor 33. Therefore, by storing this blur information in advance and reflecting the blur information in the above-mentioned matrix H, the hyperspectral image 36 can be reconstructed. Here, the blur information is represented by a point spread function (PSF). The PSF is a function that defines the degree of spread of a point image to surrounding pixels. For example, when a point image corresponding to one pixel on an image spreads to a region of k×k pixels around the pixel due to blurring, the PSF can be defined as a group of coefficients that indicate the influence on the pixel value of each pixel in that region, that is, a matrix. The hyperspectral image 36 can be reconstructed by reflecting the influence of the blurring of the encoding pattern by the PSF in the matrix H. The position at which the filter array 32 is arranged is arbitrary, but a position at which the encoding pattern of the filter array 32 does not diffuse too much and disappear can be selected.

By the above processing, a hyperspectral image 36 can be restored based on a compressed image 35 of the subject 10b acquired by the image sensor 33 via the filter array 32. Details of the method for restoring the hyperspectral image 36 are disclosed in Patent Document 2. The disclosure of Patent Document 2 is incorporated herein by reference in its entirety.

[10. Notes]
The above description of the embodiments discloses the following techniques.

(Technology 1)
acquiring an image generated by photographing a first object as a first image, an image generated by photographing a second object and including information of a plurality of wavelength bands as a second image, and the first image as a correction image for correcting the second image;
determining whether or not the first image satisfies a suitability condition for the correction image based on pixel values of the first image;
Including,
method.

This method allows for more accurate acquisition of spectral information of the subject being analyzed by preventing inappropriate corrections.

(Technology 2)
The method according to technique 1, wherein the correction of the second image based on the first image is black subtraction.

This method can prevent inappropriate blackouts when shooting in dark conditions.

(Technology 3)
Determining whether the first image satisfies the suitability condition means determining that the first image satisfies the suitability condition when a condition for determining that a pixel value of the first image is small is satisfied.
The method described in technique 2.

This method makes it possible to determine whether or not a photograph was taken in a dark environment.

(Technology 4)
The correction of the second image based on the first image is a whiteboard correction.
The method described in technique 1.

This method can prevent inappropriate white board correction when the white board is not photographed correctly.

(Technique 5)
Determining whether the first image satisfies the suitability condition means determining that the first image satisfies the suitability condition when a condition for determining that a spatial distribution of pixel values of the first image is constant is satisfied.
The method described in technique 4.

This method makes it possible to determine whether or not the whiteboard has been photographed correctly.

(Technology 6)
Determining whether the first image satisfies the suitability condition includes determining whether the first image satisfies the suitability condition by comparing spectrum information acquired from the first image with pre-stored spectrum information.
The method according to technique 4.

This method makes it possible to determine whether or not the whiteboard has been photographed correctly.

(Technology 7)
the first image is a compressed image in which information of a plurality of wavelength bands about the first object is compressed;
determining whether the first image satisfies the suitability condition based on a histogram of pixel values of the compressed image;
The method described in technique 4.

This method makes it possible to determine whether or not the whiteboard has been photographed correctly.

(Technology 8)
The method according to any one of claims 2 to 4, wherein determining whether the first image satisfies the suitability condition comprises determining whether the first image satisfies the suitability condition by detecting edges in the first image.

This method makes it possible to determine whether an image was captured in a darkened state, or whether a white board was captured correctly.

(Technology 9)
and displaying an error on a display device if the first image does not satisfy the suitability condition.
The method according to any one of techniques 1 to 8.

In this way, the user can know that the first image does not meet the suitability criteria.

(Technology 10)
The error prompts the user to confirm whether the whiteboard has been correctly photographed as the first object.
The method according to technique 9.

This method allows users to check whether they are photographing the whiteboard correctly.

(Technology 11)
the error prompts a user to check a light blocking state when photographing the first subject;
The method according to technique 9.

This method allows the user to check the light blocking status.

(Technology 12)
acquiring the second image;
correcting the second image based on the first image if the first image satisfies the suitability condition;
Further comprising:
The method according to any one of techniques 1 to 11.

This method allows the second image to be appropriately corrected based on the first image.

(Technology 13)
Obtaining an image including information of a plurality of wavelength bands generated by photographing a scene including a first object and a second object;
a sub-image corresponding to the first subject generated based on the image as a first sub-image, a sub-image corresponding to the second subject generated based on the image as a second sub-image, and based on pixel values of the first sub-image, determining whether or not the first sub-image satisfies a suitability condition for use as a correction image for correcting the second sub-image;
Including,
method.

This method allows for more accurate acquisition of spectral information of the subject being analyzed by preventing inappropriate corrections.

(Technology 14)
14. The method of claim 13, wherein the correction of the second sub-image based on the first sub-image is a whiteboard correction.

This method can prevent inappropriate white board correction when the white board is not photographed correctly.

(Technology 15)
A processing circuit for carrying out the method according to any one of techniques 1 to 14.

This processing circuitry prevents inappropriate corrections and allows for more accurate acquisition of spectral information of the subject being analyzed.

(Other form 1)
An object other than a whiteboard may be used as the object for correction. The object may have a spectral reflectance characteristic in which the reflectance of light in a specific wavelength range is high and the reflectance of light in another specific wavelength range is low. For example, the spectral reflectance characteristic of the object may be stored in the memory 54. Based on the stored spectral reflectance characteristic, correction may be performed to reduce the influence of the shape of the spectrum of the irradiated light, the irradiation distribution at the time of shooting, the peripheral light falloff of the lens, the uneven sensitivity of the image sensor, and the like.

(Other form 2)
The first image may be an image including information of three or less wavelength bands, for example. For example, the second image including information of multiple wavelength bands may be an image generated based on a wavelength band corresponding to red, a wavelength band corresponding to green, and a wavelength band corresponding to blue.

The second image including information on multiple wavelength bands may be, for example, an image including information on three or fewer wavelength bands. Also, the corrected image generated by correcting the second image using the first image may be an image including information on three or fewer wavelength bands.

The technology disclosed herein is useful, for example, in cameras and measuring devices that capture multi-wavelength or high-resolution images. The technology disclosed herein can also be applied, for example, to sensing for biomedical and cosmetic applications, food foreign body and pesticide residue inspection systems, remote sensing systems, and vehicle-mounted sensing systems.

Reference Signs List 10a1, 10a2, 10b Object 20 Light source 30 Hyperspectral camera 31, 31A, 31B Optical system 32 Filter array 33 Image sensor 34 Image processing device 35 Compressed image 36W ₁ to 36W _N restored image 38 Image 40 Display device 42 Input UI
44 Display UI
50 Processing device 52 Processing circuit 54 Memory 56 Storage device 60 Stage 70 Support 80 Adjustment device

Claims (15)

acquiring an image generated by photographing a first object as a first image, an image generated by photographing a second object and including information of a plurality of wavelength bands as a second image, and the first image as a correction image for correcting the second image;
determining whether or not the first image satisfies a suitability condition for the correction image based on pixel values of the first image;
Including,
method. The correction of the second image based on the first image is black subtraction.
The method of claim 1. Determining whether the first image satisfies the suitability condition means determining that the first image satisfies the suitability condition when a condition for determining that a pixel value of the first image is small is satisfied.
The method of claim 2. The correction of the second image based on the first image is a whiteboard correction.
The method of claim 1. Determining whether the first image satisfies the suitability condition means determining that the first image satisfies the suitability condition when a condition for determining that a spatial distribution of pixel values of the first image is constant is satisfied.
The method according to claim 4. Determining whether the first image satisfies the suitability condition includes determining whether the first image satisfies the suitability condition by comparing spectrum information acquired from the first image with pre-stored spectrum information.
The method according to claim 4. the first image is a compressed image in which information of a plurality of wavelength bands about the first object is compressed;
determining whether the first image satisfies the suitability condition based on a histogram of pixel values of the compressed image;
The method according to claim 4. Determining whether or not the first image satisfies the suitability condition includes determining whether or not the first image satisfies the suitability condition by detecting edges of the first image.
The method according to claim 2 or 4. and displaying an error on a display device if the first image does not satisfy the suitability condition.
The method according to any one of claims 1 to 7. The error prompts the user to confirm whether the whiteboard has been correctly photographed as the first object.
10. The method of claim 9. the error prompts a user to check a light blocking state when photographing the first subject;
10. The method of claim 9. acquiring the second image;
correcting the second image based on the first image if the first image satisfies the suitability condition;
Further comprising:
The method according to any one of claims 1 to 7. Obtaining an image including information of a plurality of wavelength bands generated by photographing a scene including a first object and a second object;
a sub-image corresponding to the first subject generated based on the image as a first sub-image, a sub-image corresponding to the second subject generated based on the image as a second sub-image, and based on pixel values of the first sub-image, determining whether or not the first sub-image satisfies a suitability condition for use as a correction image for correcting the second sub-image;
Including,
method. the correction of the second sub-image based on the first sub-image is a whiteboard correction;
The method of claim 13. A processing circuit for performing the method according to any one of claims 1 to 7.

Images