WO2025084081A1 - Imaging system and method used for same - Google Patents

Abstract

The present system comprises: an imaging device that detects light in a first wavelength region and generates an image signal in which N number (N being an integer of 4 or greater) of wavelength band components included in the first wavelength region are superimposed; and a processing circuit that generates, on the basis of the image signal, N number of images respectively corresponding to the N number of wavelength bands. In the case where the N number of wavelength bands are numbered in ascending or descending order of the center wavelength, the processing circuit performs a processing operation on the basis of a part or all of the images among the N number of images, excluding the first image corresponding to the first wavelength band and the Nth image corresponding to the Nth wavelength band.

Description

Imaging system and method used therein

This disclosure relates to an imaging system and a method used therein.

Compressed sensing is a technique that reconstructs more data than is observed by assuming that the data distribution of the observed object is sparse in a certain space, such as frequency space. Compressed sensing can be applied, for example, to an imaging device that reconstructs an image containing more information from a small amount of observed data. An imaging device to which compressed sensing is applied generates a reconstructed image by calculation from an image in which the spectral information of the object is compressed.

Patent Document 1 discloses an example of applying compressed sensing technology to a hyperspectral camera that captures images in multiple narrow wavelength bands. The technology disclosed in Patent Document 1 makes it possible to realize a hyperspectral camera that generates high-resolution, multi-wavelength images.

U.S. Pat. No. 9,599,511

To provide a system that can perform more accurate processing based on a restored image generated from an image with compressed spectral information.

A system according to one aspect of the present disclosure includes an imaging device that detects light in a first wavelength range and generates an image signal in which components of N wavelength bands (N is an integer equal to or greater than 4) included in the first wavelength range are superimposed, and a processing circuit that generates N images corresponding to the N wavelength bands based on the image signal, and when the N wavelength bands are numbered in ascending or descending order of central wavelength, the processing circuit performs processing operations based on some or all of the remaining images of the N images, excluding a first image corresponding to the first wavelength band and an Nth image corresponding to the Nth wavelength band.

The general or specific aspects of the present disclosure may be realized in a system, an apparatus, a method, an integrated circuit, a computer program, or a recording medium such as a computer-readable recording disk, or may be realized in any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium. The computer-readable recording medium may include a non-volatile recording medium such as a CD-ROM (Compact Disc-Read Only Memory). An apparatus may be composed of one or more devices. When an apparatus is composed of two or more devices, the two or more devices may be arranged in one device, or may be arranged separately in two or more separate devices. In this specification and the claims, "apparatus" may mean not only one device, but also a system consisting of multiple devices.

The technology disclosed herein makes it possible to realize a system that can perform more accurate processing based on a restored image generated from an image with compressed spectral information.

FIG. 1A is a diagram illustrating a schematic configuration example of an imaging system. FIG. 1B is a diagram illustrating a schematic configuration example of another imaging system. FIG. 1C is a diagram illustrating a schematic configuration example of still another imaging system. FIG. 1D is a diagram illustrating a schematic configuration example of still another imaging system. FIG. 2A is a schematic diagram illustrating an example of a filter array. FIG. 2B is a diagram showing an example of the spatial distribution of the light transmittance of each of the wavelength bands W ₁ , W ₂ , . . . , W _N included in the target wavelength range. FIG. 2C is a diagram showing an example of the spectral transmittance of the area A1 included in the filter array shown in FIG. 2A. FIG. 2D is a diagram showing an example of the spectral transmittance of the area A2 included in the filter array shown in FIG. 2A. FIG. 3 is a diagram for explaining an example of the relationship between the target wavelength range W and the wavelength bands W ₁ , W ₂ , . . . , W _N included therein. FIG. 4A is a diagram for explaining the characteristics of the spectral transmittance in a certain region of the filter array. FIG. 4B is a diagram showing the results of averaging the spectral transmittance shown in FIG. 4A for each of the wavelength bands W ₁ , W ₂ , . . . , W _N. FIG. 5A is a cross-sectional view that illustrates an example of an imaging device included in the imaging system according to this embodiment. FIG. 5B is a cross-sectional view that illustrates a schematic diagram of another example of the imaging device included in the imaging system according to this embodiment. FIG. 6 is a diagram showing an example of a calculation result of a transmission spectrum of a filter. FIG. 7 is a diagram illustrating an example of the configuration of an imaging system that performs more accurate processing based on a restored image. FIG. 8 is a diagram for explaining the processing operation executed by the processing circuit in the imaging system according to the present embodiment. FIG. 9 is a flowchart illustrating an example 1 of a processing operation performed by the processing circuit. FIG. 10 is a flowchart illustrating a second example of the processing operation performed by the processing circuit. FIG. 11 is a flowchart illustrating an example 3 of the processing operation performed by the processing circuit. FIG. 12 is a diagram illustrating an example of luminance information displayed on the display device. FIG. 13 is a flowchart illustrating an example 4 of the processing operation performed by the processing circuit. FIG. 14 is a flowchart illustrating a fifth example of the processing operation performed by the processing circuit. FIG. 15 is a flowchart illustrating an example 6 of the processing operation performed by the processing circuit.

In this disclosure, all or part of a circuit, unit, device, member, or part, or all or part of a functional block in a block diagram, may be implemented by one or more electronic circuits including, for example, a semiconductor device, a semiconductor integrated circuit (IC), or an LSI (large scale integration). The LSI or IC may be integrated into one chip, or may be configured by combining multiple chips. For example, functional blocks other than memory elements may be integrated into one chip. Here, LSI or IC are referred to as different names depending on the degree of integration, and may be referred to as a system LSI, VLSI (very large scale integration), or ULSI (ultra large scale integration). Field programmable gate arrays (FPGAs), which are programmed after the LSI is manufactured, or reconfigurable logic devices, which allow the reconfiguration of the connections inside the LSI or the setup of circuit sections inside the LSI, can also be used for the same purpose.

Furthermore, all or part of the functions or operations of a circuit, unit, device, member or part can be executed by software processing. In this case, the software is recorded on one or more non-transitory recording media such as ROMs, optical disks, hard disk drives, etc., and when the software is executed by a processor, the functions specified in the software are executed by the processor and peripheral devices. The system or device may include one or more non-transitory recording media on which the software is recorded, a processor, and necessary hardware devices, such as interfaces.

In this disclosure, "light" refers to electromagnetic waves including not only visible light (wavelengths of about 400 nm to about 700 nm), but also ultraviolet light (wavelengths of about 10 nm to about 400 nm) and infrared light (wavelengths of about 700 nm to about 1 mm).

Below, exemplary embodiments of the present disclosure are described. Note that the embodiments described below are all comprehensive or specific examples. The numerical values, shapes, components, component arrangement and connection forms, steps, and order of steps shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Furthermore, among the components in the following embodiments, components that are not described in an independent claim that indicates a top-level concept are described as optional components. Furthermore, each figure is a schematic diagram and is not necessarily a precise illustration. Furthermore, in each figure, substantially identical components are given the same reference numerals, and duplicate explanations may be omitted or simplified.

(Findings that formed the basis of this disclosure)
Before describing the embodiments of the present disclosure, the findings on which the present disclosure is based will be described.

In the field of imaging, the process of classifying multiple objects captured in an image into types is carried out in fields such as factory automation (FA) and medicine. Examples of features used in classification processing include the shape of the object and the spectral information of the object. Hyperspectral cameras can acquire multi-wavelength images that contain a lot of spectral information for each pixel, so their use is expected to become more widespread in the future. While research and development of hyperspectral cameras has been conducted around the world for many years, their use has been limited for the following reasons. For example, line-scan hyperspectral cameras have high spatial and wavelength resolution, but the shooting time is long due to the line scan. Snapshot hyperspectral cameras can take images in one shot, but the sensitivity and spatial resolution are often insufficient.

In response to these problems, it has been reported in recent years that the sensitivity and spatial resolution of hyperspectral cameras can be improved by restoring images based on sparsity. Sparsity is the property that elements that characterize an observed object exist sparsely in a certain space, such as frequency space. Sparsity is widely observed in nature. By utilizing sparsity, it becomes possible to observe necessary information efficiently. Sensing technology that utilizes sparsity is called compressed sensing technology. By utilizing compressed sensing technology, it is possible to build highly efficient devices and systems.

As a specific application example of compressed sensing technology, for example, a hyperspectral camera with improved wavelength resolution as disclosed in Patent Document 1 has been proposed. The hyperspectral camera includes, for example, an optical filter with irregular light transmission characteristics in terms of space and/or wavelength. Such an optical filter is also called an "encoding mask." The encoding mask is placed on the optical path of light incident on the image sensor, and transmits light incident from an object with different light transmission characteristics depending on the area. This process using the encoding mask is called "encoding." In the image of the object acquired through the encoding mask, the spectral information of the object is compressed. The image is called a "compressed image." Mask information indicating the light transmission properties of the encoding mask is stored in advance in a storage device as a restoration table.

The processing device of the imaging device performs restoration processing based on the compressed image and the restoration table. The restoration processing can obtain more information than the compressed image, such as image information with higher resolution or image information with more wavelengths. The restoration table can be, for example, data indicating the spatial distribution of the optical response characteristics of the coding mask. The restoration processing based on such a restoration table can generate multiple restored images from one compressed image, each corresponding to multiple wavelength bands included in the target wavelength range. In the following description, "multiple restored images" will also be simply referred to as "restored image".

The inventors investigated the restoration accuracy of multiple restored images and found that the restoration accuracy of the following two restored images among the multiple restored images is lower than that of the remaining restored images. The two restored images correspond to the wavelength bands at both ends of the target wavelength range. "The wavelength bands at both ends" refers to the wavelength band with the longest central wavelength and the wavelength band with the shortest central wavelength. When processing is performed based on multiple restored images including the two restored images with low restoration accuracy, it is not easy to perform accurate processing.

The inventors have considered the above problems and come up with a system according to an embodiment of the present disclosure that is capable of performing more accurate processing based on the restored images. In the system according to this embodiment, processing is performed based on some or all of the remaining restored images. The two restored images corresponding to the wavelength bands at both ends are not used in this processing. As a result, it becomes possible to perform more accurate processing based on the restored images.

(Embodiment)
In the following, first, an imaging system that generates a restored image from a compressed image will be described, followed by a method for performing more accurate processing based on the restored image in the imaging system.

[1. Imaging system]
FIG. 1A is a diagram showing a schematic configuration example of an imaging system. The system shown in FIG. 1A includes an imaging device 100 and an image processing device 200. The imaging device 100 has a configuration similar to that of the imaging device disclosed in Patent Document 1. The imaging device 100 includes an optical system 140, a filter array 110, and an image sensor 160. The optical system 140 and the filter array 110 are disposed on the optical path of light incident from an object 70, which is a subject. The filter array 110 in the example of FIG. 1A is disposed between the optical system 140 and the image sensor 160.

FIG. 1A illustrates an apple as an example of the object 70. The object 70 is not limited to an apple, and may be any object. The image sensor 160 generates data of a compressed image 10 in which information of a plurality of wavelength bands is compressed as a two-dimensional monochrome image. The image processing device 200 generates data representing a plurality of images corresponding one-to-one to a plurality of wavelength bands included in a predetermined target wavelength range based on the data of the compressed image 10 generated by the image sensor 160. Here, the number of wavelength bands included in the target wavelength range is N (N is an integer of 4 or more). In the following description, the N images generated based on the compressed image 10 are referred to as restored images 20W ₁ , 20W ₂ , ..., 20W _N , and these may be collectively referred to as "hyperspectral images 20".

In this embodiment, the filter array 110 is an array of multiple light-transmitting filters arranged in rows and columns. The multiple filters include multiple types of filters that differ from one another in terms of spectral transmittance, i.e., the wavelength dependency of light transmittance. The filter array 110 modulates the intensity of incident light for each wavelength and outputs it. This process performed by the filter array 110 is called "encoding," and the filter array 110 is also called an "encoding mask."

In the example shown in FIG. 1A, the filter array 110 is disposed near or directly above the image sensor 160. Here, "near" means close enough that a reasonably clear image of the light from the optical system 140 is formed on the surface of the filter array 110. "Directly above" means that the two are so close that there is almost no gap between them. The filter array 110 and the image sensor 160 may be integrated.

Optical system 140 includes at least one lens. In FIG. 1A, optical system 140 is shown as a single lens, but optical system 140 may be a combination of multiple lenses. Optical system 140 forms an image on the imaging surface of image sensor 160 through filter array 110.

The filter array 110 may be disposed away from the image sensor 160. FIGS. 1B to 1D are diagrams showing configuration examples of the imaging device 100 in which the filter array 110 is disposed away from the image sensor 160. In the example of FIG. 1B, the filter array 110 is disposed between the optical system 140 and the image sensor 160 and at a position distant from the image sensor 160. In the example of FIG. 1C, the filter array 110 is disposed between the object 70 and the optical system 140. In the example of FIG. 1D, the imaging device 100 includes two optical systems 140A and 140B, and the filter array 110 is disposed between them. As in these examples, an optical system including one or more lenses may be disposed between the filter array 110 and the image sensor 160.

The image sensor 160 is a monochrome type light detection device having a plurality of light detection elements (also referred to as "pixels" in this specification) arranged two-dimensionally. The image sensor 160 may be, for example, a CCD (Charge-Coupled Device), a CMOS (Complementary Metal Oxide Semiconductor) sensor, or an infrared array sensor. The light detection elements include, for example, photodiodes. The image sensor 160 does not necessarily have to be a monochrome type sensor. For example, a color type sensor may be used. The color type sensor may include, for example, a plurality of red (R) filters that transmit red light, a plurality of green (G) filters that transmit green light, and a plurality of blue (B) filters that transmit blue light. The color type sensor may further include a plurality of IR filters that transmit infrared light. The color type sensor may also include a plurality of transparent filters that transmit all red, green, and blue light. By using a color type sensor, the amount of information about wavelengths can be increased, and the accuracy of the reconstruction of the hyperspectral image 20 can be improved. The wavelength range to be acquired can be determined arbitrarily, and is not limited to the visible wavelength range, and may be an ultraviolet, near-infrared, mid-infrared, or far-infrared wavelength range.

The image processing device 200 may be a computer including one or more processors and one or more storage media such as a memory. The image processing device 200 generates data of decompressed images 20W ₁ , 20W ₂ , ..., 20W _N based on the compressed image 10 acquired by the image sensor 160.

FIG. 2A is a diagram showing a schematic example of a filter array 110. The filter array 110 has a number of regions arranged two-dimensionally. In this specification, each of the multiple regions may be referred to as a "cell." An optical filter having an individually set spectral transmittance is disposed in each region. The spectral transmittance is expressed as a function T(λ), where λ is the wavelength of the incident light. The spectral transmittance T(λ) can take a value between 0 and 1.

In the example shown in FIG. 2A, the filter array 110 has 48 rectangular regions arranged in 6 rows and 8 columns. This is merely an example, and in actual applications, more regions may be provided. The number may be approximately the same as the number of pixels in the image sensor 160, for example. The number of filters included in the filter array 110 is determined according to the application, ranging from tens to tens of millions, for example.

2B is a diagram showing an example of the spatial distribution of the light transmittance of each of the wavelength bands _W1 , _W2 , ..., _WN included in the target wavelength range. In the example shown in Fig. 2B, the difference in the shading of each region represents the difference in the transmittance. The lighter the region, the higher the transmittance, and the darker the region, the lower the transmittance. As shown in Fig. 2B, the spatial distribution of the light transmittance differs depending on the wavelength band.

2C and 2D are diagrams showing examples of the spectral transmittance of the region A1 and the region A2 included in the filter array 110 shown in FIG. 2A, respectively. The spectral transmittance of the region A1 and the spectral transmittance of the region A2 are different from each other. In this way, the spectral transmittance of the filter array 110 varies depending on the region. However, it is not necessary that the spectral transmittance of all the regions is different. In the filter array 110, the spectral transmittance of at least some of the multiple regions is different from each other. The filter array 110 includes two or more filters having different spectral transmittances from each other. In one example, the number of patterns of the spectral transmittance of the multiple regions included in the filter array 110 may be the same as or greater than the number N of wavelength bands included in the target wavelength range. The filter array 110 may be designed so that the spectral transmittance of more than half of the regions is different.

FIG. 3 is a diagram for explaining the relationship between the target wavelength range W and the wavelength bands W ₁ , W ₂ , ..., W _N included therein. The target wavelength range W can be set to various ranges depending on the application. The target wavelength range W can be, for example, a visible light wavelength range of about 400 nm to about 700 nm, a near-infrared wavelength range of about 700 nm to about 2500 nm, or a near-ultraviolet wavelength range of about 10 nm to about 400 nm. Alternatively, the target wavelength range W may be a wavelength range such as mid-infrared or far-infrared. In this way, the wavelength range used is not limited to the visible light range. In this specification, radiation in general, including infrared and ultraviolet rays, is referred to as "light" not limited to visible light.

In the example shown in Fig. 3, N is an arbitrary integer equal to or greater than 4, and the wavelength bands obtained by equally dividing the target wavelength range W into N are wavelength bands _W1 , _W2 , ..., _WN . However, this is not a limitation. The multiple wavelength bands included in the target wavelength range W may be set arbitrarily. For example, the bandwidths of the wavelength bands may be made non-uniform.

FIG. 4A is a diagram for explaining the characteristics of the spectral transmittance in a certain region of the filter array 110. In the example shown in FIG. 4A, the spectral transmittance has multiple maximum values P1 to P5 and multiple minimum values with respect to wavelengths in the target wavelength range W. In the example shown in FIG. 4A, the maximum value of the light transmittance in the target wavelength range W is normalized to 1 and the minimum value is 0. In the example shown in FIG. 4A, the spectral transmittance has maximum values in wavelength ranges such as wavelength band W ₂ and wavelength band W _N-1 . In this way, the spectral transmittance of each region can be designed to have maximum values in at least two wavelength ranges among the wavelength bands W ₁ , W ₂ , ..., W _N. In the example of FIG. 4A, the maximum values P1, P3, P4, and P5 are 0.5 or more.

In this way, the light transmittance of each region varies depending on the wavelength. Therefore, the filter array 110 transmits a large amount of components in a certain wavelength range among the incident light, and does not transmit components in other wavelength ranges as much. For example, the transmittance of light in k wavelength bands out of the N wavelength bands may be greater than 0.5, and the transmittance of light in the remaining N-k wavelength bands may be less than 0.5, where k is an integer satisfying 2≦k<N. If the incident light is white light that contains all visible light wavelength components evenly, the filter array 110 modulates the incident light into light having multiple discrete intensity peaks with respect to wavelength for each region, and outputs this multi-wavelength light by superimposing it.

FIG. 4B is a diagram showing an example of the spectral transmittance shown in FIG. 4A averaged for each wavelength band W ₁ , W ₂ , ..., W _N. The averaged transmittance is obtained by integrating the spectral transmittance T(λ) for each wavelength band and dividing by the bandwidth of the wavelength band. In this specification, the transmittance averaged for each wavelength band is defined as the transmittance for that wavelength band. In this example, the transmittance is remarkably high in three wavelength ranges with maximum values P1, P3, and P5. In particular, the transmittance exceeds 0.8 in two wavelength ranges with maximum values P3 and P5.

In the examples shown in Figures 2A to 2D, a grayscale transmittance distribution is assumed in which the transmittance of each region can take any value between 0 and 1 inclusive. However, it is not necessary to use a grayscale transmittance distribution. For example, a binary scale transmittance distribution may be used in which the transmittance of each region can take a value of either approximately 0 or approximately 1. In a binary scale transmittance distribution, each region transmits most of the light in at least two of the multiple wavelength ranges included in the target wavelength range, and does not transmit most of the light in the remaining wavelength ranges. Here, "most" refers to approximately 80% or more.

A part of all the cells, for example, half of the cells, may be replaced with a transparent region. Such a transparent region transmits the light of each of the wavelength bands W ₁ , W ₂ , ..., W _N included in the target wavelength range W with a similarly high transmittance, for example, a transmittance of 80% or more. In such a configuration, the multiple transparent regions may be arranged, for example, in a checkerboard pattern. That is, in two arrangement directions of the multiple regions in the filter array 110, regions whose light transmittance varies depending on the wavelength and transparent regions may be arranged alternately.

The data showing the spatial distribution of the spectral transmittance of the filter array 110 is acquired in advance based on design data or actual measurement calibration, and is stored in a storage medium provided in the image processing device 200. This data is used in the calculation processing described below.

The filter array 110 may be configured using, for example, a multilayer film, an organic material, a diffraction grating structure, a microstructure including a metal, or a metasurface. When a multilayer film is used, for example, a dielectric multilayer film or a multilayer film including a metal layer may be used. In this case, at least one of the thickness, material, and stacking order of each multilayer film is formed so that it differs for each cell. This allows different spectral characteristics to be realized for each cell. By using a multilayer film, a sharp rise and fall in the spectral transmittance can be realized. A configuration using an organic material can be realized by making the pigment or dye contained different for each cell, or by stacking different materials. A configuration using a diffraction grating structure can be realized by providing a diffraction structure with a different diffraction pitch or depth for each cell. A microstructure including a metal can be created by utilizing the spectrum due to the plasmon effect. A metasurface can be created by microfabricating a dielectric material in a size smaller than the wavelength of the incident light. In this structure, the refractive index for the incident light is spatially modulated. Alternatively, the incident light may be encoded by directly processing a plurality of pixels included in the image sensor 160 without using the filter array 110.

From the above, it can be said that the imaging device 100 has multiple light receiving areas with different optical response characteristics. When the imaging device 100 is equipped with a filter array 110 including multiple filters, and the multiple filters have optical transmission characteristics that are irregularly different from one another, the multiple light receiving areas can be realized by an image sensor 160 in which the filter array 110 is disposed nearby or directly above it. In this case, the optical response characteristics of the multiple light receiving areas are determined based on the optical transmission characteristics of the multiple filters included in the filter array 110.

Alternatively, if the imaging device 100 does not include a filter array 110, the multiple light receiving regions can be realized by, for example, an image sensor 160 in which multiple pixels are directly processed so that their photoresponse characteristics are irregularly different from one another. In this case, the photoresponse characteristics of the multiple light receiving regions are determined based on the photoresponse characteristics of the multiple pixels included in the image sensor 160.

The above multilayer film, organic material, diffraction grating structure, microstructure including metal, or metasurface can encode incident light if it is configured such that the spectral transmittance is modulated to vary depending on the position in a two-dimensional plane. Therefore, the above multilayer film, organic material, diffraction grating structure, microstructure including metal, or metasurface does not need to be configured with multiple filters arranged in an array.

Next, an example of signal processing by the image processing device 200 will be described. The image processing device 200 reconstructs a multi-wavelength hyperspectral image 20 based on the compressed image 10 output from the image sensor 160 and the spatial distribution characteristics of the transmittance for each wavelength of the filter array 110. Here, multi-wavelength means more wavelength ranges than the wavelength ranges of the three colors RGB captured by a normal color camera, for example. The number of wavelength ranges can be, for example, about 4 to 100. This number of wavelength ranges is referred to as the "number of bands." Depending on the application, the number of bands may exceed 100.

The data to be obtained is the data of the hyperspectral image 20, and the data is denoted as f. If the number of bands is denoted as N, f is data obtained by integrating data f ₁ , f ₂ , ..., f _N of N image bands. Here, the horizontal direction of the image is the x direction, and the vertical direction of the image is the y direction. If the number of pixels in the x direction of the image data to be obtained is denoted as m, and the number of pixels in the y direction is denoted as n, each of the image data f ₁ , f ₂ , ..., f _N has n x m brightness values. Therefore, the data f is data with n x m x N elements. On the other hand, the number of elements of the data g of the compressed image 10 obtained by encoding and multiplexing by the filter array 110 is n x m. The data g can be expressed by the following formula (1).

In formula (1), f represents the data of the hyperspectral image expressed as a one-dimensional vector. Each of _{f 1} , f ₂ , ..., f _N has n x m elements. Therefore, the vector on the right side is a one-dimensional vector with n x m x N rows and one column. The data g of the compressed image is calculated as a one-dimensional vector with n x m rows and one column. The matrix H represents a transformation in which each component f ₁ , f ₂ , ..., f _N of the vector f is encoded and intensity-modulated with different encoding information for each wavelength band, and then added. Therefore, H is a matrix with n x m rows and n x m x N columns. Formula (1) can also be expressed as follows.
g=(pg ₁₁ ...pg _1m ...pg _n1 ...pg _nm ) ^T =H( _f1 ... _fN ) ^T
Here, pg _ij represents the luminance value of the i-th row and j-th column of the compressed image 10 .

Given a vector g and a matrix H, it seems possible to calculate f by solving the inverse problem of equation (1). However, since the number of elements n×m×N of the desired data f is greater than the number of elements n×m of the acquired data g, this problem is an ill-posed problem and cannot be solved as is. Therefore, the image processing device 200 utilizes the sparsity of the image contained in the data f to find a solution using a compressed sensing technique. Specifically, the desired data f is estimated by solving the following equation (2).

Here, f' represents the estimated f data. The first term in the parentheses in the above equation represents the amount of deviation between the estimated result Hf and the acquired data g, the so-called residual term. Here, the sum of squares is used as the residual term, but the absolute value or the square root of the sum of squares, etc. may also be used as the residual term. The second term in the parentheses is a regularization term or stabilization term. Equation (2) means to find f that minimizes the sum of the first and second terms. The function in the parentheses in equation (2) is called the evaluation function. The image processing device 200 can converge the solution by recursive iterative calculations and calculate f that minimizes the evaluation function as the final solution f'.

The first term in the parentheses in formula (2) means an operation to obtain the sum of squares of the difference between the acquired data g and Hf obtained by transforming f in the estimation process by the matrix H. The second term Φ(f) is a constraint condition in the regularization of f, and is a function reflecting the sparse information of the estimated data. This function has the effect of smoothing or stabilizing the estimated data. The regularization term can be expressed, for example, by the discrete cosine transform (DCT), wavelet transform, Fourier transform, or total variation (TV) of f. For example, when the total variation is used, stable estimated data that suppresses the influence of noise in the observed data g can be obtained. The sparsity of the object 70 in the space of each regularization term differs depending on the texture of the object 70. A regularization term that makes the texture of the object 70 sparser in the space of the regularization term may be selected. Alternatively, multiple regularization terms may be included in the operation. τ is a weighting coefficient. The larger the weighting factor τ, the more redundant data is reduced, and the higher the compression ratio. The smaller the weighting factor τ, the weaker the convergence to a solution. The weighting factor τ is set to an appropriate value that allows f to converge to a certain extent, but does not result in over-compression.

1B and 1C, the image encoded by the filter array 110 is acquired in a blurred state on the imaging surface of the image sensor 160. Therefore, the hyperspectral image 20 can be reconstructed by storing this blur information in advance and reflecting the blur information in the above-mentioned matrix H. Here, the blur information is represented by a point spread function (PSF). The PSF is a function that defines the degree of spread of a point image to surrounding pixels. For example, when a point image corresponding to one pixel on an image spreads to a region of k×k pixels around the pixel due to blurring, the PSF can be defined as a group of coefficients that indicate the influence on the luminance value of each pixel in that region, that is, a matrix. The hyperspectral image 20 can be reconstructed by reflecting the influence of blurring of the encoding pattern by the PSF in the matrix H. The position at which the filter array 110 is arranged is arbitrary, but a position at which the encoding pattern of the filter array 110 does not diffuse too much and disappear can be selected.

By the above processing, the hyperspectral image 20 can be restored based on the compressed image 10 acquired by the image sensor 160. Details of the method for restoring the hyperspectral image 20 are disclosed in Patent Document 1. The entire disclosure of Patent Document 1 is incorporated herein by reference.

2. Configuration example of filter array 110
A more specific example of the structure of the filter array 110 will be described below with reference to FIGS. 5A and 5B.

FIG. 5A is a cross-sectional view that shows a schematic example of an imaging device 100 included in the imaging system according to this embodiment. The imaging device 100 includes a filter array 110 and an image sensor 60. In FIG. 5A, the optical system 140 is omitted.

The filter array 110 comprises a plurality of filters 112 arranged two-dimensionally. The plurality of filters 112 are arranged in rows and columns, for example, as shown in FIG. 2A. FIG. 5A shows a schematic cross-sectional structure of one row shown in FIG. 2A. Each of the plurality of filters 112 is a Fabry-Perot filter and has a resonant structure. A resonant structure means a structure in which light of a certain wavelength exists stably by forming a standing wave inside. This state of light is sometimes called a "resonant mode."

The resonant structure shown in FIG. 5A includes a first reflective layer 28a, a second reflective layer 28b, and an intermediate layer 26 between the first reflective layer 28a and the second reflective layer 28b. The first reflective layer 28a and/or the second reflective layer 28b may be formed of a dielectric multilayer film or a metal thin film. The intermediate layer 26 may be formed of a dielectric or a semiconductor that is transparent in a specific wavelength range. The intermediate layer 26 may be formed of at least one selected from the group consisting of Si, Si ₃ N ₄ , TiO ₂ , Nb ₂ O ₅ , and Ta ₂ O ₅ , for example. The refractive index and/or thickness of the intermediate layer 26 of the multiple filters 112 differs depending on the filter. The transmission spectrum of each of the multiple filters 112 has a maximum value of transmittance at multiple wavelengths. The multiple wavelengths correspond to multiple resonance modes of different orders in the above-mentioned resonant structure. In this embodiment, all of the filters 112 in the filter array 110 have the above-mentioned resonant structure. However, the filter array 110 may include a filter that does not have the above-mentioned resonant structure. For example, a filter that does not have wavelength dependency of light transmittance, such as a transparent filter or a neutral density filter (ND filter), may be included in the filter array 110. In this embodiment, each of two or more filters 112 among the multiple filters 112 includes the above-mentioned resonant structure.

The image sensor 160 includes a plurality of photodetection elements 160a. Each of the plurality of photodetection elements 160a is disposed opposite one of the plurality of filters. The aforementioned target wavelength range W may be, for example, an overlapping portion of the sensitivity range of each photodetection element 160a and the transmission range of the optical system 140. The sensitivity range of each photodetection element 160a is a wavelength range in which each photodetection element 160a is sensitive to light. The transmission range of the optical system 140 is a wavelength range in which the optical system 140 transmits light with a transmittance of, for example, 60%, 80%, or 90% or more. When the transmission range of the optical system 140 includes all of the sensitivity ranges of each photodetection element 160a, the target wavelength range W corresponds to the sensitivity range of each photodetection element 160a.

In this specification, the term "wavelength range sensitive to light" refers to a wavelength range having the substantial sensitivity required to detect light. For example, it refers to an external quantum efficiency of 1% or more in that wavelength range. The external quantum efficiency of the light detection element 160a may be 10% or more, or 20% or more. In the following description, the light detection element 160a may be referred to as a "pixel."

In the example shown in FIG. 5A, the filter array 110 and the image sensor 160 are integrally formed, but this is not the only example. The filter array 110 and the image sensor 160 may be separate. Even in this case, each of the multiple light detection elements 160a is disposed at a position where it receives light that has passed through one of the multiple filters 112. The components may be disposed so that the light that has passed through the multiple filters 112 is incident on each of the multiple light detection elements 160a via a mirror. In this case, each of the multiple light detection elements 160a is not disposed directly below one of the multiple filters.

Furthermore, in the example shown in FIG. 5A, the multiple light detection elements 160a correspond one-to-one to the multiple filters 112, but this is not limited to the example. The multiple light detection elements 160a do not have to correspond one-to-one to the multiple filters 112. For example, light that has passed through two or more filters 112 may be incident on one light detection element 160a.

5B is a cross-sectional view showing a schematic diagram of another example of the imaging device 100 included in the imaging system according to the present embodiment. The imaging device 100 further includes a bandpass filter 29 in addition to the filter array 110 and the image sensor 160. In the example shown in FIG. 5B, a bandpass filter 29 is disposed between the filter array 110 and the image sensor 160, as compared with the example shown in FIG. 5A. The bandpass filter 29 has a transmission band, which is a wavelength band that transmits light, and a blocking band, which is a wavelength band that is shorter and longer than the transmission band and suppresses the transmission of light. The blocking band may include a first blocking band and a second blocking band. If the minimum wavelength of the transmission band is λ1 and the maximum wavelength of the transmission band is λ2, the maximum wavelength of the first blocking band may be smaller than λ1 and the minimum wavelength of the second blocking band may be larger than λ2. In other words, (wavelength of the first blocking band) < λ1, λ1 ≦ (wavelength of the transmission band) ≦ λ2, and λ2 < (wavelength of the second blocking band) may be satisfied. The transmittance in the transmission band of the bandpass filter 29 may be, for example, 60% or more, 80% or more, or 90% or more. The transmittance in the stop band of the bandpass filter 29 may be, for example, 20% or less, 10% or less, or 5% or less.

When the imaging device 100 includes a bandpass filter 29, the aforementioned target wavelength range W can be, for example, the overlapping portion of the sensitivity range of each photodetector element 160a, the transmission range of the optical system 140, and the transmission range of the bandpass filter 29. When the transmission range of the optical system 140 includes all of the sensitivity ranges of each photodetector element 160a, and the sensitivity range of each photodetector element 160a includes all of the transmission ranges of the bandpass filter 29, the target wavelength range W corresponds to the transmission range of the bandpass filter 29.

Next, the transmission spectrum of the filter 112 will be described with reference to FIG. 6. FIG. 6 is a diagram showing an example of the calculation result of the transmission spectrum of the filter 112. In this example, the first reflection layer 28a in the filter 112 is formed of a dielectric multilayer film in which _TiO2 layers and _SiO2 layers are alternately stacked. The same is true for the second reflection layer 28b. That is, the second reflection layer 28b is formed of a dielectric multilayer film in which TiO2 layers and SiO2 layers are alternately stacked. The intermediate layer 26 in the filter 112 is formed of a _TiO2 layer. The solid line and the dashed line shown in FIG. 6 represent the transmission spectrum when the intermediate layer 26 has different thicknesses. DiffractMOD based on RSoft's rigorous coupled-wave analysis (RCWA) was used to calculate the transmission spectrum.

As shown in FIG. 6, the transmission spectrum varies depending on the thickness of the intermediate layer 26. In this manner, a filter array 110 having a plurality of filters 112 with different transmission spectra can be realized. For example, in a filter array 110 including 10 ³ to 10 ⁷ filters 112, two or more, three or more, four or more, or ten or more types of filters 112 with different transmission spectra are irregularly distributed. The transmission spectrum of each filter 112 has a plurality of peaks in the target wavelength range W. Each peak has a maximum value of transmittance and minimum values on both sides of the maximum value. The difference between the maximum value and each minimum value rate can be, for example, 10% or more, 20% or more, or 30% or more. The filter array 110 is an example of an optical element having a plurality of regions with different transmission spectra. The plurality of filters 112 is an example of a plurality of regions.

[3. Details of restoration process using evaluation function and issues regarding restoration accuracy]
The restoration process using the evaluation function will be described in more detail below. As expressed by the following formula (3), the evaluation function in parentheses is minimized.

In equation (3), the first term is the DF (Data Fidelity) term, and the second term is the TV (Total Variation) term. The DF and TV terms are examples of terms included in the evaluation function.

The f that minimizes the evaluation function can be estimated, for example, by the TwIST (Two-step Iterative Shrinkage/Thresholding) method using the following equations (4) and (5). The TwIST method is an example of a method for estimating the f that minimizes the evaluation function.

In the TwIST method, f is updated a predetermined number of times to minimize the evaluation function. The t-th updated f is defined as f _t . By setting the initial value f ₀ , f ₁ can be obtained using equation (4). Ψ _TV in equation (4) is a noise removal function.

f _t (t≧2) is obtained using equation (5), where α and β are preset values.

Updating f using equation (5) includes the following processes (A) and (B).
(A) Update f by the steepest descent method so that the DF term becomes smaller.
(B) By minimizing the Chambolle TV norm, f in (A) is used as the initial value, and f is updated so that the TV term is reduced.

Therefore, processes (A) and (B) are repeatedly executed by repeatedly updating f using equation (5).

In process (B), the initial value is important, and the encoding mask does not affect the update of the initial value. On the other hand, in process (A), the encoding mask affects the update of f. In process (A), the steepest descent method is performed by calculating the error in the DF term with f at that time, and subtracting the value obtained by proportionally allocating the error based on the transmittance of each region of the encoding mask from f at that time.

The coding mask can be realized by a Fabry-Perot filter, such as the filter array 110 shown in FIG. 5A and FIG. 5B. The light transmission characteristics of such a coding mask are strongly correlated between two nearby wavelength bands. There are many wavelength bands with strong correlation near the center of the target wavelength range W, while there are few wavelength bands with strong correlation at both ends of the target wavelength range W. Because of this, in the process (A), the convergence characteristics by the TwIST method are worse in the wavelength bands at both ends compared to the wavelength bands near the center. As a result, among the multiple restored images corresponding to the multiple wavelength bands included in the target wavelength range W, the restoration accuracy of the two restored images corresponding to the wavelength bands at both ends is lower than the restoration accuracy of the remaining restored images. In other words, if the restoration accuracy of the restored image corresponding to wavelength band _W1 in FIG. 3 is _rc1 , ..., and the restoration accuracy of the restored image corresponding to wavelength band _WN is _rcN , then _rc1 is lower than each of _rc2 , ..., rc _(N-1) , and _rcN is lower than each of _rc2 , ..., rc _(N-1) .

It is believed that a decrease in the restoration accuracy of the two restored images corresponding to the wavelength bands at both ends occurs when the following two conditions are satisfied.
In the reconstruction process, the optical transmission properties of the coding mask are exploited, as is the minimization of the DF term.
The correlation of the optical transmission characteristics of the coding mask used is stronger between two nearby wavelength bands than between two distant wavelength bands.

The restoration accuracy of the two restored images corresponding to the wavelength bands at both ends is low, so when processing is performed based on multiple restored images including these two restored images, it is not easy to perform accurate processing. The inventors have identified this problem and come up with an imaging system according to an embodiment of the present disclosure that is capable of performing more accurate processing based on restored images.

[4. Imaging system that performs more accurate processing based on restored images]
Fig. 7 is a diagram showing a schematic configuration example of an imaging system that performs more accurate processing based on a restored image. The imaging system shown in Fig. 7 includes an imaging device 100, an image processing device 200, and a display device 300. Lines with arrows indicate transmission and reception of signals.

The components of the imaging system shown in FIG. 7 will be described below. The imaging device 100 has been described with reference to FIGS. 1A to 1D. The imaging device 100 detects light in a target wavelength range W and generates an image signal in which components of N wavelength bands (N is an integer equal to or greater than 4) included in the target wavelength range W are superimposed. The imaging device 100 may further include a bandpass filter 29. When the target wavelength range W corresponds to the transmission range of the bandpass filter 29, the bandpass filter 29 transmits light in the target wavelength range W and suppresses the transmission of light in other wavelength ranges different from the target wavelength range W. The other wavelength range corresponds to the blocking range of the bandpass filter 29. In this specification, the target wavelength range W is also referred to as the "first wavelength range" and the other wavelength range is also referred to as the "second wavelength range". The first wavelength range is a wavelength range in which the imaging device 100 can detect light.

The image processing device 200 includes a processing circuit 210, a memory 212, and a storage device 220. The processing circuit 210 controls the operations of the imaging device 100, the storage device 220, and the display device 300. The processing circuit 210 generates N restored images corresponding to the N wavelength bands, respectively, based on the image signal. The operations performed by the processing circuit 210 based on the multiple restored images will be described in detail later.

The computer program executed by the processing circuitry 210 is stored in memory 212, such as a ROM or a RAM (Random Access Memory). The processing circuitry 210 and memory 212 may be integrated on a single circuit board or may be provided on separate circuit boards. The processing circuitry 210 may be distributed across multiple circuits. The processing circuitry 210 and/or memory 212 may be located in a remote location away from the other components via a wired or wireless communication network.

The storage device 220 includes one or more storage media. Each storage medium may be any storage medium, such as, for example, a semiconductor memory, a magnetic storage medium, or an optical storage medium. The storage device 220 stores a restoration table used to generate a restored image from the compressed image 10. The restoration table is an example of coding information indicating the light transmission characteristics of the filter array 110 that functions as an encoding mask, and is obtained by calibration before shipment. The restoration table may be, for example, table-format data indicating the matrix H in equation (2).

The display device 300 displays an input user interface (UI) 310 and a display UI 320. The input UI 310 is used for a user to input information. The information input by the user into the input UI 310 is received by the processing circuit 210. The display UI 320 is used for displaying information based on the restored image.

The input UI 310 and the display UI 320 are displayed as a graphical user interface (GUI). It can also be said that the information shown on the input UI 310 and the display UI 320 is displayed on the display device 300. The input UI 310 and the display UI 320 may be realized by a device capable of both input and output, such as a touch screen. In that case, the touch screen may function as the display device 300. When a keyboard and/or a mouse are used as the input UI 310, the input UI 310 is a device independent of the display device 300.

5. Processing operations performed by the processing circuit 210
The processing operation executed by the processing circuit 210 in the imaging system according to the present embodiment will be described below with reference to Fig. 8. Fig. 8 is a diagram for explaining the processing operation executed by the processing circuit 210 in the imaging system according to the present embodiment.

8 shows N wavelength bands included in the target wavelength range W, i.e., wavelength bands W ₁ , W ₂ , W ₃ , ..., W _N-2 , W _N-1 , and W _N. These wavelength bands are numbered in ascending order of center wavelength, but may be numbered in ascending order of center wavelength. The processing circuit 210 generates restored images 20W 1 , 20W ₂ , 20W ₃ , ..., 20W N _-2 , 20W _N-1 , and 20W _N corresponding to the wavelength bands W ₁ , W ₂ , W ₃ , _... , W _N-2 , W _N-1 , and W _N , respectively. λ _ic may be equal to (λ min (i) + λ max (i))/2. _λic is the central wavelength of wavelength band W _i , λmin(i) is the minimum wavelength included in wavelength band W _i , λmax(i) is the maximum wavelength included in wavelength band W i, and 1≦i≦N. λmin(1)=λmin and λmax(N)=λmax, or λmin(N)=λmin and λmax(1)=λmax may be satisfied. λmin is the minimum wavelength included in the target wavelength range W, and λmax is the maximum wavelength included in the target wavelength range W.

8, the N-2 wavelength bands other than the two wavelength bands at both ends, i.e., wavelength bands W ₂ , W ₃ , ..., W _N-2 , W _N-1, are shown. The processing circuit 210 performs processing operations based on a part or _all of the remaining restored images 20W ₂ , 20W ₃ , ..., 20W _N-2 , 20W _N-1 , 20W _N , excluding the restored images 20W ₁ and 20W _N , of the restored images 20W ₁ , 20W ₂ , 20W 3 , ..., 20W _N-2 , 20W _{N-1 , 20W N.} The restored image 20W ₁ corresponds to the wavelength band W ₁ with the shortest central wavelength, and the restored image 20W _N corresponds to the wavelength band W _N with the longest central wavelength. In the above process, the processing circuit 210 does not use the restored images _20W1 and _20WN , which have low restoration accuracy. Therefore, it is possible to perform more accurate processing compared to the case where processing is performed based on N restored images.

In this specification, the wavelength bands W ₁ , W ₂ , W ₃ , ..., W _N-2 , W _N-1 , and W _N are also referred to as the first wavelength band, the second wavelength band, the third wavelength band, ..., the N-2 wavelength band, the N-1 wavelength band, and the N wavelength band, respectively. Similarly, the restored images 20W ₁ , 20W ₂ , 20W ₃ , ..., 20W _N-2 , 20W _N-1 , and 20W _N are also referred to as the first restored image, the second restored image, the third restored image, ..., the N-2 restored image, the N-1 restored image, and the N restored image, respectively. The restored images are also simply referred to as "images."

Next, examples 1 to 6 of the processing operations performed by the processing circuit 210 will be described with reference to Figures 9 to 15.

Fig. 9 is a flowchart that outlines an example 1 of the processing operation executed by the processing circuit 210. The processing circuit 210 executes the operations of steps S101 to S103 shown in Fig. 9. The operation of step S103 corresponds to an example of a processing operation based on some or all of the remaining restored images 20W ₂ , 20W ₃ , ..., 20W _N-2 , 20W _N-1 .

<Step S101>
The processing circuit 210 acquires an image signal of the compressed image 10 from the imaging device 100 .

<Step S102>
The processing circuitry 210 generates restored images 20W 1 , _{20W 2} , _{20W 3 , ..., 20W N-2} , _20W N _-1 , and 20W N corresponding to the wavelength bands W ₁ , W ₂ , W ₃ , ..., W _{N -2 ,} W _N-1 , and W _N , respectively, based on the image signal acquired in step S101. That is, the processing circuitry 210 generates N restored images corresponding to the N wavelength bands included in the target wavelength range W, respectively.

<Step S103>
The processing circuit 210 outputs a part or all of the restored images 20W ₂ , 20W ₃ , ..., 20W _{N-2 ,} 20W _N-1 . More specifically, the processing circuit 210 outputs a part or all of the remaining restored images 20W ₂ , 20W ₃ , ..., 20W _N-2 , 20W _{N -1, excluding the restored image 20W 1 and the restored image 20W N} , among the N restored images. The processing circuit 210 does not output the restored image 20W ₁ and the restored image 20W _N. A part or all of the remaining output restored images 20W ₂ , 20W ₃ , ..., 20W _N-2 , 20W _N-1 may be sent to another device, such as the display device 300, the storage device 220, or an analysis device.

In this specification, "the processing circuit 210 outputs some or all of the remaining restored images 20W ₂ , 20W ₃ , ..., 20W _N-2 , 20W _N-1 " means that the processing circuit 210 outputs signals indicating some or all of the remaining restored images 20W ₂ , 20W ₃ , ..., 20W _N-2 , 20W _N-1 to the outside of the processing circuit 210.

Prior to step S101, S102, or S103, the user may specify information regarding which of the remaining restored images 20W ₂ , 20W ₃ , ..., 20W _N-2 , and 20W _N-1 should be output or which should not be output via the input UI 310 of the display device 300. The processing circuitry 210 reads the information specified by the user and executes the operation of step S103.

Alternatively, information regarding which restored images are to be output or which restored images are not to be output may be stored in memory 212 or storage device 220 before processing circuit 210 starts the processing operation. In that case, processing circuit 210 reads the information from memory 212 or storage device 220 and executes the operation of step S103.

Fig. 10 is a flowchart that outlines an example 2 of the processing operation executed by the processing circuit 210. The processing circuit 210 executes the operations of steps S201 to S203 shown in Fig. 10. The operation of step S203 corresponds to an example of a processing operation based on some or all of the remaining restored images 20W ₂ , 20W ₃ , ..., 20W _N-2 , 20W _N-1 .

<Steps S201 and S202>
The operations in steps S201 and S202 are the same as the operations in steps S101 and S102 shown in FIG.

<Step S203>
The processing circuit 210 causes the display device 300 to display part or all of the remaining restored images 20W ₂ , 20W ₃ , ..., 20W _{N-2 ,} and 20W _N-1, excluding the restored images 20W 1 and 20W N, out of the _N restored images. The processing circuit 210 does not cause the display device 300 to display the restored images 20W ₁ and 20W _N. The user can check part or all of the remaining restored images 20W ₂ , 20W ₃ , ..., 20W _N-2 , and 20W _N-1 via the display device 300.

Fig. 11 is a flowchart that outlines an example 3 of the processing operation executed by the processing circuit 210. The processing circuit 210 executes the operations of steps S301 to S303 shown in Fig. 11. The operation of step S303 corresponds to an example of a processing operation based on some or all of the remaining restored images 20W ₂ , 20W ₃ , ..., 20W _N-2 , 20W _N-1 .

<Steps S301 and S302>
The operations in steps S301 and S302 are the same as the operations in steps S101 and S102 shown in FIG.

<Step S303>
The processing circuit 210 outputs luminance information on the luminance values of images based on some or all of the remaining restored images 20W ₂ , 20W ₃ , ..., 20W _{N-2 ,} 20W _N-1 , excluding the restored image 20W ₁ and the restored image 20W _N , out of the N restored images. The luminance information may be, for example, the wavelength dependency of the luminance value of the pixel, that is, the luminance spectrum. The processing circuit 210 does not output luminance information based on the restored image 20W ₁ and the restored image 20W _N. Since the luminance information output from the processing circuit 210 is not based on the restored image 20W 1 and the restored image 20W _N , which have low restoration accuracy, more accurate luminance information can be obtained.

In this specification, "the processing circuit 210 outputs the luminance information" means that the processing circuit 210 outputs a signal indicating the luminance information to the outside of the processing circuit 210.

Processing circuitry 210 may output luminance information based on some or all of the remaining restored images 20W ₂ , 20W ₃ , ..., 20W _N-2 , 20W _N-1 and send it to an analysis device. Alternatively, processing circuitry 210 may output luminance information based on some or all of the remaining restored images 20W ₂ , 20W ₃ , ..., 20W _N-2 , 20W _N-1 and send it to display device 300. In other words, processing circuitry 210 may cause display device 300 to display the luminance information.

FIG. 12 is a diagram showing a schematic example of luminance information displayed on display device 300. FIG. 12 shows the screen of display device 300. Compressed image 10 is shown on the left side of the screen. The user specifies a point or area of object 70 shown in compressed image 10 about which the user wishes to know luminance information. The point can be specified using a cursor represented by a thick arrow, for example. The area can be specified by encircling it with the cursor, as represented by a dotted line, for example. Note that an image on which some or all of the N restored images are superimposed may be displayed on the left side of the screen, instead of compressed image 10.

On the right side of the screen, a luminance spectrum is shown as luminance information at a specified point or region. The luminance spectrum includes luminance values B ₂ , ..., B _N-1 of pixels corresponding to wavelength bands W ₂ , ..., W _N-1 at the specified point or region. When a region is specified instead of a point, each luminance value may be, for example, an average of luminance values of a plurality of pixels included in the region. In the luminance spectrum shown in FIG. 12, the wavelength corresponding to each luminance value may be, for example, the central wavelength of the wavelength band. As the luminance spectrum, all of the luminance values B ₂ , ..., B _N-1 may be displayed, or may be displayed with a predetermined number of spaces between them, such as every other or every third luminance value.

Fig. 13 is a flowchart that outlines a fourth example of the processing operation executed by the processing circuit 210. The processing circuit 210 executes the operations of steps S401 to S403 shown in Fig. 13. The operation of step S403 corresponds to an example of a processing operation based on some or all of the remaining restored images 20W ₂ , 20W ₃ , ..., 20W _N-2 , and 20W _N-1 .

<Steps S401 and S402>
The operations in steps S401 and S402 are the same as the operations in steps S101 and S102 shown in FIG.

<Step S403>
The processing circuit 210 performs an analysis process to analyze the state of the object 70 based on a part or all _{of the remaining restored images 20W 2 ,} 20W ₃ , ..., 20W _N-2 , 20W _N-1 , excluding the restored image _{20W 1} and the restored image 20W N, among the N restored images. The processing circuit 210 does not use the restored images 20W 1 and 20W _N in the analysis process. Since the restored images 20W ₁ and 20W _N , which have low restoration accuracy, are not used, more accurate analysis process can be performed. The processing circuit 210 may output the analysis result and send it to the display device 300. In other words, the processing circuit 210 may cause the display device 300 to display the analysis result.

If the object 70 is a food product, the analysis process may be, for example, a process of determining the sugar content or freshness of the object 70. Alternatively, if the object 70 is an object to be inspected, the analysis process may be a process of inspecting the object 70 and generating information indicating the inspection results. The inspection may be, for example, a process of determining whether the object 70 contains a foreign object, or whether a painted object has been appropriately painted.

Fig. 14 is a flowchart that outlines a fifth example of the processing operation executed by the processing circuit 210. The processing circuit 210 executes the operations of steps S501 to S503 shown in Fig. 14. The operation of step S503 corresponds to an example of a processing operation based on some or all of the remaining restored images 20W ₂ , 20W ₃ , ..., 20W _N-2 , and 20W _N-1 .

<Steps S501 and S502>
The operations in steps S501 and S502 are the same as the operations in steps S101 and S102 shown in FIG.

<Step S503>
The processing circuit 210 stores in the storage device 220 some or all of the remaining restored images 20W ₂ , 20W ₃ , ..., 20W _{N-2 ,} and 20W _N-1 , excluding the restored images 20W 1 and 20W N, out of the _N restored images. The processing circuit 210 does not store the restored images 20W ₁ and 20W _N in the storage device 220. The processing circuit 210 can acquire some or all of the remaining restored images 20W ₂ , 20W ₃ , ..., 20W _N-2 , and 20W _N-1 from the storage device 220 as necessary.

Fig. 15 is a flowchart that outlines a sixth example of the processing operation executed by the processing circuit 210. The processing circuit 210 executes the operations of steps S601 to S603 shown in Fig. 15. The operation of step S603 corresponds to an example of a processing operation based on some or all of the remaining restored images 20W ₂ , 20W ₃ , ..., 20W _N-2 , and 20W _N-1 .

<Steps S601 and S602>
The operations in steps S601 and S602 are the same as the operations in steps S101 and S102 shown in FIG.

<Step S603>
The processing circuit 210 generates a color image based on some or all of the remaining restored images 20W ₂ , 20W ₃ , ..., 20W _{N-2 ,} 20W _N-1 excluding the restored images 20W 1 and 20W _N among the N restored images, and causes the display device 300 to display the color image. The processing circuit 210 does not use the restored images 20W ₁ and 20W _N to generate the color image. The user can check the color image via the display device 300.

The color image is obtained by superimposing some or all of the remaining restored images 20W ₂ , 20W ₃ , ..., 20W _N-2 , 20W _N-1 colored with red, green, and blue, for example, according to the color of the central wavelength of the corresponding wavelength band.

[6. Notes]
The above description of the embodiments discloses the following techniques.

(Technique 1)
an imaging device that detects light in a first wavelength range and generates an image signal in which components of N wavelength bands (N is an integer equal to or greater than 4) included in the first wavelength range are superimposed;
a processing circuit for generating N images corresponding to the N wavelength bands based on the image signals;
Equipped with
When the N wavelength bands are numbered in ascending or descending order of center wavelength, the processing circuit performs a processing operation based on some or all of the remaining images of the N images, excluding a first image corresponding to a first wavelength band and an Nth image corresponding to an Nth wavelength band.
system.

This system allows for more accurate processing based on restored images generated from images with compressed spectral information.

(Technique 2)
the processing operation being to output some or all of the remaining image;
the processing circuit does not output the first image and the Nth image;
The system described in technique 1.

The system can send some or all of the remaining image to other devices, such as a display device, storage device, or analysis device.

(Technique 3)
the processing operation is to cause a display device to display some or all of the remaining images, and the processing circuitry does not cause the display device to display the first image and the Nth image.
The system described in technique 1.

In this system, the user can view some or all of the remaining images via a display device.

(Technique 4)
Further comprising a storage device;
the processing operation is to cause the storage device to store some or all of the remaining images, and the processing circuitry does not cause the storage device to store the first image and the Nth image.
The system described in technique 1.

In this system, the processing circuitry can retrieve some or all of the remaining image from the storage device as needed.

(Technique 5)
the processing operation is performing an analysis process based on some or all of the remaining image;
the processing circuit does not use the first image and the Nth image in the analysis process.
The system described in technique 1.

This system allows for more accurate analysis.

(Technique 6)
the processing operation being to output luminance information relating to luminance values of an image based on some or all of the remaining image;
the processing circuit does not output luminance information based on the first image and the Nth image;
The system described in technique 1.

This system allows for more accurate brightness information to be obtained.

(Technique 7)
the processing operation is generating a color image based on some or all of the remaining images, and the first image and the Nth image are not used in generating the color image;
The system described in technique 1.

In this system, the user can view a color image, for example, via a display device.

(Technique 8)
The imaging device further includes a bandpass filter that transmits the light in the first wavelength range and suppresses transmission of light in a second wavelength range different from the first wavelength range.
8. A system according to any one of techniques 1 to 7.

In this system, the first wavelength range can be defined by the transmission range of the bandpass filter.

(Technique 9)
The imaging device has a plurality of regions having different transmission spectra, and each of the transmission spectra of the plurality of regions has a plurality of peaks in the first wavelength range.
The system according to any one of techniques 1 to 8.

In this system, compressed sensing technology can be used to generate N images, each corresponding to one of the N wavelength bands.

(Technique 10)
A method executed by a computer in a system including an imaging device that detects light in a first wavelength range and generates an image signal in which components of N wavelength bands (N is an integer equal to or greater than 4) included in the first wavelength range are superimposed, the method comprising:
generating N images corresponding to the N wavelength bands based on the image signals;
performing a processing operation based on some or all of the remaining images, excluding a first image corresponding to a first wavelength band and an Nth image corresponding to an Nth wavelength band, among the N images, when the N wavelength bands are numbered in ascending or descending order of center wavelengths;
Including,
method.

This method allows for more accurate processing based on a restored image generated from an image with compressed spectral information.

(others)
The present disclosure is not limited to the above-described embodiment. As long as it does not deviate from the spirit of the present disclosure, various modifications conceived by a person skilled in the art may also be included in the present disclosure. Modifications of the embodiment of the present disclosure may be as shown below.

an imaging device that detects light passing through a filter array including a plurality of filters and generates a plurality of signals, each of the plurality of signals including a plurality of pieces of information that correspond one-to-one to a plurality of wavelength bands;
and a circuit capable of generating a plurality of images in one-to-one correspondence with the plurality of wavelength bands based on the plurality of signals;
the plurality of wavelength bands are a first wavelength band including a first wavelength, a second wavelength band including a second wavelength, and a third plurality of wavelength bands excluding the first wavelength band and the second wavelength band;
the first wavelength is the smallest of the plurality of wavelengths included in the plurality of wavelength bands;
the second wavelength is the largest among the plurality of wavelengths included in the plurality of wavelength bands,
the number of wavelength bands is N, where N is 4 or greater;
the number of wavelength bands in the third plurality is (N-2);
the plurality of images being a first image corresponding to the first wavelength band, a second image corresponding to the second wavelength band, and a third plurality of images in one-to-one correspondence with the third plurality of wavelength bands;
neither the first image nor the second image is displayed;
neither the first image nor the second image is output to one or more devices directly or indirectly connected to the circuit, whereby the one or more devices do not use both the first image and the second image;
One or more images in the third plurality of images are displayed;
g = H(f ₁ f ₃ f ₄ ... f _(N-1) f _N f ₂ ) ^T ;
g indicates information corresponding to the plurality of signals,
_f1 denotes information corresponding to a first plurality of pixel values of the first image;
_f2 denotes information corresponding to a second plurality of pixel values of the second image;
_f3 indicates information corresponding to a plurality of pixel values corresponding to image _I3 included in the third plurality of images, _f4 indicates information corresponding to a plurality of pixel values corresponding to image _I4 included in the third plurality of images, ..., f _(N-1) indicates information corresponding to a plurality of pixel values corresponding to image I _(N-1) included in the third plurality of images, _fN indicates information corresponding to a plurality of pixel values corresponding to image _IN included in the third plurality of images,
The H denotes a matrix including (p×q)×(p×q×N) values,
The number of the plurality of signals is (p×q), and the number of pixels included in each of the plurality of images is (p×q).
system.

The technology disclosed herein is useful, for example, in cameras and measuring devices that capture multi-wavelength or high-resolution images. The technology disclosed herein can also be applied, for example, to sensing for biomedical and cosmetic applications, food foreign body and pesticide residue inspection systems, remote sensing systems, and vehicle-mounted sensing systems.

REFERENCE SIGNS LIST 10, 10a, 10b Compressed image 20 Hyperspectral image _20W1 , _20W2 , ..., _20WN Reconstructed image 70 Object 100 Imaging device 110 Filter array 112 Filter 140, 140A, 140B Optical system 160 Image sensor 200 Image processing device 210 Processing circuit 212 Memory 220 Storage device 300 Display device 310 Input UI
320 Display UI

Claims (10)

an imaging device that detects light in a first wavelength range and generates an image signal in which components of N wavelength bands (N is an integer equal to or greater than 4) included in the first wavelength range are superimposed;
a processing circuit for generating N images corresponding to the N wavelength bands based on the image signals;
Equipped with
When the N wavelength bands are numbered in ascending or descending order of center wavelength, the processing circuit performs a processing operation based on some or all of the remaining images of the N images, excluding a first image corresponding to a first wavelength band and an Nth image corresponding to an Nth wavelength band.
system. the processing operation being to output some or all of the remaining image;
the processing circuit does not output the first image and the Nth image;
The system of claim 1 . the processing operation is to cause a display device to display some or all of the remaining image;
the processing circuit does not cause the display device to display the first image and the Nth image.
The system of claim 1 . Further comprising a storage device;
the processing operation is to cause the storage device to store some or all of the remaining image;
the processing circuitry does not store the first image and the Nth image in the storage device.
The system of claim 1 . the processing operation is performing an analysis process based on some or all of the remaining image;
the processing circuit does not use the first image and the Nth image in the analysis process.
The system of claim 1 . the processing operation being to output luminance information relating to luminance values of an image based on some or all of the remaining image;
the processing circuit does not output luminance information based on the first image and the Nth image;
The system of claim 1 . the processing operation is generating a color image based on some or all of the remaining images, and the first image and the Nth image are not used in generating the color image;
The system of claim 1 . The imaging device further includes a bandpass filter that transmits the light in the first wavelength range and suppresses transmission of light in a second wavelength range different from the first wavelength range.
A system according to any preceding claim. the imaging device has a plurality of regions having different transmission spectra;
The transmission spectrum of each of the plurality of regions has a plurality of peaks in the first wavelength range.
A system according to any preceding claim. A method executed by a computer in a system including an imaging device that detects light in a first wavelength range and generates an image signal in which components of N wavelength bands (N is an integer equal to or greater than 4) included in the first wavelength range are superimposed, the method comprising:
generating N images corresponding to the N wavelength bands based on the image signals;
performing a processing operation based on some or all of the remaining images, excluding a first image corresponding to a first wavelength band and an Nth image corresponding to an Nth wavelength band, among the N images, when the N wavelength bands are numbered in ascending or descending order of center wavelengths;
Including,
method.

Images