WO2016152900A1 - Image processing device and image capturing device - Google Patents

Abstract

The objective of the invention is to make it possible for analysis employing the spectral reflectivity of a target object to be performed simply. An image processing device 2 is provided with: an image acquiring unit 21 which acquires image data of a target object captured using at least three different wavelength bands; a statistical data acquiring unit 22 which acquires statistical data relating to the spectral reflectivity of a substance of the same type as the target object; an estimating unit 23 which calculates spectral data including the spectral reflectivity of the target object, in more than three wavelength bands, using the image data and the statistical data; and an image generating unit 24 which generates a display image by determining values of a plurality of primary colors which form colors in an image to be displayed, using the spectral reflectivity in at least one wavelength selected from the spectral reflectivities included in the spectral data.

Description

Image processing apparatus and imaging apparatus

The present application relates to an image processing technique that handles the spectral reflectance of an object obtained from image data.

Means for discriminating the state of an object from the spectral reflectance of the object are known. For example, Patent Document 1 below discloses a method for measuring concentrations of melanin pigment, oxidized hemoglobin pigment, and reduced hemoglobin pigment contained in the skin from the absorbance spectrum of the skin. Patent Document 2 below discloses estimating the stratum corneum thickness and keratin concentration based on the measurement of the spectral reflectance of the skin.

Further, Non-Patent Document 2 below describes that the distribution of makeup on a human face is detected from the spectral characteristics obtained by photographing the human face with a camera using a special filter. The camera uses a filter developed to detect the makeup foundation of a human face.

Further, the following Non-Patent Document 3 discloses diagnosing plant diseases using the spectral reflection characteristics of plants. For example, in the diagnosis of anthracnose on cucumber leaves, reflectances of 500 nm, 600 nm, 650 nm, and 700 nm are effective for disease identification.

In order to obtain the target spectral reflectance, for example, a device such as a hyperspectral camera can be used. Alternatively, there is a method of enhancing an optical signal so that the state of an object can be distinguished using a spectral filter and a light source that are made in accordance with a specific wavelength. In this case, an imaging system including a special spectral filter and a light source is constructed according to the analysis target.

A hyperspectral camera is an imaging device that detects a spectrum dispersed into several tens of bands or more for each pixel. Spectral data with high wavelength resolution can be obtained by the hyperspectral camera. This makes it possible to capture the characteristics and information of the target that cannot be captured by the human eye, the existing RGB camera, or the multispectral camera of several band spectroscopy at a time.

JP 2013-198576 A JP 2014-128487 A

Yoichi Miyake, “Introduction to Spectral Image Processing”, first edition, University of Tokyo Press, February 24, 2006, p. 62 Ken Nishino et al., "Optical filter for highlighting spectral features Part I: design and development of the filter for discrimination of human skin with and without an application of cosmetic foundation ,, Optics Express 6020, Optical Soty. No.7, 17 March 2011 Yutaka Sasaki et al., "Basic research on automatic diagnosis of plant diseases", Journal of Agricultural Machinery Society, Vol. 61, No. 2, 1999, p119-126

The hyperspectral camera is complicated in structure and usage and expensive. In addition, it is difficult to understand, interpret and analyze data obtained from a hyperspectral camera unless it is a trained technician. Therefore, the hyperspectral camera tends to be difficult to be used by general users who are not experts.

Also, when using a special spectral filter or light source depending on the application, it is necessary to change the filter or light source depending on the application. Therefore, the cost increases and the work burden on the user also increases.

Therefore, the present application discloses an image processing apparatus and an imaging apparatus that make it possible to easily perform analysis using the spectral reflectance of an object.

An image processing apparatus according to an embodiment of the present invention includes an image acquisition unit that acquires image data of an object photographed in at least three different wavelength bands, and statistical data of spectral reflectance of a substance of the same type as the object. A statistical data acquisition unit to acquire, an estimation unit for calculating spectral data including spectral reflectance of the object in more than three wavelength bands using the image data and the statistical data, and a color of the image to be displayed An image generation unit that generates a display image by determining each value of a plurality of primary colors constituting the image using a spectral reflectance at at least one wavelength selected from spectral reflectances included in the spectral data; Is provided.

According to the present disclosure, analysis using the spectral reflectance of an object can be easily performed.

FIG. 1 is a functional block diagram illustrating a configuration example of the imaging system according to the first embodiment. FIG. 2 is a flowchart illustrating an example of operations of the estimation unit and the image generation unit. FIG. 3 is a diagram illustrating an example of a result of principal component analysis of spectral reflectance of cheeks of a plurality of people. FIG. 4 is a graph showing an example of the spectral transmittance of the RGB color filter. FIG. 5 is a graph showing an example of the spectral emissivity of the standard light source A. FIG. 6 is a graph illustrating an example of spectral sensitivity of the image sensor. FIG. 7 is a graph showing the spectral reflectance of human skin. FIG. 8 is a graph showing calculated values and measured values of spectral reflectance. FIG. 9 is a graph showing the results of principal component analysis performed on the spectral reflectances of a plurality of samples. FIG. 10 is a graph showing the spectral reflectance of a person's bare skin. FIG. 11 is a graph showing the spectral reflectance of the makeup skin of the same person as in FIG. FIG. 12 is a graph showing calculated values and measured values of the spectral reflectance of bare skin. FIG. 13 is a graph showing calculated values and measured values of the spectral reflectance of the makeup skin. FIG. 14 is a functional block diagram illustrating a configuration example of the imaging system 1 according to the third embodiment. It is a graph which shows the absolute value of the difference of a ((lambda)) of makeup skin, and a ((lambda)) of bare skin. It is a graph which shows the absolute value of the difference of a ((lambda)) of makeup skin, and a ((lambda)) of bare skin. It is a graph which shows the absolute value of the difference of a ((lambda)) of makeup skin, and a ((lambda)) of bare skin. FIG. 18 is a diagram illustrating an example of a display image. FIG. 19 is a graph showing the results of principal component analysis performed on the spectral reflectances of a plurality of samples. FIG. 20 is a graph showing the spectral reflectance of cucumber leaves. FIG. 21 is a graph showing calculated values and measured values of the spectral reflectance of cucumber leaves. FIG. 22 is a functional block diagram illustrating a configuration example of the imaging system according to the fifth embodiment. FIG. 23 is a functional block diagram illustrating a configuration example of an imaging system according to the sixth embodiment. FIG. 24 is a diagram illustrating an example of the LUT referred to by the image generation unit.

An image processing apparatus according to an embodiment of the present invention includes an image acquisition unit that acquires image data of an object photographed in at least three different wavelength bands, and statistical data of spectral reflectance of a substance of the same type as the object. A statistical data acquisition unit to acquire, an estimation unit for calculating spectral data including spectral reflectance of the object in more than three wavelength bands using the image data and the statistical data, and a color of the image to be displayed An image generation unit that generates a display image by determining each value of a plurality of primary colors constituting the image using a spectral reflectance at at least one wavelength selected from spectral reflectances included in the spectral data; Is provided.

In the above configuration, the estimation unit uses the image data of the object photographed in three or more different wavelength bands, and the statistical data of the spectral reflectance of the same kind of substance as the object, and the spectral of the object. Spectral data including reflectance is calculated. The estimation unit can generate spectral reflectance information having a larger number of bands than the number of bands (number of wavelength bands) in the spectral reflectance information included in the image data by using statistical data. Further, the image generation unit determines each value of the primary color to be displayed using the spectral reflectance of at least some of the spectral reflectances included in the spectral data when generating the display image. Thereby, an image reflecting the spectral characteristics of the spectral reflectance of the object can be generated. That is, a display image indicating the characteristics of the spectral characteristics of the object is generated based on the spectral characteristics of the spectral reflectance in a wavelength band greater than three. Therefore, analysis using the spectral reflectance of the object is simplified.

The object can be human skin. In this case, the statistical data acquisition unit can acquire statistical data including the spectral reflectance of the human skin and the spectral reflectance of the skin of the person who has been makeup. The image generation unit uses a plurality of primary colors constituting the color of the image to be displayed, using the spectral reflectance at a wavelength selected based on a difference in spectral reflectance between human bare skin and makeup human skin. Each value of can be determined. As a result, it is possible to easily obtain an image that makes it easy to discriminate between bare skin and makeup skin.

The image processing device calculates a value indicating a variation degree of a change rate of the spectral reflectance due to a wavelength change from the spectrum data, and the skin indicated by the image data is bare skin or makeup according to the change rate. The makeup | decoration judgment part which judges can be further provided. Thereby, it can be judged more accurately whether it is makeup skin or bare skin.

The object can be human skin. In this case, the statistical data acquisition unit can acquire statistical data including the spectral reflectance of healthy skin and the spectral reflectance of skin affected by a skin disease. The image generation unit uses the spectral reflectance at a wavelength selected based on the difference in spectral reflectance between healthy skin and skin affected by a skin disease, and uses each of the primary colors constituting the color of the image to be displayed. The value can be determined. Thereby, an image that can easily distinguish between healthy skin and skin affected by a skin disease can be easily obtained.

The object can be a plant leaf or fruit. In this case, the statistical data acquisition unit can acquire statistical data including a spectral reflectance of a healthy plant leaf or fruit and a spectral reflectance of a diseased plant leaf or fruit. The image generation unit uses the spectral reflectance selected based on the difference in spectral reflectance between a healthy plant leaf or fruit and a diseased plant leaf or fruit to change the color of the image to be displayed. Each value of the primary color to constitute can be determined. Thereby, it is possible to easily obtain an image that makes it easy to distinguish between a leaf or fruit of a healthy plant and a leaf or fruit of a diseased plant.

The image processing apparatus is configured to select the statistical data acquired by the statistical data acquisition unit and a spectral reflectance wavelength used by the image generation unit to determine each value of the primary color based on a user input. A part can be further provided.

Thereby, the user can set to execute the spectral reflectance estimation processing and the image generation processing suitable for the object. Therefore, spectral reflectance estimation and image generation using statistical data suitable for the object can be performed. For example, the user can make settings for obtaining an image suitable for an object by inputting an instruction to the switching unit without reconfiguring an imaging system including a filter, a light source, and the like according to the object. .

An imaging system according to an embodiment of the present invention includes a filter, an imaging unit that obtains image data by converting light transmitted through the filter into an electrical signal, and an image processing unit that processes the image data. The image processing unit includes an image acquisition unit that acquires image data of an object photographed in at least three different wavelength bands, and a statistical data acquisition unit that acquires statistical data of spectral reflectance of a substance of the same type as the object. , Using the image data and the statistical data, an estimation unit that calculates spectral data of spectral reflectance of the object, and each value of a plurality of primary colors constituting the color of the image to be displayed, An image generation unit that generates a display image by determining using the spectral reflectance at at least one of the wavelengths.

A program according to an embodiment of the present invention acquires an image acquisition process for acquiring image data of an object photographed in at least three different wavelength bands, and acquires statistical data of spectral reflectance of a substance of the same type as the object. Statistical data acquisition processing, estimation processing for calculating spectral data of spectral reflectance of the object using the image data and the statistical data, and each value of a plurality of primary colors constituting the color of the image to be displayed The computer performs image generation processing for generating a display image by determining using spectral reflectance at at least one wavelength of the spectral data.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In the drawings, the same or corresponding parts are denoted by the same reference numerals and description thereof will not be repeated. In addition, in order to make the explanation easy to understand, in the drawings referred to below, the configuration is shown in a simplified or schematic manner, or some components are omitted. Further, the dimensional ratio between the constituent members shown in each drawing does not necessarily indicate an actual dimensional ratio.

<Embodiment 1>
FIG. 1 is a functional block diagram illustrating a configuration example of an imaging system 1 according to the first embodiment. The imaging system 1 includes a camera 3, an image processing device (image processing unit) 2, and a display 4.

The camera 3 includes a lens system 31, a filter group 32, and an image sensor 33. The camera 3 receives the light emitted from the light source 5 and reflected by the object. Light from the object passes through the lens system 31 and the filter group 32 and is detected by the image sensor 33.

The filter group 32 has spectral transmission characteristics corresponding to three different wavelength bands of R (red), G (green), and B (blue). For example, the filter group 32 may have a configuration in which a color filter having one of R, G, and B spectral transmission characteristics is arranged for each pixel.

The imaging sensor 33 has a plurality of solid-state imaging devices (for example, CMOS, CCD, etc.). One solid-state image sensor corresponds to one pixel. Each solid-state imaging device generates and outputs a signal (pixel signal) having a charge amount corresponding to the amount of light received by itself. Pixel signals (analog signals) output from the respective solid-state imaging devices are A / D converted and converted into image signals as digital signals. The image signal obtained by this conversion is image data captured in three wavelength bands (RGB). That is, the camera 3 can obtain 3-band image data.

Note that the number of bands of the camera 3 is not limited to the three RGB bands in the above example. A multiband camera (multispectral camera) that can capture images of four or more bands can also be used for the camera 3. For example, the camera 3 can include a filter having transmission characteristics of four or more different wavelength bands.

Further, the filters included in the filter group 32 of the camera 3 are not limited to the color filters. For example, filters such as an optical low-pass filter and an iR (infrared light) cut filter may be included in the filter group 32 as necessary.

The image processing apparatus 2 includes an image acquisition unit 21, a statistical data acquisition unit 22, an estimation unit 23, and an image generation unit 24. The image processing apparatus 2 inputs image data and statistical data of an object photographed by the camera 3, processes these data, and outputs a display image in which the spectral reflectance of the object is reflected. In the example illustrated in FIG. 1, the image processing apparatus 2 displays a display image on the display 4.

The image acquisition unit 21 acquires image data of an object captured by the camera 3 in three different wavelength bands (RGB). The image acquisition unit 21 receives image data from the camera 3 and stores the image data in a memory (not shown) accessible from the estimation unit 23. Moreover, the image acquisition part 21 can also output a control signal to the camera 3, and can image | photograph the image of a target object. For example, the image acquisition unit 21 can transmit a shooting operation instruction and a request for captured image data to the camera 3 in response to a user input.

The image acquisition unit 21 may extract the image portion of the target object from the images taken by the camera 3. The image acquisition unit 21 can extract, for example, an image portion of an object by image recognition using pattern matching or according to a selection operation from a user. The extracted image portion can be made available to the estimation unit 23 as image data.

The statistical data acquisition unit 22 acquires the spectral reflectance statistical data of the same kind of substance as the target object. The statistical data acquisition unit 22 reads the statistical data from the recording medium 6 on which the statistical data is recorded, and stores it in a memory (not shown) accessible from the estimation unit 23. The statistical data is recorded in advance on the recording medium 6, for example. The recording medium 6 only needs to be accessible from the image processing apparatus 2. The recording medium 6 may be, for example, a storage device built in the imaging system 1 or an external storage device connected to the imaging system 1.

Statistic data is data including statistical information related to the spectral reflectance of the object. The statistical data may include spectral reflectance measurements of multiple samples of the same material as the object. For example, when the target object is human skin, the spectral reflectance values of a plurality of human skins can be used as statistical data. The spectral reflectance value of the statistical data includes, for example, the reflectance values in more than 3 bands (wavelength bands).

Statistic data can be recorded by dividing the values of multiple sample groups of the same substance into two or more groups. For example, the spectral reflectance values of multiple people's skin may be recorded as statistical data in a state classified into the spectral reflectance of healthy human skin and the skin reflectance of people suffering from skin diseases. it can.

In addition to the spectral reflectance value, statistical data such as an average, variance, standard deviation, or intermediate value obtained from the spectral reflectance of a plurality of samples can be included in the statistical data. For example, the statistical data acquisition unit 22 may acquire principal component vectors (specific examples will be described later) obtained by performing principal component analysis on the spectral reflectances of a plurality of samples as statistical data.

The statistical data acquisition unit 22 can select and acquire necessary data from a plurality of classified groups included in the statistical data. The statistical data acquisition unit 22 can select appropriate statistical data based on, for example, a selection operation from the user or according to a preset value.

The estimation unit 23 uses the image data acquired by the image acquisition unit 21 and the statistical data acquired by the statistical data acquisition unit 22 to calculate spectral data of the spectral reflectance of the object. The estimation unit 23 uses the above statistical data to calculate spectral reflectances in a larger number of wavelength bands than the number of spectral reflectance wavelength bands (bands) indicated in the image data. That is, the estimation unit 23 uses the statistical data to estimate the spectral reflectance of the object in the wavelength band that cannot be obtained from the image data.

The estimation unit 23 can execute principal component analysis or winner estimation processing based on, for example, statistical data. Thereby, it is possible to calculate an estimated value of the spectral reflectance of the object in a wavelength band larger than the number of wavelength bands of the measurement values of the object detected by the imaging sensor 33 (three in this example, RGB). Details of the processing of the estimation unit 23 will be described later.

The estimation unit 23 can use imaging system data when calculating spectrum data. The imaging system data is data indicating the spectral characteristics of the imaging system used for shooting the object. For example, the spectral emissivity of the light source 5, the spectral transmittance of the lens system 31, the spectral reflectance of the filter group 32, the spectral sensitivity of the imaging sensor 33, and the like are included in the imaging system data.

The output value of each wavelength band (RGB) detected by the image sensor 33 includes, for example, the spectral emissivity of the light source 5, the spectral transmittance of the lens system 31, and the filter group 32 in addition to the spectral reflectance of the object. Depending on the spectral characteristics of the imaging system such as the spectral reflectance of the imaging sensor 33 and the spectral sensitivity of the imaging sensor 33. Therefore, the estimation unit 23 can obtain a value obtained by subtracting the spectral characteristic of the imaging system from the output value using the imaging system data, that is, a measured value of the spectral reflectance of the object.

For example, the estimation unit 23 has a high wavelength resolution obtained by photographing a target object with a hyperspectral camera from signals from several bands (three in this example, RGB), statistical data, and imaging system data. Spectral data equivalent to spectral data can be reproduced.

The image generation unit 24 calculates each value of a plurality of primary colors (for example, RGB) constituting the color of each pixel of the image to be displayed at at least one wavelength of the spectrum data of the corresponding pixel calculated by the estimation unit 23. Determined using spectral reflectance. Thereby, a display image is generated. Which spectral reflectance of spectral data to use in this image generation can be set in advance according to the object and the purpose of the analysis. That is, wavelength data representing the wavelength to be used can be recorded in advance on the recording medium 6 accessible by the image processing apparatus 2.

For example, when the object is human skin and the health condition of the human skin is analyzed, the difference in spectral reflectance between healthy skin and unhealthy skin is larger than other wavelengths (for example, λ1, λ2, ..., the color of the display image can be determined using the spectral reflectance value at λn). In this case, λ1, λ2,... Λn are recorded on the recording medium 6 as wavelength data. This wavelength may be a band having a certain width.

A pre-set function can be used for the process of determining each RGB value of the display image from the spectral reflectance value calculated by the estimation unit 23. For example, as an example of a function for calculating the value of R, there is a function that inputs spectral reflectance values in each wavelength band and outputs a value obtained by multiplying these by a coefficient as an R value. Alternatively, the image generation unit 24 can determine the RGB value corresponding to the spectral reflectance value calculated by the estimation unit 23 with reference to data indicating the correspondence between the spectral reflectance and each RGB value.

The image generation unit 24 displays the generated image on the display 4. Note that the form of image output is not limited to this. For example, the image generation unit 24 may transmit the image to a requested location via a network.

(Operation example)
Next, an operation example of the estimation unit 23 will be described. The above imaging system data can be, for example, data represented by vectors shown below.

The spectral reflectance of the object and the output of the camera can be expressed as follows.

In the above equation (8), the spectral reflectance o of the object can be obtained from the camera output v and the spectral product Fi of the imaging system. That is, the spectral reflectance o of the object can be obtained from the inverse matrix of F and the camera output v. However, when spectral data taking 31 points at an interval of 10 nm in the range of visible light 400 nm to 700 nm is considered, the spectral reflectance o of the object has 31-dimensional data. The output of the camera is 3 channels in the case of an RGB imaging device, and up to about 8 to 9 channels even in the case of a multispectral camera. Therefore, the inverse matrix equation cannot be solved.

If there are a large number of two-dimensional spectral data of the wavelength and reflectance of the object, it is possible to estimate the spectral reflectance of the object with higher accuracy using principal component analysis or Wiener estimation. When using principal component analysis, the estimation unit 3 performs estimation processing using eigenvectors (principal component vectors) obtained by principal component analysis of a plurality of spectral reflectances of the same substance as the object. When using the Wiener estimation, the estimation unit 3 uses a plurality of spectral reflectance values of the same substance as the object so that the set average of the square error between the true spectral reflectance and the estimated spectral reflectance is minimized. Next, a matrix used for estimation is calculated.

In addition, about the calculation using these principal component analysis and Wiener estimation, it is described in the said nonpatent literature. For example, 4.3.1, 4.3.2, 4.4.1 and 4.4.2 of Chapter 4 of the above Non-Patent Document 1 are incorporated into this specification by reference for the principal component analysis and the calculation method of the winner estimation. by reference).

In order to estimate the spectral reflectance, pre-recorded imaging system data is used. When the spectral characteristics of the imaging system are unknown, the spectral reflectance can be estimated by shooting a color chart, paint, etc. using the imaging system used for shooting. For this estimation, for example, winner estimation can be used.

(Example of processing using principal component analysis)
Next, a specific processing example when the estimation unit 23 calculates the spectral data of the spectral reflectance of the target object using principal component analysis will be described. FIG. 2 is a flowchart illustrating an example of operations of the estimation unit 23 and the image generation unit 24.

In the example shown in FIG. 2, the estimation unit 23 inputs statistical data (S1). Here, as an example, a case where a data file of a human skin category in SOCS (StandardＳＯobject (colour spectra database for colour reproduction evaluation) will be described as an example. SOCS is a spectral reflectance database and is also registered in ISO. One of the SOCS data files contains data on the spectral reflectance of the cheeks of a woman's skin (31 points of reflectance at 400 nm, 410 nm,..., 690 nm, and 700 nm) for 123 people ( Number of samples = 123).

The estimation unit 23 calculates principal component vectors u1, u2, u3 of spectral reflectances of a plurality of samples included in the statistical data. The principal component vector is obtained by obtaining the eigenvector of the covariance matrix of data from the Lagrange multiplier method.

For example, when the statistical data is the data file described above, the estimation unit 23 obtains a covariance matrix Rr of spectral reflectances of the cheeks of 123 female skins. The estimation unit 23 obtains the eigenvector vi and the eigenvalue li (i = 1, 2,..., 31) of the covariance matrix Rr. Of the eigenvectors V, the three vectors in descending order of the eigenvalue l can be the principal component vectors u1, u2, and u3.

FIG. 3 is a diagram illustrating an example of a result of a principal component analysis of spectral reflectances of a plurality of human cheeks included in the data file. The first principal component, the second principal component, and the third principal component are three eigenvectors in the descending order of eigenvalues among the obtained eigenvectors.

The estimation unit 23 inputs imaging system data (S3). The imaging system data includes, for example, the spectral reflectance of the light source 5, the spectral reflectance of the filter group 32, the spectral transmittance of the lens system 31, and the spectral sensitivity of the imaging sensor 33 in the shooting environment of the object by the camera 3. Any of these imaging system data can be data indicating changes in spectral characteristics depending on the wavelength. For example, the imaging system data can be a value indicating spectral characteristics at each wavelength between 400 nm and 700 nm.

Hereinafter, an example of imaging system data when a filter that transmits light having a wavelength in the range of 400 nm to 700 nm and cuts other wavelengths is provided on the lens side of the imaging sensor will be described.

FIG. 4 is a graph showing an example of the spectral transmittance of the RGB color filters included in the filter group 32. The spectral transmittance data of the filter as shown in this graph is included in the imaging system data. The spectral transmittance data of the filter can be represented by, for example, a matrix f as shown in the above equation (1).

FIG. 5 is a graph showing an example of the spectral emissivity of the standard light source A (tungsten bulb) used as the light source 5. The spectral emissivity data of the light source as shown in this graph is included in the imaging system data. The spectral emissivity data of the light source can be represented by, for example, a matrix E as shown in the above equation (2).

FIG. 6 is a graph showing an example of spectral sensitivity of the image sensor 33. Data of spectral sensitivity of the imaging sensor as shown in this graph is included in the imaging system data. The example shown in FIG. 6 is an example of the spectral sensitivity of a CCD image sensor in which a cover glass is stacked. The spectral sensitivity data of the image sensor can be represented by, for example, a matrix S as shown in the above equation (4).

The estimation unit 23 can obtain an imaging system matrix F = ^ft ES from the spectral transmittance data matrix f of the filter, the spectral emissivity data matrix E of the light source, and the spectral sensitivity data matrix S of the imaging sensor. The imaging system matrix is data representing the spectral characteristics of an imaging system that captures an object. In addition, the example of imaging system data is not restricted to the said example. For example, a matrix L representing the spectral transmittance of the lens system can be added to the imaging system matrix F. In S <b> 3, the estimation unit 3 can read the imaging system matrix F recorded in advance.

The estimation unit 23 calculates the average spectral reflectance m of the spectral reflectance of each wavelength of the plurality of samples included in the statistical data (S4 in FIG. 2). The estimation unit 23 calculates the GRB value when an object having an average spectral reflectance m is captured by the imaging system using the imaging system matrix F and the average spectral reflectance m (S5). As an example, the detection value [Gm Rm Bm] of the imaging sensor 33 of the camera 3 when photographing the object having the average spectral reflectance m of 123 people in the data file is calculated as follows.

The estimation unit 23 inputs image data (S6). The estimation unit 23 executes the following processes S7 to S9 for all the pixels in the image data. In S7, the estimation part 23 acquires the measured value (G1 R1 B1) of the pixel of adjacent GRB among image data. For example, when sample 1 No. 1 having the spectral reflectance shown in FIG. 7 is photographed, the output value of the camera 3, that is, the measured value (G1, R1, B1) is the following value.

In S8, this measured value (G1 R1 B1), the principal component vector obtained in S2, the imaging system data obtained in S3, the average spectral reflectance obtained in S4, and the detected value obtained in S5 (Gm Rm Bm) Is used to calculate the spectral reflectance o of the object. Here, the spectral reflectance o is spectral data including a spectral reflectance value of an object at a wavelength of 4 or more. For example, the spectral reflectance o can be expressed as 31-dimensional vector data.

Using the principal component vectors u1, u2, u3, the spectral reflectance o can be estimated by the following equation (9).

In this example, n = 3 in the above equation (9). Therefore, the above equation (9) becomes the following equation (10).

α ₁ , α ₂ , and α ₃ are coefficients, and their values are obtained from the following equation (11).

これは以下の式（１２）を変形して求めたものである。

This is obtained by modifying the following equation (12).

The estimation unit 23 includes the principal component vectors u1, u2, u3 obtained in S2, the imaging system matrix F obtained in S3, the average spectral reflectance m obtained in S4, the detection value obtained in S5 (Gm Rm Bm), S7 Substituting the measured value (G1 R1 B1) obtained in step (11) into the above equation (11), the coefficients α1, α2, and α3 can be calculated. The estimation unit 23 can calculate the spectral reflectance o by substituting these coefficients and the principal component vectors u1, u2, and u3 into the above equation (10).

FIG. 8 is a graph showing a comparison between the spectral reflectance o obtained by the above calculation and the measured value of spectral data obtained by taking points at intervals of 10 nm in the visible light range of 400 nm to 700 nm with the spectroscopic measurement device. . The calculated value and the measured value are in good agreement. In other words, if there are statistical data and output values of three wavelength bands (RGB) obtained by photographing a certain skin portion and its imaging system matrix F, 31-dimensional as if the skin portion was measured by a spectroscopic measurement device. Data equivalent to spectrum data can be estimated. In this example, an RGB camera having three bands is used. However, it is possible to further improve estimation accuracy of spectrum data by using a multispectral camera having four or more bands, that is, by increasing the number of bands. it can.

Referring to FIG. 2 again, in S9, the image generation unit 24 calculates the RGB value of the pixel of the display image corresponding to the RGB pixel read in S7 (S9). The image generation unit 24 calculates the RGB value of the display image using the spectral reflectance at one or more wavelengths among the spectral reflectances o obtained in S8. The image generation unit 24 can determine which wavelength of the spectral reflectance o to use according to the wavelength data recorded in advance.

As an example, the image generation unit 24 inputs a value of the estimated spectral reflectance at one or a plurality of wavelengths selected from a band of 400 nm to 700 nm, and uses a function fR that outputs an R value as an R The value can be determined. Similarly, an estimated spectral reflectance value at one or a plurality of wavelengths selected in a band of 400 nm to 700 nm is input, and a value calculated by a function fB having an output of B value is defined as B value. The value calculated by the function fG of the estimated spectral reflectance value at one or a plurality of wavelengths selected in the band of 400 nm to 700 nm is defined as a G value.

In addition, the image generation unit 24 can apply respective coefficients to the R, G, and B values according to a preset white balance or an automatically adjusted white balance. The image generation unit 24 outputs a display image including the R, G, and B values of each pixel calculated in S9 to the display 4 (S10).

Thus, a value based on the spectral reflectance selected from the spectral reflectance calculated by the estimation unit 3 for one pixel can be assigned as the output value in the corresponding pixel of the display image. The skin state can be visualized by displaying the synthesized image on the display.

In addition, if the spectral data of the reflectance of the skin is known by the processing of the estimation unit 23 described above, the melanin pigment and the oxygenated hemoglobin pigment of the bare skin can be obtained without a device that is spectrally separated into several tens of bands such as a hyperspectral camera. The skin condition can be analyzed by estimating the content of reduced hemoglobin pigment, the thickness of the stratum corneum of the skin, and the keratin concentration.

For example, the image generation unit 24 selects a wavelength suitable for detection of a substance to be analyzed in the skin within a band of 400 nm to 700 nm, and uses RGB spectral values in the display image using the spectral reflectance value of the selected wavelength. The value can be determined. For example, when generating an image for viewing the melanin pigment on the bare skin, the image generator 24 can select a wavelength at which the reflectance of the melanin pigment is greater than the others. Similarly, when generating an image for viewing the content of oxyhemoglobin pigment, reduced hemoglobin pigment, the thickness of the stratum corneum of the skin, and the keratin concentration, the wavelength at which the reflectance of each substance increases can be selected. Thus, visual analysis is facilitated by generating a display image using the spectral reflectance.

<Embodiment 2>
The second embodiment is an example in which the image processing apparatus 2 according to the first embodiment is applied to an apparatus that enables discrimination between human skin and makeup skin. In the second embodiment, the object is human skin. The statistical data acquisition unit 22 acquires statistical data including the spectral reflectance of the human skin and the spectral reflectance of the skin of the person who has been makeup.

As an example, a case where a data file of human skin (bare skin, forehead) included in the SOCS database is acquired as statistical data will be described. Specifically, a case where a human skin data file (sample number 123) in a human skin category and a human makeup skin data file (sample number 123) are used will be described.

The data file of human skin contains data for 123 people's spectral reflectance of human skin (31 reflectances for each wavelength of 400 nm, 410 nm,..., 690 nm, 700 nm). The data file of human makeup skin contains data on the spectral reflectance (similarly 31 points) of human makeup skin for 123 people.

FIG. 9 is a graph showing the result of principal component analysis performed on the spectral reflectances of a plurality of samples included in the two data files. The first principal component, the second principal component, and the third principal component are three eigenvectors in descending order of eigenvalues among the eigenvectors obtained by the principal component analysis.

FIG. 10 is a graph showing the spectral reflectance of a person's bare skin (sample No b1). FIG. 11 is a graph showing the spectral reflectance of the same person's makeup skin (sample1No f1). When these bare skin b1 and makeup skin f1 are measured by the same imaging system as in the first embodiment, the measured values (GRB) in the imaging sensor of the camera 3 are as follows.

Here, the imaging system includes a color filter having a spectral transmittance shown in FIG. 4, a light source having a spectral emissivity shown in FIG. 5, and an imaging sensor having a spectral sensitivity shown in FIG. In the imaging system, a filter is provided on the lens side of the imaging sensor so as to transmit light in a wavelength range of 400 nm to 700 nm and cut other wavelengths.

The estimation unit 23 uses the measured values, the principal component vectors u1, u2, u3 derived by principal component analysis for the two data files, and statistical data values such as the average value m of the spectral reflectance. Spectral data is calculated.

FIG. 12 shows the spectral reflectance indicated by the spectrum data of the bare skin b1 calculated by the estimation unit 23 and the measured value of the spectrum data obtained by taking a point at an interval of 10 nm in the range of visible light from 400 nm to 700 nm with the spectroscopic measurement device. It is a graph shown to be comparable. FIG. 13 is a similar graph for the makeup skin f1. In both cases, the calculated value and the measured value are in good agreement. In this way, even from RGB three-dimensional data, it is possible to estimate data equivalent to 31-dimensional spectrum data.

The image generation unit 24 determines each value of a plurality of primary colors constituting the color of the display image by using the spectral reflectance at one or more selected wavelengths among the spectral reflectances calculated by the estimation unit 23. To do. Said 1 or several wavelength can be made into the wavelength selected based on the difference of the spectral reflectance of a person's bare skin and the skin of the person who made up, for example.

For example, the image generation unit 24 assigns values corresponding to reflectances of 410 nm to 460 nm, 500 nm to 550 nm, and 560 nm to 610 nm, for example, to the B value, G value, and R value of the corresponding pixel of the display image, respectively. An image can be generated. By displaying the generated image on a display, the application state of makeup can be visualized.

If the spectral data of the reflectance of the skin is known in this way, the makeup state can be analyzed without using a device such as a hyperspectral camera or a special spectral filter.

<Embodiment 3>
The third embodiment is an example in which the image processing apparatus 2 according to the first embodiment is applied to an apparatus that enables discrimination between human skin and makeup skin. In the third embodiment, the object is human skin. FIG. 14 is a functional block diagram illustrating a configuration example of the imaging system 1 including the image processing device 2 according to the third embodiment.

In the image processing apparatus 2a shown in FIG. 3, the statistical data acquisition unit 22 acquires statistical data including the spectral reflectance of the human skin and the spectral reflectance of the makeup of the human skin. The image acquisition unit 21 acquires image data in which human skin is reflected as an object. The estimation unit 23 calculates spectral data of spectral reflectance of human skin using image data, statistical data, and imaging system data. The number of bands of spectrum data is greater than the number of bands of image data.

The image processing apparatus 2 further includes a makeup determination unit 25 that determines whether the skin indicated by the image data is bare skin or makeup skin. The makeup determination unit 25 calculates a value indicating the degree of change in the change rate of the spectral reflectance due to the change in wavelength from the spectrum data calculated by the estimation unit 23. The makeup determination unit 25 can determine whether the skin shown in the image data is bare skin or makeup skin using this value.

】 When the spectrum of the same person's bare skin and makeup skin is compared, the way the reflectance changes is different. For example, in the example shown in FIGS. 10 and 11, the reflectance curve of the makeup skin (FIG. 11) is more uneven than the curvature curve of the bare skin (FIG. 10), particularly in the wavelength band around 500 nm to 600 nm. Becomes smaller.

Therefore, the makeup determination unit 25 calculates a value indicating the degree of change in the rate of change of the spectral reflectance according to the wavelength in a predetermined wavelength band as an index value, and determines whether the skin is makeup skin or bare skin using this value. Can do. The makeup determination unit 25, for example, the rate of change from the reflectance o (λ) at a certain wavelength λ to the reflectance o (λ + Δλ) at the wavelength λ + Δλ, and the reflection from the reflectance o (λ) at the wavelength λ to the wavelength λ−Δλ. A value indicating a difference from the rate of change to the rate o (λ−Δλ) can be calculated as an index value.

As an example, the makeup determination unit 25 can calculate the value of a (λ) represented by the following formula (13) as an index value. Here, a (λ) is referred to as a tangential slope change rate.

15, FIG. 16 and FIG. 17 are graphs showing the results of calculating the absolute value of the difference between a (λ) of the makeup skin and a (λ) of the bare skin for a plurality of samples. 15, 16, and 17, the vertical axis indicates the cumulative probability distribution, and the horizontal axis indicates the absolute value of the tangent slope change rate a of the bare skin spectral reflectance and the change rate of the tangential slope of the cosmetic skin spectral reflectance. The absolute value difference of a is shown.

15, FIG. 16 and FIG. 17 show the spectrum of the same 123 makeup skin from the absolute value of the tangential slope change rate of the spectral reflectance spectrum of 123 people with Δλ = 10 nm in the above equation (13). The probability distribution of the value which subtracted the absolute value of the inclination change rate of the tangent of a reflectance spectrum is shown. The results for each wavelength of 10 nm from 470 nm to 640 nm are shown.

From these results, it can be seen that the difference between the absolute value of bare skin a and the absolute value of makeup skin a is highly likely to take a positive value in the wavelength band of about 500 nm to 600 nm. Further, in the 510 nm to 610 nm band, the difference in absolute value is larger than the other bands. In the 560 nm to 600 nm band, particularly in the wavelength of 580 nm, the difference in absolute value is even greater than in the other bands. Therefore, the makeup determination unit 25 can calculate, as an index value, a value indicating the degree of change in the rate of change of the spectral reflectance in these wavelength bands or wavelengths as an index value.

When the spectral data of the spectral reflectance of the skin of the analysis subject is recorded in advance on the recording medium 6, the makeup determination unit 25 captures the skin data of the same analysis subject as the pre-recorded spectrum data of the skin. With the spectrum data obtained from the data, it is possible to compare an index value indicating the degree of change in the reflectance change rate depending on the wavelength. Based on the comparison result, it can be determined whether it is makeup skin or bare skin. For example, when the tangential slope change rate is used as the index value, if the tangential slope change rate of the spectrum data obtained from the image data is smaller than that of the bare skin, the skin is makeup skin (skin with makeup remaining). It can be judged.

When there is no spectral data of the spectral reflectance of the analysis subject's bare skin, the makeup determination unit 25 determines whether the skin is makeup skin or bare skin based on an index value such as a tangential slope change rate of the spectrum data obtained from the image data. be able to. For example, the makeup determination unit 25 determines that a part of a pixel whose tangential slope change rate is smaller than surrounding pixels is a makeup remaining (a part where makeup remains), assuming that most of the skin is bare skin. Is also possible. In this way, for example, by estimating a reflection spectrum from each RGB value for each pixel, it is possible to determine whether the pixel is bare skin or a makeup state.

The determination result by the makeup determination unit 25 can be used, for example, when the image generation unit 24 generates a display image. Alternatively, the determination result by the makeup determination unit 25 can be output independently of the display image output by the image generation unit 24.

For example, the image generation unit 24 uses the spectral reflectance of wavelengths (for example, 520 nm, 580 nm, and 600 nm) at which the difference between the index values becomes large between the bare skin and the makeup skin in the spectrum data, and the RGB value of the display pixel. Can be determined. Alternatively, the RGB value can be determined by using the tangential slope change rate a (λ). As an example, the image generation unit 24 can display an RGB image synthesized by inputting R = a (520 nm), G = a (580 nm), and B = a (600 nm). In this example, the value of the primary color constituting the color displayed in the display image is calculated using a value representing the degree of change in the change rate of the spectral reflectance due to the change in wavelength. Thereby, the application state of the makeup part can be visualized. FIG. 18 is a diagram illustrating an example of a result of combining and displaying as an image by the above method.

<Embodiment 4>
The fourth embodiment is an example in which the image processing apparatus 2 according to the first embodiment is applied to an apparatus for analyzing the health condition of a plant. In Embodiment 4, the object is a plant leaf or fruit. The statistical data acquisition unit 22 acquires statistical data including the spectral reflectance of a healthy plant leaf or fruit and the spectral reflectance of a diseased plant leaf or fruit.

As an example, a case where a plant leaf data file included in the SOCS database is acquired as statistical data will be described. Specifically, the case of using the leaf data file of plants in the flower and leaf category will be described. This data file includes data of a plurality of samples of plant leaf spectral reflectances (31 reflectances at wavelengths of 400 nm, 410 nm,..., 690 nm, and 700 nm).

FIG. 19 is a graph showing the result of principal component analysis performed on the spectral reflectances of a plurality of samples included in the data file. The first principal component, the second principal component, and the third principal component are three eigenvectors in descending order of eigenvalues among the eigenvectors obtained by the principal component analysis.

FIG. 20 is a graph showing the spectral reflectance of cucumber leaves (sample No. 1). When this cucumber leaf is measured by the same imaging system as in the first embodiment, the measured value (GRB) in the imaging sensor of the camera 3 is as follows.

Here, the imaging system includes a color filter having a spectral transmittance shown in FIG. 4, a light source having a spectral emissivity shown in FIG. 5, and an imaging sensor having a spectral sensitivity shown in FIG. In the imaging system, a filter is provided on the lens side of the imaging sensor so as to transmit light in a wavelength range of 400 nm to 700 nm and cut other wavelengths.

The estimation unit 23 calculates statistical data values such as the above measured values, principal component vectors u1, u2, and u3 derived from the plant leaf data file using principal component analysis, and the average value m of the spectral reflectance. Spectral data is calculated from

FIG. 21 shows the spectral reflectance shown by the spectrum data of the cucumber leaf calculated by the estimation unit 23 and the spectrum data of the cucumber leaf taken at 10 nm intervals in the visible light range of 400 nm to 700 nm with the spectroscopic measurement device. It is a graph which shows that the measured value of can be compared. The calculated value and the measured value are in good agreement. In this way, even from RGB three-dimensional data, it is possible to estimate data equivalent to 31-dimensional spectrum data.

The image generation unit 24 determines each value of a plurality of primary colors constituting the color of the display image by using the spectral reflectance at one or more selected wavelengths among the spectral reflectances calculated by the estimation unit 23. To do. The one or more wavelengths may be selected based on, for example, the difference in spectral reflectance between healthy plant leaves or fruits and diseased plant leaves or fruits. For example, it is possible to select wavelengths of 500 nm, 600 nm, and 650 nm at which the difference in reflectance between the diseased part and the healthy part is greater than other wavelengths.

The image generation unit 24 can assign the spectral reflectance o (λ) calculated by the estimation unit 23 to each of the R value, the G value, and the B value in the display image, for example. As an example, when wavelengths of 500 nm, 600 nm, and 650 nm are selected, the image generation unit 24 can set R = o (500 nm), G = o (600 nm), and B = a (650 nm). In this way, by displaying the generated image on the display, it is possible to create an image that emphasizes the difference between the diseased part and the healthy part of the leaves or fruits of the plant.

Thus, if spectral data of the spectral reflectance of plant leaves is known, plant leaves or fruits can be obtained without using a device or a special spectral filter that is spectrally separated into several tens of bands such as a hyperspectral camera. Can analyze health status.

<Embodiment 5>
FIG. 22 is a functional block diagram illustrating a configuration example of the imaging system 1 including the image processing device 2b according to the fifth embodiment. An image processing apparatus 2b shown in FIG. 22 has a configuration in which a switching unit 26 is further provided to the image processing apparatus 2 shown in FIG. The switching unit 26 selects the statistical data acquired by the statistical data acquisition unit 22 and the spectral reflectance wavelength used by the image generation unit 24 to determine each primary color (RGB) value based on the user input.

In this embodiment, the user can select an analysis target. For example, the switching unit 26 causes the image processing apparatus 2b to operate as an analysis apparatus for the health condition of human skin as in the first embodiment, or analyzes whether the skin is skin or makeup skin as in the second and third embodiments. It is possible to switch between operating as a device that operates as an apparatus for analyzing a leaf state of a plant as in the fourth embodiment. Thereby, it becomes possible to analyze a plurality of objects with one imaging system.

The switching unit 26 causes the statistical data acquisition unit 22 to acquire appropriate statistical data according to the analysis target. Specifically, the switching unit 26 receives input of information related to the analysis target from the user. The switching unit 26 identifies statistical data corresponding to the analysis target input by the user, and causes the statistical data acquisition unit 22 to acquire the statistical data. For example, the switching unit 26 can specify statistical data corresponding to the analysis target input by the user by using data indicating the correspondence between the statistical data recorded in advance in the memory and the analysis target.

In addition, the structure which the switching part 26 notifies the information regarding the analysis object received from the user to the statistical data acquisition part 22, and the statistical data acquisition part 22 acquires the statistical data according to the analysis object may be sufficient.

Reception of user input by the switching unit 26 can be realized via a touch panel provided on the display 4, for example. Alternatively, a combination of the display 4 and an input device (for example, a button, a keyboard, or a pointing device) can be used to enable user input. The switching unit 26 can constitute a part of the user interface.

The user can designate an analysis target by selecting a desired one from a plurality of analysis targets displayed on the display 4, for example. As an example, the user can select one of analysis targets such as skin health condition analysis, skin makeup condition analysis, or plant health condition analysis.

In the recording medium 6 accessible from the image processing apparatus 2, statistical data of spectral reflectances of a plurality of types of objects are recorded in advance. The statistical data is recorded in a state that can be distinguished for each type of object, for example. For example, the statistical data may be recorded in a file format provided for each type of object. Or you may record on the table of RDB for every kind of target object. The statistical data may have a hierarchical structure according to the classification of the type of object. The data format of the statistical data is not limited to a specific one.

The statistical data acquisition unit 22 extracts the statistical data corresponding to the analysis target from the recording medium 6 according to the information received from the switching unit 26. Thereby, the statistical data acquisition part 22 can acquire the statistical data according to the analysis object.

The switching unit 26 determines the wavelength of the spectral reflectance used by the image generation unit 24 to determine the RGB value of the display pixel based on the information about the analysis target input from the user. For example, the switching unit 26 refers to data that is recorded in advance on the recording medium 6 and indicates the correspondence between the analysis target and the wavelength used for image generation, and determines the wavelength corresponding to the analysis target input by the user. be able to. This determination may be executed by the image generation unit 24 that has received notification of information about the analysis target from the switching unit 26.

In the recording medium 6 accessible by the image generation unit 24, for example, wavelength data indicating wavelengths to be selected is recorded for each of a plurality of analysis targets. This wavelength data includes data indicating which wavelength spectral reflectance of the spectral data is used when the image generation unit 24 determines RGB values. The wavelength data may be, for example, a wavelength value corresponding to an analysis target, or may be a parameter or function used when calculating RGB from spectrum data. The image generation unit 24 selects the wavelength data corresponding to the analysis target indicated by the switching unit 26 and uses it to determine each RGB value of the display image.

For example, when receiving an input indicating that the health condition of a person's skin is to be analyzed, the switching unit 26 selects a wavelength at which the difference in spectral reflectance between healthy skin and skin affected by a skin disease is larger than others. select. When receiving an input indicating that the makeup state of the human skin is to be analyzed, the switching unit 26 selects a wavelength for which the difference in spectral reflectance between the human bare skin and the skin of the human being made up is greater than others. . When receiving an input indicating that the state of the leaf of the plant is to be analyzed, the switching unit 26 compares the difference in spectral reflectance between the leaf or fruit of the healthy plant and the leaf or fruit of the diseased plant compared to the others. Select a larger wavelength. The image generation unit 24 calculates RGB values using the reflectance of the wavelength selected by the switching unit 26 among the spectrum data calculated by the estimation unit 23.

The switching unit 26 can also switch the processing itself for generating the primary color value constituting the color of the display image from the spectral data of the spectral reflectance of the object. For example, the function or table (a specific example will be described later) used when the image generation unit 24 calculates the RGB value of the display image from the spectrum data can be switched according to the analysis target.

Further, the switching target of the switching unit 26 is not limited to the above. For example, the switching unit 26 can switch the image data acquired by the image acquisition unit 21 according to a user input. For example, the camera 2 can switch the filter group 32 according to the analysis target. In this case, the image acquisition unit 21 can receive information related to the analysis target from the switching unit 26 and control the camera 2 to select a filter group 32 suitable for the analysis target based on this information. Alternatively, the imaging system 1 may include a plurality of cameras 2 suitable for each of a plurality of objects. The image acquisition unit 21 can select the camera 2 that acquires the image data based on the information on the analysis target from the switching unit 26.

According to the present embodiment, the image processing device 2b can switch the statistical data and the wavelength of the spectral reflectance used at the time of image generation according to the analysis target. This makes it possible to analyze various objects with a simple device configuration. Note that this embodiment can be applied to any of the above-described Embodiments 1 to 4, or a combination thereof.

<Embodiment 6>
FIG. 23 is a functional block diagram illustrating a configuration example of the imaging system 1 including the image processing device 2c according to the sixth embodiment. The image treatment device 2 c further includes a system data acquisition unit 27. The system data acquisition unit 27 obtains imaging system data indicating spectral characteristics of the imaging system 1 based on data obtained by imaging a reference sample whose spectral reflectance is known in advance by the imaging system 1 that captures an object. Generate.

The spectral characteristics of the imaging system 1 may vary depending on the shooting environment. For example, when the camera 3 and the light source 5 are not fixed, the spectral emissivity of the light source that irradiates the object with light varies depending on the shooting environment. In such a case, the system data acquisition unit 27 can calculate the spectral characteristics of the imaging system including the spectral emissivity of the light source based on the image data obtained by photographing the reference sample whose spectral reflectance is known. it can.

The system data acquisition unit 27 can start the system data acquisition process based on, for example, an input from the user. The system data acquisition unit 27 first outputs a control signal to the camera 3 or outputs a camera operation instruction to the user via the display 4. Thereby, the camera 3 of the imaging system 1 is caused to photograph a reference sample whose spectral reflectance is known in advance. The reference sample can be, for example, a color chart, a color chart, or a paint.

The spectral reflectance data of the reference sample can be recorded in advance in the image processing apparatus 2c. The system data acquisition unit 27 uses the value of the image data of the reference sample captured in the same environment as the environment in which the object is imaged and the spectral reflectance of the reference sample recorded in advance, and the spectrum of the imaging system in that environment. Data indicating characteristics can be calculated.

For example, even if the emissivity of the light source is not known, it is possible to estimate the spectral emissivity of the light source used by photographing the object to be photographed side by side with the color chart.

Alternatively, when the spectral characteristics of the imaging system 1 are unknown, the spectral reflectance of the object can be estimated using Wiener estimation by photographing the reference sample and the object using the imaging system 1. . Hereinafter, a specific example of this estimation process will be described.

First, the system data acquisition unit 27 inputs a reflectance data matrix o_real of a reference sample recorded in advance. A matrix v of output values of the camera 3 of an image obtained by capturing the reference sample with the imaging system 1 is input. Next, the system data acquisition unit 27 obtains a cross-correlation matrix Rov of o_real and v, and further obtains an autocorrelation matrix of v Rvv. From these, an estimation matrix G = RovRvv ⁻¹ is obtained. The image acquisition unit 21 acquires the matrix v ₁ of the output values of the camera 3 of the image of the object captured by the imaging system 1 and inputs the matrix v ₁ to the estimation unit 23. The estimation unit 23 calculates and outputs an estimated spectral reflectance o_est = Gv ₁ of the object.

<Modification 1>
In the first to sixth embodiments, the estimation unit 23 calculates the spectral data of the spectral reflectance of the object using principal component analysis. The spectral reflectance estimation process is not limited to this. For example, the estimation unit 23 can calculate the estimated value of the spectral reflectance using the winner estimation. Below, the example of calculation is demonstrated.

The estimation unit 23 inputs, as statistical data, a data file that includes the measured values of the spectral reflectance of many samples of the same substance as the target object. The estimation unit 23 calculates a spectral reflectance covariance matrix Rr and an average value matrix m of the data file.

Moreover, the estimation part 23 inputs the matrix F which shows the spectral characteristic of an imaging system as imaging system data. The estimation unit 23 calculates a correlation matrix Rn = <nn ^t > of the noise n by using the variation appearing in the multiband image obtained by photographing the uniform light intensity distribution by the imaging system 1 as the noise matrix n. Further, the estimation unit 23 calculates an estimation matrix W = CrF (FCrF + Rn). Here, Cr is a covariance matrix of spectral reflectances of a large number of samples of the same substance as the object.

The estimation unit 23 inputs a matrix v of output values of the camera 3 of an image taken by the imaging system 1. The estimation unit 23 outputs an estimated spectral reflectance r_est = m + W (v−Fm).

<Modification 2>
The image generation unit 24 can determine a primary color value constituting a color in an image to be displayed based on data indicating a correspondence between the primary color value and the spectral reflectance. For example, a look-up table (LUT (Look Up Table)) in which spectral reflectance values corresponding to the R value, G value, and B value are recorded can be recorded in the image treatment apparatus 2 in advance. The image generation unit 24 can determine RGB values corresponding to the spectral reflectance included in the spectral data calculated by the estimation unit 23 using the LUT. As a result, the processing speed can be increased.

FIG. 24 is a diagram illustrating an example of an LUT that the image generation unit 24 refers to. In the example shown in FIG. 24, a set of reflectance values of a plurality of wavelengths (400 nm, 410 nm,..., 700 nm) and a set of RGB values are recorded in association with each other. For example, the image generation unit 24 can read a set of RGB values corresponding to the reflectance values at a plurality of wavelengths indicated by the spectrum data, and use these values as output values.

<Configuration example of image processing apparatus and system>
In the first to sixth embodiments, the image processing device 2, 2a, 2b, or 2c (hereinafter collectively referred to as the image processing device 2) can be realized by a computer including one or more processors and a memory. The operations of the image acquisition unit 21, the statistical data acquisition unit 22, the estimation unit 23, the image generation unit 24, the makeup determination unit 25, the switching unit 26, and the system data acquisition unit 27 can be realized by the processor executing a predetermined program. . A program for causing the processor to execute the operations of the image acquisition unit 21, the statistical data acquisition unit 22, the estimation unit 23, the image generation unit 24, the makeup determination unit 25, the switching unit 26, or the system data acquisition unit 27 and the program are recorded. Non-transitory recording media are also included in the embodiments of the present invention. The image processing apparatus 2 is configured by an IC chip, for example. Note that a part of the image processing apparatus 2 may be configured by a logic circuit.

The imaging system 1 can be configured integrally with a portable terminal, for example. That is, the camera 3, the image processing device 2c, and the display 4 may be mounted on one housing. For example, the imaging system 1 can be constructed by installing application software for realizing the image processing apparatus 2c in a mobile phone, a smartphone, a tablet terminal, a game machine, or other camera-equipped terminals.

As described above, in the configuration in which the display 4, the camera 3, and the image processing apparatus 2 are combined, the user can immediately see the result of processing the image captured by the camera 3 on the display 4. For example, the light receiving surface of the camera 3 can be arranged in the same direction as the display surface of the display 4. In this case, the user can take an image of his / her face with the camera 3 and check an image indicating the skin condition of the face on the display 4 as it is. That is, the user can confirm the image processing result as if he / she used the display as a mirror and looked at the mirror.

Alternatively, the imaging system 1 may be configured by a plurality of devices. For example, the imaging unit including the camera 3 and the image processing apparatus 2 may be mounted in independent housings and configured to be able to communicate with each other. For example, the imaging system 1 can be configured by a computer such as a PC, a smartphone, or a tablet in which an application for realizing the image processing apparatus 2 is installed and a camera that can be connected to the computer wirelessly or by wire. The imaging unit including the camera 3 can be configured by a probe including the light source 5, the lens system 31, the filter group 32, and the imaging sensor 33, for example. In this case, the spectral characteristics of the imaging system are less likely to change depending on the environment. Therefore, it is possible to reduce the trouble of the user such as photographing the reference sample in order to acquire the imaging system data.

DESCRIPTION OF SYMBOLS 1 Imaging system 2 Image processing apparatus 21 Image acquisition part 22 Statistical data acquisition part 23 Estimation part 24 Image generation part 3 Camera 4 Display 5 Light source

Claims (7)

An image acquisition unit for acquiring image data of an object photographed in at least three different wavelength bands;
A statistical data acquisition unit for acquiring statistical data of spectral reflectance of the same kind of substance as the object;
An estimation unit that calculates spectral data including spectral reflectance of the object in more than three wavelength bands using the image data and the statistical data;
A display image is generated by determining each value of a plurality of primary colors constituting the color of an image to be displayed using a spectral reflectance at at least one wavelength selected from the spectral reflectance included in the spectral data. An image processing unit. The object is human skin,
The statistical data acquisition unit acquires statistical data including the spectral reflectance of the human skin and the spectral reflectance of the skin of the person who made up the skin,
The image generation unit uses a plurality of primary colors constituting the color of the image to be displayed, using the spectral reflectance at a wavelength selected based on a difference in spectral reflectance between human bare skin and makeup human skin. The image processing apparatus according to claim 1, wherein each value is determined. The object is human skin,
The statistical data acquisition unit acquires statistical data including spectral reflectance of healthy skin and skin spectral reflectance of skin affected by skin diseases,
The image generation unit uses the spectral reflectance at a wavelength selected based on the difference in spectral reflectance between healthy skin and skin affected by a skin disease, and uses each of the primary colors constituting the color of the image to be displayed. The image processing apparatus according to claim 1, wherein a value is determined. The object is a leaf or fruit of a plant,
The statistical data acquisition unit acquires statistical data including a spectral reflectance of a healthy plant leaf or fruit and a spectral reflectance of a diseased plant leaf or fruit,
The image generation unit uses the spectral reflectance selected based on the difference in spectral reflectance between a healthy plant leaf or fruit and a diseased plant leaf or fruit to change the color of the image to be displayed. The image processing apparatus according to claim 1, wherein each value of primary colors to be configured is determined. A switching unit that selects the statistical data acquired by the statistical data acquisition unit and a spectral reflectance wavelength used by the image generation unit to determine each value of the primary color based on a user input; The image processing apparatus according to any one of claims 1 to 4. The image generation unit is capable of accessing a recording medium on which wavelength data indicating a wavelength of spectral reflectance used for determining each value of the primary color for a plurality of analysis objects is recorded, and the switching unit receives from a user 6. The image processing apparatus according to claim 5, wherein wavelength data corresponding to the analysis target is selected based on the data related to the analysis target and used for determining each value of the primary color. Image acquisition processing for acquiring image data of an object photographed in at least three different wavelength bands;
Statistical data acquisition processing for acquiring spectral reflectance statistical data of the same kind of substance as the object,
An estimation process for calculating spectral data including spectral reflectance of the object in more than three wavelength bands using the image data and the statistical data;
A display image is generated by determining each value of a plurality of primary colors constituting the color of an image to be displayed using a spectral reflectance at at least one wavelength selected from the spectral reflectance included in the spectral data. A program that causes a computer to execute image generation processing.

Images