WO2025126817A1 - Information processing device, information processing method, and program - Google Patents

Abstract

An information processing device according to the present technology comprises a light source spectrum estimation unit that uses a light source model for expressing a light source spectrum, which is spectral information of an outdoor light source according to the weighted addition of a solar spectrum which is spectral information of emission light from the sun and a sky spectrum which is spectral information of emission light from the sky, to estimate the light source spectrum on the basis of an object spectrum which is spectral information of a subject captured by a spectroscopic camera.

Description

Information processing device, information processing method, and program

This technology relates to an information processing device, an information processing method, and a program, and in particular to a technology for estimating the components of a light source spectrum contained in a spectroscopic image obtained by a spectroscopic camera.

Spectroscopic sensors (multispectral sensors) are known that obtain multiple narrowband images that are wavelength characteristic analysis images of light from a subject, in other words, analysis images of the subject's spectral information (spectral spectrum). In addition, applications have been developed that perform various analyses of a subject based on the spectral information obtained by the spectroscopic sensor, such as estimating the vegetation state of plants or the condition of human skin, based on these multiple narrowband images.

When obtaining spectral information of a subject, the spectral information of the light source illuminating the subject becomes a noise component, and it is desirable to remove this noise component. For this reason, in the field of spectroscopic sensing, the light source spectrum, which is the spectral information of the light source, is estimated.

For example, the following Patent Document 1 discloses an image acquisition device that includes an image sensor that acquires a predetermined image, and a processor that acquires a basis based on the surrounding environment, estimates lighting information using the acquired basis, reflects the estimated lighting information, and performs color conversion related to the image.

JP 2023-048996 A

However, the technology described in Patent Document 1 uses a method for estimating lighting information (light source spectrum) by preparing a basis for each expected environment in advance and using the basis according to the results of environmental detection by a sensor, so it is not possible to estimate the light source spectrum in response to environmental changes other than the expected environment.

This technology was developed in consideration of the above issues, and aims to improve the robustness of light source spectrum estimation.

The information processing device according to the present technology includes a light source spectrum estimation unit that uses a light source model that expresses a light source spectrum, which is spectral information of an outdoor light source, by weighted addition of a solar spectrum, which is spectral information of light irradiated from the sun, and a sky spectrum, which is spectral information of light irradiated from the sky, to estimate the light source spectrum based on an object spectrum, which is spectral information of a subject obtained by a spectroscopic camera.
This makes it possible to estimate the light source spectrum by fitting a function serving as a light source model to the object spectrum detected by the spectroscopic camera.

FIG. 1 is a diagram illustrating an example of the configuration of a light source spectrum estimation system including an information processing device according to an embodiment. FIG. 2 is a block diagram showing an example of a schematic configuration of a spectroscopic camera used in the embodiment. 2 is a diagram illustrating a schematic configuration example of a pixel array unit of a spectroscopic sensor. FIG. 5A and 5B are explanatory diagrams of band narrowing processing in the embodiment. FIG. 1 is a block diagram illustrating an example of a hardware configuration of an information processing device according to an embodiment. FIG. 2 is a functional block diagram for explaining functions of an information processing device according to the first embodiment. FIG. 1 is a diagram showing an example of a solar spectrum and a sky spectrum. 1 is a flowchart illustrating an example of a processing procedure for implementing a light source spectrum estimation method according to a first embodiment. FIG. 11 is a block diagram showing an example of the configuration of a light source spectrum estimation system according to a second embodiment. FIG. 11 is a functional block diagram for explaining functions of an information processing device according to a second embodiment. FIG. 11 is a functional block diagram for explaining functions of an information processing device as a modified example.

Hereinafter, with reference to the accompanying drawings, embodiments of the present technology will be described in the following order.
1. First embodiment
1-1. System configuration
[1-2. Spectroscopic camera configuration]
[1-3. Configuration of information processing device]
[1-4. Functions of information processing device]
<2. Second embodiment>
3. Modifications
4. Summary of the embodiment
<5. This Technology>

1. First embodiment
1-1. System configuration
FIG. 1 is a diagram showing an example of the configuration of a light source spectrum estimation system including an information processing device according to an embodiment.
As shown in the figure, the light source spectrum estimation system includes an information processing device 1 , a spectroscopic camera 2 , and an upward-facing camera 3 .
Here, the spectroscopic camera 2 refers to a camera equipped with a spectroscopic sensor as a light receiving sensor. The "spectroscopic sensor" is a light receiving sensor for obtaining a plurality of narrowband images that serve as wavelength characteristic analysis images of light from a subject.

The information processing device 1 is configured as a computer device, and as described below, performs processing to estimate the light source spectrum, which is the spectral information of the light source, based on the spectral information of the subject obtained by the spectroscopic camera 2.

Here, spectral information refers to information that indicates the light intensity for each wavelength.

In this embodiment, the information processing device 1 estimates the light source spectrum of an outdoor light source (natural light source) as the light source spectrum, and estimates the light source spectrum using a light source model that expresses the light source spectrum, which is the spectral information of an outdoor light source, by weighted addition of the sunlight spectrum, which is the spectral information of light irradiated from the sun, and the sky spectrum, which is the spectral information of light irradiated from the sky; details will be explained later.

The upward facing camera 3 is a camera that captures images of the sky, in other words, a camera whose imaging direction is directed upward, and is capable of obtaining images of the sky.
The use of the upward facing camera 3 will be explained later.

[1-2. Spectroscopic camera configuration]
FIG. 2 is a block diagram showing an example of the configuration of the spectroscopic camera 2.
The spectroscopic camera 2 includes at least a spectroscopic sensor 4 and a spectroscopic image generating unit 5. As shown in the figure, the spectroscopic camera 2 of the embodiment includes a control unit 6, a timing unit 7, a GNSS (Global Navigation Satellite System) sensor 8, and a communication unit 9 in addition to the spectroscopic sensor 4 and the spectroscopic image generating unit 5.

FIG. 3 is a schematic diagram showing an example of the configuration of the pixel array unit 4 a of the spectroscopic sensor 4 .
As shown in the figure, the pixel array section 4a has a plurality of spectroscopic pixel units Pu formed therein, each of which has a plurality of pixels Px, each of which receives light of a different wavelength band, arranged two-dimensionally in a predetermined pattern. The pixel array section 4a has a plurality of spectroscopic pixel units Pu arranged two-dimensionally.

The example shown in the figure is an example in which each spectroscopic pixel unit Pu individually receives light in a total of eight wavelength bands, from λ1 to λ8, at each pixel Px, in other words, an example in which the number of wavelength bands received and separated within each spectroscopic pixel unit Pu (hereinafter referred to as the ``number of received wavelength channels'') is ``8'', but this is merely one example for explanatory purposes, and the number of received wavelength channels in the spectroscopic pixel unit Pu may be at least multiple and can be set arbitrarily.
Hereinafter, the number of light-receiving wavelength channels in the spectroscopic pixel unit Pu is defined as "N".

In FIG. 2, the spectroscopic image generating unit 5 generates M narrowband images based on a RAW image output from the spectroscopic sensor 4. Here, M>N, and as an example, N=8 and M=41, etc.

The spectroscopic image generating unit 5 has a demosaic unit 5a and a narrowband image generating unit 5b. The demosaic unit 5a performs demosaic processing on the raw image from the spectroscopic sensor 4, and the narrowband image generating unit 5b performs narrowband processing (linear matrix processing) based on the wavelength band images for N channels obtained by the demosaic processing, thereby generating M narrowband images from N wavelength band images.

FIG. 4 is an explanatory diagram of the band narrowing process for obtaining M narrowband images.
A predetermined matrix operation is performed for each pixel position based on the N-channel wavelength band images obtained by the demosaic process performed by the demosaic unit 5a, thereby obtaining M-channel narrowband images. In this manner, in order to convert the N-channel wavelength band images into M-channel narrowband images, the process of obtaining M-channel pixel values ( _I'0 to I'M _-1 in the figure) by matrix operation using the N- _channel pixel values (I0 to _IN-1 in the figure) for each pixel position is the band narrowing process.

Here, if the pixel value after demosaic processing is R, the input wavelength channel is n (0 to N-1), the narrowband coefficient is C, the output pixel value by the narrowband processing is B, and the output wavelength channel is m (0 to M-1), then the calculation formula for the narrowband processing can be expressed as the following [Equation 1].

That is, the pixel value B ₀ of the m=0th output wavelength channel is R[0]×C ₀ [0]+R[1]×C ₀ [1]+R[2]×C ₀ [2]+, ...+R[N-1]×C ₀ [N-1], and the pixel value B ₁ of the m=1st output wavelength channel is R[0]×C ₁ [0]+R[1]×C ₁ [1]+R[2]×C ₁ [2]+, ...+R[N-1]×C ₁ [N-1].
The process is similar thereafter, and the pixel value B _M- 1 of the final m=M-1th output wavelength channel is R[0]×C _M-1 [0]+R[1]×C _M-1 [1]+R[2]×C _M-1 [2]+, ...+R[N-1]×C _M-1 [N-1].
In this case, a total of N _× M narrowband coefficients C are used: C ₀ [0] to C ₀ [N-1] for obtaining pixel value B ₀ , C ₁ [0] to C ₁ [N-1] for obtaining pixel value B ₁ , ..., C _M-1 [0] to C _M-1 [N-1] for obtaining pixel value B M-1.

In FIG. 2, the control unit 6 is configured with a microcomputer having, for example, a CPU (Central Processing Unit), a ROM (Read Only Memory), and a RAM (Random Access Memory), and the CPU performs overall control of the spectroscopic camera 2 by executing processing based on, for example, a program stored in the ROM or a program loaded into the RAM.

The clock unit 7 keeps track of the current time.
The GNSS sensor 8 is an example of a position detection device that detects the current position of the spectroscopic camera 2 .
The uses of the current time information obtained by the clock unit 7 and the current position information of the spectroscopic camera 2 detected by the GNSS sensor 8 will be explained later.

The communication unit 9 performs wired or wireless data communication with an external device. For example, the communication unit 9 may be configured to perform wired data communication with an external device according to a predetermined wired communication standard such as the Universal Serial Bus (USB) communication standard, wireless data communication with an external device according to a predetermined wireless communication standard such as the Bluetooth (registered trademark) communication standard, or wireless or wired data communication with an external device via a predetermined network such as the Internet.
The control unit 6 can transmit and receive data to and from an external device (particularly, the information processing device 1 in this example) via the communication unit 9 .

[1-3. Configuration of information processing device]
FIG. 5 is a block diagram showing an example of the hardware configuration of the information processing device 1.
5, the information processing device 1 includes a CPU 11, a ROM 12, and a RAM 13. The CPU 11 functions as an arithmetic processing unit that performs various processes, and executes the various processes according to a program stored in the ROM 12 or a program loaded from the storage unit 19 to the RAM 13. The RAM 13 also stores data and the like necessary for the CPU 11 to execute the various processes, as appropriate.

The CPU 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input/output interface (I/F) 15 is also connected to this bus 14.

The input/output interface 15 is connected to an input unit 16 including an operator and an operating device.
For example, various types of operators and operation devices such as a keyboard, a mouse, keys, a dial, a touch panel, a touch pad, a remote controller, etc. are assumed as the input unit 16. In this example, the touch panel in the input unit 16 is formed on the display screen of the display unit 17, and the user can perform a touch operation on the display screen.
An operation by a user is detected by the input unit 16 , and a signal corresponding to the input operation is interpreted by the CPU 11 .

In addition, the input/output interface 15 is connected, either integrally or separately, to a display unit 17 formed of an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) panel, or the like, and an audio output unit 18 formed of a speaker, or the like.
The display unit 17 is used to display various types of information, and is configured as a display device provided in the housing of the information processing device 1, for example.

The display unit 17 displays images for various types of image processing, videos to be processed, etc., on the display screen based on instructions from the CPU 11. The display unit 17 also displays various operation menus, icons, messages, etc., i.e., a GUI (Graphical User Interface), based on instructions from the CPU 11.

A storage unit 19 and a communication unit 20 can be connected to the input/output interface 15 .
The storage unit 19 is configured with a hard disk drive (HDD) or a solid state drive (SSD) and stores various types of information.

The communication unit 20 performs communication processing via a transmission path such as the Internet, and communication with various devices via wired/wireless communication, bus communication, and the like.
In particular, in the case of this embodiment, the communication unit 20, like the communication unit 9 in the spectroscopic camera 2 described above, is configured to perform wired data communication with an external device according to a specified wired communication standard such as the USB communication standard, wireless data communication with an external device according to a specified wireless communication standard such as the Bluetooth communication standard, or wireless or wired data communication with an external device via a specified network such as the Internet.
This enables the CPU 11 to transmit and receive data to and from the spectroscopic camera 2 via the communication unit 20 .

If necessary, a drive 21 is also connected to the input/output interface 15, and a removable recording medium 22 such as a memory card or optical disk is appropriately attached.

The drive 21 makes it possible to read data files such as programs used for each process from the removable recording medium 22. The read data files are stored in the memory unit 19, and images and sounds contained in the data files are output on the display unit 17 and the audio output unit 18. In addition, computer programs and the like read from the removable recording medium 22 are installed in the memory unit 19 as necessary.

Here, the information processing device 1 is not limited to being configured by a single computer device as shown in Fig. 5, but may be configured by a system of multiple computer devices. The multiple computer devices may be systemized by a LAN (Local Area Network) or the like, or may be located in a remote location by a VPN (Virtual Private Network) using the Internet or the like. The multiple computer devices may include computer devices as a server group (cloud) available through a cloud computing service.

[1-4. Functions of information processing device]
As described above, in this embodiment, the information processing device 1 estimates a light source spectrum based on the spectral information of the subject obtained by the spectroscopic camera 2, using a predetermined light source model.
Hereinafter, various functions of the information processing device 1 according to the first embodiment, including the light source spectrum estimation function, will be described.
In the following description, the spectral information of the subject obtained by the spectroscopic camera 2 (the brightness values of each narrowband image) is referred to as the "object spectrum."

FIG. 6 is a functional block diagram for explaining various functions of the CPU 11 of the information processing device 1 according to the first embodiment.
As shown in the figure, the CPU 11 has functions as a light source spectrum estimation section F1, a first environmental parameter estimation section F2, a second environmental parameter estimation section F3, an image processing section F4, and a sun/shade area determination section F5.

As mentioned above, in this embodiment, a light source model that represents the light source spectrum by weighted addition of the sunlight spectrum and the sky spectrum is used to estimate the light source spectrum. Specifically, in this example, the Bird model is used as the light source model.

As is well known, the Bird model is an outdoor light source model that can calculate the solar spectrum and sky spectrum using the earth-sun distance, solar zenith angle, aerosol turbidity in the air, precipitable water vapor (also called "precipitable water vapor"), and ozone mass as inputs. Note that the earth-sun distance represents the distance between the earth and the sun, and the solar angle represents the amount of deviation of the sun direction from the zenith direction of the observation position on the earth as an angle.
The Bird model assumes that the target subject has sunny and shaded areas, and represents the spectrum of the light source based on the proportion of the subject that is illuminated by sunlight and the proportion of the subject that is illuminated by the sky.

In the Bird model, the solar spectrum and sky spectrum are calculated, and then their weighted addition makes it possible to approximately represent the light source spectrum from the sky in sunny or shady conditions regardless of the weather (sunny, cloudy, rainy).
in particular,
"Light source spectrum = sunlight spectrum * b0 + sky spectrum * b1"
It is possible to express a light source spectrum corresponding to an environmental type such as weather, sunshine or shade by using the above formula. Note that "b0" is a weighting coefficient for the sunlight spectrum, and "b1" is a weighting coefficient for the sky spectrum.

For reference, Figure 7 shows an example of the solar spectrum and sky spectrum calculated using the Bird model.

In the first embodiment, in order to use the solar spectrum and sky spectrum corresponding to the actual environment in estimating the light source spectrum, the environmental parameters used in calculating the solar spectrum and sky spectrum are estimated based on the image captured by the upward camera 3, the position information detected by the GNSS sensor 8, and the current time information measured by the timing unit 7.
Specifically, in this example, the solar altitude, solar angle, and amount of ozone used in calculating the solar spectrum and sky spectrum are estimated based on the position information detected by the GNSS sensor 8 and the current time information measured by the timing unit 7. This estimation is performed by the first environmental parameter estimation unit F2.
As is well known, the amount of ozone can be estimated by latitude, longitude, and time. The solar altitude can be estimated by time, and the solar angle can be estimated by latitude, longitude, and time.

Furthermore, the aerosol turbidity and the amount of precipitated water vapor used in the calculation of the solar spectrum and the sky spectrum are estimated based on images captured by the upward camera 3. This estimation is performed by a second environmental parameter estimation unit F3.
A specific method for estimating the aerosol turbidity and the amount of precipitated water vapor based on images captured by the upward camera 3 will be described later.

Here, even if a standard value for the assumed environment is used for the amount of ozone, the effect on the calculated light source spectrum is minor, so it is possible to use a fixed value (a fixed value serving as a standard value) without making an estimate based on the current location information and current time information. This point will be explained later in the second embodiment.

Furthermore, for aerosol turbidity and precipitating water vapor, the influence on the calculated light source spectrum is relatively large during sunrise and sunset, so it is desirable to estimate using the upward-facing camera 3. However, if the intended use is, for example, excluding those times, standard values may be used. This point will also be explained later in the second embodiment.

A specific example of light source spectrum estimation by the light source spectrum estimation unit F1 will be described.
First, in the estimation, an error function Fd is used to calculate an error Ds as follows: where "^" means exponentiation.

Error Ds = || object spectrum - (light source spectrum x object spectral reflectance + specular reflection) ||^2

Here, the term "spectral reflectance of an object" refers to the spectral reflectance of an object as a subject. The spectral reflectance refers to information indicating the reflectance for each wavelength.
In this example, the object spectral reflectance is modeled and handled using a polynomial of wavelength λ (particularly a function of second order or lower) as shown below.

Object spectral reflectance = m2λ^2+m1λ+m0

Furthermore, "specular reflection" is defined as "c*solar spectrum."

In the first embodiment, the light source spectrum estimation unit F1 estimates the light source spectrum for each pixel of the spectroscopic image (each narrowband image) obtained by the spectroscopic camera 2 by deriving parameters (specifically b0, b1, m0, m1, m2, c) that minimize the above-mentioned error Ds while satisfying the constraint conditions described below.

The constraints on light source spectrum estimation are as follows:
1) b1 is a common value for all pixels (because the sky spectrum is equivalent to ambient light)
2) b0, b1, and c are all equal to or greater than 0. 3) The object spectral reflectance satisfies v≦object spectral reflectance≦1.0 (v is a user-set value).
4) b0/b1≧w (w is a user-set value)
5) If b0/b1<z, then c=0 (z is a user-specified value)
The above condition 3) is a condition intended to improve the convergence of parameter derivation for spectral reflectance information modeled by a quadratic function.
The above condition 4) corresponds to the case where there is an empirical rule that "b0/b1" is always equal to or less than 0.1 even in a deep shadow area, for example, and is intended to improve the convergence of parameter derivation by allowing the user to set a lower limit value for "b0/b1".
The above condition 5) aims to improve the convergence of parameter derivation by allowing the user to set the conditions for specular reflection.

A specific example of the parameter derivation process will be described with reference to the flowchart of FIG.
First, in step S101, the CPU 11 (light source spectrum estimation unit F1) sets initial values for m0, m1, m2, and c, excluding b0 and b1. Note that it is possible to use standard values for each of the initial values.

In step S102 following step S101, the CPU 11 substitutes initial values m0, m1, m2, and c into the error function Fd to derive b0 and b1 that minimize the error Ds. That is, m0, m1, and m2 used in the calculation of the object spectral reflectance in the error function Fd, and c used in the calculation of specular reflection are fixed to their initial values, and b0 and b1 that minimize the error Ds are derived.
Here, the calculation of the error Ds requires the calculation of the light source spectrum, which in turn requires the calculation of the solar spectrum and the sky spectrum. In the Bird model, the solar spectrum and the sky spectrum are expressed by functions with the solar altitude, solar angle, amount of ozone, aerosol turbidity, and amount of precipitated water vapor as variables, respectively. Therefore, in the calculation process of the error Ds, the CPU 11 uses the solar altitude, solar angle, and amount of ozone estimated by the first environmental parameter estimation unit F2, and the aerosol turbidity and amount of precipitated water vapor estimated by the second environmental parameter estimation unit F3 as described below, to calculate the solar spectrum and the sky spectrum.

In step S103 following step S102, the CPU 11 substitutes the derived b0 and b1 and the initial value c into the error function Fd to derive m0, m1, and m2 that minimize the error Ds.
Furthermore, in the following step S104, the CPU 11 substitutes the derived b0, b1, m0, m1, and m2 into the error function Fd to derive c that minimizes the error Ds.

The above-mentioned processes from step S101 to step S104 constitute the initial derivation process.
After completing the first derivation process, the CPU 11 performs the second and subsequent derivation processes in steps S105 and subsequent steps.

Specifically, first in step S105, the CPU 11 substitutes m0, m1, m2, and c derived in the immediately preceding derivation process into the error function Fd to derive b0 and b1 that minimize the error Ds. That is, for example, if the immediately preceding derivation process is the initial derivation process, then the m0, m1, m2, and c finally derived in the initial derivation process are substituted into the error function Fd to derive b0 and b1 that minimize the error Ds.

In step S106 following step S105, the CPU 11 substitutes the derived b0 and b1 and the c derived in the immediately preceding derivation process into the error function Fd to derive m0, m1, and m2 that minimize the error Ds.
Furthermore, in the following step S107, the CPU 11 substitutes the derived b0, b1, m0, m1, and m2 into the error function Fd to derive c that minimizes the error Ds.

In step S108 following step S107, the CPU 11 determines whether the derivation end condition is satisfied. Here, as the derivation end condition, a condition is set that can estimate that the degree of parameter optimization (reduction of error Ds) has reached a desired degree. For example, a condition based on the value of error Ds calculated using the finally derived b0, b1, m0, m1, m2, and c can be set. As an example, a condition that the error Ds is equal to or less than a predetermined threshold value or a condition that an inflection point (inflection point from a decrease to an increase) of the value of the error Ds is detected can be set. Note that, when the latter condition is set, the parameters derived in the derivation process immediately before the inflection point is detected are treated as the parameters of the final derivation.
Alternatively, the derivation end condition may not be based on the value of the error Ds, but may be, for example, that the number of derivation processes reaches a specified number.

If it is determined in step S108 that the derivation end condition is not satisfied, the CPU 11 returns to step S105. As a result, the second and subsequent derivation processes are repeated until the derivation end condition is satisfied.

On the other hand, if it is determined in step S108 that the derivation end condition is satisfied, the CPU 11 proceeds to step S109, where it calculates the light source spectrum using the finally derived b0 and b1, the sunlight spectrum, and the sky spectrum. That is, the above-mentioned "light source spectrum = sunlight spectrum * b0 + sky spectrum * b1"
The light source spectrum is calculated by the above calculation.
It should be noted that the light source spectrum calculated by the above formula does not take into account the spectral sensitivity of the spectroscopic sensor 4. When the spectral sensitivity of the spectroscopic sensor 4 is taken into account, the light source spectrum (Nch) calculated by the above formula is convoluted with the spectral sensitivity of the spectroscopic sensor 4 to obtain a sensor signal, which is then narrowed in band by the above-mentioned [Formula 1] to obtain the light source spectrum of Mch.

The CPU 11 (light source spectrum estimation unit F1) executes the above-described derivation process for each pixel. This determines the light source spectrum for each pixel, and also determines the weighting coefficients b0 and b1 for each pixel.

Here, the above-mentioned light source spectrum estimation method can be said to be an estimation method as follows.
In other words, this estimation method estimates the light source spectrum by estimating the parameters of the light source model and the object spectral reflectance using the error between a trial object spectrum, which is a spectrum calculated by multiplying a trial light source spectrum, which is a light source spectrum calculated by setting candidate values for the parameters of the light source model, by a candidate value of the object spectral reflectance indicating the spectral reflectance of the object as the subject, and the object spectrum, as a reference value for estimation.

In the above, an example of estimating the light source spectrum is given of a method for deriving the weighting coefficients b0 and b1 using a parameter optimization method that minimizes the error Ds. However, it is also possible to estimate the light source spectrum using an AI model that has been machine-learned to input a spectroscopic image captured by the spectroscopic camera 2 and output the weighting coefficients b0 and b1.
Specifically, b0 and b1 are inferred for each pixel of the image captured by the spectroscopic camera 2 using a DNN (Deep Neural Network) capable of outputting the same size as the input image, such as a U-net used for segmentation, and this DNN can be trained, for example, by the error Ds. During learning, m0, m1, m2, and c are required for the calculation of the error Ds, and these can be derived, for example, by using the b0 and b1 inferred by the DNN using the same method as in steps S103 and S104 described above.

Next, we will explain the estimation process of aerosol turbidity and precipitated water vapor amount by the second environmental parameter estimation unit F3 shown in Figure 6.

As described above, the aerosol turbidity and the amount of precipitated water vapor are estimated based on the images captured by the upward-facing camera 3.
As a premise, in this example, the upward camera 3 is configured as a camera equipped with a spectroscopic sensor 4, similar to the spectroscopic camera 2, and is configured to obtain spectroscopic images (M narrowband images) as output images.
For confirmation, the solar spectrum and the sky spectrum are expressed as functions with the variables being the solar altitude, the solar angle, the amount of ozone, the aerosol turbidity, and the amount of precipitated water vapor.
For the sake of explanation, the aerosol turbidity will be represented as τ and the amount of precipitated water vapor as σ.

As with the above-described derivation process of b0 and b1, the aerosol turbidity τ and the amount of precipitated water vapor σ can be estimated by a method of defining a predetermined error function and deriving parameters that minimize the error.
Here, it can be said that spectral information expressed by the following formula is detected on the imaging plane of upward camera 3.
"Sunlight spectrum * b0' + sky spectrum * b1'"
Here, the weighting coefficient b0' of the sunlight spectrum and the weighting coefficient b1' of the sky spectrum in the above formula can be said to be coefficients indicating the degree of sunlight at the position where the upward camera 3 is disposed (imaging position).

Based on the above premise, in this example, the following error function Fd' is used as the error function used to estimate the aerosol turbidity τ and the amount of precipitated water vapor σ.

Error Ds'=
||Upward average spectral information - (sunlight spectrum * b0' + sky spectrum * b1') ||^2

Here, the "upward average spectral information" refers to the average value of the entire image of the spectral information obtained by the upward camera 3. Specifically, it is M brightness values obtained by calculating the average brightness value of the entire image for each of the M narrowband images.
As described above, the solar spectrum and sky spectrum are each expressed as a function with the solar altitude, solar angle, ozone amount, aerosol turbidity τ, and precipitated water vapor amount σ as variables, and the solar altitude, solar angle, and ozone amount estimated by the first environmental parameter estimation unit F2, as well as the aerosol turbidity τ and precipitated water vapor amount σ, are substituted into the calculation of the solar spectrum and sky spectrum in the above formula, respectively.

The second environmental parameter estimation unit F3 estimates the aerosol turbidity τ and the amount of precipitated water vapor σ in the following procedure.
First, the initial values of τ and σ are set. In this case, it is also possible to use standard values for the initial values.

Next, the second environment parameter estimation unit F3 substitutes τ and σ as initial values into the error function Fd′, and derives b0′ and b1′ that minimize the error Ds′.
Furthermore, the second environmental parameter estimation unit F3 substitutes the derived b0', b1' and σ as an initial value into the error function Fd', and derives τ that minimizes the error Ds'.
Next, the second environmental parameter estimation unit F3 substitutes the derived b0', b1', and τ into the error function Fd' to derive σ that minimizes the error Ds'.

In this manner, the initial derivation process of b0', b1', τ, and σ is executed. After that, the second and subsequent derivation processes are executed as follows until a predetermined derivation end condition is met.
That is, first, τ and σ derived in the immediately preceding derivation process are substituted into the error function to derive b0′ and b1′ that minimize the error Ds′.
Next, the derived b0 and b1 and the σ derived in the immediately preceding derivation process are substituted into the error function Fd' to derive τ that minimizes the error Ds'.
Furthermore, the derived b0', b1', and τ are substituted into the error function Fd' to derive σ that minimizes the error Ds'.
In this case as well, the derivation end condition may be set to the same condition as in the above-mentioned parameter derivation process for b0, b1, m0, m1, m2, and c.

The above derivation process makes it possible to estimate the solar spectrum according to the environment, the aerosol turbidity τ for calculating the sky spectrum, and the amount of precipitating water vapor σ.

In the above, the estimation of environmental parameters by the second environmental parameter estimation unit F3 is exemplified by a case in which an environmental parameter is derived by a parameter optimization method that minimizes the error Ds', but the estimation method of the environmental parameters by the second environmental parameter estimation unit F3 is not limited to this. For example, it is possible to use an AI (artificial intelligence) model that has been machine-learned so that it can derive environmental parameters from images captured by the upward camera 3. In this case, the machine learning of the AI model is considered to be performed by deep learning using images captured of the sky as learning input data and correct answer information for τ and σ as teacher data. In this case, it is considered to use the above error function Fd' as the cost function.

Furthermore, in estimating the environmental parameters by the second environmental parameter estimation unit F3, the upward camera 3 is not limited to being a camera equipped with a spectroscopic sensor 4 like the spectroscopic camera 2, but any camera configured to receive and separate at least a plurality of wavelength bands of light, such as an RGB camera, may be used.

In FIG. 6, an image processing unit F4 performs predetermined image processing based on the light source spectrum estimated by the light source spectrum estimation unit F1 and the spectroscopic image obtained by the spectroscopic camera 2.
The image processing by the image processing unit F4 may be a light source cancellation process, which is a process of removing the spectrum of the light source included in the object spectrum as the spectroscopic image by using the light source spectrum estimated by the light source spectrum estimation unit F1.

Alternatively, the image processing by the image processing unit F4 may include semantic segmentation processing based on the light source spectrum estimated by the light source spectrum estimation unit F1 and the spectroscopic image obtained by the spectroscopic camera 2.
Specifically, it is conceivable to perform light source cancellation processing on the spectroscopic image using the light source spectrum estimated by the light source spectrum estimation unit F1, and then perform semantic segmentation processing on the image after cancellation using an AI model.
In this case, as an AI model for performing semantic segmentation processing, rather than using a model trained on an image from which the light source component has been removed as learning input data, it is also possible to use a model trained on an image from which the light source component has not been removed and the light source spectrum as learning input data.
This makes it possible to realize highly accurate semantic segmentation processing that eliminates the influence of light source components (specifically, the influence of sunlight and shade) while eliminating the need for light source cancellation processing.

Also, in Figure 6, the sun/shade area determination unit F5 determines whether the subject captured by the spectroscopic camera 2 is in a sun or shade area using the sunlight spectrum weighting coefficient b0 and the sky spectrum weighting coefficient b1 estimated by the light source spectrum estimation unit F1.
Basically, the weighting coefficient b0 of the solar spectrum tends to be large in sunny areas compared to shaded areas, so pixels with a large weighting coefficient b0 are determined to be in sunny areas, and other pixels are determined to be in shaded areas. There are various specific methods, but one possible method is to generate a histogram using b0, find a threshold value for b0 for determining whether the area is in sunny or shade based on the histogram, and determine whether the area is in sunny or shade based on the results of comparing the threshold value with b0.

In the above, an example has been given in which image processing by the image processing unit F4 is realized by software processing by the CPU 11, but at least a part of the image processing may be realized by a method other than software processing by the CPU 11. For example, at least a part of the image processing by the image processing unit F4 may be realized by hardware processing by a signal processing unit such as a DPS (Digital Signal Processor) or GPU (Graphics Processing Unit) outside the CPU 11. In particular, in the case of semantic segmentation processing using an AI model, it is conceivable that the necessary convolution calculations will be realized by using a hardware signal processing unit such as a GPU.

Although the above describes an example in which the light source spectrum is estimated on a pixel-by-pixel basis, in the first embodiment, the light source spectrum may be estimated for each divided region obtained by dividing the spectral image into regions each having a predetermined number of pixels.
For example, it is possible to carry out the process for each divided region having a size of 2×2=4 pixels or 8×8=64 pixels.
This allows estimation of the light source spectrum at a finer granularity than the entire image, thereby improving the estimation accuracy of the light source spectrum in the spatial direction.

<2. Second embodiment>
Next, a second embodiment will be described.
In the second embodiment, the light source spectrum is estimated not for each divided region as in the first embodiment, but rather as an average light source spectrum for the entire image.
In the second embodiment, all environmental spectra are estimated in the first embodiment, but fixed values (standard values) are used for at least some environmental parameters without estimation. Specifically, in this example, standard values are used for the amount of ozone, the amount of aerosol turbidity, and the amount of precipitated water vapor, excluding the amount of solar altitude and the solar angle, among the environmental parameters of the solar altitude, the solar angle, the amount of ozone, the aerosol turbidity, and the amount of precipitated water vapor, which are used to estimate the light source spectrum.

FIG. 9 is a block diagram showing an example of the configuration of a light source spectrum estimation system according to the second embodiment.
In the following description, parts that are similar to parts that have already been described will be given the same reference numerals and description thereof will be omitted.
The changes from the first embodiment are that the upward facing camera 3 is omitted and that an information processing device 1A is used instead of the information processing device 1.

FIG. 10 is a functional block diagram for explaining functions of an information processing device 1A according to the second embodiment.
Here, the CPU 11 included in the information processing device 1A is referred to as a CPU 11A.
FIG. 10 shows the functions of the CPU 11A in blocks.

The differences from the CPU 11 in the first embodiment are that the second environmental parameter estimation unit F3 and the sun/shade area determination unit F5 are omitted, and that a first environmental parameter estimation unit F2A is provided instead of the first environmental parameter estimation unit F2, and a light source spectrum estimation unit F1A is provided instead of the light source spectrum estimation unit F1.

The first environmental parameter estimation unit F2A estimates the solar altitude and solar angle based on the current position information and current time information input from the spectroscopic camera 2.

The light source spectrum estimation unit F1A inputs the standard value of the ozone amount (standard first environmental parameter in the figure), the standard values of the aerosol turbidity and precipitated water vapor amount (standard second environmental parameter in the figure), and the solar altitude and solar angle estimated by the first environmental parameter estimation unit F2A, and estimates the average light source spectrum for the entire image based on the spectroscopic image input from the spectroscopic camera 2.
Specifically, the average light source spectrum for the entire image is realized by using the average luminance value for the entire image of each of the M narrowband images as the "object spectrum" in the calculation of the error Ds described above. That is, in the calculation of the error Ds used in the processing of FIG. 8 described above, the average luminance value for the entire image of each of the M narrowband images input as the spectral image may be used as the "object spectrum."

The image processing unit F4A performs predetermined image processing based on the spectroscopic image input from the spectroscopic camera 2 and the average light source spectrum of the entire image obtained by the light source spectrum estimation unit F1A. For example, image processing by the image processing unit F4A may involve removing the average light source spectrum of the entire image from the spectroscopic image.

In the above example, standard values are used for the amount of ozone, the amount of aerosol turbidity, and the amount of precipitated water vapor, but the combination of environmental parameters for which standard values are used is not limited to this, and it is also possible to use a standard value for only the amount of ozone, or to use standard values for only the amount of aerosol turbidity and the amount of precipitated water vapor as the second environmental parameters, etc. Alternatively, it is also possible to use a standard value for only one of the amount of aerosol turbidity and the amount of precipitated water vapor.
When estimating at least one of the aerosol turbidity and the amount of precipitated water vapor, the system is configured to have an upward camera 3 as in the first embodiment.

3. Modifications
The embodiment is not limited to the specific example described above, and various modified configurations may be adopted.
For example, in the above example, a method for estimating the weighting coefficient b0 of the sunlight spectrum and the weighting coefficient b1 of the sky spectrum is exemplified, but it is also possible to use standard values without estimating these weighting coefficients b0 and b1.
FIG. 11 is a functional block diagram for explaining the functions of a CPU 11X corresponding to this case.
Here, an example of a functional configuration is shown assuming that standard values are used for each environmental parameter, other than the solar altitude and solar angle, namely, the amount of ozone, the aerosol turbidity, and the amount of precipitated water vapor, as in the second embodiment, and the average light source spectrum for the entire image is estimated.

The difference from the CPU 11A in the second embodiment is that it has a light source spectrum estimation unit F1X instead of the light source spectrum estimation unit F1A. The light source spectrum estimation unit F1X inputs weighting coefficients b0 and b1 ("standard degree of sunlight" in the figure) that are fixed values as standard values, and estimates the average light source spectrum of the entire image using these weighting coefficients b0 and b1. Specifically, the sunlight spectrum and sky spectrum are calculated using the solar altitude and solar angle estimated by the first environmental parameter estimation unit F2A, and the amount of ozone, aerosol turbidity, and amount of precipitated water vapor as standard values, and then the light source spectrum is estimated by substituting the standard values for b0 and b1 in the aforementioned "sunlight spectrum * b0 + sky spectrum * b1".

When standard values are used for the weighting coefficients b0 and b1, the amount of ozone can be estimated based on location information and time information, as in the first embodiment. Also, at least one of the aerosol turbidity and the amount of precipitated water vapor can be estimated based on an image captured by the upward camera 3, as in the first embodiment.
In addition, in the configuration shown in Figure 11, instead of using the standard values of b0 and b1 (standard degree of sunlight) to estimate the light source spectrum, a configuration can be adopted in which b0', b1' calculated by the second environmental parameter estimation unit F3 described in Figure 6 based on the image captured by the upward camera 3 is used.

In the explanation so far, the Bird model has been given as an example of a light source model used to estimate the light source spectrum, but other light source models can be used as the light source model as long as they express the light source spectrum, which is spectral information of an outdoor light source, by weighted addition of the solar spectrum and the sky spectrum, such as the SMARTS2 (Simple Model of the Atmospheric Radiative Transfer of Sunshine 2) light source model or the MODTRAN6 (MODerate resolution atmospheric TRANsmission 6) light source model.
The SMARTS2 light source model has a larger number of environmental parameters than the Bird model, and is capable of generating a more accurate spectrum.
The MODTRAN6 light source model has a larger number of environmental parameters than the SMARTS2 light source model, and can generate more accurate spectra.
In addition, even when the number of environmental parameters increases, for example, when adopting a light source model such as SMARTS2 or MODTRAN6, the environmental parameters required for light source spectrum estimation can be derived in a similar manner, in which parameters other than the target parameter are fixed and parameters that minimize the error due to the error function are derived for the target parameters.

In addition, in the above description, an example has been given in which the information processing device according to the present technology is configured as a device separate from the spectroscopic camera 2, but the information processing device according to the present technology may also be configured as an integrated device with the spectroscopic camera 2.

4. Summary of the embodiment
As described above, the information processing device (1 or 1A) as an embodiment includes a light source spectrum estimation unit (F1, F1A) that estimates a light source spectrum based on an object spectrum that is spectral information of a subject obtained by a spectroscopic camera, using a light source model that expresses a light source spectrum that is spectral information of an outdoor light source by weighted addition of a solar spectrum that is spectral information of light irradiated from the sun and a sky spectrum that is spectral information of light irradiated from the sky.
This makes it possible to estimate the light source spectrum by fitting a function serving as a light source model to the object spectrum detected by the spectroscopic camera.
Since it is possible to eliminate the restriction imposed by the conventional technology that the environment in which the light source spectrum can be estimated is limited to the assumed environment, it is possible to improve the robustness of the light source spectrum estimation.
In addition, since the light source model used represents the light source spectrum by weighted addition of the sunlight spectrum and the sky spectrum, it is possible to realize appropriate light source spectrum estimation that takes into account whether the object is in sunlight or shade.

Furthermore, in the information processing device as an embodiment, the light source spectrum estimation unit estimates the light source spectrum by estimating the parameters of the light source model and the object spectral reflectance, using the error (Ds) between the trial object spectrum, which is a spectrum calculated by multiplying the trial light source spectrum, which is a light source spectrum calculated by setting candidate values for the parameters of the light source model, by a candidate value of the object spectral reflectance indicating the spectral reflectance of the object as the subject, and the object spectrum as a reference value for estimation.
This makes it possible to estimate candidate values that satisfy the condition that the error is small, for example, that the error between the estimated object spectrum and the object spectrum detected by a spectroscopic camera is minimized, as the correct values for the parameters of the light source model and the object spectral reflectance, and thus makes it possible to appropriately estimate the light source spectrum according to the environment.

Furthermore, in the information processing device as the embodiment, the light source spectrum estimating unit estimates the weighting coefficient (b0) of the sunlight spectrum and the weighting coefficient (b1) of the sky spectrum as parameters of the light source model based on the error.
This makes it possible to estimate an appropriate light source spectrum according to whether the environment is in sunlight or shade.

Furthermore, in the information processing device of the embodiment (same as F1), a sun/shade area determination unit (same as F5) is provided that determines the sunshine area and the shade area of the subject imaged by the spectroscopic camera using the weighting coefficient of the sunlight spectrum and the weighting coefficient of the sky spectrum estimated by the light source spectrum estimation unit.
This makes it possible to appropriately determine sunny areas and shaded areas in an image captured by the spectroscopic camera.

In addition, in the information processing device as an embodiment, a first environmental parameter estimation unit (F2 or F2A) is provided that estimates at least a portion of the environmental parameters used by the light source spectrum estimation unit to calculate the solar spectrum and the sky spectrum based on at least one of the position information and the time information of the spectroscopic camera.
This makes it possible to estimate appropriate values for environmental parameters used in calculating the solar spectrum and sky spectrum that can be estimated based on the position information of the spectroscopic camera and current time information, and makes it possible to estimate appropriate solar spectrum and sky spectrum according to the actual environment.
Therefore, the estimation accuracy of the light source spectrum can be improved.

Furthermore, in the information processing device as an embodiment, the light source spectrum estimation unit uses environmental parameters such as the solar altitude, solar angle, and ozone amount to calculate the solar spectrum and the sky spectrum, and the first environmental parameter estimation unit estimates at least one of the solar altitude, solar angle, and ozone amount based on at least one of the position information of the spectroscopic camera and current time information.
The solar altitude, solar angle, and amount of ozone are environmental parameters that can be estimated based on location information and current time information.
Therefore, according to the above configuration, it is possible to estimate appropriate spectra according to the actual environment as the sunlight spectrum and the sky spectrum, thereby improving the accuracy of estimating the light source spectrum.

Furthermore, in the information processing device as an embodiment, the light source spectrum estimation unit is equipped with a second environmental parameter estimation unit (F3) that estimates at least some of the environmental parameters used in calculating the solar spectrum and sky spectrum based on an image captured by an upward-facing camera (F3) that captures an image of the sky.
This makes it possible to estimate appropriate values for the environmental parameters used to calculate the solar spectrum and sky spectrum that can be estimated based on images captured by the upward-facing camera, and to estimate appropriate solar spectrum and sky spectrum that are suited to the actual environment.
Therefore, the estimation accuracy of the light source spectrum can be improved.

In addition, in an information processing device as an embodiment, the light source spectrum estimation unit uses environmental parameters such as aerosol turbidity and precipitated water vapor amount to calculate the solar spectrum and sky spectrum, and the second environmental parameter estimation unit estimates at least one of the aerosol turbidity and the precipitated water vapor amount based on an image captured by an upward-facing camera.
As a result, when aerosol turbidity or precipitated water vapor amount is used to calculate the solar spectrum and sky spectrum, it becomes possible to appropriately estimate these environmental parameters based on images taken from the sky, making it possible to estimate appropriate solar spectrum and sky spectrum according to the actual environment.
Therefore, the estimation accuracy of the light source spectrum can be improved.

Furthermore, in the information processing device according to the embodiment, the light source spectrum estimation unit (F1A) uses a fixed value for one of the environmental parameters used to calculate the sunlight spectrum and the sky spectrum.
This makes it possible to use standard values corresponding to a standard environment for at least one of the environmental parameters used in calculating the solar spectrum and the sky spectrum.
Some environmental parameters have little effect on the estimated light source spectrum even when standard values are used. In addition, if fixed values are used for the environmental parameters, there is no need to perform estimation processing for the environmental parameters. Therefore, with the above configuration, it is possible to reduce the processing load required for estimating the light source spectrum while suppressing a decrease in the estimation accuracy of the light source spectrum.

Furthermore, in the information processing device of the embodiment, the light source spectrum estimation unit (F1) estimates the light source spectrum using a light source model for each divided region obtained by dividing the spectroscopic image obtained by the spectroscopic camera into regions each having a predetermined number of pixels.
This allows estimation of the light source spectrum at a finer granularity, for example pixel by pixel, rather than for the entire image.
Therefore, it is possible to improve the estimation accuracy of the light source spectrum in the spatial direction.

Furthermore, in the information processing device according to the embodiment, the light source spectrum estimating unit (F1A) estimates an average light source spectrum for the entire spectroscopic image obtained by the spectroscopic camera.
The above configuration is suitable for cases where information on the average light source spectrum over the entire subject is required.

Furthermore, the information processing device according to the embodiment includes an image processing unit (F4 or F4A) that performs light source cancellation processing on the spectroscopic image obtained by the spectroscopic camera based on the light source spectrum estimated by the light source spectrum estimation unit.
As a result, the light source cancellation process is performed based on the appropriate light source spectrum estimated by the light source spectrum estimation unit.
Therefore, the accuracy of the light source cancellation process is improved.

Furthermore, the information processing device according to the embodiment includes an image processing unit (F4) that performs semantic segmentation processing based on the light source spectrum estimated by the light source spectrum estimation unit and the spectroscopic image obtained by the spectroscopic camera.
As a result, semantic segmentation processing is performed on the subject based on the appropriate light source spectrum estimated by the light source spectrum estimation unit.
Therefore, the accuracy of the semantic segmentation process is improved.

An information processing method as an embodiment is an information processing method in which an information processing device estimates a light source spectrum based on an object spectrum, which is spectral information of a subject obtained by a spectroscopic camera, using a light source model that represents a light source spectrum, which is spectral information of an outdoor light source, by weighted addition of a solar spectrum, which is spectral information of light irradiated from the sun, and a sky spectrum, which is spectral information of light irradiated from the sky.
With such an information processing method, it is possible to obtain the same functions and effects as those of the information processing device according to the above embodiment.

Here, as an embodiment, a program for realizing the functions of the light source spectrum estimation unit F1 or F1A etc. described with reference to FIG. 8 etc. in, for example, a CPU, a DSP etc., or a device including these, can be considered.
That is, the program of the embodiment is a program readable by a computer device, and causes the computer device to realize a function of estimating a light source spectrum based on an object spectrum, which is spectral information of a subject obtained by a spectroscopic camera, using a light source model that expresses a light source spectrum, which is spectral information of an outdoor light source, by weighted addition of a solar spectrum, which is spectral information of light irradiated from the sun, and a sky spectrum, which is spectral information of light irradiated from the sky.
By using such a program, the functions of the light source spectrum estimation unit F1 or F1A or the like described above can be realized in a device such as the information processing device 1 or 1A or the like.

The above-mentioned programs can be recorded in advance in a recording medium such as an HDD (Hard Disc Drive) or SSD (Solid State Drive) built into a device such as a computer device, or in a ROM within a microcomputer having a CPU.
Alternatively, the software may be temporarily or permanently stored (recorded) on a removable recording medium such as a flexible disk, a CD-ROM (Compact Disc Read Only Memory), an MO (Magneto Optical) disk, a DVD (Digital Versatile Disc), a Blu-ray Disc (registered trademark), a magnetic disk, a semiconductor memory, a memory card, etc. Such removable recording media may be provided as so-called package software.
Such a program can be installed in a personal computer or the like from a removable recording medium, or can be downloaded from a download site via a network such as a LAN or the Internet.

Furthermore, such a program is suitable for providing the light source spectrum estimation method of the embodiment in a wide range of applications. For example, by downloading the program to a personal computer, a portable information processing device, a mobile phone, a game device, a video device, a PDA (Personal Digital Assistant), etc., the personal computer, etc. can be made to function as a device that realizes the light source spectrum estimation method of the present disclosure.

It should be noted that the effects described in this specification are merely examples and are not limiting, and other effects may also be obtained.

<5. This Technology>
The present technology can also be configured as follows.
(1)
An information processing device comprising: a light source spectrum estimation unit that estimates the light source spectrum based on an object spectrum that is spectral information of a subject obtained by a spectroscopic camera, using a light source model that expresses a light source spectrum that is spectral information of an outdoor light source by weighted addition of a solar spectrum that is spectral information of light irradiated from the sun and a sky spectrum that is spectral information of light irradiated from the sky.
(2)
The light source spectrum estimation unit
The information processing device described in (1) estimates the light source spectrum by estimating the parameters of the light source model and the object spectral reflectance, using an error between a trial object spectrum, which is a spectrum calculated by multiplying a trial light source spectrum, which is a light source spectrum calculated by setting candidate values for parameters of the light source model, by a candidate value of object spectral reflectance indicating the spectral reflectance of the object as the subject, and the object spectrum, as a reference value for estimation.
(3)
The light source spectrum estimation unit
The information processing device according to (2), further comprising: estimating a weighting coefficient of the sunlight spectrum and a weighting coefficient of the sky spectrum as parameters of the light source model based on the error.
(4)
The information processing device according to (3), further comprising a sun/shade area determination unit that determines a sun area and a shade area of a subject imaged by the spectroscopic camera using the weighting coefficient of the sunlight spectrum and the weighting coefficient of the sky spectrum estimated by the light source spectrum estimation unit.
(5)
The information processing device according to any one of (1) to (4), further comprising a first environmental parameter estimation unit that estimates at least a portion of the environmental parameters used by the light source spectrum estimation unit to calculate the solar spectrum and the sky spectrum, based on at least one of position information of the spectroscopic camera and current time information.
(6)
the light source spectrum estimation unit uses environmental parameters such as a solar altitude, a solar angle, and an ozone amount to calculate the solar spectrum and the sky spectrum;
The first environmental parameter estimation unit
The information processing device according to (5), wherein at least one of the solar altitude, the solar angle, and the amount of ozone is estimated based on at least one of position information of the spectroscopic camera and current time information.
(7)
The information processing device according to any one of (1) to (6), further comprising a second environmental parameter estimation unit that estimates at least a portion of the environmental parameters used by the light source spectrum estimation unit to calculate the solar spectrum and the sky spectrum based on an image captured by an upward-facing camera that captures an image of the sky.
(8)
the light source spectrum estimation unit uses environmental parameters, such as aerosol turbidity and precipitated water vapor, in calculating the solar spectrum and the sky spectrum;
The second environment parameter estimation unit
The information processing device according to (7), wherein at least one of the aerosol turbidity and the amount of precipitated water vapor is estimated based on an image captured by the upward camera.
(9)
The light source spectrum estimation unit
The information processing device according to any one of (1) to (4), wherein a fixed value is used for any of the environmental parameters used in the calculation of the solar spectrum and the sky spectrum.
(10)
The light source spectrum estimation unit
The information processing device according to any one of (1) to (9), further comprising: a light source model for estimating the light source spectrum for each divided region obtained by dividing a spectroscopic image obtained by the spectroscopic camera into regions each having a predetermined number of pixels.
(11)
The light source spectrum estimation unit
The information processing device according to any one of (1) to (9), further comprising: estimating an average light source spectrum of an entire spectroscopic image obtained by the spectroscopic camera.
(12)
The information processing device according to any one of (1) to (11), further comprising an image processing unit that performs a light source cancellation process on a spectroscopic image obtained by the spectroscopic camera based on the light source spectrum estimated by the light source spectrum estimation unit.
(13)
The information processing device according to any one of (1) to (11), further comprising an image processing unit that performs semantic segmentation processing based on the light source spectrum estimated by the light source spectrum estimation unit and a spectroscopic image obtained by the spectroscopic camera.
(14)
An information processing device,
An information processing method for estimating a light source spectrum, which is spectral information of an outdoor light source, based on an object spectrum, which is spectral information of a subject obtained by a spectroscopic camera, using a light source model that represents a light source spectrum, which is spectral information of an outdoor light source, by weighted addition of a solar spectrum, which is spectral information of light irradiated from the sun, and a sky spectrum, which is spectral information of light irradiated from the sky.
(15)
A computer readable program,
The program causes the computer device to realize a function of estimating the light source spectrum based on the object spectrum, which is spectral information of a subject obtained by a spectroscopic camera, using a light source model that expresses the light source spectrum, which is spectral information of an outdoor light source, by weighted addition of a solar spectrum, which is spectral information of light irradiated from the sun, and a sky spectrum, which is spectral information of light irradiated from the sky.

REFERENCE SIGNS LIST 1 Information processing device 2 Spectroscopic camera 3 Upward camera 4 Spectroscopic sensor 4a Pixel array unit 5 Spectroscopic image generating unit 5a Demosaic unit 5b Narrowband image generating unit 6 Control unit 7 Timekeeping unit 8 GNSS sensor 9 Communication unit Px Pixel Pu Spectroscopic pixel unit 11, 11A CPU
12 ROM
13 RAM
14 Bus 15 Input/Output Interface 16 Input Unit 17 Display Unit 18 Audio Output Unit 19 Memory Unit 20 Communication Unit 21 Drive 22 Removable Recording Medium F1, F1A Light Source Spectrum Estimation Unit F2, F2A First Environmental Parameter Estimation Unit F3, F3A Second Environmental Parameter Estimation Unit F4, F4A Image Processing Unit

Claims (15)

An information processing device comprising: a light source spectrum estimation unit that estimates the light source spectrum based on an object spectrum that is spectral information of a subject obtained by a spectroscopic camera, using a light source model that expresses a light source spectrum that is spectral information of an outdoor light source by weighted addition of a solar spectrum that is spectral information of light irradiated from the sun and a sky spectrum that is spectral information of light irradiated from the sky. The light source spectrum estimation unit
2. The information processing device according to claim 1, wherein the light source spectrum is estimated by estimating the parameters of the light source model and the object spectral reflectance, using an error between a trial object spectrum, which is a spectrum calculated by multiplying a trial light source spectrum, which is a light source spectrum calculated by setting candidate values for parameters of the light source model, by a candidate value of object spectral reflectance indicating the spectral reflectance of the object as the subject, and the object spectrum as a reference value for estimation. The light source spectrum estimation unit
The information processing apparatus according to claim 2 , further comprising: estimating a weighting coefficient of the sunlight spectrum and a weighting coefficient of the sky spectrum as parameters of the light source model based on the error. The information processing device according to claim 3 , further comprising a sun/shade area determination unit that determines a sunshine area and a shade area of a subject imaged by the spectroscopic camera using the weighting coefficient of the sunlight spectrum and the weighting coefficient of the sky spectrum estimated by the light source spectrum estimation unit. 2. The information processing device according to claim 1, further comprising a first environmental parameter estimating unit that estimates at least a part of the environmental parameters used by the light source spectrum estimating unit to calculate the solar spectrum and the sky spectrum based on at least one of position information and time information of the spectroscopic camera. the light source spectrum estimation unit uses environmental parameters such as a solar altitude, a solar angle, and an ozone amount to calculate the solar spectrum and the sky spectrum;
The first environmental parameter estimation unit
The information processing apparatus according to claim 5 , wherein at least one of the solar altitude, the solar angle, and the amount of ozone is estimated based on at least one of position information of the spectroscopic camera and current time information. The information processing device according to claim 1 , further comprising a second environmental parameter estimation unit that estimates at least a portion of the environmental parameters used by the light source spectrum estimation unit to calculate the solar spectrum and the sky spectrum based on an image captured by an upward-facing camera that captures an image of the sky. the light source spectrum estimation unit uses environmental parameters, such as aerosol turbidity and precipitated water vapor, in calculating the solar spectrum and the sky spectrum;
The second environment parameter estimation unit
The information processing device according to claim 7 , wherein at least one of the aerosol turbidity and the amount of precipitated water vapor is estimated based on an image captured by the upward-facing camera. The light source spectrum estimation unit
The information processing apparatus according to claim 1 , wherein a fixed value is used for any one of the environmental parameters used in the calculation of the solar spectrum and the sky spectrum. The light source spectrum estimation unit
The information processing apparatus according to claim 1 , further comprising: a spectroscopic image obtained by dividing the spectroscopic image obtained by the spectroscopic camera into regions each having a predetermined number of pixels; The light source spectrum estimation unit
The information processing device according to claim 1 , further comprising: an optical fiber estimating unit configured to estimate an average light source spectrum of an entire spectroscopic image obtained by the spectroscopic camera. The information processing device according to claim 1 , further comprising an image processing unit that performs a light source cancellation process on the spectroscopic image obtained by the spectroscopic camera, based on the light source spectrum estimated by the light source spectrum estimation unit. The information processing device according to claim 1 , further comprising an image processing unit that performs a semantic segmentation process based on the light source spectrum estimated by the light source spectrum estimation unit and a spectroscopic image obtained by the spectroscopic camera. An information processing device,
An information processing method for estimating a light source spectrum, which is spectral information of an outdoor light source, based on an object spectrum, which is spectral information of a subject obtained by a spectroscopic camera, using a light source model that represents a light source spectrum, which is spectral information of an outdoor light source, by weighted addition of a solar spectrum, which is spectral information of light irradiated from the sun, and a sky spectrum, which is spectral information of light irradiated from the sky. A computer readable program,
The program causes the computer device to realize a function of estimating the light source spectrum based on the object spectrum, which is spectral information of a subject obtained by a spectroscopic camera, using a light source model that expresses the light source spectrum, which is spectral information of an outdoor light source, by weighted addition of a solar spectrum, which is spectral information of light irradiated from the sun, and a sky spectrum, which is spectral information of light irradiated from the sky.

Images