WO2025033195A1 - Detection device, learning device, detection method, learning method, and recording medium - Google Patents

Abstract

A detection device according to the present disclosure comprises a detection means for detecting an object to be detected on the basis of a composite image of the object to be detected obtained by synthesizing: a captured image to be processed which results from imaging a region of interest; and a two-dimensional image to be processed which was generated on the basis of reflected wave information indicating reflected waves of electromagnetic waves projected on the region of interest. The detection means detects the object to be detected on the basis of a learning model obtained by learning a plurality of composite images for learning obtained by synthesizing captured images for learning and two-dimensional images for learning generated on the basis of reflected wave information describing reflected waves of electromagnetic waves.

Description

Detection device, learning device, detection method, learning method, and recording medium

This disclosure relates to a detection device, a learning device, a detection method, and a program.

Technology related to the present disclosure is disclosed in Patent Document 1. Patent Document 1 discloses technology for detecting objects that reflect a lot of millimeter wave radar, such as metals, based on the reflection intensity of the millimeter wave radar.

International Publication No. 2017/057058

The technology disclosed in Patent Document 1 detects all objects that reflect a lot of millimeter wave radar, such as metals. With the technology disclosed in Patent Document 1, it is difficult to accurately detect only some of the objects that reflect a lot of millimeter wave radar.

In view of the above-mentioned problems, one example of the objective of the present disclosure is to provide a detection device, a learning device, a detection method, a learning method, and a program for detecting a desired object with high accuracy.

According to the present disclosure,
A detection device is provided that has a detection means for detecting a detection target based on a composite image of a processing target obtained by combining a captured image of the processing target obtained by capturing an image of the target area and a two-dimensional image of the processing target generated based on reflected wave information indicating the reflected waves of electromagnetic waves irradiated to the target area.

Further, according to the present disclosure,
One or more computers
A detection method is provided for detecting a detection target based on a processing target composite image obtained by combining a processing target captured image of a target area and a processing target two-dimensional image generated based on reflected wave information indicating the reflected waves of electromagnetic waves irradiated to the target area.

Further, according to the present disclosure,
Computer,
A program is provided that functions as a detection means for detecting a detection target based on a composite image of a processing target obtained by combining a captured image of the processing target obtained by capturing an image of the target area and a two-dimensional image of the processing target generated based on reflected wave information indicating the reflected waves of electromagnetic waves irradiated to the target area.

Further, according to the present disclosure,
A learning device is provided having a learning means for learning a learning model using a plurality of learning composite images obtained by combining a learning captured image with a learning two-dimensional image generated based on reflected wave information indicating a reflected electromagnetic wave.

Further, according to the present disclosure,
One or more computers
A learning method is provided for learning a learning model using a plurality of learning composite images obtained by combining learning captured images with learning two-dimensional images generated based on reflected wave information indicating reflected electromagnetic waves.

Further, according to the present disclosure,
Computer,
A program is provided that functions as a learning means for learning a learning model using a plurality of learning composite images obtained by combining learning captured images with learning two-dimensional images generated based on reflected wave information indicating reflected electromagnetic waves.

According to one aspect of the present disclosure, a detection device, a learning device, a detection method, a learning method, and a program for detecting a desired object with high accuracy are realized.

FIG. 2 is a diagram illustrating an example of a functional block diagram of a learning device according to the present disclosure. 11 is a flowchart illustrating an example of a process flow of a learning device according to the present disclosure. FIG. 2 is a diagram showing an example of a functional block diagram of a detection device according to the present disclosure. 10 is a flowchart illustrating an example of a process flow of a detection device according to the present disclosure. FIG. 11 is a diagram for explaining a process of a comparative example. FIG. 2 is a diagram for explaining processing of a learning device according to the present disclosure. FIG. 2 is a diagram illustrating an example of a hardware configuration of a learning device and a detection device according to the present disclosure. FIG. 13 is a diagram illustrating another example of a functional block diagram of a learning device according to the present disclosure. 1 is a diagram for explaining an example of processing executed by a learning device and a detection device according to the present disclosure. FIG. 11 is a diagram for explaining another example of the processing executed by the learning device and the detection device according to the present disclosure. FIG. 13 is a flowchart showing another example of the processing flow of the learning device according to the present disclosure. FIG. 13 is a diagram showing another example of a functional block diagram of a detection device according to the present disclosure. 10 is a flowchart showing another example of the processing flow of the detection device according to the present disclosure.

Below, an embodiment of the present disclosure will be described with reference to the drawings. In this disclosure, the drawings relate to one or more embodiments. In addition, in all drawings, similar components are given similar reference symbols, and descriptions will be omitted as appropriate.

First Embodiment
Fig. 1 is a functional block diagram showing an overview of the learning device 20. Fig. 2 is a flowchart showing an example of the flow of processing executed by the learning device 20.

As shown in FIG. 1, the learning device 20 has a learning unit 21. This functional unit executes the process in FIG. 2.

The learning unit 21 trains the learning model using multiple synthetic learning images that are a combination of "learning captured images" and "learning two-dimensional images generated based on reflected wave information indicating reflected electromagnetic waves" (S10).

The learning device 20, which uses such synthetic learning images to train a learning model, can generate a learning model that has learned both the "features of the captured image" and the "features of the reflected wave information indicating the reflected waves of electromagnetic waves." This learning model can perform object detection and object recognition based on both the "features of the captured image" and the "features of the reflected wave information indicating the reflected waves of electromagnetic waves." By learning not just one of the "features of the captured image" and the "features of the reflected wave information indicating the reflected waves of electromagnetic waves," but both, the accuracy of object detection and object recognition by the learning model is improved.

The learning device 20 also integrates the "features of the captured image" and the "features of the reflected wave information indicating the reflected electromagnetic waves" by a unique method of generating a synthetic image, and learns them together. Rather than learning the "features of the captured image" and the "features of the reflected wave information indicating the reflected electromagnetic waves" individually, learning them together as a synthetic image that is integrated by a unique method is expected to have a synergistic effect and improve the accuracy of object detection and object recognition.

Second Embodiment
Fig. 3 is a functional block diagram showing an overview of the detection device 10. Fig. 4 is a flowchart showing an example of the flow of processing executed by the detection device 10.

As shown in FIG. 3, the detection device 10 has a detection unit 11. This functional unit executes the process in FIG. 4.

The detection unit 11 detects the detection target based on a composite image of the processing target, which is a combination of a "processing target captured image of the target area" and a "processing target two-dimensional image generated based on reflected wave information indicating the reflected waves of the electromagnetic waves irradiated to the target area" (S20).

The detection device 10, which detects the detection target using such a composite image to be processed, can detect the detection target based on both the "characteristics of the captured image" and the "characteristics of the reflected wave information indicating the reflected wave of the electromagnetic wave." By using not just one but both of the "characteristics of the captured image" and the "characteristics of the reflected wave information indicating the reflected wave of the electromagnetic wave," the detection accuracy of the detection target is improved.

In addition, the detection device 10 integrates the "features of the captured image" and the "features of the reflected wave information indicating the reflected waves of the electromagnetic waves" by a unique method of generating a composite image, and detects the detection target based on the composite image. Rather than processing the "features of the captured image" and the "features of the reflected wave information indicating the reflected waves of the electromagnetic waves" separately, they are integrated by a unique method and processed together as a composite image, which is expected to have a synergistic effect and improve detection accuracy.

Third Embodiment
"overview"
The learning device 20 of the third embodiment is a specific embodiment of the configuration of the learning device 20 of the first embodiment.

In recent years, object recognition models that use language models to recognize detection targets and object detection models that use language models to detect detection targets have become known. These object recognition models and object detection models are models that can represent images and language in a joint embedding space.

The object recognition model and the object detection model are generated by learning the relationship between the results of object recognition/detection obtained by a technology such as a neural network and language related to the object (description or expression of the object). Based on the object recognition model and the object detection model, the object indicated by the search criteria (text) can be recognized/detected in the captured image. The technology is disclosed in, for example, the following documents.
"Radford, Alec, et al. "Learning transferable visual models from natural language supervision."International conference on machine learning. PMLR, 2021."
"Li, Liunian Harold, et al. "Grounded language-image pre-training." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022."

The object recognition model and object detection model disclosed in the above-mentioned documents are generated by learning the correlation between language related to objects (descriptions and expressions of objects) and captured images, as shown in Figure 5.

In response to this, the learning device 20 generates a learning model by learning the correlation between language relating to an object (description or expression of the object) and a synthetic image, as shown in FIG. 6. The synthetic image is an image generated by synthesizing a "captured image" and a "two-dimensional image based on reflected wave information indicating reflected electromagnetic waves." In this respect, the learning device 20 differs from the object recognition model that recognizes a detection target using the language model disclosed in the above-mentioned document and the object detection model that detects a detection target using a language model.

The configuration of the learning device 20 is explained in detail below.

"Hardware Configuration"
First, an example of the hardware configuration of the learning device 20 will be described. Each functional unit of the learning device 20 is realized by any combination of hardware and software. Those skilled in the art will understand that there are various variations in the realization method and device. The software includes programs that are stored in the device before it is shipped, and programs downloaded from recording media such as CDs (Compact Discs) and servers on the Internet.

FIG. 7 is a block diagram illustrating an example of the hardware configuration of a learning device 20. As shown in FIG. 7, the learning device 20 has a processor 1A, a memory 2A, an input/output interface 3A, a peripheral circuit 4A, and a bus 5A. The peripheral circuit 4A includes various modules. The learning device 20 does not have to have the peripheral circuit 4A. The learning device 20 may be composed of multiple devices that are physically and/or logically separated. In this case, each of the multiple devices can have the above hardware configuration.

The bus 5A is a data transmission path for the processor 1A, memory 2A, peripheral circuit 4A, and input/output interface 3A to send and receive data to each other. The processor 1A is an arithmetic processing device such as a CPU or a GPU (Graphics Processing Unit). The memory 2A is a memory such as a RAM (Random Access Memory) or a ROM (Read Only Memory). The input/output interface 3A includes interfaces for acquiring information from an input device, an external device, an external server, an external sensor, a camera, etc., and interfaces for outputting information to an output device, an external device, an external server, etc. The input/output interface 3A also includes an interface for connecting to a communication network such as the Internet. Examples of input devices include a keyboard, a mouse, a microphone, a physical button, a touch panel, etc. Examples of output devices include a display, a speaker, a printer, a mailer, etc. The processor 1A can issue commands to each module and perform calculations based on the results of those calculations.

"Function Configuration"
Next, a detailed description will be given of the functional configuration of the learning device 20. Fig. 8 shows an example of a functional block diagram of the learning device 20. As shown in the figure, the learning device 20 has a learning unit 21, a learning image acquisition unit 22, a learning reflected wave processing unit 23, a learning synthesis unit 24, and a language input unit 25.

The learning image acquisition unit 22 acquires learning images.

"Learning images" are captured images that are used as learning data when the learning device 20 learns a learning model.

A "captured image" is an image generated by taking a picture with a camera. A camera detects visible light and converts it into an image. Note that a camera may also detect other types of light, such as infrared or ultraviolet light, and convert it into an image. A captured image may be a still image or a video. A captured image may be provided with information indicating the date and time of capture and the location where the image was taken. If the captured image is a video, information that identifies the date and time of capture and the location where the image was taken may be provided for each frame image.

In one example, the camera is mounted on the measuring device. The measuring device is a device that can be moved. The measuring device may be, for example, an air vehicle such as a drone, or may be a device that moves on land. The measuring device can be moved automatically or by remote control. The measuring device can move automatically, for example, along a route registered in advance. As the measuring device moves, it captures an image of a specified data collection area with the camera. The data collection area is an area selected for the purpose of collecting learning data, detecting a detection target, etc. The data collection area may be a part of the ground, a part of a building, or something else.

In another example, the camera is carried by a worker who moves around and takes pictures of the data collection area with the camera.

The captured images generated by the camera are input to the learning device 20 by any means. For example, the camera and the learning device 20 may be configured to be able to communicate with each other. The camera may then transmit the captured images generated to the learning device 20 via the communication means. The transmission of the captured images from the camera to the learning device 20 may be performed by real-time processing or batch processing.

In addition, the captured images generated by the camera may be stored in any storage device. The storage device may be provided in the camera, or in an external device configured to be able to communicate with the camera. The captured images stored in the storage device may then be input to the learning device 20 at any time and by any means.

The learning reflected wave processing unit 23 generates a learning two-dimensional image based on the reflected wave information indicating the reflected electromagnetic waves.

The "electromagnetic wave" is, for example, a millimeter wave, and an example of its wavelength is 0.3 GHz or more and 300 GHz or less. However, the band of the electromagnetic wave is not limited to millimeter waves. The electromagnetic wave may be near infrared rays, far infrared rays, etc.

"Reflected waves" are electromagnetic waves that are reflected by objects in the irradiated area. If an object that reflects electromagnetic waves is present in that area, the strength of the reflected waves will be high. Objects that reflect electromagnetic waves are often made primarily of metal.

The irradiation of electromagnetic waves and the reception of reflected waves are achieved by an electromagnetic wave transmitting/receiving device. The electromagnetic wave transmitting/receiving device includes an electromagnetic wave transmitting unit that transmits electromagnetic waves, and an electromagnetic wave receiving unit that receives reflected waves. The electromagnetic wave transmitting/receiving device may include multiple electromagnetic wave receiving units, for example, two. These multiple electromagnetic wave receiving units are spaced apart from each other, and receive reflected waves of the electromagnetic waves irradiated by the same electromagnetic wave transmitting unit. In this way, the accuracy of detecting the position of an object is increased.

The transmission method used by the electromagnetic wave transmitter is, for example, one of FMCW (Frequency Modulated Continuous Wave), pulse, CW (Continuous Wave) Doppler, two-frequency CW, and pulse compression, but may be other than these.

In one example, the measuring device equipped with the camera described above further includes an electromagnetic wave transmitting and receiving device. As the measuring device moves, it captures images of the data collection area with the camera, and uses the electromagnetic wave transmitting and receiving device to irradiate electromagnetic waves onto the data collection area and receive the reflected waves.

In another example, the electromagnetic wave transmitting and receiving device is carried by a worker. While moving around carrying a camera and an electromagnetic wave transmitting and receiving device, the worker photographs the data collection area with the camera, and irradiates the data collection area with electromagnetic waves using the electromagnetic wave transmitting and receiving device and receives the reflected waves.

"Reflected wave information" is generated by the electromagnetic wave transmitting/receiving device. More specifically, the reflected wave information is generated based on the results of reception of the reflected wave by the electromagnetic wave receiving unit. The reflected wave information includes, for example, time series information on the strength of the reflected wave. This time series information includes a combination of the date and time the reflected wave was received and the strength of the reflected wave at that time. If multiple electromagnetic wave receiving units are provided, the reflected wave information is generated separately for each of the multiple electromagnetic wave receiving units.

The electromagnetic wave transmitting/receiving device may generate location information indicating the location of the electromagnetic wave transmitting/receiving device. The location information is indicated by, for example, latitude and longitude. The location information may include altitude in addition to latitude and longitude. This location information may be generated using, for example, GPS, or may be generated using other methods, such as SLAM (Simultaneous Localization and Mapping). The electromagnetic wave transmitting/receiving device may add location information of the electromagnetic wave transmitting/receiving device at the time the reflected wave was received to the above-mentioned reflected wave information.

The above-mentioned location information may be information separate from the reflected wave information. In this case, the location information is time-series information on the location of the electromagnetic wave transmitting/receiving device. This time-series information includes a combination of date and time and the location of the electromagnetic wave transmitting/receiving device at that date and time.

The reflected wave information generated by the electromagnetic wave transmitting/receiving device is input to the learning device 20 by any means. For example, the electromagnetic wave transmitting/receiving device and the learning device 20 may be configured to be able to communicate with each other. The electromagnetic wave transmitting/receiving device may then transmit the generated reflected wave information to the learning device 20 via the communication means. The transmission of the reflected wave information from the electromagnetic wave transmitting/receiving device to the learning device 20 may be performed by real-time processing or by batch processing.

In addition, the reflected wave information generated by the electromagnetic wave transmitting/receiving device may be stored in any storage device. The storage device may be provided within the electromagnetic wave transmitting/receiving device, or may be provided within an external device configured to be able to communicate with the electromagnetic wave transmitting/receiving device. The reflected wave information stored in the storage device may then be input to the learning device 20 at any time and by any means.

"2D training images" are 2D images generated for training purposes.

A "two-dimensional image" is a two-dimensional image generated based on the reflected wave information described above. Below, the process of generating a two-dimensional image from reflected wave information is explained.

First, the learning reflected wave processing unit 23 generates three-dimensional information by processing the reflected wave information. In detail, the reflected wave information includes a time series signal of the reflected wave intensity and the position of the electromagnetic wave transmitting and receiving device. For example, the learning reflected wave processing unit 23 calculates the distance from the electromagnetic wave receiving unit to the reflection point that is the origin of the reflected wave by performing FFT (Fast Fourier Transform) multiple times on the reflected wave that constitutes this time series signal. If there are multiple electromagnetic wave receiving units, the learning reflected wave processing unit 23 performs this processing for each electromagnetic wave receiving unit. Then, the learning reflected wave processing unit 23 calculates an estimate of the reflected wave intensity for at least one first point included in the three-dimensional space corresponding to the data collection area by integrating multiple distances based on reflected waves measured by different electromagnetic wave receiving units at the same timing. The learning reflected wave processing unit 23 then performs this processing on the reflected waves measured at multiple timings to calculate an estimate of the intensity of the reflected waves for each of the multiple first points, and treats these as three-dimensional information. Note that this estimate can be considered as a value indicating the possibility that an object that reflects electromagnetic waves is present at the first point. Hereinafter, this value will be referred to as the first value. However, the method of generating the three-dimensional information, for example the method of generating the first value, is not limited to this example.

Then, the learning reflected wave processing unit 23 generates two-dimensional information by projecting the three-dimensional information onto a specified plane. Hereinafter, this specified plane will be referred to as the projection plane. It is preferable that the angle that the projection plane makes with the surface of the data collection area (the ground, the outer surface of a building, etc.) is 10° or less. In other words, it is preferable that the projection plane is horizontal to the surface of the data collection area.

9 is a diagram for explaining an example of processing performed by the learning reflected wave processing unit 23. The learning reflected wave processing unit 23 identifies a plurality of first points corresponding to the second point. For example, the learning reflected wave processing unit 23 determines a plurality of first points that overlap with the second point when viewed from a direction perpendicular to the projection surface as the first points corresponding to the second point. Next, the learning reflected wave processing unit 23 identifies a first value corresponding to each of the identified plurality of first points, and generates a second value indicating the possibility that an object that reflects electromagnetic waves is present at the second point using the first value. The second value can be a statistical value (maximum value, minimum value, average value, mode, median, etc.) of the plurality of first values. Alternatively, the first value corresponding to the first point closest to the data collection area among the first points whose first values exceed a reference value may be determined as the second value. Then, the learning reflected wave processing unit 23 determines the second value for each second point as two-dimensional information. From this two-dimensional information, for example, black and white image data (two-dimensional image) can be generated. Note that the two-dimensional image may represent the second value of the second point in black and white, or in other colors or other means.

As a variant, the learning device 20 may be configured to learn the learning model by taking into account at least one of the geological information of the data collection area and the weather information at the time the reflected waves were generated. For example, the geology of a particular area of the data collection area may be more conducive to generating reflected waves. In addition, depending on the weather, water or snow may accumulate on the ground surface of the data collection area, affecting the reflected waves. The learning reflected wave processing unit 23 can reflect this effect in the generation of the two-dimensional image.

For example, the learning reflected wave processing unit 23 may generate three-dimensional information or two-dimensional information by multiplying a parameter corresponding to the geology of the data collection area by a first value or a second value. This parameter is set in advance. The learning reflected wave processing unit 23 may also generate three-dimensional information or two-dimensional information by multiplying a parameter corresponding to the weather at the time of measurement by a first value or a second value. This parameter is also set in advance.

In addition, the geological information and weather information may be input to the learning device 20 by, for example, a user of the learning device 20, or may be obtained by the learning device 20 from a database in which the geological information and weather information are stored.

The modified examples described here can also be applied to the detection device 10, which will be described in detail in the following embodiment.

Returning to FIG. 8, the learning synthesis unit 24 synthesizes the learning photographed image and the learning two-dimensional image to generate a learning synthesized image. It is preferable that the learning photographed image and the learning two-dimensional image are of similar size.

As shown in FIG. 10, the learning synthesis unit 24 blends (synthesizes) the learning captured image and the learning two-dimensional image at a predetermined blend ratio to generate a learning synthetic image.

The training synthesis unit 24 may generate a training synthetic image from a set of training photographic images and a training two-dimensional image. In this case, the Blend ratio may be determined in advance.

In addition, the learning synthesis unit 24 may generate multiple learning composite images having different blend ratios of the learning photographed images and learning two-dimensional images from a set of learning photographed images and learning two-dimensional images. In this way, it is preferable that a large number of learning composite images can be generated from a set of learning photographed images and learning two-dimensional images. It is also preferable that learning composite images with various blend ratios can be learned.

In addition, the learning synthesis unit 24 may generate a learning synthetic image in which the blending ratio differs for each partial region in the image. For example, the learning synthesis unit 24 may vary the blending ratio for each object region detected using widely known object detection technology, semantic segmentation, or the like. This is preferable because it makes it possible to generate a large number of learning synthetic images from a set of learning photographed images and learning two-dimensional images. It is also preferable because it makes it possible to learn from learning synthetic images with various blending ratios.

The learning synthesis unit 24 synthesizes a learning photographed image and a learning two-dimensional image containing information about the same object to generate a learning composite image. To achieve this, the learning synthesis unit 24 may synthesize a learning photographed image and a learning two-dimensional image that have been photographed/measured at the same time. Alternatively, the learning synthesis unit 24 may synthesize a learning photographed image and a learning two-dimensional image that have been photographed/measured at the same position. Alternatively, the learning synthesis unit 24 may synthesize a learning photographed image and a learning two-dimensional image that have been photographed/measured at the same time and in the same position. Here, the "same timing" may be a perfect match, or may be a concept that allows for a time difference of up to a few seconds. Also, here, the "same position" may be a perfect match, or may be a concept that allows for a position difference of up to a few centimeters to a few meters.

The learning synthesis unit 24 may synthesize the images after performing a process to approximately match the position of an object in the learning captured image with the position of the object in the learning two-dimensional image. However, this process is not essential. It is required that the learning captured image and the learning two-dimensional image to be synthesized together contain information of the same object, but it is not essential that the positions of the objects in the images match. The inventors have confirmed that sufficient learning effects and detection results can be obtained even if the positions of the objects in the images do not match. However, if the positions of the objects in the images match, it is expected that more favorable learning effects and detection results can be obtained compared to when they do not. The learning synthesis unit 24 may, for example, perform the following misalignment correction process 1 or 2.

"Misalignment correction process 1"
In one example, the misalignment is corrected by user input. The learning synthesis unit 24 simultaneously displays the captured image and the two-dimensional image as shown in FIG. 10 on the screen. The user shifts the position of at least one of the captured image and the two-dimensional image so that the positions of an object shown in each image overlap each other. The user can specify which object in the captured image corresponds to the second set of points in the two-dimensional image based on the shape of the object shown in the captured image and the shape of the second set of points in the two-dimensional image.

"Misalignment correction process 2"
The learning synthesis unit 24 may transform the captured image into the same coordinate system as the two-dimensional image based on the positions of the camera and the electromagnetic wave transmitting and receiving device. When the camera and the electromagnetic wave transmitting and receiving device are mounted on the same measurement device, the transformation can be realized based on the distance, orientation, etc. of the camera and the electromagnetic wave transmitting and receiving device. A transformation rule for transforming the captured image into a predetermined coordinate system based on the position information is prepared in advance. The learning synthesis unit 24 executes the above-mentioned transformation of the captured image using the transformation rule. Examples of the image transformation method include image transformation such as affine transformation and homography transformation, but are not limited to these. The viewpoint of the captured image after transformation is the same as that of the two-dimensional image. As a result, a predetermined position (e.g., center) in the captured image and the same predetermined position (e.g., center) in the two-dimensional image indicate information of the same position in the data collection area.

Returning to FIG. 8, the language input unit 25 acquires text that describes an object that appears in the synthetic training image. The text may be a word, a sentence, a prompt, etc. The text that describes an object may include, for example, the type of object, or the external characteristics of the object (color, size, shape, etc.). An example of text is "blue can."

For example, a user may input text describing an object appearing in the training composite image into the training device 20. The user may visually recognize the training photographed image that was the basis for the training composite image, and identify the object appearing in the training composite image. The user may also identify the object appearing in the training composite image by other means. For example, when training data is generated by placing a specific object in the data collection area and generating photographic/reflection information, the user can identify the object placed in the data collection area by any other means, without visually recognizing the training photographed image.

As another example, the recognition result obtained by inputting the learning captured image that is the basis of the learning synthetic image into a recognition model (such as a classifier) that has been generated in advance by machine learning or the like may be input to the learning device 20 as the above text.

The learning unit 21 trains a learning model using multiple training synthetic images and text that describes objects that appear in each training synthetic image. The learning model is an object recognition model or object detection model that uses a language model.

As described above, an object recognition model or object detection model using a language model is a model capable of expressing images and language in a joint embedding space. The object recognition model and object detection model are generated by learning the relationship between the results of object recognition/object detection obtained using technology such as neural networks and the language related to objects (descriptions and expressions of objects). Based on the object recognition model and object detection model, objects indicated by the search criteria (text) can be recognized/detected within the captured image.

The learning unit 21 learns the correlation between an object appearing in a synthetic training image and text expressing that object. The learning unit 21 can generate the learning model of this embodiment by utilizing a widely known technology that learns the correlation between "an object appearing in a captured image" and "text expressing that object" based on a captured image to generate an object recognition model or an object detection model. In other words, the learning unit 21 can learn the learning model by replacing the "captured image" with the "synthetic training image" in this widely known technology.

Below, an example of the processing performed by the learning unit 21 is described, but the processing of the learning unit 21 is not limited to this.

First, the learning unit 21 can divide one or more texts inputted by the language input unit 25 into tokens and extract language features from these. The language features can be extracted using any widely known technology. For example, the transformer described in the following document may be used to extract the language features. For example, the learning unit 21 may extract the language features using a 63 million parameter transformer with 12 layers, 512 hidden layer dimensions, and 8 attention heads.
"Radford, Alec, et al. "Learning transferable visual models from natural language supervision." International conference on machine learning. PMLR, 2021."

Furthermore, the learning unit 21 can extract image features from the synthetic image for learning. The image features can be extracted using any widely known technology. ResNet-50, ResNet-101, ResNet-50×4, ResNet-50×16, ResNet-50×64, ViT-B/32, ViT-B/16, ViT-L/14, etc. may be used to extract the image features. Furthermore, the image features may be extracted using, for example, the technology described in the following document.
"He, Kaiming, et al. "Deep residual learning for image recognition."Proceedings of the IEEE conference on computer vision and pattern recognition. 2016."
"Dosovitskiy, Alexey, et al. "An image is worth 16x16 words: Transformers for image recognition at scale."arXiv preprint arXiv:2010.11929(2020)."

The learning unit 21 can adjust at least one of the language features, the synthetic training image, and the image features to adapt them to the learning model generated/adjusted for the captured image.

In adjusting language features, the learning unit 21 transforms the language features (feature vectors) extracted from the text inputted by the language input unit 25 using a nonlinear function or a linear function, for example to generate features of the same dimensions as the input features. As the nonlinear function, for example, a fully connected neural network (MLP) with one or more layers can be used. Also, as the linear function, a linear function with a scale (coefficient) and an offset (addition part) as optimization parameters can be used. It is preferable that the function has a smaller number of optimization parameters than the extraction of language features.

In adjusting the training composite image, the learning unit 21 converts the training composite image generated by the training synthesis unit 24 using a nonlinear function or a linear function to generate an image of the same dimensions as the input image, for example. As the nonlinear function, for example, an MLP with one or more layers can be used. As the linear function, a linear function with the optimization parameters being the scale (coefficient) and the offset (addition part) can be used. For example, the parameters for optimizing the scale and offset correspond to the correction of brightness. This correction makes it possible to bring the statistical index of the pixel values of the training composite image (such as the range of pixel values) closer to the statistical index of the pixel values of the captured image. It is preferable that the function has a smaller number of optimization parameters than the extraction of image features. When adjusting the training composite image, the learning unit 21 extracts image features from the adjusted training composite image.

In adjusting the image features, the learning unit 21 transforms the image features (feature vectors) extracted from the training synthetic image that has not been adjusted or the training synthetic image that has been adjusted, using a nonlinear function or a linear function to generate features of the same dimension as the input features, for example. As the nonlinear function, for example, an MLP with one or more layers can be used. Also, as the linear function, a linear function with scale (coefficient) and offset (additive part) as optimization parameters can be used. This correction makes it possible to bring the statistical index of the image features extracted from the training synthetic image closer to the statistical index of the feature amount extracted from the captured image. It is preferable that the function has a smaller number of optimization parameters than the extraction of image features.

The learning unit 21 can perform the following processing using the data after the above adjustments have been made to at least one of the language features, the synthetic training image, and the image features.

The learning unit 21 calculates the correlation between the language feature and the image feature, for example, using cosine similarity. For example, the learning unit 21 performs normalization processing on each of the language feature and the image feature, and calculates the inner product between these normalized vectors to calculate the correlation. To explain in more detail, the learning unit 21 receives image features for each of a plurality of training synthetic images and language features paired with each of them. The learning unit 21 then calculates not only the cosine similarity between the paired image feature and language feature, but also the cosine similarity between unpaired image features and language features. In other words, when N pairs of language features and image features are input to the learning unit 21, the learning unit 21 calculates N x N cosine similarities.

Then, the learning unit 21 calculates a loss value based on the calculated correlation, characterized in that correct pairs have small losses and incorrect pairs have large losses. The learning unit 21 then updates at least one parameter among the above-mentioned adjustment of language features, adjustment of synthetic images for training, and adjustment of image features so as to reduce the loss value. Note that the learning unit 21 may further update other parameters.

The learning unit 21 can calculate the loss value using any well-known technique. For example, the learning unit 21 arranges N x N cosine similarities in an N x N matrix, calculates the cross-entropy error in the row direction (horizontal direction), and further calculates the loss in the column direction (vertical direction). The learning unit 21 can then calculate the average value of these cross-entropies as the loss value.

The learning unit 21 can also update the above-mentioned parameters using widely known techniques. For example, the learning unit 21 may update the parameters using backpropagation or the like.

Next, an example of the process flow of the learning device 20 will be described using the flowchart in FIG. 11. Note that the details of each process have been described above, so here, an example of the process flow will be described along with an overview of each process.

In S30, the learning device 20 acquires text that describes an object. The object is an object that appears in the learning captured image or the learning composite image.

In S31, the learning device 20 acquires a learning image. In S32, the learning device 20 generates a learning two-dimensional image based on reflected wave information indicating reflected electromagnetic waves. The learning image and the learning two-dimensional image indicate information of the same object.

In S33, the learning device 20 synthesizes the learning captured image and the learning two-dimensional image to generate a learning composite image.

In S34, the learning device 20 learns a learning model using the text acquired in S30 and the synthetic learning image generated in S33. The learning model is an object recognition model or an object detection model that uses a language model.

The learning device 20 learns a learning model using the language features extracted from the text and the image features extracted from the synthetic training image. The learning device 20 can adjust at least one of the language features, the synthetic training image, and the image features to adapt them to the learning model generated/adjusted for the captured image. After making the adjustment, the learning device 20 calculates the correlation between the language features and the image features, and updates the parameters of the adjustment so that the loss value based on the calculated correlation is smaller.

"Action and effect"
The learning device 20 learns a learning model using a synthetic learning image obtained by synthesizing a "learning captured image" and a "learning two-dimensional image generated based on reflected wave information indicating reflected waves of electromagnetic waves." Such a learning device 20 can generate a learning model that has learned both the "features of the captured image" and the "features of the reflected wave information indicating reflected waves of electromagnetic waves." This learning model can perform object detection and object recognition based on both the "features of the captured image" and the "features of the reflected wave information indicating reflected waves of electromagnetic waves." By learning not just one of the "features of the captured image" and the "features of the reflected wave information indicating reflected waves of electromagnetic waves," but both, the accuracy of object detection and object recognition by the learning model is improved.

The learning device 20 also integrates the "features of the captured image" and the "features of the reflected wave information indicating the reflected electromagnetic waves" by a unique method of generating a synthetic image, and learns them together. Rather than learning the "features of the captured image" and the "features of the reflected wave information indicating the reflected electromagnetic waves" individually, learning them together as a synthetic image that is integrated by a unique method is expected to have a synergistic effect and improve the accuracy of object detection and object recognition.

The learning device 20 can also generate multiple learning composite images from a set of learning photographed images and learning two-dimensional images, with the blending ratios of the learning photographed images and learning two-dimensional images being different from one another. The learning device 20 can also generate learning composite images with different blending ratios for each partial area within the image. In this way, it is preferable that multiple learning composite images can be generated from a set of learning photographed images and learning two-dimensional images. It is also preferable that learning composite images with various blending ratios can be learned.

The learning device 20 can also adjust at least one of the language features, the synthetic image for training, and the image features to adapt them to the learning model generated/adjusted for the captured image. The learning device 20 can then update the parameters for this adjustment so that the loss value based on the correlation between the language features and the image features becomes smaller. Such a learning device 20 can generate a learning model that learns a synthetic image that combines a captured image and a two-dimensional image using an existing learning model generated/adjusted for the captured image. In other words, the learning model of this embodiment can be generated by having an existing learning model generated/adjusted for the captured image learn a synthetic image that combines a captured image and a two-dimensional image.

Fourth Embodiment
The learning device 20 of the fourth embodiment acquires learning images by a means different from that of the learning device 20 of the third embodiment, which will be described in detail below.

The learning image acquisition unit 22 searches for images that match the search query from among multiple images, using the "text describing an object" input to the language input unit 25 as a search query. The learning image acquisition unit 22 then uses the images included in the search results as learning images.

As explained in the third embodiment, the learning photographed image and the learning two-dimensional image to be combined with each other are required to contain information about the same object, but the position of the object within the image does not necessarily have to match. "The same object" means matching in type or variety, and does not require matching of the individual object. Therefore, the learning photographed image and the learning two-dimensional image to be combined with each other do not have to be photographs of the same individual object taken at the same position and in the same place. For this reason, the learning photographed image acquisition unit 22 of this embodiment acquires the learning photographed image by the means described above.

The learning image acquisition unit 22 may search for images that match the search query from among the vast amount of images available on the Internet. Alternatively, a database that stores a large number of images may be generated in advance. The learning image acquisition unit 22 may then search for images that match the search query from within this database.

The learning image acquisition unit 22 may acquire one learning image in response to one search query, or may acquire multiple learning images in response to one search query. For example, the learning image acquisition unit 22 may acquire a predetermined number of images as learning images from among multiple images searched based on the search query, in descending order of matching rate. When multiple learning images are acquired in response to one search query, the learning synthesis unit 24 can generate multiple learning synthetic images by synthesizing one learning two-dimensional image containing information of an object with each of the multiple learning images containing information of the object. Increasing the number of learning synthetic images is preferable because it improves the learning effect.

As a modified example, the learning image acquisition unit 22 may acquire learning images using both the means described in the third embodiment and the means described in this embodiment. Increasing the number of learning images is preferable because it increases the number of learning composite images that are generated.

The rest of the configuration of the learning device 20 in this embodiment is the same as the configuration of the learning device 20 in the first and third embodiments.

The learning device 20 of this embodiment achieves the same effects as the learning device 20 of the first and third embodiments. Furthermore, the learning device 20 of this embodiment can acquire learning images from among multiple images by searching for "text expressing an object" input to the language input unit 25 as a search query. Such a learning device 20 is preferable because it can acquire a large number of learning images and generate a large number of learning composite images. Furthermore, in one example, a camera is not required, which can reduce the cost burden and the burden of equipment maintenance, etc.

Fifth embodiment
"overview"
The detection device 10 of the fifth embodiment is a specific embodiment of the configuration of the detection device 10 of the second embodiment. The detection device 10 detects a detection target using a learning model generated by the learning device 20 of the first, third, and fourth embodiments. The configuration of the detection device 10 of this embodiment will be described in detail below.

"Hardware Configuration"
First, an example of the hardware configuration of the detection device 10 will be described. Each functional unit of the detection device 10 is realized by any combination of hardware and software. Those skilled in the art will understand that there are various variations in the realization method and device. The software includes programs that are stored in the device before it is shipped, and programs downloaded from recording media such as CDs (Compact Discs) and servers on the Internet.

FIG. 7 is a block diagram illustrating an example of the hardware configuration of the detection device 10. As shown in FIG. 7, the detection device 10 has a processor 1A, a memory 2A, an input/output interface 3A, a peripheral circuit 4A, and a bus 5A. The peripheral circuit 4A includes various modules. The learning device 20 does not need to have the peripheral circuit 4A. The detection device 10 may be composed of multiple devices that are physically and/or logically separated. In this case, each of the multiple devices can have the above hardware configuration.

The bus 5A is a data transmission path for the processor 1A, memory 2A, peripheral circuit 4A, and input/output interface 3A to send and receive data to each other. The processor 1A is an arithmetic processing device such as a CPU or a GPU (Graphics Processing Unit). The memory 2A is a memory such as a RAM (Random Access Memory) or a ROM (Read Only Memory). The input/output interface 3A includes interfaces for acquiring information from an input device, an external device, an external server, an external sensor, a camera, etc., and interfaces for outputting information to an output device, an external device, an external server, etc. The input/output interface 3A also includes an interface for connecting to a communication network such as the Internet. Examples of input devices include a keyboard, a mouse, a microphone, a physical button, a touch panel, etc. Examples of output devices include a display, a speaker, a printer, a mailer, etc. The processor 1A can issue commands to each module and perform calculations based on the results of those calculations.

"Function Configuration"
Next, a detailed description will be given of the functional configuration of the detection device 10. Fig. 12 shows an example of a functional block diagram of the detection device 10. As shown in the figure, the detection device 10 has a detection unit 11, a detection-use photographed image acquisition unit 12, a detection-use reflected wave processing unit 13, and a detection-use synthesis unit 14.

The detection image acquisition unit 12 acquires the captured image to be processed.

The "image to be processed" is the image that is the subject of processing to detect the detection target. The image to be processed is an image of the target area.

The concept of a "captured image" and an example of a capture method are as explained in the third embodiment.

The "detection target" is the object that is to be detected. The learning model generated by the learning device 20 detects the detection target based on the "characteristics of the captured image" and the "characteristics of the reflected wave information indicating the reflected electromagnetic wave." Therefore, an object that reflects electromagnetic waves can be the detection target. Objects that reflect electromagnetic waves are often made mainly of metal, but are not limited to this.

The "target area" is the area being searched to see if a detection target exists. The target area may be a part of the ground, a part of a building, or something else.

The detection reflected wave processing unit 13 generates a two-dimensional image to be processed based on reflected wave information indicating the reflected waves of the electromagnetic waves irradiated to the target area. The detection reflected wave processing unit 13 can generate a two-dimensional image based on the reflected wave information using processing similar to that of the learning reflected wave processing unit 23.

The "two-dimensional image to be processed" is the two-dimensional image that is the subject of processing to detect the detection target.

The concepts of "two-dimensional images," "electromagnetic waves," "reflected waves," and "reflected wave information," as well as information related to these, and an example of a method for generating these, are as described in the third embodiment.

The detection synthesis unit 14 synthesizes the photographic image to be processed and the two-dimensional image to be processed to generate a composite image to be processed. The detection synthesis unit 14 can synthesize the photographic image to be processed and the two-dimensional image to be processed in a similar process to the synthesis of the photographic image to be processed and the two-dimensional image to be processed by the learning synthesis unit 24 to generate a composite image to be processed. It is preferable that the photographic image to be processed and the two-dimensional image to be processed are of similar size.

The detection unit 11 detects the detection target based on a composite image of the processing target, which is a composite of the captured image of the processing target and the two-dimensional image of the processing target. The detection unit 11 detects the detection target based on the learning model generated by the learning device 20 of the first, third, and fourth embodiments, and the composite image of the processing target. The learning model is an object recognition model or an object detection model that uses a language model.

The detection unit 11 can adjust at least one of the language features, the synthetic image to be processed, and the image features to adapt them to the learning model generated/adjusted for the captured image. The detection unit 11 can then use the adjusted data to detect the detection target. The adjustment of at least one of the language features, the synthetic image to be processed, and the image features by the detection unit 11 is achieved by the same means as the adjustment of at least one of the language features, the synthetic image to be processed, and the image features by the learning unit 21 described in the third embodiment.

In other words, the detection unit 11 can transform language features using a nonlinear function or a linear function, and detect the detection target based on the transformed language features. The detection unit 11 can also transform the composite image to be processed using a nonlinear function or a linear function, and detect the detection target based on the transformed composite image to be processed. The detection unit 11 can also transform image features extracted from the composite image to be processed using a nonlinear function or a linear function, and detect the detection target based on the transformed image features.

The detection unit 11 acquires search conditions that express the detection target in text. The detection target is specified by these search conditions. The detection unit 11 then detects the detection target specified by these search conditions. The detection target may be specified each time a processing target composite image is input. Alternatively, the detection target may be specified in advance, and the specified detection target may be used in processing multiple processing target composite images.

"Search criteria" is text that expresses the object to be detected. The text that constitutes the search criteria is a word, a sentence, a prompt, etc. The text that constitutes the search criteria can include, for example, the type of object, or the external characteristics of the object (color, size, shape, etc.). An example of a search criterion is "blue can."

The search criteria may further include text expressing an object to be removed from the search results. That is, the search criteria may include text expressing the detection target and text expressing the target to be removed from the search results. Hereinafter, the detection target may be referred to as a "positive example," and the target to be removed from the search results may be referred to as a "negative example." Negative examples are expressed in the same manner as positive examples. An example of a search criterion for negative examples is "white can."

The detection unit 11 acquires search conditions input by the user. The user can input search conditions to the detection device 10 via any input device, such as a keyboard, touch panel, microphone, mouse, or physical button.

The detection unit 11 can detect the detection target from within the composite image to be processed, for example, based on the correlation between the composite image to be processed and the search conditions.

In one example, the detection unit 11 can detect objects with a high correlation value (cosine similarity) with the search criteria (positive examples) from within the composite image to be processed, and output the area of the object as the detection result. For example, the detection unit 11 may detect objects with a correlation value with the search criteria (positive examples) equal to or greater than a threshold from within the composite image to be processed. The detection unit 11 may also detect a predetermined number of objects with a high correlation value with the search criteria (positive examples) from within the composite image to be processed. The detection unit 11 may also detect a predetermined number of objects with a high correlation value with the search criteria (positive examples) from within the composite image to be processed, whose correlation value with the search criteria (positive examples) is equal to or greater than a threshold.

The detection unit 11 can also detect objects having a high correlation value with the search criteria (negative examples) from within the composite image to be processed, and output the detection result without including the area of the object in it. For example, the detection unit 11 can detect objects having a correlation value with the search criteria (negative examples) equal to or greater than a threshold from within the composite image to be processed. The detection unit 11 can also detect a predetermined number of objects having a high correlation value with the search criteria (negative examples) from within the composite image to be processed. The detection unit 11 can also detect a predetermined number of objects having a high correlation value with the search criteria (negative examples) from within the composite image to be processed, whose correlation value with the search criteria (negative examples) is equal to or greater than a threshold.

In this way, the detection unit 11 detects an area in which the detection target exists from within the composite image to be processed. The detection unit 11 then outputs information indicating the detected area. For example, the detection unit 11 may output an image in which information indicating the detected area (frame, mask, etc.) is superimposed on the composite image to be processed or the photographed image to be processed. In addition, the detection unit 11 may indicate a reliability calculated by any means for each detected area. Note that this output example is merely an example and is not limited to this.

Next, an example of the process flow of the detection device 10 will be described using the flowchart in FIG. 13. Note that the details of each process have been described above, so here, an example of the process flow will be described along with an overview of each process.

In S40, the detection device 10 acquires a captured image of the target area to be processed. In S41, the detection device 10 generates a two-dimensional image of the target area to be processed based on reflected wave information indicating the reflected waves of the electromagnetic waves irradiated to the target area.

Next, in S42, the detection device 10 synthesizes the captured image to be processed and the two-dimensional image to be processed to generate a composite image to be processed. Then, the detection device 10 detects the detection target based on the composite image to be processed.

"Action and effect"
According to the detection device 10 that detects the detection target using the composite image to be processed, the detection target can be detected based on both the "characteristics of the captured image" and the "characteristics of the reflected wave information indicating the reflected wave of the electromagnetic wave." By using not only one but both of the "characteristics of the captured image" and the "characteristics of the reflected wave information indicating the reflected wave of the electromagnetic wave," the detection accuracy of the detection target is improved.

In addition, the detection device 10 integrates the "features of the captured image" and the "features of the reflected wave information indicating the reflected waves of the electromagnetic waves" by a unique method of generating a composite image, and detects the detection target based on the composite image. Rather than processing the "features of the captured image" and the "features of the reflected wave information indicating the reflected waves of the electromagnetic waves" separately, they are integrated by a unique method and processed together as a composite image, which is expected to have a synergistic effect and improve detection accuracy.

The detection device 10 can also adjust at least one of the language features, the synthetic image to be processed, and the image features to accommodate the learning model generated/adjusted for the captured image. The detection device 10 can then use the adjusted data to detect the detection target. Such a detection device 10 can detect the detection target using a learning model generated by learning an existing learning model generated/adjusted for the captured image.

Sixth Embodiment
The detection device 10 of this embodiment generates a plurality of processing target composite images having different blend ratios using a set of a processing target photographic image and a processing target two-dimensional image that are input. Then, the detection device 10 detects the detection target using the plurality of processing target composite images. This will be described in detail below.

The detection synthesis unit 14 uses an input set of a photographic image to be processed and a two-dimensional image to be processed to generate a plurality of composite images to be processed having different blending ratios. The detection synthesis unit 14 may generate composite images to be processed having different blending ratios for each partial area within the image. The detection synthesis unit 14 can achieve this synthesis by a means similar to that used by the learning synthesis unit 24 described in the third embodiment.

The detection unit 11 detects the detection target from each of a plurality of processing target composite images generated using a set of processing target photographic images and processing target two-dimensional images. The process of detecting the detection target from each processing target composite image is realized by the same means as in the fifth embodiment.

Then, the detection unit 11 detects the detection target based on the detection results of each of the multiple processing target composite images. For example, the detection unit 11 can detect, as a region where the detection target exists, an area where the detection target has been detected to exist in at least a predetermined percentage of the processing target composite images (or at least a predetermined number of processing target composite images) among the multiple processing target composite images. The predetermined percentage and the predetermined number are values that are determined in advance.

The rest of the configuration of the detection device 10 of this embodiment is the same as in the second and fifth embodiments.

The detection device 10 of this embodiment provides the same effects as the second and fifth embodiments. Furthermore, the detection device 10 of this embodiment generates a plurality of composite images of the processing target having different blend ratios using an input set of a photographic image of the processing target and a two-dimensional image of the processing target, and can detect the detection target using the plurality of composite images of the processing target. Such a detection device 10 improves the detection accuracy of the detection target.

<Modification>
Modifications that can be applied to the first to sixth embodiments will be described.

The detection synthesis unit 14 may perform a correction process on the photographed image to be processed, and then generate a composite image to be processed using the corrected photographed image to be processed.

In addition, the learning synthesis unit 24 may perform a correction process on the learning captured image, and then generate a learning composite image using the corrected learning captured image.

In addition, the detection unit 11 may perform a correction process on the processing target composite image, and then detect the detection target using the corrected processing target composite image.

In addition, the learning unit 21 may perform a correction process on the training composite image and then use the corrected training composite image to train the learning model.

An example of the correction process is a correction to adjust the brightness of an image. For example, the brightness of an image may be adjusted using the technology disclosed in the following document. The correction process may be performed when the brightness of an image satisfies a predetermined condition (brightness is equal to or less than a threshold value).
"Shibata, Takashi, Masayuki Tanaka, and Masatoshi Okutomi. "Gradient-domain image reconstruction framework with intensity-range and base-structure constraints." Proceedings of the IEEE conference on computer vision and pattern recognition. 2016."
"Tanaka, Masayuki, Takashi Shibata, and Masatoshi Okutomi. "Gradient-based low-light image enhancement." 2019 IEEE International Conference on Consumer Electronics (ICCE). IEEE, 2019."

The present disclosure has been described above with reference to the embodiments, but the present disclosure is not limited to the above-mentioned embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate.

In addition, in the flowcharts used in the above explanations, multiple steps (processing) are listed in order. However, the order in which the steps are executed in each embodiment is not limited to the order listed. In each embodiment, the order of the steps shown in the figures can be changed to the extent that does not interfere with the content.

A part or all of the above-described embodiments can be described as, but is not limited to, the following supplementary notes.
1. A detection device having a detection means for detecting a detection target based on a processing target composite image obtained by combining a processing target photographed image of a target area and a processing target two-dimensional image generated based on reflected wave information indicating reflected waves of electromagnetic waves irradiated to the target area.
2. A detection image acquisition means for acquiring the processing target photographed image;
a detection reflected wave processing means for generating the two-dimensional image of the processing object based on the reflected wave information;
a detection synthesis means for synthesizing the captured image to be processed and the two-dimensional image to be processed to generate the synthesized image to be processed;
2. The detection device according to claim 1, comprising:
3. The detection means is
A detection device as described in 1 or 2, which detects the detection target based on a learning model that has been trained using a plurality of learning composite images that are obtained by combining learning photographic images with learning two-dimensional images generated based on reflected wave information indicating reflected waves of electromagnetic waves, and the detection target based on the processing target composite image.
4. The detection device according to 3, wherein the learning model is an object recognition model or an object detection model using a language model.
5. The detection synthesis means comprises:
generating a plurality of processing target composite images having different blend ratios between the processing target photographed image and the processing target two-dimensional image;
The detection means includes:
Detecting the detection target from each of the plurality of processing target composite images;
3. The detection device according to claim 2, which detects the detection target based on detection results of each of the multiple processing target composite images.
6. The detection means is
Transforming the composite image to be processed by a nonlinear function or a linear function;
6. The detection device according to any one of claims 1 to 5, which detects the detection target based on the converted composite image of the processing target.
7. The detection means is
Extracting image features from the composite image to be processed;
Transforming the image features by a nonlinear or linear function;
7. A detection device according to any one of 1 to 6, which detects the detection target based on the transformed image features.
8. One or more computers:
A detection method for detecting a detection target based on a composite image of a processing target obtained by combining a captured image of the processing target obtained by capturing an image of the target area and a two-dimensional image of the processing target generated based on reflected wave information indicating the reflected waves of electromagnetic waves irradiated to the target area.
9. Computers,
A program that functions as a detection means for detecting a detection target based on a composite image of a processing target obtained by combining a captured image of the processing target obtained by capturing an image of the target area and a two-dimensional image of the processing target generated based on reflected wave information indicating the reflected waves of electromagnetic waves irradiated to the target area.
10. A learning device having a learning means for learning a learning model using a plurality of synthetic learning images obtained by synthesizing a captured learning image with a two-dimensional learning image generated based on reflected wave information indicating a reflected electromagnetic wave.
11. The learning device according to 10, wherein the learning model is an object recognition model or an object detection model using a language model.
12. A language input means for acquiring text expressing an object appearing in the learning synthetic image,
12. The learning device according to claim 11, wherein the learning means learns a correlation between the object appearing in the synthetic image for learning and the text.
13. A learning image acquisition means for acquiring the learning image;
a learning reflected wave processing means for generating the learning two-dimensional image based on the reflected wave information;
a learning synthesis means for synthesizing the learning photographed image and the learning two-dimensional image to generate the learning synthesized image;
13. The learning device according to any one of 10 to 12, comprising:
14. The learning synthesis means
The learning device according to claim 13, wherein a plurality of the learning composite images are generated in which the blending ratios of the learning captured images and the learning two-dimensional images are different from each other.
15. The learning means
Transforming the synthetic image for learning by a nonlinear function or a linear function;
15. The learning device according to any one of claims 10 to 14, which learns the learning model using the training synthetic image after conversion.
16. The learning means
Extracting image features from the synthetic image for training;
Transforming the image features by a nonlinear or linear function;
16. A learning device according to any one of claims 10 to 15, which learns the learning model using the transformed image features.
17. One or more computers:
A learning method for learning a learning model using a plurality of synthetic learning images obtained by synthesizing a captured learning image with a two-dimensional learning image generated based on reflected wave information indicating reflected electromagnetic waves.
18. A computer
A program that functions as a learning means for learning a learning model using multiple learning composite images that are synthesized by combining learning captured images with learning two-dimensional images generated based on reflected wave information indicating reflected electromagnetic waves.
Some or all of Appendices 2 to 7 that are dependent on the detection device of Appendix 1 described above may also be dependent on the detection method of Appendix 8 and the program of Appendix 9 in the same dependent relationship as Appendix 1 and Appendices 2 to 7. In addition, some or all of Appendices 11 to 16 that are dependent on the learning device of Appendix 10 described above may also be dependent on the learning method of Appendix 17 and the program of Appendix 18 in the same dependent relationship as Appendix 10 and Appendices 11 to 16. Furthermore, within the scope of each of the above-mentioned embodiments, some or all of the configurations described as appendices can be realized in various hardware, software, various recording means for recording software, or systems.

This application claims priority based on Japanese Patent Application No. 2023-130762, filed on August 10, 2023, the entire disclosure of which is incorporated herein by reference.

REFERENCE SIGNS LIST 10 Detection device 11 Detection section 12 Detection-use captured image acquisition section 13 Detection-use reflected wave processing section 14 Detection-use synthesis section 20 Learning device 21 Learning section 22 Learning-use captured image acquisition section 23 Learning-use reflected wave processing section 24 Learning-use synthesis section 25 Language input section 1A Processor 2A Memory 3A Input/output I/F
4A Peripheral circuit 5A Bus

Claims (20)

A detection device having a detection means for detecting a detection target based on a composite image of a processing target, which is a composite of a captured image of the processing target obtained by capturing an image of the target area and a two-dimensional image of the processing target generated based on reflected wave information indicating the reflected waves of electromagnetic waves irradiated to the target area. A detection image acquisition means for acquiring the processing target photographed image;
a detection reflected wave processing means for generating the two-dimensional image of the processing object based on the reflected wave information;
a detection synthesis means for synthesizing the captured image to be processed and the two-dimensional image to be processed to generate the synthesized image to be processed;
The detection device according to claim 1 , further comprising: The detection means includes:
The detection device according to claim 1 or 2, which detects the detection target based on a learning model that has been trained using a plurality of learning composite images that are obtained by combining learning photographic images with learning two-dimensional images generated based on reflected wave information indicating reflected waves of electromagnetic waves, and the processing target composite image. The detection device according to claim 3, wherein the learning model is an object recognition model or an object detection model that uses a language model. The detection synthesis means comprises:
generating a plurality of processing target composite images having different blend ratios between the processing target photographed image and the processing target two-dimensional image;
The detection means includes:
Detecting the detection target from each of the plurality of processing target composite images;
The detection device according to claim 2 , wherein the detection target is detected based on detection results of each of the plurality of processing target composite images. The detection means includes:
Transforming the composite image to be processed by a nonlinear function or a linear function;
The detection device according to claim 1 , wherein the detection target is detected based on the converted composite image of the processing target. The detection means includes:
Extracting image features from the composite image to be processed;
Transforming the image features by a nonlinear or linear function;
The detection device according to claim 1 , wherein the detection target is detected based on the transformed image features. One or more computers
A detection method for detecting a detection target based on a composite image of a processing target obtained by combining a captured image of the processing target obtained by capturing an image of the target area and a two-dimensional image of the processing target generated based on reflected wave information indicating the reflected waves of electromagnetic waves irradiated to the target area. Computer,
A recording medium having a program recorded thereon that functions as a detection means for detecting a detection target based on a composite image of a processing target obtained by combining a captured image of the processing target obtained by capturing an image of the target area and a two-dimensional image of the processing target generated based on reflected wave information indicating the reflected waves of electromagnetic waves irradiated to the target area. A learning device having a learning means for learning a learning model using a plurality of synthetic learning images obtained by synthesizing a captured learning image with a two-dimensional learning image generated based on reflected wave information indicating reflected electromagnetic waves. The learning device according to claim 10, wherein the learning model is an object recognition model or an object detection model using a language model. a language input means for acquiring text expressing an object appearing in the learning synthetic image,
The learning device according to claim 11 , wherein the learning means learns a correlation between the object appearing in the synthetic image for learning and the text. A learning image acquisition means for acquiring the learning image;
a learning reflected wave processing means for generating the learning two-dimensional image based on the reflected wave information;
a learning synthesis means for synthesizing the learning photographed image and the learning two-dimensional image to generate the learning synthesized image;
13. A learning device according to claim 10, comprising: The learning synthesis means includes:
The learning device according to claim 13 , wherein a plurality of the learning composite images are generated in which the blending ratios of the learning captured images and the learning two-dimensional images are different from each other. The learning means includes:
Transforming the synthetic image for learning by a nonlinear function or a linear function;
The learning device according to claim 10 , wherein the learning model is trained using the training synthetic image after conversion. The learning means includes:
Extracting image features from the synthetic image for training;
Transforming the image features by a nonlinear or linear function;
The learning device according to claim 10 , wherein the learning model is trained using the transformed image features. One or more computers
A learning method for learning a learning model using a plurality of synthetic learning images obtained by synthesizing a captured learning image with a two-dimensional learning image generated based on reflected wave information indicating reflected electromagnetic waves. The learning method according to claim 17, wherein the learning model is an object recognition model or an object detection model using a language model. Computer,
A recording medium having a program recorded thereon that functions as a learning means for learning a learning model using multiple learning composite images that are obtained by combining learning photographic images with learning two-dimensional images generated based on reflected wave information indicating reflected electromagnetic waves. The recording medium according to claim 19, wherein the learning model is an object recognition model or an object detection model using a language model.

Images