WO2022208843A1 - Training data processing device, training data processing method, and training data processing program - Google Patents

Abstract

A training data processing device according to one embodiment of the present invention comprises a data input unit, data expansion unit, and data output unit. The data input unit acquires training data to be used for training a deep generative model including: an input image and an illumination environment for the input image; a teacher image obtained by changing only the illumination environment from the input image, and an illumination environment for the teacher image; and a targeted region image indicating a brightness change region in the input image. The data expansion unit creates a brightness adjustment image obtained by subjecting the input image to brightness adjustment and also creates a mask image indicating a brightness change region in which brightness change is to be made, and synthesizes the targeted region image, the brightness adjustment image, and the mask image so as to create a data expansion image. The data output unit changes the input image in the training data to the data expansion image so as to create new training data, and outputs the new training data as training data to be used for training the deep generative model.

Description

LEARNING DATA PROCESSING DEVICE, LEARNING DATA PROCESSING METHOD AND LEARNING DATA PROCESSING PROGRAM

Embodiments of the present invention relate to a learning data processing device, a learning data processing method, and a learning data processing program.

Relighting is a technique for generating a relighted image by changing the lighting environment in the image to the desired one for the input image. This relighting technique uses deep learning to generate the desired relighted image from the input image.

In a method using deep learning, for example, as proposed in Non-Patent Document 1, a re-illuminated image is obtained using learning data in which an input image and a teacher image obtained by changing only the lighting environment from the input image are paired. Train a deep generative model that generates

Also, for example, Non-Patent Document 2 prepares a special facility surrounded by a large number of cameras and lighting when creating such learning data in a real environment, and prepares various shooting conditions and lighting conditions. I suggest shooting with

It is desirable that the input image used for learning is an image that does not have global shadows on the face region caused by the lighting environment being blocked by buildings or trees. A shadow removal method is proposed.

T. SUN, et al, "Single Image Portrait Relighting," SIGGRAPH2019. K. Guo, et al, "The Relightables: Volumetric Performance Capture of Humans with Realistic Relighting," SIGGRAPH 2020. X. Zhang, et, al, "Portrait Shadow Manipulation," SIGGRAPH2020.

Special equipment such as that proposed in Non-Patent Document 2 above requires a huge number of patterns of shooting conditions and lighting conditions to be set in order to consider irregular shadows and highlights that may occur in the actual environment. Not realistic.

Therefore, when a deep generative model that generates a re-illuminated image as in Non-Patent Document 1 is learned using the learning data created in Non-Patent Document 2, the number of lighting environment patterns in the learning data is reduced. . Therefore, after learning the deep generation model, when generating a relight image from an image with shadows or highlights that are not included in the training data as an input image, shadows or highlights are added to the generated relight image. It leaves highlights.

In order to remove the shadows in the re-illumination image, shadow removal processing like that of Non-Patent Document 3 is further required.

The present invention seeks to provide a technique that makes it possible to implement the learning of deep generative models that are robust to shadows or highlights.

In order to solve the above problems, a learning data processing device according to one aspect of the present invention includes a data input section, a data extension section, and a data output section. The data input unit inputs an input image, an illumination environment of the input image, a teacher image that is an image obtained by changing only the illumination environment from the input image, an illumination environment of the teacher image, and a brightness change target area in the input image. Learning data used for learning the deep generative model is acquired, including the target region image shown in FIG. A data extension unit creates a brightness-adjusted image obtained by performing brightness adjustment on the input image, creates a mask image indicating a brightness-changed region to which brightness is to be changed, and extracts the target region image, the brightness-adjusted image, and the brightness-adjusted image. Synthesize the mask image to create a data augmented image. The data output unit creates new learning data by changing the input image in the learning data to the data augmentation image, and uses the new learning data as the learning data used for learning the deep generative model. Output.

According to one aspect of the present invention, it is possible to provide a technology that enables learning of a deep generative model robust to shadows or highlights.

FIG. 1 is a block diagram showing an example of the configuration of a deep generative model learning system comprising a learning data processing device according to the first embodiment of the present invention. FIG. 2 is a diagram showing an example of a mask image data set possessed by the learning data processing device. FIG. 3 is a diagram illustrating an example of a hardware configuration of a learning data processing device; FIG. 4 is a flow chart showing an example of the processing operation of the learning data processing device. FIG. 5 is a diagram showing an example of an input image that is one of learning data. FIG. 6 is a diagram showing an example of a target region image, which is one of learning data. FIG. 7 is a diagram showing an example of a shadow/highlight image created by the learning data processing device during processing. FIG. 8 is a diagram showing another example of a shadow/highlight image. FIG. 9 is a diagram showing an example of a mask image. FIG. 10 is a diagram showing another example of the mask image. FIG. 11 is a diagram showing an example of a reverse shadow/highlight imparting area image to be combined. FIG. 12 is a diagram showing another example of a reverse shadow/highlight imparting area image to be combined. FIG. 13 is a diagram illustrating an example of a data augmented image created by the learning data processing device; FIG. 14 is a diagram showing another example of a data augmented image created by the learning data processing device. FIG. 15 is a flow chart showing an example of the processing operation of the learning data processing device according to the second embodiment of the present invention. FIG. 16 is a block diagram showing an example of the configuration of a deep generative model learning system including a learning data processing device according to the third embodiment of the present invention. FIG. 17 is a flow chart showing an example of the processing operation of the learning data processing device according to the third embodiment.

Hereinafter, embodiments according to the present invention will be described with reference to the drawings.

[First embodiment]
(Configuration example)
FIG. 1 is a block diagram showing an example of the configuration of a deep generative model learning system including a learning data processing device 100 according to the first embodiment of the present invention. The deep generative model learning system includes this learning data processing device 100 , a learning device 200 and a learning data storage unit 300 . Note that the deep generative model learning system may be configured such that each of these units is integrated as one device or housing, or may be configured from a plurality of devices. Also, multiple devices may be remotely located and connected via a network.

The learning data storage unit 300 stores learning data necessary for learning in the learning device 200. The learning data includes the input image and the lighting environment of the input image, the teacher image that is an image obtained by changing only the lighting environment from the input image, the lighting environment of the teacher image, and the brightness change target area, that is, the shadow or highlight in the input image. and a target area image showing the area to be applied (eg, a portion of a person's face). Of these, the lighting environment has, for example, either vector data using spherical harmonics or an environment map image expressing reflections around the image. One epoch is when all of the prepared learning data is transferred from the learning data storage unit 300 to the learning data processing device 100 once.

The learning data processing device 100 performs data preprocessing including data extension on the learning data acquired from the learning data storage unit 300 . Data augmentation refers to a process of adding an image effect that simulates shadows or highlights to learning data. The learning data processing device 100 passes the preprocessed learning data to the learning device 200 .

The learning device 200 uses the learning data passed from the learning data processing device 100 to learn the deep generative model.

Also, the learning device 200 uses the learned deep generation model to generate a re-illuminated image from an arbitrary input image. The input image may be acquired via the learning data processing device 100, or may be acquired via an input device or a network (not shown). Furthermore, the learning device 200 updates the parameters of the deep generative model and records the deep generative model by evaluating the generated re-illumination image and the learning data.

As shown in FIG. 1, the learning data processing device 100 includes a data input section 110, a data extension section 120 and a data output section .

The data input unit 110 acquires the learning data, that is, the input image and the lighting environment of the input image, the teacher image and the lighting environment of the teacher image, and the target area image from the learning data storage unit 300 . The data input unit 110 passes the input image and the lighting environment of the input image and the teacher image and the lighting environment of the teacher image among the learning data to the data output unit 130 . In addition, the data input unit 110 uses random parameters to determine whether to perform data extension that increases the influence of illumination. Hand over to 120.

The data extension unit 120 includes a luminance adjustment unit 121, a mask area creation unit 122, a mask image storage unit 123, and an image synthesis unit 124.

The brightness adjustment unit 121 passes the input image passed from the data input unit 110 to the image synthesizing unit 124 . Also, the luminance adjustment unit 121 creates a shadow/highlight image, which is a luminance-adjusted image obtained by performing luminance adjustment on the input image. Then, the brightness adjustment unit 121 passes the created shadow/highlight image and the target region image passed from the data input unit 110 to the mask region generation unit 122 .

The mask image storage unit 123 stores a mask image data set, which is a data set of irregular mask images. FIG. 2 is a diagram showing an example of this mask image data set. An existing data set may be used for the mask image, or an image created using Perlin noise may be used.

The mask area creation unit 122 creates a shadow/highlight application area image indicating an area to be shaded or highlighted from the shadow/highlight image and the target area image passed from the brightness adjustment unit 121 and the mask image. do. The mask image may be a mask image stored in advance in the mask image storage unit 123, or may be created by subjecting the shadow/highlight image to arbitrary binarization processing. The mask area creation unit 122 can determine which of the pre-stored mask image and the created mask image is to be used using a random parameter. The mask area creating unit 122 passes the shadow/highlight image and the created shadow/highlight adding area image to the image synthesizing unit 124 .

The image synthesizing unit 124 synthesizes the input image passed from the luminance adjusting unit 121, the shadow/highlight image and the shadow/highlight added area image passed from the mask area creating unit 122, and generates a data extension image. create. The image synthesizing unit 124 passes the created data augmented image to the data output unit 130 .

The data output unit 130 normalizes or standardizes each of the input image and teacher image passed from the data input unit 110 when data extension is not performed. Then, the data output unit 130 passes the normalized or standardized input image and the lighting environment of the input image, and the normalized or standardized teacher image and the lighting environment of the teacher image to the learning device 200 as learning data.

Also, when performing data extension, the data output unit 130 replaces the input image passed from the data input unit 110 with the data extended image passed from the image synthesis unit 124 of the data extension unit 120 . That is, in this case, the data output unit 130 normalizes or standardizes the input image rewritten to the data augmented image and the teacher image. Then, the data output unit 130 passes the normalized or standardized input image and the lighting environment of the input image, and the normalized or standardized teacher image and the lighting environment of the teacher image to the learning device 200 as learning data. That is, data output unit 130 passes new learning data different from the original learning data to learning device 200 .

FIG. 3 is a diagram showing an example of the hardware configuration of the learning data processing device 100. As shown in FIG. The learning data processing device 100 includes a processor 11, a program memory 12, a data memory 13, an input/output interface 14, and a communication interface 15, for example. Program memory 12 , data memory 13 , input/output interface 14 and communication interface 15 are connected to processor 11 via bus 16 . The learning data processing device 100 may be composed of, for example, a general-purpose computer such as a personal computer.

The processor 11 includes a multi-core/multi-threaded CPU (Central Processing Unit), and is capable of concurrently executing multiple pieces of information processing.

The program memory 12 includes, as a storage medium, a non-volatile memory such as a HDD (Hard Disk Drive) or an SSD (Solid State Drive) that can be written and read at any time, and a non-volatile memory such as a ROM (Read Only Memory). , and stores programs necessary for executing various control processes according to the first embodiment of the present invention by being executed by a processor 11 such as a CPU. That is, the processor 11 can function as the data input unit 110, the data expansion unit 120, and the data output unit 130 as shown in FIG. 1 by reading and executing the programs stored in the program memory 12. FIG. These processing function units may be realized by sequential processing of one CPU thread, or may be realized in a form in which simultaneous parallel processing is possible by separate CPU threads. Also, these processing function units may be realized by separate CPUs. That is, the learning data processing device 100 may include multiple CPUs. In addition, at least some of these processing function units may include integrated circuits such as ASICs (Application Specific Integrated Circuits), FPGAs (field-programmable gate arrays), GPUs (Graphics Processing Units), and various other hardware circuits. may be implemented in the form of The programs stored in the program memory 12 can include a learning data processing program as shown in FIG.

The data memory 13 uses, as a storage medium, a combination of a non-volatile memory such as an HDD or an SSD that can be written and read at any time, and a volatile memory such as a RAM (Random Access Memory). It is used to pre-store various data necessary for pre-processing data including For example, in the data memory 13, a mask image data set storage area 13A for storing mask image data sets can be reserved. That is, the data memory 13 can function as the mask image storage unit 123 . In the data memory 13, a temporary storage area 13B can also be reserved for storing various data obtained and created during the process of pre-processing data including data extension.

The input/output interface 14 is an interface with an input device such as a keyboard and mouse (not shown) and an output device such as a liquid crystal monitor. The input/output interface 14 may also include an interface with a memory card or disk medium reader/writer. If the mask image set is recorded on a memory card or disk medium and provided, the processor 11 can read it through the input/output interface 14 and store it in the mask image data set storage area 13A of the data memory 13. can.

The communication interface 15 includes, for example, one or more wired or wireless communication interface units, and enables transmission and reception of various information with devices on the network according to the communication protocol used on the network. As a wired interface, for example, a wired LAN, a USB (Universal Serial Bus) interface, etc. are used. An interface that adopts the power wireless data communication standard, etc. is used. For example, when the learning data storage unit 300 is arranged in a file server or the like on the network, the processor 11 can receive and acquire learning data from the learning data storage unit 300 via the communication interface 15 . Similarly, processor 11 can obtain mask image data sets from devices on the network. Also, when learning data storage unit 300 is arranged in a server device or the like on a network, processor 11 can transmit learning data to learning device 200 via communication interface 15 .

(motion)
Next, the operation of the learning data processing device 100 will be described.

FIG. 4 is a flowchart showing an example of the processing operation of the learning data processing device 100. FIG. When the user instructs execution of the learning data processing program from an input device (not shown) through the input/output interface 14, the processor 11 starts the operation shown in this flow chart. Alternatively, the processor 11 may start the operation shown in this flow chart in response to an execution instruction from the learning device 200 on the network via the communication interface.

First, the processor 11 operates as the data input unit 110 to acquire learning data from the learning data storage unit 300 (step S11). The acquired learning data is stored in the temporary storage area 13B of the data memory 13 . The learning data includes the input image and the lighting environment of the input image, the teacher image and the lighting environment of the teacher image, and the target area image.

Then, the processor 11 determines whether or not to perform data expansion to increase the influence of illumination using random parameters (step S12). If data extension is not to be performed (NO in step S12), the processor 11 proceeds to the process of step S20, which will be described later.

On the other hand, if data extension is to be performed (YES in step S12), the processor 11 performs the operation as the luminance adjustment unit 121 and first acquires the input image and the target area image (step S13). . That is, the processor 11 reads out the input image and the target area image from the temporary storage area 13B. Passing the input image and the target area image from the data input unit 110 to the brightness adjustment unit 121 in the description of the configuration means saving and reading to the temporary storage area 13B in this manner. This also applies to the following description.

FIG. 5 is a diagram showing an example of the input image I. FIG. FIG. 6 is a diagram showing an example of the target area image _Mf .

Then, the processor 11 performs luminance adjustment on the entire input image I to create a shadow/highlight image (step S14). This brightness adjustment includes a brightness adjustment to decrease brightness when adding effect A simulating shadow as data extension, and a brightness adjustment increasing brightness when effect B simulating highlight is added as data extension. Brightness adjustment techniques may use, for example, linear correction and gamma correction, and are preselected by the user. For the brightness adjustment parameter γ, the user sets the upper and lower limits in advance under the condition that γ<1.0 for effect A and γ>1.0 for effect B in both linear correction and gamma correction. and randomly determined within this range. A shadow/highlight image J whose luminance has been adjusted is created as shown in Equation 1 below.

FIG. 7 is a diagram showing an example of a shadow/highlight image J that has undergone luminance adjustment in the case of adding an effect A imitating a shadow as data extension. The shadow/highlight image J in this case is a shadow image. Also, FIG. 8 is a diagram showing an example of a shadow/highlight image J subjected to luminance adjustment in the case of adding an effect B simulating a highlight as data extension. The shadow/highlight image J in this case is a highlight image.

The processor 11 stores the shadow/highlight image J thus created in the temporary storage area 13B.

The processor 11 executes the operation as the mask area creating unit 122 and determines whether to use the mask image data set stored in advance in the mask image storage unit 123, that is, the mask image data set storage area 13A (step S15). ). That is, the processor 11 determines whether to use a mask image prepared in advance or to create a mask image from the shadow/highlight image J based on random parameters.

If it is determined to use the mask image data set (YES in step S15), the processor 11 acquires the mask image _Md from the mask image data set stored in the mask image storage unit 123 using, for example, random parameters. (Step S16). FIG. 9 is a diagram showing an example of the acquired mask image _Md . The processor 11 stores this acquired mask image _Md in the temporary storage area 13B.

On the other hand, if it is determined not to use the mask image data set (NO in step S15), the processor 11 selects the shadow/highlight image J stored in the temporary storage area 13B as data extension. A highlight image, which is a shadow/highlight image J subjected to luminance adjustment when applying an effect B imitating , is read. Then, the processor 11 creates a mask image _Md by performing arbitrary binarization processing on the shadow/highlight image J (step S17). FIG. 10 is a diagram showing an example of a mask image _Md created from this highlight image. The processor 11 stores this created mask image _Md in the temporary storage area 13B.

After the mask image _Md is obtained or created in this way, the processor 11 reads out the target area image _Mf and the mask image _Md from the temporary storage area 13B, and creates a shadow/highlight application area image based on them. (step S18). That is, the processor 11 obtains the logical product of the target area image M _f and the mask image M _d as shown in Equation 2 below, and applies a Gaussian filter g(k, σ) to obtain a shadow/highlight added area image Ask for M. Note that the parameters of the Gaussian filter are determined in advance by the user. By default, the filter size k=11 and σ=5.0. Then, the processor 11 saves the created shaded/highlighted area image M in the temporary storage area 13B.

After that, the processor 11 executes the operation as the image synthesizing unit 124, reads out the input image I, the shadow/highlight image J, and the shadow/highlight application area image M from the temporary storage area 13B, and converts them into the following numbers. 3 to create a data extended image I' (step S19).

The shadow/highlight image J read here is a shadow image when the mask image _Md is obtained from the mask image data set, a corresponding highlight image when the mask image Md is created from the shadow/highlight image J, becomes. The processor 11 can determine which shadow/highlight image J is to be read by storing it in the temporary storage area 13B in step S16 or step S17 and reading it. Alternatively, when the mask image _Md is stored in the temporary storage area 13B in step S16 or S17, the shadow/highlight image J not used for image synthesis may be deleted from the temporary storage area 13B.

11 and 12 are diagrams showing an example of a reverse shadow/highlight imparting area image to be combined, indicated by 1-M in Equation 3. FIG. 11 corresponds to the mask image _Md of FIG. 9, and FIG. 12 corresponds to the mask image _Md of FIG.

Then, the processor 11 saves the created data augmented image I' in the temporary storage area 13B. 13 and 14 are diagrams showing an example of the created data augmented image I'. FIG. 13 shows a data augmented image I' created from the input image I, a shadow/highlight image J which is a shadow image, and the reversed shadow/highlight added area image 1-M in FIG. 12 shows a data augmented image I' created from I, a shadow/highlight image J which is a highlight image, and the reverse shadow/highlight imparting area image 1-M of FIG.

The processor 11 then operates as the data output unit 130 and transmits learning data (step S20).

That is, if it is determined in step S12 that data expansion is not to be performed, the processor 11 reads out the input image and teacher image stored in the temporary storage area 13B, and normalizes or standardizes them. and store it again in the temporary storage area 13B. Then, the processor 11 reads out the input image, the lighting environment of the input image, and the teacher image and the lighting environment of the teacher image from the temporary storage area 13B, and transmits them to the learning device 200 via the communication interface 15 .

On the other hand, when it is determined in step S12 that data extension is to be performed and the data extension image I' is generated by the processing in steps S13 to S19, the processor 11 stores the data in the temporary storage area 13B. read out the data augmented image I'. Then, the processor 11 normalizes or standardizes the data augmented image I', and saves the result as the input image I by overwriting the input image I already saved in the temporary storage area 13B. That is, the processor 11 rewrites the input image I stored in the temporary storage area 13B to the normalized or standardized data augmented image I'. The processor 11 also reads the teacher image stored in the temporary storage area 13B, normalizes or standardizes it, and stores it in the temporary storage area 13B again. Then, the processor 11 reads out the input image, the lighting environment of the input image, and the teacher image and the lighting environment of the teacher image from the temporary storage area 13B, and transmits them to the learning device 200 via the communication interface 15 .

In the learning data processing apparatus 100 according to the first embodiment described above, the data input unit 110 obtains the input image I and the lighting environment of the input image I from the learning data storage unit 300, and only the lighting environment from the input image I. Acquiring learning data used for learning a deep generative model, including a teacher image that is a modified image, the lighting environment of the teacher image, and a target region image M _f that indicates a brightness change target region in the input image I; The data extension unit 120 creates a shadow/highlight image J, which is a brightness adjusted image obtained by performing brightness adjustment on the input image I, and acquires or creates a mask image _Md indicating a brightness change area to which brightness change is applied. Then, the target area image M _f , the shadow/highlight image J, and the mask image M _d are combined to create a data extension image I′. Then, the learning data processing device 100 uses the data output unit 130 to change the input image I in the learning data to the data augmented image I′ to create new learning data, and converts the new learning data into the deep generation model. It is output to the learning device 200 as learning data used for learning.

In this way, the learning data processing apparatus 100 according to the first embodiment creates the data augmented image I′ based on the learning data, and creates new learning data including the data augmented image I′, so that the deep layer The number of pieces of learning data used for learning the generative model can be increased. Therefore, the learning device 200 can learn a deep generative model using learning data with increased influence of an irregular lighting environment, and realize learning of a deep generative model that is robust against shadows or highlights. becomes possible.

Further, according to the first embodiment, the data extension unit 120 includes the brightness adjustment unit 121 that creates the shadow/highlight image J, which is a brightness adjusted image, by lowering the brightness of the entire input image I, and the target region image M _f , the shadow/highlight image J and the mask image _Md are combined to create a data augmented image I′ in which the portion corresponding to the luminance change area in the area corresponding to the luminance change target area in the input image I is darkened. and an image synthesizing unit 124 to create.

As described above, the learning data processing apparatus 100 according to the first embodiment adds an image effect simulating a shadow to the learning data, thereby increasing the pattern of the lighting environment in the learning data in a pseudo manner. It is possible to realize robust deep generative model learning for shadows in the device 200 .

Further, according to the first embodiment, the data extension unit 120 includes the brightness adjustment unit 121 that creates the shadow/highlight image J, which is a brightness adjustment image, by increasing the brightness of the entire input image I, and the target region image M _f , the shadow/highlight image J and the mask image M _d are combined to create a data augmented image I′ in which the portion corresponding to the luminance change area in the area corresponding to the luminance change target area in the input image I is brightened. and an image synthesizing unit 124 to create.

In this way, the learning data processing apparatus 100 according to the first embodiment adds an image effect simulating a highlight to the learning data, thereby increasing the number of pseudo lighting environment patterns in the learning data. It is possible to realize robust deep generative model learning for highlights in the learning device 200 .

Furthermore, according to the first embodiment, the data expansion unit 120 includes the mask area creation unit 122 that creates the mask image _Md by performing arbitrary binarization processing on the shadow/highlight image J, which is the brightness adjustment image. including.

In this way, the learning data processing apparatus 100 according to the first embodiment creates a mask image _Md based on the shadow/highlight image J, that is, the input image I, which is a luminance adjustment image, and uses this mask image _Md . Since the data-extended image I' is generated as a result, the probability that the data-extended image I' deviating greatly from the input image I will be generated can be reduced.

Further, according to the first embodiment, the data expansion unit 120 further extracts mask images from the mask image data set MIDS, which is a data set of irregular mask images stored in the mask image storage unit 123 in advance. It includes a mask region generator 122 that obtains M _d .

Thus, the learning data processing apparatus 100 according to the first embodiment can create the data augmented image I' using various mask images _Md independent of the input image I, and the number of learning data can be reduced to It becomes possible to increase easily.

[Second embodiment]
In the first embodiment, a data-extended image obtained by adding an image effect simulating a shadow to the learning data, or a data-extended image I′ obtained by performing data extension adding an image effect simulating a highlight to the learning data. is creating That is, one of the data extension images I' is created. However, both of these two types of data extension images I' may be created.

FIG. 15 is a flow chart showing an example of the processing operation of the learning data processing device according to the second embodiment of the present invention. In this embodiment, the determination process of step S15 in the first embodiment is omitted, and the processor 11 obtains the mask image _Md in step S16 and creates the mask image _Md in step S17. do both. When the processor 11 includes multi-threaded CPUs, these processes can be performed concurrently in separate threads. Of course, the process of step S16 and the process of step S17 may be performed sequentially. In this case, after performing the process of step S16, the process of step S17 may be performed, or the order may be reversed.

In the process of step S18, the processor 11 creates two types of shadow/highlight addition area images M using the respective mask images M _d , and in the process of step S19, two types of data extension images I ' to create.

Then, the processor 11 transmits learning data including these two types of data augmented images I' to the learning device 200 in step S20.

The learning data processing apparatus 100 according to the second embodiment described above produces a shadow/highlight image J obtained by reducing the luminance of the entire input image I and a shadow/highlight image J obtained by increasing the luminance of the entire input image I. is used to obtain a data extension image I′ in which a portion corresponding to the brightness change region is darkened in the region corresponding to the brightness change target region in the input image I, and a data extension image in which a portion corresponding to the brightness change region is brightened Create an image I'.

As described above, the learning data processing apparatus 100 according to the second embodiment adds image effects simulating shadows and highlights to the learning data, thereby increasing the number of pseudo lighting environment patterns in the learning data. This makes it possible to realize robust deep generative model learning for shadows and highlights in the learning device 200 .

[Third embodiment]
FIG. 16 is a block diagram showing an example of the configuration of a deep generative model learning system including the learning data processing device 100 according to the third embodiment of the present invention. A learning data processing apparatus 100 according to this embodiment includes an evaluation unit 140 in addition to the configuration of the first embodiment. Then, the image synthesizing unit 124 of the data extension unit 120 adds the created data extension image I′ to the data output unit 130 and also passes it to the evaluation unit 140 .

The evaluation unit 140 has an internal anomaly detection model and evaluates the data augmented image I'. Here, the anomaly detection model consists of an image group A with shadows and highlights obtained from the actual image, and an image group B with shadows and highlights arbitrarily created so as to greatly deviate from the actual image. was used as learning data, and learning was performed using distance learning. The evaluation unit 140 acquires the data extension image I' from the data extension unit 120, inputs this data extension image I' to the anomaly detection model, and obtains an evaluation value. Then, when the evaluation value exceeds the threshold set by the user, the evaluation unit 140 regards the data extension image I′ as an image greatly deviating from the actual image, discards it, and returns the image to the data extension unit 120 again. , to perform data augmentation.

FIG. 17 is a flow chart showing an example of the processing operation of the learning data processing device 100 according to the third embodiment. That is, following the processing of step S19, the processor 11 reads out the data augmented image I' stored in the temporary storage area 13B, and evaluates the data augmented image I' using the anomaly detection model learned in advance (step S31).

Then, the processor 11 determines whether the obtained evaluation value is equal to or less than the threshold (step S32). If the evaluation value is equal to or less than the threshold (YES in step S32), the data augmented image I' does not deviate greatly from the actual image, and is considered suitable for learning data of the learning device 200, and the process proceeds to step S20. . As a result, learning data including the data augmented image I′ is transmitted to the learning device 200 .

On the other hand, if the evaluation value is not equal to or less than the threshold value, that is, if it exceeds the threshold value (NO in step S32), the processor 11 temporarily determines that the data augmented image I′ is an image greatly deviating from the actual image. The data augmented image I' is deleted from the storage area 13B, and the process from step S13 is repeated. As a result, a new data augmented image I' can be created by changing the mask image.

In the learning data processing device 100 according to the third embodiment described above, the evaluation unit 140 evaluates the data augmented image I′ created by the data extension unit 120, and the data augmented image I′ is larger than the actual image. If the image is a deviated image, the data extension unit 120 is caused to create the data extension image I′ again.

In this way, the learning data processing device 100 according to the third embodiment evaluates the data augmented image I′ to create learning data that is not suitable for use in learning the deep generative model in the learning device 200. It is possible to prevent it from being done.

It goes without saying that the learning data processing device 100 according to the second embodiment described above may also include the evaluation unit 140 as in the third embodiment.

[Other embodiments]
In the first and third embodiments described above, the brightness adjustment unit 121 creates both a shadow image and a highlight image as the shadow/highlight image J, and the mask area creation unit 122 randomly selects one of them. shall be used for Instead of doing so, the brightness adjustment unit 121 randomly generates any image, and the mask area creation unit 122 uses a mask image corresponding to the shadow/highlight image J generated by the brightness adjustment unit 121. It is good to do.

In the first embodiment, one data augmented image I' of one type is created, and in the second embodiment, two types of data augmented image I' are created one by one. Alternatively, the number of data extension images I' to be created may be increased by creating them.

In the first and third embodiments, when the data augmented image I' is created in step S19, if the mask image _Md has been obtained from the mask image data set, the shadow/highlight image J However, the highlight image may be used as the shadow/highlight image J in such a case as well. Which image to use as the shadow/highlight image J may be determined by a random parameter, or both may be used to create two types of data extension images I'.

Also, the learning data storage unit 300 may be configured as part of the learning data processing device 100 . That is, the data memory 13 may be provided with a storage area as the learning data storage unit 300 .

Furthermore, the learning device 200 may incorporate the functions of the learning data processing device 100 of the embodiment.

Further, the method described in each embodiment can be executed by a computer (computer) as a program (software means), such as a magnetic disk (floppy (registered trademark) disk, hard disk, etc.), an optical disk (CD-ROM, DVD , MO, etc.), a semiconductor memory (ROM, RAM, flash memory, etc.), or the like, or may be transmitted and distributed via a communication medium. The programs stored on the medium also include a setting program for configuring software means (including not only execution programs but also tables and data structures) to be executed by the computer. A computer that realizes this apparatus reads a program recorded on a recording medium, and in some cases, builds software means by a setting program, and executes the above-described processes by controlling the operation by this software means. The term "recording medium" as used in this specification includes not only those for distribution, but also storage media such as magnetic disks, semiconductor memories, etc. provided in computers or devices connected via a network.

In short, the present invention is not limited to the above embodiments, and can be modified in various ways without departing from the gist of the invention at the implementation stage. Moreover, each embodiment may be implemented in combination as much as possible, in which case the combined effect can be obtained. Furthermore, the above-described embodiments include inventions at various stages, and various inventions can be extracted by appropriately combining a plurality of disclosed constituent elements.

DESCRIPTION OF SYMBOLS 11... Processor 12... Program memory 13... Data memory 13A... Mask image data set storage area 13B... Temporary storage area 14... Input/output interface 15... Communication interface 16... Bus 100... Learning data processing device 110... Data input unit 120... Data Extension unit 121 Luminance adjustment unit 122 Mask area creation unit 123 Mask image storage unit 124 Image synthesizing unit 130 Data output unit 140 Evaluation unit 200 Learning device 300 Learning data storage unit I Input image I' Data extension image J...Shade/highlight image Md...Mask image _Mf ...Target area image _MIDS ...Mask image data set 1-M...Inverted shadow/highlight added area image

Claims (8)

an input image and the illumination environment of the input image; a teacher image that is an image obtained by changing only the illumination environment from the input image; the illumination environment of the teacher image; a data input unit that acquires learning data used for learning the deep generative model, including
creating a brightness-adjusted image obtained by performing brightness adjustment on the input image, creating a mask image indicating a brightness-changed region to which brightness is to be changed, and synthesizing the target region image, the brightness-adjusted image, and the mask image; a data augmentation unit for creating a data augmentation image;
A data output unit that creates new learning data by changing the input image in the learning data to the data augmentation image, and outputs the new learning data as the learning data used for learning the deep generative model. When,
A learning data processing device comprising: The data extension unit combines a brightness adjustment unit that creates the brightness adjustment image by lowering the brightness of the entire input image, and combines the target area image, the brightness adjustment image, and the mask image to obtain the 2. The learning data processing device according to claim 1, further comprising an image synthesizing unit that creates the data augmented image in which a portion corresponding to the brightness change area in the area corresponding to the brightness change target area is darkened. The data extension unit synthesizes the brightness adjustment unit that creates the brightness adjustment image by increasing the brightness of the entire input image, and the target area image, the brightness adjustment image, and the mask image, 2. The learning data processing device according to claim 1, further comprising an image synthesizing unit that creates the data augmented image in which a portion corresponding to the brightness change area in the area corresponding to the brightness change target area is brightened. The learning data processing device according to claim 3, wherein the data extension unit further includes a mask area creation unit that creates the mask image by performing arbitrary binarization processing on the brightness adjustment image. 4. The data extension unit according to claim 2 or 3, further comprising a mask region creation unit that obtains the mask image from a mask image data set, which is a pre-stored data set of irregular mask images. learning data processor. 2. An evaluation unit that evaluates the data extension image and causes the data extension unit to create the data extension image again when the data extension image is an image greatly deviated from the actual image. 6. The learning data processing device according to any one of 1 to 5. A learning data processing method in a learning data processing device having a processor and creating new learning data from learning data used for learning a deep generative model,
The processor indicates an input image, an illumination environment of the input image, a teacher image that is an image obtained by changing only the illumination environment from the input image, an illumination environment of the teacher image, and a brightness change target area in the input image. obtaining the training data, including an image of the target area;
The processor creates a brightness-adjusted image obtained by performing brightness adjustment on the input image, creates a mask image indicating a brightness-changed region to which brightness is to be changed, and produces the target region image, the brightness-adjusted image, and the mask. Combining images to create a data augmented image,
The processor changes the input image in the learning data to the data augmentation image to create new learning data, and outputs the new learning data as the learning data used for learning the deep generative model. do,
Learning data processing method. A learning data processing program that causes a processor to function as each part of the learning data processing device according to any one of claims 1 to 6.

Images