CN116777925A - A generalization method for image segmentation domain based on style transfer - Google Patents

Abstract

The present invention relates to the technical field of image processing, and solves the technical problem that the previous method based on style transfer is difficult to generate diversified styles while retaining the structural information of the image. In particular, it relates to an image segmentation domain generalization method based on style transfer. It includes the following steps: S1. Extract texture structure features from the training data set P containing multiple source domains; S2. Use the training data set P to train a random style generation model based on local invariance. This invention realizes image generation through local texture invariant constraints and domain-independent random variables, and then constructs a scenario training scheme that simulates domain shifts, so that the model pays more attention to the content of the image, reduces dependence on style information, and can generate diversified styles. While retaining the structural information of the image, it effectively improves the segmentation accuracy of the model in the unknown domain and improves the efficiency of model deployment.

Description

A generalization method for image segmentation domain based on style transfer

Technical field

The present invention relates to the field of image processing technology, and in particular to an image segmentation domain generalization method based on style transfer.

Background technique

Domain generalization learns a common model from different training data sets that can generalize well to arbitrary unseen target domains without the need for target data collection and fine-tuning, where exploiting data generation is a more efficient way of downstream tasks. Intuitive and more efficient method because the quality of the generated samples is easier to assess.

Current methods based on style transfer have difficulty in generating diverse styles while retaining the structural information of images, thereby reducing the performance of downstream tasks. For example, Fourier transform is used to obtain frequency domain information, and image style conversion is achieved by exchanging low-frequency components between images. This type of method can well retain the semantic information of the image, but the generated style is not realistic. Another type of method is based on the CycleGAN architecture to achieve data generation, which can generate image styles closer to real scenes, but often changes the structure of the image during the generation process.

Contents of the invention

In view of the shortcomings of the existing technology, the present invention provides an image segmentation domain generalization method based on style transfer, which solves the technical problem that the previous method based on style transfer is difficult to generate diversified styles while retaining the structural information of the image.

In order to solve the above technical problems, the present invention provides the following technical solution: an image segmentation domain generalization method based on style transfer, which method includes the following steps:

S1. Extract texture structure features from the training data set P containing multiple source domains. The texture structure features include the original image. with generated image/> Local binary pattern features of category c/> and/> ;

S2. Use the training data set P to train a random style generation model based on local invariance. The random style generation model uses two generators. ,/> and two discriminators/> ,/> cyclic generative adversarial network;

S3. Construct a meta-learning training data set M based on the style random generation model. The training data set M includes a meta-training set and a meta-test set;

S4. Use the training data set M to train a generalization model based on the meta-learning paradigm, and promote robust optimization by simulating domain shifts during the training process to obtain a generalization model that can cope with scenarios in different styles.

Further, in step S1, it specifically includes:

S11. Extract the original image according to the label in the training data set P with generated image/> Local binary pattern features of category c/> and/> ;

S12. According to local binary pattern characteristics and/> Calculate cosine similarity to construct local similarity loss/> , which is used to keep the texture structure in the mask area unchanged during the image style generation process.

Further, in step S12, local similarity loss The calculation formula is:

In the above formula, C represents the number of categories c, and/> All are original images/> with generated image/> Local binary pattern features of class c.

Further, in step S2, the two generators and/> Respectively used to convert real images with style S/> and real images with style T/> Convert to intermediate-style generated image/> ;

The discriminator is used to distinguish real samples and generated samples. Two discriminators and/> Distinguish images separately/> and images/> .

Further, in step S2, the loss function of the random style generation model Loss by confrontation/> , cycle consistent loss/> and local similarity loss/> Composition, defined as:/> ;

Among them, against losses defined as:

in,

In the above formula, z represents a domain-independent random variable, and/> A collection of images representing the source domain and the target domain respectively, Indicates that the calculation comes from the data set/> All samples/> The expected value,/> Indicates that the calculation comes from the data set/> All samples/> the expected value;

cycle consistent loss defined as:

In the above formula, z represents a domain-independent random variable, Represents a generator that converts real images of style S into generated images, /> Represents a generator that converts real images of style T into generated images;

local similarity loss defined as:

In the above formula, C represents the number of categories c, and/> All are original images/> with generated image/> Local binary pattern features of class c.

Furthermore, for domain-independent random variables z∈[0,1], when the random variable z is closer to 0, the intermediate state The closer the picture style is to the original image/> , on the contrary, when the random variable z is closer to 1, the intermediate state/> The closer the picture style is to the style image .

Further, in step S3, it specifically includes:

S31. Randomly select a domain from the source domain S, and randomly sample n images from this domain as a meta-training set. ;

S32. Randomly generate a model based on the trained style to change the styles of n images from the original images. Convert to intermediate state/> , will generate image/> As a meta-test set/> , original image/> The corresponding generated image/> share the same semantic label.

Further, in step S4, the generalization model adopts an encoder-decoder based segmentation network as the backbone network in the meta-learning paradigm.

Furthermore, the loss function of the segmentation network By cross entropy loss/> and Dice loss/> Composition, defined as:/> ,in;

cross entropy loss defined as:

Dice loss defined as:

In the above formula, M represents the total number of pixels in the image, C represents the number of categories, and/> Represent the predicted probability and true value of the c-th category of the i-th pixel respectively.

Through the above technical solutions, the present invention provides an image segmentation domain generalization method based on style transfer, which at least has the following beneficial effects:

1. The present invention realizes image generation through local texture invariant constraints and domain-independent random variables, and then constructs a scenario training scheme that simulates domain shifts, so that the model pays more attention to the content of the image, reduces dependence on style information, and can generate diverse It optimizes the style while retaining the structural information of the image, effectively improving the model's segmentation accuracy in unknown domains and improving the efficiency of model deployment.

2. The present invention ensures local invariance by calculating the similarity of texture features, and adds a domain-independent random variable to the adversarial loss function to construct a pairwise style transfer model in multiple source domains, which can ensure the masking While the texture characteristics of the code area remain unchanged, images with different angles, light intensities and other styles are generated to simulate actual scenes with different equipment and different shooting angles, which greatly reduces the cost of data collection and labeling.

3. The present invention can develop a reusable image segmentation method based on convolutional neural network on unseen data sets. This method can generate more realistic and diverse data and be based on diverse style image design. A training paradigm that makes the model pay more attention to the content of the image and reduce its reliance on style information.

4. Based on the random style generation model, the present invention randomly selects n images in a source domain as a meta-training set in each iteration, and randomly generates n style images from the meta-training set as a meta-test set, where the meta-training set and The meta-test set shares the same semantic labels, which promotes robust optimization by simulating domain shifts during training, thereby enabling the model to have good segmentation capabilities in unknown scenarios.

Description of drawings

The drawings described here are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation of the present application. In the attached picture:

Figure 1 is a flow chart of the image segmentation domain generalization method of the present invention;

Figure 2 is a network structure diagram of the random style generation model based on local invariance of the present invention;

Figure 3 is a network structure diagram of the generalization model based on the meta-learning paradigm of the present invention.

Detailed ways

In order to make the above objects, features and advantages of the present invention more obvious and understandable, the present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. In this way, the implementation process of how this application applies technical means to solve technical problems and achieve technical effects can be fully understood and implemented accordingly.

Deep learning technology has made significant progress in image segmentation. However, these achievements largely rely on the assumption that source domain data and target domain data are equally distributed. Directly applying related methods without addressing distribution shifts will lead to poor practical application results. Significant degradation. In this context, domain generalization has become an active research direction and has gained widespread interest and prospects in the image analysis community. Domain generalization learns a common model from different training data sets that generalizes well to arbitrary unseen target domains without the need for target data collection and fine-tuning.

Existing domain generalization research can generally be divided into three categories: domain-invariant representation learning, meta-learning and data enhancement. Domain-invariant representation learning performs explicit feature alignment between domains and invariant risk minimization to learn domain-invariant representations. Meta-learning increases the generalization performance of the model by partitioning the source domain to simulate the unseen target domain. Recently, domain generalization methods based on data augmentation have become more popular, which focus on manipulating the input data to expand the training distribution and try to match the test distribution. Among them, leveraging data generation is a more intuitive and efficient method for downstream tasks because the quality of the generated samples is easier to evaluate.

Current methods based on style transfer have difficulty in generating diverse styles while retaining the structural information of images, thereby reducing the performance of downstream tasks. For example, Fourier transform is used to obtain frequency domain information, and image style conversion is achieved by exchanging low-frequency components between images. This type of method can well retain the semantic information of the image, but the generated style is not realistic. Another type of method is based on the CycleGAN architecture to achieve data generation, which can generate image styles closer to real scenes, but often changes the structure of the image during the generation process.

Please refer to Figures 1-3, which illustrates a specific implementation of this embodiment. This embodiment uses local invariance to construct a pair of random style generation models in multiple source domains, while ensuring the texture characteristics of the mask area. Under the same condition, images with different angles, light intensity and other styles are generated to simulate actual scenes with different devices and different shooting angles; then, during the training process of the generalization model based on the meta-learning paradigm, through the above-mentioned pre-trained random styles The generative model generates two sets of images as a meta-training set and a meta-test set respectively, and promotes robust optimization by simulating domain shifts during the training process, so that the generalization model can cope with scenes in different styles.

Please refer to Figure 1. This embodiment proposes an image segmentation domain generalization method based on style transfer. The method includes the following steps:

S1. Extract texture structure features from the training data set P containing multiple source domains. The texture structure features include the original image. with generated image/> Local binary pattern features of category c/> and/> ; In this embodiment, in order to illustrate the implementation of step S1, the specific process is implemented through the following steps, and the detailed implementation method is as follows:

S11. Extract the original image according to the label in the training data set P with generated image/> Local binary pattern features of category c/> and/> ;

S12. According to local binary pattern characteristics and/> Calculate cosine similarity to construct local similarity loss/> , which is used to keep the texture structure in the mask area unchanged during the image style generation process.

In step S12, local similarity loss The calculation formula is:

In the above formula, C represents the number of categories c, and/> All are original images/> with generated image/> Local binary pattern features of class c.

In this step, for a training data set containing multiple source domains, as shown in Figure 2, the original image is extracted according to the label with generated image/> The local binary pattern feature of category c is/> and/> , construct local similarity loss by calculating cosine similarity/> , local similarity loss/> Keep original image/> and generate images The texture of the local information (mask area) remains unchanged to ensure the authenticity of the mask, so that only the style is changed during the image generation process, while the texture structure in the mask area remains unchanged.

In this step, local invariance is ensured by calculating the similarity of texture features, and a domain-independent random variable is added to the adversarial loss function to build a pairwise style transfer model in multiple source domains, which can ensure While the texture characteristics of the mask area remain unchanged, images with different angles, light intensity and other styles are generated to simulate actual scenes with different devices and different shooting angles, which greatly reduces the cost of data collection and annotation.

S2. Use the training data set P to train a random style generation model based on local invariance. The random style generation model uses two generators. ,/> and two discriminators/> ,/> A recurrent generative adversarial network; two generators/> and/> Respectively used to convert real images with style S/> and real images with style T/> Convert to intermediate-style generated image/> ;

The discriminator is used to distinguish real samples and generated samples. Two discriminators and/> Distinguish images separately/> and images/> . The output of the discriminator is a probability value, used to represent the probability that the input image is a real sample, for/> Discriminator, if the input is a real sample of style S/> , then/> The output is close to 1, indicating that it is very likely to be a real sample; if the input is a generated sample with style S/> , then/> The output is close to 0, indicating that it is likely to be a generated sample. Similarly, for the discriminator/> , judge the sample with style T.

The cyclic generative adversarial network can realize the mutual conversion between images in one source domain and images in another source domain. On this basis, a domain-independent random variable z is input. As shown in Figure 2, the style S Source domain image Generate intermediate-style images/> , not just another source domain image/> style, thereby obtaining diverse images with random styles.

In step S2, the loss function of the random style generation model Loss by confrontation/> , cycle consistent loss and local similarity loss/> Composition, defined as:/> .

Among them, against losses defined as:

in,

In the above formula, z represents a domain-independent random variable, and/> A collection of images representing the source domain and the target domain respectively, Indicates that the calculation comes from the data set/> All samples/> The expected value,/> Indicates that the calculation comes from the data set/> All samples/> expected value.

Among them, the cycle-consistent loss It is guaranteed that the generated image can be reconstructed back to the original image, thereby ensuring that the semantic information of the generated image has not changed, and the loop consistency loss/> defined as:

In the above formula, Represents a generator that converts real images of style S into generated images, /> Represents a generator that converts real images of style T into generated images.

Local similarity loss preserves the original image with generated image/> The texture of the local information (mask area) remains unchanged to ensure the authenticity of the mask, and local similarity is lost/> defined as:

In the above formula, C represents the number of categories c, and/> All are original images/> with generated image/> Local binary pattern features of class c.

Domain-independent random variable z∈[0,1], the closer the random variable z is to 0, the intermediate state The closer the picture style is to the original image/> , on the contrary, when the random variable z is closer to 1, the intermediate state/> The closer the picture style is to the style image/> .

In this step, the loss function of the style random generation model based on local invariance consists of adversarial loss, cycle consistent loss and local similarity loss, and a style transfer model that can generate realistic and diverse images is obtained through stochastic gradient descent.

S3. Construct a meta-learning training data set M based on the style random generation model. The training data set M includes a meta-training set and a meta-test set; in this embodiment, in order to illustrate the implementation of step S3, the specific process is as follows The detailed implementation method is as follows:

S31. Randomly select a domain from the source domain S, and randomly sample n images from this domain as a meta-training set. ;

S32. Randomly generate a model based on the trained style to change the styles of n images from the original images. Convert to intermediate state/> , will generate image/> As a meta-test set/> , original image/> The corresponding generated image/> share the same semantic label.

In this step, as shown in a in Figure 3, at each iteration, a domain is randomly selected from the source domain S, and n images are randomly sampled from this domain as a meta-training set , according to the trained style random generation model based on local invariance, the style of n images is changed from the original image/> Convert to intermediate state/> , will generate image/> as a meta-test set , original image/> The corresponding generated image/> share the same semantic label.

In this embodiment, based on the random style generation model, the present invention randomly selects n images in a source domain as a meta-training set in each iteration, and randomly generates n style images from the meta-training set as a meta-test set, where The meta-training set and meta-test set share the same semantic labels, which promotes robust optimization by simulating domain shifts during training, thereby enabling the model to have good segmentation capabilities in unknown scenarios.

S4. Use the training data set M to train a generalization model based on the meta-learning paradigm, and promote robust optimization by simulating domain shifts during the training process to obtain a generalization model that can cope with scenarios in different styles. The generalization model uses a Encoder-decoder based segmentation network as the backbone network in the meta-learning paradigm; loss function of the segmentation network By cross entropy loss/> and Dice loss/> Composition, defined as:/> ,in;

cross entropy loss defined as:

Dice loss defined as:

In the above formula, M represents the total number of pixels in the image, C represents the number of categories, and/> Represent the predicted probability and true value of the c-th category of the i-th pixel respectively.

As shown in b in Figure 3, the meta-training set obtained above is trained through the backbone network to obtain the meta-training loss. , meta-training loss/> The expression is:

In the above formula, is the loss function of the segmentation network,/> is the meta-training set,/> About weight/> function, representing the gradient.

Updating meta-optimizer weights based on stochastic gradient descent and get the updated weight/> ,/> , and then in the meta-test set/> based on the updated weights/> Calculate meta-test loss , meta-test loss/> The expression is:

In the above formula, is the meta-test set,/> About weight/> function, representing the gradient.

Finally, the weights are updated via stochastic gradient descent ,/> , where,/> is the learning rate in the meta-test phase,/> is the learning rate in the meta-test phase.

In this process, the random selection of the source domain and the random value of the domain-independent random variable z in each iteration ensure that the meta-training set in the meta-training phase and the meta-test set in the meta-testing phase/> The differentiation of style, and the local invariance ensures the authenticity of the target area, so after training iterations, the generalization model can improve the robustness to image style, thereby achieving accurate segmentation of images in the invisible domain.

The image segmentation domain generalization method based on style transfer proposed in this embodiment realizes image generation through local texture invariant constraints and domain-independent random variables, and then constructs a scenario training scheme that simulates domain shift, making the model pay more attention to the image. content, reducing the dependence on style information, retaining the structural information of the image while generating diversified styles, effectively improving the segmentation accuracy of the model in unknown domains, and improving the efficiency of model deployment.

The present invention can develop a reusable image segmentation method based on convolutional neural network on unseen data sets, which can generate more realistic and diverse data, and design a training method based on diverse style images. Paradigm makes the model pay more attention to the content of the image and reduce its dependence on style information.

Those of ordinary skill in the art can understand that all or part of the steps in implementing the methods of the above embodiments can be completed by instructing relevant hardware through programs. Therefore, this application can adopt a complete hardware embodiment, a complete software embodiment, or a combination of software and Hardware embodiments. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

The above embodiments introduce the present invention in detail. Specific examples are used in this article to illustrate the principles and implementation modes of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for Those of ordinary skill in the art will make changes in the specific implementation and application scope based on the ideas of the present invention. In summary, the contents of this description should not be understood as limiting the present invention.

Claims (9)

1. The image segmentation domain generalization method based on style migration is characterized by comprising the following steps of:

s1, extracting texture structure features from a training data set P containing a plurality of source domains, wherein the texture structure features comprise original imagesAnd generate image +.>Partial binary pattern feature of class c> and />；

S2, training a random style generation model based on local invariance by adopting a training data set P, wherein the random style generation model comprises two generators、/>And two discriminators->、/>Is a loop generation countermeasure network;

s3, constructing a meta-learning training data set M based on a style random generation model, wherein the training data set M comprises a meta-training set and a meta-testing set;

s4, training a generalization model based on a meta-learning paradigm by adopting a training data set M, and promoting robust optimization by simulating domain offset in the training process so as to obtain the generalization model which can cope with scenes in different styles.

2. The image segmentation domain generalization method according to claim 1, specifically comprising, in step S1:

s11, extracting an original image from the training data set P according to the labelAnd generate image +.>Partial binary pattern feature of class c> and />；

S12, according to the local binary pattern characteristics and />Calculating cosine similarity to construct local similarity loss +.>Which is used to keep the texture in the mask area unchanged during the image style generation.

3. The image segmentation domain generalization method according to claim 2, characterized in that in step S12, local similarity is lostThe calculation formula of (2) is as follows:

；

in the above formula, C represents the number of categories C, and />Are all original images +.>And generate image +.>A local binary pattern feature of category c.

4. The image segmentation domain generalization method according to claim 1, characterized in that in step S2, two generators and />For the real images of style S respectively +.>And true image with style T +.>Generating image converted into intermediate form style +.>Wherein z represents a domain independent random variable;

the discriminators are used for distinguishing the real sample from the generated sample, and two discriminators and />Respectively to distinguish imagesAnd image->。

5. The method of image segmentation generalizing according to claim 1, characterized in that in step S2, a loss function of a model is generated in a random styleBy countering losses->Loss of cycle consistency>And local similarity loss->The composition is defined as: />；

Wherein, countering the lossThe definition is as follows:

；

wherein ,

；

；

in the above equation, z represents a domain independent random variable, and />Sets of pictures representing source domain and target domain, respectively,/->Representing computation from dataset +.>Is>Is>Representing computation from dataset +.>Is>Is a desired value of (2);

loss of cyclic uniformityThe definition is as follows:

；

in the above equation, z represents a domain independent random variable,representing a generator for converting a real image of style S into a generated image, < >>A generator for converting the real image representing the style T into a generated image;

loss of local similarityThe definition is as follows:

；

in the above formula, C represents the number of categories C, and />Are all original images +.>And generate image +.>A local binary pattern feature of category c.

6. The method of image segmentation domain generalization according to claim 5, characterized in that the domain independent random variable z e [0,1 ]]The closer the random variable z is to 0, the intermediate stateThe closer the picture style of (a) is to the original image +.>Whereas when the random variable z is closer to 1, the intermediate state +.>The closer the picture style of (a) is to the style image +.>。

7. The image segmentation domain generalization method according to claim 1, specifically comprising, in step S3:

s31, randomly selecting a domain from the source domain S, and randomly sampling n images from the domain as a meta-training set；

S32, randomly generating a model according to the trained styles, and changing the styles of the n images from the original imagesTransition to intermediate state->Will generate an image +.>As meta-test set->Original image +.>Corresponding to the generated image +.>Share the same semantic tags.

8. The image segmentation domain generalization method according to claim 1, characterized in that in step S4, the generalization model adopts a segmentation network based on encoder-decoder as a backbone network in a meta-learning paradigm.

9. The method of image segmentation domain generalization according to claim 8, characterized in that the loss function of the segmentation networkBy cross entropy loss->And Dice loss->The composition is defined as: />, wherein ；

cross entropy lossThe definition is as follows:

；

dice lossThe definition is as follows:

；

in the above equation, M represents the sum of pixels in the image, C represents the number of classes, and />Representing the predicted probability and the true value, respectively, of the c-th class of the i-th pixel.