WO2024125267A1 - Image processing method and apparatus, computer-readable storage medium, electronic device and computer program product - Google Patents

Abstract

Provided in the present disclosure are an image processing method and apparatus, a computer-readable storage medium, an electronic device and a computer program product. The method comprises: acquiring an original image to be processed, the original image comprising an object of a first category, and the object of the first category corresponding to a first area in the original image; on the basis of the first area, determining a mask image corresponding to the original image; segmenting the original image to obtain segmented images having at least some of semantics of preset categories; and, on the basis of the original image, the mask image and the segmented images, obtaining a target image, the target image comprising an object of a second category converted from the object of the first category in the original image. The embodiments of the present disclosure may allow target objects converted from designated objects in the target images to have clearer edges and richer texture details, thus making images more realistic, improving the display effect, and solving the problem of color anomaly.

Description

Image processing method, device, computer readable storage medium, electronic device and computer program product

This application claims priority to Chinese Patent Application No. 202211594249.0 filed on December 13, 2022, and the contents of the above-mentioned Chinese patent application disclosure are hereby cited in their entirety as a part of this application.

Technical Field

The present disclosure relates to an image processing method, an apparatus, a computer-readable storage medium, an electronic device, and a computer program product.

Background technique

Artificial intelligence technology is increasingly being used in the field of images. Artificial intelligence technology is usually used to convert a specified object in an original image into a target object, thereby obtaining a target image including the target object. At present, the target image obtained by processing the original image using related technologies has problems such as unclear edges of the target object, poor display effects of the processed area, abnormal colors, and missing texture details. Therefore, a solution is needed to convert a specified object in an original image into a target object.

Summary of the invention

The present disclosure provides an image processing method, an apparatus, a computer-readable storage medium, an electronic device, and a computer program product.

According to a first aspect, there is provided an image processing method, the method comprising:

Acquire an original image to be processed; the original image includes a first category of objects; the first category of objects corresponds to a first area in the original image;

Based on the first area, determining a mask image corresponding to the original image;

Performing segmentation processing on the original image to obtain a segmented image including at least part of the preset category semantics;

Based on the original image, the mask image and the segmented image, a target image is obtained; the target image includes a second category converted from the first category object in the original image Other objects.

According to a second aspect, an image processing apparatus is provided, the method comprising:

An acquisition module, configured to acquire an original image to be processed; the original image includes a first object of a first category; the first object of the first category corresponds to a first area in the original image;

A determination module, configured to determine a mask image corresponding to the original image based on the first area;

A segmentation module, used for performing segmentation processing on the original image to obtain a segmented image including at least part of the preset category semantics;

A processing module is used to obtain a target image based on the original image, the mask image and the segmented image; the target image includes second category objects converted from the first category objects in the original image.

According to a third aspect, a computer-readable storage medium is provided, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of the above-mentioned first aspects is implemented.

According to a fourth aspect, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method described in any one of the first aspects is implemented.

According to a fifth aspect, a computer program product is provided, comprising a computer program, wherein the computer program implements the method described in any one of the first aspects when executed by a processor.

It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

FIG1 is a schematic diagram of an image processing scene according to an exemplary embodiment of the present disclosure;

FIG2 is a schematic diagram of a training scenario of an image processing model according to an exemplary embodiment of the present disclosure;

FIG3 is a flow chart of an image processing method according to an exemplary embodiment of the present disclosure;

FIG4 is a flow chart of a method for training an image processing model according to an exemplary embodiment of the present disclosure;

FIG5 is a block diagram of an image processing device according to an exemplary embodiment of the present disclosure;

FIG6 is a schematic block diagram of an electronic device provided by some embodiments of the present disclosure;

FIG7 is a schematic block diagram of another electronic device provided by some embodiments of the present disclosure;

FIG. 8 is a schematic diagram of a storage medium provided by some embodiments of the present disclosure.

Detailed ways

In order to enable those skilled in the art to better understand the technical solutions in the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in the field without creative work should fall within the scope of protection of the present disclosure.

When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are only examples of devices and methods consistent with some aspects of the present disclosure as detailed in the attached claims.

The terms used in this disclosure are only for the purpose of describing specific embodiments and are not intended to limit the disclosure. The singular forms of "a", "the" and "the" used in this disclosure are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and/or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

It should be understood that although the terms first, second, third, etc. may be used in the present disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present disclosure, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

Artificial intelligence technology is increasingly being used in the field of images. Artificial intelligence technology is usually used to convert a specified object in an original image into a target object, thereby obtaining a target image that includes the target object. For example, the hairstyle of a person in a person image can be changed (e.g., long hair can be changed to short hair), or the person in a person image can be changed into a different costume, etc. At present, the target image obtained by processing the original image using related technologies has problems such as unclear edges of the target object, poor display effect of the processed area, abnormal colors, and missing texture details. Therefore, a solution is needed to convert a specified object in an original image into a target object.

The present disclosure provides an image processing solution, which combines the mask image and the segmented image corresponding to the original image, converts the first category object specified in the original image into the second category object to obtain the target image. Since the conversion is performed on the specified object in the original image, only the partial area corresponding to the specified object needs to be significantly changed. In the process of processing the image, the present solution combines the mask image and the segmented image, so that the process of converting the specified object in the original image is more targeted, and the changed area can be repaired through the guidance of the mask image and the segmented image. As a result, the edge of the target object in the target image is clearer, the texture details are richer, the image is more realistic, the display effect is improved, and the problem of color abnormality is solved.

Referring to Fig. 1, a schematic diagram of an image processing scenario according to an exemplary embodiment is shown. Referring to Fig. 1, the solution of the present disclosure is schematically described in combination with a complete and specific application example. The application example describes a specific image processing process.

As shown in FIG1 , the original image A1 is the image to be processed, and the original image A1 includes an object m. The mask image B1 corresponding to the original image A1 can be generated according to the region S corresponding to the object m in the original image A1. For example, the value of the pixel corresponding to the region S can be set to 1, and the value of the pixel of the region S' other than the region S can be set to 0. At the same time, the original image A1 can also be semantically parsed to obtain the segmented image C1 corresponding to the original image A1. The original image A1, the mask image B1 and the segmented image C1 are merged, for example, stacked, to obtain the merged data D1. The number of channels of the merged data D1 is the sum of the number of channels of the original image A1, the mask image B1 and the segmented image C1. For example, if the number of channels of the original image A1 is a, the number of channels of the mask image B1 is b, and the number of channels of the segmented image C1 is c, then the number of channels of the merged data D1 is a+b+c.

Next, the merged data D1 can be input into the image processing network, and the image processing network processes the merged data D1 of multiple channels. When processing the merged data D1, the same/different convolution kernels can be used in the shallow layer of the image processing network to perform convolution processing on each channel of the merged data D1, and the convolution processing results corresponding to each channel are added. Therefore, the information of the mask image B1 and the segmented image C1 can guide the image processing process.

Finally, the image processing network can output image data with a+b+c channels, where data with a channels can constitute the target image A2, data with b channels can constitute the mask image B2, and data with the remaining c channels can constitute the segmented image C2. The target image A2 includes object n, which is converted from object m included in the original image A1. The mask image B2 and the segmented image C2 are both for object n and correspond to the target image A2. The target image A2 can be obtained based on the image data with a+b+c channels output by the image processing network.

Referring to Fig. 2, a schematic diagram of a training scenario of an image processing model according to an exemplary embodiment is shown. Referring to Fig. 2, the scheme of the present disclosure is schematically described in combination with a complete and specific application example. The application example describes the training process of a specific image processing model.

As shown in FIG2 , first, a sample image A3, a sample mask image B3 corresponding to the sample image A3, a sample segmentation image C3 and a label image A5, and a label mask image B5 and a label segmentation image C5 corresponding to the label image A5 can be obtained from a pre-created training set. The sample image A3, the sample mask image B3 and the sample segmentation image C3 are merged to obtain merged data D2. The number of channels of the merged data D2 is the sum of the number of channels of the sample image A3, the sample mask image B3 and the sample segmentation image C3. For example, if the number of channels of the sample image A3 is a, the number of channels of the sample mask image B3 is b, and the number of channels of the sample segmentation image C3 is c, then the number of channels of the merged data D2 is a+b+c. The merged data D2 can be input into the image processing network to be trained, and the multi-channel merged data D2 can be processed by the image processing network to be trained.

Then, the image processing network to be trained can output image data with a+b+c channels, where data with a channels can constitute the predicted image A4, data with b channels can constitute the predicted mask image B4, and data with the remaining c channels can constitute the predicted segmentation image C4. The predicted image A4, the predicted mask image B4, and the predicted segmentation image C4 can be obtained according to the image data with a+b+c channels output by the image processing network to be trained. The predicted image A4 and the label image A5 are respectively input to the discriminator P to be trained, and the discriminator P can be used to discriminate the authenticity of the predicted image A4 and the label image A5.

Finally, based on the predicted image A4, the predicted mask image B4, the predicted segmented image C4 and the output of the discriminator P, the network parameters of the image processing network to be trained and the discriminator P can be adjusted in turn with reference to the label image A5, the label mask image B5 and the label segmented image C5. Specifically, in the discriminator training stage, the network parameters of the discriminator P can be adjusted based on the output of the discriminator P, so that its judgment results on the authenticity of the image are more and more accurate.

In the training phase of the image processing network, the prediction loss can be determined based on the prediction image A4, the prediction mask image B4, the prediction segmentation image C4, the result output by the discriminator P, the label image A5, the label mask image B5 and the label segmentation image C5. The prediction loss can be composed of the sum of loss items L1, L2, L3, L4, L5, L6 and L7, and the network parameters of the image processing network are adjusted with the goal of reducing the prediction loss.

Among them, loss terms L1, L2 and L3 are all determined based on the predicted image A4 and the label image A5. The loss term L1 can represent the difference in image features between the predicted image A4 and the label image A5, and can be specifically determined based on the mean absolute error of the pixel values of the pixels of the predicted image A4 and the label image A5. The loss term L2 can represent the difference in visual perception between the predicted image A4 and the label image A5, and the loss term L3 can represent the difference in image style between the predicted image A4 and the label image A5. Specifically, the predicted image A4 and the label image A5 can be respectively input into a pre-trained convolutional neural network to obtain two feature maps, and the loss term L2 is obtained by calculating the difference between the two feature maps. The loss term L3 is obtained by calculating the difference in the Lagum matrices corresponding to the two feature maps.

The loss term L4 is determined based on the difference between the target area M in the predicted image A4 and the label image A5, where the target area M is the area where there is a difference between the sample mask image B3 and the label mask image B5. The loss term L4 can represent the difference in color of the target area M between the predicted image A4 and the label image A5.

The loss term L5 is determined based on the predicted mask image B4 and the label mask image B5, and can be a weighted sum of the binary cross entropy loss and the regional mutual information loss between the predicted mask image B4 and the label mask image B5. The loss term L6 is determined based on the predicted segmented image C4 and the label segmented image C5, and can be a weighted sum of the binary cross entropy loss and the regional mutual information loss between the predicted segmented image C4 and the label segmented image C5. The loss term L7 is determined based on the result output by the above-mentioned discriminator P, and can be a loss term used to represent the loss of the generative adversarial network. It can be understood that the prediction loss can also include other loss terms, and this embodiment is not limited to this aspect.

The present disclosure will be described in detail below in conjunction with specific embodiments.

FIG3 is a flow chart of an image processing method according to an exemplary embodiment. The execution subject of the method can be implemented as any device, platform, server or device cluster with computing and processing capabilities. The method includes the following steps:

As shown in FIG. 3 , in step 301 , an original image to be processed is obtained.

In this embodiment, the original image is an image to be processed, and the original image includes a first category object. The processing of the original image involved in this embodiment may be to convert the first category object included in the original image into a second category object. For example, in one scenario, the original image may be a person image, and the object to be converted in the original image may be the person's hair, and the first category object may be, for example, the long hair of the person in the original image. Processing the original image may be to convert the long hair included in the original image into short hair to obtain a target image. The target image includes the short hair of the person after conversion (i.e., the second category object).

For another example, in another scenario, the original image may be a full-body image of a person, the object to be converted in the original image may be the person's clothing, and the first category object may be, for example, a long skirt of the person in the original image. Processing the original image may be converting the long skirt included in the original image into a short skirt to obtain a target image. The target image may include the short skirt of the person after conversion (i.e., the second category object). It can be understood that the present solution may also be applied in other scenarios, and the present embodiment does not limit the specific application scenarios.

In step 302, a mask image is determined based on a first region corresponding to an object of a first category in an original image.

In this embodiment, the first category object corresponds to the first area in the original image, and a mask image corresponding to the original image can be generated based on the first area. The mask image can be a single-channel binary image. For example, the value of the pixel corresponding to the first area can be set to 1 or 255, and the value of the pixel in other areas except the first area can be set to 0. Therefore, in the scene where the object to be converted is the hair of a person, the mask image corresponding to the original image presents an effect that the hair area is white and the other areas are black. In the scene where the object to be converted is the clothing of a person, the mask image corresponding to the original image presents an effect that the clothing area of the person is white and the other areas are black.

In step 303, the original image is segmented to obtain a segmented image.

In this embodiment, the original image may be subjected to, for example, semantic segmentation processing to obtain a segmented image including at least part of the semantics of a preset category. Optionally, a semantic segmentation model may be used to process the original image, wherein the semantic segmentation model is trained based on an image including the semantics of a preset category. For example, n preset categories of semantics may be set in advance according to the scene and needs, and then a semantic segmentation model may be trained based on the semantics of the n preset categories and an image including the semantics of the n preset categories. The semantic segmentation model is used to perform semantic segmentation processing on the original image, and the obtained semantic image includes any number of semantics of the n preset categories. For example, when the object to be converted is a person's head In the scenario of posting, the semantics of the n preset categories may include but are not limited to: background, hair, face, clothes, body skin, hands, glasses, hats, etc.

In step 304, a target image is obtained based on the original image, the mask image and the segmented image.

In this embodiment, based on the original image, the mask image and the segmented image, the first category object included in the original image can be converted into the second category object to obtain a target image. Optionally, the original image, the mask image and the segmented image can be converted using a pre-trained target model to obtain a target image. The target model can be a deep convolutional neural network model, and optionally, the target model can specifically be a u2ne type model. The target model can be trained by traditional supervised learning, or by a supervised adversarial generative network. It can be understood that this embodiment is not limited to this aspect.

Specifically, the original image, the mask image and the segmented image can be merged to obtain merged data, so that the number of channels of the merged data is the sum of the number of channels of the original image, the mask image and the segmented image. Among them, the merging process can be a channel-based merging process (such as stacking process, etc.). For example, the original image can be a 3-channel RGB image, the mask image can be a 1-channel binary image, and the segmented image can be an n-channel semantic segmentation image (n is the number of preset semantic categories), then after the merging process, 4+n channel merged data can be obtained. It should be noted that after the merging process, the information of each channel has not changed. The merged data can be input into the target model for conversion processing to obtain the target image.

The present disclosure provides an image processing method, which combines a mask image and a segmented image corresponding to the original image, and converts a first category object specified in the original image into a second category object to obtain a target image. Since the specified object in the original image is converted, only a part of the area corresponding to the specified object needs to be significantly changed. In the process of processing the image, the present solution combines the mask image and the segmented image, so that the process of converting the specified object in the original image is more targeted, and the changed area can be repaired through the guidance of semantic information. As a result, the edge of the target object converted from the specified object in the target image is clearer, the texture details are richer, the image is more realistic, the display effect is improved, and the problem of color abnormality is solved.

FIG4 is a flow chart of a training method for an image processing model according to an exemplary embodiment. The model may be the target model for processing images involved in the embodiment of FIG3 . The execution subject of the method may be implemented as any device, platform, server or device cluster with computing and processing capabilities. The method may include iteratively executing the following steps:

As shown in FIG. 4 , in step 401 , a sample image, a sample mask image, a sample segmented image, a label image, a label mask image and a label segmented image are obtained.

In this embodiment, a large number of candidate images for training can be collected in advance, each candidate image includes a first category object, and the candidate label image corresponding to each candidate image is obtained. The candidate label image corresponding to any candidate image includes a second category object converted from the first category object in the candidate image. For example, in a scene where the object to be converted is a person's hair, if the first category object is a person's long hair and the second category object is a person's short hair, then each candidate image includes the person's long hair, and the candidate label image corresponding to each candidate image includes the person's short hair converted from the person's long hair in the candidate image. For another example, in a scene where the object to be converted is a person's clothing, if the first category object is a person's long skirt and the second category object is a person's short skirt, then each candidate image includes a person's long skirt, and the candidate label image corresponding to each candidate image includes the person's short skirt converted from the person's long skirt in the candidate image.

It is understandable that the candidate label image corresponding to the candidate image can be obtained by any reasonable method. For example, the candidate image can be manually modified to obtain the candidate label image corresponding to the candidate image. The candidate image can also be processed by other models with similar functions but poor effects, and then manually modified to obtain the candidate label image corresponding to the candidate image. This embodiment does not limit the specific method for obtaining the candidate label image.

Then, obtain the mask image and segmentation image corresponding to each candidate image (the acquisition method can refer to the acquisition method of the mask image and segmentation image corresponding to the original image in the embodiment of FIG. 3), and obtain the mask image and segmentation image corresponding to each candidate label image. Then, use the candidate images and their corresponding mask images, segmentation images, candidate label images, and the mask images and segmentation images corresponding to the candidate label images to construct a training set.

When training the target model, an alternative image can be taken from the training set as a sample image, and a sample mask image, a sample segmentation image, and a label image corresponding to the sample image can be obtained, as well as a label mask image and a label segmentation image corresponding to the label image. The sample image includes a first category object, the first category object corresponds to a second region in the sample image, and the sample mask image is determined based on the second region. The label image is an image obtained by converting the first category object in the sample image into a second category object, the second category object corresponds to a third region in the label image, and the label mask image is determined based on the third region.

In step 402, based on the sample image, the sample mask image, the sample segmentation image and the training image, The trained target model is used to obtain the predicted image, predicted mask image and predicted segmentation image.

In this embodiment, the sample image, the sample mask image and the sample segmentation image may be merged (such as stacking) to obtain merged sample data. The number of channels of the merged sample data is the sum of the number of channels of the sample image, the sample mask image and the sample segmentation image. Then, the merged sample data is input into the target model to be trained, and the target model to be trained may output a predicted image, a predicted mask image and a predicted segmentation image corresponding to the predicted image. The predicted image includes a second category object converted from a first category object in the sample image.

It should be noted that the second category objects included in the label image and the second category objects included in the predicted image are not completely the same, although they are both converted from the first category objects included in the sample image. In addition, the label image is a reference image for comparison with the predicted image, which has a better display effect.

In step 403, the prediction loss is determined based on the predicted image, the predicted mask image, the predicted segmented image, the label image, the label mask image and the label segmented image, and in step 404, the model parameters of the target model to be trained are adjusted with the goal of reducing the prediction loss.

In this embodiment, the prediction loss can be determined based on the predicted image, the predicted mask image, the predicted segmented image, the label image, the label mask image and the label segmented image. The prediction loss may include a first loss term, a second loss term and a third loss term. The first loss term is determined based on the predicted image and the label image, and may be a loss term representing the difference in image features between the predicted image and the label image, and may be specifically determined based on the mean absolute error between the predicted image and the label image. The second loss term is determined based on the predicted mask image and the label mask image, and may be a weighted sum of a binary cross entropy loss and a regional mutual information loss between the predicted mask image and the label mask image. The third loss term is determined based on the predicted segmented image and the label segmented image, and may be a weighted sum of a binary cross entropy loss and a regional mutual information loss between the predicted segmented image and the label segmented image.

Optionally, the prediction loss may also include a fourth loss term, which is determined based on the difference between the target area in the predicted image and the label image. The target area is the area where there is a difference between the sample mask image and the label mask image. Specifically, the target area can be determined by calculating the absolute value of the difference between the pixel values corresponding to each pixel point of the sample mask image and the label mask image. Since the fourth loss term for guiding the adjustment of the model parameters is obtained based on the target area in this embodiment, the trained target model can process the specified object in the original image more accurately and more targeted.

Optionally, the fourth loss term can be determined in the following manner: Specifically, the first difference value of the target area in the predicted image and the label image under the first color space can be determined first, and the second difference value of the target area in the predicted image and the label image under the second color space can be determined, and the fourth loss term is determined based on the weighted sum of the first difference value and the second difference value. Among them, the first color space can be, for example, an RGB color space, and the second color space can be, for example, a LAB color space. The first difference value can be the mean absolute error of the target area in the predicted image and the label image under the RGB color space, and the second difference value can be the mean absolute error of the target area in the predicted image and the label image under the LAB color space. The corresponding weight of the first difference value is greater than the corresponding weight of the second difference value. Since this embodiment obtains the loss term for guiding the adjustment of model parameters based on the weighted sum of the difference values of the target area in the predicted image and the label image under different color spaces, the color effect of the image obtained by processing the original image by the trained target model is better.

Further optionally, the target model can be trained by a supervised generative adversarial network. Therefore, the predicted image and the label image need to be input into the discriminator to be trained, and the discriminator is adjusted based on the result output by the discriminator. In addition, the fifth loss term included in the prediction loss can also be determined based on the result output by the discriminator. The fifth loss term can be a loss term used to represent the loss of the generative adversarial network. It can be understood that the prediction loss can also include other loss terms, and this embodiment is not limited to this aspect.

In the process of training the target model, the present embodiment combines the label image, label mask image and label segmentation image corresponding to the sample image to obtain the prediction loss. The target model is trained based on the prediction loss, so that the process of converting the specified object in the original image by the target model is more targeted, so that the edge of the target object in the target image is clearer, the texture details are richer, the image is more realistic, the display effect is improved, and the problem of color abnormality is solved.

It should be noted that although the operations of the method of the disclosed embodiment are described in a specific order in the above embodiments, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. On the contrary, the steps depicted in the flowchart can change the order of execution. Additionally or alternatively, some steps can be omitted, multiple steps can be combined into one step for execution, and/or one step can be decomposed into multiple steps for execution.

Corresponding to the aforementioned image processing method embodiment, the present disclosure also provides an embodiment of an image processing device.

As shown in FIG5 , FIG5 is a block diagram of an image processing device according to an exemplary embodiment of the present disclosure, the device may include: an acquisition module 501, a determination module 502, a segmentation module 503 and Processing module 504.

The acquisition module 501 is used to acquire an original image to be processed, wherein the original image includes a first category object, and the first category object corresponds to a first area in the original image.

The determination module 502 is configured to determine a mask image corresponding to the original image based on the first area.

The segmentation module 503 is used to perform segmentation processing on the original image to obtain a segmented image including at least part of the preset category semantics.

The processing module 504 is used to obtain a target image based on the original image, the mask image and the segmented image. The target image includes objects of the second category converted from objects of the first category in the original image.

In some embodiments, the segmentation module 503 is configured to: process the original image using a semantic segmentation model to obtain a segmented image, wherein the semantic segmentation model is trained based on an image including a preset category semantics.

In some other implementations, the processing module 504 may include: a conversion submodule (not shown in the figure).

The conversion submodule is used to convert the original image, the mask image and the segmented image using the target model to obtain the target image. The conversion process is to convert the first category object into the second category object.

In other embodiments, the conversion submodule is configured to: merge the original image, the mask image and the segmented image to obtain merged data. The number of channels of the merged data is the sum of the number of channels of the original image, the mask image and the segmented image, and the merged segmentation is input into the target model for the above conversion process to obtain the target image.

In other embodiments, the target model is trained by iteratively executing the following steps: obtaining a sample image and a label image, wherein the second region in the sample image corresponds to a first category object, and the third region in the label image corresponds to a second category object. Based on the sample image and the target model to be trained, a predicted image is obtained, and a prediction loss is determined based on the predicted image and the label image. The prediction loss includes a fourth loss term, which is determined based on the difference between the target region in the predicted image and the label image. The target region is determined based on the second region and the third region, and the model parameters of the target model to be trained are adjusted with the goal of reducing the prediction loss.

In some other embodiments, the fourth loss term can be determined by: determining a first difference value between the target area in the predicted image and the label image in the first color space, determining a second difference value between the target area in the predicted image and the label image in the second color space, and based on the first difference value and the second difference value, The weighted sum of the two difference values determines the fourth loss term.

For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the embodiment scheme of the present disclosure. A person of ordinary skill in the art can understand and implement it without paying creative labor.

FIG6 is a schematic block diagram of an electronic device provided by some embodiments of the present disclosure. As shown in FIG6 , the electronic device 910 includes a processor 911 and a memory 912, which can be used to implement a client or a server. The memory 912 is used to store computer executable instructions (e.g., one or more computer program modules) non-transiently. The processor 911 is used to run the computer executable instructions, and when the computer executable instructions are run by the processor 911, one or more steps in the image processing method described above can be executed, thereby implementing the image processing method described above. The memory 912 and the processor 911 can be interconnected via a bus system and/or other forms of connection mechanisms (not shown).

For example, the processor 911 may be a central processing unit (CPU), a graphics processing unit (GPU), or other forms of processing units having data processing capabilities and/or program execution capabilities. For example, the central processing unit (CPU) may be an X86 or ARM architecture, etc. The processor 911 may be a general-purpose processor or a dedicated processor, and may control other components in the electronic device 910 to perform desired functions.

For example, the memory 912 may include any combination of one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and/or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and/or cache memory (cache), etc. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, erasable programmable read-only memory (EPROM), portable compact disk read-only memory (CD-ROM), USB memory, flash memory, etc. One or more computer program modules may be stored on the computer-readable storage medium, and the processor 911 may run one or more computer program modules to implement various functions of the electronic device 910. Various applications and various data, as well as various data used and/or generated by the application, etc. may also be stored in the computer-readable storage medium.

It should be noted that, in the embodiment of the present disclosure, the specific functions and technical effects of the electronic device 910 can refer to the above description of the image processing method, which will not be repeated here.

FIG7 is a schematic block diagram of another electronic device provided by some embodiments of the present disclosure. The electronic device 920 is suitable for implementing the image processing method provided by the embodiment of the present disclosure, for example. The electronic device 920 may be a terminal device, etc., and may be used to implement a client or a server. The electronic device 920 may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), wearable electronic devices, etc., and fixed terminals such as digital TVs, desktop computers, smart home devices, etc. It should be noted that the electronic device 920 shown in FIG7 is only an example, which does not impose any restrictions on the functions and scope of use of the embodiments of the present disclosure.

As shown in FIG. 7 , the electronic device 920 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 921, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 922 or a program loaded from a storage device 928 to a random access memory (RAM) 923. In the RAM 923, various programs and data required for the operation of the electronic device 920 are also stored. The processing device 921, the ROM 922, and the RAM 923 are connected to each other via a bus 924. An input/output (I/O) interface 925 is also connected to the bus 924.

Typically, the following devices may be connected to the I/O interface 925: input devices 926 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 927 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 928 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 929. The communication devices 929 may allow the electronic device 920 to communicate with other electronic devices wirelessly or by wire to exchange data. Although FIG. 7 shows an electronic device 920 having various devices, it should be understood that it is not required to implement or have all of the devices shown, and the electronic device 920 may alternatively implement or have more or fewer devices.

For example, according to an embodiment of the present disclosure, the above-mentioned image processing method can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the above-mentioned image processing method. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 929, or installed from the storage device 928, or installed from the ROM 922. When the computer program is executed by the processing device 921, the functions defined in the image processing method provided by the embodiment of the present disclosure can be implemented.

FIG8 is a schematic diagram of a storage medium provided by some embodiments of the present disclosure. For example, as shown in FIG8 , the storage medium 930 may be a non-transitory computer-readable storage medium for storing non-transitory When the non-transitory computer executable instruction 931 is executed by a processor, the image processing method described in the embodiment of the present disclosure can be implemented. For example, when the non-transitory computer executable instruction 931 is executed by a processor, one or more steps in the image processing method described above can be performed.

For example, the storage medium 930 may be applied to the above-mentioned electronic device. For example, the storage medium 930 may include a memory in the electronic device.

For example, the storage medium may include a memory card of a smart phone, a storage component of a tablet computer, a hard disk of a personal computer, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disk read-only memory (CD-ROM), flash memory, or any combination of the above storage media, or other applicable storage media.

For example, the description of the storage medium 930 can refer to the description of the memory in the embodiment of the electronic device, and the repeated parts are not repeated. The specific functions and technical effects of the storage medium 930 can refer to the description of the image processing method above, and are not repeated here.

It should be noted that in the context of the present disclosure, a computer-readable medium may be a tangible medium that may contain or store a program for use by an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. A computer-readable storage medium may be, for example, but not limited to: an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that may be used by an instruction execution system, device or device or used in combination with it. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may be sent, propagated, or transmitted for use by or in conjunction with an instruction execution system, apparatus, or device. Programs used. The program code contained on the computer readable medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the present disclosure. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The description and examples are to be regarded as exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims (10)

An image processing method, comprising: Acquire an original image to be processed, wherein the original image includes a first category of objects, and the first category of objects corresponds to a first area in the original image; Based on the first area, determining a mask image corresponding to the original image; Performing segmentation processing on the original image to obtain a segmented image including at least part of the preset category semantics; A target image is obtained based on the original image, the mask image and the segmented image, wherein the target image includes objects of the second category converted from objects of the first category in the original image. The method according to claim 1, wherein the segmenting of the original image to obtain a segmented image including at least part of the preset category semantics comprises: The original image is processed by using a semantic segmentation model to obtain the segmented image; wherein the semantic segmentation model is trained based on an image including the preset category semantics. The method according to claim 1 or 2, wherein obtaining the target image based on the original image, the mask image and the segmented image comprises: The target image is obtained by performing conversion processing on the original image, the mask image and the segmented image using the target model, wherein the conversion processing is a processing of converting the first category object into the second category object. The method according to claim 3, wherein the converting process of the original image, the mask image and the segmented image using the target model to obtain the target image comprises: Merging the original image, the mask image and the segmented image to obtain merged data, wherein the number of channels of the merged data is the sum of the number of channels of the original image, the mask image and the segmented image; The combined data is input into the target model for the conversion process to obtain the target image. According to the method of claim 3 or 4, the target model is trained by iteratively performing the following steps: Obtain a sample image and a label image; wherein the second area in the sample image corresponds to the first category object; the third area in the label image corresponds to the second category object; Based on the sample image and the target model to be trained, a predicted image is obtained; Determining a prediction loss based on the predicted image and the label image, wherein the prediction loss includes a fourth loss term, the fourth loss term is determined based on a difference between a target region in the predicted image and the label image, and the target region is determined based on the second region and the third region; With the goal of reducing the prediction loss, the model parameters of the target model to be trained are adjusted. The method according to claim 5, wherein the fourth loss term is determined by: Determine a first difference value of the target area in the predicted image and the label image in a first color space; Determine a second difference value of the target area in the predicted image and the label image in a second color space; The fourth loss term is determined based on a weighted sum of the first difference value and the second difference value. An image processing device, comprising: An acquisition module is configured to acquire an original image to be processed, wherein the original image includes a first category of objects, and the first category of objects corresponds to a first area in the original image; a determination module, configured to determine a mask image corresponding to the original image based on the first area; a segmentation module, configured to perform segmentation processing on the original image to obtain a segmented image including at least part of the preset category semantics; and The processing module is configured to obtain a target image based on the original image, the mask image and the segmented image, wherein the target image includes objects of the second category converted from objects of the first category in the original image. A computer-readable storage medium stores a computer program, wherein when the computer program is executed in a computer, the computer is caused to execute the method according to any one of claims 1 to 6. An electronic device comprises a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method according to any one of claims 1 to 6 is implemented. A computer program product comprises a computer program, wherein the computer program implements the method according to any one of claims 1 to 6 when being executed by a processor.