WO2025218588A1 - Image editing method and apparatus, device, and storage medium - Google Patents

Abstract

Embodiments of the present disclosure provide an image editing method and apparatus, a device, and a storage medium. The image editing method comprises: acquiring an area to be edited in an original image and a target prompt word corresponding to said area, wherein the target prompt word is text information used for describing an expected effect of image editing; determining a target mask image on the basis of said area, and adding preset noise to said area on the basis of the target mask image by means of an image editing model to obtain a local noise image; and on the basis of the target prompt word and by means of the image editing model, performing noise prediction processing and image generation processing on an area to be edited in the local noise image, and outputting a target image on the basis of a noise prediction result and an image generation result. The embodiments of the present disclosure solve the problem that image editing methods in the related art cannot accurately modify and edit local areas, thereby improving image-text consistency and image generation quality.

Description

Image editing method, device, equipment and storage medium

This application claims priority to Chinese Patent Application No. 202410455156.2 filed on April 15, 2024, and the contents of the above-mentioned Chinese patent application disclosure are hereby incorporated by reference in their entirety as a part of this application.

Technical Field

The embodiments of the present disclosure relate to an image editing method, apparatus, device, and storage medium.

Background Art

With the development of computer technology, the demand for editing local areas in original images based on text descriptions is increasing.

Currently, in image editing tasks based on text descriptions, semantic image editing will indiscriminately process the entire image and cannot accurately modify and edit the specified area, resulting in differences between the generated image and the text description, affecting the image generation quality.

Summary of the Invention

The present disclosure provides an image editing method, apparatus, device, and storage medium, which can accurately edit a specified area, improve image-text matching, and improve image generation quality.

In a first aspect, an embodiment of the present disclosure provides an image editing method, comprising:

Acquire a region to be edited in the original image and a target prompt word corresponding to the region to be edited, wherein the target prompt word is text information used to describe the expected effect of image editing;

Determining a target mask image according to the area to be edited, and adding preset noise to the area to be edited based on the target mask image using an image editing model to obtain a local noise image;

The image editing model performs noise prediction processing and image generation processing on the to-be-edited area of the local noise image based on the target prompt word, and outputs a target image according to the noise prediction result and the image generation result.

In a second aspect, an embodiment of the present disclosure further provides an image editing device, the device comprising:

An acquisition module, configured to acquire a region to be edited in an original image and a target prompt word corresponding to the region to be edited, wherein the target prompt word is text information used to describe an expected effect of image editing;

a noise adding module, configured to determine a target mask image according to the area to be edited, and add preset noise to the area to be edited based on the target mask image by an image editing model to obtain a local noise image;

An image generation module is used to perform noise prediction processing and image generation processing on the to-be-edited area of the local noise image based on the target prompt word through the image editing model, and output a target image according to the noise prediction result and the image generation result.

In a third aspect, an embodiment of the present disclosure further provides an electronic device, the electronic device comprising:

one or more processors;

a storage device for storing one or more programs,

When the one or more programs are executed by the one or more processors, the one or more processors implement the image editing method as described in any embodiment of the present disclosure.

In a fourth aspect, an embodiment of the present disclosure further provides a storage medium comprising computer-executable instructions, which, when executed by a computer processor, are used to execute the image editing method as described in any embodiment of the present disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other features, advantages, and aspects of the various embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that the originals and elements are not necessarily drawn to scale.

FIG1 is a schematic flow chart of an image editing method provided by an embodiment of the present disclosure;

FIG2 is a schematic diagram of an editing interface provided by an embodiment of the present disclosure;

FIG3 is a flow chart of another image editing method provided by an embodiment of the present disclosure;

FIG4 is a schematic diagram of training an image editing model provided by an embodiment of the present disclosure;

FIG5 is a schematic structural diagram of an image editing device provided by an embodiment of the present disclosure; and

FIG6 is a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure.

DETAILED DESCRIPTION

The following describes embodiments of the present disclosure in more detail with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and/or in parallel. In addition, the method embodiments may include additional steps and/or omit the steps shown. The scope of the present disclosure is not limited in this respect.

As used herein, the term "including" and its variations are open-ended, i.e., "including but not limited to." The term "based on" means "based, at least in part, on." The term "one embodiment" means "at least one embodiment," the term "another embodiment" means "at least one additional embodiment," and the term "some embodiments" means "at least some embodiments." Other terms are defined in the following description.

It should be noted that the concepts of "first" and "second" mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

It should be noted that the modifications of "one" and "multiple" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, they should be understood as "one or more".

The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

It is understandable that before using the technical solutions disclosed in the various embodiments of this disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved in this disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

For example, in response to a user's active request, a prompt message is sent to the user to clearly inform the user that the operation requested will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the electronic device, application, server, storage medium, or other software or hardware that performs the operations of the disclosed technical solution based on the prompt message.

As an optional but non-limiting implementation, in response to receiving a user's active request, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. Furthermore, the pop-up window may also contain a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device.

It is understandable that the above notification and user authorization process are merely illustrative and do not limit the implementation of the present disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of the present disclosure.

It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) must comply with the requirements of relevant laws, regulations and relevant provisions.

It should be noted that in the embodiments of the present disclosure, certain software, components, models and other existing solutions in the industry may be mentioned. They should be regarded as exemplary and their purpose is only to illustrate the feasibility of implementing the technical solution of the present disclosure, but it does not mean that the applicant has or will necessarily use the solution.

FIG1 is a flow chart of an image editing method provided by an embodiment of the present disclosure. The embodiment of the present disclosure is applicable to the situation of local editing. For example, the user inputs an area selection operation for the original image to determine the area to be edited in the original image, and inputs a target prompt word for the area to be edited. The image editing model generates a target object in the area to be edited based on the original image, the area to be edited, and the target prompt word, obtains a target image with the target object, and outputs it. The target object representation accurately generates the area to be edited based on the target prompt word. The method can be executed by an image editing device, which can be implemented in the form of software and/or hardware. Optionally, it can be implemented by an electronic device, which can be a mobile terminal, a PC, or a server, etc.

As shown in FIG1 , the method includes:

S110: Acquire a region to be edited in the original image and a target prompt word corresponding to the region to be edited.

In the embodiments of the present disclosure, the original image represents a picture or video frame of a video to be edited. The original image may include a picture or video frame stored in an electronic device. Alternatively, the original image may include a picture or video frame downloaded from a server. Alternatively, the original image may also include a picture or video frame taken by a user. If image editing operations are required for a local area, the original image may be displayed through an editing interface. The editing interface may be an interactive interface that is triggered and displayed by a local editing control in the client. The client may include an application, a mini-program, or a web client, etc. The local editing control is used to trigger the display of the editing interface.

The area to be edited represents the image area where image editing operations are to be performed. The area to be edited can be determined by performing an area selection operation on the original image in the editing interface. Area selection operations include smearing, framing, or sliding operations. For example, a smearing operation on the original image in the editing interface is obtained, and the area to be edited is determined based on the smeared area. Alternatively, a framing operation on the original image in the editing interface is obtained, and the area to be edited is determined based on the framed area. Alternatively, a sliding operation on the original image in the editing interface is obtained, and the area to be edited is determined based on the sliding trajectory.

The target prompt word is text information used to describe the expected effect of image editing. Specifically, the target prompt word represents a descriptive text of the target object to be generated in the area to be edited. For example, the target prompt word may represent a car, indicating that the expected effect of image editing is to replace the object in the area to be edited with a car.

Exemplarily, an original image is displayed in an editing interface; an area selection operation for the original image is obtained, and an area to be edited in the original image is determined based on the area selection operation; a text input operation for the area to be edited is obtained, and a target prompt word corresponding to the area to be edited is determined based on the text input operation.

FIG2 is a schematic diagram of an editing interface provided by an embodiment of the present disclosure. As shown in FIG2 , the editing interface 200 includes an editing area 210 and a result display area 220, etc. The editing area 210 includes an image loading control 230 and a prompt word input control 240. In response to a triggering event of the image loading control 230, the original image 250 is obtained, and the original image 250 is displayed at a position corresponding to the image loading control 230. In response to a smearing operation on the original image 250, the smeared area in the original image 250 is determined as the area to be edited. In response to a text input operation on the prompt word input control 240, the target prompt word corresponding to the area to be edited is obtained. The prompt word input control 240 includes a text box control.

S120 , determining a target mask image according to the area to be edited, and adding preset noise to the area to be edited based on the target mask image by an image editing model to obtain a local noise image.

The target mask is used to mask the non-to-be-edited area in the original image, prompting the image editing model to perform image editing on the to-be-edited area. For example, an original mask with the same size as the original image is generated, the target object in the to-be-edited area of the original image is identified, and the to-be-edited area in the original mask is determined based on the coordinates of the target object. The to-be-edited area in the original mask is filled with white, and the non-to-be-edited area is filled with black to obtain the target mask. Optionally, the target object in the to-be-edited area of the original image can be dilated to expand the boundary of the target object, and then the target mask is determined based on the dilated target object.

The image editing model may be a diffusion model trained based on a sample atlas, a sample mask atlas, and a descriptive text set. The sample mask atlas is determined based on a sample dilated image corresponding to a sample image in the sample atlas. The sample dilated image represents an image after a dilation operation is performed on a target region corresponding to the sample image. The descriptive text set is determined based on the image content of the target region. The target region may represent a semantically significant target object segmented in the sample image using a panoptic segmentation model.

The diffusion model includes an encoder, a noise prediction network, and a decoder. The encoder input includes the original image, which is used to compress the original image into a low-dimensional space to obtain a latent feature map. The decoder is used to restore the low-dimensional image, after completing the image editing task, to the original image size, obtaining the target image. The noise prediction network is used to add noise to the to-be-edited region in the latent feature map, subject to the constraints of the target mask map, while preserving the latent features in the non-to-be-edited regions, to obtain a local noise image. Furthermore, under the constraints of the target cue word, the to-be-edited region in the local noise image zT is subjected to noise prediction and image generation, obtaining the predicted noise and image-text correlation corresponding to time step t=T. Then, based on the image-text correlation and the local noise image zT, a local edited image corresponding to time step t=T is generated. The predicted noise corresponding to time step t=T is subtracted from the local edited image to obtain the local noise image zT-1 corresponding to time step t=T-1. The local noise image zT-1 corresponding to time step t=T-1 is input into the noise prediction network to generate the local noise image zT-2 at time step t=i-2, and so on, until the local noise image z0 corresponding to time step t=0 is generated, that is, the low-dimensional feature map of the target image is obtained.

Optionally, the noise prediction network can be a Unet network, comprising a convolutional layer, a downsampling layer, a downsampling layer based on a multi-head attention mechanism, an intermediate network, an upsampling layer based on a multi-head attention mechanism, and an upsampling layer. By introducing the multi-head attention mechanism, text and images are associated. The downsampling layer includes multiple residual modules. The downsampling layer based on the multi-head attention mechanism may include a residual module and a Transformer module, and the diffusion model may include at least one downsampling layer based on the multi-head attention mechanism. If there are multiple downsampling layers based on the multi-head attention mechanism, different downsampling layers based on the multi-head attention mechanism may include different numbers of Transformer modules. The intermediate network includes residual modules and Transformer modules. The upsampling layer based on the multi-head attention mechanism may include a residual module and a Transformer module, and the diffusion model may include at least one upsampling layer based on the multi-head attention mechanism. If there are multiple upsampling layers based on the multi-head attention mechanism, different upsampling layers based on the multi-head attention mechanism may include different numbers of Transformer modules. The upsampling layer includes multiple residual modules.

The preset noise can be noise whose distribution properties in the noise feature map meet preset requirements. For example, the preset noise can be random noise that meets Gaussian distribution, etc. Among them, the noise feature map can be a noise image with the same size as the original image. The noise feature area can be determined by superimposing the target mask map and the noise feature map. The noise feature area represents a specific area in the noise feature map that corresponds to the area to be edited. Since the area to be edited in the target mask map is white and the remaining areas are black, superimposing the target mask map on the noise feature map can cover the non-to-edit area in the noise feature map, thereby determining the noise feature area corresponding to the area to be edited.

The local noise image represents the image obtained by adding a preset noise to the region to be edited in the original image. Since only the region to be edited exhibits noise after the noise is added, the noisy image is considered the local noise image. Optionally, to reduce computational complexity, the original image is compressed into a low-dimensional latent space using an encoder to obtain a latent feature map of the original image. The preset noise is then added to the region to be edited in the latent feature map to obtain the local noise image.

Exemplarily, a target mask map is determined according to the area to be edited, and preset noise is added to the area to be edited based on the target mask map through an image editing model to obtain a local noise image, including: determining the target mask map according to the foreground content in the area to be edited; generating a potential feature map of the original image through the image editing model, and determining the area to be edited in the potential feature map by combining the target mask map and the potential feature map; performing a set number of noise adding operations based on the area to be edited in the potential feature map to obtain a local noise image.

For example, if a local area in the original image is smeared, the foreground content of the smeared area in the original image is identified to obtain the target object. A target mask image is generated based on the target object in the original image. The original image is compressed by the encoder in the image editing model to obtain a potential feature map of the original image. Since the area to be edited in the target mask image is white and the remaining areas are black, superimposing the target mask image on the potential feature map can cover the non-to-be-edited area in the potential feature map, thereby determining the area to be edited in the potential feature map. The image editing model performs a set number of noise addition operations based on the area to be edited in the potential feature map to obtain a local noise image.

In some embodiments, the latent feature map is represented as z0. The image editing model determines the region to be edited within z0 under the constraints of the target mask map. Random noise is then added to the region to be edited within z0 to obtain a local noise map z1. The image editing model then adds random noise to the region to be edited within the local noise map z1 to obtain a local noise map z2. The noise addition steps are iteratively performed until a local noise map zT is obtained.

Optionally, performing a set number of noise adding operations based on the area to be edited in the potential feature map to obtain a local noise image includes: obtaining a noise feature map, wherein the noise feature map represents a preset noise; determining a noise feature area in combination with the target mask map and the noise feature map; performing a set number of noise adding operations based on the area to be edited and the noise feature area to obtain a local noise image.

For example, the latent feature map is represented as z0, and the noise feature map is represented as S. z0, S, and the target mask map are input into the Unet network of the image editing model. The Unet network determines the region to be edited in z0 under the constraints of the target mask map. The Unet network also determines the noise feature region under the constraints of the target mask map. The noise features in the noise feature region are superimposed on the pixel features in the region to be edited in z0 to obtain the local noise map z1. Then, using the same method, random noise is added to the region to be edited in the local noise map z1 through the Unet network to obtain the local noise map z2. The noise addition step is iterated until the local noise map zT is obtained.

S130 , performing noise prediction processing and image generation processing on the to-be-edited area of the local noise image based on the target prompt word through the image editing model, and outputting a target image according to the noise prediction result and the image generation result.

Exemplarily, the image editing model determines predicted noise based on image features in the to-be-edited area of the local noise image; the image editing model generates text features based on the target prompt word, and determines the correlation between the text features and the local noise image; a local edited image is generated based on the correlation and the local noise image, and a denoising operation is performed on the local edited image based on the predicted noise, and a target image is output.

For example, the image editing model includes a text mapping module for mapping the target prompt word into a text vector as a text feature. The text features, the local noise image ZT, and the time step t=T are input into the Unet network of the image editing model. The Unet network outputs the predicted noise corresponding to time step t=T, calculates the correlation between the text features and the local noise image ZT, and adjusts the pixel distribution of the local noise image ZT based on the correlation to obtain the local edited image corresponding to time step t=T. The predicted noise corresponding to time step t=T is subtracted from the local edited image corresponding to time step t=T to obtain the local noise image at time step t=T-1. The text features, the local noise image ZT-1, and the time step t=T-1 are input into the Unet network of the image editing model. Using a similar method, the local noise image ZT-2 at time step t=T-2 can be obtained. Similarly, after performing T rounds of denoising and image generation, the local noise image Z0 at time step t=0 is obtained. The local noise image Z0 is input into the decoder of the image editing model, and the local noise image Z0 after denoising and image generation processing is decompressed by the decoder to restore it to the size of the original image to obtain the target image.

Optionally, the method further includes: obtaining mask setting information, the mask setting information including a blur level; determining predicted noise based on image features in the to-be-edited region of the local noise image using the image editing model; generating text features based on the target prompt word using the image editing model, and determining the correlation between the text features and the local noise image; determining the degree to which the generated object fills the to-be-edited region based on the blur level; generating a local edited image based on the correlation, the filling level, and the local noise image, wherein the to-be-edited region in the local edited image includes the generated object; and performing a denoising operation on the local edited image based on the predicted noise to output a target image.

The fill level represents the distribution of the generated object within the area to be edited. At different blur levels, the generated object fills the area to varying degrees. With a high fill level, the generated object is positioned close to the edge of the area to be edited. With a low fill level, the generated object is further away from the edge of the area to be edited.

For example, the editing interface also includes a mask setting control for inputting mask setting information. If a trigger operation for the mask setting control is detected, the mask setting information is obtained and parsed to obtain a blur level. After generating text features based on the target prompt word using the image editing model and determining the correlation between the text features and the local noise image, the degree to which the generated object fills the to-be-edited area is determined based on the blur level. A local edited image is generated based on the correlation, the filling level, and the local noise image, wherein the to-be-edited area in the local edited image includes the generated object. A denoising operation is then performed on the local edited image based on the predicted noise, and a target image is output.

The technical solution of the embodiment of the present disclosure accurately describes the expected editing effect of the area to be edited by the target prompt word for the area to be edited, determines the target mask map based on the area to be edited, adds preset noise to the area to be edited based on the target mask map through the image editing model, and obtains a local noise image. Then, the area to be edited of the local noise image is denoised and processed by the image editing model based on the target prompt word to generate a target object that meets the expected editing effect in the area to be edited, thereby achieving precise editing of the designated area and improving the image-text matching and image generation quality. Since only the area to be edited is subjected to noise addition and denoising, the difficulty of image processing and the color difference between the target object and the original image are reduced. The embodiment of the present disclosure solves the problem that the image editing method in the related art cannot accurately modify and edit local areas, and improves the consistency of image and text and the image generation quality.

Figure 3 is a flow chart of another image editing method provided by an embodiment of the present disclosure. The embodiment of the present disclosure is applicable to the situation of training an image editing model. Based on the above embodiments, the embodiment of the present disclosure additionally defines the training process of the image editing model.

As shown in FIG3 , the method includes:

S310: Obtain a sample atlas.

The sample atlas is a collection of sample images. The sample images may include countable instance objects, such as people, cars, and animals. The sample images may also include regions without fixed shapes, such as the sky, grass, snow, and trees.

S320 . For a sample image in the sample atlas, determine a target area in the sample image, perform a dilation operation on the target area to obtain a sample dilated image, and determine a sample mask atlas based on the sample dilated image corresponding to the sample atlas.

The target region represents an instance target and/or a region without a fixed shape in the sample image that carries semantic information. The dilation operation is a morphological operation in image processing that is used to enlarge the target region and roughen its boundaries.

A panoptic segmentation model is used to segment a sample image, obtaining instance targets and/or shapeless regions within the sample image. A dilation operation is performed on the instance targets and/or shapeless regions within the sample image to obtain a dilated sample image. By randomly dilating the sample image, the disclosed embodiment allows the target mask input during the model prediction process to exceed the target region boundary, completely resolving the model's incomplete fill problem. This makes the generated objects in the target image's to-be-edited region more realistic and natural, thereby improving image generation quality.

A sample mask atlas set is determined based on the sample dilated images corresponding to each sample image in the sample atlas. Exemplarily, for the sample dilated images corresponding to the sample atlas, a reference mask image of the sample image is generated based on a target region in the sample dilated image; Gaussian blur processing is performed on the reference mask image to obtain at least two sample mask images corresponding to the sample image; and a sample mask atlas set is determined based on the at least two sample mask images corresponding to the sample images in the sample atlas, wherein the at least two sample mask images are associated with the description text.

In the disclosed embodiment, a reference mask of the sample image is generated based on the target region in the sample dilated image. The to-be-edited region of the reference mask of the sample image is filled with white, while the non-to-be-edited region is filled with black. The reference mask includes instance masks and semantic masks.

Gaussian kernels of different sizes are used to perform Gaussian blur processing on the reference mask image to obtain at least two sample mask images with different degrees of roughness corresponding to the sample image. A sample mask atlas is formed based on at least two sample mask images corresponding to each sample image in the sample atlas to meet image editing needs of different precisions. For the case where the target area represents an instance target, the image editing model is trained by at least two sample mask images with different degrees of roughness, which can support fine instance target boundaries and rough rectangles as mask images to input the image editing model, and output a target image with a high degree of image-text matching. For the case where the target area represents an area without a fixed shape, the image editing model is trained by at least two sample mask images with different degrees of roughness, which can support fine instance target boundaries and rough rectangles as mask images to input the image editing model, and the generated object in the output target image fills the entire area to be edited.

S330 , performing image content recognition on the target area to obtain description text corresponding to the target area, and determining a description text set based on the description text of the target area corresponding to the sample atlas.

For example, a pre-set image description generation model is used to identify and understand the image content of the target area and output a description text corresponding to the target area. A description text set is formed based on the description text of the target area of each sample image in the sample atlas. A correlation is established between the description text of the sample image and the sample mask image to obtain an image-text data pair.

S340: Train a preset editing model based on the sample atlas, the sample mask atlas, and the description text set to obtain an image editing model.

The preset editing model may include an encoder, a noise prediction network, and a decoder. Optionally, the noise prediction network may be a Unet network, comprising a convolutional layer, a downsampling layer, a downsampling layer based on a multi-head attention mechanism, an intermediate network, an upsampling layer based on a multi-head attention mechanism, and an upsampling layer. By introducing a multi-head attention mechanism, text and images are associated. The downsampling layer includes multiple residual modules. The downsampling layer based on the multi-head attention mechanism may include a residual module and a Transformer module, and the diffusion model may include at least one downsampling layer based on the multi-head attention mechanism. If there are multiple downsampling layers based on the multi-head attention mechanism, different downsampling layers based on the multi-head attention mechanism may include different numbers of Transformer modules. The noise prediction network includes residual modules and Transformer modules. The upsampling layer based on the multi-head attention mechanism may include a residual module and a Transformer module, and the diffusion model may include at least one upsampling layer based on the multi-head attention mechanism. If there are multiple upsampling layers based on the multi-head attention mechanism, different upsampling layers based on the multi-head attention mechanism may include different numbers of Transformer modules. The upsampling layer consists of multiple residual modules.

Figure 4 is a schematic diagram of training an image editing model provided by an embodiment of the present disclosure. As shown in Figure 4, the encoder 410 input includes a sample image 420, which is used to compress the sample image 420 into a low-dimensional space to obtain a latent feature map 430. The decoder 440 is used to restore the low-dimensional image 450 that has completed the image editing task to the size of the sample image, obtaining an edited result image 460. The noise prediction network 470 is used to iteratively perform T noise addition operations on the to-be-edited region 490 in the latent feature map 430 under the constraints of the sample mask map 480, while maintaining the latent features of the non-to-be-edited regions unchanged, to obtain a local noise image 4100. Furthermore, under the constraints of the time step and the descriptive text, the to-be-edited region in the local noise image zT is subjected to noise prediction and image generation processing to obtain the predicted noise and image-text correlation corresponding to time step t = T. Then, based on the image-text correlation corresponding to time step t = T and the local noise image zT, a local edited image corresponding to time step t = T is generated. The predicted noise at time step t = T is subtracted from the local edited image at time step t = T to obtain the local noise image zT-1 at time step t = T-1. The local noise image zT-1 at time step t = T-1 is input into the noise prediction network to generate the local noise image zT-2 at time step t = i-2. This is repeated until the local noise image z0 at time step t = 0 is generated, thus obtaining the low-dimensional image 450 of the edited result image 460.

Low-dimensional image 450 is input to decoder 440, which decompresses low-dimensional image 450 to obtain an edited image 460 of the same size as sample image 420. A loss value is calculated between the generated object in the to-be-edited region of edited image 460 and the real instance object or the region with no fixed shape. The model parameters are updated based on the loss value during backpropagation, ultimately yielding a trained image editing model.

Optionally, after performing Gaussian blur processing on the reference mask image to obtain at least two sample mask images corresponding to the sample image, the method further includes: determining the blur level of the sample mask image based on the size of the Gaussian kernel used in the Gaussian blur processing, and marking the corresponding sample mask image according to the blur level. Since Gaussian blur processing is performed on the reference mask image using Gaussian kernels of different sizes, the resulting sample mask images have different degrees of roughness. The sample mask images can be marked with a blur level based on the size of the Gaussian kernel used in the Gaussian blur processing. For example, a fine sample mask image corresponds to a blur level of 0, and as the roughness of the sample mask image increases, the blur level corresponding to the sample mask image increases.

The method of training a preset editing model based on the sample atlas, the sample mask atlas, and the descriptive text set to obtain an image editing model includes: training the preset editing model based on the sample atlas, the sample mask atlas, the blur levels corresponding to the sample mask images, and the descriptive text set to obtain the image editing model. Specifically, the blur levels are mapped to blur level vectors, and the blur level vectors are injected into the noise prediction network so that the model learns the correspondence between different blur levels and sample mask images of different roughness levels, thereby controlling the degree to which the generated object fills the area to be edited.

The technical solution of the disclosed embodiment adds noise and denoise to the area to be edited, increasing local correlation and improving the matching degree between the edited image and the description text. This improves the generation stability of the model and avoids the generation errors that may occur during the denoising and denoising of the sample image as a whole, as the description text is a description of the entire image, resulting in the edited image not meeting expectations. For areas not to be edited, the latent features of the sample image are used as input in the denoising and denoising operations corresponding to all time steps of model training. This reduces the reconstruction difficulty of the model in image editing tasks, reduces the color difference of the generated objects, and improves model consistency.

Figure 5 is a structural diagram of an image editing device provided by an embodiment of the present disclosure. The device can be implemented in the form of software and/or hardware. Optionally, it can be implemented by an electronic device, which can be a mobile terminal, a PC or a server.

As shown in FIG5 , the apparatus includes: an acquisition module 510 , a noise adding module 520 and an image generating module 530 .

An acquisition module 510 is configured to acquire a region to be edited in an original image and a target prompt word corresponding to the region to be edited, wherein the target prompt word is text information used to describe an expected effect of image editing;

a noise adding module 520 for determining a target mask image according to the region to be edited, and adding preset noise to the region to be edited based on the target mask image by an image editing model to obtain a local noise image;

The image generation module 530 is configured to perform noise prediction processing and image generation processing on the to-be-edited area of the local noise image based on the target prompt word through the image editing model, and output a target image according to the noise prediction results and the image generation results.

Optionally, the image editing model is trained by:

Get the sample atlas;

For a sample image in the sample atlas, determining a target area in the sample image, performing a dilation operation on the target area to obtain a sample dilated image, and determining a sample mask atlas according to the sample dilated image corresponding to the sample atlas;

Performing image content recognition on the target area to obtain description text corresponding to the target area, and determining a description text set based on the description text of the target area corresponding to the sample atlas;

A preset editing model is trained based on the sample atlas, sample mask atlas and description text set to obtain an image editing model.

Furthermore, determining a sample mask atlas according to the sample dilation image corresponding to the sample atlas includes:

For the sample dilated image corresponding to the sample atlas, generating a reference mask image of the sample image according to the target area in the sample dilated image;

Performing Gaussian blur processing on the reference mask image to obtain at least two sample mask images corresponding to the sample image;

A sample mask atlas is determined according to at least two sample mask images corresponding to the sample images in the sample atlas.

Optionally, after performing Gaussian blur processing on the reference mask image to obtain at least two sample mask images corresponding to the sample image, the method further includes:

Determining a blur level of the sample mask image according to a size of a Gaussian kernel used in Gaussian blur processing, and marking a corresponding sample mask image according to the blur level;

The method of training a preset editing model based on the sample atlas, the sample mask atlas, and the description text set to obtain an image editing model includes:

A preset editing model is trained according to the sample atlas, the sample mask atlas, the blur levels corresponding to the sample mask images, and the description text set to obtain an image editing model.

Optionally, the acquisition module 510 is specifically configured to:

Display the original image in the editing interface;

Obtaining a region selection operation for the original image, and determining a region to be edited in the original image according to the region selection operation;

A text input operation for the area to be edited is acquired, and a target prompt word corresponding to the area to be edited is determined according to the text input operation.

Optionally, the noise adding module 520 is specifically configured to:

Determining a target mask image according to the foreground content in the area to be edited;

Generate a potential feature map of the original image by using the image editing model, and determine a region to be edited in the potential feature map by combining the target mask map and the potential feature map;

A noise adding operation is performed a set number of times based on the area to be edited in the potential feature map to obtain a local noise image.

Furthermore, performing a set number of noise addition operations based on the area to be edited in the potential feature map to obtain a local noise image includes:

Acquire a noise characteristic graph, wherein the noise characteristic graph represents a preset noise;

Determining a noise feature area by combining the target mask image and the noise feature image;

A noise adding operation is performed a set number of times based on the area to be edited and the noise feature area to obtain a local noise image.

Optionally, the image generation module 530 is specifically configured to:

Determining predicted noise according to image features in the to-be-edited area of the local noise image by the image editing model;

generating text features based on the target prompt word using the image editing model, and determining the correlation between the text features and the local noise image;

A local edited image is generated according to the correlation and the local noise image, a denoising operation is performed on the local edited image based on the predicted noise, and a target image is output.

Optionally, it also includes:

A level setting module, configured to obtain mask setting information, wherein the mask setting information includes a blur level;

Furthermore, generating a local edited image according to the correlation and the local noise image includes:

Determining a filling degree of the area to be edited by the generated object based on the blur level;

A local editing image is generated according to the correlation, the filling degree and the local noise image, wherein the area to be edited in the local editing image includes the generated object.

The image editing device provided by the embodiments of the present disclosure can execute the image editing method provided by any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

It is worth noting that the various units and modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the embodiments of the present disclosure.

FIG6 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure. Referring to FIG6 , a schematic diagram of the structure of an electronic device (such as a terminal device or server in FIG6 ) 600 suitable for implementing an embodiment of the present disclosure is shown below. The terminal device in the embodiment of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. The electronic device shown in FIG6 is merely an example and should not impose any limitations on the functions and scope of use of the embodiments of the present disclosure.

As shown in FIG6 , electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. Various programs and data required for the operation of electronic device 600 are also stored in RAM 603. Processing device 601, ROM 602, and RAM 603 are connected to each other via a bus 604. An input/output (I/O) interface 605 is also connected to bus 604.

Typically, the following devices may be connected to the I/O interface 605: an input device 606 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. Although FIG. 6 shows the electronic device 600 with various devices, it should be understood that not all of the devices shown are required to be implemented or present. More or fewer devices may alternatively be implemented or present.

In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 609, or installed from the storage device 608, or installed from the ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of the embodiment of the present disclosure are performed.

The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

The electronic device provided by the embodiment of the present disclosure and the image editing method provided by the above embodiment belong to the same inventive concept. For technical details not fully described in this embodiment, please refer to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

An embodiment of the present disclosure provides a computer storage medium having a computer program stored thereon. When the program is executed by a processor, the image editing method provided by the above embodiment is implemented.

It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device, or component. In the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

In some embodiments, the client and server can communicate using any currently known or later developed network protocol, such as HTTP (HyperText Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or later developed network.

The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device:

Acquire a region to be edited in the original image and a target prompt word corresponding to the region to be edited, wherein the target prompt word is text information used to describe the expected effect of image editing;

Determining a target mask image according to the area to be edited, and adding preset noise to the area to be edited based on the target mask image using an image editing model to obtain a local noise image;

The image editing model performs noise prediction processing and image generation processing on the to-be-edited area of the local noise image based on the target prompt word, and outputs a target image according to the noise prediction result and the image generation result.

Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including, but not limited to, object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and/or flowchart, and the combination of the boxes in the block diagram and/or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

The units involved in the embodiments described in this disclosure may be implemented in software or hardware, wherein the name of a unit does not necessarily limit the unit itself.

The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

The above description is merely a preferred embodiment of the present disclosure and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but also includes other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned disclosed concepts. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this disclosure.

In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details have been included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

Although the subject matter has been described in language specific to structural features and/or methodological logical acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims.

Claims (12)

An image editing method, comprising: Acquire a region to be edited in the original image and a target prompt word corresponding to the region to be edited, wherein the target prompt word is text information used to describe the expected effect of image editing; Determining a target mask image according to the area to be edited, and adding preset noise to the area to be edited based on the target mask image using an image editing model to obtain a local noise image; The image editing model performs noise prediction processing and image generation processing on the to-be-edited area of the local noise image based on the target prompt word, and outputs a target image according to the noise prediction result and the image generation result. The method according to claim 1, wherein the training method of the image editing model comprises: Get the sample atlas; For a sample image in the sample atlas, determining a target area in the sample image, performing a dilation operation on the target area to obtain a sample dilated image, and determining a sample mask atlas according to the sample dilated image corresponding to the sample atlas; Performing image content recognition on the target area to obtain description text corresponding to the target area, and determining a description text set based on the description text of the target area corresponding to the sample atlas; A preset editing model is trained based on the sample atlas, sample mask atlas and description text set to obtain an image editing model. The method according to claim 2, wherein determining a sample mask atlas based on the sample dilation image corresponding to the sample atlas comprises: For the sample dilated image corresponding to the sample atlas, generating a reference mask image of the sample image according to the target area in the sample dilated image; Performing Gaussian blur processing on the reference mask image to obtain at least two sample mask images corresponding to the sample image; A sample mask atlas set is determined according to at least two sample mask images corresponding to the sample images in the sample atlas set, wherein the at least two sample mask images are associated with description texts. The method according to claim 3, wherein, after performing Gaussian blur processing on the reference mask image to obtain at least two sample mask images corresponding to the sample image, the method further comprises: Determining a blur level of the sample mask image according to a size of a Gaussian kernel used in Gaussian blur processing, and marking a corresponding sample mask image according to the blur level; The method of training a preset editing model based on the sample atlas, the sample mask atlas, and the description text set to obtain an image editing model includes: A preset editing model is trained according to the sample atlas, the sample mask atlas, the blur levels corresponding to the sample mask images, and the description text set to obtain an image editing model. The method according to any one of claims 1 to 4, wherein obtaining the area to be edited in the original image and the target prompt word corresponding to the area to be edited comprises: Display the original image in the editing interface; Obtaining a region selection operation for the original image, and determining a region to be edited in the original image according to the region selection operation; A text input operation for the area to be edited is acquired, and a target prompt word corresponding to the area to be edited is determined according to the text input operation. The method according to any one of claims 1 to 5, wherein determining a target mask image according to the area to be edited, and adding preset noise to the area to be edited based on the target mask image using an image editing model to obtain a local noise image comprises: Determining a target mask image according to the foreground content in the area to be edited; Generate a potential feature map of the original image by using the image editing model, and determine a region to be edited in the potential feature map by combining the target mask map and the potential feature map; A noise adding operation is performed a set number of times based on the area to be edited in the potential feature map to obtain a local noise image. The method according to claim 6, wherein performing a set number of noise addition operations based on the area to be edited in the potential feature map to obtain a local noise image comprises: Acquire a noise characteristic graph, wherein the noise characteristic graph represents a preset noise; Determining a noise feature area by combining the target mask image and the noise feature image; A noise adding operation is performed a set number of times based on the area to be edited and the noise feature area to obtain a local noise image. The method according to any one of claims 1 to 7, wherein the performing noise prediction processing and image generation processing on the to-be-edited area of the local noise image based on the target prompt word by the image editing model, and outputting the target image according to the noise prediction result and the image generation result, comprises: Determining predicted noise according to image features in the to-be-edited area of the local noise image by the image editing model; generating text features based on the target prompt word using the image editing model, and determining the correlation between the text features and the local noise image; A local edited image is generated according to the correlation and the local noise image, a denoising operation is performed on the local edited image based on the predicted noise, and a target image is output. The method according to claim 8, further comprising: Obtaining mask setting information, wherein the mask setting information includes a blur level; Generating a local edited image according to the correlation and the local noise image includes: Determining a filling degree of the area to be edited by the generated object based on the blur level; A local editing image is generated according to the correlation, the filling degree and the local noise image, wherein the area to be edited in the local editing image includes the generated object. An image editing device, comprising: an acquisition module configured to acquire a region to be edited in an original image and a target prompt word corresponding to the region to be edited, wherein the target prompt word is text information used to describe an expected effect of image editing; a noise adding module configured to determine a target mask image according to the area to be edited, and add preset noise to the area to be edited based on the target mask image by an image editing model to obtain a local noise image; The image generation module is configured to perform noise prediction processing and image generation processing on the to-be-edited area of the local noise image based on the target prompt word through the image editing model, and output the target image according to the noise prediction result and the image generation result. An electronic device, comprising: one or more processors; A storage device for storing one or more programs, wherein: When the one or more programs are executed by the one or more processors, the one or more processors implement the image editing method according to any one of claims 1 to 9. A storage medium comprising computer-executable instructions, wherein the computer-executable instructions, when executed by a computer processor, are used to perform the image editing method according to any one of claims 1 to 9.