WO2025152653A1 - Method and apparatus for image processing, and device and storage medium - Google Patents

Abstract

According to the embodiments of the present disclosure, provided are a method and apparatus for image processing, and a device and a storage medium. The method comprises: acquiring a first reference image feature of a first reference image; determining a style-related feature of the first reference image on the basis of the first reference image feature, a first predetermined prompt for style feature extraction, and a first query representation; and generating a first target image on the basis of the style-related feature of the first reference image and first input text indicating target image content, wherein the first target image matches both the image style of the first reference image and the target image content. In this way, a style feature of a reference image can be decoupled from a semantic feature thereof, thus facilitating the generation of an image that has the style of the reference image and also meets the semantics of input text.

Description

Method, device, apparatus and storage medium for image processing

This application claims priority to the Chinese invention patent application entitled “Method, device, apparatus and storage medium for image processing” and application number 202410070352.8 filed on January 17, 2024, the entire contents of which are incorporated by reference into this application.

Technical Field

Example embodiments of the present disclosure generally relate to the field of computers, and more particularly, to methods, devices, apparatuses, and computer-readable storage media for image processing.

Background Art

In the field of computer vision (CV), various image generation techniques based on machine learning have been significantly developed and have been widely used. For example, in many application scenarios such as social networking, games, and image editing, it is expected to generate and use images with a specific style. Image generation techniques based on machine learning can be used in such application scenarios to improve the image generation effect.

Summary of the invention

In a first aspect of the present disclosure, an image processing method is provided. The method includes: obtaining a first reference image feature of a first reference image; determining a style-related feature of the first reference image based on the first reference image feature, a first predetermined prompt for style feature extraction, and a first query representation; and generating a first target image based on the style-related feature of the first reference image and a first input text indicating the content of the target image, the first target image matching the image style and the target image content of the first reference image.

In a second aspect of the present disclosure, a device for image processing is provided. The device includes: a first image feature extraction module configured to obtain a first reference image feature of a first reference image; a style-related feature module configured to determine the style-related features of the first reference image based on the first reference image feature, a first predetermined prompt for style feature extraction, and a first query representation; and a first target image generation module configured to generate a first target image based on the style-related features of the first reference image and a first input text indicating the content of the target image, the first target image matching the image style and the target image content of the first reference image.

In a third aspect of the present disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory, the at least one memory is coupled to the at least one processing unit and stores instructions for execution by the at least one processing unit. When the instructions are executed by the at least one processing unit, the device executes the method of the first aspect.

In a fourth aspect of the present disclosure, a computer-readable storage medium is provided, wherein a computer program is stored on the computer-readable storage medium, and the computer program can be executed by a processor to implement the method of the first aspect.

In a fifth aspect of the present disclosure, a computer program product is provided, which is tangibly stored in a computer storage medium and includes computer executable instructions, which when executed by a device cause the device to execute to implement the method of the first aspect.

It should be understood that the contents described in this content section are not intended to limit the key features or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

FIG1 is a schematic diagram showing an example environment in which embodiments of the present disclosure can be implemented;

FIG2 shows a schematic diagram of an example architecture for generating an image having a reference style and a target content according to some embodiments of the present disclosure;

FIG3 shows a schematic diagram of an example architecture for generating an image with reference content and a target style according to some embodiments of the present disclosure;

FIG4 shows a schematic diagram of an application of a cross-attention mechanism according to some embodiments of the present disclosure;

FIG5A shows a schematic diagram of a training task for style feature extraction according to some embodiments of the present disclosure;

FIG5B shows a schematic diagram of a training task for semantic feature extraction according to some embodiments of the present disclosure;

FIG5C shows a schematic diagram of a training task for image reconstruction according to some embodiments of the present disclosure;

FIG6 shows a flowchart of an image processing process according to some embodiments of the present disclosure;

FIG7 shows a block diagram of an apparatus for image processing according to some embodiments of the present disclosure; and

FIG8 shows a block diagram of a device capable of implementing various embodiments of the present disclosure.

DETAILED DESCRIPTION

It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

It should be noted that the titles of any sections/subsections provided herein are not restrictive. Various embodiments are described throughout this article, and any type of embodiment may be included under any section/subsection. In addition, the embodiments described in any section/subsection may be combined in any manner with any other embodiments described in the same section/subsection and/or different sections/subsections.

Herein, unless explicitly stated, executing a step “in response to A” does not mean executing the step immediately after “A” but may include one or more intermediate steps.

In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may be included below. The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may be included below.

As used herein, the term "model" can learn the association between the corresponding input and output from the training data, so that after the training is completed, the corresponding output can be generated for a given input. The generation of the model can be based on machine learning technology. Deep learning is a machine learning algorithm that processes inputs and provides corresponding outputs by using multiple layers of processing units. In this article, "model" may also be referred to as "machine learning model", "machine learning network" or "network", which are used interchangeably in this article. A model may also include different types of processing units or networks.

As used herein, a target image matches an image style, which may refer to the target image visually having a style consistent with or similar to the image style. A target image matches an image content, which may refer to the target image visually containing the same or similar content as the target content.

Example Environment

1 shows a schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented. In the environment 100 , an image processing system 120 , also referred to as system 120 , is deployed in an electronic device 110 . The image processing system 120 is configured to generate a target image 105 based on an input text 102 and a reference image 101 .

The reference image 101 can be used to provide or indicate an image element that is desired to be included in the target image 105, such as an image style, image content, etc. The input text can be used to indicate another image element that is desired to be included in the target image 105. As used herein, the term "image element" can indicate various explicit elements or implicit elements that constitute an image. Exemplarily, an image element can include an image style or image content. Examples of image styles include, but are not limited to, watercolor, crayons, sketches, comics, etc. Examples of image content can include, but are not limited to, at least a portion of the foreground of an image, at least a portion of the background of an image, etc.

In some embodiments, the image processing system 120 may generate an image having the same style as the reference image 101. Hereinafter, the image style of the reference image 101 is also referred to as the reference style. In such embodiments, the input text 102 may indicate the target image content. The generated target image 105 may match the image style and target image content of the reference image 101. For example, the target image 105 may have the image style of the reference image 101 and include the target image content.

Alternatively or additionally, in some embodiments, the image processing system 120 may generate an image having the same content as the reference image 101. In such embodiments, the input text 102 may indicate a target image style. The generated target image 105 may match the target image style and the image content in the reference image 101. For example, the target image 105 may contain the image content in the reference image 101 and have the style indicated by the input text 102.

In environment 100, electronic device 110 can be any type of device with computing capabilities, including terminal devices or server devices. The terminal device can be any type of mobile terminal, fixed terminal or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, personal communication systems (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio/video players, digital cameras/camcorders, positioning devices, television receivers, radio broadcast receivers, e-book devices, gaming devices, or any combination of the foregoing, including accessories and peripherals of these devices or any combination thereof. The server device can include, for example, a computing system/server, such as a mainframe, an edge computing node, an electronic device in a cloud environment, and the like.

It should be understood that the structure and functionality of environment 100 are described for exemplary purposes only and does not imply any limitation on the scope of the present disclosure.

As mentioned above, it is desirable to generate an image having a reference style of a reference image. To this end, in some conventional schemes, all or part of the parameters of a basic image generation model (such a model can generate an image based on input text) can be fine-tuned to generate a stylized image. However, such a scheme is time-consuming (e.g., at least minutes) and requires manual intervention.

In other conventional schemes, a two-stage encoder is sampled, which includes a fixed backbone network and a head network that needs to be trained to extract features of the reference image. The features obtained by the encoder can be further combined with a denoising network. However, in this conventional method, the training task is usually a reconstruction task, so what the encoder learns is a mixture of content and style. This causes the semantics in the reference image to conflict with the semantics indicated by the input text, resulting in the generated image possibly containing the semantics of the reference image.

To this end, an embodiment of the present disclosure provides an improved solution for image generation. In the embodiment of the present disclosure, a predetermined prompt and query representation specific to style feature extraction are introduced to extract style-related features in a reference image. Then, a target image matching the image style and target image content of the reference image can be generated based on the extracted style-related features and input text indicating the content of the target image.

In the embodiments of the present disclosure, feature extraction is performed using predetermined prompts and query representations specific to style feature extraction to decouple the style features of the reference image from the semantic features. This can alleviate the inconsistency problem between the semantics in the reference image and the semantics of the input text. As a result, an image that has both the style of the reference image and the semantics of the input text can be advantageously generated.

Example Image Generation Architecture

In some embodiments, input text 102 may indicate the content that the target image is expected to have, also referred to as target image content or simply target content. In such embodiments, reference image 101 indicates an image style. Target image 105 has a reference style and target content. FIG. 2 shows a schematic diagram of an example architecture 200 for generating an image with a reference style and target content according to some embodiments of the present disclosure. Architecture 200 may be implemented in system 120. For the extraction of image features, architecture 200 adopts a two-stage structure.

As shown in FIG2 , the reference style of the reference image 201 is style A. The system 120 may obtain image features of the reference image 201, also referred to as reference image features 202. Exemplarily, the image encoder 220 may generate the reference image features 202 based on the reference image 201. The image encoder 220 may adopt any suitable network structure, and the embodiments of the present disclosure are not limited in this respect.

The reference image features 202 can be used as inputs to the feature transformation model 230. In addition, the feature transformation model 230 has other inputs, including a predetermined prompt 204 (also referred to as a first predetermined prompt) for style feature extraction and a query representation 203 (also referred to as a first query representation) for style feature extraction. The predetermined prompt 204 can be any suitable text or prompt word. In this example, the predetermined prompt 204 is shown as the text "style" for exemplary purposes only and is not intended to be limiting. In an embodiment of the present disclosure, the first predetermined prompt is specific to style feature extraction and is independent of the reference image. That is, the first predetermined prompt does not change with the reference image. The feature transformation model 230 can be implemented with a network of any suitable structure. As an example and without any limitation, the feature transformation model 230 can be a query transformer.

The query representation 203 is learnable. The query representation 203 may be obtained during the training of the feature transformation model 230. For example, the query representation 203 may be initialized, and during the training of the feature transformation model 230, the query representation 203 may be updated together until the training is completed and solidified. Such a query representation 203 may be used to instruct the feature transformation model 230 to extract features related to the image style. As shown in FIG. 2 , the query representation 203 may be a vectorized representation of any suitable dimension. The query representation 203 is obtained for style feature extraction.

The feature transformation model 230 can generate style-related features 205 of the reference image 201 based on the reference image features 202, the first predetermined hint 204, and the first query representation 203. In other words, under the hint or guidance of the first predetermined hint 204 and the first query representation 203, the feature transformation model 230 can extract features related to the image style from the reference image 201.

Next, the system 120 can generate a target image 210 based on the style-related features 205 and the input text 209 indicating the content of the target image. The input text 209 can indicate the content that the target image is expected to contain in any suitable characters, such as but not limited to people, animals, objects, scenery, buildings, etc. In this example, the target content indicated by the input text 209 is "panda", but this is only exemplary and not intended to be any limitation. As shown in Figure 2, the target image 210 has the style of the reference image 201 (i.e., style A) and contains the content indicated by the input text 209 (i.e., panda).

System 120 may use any suitable algorithm or model to generate the target image. As shown in FIG2 , in some embodiments, model 240, also referred to as a first model, may be used. Model 240 may be implemented using any suitable network structure. Exemplarily, model 240 may be a diffusion model that may perform multiple denoising steps. Model 240 may also be of other types, such as a generative adversarial network.

Specifically, the text encoder 250 can generate text features 207 of the input text 209. The text features 207 can be provided to the model 240 as text conditions. In addition, the model 240 can also receive style-related features 205. The model 240 can generate target image features 208 based on the text features 207 and the style-related features 205. The target image features 208 can then be used to generate the target image 210. For example, the system 120 can utilize an image decoder (not shown) to convert the target image features 208 into the target image 210. The embodiments of the present disclosure are not limited in how to convert the target image features into the target image. In addition, in each step (e.g., a denoising step) performed using the model 240, the model 240 also receives the image features output by the previous step as input.

In some embodiments, the model 240 may include multiple processing layers, and the multiple processing layers are respectively used to process features of corresponding sizes. In this embodiment, the style-related features 205 are provided to a first number of processing layers whose sizes are greater than the first threshold size. For example, the model 240 can be a denoising U-shaped network. Exemplarily, as shown in Figure 2, the processing layers close to the model input and output are used to process features of larger sizes, while the processing layers close to the middle of the model are used to process features of smaller sizes. Accordingly, the style-related features 205 are provided to the processing layers close to the model input and output. As an example and without any limitation, it is assumed that the model 240 includes 16 processing layers, which are numbered 0, 1, ..., 15 in sequence. The style-related features 205 can be provided to layers 0 to 3 and 9 to 15. The combination of style-related features and text features will be described below with reference to Figure 4.

Features with larger sizes include more details of the image and correspond to the fine processing layer. It is generally believed that such fine processing layers are mainly responsible for the generation of style-related elements such as color and structure of the image. Features with smaller sizes include high-level semantic information of the image and correspond to the coarser processing layer. It is generally believed that such coarser processing layers are mainly responsible for the semantic generation of the image. Therefore, by injecting style-related features only into the fine processing layer and not into the coarser processing layer, the style and semantics of the reference image can be further decoupled. In this embodiment, it is advantageous to enable the model 240 to focus more on the style of the reference image in image generation.

An example implementation of generating an image having a reference style and a target content is described above. An example implementation of generating an image having a target style and a reference content is described below.

In some embodiments, the input text 102 may indicate the style of the desired target image, also referred to as the target image style or simply the target style. In this embodiment, the reference image 101 indicates the image content, also referred to as the reference content. The target image 105 has the reference content and the target style. FIG3 shows a schematic diagram of an example architecture 300 for generating an image with reference content and a target style according to some embodiments of the present disclosure. The architecture 300 may be implemented in the system 120. For the extraction of image features, the architecture 300 adopts a two-stage structure. Note that the text encoder 350 in FIG3 may be the same or different from the text encoder 250 in FIG2, and the image encoder 320 in FIG3 may be the same or different from the image encoder 220 in FIG2. The embodiments of the present disclosure are not limited in this respect.

As shown in FIG3 , the reference style of the reference image 301 is style A, and the image content is a smiling face. The system 120 may obtain image features of the reference image 301, also referred to as reference image features 302. Exemplarily, the image encoder 320 may generate the reference image features 302 based on the reference image 301. The image encoder 320 may adopt any suitable network structure, and the embodiments of the present disclosure are not limited in this respect.

The reference image features 302 can be used as inputs to the feature transformation model 330. In addition, the feature transformation model 330 has other inputs, including a predetermined prompt 304 (also referred to as a second predetermined prompt) for semantic feature extraction and a query representation 303 (also referred to as a second query representation) for semantic feature extraction. The predetermined prompt 304 can be any suitable text or prompt word. In this example, the predetermined prompt 304 is shown as the text "content" for exemplary purposes only and is not intended to be limiting. In an embodiment of the present disclosure, the second predetermined prompt is specific to semantic feature extraction and is independent of the reference image. That is, the second predetermined prompt does not change with the reference image. The feature transformation model 230 can be implemented with a network of any suitable structure. As an example and without any limitation, the feature transformation model 230 can be a query transformer.

The query representation 303 is learnable. The query representation 303 may be obtained during the training of the feature transformation model 330. For example, the query representation 303 may be initialized, and during the training of the feature transformation model 330, the query representation 303 may be updated together until the training is completed and solidified. Such a query representation 303 may be used to instruct the feature transformation model 330 to extract features related to the image content, that is, semantically related features. As shown in FIG. 3 , the query representation 303 may be a vectorized representation of any suitable dimension. The query representation 303 is obtained for semantic feature extraction.

The feature transformation model 330 can generate semantically relevant features 305 of the reference image 301 based on the reference image features 302, the second predetermined prompt 304, and the second query representation 303. In other words, under the prompt or guidance of the second predetermined prompt 304 and the second query representation 303, the feature transformation model 330 can extract features related to the image semantics from the reference image 301.

Next, the system 120 can generate a target image 310 based on the semantically relevant features 305 and the input text 309 indicating the style of the target image. The input text 309 can indicate the style that the target image is expected to have in any suitable characters. In this example, the target style indicated by the input text 309 is "Style B", but this is only exemplary and not intended to be limiting. As shown in Figure 3, the target image 310 has the content of the reference image 301 (i.e., a smiling face) and contains the style indicated by the input text 309 (i.e., Style B).

Similar to what is described with reference to FIG. 2 , the system 120 may use any suitable algorithm or model to generate the target image 310. As shown in FIG. 3 , in some embodiments, the model 240 may be used. Specifically, the text encoder 350 may generate text features 307 of the input text 309. The text features 307 may be provided to the model 240 as text conditions. In addition, the model 240 may also receive semantically relevant features 305. The model 240 may generate target image features 308 based on the text features 307 and the semantically relevant features 305. The target image features 308 may then be used to generate the target image 310. For example, the system 120 may utilize an image decoder (not shown) to convert the target image features 308 into the target image 310. The embodiments of the present disclosure are not limited in how to convert the target image features into the target image. In addition, in each step (e.g., a denoising step) performed using the model 240, the model 240 also receives the image features output by the previous step as input.

In some embodiments, model 240 may include multiple processing layers, and multiple processing layers are used to process features of corresponding sizes. In this embodiment, semantically relevant features 305 are provided to a second number of processing layers whose size is less than a second threshold size. The second threshold size may be the same or different from the first threshold size described above. For example, model 240 may be a denoising U-shaped network. Exemplarily, as shown in Figure 3, the processing layers close to the model input and output are used to process features with larger sizes, while the processing layers close to the middle of the model are used to process features with smaller sizes. Accordingly, semantically relevant features 305 are provided to the processing layers close to the middle of the model. As an example and without any intention of limitation, it is assumed that model 240 includes 16 processing layers, which are numbered 0, 1, ..., 15 in sequence. Semantically relevant features 305 may be provided to 4 to 8 layers. The combination of semantically relevant features and text features will be described below with reference to Figure 4.

As described with reference to FIG2 , features with larger sizes include more details of the image and correspond to a fine processing layer. It is generally believed that such a fine processing layer is mainly responsible for the generation of style-related elements such as color and structure of the image. Features with smaller sizes include high-level semantic information of the image and correspond to a coarser processing layer. It is generally believed that such a coarser processing layer is mainly responsible for the semantic generation of the image. Therefore, by injecting semantically relevant features only into the coarser processing layer and not into the fine processing layer, the style and semantics of the reference image can be further decoupled. In this embodiment, the model 240 can be made to focus more on the content of the reference image in image generation.

Combination of reference image features with text features

As mentioned above, in some embodiments, style-related features and semantic-related features can be provided to certain processing layers of model 240. Hereinafter, for the purpose of illustration only, style-related features and semantic-related features are collectively referred to as image-related features. In such processing layers, image features and text features of input text can be used as conditions for target image generation. In some embodiments, model 240 can be based on a cross-attention mechanism. Accordingly, in the processing layers, cross-attention related to image-related features and text features can be applied.

FIG4 shows a schematic diagram of the application of the cross-attention mechanism according to some embodiments of the present disclosure. In FIG4 , the corresponding size is shown below each feature. As shown in FIG4 , in a certain processing layer, the input image feature 411 of the processing layer (which is represented by Z and can come from the previous processing layer or the previous denoising step) is converted into a query feature 421. By converting the text feature 412 (which is represented by C _t ) and the image-related feature 413 (which is represented by C _i and can be a semantic-related feature or a style-related feature), a key feature 431 and a value feature 432 are generated. Then, based on the query feature 421, the key feature 431 and the value feature 432, the output image feature 450 of the processing layer can be generated. As shown in FIG4 , the dot product of the query feature 421 and the key feature 431 can obtain an attention map. The output image feature 450 can be obtained by matrix multiplication of the attention map and the value feature 432.

In some embodiments, the parameters of such a processing layer may be obtained through different training modes, and the different training modes may include, for example, pre-training and fine-tuning. As shown in FIG4 , the processing layer may include a first conversion unit 401 for converting an input image feature 411 into a query feature 421. The processing layer may also include a second conversion unit 402, a third conversion unit 403, a fourth conversion unit 404, and a fifth conversion unit 405. The second conversion unit 402 is used to convert a text feature 412 into a first intermediate text feature 422 as part of a key feature. The third conversion unit 403 is used to convert a text feature 412 into a second intermediate text feature 423 as part of a value feature. The fourth conversion unit 404 is used to convert an image-related feature 413 into a first intermediate image-related feature 424 as part of a key feature. The fifth conversion unit 405 is used to convert an image-related feature 413 into a second intermediate image-related feature 425 as part of a value feature.

The first intermediate text feature 422 and the first intermediate image-related feature 424 may be combined into a key feature 431, for example, may be concatenated into the key feature 431. Similarly, the second intermediate text feature 423 and the second intermediate image-related feature 425 may be combined into a value feature 432, for example, may be concatenated into the value feature 432.

In some embodiments, the first conversion unit 401, the second conversion unit 402, and the third conversion unit 403 are obtained through a first training mode, and the fourth conversion unit 404 and the fifth conversion unit 405 are obtained through a second training mode different from the first training mode. For example, the first conversion unit 401, the second conversion unit 402, and the third conversion unit 403 may be obtained through pre-training, and the fourth conversion unit 404 and the fifth conversion unit 405 may be obtained through fine-tuning. In this way, additional units can be added to the pre-trained base model, so that the additional units can be fine-tuned without training the parts included in the base model. This can effectively reduce the training cost.

In the above-described embodiment, the semantic information and style information in the reference image are decoupled. This can be regarded as a double decoupled representation extraction (DDRE). Specifically, a predetermined prompt text specific to style feature extraction (e.g., "style") and a predetermined prompt text specific to semantic feature extraction (e.g., "content") are input to the feature transformation model respectively, so that the feature transformation model can obtain image features aligned with the prompt text. That is, style-related features and semantic-related features can be obtained respectively.

Training tasks and samples

In order to achieve the above-mentioned dual disentangled representation extraction, various suitable training tasks can be performed during training.

In some embodiments, a training task for style feature extraction, also known as a style representation extraction (STRE) task, may be performed. In the STRE task, pairs of different sample images with the same style are used for training. FIG5A shows a schematic diagram of a training task 500A for style feature extraction according to some embodiments of the present disclosure.

In the training task 500A, a pair of sample images 511 and sample images 520 have the same style, for example, style A. The sample image 511 is used as a reference, and the sample image 520 is used as a target. In the training, the first query representation 203 is initialized. The feature transformation model 230 can generate style-related features 515 of the sample image 511 based on the first sample image features of the sample image 511 (for example, the output of the image encoder 220), the first predetermined prompt 204, and the first query representation 203. With the style-related features 515 of the sample image 511 and the text 519 indicating the image content of the sample image 520 as conditions, the feature transformation model 230 and the initialized first query representation 203 are updated by denoising and denoising the sample image 520. When the predetermined condition is met, the updating of the feature transformation model 230 and the first query representation 203 is stopped. In some embodiments, if the model 240 includes a trainable conversion unit (for example, as described with reference to FIG. 4 ), such a conversion unit is also updated with the training.

The STRE task described above is a non-reconstruction task. Through such a training task, it is beneficial to decouple the style and semantics of the reference image, and it can also ensure that the image information (in this example, the style information) during the training process is not too strong, thereby "drowning" the prompt information of the input text.

In some embodiments, a training task for semantic feature extraction, also known as a semantic representation extraction (SERE) task, may be performed. In the SERE task, pairs of different sample images with the same semantics are used for training. FIG5B shows a schematic diagram of a training task 500B for semantic feature extraction according to some embodiments of the present disclosure.

In the training task 500B, a pair of sample images 521 and sample images 530 have the same content, such as musical notes. The sample image 521 is used as a reference and the sample image 530 is used as a target. In the training, the second query representation 303 is initialized. The feature transformation model 330 can generate semantically relevant features 525 of the sample image 521 based on the sample image features of the sample image 531 (e.g., the output of the image encoder 320), the second predetermined prompt 304, and the second query representation 303. With the semantically relevant features 525 of the sample image 521 and the text 529 indicating the style of the sample image 530 as conditions, the feature transformation model 330 and the initialized second query representation 303 are updated by denoising and denoising the sample image 530. When the predetermined condition is met, the updating of the feature transformation model 330 and the second query representation 303 is stopped. In some embodiments, if the model 240 includes a trainable conversion unit (e.g., as described with reference to FIG. 4), such a conversion unit is also updated with the training.

The SERE task described above is a non-reconstruction task. Through such a training task, it is beneficial to decouple the style and semantics of the reference image, and it can also ensure that the image information (semantic information in this example) during training is not too strong, thereby "drowning" the prompt information of the input text.

In some embodiments, in order to avoid image information omission caused by non-reconstruction tasks, a reconstruction task may be additionally performed. Fig. 5C shows a schematic diagram of a training task 500C for image reconstruction according to some embodiments of the present disclosure.

5C , the upper half branch is a style branch. During training, the feature transformation model 230 can generate style-related features 533 of the sample image 531 based on the sample image features of the sample image 531 (eg, output of the image encoder 220 ), the first predetermined hint 204 , and the first query representation 203 .

In the semantic branch, semantically relevant features 532 of the sample image 531 may be obtained. For example, the acquisition of the semantically relevant features 532 may utilize the feature transformation model 330. As shown in the figure, the feature transformation model 330 may generate the semantically relevant features 532 of the sample image 531 based on the sample image features of the sample image 531 (e.g., the output of the image encoder 320), the second predetermined hint 304, and the second query representation 303.

In this way, the feature transformation model 230 and the first query representation 203 can be updated by reconstructing the sample image 531 with the style-related features 533 and the semantic-related features 532 as conditions. Accordingly, the feature transformation model 330 and the second query representation 303 are also updated. In some embodiments, if the model 240 includes a trainable transformation unit (e.g., as described with reference to FIG. 4 ), such a transformation unit is also updated with the training based on the reconstruction task.

As described above, in the execution of training tasks 500A, 500B and 500C, feature transformation model 230, feature transformation model 330, first query representation 203 and second query representation 303 are trainable, model 240 is partially trainable (such as the part described in reference Figure 4), and the remaining models are frozen.

In some embodiments, in order to support the above-mentioned training tasks, corresponding sample sets can be created. Exemplarily, a group of style words indicating different styles and a group of subject words indicating different contents can be determined. Through the combination of style words and subject words, multiple prompt information can be obtained, for example, multiple prompt words. Each prompt information indicates the style and content to be generated. For example, the same prompt information can generate multiple sample images. In some embodiments, any pair of sample images among the multiple sample images generated by the same prompt information can be used for the training task of style feature extraction. For example, sample image 511 and sample image 520 in Figure 5A are generated by the same prompt information. Compared with image pairs using the same style words but different subject words, image pairs generated using the same prompt information can obtain better stylization effects.

In some embodiments, sample image pairs with the same subject word and different style words can be used for the training task of semantic feature extraction. For example, sample image 521 and sample image 530 in FIG5B can be generated based on the same subject word and different style words.

Example procedures, devices and equipment

FIG6 shows a flowchart of a process 600 of image processing according to some embodiments of the present disclosure. The process 600 may be implemented at the electronic device 110.

In block 610 , the electronic device 110 acquires a first reference image feature of a first reference image.

In block 610 , the electronic device 110 determines style-related features of the first reference image based on the first reference image features, a first predetermined cue for style feature extraction, and a first query representation.

At block 610 , the electronic device 110 generates a first target image based on the style-related features of the first reference image and a first input text indicating the content of the target image. The first target image matches the image style of the first reference image and the content of the target image.

In some embodiments, generating a first target image includes: obtaining a first text feature of a first input text; generating a first target image feature using a first model based on style-related features and the first text feature, wherein the first model includes multiple processing layers, the multiple processing layers are respectively used to process features of corresponding sizes, and the style-related features are provided to a first number of processing layers having a size greater than a first threshold size; and generating a first target image based on the first target image feature.

In some embodiments, generating a first target image feature using a first model includes: for a given processing layer in the first number of processing layers: converting the input image features of the given processing layer into query features; generating key features and value features by converting first text features and style-related features; and generating output image features of the given processing layer based on the query features, key features, and value features.

In some embodiments, input image features are converted into query features using a first conversion unit, and generating key features and value features includes: converting the first text feature into a first intermediate text feature using a second conversion unit; converting the first text feature into a second intermediate text feature using a third conversion unit; converting the style-related feature into a first intermediate style feature using a fourth conversion unit; converting the style-related feature into a second intermediate style feature using a fifth conversion unit; combining the first intermediate text feature and the first intermediate style feature into a key feature; and combining the second intermediate text feature and the second intermediate style feature into a value feature.

In some embodiments, the first transformation unit, the second transformation unit, and the third transformation unit are obtained through a first training pattern, and the fourth transformation unit and the fifth transformation unit are obtained through a second training pattern different from the first training pattern.

In some embodiments, process 600 also includes: obtaining second reference image features of the second reference image; determining semantically relevant features of the second reference image based on the second reference image features, a second predetermined prompt for semantic feature extraction, and a second query representation; and generating a second target image based on the semantically relevant features and a second input text indicating the style of the target image, the second target image matching the image content and the target image style of the second reference image.

In some embodiments, generating a second target image includes: obtaining second text features of a second input text; generating second target image features according to a first model based on semantically related features and the second text features, wherein the first model includes multiple processing layers, the multiple processing layers are respectively used to process features of corresponding sizes, and the semantically related features are provided to a second number of processing layers whose sizes are less than a second threshold size; and generating a second target image based on the second target image features.

In some embodiments, style-related features are determined using a feature transformation model, the first query representation is obtained by training the feature transformation model, and the training of the feature transformation model includes: obtaining a first sample image and a second sample image having the same image style; determining style-related features of the first sample image using the feature transformation model based on first sample image features of the first sample image, a first predetermined prompt, and a first query representation; and updating the feature transformation model and the first query representation by denoising and denoising the second sample image, taking the style-related features of the first sample image and text indicating the image content of the second sample image as conditions.

In some embodiments, the first sample image and the second sample image are generated based on the same prompt information, where the prompt information indicates the style and content of the image to be generated.

In some embodiments, the training of the feature transformation model also includes: determining style-related features of the third sample image using the feature transformation model based on third sample image features of the third sample image, a first predetermined prompt and a first query representation; obtaining semantic-related features of the third sample image; and updating the feature transformation model and the first query representation by reconstructing the third sample image with the style-related features and the semantic-related features as conditions.

7 shows a schematic structural block diagram of an apparatus 700 for image processing according to some embodiments of the present disclosure. The apparatus 700 may be implemented as or included in the electronic device 110. Each module/component in the apparatus 700 may be implemented by hardware, software, firmware or any combination thereof.

As shown in the figure, the device 700 includes a first image feature extraction module 710, which is configured to obtain a first reference image feature of a first reference image. The device 700 also includes a style-related feature module 720, which is configured to determine the style-related features of the first reference image based on the first reference image feature, a first predetermined prompt for style feature extraction, and a first query representation. The device 700 also includes a first target image generation module 730, which is configured to generate a first target image based on the style-related features of the first reference image and a first input text indicating the content of the target image, and the first target image matches the image style and the target image content of the first reference image.

In some embodiments, the first target image generation module 730 is further configured to: obtain a first text feature of a first input text; generate a first target image feature using a first model based on the style-related feature and the first text feature, wherein the first model includes multiple processing layers, the multiple processing layers are respectively used to process features of corresponding sizes, and the style-related features are provided to a first number of processing layers whose sizes are greater than a first threshold size; and generate a first target image based on the first target image feature.

In some embodiments, the first target image generation module 730 is further configured to: for a given processing layer in the first number of processing layers: convert the input image features of the given processing layer into query features; generate key features and value features by converting the first text features and style-related features; and generate output image features of the given processing layer based on the query features, key features, and value features.

In some embodiments, the input image feature is converted into a query feature using a first conversion unit, and the first target image generation module 730 is further configured to: convert the first text feature into a first intermediate text feature using a second conversion unit; convert the first text feature into a second intermediate text feature using a third conversion unit; convert the style-related feature into a first intermediate style feature using a fourth conversion unit; convert the style-related feature into a second intermediate style feature using a fifth conversion unit; combine the first intermediate text feature and the first intermediate style feature into a key feature; and combine the second intermediate text feature and the second intermediate style feature into a value feature.

In some embodiments, the first transformation unit, the second transformation unit, and the third transformation unit are obtained through a first training pattern, and the fourth transformation unit and the fifth transformation unit are obtained through a second training pattern different from the first training pattern.

In some embodiments, the device 700 also includes: a second image feature extraction module, configured to obtain second reference image features of the second reference image; a semantic related feature module, configured to determine the semantic related features of the second reference image based on the second reference image features, a second predetermined prompt for semantic feature extraction and a second query representation; and a second target image generation module, configured to generate a second target image based on the semantic related features and a second input text indicating the style of the target image, the second target image matching the image content and the target image style of the second reference image.

In some embodiments, the second target image generation module is further configured to: obtain second text features of a second input text; generate second target image features according to a first model based on semantically related features and the second text features, wherein the first model includes multiple processing layers, the multiple processing layers are respectively used to process features of corresponding sizes, and the semantically related features are provided to a second number of processing layers whose sizes are less than a second threshold size; and generate a second target image based on the second target image features.

In some embodiments, style-related features are determined using a feature transformation model, the first query representation is obtained by training the feature transformation model, and the training of the feature transformation model includes: obtaining a first sample image and a second sample image having the same image style; determining style-related features of the first sample image using the feature transformation model based on first sample image features of the first sample image, a first predetermined prompt, and a first query representation; and updating the feature transformation model and the first query representation by denoising and denoising the second sample image, taking the style-related features of the first sample image and text indicating the image content of the second sample image as conditions.

In some embodiments, the first sample image and the second sample image are generated based on the same prompt information, where the prompt information indicates the style and content of the image to be generated.

In some embodiments, the training of the feature transformation model also includes: determining style-related features of the third sample image using the feature transformation model based on third sample image features of the third sample image, a first predetermined prompt and a first query representation; obtaining semantic-related features of the third sample image; and updating the feature transformation model and the first query representation by reconstructing the third sample image with the style-related features and the semantic-related features as conditions.

FIG8 shows a block diagram of an electronic device 800 in which one or more embodiments of the present disclosure may be implemented. It should be understood that the electronic device 800 shown in FIG8 is merely exemplary and should not constitute any limitation on the functionality and scope of the embodiments described herein. The electronic device 800 shown in FIG8 may be used to implement the electronic device 110 of FIG1 .

As shown in FIG8 , the electronic device 800 is in the form of a general electronic device. The components of the electronic device 800 may include, but are not limited to, one or more processors or processing units 810, a memory 820, a storage device 830, one or more communication units 840, one or more input devices 850, and one or more output devices 860. The processing unit 810 may be an actual or virtual processor and is capable of performing various processes according to a program stored in the memory 820. In a multi-processor system, multiple processing units execute computer executable instructions in parallel to improve the parallel processing capability of the electronic device 800.

The electronic device 800 typically includes a plurality of computer storage media. Such media can be any accessible media that can be obtained by the electronic device 800, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 820 can be a volatile memory (e.g., a register, a cache, a random access memory (RAM)), a non-volatile memory (e.g., a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The storage device 830 can be a removable or non-removable medium, and can include a machine-readable medium, such as a flash drive, a disk, or any other medium, which can be used to store information and/or data and can be accessed within the electronic device 800.

The electronic device 800 may further include additional removable/non-removable, volatile/non-volatile storage media. Although not shown in FIG. 8 , a disk drive for reading or writing from a removable, non-volatile disk (e.g., a “floppy disk”) and an optical drive for reading or writing from a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to a bus (not shown) by one or more data media interfaces. The memory 820 may include a computer program product 825 having one or more program modules configured to perform various methods or actions of various embodiments of the present disclosure.

The communication unit 840 implements communication with other electronic devices through a communication medium. Additionally, the functions of the components of the electronic device 800 can be implemented with a single computing cluster or multiple computing machines that can communicate through a communication connection. Therefore, the electronic device 800 can operate in a networked environment using a logical connection with one or more other servers, a network personal computer (PC), or another network node.

The input device 850 may be one or more input devices, such as a mouse, a keyboard, a tracking ball, etc. The output device 860 may be one or more output devices, such as a display, a speaker, a printer, etc. The electronic device 800 may also communicate with one or more external devices (not shown) through the communication unit 840 as needed, such as a storage device, a display device, etc., communicate with one or more devices that allow a user to interact with the electronic device 800, or communicate with any device that allows the electronic device 800 to communicate with one or more other electronic devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input/output (I/O) interface (not shown).

According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored, wherein the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above.

Various aspects of the present disclosure are described herein with reference to the flowcharts and/or block diagrams of the methods, devices, equipment, and computer program products implemented according to the present disclosure. It should be understood that each box in the flowchart and/or block diagram and the combination of each box in the flowchart and/or block diagram can be implemented by computer-readable program instructions.

These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions/actions specified in one or more boxes in the flowchart and/or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and/or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions/actions specified in one or more boxes in the flowchart and/or block diagram.

Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable data processing apparatus, or other device implement the functions/actions specified in one or more boxes in the flowchart and/or block diagram.

The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple implementations of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of a module, program segment or instruction includes one or more executable instructions for realizing the logical function of the specification. In some implementations as replacements, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and/or flow chart, and the combination of the square boxes in the block diagram and/or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.

The above descriptions of various implementations of the present disclosure are exemplary, non-exhaustive, and not limited to the disclosed implementations. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described implementations. The selection of terms used herein is intended to best explain the principles of the implementations, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the various implementations disclosed herein.

Claims (14)

An image processing method, comprising: Acquire a first reference image feature of a first reference image; determining style-related features of the first reference image based on the first reference image features, a first predetermined cue for style feature extraction, and a first query representation; and Based on the style-related features of the first reference image and first input text indicating content of a target image, a first target image is generated, the first target image matching the image style of the first reference image and the content of the target image. The method of claim 1, wherein generating the first target image comprises: Acquire a first text feature of the first input text; Based on the style-related feature and the first text feature, generating a first target image feature using a first model, wherein the first model includes a plurality of processing layers, the plurality of processing layers are respectively used to process features of corresponding sizes, and the style-related feature is provided to a first number of processing layers whose sizes are greater than a first threshold size; and Based on the first target image feature, the first target image is generated. The method according to claim 2, wherein generating the first target image feature using a first model comprises: For a given processing layer in the first number of processing layers: Converting input image features of the given processing layer into query features; Generate a key feature and a value feature by converting the first text feature and the style-related feature; An output image feature of the given processing layer is generated based on the query feature, the key feature, and the value feature. The method according to claim 3, wherein the input image feature is converted into the query feature using a first conversion unit, and generating the key feature and the value feature comprises: Using a second conversion unit to convert the first text feature into a first intermediate text feature; Using a third conversion unit to convert the first text feature into a second intermediate text feature; Using a fourth conversion unit, converting the style-related feature into a first intermediate style feature; Using a fifth conversion unit, the style-related feature is converted into a second intermediate style feature; combining the first intermediate text feature and the first intermediate style feature into the key feature; and The second intermediate text feature and the second intermediate style feature are combined into the value feature. The method according to claim 4, wherein the first conversion unit, the second conversion unit and the third conversion unit are obtained through a first training mode, and the fourth conversion unit and the fifth conversion unit are obtained through a second training mode different from the first training mode. The method according to claim 1, further comprising: Acquire a second reference image feature of the second reference image; determining semantically relevant features of the second reference image based on the second reference image features, a second predetermined cue for semantic feature extraction, and a second query representation; and Based on the semantically relevant features and a second input text indicating a style of a target image, a second target image is generated, the second target image matching the image content of the second reference image and the style of the target image. The method of claim 6, wherein generating the second target image comprises: Acquire a second text feature of the second input text; generating a second target image feature according to a first model based on the semantically related feature and the second text feature, wherein the first model includes a plurality of processing layers, the plurality of processing layers are respectively used to process features of corresponding sizes, and the semantically related features are provided to a second number of processing layers whose sizes are smaller than a second threshold size; and Based on the second target image feature, the second target image is generated. The method according to claim 1, wherein the style-related features are determined using a feature transformation model, the first query representation is obtained by training the feature transformation model, and the training of the feature transformation model comprises: Acquire a first sample image and a second sample image having the same image style; determining, using the feature transformation model, style-related features of the first sample image based on a first sample image feature of the first sample image, the first predetermined prompt, and the first query representation; The feature transformation model and the first query representation are updated by adding noise and denoising the second sample image, taking the style-related features of the first sample image and the text indicating the image content of the second sample image as conditions. The method according to claim 8, wherein the first sample image and the second sample image are generated based on the same prompt information, and the prompt information indicates the style and content of the image to be generated. The method according to claim 8, wherein the training of the feature transformation model further comprises: determining, using the feature transformation model, style-related features of the third sample image based on third sample image features of the third sample image, the first predetermined prompt, and the first query representation; Acquiring semantically relevant features of the third sample image; and The feature transformation model and the first query representation are updated by reconstructing the third sample image based on the style-related features and the semantics-related features. A device for image processing, comprising: A first image feature extraction module is configured to obtain a first reference image feature of a first reference image; a style-related feature module configured to determine style-related features of the first reference image based on the first reference image features, a first predetermined hint for style feature extraction, and a first query representation; and The first target image generation module is configured to generate a first target image based on the style-related features of the first reference image and a first input text indicating the content of the target image, wherein the first target image matches the image style of the first reference image and the content of the target image. An electronic device, comprising: at least one processing unit; and At least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to any one of claims 1 to 10 when executed by the at least one processing unit. A computer-readable storage medium having a computer program stored thereon, wherein the computer program can be executed by a processor to implement the method according to any one of claims 1 to 10. A computer program product, which is tangibly stored in a computer storage medium and includes computer executable instructions, which when executed by a device cause the device to perform the method according to any one of claims 1-10.