WO2025112948A1 - Image generation method, automatic question answering method, and parameter generation model training method - Google Patents

Abstract

Embodiments of the present disclosure provide an image generation method, an automatic question answering method, and a parameter generation model training method. The image generation method comprises: acquiring an image description text; inputting the image description text and generation prompt information into a parameter generation model to obtain image generation parameters corresponding to the image description text, wherein the image generation parameters are used for describing visual features of an image, and the parameter generation model is obtained by performing training on the basis of a plurality of sample image and text pairs and sample parameter information carried by the plurality of sample image and text pairs; and inputting the image generation parameters and the image description text into an image generation model to obtain a target image corresponding to the image description text. By using a parameter generation model to perform semantic decomposition on visual elements of an image to obtain image generation parameters, and further completing accurate image generation on the basis of the image generation parameters, a target image can clearly express an image description text and the image generation parameters, thereby improving the interpretability and controllability of image generation.

Description

Image generation, automatic question answering, and parameter generation model training methods

This disclosure claims priority to a Chinese patent application filed with the Chinese Patent Office on November 28, 2023, with application number 2023116226401 and application name “Image Generation, Automatic Question Answering, and Parameter Generation Model Training Method,” the entire contents of which are incorporated by reference in this disclosure.

Technical Field

The embodiments of the present disclosure relate to the field of computer technology, and in particular to image generation, automatic question answering, and parameter generation model training methods.

Background Art

With the development of computer technology, text-generated image technology has gradually become the core technology in the field of artificial intelligence generation (AIGC, AI Generated Content). Text-generated image technology can generate images through text descriptions, and can be transformed and adjusted according to user requirements and input content, making it easier for users to create works of art with unique styles, and has been widely used in the field of digital art.

Currently, the common text-generated graph architecture is the Hidden Space Diffusion Model (HSDM). However, since the Hidden Space Diffusion Model operates in the hidden space, the generation process is difficult to understand and the generated image is quite different from the text originally input by the user.

Summary of the invention

In view of this, an embodiment of the present disclosure provides an image generation method. One or more embodiments of the present disclosure also relate to an automatic question-answering method, a parameter generation model training method, an image generation device, an automatic question-answering device, a parameter generation model training device, a computing device, a computer-readable storage medium, and a computer program to solve the technical defects existing in the prior art.

According to a first aspect of an embodiment of the present disclosure, there is provided an image generating method, comprising:

Get the image description text;

Input the image description text and the generation prompt information into the parameter generation model to obtain the image generation parameters corresponding to the image description text, wherein the image generation parameters are used to describe the visual features of the image, and the parameter generation model is trained based on multiple sample image-text pairs and sample parameter information carried by the multiple sample image-text pairs;

The image generation parameters and the image description text are input into the image generation model to obtain the target image corresponding to the image description text.

According to a second aspect of an embodiment of the present disclosure, a parameter generation model training method is provided, which is applied to a cloud-side device, including:

Acquire a sample set, wherein the sample set includes a plurality of sample image-text pairs, the sample image-text pairs include a sample image and a sample description text, and the sample image-text pairs carry sample parameter information;

Inputting a plurality of sample image-text pairs and prediction prompt information into a pre-trained language model to obtain image prediction parameters corresponding to the plurality of sample image-text pairs;

According to the image prediction parameters and sample parameter information, the model parameters of the pre-trained language model are adjusted to obtain a parameter generation model that has completed training.

According to a third aspect of an embodiment of the present disclosure, there is provided an automatic question-answering method, comprising:

receiving an image question answering request, wherein the image question answering request carries an image description text;

Input the image description text and the generation prompt information into the parameter generation model to obtain the image generation parameters corresponding to the image description text, wherein the image generation parameters are used to describe the visual features of the image, and the parameter generation model is trained based on multiple sample image-text pairs and sample parameter information carried by the multiple sample image-text pairs;

The image generation parameters and image description text are input into the image generation model to obtain the reply image corresponding to the image question answering request.

According to a fourth aspect of an embodiment of the present disclosure, there is provided an image generating device, including:

A first acquisition module is configured to acquire image description text;

A first input module is configured to input the image description text and the generation prompt information into a parameter generation model to obtain image generation parameters corresponding to the image description text, wherein the image generation parameters are used to describe the visual features of the image, and the parameter generation model is trained based on a plurality of sample image-text pairs and sample parameter information carried by the plurality of sample image-text pairs;

The second input module is configured to input the image generation parameters and the image description text into the image generation model to obtain the target image corresponding to the image description text.

According to a fifth aspect of an embodiment of the present disclosure, a parameter generation model training device is provided, which is applied to a cloud-side device, including:

A second acquisition module is configured to acquire a sample set, wherein the sample set includes a plurality of sample image-text pairs, the sample image-text pairs include a sample image and a sample description text, and the sample image-text pairs carry sample parameter information;

A third input module is configured to input a plurality of sample image-text pairs and prediction prompt information into the pre-trained language model to obtain image prediction parameters corresponding to the plurality of sample image-text pairs respectively;

The adjustment module is configured to adjust the model parameters of the pre-trained language model according to the image prediction parameters and the sample parameter information to obtain a parameter generation model that has completed the training.

According to a sixth aspect of an embodiment of the present disclosure, there is provided an automatic question-answering device, comprising:

A first receiving module is configured to receive an image question and answer request, wherein the image question and answer request carries an image description text;

a fourth input module, configured to input the image description text and the generation prompt information into a parameter generation model, and obtain image generation parameters corresponding to the image description text, wherein the image generation parameters are used to describe the visual features of the image, and the parameter generation model is trained based on a plurality of sample image-text pairs and sample parameter information carried by the plurality of sample image-text pairs;

The fifth input module is configured to input the image generation parameters and the image description text into the image generation model to obtain a reply image corresponding to the image question and answer request.

According to a seventh aspect of an embodiment of the present disclosure, there is provided a computing device, including:

Memory and processor;

The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the method provided in the first aspect, the second aspect, or the third aspect are implemented.

According to an eighth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, which stores computer-executable instructions. When the instructions are executed by a processor, the steps of the method provided in the first aspect, the second aspect, or the third aspect are implemented.

According to a ninth aspect of an embodiment of the present disclosure, a computer program is provided, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the method provided in the first aspect, the second aspect, or the third aspect above.

An image generation method provided by an embodiment of the present disclosure obtains an image description text; inputs the image description text and generation prompt information into a parameter generation model to obtain image generation parameters corresponding to the image description text, wherein the image generation parameters are used to describe the visual features of the image, and the parameter generation model is trained based on multiple sample image-text pairs and sample parameter information carried by the multiple sample image-text pairs; the image generation parameters and the image description text are input into the image generation model to obtain a target image corresponding to the image description text. The image generation parameters are obtained by semantically decomposing the visual elements of the image using the parameter generation model, and the precise image generation is further completed based on the image generation parameters, so that the target image can clearly express the image description text and the image generation parameters, thereby improving the interpretability and controllability of the image generation.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is an architecture diagram of an image generation system provided by an embodiment of the present disclosure;

FIG2 is an architecture diagram of another image generation system provided by an embodiment of the present disclosure;

FIG3 is a flow chart of an image generating method provided by an embodiment of the present disclosure;

FIG4 is a process flow chart of an image generation method provided by an embodiment of the present disclosure;

FIG5 is a flow chart of a parameter generation model training method provided by an embodiment of the present disclosure;

FIG6 is a flow chart of another parameter generation model training method provided by an embodiment of the present disclosure;

FIG7 is a flow chart of an automatic question-answering method provided by an embodiment of the present disclosure;

FIG8 is a process flow chart of another image generation method provided by an embodiment of the present disclosure;

FIG9 is a schematic diagram of a processing process of a parameter generation model provided by an embodiment of the present disclosure;

FIG10 is a schematic diagram of an image generation interface provided by an embodiment of the present disclosure;

FIG11 is a schematic diagram of the structure of an image generating device provided by an embodiment of the present disclosure;

FIG12 is a schematic diagram of the structure of a parameter generation model training device provided by an embodiment of the present disclosure;

FIG13 is a schematic diagram of the structure of an automatic question-answering device provided by an embodiment of the present disclosure;

FIG. 14 is a structural block diagram of a computing device provided by an embodiment of the present disclosure.

DETAILED DESCRIPTION

Many specific details are described in the following description to facilitate a full understanding of the present disclosure. However, the present disclosure can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of the present disclosure, so the present disclosure is not limited by the specific implementation disclosed below.

The terms used in one or more embodiments of the present disclosure are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of the present disclosure. The singular forms of "a", "said" and "the" used in one or more embodiments of the present disclosure and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and/or" used in one or more embodiments of the present disclosure refers to and includes any or all possible combinations of one or more associated listed items.

It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of the present disclosure, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of the present disclosure, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of the present disclosure are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions, and corresponding operation entrances are provided for users to choose to authorize or refuse.

In one or more embodiments of the present disclosure, a large model refers to a deep learning model with large-scale model parameters, which usually contains hundreds of millions, tens of billions, hundreds of billions, trillions, or even more than ten trillion model parameters. A large model can also be called a foundation model/foundation model. The large model is pre-trained with large-scale unlabeled corpus to produce a pre-trained model with more than 100 million parameters. This model can adapt to a wide range of downstream tasks, and the model has good generalization ability, such as a large-scale language model (LLM, Large Language Model), a multi-modal pre-training model, etc.

When the big model is used in practice, only a small number of samples are needed to fine-tune the pre-trained model and it can be applied to different tasks. The big model can be widely used in natural language processing (NLP), computer vision and other fields. Specifically, it can be applied to computer vision tasks such as visual question answering (VQA), image description (IC, Image Caption), image generation, as well as natural language processing tasks such as text-based sentiment classification, text summary generation, and machine translation. The main application scenarios of the big model include digital assistants, intelligent robots, search, online education, office software, e-commerce, intelligent design, etc.

First, the terms involved in one or more embodiments of the present disclosure are explained.

Latent space diffusion model: Latent space diffusion model refers to an artificial intelligence model widely used in image editing and image generation. A technique for encoding images into a latent space for denoising and de-noising.

Attention mechanism: Attention mechanism is a machine learning method that can assign different weights according to the importance of each part of the input data.

CLIP: CLIP (Contrastive Language-Image Pre-training) is a model for measuring the correlation between text and images, and is widely used in text and image representation extraction.

Large-scale language models: Large-scale language models are trained on massive amounts of text data to predict and generate various language expressions, such as text, sentences, paragraphs, etc.

LoRA: A language model fine-tuning method based on low-rank adaptation, which can achieve good results by training only a small number of parameters when using a large model to adapt downstream tasks.

The text-generated graph technology outputs images based on the image description text input by the user, which has great value in many scenarios such as advertising recommendation, interactive entertainment, art design, etc. The common text-generated graph architecture is the latent space diffusion model. Since the latent space diffusion model operates in the hidden space, it is difficult to understand and explain its internal working mechanism and decision-making process, which makes the generation process difficult to understand. In addition, this type of method relies on the feature extraction module (CLIP encoder) to understand the text, and cannot control the semantics of the visual elements in the generation stage, which leads to a large difference between the final generated result and the image description text originally input by the user.

In order to improve the accuracy of the image generation model, the disclosed embodiment proposes an image generation method combined with a large-scale language model, which uses the powerful semantic understanding and associative ability of the large-scale language model to complete the semantic decomposition and reorganization of the visual elements of the image, and inputs the parameterized visual elements into the image generation model to complete accurate image generation, so as to meet the stronger interpretability and controllability of the image generation process. In addition, the large-scale language model can align complex and long texts, laying the technical foundation for more user interaction methods. In addition, by combining the large-scale language model and the image generation model, convenient and refined image editing, similar structure image generation and other extended applications can be realized.

Specifically, an embodiment of the present disclosure proposes an image generation scheme to obtain image description text; input the image description text and generation prompt information into a parameter generation model to obtain image generation parameters corresponding to the image description text, wherein the image generation parameters are used to describe the visual features of the image, and the parameter generation model is trained based on multiple sample image-text pairs and sample parameter information carried by the multiple sample image-text pairs; input the image generation parameters and the image description text into the image generation model to obtain a target image corresponding to the image description text.

In the present disclosure, an image generation method is provided. The present disclosure also relates to an automatic question-answering method, a parameter generation model training method, an image generation device, an automatic question-answering device, a parameter generation model training device, a computing device, and a computer-readable storage medium, which are described in detail one by one in the following embodiments.

Referring to FIG. 1 , FIG. 1 shows an architecture diagram of an image generation system provided by an embodiment of the present disclosure. The image generation system may include a client 100 and a server 200;

The client 100 is used to send image description text to the server 200;

The server 200 is used to input the image description text and the generation prompt information into the parameter generation model to obtain the image generation parameters corresponding to the image description text, wherein the image generation parameters are used to describe the visual features of the image. Based on a plurality of sample image-text pairs and sample parameter information carried by the plurality of sample image-text pairs, the training is obtained; the image generation parameters and the image description text are input into the image generation model to obtain a target image corresponding to the image description text; and the target image is sent to the client 100;

The client 100 is also used to receive the target image sent by the server 200.

By applying the solution of the embodiment of the present disclosure, the visual elements of the image are semantically decomposed by utilizing a parameter generation model to obtain image generation parameters, and accurate image generation is further completed based on the image generation parameters, so that the target image can clearly express the image description text and image generation parameters, thereby improving the interpretability and controllability of image generation.

Referring to FIG. 2 , FIG. 2 shows an architecture diagram of another image generation system provided by an embodiment of the present disclosure. The image generation system may include multiple clients 100 and a server 200, wherein the client 100 may include a terminal-side device, and the server 200 may include a cloud-side device. A communication connection may be established between multiple clients 100 through the server 200. In the image generation scenario, the server 200 is used to provide image generation services between multiple clients 100. Multiple clients 100 may serve as a sender or a receiver, respectively, and realize communication through the server 200.

The user can interact with the server 200 through the client 100 to receive data sent by other clients 100, or send data to other clients 100, etc. In the image generation scenario, the user can publish a data stream to the server 200 through the client 100, and the server 200 generates a target image according to the data stream and pushes the target image to other clients that have established communication.

The client 100 and the server 200 are connected via a network. The network provides a medium for a communication link between the client 100 and the server 200. The network may include various connection types, such as wired or wireless communication links or optical fiber cables, etc. The data transmitted by the client 100 may need to be encoded, transcoded, compressed, etc. before being released to the server 200.

The client 100 may be a browser, an APP (Application), or a web application such as an H5 (HyperText Markup Language5, the fifth edition of Hypertext Markup Language) application, or a light application (also known as a mini-program, a lightweight application) or a cloud application, etc. The client 100 may be based on the software development kit (SDK, Software Development Kit) of the corresponding service provided by the server 200, such as based on the real-time communication (RTC, Real Time Communication) SDK development and acquisition, etc. The client 100 may be deployed in an electronic device, and may need to rely on the device to run or some APPs in the device to run, etc. For example, the electronic device may have a display screen and support information browsing, etc., such as a personal mobile terminal such as a mobile phone, a tablet computer, a personal computer, etc. Various other types of applications may also be configured in the electronic device, such as human-computer dialogue applications, model training applications, text processing applications, web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

The server 200 may include servers that provide various services, such as servers that provide communication services to multiple clients, servers that provide background training to support models used on clients, and servers that process data sent by clients. It should be noted that the server 200 can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. The server can also be a server of a distributed system, or a server combined with a blockchain. The server can also be a cloud service, cloud database, cloud computing, cloud function, cloud storage, Cloud servers that provide basic cloud computing services such as network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), big data and artificial intelligence platforms, or intelligent cloud computing servers or intelligent cloud hosts with artificial intelligence technology.

It is worth noting that the image generation method provided in the embodiment of the present disclosure is generally executed by the server, but in other embodiments of the present disclosure, the client may also have similar functions as the server, thereby executing the image generation method provided in the embodiment of the present disclosure. In other embodiments, the image generation method provided in the embodiment of the present disclosure may also be jointly executed by the client and the server.

Referring to FIG. 3 , FIG. 3 shows a flow chart of an image generation method provided by an embodiment of the present disclosure, which specifically includes the following steps:

Step 302: Obtain image description text.

In one or more embodiments of the present disclosure, when image generation begins, image description text may be acquired, and a target image that meets actual needs of the user may be generated based on the image description text.

Specifically, the image description text represents the user's image generation needs. The image description text can be description text in different languages, such as English description text, Chinese description text, etc. Since the image generation process introduces a large-scale language model, the image description text can be a short text or a long text. For example, the image description text is "two black cats and a white dog on an orange sofa".

In practical applications, there are many ways to obtain image description text, which can be selected according to actual conditions, and the embodiments of the present disclosure do not limit this. In one possible implementation of the present disclosure, the image description text sent by the user through the client can be received. In another possible implementation of the present disclosure, the image description text can be read from other data acquisition devices or databases.

Step 304: Input the image description text and generation prompt information into a parameter generation model to obtain image generation parameters corresponding to the image description text, wherein the image generation parameters are used to describe the visual features of the image, and the parameter generation model is trained based on multiple sample image-text pairs and sample parameter information carried by the multiple sample image-text pairs.

In one or more embodiments of the present disclosure, after obtaining the image description text, the image description text and generation prompt information may be further input into a parameter generation model to obtain image generation parameters corresponding to the image description text.

Specifically, the parameter generation model is obtained by training a pre-trained language model based on a sample set. The pre-trained language model can be a large-scale language model, such as a multimodal large model, or a language processing model trained using the first training set. Image generation parameters can be understood as parameterized image generation conditions. Image generation parameters include but are not limited to size parameters, position parameters, shape parameters, and the like. Generation prompt information can be understood as a parameter generation paradigm, also known as a generation condition paradigm. Generation prompt information is used to guide the parameter generation model to generate image generation parameters. Generation prompt information includes but is not limited to image generation condition elements of shape, color, size, position, blur, and key points. For example, generation prompt information can be "what are the elements in the image."

It should be noted that in the embodiment of the present disclosure, the generation conditions are unified into parameters that can be described in natural language rather than image conditions, so as to facilitate semantic understanding of large-scale language models.

Step 306: input the image generation parameters and the image description text into the image generation model to obtain the target image corresponding to the image description text.

In one or more embodiments of the present disclosure, an image description text is obtained, the image description text and generation prompt information are input into a parameter generation model, and after the image generation parameters corresponding to the image description text are obtained, the image generation parameters and the image description text can be further input into the image generation model to obtain a target image corresponding to the image description text.

Specifically, the image generation model can be a latent space diffusion model, or an image generation model trained using the second training set. The image generation model is used to generate a final target image based on image generation parameters and image description text. The target image can be a black and white image or a color image (RGB Image).

By applying the solution of the embodiment of the present disclosure, the visual elements of the image are semantically decomposed by utilizing a parameter generation model to obtain image generation parameters, and accurate image generation is further completed based on the image generation parameters, so that the target image can clearly express the image description text and image generation parameters, thereby improving the interpretability and controllability of image generation.

In an optional embodiment of the present disclosure, image generation parameters and image description text are input into an image generation model, and a side branch structure can be used in the image generation model to process the image generation parameters, that is, on the basis of the encoding and decoding unit originally possessed by the image generation model, a parameter encoding unit is additionally added as a side branch structure, that is, the image generation model includes a parameter encoding unit and an encoding and decoding unit; the above-mentioned inputting the image generation parameters and the image description text into the image generation model to obtain the target image corresponding to the image description text may include the following steps:

Inputting image generation parameters and image description text into the image generation model, and encoding the image generation parameters through the parameter encoding unit to obtain parameter encoding features;

The encoding and decoding unit generates a target image corresponding to the image description text according to the parameter encoding features and the image description text.

Refer to Figure 4, which shows a processing flow chart of an image generation method provided by an embodiment of the present disclosure. As shown in Figure 4, after the parameter generation model generates the image generation parameters, the image description text and the image generation parameters can be input into the image generation model together. In the image generation model, the image generation parameters are first encoded by the parameter encoding unit to obtain parameter encoding features. Secondly, the encoding and decoding unit generates a target image corresponding to the image description text according to the parameter encoding features and the image description text.

It should be noted that the encoding and decoding unit includes an encoding unit and a decoding unit. After obtaining the parameter encoding features, the parameter encoding features can be input into the encoding unit in the latent space for parameterized conditional control, and the residual is sent to the decoding unit to realize the final parameter fusion generation.

In practical applications, the parameter encoding unit can directly encode the image generation parameters to obtain parameter encoding features. Furthermore, since the image generation parameters include parameters of different dimensions, the parameter encoding unit can also encode the image generation parameters into parameter encoding features of different latitudes in view of the diversity of the image generation parameter dimensions.

Applying the solution of the embodiment of the present disclosure, the image generation parameters and the image description text are input into the image generation model, and the parameter encoding unit encodes the image generation parameters to obtain parameter encoding features; the encoding and decoding unit generates a target image corresponding to the image description text according to the parameter encoding features and the image description text. Parameters are controlled to ensure the accuracy of the target image.

In an optional embodiment of the present disclosure, the parameter encoding unit includes a one-dimensional parameter encoding unit, a two-dimensional parameter encoding unit and a feature aggregation unit; the parameter encoding unit generates parameter encoding for the image to obtain the parameter encoding feature, which may include the following steps:

Through a one-dimensional parameter encoding unit, encoding the one-dimensional parameter in the image generation parameter to obtain a one-dimensional encoding feature;

The two-dimensional parameter encoding unit encodes the two-dimensional parameters in the image generation parameters to obtain a two-dimensional encoding feature;

The one-dimensional coding features and the two-dimensional coding features are aggregated by the feature aggregation unit to obtain parameter coding features.

Specifically, the image generation parameters may include parameters of different dimensions, such as one-dimensional parameters and two-dimensional parameters. Among them, the one-dimensional parameters include but are not limited to color parameters and blur parameters. The two-dimensional parameters include but are not limited to position parameters and shape parameters. Therefore, in order to better encode the image generation parameters, the embodiment of the present disclosure divides the parameter encoding unit to obtain a one-dimensional parameter encoding unit for encoding the one-dimensional parameters and a two-dimensional parameter encoding unit for encoding the two-dimensional parameters.

Furthermore, since the image generation parameters characterize the overall generation conditions of the target image, a feature aggregation unit may be additionally added to the parameter encoding unit, so that the one-dimensional encoding features and the two-dimensional encoding features are aggregated through the feature aggregation unit to obtain parameter encoding features.

By applying the scheme of the embodiment of the present disclosure, the one-dimensional parameters in the image generation parameters are encoded by a one-dimensional parameter encoding unit to obtain a one-dimensional encoding feature; the two-dimensional parameters in the image generation parameters are encoded by a two-dimensional parameter encoding unit to obtain a two-dimensional encoding feature; and the one-dimensional encoding feature and the two-dimensional encoding feature are aggregated by a feature aggregation unit to obtain a parameter encoding feature, thereby encoding parameters of different dimensions respectively, thereby ensuring the accuracy of the parameter encoding feature.

In an optional embodiment of the present disclosure, after inputting the image generation parameters and the image description text into the image generation model to obtain the target image corresponding to the image description text, further, an image update parameter for the target image may be received, and the target image may be edited based on the image update parameter, or an image with a structure similar to the target image may be generated. That is, after inputting the image generation parameters and the image description text into the image generation model to obtain the target image corresponding to the image description text, the following steps may also be included:

Obtain image update parameters for a target image;

determining an image editing region in a target image according to an image updating parameter;

Mask the image editing area to obtain a mask generation sequence;

The mask generation sequence and the target image are input into the image generation model to obtain the updated target image.

Specifically, the image update parameter is used to modify or adjust the target image to obtain an updated target image. The image update parameter may be an image generation condition independent of the image generation parameter, such as if the image generation parameter is "the number of black cats is 2", and the image update parameter is "the number of black cats is 3"; for example, if the image generation parameter is "the number of black cats is 2", the image update parameter is "the number of black cats plus one". There are many ways to obtain image update parameters for a target image, which can be selected based on actual conditions, and the embodiments of the present disclosure do not impose any limitations on this. In one possible implementation of the present disclosure, an image update parameter sent by a user through a client may be received. In another possible implementation of the present disclosure, an image update parameter may be obtained from other data. Get the image update parameters for the target image from the device or database.

It should be noted that after obtaining the image update parameters for the target image and determining that the image update parameters have changed compared to the image generation parameters, the image editing region in the target image can be determined. The image editing region refers to the region in the target image that needs to be parameter updated. When determining the image editing region in the target image based on the image update parameters, the image editing region in the attention map output by the attention mechanism can be determined, and the image editing region in the attention map can be masked to generate a mask generation sequence.

By applying the solution of the embodiment of the present disclosure, image update parameters for the target image are obtained; according to the image update parameters, the image editing area in the target image is determined; the image editing area is masked to obtain a mask generation sequence; the mask generation sequence and the target image are input into the image generation model to obtain an updated target image. By regenerating the image editing area in the attention map based on the mask generation sequence, other areas where parameters are not updated are not affected, and identity-preserving image editing and similar structure image generation are achieved.

In an optional embodiment of the present disclosure, taking the image generation model as a diffusion model as an example, the above-mentioned inputting the mask generation sequence and the target image into the image generation model to obtain the updated target image may include the following steps:

For the current diffusion step of the image generation model, the diffusion result of the current diffusion step is generated according to the diffusion result of the completed diffusion step and the mask generation sequence, until the iteration stop condition is reached, and the updated target image is obtained.

Specifically, the diffusion generation model includes multiple diffusion steps in its processing. Each diffusion step can be regarded as a transformation of the image. These transformations can help the diffusion model better simulate the image degradation process and restore the noise-free image from the noisy image. Generally speaking, the multiple diffusion steps of the diffusion model can be divided into two stages, namely the diffusion stage and the denoising stage. In the diffusion stage, the diffusion model can simulate the image degradation process by gradually adding noise to the input image; in the denoising stage, the diffusion model can find and eliminate noise from the noisy image to restore the original noise-free image.

In the disclosed embodiment, in the image generation model, the next editable image generation can be performed based on the diffusion results of the previous diffusion steps. During the editable image generation process, the diffusion result and the mask generation sequence can be multiplied to obtain the diffusion result of the current diffusion step, until the iteration stop condition is reached, thereby obtaining the updated target image.

By applying the solution of the embodiment of the present disclosure, for the current diffusion step of the image generation model, the diffusion result of the current diffusion step is generated according to the diffusion result of the completed diffusion step and the mask generation sequence, until the iteration stop condition is reached, and the updated target image is obtained, thereby realizing identity-preserving image editing and similar structure image generation.

In an optional embodiment of the present disclosure, after inputting the mask generation sequence and the target image into the image generation model to obtain the updated target image, the following steps may also be included:

The target image and the updated target image are sent to the client, so that the client displays the target image and the updated target image to the user.

It should be noted that after obtaining the updated target image, the target image and the updated target image can be sent to the client at the same time, and the user can select the target image that meets the actual requirements from the target image and the updated target image displayed on the client. images that meet actual needs.

In actual applications, there are many ways to send the target image and the updated target image to the client, and the specific selection is based on the actual situation. The embodiments of the present disclosure do not impose any restrictions on this. In one possible implementation of the present disclosure, the target image and the updated target image can be directly sent to the client so that the client displays the target image and the updated target image to the user. In another possible implementation of the present disclosure, the target image and the updated target image can be sent to the client according to the user's display demand information. Among them, the display demand information represents the user's demand for viewing the target image. The display demand information includes but is not limited to only displaying the target image and the updated target image, displaying the image description information, the target image and the updated target image. The display demand information is specifically set according to the actual needs of the user, and the embodiments of the present disclosure do not impose any restrictions on this.

By applying the solution of the embodiment of the present disclosure, the target image and the updated target image are sent to the client, so that the client displays the target image and the updated target image to the user, providing the user with a variety of choices, and the user can intuitively compare the target image and the updated target image to determine whether the updated target image meets his or her needs.

In an optional embodiment of the present disclosure, after sending the target image and the updated target image to the client, the following steps may also be included:

Receive image selection information sent by the user through the client, and adjust model parameters of the image generation model based on the image selection information.

Specifically, the image selection information is used to identify the image selected by the user. The image selection information includes but is not limited to the image number and the image position, and is selected according to actual conditions, and the embodiment of the present disclosure does not impose any limitation on this.

It should be noted that the user may send image selection information in a manner including but not limited to voice input, touch, and text input. Based on the image selection information, an image that meets the user's needs may be determined, and the image corresponding to the image selection information may be used as a real image to adjust parameters of the image generation model.

By applying the solution of the embodiment of the present disclosure, image selection information sent by the user through the client is received, and the model parameters of the image generation model are adjusted based on the image selection information, so that the image generation model has the ability to interact with the user, so that the user can adjust the image generation result in an interactive manner, thereby improving user satisfaction.

In an optional embodiment of the present disclosure, after inputting the image generation parameters and the image description text into the image generation model and obtaining the target image corresponding to the image description text, the following steps may also be included:

Receive adjustment sample data sent by the user through the client, and adjust the model parameters of the image generation model according to the adjustment sample data, wherein the adjustment sample data is constructed based on the target image.

In practical applications, the image generation parameters and the image description text are input into the image generation model. After obtaining the target image corresponding to the image description text, the target image can be sent to the client. Furthermore, the user can process the target image by himself. If he is not satisfied with the target image, he can send the adjustment sample data to train the model again. Specifically, after receiving the adjustment sample data sent by the user through the client, in a first possible implementation method, the model parameters of the image generation model can be adjusted according to the adjustment sample data; in a second possible implementation method, the model parameters of the parameter generation model can be adjusted according to the adjustment sample data; in a third possible implementation method, the adjustment sample data can be adjusted according to the adjustment sample data. Model parameters of the image generation model and the parameter generation model. The adjusted sample data can be obtained by the user adding labels to the target image (such as positive and negative sample labels), or by adjusting parameters of the target image (such as adjusting size parameters). The embodiments of the present disclosure do not impose any restrictions on the generation method of the adjusted sample data.

It should be noted that the implementation method of "adjusting the model parameters of the image generation model according to the adjusted sample data" is the same as the training method of the image generation model, and will not be repeated in the embodiment of the present disclosure.

By applying the solution of the embodiment of the present disclosure, a target image is sent to a client so that the client displays the target image to a user; adjustment sample data sent by the user through the client is received, and model parameters of the image generation model are adjusted according to the adjustment sample data, so that the image generation model has the ability to interact with the user, so that the user can adjust the image generation result in an interactive manner, thereby improving user satisfaction.

In an optional embodiment of the present disclosure, before inputting the image description text and the generation prompt information into the parameter generation model to obtain the image generation parameters corresponding to the image description text, the following steps may also be included:

Acquire a sample set, wherein the sample set includes a plurality of sample image-text pairs, the sample image-text pairs include a sample image and a sample description text, and the sample image-text pairs carry sample parameter information;

Inputting a plurality of sample image-text pairs and prediction prompt information into a pre-trained language model to obtain image prediction parameters corresponding to the plurality of sample image-text pairs;

According to the image prediction parameters and sample parameter information, the model parameters of the pre-trained language model are adjusted to obtain a parameter generation model that has completed training.

Specifically, the training method of the parameter generation model is supervised training based on prompt learning, that is, each sample image-text pair in the sample set carries real sample parameter information, and the sample parameter information is the generation target of the parameter generation model, which is used to guide the training process of the parameter generation model.

The prediction prompt information is used to prompt the prediction process of the pre-trained language model. The prediction prompt information is set according to the actual situation, and the present disclosure embodiment does not impose any restrictions on this. For example, the prediction prompt information can be "You are an intelligent bounding box generation model. I will provide you with images and image description texts. The format of each bounding box should be (object name, center point horizontal coordinate, center point vertical coordinate, bounding box width, bounding box height)". The prediction prompt information can enable the pre-trained language model to output prediction results in a fixed data format.

It should be noted that the implementation method of "inputting multiple sample image-text pairs and prediction prompt information into the pre-trained language model to obtain image prediction parameters corresponding to multiple sample image-text pairs" can refer to the implementation method of "inputting image description text and generation prompt information into the parameter generation model to obtain image generation parameters corresponding to the image description text", and this disclosure will not repeat it. When adjusting the model parameters of the pre-trained language model according to the image prediction parameters and sample parameter information, the LoRA method can be used, and rank=8 can be set to achieve model adjustment without forgetting the semantic understanding ability of the basic model.

In practical applications, there are many ways to obtain sample sets, which can be selected according to actual conditions. The embodiments of the present disclosure do not limit this. In one possible implementation of the present disclosure, a large number of sample image text pairs carrying sample parameter information input by the user can be received to form a sample set. In another possible implementation of the present disclosure, the sample set can be obtained from other data. A large number of sample image-text pairs carrying sample parameter information are read from an acquisition device or a database to form a sample set.

It is worth noting that the sample set may include at least one of an original sample subset, a generative sample subset, and a constructed sample subset. Specifically, a plurality of sample images carrying sample description texts may be obtained to form an original sample subset. The image understanding capability of a large model may be used to perform sample description texts on sample images to form a generative sample subset, and when forming a generative sample subset, a collaborative detection and segmentation model may be used to extract a variety of sample parameter information. Parameter configuration conditions may be set based on prior knowledge, such as output rules, object rules, element number rules, etc., to construct pseudo data to form a constructed sample subset, thereby using the constructed sample subset to enable the parameter generation model to have a purposeful learning capability.

Referring to FIG5 , FIG5 shows a flow chart of a parameter generation model training method provided by an embodiment of the present disclosure, wherein the original sample subset, the generative sample subset and the constructed sample subset are mixed to obtain a sample set, multiple sample image-text pairs and prediction prompt information in the sample set are input into a pre-trained language model, image prediction parameters corresponding to the multiple sample image-text pairs are obtained, model parameters of the pre-trained language model are adjusted according to the image prediction parameters and sample parameter information, and a parameter generation model that has been trained is obtained. By constructing a sample set including three sample subsets, the image generation parameters output by the parameter generation model can be made more reasonable.

By applying the solution of the embodiment of the present disclosure, the model parameters of the pre-trained language model are adjusted according to the image prediction parameters and sample parameter information to obtain a parameter generation model that has completed training. By continuously adjusting the model parameters of the pre-trained language model, the parameter generation model finally obtained can be made more accurate.

In an optional embodiment of the present disclosure, taking the generative sample subset included in the sample set as an example, the above-mentioned acquisition of the sample set may include the following steps:

Acquire multiple sample images;

For the first sample image, input the first sample image and the construction prompt information into the pre-trained language model to obtain a first sample description text of the first sample image;

Generate first sample parameter information of the first sample image according to the first sample image and the first sample description text;

A sample set is constructed according to a plurality of sample images, sample description texts of the plurality of sample images, and sample parameter information.

Specifically, the construction prompt information is used to prompt the pre-trained language model to generate a sample description text of the sample image. The construction prompt information is set according to the actual situation, and the embodiment of the present disclosure does not make any limitation on this. For example, the prediction prompt information may be "Please describe the image".

It should be noted that there are multiple ways to obtain multiple sample images, which can be selected according to actual conditions, and the embodiments of the present disclosure do not limit this. In one possible implementation of the present disclosure, multiple sample images input by a user can be received. In another possible implementation of the present disclosure, multiple sample images can be read from other data acquisition devices or databases.

Apply the solution of the embodiment of the present disclosure to obtain multiple sample images; for a first sample image, input the first sample image and the construction prompt information into the pre-trained language model to obtain a first sample description text of the first sample image; generate first sample parameter information of the first sample image according to the first sample image and the first sample description text; according to the multiple sample images The sample description text and sample parameter information of multiple sample images are used to build a sample set. The sample description text is generated through the image understanding ability of the pre-trained language model, ensuring the readiness of the sample description text.

In practical applications, there are multiple ways to generate the first sample parameter information of the first sample image based on the first sample image and the first sample description text, which are selected according to the actual situation, and the embodiments of the present disclosure do not limit this. In a possible implementation of the present disclosure, relevant keywords, sentence structure and other information can be obtained from the first sample description text through methods such as text mining and image analysis, and this information can be used as the first sample parameter information.

In another possible implementation of the present disclosure, the segmentation model may be collaboratively detected to realize the extraction of multiple sample parameter information. That is, the above-mentioned generation of the first sample parameter information of the first sample image according to the first sample image and the first sample description text may include the following steps:

Performing region detection on the first sample image according to the first sample description text to determine key region position information of the first sample image;

Visually segment the first sample image according to the key area position information to obtain first sample parameter information of the first sample image.

Specifically, the key area position information refers to the coordinates of the center point of the key area in the first sample image, and the key area can be understood as a bounding box. For example, the first sample image is a little girl and a dog running, and the key area in the first sample image is the area where the little girl is and the area where the dog is. The first sample parameter information includes but is not limited to the height and width of the key area.

It should be noted that when performing region detection on the first sample image according to the first sample description text, the first sample description text and the first sample image can be input into the detection segmentation model, and the key region position information of the first sample image can be obtained through region detection by the detection segmentation model. When performing visual segmentation on the first sample image according to the key region position information, the key region position information and the first sample image can be input into the visual segmentation model (SAM, Segmentation As A Module) to obtain the first sample parameter information of the first sample image.

By applying the solution of the embodiment of the present disclosure, the first sample image is subjected to region detection according to the first sample description text to determine the key region position information of the first sample image; the first sample image is subjected to visual segmentation according to the key region position information to obtain the first sample parameter information of the first sample image. The accuracy of the first sample parameter information is improved through region detection and visual segmentation.

In an optional embodiment of the present disclosure, a constructed sample subset included in a sample set is taken as an example to illustrate a construction method of the constructed sample subset. After generating the first sample parameter information of the first sample image according to the first sample image and the first sample description text, the following steps may also be included:

Get parameter configuration conditions;

When the first sample parameter information does not meet the parameter configuration condition, adjust the first sample parameter information to obtain adjusted first sample parameter information;

A sample set is constructed according to a plurality of sample images, sample description texts of the plurality of sample images, and sample parameter information, including:

A sample set is constructed according to the multiple sample images, the sample description texts of the multiple sample images and the adjusted sample parameter information.

Specifically, the parameter configuration conditions include but are not limited to brightness greater than a preset brightness threshold, clarity greater than a preset clarity threshold, etc., which are selected based on actual conditions, and the embodiments of the present disclosure do not impose any limitations on this.

In practical applications, there are many ways to obtain parameter configuration conditions, which are selected according to actual conditions, and the embodiments of the present disclosure do not limit this. In one possible implementation of the present disclosure, the parameter configuration conditions input by the user can be received. In another possible implementation of the present disclosure, the parameter configuration conditions can be read from other data acquisition devices or databases.

It should be noted that after obtaining the parameter configuration condition, it can be further determined whether the first sample parameter information meets the parameter configuration condition. If the first sample parameter information meets the parameter configuration condition, the first sample parameter is not adjusted. If the first sample parameter information does not meet the parameter configuration condition, the first sample parameter information is adjusted to obtain the adjusted first sample parameter information.

By applying the solution of the embodiment of the present disclosure, the parameter configuration condition is obtained; when the first sample parameter information does not meet the parameter configuration condition, the first sample parameter information is adjusted to obtain the adjusted first sample parameter information; a sample set is constructed according to multiple sample images, sample description texts of the multiple sample images and the adjusted sample parameter information. The reliability of the sample parameter information is monitored through the parameter configuration condition, thereby improving the accuracy of the sample parameter information.

In an optional embodiment of the present disclosure, before inputting the image generation parameters and the image description text into the image generation model to obtain the target image corresponding to the image description text, the following steps may also be included:

Acquire a sample set, wherein the sample set includes a plurality of sample image-text pairs, the sample image-text pairs include a sample image and a sample description text, and the sample image-text pairs carry sample parameter information;

Inputting a plurality of sample description texts and sample parameter information corresponding to the plurality of sample description texts into an initial generation model to obtain prediction images corresponding to the plurality of sample description texts respectively;

According to the predicted image and the sample image, the model parameters of the initial generation model are adjusted to obtain a trained image generation model.

Specifically, the training method of the image generation model is supervised training, that is, the sample description text in the sample set carries the real sample image, and the sample image is the generation target of the image generation model, which is used to guide the training process of the image generation model.

It should be noted that the method of "obtaining a sample set" can refer to the method of obtaining a sample set in the training process of the parameter generation model mentioned above. The implementation method of "inputting multiple sample description texts and sample parameter information corresponding to the multiple sample description texts into the initial generation model to obtain predicted images corresponding to the multiple sample description texts" can refer to the implementation method of "inputting image generation parameters and image description texts into the image generation model to obtain target images corresponding to the image description texts" mentioned above, and the embodiments of the present disclosure will not be repeated here.

In practical applications, when adjusting the model parameters of the initial generation model according to the predicted image and the sample image, the loss value can be calculated according to the predicted image and the sample image, and the model parameters of the initial generation model can be adjusted according to the loss value until the predicted image is reached. Set a stopping condition to obtain a trained image generation model, where there are many functions for calculating loss values, such as cross entropy loss function, L1 norm loss function, maximum loss function, mean square error loss function, logarithmic loss function, etc. The specific selection is based on the actual situation, and the embodiments of the present disclosure do not impose any restrictions on this.

In a possible implementation of the present disclosure, the preset stop condition includes that the loss value is less than or equal to a preset threshold value. After the loss value is calculated based on the predicted image and the sample image, the loss value is compared with the preset threshold value.

Specifically, if the loss value is greater than the preset threshold, it means that the difference between the predicted image and the sample image is large, and the initial generation model has poor prediction ability for the predicted image. At this time, the model parameters of the initial generation model can be adjusted, and the initial generation model can continue to be trained until the loss value is less than or equal to the preset threshold, indicating that the difference between the predicted image and the sample image is small, and the preset stopping condition is met, and a trained image generation model is obtained.

In another possible implementation of the present disclosure, in addition to comparing the loss value and the preset threshold, it is also possible to determine whether the current initial generation model has been trained in combination with the number of iterations.

Specifically, if the loss value is greater than a preset threshold, the model parameters of the initial generation model are adjusted, and the initial information extraction model continues to be trained until the preset number of iterations is reached, and the iteration is stopped to obtain a trained image generation model, wherein the preset threshold and the preset number of iterations are selected according to actual conditions, and the embodiments of the present disclosure do not impose any limitations on this.

By applying the solution of the embodiment of the present disclosure, the model parameters of the initial generation model are adjusted according to the predicted image and the sample image to obtain a trained image generation model. By continuously adjusting the model parameters of the initial generation model, the final image generation model can be made more accurate.

Referring to FIG. 6 , FIG. 6 shows a flow chart of another parameter generation model training method provided by an embodiment of the present disclosure. The parameter generation model training is applied to a cloud-side device and specifically includes the following steps:

Step 602: Acquire a sample set, wherein the sample set includes a plurality of sample image-text pairs, the sample image-text pairs include a sample image and a sample description text, and the sample image-text pairs carry sample parameter information.

Step 604: input the plurality of sample image-text pairs and prediction prompt information into the pre-trained language model to obtain image prediction parameters corresponding to the plurality of sample image-text pairs.

Step 606: According to the image prediction parameters and sample parameter information, the model parameters of the pre-trained language model are adjusted to obtain a parameter generation model that has completed training.

It should be noted that the implementation method of steps 602 to 606 is detailed in the training method of the parameter generation model in the above-mentioned image generation method, and the present disclosed embodiment does not impose any limitation on this.

In actual applications, after obtaining the trained parameter generation model, the model parameters of the trained parameter generation model can be sent to the terminal device, so that the user can locally build the parameter generation model based on the model parameters to generate image generation parameters.

By applying the solution of the embodiment of the present disclosure, the model parameters of the pre-trained language model are adjusted according to the image prediction parameters and the sample parameter information to obtain a parameter generation model that has completed the training. Adjustment can make the final parameter generation model more accurate.

Referring to FIG. 7 , FIG. 7 shows a flow chart of an automatic question-answering method provided by an embodiment of the present disclosure, which specifically includes the following steps:

Step 702: Receive an image question and answer request, wherein the image question and answer request carries an image description text.

Step 704: Input the image description text and generation prompt information into a parameter generation model to obtain image generation parameters corresponding to the image description text, wherein the image generation parameters are used to describe the visual features of the image, and the parameter generation model is trained based on multiple sample image-text pairs and sample parameter information carried by the multiple sample image-text pairs.

Step 706: Input the image generation parameters and the image description text into the image generation model to obtain a reply image corresponding to the image question and answer request.

It should be noted that the implementation of steps 702 to 706 is detailed in the above steps 302 to 306, and the present embodiment does not impose any limitation on this.

By applying the solution of the embodiment of the present disclosure, the visual elements of the image are semantically decomposed by utilizing a parameter generation model to obtain image generation parameters, and accurate image generation is further completed based on the image generation parameters, so that the target image can clearly express the image description text and image generation parameters, thereby improving the interpretability and controllability of the automatic question-answering process.

Referring to FIG8 , FIG8 shows a flowchart of the processing process of another image generation method provided by an embodiment of the present disclosure. During the image generation process, the powerful semantic understanding and associative ability of the parameter generation model is used to complete the semantic decomposition and reorganization of the visual elements of the image, and the parameters are input into the image generation model to complete accurate image generation. The image generation stage can be divided into two stages: the parameter generation model processing stage and the image generation model processing stage, wherein the parameter generation model processing stage includes fine-tuning the pre-trained language model, constructing the conditional generation sample set, and designing the generated prompt information;

As shown in FIG8 , a parameter generation model is obtained by training a pre-trained language model based on multiple sample image-text pairs (composed of sample description texts and sample images) and sample parameter information carried by multiple sample image-text pairs; the image description text is obtained; the image description text and generation prompt information are input into the parameter generation model to obtain image generation parameters corresponding to the image description text; the image generation parameters and the image description text are input into the image generation model to obtain a target image corresponding to the image description text.

By applying the solution of the embodiment of the present disclosure, a two-stage diffusion generation architecture is used to implement auxiliary generation based on a pre-trained language model. In the parameter generation model processing stage, the emergence ability and appropriate fine-tuning of the pre-trained language model are used to output the image generation parameters required in the image generation model processing stage. Subsequently, in the image generation model processing stage, image generation based on image generation parameter control is performed to output an image with precise semantics. Moreover, since the generated image can clearly express the generation prompt information or image generation parameters used in the generation process, the user can understand and verify the accuracy of the generation result.

Referring to FIG. 9 , FIG. 9 shows a schematic diagram of a processing process of a parameter generation model provided by an embodiment of the present disclosure. As shown in FIG. 9 , the image description text is obtained as “two black cats and a white dog on an orange sofa”, and the image description text and the generated prompt information 1 are combined. The title of my image is “two black cats and a white dog on an orange sofa” and “elements in the image” What is it? " Input parameter generation model, get the image generation parameter 1 corresponding to the image description text "[(black cat, [30, 171, 212, 286]), (black cat, [40, 200, 123, 412]), (white dog, [24, 543, 231, 332]), (sofa, [264, 173, 222, 221])]"; combine the image description text and generate prompt information 2 "What is their bounding box?" Input parameter generation model, get the image generation parameter 2 corresponding to the image description text "[(black cat, [30, 171, 212, 286]), (black cat, [40, 200, 123, 412]), (white dog, [24, 543, 231, 332]), (sofa, [264, 173, 222, 221])]"; combine the image description text and generate prompt information 3 "What is the main color of each element? "Input parameter generation model, obtain image generation parameter 3 corresponding to the image description text "[(black cat, [black, gray, dark gray]),..., (white dog, [white, light gray, brown]), (sofa, [orange, brown, gray])]"; finally, aggregate image generation parameter 1, image generation parameter 2 and image generation parameter 3 to obtain the final image generation parameter.

Referring to FIG. 10 , FIG. 10 shows a schematic diagram of an interface of an image generation interface provided by an embodiment of the present disclosure. The image generation interface is divided into a request input interface and a result display interface. The request input interface includes a request input box, an “OK” control, and a “Cancel” control. The result display interface includes a result display box.

The user inputs an image generation request through the request input box displayed on the client, wherein the image generation request carries the image description text, clicks the "OK" control, and the server receives the image description text sent by the client, inputs the image description text and the generation prompt information into the parameter generation model, and obtains the image generation parameters corresponding to the image description text, wherein the image generation parameters are used to describe the visual features of the image, and the parameter generation model is trained based on multiple sample image-text pairs and sample parameter information carried by multiple sample image-text pairs; the image generation parameters and the image description text are input into the image generation model, and the target image corresponding to the image description text is obtained, and the target image is sent to the client. The client displays the target image in the result display box.

In actual applications, users can operate controls by clicking, double-clicking, touching, hovering the mouse, sliding, long pressing, voice control, or shaking, etc. The selection is based on actual conditions, and the embodiments of the present disclosure do not impose any limitations on this.

Corresponding to the above-mentioned image generation method embodiment, the present disclosure also provides an image generation device embodiment, and FIG11 shows a structural schematic diagram of an image generation device provided by an embodiment of the present disclosure. As shown in FIG11 , the device includes:

A first acquisition module 1102 is configured to acquire image description text;

The first input module 1104 is configured to input the image description text and the generation prompt information into a parameter generation model to obtain image generation parameters corresponding to the image description text, wherein the image generation parameters are used to describe the visual features of the image, and the parameter generation model is trained based on a plurality of sample image-text pairs and sample parameter information carried by the plurality of sample image-text pairs;

The second input module 1106 is configured to input the image generation parameters and the image description text into the image generation model to obtain a target image corresponding to the image description text.

Optionally, the image generation model includes a parameter encoding unit and a coding unit; the second input module 1106 is further configured to input the image generation parameters and the image description text into the image generation model, and encode the image generation parameters to obtain parameter encoding features through the parameter encoding unit; and generate the image generation parameters according to the parameter encoding features and the image description text through the coding unit. The target image corresponding to the image description text.

Optionally, the parameter coding unit includes a one-dimensional parameter coding unit, a two-dimensional parameter coding unit and a feature aggregation unit; the second input module 1106 is further configured to encode the one-dimensional parameters in the image generation parameters through the one-dimensional parameter coding unit to obtain one-dimensional coding features; encode the two-dimensional parameters in the image generation parameters through the two-dimensional parameter coding unit to obtain two-dimensional coding features; and aggregate the one-dimensional coding features and the two-dimensional coding features through the feature aggregation unit to obtain parameter coding features.

Optionally, the device also includes: a third acquisition module, configured to obtain image update parameters for the target image; determine the image editing area in the target image based on the image update parameters; mask the image editing area to obtain a mask generation sequence; input the mask generation sequence and the target image into the image generation model to obtain an updated target image.

Optionally, the device further includes: a first sending module configured to send the target image and the updated target image to the client, so that the client displays the target image and the updated target image to the user.

Optionally, the device further includes: a second receiving module configured to receive image selection information sent by a user through a client, and adjust model parameters of the image generation model based on the image selection information.

Optionally, the device further includes: a third receiving module, configured to receive adjustment sample data sent by a user through a client, and adjust model parameters of the image generation model according to the adjustment sample data, wherein the adjustment sample data is constructed based on the target image.

Optionally, the device also includes: a parameter generation model training module, configured to obtain a sample set, wherein the sample set includes multiple sample image-text pairs, the sample image-text pairs include sample images and sample description texts, and the sample image-text pairs carry sample parameter information; input the multiple sample image-text pairs and prediction prompt information into a pre-trained language model to obtain image prediction parameters corresponding to the multiple sample image-text pairs; adjust the model parameters of the pre-trained language model according to the image prediction parameters and the sample parameter information to obtain a parameter generation model that has completed training.

Optionally, the parameter generation model training module is further configured to obtain a plurality of sample images;

For a first sample image, the first sample image and construction prompt information are input into a pre-trained language model to obtain a first sample description text of the first sample image; based on the first sample image and the first sample description text, first sample parameter information of the first sample image is generated; based on multiple sample images, sample description texts of multiple sample images, and sample parameter information, a sample set is constructed.

Optionally, the parameter generation model training module is further configured to perform region detection on the first sample image according to the first sample description text to determine key region position information of the first sample image; perform visual segmentation on the first sample image according to the key region position information to obtain first sample parameter information of the first sample image.

Optionally, the device also includes: a fourth acquisition module, configured to acquire parameter configuration conditions; when the first sample parameter information does not meet the parameter configuration conditions, adjusting the first sample parameter information to obtain the adjusted first sample parameter information; a parameter generation model training module, further configured to construct a sample set based on multiple sample images, sample description texts of multiple sample images and the adjusted sample parameter information.

Optionally, the device further comprises: an image generation model training module configured to obtain a sample set, wherein the sample set comprises a plurality of sample image-text pairs, the sample image-text pairs comprise a sample image and a sample description text, the sample image-text pairs This pair carries sample parameter information; multiple sample description texts and sample parameter information corresponding to the multiple sample description texts are input into the initial generation model to obtain predicted images corresponding to the multiple sample description texts; according to the predicted images and the sample images, the model parameters of the initial generation model are adjusted to obtain a trained image generation model.

By applying the solution of the embodiment of the present disclosure, the visual elements of the image are semantically decomposed by utilizing a parameter generation model to obtain image generation parameters, and accurate image generation is further completed based on the image generation parameters, so that the target image can clearly express the image description text and image generation parameters, thereby improving the interpretability and controllability of image generation.

The above is a schematic scheme of an image generating device of this embodiment. It should be noted that the technical scheme of the image generating device and the technical scheme of the above image generating method belong to the same concept, and the details not described in detail in the technical scheme of the image generating device can be referred to the description of the technical scheme of the above image generating method.

Corresponding to the above-mentioned parameter generation model training method embodiment, the present disclosure also provides a parameter generation model training device embodiment, and FIG12 shows a structural schematic diagram of a parameter generation model training device provided by an embodiment of the present disclosure. As shown in FIG12, the device is applied to a cloud-side device, including:

The second acquisition module 1202 is configured to acquire a sample set, wherein the sample set includes a plurality of sample image-text pairs, the sample image-text pairs include a sample image and a sample description text, and the sample image-text pairs carry sample parameter information;

The third input module 1204 is configured to input a plurality of sample image-text pairs and prediction prompt information into the pre-trained language model to obtain image prediction parameters corresponding to the plurality of sample image-text pairs respectively;

The adjustment module 1206 is configured to adjust the model parameters of the pre-trained language model according to the image prediction parameters and the sample parameter information to obtain a parameter generation model that has completed training.

By applying the solution of the embodiment of the present disclosure, the model parameters of the pre-trained language model are adjusted according to the image prediction parameters and sample parameter information to obtain a parameter generation model that has completed training. By continuously adjusting the model parameters of the pre-trained language model, the parameter generation model finally obtained can be made more accurate.

The above is a schematic scheme of a parameter generation model training device of this embodiment. It should be noted that the technical scheme of the parameter generation model training device and the technical scheme of the parameter generation model training method mentioned above belong to the same concept, and the details not described in detail in the technical scheme of the parameter generation model training device can be found in the description of the technical scheme of the parameter generation model training method mentioned above.

Corresponding to the above-mentioned automatic question-answering method embodiment, the present disclosure also provides an automatic question-answering device embodiment. FIG13 shows a schematic diagram of the structure of an automatic question-answering device provided by an embodiment of the present disclosure. As shown in FIG13 , the device includes:

The first receiving module 1302 is configured to receive an image question and answer request, wherein the image question and answer request carries an image description text;

The fourth input module 1304 is configured to input the image description text and the generation prompt information into the parameter generation model to obtain the image generation parameters corresponding to the image description text, wherein the image generation parameters are used to describe the visual features of the image, and the parameter generation model is trained based on multiple sample image-text pairs and sample parameter information carried by the multiple sample image-text pairs;

The fifth input module 1306 is configured to input the image generation parameters and the image description text into the image generation model to obtain a reply image corresponding to the image question and answer request.

By applying the solution of the embodiment of the present disclosure, the visual elements of the image are semantically decomposed by utilizing a parameter generation model to obtain image generation parameters, and accurate image generation is further completed based on the image generation parameters, so that the target image can clearly express the image description text and image generation parameters, thereby improving the interpretability and controllability of the automatic question-answering process.

The above is a schematic scheme of an automatic question-answering device of this embodiment. It should be noted that the technical scheme of the automatic question-answering device and the technical scheme of the automatic question-answering method described above are of the same concept, and the details not described in detail in the technical scheme of the automatic question-answering device can be found in the description of the technical scheme of the automatic question-answering method described above.

14 shows a block diagram of a computing device provided by an embodiment of the present disclosure. The components of the computing device 1400 include but are not limited to a memory 1410 and a processor 1420. The processor 1420 is connected to the memory 1410 via a bus 1430, and a database 1450 is used to store data.

The computing device 1400 also includes an access device 1440 that enables the computing device 1400 to communicate via one or more networks 1460. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 1440 may include one or more of any type of network interface, wired or wireless (e.g., a network interface card (NIC)), such as an IEEE802.11 wireless local area network (WLAN) wireless interface, a World Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.

In one embodiment of the present disclosure, the above components of the computing device 1400 and other components not shown in FIG. 14 may also be connected to each other, for example, through a bus. It should be understood that the computing device structure block diagram shown in FIG. 14 is only for illustrative purposes and is not intended to limit the scope of the present disclosure. Those skilled in the art may add or replace other components as needed.

Computing device 1400 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). Computing device 1400 may also be a mobile or stationary server.

Among them, the processor 1420 is used to execute the following computer-executable instructions, which, when executed by the processor, implement the steps of the above-mentioned image generation method or parameter generation model training method or automatic question-answering method.

The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical schemes of the above-mentioned image generation method, parameter generation model training method and automatic question answering method belong to the same concept. For details not described in detail in the technical scheme of the computing device, please refer to the above-mentioned image generation method or parameter generation model training method. A description of the technical solution for the data generation model training method or the automatic question answering method.

An embodiment of the present disclosure also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, can implement the steps of the above-mentioned image generation method or parameter generation model training method or automatic question-answering method.

The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the above-mentioned image generation method, parameter generation model training method and automatic question answering method belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the technical scheme of the above-mentioned image generation method or parameter generation model training method or automatic question answering method.

An embodiment of the present disclosure further provides a computer program, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the above-mentioned image generation method or parameter generation model training method or automatic question-answering method.

The above is a schematic scheme of a computer program of this embodiment. It should be noted that the technical scheme of the computer program and the technical schemes of the above-mentioned image generation method, parameter generation model training method and automatic question answering method belong to the same concept, and the details not described in detail in the technical scheme of the computer program can be found in the description of the technical schemes of the above-mentioned image generation method or parameter generation model training method or automatic question answering method.

The above describes specific embodiments of the present disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should know that the embodiments of the present disclosure are not limited by the described action sequence, because according to the embodiments of the present disclosure, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of the present disclosure.

In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

The preferred embodiments of the present disclosure disclosed above are only used to help explain the present disclosure. The optional embodiments are not described in detail. All details are provided, and the invention is not limited to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of the present disclosure. The present disclosure selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of the present disclosure, so that those skilled in the art can understand and use the present disclosure well. The present disclosure is limited only by the claims and their full scope and equivalents.

Claims (17)

A method for generating an image, comprising: Get the image description text; Input the image description text and the generation prompt information into a parameter generation model to obtain image generation parameters corresponding to the image description text, wherein the image generation parameters are used to describe the visual features of the image, and the parameter generation model is trained based on multiple sample image-text pairs and sample parameter information carried by the multiple sample image-text pairs; The image generation parameters and the image description text are input into an image generation model to obtain a target image corresponding to the image description text. The method according to claim 1, wherein the image generation model comprises a parameter encoding unit and a coding and decoding unit; The step of inputting the image generation parameters and the image description text into an image generation model to obtain a target image corresponding to the image description text includes: Inputting the image generation parameters and the image description text into an image generation model, and encoding the image generation parameters through the parameter encoding unit to obtain parameter encoding features; The encoding and decoding unit generates a target image corresponding to the image description text according to the parameter encoding features and the image description text. According to the method of claim 2, the parameter encoding unit includes a one-dimensional parameter encoding unit, a two-dimensional parameter encoding unit and a feature aggregation unit; The parameter encoding unit generates parameter encoding for the image to obtain parameter encoding features, including: The one-dimensional parameter encoding unit encodes the one-dimensional parameter in the image generation parameter to obtain a one-dimensional encoding feature; The two-dimensional parameter encoding unit encodes the two-dimensional parameters in the image generation parameters to obtain two-dimensional encoding features; The one-dimensional coding feature and the two-dimensional coding feature are aggregated by the feature aggregation unit to obtain a parameter coding feature. The method according to any one of claims 1 to 3, after inputting the image generation parameters and the image description text into an image generation model to obtain a target image corresponding to the image description text, further comprising: Acquire image update parameters for the target image; Determining an image editing area in the target image according to the image update parameters; Masking the image editing area to obtain a mask generation sequence; The mask generation sequence and the target image are input into the image generation model to obtain an updated target image. The method according to claim 4, after inputting the mask generation sequence and the target image into the image generation model to obtain the updated target image, further comprising: The target image and the updated target image are sent to a client, so that the client displays the target image and the updated target image to a user. The method according to claim 5, after sending the target image and the updated target image to the client, further comprising: The image selection information sent by the user through the client is received, and the model parameters of the image generation model are adjusted based on the image selection information. The method according to any one of claims 1 to 6, after inputting the image generation parameters and the image description text into an image generation model to obtain a target image corresponding to the image description text, further comprising: Receive adjustment sample data sent by the user through the client, and adjust the image generation according to the adjustment sample data The adjustment sample data is constructed based on the target image. According to any one of claims 1 to 7, before inputting the image description text and the generation prompt information into a parameter generation model to obtain image generation parameters corresponding to the image description text, the method further comprises: Acquire a sample set, wherein the sample set includes a plurality of sample image-text pairs, the sample image-text pairs include a sample image and a sample description text, and the sample image-text pairs carry sample parameter information; Inputting the plurality of sample image-text pairs and prediction prompt information into a pre-trained language model to obtain image prediction parameters corresponding to the plurality of sample image-text pairs respectively; According to the image prediction parameters and the sample parameter information, the model parameters of the pre-trained language model are adjusted to obtain a parameter generation model that has completed training. According to the method of claim 8, obtaining the sample set comprises: Acquire multiple sample images; For a first sample image, inputting the first sample image and construction prompt information into a pre-trained language model to obtain a first sample description text of the first sample image; generating first sample parameter information of the first sample image according to the first sample image and the first sample description text; A sample set is constructed according to the multiple sample images, the sample description texts of the multiple sample images, and the sample parameter information. According to the method of claim 9, generating first sample parameter information of the first sample image according to the first sample image and the first sample description text comprises: Performing region detection on the first sample image according to the first sample description text to determine key region position information of the first sample image; Visually segment the first sample image according to the key area position information to obtain first sample parameter information of the first sample image. The method according to claim 9, after generating the first sample parameter information of the first sample image according to the first sample image and the first sample description text, further comprising: Get parameter configuration conditions; If the first sample parameter information does not meet the parameter configuration condition, adjust the first sample parameter information to obtain adjusted first sample parameter information; The constructing a sample set according to the plurality of sample images, sample description texts of the plurality of sample images and sample parameter information includes: A sample set is constructed according to the multiple sample images, the sample description texts of the multiple sample images, and the adjusted sample parameter information. According to any one of claims 1 to 11, before inputting the image generation parameters and the image description text into an image generation model to obtain a target image corresponding to the image description text, the method further comprises: Acquire a sample set, wherein the sample set includes a plurality of sample image-text pairs, the sample image-text pairs include a sample image and a sample description text, and the sample image-text pairs carry sample parameter information; Inputting a plurality of sample description texts and sample parameter information corresponding to the plurality of sample description texts into an initial generation model to obtain prediction images corresponding to the plurality of sample description texts respectively; According to the predicted image and the sample image, the model parameters of the initial generation model are adjusted to obtain a completed training Image generation model trained. A parameter generation model training method, applied to a cloud-side device, includes: Acquire a sample set, wherein the sample set includes a plurality of sample image-text pairs, the sample image-text pairs include a sample image and a sample description text, and the sample image-text pairs carry sample parameter information; Inputting the plurality of sample image-text pairs and prediction prompt information into a pre-trained language model to obtain image prediction parameters corresponding to the plurality of sample image-text pairs respectively; According to the image prediction parameters and the sample parameter information, the model parameters of the pre-trained language model are adjusted to obtain a parameter generation model that has completed training. An automatic question answering method, comprising: receiving an image question and answer request, wherein the image question and answer request carries an image description text; Input the image description text and the generation prompt information into a parameter generation model to obtain image generation parameters corresponding to the image description text, wherein the image generation parameters are used to describe the visual features of the image, and the parameter generation model is trained based on multiple sample image-text pairs and sample parameter information carried by the multiple sample image-text pairs; The image generation parameters and the image description text are input into an image generation model to obtain a reply image corresponding to the image question and answer request. A computing device comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the method described in any one of claims 1 to 12 or claim 13 or claim 14 are implemented. A computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the method described in any one of claims 1 to 12 or claim 13 or claim 14. A computer program, which, when executed by a computer, implements the method according to any one of claims 1 to 12 or claim 13 or claim 14.