CN117853600A - Image generation method, device, electronic equipment and storage medium - Google Patents

Abstract

The disclosure provides an image generation method, an image generation device, electronic equipment and a storage medium, relates to the technical field of artificial intelligence, and particularly relates to the fields of natural language processing, computer vision, deep learning and the like. The image generation method comprises the following steps: acquiring first description data input by a user in a current round of dialogue, wherein the first description data is used for describing a first image to be generated; generating a first image based on the first description data; and outputting the first image and at least one control for the first image as response data of the current wheel dialogue, wherein the at least one control corresponds to at least one action for the first image respectively, and any control in the at least one control is configured to respond to the operation of the control by a user to execute the corresponding action on the first image.

Description

Image generation method, device, electronic device and storage medium

Technical Field

The present disclosure relates to the field of artificial intelligence technology, in particular to technical fields such as natural language processing, computer vision, and deep learning, and specifically to an image generation method and device, an electronic device, a computer-readable storage medium, and a computer program product.

Background technique

Artificial Intelligence (AI) is a discipline that studies how computers can simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.). It includes both hardware-level and software-level technologies. AI hardware technologies generally include technologies such as sensors, dedicated AI chips, cloud computing, distributed storage, and big data processing; AI software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, as well as machine learning/deep learning, big data processing technology, and knowledge graph technology.

Large Language Model (LLM) is a deep learning model trained with a large amount of text data, which can generate natural language text or understand the meaning of natural language text. Large language models can handle a variety of natural language tasks, such as dialogue, text classification, text generation, etc., and are an important path to artificial intelligence. Some large language models also have multimodal data processing capabilities, such as the ability to process multimodal data such as text, images, and videos.

The methods described in this section are not necessarily methods that have been previously conceived or employed. Unless otherwise indicated, it should not be assumed that any method described in this section is considered to be prior art simply because it is included in this section. Similarly, unless otherwise indicated, the issues mentioned in this section should not be considered to have been recognized in any prior art.

Summary of the invention

The present disclosure provides an image generating method and device, an electronic device, a computer-readable storage medium, and a computer program product.

According to one aspect of the present disclosure, there is provided an image generation method, comprising: obtaining first description data input by a user in a current round of dialogue, wherein the first description data is used to describe a first image to be generated; generating the first image based on the first description data; and outputting the first image and at least one control for the first image as response data for the current round of dialogue, wherein the at least one control corresponds to at least one action for the first image, respectively, and any one of the at least one control is configured to perform a corresponding action on the first image in response to the user's operation on the control.

According to one aspect of the present disclosure, there is provided an image generating device, comprising: a first acquisition module, configured to acquire first description data input by a user in a current round of dialogue, wherein the first description data is used to describe a first image to be generated; a first generation module, configured to generate the first image based on the first description data; and a first output module, configured to output the first image and at least one control for the first image as response data for the current round of dialogue, wherein the at least one control corresponds to at least one action for the first image, respectively, and any one of the at least one control is configured to perform a corresponding action on the first image in response to the user's operation on the control.

According to one aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the above method.

According to one aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to execute the above method.

According to one aspect of the present disclosure, a computer program product is provided, comprising computer program instructions, which implement the above method when executed by a processor.

According to one or more embodiments of the present disclosure, conversational image generation can be achieved, and users are supported to interact with a graphical interface, thereby reducing operation complexity and improving image generation efficiency.

It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings exemplarily illustrate the embodiments and constitute a part of the specification, and together with the text description of the specification, are used to explain the exemplary implementation of the embodiments. The embodiments shown are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to similar but not necessarily identical elements.

FIG1 shows a schematic diagram of an exemplary system in which various methods described herein may be implemented according to an embodiment of the present disclosure;

FIG2 shows a flow chart of an image generating method according to an embodiment of the present disclosure;

3A-3P are schematic diagrams showing a dialogue interface according to an embodiment of the present disclosure;

FIG4 shows a structural block diagram of an image generating device according to an embodiment of the present disclosure; and

FIG. 5 shows a structural block diagram of an exemplary electronic device that can be used to implement an embodiment of the present disclosure.

Detailed ways

The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

In the present disclosure, unless otherwise specified, the use of the terms "first", "second", etc. to describe various elements is not intended to limit the positional relationship, temporal relationship, or importance relationship of these elements, and such terms are only used to distinguish one element from another element. In some examples, the first element and the second element may refer to the same instance of the element, and in some cases, based on the description of the context, they may also refer to different instances.

The terms used in the description of various examples in this disclosure are only for the purpose of describing specific examples and are not intended to be limiting. Unless the context clearly indicates otherwise, if the number of elements is not specifically limited, the element can be one or more. In addition, the term "and/or" used in this disclosure covers any one of the listed items and all possible combinations. "Multiple" refers to two or more.

In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

Image generation tasks can be divided into two categories: image creation and image editing. Image creation refers to creating a new image according to the user's requirements. Image editing refers to making specified changes to an existing image (e.g., an already created image) to obtain a new image.

In the related art, traditional image generation tools, such as CorelDRAW, PhotoShop and other drawing software, are often used to generate images. These image generation tools have high barriers to use and require users to undergo special training. In addition, the operation process is complex and cumbersome, resulting in low efficiency and high cost of image generation, which is difficult to meet user needs.

AIGC (AI Generated Content) technology has shown great potential in image generation tasks. The current mainstream AI image generation technology is the Vincent graph technology, that is, the prompt text (i.e., prompt) given by the user is input into the Vincent graph model, and the Vincent graph model will output the generated image. Although the Vincent graph technology has effectively improved the image generation efficiency compared to traditional image generation tools, the image generation effect of the Vincent graph technology depends on the quality of the prompt text given by the user. Users usually need to modify the prompt text multiple times to obtain a relatively ideal image generation effect, so the threshold for using this technology is still high. In addition, this technology can only perform a single image generation and cannot achieve continuous image generation (for example, create an image first and then edit the image). Therefore, the image generation efficiency of the Vincent graph technology is still low and it is difficult to meet user needs.

As can be seen from the above, the image generation solutions in the related art are not universal, the user threshold is high, the operation process is complicated and cumbersome, continuous image generation cannot be achieved, the image generation efficiency is low, and it is difficult to meet user needs.

In view of the above problems, the embodiments of the present disclosure provide a conversational image generation method that integrates a graphical interface. Users can complete image generation by having a conversation with the AI image generation system, which greatly reduces the complexity of operation and improves the efficiency of image generation.

In each round of dialogue, based on the first description data input by the user, the user's image generation requirements are understood and a first image is generated, and the generated first image and at least one control are returned to the user as the response data of the current round of dialogue. The user can further process the first image by operating the control, so that the user can complete continuous image generation in the same dialogue interface, which simplifies the operation, improves the image generation efficiency, and thus improves the user experience.

The embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

FIG1 shows a schematic diagram of an exemplary system 100 in which various methods and apparatuses described herein may be implemented according to an embodiment of the present disclosure. Referring to FIG1 , the system 100 includes one or more client devices 101, 102, 103, 104, 105, and 106, a server 120, and one or more communication networks 110 coupling the one or more client devices to the server 120. The client devices 101, 102, 103, 104, 105, and 106 may be configured to execute one or more applications.

In an embodiment of the present disclosure, the client devices 101 , 102 , 103 , 104 , 105 , and 106 and the server 120 may run one or more services or software applications that enable execution of the image generation method or the image generation method.

In some embodiments, server 120 may also provide other services or software applications, which may include non-virtualized environments and virtualized environments. In some embodiments, these services may be provided as web-based services or cloud services, such as provided to users of client devices 101, 102, 103, 104, 105, and/or 106 under a software as a service (SaaS) model.

In the configuration shown in FIG. 1 , server 120 may include one or more components that implement the functions performed by server 120. These components may include software components, hardware components, or a combination thereof that can be executed by one or more processors. Users operating client devices 101, 102, 103, 104, 105, and/or 106 may in turn utilize one or more client applications to interact with server 120 to utilize the services provided by these components. It should be understood that a variety of different system configurations are possible, which may be different from system 100. Therefore, FIG. 1 is an example of a system for implementing the various methods described herein and is not intended to be limiting.

Client devices 101, 102, 103, 104, 105 and/or 106 may provide an interface that enables a user of the client device to interact with the client device. The client device may also output information to the user via the interface. Although FIG. 1 depicts only six client devices, those skilled in the art will appreciate that the present disclosure may support any number of client devices.

Client devices 101, 102, 103, 104, 105 and/or 106 may include various types of computer devices, such as portable handheld devices, general-purpose computers (such as personal computers and laptop computers), workstation computers, wearable devices, smart screen devices, self-service terminal devices, service robots, vehicle-mounted devices, game systems, thin clients, various messaging devices, sensors or other sensing devices, etc. These computer devices may run various types and versions of software applications and operating systems, such as MICROSOFT Windows, APPLE iOS, UNIX-like operating systems, Linux or Linux-like operating systems; or include various mobile operating systems, such as MICROSOFT Windows Mobile OS, iOS, Windows Phone, Android. Portable handheld devices may include cellular phones, smart phones, tablet computers, personal digital assistants (PDAs), etc. Wearable devices may include head-mounted displays (such as smart glasses) and other devices. Game systems may include various handheld game devices, Internet-enabled game devices, etc. Client devices are capable of executing a variety of different applications, such as various Internet-related applications, communication applications (such as email applications), short message service (SMS) applications, and may use various communication protocols.

The network 110 may be any type of network known to those skilled in the art that may support data communications using any of a variety of available protocols, including but not limited to TCP/IP, SNA, IPX, etc. By way of example only, the one or more networks 110 may be a local area network (LAN), an Ethernet-based network, a token ring, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a blockchain network, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., Bluetooth, Wi-Fi), and/or any combination of these and/or other networks.

Server 120 may include one or more general purpose computers, dedicated server computers (e.g., PC (personal computer) servers, UNIX servers, mid-range servers), blade servers, mainframe computers, server clusters, or any other suitable arrangement and/or combination. Server 120 may include one or more virtual machines running virtual operating systems, or other computing architectures involving virtualization (e.g., one or more flexible pools of logical storage devices that may be virtualized to maintain a server's virtual storage device). In various embodiments, server 120 may run one or more services or software applications that provide the functionality described below.

The computing units in the server 120 may run one or more operating systems including any of the above operating systems and any commercially available server operating systems. The server 120 may also run any of a variety of additional server applications and/or middle-tier applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, etc.

In some implementations, server 120 may include one or more applications to analyze and consolidate data feeds and/or event updates received from users of client devices 101, 102, 103, 104, 105, and/or 106. Server 120 may also include one or more applications to display data feeds and/or real-time events via one or more display devices of client devices 101, 102, 103, 104, 105, and/or 106.

In some embodiments, the server 120 may be a server of a distributed system, or a server combined with a blockchain. The server 120 may also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology. A cloud server is a host product in a cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and virtual private servers (VPS) services.

The system 100 may also include one or more databases 130. In some embodiments, these databases may be used to store data and other information. For example, one or more of the databases 130 may be used to store information such as audio files and video files. The databases 130 may reside in various locations. For example, the database used by the server 120 may be local to the server 120, or may be remote from the server 120 and may communicate with the server 120 via a network-based or dedicated connection. The databases 130 may be of different types. In some embodiments, the databases used by the server 120 may be, for example, relational databases. One or more of these databases may store, update, and retrieve data to and from the databases in response to commands.

In some embodiments, one or more of the databases 130 may also be used by applications to store application data. The databases used by the applications may be different types of databases, such as a key-value store, an object store, or a conventional store backed by a file system.

The system 100 of FIG. 1 may be configured and operated in various ways to enable application of various methods and apparatuses described according to the present disclosure.

According to some embodiments, the client devices 101-106 can execute the image generation method of the embodiment of the present disclosure to provide an immersive image generation service for the user. Specifically, the user can input the query data for each round of dialogue by operating the client devices 101-106 (for example, operating input devices such as a mouse and a touch screen) to express his or her image generation needs. The client devices 101-106 generate the image desired by the user by executing the image generation method of the embodiment of the present disclosure, and output the image and the corresponding controls as the response data of the current round of dialogue to the user (for example, output through a display).

According to some embodiments, the server 120 may also execute the image generation method according to the embodiment of the present disclosure. Specifically, the user may input the question data for each round of dialogue by operating the client devices 101-106 (for example, operating input devices such as a mouse and a touch screen) to express his or her image generation needs. The client devices 101-106 send the question data for each round of dialogue of the user to the server 120. The server 120 generates the image desired by the user by executing the image generation method of the embodiment of the present disclosure, and outputs the image and the corresponding controls as the response data of the current round of dialogue to the client devices 101-106. The client devices 101-106 further output the response data to the user (for example, through a display).

Fig. 2 shows a flow chart of an image generation method 200 according to an embodiment of the present disclosure. As described above, the execution subject of the method 200 may be a client device, such as the client devices 101-106 shown in Fig. 1, or a server, such as the server 120 shown in Fig. 1.

As shown in FIG. 2 , the method 200 includes steps S210 - S230 .

In step S210, first description data input by the user in the current round of dialogue is obtained. The first description data is used to describe the first image to be generated.

In step S220 , a first image is generated based on the first description data.

In step S230, the first image and at least one control for the first image are output as response data for the current round of dialogue, wherein at least one control corresponds to at least one action for the first image, and any one of the at least one control is configured to perform a corresponding action on the first image in response to a user's operation on the control.

According to an embodiment of the present disclosure, a conversational image generation method integrating a graphical interface is provided. A user can complete image generation by having a conversation with an AI image generation system and interacting with the graphical interface, which greatly reduces the complexity of operation and improves the efficiency of image generation.

In each round of dialogue, based on the first description data input by the user, the user's image generation requirements are understood and a first image is generated, and the generated first image and at least one control are returned to the user as the response data of the current round of dialogue. The user can further process the first image by operating the control, so that the user can complete continuous image generation in the same dialogue interface, which simplifies the operation, improves the image generation efficiency, and thus improves the user experience.

The following is a detailed description of each step of method 200.

In step S210, the first description data input by the user in the current round of dialogue is obtained.

In an embodiment of the present disclosure, a dialogue refers to an interactive process in which a user inputs a question (query) and an AI image generation system outputs a response (response). Depending on the number of interactions between the user and the AI image generation system, the dialogue can be divided into a single-round dialogue and a multi-round dialogue. In a single-round dialogue, the user interacts with the AI image generation system only once. After the user inputs a question and gets a response output by the system, the dialogue ends. In a multi-round dialogue, the user interacts with the AI image generation multiple times. Each interaction is called a "round" in the dialogue, including the question input by the user and the response output by the system to the question.

The current round of dialogue may be any round in the dialogue, for example, it may be the first round of dialogue, the second round of dialogue, the third round of dialogue, and the like.

The first description data is used to describe the first image to be generated, that is, to describe the image generation requirements of the user. For example, the first description data may describe the style, color, size, elements included in the image, etc. of the image that the user expects to generate.

The first description data may include data of any modality, such as text, image, voice, etc. According to some embodiments, the first description data may include only the first description text. According to some embodiments, the first description data may include both the first description text and the reference image. According to some embodiments, the first description data may include only the first voice. In the case where the first description data includes voice, the voice may be converted into text by voice recognition technology, thereby simplifying subsequent data processing steps and improving computing efficiency.

FIG3A shows a schematic diagram of a dialogue interface according to an embodiment of the present disclosure. The dialogue interface includes an input box 301. The user can enter a first description text "Help me draw a big white rabbit on the grass by the stream, in a cartoon style" in the input box 301, and send the first description text to the AI image generation system (also referred to as "AI painting assistant") by clicking button 302. The dialogue interface after sending the first description text is shown in FIG3B. In the dialogue interface shown in FIG3B, U represents the user and AI represents the AI image generation system.

Returning to FIG. 3A , in addition to inputting the first description text “Help me draw a big white rabbit on the grass beside the stream, in a cartoon style” in the input box 301 , the user can also input a reference image by clicking button 303 . As shown in FIG. 3A , when the user has not input a reference image, button 303 displays the words “Upload reference image” to prompt the user to input a reference image. After the user inputs a reference image, button 303 displays a thumbnail of the input reference image, as shown in FIG. 3C .

In the dialogue interface shown in FIG3C , the user can send the first description text and the reference image as the first description data to the AI image generation system by clicking button 302. The dialogue interface after sending the first description data is shown in FIG3D . Compared with the dialogue interface shown in FIG3B , the dialogue interface shown in FIG3D adds a bubble 305. Bubble 305 indicates that the user has entered a reference image. By performing interactive operations on bubble 305 (such as clicking, hovering the mouse pointer, etc.), a thumbnail of the reference image will pop up above bubble 305.

In the conversation interface shown in FIGS. 3A-3D , the user can end the current conversation and start a new conversation by clicking button 304 .

In some cases, the user may not be clear about what kind of image he wants to generate, that is, it is not clear how the first description text should be expressed. To avoid this situation, according to some embodiments, before the user enters the first description text, the AI image generation system can actively recommend a second image to the user and output a second description text for describing the second image to inspire the user's image creation inspiration and guide the user to enter the first description text based on the second description text.

The second image may be calculated according to any recommendation algorithm. There may be one or more second images.

According to some embodiments, the second image and its second description text may be output simultaneously without the user having to perform additional operations, so as to inspire the user's creative inspiration.

According to some embodiments, only the second image may be output. The description text of the second image is not acquired in the initial state or is acquired but not output, that is, the description text of the second image is not visible without additional operation by the user. In response to the user's selection operation on a second image, the second description text of the second image is output. According to this embodiment, only the second description text of the second image that the user is interested in is displayed, which can avoid the user from being unable to obtain inspiration due to providing too much redundant information to the user, and can save the display space of the dialogue interface.

3E-3H show schematic diagrams of a conversation interface for providing creative inspiration to a user according to an implementation of the present disclosure.

As shown in FIG3E , when the user enters the conversation interface for the first time or starts a new conversation, the second images 306-309 are actively recommended to the user. At this time, the second description text of each second image is invisible. If the user is not satisfied with the currently recommended second images 306-309, he can obtain new second images by clicking the "Change a Batch" label in the message.

When the user selects the second image 306 (for example, the user clicks the second image 306, the time the mouse pointer hovers over the second image 306 exceeds a threshold, etc.), a button 310 for obtaining the second description text of the image is displayed in the second image 306, as shown in FIG3F. The user can obtain the second description text of the second image 306 by clicking the button 310.

After the user clicks button 310, the second description text "little white rabbit, grass, streamside, clouds, grass, low saturation atmosphere, exquisite details, fine drawing, wide-angle lens, comic style" corresponding to the second image 306 is obtained and output to the text box 301, as shown in Figure 3G. The user can directly click the send button 302 to send the second description text as the first description text describing its image generation requirements to the AI image generation system. Alternatively, the user can also modify the second description text to obtain the first description text, as shown in Figure 3H.

It should be noted that the user's input data in the current round of dialogue does not necessarily always express image generation requirements, but may also express other requirements, such as chatting, asking how to use various functions of the AI image generation system, etc. It can be understood that only when the user's input data expresses image generation requirements, the input data belongs to the first description data describing the first image to be generated. If the user's input data expresses non-image generation requirements, the input data does not belong to the first description data.

According to some embodiments, the user's input data of the current round of dialogue may be used for intent recognition to determine whether the user currently has an image generation requirement, that is, to determine whether the current input data is first description data describing a first image to be generated.

Specifically, the user's input data in the current round of dialogue and the historical dialogue data in the historical round of dialogue (including historical user input and historical system response) can be obtained. Based on the input data and the historical dialogue data, the user's intention in the current round of dialogue is identified. In response to the intention, the image generation intention is generated, and the input data of the current round of dialogue is used as the first description data.

According to the above embodiments, the needs of users can be accurately identified, and different processing can be performed for different needs. For example, for image generation needs, targeted image generation processing can be performed; for other non-image generation needs, users can be replied to based on corresponding reply strategies. This can improve the pertinence and accuracy of image generation, avoid unnecessary image generation processing, and thus improve the user's image generation experience.

According to some embodiments, a trained large language model can be used to identify user intent. For example, the user's input data of the current round of dialogue and historical dialogue data can be filled into a preset prompt template to obtain query data. The query data is input into the large language model to obtain the intent recognition result output by the large language model.

According to some embodiments, when the user has non-image generation needs, the response data for the current round of dialogue can be generated based on the corresponding reply strategy. For example, the large language model can be trained (fine-tuned) using a preset question and answer template, and then the trained large language model can be used to generate answers to user questions and return them to the user. For another example, a help document for the AI image generation system can be pre-set. When a user asks how to use the AI image generation system, the help document can be retrieved to obtain answers to the user's questions and returned to the user.

FIG3I shows a schematic diagram of a dialogue interface for responding to a user's non-image generation needs. As shown in FIG3I , the user's input data in the current round of dialogue is "Who are you? What are your abilities?" The AI image generation system uses a large language model to generate an answer to this question: "I am an AI painting assistant. I can create works for you based on your input, modify and redraw according to your suggestions, etc. What kind of work do you want to draw? Please tell me. If you don't know what to draw, I can also provide you with inspiration~", which is output to the user as the response data of the current round of dialogue.

After determining that the user's intention in the current round of conversation is the intention to generate an image through intention recognition, the input data of the current round of conversation can be used as first description data describing the first image that the user expects to generate, and by executing step S220, a first image is generated based on the first description data. There can be multiple first images.

According to some embodiments, step S220 may include: determining an image generation model for generating the first image based on the data modality of the first description data; and inputting the first description data into the image generation model to obtain the first image output by the image generation model.

The image generation model includes a text-to-image model (the input data of the model is text), an image-to-image model (the input data of the model is an image), a text-to-image mixed image model (the input data of the model is text and an image), etc. In step S220, an image generation model whose input data has the same modality as the first description data is selected to generate the first image.

According to the above embodiments, the image generation capability of the image generation model can be made consistent with the needs of the user, thereby ensuring the accuracy of image generation.

According to some embodiments, the first description data includes a first description text. Accordingly, step S220 may include steps S221 and S222.

In step S221, the first description text is rewritten to generate second description data. The second description data includes the rewritten first description text.

In step S222, a first image is generated based on the second description data.

According to the above embodiment, the user only needs to express his or her image generation needs in colloquial language without describing the complete image details. The AI image generation system can intelligently rewrite the user's natural language instructions, fully understand and refine the user's image generation needs, and generate a richer and more diverse first image that meets the user's needs, thereby improving the accuracy of image generation.

According to some embodiments, step S221 may further include steps S2211 - S2213 .

In step S2211, the clarity of the first description text is determined.

In step S2212, a rewriting strategy for the first description text is determined based on the clarity of the first description text.

In step S2213, the first description text is rewritten using the above rewriting strategy to obtain a rewritten first description text.

According to the above embodiment, the user only needs to express his or her image generation needs in colloquial language without describing the complete image details. The AI image generation system can make targeted rewrites of different degrees of first description texts with different degrees of clarity, accurately understand the user's image generation needs, and generate a variety of first images that meet the user's needs for the user, thereby improving the accuracy of image generation.

According to some embodiments, in step S2211, the clarity of the first description text may be determined using a trained AI model. The AI model may be, for example, a binary classification model (the model determines whether the first description text is clear), a regression model (outputting a probability value that the first description text is clear), a large language model, etc.

Different clarity corresponds to different rewriting strategies. The rewriting strategy can be implemented as a trained AI rewriting model. Different AI rewriting models can rewrite the first description text to different degrees. According to some embodiments, the correspondence between clarity and AI rewriting model can be pre-configured, and the first description text is rewritten using the AI rewriting model corresponding to the clarity of the first description text to obtain the rewritten first description text.

For example, when the first description text is clear, the first rewriting model can be used to rewrite the first description text in terms of style and modifiers. For example, the first description text "Draw a cat" contains a clear subject (cat). The first rewriting model can be used to rewrite it, and the resulting text can be "a cute British shorthair cat rolling on the ground", "a tabby cat standing on the table in a daze", "a cute little white cat sleeping soundly on the carpet", etc. When the first description text is unclear, the second rewriting model can be used to rewrite the first description text in depth and intensification, such as adding a subject, describing a concrete picture, etc. For example, the first description text "Draw a painting where a bosom friend is hard to find" describes abstract content. The second rewriting model can be used to rewrite it in depth, and the resulting text can be "a bosom friend is hard to find, a scholar plays the piano alone, with only the cold moonlight and the lonely mountain in the distance in front of him, in the style of oil painting", "the center of the picture is two ancient people with fluttering sleeves, they sit opposite each other, playing the piano and talking, surrounded by green mountains and green waters, in the style of ink painting", etc.

According to some embodiments, in step S221, the first description text may be rewritten using the trained language model, thereby making it possible to use the language understanding and generation capabilities of the large language model to achieve intelligent rewriting, deeply understand the user's image generation needs, and make the rewritten text consistent with the user's needs, thereby improving the rewriting efficiency and accuracy.

According to some embodiments, in step S222, an image generation model for generating the first image can be determined based on the data modality of the second description data. Subsequently, the second description data is input into the image generation model to obtain the first image output by the image generation model. According to this embodiment, the image generation capability of the image generation model can be made consistent with the needs of the user, thereby ensuring the accuracy of image generation. Moreover, since the second description data is the description data after rewriting and optimization, the first image generated based on the second description data is more in line with the needs of the user than the first image generated based on the first description data, thereby improving the accuracy of image generation.

According to some embodiments, after the first image is generated by step S220, in step S230, the first image and at least one control for the first image are output as response data for the current round of dialogue. At least one control corresponds to at least one action for the first image. Each control is configured to perform a corresponding action on the first image in response to a user's interactive operation on the control (e.g., click, drag, mouse pointer stay, etc.). Thus, the user can further process the first image by operating the control, which greatly reduces the complexity of the operation, improves the efficiency of image generation, and thus improves the user experience.

According to some embodiments, the frequency of users' use of various functions of the AI image generation system can be counted, and the functions with higher frequency of use can be configured as controls to facilitate user operation, thereby reducing operation complexity and improving image generation efficiency.

According to some embodiments, the at least one control mentioned above may include at least one of: a first control for regenerating the first image; a second control for referencing the first image; a third control for managing the first image; or a fourth control for collecting user feedback data on the first image.

According to the above embodiments, by providing users with different types of controls, the users' high-frequency and diversified image processing needs can be met without the users having to repeatedly input natural language instructions, which greatly reduces the complexity of operations, improves image generation efficiency, and thus improves user experience.

Figures 3J and 3K show a dialog interface showing the first image and corresponding controls. It should be understood that the controls shown in Figures 3J-3M are only examples. The number, function, type (such as buttons, labels, sliders, etc.), and layout of the controls can be set according to actual needs.

As shown in Figure 3J, the AI image generation system generates four first images 311-314 for the user, and controls 315-318 for the first images 311-314 are displayed in the dialogue interface. Among them, control 315 is a reference button for referencing the first images 311-314. Control 316 is a regenerate button for regenerating the first image. Controls 317 and 318 are used to collect user feedback data on the first image. Specifically, control 317 is a like button for collecting positive feedback data from the user on the first image; control 318 is a dislike button for collecting negative feedback data from the user on the first image. Users can trigger corresponding actions (reference, regenerate, like, dislike) by clicking controls 315-318.

According to some embodiments, when a user hovers the mouse pointer over a first image, one or more controls for managing the first image may pop up on the first image. As shown in FIG3K , when a user hovers the mouse pointer over a first image 311, controls 319-321 pop up on the first image 311. Among them, control 319 is used to save the first image 311 online, control 320 is used to download the first image 311 locally, and control 321 is used to share the first image 311. By clicking controls 319-321, the user can trigger corresponding actions (save, download, share).

According to some embodiments, method 200 further includes steps S240 - S270 .

In step S240, in response to the user's operation on the second control, the first image is referenced in the next round of dialogue.

In step S250, a third description text input by the user in the next round of dialogue is obtained, wherein the third description text is used to describe the user's editing requirements for the first image.

In step S260, the first image and the third description text are used as third description data for describing a fourth image to be generated in the next round of dialogue.

In step S270 , a fourth image is generated based on the third description data.

According to the above embodiment, the first image to be edited can be accurately located by referencing the control (i.e., the second control), and the editing requirements for the image to be edited can be described through natural language instructions (i.e., the third description text), thereby enabling image creation and further editing of the created image in the same dialogue interface, thereby achieving efficient and smooth image generation.

According to some embodiments, in step S270, the third description text may be rewritten to generate fourth description data. The fourth description data includes the first image and the rewritten third description text. Subsequently, a fourth image is generated based on the fourth description data.

The specific implementation of step S270 is similar to that of the above step S220 and will not be repeated here.

3L and 3M illustrate the reference editing function in the AI image generation system. As shown in FIG3L , a reference control 315 is provided below the first image 311-314. By clicking the control 315, the user can reference the first image 311-314, and fill the referenced content into the input box 301 in the form of a bubble 322. The user can further enter a natural language instruction (i.e., a third description text) in the input box 301 "Change the rabbit in the second picture to gray". The user can send the natural language instruction and the referenced content to the AI image generation system together by clicking the send button. As shown in FIG3M , the referenced content is displayed below the user's natural language instruction in the form of a bubble 323. Bubble 322 indicates that the user has referenced a historically generated image. The user interacts with bubble 322 (e.g., clicks, hovers the mouse pointer, etc.), and a thumbnail of the referenced image will pop up above bubble 322.

According to some embodiments, after the first image and at least one control are outputted in step S230, guidance information for guiding the user to conduct the next round of dialogue may be generated and outputted based on the first image. According to this embodiment, after the image generation of the current round is completed, guidance information is further outputted to provide guidance for the user's next operation, thereby reducing the user's operation complexity and improving the efficiency of image generation.

According to some embodiments, the guidance information may describe feasible operations on the first image, such as "change to ink painting effect", "the first one is good, draw some more", etc. Such guidance information may be selected from a plurality of preset guidance information templates.

According to some other embodiments, the guidance information may also be a recommended description text generated based on the first description text of the first image. For example, the user generates the first image through the first description text "Help me draw a big white rabbit on the grass beside the stream". By modifying the first description text, a new description text "Help me draw a little rabbit, on the grass, beside the stream" may be generated. This type of guidance information may be generated, for example, by a large language model.

According to some embodiments, the guidance information may be output in the form of a control. By clicking the control, the user can fill the guidance information into the input box of the dialogue interface. The user may directly use the guidance information as input data for the next round of dialogue, or may modify the guidance information and use the modified data as input data for the next round of dialogue.

3N and 3O are schematic diagrams showing guidance information according to an embodiment of the present disclosure.

FIG3N shows four pieces of guidance information 324 in the form of bubble controls, namely, “Change to ink painting effect”, “The first one is good, draw a few more”, “Regenerate a few for me” and “Add "detailed depiction, delicate texture"”. These guidance information are used to describe the feasible operations on the first images 311-314. By clicking on the guidance information "Regenerate a few for me", the user can fill the guidance information into the input box 301 as input data for the next round of dialogue. The user can obtain new guidance information by clicking on the "Change a batch" label in the dialogue interface.

FIG3O shows three pieces of guidance information 325 in the form of bubble controls, namely, “Help me draw a little rabbit, grass, streamside”, “Help me draw a little rabbit, streamside, exquisite details” and “Help me draw a little rabbit, grass, streamside, exquisite details, fine drawing, wide-angle lens”. These guidance information are description texts recommended by the system. The user can obtain new guidance information by clicking the “Change a batch” label in the dialogue interface.

According to some embodiments, after the first image is output, the user's behavior data may be obtained. Based on the user's behavior data, the user's satisfaction with the first image is determined. In response to the user being dissatisfied with the first image, a third image to be recommended to the user is determined based on the first description data, and the third image is output through the dialogue interface. In this way, when the user has a negative behavior, relevant images can be actively recommended to the user, providing the user with creative inspiration, thereby improving the efficiency of image generation.

According to some embodiments, the user's behavior data includes the user's operation data on at least one of the above controls and/or the user's input data in the next round of dialogue. The user's behavior data can reflect the user's satisfaction with the first image. For example, the user clicks the like, save, download, share and other buttons, and enters positive feedback data "Great, thanks~" in the next round of dialogue, which means that the user is satisfied with the first image. The user clicks the thumbs-down button and enters negative feedback data "The generation effect is not good, I am not satisfied" in the next round of dialogue, which means that the user is dissatisfied with the first image.

According to some embodiments, multiple behavior data of users can be obtained and represented as feature vectors. The feature vectors are input into a trained satisfaction analysis model to obtain satisfaction output by the model. The value range of satisfaction can be, for example, 0 to 1.

According to some embodiments, when the user is not satisfied with the first image, an image similar to the first description data input by the user, that is, an image meeting the user's needs, may be searched in the image library as a third image to be recommended to the user.

According to some embodiments, after determining the user's satisfaction with the first image, the image generation model that generates the first image may be optimized based on the first image and the satisfaction, thereby improving the image generation effect of the image generation model.

For example, the first image with higher satisfaction can be used as a positive sample, and the first image with lower satisfaction can be used as a negative sample to fine-tune the image generation model, thereby continuously improving the text/image understanding and image generation capabilities of the image generation model, making the generated image consistent with user needs, thereby improving the accuracy of image generation.

According to some embodiments, as shown in FIG3P , the dialogue interface between the user and the AI image generation system may be divided into a historical dialogue record area 326 and a dialogue area 328. The historical dialogue record area 326 is used to display historical dialogues and manage (e.g., delete) historical dialogues. The historical dialogue record area 326 may include a control 327 for starting a new dialogue. By clicking on the control 327, the user can end the current dialogue and start a new dialogue. For example, a historical dialogue may be named after the data that the user first entered in the dialogue. Multiple historical dialogues can be sorted according to the time when the user first sent the data. The closer the time when the user first sent the data is to the current time, the higher the sorting.

According to the embodiments of the present disclosure, a conversational AI painting assistant is provided for users, which provides users with immersive image generation services in a conversation interface, understands the content of user vernacular input, clarifies user needs in multiple rounds of conversations, and generates and modifies pictures for users. The embodiments of the present disclosure innovatively integrate the language interaction form and the graphical interface form. While operating based on conversation instructions, the user's high-frequency needs are designed as quick click buttons, such as regeneration, clicking to view detailed content, feedback (likes, thumbs down), etc., which greatly simplifies user operations and helps users to immersively create with low thresholds, edit smoothly, and efficiently generate satisfactory content. In addition, the embodiments of the present disclosure can actively provide users with guidance on creative inspiration and creative ability. At the same time, it supports the automatic accumulation of feedback data through user conversations and click behaviors, and rapid feedback of algorithm model iterations.

The embodiments of the present disclosure make up for the high threshold for use and long creation process of existing painting products, and provide a convenient solution for users who have no experience in using AI raw image products. The embodiments of the present disclosure mainly have the following advantages:

1. Lowering the threshold for creation: The system can understand the vernacular content input by users and clarify user needs through multiple rounds of dialogue. Users can generate satisfactory works without having to modify prompt words multiple times, or complicated parameter settings and function selections.

2. Shorten the creation link: Using text-controlled editing technology (using natural language instructions as control instructions to modify image content), it integrates a variety of editing functions, including image segmentation, local image editing (inpainting), image expansion (outpainting), style conversion (such as comic style, oil painting style, ink painting style, etc.), size modification, adding, modifying, and deleting elements in a specified position or area. Continuous editing can be completed on the same page according to user instructions, without having to cross multiple products and functions. At the same time, it can intelligently recommend works that users may be interested in based on user feedback, provide creation guidance, and help users improve their creation efficiency.

3. Improved feedback collection efficiency: Feedback data is automatically accumulated through feedback expressions in user conversations and clicks on shortcut buttons. There is no need to guide users to actively give feedback. At the same time, the feedback collection capability is intelligently activated based on user behavior, which can greatly increase the amount of feedback data without disturbing the user experience.

The embodiments of the present disclosure can be applied to fields such as AI painting, graphic design, film and television post-production, image editing, and video editing.

According to an embodiment of the present disclosure, an image generating device is also provided. Fig. 4 shows a structural block diagram of an image generating device 400 according to an embodiment of the present disclosure. As shown in Fig. 4, the device 400 includes a first acquisition module 410, a first generation module 420 and a first output module 430.

A first acquisition module is configured to acquire first description data input by the user in a current round of dialogue, wherein the first description data is used to describe a first image to be generated;

A first generating module, configured to generate the first image based on the first description data; and

The first output module is configured to output the first image and at least one control for the first image as response data for the current round of dialogue, wherein the at least one control corresponds to at least one action for the first image, and any one of the at least one control is configured to perform a corresponding action on the first image in response to the user's operation on the control.

According to an embodiment of the present disclosure, a conversational image generation solution integrating a graphical interface is provided. Users can complete image generation by interacting with the AI image generation system through a conversation and a graphical interface, which greatly reduces the complexity of operation and improves the efficiency of image generation.

In each round of dialogue, based on the first description data input by the user, the user's image generation requirements are understood and a first image is generated, and the generated first image and at least one control are returned to the user as the response data of the current round of dialogue. The user can further process the first image by operating the control, so that the user can complete continuous image generation in the same dialogue interface, which simplifies the operation, improves the image generation efficiency, and thus improves the user experience.

According to some embodiments, the first acquisition module includes: an acquisition unit, configured to acquire the input data of the user in the current round of conversation and the historical conversation data in the historical round of conversation; an identification unit, configured to identify the intention of the user in the current round of conversation based on the input data and the historical conversation data; and a first determination unit, configured to use the input data as the first description data in response to the intention being an image generation intention.

According to some embodiments, the first generation module includes: a second determination unit, configured to determine an image generation model for generating the first image based on a data modality of the first description data; and a first input unit, configured to input the first description data into the image generation model to obtain the first image output by the image generation model.

According to some embodiments, the first description data includes a first description text, and wherein the first generation module includes: a rewriting unit configured to rewrite the first description text to generate second description data, wherein the second description data includes the rewritten first description text; and a generation unit configured to generate the first image based on the second description data.

According to some embodiments, the rewriting unit includes: a first determination subunit, configured to determine the clarity of the first description text; a second determination subunit, configured to determine a rewriting strategy for the first description text based on the clarity; and a rewriting subunit, configured to rewrite the first description text using the rewriting strategy.

According to some embodiments, the rewriting unit is further configured to: rewrite the first description text using a trained language model.

According to some embodiments, the generation unit includes: a third determination subunit, configured to determine an image generation model for generating the first image based on a data modality of the second description data; and an input subunit, configured to input the second description data into the image generation model to obtain the first image output by the image generation model.

According to some embodiments, the first description data includes a first description text, and the device also includes: a recommendation module configured to recommend a second image to the user; and a second output module configured to output a second description text for describing the second image to guide the user to input the first description text.

According to some embodiments, the second output module is further configured to: output the second description text in response to the user's selection operation on the second image.

According to some embodiments, the apparatus 400 further includes: a second generating module configured to generate and output, based on the first image, guiding information for guiding the user to conduct a next round of dialogue.

According to some embodiments, the device 400 also includes: a first determination module, configured to determine the user's satisfaction with the first image based on the user's behavioral data; a second determination module, configured to determine a third image to be recommended to the user based on the first description data in response to the user being dissatisfied with the first image; and a third output module, configured to output the third image.

According to some embodiments, the device 400 also includes: a first determination module, configured to determine the user's satisfaction with the first image based on the user's behavioral data; and an optimization module, configured to optimize the image generation model that generates the first image based on the first image and the satisfaction.

According to some embodiments, the behavior data includes operation data of the user on the at least one control and/or input data of the user in the next round of dialogue.

According to some embodiments, the at least one control includes at least one of: a first control for regenerating the first image; a second control for referencing the first image; a third control for managing the first image; or a fourth control for collecting feedback data of the user on the first image.

According to some embodiments, the device 400 also includes: a reference module, configured to reference the first image in the next round of conversation in response to the user's operation on the second control; a second acquisition module, configured to acquire a third description text input by the user in the next round of conversation, wherein the third description text is used to describe the user's editing requirements for the first image; a third determination module, configured to use the first image and the third description text as third description data for describing a fourth image to be generated in the next round of conversation; and a third generation module, configured to generate the fourth image based on the third description data.

It should be understood that the various modules and units of the device 400 shown in FIG4 may correspond to the various steps in the method 200 described with reference to FIG2. Thus, the operations, features and advantages described above for the method 200 are also applicable to the device 400 and the modules and units included therein. For the sake of brevity, some operations, features and advantages are not described in detail herein.

Although specific functionality is discussed above with reference to specific modules, it should be noted that the functionality of the various modules discussed herein may be separated into multiple modules, and/or at least some functionality of multiple modules may be combined into a single module.

It should also be understood that various technologies can be described herein in the general context of software hardware elements or program modules. The various units described above with respect to Figure 4 can be implemented in hardware or in hardware in combination with software and/or firmware. For example, these units can be implemented as computer program codes/instructions, which are configured to be executed in one or more processors and stored in a computer-readable storage medium. Alternatively, these units can be implemented as hardware logic/circuits. For example, in some embodiments, one or more of modules 410-430 can be implemented together in a system on chip (System on Chip, SoC). SoC can include an integrated circuit chip (which includes a processor (e.g., a central processing unit (Central Processing Unit, CPU), a microcontroller, a microprocessor, a digital signal processor (Digital Signal Processor, DSP) etc.), a memory, one or more communication interfaces, and/or one or more components in other circuits), and can optionally execute the received program code and/or include embedded firmware to perform functions.

According to an embodiment of the present disclosure, there is also provided an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor, the memory storing instructions executable by the at least one processor, the instructions being executed by the at least one processor so that the at least one processor can execute the image generation method of the embodiment of the present disclosure.

According to an embodiment of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is further provided. The computer instructions are used to enable a computer to execute the image generating method of the embodiment of the present disclosure.

According to an embodiment of the present disclosure, a computer program product is further provided, including computer program instructions, which implement the image generating method of the embodiment of the present disclosure when executed by a processor.

With reference to Figure 5, a block diagram of an electronic device 500 that can be used as a server or client of the present disclosure will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples, and are not intended to limit the implementation of the present disclosure described and/or required herein.

As shown in Figure 5, electronic device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 into a random access memory (RAM) 503. In RAM 503, various programs and data required for the operation of electronic device 500 can also be stored. Computing unit 501, ROM 502 and RAM 503 are connected to each other via bus 504. Input/output (I/O) interface 505 is also connected to bus 504.

Multiple components in the electronic device 500 are connected to the I/O interface 505, including: an input unit 506, an output unit 507, a storage unit 508, and a communication unit 509. The input unit 506 can be any type of device that can input information to the electronic device 500. The input unit 506 can receive input digital or character information and generate key signal input related to user settings and/or function control of the electronic device, and can include but is not limited to a mouse, a keyboard, a touch screen, a track pad, a track ball, a joystick, a microphone, and/or a remote controller. The output unit 507 can be any type of device that can present information, and can include but is not limited to a display, a speaker, a video/audio output terminal, a vibrator, and/or a printer. The storage unit 508 can include but is not limited to a disk, an optical disk. The communication unit 509 allows the electronic device 500 to exchange information/data with other devices through a computer network such as the Internet and/or various telecommunication networks, and can include but is not limited to a modem, a network card, an infrared communication device, a wireless communication transceiver, and/or a chipset, such as a Bluetooth device, an 802.11 device, a Wi-Fi device, a WiMAX device, a cellular communication device, and/or the like.

The computing unit 501 may be a variety of general and/or special processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 501 performs the various methods and processes described above, such as method 200. For example, in some embodiments, the method 200 may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 508. In some embodiments, part or all of the computer program may be loaded and/or installed on the electronic device 500 via the ROM 502 and/or the communication unit 509. When the computer program is loaded into the RAM 503 and executed by the computing unit 501, one or more steps of the method 200 described above may be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to perform the method 200 in any other appropriate manner (e.g., by means of firmware).

Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and/or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, implements the functions/operations specified in the flow chart and/or block diagram. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document is not limited here.

Although the embodiments or examples of the present disclosure have been described with reference to the accompanying drawings, it should be understood that the above-mentioned methods, systems and devices are merely exemplary embodiments or examples, and the scope of the present disclosure is not limited by these embodiments or examples, but is only limited by the claims after authorization and their equivalent scope. Various elements in the embodiments or examples may be omitted or replaced by their equivalent elements. In addition, each step may be performed in an order different from that described in the present disclosure. Further, the various elements in the embodiments or examples may be combined in various ways. It is important that with the evolution of technology, many of the elements described herein may be replaced by equivalent elements that appear after the present disclosure.

Claims (33)

1. A method for generating an image, comprising: Acquire first description data input by the user in the current round of dialogue, wherein the first description data is used to describe a first image to be generated; generating the first image based on the first description data; and The first image and at least one control for the first image are output as response data for the current round of dialogue, wherein the at least one control corresponds to at least one action for the first image, and any one of the at least one control is configured to perform a corresponding action on the first image in response to the user's operation on the control. 2. The method according to claim 1, wherein the step of obtaining the first description data input by the user in the current round of dialogue comprises: Acquire input data of the user in the current round of dialogue and historical dialogue data in historical rounds of dialogue; Based on the input data and the historical conversation data, identifying the user's intention in the current round of conversation; and In response to the intention being an image generation intention, the input data is used as the first description data. 3. The method according to claim 1 or 2, wherein generating the first image based on the first description data comprises: determining, based on the data modality of the first description data, an image generation model for generating the first image; and The first description data is input into the image generation model to obtain the first image output by the image generation model. 4. The method according to claim 1 or 2, wherein the first description data comprises a first description text, and wherein the generating the first image based on the first description data comprises: rewriting the first description text to generate second description data, wherein the second description data includes the rewritten first description text; and Based on the second description data, the first image is generated. 5. The method according to claim 4, wherein rewriting the first description text comprises: determining the clarity of the first description text; Based on the clarity, determining a rewriting strategy for the first description text; and The first description text is rewritten using the rewriting strategy. 6. The method according to claim 4 or 5, wherein rewriting the first description text comprises: The first description text is rewritten using the trained language model. 7. The method according to any one of claims 4 to 6, wherein generating the first image based on the second description data comprises: determining an image generation model for generating the first image based on the data modality of the second description data; and The second description data is input into the image generation model to obtain the first image output by the image generation model. 8. The method according to any one of claims 1 to 7, wherein the first description data comprises a first description text, and the method further comprises: recommending a second image to the user; and A second description text for describing the second image is output to guide the user to input the first description text. 9. The method according to claim 8, wherein the outputting a second description text for describing the second image comprises: In response to the user's selection operation on the second image, the second description text is output. 10. The method according to any one of claims 1 to 9, further comprising: Based on the first image, guidance information for guiding the user to conduct a next round of dialogue is generated and output. 11. The method according to any one of claims 1 to 10, further comprising: determining, based on the user's behavior data, the user's satisfaction with the first image; In response to the user being dissatisfied with the first image, determining a third image to be recommended to the user based on the first description data; and The third image is output. 12. The method according to any one of claims 1 to 10, further comprising: determining the user's satisfaction with the first image based on the user's behavior data; and An image generation model for generating the first image is optimized based on the first image and the satisfaction level. 13. The method according to claim 11 or 12, wherein the behavior data includes the user's operation data on the at least one control and/or the user's input data in the next round of dialogue. 14. The method according to any one of claims 1 to 13, wherein the at least one control comprises at least one of the following: a first control for regenerating the first image; a second control for referencing the first image; a third control for managing the first image; or A fourth control is used to collect feedback data of the user on the first image. 15. The method according to claim 14, further comprising: In response to the user's operation on the second control, referencing the first image in a next round of dialogue; Acquire a third description text input by the user in the next round of dialogue, wherein the third description text is used to describe the user's editing requirement for the first image; using the first image and the third description text as third description data for describing a fourth image to be generated in the next round of dialogue; and Based on the third description data, the fourth image is generated. 16. An image generating device, comprising: A first acquisition module is configured to acquire first description data input by the user in a current round of dialogue, wherein the first description data is used to describe a first image to be generated; A first generating module, configured to generate the first image based on the first description data; and The first output module is configured to output the first image and at least one control for the first image as response data for the current round of dialogue, wherein the at least one control corresponds to at least one action for the first image, and any one of the at least one control is configured to perform a corresponding action on the first image in response to the user's operation on the control. 17. The device according to claim 16, wherein the first acquisition module comprises: An acquisition unit, configured to acquire input data of the user in the current round of dialogue and historical dialogue data in historical rounds of dialogue; an identification unit configured to identify the user's intention in the current round of dialogue based on the input data and the historical dialogue data; and The first determining unit is configured to use the input data as the first description data in response to the intention being an image generation intention. 18. The apparatus according to claim 16 or 17, wherein the first generating module comprises: a second determining unit configured to determine an image generation model for generating the first image based on a data modality of the first description data; and The first input unit is configured to input the first description data into the image generation model to obtain the first image output by the image generation model. 19. The apparatus according to claim 16 or 17, wherein the first description data comprises a first description text, and wherein the first generation module comprises: a rewriting unit configured to rewrite the first description text to generate second description data, wherein the second description data includes the rewritten first description text; and The generating unit is configured to generate the first image based on the second description data. 20. The device according to claim 19, wherein the rewriting unit comprises: A first determining subunit, configured to determine the clarity of the first description text; A second determining subunit is configured to determine a rewriting strategy for the first description text based on the clarity; and The rewriting subunit is configured to rewrite the first description text using the rewriting strategy. 21. The apparatus according to claim 19 or 20, wherein the rewriting unit is further configured to: The first description text is rewritten using the trained language model. 22. The apparatus according to any one of claims 19 to 21, wherein the generating unit comprises: a third determining subunit, configured to determine an image generation model for generating the first image based on the data modality of the second description data; and The input subunit is configured to input the second description data into the image generation model to obtain the first image output by the image generation model. 23. The apparatus according to any one of claims 16 to 22, wherein the first description data comprises a first description text, and the apparatus further comprises: a recommendation module configured to recommend a second image to the user; and The second output module is configured to output a second description text for describing the second image to guide the user to input the first description text. 24. The apparatus according to claim 23, wherein the second output module is further configured to: In response to the user's selection operation on the second image, the second description text is output. 25. The apparatus according to any one of claims 16 to 24, further comprising: The second generating module is configured to generate and output guiding information for guiding the user to have a next round of dialogue based on the first image. 26. The apparatus according to any one of claims 16 to 25, further comprising: A first determination module is configured to determine the user's satisfaction with the first image based on the user's behavior data; a second determining module, configured to determine, in response to the user being dissatisfied with the first image, a third image to be recommended to the user based on the first description data; and The third output module is configured to output the third image. 27. The apparatus according to any one of claims 16 to 25, further comprising: A first determination module is configured to determine the user's satisfaction with the first image based on the user's behavior data; and An optimization module is configured to optimize an image generation model for generating the first image based on the first image and the satisfaction level. 28. The device according to claim 26 or 27, wherein the behavior data includes the user's operation data on the at least one control and/or the user's input data in the next round of dialogue. 29. The apparatus of any one of claims 16 to 28, wherein the at least one control comprises at least one of the following: a first control for regenerating the first image; a second control for referencing the first image; a third control for managing the first image; or A fourth control is used to collect feedback data of the user on the first image. 30. The apparatus of claim 29, further comprising: a reference module, configured to reference the first image in a next round of dialogue in response to the user's operation on the second control; a second acquisition module, configured to acquire a third description text input by the user in the next round of dialogue, wherein the third description text is used to describe the user's editing requirements for the first image; a third determining module, configured to use the first image and the third description text as third description data for describing a fourth image to be generated in the next round of dialogue; and The third generating module is configured to generate the fourth image based on the third description data. 31. An electronic device comprising: at least one processor; and a memory communicatively coupled to the at least one processor; wherein The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 15. 32. A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the method according to any one of claims 1-15. 33. A computer program product comprising computer program instructions, wherein the computer program instructions, when executed by a processor, implement the method of any one of claims 1-15.