WO2024138720A1 - Image generation method and apparatus, and computer device and storage medium - Google Patents

Abstract

Disclosed in the embodiments of the present invention are an image generation method and apparatus, and a computer device and a storage medium. The method may comprise: receiving an image generation instruction, wherein the image generation instruction comprises image attribute information of a target image which is required to be generated; acquiring environment information, wherein the environment information is collected by means of a display device which displays the target image; performing data mapping on the environment information by means of a preset image generation model, so as to generate a content image corresponding to the environment information; and extracting a style feature from at least one preset style reference image, and extracting a content feature from the content image, so as to obtain a target image on the basis of the fusion of the content feature and the style feature. In the embodiments of the present invention, an image generation model may perform mapping according to environment information to obtain a content image, and a content feature in the content image can be fused with a style feature in a style reference image, such that a target image is finally generated, and thus various styles of wallpaper images with rich content can be generated, thereby improving the visual experience of a user.

Description

Image generation method, device, computer equipment and storage medium Technical Field

The present invention relates to the technical field of image processing, and in particular to an image generating method, device, computer equipment and storage medium.

Background technique

With the rapid development of the current economy, various display devices are increasingly widely used in people's lives. Generally, in order to enrich the visual experience of users, display devices can display some wallpaper images with different contents and styles by default.

At present, the main method used to display wallpaper images is that the technicians pre-set a set of wallpaper images, and when a wallpaper image needs to be displayed, one is randomly selected from the set for display. However, this solution cannot guarantee that the content of the displayed wallpaper image fits the current environment, and the number of wallpaper images is limited, so there may be uneven image quality, which affects the user's visual experience.

technical problem

The prior art cannot ensure that the content of the displayed wallpaper image fits the current environment, and the number of wallpaper images is also limited, and there may be problems of uneven image quality, which affects the user's visual experience.

Technical Solutions

The embodiments of the present invention provide an image generation method, apparatus, computer equipment and storage medium, which can generate wallpaper images with rich content and diverse styles, so that the content of the wallpaper image fits the real environment and enhances the user's visual experience.

An embodiment of the present invention provides an image generation method, comprising:

receiving an image generation instruction, wherein the image generation instruction includes image attribute information of a target image to be generated;

Acquiring environmental information, wherein the environmental information is collected by a display device that displays the target image;

Performing data mapping on the environmental information through a preset image generation model to generate a content image corresponding to the environmental information;

Style features are extracted from at least one preset style reference image, and content features are extracted from the content image, and a target image is obtained based on the fusion of the content features and the style features.

Accordingly, an embodiment of the present invention further provides an image generating device, comprising:

An instruction receiving unit, used for receiving an image generation instruction, wherein the image generation instruction includes image attribute information of a target image to be generated;

An environmental information acquisition unit, used to acquire environmental information, wherein the environmental information is collected by a display device that displays the target image;

A content image generating unit, configured to perform data mapping on the environmental information through a preset image generating model, and generate a content image corresponding to the environmental information;

The target image generation unit is used to extract style features from at least one preset style reference image and content features from the content image, and obtain the target image based on the fusion of the content features and the style features.

Optionally, the image generation device provided by the embodiment of the present invention further includes an image generation model training unit, which is used to obtain a generative adversarial model to be trained, wherein the generative adversarial model includes an image generation model to be trained and a discriminant model to be trained, and the generative adversarial model is provided with at least one real content sample image;

Performing data mapping on the training input parameters through the image generation model to be trained to generate a training content image corresponding to the training input parameters;

Calculating the authenticity parameter of the training content image relative to the real content sample image by using the discriminant model to be trained;

Adjusting the image generation model to be trained and the discriminant model to be trained based on the truthfulness parameter;

Return to execute the step of performing data mapping on the training input parameters through the image generation model to be trained until a preset training end condition is met, thereby obtaining a trained image generation model.

Optionally, the image generation apparatus provided by the embodiment of the present invention further includes an image generation model setting unit, which is used to obtain display parameters of the display device;

Based on the display parameters, the model parameters of the image generation model to be trained are set to obtain a new image generation model to be trained.

Optionally, a target image generating unit is used to extract style features from at least one preset style reference image according to style feature mapping parameters of a style extraction layer in a style transfer model to obtain style features corresponding to the style reference image;

According to the content feature mapping parameters of the content extraction layer in the style transfer model, content features are extracted from the content image to obtain content features corresponding to the content image.

Optionally, the image generation device provided by the embodiment of the present invention further includes a style transfer model training unit, which is used to extract style features of the sample style reference image through the style transfer model to be trained, so as to obtain sample style features corresponding to the sample style reference image;

Extracting content features of the sample content image by using the style transfer model to be trained to obtain sample content features corresponding to the sample content image;

Performing feature fusion on the sample style feature and the sample content feature to obtain a sample style transfer image;

Calculating a model loss of the style transfer model to be trained based on the sample style transfer image, the sample style reference image and the sample content image;

According to the model loss, the model parameters of the style transfer model to be trained are adjusted to obtain a trained style transfer model.

Optionally, the environment information acquisition unit is used to collect environment sound audio through a display device;

The environmental sound audio is converted into a model input format of the image generation model to obtain environmental information.

Optionally, the target image generation unit is used to select at least one target style reference image corresponding to the image style type from at least one preset style reference image;

Extracting style features for each of the target style reference images, and extracting content features from the content images;

The content features and the style features are weightedly fused to obtain a target image.

Correspondingly, an embodiment of the present invention further provides a computer device, comprising a memory and a processor; the memory stores an application program, and the processor is used to run the application program in the memory to execute the steps in any one of the image generation methods provided in the embodiments of the present invention.

Correspondingly, an embodiment of the present invention further provides a computer-readable storage medium, which stores a plurality of instructions, and the instructions are suitable for a processor to load to execute the steps in any one of the image generation methods provided in the embodiments of the present invention.

In addition, an embodiment of the present invention further provides a computer program product, including a computer program or instructions, which, when executed by a processor, implements the steps in any one of the image generating methods provided by the embodiments of the present invention.

Beneficial Effects

Beneficial effect: Compared with the prior art, the present invention provides an image generation method, which can receive an image generation instruction, wherein the image generation instruction includes image attribute information of a target image to be generated, obtains environmental information, the environmental information is collected by a display device that displays the target image, performs data mapping on the environmental information through a preset image generation model, generates a content image corresponding to the environmental information, extracts style features from at least one preset style reference image, and extracts content features from the content image, and obtains the target image based on the fusion of the content features and the style features; in an embodiment of the present invention, the image generation model can obtain the content image by mapping according to the environmental information, therefore, it can be ensured that the content of the image fits the environment, and the content features in the content image can be fused with the style features in the style reference image to finally generate the target image, therefore, wallpaper images with rich content and diverse styles can be generated to enhance the user's visual experience.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

FIG1 is a schematic diagram of a scene of an image generation method provided by an embodiment of the present invention;

FIG2 is a flow chart of an image generating method provided by an embodiment of the present invention;

FIG3 is a schematic diagram of a technical implementation of generating a target image provided by an embodiment of the present invention;

FIG4 is a schematic diagram of an application on a smart TV provided by an embodiment of the present invention;

5 is a schematic diagram of the structure of an image generating device provided by an embodiment of the present invention;

FIG6 is another schematic diagram of the structure of an image generating device provided by an embodiment of the present invention;

FIG. 7 is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention.

Embodiments of the present invention

The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

The embodiments of the present invention provide an image generation method, an apparatus, a computer device and a computer-readable storage medium. Specifically, the embodiments of the present invention provide an image generation method applicable to an image generation apparatus, and the image generation apparatus can be integrated in a computer device.

The computer device may be a terminal or other device, including but not limited to a mobile terminal and a fixed terminal. For example, a mobile terminal includes but is not limited to a smart phone, a smart watch, a tablet computer, a laptop computer, a smart car-mounted terminal, etc., wherein a fixed terminal includes but is not limited to a desktop computer, a smart TV, etc.

The computer device may also be a server or other device. The server may be an independent physical server or a server cluster or distributed system composed of multiple physical servers. It may also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms, but is not limited to these.

The image generating method according to the embodiment of the present invention may be implemented by a server, or may be implemented by a terminal and a server together.

The method is described below by taking the example of a terminal and a server jointly implementing the image generation method.

As shown in FIG1 , the image generation system provided by the embodiment of the present invention includes a terminal 10 and a server 20, etc. The terminal 10 and the server 20 are connected via a network, such as a wired or wireless network connection, etc., wherein the terminal 10 can exist as a terminal for displaying a target image and sending an image generation instruction of the target image to the server 20.

The terminal 10 may be a terminal for a user to view a target image, and is used to send an image generation instruction for the target image to the server 20 .

The server 20 can be used to receive an image generation instruction, where the image generation instruction includes image attribute information of a target image to be generated, obtain environmental information, where the environmental information is collected through a display device that displays the target image, perform data mapping on the environmental information through a preset image generation model, generate a content image corresponding to the environmental information, extract style features from at least one preset style reference image, and extract content features from the content image, and obtain the target image based on a fusion of the content features and the style features.

The server 20 may send the target image to the terminal 10 for display.

In some optional embodiments, the step of generating the target image performed by the server 20 may also be performed by the terminal 10, which is not limited in the embodiment of the present invention.

It should be noted that the description order of the following embodiments is not intended to limit the preferred order of the embodiments.

The embodiments of the present invention will be described from the perspective of an image generating device, which may be specifically integrated in a server and/or a terminal.

As shown in FIG. 2 , the specific process of the image generation method of this embodiment may be as follows:

201. Receive an image generation instruction, where the image generation instruction includes image attribute information of a target image to be generated.

Specifically, the image generation instruction is an instruction for instructing to generate a target image. The image generation instruction may be generated based on a user's operation or may be actively generated by the display device.

For example, the image generation instruction may be generated according to the user's power-on operation, return to the desktop operation, etc.; or, the image generation instruction may be generated regularly or irregularly during the operation of the display device, and so on.

The image attribute information is information related to the target image itself, for example, the image attribute information may include but is not limited to the style type, resolution, category of image content, etc. of the target image.

202. Acquire environmental information, where the environmental information is collected through a display device that displays the target image.

Specifically, the environmental information may be obtained by collecting information about the environment in which the display device is located. For example, the environmental information may include, but is not limited to, information related to environmental brightness, information related to environmental sound, information related to environmental temperature, information related to environmental weather, and the like.

Taking the environmental information including the information related to the environmental sound as an example, the step of “obtaining the environmental information” may specifically include:

Collect environmental sound audio through display devices;

The environmental sound audio is converted into a model input format of the image generation model to obtain environmental information.

For example, the audio signal can be received by a microphone device on the smart TV to obtain ambient sound audio; the audio information (ambient sound audio) is preprocessed to obtain sound waves from the audio, and then numerical values such as amplitude and pulse are extracted from the sound waves and converted into a model input format of an image generation model, such as a one-dimensional array format, as environmental information for changing the image content in the target image.

The one-dimensional numerical value converted from audio information is used as input, and the generative network generates a real image with unique content that is indistinguishable from the real image. The style transfer network stylizes it into an artistic abstract image, and the stylized 16:9 artistic abstract image is displayed in full screen on the interface. Multi-threaded sound data is received, and wallpaper is continuously generated and then refreshed.

It is understandable that the environmental information may also include other information besides sound. For example, the step of “obtaining environmental information” may specifically include:

Collecting ambient sound audio through the display device to obtain ambient sound information;

Obtaining environmental weather information corresponding to the display device;

Fusing the ambient sound information and the ambient weather information to generate fused ambient information;

The fused environmental information is converted into a model input format of the image generation model to obtain environmental information.

Among them, a weighted calculation method can be used when integrating the ambient sound information and the ambient weather information. The weights of the ambient sound information and the ambient weather information can be the same or different, and technicians and users can set them according to actual display requirements.

203. Perform data mapping on the environmental information through a preset image generation model to generate a content image corresponding to the environmental information.

In the embodiment of the present invention, the image generation model is a generative model in GAN (Generative Adversarial Network). GAN is a related technology of deep learning. It uses the mutual game learning of the generative model and the discriminative model to generate photos that are almost indistinguishable from the actual photos.

However, in the field of GAN, there are problems such as high requirements on the quality and quantity of training data, difficulty in training, instability, and easy collapse of details. Artistic images often have abstract image content, a wide variety of image content, one style for each image, limited data volume, and uneven quality. The quality of directly training artistic images is poor.

Therefore, in the embodiment of the present invention, images with real content rather than artistic images are used to train GAN, which ensures the authenticity of the image content in the target image and avoids problems such as abstract image content. In addition, the quality of real images is easier to control and obtain than that of artistic images, which can improve the image quality of the target image.

In order to enable the image generation model to achieve better image generation performance, the parameters of the image generation model can be adjusted through the pre-training process. That is, before the step of "data mapping the environmental information through a preset image generation model to generate a content image corresponding to the environmental information", the image generation method provided by the embodiment of the present invention may also include:

Obtaining a generative adversarial model to be trained, wherein the generative adversarial model includes an image generation model to be trained and a discriminant model to be trained, and the generative adversarial model is provided with at least one real content sample image;

Performing data mapping on the training input parameters through the image generation model to be trained to generate a training content image corresponding to the training input parameters;

Calculating the authenticity parameter of the training content image relative to the real content sample image by using the discriminant model to be trained;

Adjusting the image generation model to be trained and the discriminant model to be trained based on the truthfulness parameter;

Return to execute the step of performing data mapping on the training input parameters through the image generation model to be trained until a preset training end condition is met, thereby obtaining a trained image generation model.

The image contents in the real content sample images, such as flowers, plants, people, etc., all exist in the real environment. The embodiment of the present invention does not limit the content type in the real content sample images.

Specifically, the authenticity parameter is used to describe the authenticity of the image content in the training content image, and the authenticity can be determined by the similarity between the training content image and the real content sample image. For example, the discriminant model can score the training content image and output a confidence level between 0 and 1 corresponding to the training content image.

In actual training, the optimization of the image generation model and the discriminant model may not be alternating one after another. Instead, the image generation model is trained once after training the discriminant model k times. This ensures that the image generation model changes slowly enough so that the discriminant model can always remain near its optimal solution.

Among them, the training end condition may be that the number of model adjustments to the image generation model reaches a preset adjustment number threshold, or that the authenticity parameter of the discriminant model output is greater than a preset threshold for N consecutive times, and so on.

In the process of pre-training the GAN network, as shown in Figure 3, a large number of real images with high-quality resolution (720p, 1080p, 2k) can be collected as training sets. The image content is atmospheric and beautiful, such as landscape pictures. A GAN network (such as the representative StyleGAN[1] and BigGAN[2]) is selected. The GAN network defaults to random one-dimensional input. This GAN network is trained and the model parameters are updated. The network continuously extracts detailed features of the image content until an image that is almost indistinguishable from the actual image is generated during testing.

It is understandable that the display parameters of different display devices may be different, and the model parameters of the image generation model may be adjusted in advance according to the display parameters. That is, before the step of "data mapping the training input parameters through the image generation model to be trained to generate the training content image corresponding to the training input parameters", the image generation method provided by the embodiment of the present invention may also include:

Acquire display parameters of the display device;

Based on the display parameters, the model parameters of the image generation model to be trained are set to obtain a new image generation model to be trained.

For example, the size of the convolution kernel in the model parameters of the image generation model to be trained can be 4*4, and the generated image ratio is 1:1. However, the display parameters of the display device may require a wallpaper image with a ratio of 16:9. In this case, the size of the convolution kernel can be changed to 3:5, and so on.

As shown in FIG. 4 , the image generation model can be set on a terminal such as a smart TV, and the smart TV can generate a target image in memory based on environmental disturbances (i.e., environmental information), an image generation model (GAN), etc. as an artistic wallpaper of the smart TV.

204. Extract style features from at least one preset style reference image, and extract content features from the content image, and obtain a target image based on a fusion of the content features and the style features.

Among them, style can be understood as the subjective feeling that an image brings to the viewer, and style can be reflected through the color, texture, etc. in the image. For example, images created by different people generally have different personal characteristics, and this personal characteristic shown through the image is style.

Optionally, the style transfer in the style reference image can be implemented based on deep learning technology. For example, a style transfer model can be constructed through deep learning technology to extract style features and content features, and then feature fusion can be performed to achieve style transfer. The step of "extracting style features from at least one preset style reference image, and extracting content features from the content image" can specifically include:

Extracting style features from at least one preset style reference image according to style feature mapping parameters of a style extraction layer in a style transfer model to obtain style features corresponding to the style reference image;

According to the content feature mapping parameters of the content extraction layer in the style transfer model, content features are extracted from the content image to obtain content features corresponding to the content image.

For example, the style extraction layer and the content extraction layer in the style transfer model can be models with one of the machine learning networks such as CNN, DNN, GAN as the framework.

The style transfer model is a network structure that can extract features of an image. For example, the style transfer model may include a convolutional layer, and the convolutional layer can extract features of an image through a convolution operation.

In some optional embodiments, the style transfer model is obtained through pre-training. Through the pre-training process, the parameters of the style transfer model can be adjusted so that the style transfer model can achieve better style transfer performance. Before the step of "extracting style features from at least one preset style reference image according to the style feature mapping parameters of the style extraction layer in the style transfer model to obtain the style features corresponding to the style reference image", the image generation method provided by the embodiment of the present invention may also include:

By using the style transfer model to be trained, style features are extracted from the sample style reference image to obtain sample style features corresponding to the sample style reference image;

Extracting content features of the sample content image by using the style transfer model to be trained to obtain sample content features corresponding to the sample content image;

Performing feature fusion on the sample style feature and the sample content feature to obtain a sample style transfer image;

Calculating a model loss of the style transfer model to be trained based on the sample style transfer image, the sample style reference image and the sample content image;

According to the model loss, the model parameters of the style transfer model to be trained are adjusted to obtain a trained style transfer model.

The sample style reference image may be an image with any style, and the sample content image may be an image with any content.

The model parameters may specifically include the number of feature extraction layers used to extract feature information in the style transfer model, the number of input channels of the feature extraction layers, and the like.

For example, when pre-training a style transfer network, you can select any image in the style of Van Gogh or Monet as a style image for training. The training set can be the large real image set of ImageNet. Select a style transfer network (such as Fast style transfer[3]) for training and update the model parameters. The trained model combines the feature weights extracted from the image content and the feature weights extracted from the style image. When testing, the stylized image retains both the content details of the real image and the style details of the style image.

In some optional examples, the model loss can be calculated based on the style loss between the sample style transfer image and the sample style reference image, and the content loss between the sample content image and the sample style transfer image. That is, the step of "calculating the model loss of the style transfer model to be trained based on the sample style transfer image, the sample style reference image and the sample content image" may specifically include:

Calculate the style similarity between the sample style transfer image and the sample style reference image as the style loss of the style transfer model to be trained;

Calculate the content similarity between the sample style transfer image and the sample content image, and calculate the content loss of the style transfer model to be trained;

Based on the style loss and content loss, the model loss of the style transfer model to be trained is calculated.

Specifically, the model loss can be calculated by weighting the style loss and the content loss, and the weights of the style loss and the content loss can be set by the technicians themselves.

In actual application, style transfer models of multiple style types may be trained for backup. Optionally, the image attribute information includes the image style type. The step of "extracting style features from at least one preset style reference image, and extracting content features from the content image, and obtaining a target image based on the fusion of the content features and the style features" may specifically include:

Selecting at least one target style reference image corresponding to the image style type from at least one preset style reference image;

Extracting style features for each of the target style reference images, and extracting content features from the content images;

The content features and the style features are weightedly fused to obtain a target image.

The weights corresponding to the content features and the style features may be the same or different, which is not limited in the embodiment of the present invention.

As can be seen from the above, an embodiment of the present invention can receive an image generation instruction, wherein the image generation instruction includes image attribute information of a target image to be generated, obtains environmental information, the environmental information is collected by a display device that displays the target image, performs data mapping on the environmental information through a preset image generation model, generates a content image corresponding to the environmental information, extracts style features from at least one preset style reference image, and extracts content features from the content image, and obtains the target image based on the fusion of the content features and the style features; since in an embodiment of the present invention, the image generation model can obtain the content image by mapping according to the environmental information, it can ensure that the content of the image fits the environment, and the content features in the content image can be fused with the style features in the style reference image to finally generate the target image, therefore, wallpaper images with rich content and diverse styles can be generated to enhance the user's visual experience.

In order to better implement the above method, accordingly, an embodiment of the present invention further provides an image generating device.

Referring to FIG5 , the apparatus may include:

The instruction receiving unit 501 may be used to receive an image generation instruction, wherein the image generation instruction may include image attribute information of a target image to be generated;

The environment information acquisition unit 502 may be used to acquire environment information, wherein the environment information is collected by a display device that displays the target image;

The content image generation unit 503 may be used to perform data mapping on the environment information through a preset image generation model to generate a content image corresponding to the environment information;

The target image generating unit 504 may be configured to extract style features from at least one preset style reference image and content features from the content image, and obtain a target image based on a fusion of the content features and the style features.

In some optional embodiments, as shown in FIG6 , the image generation apparatus provided by the embodiment of the present invention may further include an image generation model training unit 505, which may be used to obtain a generative adversarial model to be trained, wherein the generative adversarial model may include an image generation model to be trained and a discriminant model to be trained, and the generative adversarial model is provided with at least one real content sample image;

Performing data mapping on the training input parameters through the image generation model to be trained to generate a training content image corresponding to the training input parameters;

Calculating the authenticity parameter of the training content image relative to the real content sample image by using the discriminant model to be trained;

Adjusting the image generation model to be trained and the discriminant model to be trained based on the truthfulness parameter;

Return to execute the step of performing data mapping on the training input parameters through the image generation model to be trained until a preset training end condition is met, thereby obtaining a trained image generation model.

In some optional embodiments, as shown in FIG6 , the image generation apparatus provided by the embodiment of the present invention may further include an image generation model setting unit 506, which may be used to obtain display parameters of the display device;

Based on the display parameters, the model parameters of the image generation model to be trained are set to obtain a new image generation model to be trained.

In some optional embodiments, the target image generation unit 504 may be configured to extract style features from at least one preset style reference image according to style feature mapping parameters of a style extraction layer in a style transfer model, to obtain style features corresponding to the style reference image;

According to the content feature mapping parameters of the content extraction layer in the style transfer model, content features are extracted from the content image to obtain content features corresponding to the content image.

In some optional embodiments, the image generation apparatus provided by the embodiment of the present invention may further include a style transfer model training unit 507, which may be used to extract style features of the sample style reference image through the style transfer model to be trained, so as to obtain sample style features corresponding to the sample style reference image;

Extracting content features of the sample content image by using the style transfer model to be trained to obtain sample content features corresponding to the sample content image;

Performing feature fusion on the sample style feature and the sample content feature to obtain a sample style transfer image;

Calculating a model loss of the style transfer model to be trained based on the sample style transfer image, the sample style reference image and the sample content image;

According to the model loss, the model parameters of the style transfer model to be trained are adjusted to obtain a trained style transfer model.

In some optional embodiments, the environment information acquisition unit may be used to collect environment sound audio through a display device;

The environmental sound audio is converted into a model input format of the image generation model to obtain environmental information.

In some optional embodiments, the target image generation unit may be configured to select at least one target style reference image corresponding to the image style type from at least one preset style reference image;

Extracting style features for each of the target style reference images, and extracting content features from the content images;

The content features and the style features are weightedly fused to obtain a target image.

As can be seen from the above, through the image generation device, an image generation instruction can be received, the image generation instruction includes image attribute information of a target image to be generated, and environmental information is obtained, the environmental information is collected by a display device that displays the target image, and the environmental information is data mapped by a preset image generation model to generate a content image corresponding to the environmental information, and style features are extracted from at least one preset style reference image, and content features are extracted from the content image, and the target image is obtained based on the fusion of the content features and the style features; because in an embodiment of the present invention, the image generation model can map the content image according to the environmental information, therefore, it can be ensured that the content of the image fits the environment, and the content features in the content image can be fused with the style features in the style reference image to finally generate the target image, therefore, wallpaper images with rich content and diverse styles can be generated to enhance the user's visual experience.

In addition, an embodiment of the present invention further provides a computer device, which may be a terminal or a server, etc. As shown in FIG. 7 , a schematic diagram of the structure of the computer device involved in the embodiment of the present invention is shown. Specifically:

The computer device may include a radio frequency (RF) circuit 701, a memory 702 including one or more computer-readable storage media, an input unit 703, a display unit 704, a sensor 705, an audio circuit 706, a wireless fidelity (WiFi) module 707, a processor 708 including one or more processing cores, and a power supply 709. Those skilled in the art will appreciate that the computer device structure shown in FIG. 7 does not constitute a limitation on the computer device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange components differently. Among them:

The RF circuit 701 can be used for receiving and sending signals during information transmission or calls. In particular, after receiving the downlink information of the base station, it is handed over to one or more processors 708 for processing; in addition, the data related to the uplink is sent to the base station. Usually, the RF circuit 701 includes but is not limited to an antenna, at least one amplifier, a tuner, one or more oscillators, a subscriber identity module (SIM) card, a transceiver, a coupler, a low noise amplifier (LNA), a duplexer, etc. In addition, the RF circuit 701 can also communicate with the network and other devices through wireless communication. Wireless communication can use any communication standard or protocol, including but not limited to the Global System of Mobile Communication (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, Short Messaging Service (SMS), etc.

The memory 702 can be used to store software programs and modules. The processor 708 executes various functional applications and data processing by running the software programs and modules stored in the memory 702. The memory 702 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the computer device (such as audio data, a phone book, etc.), etc. In addition, the memory 702 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 702 may also include a memory controller to provide the processor 708 and the input unit 703 with access to the memory 702.

The input unit 703 can be used to receive input digital or character information, and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control. Specifically, in a specific embodiment, the input unit 703 may include a touch-sensitive surface and other input devices. The touch-sensitive surface, also known as a touch display screen or a touch pad, can collect the user's touch operations on or near it (such as the user's operation on or near the touch-sensitive surface using any suitable object or accessory such as a finger, a stylus, etc.), and drive the corresponding connection device according to a pre-set program. Optionally, the touch-sensitive surface may include a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch direction, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into the touch point coordinates, and then sends it to the processor 708, and can receive and execute the command sent by the processor 708. In addition, the touch-sensitive surface can be implemented using multiple types such as resistive, capacitive, infrared and surface acoustic wave. In addition to the touch-sensitive surface, the input unit 703 may also include other input devices. Specifically, other input devices may include, but are not limited to, one or more of a physical keyboard, function keys (such as a volume control key, a switch key, etc.), a trackball, a mouse, a joystick, and the like.

The display unit 704 can be used to display information input by the user or information provided to the user and various graphical user interfaces of the computer device, which can be composed of graphics, text, icons, videos and any combination thereof. The display unit 704 may include a display panel. Optionally, the display panel may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), etc. Further, the touch-sensitive surface may cover the display panel. When the touch-sensitive surface detects a touch operation on or near it, it is transmitted to the processor 708 to determine the type of touch event, and then the processor 708 provides corresponding visual output on the display panel according to the type of touch event. Although in FIG. 7, the touch-sensitive surface and the display panel are implemented as two independent components to implement input and output functions, in some embodiments, the touch-sensitive surface and the display panel can be integrated to implement input and output functions.

The computer device may also include at least one sensor 705, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel according to the brightness of the ambient light, and the proximity sensor may turn off the display panel and/or backlight when the computer device is moved to the ear. As a type of motion sensor, the gravity acceleration sensor can detect the magnitude of acceleration in each direction (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that identify the posture of the mobile phone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; As for other sensors such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc. that can also be configured in the computer device, they will not be repeated here.

The audio circuit 706, the speaker, and the microphone can provide an audio interface between the user and the computer device. The audio circuit 706 can transmit the electrical signal converted from the received audio data to the speaker, which is converted into a sound signal for output; on the other hand, the microphone converts the collected sound signal into an electrical signal, which is received by the audio circuit 706 and converted into audio data, and then the audio data is output to the processor 708 for processing, and then sent to another computer device through the RF circuit 701, or the audio data is output to the memory 702 for further processing. The audio circuit 706 may also include an earphone jack to provide communication between an external headset and the computer device.

WiFi is a short-range wireless transmission technology. Computer devices can help users send and receive emails, browse web pages, and access streaming media through WiFi module 707, which provides users with wireless broadband Internet access. Although FIG. 7 shows WiFi module 707, it is understandable that it is not a necessary component of the computer device and can be omitted as needed without changing the essence of the invention.

The processor 708 is the control center of the computer device, which uses various interfaces and lines to connect various parts of the entire mobile phone, and executes various functions of the computer device and processes data by running or executing software programs and/or modules stored in the memory 702, and calling data stored in the memory 702. Optionally, the processor 708 may include one or more processing cores; preferably, the processor 708 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 708.

The computer device also includes a power supply 709 (such as a battery) for supplying power to each component. Preferably, the power supply can be logically connected to the processor 708 through a power management system, so that the power management system can manage charging, discharging, and power consumption management. The power supply 709 can also include one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

Although not shown, the computer device may also include a camera, a Bluetooth module, etc., which will not be described in detail here. Specifically in this embodiment, the processor 708 in the computer device will load the executable files corresponding to the processes of one or more application programs into the memory 702 according to the following instructions, and the processor 708 will run the application programs stored in the memory 702 to implement various functions, as follows:

receiving an image generation instruction, wherein the image generation instruction includes image attribute information of a target image to be generated;

Acquiring environmental information, wherein the environmental information is collected by a display device that displays the target image;

Performing data mapping on the environmental information through a preset image generation model to generate a content image corresponding to the environmental information;

Style features are extracted from at least one preset style reference image, and content features are extracted from the content image, and a target image is obtained based on the fusion of the content features and the style features.

A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

To this end, an embodiment of the present invention provides a computer-readable storage medium, in which a plurality of instructions are stored, and the instructions can be loaded by a processor to execute the steps in any one of the image generation methods provided in the embodiments of the present invention. For example, the instructions can execute the following steps:

receiving an image generation instruction, wherein the image generation instruction includes image attribute information of a target image to be generated;

Acquiring environmental information, wherein the environmental information is collected by a display device that displays the target image;

Performing data mapping on the environmental information through a preset image generation model to generate a content image corresponding to the environmental information;

Style features are extracted from at least one preset style reference image, and content features are extracted from the content image, and a target image is obtained based on the fusion of the content features and the style features.

The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

The computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

Since the instructions stored in the computer-readable storage medium can execute the steps in any image generation method provided in the embodiments of the present invention, the beneficial effects that can be achieved by any image generation method provided in the embodiments of the present invention can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

According to one aspect of the present application, a computer program product or a computer program is also provided, the computer program product or the computer program including computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the methods provided in various optional implementations in the above-mentioned embodiments.

The above is a detailed introduction to an image generation method, device, computer equipment and storage medium provided in an embodiment of the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims (20)

A method for generating an image, comprising: receiving an image generation instruction, wherein the image generation instruction includes image attribute information of a target image to be generated; Acquiring environmental information, wherein the environmental information is collected by a display device that displays the target image; Performing data mapping on the environmental information through a preset image generation model to generate a content image corresponding to the environmental information; Style features are extracted from at least one preset style reference image, and content features are extracted from the content image, and a target image is obtained based on the fusion of the content features and the style features. The image generation method according to claim 1, characterized in that before performing data mapping on the environmental information through a preset image generation model to generate a content image corresponding to the environmental information, the method further comprises: Obtaining a generative adversarial model to be trained, wherein the generative adversarial model includes an image generation model to be trained and a discriminant model to be trained, and the generative adversarial model is provided with at least one real content sample image; Performing data mapping on the training input parameters through the image generation model to be trained to generate a training content image corresponding to the training input parameters; Calculating the authenticity parameter of the training content image relative to the real content sample image by using the discriminant model to be trained; Adjusting the image generation model to be trained and the discriminant model to be trained based on the truthfulness parameter; Return to execute the step of performing data mapping on the training input parameters through the image generation model to be trained until a preset training end condition is met, thereby obtaining a trained image generation model. The image generation method according to claim 2 is characterized in that, before performing data mapping on the training input parameters through the image generation model to be trained to generate the training content image corresponding to the training input parameters, the method further comprises: Acquire display parameters of the display device; Based on the display parameters, the model parameters of the image generation model to be trained are set to obtain a new image generation model to be trained. The image generation method according to claim 1, characterized in that extracting style features from at least one preset style reference image and extracting content features from the content image include: Extracting style features from at least one preset style reference image according to style feature mapping parameters of a style extraction layer in a style transfer model to obtain style features corresponding to the style reference image; According to the content feature mapping parameters of the content extraction layer in the style transfer model, content features are extracted from the content image to obtain content features corresponding to the content image. The image generation method according to claim 1 is characterized in that, before extracting style features from at least one preset style reference image according to the style feature mapping parameters of the style extraction layer in the style transfer model to obtain the style features corresponding to the style reference image, the method further comprises: By using the style transfer model to be trained, style features are extracted from the sample style reference image to obtain sample style features corresponding to the sample style reference image; Extracting content features of the sample content image by using the style transfer model to be trained to obtain sample content features corresponding to the sample content image; Performing feature fusion on the sample style feature and the sample content feature to obtain a sample style transfer image; Calculating a model loss of the style transfer model to be trained based on the sample style transfer image, the sample style reference image and the sample content image; According to the model loss, the model parameters of the style transfer model to be trained are adjusted to obtain a trained style transfer model. The image generation method according to claim 5, characterized in that the step of calculating the model loss of the style transfer model to be trained based on the sample style transfer image, the sample style reference image and the sample content image comprises: Calculate the style similarity between the sample style transfer image and the sample style reference image as the style loss of the style transfer model to be trained; Calculate the content similarity between the sample style transfer image and the sample content image, and calculate the content loss of the style transfer model to be trained; Based on the style loss and content loss, the model loss of the style transfer model to be trained is calculated. The image generation method according to claim 1, characterized in that the obtaining of environmental information comprises: Collect environmental sound audio through display devices; The environmental sound audio is converted into a model input format of the image generation model to obtain environmental information. The image generation method according to claim 1, characterized in that the obtaining of environmental information comprises: Collecting ambient sound audio through a display device to obtain ambient sound information; Obtaining environmental weather information corresponding to the display device; Fusing the ambient sound information and the ambient weather information to generate fused ambient information; The fused environmental information is converted into a model input format of the image generation model to obtain environmental information. The image generation method according to claim 1, characterized in that the image attribute information includes an image style type, the extracting style features from at least one preset style reference image, and extracting content features from the content image, and obtaining a target image based on a fusion of the content features and the style features, comprises: Selecting at least one target style reference image corresponding to the image style type from at least one preset style reference image; Extracting style features for each of the target style reference images, and extracting content features from the content images; The content features and the style features are weightedly fused to obtain a target image. An image generating device, comprising: An instruction receiving unit, used for receiving an image generation instruction, wherein the image generation instruction includes image attribute information of a target image to be generated; An environmental information acquisition unit, used to acquire environmental information, wherein the environmental information is collected by a display device that displays the target image; A content image generating unit, configured to perform data mapping on the environmental information through a preset image generating model, and generate a content image corresponding to the environmental information; The target image generation unit is used to extract style features from at least one preset style reference image and content features from the content image, and obtain the target image based on the fusion of the content features and the style features. The image generation device according to claim 10, characterized in that the image generation device further comprises an image generation model training unit, which is used to obtain a generative adversarial model to be trained, wherein the generative adversarial model comprises an image generation model to be trained and a discriminant model to be trained, and the generative adversarial model is provided with at least one real content sample image; Performing data mapping on the training input parameters through the image generation model to be trained to generate a training content image corresponding to the training input parameters; Calculating the authenticity parameter of the training content image relative to the real content sample image by using the discriminant model to be trained; Adjusting the image generation model to be trained and the discriminant model to be trained based on the truthfulness parameter; Return to execute the step of performing data mapping on the training input parameters through the image generation model to be trained until a preset training end condition is met, thereby obtaining a trained image generation model. The image generating device according to claim 11, characterized in that the image generating device further comprises an image generating model setting unit, which is used to obtain display parameters of the display device; Based on the display parameters, the model parameters of the image generation model to be trained are set to obtain a new image generation model to be trained. The image generation device according to claim 10, characterized in that the target image generation unit is used to extract style features from at least one preset style reference image according to the style feature mapping parameters of the style extraction layer in the style transfer model to obtain the style features corresponding to the style reference image; According to the content feature mapping parameters of the content extraction layer in the style transfer model, content features are extracted from the content image to obtain content features corresponding to the content image. The image generation device according to claim 10, characterized in that the image generation device further comprises a style transfer model training unit, configured to extract style features from the sample style reference image through the style transfer model to be trained, so as to obtain the sample style features corresponding to the sample style reference image; Extracting content features of the sample content image by using the style transfer model to be trained to obtain sample content features corresponding to the sample content image; Performing feature fusion on the sample style feature and the sample content feature to obtain a sample style transfer image; Calculating a model loss of the style transfer model to be trained based on the sample style transfer image, the sample style reference image and the sample content image; According to the model loss, the model parameters of the style transfer model to be trained are adjusted to obtain a trained style transfer model. The image generation device according to claim 14, characterized in that the style transfer model training unit is used to calculate the style similarity between the sample style transfer image and the sample style reference image as the style loss of the style transfer model to be trained; Calculate the content similarity between the sample style transfer image and the sample content image, and calculate the content loss of the style transfer model to be trained; Based on the style loss and content loss, the model loss of the style transfer model to be trained is calculated. The image generating device according to claim 10, characterized in that the environmental information acquisition unit is used to collect environmental sound audio through a display device; The environmental sound audio is converted into a model input format of the image generation model to obtain environmental information. The image generation device according to claim 10, characterized in that the environmental information acquisition unit is used to collect environmental sound audio through a display device to obtain environmental sound information; Obtaining environmental weather information corresponding to the display device; Fusing the ambient sound information and the ambient weather information to generate fused ambient information; The fused environmental information is converted into a model input format of the image generation model to obtain environmental information. The image generation device according to claim 10, characterized in that the target image generation unit is used to select at least one target style reference image corresponding to the image style type from at least one preset style reference image; Extracting style features for each of the target style reference images, and extracting content features from the content images; The content features and the style features are weightedly fused to obtain a target image. A computer device, characterized in that it includes a memory and a processor; the memory stores an application program, and the processor is used to run the application program in the memory to execute the steps in the image generation method described in any one of claims 1 to 9. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a plurality of instructions, wherein the instructions are suitable for a processor to load to execute the steps in the image generation method according to any one of claims 1 to 9.