WO2025118634A1 - Video generation method, electronic device, and computer readable storage medium - Google Patents

Abstract

Provided are a video generation method, an electronic device, and a computer readable storage medium, relating to the technical fields of computers and video processing. The video generation method comprises: acquiring a target text, wherein the target text is used for describing video content to be generated (S21); and using a target video generation model to perform video generation processing on the target text to obtain a target video, wherein the target video generation model is a model obtained by performing model alignment on an initial video generation model and a preset reward model in a fine-tuning manner (S22). The present invention solves the technical problems in the related art that a video generated by a video generation model obtained by training based on network data has poor quality and does not meet user expectations.

Description

Video generation method, electronic device and computer readable storage medium Technical Field

The present disclosure relates to the fields of computer technology and video processing technology, and in particular to a video generation method, an electronic device, and a computer-readable storage medium.

Background Art

With the popularity of video content on the Internet, the demand for high-quality and personalized video generation is increasing. Video generation models, as a video generation tool, can generate realistic video content based on given input.

At present, video generation models usually use data on the Internet for model training. Since most of the data on the Internet are of uneven quality, the videos generated by the trained video generation models are of poor quality and do not meet user expectations.

To address the above-mentioned problems, no effective solution has been proposed yet.

Summary of the invention

The embodiments of the present disclosure provide a video generation method, an electronic device, and a computer-readable storage medium to at least solve the technical problem in the related art that a video generation model is trained based on network data, resulting in that the video quality generated by the trained video generation model is poor and does not meet user expectations.

According to one aspect of an embodiment of the present disclosure, there is provided a video generation method, comprising: obtaining a target text, wherein the target text is used to describe the content of a video to be generated; performing video generation processing on the target text using a target video generation model to obtain a target video, wherein the target video generation model is a model obtained by aligning an initial video generation model with a preset reward model using a fine-tuning method.

According to another aspect of an embodiment of the present disclosure, a video generation method is also provided, which provides a graphical user interface through a terminal device, and the content displayed by the graphical user interface at least partially includes a video generation scene, including: in response to a first touch operation applied to the graphical user interface, inputting a target text, wherein the target text is used to describe the video content to be generated; in response to a second touch operation applied to the graphical user interface, performing video generation processing on the target text using a target video generation model to obtain a target video, wherein the target video generation model is a model obtained by aligning an initial video generation model with a preset reward model using a fine-tuning method; and displaying the target video in the graphical user interface.

According to another aspect of the embodiment of the present disclosure, a video generation method is also provided, including: obtaining a currently input video generation dialogue request, wherein the information carried in the video generation dialogue request includes: a target text, which is used to describe the video content to be generated; in response to the video generation dialogue request, returning a video generation dialogue reply, wherein the information carried in the video generation dialogue reply includes: a target video, which is obtained by performing video generation processing on the target text using a target video generation model, wherein the target video generation model is a fine-tuned method for The model obtained after model alignment between the initial video generation model and the preset reward model; video generation dialogue response is displayed in the graphical user interface.

According to another aspect of an embodiment of the present disclosure, an electronic device is further provided, including: a memory storing an executable program; and a processor for running the program, wherein any one of the above-mentioned video generation methods is executed when the program is running.

According to another aspect of an embodiment of the present disclosure, a computer-readable storage medium is further provided, the computer-readable storage medium including a stored executable program, wherein when the executable program runs, the device where the computer-readable storage medium is located is controlled to execute any one of the above-mentioned video generation methods.

In the disclosed embodiment, a target text for describing the video content to be generated is obtained, and then a video generation process is performed on the target text based on a target video generation model obtained by aligning an initial video generation model with a preset reward model by fine-tuning. That is, a pre-trained U-shaped network is aligned with a picture reward model by fine-tuning to obtain a fine-tuned target video generation model, and a video is generated based on the target text using the fine-tuned target video generation model, thereby obtaining a target video. This achieves the purpose of generating a target video that meets user expectations, thereby achieving a technical effect of making the generated target video more consistent with human aesthetic preferences and the target text content, improving the video quality of the generated target video, and making the generated target video more popular with humans. This further solves the technical problem in the related art of training a video generation model based on network data, resulting in a poor quality video generated by the trained video generation model that does not meet user expectations.

It is easily noted that the above general description and the following detailed description are merely for exemplifying and explaining the present disclosure, and do not constitute a limitation of the present disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

The drawings described herein are used to provide a further understanding of the present disclosure and constitute a part of the present disclosure. The illustrative embodiments of the present disclosure and their descriptions are used to explain the present disclosure and do not constitute an improper limitation on the present disclosure. In the drawings:

FIG1 is a schematic diagram of an application scenario of a video generation method according to Embodiment 1 of the present disclosure;

FIG2 is a flow chart of a video generation method according to Embodiment 1 of the present disclosure;

FIG3 is a schematic diagram of a fine-tuning process according to Embodiment 1 of the present disclosure;

FIG4 is a schematic diagram of a video generation method according to Embodiment 1 of the present disclosure;

FIG5 is a flow chart of a video generation method according to Embodiment 2 of the present disclosure;

FIG6 is a flow chart of a video generation method according to Embodiment 3 of the present disclosure;

FIG7 is a schematic structural diagram of a video generating device according to Embodiment 4 of the present disclosure;

FIG8 is a schematic structural diagram of another video generating device according to Embodiment 4 of the present disclosure;

FIG9 is a schematic structural diagram of another video generating device according to Embodiment 4 of the present disclosure;

FIG10 is a structural block diagram of a computer terminal according to Embodiment 5 of the present disclosure.

DETAILED DESCRIPTION

In order to enable those skilled in the art to better understand the scheme of the present disclosure, the technical scheme in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in the field without creative work should fall within the scope of protection of the present disclosure.

It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

First, some nouns or terms that appear in the process of describing the embodiments of the present disclosure are subject to the following explanations:

Video diffusion models: A deep learning-based generative model used to generate or modify video content. The model generates new video frame sequences by simulating the distribution of video data. Video diffusion models are usually trained with large amounts of data to learn how to generate realistic videos.

Human preference: refers to the subjective preferences or choices of human users when reviewing or evaluating content. In the context of AI-generated content, human preference usually refers to the user's preference for content quality, style, accuracy, etc.

Human preference model: A machine learning model that aims to capture and imitate human preference judgments. The model learns by analyzing human evaluations of content in order to produce results that are more in line with user preferences in subsequent generation processes. The model usually requires a large amount of manually annotated data for training.

Alignment: In the context of AI-generated content, alignment usually refers to the process of adjusting the generative model so that the content produced by the generative model better meets specific standards or goals, such as user preferences, requirements of specific tasks, etc.

Reward model: In machine learning, a reward model is used to evaluate the effect of an action or output, and is usually used in reinforcement learning/reward learning. In the context of video generation in the disclosed embodiments, a reward model can be used to evaluate the quality of the generated video to guide the model to produce higher quality output.

Reward score: The reward score is an indicator used to quantify the quality or conformity of the model output. In the application of the video generation model of the embodiment of the present disclosure, reward fine-tuning specifically refers to the evaluation score of the generated video based on the reward model (such as the human preference model). The score reflects the consistency between the generated content and human preferences, target standards or expected goals, and is usually used to guide and optimize the training process of the model to generate more in line with user preferences or Higher quality video content.

U-Net: A deep learning neural network structure that is widely used in the field of computer vision, especially in image segmentation tasks. The U-Net structure consists of an encoder and a decoder, and its structure resembles a U shape, hence the name. The encoder is responsible for feature extraction and dimensionality reduction of the input image, while the decoder is responsible for restoring the encoded feature map to the original image size and performing pixel-level classification or segmentation.

Model fine-tuning: refers to adjusting the parameters of a model based on a pre-trained model through a small amount of data or domain-related data to adapt it to a specific task or dataset. Usually, fine-tuning is performed on a model that has been trained on a large dataset. This model is usually a deep learning model that performs well on general tasks, or a natural language processing model pre-trained on a large text corpus.

Resampling: A commonly used data processing method used to adjust the size, distribution or time interval of data samples. In statistics and machine learning, resampling is often used to solve problems such as sample imbalance, missing data or inconsistent data collection frequency. In the disclosed embodiment, resampling the video refers to adjusting the sampling rate of the original video to change the playback speed of the video or adapt to different playback devices. Resampling can be to increase the sampling rate to improve video quality, or to reduce the sampling rate to reduce the file size or to adapt to specific playback requirements. Resampling usually causes changes in the image quality and smoothness of the video.

Discriminative Dimensionality Reduction for Imitation Learning (DDIM) sampling: DDIM sampling is a method for the generative process that aims to extract important information from the data and learn it. This method is mainly used in the field of imitation learning to build a model that can imitate human behavior. The main idea of DDIM sampling is to reduce the dimensionality of the data by distinguishing between important and unimportant dimensions. It discriminates the dimensions of the data to find the most useful dimensions for the model training task, and then uses these dimensions for model training and generation. Specifically, DDIM sampling first finds the most useful features for the model task through feature selection methods or feature extraction methods. Then, it maps the data to the dimensions where these important features are located by learning a discriminative dimensionality reduction model. Finally, these important dimensions are used for model training and generation.

Time-Decay Reward (TAR): A technique used in reinforcement learning to handle situations where the value of future rewards decreases over time. In reinforcement learning, a discount factor is often used to measure the importance of future rewards, but in some cases, the value of future rewards decreases over time, such as in some tasks where earlier rewards may be more important than later rewards. TAR reflects the effect of time by giving higher weights to earlier rewards. This can be achieved by introducing a time decay function when calculating the reward, such as an exponential decay function or a polynomial decay function.

Sparse sampling: A data sampling method used to select a portion of samples in a large data set for analysis or processing. In sparse sampling, only a small portion of the data set is selected to represent the entire data set to reduce computational cost and time. Sparse sampling can be achieved by random sampling, stratified sampling, or other sampling methods to ensure that all samples are The selected samples can represent the characteristics of the entire dataset.

Low Power Wide Area Network (LoRA): A low power wide area network technology that enables low power communication over long distances. LoRA technology uses a modulation technique called spread spectrum, which enables long-distance communication at low power. LoRA technology can achieve more efficient and reliable communication by fine-tuning the parameters of LoRA devices, that is, achieving efficient fine-tuning. These parameters include transmit power, data rate, receive sensitivity, etc. By adjusting these parameters reasonably, better performance can be achieved in different application scenarios.

The related art of training video generation models based on network data has the following defects.

Defect 1: Since most of the data on the Internet are of varying quality, the quality of the videos generated by the trained video generation model is poor, which does not meet user expectations.

Defect 2: The video diffusion model in related technologies cannot fully consider human aesthetic preferences and content relevance.

With respect to the above-mentioned defects, no effective solution has been proposed before the present disclosure.

Example 1

According to an embodiment of the present disclosure, a video generation method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

The method embodiment provided in the first embodiment of the present disclosure can be executed in a mobile terminal, a computer terminal or a similar computing device. FIG1 shows a hardware structure block diagram of a computer terminal (or mobile device) for implementing a video generation method. As shown in FIG1 , the computer terminal 10 (or mobile device) may include one or more (102a, 102b, ..., 102n are used in the figure to show) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input/output interface (I/O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and/or a camera. It can be understood by those skilled in the art that the structure shown in FIG1 is only for illustration and does not limit the structure of the above-mentioned electronic device. For example, the computer terminal 10 may also include more or fewer components than those shown in FIG1 , or have a configuration different from that shown in FIG1 .

It should be noted that the one or more processors 102 and/or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10 (or mobile device). As involved in the embodiments of the present disclosure, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

The memory 104 can be used to store software programs and modules of application software, such as the video The program instructions/data storage device corresponding to the generation method, the processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, the above-mentioned video generation method is realized. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (Radio Frequency, RF) module, which is used to communicate with the Internet wirelessly.

The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 (or mobile device).

In the above operating environment, the present disclosure provides a video generation method as shown in FIG2. FIG2 is a flow chart of a video generation method according to Embodiment 1 of the present disclosure. As shown in FIG2, the method may include the following steps:

Step S21, obtaining a target text, wherein the target text is used to describe the video content to be generated;

Step S22, using a target video generation model to perform video generation processing on the target text to obtain a target video, wherein the target video generation model is a model obtained by aligning the initial video generation model with the preset reward model in a fine-tuning manner.

The target text can be understood as the text input into the video generation model, such as the text input into the target video generation model in the embodiment of the present disclosure, that is, as the input of the target video generation model. The target text is used to describe the video content to be generated, for example, to describe the video content that the user expects to generate.

For example, if the video content that the user desires to generate is a video of Dogwood blossoms are blowing in the wind, the target text may be "狗玉花在风漂荡", or "狗木 开花s are blowing in the wind", etc. It is understandable that the target text may be described in natural language, such as Chinese, English, Japanese, etc., which is not limited here.

The initial video generation model can be understood as an initial model for generating videos based on text. For example, the initial video generation model can be a pre-trained U-type network (Pre-trained UNet), that is, a U-type network model pre-trained on a large-scale dataset. Considering that pre-training on a large-scale dataset can enable the model to learn rich image features and semantic information, thereby improving the generalization ability and accuracy of the model on specific tasks, the present disclosure uses a pre-trained U-type network model as the initial video generation model to make the video generated by the final target video generation model more accurate.

The preset reward model can be used to evaluate the quality of the generated video, thereby guiding the video generation model to produce higher A model of output quality, such as a preset reward model, can evaluate and score the generated video, thereby quantifying the output quality or conformity of the video generation model through a reward score.

It should be noted that, considering that the final target video generation model should fully consider human aesthetic preferences and content relevance in order to generate videos that meet user expectations, the preset reward model in the embodiment of the present disclosure adopts an image-based human preference model, namely an image reward model. The image reward model can evaluate the quality of the generated video based on human aesthetic preferences and content relevance, so that the video generated by the final target video generation model is more in line with user expectations.

The target video generation model is a model obtained by aligning the initial video generation model with the preset reward model by fine-tuning the embodiment of the present disclosure, that is, a model obtained by aligning the pre-trained U-type network with the image reward model by fine-tuning the model, so that video content that meets user preferences and expectations can be accurately generated according to the target video generation model.

In the disclosed embodiment, by obtaining a target text for describing the video content to be generated, and then performing video generation processing on the target text based on a target video generation model obtained by aligning the initial video generation model with a preset reward model in a fine-tuning manner, that is, aligning the pre-trained U-type network with the image reward model through fine-tuning to obtain a fine-tuned target video generation model, and using the fine-tuned target video generation model to generate a video based on the target text, thereby obtaining a target video. This can make the generated target video more consistent with human aesthetic preferences and the target text content, improve the video quality of the generated target video, that is, obtain a video that better meets user expectations, thereby improving the personalization level of the generated video content, and opening up new application prospects in the field of video generation.

The above-mentioned video generation method provided by the embodiments of the present disclosure can be applied to, but is not limited to, application scenarios involving video generation in the fields of e-commerce services, educational services, legal services, medical services, conference services, social network services, financial product services, logistics services, and navigation services. For example: the scenario of generating product display content in e-commerce services, the scenario of generating learning content videos in educational services, the scenario of generating case-related videos in legal services, etc., which are not limited here.

According to the disclosed embodiment, a target text for describing the content of a video to be generated is obtained, and then a video generation process is performed on the target text based on a target video generation model obtained by aligning an initial video generation model with a preset reward model by fine-tuning. That is, a pre-trained U-shaped network is aligned with a picture reward model by fine-tuning to obtain a fine-tuned target video generation model, and a video is generated based on the target text using the fine-tuned target video generation model, thereby obtaining a target video. This achieves the purpose of generating a target video that meets user expectations, thereby achieving a technical effect of making the generated target video more consistent with human aesthetic preferences and the content of the target text, improving the video quality of the generated target video, and making the generated target video more popular with humans. This further solves the technical problem in the related art of training a video generation model based on network data, resulting in a poor quality video generated by the trained video generation model that does not meet user expectations.

In an optional embodiment, the target video generation model is a video diffusion model, and the preset reward model is a graph Image reward model, where the image reward model is used to perform preference learning on the video diffusion model.

In the disclosed embodiment, the target video generation model may be a video diffusion model, an autoregressive model, etc., and the preset reward model may be an image reward model for performing preference learning on the video diffusion model.

In an optional embodiment, the video generation method further includes the following method steps:

Step S23, using training samples to perform sampling processing on the initial video generation model to generate sampled videos, wherein the training samples include: a plurality of video-text pairs, and the plurality of video-text pairs each include: a training video and a training text, and the training text is used to describe the video content of the training video;

Step S24, using a preset reward model to calculate rewards for the sampled video to obtain a target reward result;

Step S25, adjusting the model parameters of the initial video generation model based on the target reward result to generate a target video generation model.

The training samples can be understood as training samples to be used for model fine-tuning. For example, they can be part of the video text selected from the pre-training data set. The training samples can be selected according to any rules, which is not limited in the embodiments of the present disclosure.

The training samples include a plurality of video-text pairs, each of which includes a training video and a corresponding training text, wherein the training text is text content used to describe the video content of the training video.

The sampled video can be understood as a video obtained by sampling the training samples using the initial video generation model, that is, a video generated by sampling the training video and training text using the pre-trained U-shaped network model.

The target reward result is the reward score of the sampled video obtained by calculating the reward for the sampled video according to the preset reward model. The target reward result is used to reflect whether the sampled video is consistent with human aesthetic preferences and text content, and can be denoted as R.

In the disclosed embodiment, before the fine-tuning phase begins, some video texts can be selected from the pre-training data set as training samples, and then the initial video generation model is sampled using the training samples, that is, the training videos and training texts in the training samples are sampled using the pre-trained U-shaped network model to generate a sampled video. Then, a preset reward model is used to calculate rewards for the sampled video, that is, an image reward model is used to calculate rewards for the generated sampled video to obtain the target reward result corresponding to the sampled video. Finally, the model parameters of the initial video generation model are adjusted based on the target reward result, that is, the initial video generation model is fine-tuned based on the target reward result to obtain the target video generation model.

In an optional embodiment, in step S23, the initial video generation model is sampled using the training sample to generate a sampled video, including the following method steps:

Step S231, performing noise processing on the training video to obtain a noisy video;

Step S232, using the noisy video and the training text to perform video resampling on the initial video generation model to generate a sampled video.

In the embodiment of the present disclosure, when the training sample is used to sample the initial video generation model, the training video in the training sample can be denoised to obtain a noisy video, and then the obtained noisy video and the training text in the training sample are used to resample the initial video generation model to generate a sampled video.

Exemplarily, a diffusion process with noise may be performed on the training video to obtain a noisy video, and then the initial video generation model is sampled by DDIM using the obtained noisy video and the training text in the training sample to generate a sampled video.

It can be seen that compared with the traditional generation process method, the resampling of videos through DDIM sampling in the present invention can more effectively utilize data information, improve the generation and generalization capabilities of the model, and help the model better understand the structure and characteristics of the data, thereby better imitating human behavior.

In an optional embodiment, in step S231, performing noise processing on the training video to obtain a noisy video includes the following method steps:

Step S2311, obtaining the number of noise adding steps and the noise level corresponding to the training video, wherein the number of noise adding steps is used to determine the number of steps to be noise added to the training video through a preset noise adding function, and the noise level is used to determine the degree of damage to the training video;

Step S2312: Noise the training video based on the number of noise addition steps and the noise level to obtain a noisy video.

The preset noise adding function can be expressed as d(τ, D), which is used to calculate how many steps the training video should be noised to according to the value of the noise level τ and the number of noise adding steps D. The output result is usually between 1 and 1000.

The number of denoising steps D is used to determine the number of steps to be denoised for the training video by using a preset denoising function d(), and is usually set to 20.

The noise level τ is used to determine the degree of damage to the training video, and its value range is between 0 and 1.

Exemplarily, if the video generation model selected in the embodiment of the present disclosure is a diffusion model, the number of noise addition steps D represents the number of steps used in the diffusion model generation process. The preset noise addition function d() can determine the degree of video noise addition according to the noise level τ and the number of noise addition steps D, so as to generate a video with a certain degree of noise in the diffusion model.

In the disclosed embodiment, when performing noise processing on a training video, the number of noise adding steps d and the noise level τ corresponding to the training video can be obtained, and then the training video can be noise processed based on the number of noise adding steps d and the noise level τ, thereby generating a noisy video with a certain degree of noise.

It can be seen that compared with the related art of starting the noise addition process from the text, the embodiment of the present disclosure adopts DDIM sampling. The method of starting the noise addition process from the video only requires a τ proportion of the calculation amount of the complete generation process, so the calculation amount is small, which can effectively save computing resources.

In an optional embodiment, in step S24, a preset reward model is used to calculate a reward for the sampled video to obtain a target reward result, including the following method steps:

Step S241, using a preset reward model to calculate rewards for the sampled video to obtain an initial reward result;

Step S242, using a time decay reward method to adjust the initial reward weight corresponding to the initial reward result to generate a target reward result, wherein the initial reward weight is the default reward weight corresponding to the video frame sequence contained in the sampled video. Heavy.

In the disclosed embodiment, after the sampled video is generated, when the preset reward model is used to calculate the reward for the sampled video, the preset reward model can be used to calculate the reward for the sampled video, that is, an image-based human preference model is used, that is, an image reward model is used to calculate the reward, thereby obtaining an initial reward result, that is, the original output result of the preset reward model.

For example, the generated sample video and the text input when generating the sample video can be input into the image reward model, so as to obtain a score value based on the output of the image reward model, that is, to obtain an initial reward result. The score value output by the image reward model can be between 0 and 1, and a larger score indicates a better video quality, which is not limited here.

In addition, in order to improve the training effect of the model, after obtaining the initial reward result, the time-decay reward (TAR) method can be used to adjust the initial reward weight corresponding to the initial reward result, that is, adjust the default reward weight corresponding to the video frame sequence contained in the sampled video to generate the target reward result.

In an optional embodiment, in step S241, a preset reward model is used to calculate a reward for a sampled video to obtain an initial reward result, including the following method steps:

Step S2411, performing video segment sampling on the video frame sequence to obtain segment sampling results;

Step S2412, using a preset reward model to calculate rewards for the segmented sampling results to obtain an initial reward result.

In the disclosed embodiments, when a preset reward model is used to calculate rewards for sampled videos, in order to improve the efficiency of the learning process, segmented video rewards may be used to perform segmental sampling on video frame sequences, that is, sparse sampling is performed on the video, the video is split into several segments, and multiple continuous video frames are grouped to obtain segmented sampling results, and then the preset reward model is used to calculate rewards for the segmented sampling results to obtain initial reward results.

For example, taking a sampled video with 16 frames as an example, the 16-frame video can be divided into 4 groups with 4 frames in each group, and the preset reward model is used to calculate the rewards for the 4 groups to obtain the initial reward results, so as to improve the training effect of the model.

In an optional embodiment, in step S2411, performing video segment sampling on the video frame sequence to obtain segment sampling results includes the following method steps:

Step S24111, obtaining a feature space representation of a video frame sequence;

Step S24112, performing video segment sampling on the feature space representation to obtain a color space representation of the segmented video frame;

Step S24113, determine the segmented sampling result based on the color space representation.

In the disclosed embodiment, when performing video segmentation sampling on a video frame sequence, the feature space representation z_0 of the video frame sequence can be obtained, and then the feature space representation z_0 can be subjected to video segmentation sampling to obtain the color (RGB) space representation x_0^g of the segmented video frame, and finally the segmentation sampling result can be determined based on the color space representation x_0^g.

In an optional embodiment, in step S242, the initial reward weight corresponding to the initial reward result is adjusted by using a time decay reward method to generate a target reward result, including the following method steps:

Step S2421, using a time decay reward method to differentially adjust the initial reward weight corresponding to the initial reward result to obtain a target reward weight, wherein the target reward weight is used to indicate that the current reward weight of the first video frame in the video frame sequence is higher than the current reward weight of the second video frame, the first video frame is located in the middle of the video frame sequence, and the second video frame is located at the edge of the video frame sequence;

Step S2422, generating a target reward result based on the initial reward result and the target reward weight.

In the disclosed embodiment, after obtaining the initial reward result, when the time decay reward method is used to adjust the initial reward weight corresponding to the initial reward result, the time decay reward method can be used to perform differentiated adjustment on the initial reward weight corresponding to the initial reward result. By adjusting the weight of the middle frame of the video to a higher value and the weight of the edge frame to a lower value, effective and efficient fine-tuning can be achieved to obtain the target reward weight.

That is, the present disclosure can use a time decay reward method to adjust the current reward weight of the first video frame located in the middle of the video frame sequence to be higher than the current reward weight of the second video frame located at the edge of the video frame sequence. For example, taking a video with 16 frames as an example, the reward weight of each video frame is adjusted by time decay reward, and the coefficient of the middle video frame among the 16 frames is adjusted to 1, and the coefficient of the side video frame is adjusted to Thus generating the target reward result.

In an optional embodiment, a graphical user interface is provided by a terminal device, and the content displayed by the graphical user interface at least partially includes a video generation scene. The video generation method further includes:

Step S26, in response to the first touch operation on the graphical user interface, inputting a target text, wherein the target text is used to describe the video content to be generated;

Step S27, in response to the second touch operation on the graphical user interface, using the target video generation model to perform video generation processing on the target text to obtain a target video, wherein the target video generation model is a model obtained by aligning the initial video generation model with the preset reward model in a fine-tuning manner;

Step S28, displaying the target video in the graphical user interface.

In the embodiment of the present disclosure, at least a video generation scene is displayed in the graphical user interface, and the user can input the target text in the image encoding scene by executing control operations, and use the target video generation model to perform video generation processing on the target text to obtain the target video and other steps. It can be understood that the above video generation scene can be, but is not limited to, application scenes involving video generation in the fields of e-commerce, education, medical treatment, conferences, social networks, financial products, logistics, and navigation.

The graphical user interface also includes a first control (or a first touch area). When a first touch operation acting on the first control (or the first touch area) is detected, a target text input by the user can be obtained. The target text can be input by the user from a text box in the graphical user interface through the first touch operation. The first touch operation can be a point selection, a box selection, a check, a conditional screening, etc., which are not limited here.

The above-mentioned graphical user interface also includes a second control (or a second touch area), when the second control is detected When the second touch operation is performed on the target text input by the user (or the second touch area), the target video generation model can be used to generate a video for the target text input by the user to obtain the target video. The above second touch operation can be operations such as point selection, box selection, check selection, conditional screening, etc., which are not limited here.

After the target video is obtained, the target video can be displayed in a graphical user interface to provide feedback to the user.

It should be noted that the first touch operation and the second touch operation can both be operations in which a user touches the display screen of the terminal device with a finger and touches the terminal device. The touch operation can include single-point touch and multi-point touch, wherein the touch operation of each touch point can include click, long press, heavy press, swipe, etc. The first touch operation and the second touch operation can also be touch operations implemented by input devices such as a mouse and a keyboard, which are not limited here.

Figure 3 is a schematic diagram of the fine-tuning process according to Example 1 of the present disclosure, wherein z is the representation of the sampled video in the feature space. c is the text corresponding to the sampled video z. It can be understood that z and c together constitute a corresponding video-text pair. In Figure 3, the text data c is "Cornus flowers are flying in the air" and the video data z is the video corresponding to the text data c as an example. z_d(τ·D) represents the noisy video, wherein d() is a function for calculating how many steps the video should be noisy, D is the number of noisy steps, and τ is the noise level. z_0 is the representation of the generated video in the feature space, x_0^g is the representation of the generated video in the RGB space after sampling, and R is the calculated reward score.

As shown in FIG3 , before the fine-tuning phase begins, some video-text pairs are first selected from the pre-training data set for fine-tuning. After selecting some video-text pairs, a diffusion process with noise is implemented on the video data z in the video-text pairs, wherein the noise level is set to τ and the number of noise steps is D, that is, the video data z is subjected to noise processing through the diffusion model to obtain a noisy video z_d (τ·D). Then, based on this noise processing, the noisy video z_d (τ·D) and the corresponding text data c are resampled using DDIM sampling to generate a sampled video z_0. After the sampled video z_0 is generated, the present disclosure adopts an image-based human preference model as a reward model, that is, an image reward model is used for reward calculation, and when calculating the reward score, in order to improve the efficiency of the learning process, a segmented sampling and decoding method is used to segment the sampled video z_0, and the color space representation x_0^g of the segmented video frame and the segmented sampling result are obtained. Then, the reward of the segmented sampling result is calculated through the image reward model and the corresponding text data c, and the reward weight of each video frame is adjusted through the time decay reward (TAR) to achieve effective and efficient fine-tuning, so as to obtain the reward score R, that is, the target reward result.

It is understandable that after obtaining the reward score R, the noisy video z_d(1) can be obtained through gradient back propagation based on the reward score R to adjust the network parameters to minimize the error, thereby optimizing the model. I will not go into details here.

FIG4 is a schematic diagram of a video generation method according to Embodiment 1 of the present disclosure. As shown in FIG4, compared with the method of the related art, that is, generating a video by inputting text into a pre-trained U-shaped network with fixed model parameters, the generated video does not meet the user's expectations. The method of the present disclosure, through fine-tuning, aligns the video generation model adopted by the present disclosure with the human preference model of the image based on the text and the noisy video, that is, aligns the pre-trained U-shaped network model with trainable model parameters with the image reward model, and in the fine-tuning process, adopts an efficient fine-tuning method. The LoRA technology improves the performance of the model by making slight adjustments to the parameters of the model. In addition, it can also perform gradient processing optimization on the model based on the image reward model to obtain the target video generation model. Finally, the target video generation model obtained by the present disclosure is used to perform video generation processing on the input text, so as to obtain the target video that meets the user's expectations.

It can be seen that the method of the embodiment of the present disclosure can use the image reward model to learn human preferences for the video diffusion model without a video reward model (such a model has not been trained in the relevant technology), so that after the fine-tuning method designed by the present disclosure, the output results of the video diffusion model are more in line with user expectations and more popular with humans.

It is easy to understand that the beneficial effects of the video generation method provided by the present disclosure include the following points.

Beneficial effect (1): In order to align the video diffusion model with human preferences, the present disclosure proposes to align the video diffusion model with the human preference model of images, so that the video diffusion model obtained after alignment can fully consider human aesthetic preferences and content relevance, output video content that better meets user expectations, improve the personalization level of content, and open up new application prospects in the field of video generation;

Beneficial effect (2): The present disclosure proposes segmented video rewards and time decay rewards in video reward fine-tuning learning to enable the model to be effectively fine-tuned.

It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

In addition, it should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the present disclosure is not limited by the order of the actions described, because according to the present disclosure, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present disclosure.

Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the technical solution of the present disclosure, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM/RAM, a disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present disclosure.

Example 2

In the operating environment as in Example 1, the present disclosure provides a video generation method as shown in FIG5 . FIG5 is a flow chart of a video generation method according to Example 2 of the present disclosure. As shown in FIG5 , the method includes:

Step S51, obtaining a video generation call request through a first application programming interface, wherein the request data carried in the video generation call request includes: a target text, the target text is used to describe the video content to be generated, the video generation call request is used to request to call a target video generation model to perform video generation processing on the target text to obtain a target video, and the target video generation model is a model obtained by aligning an initial video generation model with a preset reward model in a fine-tuning manner;

Step S52: Return a video generation call response through the second application programming interface, wherein the response data carried in the video generation call response includes: the target video.

Both the first application programming interface and the second application programming interface can be understood as application programming interfaces (Application Programming Interface, API). By calling the first application programming interface, a video generation call request carrying the target text can be obtained, and by calling the second application programming interface, a video generation call response can be returned.

The first application programming interface and the second application programming interface in the embodiment of the present disclosure may be the same application programming interface or different application programming interfaces, which is not limited here.

The target text can be understood as the text input into the video generation model, such as the text input into the target video generation model in the embodiment of the present disclosure, that is, as the input of the target video generation model. The target text is used to describe the video content to be generated, for example, to describe the video content that the user expects to generate.

For example, if the video content that the user desires to generate is a video of Dogwood blossoms are blowing in the wind, the target text may be "狗玉花在风漂荡", or "狗木 开花s are blowing in the wind", etc. It is understandable that the target text may be described in natural language, such as Chinese, English, Japanese, etc., which is not limited here.

The initial video generation model can be understood as an initial model for generating videos based on text. For example, the initial video generation model can be a pre-trained U-type network (Pre-trained UNet), that is, a U-type network model pre-trained on a large-scale dataset. Considering that pre-training on a large-scale dataset can enable the model to learn rich image features and semantic information, thereby improving the generalization ability and accuracy of the model on specific tasks, the present disclosure uses a pre-trained U-type network model as the initial video generation model to make the video generated by the target video generation model more accurate.

The preset reward model may be a model used to evaluate the quality of generated videos, thereby guiding the video generation model to produce higher quality outputs. For example, the preset reward model may evaluate and score the generated videos, thereby quantifying the output quality or conformity of the video generation model through reward scores.

It should be noted that, considering that the final target video generation model should fully consider human aesthetic preferences and content relevance in order to generate videos that meet user expectations, the preset reward model in the embodiment of the present disclosure adopts an image-based human preference model, namely an image reward model. The image reward model can evaluate the quality of the generated video based on human aesthetic preferences and content relevance, so that the video generated by the final target video generation model is more in line with user expectations.

The target video generation model is a model obtained by aligning the initial video generation model with the preset reward model by fine-tuning the embodiment of the present disclosure, that is, a model obtained by aligning the pre-trained U-type network with the image reward model by fine-tuning the model, so that video content that meets user preferences and expectations can be accurately generated according to the target video generation model.

In the disclosed embodiment, a video generation call request can be obtained by calling a first application programming interface, and the video generation call request carries a target text for describing the content of the video to be generated. According to the obtained video generation call request, a target video generation model is used to perform video generation processing on the target text, so that a target video can be obtained, and then a video generation call response including the target video can be returned by calling a second application programming interface. According to this method, the generated target video can be made more consistent with human aesthetic preferences and the content of the target text, and the video quality of the generated target video can be improved, that is, a video that better meets the user's expectations can be obtained, thereby improving the personalization level of the generated video content, and opening up new application prospects in the field of video generation.

The above-mentioned video generation method provided by the embodiments of the present disclosure can be applied to, but is not limited to, application scenarios involving video generation in the fields of e-commerce services, educational services, legal services, medical services, conference services, social network services, financial product services, logistics services, and navigation services. For example: the scenario of generating product display content in e-commerce services, the scenario of generating learning content videos in educational services, the scenario of generating case-related videos in legal services, etc., which are not limited here.

By adopting the embodiment of the present disclosure, a video generation call request can be obtained by calling a first application programming interface, and the video generation call request carries a target text for describing the video content to be generated. According to the obtained video generation call request, a target video generation model is used to perform video generation processing on the target text, so that a target video can be obtained. Then, by calling a second application programming interface, a video generation call response including the target video can be returned, thereby achieving the purpose of generating a target video that meets user expectations, thereby achieving the technical effect of making the generated target video more consistent with human aesthetic preferences and target text content, improving the video quality of the generated target video, and making the generated target video more popular with humans, thereby solving the technical problem in the related technology of training a video generation model based on network data, resulting in the video generated by the trained video generation model having poor quality and not meeting user expectations.

It should be noted that the preferred implementation of this embodiment can refer to the relevant description in Example 1, which will not be repeated here.

Example 3

In the operating environment as in Example 1, the present disclosure provides a video generation method as shown in FIG6 . FIG6 is a flow chart of a video generation method according to Example 3 of the present disclosure. As shown in FIG6 , the method includes:

Step S61, obtaining a currently input video generation dialogue request, wherein the information carried in the video generation dialogue request includes: a target text, the target text is used to describe the video content to be generated;

Step S62, in response to the video generation dialogue request, returns a video generation dialogue reply, wherein the video generation The information carried in the dialogue response includes: a target video, which is obtained by processing the target text with a target video generation model, and the target video generation model is a model obtained by aligning the initial video generation model with the preset reward model by fine-tuning;

Step S63, displaying the video-generated dialogue response in the graphical user interface.

The video generation dialogue request can be understood as a dialogue request (request) initiated by a user to a computer or a robot, and the video generation dialogue request carries a target text for describing the video content to be generated.

The target text can be understood as the text input into the video generation model, such as the text input into the target video generation model in the embodiment of the present disclosure, that is, as the input of the target video generation model. The target text is used to describe the video content to be generated, for example, to describe the video content that the user expects to generate.

For example, if the video content that the user desires to generate is a video of Dogwood blossoms are blowing in the wind, the target text may be "狗玉花在风漂荡", or "狗木 开花s are blowing in the wind", etc. It is understandable that the target text may be described in natural language, such as Chinese, English, Japanese, etc., which is not limited here.

In the disclosed embodiment, in response to the acquired video generation dialogue request, a video generation dialogue reply (response) may be returned, wherein the video generation dialogue reply carries a target video obtained by performing video generation processing on the target text using a target video generation model, and the target video generation model is a model obtained by aligning the initial video generation model with a preset reward model using a fine-tuning method.

The initial video generation model can be understood as an initial model for generating videos based on text. For example, the initial video generation model can be a pre-trained U-type network (Pre-trained UNet), that is, a U-type network model pre-trained on a large-scale dataset. Considering that pre-training on a large-scale dataset can enable the model to learn rich image features and semantic information, thereby improving the generalization ability and accuracy of the model on specific tasks, the present disclosure uses a pre-trained U-type network model as the initial video generation model to make the video generated by the target video generation model more accurate.

The preset reward model may be a model used to evaluate the quality of generated videos, thereby guiding the video generation model to produce higher quality outputs. For example, the preset reward model may evaluate and score the generated videos, thereby quantifying the output quality or conformity of the video generation model through reward scores.

It should be noted that, considering that the final target video generation model should fully consider human aesthetic preferences and content relevance in order to generate videos that meet user expectations, the preset reward model in the embodiment of the present disclosure adopts an image-based human preference model, namely an image reward model. The image reward model can evaluate the quality of the generated video based on human aesthetic preferences and content relevance, so that the video generated by the final target video generation model is more in line with user expectations.

The target video generation model is a model obtained by aligning the initial video generation model with the preset reward model by fine-tuning the embodiment of the present disclosure, that is, a model obtained by aligning the pre-trained U-shaped network with the image reward model by fine-tuning the model, so as to be able to generate the target video according to the target. The video generation model accurately generates video content that conforms to user preferences and meets user expectations.

After returning to the video-generated dialogue response, the video-generated dialogue response can be displayed in the graphical user interface.

The above-mentioned video generation method provided by the embodiments of the present disclosure can be applied to, but is not limited to, application scenarios involving video generation in the fields of e-commerce services, educational services, legal services, medical services, conference services, social network services, financial product services, logistics services, and navigation services. For example: the scenario of generating product display content in e-commerce services, the scenario of generating learning content videos in educational services, the scenario of generating case-related videos in legal services, etc., which are not limited here.

According to the disclosed embodiment, a video generation dialogue request carrying a target text for describing the content of a video to be generated is obtained, and then in response to the video generation dialogue request, a video generation process is performed on the target text based on a target video generation model obtained by aligning the initial video generation model with a preset reward model in a fine-tuning manner, that is, a pre-trained U-shaped network is aligned with a picture reward model by fine-tuning to obtain a fine-tuned target video generation model, and a video is generated based on the target text using the fine-tuned target video generation model, thereby obtaining a target video, that is, a video generation dialogue response carrying the target video is obtained, and finally the video generation dialogue response is displayed in a graphical user interface, thereby achieving the purpose of generating a target video that meets the user's expectations, thereby achieving the technical effect of making the generated target video more consistent with human aesthetic preferences and the content of the target text, improving the video quality of the generated target video, and making the generated target video more popular with humans, thereby solving the technical problem in the related art of training a video generation model based on network data, resulting in a poor quality of the video generated by the trained video generation model that does not meet the user's expectations.

It should be noted that the preferred implementation of this embodiment can refer to the relevant description in Example 1, which will not be repeated here.

Example 4

According to an embodiment of the present disclosure, a device embodiment for implementing the above-mentioned video generation method is also provided. FIG7 is a structural schematic diagram of a video generation device according to Embodiment 4 of the present disclosure. As shown in FIG7 , the device includes:

An acquisition module 701 is configured to acquire a target text, wherein the target text is used to describe the video content to be generated;

The processing module 702 is configured to use a target video generation model to perform video generation processing on the target text to obtain a target video, wherein the target video generation model is a model obtained by aligning the initial video generation model with the preset reward model using a fine-tuning method.

Optionally, it also includes: a training module, which is configured to use training samples to sample the initial video generation model to generate a sampled video, wherein the training samples include: multiple video-text pairs, and the multiple video-text pairs all include: training videos and training texts, and the training texts are used to describe the video content of the training videos; using a preset reward model to calculate rewards for the sampled videos to obtain target reward results; based on the target reward results, adjusting the model parameters of the initial video generation model to generate a target video generation model.

Optionally, the training module is further configured to: perform noise processing on the training video to obtain a noisy video; and perform video resampling on the initial video generation model using the noisy video and the training text to generate a sampled video.

Optionally, the above-mentioned training module is also configured to: obtain the number of noise addition steps and the noise level corresponding to the training video, wherein the number of noise addition steps is used to determine the number of steps to be noised for the training video through a preset noise addition function, and the noise level is used to determine the degree of damage to the training video; and perform noise addition processing on the training video based on the number of noise addition steps and the noise level to obtain a noisy video.

Optionally, the above-mentioned training module is also configured to: use a preset reward model to calculate the reward for the sampled video to obtain an initial reward result; use a time-decayed reward method to adjust the initial reward weight corresponding to the initial reward result to generate a target reward result, wherein the initial reward weight is the default reward weight corresponding to the video frame sequence contained in the sampled video.

Optionally, the training module is further configured to: perform video segment sampling on the video frame sequence to obtain segment sampling results; and perform reward calculation on the segment sampling results using a preset reward model to obtain an initial reward result.

Optionally, the above training module is also configured to: obtain a feature space representation of a video frame sequence; perform video segmentation sampling on the feature space representation to obtain a color space representation of the segmented video frame; and determine the segmentation sampling result based on the color space representation.

Optionally, the above-mentioned training module is also configured to: use a time-decayed reward method to differentially adjust the initial reward weight corresponding to the initial reward result to obtain a target reward weight, wherein the target reward weight is used to indicate that the current reward weight of the first video frame in the video frame sequence is higher than the current reward weight of the second video frame, the first video frame is located in the middle position of the video frame sequence, and the second video frame is located at the edge position of the video frame sequence; generate the target reward result based on the initial reward result and the target reward weight.

Optionally, the target video generation model is a video diffusion model, and the preset reward model is an image reward model, wherein the image reward model is used to perform preference learning on the video diffusion model.

According to the disclosed embodiment, a target text for describing the content of a video to be generated is obtained, and then a video generation process is performed on the target text based on a target video generation model obtained by aligning an initial video generation model with a preset reward model by fine-tuning. That is, a pre-trained U-shaped network is aligned with a picture reward model by fine-tuning to obtain a fine-tuned target video generation model, and a video is generated based on the target text using the fine-tuned target video generation model, thereby obtaining a target video. This achieves the purpose of generating a target video that meets user expectations, thereby achieving a technical effect of making the generated target video more consistent with human aesthetic preferences and the content of the target text, improving the video quality of the generated target video, and making the generated target video more popular with humans. This further solves the technical problem in the related art of training a video generation model based on network data, resulting in a poor quality video generated by the trained video generation model that does not meet user expectations.

It should be noted that the acquisition module 701 and the processing module 702 correspond to step S21 and step S22 in Example 1. The examples and application scenarios implemented by the two modules and the corresponding steps are the same, but are not limited to the contents disclosed in the above-mentioned Example 1. It should be noted that the above-mentioned modules or units can be stored in a memory (for example, For example, a hardware component or software component in memory 104) and processed by one or more processors (for example, processors 102a, 102b, ..., 102n), the above module can also be run in the computer terminal 10 provided in Example 1 as part of the device.

According to an embodiment of the present disclosure, another device embodiment for implementing the above-mentioned video generation method is also provided. FIG8 is a structural schematic diagram of another video generation device according to embodiment 4 of the present disclosure. As shown in FIG8 , the device includes:

The acquisition module 801 is configured to acquire a video generation call request through a first application programming interface, wherein the request data carried in the video generation call request includes: a target text, the target text is used to describe the video content to be generated, the video generation call request is used to request to call a target video generation model to perform video generation processing on the target text to obtain a target video, and the target video generation model is a model obtained by aligning an initial video generation model with a preset reward model in a fine-tuning manner;

The return module 802 is configured to return a video generation call response through a second application programming interface, wherein the response data carried in the video generation call response includes: a target video.

By adopting the embodiment of the present disclosure, a video generation call request can be obtained by calling a first application programming interface, and the video generation call request carries a target text for describing the video content to be generated. According to the obtained video generation call request, a target video generation model is used to perform video generation processing on the target text, so that a target video can be obtained. Then, by calling a second application programming interface, a video generation call response including the target video can be returned, thereby achieving the purpose of generating a target video that meets user expectations, thereby achieving the technical effect of making the generated target video more consistent with human aesthetic preferences and target text content, improving the video quality of the generated target video, and making the generated target video more popular with humans, thereby solving the technical problem in the related technology of training a video generation model based on network data, resulting in the video generated by the trained video generation model having poor quality and not meeting user expectations.

It should be noted that the acquisition module 801 and the return module 802 correspond to step S51 and step S52 in Example 2, and the examples and application scenarios implemented by the two modules and the corresponding steps are the same, but are not limited to the contents disclosed in the above-mentioned Example 1. It should be noted that the above-mentioned modules or units can be hardware components or software components stored in a memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n), and the above-mentioned modules can also be run in the computer terminal 10 provided in Example 1 as part of the device.

According to an embodiment of the present disclosure, another device embodiment for implementing the above-mentioned video generation method is also provided. FIG9 is a structural schematic diagram of another video generation device according to embodiment 4 of the present disclosure. As shown in FIG9 , the device includes:

The acquisition module 901 is configured to acquire a currently input video generation dialogue request, wherein the information carried in the video generation dialogue request includes: a target text, the target text is used to describe the video content to be generated;

The first response module 902 is configured to respond to the video generation dialogue request and return a video generation dialogue reply. The information carried in the video generation dialogue response includes: a target video, which is obtained by performing video generation processing on the target text using a target video generation model, and the target video generation model is a model obtained by aligning the initial video generation model with the preset reward model using a fine-tuning method;

The display module 903 is configured to display the video-generated dialogue response in the graphical user interface.

According to the disclosed embodiment, a video generation dialogue request carrying a target text for describing the content of a video to be generated is obtained, and then in response to the video generation dialogue request, a video generation process is performed on the target text based on a target video generation model obtained by aligning the initial video generation model with a preset reward model in a fine-tuning manner, that is, a pre-trained U-shaped network is aligned with a picture reward model by fine-tuning to obtain a fine-tuned target video generation model, and a video is generated based on the target text using the fine-tuned target video generation model, thereby obtaining a target video, that is, a video generation dialogue response carrying the target video is obtained, and finally the video generation dialogue response is displayed in a graphical user interface, thereby achieving the purpose of generating a target video that meets the user's expectations, thereby achieving the technical effect of making the generated target video more consistent with human aesthetic preferences and the content of the target text, improving the video quality of the generated target video, and making the generated target video more popular with humans, thereby solving the technical problem in the related art of training a video generation model based on network data, resulting in a poor quality of the video generated by the trained video generation model that does not meet the user's expectations.

It should be noted that the acquisition module 901, the first response module 902 and the display module 903 correspond to steps S61 to S63 in Example 3, and the three modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above-mentioned Example 1. It should be noted that the above-mentioned modules or units may be hardware components or software components stored in a memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n), and the above-mentioned modules may also be part of the device and may be run in the computer terminal 10 provided in Example 1.

It should be noted that the preferred implementation scheme involved in the above embodiments of the present disclosure is the same as the scheme provided in Example 1, as well as the application scenario and implementation process, but is not limited to the scheme provided in Example 1.

Example 5

The embodiment of the present disclosure may provide a computer terminal, which may be any computer terminal device in a computer terminal group. Optionally, in this embodiment, the computer terminal may also be replaced by a terminal device such as a mobile terminal.

Optionally, in this embodiment, the computer terminal may be located in at least one network device among a plurality of network devices of the computer network.

In this embodiment, the above-mentioned computer terminal can execute the program code of the following steps in the video generation method: obtaining a target text, wherein the target text is used to describe the video content to be generated; using a target video generation model to perform video generation processing on the target text to obtain a target video, wherein the target video generation model is a model obtained by aligning the initial video generation model with a preset reward model using a fine-tuning method.

Optionally, FIG10 is a structural block diagram of a computer terminal according to Embodiment 5 of the present disclosure. As shown, the computer terminal A may include: one or more (only one is shown in the figure) processors 1002, a memory 1004, a storage controller, and a peripheral interface, wherein the peripheral interface is connected to a radio frequency module, an audio module, and a display.

Among them, the memory can be configured to store software programs and modules, such as program instructions/modules corresponding to the video generation method and device in the embodiment of the present disclosure, and the processor executes various functional applications and data processing by running the stored software programs and modules, that is, realizing the above-mentioned video generation method. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the computer terminal A via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

The processor can call the information and application stored in the memory through the transmission device to execute the following steps: obtain the target text, wherein the target text is used to describe the video content to be generated; use the target video generation model to perform video generation processing on the target text to obtain the target video, wherein the target video generation model is a model obtained by aligning the initial video generation model with the preset reward model using a fine-tuning method.

Optionally, the processor may also execute the program code of the following steps: using training samples to perform sampling processing on the initial video generation model to generate a sampled video, wherein the training samples include: multiple video-text pairs, and the multiple video-text pairs each include: a training video and a training text, and the training text is used to describe the video content of the training video; using a preset reward model to calculate the reward for the sampled video to obtain a target reward result; adjusting the model parameters of the initial video generation model based on the target reward result to generate a target video generation model.

Optionally, the processor may also execute the program code of the following steps: performing noise processing on the training video to obtain a noisy video; and performing video resampling on the initial video generation model using the noisy video and the training text to generate a sampled video.

Optionally, the processor may also execute the following program code: obtaining the number of noise addition steps and the noise level corresponding to the training video, wherein the number of noise addition steps is used to determine the number of steps to be noised for the training video through a preset noise addition function, and the noise level is used to determine the degree of damage to the training video; performing noise addition processing on the training video based on the number of noise addition steps and the noise level to obtain a noisy video.

Optionally, the processor may also execute the program code of the following steps: using a preset reward model to calculate rewards for the sampled video to obtain an initial reward result; using a time-decayed reward method to adjust an initial reward weight corresponding to the initial reward result to generate a target reward result, wherein the initial reward weight is a default reward weight corresponding to a video frame sequence contained in the sampled video.

Optionally, the processor may also execute the program code of the following steps: performing video segment sampling on the video frame sequence to obtain segment sampling results; performing reward calculation on the segment sampling results using a preset reward model to obtain an initial reward result.

Optionally, the processor may further execute the program code of the following steps: obtaining a feature space of a video frame sequence; The method comprises the following steps: performing video segmentation sampling on the feature space representation to obtain a color space representation of the segmented video frame; and determining the segmentation sampling result based on the color space representation.

Optionally, the processor may also execute the program code of the following steps: using a time-decayed reward method to perform differentiated adjustments on the initial reward weight corresponding to the initial reward result to obtain a target reward weight, wherein the target reward weight is used to indicate that the current reward weight of the first video frame in the video frame sequence is higher than the current reward weight of the second video frame, the first video frame is located in the middle of the video frame sequence, and the second video frame is located at the edge of the video frame sequence; generating the target reward result based on the initial reward result and the target reward weight.

Optionally, the target video generation model is a video diffusion model, and the preset reward model is an image reward model, wherein the image reward model is used to perform preference learning on the video diffusion model.

According to the disclosed embodiment, a target text for describing the content of a video to be generated is obtained, and then a video generation process is performed on the target text based on a target video generation model obtained by aligning an initial video generation model with a preset reward model by fine-tuning. That is, a pre-trained U-shaped network is aligned with a picture reward model by fine-tuning to obtain a fine-tuned target video generation model, and a video is generated based on the target text using the fine-tuned target video generation model, thereby obtaining a target video. This achieves the purpose of generating a target video that meets user expectations, thereby achieving a technical effect of making the generated target video more consistent with human aesthetic preferences and the content of the target text, improving the video quality of the generated target video, and making the generated target video more popular with humans. This further solves the technical problem in the related art of training a video generation model based on network data, resulting in a poor quality video generated by the trained video generation model that does not meet user expectations.

It can be understood by those skilled in the art that the structure shown in FIG. 10 is for illustration only, and the computer terminal A may also be a terminal device such as a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, and a mobile Internet device (Mobile Internet Devices, MID), PAD, etc. FIG. 10 does not limit the structure of the above-mentioned electronic device. For example, the computer terminal A may also include more or fewer components (such as a network interface, a display device, etc.) than those shown in FIG. 10, or have a configuration different from that shown in FIG. 10.

A person of ordinary skill in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing the hardware related to the terminal device through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

Example 6

The embodiment of the present disclosure further provides a computer-readable storage medium. Optionally, in this embodiment, the computer-readable storage medium can be used to store the program code executed by the video generation method provided in the first embodiment.

Optionally, in this embodiment, the computer-readable storage medium may be located in any one of the computer terminals in a computer terminal group in a computer network, or in any one of the mobile terminals in a mobile terminal group.

Optionally, in this embodiment, the computer readable storage medium is configured to store the following steps: Program code: Obtain a target text, wherein the target text is used to describe the content of the video to be generated; use a target video generation model to perform video generation processing on the target text to obtain a target video, wherein the target video generation model is a model obtained by aligning an initial video generation model with a preset reward model using a fine-tuning method.

Optionally, in this embodiment, the computer-readable storage medium is configured to store program codes for executing the following steps: sampling the initial video generation model using training samples to generate a sampled video, wherein the training samples include: multiple video-text pairs, and the multiple video-text pairs each include: a training video and a training text, and the training text is used to describe the video content of the training video; using a preset reward model to calculate rewards for the sampled video to obtain a target reward result; adjusting the model parameters of the initial video generation model based on the target reward result to generate a target video generation model.

Optionally, in this embodiment, the computer-readable storage medium is configured to store program codes for executing the following steps: performing noise processing on the training video to obtain a noisy video; and performing video resampling on the initial video generation model using the noisy video and the training text to generate a sampled video.

Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for executing the following steps: obtaining the number of noise addition steps and the noise level corresponding to the training video, wherein the number of noise addition steps is used to determine the number of steps to be noised for the training video through a preset noise addition function, and the noise level is used to determine the degree of damage to the training video; performing noise addition processing on the training video based on the number of noise addition steps and the noise level to obtain a noisy video.

Optionally, in this embodiment, the computer-readable storage medium is configured to store program codes for executing the following steps: using a preset reward model to calculate rewards for the sampled video to obtain an initial reward result; using a time-decayed reward method to adjust an initial reward weight corresponding to the initial reward result to generate a target reward result, wherein the initial reward weight is a default reward weight corresponding to a video frame sequence contained in the sampled video.

Optionally, in this embodiment, the computer-readable storage medium is configured to store program codes for executing the following steps: performing video segment sampling on a video frame sequence to obtain segment sampling results; and performing reward calculation on the segment sampling results using a preset reward model to obtain an initial reward result.

Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for executing the following steps: obtaining a feature space representation of a video frame sequence; performing video segmentation sampling on the feature space representation to obtain a color space representation of the segmented video frame; and determining the segmentation sampling result based on the color space representation.

Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for executing the following steps: using a time-decayed reward method to differentially adjust the initial reward weight corresponding to the initial reward result to obtain a target reward weight, wherein the target reward weight is used to indicate that the current reward weight of the first video frame in the video frame sequence is higher than the current reward weight of the second video frame, the first video frame is located in the middle position of the video frame sequence, and the second video frame is located at the edge position of the video frame sequence; generating the target reward result based on the initial reward result and the target reward weight.

Optionally, the target video generation model is a video diffusion model, and the preset reward model is an image reward model, wherein the image reward model is used to perform preference learning on the video diffusion model.

The serial numbers of the above-mentioned embodiments of the present disclosure are only for description and do not represent the advantages or disadvantages of the embodiments.

In the above embodiments of the present disclosure, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

In the several embodiments provided in the present disclosure, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

The above is only a preferred embodiment of the present disclosure. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present disclosure. These improvements and modifications should also be regarded as the scope of protection of the present disclosure.

Claims (13)

A video generation method, comprising: Obtaining a target text, wherein the target text is used to describe the video content to be generated; The target text is processed by video generation using a target video generation model to obtain a target video, wherein the target video generation model is a model obtained by aligning an initial video generation model with a preset reward model using a fine-tuning method. The video generation method according to claim 1, wherein the video generation method further comprises: The initial video generation model is sampled using training samples to generate sampled videos, wherein the training samples include: a plurality of video-text pairs, each of the plurality of video-text pairs includes: a training video and a training text, and the training text is used to describe the video content of the training video; Using the preset reward model to calculate the reward for the sampled video to obtain a target reward result; The model parameters of the initial video generation model are adjusted based on the target reward result to generate the target video generation model. The video generation method according to claim 2, wherein the initial video generation model is sampled using the training sample to generate the sampled video, comprising: Performing noise processing on the training video to obtain a noisy video; The noisy video and the training text are used to perform video resampling on the initial video generation model to generate the sampled video. The video generation method according to claim 3, wherein the step of performing noise processing on the training video to obtain the noisy video comprises: Obtaining the number of noise adding steps and the noise level corresponding to the training video, wherein the number of noise adding steps is used to determine the number of steps to be noise added to the training video through a preset noise adding function, and the noise level is used to determine the degree of damage to the training video; The training video is subjected to noise addition processing based on the noise addition step number and the noise level to obtain the noisy video. The video generation method according to claim 2, wherein the preset reward model is used to perform reward calculation on the sampled video to obtain the target reward result, comprising: Using the preset reward model to calculate the reward for the sampled video to obtain an initial reward result; An initial reward weight corresponding to the initial reward result is adjusted by adopting a time decay reward method to generate the target reward result, wherein the initial reward weight is a default reward weight corresponding to the video frame sequence included in the sampled video. The video generation method according to claim 5, wherein the reward calculation for the sampled video is performed using the preset reward model to obtain the initial reward result comprises: Performing video segment sampling on the video frame sequence to obtain segment sampling results; The preset reward model is used to calculate the reward for the segmented sampling result to obtain the initial reward result. The video generation method according to claim 6, wherein performing video segment sampling on the video frame sequence to obtain the segment sampling result comprises: Obtaining a feature space representation of the video frame sequence; Performing video segment sampling on the feature space representation to obtain a color space representation of the segmented video frame; The segment sampling result is determined based on the color space representation. The video generation method according to claim 7, wherein the time decay reward method is used to adjust the initial reward weight corresponding to the initial reward result, and generating the target reward result comprises: The time decay reward method is used to differentially adjust the initial reward weight corresponding to the initial reward result to obtain a target reward weight, wherein the target reward weight is used to indicate that the current reward weight of the first video frame in the video frame sequence is higher than the current reward weight of the second video frame, the first video frame is located in the middle of the video frame sequence, and the second video frame is located at the edge of the video frame sequence; The target reward result is generated based on the initial reward result and the target reward weight. According to the video generation method according to claim 1, wherein the target video generation model is a video diffusion model, and the preset reward model is an image reward model, wherein the image reward model is used to perform preference learning on the video diffusion model. A video generation method, comprising: Obtaining a video generation call request through a first application programming interface, wherein the request data carried in the video generation call request includes: a target text, the target text is used to describe the video to be generated The video generation call request is used to request to call a target video generation model to perform video generation processing on the target text to obtain a target video, and the target video generation model is a model obtained by aligning the initial video generation model with the preset reward model in a fine-tuning manner; A video generation call response is returned through the second application programming interface, wherein the response data carried in the video generation call response includes: the target video. A video generation method, comprising: Acquire a currently input video generation dialogue request, wherein the information carried in the video generation dialogue request includes: a target text, wherein the target text is used to describe the video content to be generated; In response to the video generation dialogue request, a video generation dialogue reply is returned, wherein the information carried in the video generation dialogue reply includes: a target video, the target video is obtained by performing video generation processing on the target text using a target video generation model, and the target video generation model is a model obtained by aligning the initial video generation model with the preset reward model in a fine-tuning manner; Presenting the video within a graphical user interface generates a conversation response. An electronic device, comprising: A memory storing an executable program; A processor, configured to run the program, wherein the program executes the method according to any one of claims 1 to 11 when running. A computer-readable storage medium, the computer-readable storage medium comprising a stored executable program, wherein when the executable program is run, the device where the computer-readable storage medium is located is controlled to execute the method described in any one of claims 1 to 11.