CN109936766B - An end-to-end audio generation method for water scenes - Google Patents

Abstract

The invention belongs to the technical field of audio processing, and in particular relates to an end-to-end water scene audio generation method, comprising the following steps: Step 1, select various water scene videos, and perform preprocessing; The data of the generator model is obtained through training; step 3, the silent video is preprocessed, loaded into the trained generator model, and the audio corresponding to the silent video is output; step 4, the envelope is generated according to the sequence of the audio, and loaded To the trained timbre enhancer model, output the timbre-enhanced audio. The invention can realize the automatic generation of end-to-end outdoor water scene sound, solve the problem of time-consuming and laborious dubbing of the scene, and at the same time, use the model obtained by training to generate water scene audio, which can improve the generation speed and synchronization, thereby improving work efficiency .

Description

An end-to-end audio generation method for water scenes

technical field

The invention belongs to the technical field of audio processing, and in particular relates to an end-to-end audio generation method for water scenes.

Background technique

With the continuous development of computer graphics technology, people put forward higher requirements for the sound quality of video and animation. Water scenes, especially outdoor water scenes exist in movies and games, so it is very necessary to develop a method that can automatically generate corresponding scene sounds based on outdoor water scene videos. Currently, people mostly use physics-based methods to generate the sound of water scenes.

The physics-based sound generation method for water scenes is mainly based on a theory that the formation and resonance of air bubbles are the most important sources of water sound. Zheng et al. proposed a harmonic bubble-based water flow sound generation method in the harmonic bubble. By considering the sound propagation process, it generated a variety of flowing water sounds including faucet flowing water, but the generated results need to go through After tedious artificial adjustment, Langlois et al. proposed a sound generation method based on two-phase incompressible fluid simulation in the water simulation based on complex acoustic bubbles. The bubble no longer adopts a random model, but generates more realistic bubbles according to the state of the fluid, which also makes the final sound effect more realistic, but the main research objects of these methods are limited to small-scale water flow, and, with the sound The continuous improvement of the results and the increasing complexity of the algorithms make them unable to be applied to the sound synthesis of outdoor water scenes.

The sound generation method of deep learning, based on the video to generate the corresponding sound. Owens et al. proposed a neural network composed of a convolutional neural network (CNN) and a long short-term memory unit (LSTM) in the visual representation of the sound. The image features of the composed spacetime map, output the sound cochlear map corresponding to the video, and then go to the sound library to find the sound sample that best matches this map to splicing to generate the final result. Chen et al. proposed in deep cross-modal audiovisual generation Using the GAN network, two conversion modes are designed, which convert the logarithmic amplitude Mel-spectrogram (LMS) of the input instrument sound into the corresponding instrument diagram, and convert the instrument diagram into the corresponding LMS diagram, and then look for a match with the LMS. The output of the deep network of these two algorithms is similar to the spectrogram of the image, and does not directly generate the original sound signal. Zhou et al. in Video to Sound: Sound Generation for Outdoor Video proposed to use the SampleRNN model to The sound of natural scene video is tentatively generated, and the corresponding sound signal is directly generated by extracting the features of the video image or optical flow graph as the input of the RNN. However, there are still some problems in the synchronization of audio and video.

SUMMARY OF THE INVENTION

The object of the present invention is to: aiming at the deficiencies of the prior art, provide a kind of generation method based on end-to-end water scene audio, can realize the automatic generation of end-to-end outdoor water scene sound, solve the problem of time-consuming and laborious dubbing for the scene, At the same time, using the trained model to generate water scene audio can improve the generation speed and synchronization, thereby improving work efficiency.

In order to achieve the above object, the present invention adopts the following technical solutions:

An end-to-end water scene audio generation method, comprising the following steps:

Step 1, select all kinds of water scene videos, and perform preprocessing;

Step 2: Obtain a generator model through training according to the preprocessed data;

Step 3, preprocess the silent video, load it into the trained generator model, and output the audio corresponding to the silent video;

Step 4: Generate an envelope according to the audio sequence, load it into the trained timbre enhancer model, and output the timbre-enhanced audio.

It should also be noted that: in the generation method of the present invention, in step 1, various types of water scene videos are selected for training, which is helpful for optimizing the training of the model and reducing errors. If there is a large dimensional difference, the image information and sound can be in the same dimension through preprocessing; in step 2, by training the generator model on the preprocessed data, the fluid sound synchronized with the video of the outdoor water scene can be automatically synthesized. There is no need for a professional foley engineer to synthesize the synchronized water scene sounds, and there is no need to manually design different algorithms to generate the sounds of various scenes according to different scene characteristics, which saves manpower and material resources and improves the accuracy of the generator model. At the same time, a discriminator needs to be set up to evaluate the quality of the generated results of the generator, and the evaluation results are fed back to the generator model. After many feedback and adjustment processes, the generator model can achieve Effectively train the generator model, thereby improving the accuracy of the generator model, and adding sound to the silent video synchronously; in step 3, the silent video does not have sound, and the trained generator model needs to be based on the information vector of each second of the silent video. , generate the corresponding audio data, so as to complete the sound for the silent video; in step 4, since the audio data output by the generator model may not conform to the actual water scene, such as the waterfall scene, the sound quality needs to be enhanced to conform to the actual water scene. At the same time, in order to further improve the level of automation, the trained timbre enhancer model is also used to enhance the timbre to realize the automatic generation of end-to-end outdoor water scene sounds. The trained timbre enhancer model can be based on the envelope of the sound, The enhanced audio is directly obtained, eliminating the intermediate physical methods, such as image method, comparison method, synthesis method, control variable method and transformation method, etc., which greatly improves the processing speed and reduces the waiting time of users.

As an improvement of the end-to-end water scene audio generation method of the present invention, in step 1, the preprocessing method includes the following steps:

A1. Extract the features of the video frame and obtain the information of the video;

A2. Convert the video information per second into a vector with the same dimension as the audio.

As an improvement of the end-to-end water scene audio generation method according to the present invention, in step 2, the training method of the generator model includes the following steps:

B1, input the vector of described video information, output audio signal through described generator model;

B2. Evaluate the audio signal, and if it does not correspond, feed it back to the generator model, and re-adjust until the corresponding audio signal is output; if it corresponds, continue to train the next video information.

As an improvement of the end-to-end water scene audio generation method according to the present invention, in step 4, the training method of the timbre enhancer model includes the following steps:

C1, input the envelope of target audio frequency, output the sequence of described audio frequency through described timbre enhancer model;

C2. Evaluate the audio sequence, and if it is not the target sequence, feed it back to the timbre enhancer model, and re-adjust until the target audio sequence is output; if it is the target sequence, proceed to the next timbre enhancement training.

As an improvement of the end-to-end water scene audio generation method of the present invention, in step 4, the envelope generation method includes the following steps:

D1. Input a segment of audio sequence G _V and the sampling interval L _step of the envelope;

D2. Take the maximum value of the absolute value in each sampling interval L _step in the audio sequence G _V as an envelope point pi in this interval;

D3. The array Ep formed by connecting the envelope points pi in all sampling intervals is linearly interpolated to form a sequence E(1:len) with the same length as G _V , which is the envelope corresponding to the audio sequence G _V ,

表示连接操作。Among them, Pi∈Gv, interp() represents linear interpolation,

Represents a connect operation.

As an improvement of the method for generating end-to-end water scene audio according to the present invention, in step A2, the video information conversion formula is:

G(y ₁ , ..., y _m ) → x ₁ , ..., x _n , x ∈ {audio), y ∈ {video}

Where y1,...,ym represents the color channel information of the video frame, each channel is a matrix composed of numbers between 0 and 255, G(y1,...,ym) represents the video frame based on The value of the generated audio signal (the value range is -1 to 1), x1, . . . , xn represents the value of the audio signal corresponding to the video (the change range is -1 to 1).

As an improvement of the end-to-end water scene audio generation method according to the present invention, in step 2, the loss function used for outputting the audio signal is:

其中，λ＝100，其中，X表示声音真实值，V表示视频帧信息，G表示生成器生成的结果，D表示评估的结果，E表示求均值。

Among them, λ=100, where X represents the real sound value, V represents the video frame information, G represents the result generated by the generator, D represents the evaluation result, and E represents the mean value.

As an improvement of the end-to-end water scene audio generation method according to the present invention, in step 2, the loss function used for evaluating the audio signal is:

其中，V表示视频帧信息，G表示生成器生成的结果，D表示评估的结果，E表示求均值。

Among them, V represents the video frame information, G represents the result generated by the generator, D represents the evaluation result, and E represents the mean value.

As an improvement of the end-to-end water scene audio generation method of the present invention, the water scene audio generation method is based on a GAN network, and the GAN network includes a generator, a discriminator and a timbre enhancer.

As an improvement of the end-to-end water scene audio generation method of the present invention, the water scene audio generation method is based on a GAN network. In step 1, the vector V _t generated by the preprocessed video frame is It can be expressed in the following form:

表示连接操作，

v_t,q表示第t秒的第q帧所提取的特征，Floor表示向下取整；in,

represents the connection operation,

v _t,q represents the feature extracted from the qth frame of the tth second, and Floor represents the rounding down;

The sound generation task can be further expressed as the following form:

G(V ₁ , V ₂ , ..., V _Δt )→X ₁ , X ₂ , ..., X _Δt

where X _t = {X _{t, 1} , x _{t, 2} , ..., X _t , SR _audio }, t∈{1, 2, ..., Δt}.

The beneficial effect of the present invention is that the present invention includes the following steps: step 1, selecting various water scene videos and preprocessing; step 2, obtaining a generator model through training according to the preprocessed data; step 3, adding the silent The video is preprocessed, loaded into the trained generator model, and the audio corresponding to the silent video is output; in step 4, the envelope is generated according to the audio sequence, and loaded into the trained timbre enhancer model, and the timbre-enhanced audio is output . In the generation method of the present invention, in step 1, various water scene videos are selected for training, which helps to optimize the training of the model and reduce errors. At the same time, because there is a large dimensional difference between the image information of the video and the sound , through preprocessing, the image information and the sound can be in the same dimension; in step 2, by training the generator model on the preprocessed data, the fluid sound synchronized with the video of the outdoor water scene can be automatically synthesized, without the need for professional simulation. The sound engineer synthesizes the synchronized water scene sounds, and does not need to manually design different algorithms according to different scene characteristics to generate the sounds of various scenes, saving manpower and material resources, and improving the accuracy of the generator model to meet people's needs. At the same time, it is also necessary to set a discriminator to evaluate the quality of the results generated by the generator, and to feed the evaluation results back to the generator model. Effective training, thereby improving the accuracy of the generator model, and adding sound to the silent video synchronously; in step 3, the silent video does not have sound, and the trained generator model needs to generate the corresponding audio according to the information vector of the silent video every second In step 4, since the audio data output by the generator model may not meet the actual water scene, such as the waterfall scene, the sound quality needs to be enhanced to meet the needs of the actual water scene. At the same time, in order to To further improve the level of automation, the trained timbre enhancer model is also used to enhance the timbre to realize the automatic generation of end-to-end outdoor water scene sounds. The trained timbre enhancer model can directly obtain the enhanced sound according to the envelope of the sound. Audio, without intermediate physical methods, such as image method, comparison method, synthesis method, control variable method and transformation method, etc., greatly improve the processing speed and reduce the waiting time of users. The invention can realize the automatic generation of end-to-end outdoor water scene sound, solve the problem of time-consuming and laborious dubbing of the scene, and at the same time, use the model obtained by training to generate water scene audio, which can improve the generation speed and synchronization, thereby improving work efficiency .

Description of drawings

Fig. 1 is the schematic flow chart of the present invention;

Fig. 2 is the working schematic diagram of the present invention;

Fig. 3 is the waveform diagram of the water scene in the present invention and its corresponding audio signal;

FIG. 4 is a spectrum comparison diagram before and after timbre enhancement in the present invention.

Detailed ways

As used in the specification and claims, certain terms are used to refer to particular components. It should be understood by those skilled in the art that hardware manufacturers may refer to the same component by different nouns. The description and claims do not use the difference in name as a way to distinguish components, but use the difference in function of the components as a criterion for distinguishing. As mentioned in the entire specification and claims, "comprising" is an open-ended term, so it should be interpreted as "including but not limited to". "Approximately" means that within an acceptable error range, those skilled in the art can solve technical problems within a certain error range, and basically achieve technical effects.

In the description of the present invention, it should be understood that the orientation or positional relationship indicated by the terms "upper", "lower", "front", "rear", "left", "right", horizontal" etc. is based on the accompanying drawings The orientation or positional relationship shown is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the indicated device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be construed as a reference to the present invention. limits.

In the invention, unless otherwise expressly specified and limited, the terms "installation", "connection", "connection", "fixation" and other terms should be understood in a broad sense, for example, it may be a fixed connection or a detachable connection, Or integrally connected; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, and it can be the internal communication between the two components. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific situations.

The present invention will be further described in detail below in conjunction with accompanying drawings 1 to 4, but it is not intended to limit the present invention.

Example 1

An end-to-end water scene audio generation method, comprising the following steps:

Step 1, select all kinds of water scene videos, and perform preprocessing;

Step 2: Obtain a generator model through training according to the preprocessed data;

Step 3: Preprocess the silent video, load it into the trained generator model, and output the audio corresponding to the silent video;

In step 4, the envelope is generated according to the audio sequence, and loaded into the trained timbre enhancer model, and the timbre-enhanced audio is output.

It should also be noted that: in the generation method of the present invention, in step 1, various types of water scene videos are selected for training, which is helpful for optimizing the training of the model and reducing errors. If there is a large dimensional difference, the image information and sound can be in the same dimension through preprocessing; in step 2, by training the generator model on the preprocessed data, the fluid sound synchronized with the video of the outdoor water scene can be automatically synthesized. There is no need for a professional foley engineer to synthesize the synchronized water scene sounds, and there is no need to manually design different algorithms to generate the sounds of various scenes according to different scene characteristics, which saves manpower and material resources and improves the accuracy of the generator model. At the same time, a discriminator needs to be set up to evaluate the quality of the generated results of the generator, and the evaluation results are fed back to the generator model. After many feedback and adjustment processes, the generator model can achieve Effectively train the generator model, thereby improving the accuracy of the generator model, and adding sound to the silent video synchronously; in step 3, the silent video does not have sound, and the trained generator model needs to be based on the information vector of each second of the silent video. , generate the corresponding audio data, so as to complete the sound for the silent video; in step 4, since the audio data output by the generator model may not conform to the actual water scene, such as the waterfall scene, the sound quality needs to be enhanced to conform to the actual water scene. At the same time, in order to further improve the level of automation, the trained timbre enhancer model is also used to enhance the timbre to realize the automatic generation of end-to-end outdoor water scene sounds. The trained timbre enhancer model can be based on the envelope of the sound, The enhanced audio is directly obtained, eliminating the intermediate physical methods, such as image method, comparison method, synthesis method, control variable method and transformation method, etc., which greatly improves the processing speed and reduces the waiting time of users.

Preferably, in step 1, the method for preprocessing includes the following steps:

A1. Extract the features of the video frame and obtain the information of the video;

A2. Convert the video information per second into a vector with the same dimension as the audio.

In the above preprocessing method, in step A1, because the complete water scene video occupies a larger memory space, which is not conducive to obtaining video information, and the amount of calculation is large, the amount of calculation can be reduced by extracting the characteristics of the video frame, At the same time, the purpose of obtaining the information of the video is achieved, and the operation speed is improved; in step A2, due to the large dimensional difference between the image information of the video and the sound, it not only requires a lot of calculation, but also increases the error of the generator model and reduces the water scene. The effect of pairing sound with video.

Preferably, in step 2, the training method of the generator model includes the following steps:

B1. Input the vector of video information, and output the audio signal through the generator model;

B2. Evaluate the audio signal, and if it does not correspond, feed it back to the generator model, and adjust it again until the corresponding audio signal is output; if it corresponds, continue the training of the next video information.

In the above training method, in step B2, the initial generator model has not been trained, and the output audio signal may not correspond to the vector of frequency information one-to-one. It is trained through various water scene videos and fed back to the generator model in real time. Helps to optimize the training of the model and reduce the error of the output.

Preferably, in step 4, the training method of the timbre enhancer model includes the following steps:

C1. Input the envelope of the target audio, and output the audio sequence through the timbre enhancer model;

C2. Evaluate the audio sequence. If it is not the target sequence, feed it back to the timbre enhancer model and re-adjust until the target audio sequence is output; if it is the target sequence, proceed to the next timbre enhancement training.

In the above training method, in step C2, the initial timbre enhancer model has not been trained, and the output audio sequence may not correspond to the envelope of the target audio. It helps to optimize the training of the model and reduce the error of the output.

Preferably, in step 4, the method for generating the envelope includes the following steps:

D1. Input a segment of audio sequence G _V and the sampling interval L _step of the envelope;

D2. Take the maximum value of the absolute value in each sampling interval L _step in the audio sequence G _V as an envelope point pi in this interval;

D3. The array Ep formed by connecting the envelope points pi in all sampling intervals is linearly interpolated to form a sequence E(1:len) with the same length as G _V , which is the envelope corresponding to the audio sequence G _V ,

表示连接操作。Among them, Pi∈Gv, interp() represents linear interpolation,

Represents a connect operation.

Preferably, in step A2, the video information conversion formula is:

G(y ₁ , ..., y _m ) → x ₁ , ..., x _n , x ∈ {audio}, y ∈ {video}

where y ₁ ,...,ym represents the color channel information of the video frame, each channel is a matrix composed of numbers between 0 and 255, G(y1,...,ym) represents the video-based The value of the audio signal generated by the frame (value range is -1 to 1), x1, . . . , xn represent the value of the audio signal corresponding to the video (the change range is -1 to 1).

Preferably, in step 2, the loss function used to output the audio signal is:

其中，λ＝100，其中，X表示声音真实值，V表示视频帧信息，G表示生成器生成的结果，D表示评估的结果，E表示求均值。

Among them, λ=100, where X represents the real sound value, V represents the video frame information, G represents the result generated by the generator, D represents the evaluation result, and E represents the mean value.

Preferably, in step 2, the loss function used for evaluating the audio signal is:

其中，V表示视频帧信息，G表示生成器生成的结果，D表示评估的结果，E表示求均值。

Among them, V represents the video frame information, G represents the result generated by the generator, D represents the evaluation result, and E represents the mean value.

Example 2

The difference from Embodiment 1 is that in the video preprocessing of this embodiment, the image sizes of different input videos are usually different. 3 images, and then encode 30 256×256×3 images per second into 1×4096×1 corresponding to the audio scale. First, for each video frame _yi , extract its feature vector vi under the _VGG19 network, and its dimension is 1×4096×1. Let SR _videc and SR _audio be the sampling rates of video and audio, which are 30 and 44100 in the present invention. For the video of the t second second, the corresponding video preprocessed vector V _t can be expressed as the following form:

表示连接操作，

v_t,q表示第t秒的第q帧所提取的VGG19特征，Floor表示向下取整。所以，在本发明中p ＝10，q＝3。对于拼接中由于四舍五入所导致的最终长度的缺失，本发明均匀的在空缺处补零。如此，原本的视频到音频的转换可以表示为如下形式：in,

represents the connection operation,

v _t,q represents the VGG19 feature extracted from the qth frame of the tth second, and Floor represents the rounding down. Therefore, in the present invention, p=10, q=3. For the lack of the final length due to rounding in the splicing, the present invention evenly fills the vacancy with zeros. In this way, the original video-to-audio conversion can be expressed as follows:

G(V ₁ , V ₂ , ..., V _Δt )→X ₁ , X ₂ , ..., X _Δt

where X _t ={x _t,1 , x _t,2 ,...,x _t ,SR _audio }, t∈{1,2,...,Δt}. Vt and Xt have the same dimension at this point.

Example 3

The difference from Embodiment 1 is that the method for generating audio of a water scene in this embodiment is based on a GAN network, and the GAN network includes a generator, a discriminator, and a timbre enhancer. The network in the present invention adjusts the input and output according to the requirements of sound generation, so that the receptive field of each layer of convolution in the original image network (receptive field: in the convolutional neural network CNN, determines the output result of a certain layer The region size of the input layer corresponding to an element) is no longer applicable. In image networks, convolutional layers with a receptive field of 3×3 are usually used. Corresponding to the 44100-dimensional input and output of the present invention, the receptive fields of the convolutional layers of the generator and the discriminator are also changed, and a larger receptive field is used to complete the corresponding convolution operation. In addition, in the convolution process, the two-dimensional filter used in the image is discarded. For the characteristics of the sound dimension, the present invention uses a one-dimensional filter for convolution, in order to remove some unwanted frequency information in the sound results , a filter is added at the end of the generator to filter out part of the frequency information in the result, and the length of the output sequence remains unchanged during the filtering process. The specific structure of generator and discriminator can refer to Table 1 and Table 2

Table 1

Table 2

Among them, because the corresponding parts of Relu, LeakRelu and BatchNorm layers after the convolutional layer (Conv1D) and the deconvolutional layer (Trans Conv1D) do not involve convolution kernels and changes to the output size, they are not listed in the table. Stride represents the convolution stride in the convolution or deconvolution process. The three parameters corresponding to the column of "convolution kernel size" refer to the size of the receptive field, the number of input channels of this layer, and the number of output channels of this layer. The three parameters corresponding to the column "Output Shape" refer to the batch size, input dimension and number of channels of this layer respectively. In order to ensure the correspondence between the convolution and the deconvolution process, in the process of convolution or deconvolution, the present invention adopts the constantly changing receptive field and the convolution step size, so that the conversion between the input and the output in the layer is not discarded. Dimension or the process of adding dimension.

Based on the disclosure and teaching of the above specification, those skilled in the art to which the present invention pertains can also make changes and modifications to the above-described embodiments. Therefore, the present invention is not limited to the above-mentioned specific embodiments, and any obvious improvement, replacement or modification made by those skilled in the art on the basis of the present invention falls within the protection scope of the present invention. In addition, although some specific terms are used in this specification, these terms are only for convenience of description and do not constitute any limitation to the present invention.

Claims (7)

1. An end-to-end-based method for generating audio of a water scene, comprising the steps of:

selecting various water scene videos and preprocessing the videos;

step two, obtaining a generator model through training according to the preprocessed data;

preprocessing a silent video, loading the silent video to the trained generator model, and outputting an audio corresponding to the silent video;

generating an envelope according to the sequence of the audio, loading the envelope to a trained tone enhancer model, and outputting the audio after tone enhancement;

wherein, in the first step, the pretreatment method comprises the following steps,

a1, extracting the characteristics of video frames to obtain the information of videos;

a2, converting video information per second into a vector with the same dimension as audio;

in step a2, the video information conversion formula is,

；

wherein y1, ym represents color channel information of the video frame, each channel is a matrix composed of an array of 0 to 255, G (y1, ym) represents a value of an audio signal generated based on the video frame, ranging from-1 to 1,

x 1.. times, xn represents the value of the audio signal corresponding to the video, and the variation range is-1 to 1;

in step one, the vector Vt generated by the pre-processed video frame can be expressed as follows,

；

wherein,

it is shown that the connection operation is performed,

，

，

and

representing video and audioRespectively, 30 and 44100,

represents the extracted features of the qth frame of the tth second, Floor represents rounding down;

the task of generating sound may be further expressed in the form of,

；

wherein,

。

2. the method for generating an end-to-end-based waterscape audio frequency according to claim 1, wherein in the second step, the training method of the generator model comprises the following steps:

b1, inputting the vector of the video information, and outputting an audio signal through the generator model;

b2, evaluating the audio signals, if the audio signals do not correspond to the audio signals, feeding back the audio signals to the generator model, and readjusting the audio signals until the corresponding audio signals are output; and if so, continuing to train the next video information.

3. The method for generating end-to-end water scene audio according to claim 1, wherein in step four, the method for training the timbre enhancer model comprises the following steps:

c1, inputting the envelope of the target audio, and outputting the sequence of the audio through the tone enhancer model;

c2, evaluating the audio sequence, if the audio sequence is not the target sequence, feeding back the audio sequence to the tone intensifier model, and readjusting until the target audio sequence is output; if the target sequence is obtained, continuing the next tone enhancement training.

4. The method for generating audio for a water scene on an end-to-end basis as claimed in claim 1, wherein in step four, the method for generating the envelope comprises the steps of:

d1, inputting a segment of audio sequenceGVAnd sampling interval of the envelopeLstep；

D2, getting audio sequenceGVEach sampling interval inLstepThe maximum of the absolute value in this interval is taken as an envelope point pi in this interval;

d3, an array Ep formed by connecting envelope points pi in all sampling intervals, and length sum formed by linear interpolationGVThe same sequence E (1: len), i.e. an audio sequenceGVThe corresponding envelope of the envelope is then determined,

E(1 :len) = interp( max p1,……pLstep ⊕ ... ⊕ max plen − Lstep + 1,……plen ) ，

pi ϵ GV and interp () indicate linear interpolation, and ≧ indicate join operation.

5. The method as claimed in claim 2, wherein in step two, the loss function used for outputting the audio signal is:

,

where λ = 100, where X denotes a true sound value, V denotes video frame information, G denotes a result generated by the generator, D denotes a result of evaluation, and E denotes averaging.

6. The method as claimed in claim 2, wherein in step two, the loss function used for evaluating the audio signal is:

,

where V denotes video frame information, G denotes a result generated by the generator, D denotes a result of evaluation, and E denotes averaging.

7. The method as claimed in claim 1, wherein the method for generating the water scene audio is based on a GAN network, and the GAN network comprises a generator, a discriminator and a timbre enhancer.

Images