CN116665696A - Piano playing video generation method, device, computer equipment and storage medium - Google Patents

Abstract

The invention relates to the field of speech analysis, in particular to a method, device, computer equipment and storage medium for generating a piano playing video. The method includes: obtaining audio stream data; inputting the audio stream data into an audio encoder for encoding processing to obtain audio encoding; performing code conversion on the audio encoding through a piano video code conversion model to obtain a sequence of piano video codebooks; The decoder decodes the piano video codebook sequence to obtain the piano video stream data; the piano video stream data refers to the video stream of people playing music corresponding to the audio encoding on the piano; the piano video stream data and the audio stream data are combined Merge to get the piano playing video. The present invention converts audio stream data into video stream data with a picture of human hands playing the piano, and finally generates a video containing both audio and human hands playing the audio on the piano, so that the video effect and quality are better, and user experience is improved feel.

Description

Piano playing video generation method, device, computer equipment and storage medium

technical field

The invention relates to the field of speech analysis, in particular to a method, device, computer equipment and storage medium for generating a piano playing video.

Background technique

With the development of technology and the entertainment industry, people's entertainment methods have changed from simple text reading and picture display to multimedia such as audio and video. Especially with the rise of short videos, the demand for music videos is also increasing. Existing channels for obtaining music are usually obtained through music software, but often only the audio of playing the piano is obtained, and there is no complete piano playing video including human hands playing the piano. Secondly, even some piano music generates a corresponding piano playing video containing human hands playing the piano. However, the existing piano playing videos are usually generated by speaking to drive the movement of the virtual human, which makes the generated piano playing videos not smooth enough and of poor quality, which affects the user experience.

Contents of the invention

Based on this, it is necessary to provide a piano playing video generation method, device, computer equipment and storage medium for the above technical problems, so as to solve the problem of poor video effect existing in the existing piano playing video generation technology.

A method for generating a piano playing video, comprising:

Get audio stream data;

Inputting the audio stream data into an audio encoder for encoding processing to obtain audio encoding;

Carry out transcoding to described audio coding by piano video transcoding model, obtain the piano video codebook sequence corresponding to described audio coding;

The piano video codebook sequence is decoded by a piano video codebook decoder to obtain piano video stream data; the piano video stream data refers to a video stream of people playing music corresponding to the audio code on the piano ;

Combining the piano video stream data and the audio stream data to obtain a piano playing video.

A device for generating a piano playing video, comprising:

The audio stream data module is used to obtain audio stream data;

An audio encoding module, configured to input the audio stream data into an audio encoder for encoding processing to obtain audio encoding;

A piano video codebook sequence module, used for transcoding the audio code through a piano video code conversion model, to obtain a piano video codebook sequence corresponding to the audio code;

The piano video stream data module is used to decode the piano video codebook sequence through the piano video codebook decoder to obtain piano video stream data; The video stream of the music corresponding to the audio encoding;

The piano playing video module is used to combine the piano video stream data and the audio stream data to obtain a piano playing video.

A computer device, comprising a memory, a processor, and computer-readable instructions stored in the memory and operable on the processor, when the processor executes the computer-readable instructions, the above-mentioned piano playing video is realized generate method.

One or more readable storage media storing computer-readable instructions, when the computer-readable instructions are executed by one or more processors, the one or more processors execute the method for generating a piano playing video as described above .

The above-mentioned piano playing video generation method, device, computer equipment and storage medium obtain audio stream data; input the audio stream data into an audio encoder for encoding processing to obtain audio encoding; Encoding is carried out transcoding, obtains the piano video codebook sequence corresponding to described audio coding; The piano video codebook sequence is decoded by piano video codebook decoder, obtains piano video stream data; Described piano video stream The data and the audio stream data are combined to obtain a piano playing video. The present invention encodes the audio data and converts the code into a piano video codebook sequence, and then decodes the piano video codebook sequence into piano video stream data (the piano video stream data refers to the human hand playing on the piano and the piano video stream data). Audio encoding corresponds to the video stream of the music). Further, the piano video stream data and audio stream data are combined to obtain the piano playing video, and finally the audio stream data is converted into a piano playing video containing both audio and the audio played by the human hand on the piano. According to the audio stream data The generated piano playing video is smoother, the video effect and quality are better, and the user experience is improved.

Description of drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the accompanying drawings that need to be used in the description of the embodiments of the present invention. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention , for those skilled in the art, other drawings can also be obtained according to these drawings without paying creative labor.

Fig. 1 is a schematic diagram of an application environment of a piano playing video generation method in an embodiment of the present invention;

Fig. 2 is a schematic flow chart of the piano playing video generation method in an embodiment of the present invention;

Fig. 3 is a structural representation of a piano playing video generating device in an embodiment of the present invention;

FIG. 4 is a schematic diagram of a computer device in an embodiment of the present invention.

Detailed ways

The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some of the embodiments of the present invention, but not all of them. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without creative efforts fall within the protection scope of the present invention.

The method for generating a piano playing video provided in this embodiment can be applied in an application environment as shown in FIG. 1 , where the client communicates with the server. Among them, clients include but are not limited to various personal computers, notebook computers, smart phones, tablet computers and portable wearable devices. The server can be implemented by an independent server or a server cluster composed of multiple servers.

In one embodiment, as shown in Figure 2, a method for generating a piano playing video is provided, and the method is applied to the server in Figure 1 as an example for illustration, including the following steps:

S10. Acquire audio stream data.

Understandably, the audio stream data refers to audio data. For example, music data obtained from music platforms.

S20. Input the audio stream data into an audio encoder for encoding processing to obtain audio encoding.

Understandably, an audio encoder refers to an encoder that encodes audio data. Audio coding refers to coding of audio data, and the audio coding may be a coding composed of several characters. Specifically, before the audio stream data is input into the audio encoder, the audio stream data can be converted into Mel spectrum data first, and the Mel spectrum feature of the audio stream data can be obtained according to the Mel spectrum data, and the Mel spectrum feature of the audio stream data can be converted into The spectral features are input into an audio encoder for encoding processing.

S30. Perform code conversion on the audio code by using a piano video code conversion model to obtain a piano video codebook sequence corresponding to the audio code.

Understandably, the piano video code conversion model is a trained neural network model, which is used to convert audio codes into piano video codebook sequences of human hands playing the piano. Wherein, the piano video transcoding model may be a GPT2 model (language model), and the GPT2 model is a type of Transformer (self-attention network) model. Transcoding refers to the process of transcoding audio codes into piano video codebook sequences, that is, the process of converting audio codes into video codes. Piano video codebook sequences are video codes obtained by transcoding audio codes. The piano video codebook sequence refers to a code sequence of a video of a human hand playing music corresponding to audio coding on a piano.

S40. Decode the piano video codebook sequence by a piano video codebook decoder to obtain piano video stream data; the piano video stream data refers to the music corresponding to the audio code being played on the piano by hand video stream.

It can be understood that the piano video codebook decoder is a trained neural network model, which is used to decode the piano video codebook sequence, and decode the piano video codebook sequence into piano video stream data. For example, the piano video codebook decoder can be a VqGAN model (Vector Quantized General Adversarial Network, image generation model), which can generate corresponding images according to input data. The decoding process refers to the process of decoding the piano video codebook sequence into piano video stream data. The piano video stream data refers to data including several video frames. The piano video stream data refers to a video stream of people playing music corresponding to the audio code on the piano.

S50. Combine the piano video stream data and the audio stream data to obtain a piano playing video.

Understandably, after obtaining the video stream of people playing music corresponding to audio coding on the piano, the video stream and audio stream data of people playing music corresponding to audio coding on the piano are combined to obtain both audio and video streams. It also includes a video of people playing the audio on the piano, that is, a piano playing video.

In steps S10-S50, by acquiring audio stream data; inputting the audio stream data into an audio encoder for encoding processing to obtain audio encoding; transcoding the audio encoding through the piano video transcoding model to obtain the same as the described audio encoding. A piano video codebook sequence corresponding to the audio encoding; decoding the piano video codebook sequence by a piano video codebook decoder to obtain piano video stream data; merging the piano video stream data and the audio stream data , to get the piano playing video. In this embodiment, the audio data is encoded and converted into a piano video codebook sequence, and then the piano video codebook sequence is decoded into piano video stream data (the piano video stream data refers to the human hand playing on the piano and the piano video stream data). video stream of the music corresponding to the audio encoding described above). Further, the piano video stream data and audio stream data are combined to obtain the piano playing video, and finally the audio stream data is converted into a piano playing video containing both audio and the audio played by the human hand on the piano. According to the audio stream data The generated piano playing video is smoother, the video effect and quality are better, and the user experience is improved.

Optionally, before step S30, that is, before performing code conversion on the audio code through the piano video code conversion model to obtain a piano video codebook sequence corresponding to the audio code, the method includes:

S301. Obtain a piano playing video sample;

S302. Perform encoding processing on the piano playing video sample to obtain a first video codebook sequence and audio sample encoding;

S303. Encode the audio samples into the initial piano video transcoding model to obtain a target video codebook sequence;

S304. Determine a video loss value according to the first video codebook sequence and the target video codebook sequence;

S305. When the video loss value does not reach the preset convergence condition, iteratively update the initial parameters of the initial piano video transcoding model until the video loss value reaches the preset convergence condition, after convergence The initial piano video transcoding model of is used as the piano video transcoding model.

It can be understood that the piano playing video sample refers to a video collected for training a piano video transcoding model that contains both audio and human hands playing the audio on the piano. Encoding the piano playing video samples refers to a process of encoding the piano playing video samples into a first video codebook sequence and encoding audio samples. Specifically, first stream the piano playing video samples to obtain the first piano video stream samples and audio stream samples; then, encode the first piano video stream samples and audio stream samples respectively to obtain the first video codebook sequence and audio sample encoding. The initial piano video transcoding model is used to train and learn the first video codebook sequence and audio sample encoding, and finally generate a piano video transcoding model that can convert the audio sample encoding into a video codebook sequence. The initial piano video code conversion model can be a GPT2 model (language model), and the GPT2 model is a type of Transformer (self-attention network) model, which can predict output results according to input data. The transcoding ability of the initial piano video transcoding model is weak, and it needs to be updated continuously, and finally a piano video transcoding model with strong transcoding ability that satisfies the convergence condition is obtained. The target video codebook sequence is the video codebook sequence obtained by transcoding the audio sample encoding through the initial piano video transcoding model. The video loss value refers to the loss value between the first video codebook sequence and the target video codebook sequence. When the video loss value is smaller, it indicates that the target video codebook sequence is closer to the first video codebook sequence, which means that the transcoding ability of the initial piano video transcoding model is stronger. Preferably, the video loss value can be obtained through a loss function based on cross entropy (Cross Entropy).

In this embodiment, model training is performed on the initial piano video transcoding model based on piano playing video samples, so that the obtained piano video transcoding model has strong transcoding capability and improves the conversion accuracy of the piano video transcoding model.

Optionally, in step S303, the encoding of the audio samples is input into the initial piano video code conversion model to obtain the target video codebook sequence, including:

S3031. Encode the audio sample as first input data, and input it into the initial piano video transcoding model to obtain a first frame video codebook sequence;

S3032. Perform code splicing of the audio sample code and the first frame video codebook sequence as second input data, and input it into the initial piano video code conversion model to obtain a second frame video codebook sequence;

S3033. Perform code concatenation on the audio sample code, the first frame video codebook sequence, and the second frame video codebook sequence as third input data, and input them into the initial piano video code conversion model to obtain the first A three-frame video codebook sequence;

S3034. Obtain the target video codebook sequence when all the audio sample codes are converted into a video codebook sequence.

Understandably, the audio sample code may be a code composed of several characters. A video codebook sequence may be a code consisting of several characters. After inputting the entire audio sample encoding as the first input data into the initial piano video transcoding model, when the initial piano video transcoding model transcodes the audio sample encoding, it first predicts the code sequence of a video frame, and obtains the code sequence of a video frame The video codebook sequence is recorded as the first frame video codebook sequence. Furthermore, the video codebook sequence of the first frame predicted and the audio sample coding are concatenated as the second input data, and the video codebook sequence of the next video frame is predicted, and the video codebook sequence is recorded as the second frame video code thin sequence. Similarly, code concatenation of the audio sample code, the first frame of video codebook sequence and the second frame of video codebook sequence is used as the third input data, input into the initial piano video code conversion model, and the third frame of video codebook sequence can be obtained . The target video codebook sequence is obtained when all characters encoded up to the audio sample are converted to a video codebook sequence. For example, when the first input data is (a1, a2, a3, . . . , <BOS>), the corresponding output data may be (a2, a3, a4, . . . , <BOS>, v1). Among them, (a1, a2, a3, ...) is the audio sample code, <BOS> is the separator, and v1 is the video codebook sequence of the first frame; then the second input data is (a1, a2, a3, ..., <BOS >, v1), the corresponding output data can be (a3, a4, a5, ..., <BOS>, v1, v2), wherein, v2 is the second frame video codebook sequence; the third input data is (a1, a2 , a3, ..., <BOS>, v1, v2), the corresponding output data can be (a4, a5, a6, ..., <BOS>, v1, v2, v3), where v3 is the video codebook of the third frame Sequence; when all audio sample codes are converted into video codebook sequences, the corresponding output data can be (v1, v2, v3, ...), where (v1, v2, v3, ...) is the target video codebook sequence.

In this embodiment, the video codebook sequence of the first frame is predicted through the entire audio coding, and then the video codebook sequence prediction of the next video frame is performed based on the entire audio coding and the obtained video codebook sequence, so that each video frame The video codebook sequence fully learns the entire audio code and the content of the video codebook sequence of all video frames before this video frame, which improves the accuracy of the target video codebook sequence.

Optionally, in step S302, that is, performing encoding processing on the piano playing video sample to obtain the first video codebook sequence and audio sample encoding, including:

S3021. Perform streaming processing on the piano playing video sample to obtain a first piano video stream sample and audio stream sample;

S3022. Input the first piano video stream sample into a codebook encoder to obtain the first video codebook sequence;

S3023. Input the audio stream sample into the audio encoder to obtain the audio sample code.

It can be understood that the splitting process refers to the process of splitting the piano playing video sample into a first piano video stream sample and an audio stream sample by using video splitting technology. Wherein, the first piano video stream sample refers to a video stream segmented from the piano playing video sample, including several video frames of people playing the piano by hand. The audio stream sample refers to an audio stream segmented from the piano playing video sample. A codebook encoder is an encoder that encodes a video stream. The first piano video stream samples may be encoded into a first video codebook sequence by a codebook encoder. An audio encoder refers to an encoder that encodes audio data. Audio stream samples can be encoded into audio sample codes by an audio encoder.

In this embodiment, the piano playing video samples are streamed, and the video streams and audio streams are coded separately, so that the first video codebook sequence and audio sample codes can be quickly and accurately obtained, improving the efficiency of model training .

Optionally, before step S40, before the piano video codebook sequence is decoded by the piano video codebook decoder to obtain piano video stream data, the steps include:

S401. Obtain a second piano video stream sample;

S402. Extract the first skeletal key point video stream from the second piano video stream sample by using the hand model;

S403. Use a codebook encoder to encode the second piano video stream sample to obtain a second video codebook sequence;

S404. Perform quantization processing on the second video codebook sequence to obtain a video quantized codebook sequence;

S405. Input the video quantization codebook sequence into the initial piano video codebook decoder for decoding processing, and obtain the target piano video stream and the second bone key point video stream;

S406. According to the second piano video stream sample, the target piano video stream, the first skeletal key point video stream, the second skeletal key point video stream, the second video codebook sequence, and the The video quantization codebook sequence determines the total loss value;

S407. When the total loss value does not reach the preset convergence condition, iteratively update the initial parameters of the initial piano video codebook decoder until the total loss value reaches the preset convergence condition, and converge The subsequent initial piano video codebook decoder serves as the piano video codebook decoder.

It can be understood that the second piano video stream sample refers to a video stream segmented from the piano playing video sample, which includes several video frames of people playing the piano by hand. The hand model refers to the trained neural network model used to recognize human hands from the input data. Through the hand model, the skeleton key points of the human hand in each video frame in the second piano video stream sample can be extracted, and the skeleton key points of the human hand in each video frame obtained are combined into a video, and the output is the first bone key point video flow. The second video codebook sequence refers to the video codebook sequence obtained by encoding the second piano video stream samples by the codebook encoder. Quantization processing refers to a process of querying the codebook library for a video codebook sequence corresponding to the second video codebook sequence through a quantization function. Among them, the code book library is an existing code library containing several piano video codes. The video quantization codebook sequence refers to the video codebook sequence corresponding to the second video codebook sequence that is queried in the codebook library through the quantization function. The initial piano video codebook decoder can be a VqGAN model (Vector Quantised General Adversarial Network, image generation model), which can generate corresponding images according to input data. The video quantized codebook sequence can be decoded into a video stream composed of several images through the initial piano video codebook decoder, that is, the target piano video stream. Similarly, referring to the acquisition method of the first skeletal key point video stream, the second skeletal key point video stream can be obtained according to the target piano video stream. Furthermore, according to the second piano video stream sample, the target piano video stream, the first skeletal key point video stream, the second skeletal key point video stream, the second video codebook sequence and the video quantization codebook sequence, determine the initial piano video codebook decoding The total loss value of the decoder, and iteratively update the initial parameters of the initial piano video codebook decoder according to the total loss value, until the initial piano video codebook decoder meets the convergence condition, and the piano video codebook decoder is obtained.

In this embodiment, model training is carried out to the initial piano video codebook decoder based on the second piano video stream sample, so that the obtained piano video codebook decoder has stronger decoding ability, and the decoding performance of the piano video codebook decoder is improved. Accuracy.

Optionally, in step S406, that is, according to the second piano video stream sample, the target piano video stream, the first skeletal key point video stream, the second skeletal key point video stream, the The second video codebook sequence and the video quantization codebook sequence determine a total loss value, comprising:

S4061. Determine a first loss value according to the second piano video stream sample and the target piano video stream;

S4062. Determine a second loss value according to the first skeletal key point video stream and the second skeletal key point video stream;

S4063. Determine a third loss value according to the second video codebook sequence and the video quantized codebook sequence;

S4064. Determine the total loss value according to the first loss value, the second loss value, and the third loss value.

It can be understood that the first loss value refers to the loss value between the second piano video stream sample and the target piano video stream, and the loss value between the two can be calculated by a discriminator. Similarly, the second loss value refers to the loss value between the second video codebook sequence and the video quantization codebook sequence, and the third loss value refers to the loss value between the second video codebook sequence and the video quantization codebook sequence . Wherein, the second loss value and the third loss value may be calculated based on Euclidean distance. Furthermore, the sum of the first loss value, the second loss value and the third loss value may be determined as the total loss value.

In this embodiment, the total loss value takes into account the interval between the second piano video stream sample and the target piano video stream, between the first skeleton keypoint video stream and the second skeleton keypoint video stream, and the second video codebook sequence The loss value between the video quantization codebook sequence and the video quantization codebook sequence, so that the initial piano video codebook decoder fully learns the key points of the human hand skeleton, the video quantization codebook sequence, and the piano video stream during the model training process, so that the generated piano playing The human hands in the video are clearer, and the playing screen of the video is smoother, which improves the user experience.

It should be understood that the sequence numbers of the steps in the above embodiments do not mean the order of execution, and the execution order of each process should be determined by its functions and internal logic, and should not constitute any limitation to the implementation process of the embodiment of the present invention.

In one embodiment, a device for generating a piano playing video is provided, and the device for generating a piano playing video corresponds to the method for generating a piano playing video in the above-mentioned embodiments. As shown in FIG. 3 , the piano playing video generating device includes an audio stream data module 10 , an audio encoding module 20 , a piano video codebook sequence module 30 , a piano video streaming data module 40 and a piano playing video module 50 . The detailed description of each functional module is as follows:

Audio stream data module 10, for obtaining audio stream data;

An audio encoding module 20, configured to input the audio stream data into an audio encoder for encoding processing to obtain audio encoding;

Piano video codebook sequence module 30, for transcoding described audio coding by piano video transcoding model, obtains the piano video codebook sequence corresponding to described audio coding;

The piano video stream data module 40 is used to decode the piano video codebook sequence by a piano video codebook decoder to obtain piano video stream data;

The piano playing video module 50 is used for merging the piano video stream data and the audio stream data to obtain the piano playing video; The corresponding music video stream.

Optionally, the piano playing video generation device also includes:

Piano playing video sample module, used to obtain piano playing video samples;

An encoding processing module, configured to encode the piano playing video sample to obtain the first video codebook sequence and audio sample encoding;

A target video codebook sequence module for encoding the audio samples into the initial piano video transcoding model to obtain a target video codebook sequence;

a video loss value module, configured to determine a video loss value based on the first video codebook sequence and the target video codebook sequence;

A piano video transcoding model module, configured to iteratively update the initial parameters of the initial piano video transcoding model when the video loss value does not reach a preset convergence condition, until the video loss value reaches the preset convergence condition When the convergence condition is used, the initial piano video transcoding model after convergence is used as the piano video transcoding model.

Optionally, the target video codebook sequence module includes:

A first frame video codebook sequence unit, configured to encode the audio samples as first input data and input them into the initial piano video transcoding model to obtain a first frame video codebook sequence;

The second frame video codebook sequence unit is used to code the audio sample code and the first frame video codebook sequence as the second input data and input it into the initial piano video code conversion model to obtain the second frame video codebook sequence;

The third frame video codebook sequence unit is used to code the audio sample code, the first frame video codebook sequence and the second frame video codebook sequence as the third input data, and input the initial In the piano video code conversion model, the third frame video codebook sequence is obtained;

A target video codebook sequence unit, configured to obtain the target video codebook sequence when all audio sample codes are converted into video codebook sequences.

Optionally, the encoding processing module includes:

A streaming processing unit, configured to perform streaming processing on the piano playing video sample to obtain a first piano video stream sample and an audio stream sample;

A first video codebook sequence unit, configured to input the first piano video stream sample into a codebook encoder to obtain the first video codebook sequence;

An audio sample encoding unit, configured to input the audio stream sample into the audio encoder to obtain the audio sample encoding.

Optionally, the piano playing video generation device also includes:

The second piano video stream sample module is used to obtain the second piano video stream sample;

The first skeletal key point video stream module is used to extract the first skeletal key point video stream from the second piano video stream sample through the hand model;

The second video codebook sequence module is used to encode the second piano video stream sample through a codebook encoder to obtain a second video codebook sequence;

A video quantization codebook sequence module, configured to perform quantization processing on the second video codebook sequence to obtain a video quantization codebook sequence;

The decoding processing module is used to input the video quantization codebook sequence into the initial piano video codebook decoder for decoding processing to obtain the target piano video stream and the second skeleton key point video stream;

The total loss value module is used for according to the second piano video stream sample, the target piano video stream, the first skeleton key point video stream, the second skeleton key point video stream, the second video code A thin sequence and said video quantization codebook sequence determine a total loss value;

A piano video codebook decoder module, configured to iteratively update the initial parameters of the initial piano video codebook decoder until the total loss value reaches the preset convergence condition. When the convergence condition is set, the initial piano video codebook decoder after convergence is used as the piano video codebook decoder.

Optionally, the total loss value module includes:

A first loss value unit, configured to determine a first loss value according to the second piano video stream sample and the target piano video stream;

A second loss value unit, configured to determine a second loss value according to the first skeletal key point video stream and the second skeletal key point video stream;

a third loss value unit, configured to determine a third loss value according to the second video codebook sequence and the video quantized codebook sequence;

A total loss value unit, configured to determine the total loss value according to the first loss value, the second loss value and the third loss value.

For specific limitations on the apparatus for generating a piano playing video, please refer to the above-mentioned limitations on the method for generating a piano playing video, which will not be repeated here. Each module in the above-mentioned piano playing video generating device can be fully or partially realized by software, hardware and combinations thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, and can also be stored in the memory of the computer device in the form of software, so that the processor can invoke and execute the corresponding operations of the above-mentioned modules.

In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure may be as shown in FIG. 4 . The computer device includes a processor, memory, network interface and database connected by a system bus. Wherein, the processor of the computer device is used to provide calculation and control capabilities. The memory of the computer device includes a readable storage medium and an internal memory. The readable storage medium stores an operating system, computer readable instructions and a database. The internal memory provides an environment for the execution of the operating system and computer readable instructions in the readable storage medium. The database of the computer device is used to store the data involved in the method for generating the piano playing video. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer-readable instructions are executed by the processor, a method for generating a piano performance video is realized. The readable storage medium provided in this embodiment includes a non-volatile readable storage medium and a volatile readable storage medium.

In one embodiment, a computer device is provided, including a memory, a processor, and computer-readable instructions stored on the memory and operable on the processor. When the processor executes the computer-readable instructions, the following steps are implemented:

Get audio stream data;

Inputting the audio stream data into an audio encoder for encoding processing to obtain audio encoding;

Carry out transcoding to described audio coding by piano video transcoding model, obtain the piano video codebook sequence corresponding to described audio coding;

The piano video codebook sequence is decoded by a piano video codebook decoder to obtain piano video stream data; the piano video stream data refers to a video stream of people playing music corresponding to the audio code on the piano ;

Combining the piano video stream data and the audio stream data to obtain a piano playing video.

In one embodiment, one or more computer-readable storage media storing computer-readable instructions is provided. The readable storage media provided in this embodiment include non-volatile readable storage media and volatile readable storage media. storage medium. Computer-readable instructions are stored on the readable storage medium, and when the computer-readable instructions are executed by one or more processors, the following steps are implemented:

Get audio stream data;

Inputting the audio stream data into an audio encoder for encoding processing to obtain audio encoding;

Carry out transcoding to described audio coding by piano video transcoding model, obtain the piano video codebook sequence corresponding to described audio coding;

The piano video codebook sequence is decoded by a piano video codebook decoder to obtain piano video stream data; the piano video stream data refers to a video stream of people playing music corresponding to the audio code on the piano ;

Combining the piano video stream data and the audio stream data to obtain a piano playing video.

Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware through computer-readable instructions, and the computer-readable instructions can be stored in a non-volatile memory When being read from a storage medium or a volatile readable storage medium, the computer-readable instructions may include the processes of the embodiments of the above-mentioned methods when executed. Wherein, any references to memory, storage, database or other media used in the various embodiments provided in the present application may include non-volatile and/or volatile memory. Nonvolatile memory can include read only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced SDRAM (ESDRAM), Synchronous Chain Synchlink DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

Those skilled in the art can clearly understand that for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used for illustration. In practical applications, the above-mentioned functions can be assigned to different functional units, Completion of modules means that the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above.

The above-described embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: it can still carry out the foregoing embodiments Modifications to the technical solutions recorded in the examples, or equivalent replacement of some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the various embodiments of the present invention, and should be included in within the protection scope of the present invention.

Claims (10)

1. A piano-playing video generation method, characterized by comprising:

acquiring audio stream data;

inputting the audio stream data into an audio encoder for encoding processing to obtain audio codes;

Transcoding the audio codes through a piano video transcoding model to obtain piano video code book sequences corresponding to the audio codes;

decoding the piano video code book sequence through a piano video code Bao Jiema device to obtain piano video stream data; the piano video stream data refers to a video stream in which a human hand plays music corresponding to the audio code on a piano;

and combining the piano video stream data and the audio stream data to obtain piano playing video.

2. The piano-playing video generating method of claim 1, wherein before said transcoding said audio codes through a piano video transcoding model, obtaining a piano video codebook sequence corresponding to said audio codes, comprising:

acquiring a piano playing video sample;

performing coding processing on the piano playing video sample to obtain a first video code book sequence and an audio sample code;

inputting the audio sample codes into an initial piano video code conversion model to obtain a target video code book sequence;

determining a video loss value from the first video codebook sequence and the target video codebook sequence;

And when the video loss value does not reach a preset convergence condition, iteratively updating initial parameters of the initial piano video transcoding model, and taking the initial piano video transcoding model after convergence as the piano video transcoding model until the video loss value reaches the preset convergence condition.

3. The piano-playing video generation method of claim 2, wherein said encoding said audio samples into an initial piano video transcoding model results in a sequence of target video codebooks, comprising:

the audio sample codes are used as first input data and are input into the initial piano video transcoding model, and a first frame video code book sequence is obtained;

performing code splicing on the audio sample codes and the first frame video code book sequence to obtain second input data, and inputting the second input data into the initial piano video code conversion model to obtain a second frame video code book sequence;

performing code splicing on the audio sample codes, the first frame video code book sequence and the second frame video code book sequence to obtain third input data, and inputting the third input data into the initial piano video code conversion model to obtain a third frame video code book sequence;

And when the audio sample codes are all converted into video code book sequences, obtaining the target video code book sequences.

4. The piano-playing video generating method of claim 2, wherein said encoding the piano-playing video sample to obtain a first video codebook sequence and an audio sample code comprises:

carrying out splitting processing on the piano playing video sample to obtain a first piano video stream sample and an audio stream sample;

inputting the first piano video stream sample into a code Bao Bianma device to obtain the first video code book sequence;

and inputting the audio stream samples into the audio encoder to obtain the audio sample codes.

5. The piano action video generating method of claim 1, wherein before said decoding process is performed on said piano video codebook sequence by means of a piano video coder Bao Jiema, comprising:

acquiring a second piano video stream sample;

extracting a first skeleton key point video stream from the second piano video stream sample through a hand model;

the second piano video stream sample is subjected to coding processing through a code Bao Bianma device to obtain a second video code book sequence;

Performing quantization processing on the second video code book sequence to obtain a video quantized code book sequence;

inputting the video quantized code book sequence into an initial piano video code Bao Jiema device for decoding processing to obtain a target piano video stream and a second bone key point video stream;

determining a total loss value from the second piano video stream sample, the target piano video stream, the first skeletal keypoint video stream, the second video codebook sequence and the video quantization codebook sequence;

and when the total loss value does not reach the preset convergence condition, iteratively updating the initial parameters of the initial piano video code Bao Jiema device until the total loss value reaches the preset convergence condition, and taking the initial piano video code Bao Jiema device after convergence as the piano video code Bao Jiema device.

6. The piano action video generating method of claim 5, wherein said determining a total loss value from said second piano video stream sample, said target piano video stream, said first skeletal keypoint video stream, said second video codebook sequence and said video quantization codebook sequence comprises:

Determining a first loss value according to the second piano video stream sample and the target piano video stream;

determining a second loss value according to the first skeletal keypoint video stream and the second skeletal keypoint video stream;

determining a third loss value based on the second video codebook sequence and the video quantization codebook sequence;

determining the total loss value from the first loss value, the second loss value, and the third loss value.

7. A piano-playing video generating apparatus, characterized by comprising:

the audio stream data module is used for acquiring audio stream data;

the audio coding module is used for inputting the audio stream data into an audio coder for coding processing to obtain audio codes;

the piano video code book sequence module is used for carrying out code conversion on the audio codes through a piano video code conversion model to obtain piano video code book sequences corresponding to the audio codes;

the piano video stream data module is used for decoding the piano video code book sequence through a piano video code Bao Jiema device to obtain piano video stream data; the piano video stream data refers to a video stream in which a human hand plays music corresponding to the audio code on a piano;

And the piano playing video module is used for combining the piano video stream data and the audio stream data to obtain piano playing video.

8. The piano-playing video generating device of claim 7, comprising, prior to said piano video codebook sequence block:

the piano playing video sample module is used for acquiring piano playing video samples;

the coding processing module is used for carrying out coding processing on the piano playing video sample to obtain a first video code book sequence and an audio sample code;

the target video code book sequence module is used for inputting the audio sample codes into an initial piano video code conversion model to obtain a target video code book sequence;

a video loss value module for determining a video loss value from the first video codebook sequence and the target video codebook sequence;

and the piano video transcoding model module is used for iteratively updating the initial parameters of the initial piano video transcoding model when the video loss value does not reach the preset convergence condition, and taking the initial piano video transcoding model after convergence as the piano video transcoding model until the video loss value reaches the preset convergence condition.

9. A computer device comprising a memory, a processor, and computer readable instructions stored in the memory and executable on the processor, wherein the processor, when executing the computer readable instructions, implements the piano action video generating method of any one of claims 1 to 6.

10. One or more readable storage media storing computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform the piano action video generation method of any one of claims 1 to 6.