WO2018121798A1 - Video coding and decoding device and method based on depth automatic coder - Google Patents

Abstract

The present disclosure provides a video coding and decoding device and method based on depth automatic coder. The device comprises: a depth automatic coder module, a neural network codec module, and a mixed codec module. The depth automatic coder module comprises a depth automatic coder. The depth automatic coder comprises a coding end. The coding end is used for performing first compression on an original video to obtain first-compressed data. The neural network codec module is used for performing coding and compression on a decoding end parameter, so as to generate a decoding end parameter that has been coded. The mixed codec module is used for performing mixed coding on the first-compressed data and the decoding end parameter that has been coded, so as to obtain the video compression data. In the present disclosure, the compression rate of video data is improved by constructing a coding end and a decoding end that are symmetrical in the structure and by performing secondary compression and decompression on the video data. Because of a non-linear characteristic of an artificial neural network, the integration of video data compression and encryption is implemented, by using a parameter of the artificial neural network as a key. The coding result of the video data contains a characteristic of the video data, which facilitates the classification and search of the video data and provides a broad development space and application prospect. Without the complicated coding and decoding process in manual design, the automatic extraction of a data characteristic by the depth automatic coder greatly reduces manual intervention, so that the automation of the coding process is implemented, the implementation is simple, and the scalability is good, and accordingly, the present invention can be used to compress video data and can also compresses other data.

Description

Video codec device and method based on depth automatic encoder Technical field

The present disclosure relates to the field of video compression and decompression, and in particular, to a video encoding and decoding apparatus and method based on a depth auto-encoder.

Background technique

With the advent of the Internet era, the massive generation of video data puts higher demands on transmission capabilities. In order to alleviate the transmission pressure, video coding and decoding technology came into being, and played a huge role in compressing video for transmission.

The traditional video coding technology is to eliminate the various types of redundancy existing in the video by different methods to achieve the purpose of compressing video. For example, techniques for temporal redundancy, spatial redundancy, visual redundancy, and coding redundancy for video use inter-frame coding, intra-frame coding, quantization, and entropy coding, respectively. Transforming is also a common method of removing spatial redundancy. Each video encoding method has a corresponding decoding method. Complex coding standards achieve better compression ratios by combining different methods and using different implementations.

Although the traditional video coding technology is relatively mature, it is more complicated and requires sophisticated manual design to achieve better compression.

Public content

In view of this, the main purpose of the present disclosure is to provide a video encoding and decoding apparatus and method based on a depth auto-encoder.

The depth codec-based video codec device of the present disclosure includes: a depth auto-encoder module, including a depth auto-encoder, the depth auto-encoder includes an encoding end, and the encoding end is used to compress the original video for the first time. Obtaining the first compressed data; the neural network codec module is configured to encode and compress the decoding end parameters to generate the encoded decoding end parameters; the hybrid codec module is configured to perform hybrid encoding on the first compressed data and the encoded decoding end parameters. , get video compression data.

In some embodiments of the disclosure, the encoding end is an N-layer artificial neural network structure.

In some embodiments of the present disclosure, the first layer of the N-layer artificial neural network structure is an input layer, the second to N layers are hidden layers, the inter-layer units are fully connected, the intra-layer elements are not connected, and the N-th layer is implicit. The number of hidden cells in the layer is less than the number of input cells in the input layer.

In some embodiments of the disclosure, the hybrid encoding comprises entropy encoding.

In some embodiments of the disclosure, the entropy encoding comprises Huffman encoding.

In some embodiments of the present disclosure, the method further includes: a storage module, configured to store the first compressed data, the decoding end parameter, and the video compressed data.

In some embodiments of the present disclosure, the neural network codec module is configured to read the decoding end parameter from the storage module to encode and compress the decoding end parameter.

In some embodiments of the present disclosure, the hybrid codec module is configured to read the first compressed data from the storage module, and read the encoded decoding end parameters from the neural network codec module to perform The hybrid encoding and storing the video compressed data to the storage module.

In some embodiments of the disclosure, the depth auto-encoder further includes: a decoding end; the hybrid codec module is further configured to decode the video compressed data to obtain the first decompressed data and the encoded decoding end parameter; The neural network codec module is further configured to decode the encoded decoding end parameter to obtain a decoding end parameter; the decoding end is configured to decode the first decompressed data to obtain original video data.

In some embodiments of the disclosure, the storage module is further configured to store the first decompressed data, the encoded decoding end parameter, and the original video data.

In some embodiments of the present disclosure, the hybrid codec module is further configured to read the video compressed data from the storage module to decode the video compressed data.

In some embodiments of the disclosure, the neural network codec module is further configured to read the encoded decoding end parameter from the storage module to decode the encoded decoding end parameter.

In some embodiments of the present disclosure, the depth autoencoder module is further configured to read the first decompressed data from the storage module, and read parameters of the decoding end from the neural network codec module, so that The decoding end decodes the first decompressed data.

In some embodiments of the present disclosure, the decoding end is an N-layer artificial neural network structure that is symmetric with the encoding end structure.

In some embodiments of the disclosure, the nth layer of the decoding end is the (N-n+1)th layer of the encoding end, and the weight matrix between the nth layer and the n+1th layer of the decoding end, Is a transposition of a weight matrix between the (Nn)th layer and the (N-n+1)th layer of the encoding end, where 1≤n≤N.

In some embodiments of the present disclosure, the depth autoencoder module is further configured to initialize the depth auto-encoder and train the depth auto-encoder by using training video to obtain depth automatic for video coding. Encoder.

In some embodiments of the disclosure, the depth autoencoder module is further configured to perform training on the depth autocoder by using training video, including: using two adjacent layers of the depth autoencoder encoding end as a limitation a Boltzmann machine; initializing the limited Boltzmann machine; training the limited Boltzmann machine with the training video data; fine-tuning the depth autoencoder encoding end with a backpropagation algorithm A weight matrix to minimize the reconstruction error to the original input.

In some embodiments of the present disclosure, a controller is further included, interconnected with the depth autoencoder module, the neural network codec module, and the hybrid codec module for controlling the above modules.

The present disclosure also provides a video encoding method based on a depth auto-encoder, which uses the video encoding and decoding apparatus of any of the above to perform video encoding, including: compressing the original video for the first time, and obtaining the first compressed data; Encoding compression, obtaining encoded decoding end parameters; performing hybrid encoding on the first compressed data and the encoded decoding end parameters to obtain video compressed data.

In some embodiments of the present disclosure, the original video is first compressed using a first N-layer artificial neural network structure.

In some embodiments of the disclosure, the first layer of the first N-layer artificial neural network structure is an input layer, the second to N layers are hidden layers, the inter-layer units are fully connected, and the intra-layer elements are not connected. The number of hidden cells of the N-layer hidden layer is smaller than the number of input cells of the input layer.

In some embodiments of the disclosure, the hybrid encoding comprises entropy encoding.

In some embodiments of the disclosure, the entropy encoding comprises Huffman encoding.

In some embodiments of the present disclosure, the method further includes: storing the first compressed data, the decoding end parameter, and the video compressed data.

In some embodiments of the present disclosure, the decoding end parameters are read to encode and compress the decoding end parameters.

In some embodiments of the present disclosure, the first compressed data and the encoded decoded end parameters are read to perform the hybrid encoding, and the video compressed data is stored.

In some embodiments of the present disclosure, the method further includes: decoding the video compressed data to obtain first decompressed data and the encoded decoding end parameter; and decoding the encoded decoding end parameter to obtain a decoding end parameter; Decoding the first decompressed data to obtain original video data.

In some embodiments of the present disclosure, the method further includes: storing the first decompressed data, the encoded decoding end parameters, and the original video data.

In some embodiments of the disclosure, the video compression data is read to decode the video compression data.

In some embodiments of the present disclosure, the encoded decoder parameters are read to decode the encoded decoder parameters.

In some embodiments of the present disclosure, the first decompressed data and parameters of the decoding end are read to decode the first decompressed data.

In some embodiments of the present disclosure, the first decompressed data is decoded using a second N-layer artificial neural network structure that is symmetric with the first N-layer artificial neural network structure.

In some embodiments of the present disclosure, the nth layer of the second N-layer artificial neural network structure is the (N-n+1)th layer of the first N-layer artificial neural network structure, and the second N The weight matrix between the nth layer and the n+1th layer of the layer artificial neural network structure is the weight between the (Nn)th layer and the (N-n+1)th layer of the first N layer artificial neural network structure The transposition of the matrix, where 1 ≤ n ≤ N.

In some embodiments of the present disclosure, before the first compression of the original video, the method further includes: initializing a depth auto-encoder; and training the deep auto-encoder with the training video data.

In some embodiments of the present disclosure, the training the depth autoencoder with the training video data comprises: using two adjacent layers of the depth autoencoder encoding end as a limited Boltzmann machine; initializing the limiting glass Using the training video data to train the restricted Boltzmann machine; using a backpropagation method to adjust the weight matrix of the depth autoencoder encoding end, minimizing the reconstruction of the original input error.

In some embodiments of the present disclosure, the method further includes: controlling the foregoing steps by using a controller.

In some embodiments of the present disclosure, a forward operation is sequentially performed on each layer in the N-layer artificial neural network structure; in an inverse order of the forward operation, in the N-layer artificial neural network structure Each layer sequentially performs a reverse operation; performs weight update on each layer in the N-layer artificial neural network structure; and repeatedly performs the above steps multiple times to complete the training of the N-layer artificial neural network structure.

In some embodiments of the disclosure, the performing the inverse operations sequentially on the layers in the N-layer artificial neural network structure includes: a first computing portion: obtained by the output neuron gradient and the input neurons Weight gradient; second operation portion: calculating the input neuron gradient using the output neuron gradient and weight.

In some embodiments of the disclosure, performing weight update on each layer in the N-layer artificial neural network structure includes: updating the weight by using the weight gradient to obtain an updated weight .

It can be seen from the above technical solution that the video encoding and decoding apparatus and method based on the depth auto-encoder of the present disclosure have the following beneficial effects:

(1) Using the artificial neural network degree video to encode the video twice, which improves the compression ratio of the video data;

(2) Since the artificial neural network has nonlinear characteristics, the compression and encryption integration of video data is realized by using the parameters of the artificial neural network as the secret key;

(3) The encoding result of video data by deep automatic encoder includes the characteristics of video data, which facilitates the classification and search of video data, and introduces machine learning into the field of video coding, which has broad development space and application prospects;

(4) It is not necessary to manually design complex codec process, and the function of automatically extracting data features by using deep automatic encoder, greatly reducing manual intervention, realizing automation of coding process, simple implementation, and good expandability, not only for video Data compression can also be used for other data compression.

DRAWINGS

The drawings are intended to provide a further understanding of the disclosure, and are in the In the drawing:

1 is a schematic structural diagram of a video codec apparatus according to an embodiment of the present disclosure;

2 is a schematic diagram of a depth autoencoder of an embodiment of the present disclosure;

3 is a coding flowchart of a video encoding and decoding method according to an embodiment of the present disclosure;

4 is a flowchart of a deep autoencoder training of a video encoding and decoding method according to an embodiment of the present disclosure;

FIG. 5 is a flowchart of decoding of a video encoding and decoding method according to an embodiment of the present disclosure.

【Symbol Description】

10-controller; 20-depth autoencoder module; 30-neural network codec module; 40-hybrid codec module; 50-storage module 50.

detailed description

The present disclosure will be further described in detail below with reference to the specific embodiments thereof and the accompanying drawings.

With the advent of the intelligent era, the introduction of artificial intelligence methods into the field of video codec, in order to seek greater breakthroughs should become the future development trend. The embodiment of the present disclosure provides a video encoding and decoding device based on a depth auto-encoder, and FIG. 1 is a schematic structural diagram of the video encoding and decoding device, including a controller 10, a depth auto-encoder module 20, and a neural network codec module. 30, a hybrid codec module 40, a storage module 50; wherein

Controller 10 is interconnected with depth autoencoder module 20, neural network codec module 30, and hybrid codec module 40. Controller 10 includes a local command queue. The controller 10 is configured to store the control instructions compiled by the user program in the instruction queue, and decode them into control signals to control each module to complete its respective functions, and implement video encoding and decoding. The storage module 50 is also interconnected with the depth autoencoder module 20, the neural network codec module 30, and the hybrid codec module 40 for storing various data and parameters in the video codec process.

The depth autoencoder module 20 includes a depth autoencoder including a structurally symmetric encoding end and a decoding end, the encoding end being an N-layer artificial neural network structure, wherein the first layer is an input layer, and the second to N The layer is a hidden layer, the inter-layer unit is fully connected, and the intra-layer unit has no connection. The number of hidden units in the hidden layer of the N-th layer is smaller than the number of input units in the input layer, so that the effect of video compression can be achieved, wherein N is greater than or equal to 2.

The decoding end is an N-layer artificial neural network structure symmetric with the coding end structure. Specifically, the first layer (ie, the input layer) of the decoding end is the Nth layer hidden layer of the coding end, and the second layer (ie, the first layer is hidden) The layer containing) is the N-1 layer hidden layer of the encoding end, and the weight matrix between the first layer and the second layer of the decoding end is the transposition of the weight matrix between the N-1th layer and the Nth layer of the encoding end.

The third layer of the decoding end (ie, the second layer hidden layer) is the N-2 layer hidden layer of the encoding end, and the weight matrix between the second layer and the third layer of the decoding end is the N-2 layer and the encoding end of the encoding end. Transpose of the weight matrix between the N-1 layers.

By analogy, the Nth layer of the decoding end (ie, the Nth layer hidden layer) is the first layer of the encoding end (ie, the input layer), and the weight matrix between the N-1th layer and the Nth layer of the decoding end is the first of the encoding end. The transpose of the weight matrix between the layer and the second layer.

That is, the nth layer of the decoding end is the N-n+1 layer of the encoding end, and the weight matrix between the two adjacent layers (the nth layer and the n+1th layer) of the decoding end is the adjacent two layers of the coding end (the first layer) Transposition of the weight matrix between the Nn layer and the N-n+1th layer).

The artificial neural network structure can be trained. The training step is to perform a forward operation on each layer in a (multi-layer) artificial neural network, and then perform reverse operations in the order of the opposite layers, and finally calculate The gradient of the weights is used to update the weights; this is the sequential iteration of the training of the neural network, and the entire training process needs to be repeated several times. Specifically, the method for implementing artificial neural network training using an artificial neural network structure includes the following contents:

Forward operation steps: First, a forward operation is sequentially performed on each layer in the multi-layer artificial neural network to obtain output neurons of each layer.

Reverse operation steps: Then, in the reverse order of the forward operation, the layers in the multi-layer artificial neural network are sequentially subjected to an inverse operation to obtain a weight gradient of each layer and an input neuron gradient.

This step includes a first arithmetic part and a second arithmetic part. The first arithmetic part is used to calculate the weight gradient. For each layer of the artificial neural network, the gradient of the weight of the layer is obtained by matrix multiplication or convolution of the output neuron gradient of the layer and the input neurons. The second computational portion is used to calculate the input neuron gradient. For each layer of the artificial neural network, the input neuron gradient and weight can be used to calculate the input neuron gradient.

Weight update step: Next, weight updates are performed on each layer in the multi-layer artificial neural network to obtain an updated weight. In this step, for each layer of the artificial neural network, the weight is updated with a weight gradient to obtain an updated weight.

The forward operation step, the reverse operation step, and the weight update step are repeatedly performed multiple times to complete the training of the multi-layer artificial neural network.

The entire training method requires repeated execution of the above process until the parameters of the artificial neural network meet the requirements, and the training process is completed.

As shown in FIG. 2, a schematic diagram of a depth auto-encoder is exemplarily given. The encoding end and the decoding end are five-layer artificial neural network structures, wherein the first layer hidden layer of the deep auto-encoder has 2000 units, the second layer has 1000 cells in the hidden layer, the hidden layer in the third layer has 500 cells, the hidden layer in the fourth layer has 30 cells, and the weight between the input layer and the hidden layer in the first layer The matrix is W1, the weight matrix between the first layer hidden layer and the second layer hidden layer is W2, and the weight matrix between the second layer hidden layer and the third layer hidden layer is W3, and the third layer is hidden. The weight matrix between the containing layer and the layer 4 hidden layer is W4. Correspondingly, the input layer of the decoding end has 30 units, the first layer has 500 cells, the second layer has 1000 cells, and the third layer has 2000 cells, the input layer and the first layer. The weight matrix between the layer hidden layers is W ^T ₄ , and the weight matrix between the first layer hidden layer and the second layer hidden layer is W ^T ₃ , the second layer hidden layer and the third layer hidden layer The weight matrix between them is W ^T ₂ , and the weight matrix between the layer 3 hidden layer and the layer 4 hidden layer is W ^T ₁ .

The depth autoencoder module 20 uses the encoding end of the depth autoencoder to compress the original video for the first time. The original video data is input to the input layer of the encoding end, and is compressed by each layer of the encoding end and output by the hidden layer of the Nth layer to obtain the first compressed data. And stored in the storage module 50, and the parameters of the decoding end are stored in the storage module 50, the parameters include the number of layers N of the decoding end, the number of units of each layer, and the weight matrix between the layers.

The neural network codec module 30 reads the parameters of the decoding end from the storage module 50, and encodes and compresses the parameters to generate the encoded decoding end parameters. Among them, the parameters can be encoded by a common coding method.

The hybrid codec module 40 performs secondary compression on the first compressed data. Specifically, it reads the first compressed data from the storage module 50, and reads the encoded decoding end parameters from the neural network codec module 30, and for the first time. The compressed data and the encoded decoding end parameters are mixed and encoded to obtain video compressed data, and stored in the storage module 50 to complete video compression. Among them, the hybrid coding can adopt the Huffman coding isentropic coding mode.

The video codec device of the present disclosure uses the artificial neural network degree video to compress and compress the video twice, thereby improving the compression ratio of the video data, and because the artificial neural network has nonlinear characteristics, the parameters of the artificial neural network are taken as secrets. The key realizes the integration of compression and encryption of video data. The encoding result of video data by deep automatic encoder includes the characteristics of video data, which facilitates the classification and search of video data, and introduces machine learning into the field of video coding, which has broad development space and application prospects.

Further, the video codec device of this embodiment may decode the video compressed data to reconstruct the original video data.

The hybrid codec module 40 decompresses the video compressed data for the first time. Specifically, it reads the video compressed data from the storage module 50, and decodes the video compressed data to obtain the first decompressed data and the encoded decoding end parameters, and stores the data in the decoded data. Storage module 50. The decoding adopts a decoding manner corresponding to the hybrid encoding, and the first decompressed data is the first compressed data in the encoding process.

The neural network codec module 30 reads the encoded decoding end parameters from the storage module 50, and decodes the encoded decoding end parameters to obtain parameters of the decoding end. The decoding adopts a decoding manner corresponding to the encoding mode of the decoding end parameter in the encoding process.

The depth autoencoder module 20 uses the decoding end to perform secondary decompression on the first decompressed data. Specifically, the deep autoencoder module 20 reads the first decompressed data from the storage module 50, and reads the parameters of the decoding end from the neural network codec module 30. The input layer of the data input decoding end is decompressed for the first time, and is decompressed by each layer of the decoding end and output by the hidden layer of the Nth layer to obtain original video data, and is stored in the storage module 50.

It can be seen that the video codec device of the present disclosure does not need to manually design a complicated codec process, and automatically extracts data features by using a deep automatic encoder, thereby greatly reducing manual intervention, realizing automation of the encoding process, and realizing simplicity and Good scalability, not only for video data compression, but also for other data compression.

Further, in the video codec device of the present disclosure, the depth auto-encoder is generated by training. The depth autoencoder module 20 first initializes a depth autoencoder, and then trains the encoding end of the depth autoencoder with the training video to obtain a depth autoencoder encoding end for video encoding. Specifically,

First, the adjacent two layers of the deep autoencoder encoding end are used as a limited Boltzmann machine, the upper layer of the adjacent two layers is used as the visible layer, and the next layer is used as the hidden layer to limit the Boltzmann machine. Train.

Limiting the Boltzmann machine to use a binary unit whose energy function is:

Where v _i is the i-th visible unit, h _j is the j-th hidden unit, a _i is the offset of the i-th visible unit v _i , b _j is the offset of the j-th hidden unit h _j , w _{j, i} is the weight connecting the jth hidden unit and the ith visible unit, and n _v and n _h are the number of visible and hidden units, respectively.

Then: Initialize the limit Boltzmann machine. Including: training video as training sample set S(|S|=n _s ), setting training period J, learning rate η, CD-K algorithm parameter k; specifying visible layer and hidden layer unit numbers n _v and n _h ; set the offset vectors a, b and the weight matrix w.

Wherein, the offset a _i of the i-th visible cell v _i is the i-th term of the offset vector a, and the offset b _j of the j-th hidden cell h _j is the j-th term of the offset vector b, w _j,i Is the element of the i-th column of the jth row in the weight matrix W, and n _s is the number of cells of the training sample set.

Next, the Boltzmann machine is trained. include:

First, ΔW, Δa and Δb are obtained using the CD-K algorithm;

Then, use ΔW, Δa and Δb to update the parameters that limit the Boltzmann machine:

The above two steps are cycled J times to obtain a trained limited Boltzmann machine as a depth auto-encoder.

Among them, the steps of obtaining ΔW, Δa and Δb using the CD-K algorithm are as follows:

Initialization: ΔW=0, Δa=0, Δb=0;

The following loop is performed for each sample v in the training sample set S:

(1) Initialization v ₀ = v

(2) for k samples in each sample, the visible start sampling the cell group V _t H _t group of hidden units, and from the group of hidden units H _t sampling a visible cell group v _{t + 1,} wherein t is an integer And 0 ≤ t ≤ k-1.

(3) For each of i and j (i and j are integers, 1 ≤ i ≤ n _h , 1 ≤ j ≤ n _v ), the following calculation is performed:

Δb _i =Δb _i +[P(h _i =1|v ₀ )-p(h _i =1|v _k )]

和

分别为序号为0的可见单元组中的第j个可见单元和序号为k的可见单元组中第j个可见单元。 among them,

with

The jth visible unit in the visible unit group with the sequence number 0 and the jth visible unit in the visible unit group with the sequence number k, respectively.

Finally, the back propagation algorithm is used to fine tune the weight matrix of the depth autoencoder's encoding end to minimize the reconstruction error to the original input. For example, when the weight matrix of the encoding end of the depth autoencoder is finely adjusted, the input/output unit and the hidden unit of the encoding end are no longer regarded as the unit of the Boltzmann machine, but the real output value of each unit is directly used. Since the encoder has been trained, a backpropagation algorithm can be used to adjust the weight matrix to minimize the reconstruction error of the encoder output.

Another embodiment of the present disclosure provides a video encoding and decoding method based on a depth auto-encoder. Referring to FIG. 3, the method includes:

In step S101, the controller 10 sends an encoding instruction to the depth autoencoder module 20, and the encoding end of the deep autoencoder first compresses the original video.

In step S102, the controller 10 sends an IO command to the depth autoencoder module 20, and the parameters of the first compressed data and the decoding end are stored in the storage module 50.

In step S103, the controller 10 sends an IO command to the neural network codec module 30, and the neural network codec module 30 reads the parameters of the decoding end from the storage module 50.

In step S104, the controller 10 sends an encoding instruction to the neural network codec module 30, and the neural network codec module 30 encodes and compresses the parameters.

In step S105, the controller 10 sends an IO command to the hybrid codec module 40, and the hybrid codec module 40 reads the first compressed data from the storage module 50, and reads the encoded decoded terminal parameters from the neural network codec module 30.

In step S106, the controller 10 sends an encoding instruction to the hybrid codec module 40, and the hybrid codec module 40 performs hybrid encoding on the first compressed data and the encoded decoding end parameters to obtain video compressed data.

In step S107, the controller 10 sends an IO command to the hybrid codec module 40, and the hybrid codec module 40 stores the video compressed data in the storage module 50.

Wherein, referring to FIG. 4, before step S101, the method may further include:

Reading training video data from the storage module 50;

The depth autoencoder is trained using training video data.

Referring to FIG. 5, the video encoding and decoding method further includes:

In step S201, the controller 10 sends an IO command to the hybrid codec module 40, and the hybrid codec module 40 reads the video compressed data from the storage module 50.

In step S202, the controller 10 sends a decoding instruction to the hybrid codec module 40, and the hybrid codec module 40 decodes the video compressed data to obtain the first decompressed data and the encoded decoding end parameters.

In step S203, the controller 10 sends an IO command to the hybrid codec module 40, and the hybrid codec module 40 stores the first decompressed data and the encoded decoder parameters in the storage module 50.

In step S204, the controller 10 sends an IO command to the neural network codec module 30, and the neural network codec module 30 reads the encoded decoder parameters from the storage module 50.

In step S205, the controller 10 sends a decoding instruction to the neural network codec module 30, and the neural network codec module 30 decodes the encoded decoding end parameters to obtain parameters of the decoding end.

In step S206, the controller 10 sends an IO command to the depth autoencoder module 20, and the deep autoencoder module 20 reads the first decompressed data from the storage module 50, and reads the parameters of the decoding end from the neural network codec module 30.

In step S207, the controller 10 sends a decoding instruction to the depth autoencoder module 20, and the depth autoencoder module 20 performs second decompression on the first decompressed data to obtain original video data.

In step S208, the controller 10 sends an IO command to the depth autoencoder module 20, and the depth autoencoder module 20 stores the original video data in the storage module 50.

In one embodiment, the present disclosure discloses a chip that includes the video codec described above.

In one embodiment, the present disclosure discloses a chip package structure that includes the chip described above.

In one embodiment, the present disclosure discloses a board that includes the chip package structure described above.

In one embodiment, the present disclosure discloses an electronic device that includes the above-described card.

Electronic devices include data processing devices, robots, computers, printers, scanners, tablets, smart terminals, mobile phones, driving recorders, navigators, sensors, cameras, cloud servers, cameras, cameras, projectors, watches, headphones, mobile Storage, wearable device vehicles, household appliances, and/or medical devices.

The vehicle includes an airplane, a ship, and/or a vehicle; the household appliance includes a television, an air conditioner, a microwave oven, a refrigerator, a rice cooker, a humidifier, a washing machine, an electric lamp, a gas stove, a range hood; the medical device includes a nuclear magnetic resonance instrument, B-ultrasound and / or electrocardiograph.

In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of a software program module.

The integrated unit, if implemented in the form of a software program module and sold or used as a standalone product, may be stored in a computer readable memory. Based on such understanding, the technical solution of the present disclosure may be embodied in the form of a software product in the form of a software product in essence or in the form of a contribution to the prior art, and the computer software product is stored in a memory. A number of instructions are included to cause a computer device (which may be a personal computer, server or network device, etc.) to perform all or part of the steps of the methods described in the various embodiments of the present disclosure. The foregoing memory includes: a U disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a removable hard disk, a magnetic disk, or an optical disk, and the like, which can store program codes.

Each functional unit/module may be hardware, such as the hardware may be a circuit, including digital circuits, analog circuits, and the like. Physical implementations of hardware structures include, but are not limited to, physical devices including, but not limited to, transistors, memristors, and the like. The computing modules in the computing device can be any suitable hardware processor, such as a CPU, GPU, FPGA, DSP, ASIC, and the like. The storage unit may be any suitable magnetic storage medium or magneto-optical storage medium such as RRAM, DRAM, SRAM, EDRAM, HBM, HMC, and the like.

The processes or methods depicted in the preceding figures may include hardware (eg, circuitry, dedicated logic, etc.), firmware, software (eg, software embodied on a non-transitory computer readable medium), or both The combined processing logic is executed. Although the processes or methods have been described above in some order, it should be understood that certain operations described can be performed in a different order. Moreover, some operations may be performed in parallel rather than sequentially.

It should be noted that the implementations that are not shown or described in the drawings or the text of the specification are all known to those of ordinary skill in the art and are not described in detail. In addition, the above definitions of the various elements are not limited to the specific structures and shapes mentioned in the embodiments, and those skilled in the art may simply change or replace them; an example of parameters including specific values may be provided herein. However, these parameters need not be exactly equal to the corresponding values, but may approximate the corresponding values within acceptable error tolerances or design constraints; the directional terms mentioned in the examples, such as "upper", "lower", "front" , "post", "left", "right", etc., are merely referring to the directions of the drawings, and are not intended to limit the scope of protection of the present disclosure; the above embodiments may be used in combination or in combination with each other based on design and reliability considerations. Other embodiments are used in combination, that is, the technical features in different embodiments can be freely combined to form more embodiments.

The specific embodiments of the present invention have been described in detail with reference to the specific embodiments of the present disclosure. It is understood that the above description is only the specific embodiment of the disclosure, and is not intended to limit the disclosure. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of this disclosure are intended to be included within the scope of the disclosure.

Claims (39)

A video encoding and decoding device based on a depth auto-encoder, comprising: a depth auto-encoder module, including a depth auto-encoder, the depth auto-encoder includes an encoding end, and the encoding end is used for first compressing the original video to obtain first-time compressed data; a neural network codec module, configured to encode and compress parameters of the decoding end, and generate encoded decoding end parameters; The hybrid codec module is configured to perform hybrid coding on the first compressed data and the encoded decoding end parameters to obtain video compressed data. The video codec device of claim 1 wherein: The coding end is an N-layer artificial neural network structure. The video encoding and decoding apparatus according to claim 2, wherein the first layer of the N-layer artificial neural network structure is an input layer, and the second to N layers are hidden layers, and the inter-layer units are fully connected, and the layers are The unit has no connection, and the number of hidden units of the hidden layer of the Nth layer is smaller than the number of input units of the input layer. The video codec device according to any one of claims 1 to 3, wherein the hybrid coding comprises entropy coding. The video encoding and decoding apparatus according to claim 4, wherein said entropy encoding comprises Huffman encoding. The video codec device according to any one of claims 1 to 5, further comprising: And a storage module, configured to store the first compressed data, the decoding end parameter, and the video compressed data. The video codec device of claim 6 wherein: The neural network codec module is configured to read the decoding end parameter from the storage module to encode and compress the decoding end parameter. A video encoding and decoding apparatus according to claim 6 or 7, wherein: The hybrid codec module is configured to read the first compressed data from the storage module, and read the encoded decoding end parameters from the neural network codec module to perform the hybrid encoding, and Video compression data is stored to the storage module. The video encoding and decoding apparatus according to any one of claims 2 to 8, wherein the depth automatic encoder further comprises: a decoding end; The hybrid codec module is further configured to decode video compressed data to obtain first decompressed data and encoded decoding end parameters; The neural network codec module is further configured to decode the encoded decoding end parameter to obtain a decoding end parameter; The decoding end is configured to decode the first decompressed data to obtain original video data. The video encoding and decoding apparatus according to claim 9, wherein the storage module is further configured to store the first decompressed data, the encoded decoding end parameter, and the original video data. A video encoding and decoding apparatus according to claim 10, wherein The hybrid codec module is further configured to read the video compressed data from the storage module to decode the video compressed data. The video encoding and decoding apparatus according to claim 10 or 11, wherein the neural network codec module is further configured to read the encoded decoding end parameter from the storage module, after the encoding The decoding end parameters are decoded. The video encoding and decoding apparatus according to any one of claims 10 to 12, wherein the depth autoencoder module is further configured to read the first decompressed data from the storage module, from the neural network. The decoding module reads the parameters of the decoding end to cause the decoding end to decode the first decompressed data. The video encoding and decoding apparatus according to any one of claims 9 to 13, wherein the decoding end is an N-layer artificial neural network structure symmetric with the coding end structure. The video encoding and decoding apparatus according to claim 14, wherein the nth layer of the decoding end is the (N-n+1)th layer of the encoding end, and the nth layer and the n+1th of the decoding end are The weight matrix between the layers is a transposition of the weight matrix between the (Nn)th layer and the (N-n+1)th layer of the encoding end, where 1≤n≤N. A video encoding and decoding apparatus according to any one of claims 1 to 15, wherein The depth autoencoder module is further configured to initialize the depth auto-encoder, and train the depth auto-encoder by using training video to obtain a depth auto-encoder for video coding. The video encoding and decoding apparatus according to claim 16, wherein the depth autoencoder module is further configured to train the depth autoencoder by using training video, including: The adjacent two layers of the code end of the depth autoencoder are used as a limited Boltzmann machine; Initializing the restricted Boltzmann machine; Training the restricted Boltzmann machine with the training video data; The weight matrix of the codec of the depth autoencoder is finely adjusted by a back propagation algorithm to minimize the reconstruction error to the original input. The video codec device according to any one of claims 1 to 17, further comprising: And a controller interconnected with the depth autoencoder module, the neural network codec module, and the hybrid codec module for controlling the module. A video encoding method based on a depth auto-encoder, wherein the video encoding and decoding apparatus according to any one of claims 1 to 18 is used for video encoding, comprising: The first compression of the original video, the first compressed data; Encoding and compressing the parameters of the decoding end to obtain the encoded decoding end parameters; The first compressed data and the encoded decoding end parameters are mixed and encoded to obtain video compressed data. The video encoding and decoding method according to claim 19, wherein the original video is first compressed using a first N-layer artificial neural network structure. The video encoding and decoding method according to claim 20, wherein the first layer of the first N-layer artificial neural network structure is an input layer, the second to N layers are hidden layers, and the inter-layer units are fully connected. There is no connection in the intra-layer unit, and the number of hidden units in the hidden layer of the N-th layer is smaller than the number of input units in the input layer. The video encoding and decoding method according to any one of claims 19 to 21, wherein the hybrid encoding comprises entropy encoding. The video encoding and decoding method according to claim 22, wherein said entropy encoding comprises Huffman encoding. The video encoding and decoding method according to any one of claims 19 to 23, further comprising: The first compressed data, the decoding end parameters, and the video compressed data are stored. A video encoding and decoding method according to claim 24, wherein The decoding end parameter is read to encode and compress the decoding end parameter. A video encoding and decoding method according to claim 24 or 25, wherein The first compressed data and the encoded decoding end parameters are read to perform the hybrid encoding, and the video compressed data is stored. The video encoding and decoding method according to any one of claims 20 to 26, further comprising: Decoding the video compressed data to obtain first decompressed data and encoded decoding end parameters; Decoding the encoded decoding end parameter to obtain a decoding end parameter; Decoding the first decompressed data to obtain original video data. The video encoding and decoding method according to claim 27, further comprising: storing the first decompressed data, the encoded decoding end parameters, and the original video data. A video encoding and decoding method according to claim 28, wherein The video compressed data is read to decode the video compressed data. The video encoding and decoding method according to claim 28 or 29, wherein the encoded decoding end parameter is read to decode the encoded decoding end parameter. The video encoding and decoding method according to any one of claims 28 to 30, wherein the first decompressed data and the parameters of the decoding end are read to decode the first decompressed data. The video encoding and decoding method according to any one of claims 27 to 31, wherein the first decompressed data is decoded by using a second N-layer artificial neural network structure, and the second N-layer artificial neural network structure and The structure of the first N-layer artificial neural network is symmetrical. The video encoding and decoding method according to claim 32, wherein the nth layer of the second N-layer artificial neural network structure is the (N-n+1) of the first N-layer artificial neural network structure a weight matrix between the nth layer and the n+1th layer of the second N layer artificial neural network structure, which is the (Nn)th layer and the (N-n+) of the first N layer artificial neural network structure 1) Transposition of a weight matrix between layers, where 1 ≤ n ≤ N. The video encoding method according to any one of claims 19 to 33, further comprising: before the first compression of the original video: Initialize the depth autoencoder; The depth autoencoder is trained using training video data. The video encoding method according to claim 34, wherein said training the depth auto-encoder by using video data for training comprises: The adjacent two layers of the coding end of the depth autoencoder are used as a limited Boltzmann machine; Initializing the restricted Boltzmann machine; Training the restricted Boltzmann machine with the training video data; The weighting matrix of the encoding end of the depth autoencoder is adjusted by a backpropagation method to minimize the reconstruction error of the original input. The video encoding and decoding method according to any one of claims 19 to 35, further comprising: controlling the step by using a controller. A video encoding and decoding method according to any one of claims 19 to 36, characterized in that Performing a forward operation on each layer in the N-layer artificial neural network structure in sequence; Performing inverse operations on the layers in the N-layer artificial neural network structure in the reverse order of the forward operation; Performing weight update on each layer in the N-layer artificial neural network structure; The above steps are repeatedly performed multiple times to complete the training of the N-layer artificial neural network structure. The video encoding and decoding method according to claim 37, wherein the performing the inverse operations sequentially on the layers in the N-layer artificial neural network structure comprises: a first operation portion: obtaining a weight gradient from the output neuron gradient and the input neuron; A second operation portion: calculating an input neuron gradient using the output neuron gradient and weight. The video encoding and decoding method according to claim 38, wherein the performing weight update on each layer in the N-layer artificial neural network structure comprises: updating the weight by using the weight gradient , get the updated weight.

Images