TW202446075A - Efficient warping-based neural video coder - Google Patents

Abstract

An example computing device may include memory and one or more processors. The one or more processors may be configured to parallel entropy decode encoded video data from a received bitstream to generate entropy decoded data. The one or more processors may be configured to predict a motion vector based on the entropy decoded data. The one or more processors may be configured to decode a motion vector residual from the entropy decoded data. The one or more processors may be configured to add the motion vector residual and motion vector. The one or more processors may be configured to warp previous reconstructed video data with an overlapped block-based warp function using the motion vector to generate predicted current video data. The one or more processors may be configured to sum the predicted current video data with a residual block to generate current reconstructed video data.

Description

Efficient Twisted-based neural video codec

This application claims the benefit of U.S. Provisional Application No. 63/489,306, filed on March 9, 2023, and U.S. Provisional Application No. 63/497,411, filed on April 20, 2023, the entire contents of each of which are incorporated herein by reference.

The present disclosure relates to video encoding and decoding, including encoding and decoding of image and video data.

Digital media capabilities can be incorporated into a wide range of devices, including digital televisions, digital live broadcast systems, wireless broadcast systems, personal digital assistants (PDAs), laptop or desktop computers, tablet computers, e-book readers, digital cameras, digital recording devices, digital media players, video game equipment, video gaming consoles, cellular or satellite telephones, so-called "smart phones", video teleconferencing equipment, video streaming equipment, and many others. Digital video devices implement video codec technologies, such as those described in the standards defined by MPEG-2, MPEG-4, ITU-T H.263, ITU-T H.264/MPEG-4 Part 10 Advanced Video Codec (AVC), ITU-T H.265/High Efficiency Video Codec (HEVC), ITU-T H.266/Versatile Video Codec (VVC), and extensions of such standards, as well as proprietary video codecs/formats, such as AOMedia Video 1 (AV1) developed by the Alliance for Open Media. Video devices can more efficiently send, receive, encode, decode and/or store digital video information by implementing such video codec technologies.

In general, this disclosure describes techniques for media compression, including techniques for video and/or image encoding and decoding. Neural network-based media (e.g., image and/or video) compression methods can compete with current standards and provide several advantages. Neural-based encoding and decoding techniques are typically designed and tested using high-precision floating-point mathematics. However, as neural network-based media compression techniques enter actual implementations and deployments, neural network weights and activation functions are often quantized and represented using low-precision integers rather than high-precision floating-point numbers to improve speed and power consumption.

This disclosure addresses the problem that occurs when quantizing neural network variables associated with entropy coding and decoding. Neural network variables may be important for the design of neural-based video/image compression schemes, because such variables define the compression efficiency. In addition, general tools for optimizing quantization in neural networks do not consider the very specific properties of entropy coding and decoding variables. Tests show that the worst quantization effects may happen to occur on some of the most common use cases, and the loss caused by the worst quantization effects may not be recovered by retraining the neural network.

This disclosure describes techniques for optimizing the definition of trained entropy coding and decoding variables so that the information most important for efficient entropy coding and decoding is best preserved when represented by low-precision integers. Tests also show how the techniques described herein can be used to reduce or minimize the amount of memory required for entropy coding and decoding. This disclosure describes a general approach for entropy coding and decoding design, as well as specific solutions and implementations for commonly used Gaussian distributions. The techniques of this disclosure can generally be applied to any neural-based compression technology, but the examples described below focus on techniques for image and video compression.

In one example, a device for decoding video data includes: a memory for storing video data, the video data including previously reconstructed video data and currently reconstructed video data; and one or more processors configured to: perform parallel entropy decoding on encoded video data from a received bit stream to generate entropy decoded data; predict block-based motion vectors based on the entropy decoded data to generate predicted motion vector; decoding a motion vector residual from the entropy decoded data; adding the motion vector residual to the predicted motion vector to generate the block-based motion vector; using the block-based motion vector to warp the previously reconstructed video data with an overlapping block-based warping function to generate predicted current video data; and summing the predicted current video data with the residual block to generate the current reconstructed video data.

In another example, a method for decoding video data includes: performing parallel entropy decoding on encoded video data from a received bit stream to generate entropy decoded data; predicting block-based motion vectors based on the entropy decoded data to generate predicted motion vectors; decoding motion vector residues from the entropy decoded data; adding the motion vector residues to the predicted motion vectors to generate the block-based motion vectors; using the block-based motion vectors to warp previously reconstructed video data using a warping function based on overlapping blocks to generate predicted current video data; and summing the predicted current video data with the residue block to generate current reconstructed video data.

In another example, an apparatus for decoding video data includes: an apparatus for performing parallel entropy decoding on coded video data from a received bit stream to generate entropy decoded data; an apparatus for predicting block-based motion vectors based on the entropy decoded data to generate predicted motion vectors; an apparatus for decoding motion vector residues from the entropy decoded data; and a method for extracting motion vector residues from the entropy decoded data. A device for adding the motion vector residual to the predicted motion vector to generate the block-based motion vector; a device for using the block-based motion vector to warp previously reconstructed video data with a warping function based on overlapping blocks to generate predicted current video data; and a device for summing the predicted current video data with the residual block to generate current reconstructed video data.

In another example, a computer-readable storage medium is encoded with instructions that, when executed, cause a programmable processor to: perform parallel entropy decoding on encoded video data from a received bitstream to produce entropy-decoded data; predict block-based motion vectors based on the entropy-decoded data to generate predicted motion vectors; and decoding motion vector residuals; adding the motion vector residuals to the predicted motion vectors to generate the block-based motion vectors; using the block-based motion vectors to warp previously reconstructed video data with a warping function based on overlapping blocks to generate predicted current video data; and summing the predicted current video data with the residual block to generate current reconstructed video data.

The details of one or more examples of the present disclosure are described in the accompanying drawings and the following description. Other features, objectives and advantages of the present disclosure will be apparent from the description and the accompanying drawings as well as from the scope of the application.

This disclosure describes techniques for encoding and decoding media data (e.g., images or videos) using neural network-based media codec techniques. Specifically, this disclosure describes techniques for decoding encoded media data using warping. Specifically, this disclosure describes techniques for neural network-based media codecs using block-based warping. Example techniques of this disclosure include 1080p YUV420 architecture, predictive modeling for improving compression performance, quantization-aware training, parallel entropy coding (e.g., on a GPU), and/or pipelined inferencing. The techniques of this disclosure can improve the performance of neural network-based media codecs. This improved neural network-based media codec can be used in battery-powered devices such as mobile devices (e.g., smartphones).

Neural video codecs have recently become competitive with standard codecs (e.g., HEVC) in low-latency settings. However, most neural codecs consist of large floating-point networks that use pixel-dense warp operations for temporal modeling, making them too computationally intensive for deployment on mobile devices. This disclosure describes techniques for the adaptation of strong neural encoders/decoders (codecs) for mobile deployment.

Existing neural compression models for video codecs exhibit relatively good (e.g., high quality) compression performance, but are relatively computationally expensive to run, especially on battery-powered devices (such as mobile devices). Thus, there may be a need for a media codec, such as a video codec, that has a relatively low computational footprint while also providing relatively good compression performance.

According to the technology of the present disclosure, a relatively efficient model architecture is provided for 1080p YUV420 video. Such an architecture uses a prediction model to improve compression performance. The video codec can adopt quantization-aware training. The video codec can adopt efficient block-based warping. The video codec can adopt parallel entropy coding and decoding on the GPU (for example, in addition to or instead of entropy coding and decoding on the CPU). In some examples, the video codec can use pipeline inference to achieve a throughput of >30 fps for 1080x2048 video frames.

Specifically, the present disclosure describes techniques for performing low-precision weights and activation quantization in a mean-scale hyperprior model. The present disclosure also describes an efficient overlapping block motion compensation technique that can be used in a neural accelerator to replace pixel-dense distortion. The techniques of the present disclosure can provide a codec that may outperform other practical neural codecs by a relatively large margin, with up to 40% Bjontegaard-Delta (BD) rate savings, while reducing the FLOP count to approximately 1/9 on the receiver side. This reduction in complexity allows for expansion to real-time full-HD video decoding on the device. In addition, the resulting codec can operate in a perceptually relevant YUV color space and perform entropy encoding and decoding in parallel on a mobile GPU. This disclosure also describes ablations of encoder architectures to provide a discussion of the effects of motion compensation schemes and quantization.

Significant progress has been made in neural video compression in recent years. In the low-latency P-frame setting, recent neural network-based compression techniques outperform reference implementations of standard codecs such as ITU-T H.265/HEVC. However, current neural codecs are often computationally expensive, encoding and decoding are not real-time, and reported runtimes are often measured on powerful desktop GPUs. In addition, many neural video codecs assume the use of pixel-based or feature-based warping operations, which can be memory intensive and difficult to implement efficiently on resource-constrained devices such as mobile phones.

On the other hand, standard codecs typically have fast software implementations, or efficient silicon implementations designed specifically for consumer hardware. Although efficient neural codecs exist, they typically (1) replace dense optical flow warps with convolutional motion compensation networks, and (2) use a scale-only hyperprior. However, both choices have a negative impact on rate-distortion (R-D) performance, even before applying weights and enabling quantization.

This disclosure describes a neural P-frame codec architecture designed for deployment in mobile devices (sometimes referred to as "QODEC" or quantized on-device end-to-end codec). In some examples, the architecture can include three mean-scaled hyper-prior and prediction networks for flow and residue. Reducing model width, removing residue predictors, and removing redundant warping operations can reduce computational complexity. The warping operator itself can be efficiently implemented using a block-based motion compensation algorithm.

In some examples, the quantization weights and activations may be 8-bit values, which may further improve efficiency. The disclosed techniques may use an efficient quantization scheme for scaling of each mean-scaled super-prior. However, naive quantization of the super-prior mean may result in a catastrophic loss of R-D performance. Therefore, instead, the disclosed techniques describe an alternative technique involving low-precision quantization of the mean-scaled super-prior mean and scaling parameters. GPUs may perform parallel entropy encoding and decoding to massively increase parallelism to tens of thousands of threads, allowing extremely efficient entropy decoding on mobile devices.

These techniques can lead to BD rate savings while enabling, for example, 30 fps full HD real-time decoding on mobile devices. Additionally, the warp operator and quantization can be chosen, which allows determining key factors for an efficient, mobile-friendly design of a neural codec.

A neural codec is a machine learning system based on a neural network that is trained to compress data from examples. One model used for neural image compression is the mean-scaled hyper-prior. This model is a hierarchical variational autoencoder with quantized latent variables, sometimes called a compressive autoencoder.

After their success in the image domain (e.g., encoding and decoding of still images), neural codecs were extended to the video setting. Subsequent research used motion compensation and residue codecs using task-specific autoencoders. These architectures were further enhanced using prediction models of the prediction stream, the residue, or both, leading to improvements in RD performance. Recent work has shown that conditional codecs can be more powerful than residue codecs, but conditional codecs can lead to convergence errors.

Neural image codecs outperform the strongest standard image codecs, but typically, their powerful performance comes with increased computational complexity. Although many works report runtime or memory usage, the deployment of neural image codecs has received less attention.

A common approach to reducing complexity is quantization. For neural compression, cross-platform reproducibility of neural quantization has been studied, as entropy encoding decoding is sensitive and may break due to floating-point rounding errors. Post-training quantization has been studied for both weights and activations, with the goal of improving the BD-rate gap between integer quantized models and their floating-point counterparts. For example, channel splitting involves splitting and quantizing the convolution output channels that are most sensitive to quantization using a custom dynamic range, while pruning away other channels to keep the floating-point operation (FLOP) complexity somewhat constant. However, such techniques typically assume per-channel quantization. Studies have shown that per-channel weight quantization can benefit from efficient integer algorithms.

Because the techniques of this disclosure can be implemented on a device, the techniques of this disclosure can use hardware-friendly per-channel weight and per-layer activation quantization schemes for the final model. Bottleneck quantization in a mean-scaled hyper-prior structure can be implemented in various ways, and may require careful consideration when quantization is enabled and entropy coding and decoding is performed. This disclosure describes how to perform quantization in the bottleneck, especially for latent and mean paths. This disclosure also describes how parameterization of scaling parameters can achieve essentially no loss in rate performance when 8-bit quantization is performed on scaling.

In the video setting, some techniques measure computational complexity by runtime or multiply-add operations. For example, specific convolution blocks can be used to improve inference speed and BD rate. Overfitting the codec to instances for compression can significantly reduce receiver-side computational complexity. A neural codec that decodes video on the device can use per-channel model quantization and parallel entropy encoding and decoding, including a motion compensation subnetwork.

The disclosed techniques include efficient block-based warping techniques combined with a prediction model architecture in YUV-420 space. In addition, these techniques include massively parallel entropy coding and decoding on GPUs. Therefore, these techniques can decode full HD video (1080p resolution) at 30fps on mobile devices.

FIG. 1 is a block diagram illustrating an example media encoding and decoding system that may perform the techniques of the present disclosure. In the context of the present disclosure, media may include any digital file to be compressed, including video data and/or images. The example techniques of the present disclosure generally relate to encoding and decoding (encoding and/or decoding) video data and/or image data. Although the example of FIG. 1 will be described with reference to media encoding and decoding, the techniques of the present application are equally applicable to encoding and decoding any type of data file using neural-based compression techniques.

1 , in this example, system 100 includes a source device 102 that provides encoded media data to be decoded and displayed by a destination device 116. In particular, source device 102 provides the media data to destination device 116 via computer-readable medium 110. Source device 102 and destination device 116 may include any of a wide range of devices, including desktop computers, notebook (i.e., laptop) computers, mobile devices, tablet computers, set-top boxes, telephone handsets such as smart phones, televisions, cameras, display devices, digital media players, video game consoles, video streaming devices, broadcast receiver devices, or the like. In some cases, source device 102 and destination device 116 may be equipped for wireless communications, and thus may be referred to as wireless communication devices.

In the example of Fig. 1, source device 102 includes video source 104, memory 106, video encoder 200 and output interface 108. Destination device 116 includes input interface 122, video decoder 300, memory 120 and display device 118. According to the present disclosure, the video encoder 200 of source device 102 and the video decoder 300 of destination device 116 can be configured to apply the technology for entropy encoding and decoding of neural-based media compression system. Therefore, source device 102 represents an example of video encoding device, and destination device 116 represents an example of video decoding device. In other examples, source device and destination device may include other components or arrangements. For example, source device 102 may receive video data from an external video source (e.g., an external camera). Similarly, destination device 116 may interface with an external display device rather than including an integrated display device.

As shown in FIG. 1 , system 100 is only an example. In general, any digital media encoding and/or decoding device can implement techniques for entropy encoding and decoding of a neural-based media compression system. Source device 102 and destination device 116 are only examples of such codec devices where source device 102 generates encoded and decoded video data for transmission to destination device 116. This disclosure refers to a "codec" device as a device that performs encoding and decoding (encoding and/or decoding) of data. Therefore, video encoder 200 and video decoder 300 represent examples of codec devices, respectively, a video encoder and a video decoder. In some examples, source device 102 and destination device 116 can operate in a substantially symmetrical manner such that each of source device 102 and destination device 116 includes video encoding and decoding components. Thus, system 100 can support one-way or two-way video transmission between source device 102 and destination device 116, such as for video streaming, video playback, video broadcasting, or video telephony.

Generally, video source 104 represents a source of video data (i.e., raw, unencoded video data) and provides a series of consecutive pictures (also referred to as "frames") of the video data to video encoder 200, which encodes the picture data. Video source 104 of source device 102 may include a video capture device such as a camera, a video archive containing previously captured raw video, and/or a video feed interface for receiving video from a video content provider. As another alternative, video source 104 may generate computer graphics-based data as source video, or a combination of live video, archived video, and computer-generated video. In each case, video encoder 200 encodes captured, pre-captured, or computer-generated video data. Video encoder 200 may rearrange the pictures from the order in which they are received (sometimes referred to as "display order") into a codec order for encoding and decoding. Video encoder 200 may generate a bitstream comprising the encoded video data. Source device 102 may then output the encoded video data onto computer-readable medium 110 via output interface 108 for receipt and/or retrieval by, for example, input interface 122 of destination device 116 .

The memory 106 of the source device 102 and the memory 120 of the destination device 116 represent general purpose memories. In some examples, the memories 106, 120 can store raw video data, such as raw video from the video source 104 and raw decoded video data from the video decoder 300. Additionally or alternatively, the memories 106, 120 can store software instructions that can be executed by, for example, the video encoder 200 and the video decoder 300, respectively. Although in this example the memory 106 and the memory 120 are shown separately from the video encoder 200 and the video decoder 300, it should be understood that the video encoder 200 and the video decoder 300 may also include internal memory for functionally similar or equivalent purposes. In addition, the memory 106, 120 may store encoded video data, for example, output from the video encoder 200 and the input interface 122 to the video decoder 300. In some examples, portions of the memory 106, 120 may be allocated as one or more buffers, for example to store original, decoded and/or encoded video data.

Computer-readable medium 110 may represent any type of medium or device capable of transmitting encoded video data from source device 102 to destination device 116. In one example, computer-readable medium 110 represents a communication medium that enables source device 102 to transmit encoded video data directly to destination device 116 in real time, such as via a radio frequency network or a computer-based network. Output interface 108 may modulate a transmit signal containing the encoded video data, and input interface 122 may demodulate a received transmit signal, according to a communication standard (e.g., a wireless communication protocol). The communication medium may include any wireless or wired communication medium, such as a radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium may form part of a packet-based network such as a local area network, a wide area network, or a global network such as the Internet. The communication medium may include routers, switches, base stations, or any other equipment that can be used to facilitate communication from the source device 102 to the destination device 116.

In some examples, source device 102 can output the encoded data from output interface 108 to storage device 112. Similarly, destination device 116 can access the encoded data from storage device 112 via input interface 122. Storage device 112 can include any of a variety of distributed or locally accessed data storage, such as a hard drive, Blu-ray Disc, DVD, CD-ROM, flash memory, volatile or non-volatile memory, or any other suitable digital storage for storing encoded video data.

In some examples, source device 102 may output the encoded video data to file server 114 or another intermediate storage device that may store the encoded video data generated by source device 102. Destination device 116 may access the stored video data from file server 114 via streaming or downloading.

The file server 114 may be any type of server device capable of storing encoded video data and sending the encoded video data to the destination device 116. The file server 114 may represent a web server (e.g., for a website), a server configured to provide file transfer protocol services (e.g., File Transfer Protocol (FTP) or File Delivery over One-Way Transport (FLUTE) protocol), a content delivery network (CDN) device, a hypertext transfer protocol (HTTP) server, a Multimedia Broadcast Multicast Service (MBMS) or enhanced MBMS (eMBMS) server, and/or a network attached storage (NAS) device. The file server 114 may additionally or alternatively implement one or more HTTP streaming protocols, such as Dynamic Adaptive Streaming over HTTP (DASH), HTTP Live Streaming (HLS), Real-Time Streaming Protocol (RTSP), HTTP Dynamic Streaming, or the like.

Destination device 116 may access the encoded video data from file server 114 through any standard data connection, including an Internet connection. This may include a wireless channel (e.g., a Wi-Fi connection), a wired connection (e.g., a digital subscriber line (DSL), a cable modem, etc.), or a combination of both that is suitable for accessing the encoded video data stored on file server 114. Input interface 122 may be configured to operate according to any one or more of the various protocols discussed above for retrieving or receiving video data from file server 114 or other such protocols for retrieving video data.

The output interface 108 and the input interface 122 may represent wireless transmitters/receivers, modems, wired networking components (e.g., Ethernet cards), wireless communication components operating according to any of the various IEEE 802.11 standards, or other physical components. In examples where the output interface 108 and the input interface 122 include wireless components, the output interface 108 and the input interface 122 may be configured to transmit data (e.g., encoded video data) according to a cellular communication standard (e.g., 4G, 4G-LTE (Long Term Evolution), LTE Advanced, 5G, or the like). In some examples where the output interface 108 includes a wireless transmitter, the output interface 108 and the input interface 122 may be configured to transmit data (e.g., encoded video data) according to other wireless standards (e.g., IEEE 802.11 specifications, IEEE 802.15 specifications (e.g., ZigBee™), Bluetooth™ standards, or the like). In some examples, the source device 102 and/or the destination device 116 may include corresponding single-chip system (SoC) devices. For example, the source device 102 may include a SoC device for performing functions attributed to the video encoder 200 and/or the output interface 108, and the destination device 116 may include a SoC device for performing functions attributed to the video decoder 300 and/or the input interface 122.

The techniques disclosed herein may be applied to video encoding in support of any of a variety of multimedia applications, such as over-the-air television broadcasting, cable television transmission, satellite television transmission, Internet streaming video transmission (e.g., Dynamic Adaptive Streaming over HTTP (DASH)), digital video encoded to data storage media, decoding of digital video stored on data storage media, or other applications.

The input interface 122 of the destination device 116 receives the encoded video bit stream from the computer-readable medium 110 (e.g., communication media, storage device 112, file server 114, etc.). The encoded video bit stream may include signaling information defined by the video encoder 200, which is also used by the video decoder 300. The display device 118 displays the decoded picture of the decoded video data to the user. The display device 118 may represent any of a variety of display devices, such as a liquid crystal display (LCD), a plasma display, an organic light emitting diode (OLED) display, or another type of display device.

According to the technology of the present disclosure, the video decoder 300 can be an efficient encoder on a quantized device. The video decoder 300 can use a hyper-prior model based on distortion and residue designed for YUV 4:2:0 color space that runs efficiently on the device. The video decoder 300 can use an efficient block-based distortion algorithm and take advantage of the lower dimensionality of stream and pixel space inputs, which can reduce model complexity. In addition, the video decoder 300 can use 8-bit integers to model weights and activations. The video decoder 300 can also use parallel entropy encoding and decoding algorithms on GPU cores and use pipeline inference algorithms that take advantage of multiple available cores in a neural processor. Together, these various techniques may allow the video decoder 300 to decode full HD video (1080×1920) at greater than 30 fps.

For example, the video decoder 300 may receive a YUV 4:2:0 input and use a flow predictor. The YUV color space may be better aligned with human perception quality than other color spaces, and the 4:2:0 subsampling scheme may take advantage of differences in human eye sensitivity between brightness and color. In particular, the video decoder 300 may subsample the chroma channels by a factor of two (2x) along the height and width dimensions, resulting in a 2x reduction in the number of elements compared to RGB or YUV 4:4:4 color spaces, which in turn reduces network complexity.

Although not shown in FIG. 1 , in some examples, the video encoder 200 and the video decoder 300 may each be integrated with an audio encoder and/or an audio decoder and may include appropriate MUX-DEMUX units or other hardware and/or software to process multiplexed streams including both audio and video in a common data stream.

The video encoder 200 and the video decoder 300 can each be implemented as any of a variety of suitable encoder and/or decoder circuits, such as one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic, software, hardware, firmware, or any combination thereof. When the technology is partially implemented in software, the device can store instructions for the software in a suitable non-transitory computer-readable medium and execute the instructions in hardware using one or more processors to perform the technology of the present disclosure. Each of the video encoder 200 and the video decoder 300 may be included in one or more encoders or decoders, either of which may be integrated as part of a combined encoder/decoder (CODER) in a corresponding device. A device including the video encoder 200 and/or the video decoder 300 may include an integrated circuit, a microprocessor, and/or a wireless communication device, such as a cellular phone.

Entropy coding and decoding is an essential part of video compression systems. The entropy coding and decoding process is responsible for optimizing the conversion between video information and compressed data bit stream, aiming to obtain the most compact representation possible. Unlike other elements of video compression, entropy coding and decoding is a lossless process, that is, it fully preserves the information.

Several techniques have been developed to achieve efficient entropy coding and decoding in image and video compression standards. Recently, it has been shown that new compression methods based on deep learning and neural networks are approaching the performance of conventional methods while offering several other practical advantages.

FIG. 2 is a block diagram illustrating an example computing device that may implement the techniques of the present disclosure. Computing device 402 may include a mobile device (such as, for example, a smart phone, a mobile phone, a cellular phone, a satellite phone, and/or a mobile phone handset), a personal computer, a desktop computer, a laptop computer, a computer workstation, a video game platform or console, a landline phone, an Internet phone, a handheld device (such as a portable video game device or a personal digital assistant (PDA)), a personal music player, a video player, a display device, a television, a television set-top box, a server, a middle network device, a mainframe computer, a mobile computing device, a vehicle head end unit, a self-driving or autonomous vehicle, a robot, or any other type of device with imaging or video capabilities.

2 , computing device 402 includes user input interface 404, CPU 406, memory controller 408, system memory 410, graphics processing unit (GPU) 412, neural network signal processor (NSP) 430, frame interpolation (FINT) core 432 that may be implemented in NSP 430, local memory 414, display interface 416, display 418, bus 420, and one or more cameras 424. User input interface 404, CPU 406, memory controller 408, GPU 412, FINT core 432, NSP 430, display interface 416, and one or more cameras 424 may communicate with each other using bus 420. The bus 420 may be any of a variety of bus structures, such as a third generation bus (e.g., a SuperTransport bus or an InfiniBand bus), a second generation bus (e.g., an Advanced Graphics Port bus, a Peripheral Component Interconnect (PCI) Express bus, or an Advanced eXtensible Interface (AXI) bus), or another type of bus or device interconnect. It should be noted that the specific configuration of the bus and communication interface between the different components shown in FIG. 2 is merely exemplary, and other configurations of computing devices and/or other graphics processing systems having the same or different components may be used to implement the techniques of the present disclosure.

The one or more cameras 424 may include any image capture hardware including one or more image sensors and one or more lenses, and is configured to capture at least one frame of image data and transmit at least one frame of image data to the CPU 406, GPU 412, NSP 430 and/or FINT core 432.

(Multiple) CPU 406 may include one or more general and/or special purpose processors that control the operation of computing device 402. A user may provide input to computing device 402 to cause CPU 406 to execute one or more software applications. Software applications executed on CPU 406 may include, for example, an operating system, a word processor application, an email application, a spreadsheet application, a media player application, a video game application, a graphical user interface application, and/or other programs. A user may provide input to computing device 402 via one or more input devices (not shown), such as a keyboard, a mouse, a microphone, a touchpad, or another input device coupled to computing device 402 via user input interface 404.

The memory controller 408 facilitates the transfer of data into and out of the system memory 410. For example, the memory controller 408 can receive memory read and write commands and service such commands with respect to the system memory 410 to provide memory services to the components in the computing device 402. The memory controller 408 is communicatively coupled to the system memory 410. Although the memory controller 408 is shown in the example computing device 402 of FIG. 1 as a processing module separate from both the CPU(s) 406 and the system memory 410, in other examples, some or all of the functionality of the memory controller 408 may be implemented on one or both of the CPU(s) 406 and the system memory 410.

System memory 410 may store program modules and/or instructions accessible for execution by CPU 406 and/or data used by programs executing on CPU 406. For example, system memory 410 may store user applications and graphics data associated with the applications. System memory 410 may additionally store information used and/or generated by other components of computing device 402. For example, system memory 410 may serve as device memory for one or more GPUs 412 and may store data to be operated on by GPU 412 and data generated by operations performed by GPU 412. The system memory 410 may include one or more volatile or nonvolatile memory or storage devices, such as random access memory (RAM), static RAM (SRAM), dynamic RAM (DRAM), read-only memory (ROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, magnetic data media, or optical storage media.

In some aspects, the system memory 410 may include instructions that cause the CPU 406, GPU 412, NSP 430, and/or FINT core 432 to perform functions attributed in the present disclosure to the CPU 406, GPU 412, NSP 430, and FINT core 432. Thus, the system memory 410 may be a computer-readable storage medium having stored thereon instructions that, when executed, cause one or more processors (e.g., CPU(s) 406, GPU(s) 412, NSP(s) 430, and FINT core 432) to perform various functions.

In some examples, system memory 410 is a non-transitory storage medium. The term "non-transitory" indicates that the storage medium is not embodied in a carrier or propagated signal. However, the term "non-transitory" should not be interpreted to mean that system memory 410 is non-removable or that its contents are static. As one example, system memory 410 can be removed from computing device 402 and moved to another device. As another example, memory substantially similar to system memory 410 can be inserted into computing device 402. In some examples, non-transitory storage media can store data that can change over time (e.g., in RAM).

The GPU 412 may be configured to perform graphics operations to render one or more graphics primitives to the display 418. Thus, when one of the software applications executing on the CPU(s) 406 requires graphics processing, the CPU(s) 406 may provide graphics commands and graphics data to the GPU(s) 412 for rendering to the display 418. The graphics commands may include, for example, drawing commands (such as draw calls), GPU state programming commands, memory transfer commands, general computation commands, kernel execution commands, etc. In some examples, the CPU 406 may provide the commands and graphics data to the GPU 412 by writing the commands and graphics data to the system memory 410 that may be accessed by the GPU 412. In some examples, GPU(s) 412 may be further configured to perform general purpose computations for applications executing on CPU(s) 406.

In some cases, GPU 412 may be constructed with a highly parallel structure that provides more efficient vector operation processing than CPU 406. For example, GPU(s) 412 may include multiple processing elements configured to operate on multiple vertices or pixels in a parallel manner. In some cases, the highly parallel nature of GPU 412 may allow GPU 412 to draw graphical images (e.g., GUIs and two-dimensional (2D) and/or three-dimensional (3D) graphical scenes) to display 418 faster than using CPU 406 to draw scenes directly to display 418. In addition, the highly parallel nature of GPU(s) 412 may allow GPU(s) 412 to process certain types of vector and matrix operations for general computing applications faster than CPU(s) 406. In some examples, video encoder 200 or video decoder 300 may utilize the highly parallel structure of GPU 412 to perform parallel entropy encoding and decoding according to the techniques of this disclosure.

In some examples, GPU(s) 412 may be integrated into the motherboard of computing device 402. In other examples, GPU(s) 412 may reside on a graphics card that is mounted in a port in the motherboard of computing device 402, or may otherwise be incorporated into a peripheral device configured to interoperate with computing device 402. In further examples, GPU(s) 412 may be located on the same microchip as CPU(s) 406, thereby forming a system on a chip (SoC). GPU 412 and CPU 406 may include one or more processors, such as one or more microprocessors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), digital signal processors (DSPs), or other equivalent integrated or discrete logic circuits.

CPU(s) 406, GPU(s) 412, NSP(s) 430, and FINT core 432 may be collectively referred to as one or more processors 440. When describing various techniques that may be performed by one or more processors 440, it should be understood that such techniques may be performed by one or more of the CPU 406, GPU 412, NSP 430, and FINT core 432. It should be understood that the techniques disclosed herein are not necessarily limited to being performed by the CPU 406, GPU 412, NSP 430, and/or FINT core 432, but may also be performed by any other suitable hardware, device, logic, circuitry, processing unit, etc. of the computing device 402.

The GPU 412 may be directly coupled to the local memory 414. Thus, the (multiple) GPUs 412 may read and write data from and to the local memory 414 without having to use the bus 420. In other words, the (multiple) GPUs 412 may process data locally using local storage devices rather than off-chip memory. This allows the GPUs 412 to operate in a more efficient manner by eliminating the need for the GPUs 412 to read and write data via the bus 420, which may experience heavy bus traffic. However, in some instances, the (multiple) GPUs 412 may not include a separate cache, but instead utilize the system memory 410 via the bus 420. The local memory 414 may include one or more volatile or nonvolatile memory or storage devices, such as random access memory (RAM), static RAM (SRAM), dynamic RAM (DRAM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, magnetic data media, or optical storage media.

As described, CPU(s) 406 may offload graphics processing to GPU(s) 412, such as tasks that require massive parallelism. As one example, graphics processing requires massive parallelism, and CPU 406 may offload such graphics processing tasks to GPU 412. However, other operations, such as matrix operations, may also benefit from the parallel processing capabilities of GPU 412. In these examples, CPU 406 may utilize the parallel processing capabilities of GPU 412 to cause GPU 412 to perform non-graphics related operations.

The CPU(s) 406, the GPU(s) 412, the NSP(s) 430, and/or the FINT core 432 may store the rendered image data in an allocated frame buffer within the system memory 410. The display interface 416 may retrieve the data from the frame buffer and configure the display 418 to display an image represented by the rendered image data. In some examples, the display interface 416 may include a digital/analog converter (DAC) configured to convert the digital values retrieved from the frame buffer into analog signals consumable by the display 418. In other examples, the display interface 416 may pass the digital values directly to the display 418 for processing.

Display 418 may include a monitor, a television, a projection device, a liquid crystal display (LCD), a plasma display panel, a light emitting diode (LED) array, a cathode ray tube (CRT) display, electronic paper, a surface conduction electron emission display (SED), a laser television display, a nanocrystalline display, an organic light emitting diode (OLED) display, or another type of display unit. Display 418 may be integrated into computing device 402. For example, display 418 may be the screen of a mobile phone handset or a tablet computer. Alternatively, display 418 may be a separate device coupled to computing device 402 via a wired or wireless communication link. For example, display 418 may be a computer monitor or flat panel display connected to a personal computer via a cable or wireless link.

The system memory may store a neural network model 422. The neural network model 422 may include one or more artificial neural networks (also referred to as neural networks) trained to receive one or more types of input data and to provide one or more types of output data in response.

A neural network (e.g., neural network model 422) may include a trainable or adaptive algorithm that utilizes nodes that define rules. For example, a corresponding node in a plurality of nodes may utilize a function (e.g., a nonlinear function or an if-then rule) to generate an output based on an input. A corresponding node in a plurality of nodes may be connected to one or more different nodes in a plurality of nodes along an edge so that the output of the corresponding node includes the input of the different node. The function may include parameters that may be determined or adjusted using a training set of inputs and desired outputs and a learning rule (e.g., a back-propagation learning rule). Back-propagation learning rules can utilize one or more error measures that compare a desired output to an output produced by a neural network to train the neural network by varying parameters to minimize the one or more error measures.

In some examples, the neural network model 422 is trained to perform classification of input data. That is, the neural network model 422 can be trained to label the input data to classify the input data into one or more categories or classes. The neural network model 422 can perform classification of the input data by determining a confidence score for each of a plurality of classes for the input data, the confidence score indicating the degree to which the input data is believed to be classified into the corresponding class. In other examples, the neural network model 422 can determine a probability distribution over a set of classes to indicate the probability that the input data belongs to each class in the set of classes.

In some examples, the neural network model 422 can be trained to perform computer vision tasks such as image classification, object detection, and/or image segmentation. Such computer vision tasks may be useful for computer vision applications such as autonomous driving. For example, the neural network model 422 can be trained to perform image classification to determine which objects are in an image or video, such as by being trained to classify an image as including a particular object or not including a particular object and by assigning one or more labels to the image. In another example, the neural network model 422 can be trained to perform object detection to detect objects in an image or video and specify the location of each object in the image, and the neural network model 422 can be trained to assign one or more labels to each of the one or more objects in the image. In some examples, the neural network model 422 can be trained to perform image segmentation to separate an image into regions depicting potentially meaningful regions for further processing.

In some examples, the neural network model 422 can perform one or more computer vision tasks on images captured by the one or more cameras 424. That is, the one or more cameras 424 can capture images, and the one or more processors 440 can input the images captured by the one or more cameras 424 into the neural network model 422 to perform one or more computer vision tasks on the images, such as image classification, object detection, and/or image segmentation.

According to one or more aspects of the present disclosure, the video decoder 300 may be configured to: perform parallel entropy decoding on encoded video data from a received bit stream to generate entropy decoded data; predict block-based motion vectors based on the entropy decoded data to generate predicted motion vectors; decode motion vector residuals from the entropy decoded data; add the motion vector residuals to the predicted motion vectors to generate block-based motion vectors; use the block-based motion vectors to warp the previously reconstructed video data using an overlapping block-based warping function to generate predicted current video data; and sum the predicted current video data with the residual block to generate current reconstructed video data.

According to aspects of the present disclosure, the video decoder 300 can decode a hyperlatent; extrapolate a first stream; decode a mean and a scale from the hyperlatent; decode a latent based on the scale; reconstruct a second stream and a residue based on the latent and the first stream; warp a previously reconstructed frame using the second stream to generate a warped frame; and add the residue to the warped frame.

3 is a conceptual diagram illustrating an example of end-to-end deep learning of a neural video encoder according to one or more aspects of the present disclosure. For example, a video codec (e.g., video codec 200 and/or video decoder 300) can utilize neural network-based I-frame and P-frame compression.

For example, at time T1, the video encoder 200 may obtain an I frame (ground truth (GT) 500) of video data and encode the I frame using a neural network model 422, which may include elements of the video encoder 200 discussed herein. The video encoder 200 may encode the I frame using an I frame encoder (IFE) 502, and entropy encode the encoded I frame using an entropy encoder (EE) 504.

The video decoder 300 may decode the coded I frame using the neural network model 422 to reproduce a representation of the original I frame. For example, the video decoder 300 may entropy decode the entropy coded I frame using an entropy decoder (ED) 506. The video decoder 300 may decode the coded I frame using an I frame decoder (IFD) 508 to produce a reconstructed frame (recon) 510. The video decoder may use the reconstructed frame 510 as an input to a warp function (warp) 512 for use with future decoded P frames.

At time T2 (after time T1), the video encoder 200 may use its own recon 510 representation of the original I frame (the video encoder 200 may include a local version of the video decoder 300) and the P frame (GT 514). The video encoder 200 may use the I frame (recon 510) as a reference frame to perform motion estimation (ME) 516 of the GT 514 (P frame) of the video data. The motion estimation 516 may generate motion vectors and/or motion vector residuals, which may be encoded as part of encoding the P frame. The video encoder 200 may encode the P frame using the neural network model 422. For example, the video encoder 200 may encode the GT 514 using the P-frame encoder (PFE) 518. Then, the video encoder 200 may entropy encode the P-frame using the entropy encoder 504.

The video decoder 300 may entropy decode the entropy encoded P frame using an entropy decoder 506. The video decoder 300 may decode the output of the entropy decoder using a neural network model 422 to generate motion vectors and block residues associated with the P frame. For example, the video decoder 300 may use a P frame decoder (PFD) 520 to generate motion vectors (MVs) 522 and block residues (resid) 524. The video decoder 300 may use a warp 512 to warp the reconstructed I frame 510 using the motion vectors 522 and use the resulting warped frame to sum with the block residues 524 to generate a reconstructed P frame (recon 528). The output 530 of the neural network model 422 (e.g., decoded P frame information, such as motion vectors 522, block residues 524, or other decoded P frame information) can be fed back to the neural network (e.g., the P frame encoder 518 and/or the P frame decoder 520 of the neural network model 422) to train the P frame encoder 518 and the P frame decoder 520, as shown, for encoding and/or decoding new P frames (GT 532).

4 is an architectural diagram illustrating an example of a neural video codec according to one or more aspects of the present disclosure. The neural codec 600 may be an example of the video codec 200 and/or the video decoder 300. For example, the transmitter 602 may be an example of the source device 102 including the video codec 200, and the receiver 604 may be an example of the destination device 116 including the video decoder 300. Although the neural codec 600 is depicted as including elements of a video codec (of the transmitter 602) and elements of a video decoder (of the receiver 604), it should be noted that the video decoder 300 may be implemented in a device alone or together with the video codec 200.

The input video data _{Xt that} the video encoder 200 may encode and the reconstructed previous frame X̂ _{( t-1 )} may be in the YUV420 color space. By using the YUV420 color space, the complexity of calculations that the video encoder 200 or the video decoder 300 may be performed may be reduced compared to other video data in other color spaces (such as RGB). In addition, the YUV420 color space may be better aligned with human perception quality and may therefore improve human perception of quality in decoded reproduction of the input video data.

The video encoder 200 or the video decoder 300 may include a prediction architecture with a relatively small stream extrapolator 610. The stream extrapolator 610 may add minimal computational overhead, but may add improved compression performance. Therefore, the stream extrapolator 610 may be an efficient stream extrapolator that may provide better compression performance with minimal computational overhead.

The video encoder 200 or the video decoder 300 may include a FINT core 432, which may be implemented in the NSP 430, to use the motion vector Previous frame of reconstruction A block-based warp is performed 612, which may be an overlapping block-based warp.

The video encoder 200 or the video decoder 300 may use a parallelized entropy coding and decoding algorithm to quickly and efficiently encode or decode a bit stream, for example, on the GPU 412. For example, the video encoder 200 or the video decoder 300 may use a residual auto-encoder 620 to perform parallelized entropy coding and decoding. For example, parallel entropy coding and decoding may include utilizing multiple entropy decoding instances or operations simultaneously. The auto-encoder 620 may be implemented on the GPU 412.

The neural network components of the video encoder 200 or the video decoder 300 may be pipelined so that all subsystems (e.g., CPU(s) 406, GPU(s) 412, NSP(s) 430, FINT core 432, etc.) work simultaneously to achieve a desired frame rate.

The neural network model 422 can use int8 quantization for efficient reasoning. int8 uses 8-bit integers instead of floating point numbers, and uses integer mathematics instead of floating point mathematics, which can reduce memory and computation requirements. In some examples, the video encoder 200 or the video decoder 300 can use customized quantization operations for the entropy encoding and decoding process to improve compression performance. The training performed on the neural network model 422 can be quantization-aware.

Using additive flow prediction (eg, addition 630) rather than warping the flow can remove a computationally expensive warping operation from the video decoder 300. Using only the Y channel (luminance only) 640 as input to the flow autoencoder 650 can provide greater computational efficiency for the flow estimation.

Due to the overlapping block-based warping, the motion vectors are low-dimensional, resulting in reduced computational complexity. The stream autoencoder 650 is further optimized by inputting only the Y channel 640 of the pixel space input. The example neural codec 600 (which can be a neural video codec) can be compared to previous neural codecs. Specifically, the neural codec 600 can be made hardware-friendly by using an efficient block-based warp operator 612 and making the stream extrapolator network 610 more lightweight. In addition, the computation graph can be changed to reduce the number of warp operations compared to other neural codecs.

The neural codec 600 may include three mean-scaled hyper-prior auto-coders, e.g., a stream extrapolator 610, a residual auto-coder 620, and a stream auto-coder 650. The I-frame auto-coder (e.g., the I-frame encoder 502 and/or the I-frame decoder 508 of FIG. 3 ) compresses the first frame x ₁ in each group of pictures (GoP) and/or outputs a reconstructed frame _1. A P-frame model (eg, P-frame encoder 518 and/or P-frame decoder 520 of FIG. 3) may include a flow predictor, a flow extrapolator 610, based on the flow from the last time step The reconstructed flow of _t-1 is equivalent to the current flow Then, the P frame encoder compresses the residual stream relative to the prediction: the stream autoencoder 650 converts the reference truth frame x _𝑡 and the current stream into The P-frame decoder then outputs the 𝛿 stream, which is added to the prediction stream to obtain the reconstructed stream _t = 𝛿f _𝑡 + Finally, using the reconstructed stream =warp to distort the frame and the ground truth residual r = x _𝑡 − Compression, decoding ( ) and add it to the warped frame on the decoder side: _𝑡 = + .

For example, the video decoder 300 may perform entropy decoding in parallel on the encoded video data from the received bitstream to generate entropy decoded data. The video decoder 300 may predict (eg, using the stream extrapolator 610) block-based motion vectors based on the entropy decoded data. , to generate the predicted motion vector The video decoder 300 may decode the motion vector residual 𝛿f _𝑡 from the entropy decoded data (eg, using the stream autoencoder 650). The video decoder 300 may add (eg, using the addition 630) the motion vector residual 𝛿f _𝑡 to the predicted motion vector , to generate block-based motion vectors The video decoder 300 may use block-based motion vectors Warping the previously reconstructed video data using a block-based warping function (e.g., using warp 612) , to generate the predicted current video data The video decoder 300 can predict the current video data. and residual block summed (e.g., using summation 614) to generate the currently reconstructed video data .

FIG5 is a conceptual diagram showing an example of a block-based warp according to one or more aspects of the present disclosure. Some neural video codecs employ dense RGB scaling spatial warping, which requires high-dimensional RGB input and output and high-dimensional motion vector input. Processing such high-dimensional data is computationally expensive and power-consuming. By using an efficient block-based warp 612, for example, on a FINT core 432, computational complexity can be reduced and power can be saved. In the example of FIG5 , the warp 612 can include an efficient block-based YUV420 warp technique that can be utilized by a video encoder 200 or a video decoder 300. The dimensionality of the YUV420 input is lower than RGB (e.g., represented by fewer bits). Using block-based motion vectors can reduce stream size. The video encoder 200 or the video decoder 300 may apply a hybrid filter to reduce block artifacts. In some examples, the example of FIG. 5 may be implemented in the FINT core 432.

Motion compensation is considered as an essential component of neural video codecs. In many neural video codecs, pixel dense warping with backward mapping is used, where a frame x is warped using a flow field f. For each pixel i , j in the warped frame, the value is looked up from the reference frame as follows: warp _dense (x, f) _i,j = x[ i + _f𝑥 [ i,j ], j + _f𝑦 [ i,j ]]. Here [·] refers to the array index, and the 𝑦 sub-index indicates the retrieval of the corresponding coordinate in the vector field f. For non-integer motion vectors, bilinear or bicubic interpolation can often be used to compute the pixel intensities. Many previous neural codecs use a more space-space based version of warping, where a third blurred coordinate is given. While this space-space based warping can improve compression performance, it can also introduce additional complexity.

Since the motion vector f (which can also be called flow or optical flow) can generally have a large uniform area, block-based warping can be used as an alternative to dense warping with less computational complexity. In block-based warping, the warped frame can be divided into blocks of size 𝑏 × 𝑏. All pixels inside the block can be looked up from a reference frame using a single shared motion vector. The frame can be warped as follows: warp _block (x, f, 𝑏) _{i, j} =

Block-based warping can be more efficient than other types of warping due to block-by-block memory access. However, one drawback of block-based warping is that artifacts can occur around block edges with blocks of different motion vectors. The possibility of such artifacts can be addressed or resolved by using overlapping block-based motion compensation, where each block is warped multiple times using N-1 surrounding motion vectors (f), and the results can be averaged using a kernel ^w∈RbxbxN that decays towards the ends of the block.

For example, the overlap block warp motion compensation can be expressed as: warp _{block-overlap} (x, f, w, 𝑏) _i,j = ,in is the relative position of the neighboring block (e.g., top left, center, bottom, etc.), x is the pixel value of the frame to be warped, f is the motion vector (e.g., x and y displacement), w is the weight of the blending kernel, and b is the block size. Note that the warping is done in both the x and y directions.

As discussed below, overlapping block-based warps can lead to better compression performance than block-based warps and can match the performance of dense warps while leading to better efficiency.

FIG6 is an architectural diagram illustrating an example of pipeline reasoning in a video decoder 300. For example, the video decoder 300 can process 3 frames in parallel (frame t, frame t+1, and frame t+2). This allows a throughput of >30 FPS for full HD video. In the example of FIG6 , each stage of the pipeline 700 is depicted along with its corresponding inputs and outputs and is labeled based on an example of which processors of the computing device 402 can execute a particular stage of the pipeline 900. The width of each block of the pipeline 700 in the horizontal direction represents the approximate runtime of that particular stage.

For example, the video decoder 300 can decode the superlatent from the received bit stream. In some examples, the video decoder 300 uses the GPU 412 to decode the superlatent. The video decoder 300 can infer the stream and decode the mean and scaling from the superlatent on the NSP 430. Then, the video decoder 300 (e.g., via the GPU 412) can decode the potential based on the scaling. The video decoder 300 (e.g., via the NSP 430) can reconstruct the stream and the residual based on the potential and the extrapolated stream. The video decoder 300 can use the reconstructed stream to warp, for example, a reconstructed frame of a previous timestamp on the FINT core 432. The video decoder 300 may then add the reconstructed residuals to the distorted frames, for example on the CPU 406, to produce reconstructed P frames.

The neural codec 600 can use integer quantization for efficient reasoning. Most weights and activations can be quantized with little performance loss. It should be understood that the quantization of the potential used in entropy coding as well as the mean and scale can use specific implementations to avoid large performance degradation.

The neural codec 600 may include a neural network model (e.g., neural network model 422) that can achieve a throughput of 30+ frames per second (FPS) for full HD (1080×1920 pixels) YUV 4:2:0 video. Since different parts of the inference can be implemented on different processors (e.g., of the NSP 430) or subsystem processors, the pipeline architecture shown in FIG. 6 can efficiently utilize the available computing power. In the example pipeline 700, data from up to three time steps in the past can be processed simultaneously by using the GPU 412, NSP 430, CPU 406, and FINT 432 in parallel. In order to obtain the desired throughput, it may be necessary to use parallelization. Unlike multi-core CPU implementations that are used for a variety of tasks simultaneously, (multiple) GPUs 412 can be relatively easily used to perform entropy encoding exclusively. In the case where the NSP 430 and GPU 412 are configured to share memory, there may be no delay in copying data between the processing elements. By using only 8-bit integers for data elements, the total amount of entropy encoded decoded data to be processed can be reduced or minimized. The hyper-prior network can be trained to minimize the compression loss caused by using low-precision integer parameters for entropy encoding. This method can be used to limit the amount of memory required for the codec table to only 1,228 bytes.

The arithmetic codec function may be implemented, for example, in OpenCL, which includes small-sized tables. With OpenCL's small-sized tables, parallelism is mainly defined by the number of OpenCL work-items, and workgroup organization may not be critical.

FIG7 is a conceptual diagram of an example neural video codec functionality according to one or more aspects of the present disclosure. In the floating point example of the neural codec 600, only rounding (round operator in the figure) is used to quantize the symbol s. For the quantization example of the neural codec 600, a quantizer shown as a triangle is used. For example, the neural codec 600 can perform quantization, such as int8, at the location of the triangle depicted in FIG7.

In the example of Figure 7, 𝒚 represents the potential before transmission, 𝒔 represents the sign, Indicates the integer sign used for entropy encoding and decoding. represents the noise symbol for the rate proxy used during training, represents the reconstructed latent for decoding, σ represents the standard deviation for entropy coding, and μ represents the mean of the latent. In the example of FIG. 7 , the symbol may be an integer grid for entropy coding (e.g., bin width = 1). The latent before transmission and the mean over the latent 𝝁 may have sub-integer precision for good compression performance. For example, the latent before transmission 𝒚 and the mean over the latent 𝝁 may have a 16-bit length (e.g., using 16-bit quantization) and/or have a bin width of 0.25. It should be noted that this may result in fewer extreme values.

In some examples, special parameterizations of scaling 𝝈 may be used, as described in U.S. patent application No. 17/931,073 filed on September 9, 2022 and/or in U.S. patent application No. 17/814,426 filed on July 22, 2022, the entire contents of both of which are incorporated herein by reference.

Existing neural video codecs are not necessarily designed for YUV color space, may require expensive motion compensation techniques, employ theoretical bit rate or slow entropy coding, and utilize expensive floating point operations. Therefore, existing neural video codecs are challenging to implement on battery-constrained devices such as mobile devices (e.g., smartphones).

Because of the interaction between rounding and adding uniform noise and enabling quantization, care should be taken when quantizing potential bottlenecks. The potential bottleneck of the mean-scaled hyper-prior is shown in Figure 7. Function 800 includes functions executed during inference, while function 850 includes functions executed during training. As described above, for the floating-point version of the neural codec 600, no quantizer is added as shown by the diamond.

Since the signs passed to the video decoder 300 are always rounded in the quantized version of the neural codec 600, the sign quantizer can have a bin width of 1, since E[y−𝜇] = 0 and the quantizer is a 0-centered quantizer. The values can have a large dynamic range, and quantizing 𝜎 to a uniform grid can be detrimental to performance. This can be avoided by using an exponential activation function 𝑓(𝜌) = ...., where 𝜌∈[0,1]. This activation function enables quantization of 𝜎 without loss of performance (see row VI of Figure 8).

FIG8 is a table depicting example results of ablation of a neural encoder model architecture according to one or more aspects of the present disclosure. The neural encoders represented in the table of FIG8 are floating point codecs trained using only the first training phase (e.g., 1 million training steps), except for Model VII trained for the first training phase and the second training phase (see FIG13 below). The second training phase can be a fine-tuning phase using, for example, 250,000 steps. Parameters and kMACS/px are shown only for the P-frame model, and kMACS/px is calculated on a 1080×1920YUV420 input frame. For a detailed complexity analysis, see FIG14 below.

The question then remains how to quantize the latent y and the mean 𝜇. It might seem reasonable to quantize the latent and the mean to the same grid as with an interval width of 1 and a bias of 0. However, as shown in row VII of Figure 8, this results in an unfavorable degradation in compression performance. Instead, the neural codec 600 can be configured to have sub-integer precision for the mean so that the neural codec 600 can correct for rounding errors in the latent. With an interval width of 0.25, performance is improved almost by a factor of 2, although the range of possible latents is reduced (see row VIII of Figure 8). Note that scaling the hyper-prior alone does not suffer from these problems, but as shown in the table of Figure 8, the model has worse compression performance.

All numbers reported so far were obtained using post-training quantization (PTQ). This is known to be a suboptimal quantization technique and actually results in poor compression performance. The neural codec 600 can avoid this poor compression performance by following the PTQ stage with a quantization-aware training (QAT) stage, where gradient descent is used to optimize the model and quantizer parameters. For example, by following the PTQ stage with the QAT stage, compression performance is improved and provides a final performance of an 83% increase in BD rate relative to the floating point model.

9 is a conceptual diagram illustrating example techniques that may be implemented in an example video codec according to one or more aspects of the present disclosure. For example, the video codec 200 or the video decoder 300 may include a YUV420 compression architecture as disclosed herein. For example, the video codec 200 or the video decoder 300 may include block-based distortion as disclosed herein. For example, the video codec 200 or the video decoder 300 may include parallel entropy coding as disclosed herein. For example, the video codec 200 or the video decoder 300 may include quantization and quantization-aware training, such as the AI Model Efficiency Toolkit (AIMET) channel-by-channel quantization-aware training, to improve quantization model accuracy. AIMET is an open source library for optimizing trained neural network models.

FIG10 is a conceptual diagram illustrating an example of a neural video codec on a device according to one or more aspects of the present disclosure. In the example of FIG10 , a device such as a computing device 402 may include a video codec, which may include a video codec 200 and/or a video decoder 300. The computing device 402 may capture video data using a camera 424, for example, in a 1920×1080 (1080p) format. The video codec 200 may encode the captured video data using a neural network model 422, and may entropy encode the encoded video data into a bit stream. The video decoder 300 may entropy decode the received bit stream. The video decoder 300 may decode the entropy decoded data to reconstruct the captured video data. The computing device 402 may display the reconstructed captured video data via the display 418 at greater than or equal to 30 frames per second in 1080p.

FIG11 is a set of graphs comparing the compression performance and computation and rate-distortion performance of various techniques with the technique of the present disclosure. As can be seen from FIG11 , quantization may have a relatively large impact on performance.

The model complexity analysis 900 on the left shows compression performance vs. calculations. The BD rate can be calculated based on the curves in the rate/distortion plot 950, where lower numbers are better. The kMACS/pixel and BOPs/px can be calculated using full HD input. Unlike MAC, the binary operations (BOP) take into account the efficiency gains due to model quantization. The rate/distortion plot on the right shows the rate-distortion performance of the disclosed technique and various baselines.

The rate-distortion curve 950 shows the rate-distortion performance of the neural codec 600 and various baselines. The model complexity analysis 900 shows the BD rate and model complexity of various neural codecs. In addition to Figure 11, the BD rate results are summarized in Figure 15 below for ease of reading. In the examples of Figure 11, etc., the neural codec 600 can be represented as QODEC, fp32 indicates a 32-bit floating point value, and int8 indicates an 8-bit fixed point value. When looking at the floating-point neural codec, it can be seen that the best performing model is DCVC-DC, but it also has the largest model complexity. The floating-point version of the neural codec 600 has the lowest model complexity (24.5kmacs/pixel at the receiver end) and matches the BD rate of the SSF model, which has up to 8 times the FLOP count. Compared to the MobileCoder, which is also designed for on-device inference, the Neural Coder 600 improves compression performance by 48% while also reducing model complexity to less than 1/10. It can be noted that at bit rates above 1.Mb/s, the MobileCoder can outperform the Neural Coder 600 in compression performance. However, these bit rates are rarely used in practice.

Quantized neural codecs can benefit from efficient integer hardware cores. None of these neural codec models can outperform floating point neural baselines in terms of compression performance. However, these are the only end-to-end neural video codecs that have been demonstrated to decode in real-time on mobile devices. Neural Codec 600 outperforms MobileCoder int8, the only product to show real-time video decoding and save 40% of BD rate.

On HEVC-B, the Neural Codec 600 achieves an inference speed of >30 FPS. Although the encoding pipeline of the Neural Codec 600 is not optimized, the Neural Codec 600 still achieves an effective encoding rate. As mentioned above, Figure 6 illustrates the approximate duration of each stage of the pipeline. The inference speed is limited by the network processing, and due to parallelization, the warping operation and entropy coding and decoding do not incur any additional speed overhead. The Neural Codec 600 not only improves the compression performance compared to MobileCoder, but also improves the throughput because the Neural Codec 600 processes full HD (1080×1920) video instead of (720×1280) video.

Now let's discuss integer model quantization. Quantizing all weights and activations of a neural network can greatly improve its power efficiency. The disclosed techniques include quantizing a neural codec 600 to 8 bits using integer quantization with a learned uniform grid defined by an interval width 𝑠 and zero-entry parameters 𝑎 . For network weights, the neural codec 600 can learn a grid for each output channel without using zero entries (e.g., symmetric per-channel quantization). For activations, the neural codec 600 can learn a hardware-friendly single-interval width grid that includes zero entries (e.g., asymmetric tensor quantization). This quantization is different from the quantization of other neural codecs that use per-channel activation quantization to quantize learned image compression networks. Using per-channel activation quantization may require rescaling the accumulator for each input channel and therefore may not benefit from optimized integer arithmetic kernels on hardware (unless the scaling is a factor of 2). In some examples, the biases of the neural codec 600 are not quantized.

FIG12 is a table depicting example BD rate performance results for various quantizations of an example neural codec according to one or more aspects of the present disclosure. In the example of FIG12 , û indicates that the value is not quantized but remains in floating point 32. ü indicates that the value is quantized to 8 bits using a learned grid.

Extensive model quantization experiments were conducted and it was found that the relatively popular mean-scaled hyper-prior model is very sensitive to the quantizer settings in potential bottlenecks. For the neural codec 600, some of the grids of the starting quantizer can be manually fixed, and the grid of the residual quantizer can be learned, as shown in Figure 12. The impact of specific features of the neural codec 600 on performance is evaluated by ablating various features and evaluating the neural codecs with various features ablated. The results are shown in Figure 12.

We now discuss the experimental results. The two main axes of evaluation for the neural codec 600 are rate-distortion (RD) performance and computational efficiency. The experimental results show that the computational complexity is reduced while improving the compression performance and saving BD rate compared to previous neural on-device codecs. Relative to the BD rate of x.265 PSNR YUV 611 PSNR Y only QODEC int8 MobileCoder int8 (Le, 2022) 141.6% 340.2% 107.7% 288.4% QODEC fp32 MobileCoder fp32 (Le, 2022) 49.5% 191.0% 30.7% 141.6% DCVC-DC (Li, 2023) SSF-Pred (Pourreza, 2023) SSF-YUV (Pourreza, 2023) -58.5 -24.3 % 54.4 % -57.62% 10.2% 10.1% Table 1: BD rate savings for different baselines on the HEVC-B dataset

Row I depicts the compression performance of the unquantized model. Row III depicts the results of quantizing the activations with a known grid to 8-bit integers. These activations include pixel-space inputs, motion vectors, and rounding signs (mean-shifted latent). This quantization results in an 18.6% reduction in compression performance, primarily due to flow quantization, and provides an upper bound on the quantization performance. Row IV depicts the results of quantizing the weights using the per-channel learned quantization grid in row , which further reduces the compression performance to 27.1%. Row V depicts the results of additionally quantizing all activations except those in the latent and super-latent bottlenecks, and the compression performance drops to 135.9%.

In the model ablations, it is shown that overlapping block warping is a better choice for efficient neural codecs than other types of warping. In the quantization ablations, it is shown that problems may arise when quantizing the mean-scale hyper-prior compression model, and how to circumvent these problems.

Regarding warping, as expected, the model with dense warping (row III) has better RD performance than the overlapping block warping model (row I). However, the performance gap is relatively small (at the cost of 6% BD rate), and due to the higher flow dimensionality, the dense warping model has more than 4 times the model complexity of the overlapping block warping model. When the overlapping block warping model is compared with a vanilla block-based warping scheme (II) without block overlapping, there is a 19% drop in compression performance. Alternatively, a flow-agnostic neural network model can be used. For example, the model including its variants uses a conditional convolutional network that can implicitly warp the model (details are shown in Figure 16 below). It can be seen that the model of FIG. 16 shown in row IV is suboptimal for the overlapping block warping model in terms of both computation and compression performance.

The version of the neural codec 600 with only a scaling prior instead of a mean-scaling prior has significantly reduced compression performance (9.6% increase in BD rate) while having minimal efficiency gain (see row VI).

The effect of the second training phase is shown in row VII. Row VII shows the effect of training the neural encoder 600 (row I) for an additional 250 steps according to the auxiliary loss described in this article.

In the post-training quantization (PTQ) phase, the neural codec 600 can learn the quantizer by passing a small amount of data through the network and updating the quantizer bin width and zero terms using the mean squared error (MSE) loss on the quantizer quantized weights or activations. To enhance performance, the neural codec 600 can follow the PTQ phase with a quantization-aware training (QAT) phase, in which the neural codec 600 uses gradient descent to update both the network and quantizer parameters.

FIG13 is a table depicting different example hyperparameters for different training phases according to one or more aspects of the present disclosure. Examples of hyperparameters for different training phases can be found in FIG13. In some examples, the AIMET toolkit can be used to quantize the neural encoder/decoder 600.

Now let's discuss the loss function. The rate loss of the neural codec 600 can be the sum of the bit rates of the latent and superlatent of each of the three autoencoders of the neural codec 600 discussed above. Note that unlike other hyper-prior models, the hyper-prior model of the present disclosure can include a mean-scaled hyper-prior for an entropy model on the latent, and a normal distribution with a learned variance and zero mean, rather than a non-parametric distribution for an entropy model on the superlatent, to facilitate entropy coding and decoding.

For example, the neural codec 600 may re-weight the distortion loss of the Y:U:V channels with a weight of 6:1:1 so that the distortion loss is aligned with the evaluation metric, such that: D (x, ） =

One challenge with training small models at lower bit rates is that frame quality may deteriorate over time due to error accumulation. The neural codec 600 can avoid this problem. First, the neural codec 600 can halve the value of the rate loss multiplier for I frames s,t. This makes the PSNR values of the selected operating points for I frames and P frames more similar to previous codecs. Second, the neural codec 600 can use exponentially modulated P frame loss, where P frames farther away from the I frame have higher penalties, for example: 𝐷 _mod (x, , 𝜏） = （x _i , ）

In addition, the neural codec 600 can use an auxiliary loss during the first phase of training to force the network to learn meaningful extrapolated and reconstructed flow vectors (f ^P , ). These losses can include the YUV 611 mean square error between the original frame and the _previous reconstruction using motion field warping: Dflow (f, _{t -1,} x _t ) = D (warp (x _{t -1,} f) , x _t ).

The neural codec 600 can use rounding of potential and super-potential when evaluating. During training, the neural codec 600 can use additive noise to estimate rate loss and rounding of potential and super-potential paths fed into a decoder (e.g., video decoder 300).

The final loss of the neural codec 600 may include a weighted combination of all loss terms: L (x) = 𝛽𝑅( _x1 )+𝐷( _x1 , ₁ )+2𝛽𝑅（x _＞1 ）+𝐷 _mod （x _＞1 , _＞1 , 𝜏) + 𝜆𝐷 _flow (f ^𝑃 ) + 𝜆𝐷 _flow ( ).

The neural codec 600 may include a newly trained model for each value of 𝛽, and may perform different training phases using different values of 𝜆 and 𝜏 (see FIG12 ).

Experiments were conducted on the neural codec 600. The training of the neural codec 600 can be divided into four stages: the first two stages consist of training a floating point model end-to-end on a rate-distortion loss and differ only in its hyperparameters. The third stage may include post-training quantization (PTQ), where the quantizer is fitted while keeping the model parameters fixed. The final quantization-aware training (QAT) stage may include fine-tuning both the model parameters and the quantization parameters of the quantized model using a pass-through estimator. The hyperparameters for the various training stages can be found in FIG. 13 .

To evaluate the compression performance, the Peak Signal to Noise Ratio (PSNR) is calculated on the Y, U and V channels separately. Following common evaluation protocols, the PSNR is averaged over the Y:U:V channels with a weighting of 6:1:1. The BJtegaard-delta bitrate (BD-rate) is used to summarize the rate-distortion performance in a single metric. To calculate this metric, the bits per pixel (bpp) and the YUV 6:1:1 PSNR metric are used. After interpolating each curve, all points with a bitrate lower than 0.25bpp are selected and points outside the distortion range defined by the intersection of the support of all RD curves are discarded for a fair comparison.

FIG14 is a table depicting example model complexity for various neural encoders according to one or more aspects of the present disclosure. FLOP counts were measured using DeepSpeed. Since different packets of, for example, PTflops are known to produce different FLOP counts, FLOP counts associated with previous codecs have been recalculated using the same packets as much as possible. Details can be found in FIG14. For hardware-based results, encoder (e.g., video encoder 200) and decoder (e.g., video decoder 300) algorithms are run on a device (e.g., computing device 402). The size of the entropy-encoded decoded bitstream is measured to calculate the rate and the PSNR is calculated based on the decoded video. The rate-distortion results of the AIMET simulation are comparable to numbers reported for another neural codec.

Figure 14 highlights the model complexity of the neural codec 600 and the other codecs divided into subnetworks. Because the neural codec 600 is optimized for receiver inference speed, the receiver of the neural codec 600 has 25% of the kMACs of the transmitter of the neural codec 600, while for the other models this is 70-80%. Additionally, it can be seen that due to the low dimensional flow vectors of the neural codec 600, the complexity of all components of the neural codec 600 processing motion is very low compared to the other codecs. Finally, the motion autoencoder of the neural codec 600 is also more efficient on the receiver side due to the fact that only the Y channel is used. It can be noted that the warp operation itself is not considered in the MAC count, but as shown and described above with respect to Figure 6, the warp can be performed in parallel with the neural network and therefore does not incur additional inference time.

The neural codec 600 is compared to neural video compression techniques that report YUV performance but not performance on full video sequences. In this case, the techniques are re-evaluated when possible. The neural codec technique for which results for mobile device implementations are reported is trained for RGB. Since the model was originally trained for RGB, the model is modified by fine-tuning it on YUV 6:1:1 R-D loss. Both H.265 and H.264 implementations are tested. For a fair comparison, B frames are not used.

The neural codec 600 is trained on Vimeo90k, and the Xiph5N dataset is used for validation and early stopping. The neural codec 600 is evaluated on many standard video compression benchmarks, including the HEVC-B test sequence, the UVG-1k sequence, and finally the MCL-JVC sequence.

15 is a table depicting BD rate performance results for various example neural codecs according to one or more aspects of the present disclosure. As described above, the neural codec 600 can be represented as qodec_int8 for a quantized version of the neural codec 600, and qodec_fp32 for a floating point version of the neural codec 600.

Fig. 16 is a conceptual diagram showing a model architecture of a stream agnostic model according to the technology of the present disclosure. The architecture 1000 of Fig. 16 is mentioned above with respect to Fig. 12, and represents a stream agnostic neural codec that performs implicit warping.

FIG. 17 is a flow chart illustrating an example decoding technique according to one or more aspects of the present disclosure. The technique of FIG. 17 is described above with respect to FIGS. 2-4 , but may be practiced by any (one or more) devices capable of doing so. One or more processors 440 may perform parallel entropy decoding of encoded video data from a received bitstream to generate entropy-decoded data (1100). For example, one or more processors 440 and/or a GPU 412 of an entropy decoder 506 may perform parallel entropy decoding of encoded video data from a received bitstream to generate entropy-decoded data.

The one or more processors 440 may predict block-based motion vectors based on the entropy decoded data. To generate a predicted motion vector (1102). For example, the one or more processors 440 may predict a block-based motion vector based on the entropy decoded data. To generate the predicted motion vector . The predicted motion vector The one or more processors 440 may decode the motion vector residual from the entropy decoded data (1104). For example, the one or more processors 440 may execute the neural network model 422 to decode the motion vector residual 𝛿f _𝑡 from the entropy decoded data. The one or more processors 440 may add the motion vector residual 𝛿f _𝑡 to the predicted motion vector To generate a block-based motion vector (1106). For example, one or more processors 440 may add the motion vector residual 𝛿f _𝑡 to the predicted motion vector using an addition stream prediction (addition 630). , to generate block-based motion vectors The one or more processors 440 may use the block-based motion vectors to warp previously reconstructed video data using an overlapping block-based warping function to generate predicted current video data (1108). For example, the one or more processors 440 may use the block-based motion vectors to warp previously reconstructed video data using an overlapping block-based warping function to generate predicted current video data (1108). Applying a block-based warping function (e.g., warping 612 of FIG. 4 ) to the previously reconstructed video data .

The one or more processors 440 may add the predicted current video data to the residual block to generate the current reconstructed video data (1110). For example, the one or more processors 440 may add the predicted current video data to the residual block to generate the current reconstructed video data (1110). and residual block Added together to generate the currently reconstructed video data .

In some examples, as part of decoding the motion vector residual, the one or more processors 440 are configured to decode the pixel-based motion vector residual using the neural network model 422. In some examples, the neural network model 422 is quantization-aware trained.

In some examples, the block-based warping function (e.g., warp 612) is configured to warp a block of previously reconstructed video data multiple times using corresponding motion vectors of corresponding surrounding blocks to generate warped results and average the warped results using attenuation. In some examples, as part of performing parallel entropy decoding on the coded video data, the one or more processors 440 are configured to perform parallel entropy decoding on the coded video data using at least one graphics processing unit 412. In some examples, as part of warping the previously reconstructed video data, the one or more processors 440 are configured to perform block-based frame interpolation (FINT) warping on the previously reconstructed video data. In some examples, as part of performing block-based FINT warping on previously reconstructed video data, one or more processors are configured to perform FINT warping on previously reconstructed video data using FINT core 432. In some examples, FINT core 432 is implemented in neural network signal processor 430.

In some examples, the encoded video data represents YUV420 video data, and the currently reconstructed video includes YUV420 video data. In some examples, the one or more processors 440 are configured to quantize at least a portion of the entropy decoded data. In some examples, as part of quantizing at least a portion of the entropy decoded data, the one or more processors 440 are configured to quantize at least one of a potential, a mean, or a scale. In some examples, as part of quantizing at least a portion of the entropy decoded data, the one or more processors 440 are configured to quantize at least a portion of the entropy decoded data using int8.

In some examples, the one or more processors 440 are further configured to apply the stream extrapolator 610 to the entropy decoded data to generate an extrapolated stream. In some examples, the one or more processors 440 are further configured to perform additive stream prediction 630 using the extrapolated stream. In some examples, the encoded video data includes luminance data.

18 is a table depicting inference speed and rate overhead for different parallelization techniques according to one or more aspects of the present disclosure. In some examples, one or more processors 440 may use one of the settings illustrated in FIG. 18, such as 512 threads on GPU 412.

FIG. 19 is a conceptual diagram illustrating an example model architecture of a neural network of a P-frame model according to one or more aspects of the present disclosure. The example of FIG. 19 may represent a model architecture of a neural network model 422. Convolutional layers are shown as 𝑘 × 𝑘 𝑐, where 𝑘 refers to the kernel size and 𝑐 refers to the number of output channels. A convolution with a stride 𝑠 is represented by ↓ 𝑠, and a transposed convolution with a stride 𝑠 is shown as ↑ 𝑠.

FIG20 is a graphical diagram showing test results of model ablation and quantitative ablation according to one or more aspects of the present disclosure. Model ablation curve graph 1200 is described in more detail in the table of FIG8. Quantitative ablation curve graph 1202 is described in more detail in the table of FIG12.

FIG21 is a graph showing the rate-distortion performance of different neural video codecs on UVG and MCL.

Aspects of the technology disclosed herein include the following items.

Clause 1A. A method for decoding video data, the method comprising: entropy decoding encoded video data from a received bit stream to produce entropy decoded data; decoding the entropy decoded data using a neural network model to generate motion vectors and residues; warping the motion vectors using a warping function to generate warped motion vectors; and adding the warped motion vectors to the residues to generate reconstructed video data.

Clause 2A. A method according to Clause 1A, wherein the motion vectors include motion vectors of blocks of the video data.

Clause 3A. A method according to clause 1A or clause 2A, wherein entropy decoding the encoded video data includes entropy decoding the encoded video data in parallel.

Clause 4A. The method of clause 3A, wherein performing parallel entropy decoding on the encoded video data comprises performing parallel entropy decoding on the encoded video data using a graphics processing unit.

Clause 5A. The method of any of Clauses 2A-4A, wherein the warp motion vector comprises a block-based frame interpolation (FINT) warp motion vector.

Clause 6A. Clause 5A. The method of clause 5A, wherein performing a block-based FINT warp on the motion vector comprises performing a FINT warp on the motion vector using a FINT core.

Clause 7A. A method according to Clause 6A, wherein the FINT core is implemented in a neural network signal processor.

Clause 8A. A method as described in any of Clauses 1A-7A, wherein the encoded video data represents YUV420 video data and the reconstructed video includes YUV420 video data.

Clause 9A. A method according to any of Clauses 1A-8A, wherein decoding the entropy-decoded data using the neural network includes quantizing the entropy-decoded data.

Clause 10A. A method according to clause 9A, wherein quantizing the entropy decoded data includes quantizing at least one of a potential, a mean, or a scaling.

Clause 11A. A method according to Clause 9A or Clause 10A, wherein quantizing the entropy decoded data includes quantizing at least a portion of the entropy decoded data using int8.

Clause 12A. A method according to any of Clauses 9A-11A, wherein the neural network model is trained using quantized perception.

Clause 13A. The method of any of Clauses 1A-12A, further comprising applying a stream extrapolator to the entropy decoded data to generate an extrapolated stream.

Clause 14A. The method of Clause 13A, further comprising using the extrapolated stream to perform additive stream prediction.

Clause 15A. A method according to any of Clauses 1A-14A, wherein the video data includes brightness data.

Clause 16A. A method for decoding video data, the method comprising: decoding a superlatent; extrapolating a first stream; decoding a mean and a scaling from the superlatent; decoding the latent based on the scaling; reconstructing a second stream and a residue based on the latent and the first stream; warping a previously reconstructed frame using the second stream to generate a warped frame; and adding the residue to the warped frame.

Clause 17A. A method of encoding video data decoded using the method described in any of Clauses 1A-16A.

Clause 18A. A device for encoding and decoding video data, the device comprising: a memory for storing the video data; and one or more processors configured to execute the method described in any one of Clauses 1A-17A.

Clause 19A. A computer-readable storage medium storing instructions that, when executed, cause one or more processors to perform the method of any of Clauses 1A-17A.

Clause 20A. An apparatus for encoding or decoding video data, the apparatus comprising one or more means for performing the method of any of clauses 1A-17A.

Clause 1B. A device for encoding and decoding video data, the device comprising: a memory for storing the video data, the video data comprising previously reconstructed video data and currently reconstructed video data; and one or more processors configured to: perform parallel entropy decoding on the encoded video data from a received bit stream to generate entropy decoded data; predict block-based motion vectors based on the entropy decoded data to generate predicted The method comprises the steps of: detecting a motion vector of the video data of the video frame of the embodiment of the present invention; decoding a motion vector residual from the entropy decoded data; adding the motion vector residual to a block-based motion vector to generate the block-based motion vector; using the block-based motion vector to warp the previously reconstructed video data with a warping function based on overlapping blocks to generate predicted current video data; and summing the predicted current video data with the residual block to generate the currently reconstructed video data.

Clause 2B. A device as described in Clause 1B, wherein as part of decoding the motion vector residual, the one or more processors are configured to decode the pixel-based motion vector residual using a neural network model.

Clause 3B. An apparatus as described in Clause 2B, wherein the neural network model is trained for quantized perception.

Clause 4B. An apparatus according to any of Clauses 1B-3B, wherein the overlapping block-based warping function is configured to: warp the block of previously reconstructed video data multiple times using corresponding motion vectors of corresponding surrounding blocks to generate a warped result; and average the warped results using attenuation.

Clause 5B. An apparatus as described in any of Clauses 1B-4B, wherein as part of parallel entropy decoding of the encoded video data, the one or more processors are configured to utilize at least one graphics processing unit to parallel entropy decode the encoded video data.

Clause 6B. An apparatus as described in any of Clauses 1B-5B, wherein as part of warping previously reconstructed video data, the one or more processors are configured to perform block-based frame interpolation (FINT) warping on the previously reconstructed video data.

Clause 7B. An apparatus as described in Clause 6B, wherein as part of performing block-based FINT warping on the previously reconstructed video data, the one or more processors are configured to perform FINT warping on the previously reconstructed video data using FINT checks.

Clause 8B. A device according to Clause 7B, wherein the FINT core is implemented in a neural network signal processor.

Clause 9B. An apparatus as described in any of clauses 1B-8B, wherein the encoded video data represents YUV420 video data and the currently reconstructed video data includes YUV420 video data.

Clause 10B. An apparatus as described in any of clauses 1B-9B, wherein the one or more processors are further configured to quantize at least a portion of the entropy decoded data.

Clause 11B. An apparatus as described in Clause 10B, wherein as part of quantizing at least a portion of the entropy decoded data, the one or more processors are configured to quantize at least one of a potential, a mean, or a scale.

Clause 12B. An apparatus as described in Clause 10B or Clause 11B, wherein as part of quantizing at least a portion of the entropy decoded data, the one or more processors are configured to quantize at least a portion of the entropy decoded data using int8.

Clause 13B. An apparatus as described in any of clauses 1B-12B, wherein the one or more processors are further configured to apply a stream extrapolator to the entropy decoded data to generate an extrapolated stream.

Clause 14B. The apparatus of clause 13B, wherein the one or more processors are further configured to perform additive stream prediction using the extrapolated stream.

Clause 15B. An apparatus as described in any of Clauses 1B-14B, wherein the encoded video data includes luminance data.

Clause 16B. A method for decoding video data, the method comprising: performing parallel entropy decoding on encoded video data from a received bit stream to produce entropy decoded data; predicting block-based motion vectors based on the entropy decoded data to generate predicted motion vectors; decoding motion vector residues from the entropy decoded data; adding the motion vector residues to the predicted motion vectors to produce the block-based motion vectors; using the block-based motion vectors to warp previously reconstructed video data using a warping function based on overlapping blocks to produce predicted current video data; and summing the predicted current video data with the residue block to produce current reconstructed video data.

Clause 17B. A method according to Clause 16B, wherein decoding the motion vector residual comprises decoding pixel-based motion vector residual using a neural network model.

Clause 18B. A method according to Clause 17B, wherein the neural network model is trained for quantitative perception.

Clause 19B. A method according to any one of Clauses 16B-18B, wherein warping the previously reconstructed video data using the overlapping block-based warping function includes: multiple warping of the blocks of the previously reconstructed video data using corresponding motion vectors of corresponding surrounding blocks to generate a warped result; and averaging the warped results using attenuation.

Clause 20B. A method according to any one of clauses 16B to 19B, wherein the parallel entropy decoding of the encoded video data includes parallel entropy decoding of the encoded video data using at least one graphics processing unit.

Clause 21B. A method as described in any of Clauses 16B-20B, wherein warping the previously reconstructed video data includes performing block-based frame interpolation (FINT) warping on the previously reconstructed video data.

Clause 22B. A method according to Clause 21B, wherein performing a block-based FINT warp on the previously reconstructed video data includes performing the FINT warp on the previously reconstructed video data using a FINT check.

Clause 23B. A method according to Clause 22B, wherein the FINT core is implemented in a neural network signal processor.

Clause 24B. A method as described in any of clauses 16B-23B, wherein the encoded video data represents YUV420 video data and the reconstructed video includes YUV420 video data.

Clause 25B. A method according to any one of clauses 16B to 24B, further comprising quantizing at least a portion of the entropy decoded data.

Clause 26B. A method according to clause 25B, wherein quantizing at least a portion of the entropy decoded data includes quantizing at least one of a potential, a mean, or a scaling.

Clause 27B. A method according to Clause 25B or Clause 26B, wherein quantizing at least a portion of the entropy decoded data includes quantizing at least a portion of the entropy decoded data using int8.

Clause 28B. A method according to any of clauses 16B to 27B, further comprising applying a stream extrapolator to the entropy decoded data to generate an extrapolated stream.

Clause 29B. The method according to Clause 28B also includes using an extrapolated stream to perform additive stream prediction.

Clause 30B. A method according to any of clauses 16B to 29B, wherein the encoded video data includes luminance data.

Clause 31B. An apparatus for encoding and decoding video data, the apparatus comprising: means for performing parallel entropy decoding of encoded video data from a received bit stream to produce entropy decoded data; means for predicting block-based motion vectors based on the entropy decoded data to produce predicted motion vectors; means for decoding motion vector residuals from the entropy decoded data; means for adding the motion vector residual to the predicted motion vector to generate the block-based motion vector; means for warping previously reconstructed video data using the block-based motion vector with a warping function based on overlapping blocks to generate predicted current video data; and means for summing the predicted current video data with the residual block to generate current reconstructed video data.

Clause 32B. A non-transitory computer-readable storage medium storing instructions that, when executed, cause one or more processors to: perform parallel entropy decoding of encoded video data from a received bitstream to produce entropy-decoded data; predict block-based motion vectors based on the entropy-decoded data to generate predicted motion vectors; and The method comprises the steps of: decoding a motion vector residual of the video data; adding the motion vector residual to the predicted motion vector to generate the block-based motion vector; using the block-based motion vector to warp the previously reconstructed video data with a warping function based on overlapping blocks to generate predicted current video data; and summing the predicted current video data with the residual block to generate the current reconstructed video data.

In one or more examples, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored as one or more instructions or codes on a computer-readable medium and executed by a hardware-based processing unit. Computer-readable media may include computer-readable storage media, which corresponds to tangible media such as data storage media. In this manner, computer-readable media may generally correspond to non-transitory, tangible computer-readable storage media. Data storage media may be any available media that can be accessed by one or more computers or one or more processors to retrieve instructions, codes, and/or data structures for implementation in the technology described in this disclosure. A computer program product may include computer-readable media.

By way of example and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage devices, magnetic disk storage devices or other magnetic storage devices, flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures and can be accessed by a computer. It should be understood that computer-readable storage media and data storage media do not include carrier waves, signals, or other temporary media, but are directed to non-temporary tangible storage media. As used herein, magnetic disks and optical disks include compressed discs (CDs), laser discs, optical discs, digital versatile discs (DVDs), floppy disks, and Blu-ray discs, where magnetic disks typically reproduce data magnetically, while optical discs use lasers to reproduce data optically. The above combinations should also be included within the scope of computer-readable media.

The instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Therefore, the term "processor" as used herein may refer to any of the aforementioned structures or any other structure suitable for implementation in the techniques described herein. In addition, in some aspects, the functionality described herein may be provided in dedicated hardware and/or software modules configured for encoding and decoding, or incorporated in a combined encoder. Furthermore, these techniques may be fully implemented in one or more circuits or logic elements.

The techniques of this disclosure may be implemented in a wide variety of devices or apparatuses, including wireless handsets, integrated circuits (ICs), or collections of ICs (e.g., chipsets). Various components, modules, or units are described in this disclosure to emphasize functional aspects of a device configured to perform the disclosed techniques, but do not necessarily need to be implemented by different hardware units. Specifically, as described above, the various units may be combined in an encoder hardware unit, or provided by a collection of interoperable hardware units (including one or more processors as described above) in conjunction with appropriate software and/or firmware.

This disclosure also includes the attached appendix, which forms a part of this disclosure and is explicitly incorporated herein. The technology disclosed in the appendix can be combined or performed separately with the technology disclosed herein.

Various examples have been described. These and other examples are within the scope of the appended claims.

100: System 102: Source device 104: Video source 106: Memory 108: Output interface 110: Computer-readable media 112: Storage device 114: File server 116: Destination device 118: Display device 120: Memory 122: Input interface 200: Video encoder 300: Video decoder 402: Computing device 404: User input interface 406: CPU 408: Memory controller 410: System memory 412: Graphics processing unit (GPU) 414: Local memory 416: Display interface 418: Display 420: Bus 422: Neural Network Model 424: Camera 430: Neural Network Signal Processor (NSP) 432: Frame Interpolation (FINT) Core 500: Ground Truth 502: I-Frame Encoder (IFE) 504: Entropy Encoder (EE) 506: Entropy Decoder (ED) 508: I-Frame Decoder (IFD) 510: Reconstructed Frame (recon) 512: Warp Function (warp)/Warp 514: GT 516: Motion Estimation (ME) 518: P-Frame Encoder (PFE) 520: P-Frame Decoder (PFD) 522: Motion Vector (MV) 524: Block Resid 528: Recon 530: Output 532: GT 600: Neural Encoder/Decoder 602: Transmitter 604: Receiver 610: Stream Extrapolator/Stream Extrapolator Network 612: Warp/Warp Operator 614: Sum 620: Resid Autoencoder/Autoencoder 630: Addition 640: Y Channel 650: Stream Autoencoder 700: Pipeline 800: Function 900: Pipeline/Model Complexity Analysis 950: Rate/Distortion Plot 1000: Architecture 1100, 1102, 1104, 1106, 1108, 1110: Step 1200: Model ablation curve 1202: Quantitative ablation curve

FIG1 is a block diagram illustrating an example media encoding and decoding system in which the techniques of the present disclosure may be implemented.

FIG. 2 is a block diagram illustrating an example computing device that may implement the techniques of this disclosure.

FIG3 is a conceptual diagram illustrating an example of end-to-end deep learning of a neural video encoder according to one or more aspects of the present disclosure.

Figure 4 is an architectural diagram showing an example of a neural video encoder according to one or more aspects of the present disclosure.

FIG5 illustrates a conceptual diagram of an example of block-based warping according to one or more aspects of the present disclosure.

FIG. 6 is an architectural diagram illustrating an example of pipeline reasoning in the video decoder 300. As shown in FIG.

Figure 7 is a conceptual diagram of an example neural video encoder according to one or more aspects of the present disclosure.

Figure 8 is a table depicting example results of ablation of a neural encoder model architecture according to one or more aspects of the present disclosure.

FIG. 9 is a conceptual diagram illustrating example techniques that may be implemented in an example video encoder in accordance with one or more aspects of the present disclosure.

Figure 10 is a conceptual diagram illustrating an example of a neural video encoder on a device according to one or more aspects of the present disclosure.

FIG11 is a set of graphs comparing the compression performance and computational and rate-distortion performance of various technologies with the technology disclosed herein.

Figure 12 is a table depicting various quantized example BD rate performance results for an example neural encoder according to one or more aspects of the present disclosure.

Figure 13 is a table depicting different example hyperparameters for different training stages according to one or more aspects of the present disclosure.

Figure 14 is a table depicting example model complexities for various neural encoders according to one or more aspects of the present disclosure.

Figure 15 is a table depicting BD rate performance results for various example neural encoders according to one or more aspects of the present disclosure.

Figure 16 is a conceptual diagram showing the model architecture of the flow-agnostic model according to the technology of the present disclosure.

FIG. 17 is a flow chart illustrating an example decoding technique according to one or more aspects of the present disclosure.

FIG. 18 is a table depicting inference speed and rate overhead for different parallelization techniques according to one or more aspects of the present disclosure.

Figure 19 is a conceptual diagram illustrating an example model architecture of a neural network of a P-frame model according to one or more aspects of the present disclosure.

Figure 20 is a graph showing test results of model ablation and quantitative ablation according to one or more aspects of the present disclosure.

Figure 21 is a graph showing the rate-distortion performance of different neural video encoders on UVG and MCL.

600:Neural codec

602: Transmitter

604: Receiver

610: Stream Extrapolator/Stream Extrapolator Network

612: Distortion/Twist Operator

614:Summary

620: Residual error automatic encoder/automatic encoder

630: Addition

640:Y channel

650: Streaming automatic encoder

Claims (32)

A device for encoding video data, the device comprising: A memory for storing the video data, the video data comprising previously reconstructed video data and currently reconstructed video data; and One or more processors configured to: Perform parallel entropy decoding on the encoded video data from a received bit stream to generate entropy decoded data; Predict block-based motion vectors based on the entropy decoded data to generate predicted motion vectors; Decode motion vector residuals from the entropy decoded data; Add the motion vector residuals to the predicted motion vectors to generate block-based motion vectors; Using the block-based motion vectors to warp the previously reconstructed video data with an overlapping block-based warping function to generate predicted current video data; and summing the predicted current video data with a residual block to generate the currently reconstructed video data. A device as described in claim 1, wherein, as part of decoding the motion vector residual, the one or more processors are configured to decode the pixel-based motion vector residual using a neural network model. An apparatus according to claim 2, wherein the neural network model is trained using quantized perception. The device of claim 1, wherein the overlapping block-based warping function is configured to: warp the block of the previously reconstructed video data multiple times using corresponding motion vectors of corresponding surrounding blocks to generate a warped result; and average the warped results using attenuation. The apparatus of claim 1, wherein, as part of performing parallel entropy decoding on the coded video data, the one or more processors are configured to perform parallel entropy decoding on the coded video data using at least one graphics processing unit. The apparatus of claim 1, wherein, as part of warping the previously reconstructed video data, the one or more processors are configured to perform a block-based frame interpolation (FINT) warp on the previously reconstructed video data. The apparatus of claim 6, wherein, as part of performing block-based FINT warping on the previously reconstructed video data, the one or more processors are configured to perform FINT warping on the previously reconstructed video data using FINT checks. An apparatus as described in claim 7, wherein the FINT core is implemented in a neural network signal processor. The apparatus of claim 1, wherein the encoded video data represents YUV420 video data and the currently reconstructed video data comprises YUV420 video data. A device as described in claim 1, wherein the one or more processors are also configured to quantize at least a portion of the entropy decoded data. The apparatus of claim 10, wherein, as part of quantizing at least a portion of the entropy decoded data, the one or more processors are configured to quantize at least one of a potential, a mean, or a scale. The apparatus of claim 10, wherein, as part of quantizing at least a portion of the entropy decoded data, the one or more processors are configured to quantize at least a portion of the entropy decoded data using int8. The apparatus of claim 1, wherein the one or more processors are further configured to apply a stream extrapolator to the entropy decoded data to generate an extrapolated stream. The apparatus of claim 13, wherein the one or more processors are further configured to perform additive stream prediction using the extrapolated stream. The apparatus of claim 1, wherein the encoded video data includes luminance data. A method for decoding video data, the method comprising: performing parallel entropy decoding on encoded video data from a received bitstream to produce entropy decoded data; predicting block-based motion vectors based on the entropy decoded data to generate predicted motion vectors; decoding motion vector residuals from the entropy decoded data; adding the motion vector residuals to the predicted motion vectors to produce the block-based motion vectors; using the block-based motion vectors to warp previously reconstructed video data with a warping function based on overlapping blocks to produce predicted current video data; and summing the predicted current video data with the residual block to produce current reconstructed video data. A method according to claim 16, wherein decoding the motion vector residual includes decoding the pixel-based motion vector residual using a neural network model. A method according to claim 17, wherein the neural network model is trained using quantized perception. The method of claim 16, wherein warping the previously reconstructed video data using the overlapping block-based warping function comprises: warping the block of the previously reconstructed video data multiple times using corresponding motion vectors of corresponding surrounding blocks to generate a warped result; and averaging the warped results using attenuation. The method of claim 16, wherein the parallel entropy decoding of the encoded video data comprises parallel entropy decoding of the encoded video data using at least one graphics processing unit. The method of claim 16, wherein warping the previously reconstructed video data comprises performing a block-based frame interpolation (FINT) warp on the previously reconstructed video data. The method of claim 21, wherein performing block-based FINT warping on the previously reconstructed video data comprises performing FINT warping on the previously reconstructed video data using a FINT check. A method according to claim 22, wherein the FINT core is implemented in a neural network signal processor. The method of claim 16, wherein the encoded video data represents YUV420 video data and the currently reconstructed video includes YUV420 video data. The method according to claim 16 also includes quantizing at least a portion of the entropy decoded data. A method as claimed in claim 25, wherein quantizing at least a portion of the entropy decoded data comprises quantizing at least one of a potential, a mean, or a scaling. A method according to claim 25, wherein quantizing at least a portion of the entropy decoded data includes quantizing at least a portion of the entropy decoded data using int8. The method according to claim 16 also includes: applying a stream extrapolator to the entropy decoded data to generate an extrapolated stream. The method of claim 28 further comprises using the extrapolated stream to perform additive stream prediction. A method according to claim 16, wherein the encoded video data includes brightness data. A device for encoding and decoding video data, the device comprising: Means for performing parallel entropy decoding of encoded video data from a received bit stream to generate entropy decoded data; Means for predicting block-based motion vectors based on the entropy decoded data to generate predicted motion vectors; Means for decoding motion vector residues from the entropy decoded data; Means for adding the motion vector residues to the predicted motion vectors to generate block-based motion vectors; Means for warping previously reconstructed video data using the block-based motion vectors with a warping function based on overlapping blocks to generate predicted current video data; and Means for summing predicted current video data with a residual block to produce current reconstructed video data. A non-transitory computer-readable storage medium storing instructions that, when executed, cause one or more processors to: perform parallel entropy decoding of encoded video data from a received bitstream to produce entropy-decoded data; predict block-based motion vectors based on the entropy-decoded data to generate predicted motion vectors; decode motion vector residuals from the entropy-decoded data; add the motion vector residuals to the predicted motion vectors to produce block-based motion vectors; warp previously reconstructed video data using the block-based motion vectors with a warping function based on overlapping blocks to produce predicted current video data; and The predicted current video data and the residual block are summed to produce the current reconstructed video data.