HK1226567B - Methods and systems for reducing latency in video encoding and decoding - Google Patents

Description

Method and system for reducing delay in video encoding and decoding

Background Art

Engineers use compression (also known as source coding) to reduce the bit rate of digital video. Compression reduces the cost of storing and transmitting video information by converting it into a lower bit rate form. Decompression (also known as decoding) reconstructs a version of the original information from the compressed form. A "codec" is an encoder/decoder system.

Over the past two decades, various video codec standards have been adopted, including H.261, H.262 (MPEG-2 or ISO/IEC 13818-2), H.263, and H.264 (AVC or ISO/IEC 14496-10), as well as MPEG-1 (ISO/IEC 11172-2), MPEG-4 Visual (ISO/IEC 14496-2), and SMPTE 421M. More recently, the HEVC standard is under development. Video codec standards typically define syntax options for coded video bitstreams, detailing parameters within the bitstream when specific features are used during encoding and decoding. In many cases, video codec standards also provide details on the decoding operations that a decoder should perform to achieve correct decoding results.

The fundamental goal of compression is to provide good rate-distortion performance. Thus, for a given bitrate, an encoder attempts to provide the highest quality video. Alternatively, for a given quality level/fidelity level to the original video, an encoder attempts to provide the lowest bitrate encoded video. In practice, depending on the context of use, considerations such as encoding time, encoding complexity, encoding resources, decoding time, decoding complexity, decoding resources, total latency, and/or smoothness during playback also influence the decisions made during encoding and decoding.

For example, consider use cases such as playing back video from storage, playing back video from encoded data streamed over a network connection, and transcoding video (from one bitrate to another, or from one standard to another). On the encoder side, such applications can allow for completely time-insensitive offline encoding. Consequently, the encoder can increase encoding time and resources used during encoding to find the most efficient way to compress the video and thereby improve rate-distortion performance. If a small amount of latency is also acceptable on the decoder side, the encoder can further improve rate-distortion performance, for example, by exploiting inter-picture correlations from earlier pictures in the sequence.

On the other hand, consider use cases such as remote desktop conferencing, surveillance video, video telephony, and other real-time communication scenarios. Such applications are time-sensitive. Low latency between recording the input image and replaying the output image is a key performance factor. When encoding/decoding tools adapted for non-real-time communication are applied to real-time communication scenarios, the overall latency is often unacceptably high. The delays introduced by these tools during encoding and decoding may improve the performance of regular video playback, but they undermine real-time communication.

Summary of the Invention

In summary, the detailed description presents techniques and tools for reducing latency in video encoding and decoding. The techniques and tools can reduce latency to improve responsiveness in real-time communications. For example, the techniques and tools reduce overall latency by constraining the latency due to video frame reordering and by indicating the constraints on the frame reordering delay using one or more syntax elements accompanying the encoded data for the video frame.

According to one aspect of the techniques and tools described herein, a tool, such as a video encoder, a real-time communication tool with a video encoder, or other tool, sets one or more syntax elements indicating a constraint on a delay, such as a constraint on a frame reordering delay consistent with inter-frame dependencies between multiple frames of a video sequence. The tool outputs the syntax element(s) to facilitate a simpler and faster determination of when a reconstructed frame is ready for output in its output order.

According to another aspect of the techniques and tools described herein, a tool, such as a video decoder, a real-time communication tool having a video decoder, or other tool, receives and parses one or more syntax elements indicating a constraint on delay (e.g., a constraint on a frame reordering delay). The tool also receives encoded data for a plurality of frames of a video sequence. At least some of the encoded data is decoded to reconstruct one of the plurality of frames. The tool may determine the constraint on the delay based on the syntax element(s), and then use the constraint on the delay to determine when the reconstructed frame is ready for output (in output order). The tool outputs the reconstructed frame.

The foregoing and other objects, features and advantages of the present invention will become more apparent from the following detailed description made with reference to the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a diagram of an example computing system in which some described embodiments may be implemented.

2a and 2b are diagrams of example network environments in which some described embodiments may be implemented.

3 is a diagram of an example encoder system in conjunction with which some described embodiments may be implemented.

4 is a diagram of an example decoder system in conjunction with which some described embodiments may be implemented.

5a-5e are diagrams showing the encoding order and output order of frames for several example series.

6 is a flow diagram illustrating an example technique for setting and outputting one or more syntax elements indicating constraints on delays.

7 is a flow chart illustrating an example technique for decoding with reduced latency.

DETAILED DESCRIPTION

The detailed description of the embodiments presents techniques and tools for reducing latency in video encoding and decoding. These techniques and tools can help reduce latency to improve responsiveness in real-time communications.

In video encoding/decoding scenarios, some delay between the time an input video frame is received and the time it is played back is unavoidable. The frame is encoded by the encoder, fed to the decoder, and decoded by the decoder, and some amount of delay is caused by practical limitations on encoding resources, decoding resources, and/or network bandwidth. However, other delays can be avoided. For example, to improve rate-distortion performance (e.g., to exploit inter-frame correlations from earlier images in the sequence), encoders and decoders may introduce delays. Such delays can be reduced, although there may be a loss in rate-distortion performance, processor usage, or playback smoothness.

Using the techniques and tools described herein, latency is reduced by constraining the delay (thereby limiting the time range of inter-frame correlations) and indicating this constraint on the delay to the decoder. For example, the constraint on the delay is a constraint on the frame reordering delay. Alternatively, the constraint on the delay is a constraint in seconds, milliseconds, or another time metric. The decoder can then determine the constraint on the delay and use it when determining which frames are ready for output. This can reduce latency for remote desktop conferencing, video telephony, video surveillance, webcam video, and other real-time communication applications.

Some of the innovations described herein are illustrated with reference to syntax elements and operations specific to the H.264 and/or HEVC standards. Such innovations may also be implemented for other standards or formats.

More generally, various alternatives to the examples described herein are possible. Certain techniques described with reference to the flowcharts can be modified by changing the order of the stages shown in the flowcharts, by splitting, repeating, or omitting certain stages, and so on. Various aspects of delay reduction for video encoding and decoding can be used in combination or individually. Different embodiments utilize one or more of the described techniques and tools. Some of the techniques and tools described herein address one or more of the problems identified in the background. Typically, a given technique/tool does not address all such problems.

I. Example Computing System

Figure 1 illustrates a generalized example of a suitable computing system (100) in which several of the described techniques and tools may be implemented. The computing system (100) is not intended to suggest any limitation as to scope of use or functionality, as the techniques and tools may be implemented in a wide variety of general-purpose or special-purpose computing systems.

Referring to FIG1 , the computing system (100) includes one or more processing units (110, 115) and memory (120, 125). In FIG1 , this most basic configuration (130) is enclosed within the dashed lines. The processing units (110, 115) run computer-executable instructions. The processing units can be general-purpose central processing units (CPUs), processors in application-specific integrated circuits (ASICs), or other types of processors. In a multi-processing system, multiple processing units run computer-executable instructions to increase processing power. For example, FIG1 shows a central processing unit (110) and a graphics processing unit or co-processing unit (115). The tangible memory (120, 125) can be volatile memory (e.g., registers, cache, RAM), non-volatile memory (e.g., ROM, EEPROM, flash memory, etc.), or some combination of the two, accessible to the processing unit(s). The memory (120, 125) stores software (180) implementing one or more innovations for reducing latency in video encoding and decoding in the form of computer-executable instructions suitable for execution by the processing unit(s).

The computing system may have additional features. For example, the computing system (100) includes storage (140), one or more input devices (150), one or more output devices (160), and one or more communication connections (170). An interconnection mechanism (not shown) such as a bus, controller, or network interconnects the components of the computing system (100). Typically, operating system software (not shown) provides an operating environment for other software running in the computing system (100) and coordinates the activities of the components of the computing system (100).

The tangible storage device (140) may be removable or non-removable and includes a disk, tape or cartridge, CD-ROM, DVD, or any other medium that can be used to store information in a non-transitory manner and that can be accessed within the computing system (100). The storage device (140) stores instructions for the software (180) that implement one or more innovations for delay reduction in video encoding and decoding.

The input device(s) (150) may be a touch input device such as a keyboard, mouse, stylus, or trackball, a voice input device, a scanning device, or another device that provides input to the computing system (100). For video encoding, the input device(s) (150) may be a camera, a video card, a TV tuner card, or a similar device that accepts input in analog or digital form, or a CD-ROM or CD-RW that reads video samples into the computing system (100). The output device(s) (160) may be a display, a printer, speakers, a CD burner, or another device that provides output from the computing system (100).

The communication connection(s) (170) enable communication to another computing entity via a communication medium. The communication medium carries information such as computer-executable instructions, audio or video input or output, or other data in a modulated data signal. A modulated data signal is a signal that has one or more of its characteristics set or changed in such a manner as to facilitate encoding information in the signal. By way of example and not limitation, the communication medium may employ an electrical, optical, RF, or other carrier.

The techniques and tools may be described in the general context of computer-readable media. Computer-readable media is any available tangible media that can be accessed within a computing environment. By way of example and not limitation, with respect to computing system (100), computer-readable media includes memory (120, 125), storage (140), and any combination thereof.

The techniques and tools may be described in the general context of computer-executable instructions, such as those contained in program modules and executed on a real or virtual target processor in a computing system. Generally, program modules include routines, programs, libraries, objects, classes, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The functionality of the program modules may be combined or split between program modules as desired in various embodiments. The computer-executable instructions for the program modules may be executed within a local or distributed computing system.

The terms "system" and "device" are used interchangeably herein. Unless the context clearly indicates otherwise, neither term implies any limitation on the type of computing system or computing device. Generally, a computing system or computing device can be local or distributed and can include any combination of specialized hardware and/or general-purpose hardware with software that implements the functionality described herein.

For presentation purposes, this description uses terms like "determine" and "use" to describe computer operations in a computing system. These terms are high-level abstractions of the operations performed by a computer and should not be confused with actions performed by a person. The actual computer operations corresponding to these terms vary depending on the implementation.

II. Sample Network Environment

Figures 2a and 2b illustrate an example network environment (201, 202) including a video encoder (220) and a video decoder (270). The encoder (220) and decoder (270) are connected over a network (250) using an appropriate communication protocol. The network (250) may include the Internet or another computer network.

In the network environment (201) shown in Figure 2a, each real-time communication ("RTC") tool (210) includes both an encoder (220) and a decoder (270) for two-way communication. A given encoder (220) can produce output that complies with the SMPTE 421M standard, the ISO-IEC 14496-10 standard (also known as H.264 or AVC), the HEVC standard, other standards, or a proprietary format, and the corresponding decoder (270) accepts encoded data from the encoder (220). The two-way communication can be part of a video conference, a video phone call, or other two-party communication scenario. Although the network environment (201) in Figure 2a includes two real-time communication tools (210), as an alternative, the network environment (201) can include three or more real-time communication tools (210) participating in multi-party communication.

The real-time communication tool (210) manages encoding by the encoder (220). FIG3 shows an example encoder system (300) that can be included in the real-time communication tool (210). Alternatively, the real-time communication tool (210) uses another encoder system. The real-time communication tool (210) also manages decoding by the decoder (270). FIG4 shows an example decoder system (400) that can be included in the real-time communication tool (210). Alternatively, the real-time communication tool (210) uses another encoder system.

In the network environment (202) shown in Figure 2b, an encoding tool (212) includes an encoder (220) that encodes video for delivery to multiple playback tools (214), wherein the multiple playback tools include decoders (270). One-way communication can be provided for video surveillance systems, camera surveillance systems, remote desktop conference presentations, or other scenarios in which video is encoded and sent from one location to one or more other locations. Although the network environment (202) in Figure 2b includes two playback tools (214), the network environment (202) can include more or fewer playback tools (214). Generally, the playback tool (214) communicates with the encoding tool (212) to determine the video stream to be received by the playback tool (214). The playback tool (214) receives the stream, buffers the received encoded data for an appropriate period of time, and then begins decoding and playback.

FIG3 illustrates an example encoder system (300) that may be included in the encoding tool (212). Alternatively, the encoding tool (212) uses another encoder system. The encoding tool (212) may also include server-side controller logic for managing connections with one or more playback tools (214). FIG4 illustrates an example decoder system (400) that may be included in the playback tool (214). Alternatively, the playback tool (214) uses another decoder system. The playback tool (214) may also include client-side controller logic for managing connections with the encoding tool (212).

In some cases, the use of syntax elements that indicate delay (e.g., frame reordering delay) is specific to a particular standard or format. For example, the coded data may contain one or more syntax elements indicating constraints on delay as part of a syntax of an elementary coded video bitstream defined in accordance with the standard or format, or as media metadata defined about the coded data. In these cases, the real-time communication tool (210), encoding tool (212), and/or playback tool (214) with reduced delay may be codec-dependent in that the decisions they make may depend on the bitstream syntax used for the particular standard or format.

In other cases, the use of syntax elements indicating constraints on delay (e.g., frame reordering delay) is outside of a specific standard or format. For example, the syntax element(s) indicating the constraints on delay can be signaled as part of the syntax of a media transport stream, a media storage file, or more generally, a media system multiplexing protocol or transport protocol. Alternatively, the syntax element(s) indicating the delay can be negotiated between the real-time communication tool (210), the encoding tool (212), and/or the playback tool (214) according to a media attribute negotiation protocol. In these cases, the real-time communication tool (210), the encoding tool (212), and the playback tool (214) with reduced delay can be codec-independent in that they can work with any available video encoder and decoder, assuming that the level of control over inter-frame correlation is fixed during encoding.

III. Example Encoder System

FIG3 is a block diagram of an example encoder system (300) in conjunction with which some of the described embodiments may be implemented. The encoder system (300) may be a general-purpose encoding tool capable of operating in any of a plurality of encoding modes, such as a low-latency encoding mode for real-time communication, a transcoding mode, and a conventional encoding mode for playback of media from a file or stream, or it may be a dedicated encoding tool adapted for use in one such encoding mode. The encoder system (300) may be implemented as an operating system module, as part of an application library, or as a standalone application. In general, the encoder system (300) receives a sequence of source video frames (311) from a video source (310) and produces encoded data that is output to a channel (390). The encoded data output to the channel may contain one or more syntax elements indicating constraints on delay (e.g., frame reordering delay) to facilitate decoding with reduced delay.

The video source (310) can be a camera, a tuner card, a storage medium, or other digital video source. The video source (310) produces a sequence of video frames at a frame rate of, for example, 30 frames per second. As used herein, the term "frame" generally refers to source, encoded, or reconstructed image data. For progressive scan video, a frame is a progressive scan video frame. For interlaced scan video, in an example embodiment, the interlaced scan video frame is deinterlaced prior to encoding. Alternatively, two complementary interlaced scan video fields are encoded as one interlaced scan video frame or separate fields. In addition to indicating a progressive scan video frame, the term "frame" can indicate a single unpaired video field, a complementary pair of video fields, a video object plane representing a video object at a given time, or a region of interest in a larger image. The video object plane or region can be a portion of a larger image containing multiple objects or regions of a scene.

Arriving source frames (311) are stored in a source frame temporary memory storage area (320) comprising a plurality of frame buffer storage areas (321, 322, ... 32n). A frame buffer (321, 322, etc.) stores a source frame in the source frame storage area (320). After one or more source frames (311) have been stored in the frame buffers (321, 322, etc.), a frame selector (330) periodically selects a single source frame from the source frame storage area (320). The order in which the frame selector (330) selects frames to input to the encoder (340) may be different from the order in which the video source (310) generates frames, for example, the order of a frame may be in front to facilitate temporal backward prediction. Prior to the encoder (340), the encoder system (300) may include a preprocessor (not shown) that performs preprocessing (e.g., filtering) on the selected frame (331) before encoding.

The encoder (340) encodes the selected frame (331) to produce an encoded frame (341) and also generates a memory management control signal (342). If the current frame is not the first frame to be encoded, when performing its encoding process, the encoder (340) may use one or more previously encoded/decoded frames (369) that have been stored in a decoded frame temporary memory storage area (360). Such stored decoded frames (369) are used as reference frames for inter-frame prediction of the content of the current source frame (331). Generally, the encoder (340) includes multiple encoding modules that perform encoding tasks such as motion estimation and compensation, frequency conversion, quantization, and entropy encoding. The exact operations performed by the encoder (340) may vary depending on the compression format. The format of the output encoded data may be Windows Media Video format, VC-1 format, MPEG-x format (e.g., MPEG-1, MPEG-2, or MPEG-4), H.26x format (e.g., H.261, H.262, H.263, H.264), HEVC format, or other formats.

The coded frame (341) and the memory management control signal (342) are processed by a decoding process simulator (350). The decoding process simulator (350) implements some functionality of the decoder, such as the decoding task of reconstructing reference frames used by the encoder (340) in motion estimation and compensation. The decoding process simulator (350) uses the memory management control signal (342) to determine whether a given coded frame (341) needs to be reconstructed and stored for use as a reference frame in inter-frame prediction of subsequent frames to be encoded. If the control signal (342) indicates that the coded frame (341) needs to be stored, the decoding process simulator (350) models a decoding process that can be implemented by a decoder that receives the coded frame (341) and produces a corresponding decoded frame (351). In doing so, when the encoder (340) has used decoded frame(s) (369) already stored in the decoded frame storage area (360), the decoding process simulator (350) also uses the decoded frame(s) (369) from the storage area (360) as part of the decoding process.

The decoded frame temporary memory storage area (360) includes a plurality of frame buffer storage areas (361, 362, ..., 36n). The decoding process simulator (350) uses the memory management control signal (342) to manage the contents of the storage area (360) to identify any frame buffers (361, 362, etc.) that contain frames that are no longer needed as reference frames by the encoder (340). After modeling the decoding process, the decoding process simulator (350) stores the newly decoded frames (351) in the frame buffers (361, 362, etc.) that have been identified in this manner.

The coded frames (341) and memory management control signals (342) are also buffered in a temporary coded data area (370). The coded data accumulated in the coded data area (370) may contain one or more syntax elements indicating a constraint on delay as part of the syntax of the elementary coded video bitstream. Alternatively, the coded data accumulated in the coded data area (370) may contain syntax element(s) indicating a constraint on delay as part of media metadata associated with the coded video data (e.g., as one or more parameters in one or more Supplemental Enhancement Information ("SEI") messages or Video Usability Information ("VUI") messages).

Aggregated data (371) from the temporary coded data area (370) is processed by a channel encoder (380). The channel encoder (380) may package the aggregated data for transmission as a media stream, in which case the channel encoder (380) may add (multiple) syntax elements indicating the constraints on delay as part of the syntax of the media transmission stream. Or the channel encoder (380) may organize the aggregated data for storage as a file, in which case the channel encoder (380) may add (multiple) syntax elements indicating the constraints on delay as part of the syntax of the media storage file. Or, more generally, the channel encoder (380) may implement one or more media system multiplexing protocols or transport protocols, in which case the channel encoder (380) may add (multiple) syntax elements indicating the constraints on delay as part of the syntax of the protocol(s). The channel encoder (380) provides an output to a channel (390), which represents storage, a communication connection, or another channel for the output.

IV. Example Decoder System

FIG4 is a block diagram of an example decoder system (400) in conjunction with which some of the described embodiments may be implemented. The decoder system (400) may be a general purpose decoding tool capable of operating in any of a plurality of decoding modes, such as a low-latency decoding mode for real-time communications and a conventional decoding mode for playback of media from a file or stream, or it may be a dedicated decoding tool adapted for use in one such decoding mode. The decoder system (400) may be implemented as an operating system module, as part of an application library, or as a standalone application. In general, the decoder system (400) receives encoded data from a channel (410) and produces reconstructed frames as output for an output destination (490). The encoded data may contain one or more syntax elements indicating constraints on delay (e.g., frame reordering delay) to facilitate reduced-latency decoding.

The decoder system (400) includes a channel (410), which can represent storage, a communication connection, or another channel for coded data as input. The channel (410) produces coded data that has been channel-coded. A channel decoder (420) can process the coded data. For example, the channel decoder (420) depacketizes data that has been aggregated for transmission as a media stream, in which case the channel decoder (420) can parse (multiple) syntax elements indicating delay constraints as part of the syntax of the media transmission stream. Alternatively, the channel decoder (420) depacketizes coded video data that has been aggregated for storage as a file, in which case the channel decoder (420) can parse (multiple) syntax elements indicating delay constraints as part of the syntax of the media storage file. Alternatively, more generally, the channel decoder (420) can implement one or more media system demultiplexing protocols or transport protocols, in which case the channel decoder (420) can parse (multiple) syntax elements indicating delay constraints as part of the syntax of the protocol(s).

The coded data (421) output from the channel decoder (420) is stored in a temporary coded data area (430) until a sufficient amount of such data has been received. The coded data (421) includes coded frames (431) and memory management control signals (432). The coded data (421) in the coded data area (430) may contain one or more syntax elements indicating constraints on delay as part of the syntax of the elementary coded video bitstream. Alternatively, the coded data (421) in the coded data area (430) may contain (a plurality of) syntax elements indicating constraints on delay as part of media metadata related to the coded video data (e.g., as one or more parameters in one or more SEI messages or VUI messages). Generally, the coded data area (430) temporarily stores coded data (421) until the decoder (450) uses such coded data (421). At that time, the coded data for the coded frame (431) and the memory management control signal (432) are passed from the coded data area (430) to the decoder (450). As decoding continues, new coded data is added to the coded data area (430) and the oldest coded data remaining in the coded data area (430) is passed to the decoder (450).

The decoder (450) periodically decodes coded frames (431) to produce corresponding decoded frames (451). If appropriate, when performing its decoding process, the decoder (450) may use one or more previously decoded frames (469) as reference frames for inter-frame prediction. The decoder (450) reads such previously decoded frames (469) from a decoded frame temporary memory storage area (460). Generally, the decoder (450) includes a plurality of decoding modules that perform decoding tasks such as entropy decoding, inverse quantization, inverse frequency transform, and motion compensation. The exact operations performed by the decoder (450) may vary depending on the compression format.

The decoded frame temporary memory storage area (460) includes a plurality of frame buffer storage areas (461, 462, ..., 46n). The decoded frame storage area (460) is an example of a decoded image buffer. The decoder (450) uses the memory management control signal (432) to identify a frame buffer (461, 462, etc.) in which it can store the decoded frame (451). The decoder (450) stores the decoded frame (451) in the frame buffer.

An output sequencer (480) uses the memory management control signal (432) to identify when the next frame to be generated in output order is available in the decoded frame storage area (460). To reduce the latency of the encoding-decoding system, the output sequencer (480) uses syntax elements that indicate constraints on latency to speed up the identification of frames to be generated in output order. When the next frame to be generated in output order (480) is available in the decoded frame storage area (460), it is read by the output sequencer (480) and output to an output destination (490) (e.g., a display). Generally, the order in which the output sequencer (480) outputs frames from the decoded frame storage area (460) can be different from the order in which the frames are decoded by the decoder (450).

V. Syntax Elements to Facilitate Low-Latency Encoding and Decoding

In most video codec systems, encoding order (also known as decoding order or bitstream order) is the order in which video frames are represented in the coded data in the bitstream and, therefore, processed during decoding. This encoding order can differ from the order in which frames were captured by the camera before encoding and from the order in which decoded frames are displayed, stored, or otherwise output after decoding (output order or display order). Reordering frames relative to output order can have benefits (primarily in terms of compression capabilities), but it increases the end-to-end latency of the encoding and decoding process.

The techniques and tools described herein reduce the delay caused by the reordering of video frames and, by providing information to a decoder system regarding constraints on the reordering delay, further facilitate latency reduction in the decoder system. Such latency reduction is useful for many purposes. For example, it can be used to reduce the time lag that occurs in interactive video communications using video conferencing systems, thereby making the flow of conversation and communication interactions between remote participants more rapid and natural.

A. Output Timing and Output Sorting Methods

According to the H.264 standard, a decoder can use two methods to determine when a decoded frame is ready to be output. The decoder can use timing information in the form of a decoding timestamp and an output timestamp (e.g., signaled in a picture timing SEI message). Alternatively, the decoder can use buffer capacity limits signaled using various syntax elements to determine when a decoded frame is ready to be output.

Timing information can be associated with each decoded frame. The decoder can use this timing information to determine when the decoded frame can be output. However, in practice, such timing information may not be available to the decoder. Moreover, even when timing information is available, some decoders do not actually use it (for example, because the decoder is designed to operate regardless of whether timing information is available).

According to the H.264 standard (and draft versions of the HEVC standard), buffer capacity limits are indicated by several syntax elements, including the syntax element max_dec_frame_buffering, the syntax element num_reorder_frames, related ordering information (called "picture order count"), and other memory management control information signaled in the bitstream. The syntax element max_dec_frame_buffering (or its derivative, designated as MaxDpbFrames) specifies the required decoded picture buffer (DPB) size in frame buffer units. Similarly, the syntax element max_dec_frame_buffering indicates the top-level memory capacity used to encode a video sequence so that the decoder can output pictures in the correct order. The syntax element num_reorder_frames (or max_num_reorder_frames) indicates the maximum number of frames (or complementary field pairs, or unpaired fields) that can precede any frame (or complementary field pair, or unpaired fields) in coding order and follow it in output order. In other words, num_reorder_frames specifies a constraint on the memory capacity required for picture reordering. The syntax element max_num_ref_frames specifies the maximum number of short-term and long-term reference frames (or complementary reference field pairs, or unpaired reference fields) that can be used by the decoding process for inter prediction of any picture in the sequence. The syntax element max_num_ref_frames also determines the size of the sliding window used for decoding reference picture marking. Like num_reorder_frames, max_num_ref_frames specifies a constraint on the amount of memory required.

The decoder uses the max_dec_frame_buffering (or MaxDpbFrames) and num_reorder_frames syntax elements to determine when the buffer capacity limit has been exceeded. This occurs, for example, when a new decoded frame needs to be stored in the DPB, but there is no free space left in the DPB. In this case, the decoder uses the picture order count information to identify which of the already decoded pictures is oldest in output order. The oldest pictures in output order are output last. This process is sometimes called "bumping" because the arrival of a new picture that needs to be stored "bumps" a picture out of the DPB.

The information indicated by the max_dec_frame_buffering (or MaxDpbFrames) and num_reorder_frames syntax elements is sufficient to determine the amount of memory required in the decoder. However, the use of this information can introduce unnecessary delay when used to control the "bumping" process for picture output. As defined in the H.264 standard, the max_dec_frame_buffering and num_reorder_frames syntax elements do not establish a limit on the number of reorderings that can be used for any particular picture and, therefore, do not establish a limit on the end-to-end delay. Regardless of the value of these syntax elements, a particular picture can remain in the DPB for an arbitrarily long time before being output, which corresponds to a significant amount of delay added by the encoder's pre-buffering of source pictures.

B. Syntax Elements Indicating Constraints on Frame Reordering Delays

The techniques and tools described herein reduce latency in video communication systems. Coding tools, real-time communication tools, or other tools set limits on the range of reordering that can be used for any frame in a coded video sequence. For example, the limit can be expressed as the number of frames in the coded video sequence that can precede any given frame in output order and follow the given frame in coding order. The limit constrains the reordering delay allowed for any particular frame in the sequence. In other words, the limit constrains the time range (in frames) that can be used for reordering between the coding order and the output order of any particular frame. Limiting the range of reordering helps reduce end-to-end latency. Similarly, establishing such limits can be useful in real-time system negotiation protocols or application specifications for use scenarios where reducing latency is very important.

One or more syntax elements indicate constraints on frame reordering delay. Signaling constraints on frame reordering delay facilitates system-level negotiation for interactive real-time communication or other use cases. It provides a way to directly express constraints on frame reordering delay and characterize the characteristics of a media stream or session.

A video decoder can use the indicated constraints on frame reordering delay to enable reduced-latency output of decoded video frames. Specifically, compared to a frame "bumping" process, signaling constraints on frame reordering delay enables the decoder to more easily and quickly identify frames in the DPB that are ready for output. For example, the decoder can determine the delay status of a frame in the DPB by calculating the difference between the encoding order and the output order for that frame. By comparing the delay status of the frame with the constraints on frame reordering delay, the decoder can determine when the constraints on frame reordering delay have been met. The decoder can immediately output any frames that have reached this limit. This helps the decoder identify frames that are ready for output more quickly than a "bumping" process that uses multiple syntax elements and tracking structures. As a result, the decoder can quickly (and earlier) determine when a decoded frame can be output. The faster (and earlier) the decoder identifies when a frame can be output, the faster (and earlier) the decoder can output video to a display or subsequent processing stage.

Thus, by using a constraint on the frame reordering delay, the decoder can start outputting frames from the decoded frame store before it is full, but still provide standard-compliant decoding (i.e., decoding all frames so that they are bit-exactly matched to frames decoded using an otherwise conventional scheme). This significantly reduces the delay when the delay (in frames) indicated by the delay syntax element is much smaller than the size (in frames) of the decoded frame store.

5a-5e illustrate a series of frames (501-505) with different inter-frame dependencies. The series is characterized by different values of the following constraints: (1) a constraint on the memory capacity required for image reordering (i.e., the number of frame buffers used to store reference frames for reordering purposes, for example, by the syntax element num_reorder_frames), and (2) a constraint on the frame reordering delay, for example, specified by the variable MaxLatencyFrames. In FIG5a-5e, for a given frame _Fjk ^, the subscript j indicates the position of the frame in output order and the superscript k indicates the position of the frame in coding order. The frames are shown in output order - the output order subscript values increase from left to right. The arrows illustrate the inter-frame dependencies used for motion compensation, according to which the preceding frame in coding order is used to predict the subsequent frame in coding order. For simplicity, Figures 5a-5e illustrate inter-frame correlation at the frame level (rather than at the macroblock level, block level, etc., where the reference frame may change), and Figures 5a-5e illustrate that at most two frames are used as reference frames for a given frame. In practice, in some implementations, different macroblocks, blocks, etc. in a given frame may use different reference frames, and more than two reference frames may be used for a given frame.

In FIG5 a, ^a sequence (501) contains nine frames. The last frame _F81 in output order uses the first ^frame _F00 as a reference frame. The other frames in the sequence (501) use both the last ^frame _F81 and ^the first frame _F00 as reference frames. This means that frame _F00 ^is decoded first, followed by frame _F81 , then ^frame _F12 , and so on. In the sequence (501) shown in FIG5 a, the value of num_reorder_frames is 1. At any point in the processing of the decoder system, ^only one frame ( _F81 ) of the frames shown in ^FIG5 a is stored in the decoded frame store for reordering purposes . (The first frame F ₀ ⁰ is also used as a reference frame and is stored, but not for reordering purposes. Because the output order of the first frame F ₀ ⁰ is less than the output order of the intermediate frames, the first frame F ₀ ⁰ is not counted for the purposes of num_reorder_frames.) Despite the small value of num_reorder_frames, the series (501) has a relatively high latency - the value of MaxLatencyFrames is 7. After encoding the first frame F ₀ ⁰ , the encoder waits until it has buffered another eight source frames before encoding the next frame F ₁ ² in output order because the next frame F ₁ ² depends on the last frame F ₈ ¹ in the series (501). The value of MaxLatencyFrames is actually the maximum allowed difference between the subscript value and the superscript value for any particular coded frame.

In Figure 5b, series (502) contains nine frames, just like series (501) in Figure 5a, but the inter-frame correlation is different. Temporal reordering of frames occurs over a short timeframe. As a result, this series (502) has a much lower latency—the value of MaxLatencyFrames is 1. The value of num_reorder_frames remains 1.

In Figure 5c, series (503) contains ten frames. The longest inter-frame correlation (in time) is shorter than the longest inter-frame correlation in Figure 5a, but longer than the longest inter-frame correlation in Figure 5b. This series (503) has the same low value of 1 for num_reorder_frames, and it has a relatively low value of 2 for MaxLatencyFrames. This series (503) therefore allows for lower end-to-end latency than series (501) in Figure 5a, although not as low as the latency allowed by series (501) in Figure 5b.

In FIG5 d , a sequence ( 504 ) comprises frames organized in a temporal hierarchy having three temporal layers according to inter-frame dependencies. The lowest temporal resolution layer comprises a first frame F ₀ ⁰ and a last frame F ₈ ¹ . The next temporal resolution layer adds frame F ₄ ² , which depends on the first frame F ₀ ⁰ and the last frame F ₈ ¹ . The highest temporal resolution layer adds the remaining frames. The sequence ( 504 ) shown in FIG5 d has a relatively low value of 2 for num_reorder_frames but a relatively high value of 7 for MaxLatencyFrames , at least for the highest temporal resolution layer, due to the difference between the encoding order and the output order of the last frame F ₈ ¹ . If only an intermediate temporal resolution layer or the lowest temporal resolution layer is decoded, the constraint on the frame reordering delay can be reduced to 1 (for the intermediate layer) or 0 (for the lowest layer). To facilitate reduced-delay decoding in various temporal resolutions, syntax elements can indicate the constraints on the frame reordering delay for different layers in the temporal hierarchy.

In FIG5e , the series ( 505 ) contains frames organized in a temporal hierarchy with three temporal layers according to different inter-frame dependencies. The lowest temporal resolution layer contains the first frame F ₀ ⁰ , the middle frame F ₄ ¹ , and the last frame F ₈ ⁵ . The next temporal resolution layer adds frames F ₂ ² (which depends on the first frame F ₀ ⁰ and the middle frame F ₄ ¹ ) and F ₆ ⁶ (which depends on the middle frame F ₄ ¹ and the last frame F ₈ ⁵ ). The highest temporal resolution layer adds the remaining frames. Compared to the series ( 504 ) of FIG5d , the series ( 505 ) of FIG5e still has a relatively low value of 2 for num_reorder_frames but a lower value of 3 for MaxLatencyFrames, at least for the highest temporal resolution layer, due to the difference between the encoding order and the output order of the middle frame F ₄ ¹ and the last frame F ₈ ⁵ . If only the middle temporal resolution layer or the lowest temporal resolution layer is decoded, the constraint on the frame reordering delay can be reduced to 1 (for the middle layer) or 0 (for the lowest layer).

In the example shown in Figures 5a-5e, if the value of MaxLatencyFrames is known, the decoder can identify certain frames that are ready to be output as soon as the previous frame in output order is received. For a given frame, the output order value of the frame minus the coding order value of the frame can be equal to the value of MaxLatencyFrames. In this case, as soon as the previous frame in output order is received, the given frame is ready for output. (In contrast, using num_reorder_frames alone, such a frame cannot be identified as ready for output until additional frames are received or the end of the sequence is reached.) Specifically, the decoder can use the value of MaxLatencyFrames to enable early output of the following frames:

• In the series ( ⁵⁰¹ ) of Figure 5a, frame _F81 .

• In ^the series (502) of Figure 5b, ^{frames F21} _{, F43} , _F65 ^and _F87 ^.

• In ^the series (503) of Figure 5c, frames _F31 , _F64 ^and _F97 ^.

• In ^the series (504) of Figure 5d, frame _F81 .

• In _the series (505) of Figure 5e, ^frames _F41 and ^F85 .

Additionally, declaring or negotiating a value for MaxLatencyFrames at the system level may provide a high-level representation of the latency characteristics of a bitstream or session in a way that is not achievable by measuring reorder storage capacity and using num_reorder_frames to indicate that capacity.

C. Example Implementation

The syntax element indicating the constraint on the frame reordering delay can be signaled in various ways, depending on the implementation. The syntax element can be signaled as part of a sequence parameter set ("SPS"), a picture parameter set ("PPS"), or other elements of the bitstream, as part of an SEI message, a VUI message, or other metadata, or in some other way. In any implementation, the syntax element indicating the constraint value can be encoded using unsigned exponential-Golomb coding, some other form of entropy coding, or fixed-length coding and then signaled. The decoder performs the corresponding decoding upon receiving the syntax element.

In a first implementation, a flag, max_latency_limitation_flag, is signaled. If the flag has a first binary value (e.g., 0), no constraints are imposed on the frame reordering delay. In this case, the value of the max_latency_frames syntax element is not signaled or is ignored. Otherwise (the flag has a second binary value, such as 1), the value of the max_latency_frames syntax element is signaled to indicate the constraints on the frame reordering delay. For example, in this case, the value of the signaled max_latency_frames syntax element can be any non-negative integer value.

In a second implementation, a syntax element, max_latency_frames_plus1, is signaled to indicate a constraint on the frame reordering delay. If max_latency_frames_plus1 has a first value (e.g., 0), no constraint is imposed on the frame reordering delay. For other values (e.g., non-zero values), the value of the constraint on the frame reordering delay is set to max_latency_frames_plus1-1. For example, the value of max_latency_frames_plus1 is in the range of 0 to 2 ³² -2, inclusive.

Similarly, in a third implementation, a syntax element max_latency_frames is signaled to indicate a constraint on the frame reordering delay. If max_latency_frames has a first value (e.g., a maximum value), no constraint is imposed on the frame reordering delay. For other values (e.g., values less than the maximum value), the value of the constraint on the frame reordering delay is set to max_latency_frames.

In a fourth implementation, the constraint on the frame reordering delay is indicated relative to the maximum frame memory size. For example, the delay constraint is signaled as an increment relative to the num_reorder_frames syntax element. Typically, the constraint on the frame reordering delay (in frames) is greater than or equal to num_reorder_frames. To save bits in signaling the delay constraint, the difference between the delay constraint and num_reorder_frames is encoded (e.g., using unsigned exponential Golomb coding or some other form of entropy coding) and then signaled. The constraint on the frame reordering delay is signaled using the syntax element max_latency_increase_plus1. If max_latency_increase_plus1 has a first value (e.g., 0), no constraint is imposed on the frame reordering delay. For other values (e.g., non-zero values), the value of the constraint on the frame reordering delay is set to num_reorder_frames + max_latency_increase_plus1 - 1. For example, the value of max_latency_increase_plus1 ranges from 0 to 2 ³² -2, inclusive.

Alternatively, the syntax element or elements indicating the constraints on the frame reordering delay are signaled in some other manner.

D. Other ways to indicate constraints on latency

In many of the aforementioned examples, the delay constraint is a constraint on the frame reordering delay expressed in units of frame counts. More generally, the delay constraint is a constraint on the delay expressed in units of frame counts or in units of seconds, milliseconds, or another time measure. For example, the delay constraint can be expressed as an absolute time measure, such as 1 second or 0.5 seconds. The encoder can convert such a time measure into a frame count (taking into account the frame rate of the video) and then encode the video so that the inter-frame correlation between multiple frames of the video sequence is consistent with the frame count. Alternatively, regardless of frame reordering and inter-frame correlation, the encoder can use the time measure to limit the extent to which the delay can be used to smooth out short-term fluctuations in the bit rate, encoding complexity, network bandwidth, etc. of the encoded video. The decoder can use the time measure to determine when a frame can be output from the decoded picture buffer.

Delay constraints can be negotiated between the transmitter and receiver to balance responsiveness (lack of delay) with the ability to smooth short-term fluctuations in the bit rate of the encoded video, the ability to smooth short-term fluctuations in encoding complexity, the ability to smooth short-term fluctuations in network bandwidth, and/or other factors that benefit from increased delay. In such negotiations, it may be helpful to establish and characterize the delay constraints in a frame rate-independent manner. The constraints can then be applied during encoding and decoding, taking into account the frame rate of the video. Alternatively, the constraints can be applied during encoding and decoding regardless of the frame rate of the video.

E. Generalized Techniques for Setting and Outputting Syntax Elements

FIG6 shows an example technique (600) for setting and outputting syntax elements that facilitate reduced-latency decoding. For example, the real-time communication tool or encoding tool described with reference to FIG2a and 2b performs the technique (600). Alternatively, another tool performs the technique (600).

First, the tool sets (610) one or more syntax elements indicating a constraint on a delay (e.g., a frame reordering delay, a delay in time units) consistent with an inter-frame correlation between a plurality of frames of a video sequence. When the tool comprises a video encoder, the same tool can also receive frames, encode the frames to produce coded data (using the inter-frame correlation consistent with the constraint on the frame reordering delay), and output the coded data for storage or transmission.

Typically, the constraint on the frame reordering delay is the reordering delay that is allowed for any frame in the video sequence. The constraint can be expressed in various ways, however, different ways have different additional meanings. For example, the constraint can be expressed as a maximum count of frames that can precede a given frame in output order but follow the given frame in encoding order. Alternatively, the constraint can be expressed as the maximum difference between the encoding order and the output order of any frame in the video sequence. Alternatively, focusing on individual frames, the constraint can be expressed as a reordering delay associated with a given particular frame in the video sequence. Alternatively, focusing on a group of frames, the constraint can be expressed as a reordering delay associated with the group of frames in the video sequence. Alternatively, the constraint can be expressed in some other way.

Next, the tool outputs (620) the syntax element(s). This facilitates determining when a reconstructed frame is ready for output in the output order of the multiple frames. The syntax element(s) may be output as part of a sequence parameter set or picture parameter set in an elementary coded video bitstream, as part of the syntax of a media storage file or media transport stream that also contains the coded data for the frame, as part of a media feature negotiation protocol (e.g., during the exchange of stream or session parameter values in a system-level negotiation), as part of media system information multiplexed with the coded data for the frame, or as part of media metadata related to the coded data for the frame (e.g., in an SEI message or a VUI message). Different syntax elements may be output to indicate memory capacity requirements. For example, a buffer size syntax element (e.g., max_dec_frame_buffering) may indicate the maximum size of the DPB, while a frame memory syntax element (e.g., num_reorder_frames) may indicate the maximum size of the frame memory used for reordering.

The value of the delay constraint can be expressed in various ways, as described in Section V.C. For example, the tool outputs a flag indicating the presence or absence of the syntax element(s). If the flag indicates the absence of the syntax element(s), then the delay constraint is undefined or has a default value. Otherwise, the syntax element(s) follow and indicate the delay constraint. Alternatively, one value of the syntax element(s) may indicate that the delay constraint is undefined or has a default value, while other possible values of the syntax element(s) indicate an integer count of the delay constraint. Alternatively, for the case where the delay constraint is a constraint on the frame reordering delay, a given value of the syntax element(s) indicates an integer value of the frame reordering delay constraint relative to the maximum size of the frame memory used for reordering, which is indicated by a different syntax element such as num_reorder_frames. Alternatively, the delay constraint can be expressed in some other way.

In some implementations, the frames of a video sequence are organized according to a temporal hierarchy. In this case, different syntax elements may indicate different constraints on the frame reordering delay for different temporal layers of the temporal hierarchy.

F. Generalized Techniques for Receiving and Using Syntactic Elements

FIG7 shows an example technique (700) for receiving and using syntax elements that facilitate reduced-latency decoding. For example, the real-time communication tool or playback tool described with reference to FIG2a and 2b performs the technique (700). Alternatively, another tool performs the technique (700).

First, the tool receives and parses (710) one or more syntax elements indicating a constraint on a delay (e.g., a frame reordering delay, a delay in units of time). For example, the parsing includes reading one or more syntax elements indicating a constraint on the delay from a bitstream. The tool also receives (720) coded data for a plurality of frames of a video sequence. The tool can parse the syntax element(s) and, based on the syntax element(s), determine a constraint on the delay. Typically, the constraint on the frame reordering delay is a reordering delay that is allowed for any frame in the video sequence. The constraint can be expressed in various ways, however, different ways have different additional meanings, as described in the previous section. The syntax element(s) can be signaled as part of a sequence parameter set or a picture parameter set in an elementary coded video bitstream, part of a syntax for a media storage file or a media transport stream, part of a media feature negotiation protocol, part of media system information multiplexed with the coded data, or part of media metadata associated with the coded data. The tool may receive and parse different syntax elements indicating memory capacity requirements, such as a buffer size syntax element such as max_dec_frame_buffering and a frame memory syntax element such as num_reorder_frames.

The value of the delay constraint can be expressed in various ways, as described in Section V.C. For example, the tool receives a flag indicating the presence or absence of the syntax element(s). If the flag indicates that the syntax element(s) are absent, then the delay constraint is undefined or has a default value. Otherwise, the syntax element(s) follow and indicate the delay constraint. Alternatively, one value of the syntax element(s) may indicate that the delay constraint is undefined or has a default value, while other possible values of the syntax element(s) indicate an integer count of the delay constraint. Alternatively, for the case where the delay constraint is a constraint on the frame reordering delay, a given value of the syntax element(s) indicates an integer count of the frame reordering delay constraint relative to the maximum size of the frame memory used for reordering, as indicated by a different syntax element such as num_reorder_frames. Alternatively, the delay constraint is signaled in some other manner.

Returning to FIG7 , the tool decodes ( 730 ) at least some of the encoded data to reconstruct a frame. The tool outputs ( 740 ) the reconstructed frame. In doing so, the tool can use a constraint on latency to determine when the reconstructed frame is ready for output, for example, according to the output order of the frames of the video sequence.

In some implementations, the frames in the video sequence are organized according to a temporal hierarchy. In this case, different syntax elements may indicate different constraints on the frame reordering delay for different temporal layers of the temporal hierarchy. The tool may select one of the different constraints on the frame reordering delay based on the temporal resolution of the output.

In view of the many possible embodiments that can be applied to the principles of the disclosed invention, it should be recognized that the described embodiments are merely preferred examples of the invention and should not be taken as limiting the scope of the invention. Rather, the scope of the invention is defined by the appended claims. I therefore claim protection as my invention for all inventions that come within the scope and spirit of these claims.

Claims (26)

1. A method in a computing system for implementing a video decoder, comprising: Receive and parse one or more syntax elements that indicate constraints on frame reordering delay, wherein the constraints on frame reordering delay are represented by the maximum count of frames that can precede any frame in the video sequence in output order but follow that frame in encoding order. Receive encoded data of multiple frames from a video sequence; Using the video decoder, at least some of the encoded data is decoded to reconstruct one of the multiple frames; and Output the reconstructed frame. 2. The method according to claim 1, further comprising: The constraints on frame reordering delay are determined based on one or more of these syntax elements; and This constraint on frame reordering delay is used to determine when the reconstructed frame is ready for output based on the output order of the multiple frames in the video sequence. 3. The method of claim 2, wherein the plurality of frames of the video sequence are organized according to a time hierarchy, wherein different syntax elements indicate different constraints on frame reordering delay for different time levels of the time hierarchy, the method further comprising selecting one of the different constraints on frame reordering delay according to the temporal resolution of the output. 4. The method of claim 1, wherein the constraint on the frame reordering delay is defined as the maximum difference between the encoding order and the output order of any frames in the video sequence. 5. The method of claim 1, wherein the one or more syntax elements and the encoded data are signaled as part of a syntax for encoding a video bitstream, the method further comprising: Receive and parse a buffer size syntax element that indicates the maximum size of the decoded image buffer, wherein the buffer size syntax element is different from one or more syntax elements that indicate constraints on frame reordering delay. 6. The method of claim 1, wherein the one or more syntax elements are signaled as part of a sequence parameter set, a picture parameter set, a syntax for a media storage file that also contains the encoded data, a syntax for a media transport stream that also contains the encoded data, a media feature negotiation protocol, media system information multiplexed with the encoded data, or media metadata related to the encoded data. 7. The method according to claim 1, further comprising: Receive a flag indicating the presence or absence of the one or more syntax elements, wherein if the flag indicates the absence of the one or more syntax elements, then the constraint on the frame reordering delay is undefined or has a default value. 8. The method according to claim 1, wherein: One possible value of the one or more syntax elements indicates that the constraint on frame reordering delay is undefined or has a default value, and the other possible values of the one or more syntax elements indicate an integer count of the constraint on frame reordering delay. 9. The method according to claim 1, wherein: A value of one or more syntax elements indicates an integer count of constraints on frame reordering delay relative to the maximum size of the frame memory used for reordering, the maximum size of which is indicated by different syntax elements. 10. The method according to claim 9, wherein: The constraint on frame reordering delay can be determined as the maximum count of the maximum size of the frame memory used for reordering plus the integer count of the constraint on frame reordering delay minus one. 11. A method in a computing system, comprising: Syntax elements that set one or more constraints on frame reordering delays, the constraints on frame reordering delays being consistent with the inter-frame correlations between multiple frames in a video sequence, wherein the constraints on frame reordering delays are represented by the maximum count of frames that, in output order, precede any frame in the video sequence but follow that frame in coding order; and Output one or more syntax elements to facilitate the determination of when a frame is ready for output based on the output order of the multiple frames. 12. The method of claim 11, wherein the computing system implements a video encoder, the method further comprising: Receive the multiple frames of the video sequence; Using this video encoder, the multiple frames are encoded to produce encoded data, wherein the encoding uses inter-frame correlation consistent with constraints on frame reordering delay; and Output the encoded data for storage or transmission. 13. The method of claim 11, wherein the one or more syntax elements and the encoded data are output as part of a syntax for encoding a video bitstream, the method further comprising: The output is a buffer size syntax element that indicates the maximum size of the decoded image buffer, wherein the buffer size syntax element is different from the one or more syntax elements that indicate constraints on frame reordering delay. 14. The method of claim 11, wherein the one or more syntax elements are output as part of a sequence parameter set, a picture parameter set, a syntax for a media storage file that also contains encoded data of the plurality of frames, a syntax for a media transport stream that also contains encoded data of the plurality of frames, a media feature negotiation protocol, media system information multiplexed with the encoded data of the plurality of frames, or media metadata relating to the encoded data of the plurality of frames. 15. The method of claim 11, further comprising: The output is a flag indicating whether the one or more syntax elements exist or not, wherein if the flag indicates that the one or more syntax elements do not exist, then the constraint on the frame reordering delay is undefined or has a default value. 16. The method of claim 11, wherein: One possible value of the one or more syntax elements indicates that the constraint on frame reordering delay is undefined or has a default value, and the other possible values of the one or more syntax elements indicate an integer count of the constraint on frame reordering delay. 17. The method of claim 11, wherein: A value of one or more syntax elements indicates an integer count of constraints on frame reordering delay relative to the maximum size of the frame memory used for reordering, the maximum size of which is indicated by different syntax elements. 18. The method of claim 17, wherein: The constraint on frame reordering delay can be determined as the maximum count of the maximum size of the frame memory used for reordering plus the integer count of the constraint on frame reordering delay minus one. 19. A video decoding system, the video decoding system comprising: The memory is configured to receive one or more syntax elements that indicate constraints on frame reordering delays and to receive encoded data for multiple frames of a video sequence. The decoder is configured to: The constraints on frame reordering delay are determined based on one or more syntax elements, wherein the constraints on frame reordering delay are represented by the maximum count of frames that, in output order, precede a given frame but in encoding order, follow that given frame; and At least some of the encoded data needs to be decoded to reconstruct one of the multiple frames; and A frame buffer is configured to store reconstructed frames, wherein the video decoding system is configured to use constraints on frame reordering delays to determine when the reconstructed frames are ready for output according to the output order of the plurality of frames of the video sequence. 20. The video decoding system of claim 19, wherein the one or more syntax elements are signaled as part of a sequence parameter set or media metadata relating to the encoded data of the plurality of frames. 21. The video decoding system according to claim 19, wherein: One of the values of the one or more syntax elements is an integer count indicating the constraint on frame reordering latency relative to the maximum size of the frame memory used for reordering. 22. The video decoding system according to claim 21, wherein: The constraint on frame reordering delay can be determined as the maximum count of the maximum size of the frame memory used for reordering plus the integer count of the constraint on frame reordering delay minus one. 23. A video encoding system, the video encoding system comprising: A frame buffer is configured to receive multiple frames of a video sequence. The encoder is configured to: Syntax elements that set one or more constraints on frame reordering delays, which correspond to the inter-frame correlations between multiple frames in a video sequence, wherein the constraints on frame reordering delays are represented by the maximum count of frames that can precede any frame in the video sequence in output order but follow that frame in encoding order; and The multiple frames are encoded to produce encoded data, wherein the encoding uses inter-frame correlation consistent with the constraint on frame reordering delay; and The memory is configured to buffer the encoded data and one or more syntax elements for output, thereby facilitating the determination of when a frame is ready for output based on the output order of the multiple frames. 24. The video coding system of claim 23, wherein the one or more syntax elements are signaled as part of a sequence parameter set or media metadata relating to the encoded data of the plurality of frames. 25. The video encoding system according to claim 23, wherein: One of the values of the one or more syntax elements is an integer count indicating the constraint on frame reordering latency relative to the maximum size of the frame memory used for reordering. 26. The video encoding system according to claim 25, wherein: The constraint on frame reordering delay can be determined as the maximum count of the maximum size of the frame memory used for reordering plus the integer count of the constraint on frame reordering delay minus one.