RU2782245C1 - Method for supporting network abstraction layer (nal) blocks of mixed types in an encoded image - Google Patents

Abstract

FIELD: video decoding.

SUBSTANCE: invention relates to means of video decoding. A first network abstraction layer (NAL) block of the video coding layer (VCL) of the first slice of the encoded image and a second NAL block of the VCL of the second slice of the encoded image are received. The first NAL block of the VCL belongs to the first type of NAL blocks of the VCL, and the second NAL block of the VCL belongs to the second type of NAL blocks of the VCL, different from the first type of NAL blocks of the VCL. The encoded image is decoded, said decoding including determining the type of the encoded image based on the first type of the NAL block of the VCL and the second type of the NAL block of the VCL, or based on the indicator indicating that the encoded image contains NAL blocks of the VCL of mixed types. Said determining herein includes determining that the encoded image is a subsequent image, based on the first type of the NAL block of the VCL, indicating that the first NAL block of the VCL contains an encoded gradual decoding refresh (GDR) image slice, and the second type of the NAL block of the VCL, indicating that the second NAL block of the VCL contains an encoded instant decoding refresh (IDR) image slice or encoded clear random access (CRA) image slice.

EFFECT: increased efficiency of video decoding.

7 cl, 7 dwg, 6 tbl

Description

LINK TO RELATED APPLICATIONS

[0001] This application claims priority of US Provisional Application No. 62/956,254, filed January 1, 2020, and US Patent Application No. 17/077,035, filed October 22, 2020, which are incorporated herein in their entirety.

FIELD OF TECHNOLOGY TO WHICH THE INVENTION RELATES

[0002] Embodiments of the present invention relate to video encoding and decoding, and in particular to supporting the use of mixed types of Network Abstraction Layer (NAL) blocks in an encoded image.

BACKGROUND OF THE INVENTION

[0003] Versatile Video Coding (VVC) Specification Draft JVET-P2001 (incorporated herein in its entirety) (with editorial changes JVET-Q0041) supports a mixed network abstraction layer (NAL) block type feature that allows have one or more slice NAL units. having the type of internal random access point (IRAP, intra random access point) or pure random access (CRA, clean random access), and one or more slice NAL units having a different type and not being IRAP units. This function can be used to merge two different bitstreams into one or to support different random access periods for each local area (sub-picture). The following syntax and semantics are currently defined to support this functionality:

[0004] Table 1 below shows an example of the raw byte sequence payload (RBSP) image parameter set syntax.

[0005] If the value of the mixed_nalu_types_in_pic_flag syntax element is 1, then this means that each picture belonging to the Picture Parameter Set (PPS) has more than one Video Coding Layer (VCL) NAL unit, with In this case, the values of the nal_unit_type parameters of the VCL NAL units are different and the picture is not an IRAP picture. If the value of the mixed_nalu_types_in_pic_flag syntax element is 0, then this means that each picture referring to this PPS has one or more VCL NAL units, and the VCL NAL units of each picture referring to this PPS have the same nal_unit_type parameter values.

[0006] If the value of the no_mixed_nalu_types_in_pic_constraint_flag syntax element is 1, the value of the mixed nalu_types_in_pic_flag syntax element shall be 0.

[0007] According to the current VVC specification, NAL unit type codes and classes are defined as shown in Table 2 below.

[0008] For each slice with a nal_unit_type parameter value of nalUnitTypeA ranging from IDR_W_RADL to CRA_NUT inclusive in the picA image, which also contains one or more slices with a different nal_unit_type value (i.e., the value of the mixed_nalu_types_in_pic_flag flag for the picA image is 1), the following apply regulations:

[0009] (A) The slice must belong to a subpicture subpicA for which the value of the corresponding subpic treated as_pic flag[i] syntax element is 1.

[0010] (B) A slice must not belong to a picA subimage containing VCL NAL units with a nal_unit_type syntax element not equal to nalUnitTypeA.

[0011] (C) For all of the following (in decoding order) PUs in the Coded Layer Video Sequence (CLVS), neither RefPicList[0] nor RefPicListf[1] of the slice in subpicture subpicA shall include in the active entry any picture preceding (in decoding order) picA.

[0012] For VCL NAL units of any single picture, the following rules apply:

[0013] If the value of the mixed nalu_types_in_pic_flag syntax element is 0, the value of the nal_unit_type syntax element must be the same for all NAL units of the encoded picture slice. The picture or PU is considered to be of the same NAL unit type as the encoded picture slice or PU NAL units. [0014] Otherwise (if the value of the mixed_nalu_types_in_pic_flag syntax element is 1), one or more VCL NAL units must all have a specific nal_unit_type parameter value in the range IDR_W_RADL through CRA_NUT inclusive, and all other VCL NAL units must have a specific nal_unit_type parameter value in range from TRAIL_NUT to RSV VCL 6 inclusive or equal to GDR_NUT.

SUMMARY OF THE INVENTION

[0015] There may be some problems with the current concept of using mixed type VCL NAL units, as described in the Background section above.

[0016] In some cases, when an image consists of mixed types of VCL NAL units, it may be difficult to unambiguously determine the image type.

[0017] Sometimes, when NAL units of different types are mixed in the same PU (image), conflicts with temporal identifier constraints (eg, Temporalld) may occur.

[0018] For example, the current VVC specification has the following restrictions on Temporalld: If the value of the nal_unit_type syntax element is in the range of IDR_W_RADL through RSV IRAP12, then the value of the Temporalld syntax element must be 0. If the value of the nal_unit_type syntax element is STSA_NUT, the value of the element syntax Temporalld should be 0.

[0019] In some cases, if the mixed_nalu_types_in_pic_flag syntax element is conveyed in a PPS set, then the slice NAL units in the CLVS must refer to at least two PPS NAL units. Also, when extracting a sub-picture, its associated PPS must be overwritten by changing the value of the mixed nalu_types_in_pic_flag syntax element.

[0020] In some cases, the existing concept may not support the simultaneous existence of advanced decoded random access NAL units (RADL, Random Access Decodable Leading) / advanced skipped random access NAL units (RASL, Random Access Skipped Leading) with subsequent images (Trail pictures) in picture (PU).

[0021] In some cases, when an image in a layer refers to another image in another layer, the mixed_nalu_types_in_pic flag syntax element may be misaligned.

[0022] Embodiments of the present invention may solve one or more of the above problems and/or other problems.

[0023] According to one or more embodiments, a method is provided that is executed by at least one processor. According to this method: the first NAL unit of the video coding layer (VCL) of the first encoded image slice and the second NAL VCL unit of the second encoded image slice are received, the first VCL NAL unit belongs to the first VCL NAL unit type, and the second VCL NAL unit refers to the second VCL NAL unit type, which is different from the first VCL NAL unit type; and decoding the encoded picture, the decoding comprising determining the type of the encoded picture based on the first VCL NAL unit type and the second VCL NAL unit type, or based on an indicator obtained by at least one processor that indicates that the encoded picture contains VCL NAL units of different types.

[0024] According to one embodiment, said determination comprises determining that an encoded picture is a trailing picture based on a first VCL NAL block type indicating that the first VCL NAL block contains an encoded slice of the subsequent picture, and a second block type NAL VCL indicating that the second NAL VCL contains an Instantaneous Decoding Refresh (IDR) encoded image slice or a Clean Random Access (CRA) encoded image slice.

[0025] According to one embodiment, said determination comprises determining that the encoded picture is a forward decodable random access (RADL) picture, based on a first VCL NAL unit type indicating that the first VCL NAL unit contains an encoded RADL picture slice, and a second a VCL NAL unit type indicating that the second VCL NAL unit contains an instant decoding update (IDR) encoded picture slice or a pure random access (CRA) encoded picture slice.

[0026] According to one embodiment, said definition comprises determining that the encoded picture is a step-wise temporal sub-layer access (STSA) picture based on a first VCL NAL unit type indicating that the first unit The VCL NAL contains an encoded picture slice STS A, and a second VCL NAL unit type indicating that the second VCL NAL unit does not contain an instant decoding refresh (IDR) encoded picture slice.

[0027] According to one embodiment, said determination comprises determining that an encoded picture is a subsequent picture based on a first VCL NAL unit type indicating that the first VCL NAL unit contains a STSA encoded picture slice, and a second VCL NAL unit type indicating that the second VCL NAL unit does not contain a pure random access (CRA) encoded picture slice.

[0028] According to one embodiment, said definition comprises determining that the encoded picture is a subsequent picture based on the first VCL NAL block type indicating that the first VCL NAL block contains a Gradual Decoding Refresh (GDR) encoded picture slice , and a second VCL NAL unit type indicating that the second VCL NAL unit does not contain an instant decoding update (IDR) encoded picture slice or a pure random access (CRA) encoded picture slice.

[0029] According to one embodiment, the indicator is a flag, wherein the determination process determines that the encoded picture is a subsequent picture based on a flag that indicates that the encoded picture contains mixed type VCL NAL units.

[0030] According to one embodiment, the indicator is a flag, wherein the process of decoding the encoded picture further determines that the identifier TemporallD of the encoded picture is 0 based on a flag that indicates that the encoded picture contains mixed type VCL NAL units .

[0031] According to one embodiment, the indicator is a flag, in accordance with the proposed method, the flag is received in the image header or slice header.

[0032] According to one of the embodiments, the indicator is a flag, and the encoded image is in the first level, and in accordance with the proposed method, the flag is further received and it is determined that the additional encoded image located in the second level, which is the reference level of the first level , contains mixed type VCL NAL units, based on a flag that indicates that the encoded picture contains mixed type VCL NAL units.

[0033] According to one or more embodiments, a system is provided that includes: a memory configured to store computer program code; and at least one processor configured to receive at least one encoded video stream, access the computer program code, and perform operations in accordance with instructions from the computer program code. The computer program code comprises: a decoding code configured such that at least one processor decodes an encoded image from at least one encoded video stream, wherein the decoding code comprises a determination code configured such that at least one processor determines the type of encoded image based on the first VCL NAL unit type of the first coded picture slice and the second VCL NAL unit type of the second coded picture slice, or based on an indicator obtained by at least one processor indicating that the encoded picture contains mixed-type VCL NAL units, wherein the first the VCL NAL unit type is different from the second VCL NAL unit type.

[0034] According to one embodiment, the determination code is configured such that at least one processor determines that the encoded picture is a subsequent picture, based on the first VCL NAL unit type indicating that the first VCL NAL unit contains an encoded slice of the subsequent picture, and a second VCL NAL unit type indicating that the second VCL NAL unit comprises an instant decode refresh (IDR) encoded picture slice or a pure random access (CRA) encoded picture slice.

[0035] According to one embodiment, the determination code is configured such that at least one processor determines that the encoded picture is a random access decodable advance (RADL) picture based on the first VCL NAL unit type indicating that the first VCL NAL unit contains an encoded RADL picture slice, and a second VCL NAL unit type indicating that the second VCL NAL unit contains an instant decoding update (IDR) encoded picture slice or a pure random access (CRA) encoded picture slice.

[0036] According to one embodiment, the determination code is configured such that at least one processor determines that the encoded picture is a stepwise temporal sublayer access (STSA) picture based on the first VCL NAL unit type indicating that the first NAL unit The VCL contains an STSA encoded picture slice, and a second VCL NAL unit type indicating that the second VCL NAL unit does not contain an instant decoding update (IDR) encoded picture slice.

[0037] According to one embodiment, the determination code is configured such that at least one processor determines that the encoded picture is a subsequent picture, based on the first VCL NAL unit type indicating that the first VCL NAL unit contains an encoded incremental access picture slice to the temporal sublayer (STSA), and a second VCL NAL unit type indicating that the second VCL NAL unit does not contain a pure random access (CRA) encoded picture slice.

[0038] According to one embodiment, the determination code is configured such that at least one processor determines that the encoded picture is a subsequent picture, based on the first VCL NAL unit type indicating that the first VCL NAL unit contains a coded picture slice with gradual decoding update (GDR), and a second VCL NAL unit type indicating that the second VCL NAL unit does not contain an instant decoding update (IDR) encoded picture slice or a pure random access (CRA) encoded picture slice.

[0039] According to one embodiment, the indicator is a flag, wherein the determination code is configured such that at least one processor determines that the encoded picture is a subsequent picture based on a flag that indicates that the encoded picture contains mixed type VCL NAL units.

[0040] According to one embodiment, the indicator is a flag, wherein the determination code is further configured such that at least one processor determines that the identifier TemporallD of the encoded picture is 0 based on a flag indicating that the encoded picture contains mixed type VCL NAL units.

[0041] According to one embodiment, the indicator is a flag, wherein at least one processor is configured to receive the flag in an image header or a slice header.

[0042] One or more embodiments provide a computer-readable medium that stores computer instructions. When these computer instructions are executed by at least one processor, the latter: decodes an encoded image from at least one encoded video stream, while in the process of decoding determines the type of the encoded image based on the first type of the VCL NAL unit of the first slice of the encoded image and the second type of the NAL VCL unit of the second slice of the encoded picture, or based on an indicator obtained by at least one processor indicating that the encoded picture contains mixed VCL NAL unit types, wherein the first VCL NAL unit type is different from the second VCL NAL unit type.

BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Other features, nature and various advantages of the present invention will become more apparent from the following detailed description and the accompanying drawings:

[0044] FIG. 1 is a simplified block diagram of a communication system in accordance with one embodiment.

[0045] FIG. 2 is a simplified block diagram of a communication system in accordance with one embodiment.

[0046] FIG. 3 is a simplified block diagram of a decoder according to one embodiment.

[0047] FIG. 4 is a simplified block diagram of an encoder in accordance with one embodiment.

[0048] FIG. 5 is a block diagram of NAL units in accordance with one embodiment.

[0049] FIG. 6 is a block diagram of a decoder according to one embodiment.

[0050] FIG. 7 shows a computer system suitable for implementing the embodiments.

DETAILED DESCRIPTION

[0051] FIG. 1 is a simplified block diagram of a computer system (100) in accordance with one embodiment of the present invention. The system (100) may include at least two terminals (110, 120) connected via a network (150). For unidirectional data transmission, the first terminal (110) may encode video data in local mode for transmission to another terminal (120) over the network (150). The second terminal (120) may receive the encoded video data of another terminal over the network (150), decode the encoded data, and display the recovered video data. Unidirectional data transmission can be widely used in applications for multimedia services and the like.

[0052] In FIG. 1 shows a second pair of terminals (130, 140) designed to support bi-directional transmission of coded video, which may occur during video conferences, for example. For bi-directional data transmission, the first terminal (130, 140) may encode the video data in local mode for transmission to another terminal over the network (150). Each terminal (130, 140) may also receive the encoded video data transmitted by the other terminal, may decode the encoded data, and display the reconstructed video data on a local display device.

[0053] The terminals (110-140) illustrated in FIG. 1 may be servers, personal computers, smartphones, and/or any other type of terminal. For example, terminals (110-140) may be laptops, tablet computers, media players, and/or dedicated video conferencing equipment. Network (150) is any collection of networks that transmit encoded video data between terminals (110-140), including, for example, wired and/or wireless communication networks. The communication network (150) may transmit data in circuit switched and/or packet switched mode. The networks used include telecommunications networks, local area networks, global area networks and/or the Internet. The architecture and topology of the network (150) is not relevant to the implementation of the present invention, unless otherwise specified below in this document.

[0054] FIG. 2 shows an embodiment of the present invention in which a video encoder and a video decoder are placed in a streaming media. The described invention can be equally used in other video-enabled applications such as videoconferencing, digital television, compressed video storage on digital media including CDs, DVDs, memory cards, and the like, and so on.

[0055] As shown in FIG. 2, the streaming system (200) may include a capture subsystem (213), which may consist of a video source (201) and an encoder (203). The video source (201) may be, for example, a digital camera and may be configured to produce an uncompressed video sample stream (202). The uncompressed video sample stream (202) may generate a large amount of data compared to the encoded video bitstreams and may be processed by an encoder (203) connected to a camera (201). The encoder (203) may include hardware, software, or a combination thereof to provide or implement aspects of the disclosed invention, as described in more detail below. The encoded video bitstream (204) may contain less data than the sample stream and may be stored on a streaming server (205) for later use. One or more streaming clients (206) may contact the streaming server (205) to receive video bitstreams (209), which may be copies of the encoded video bitstream (204).

[0056] In embodiments, the streaming server (205) may also function as a media-aware network element (MANE). For example, the streaming server (205) may be configured to limit the encoded video bitstream (204) and match potentially different bitstreams for one or more streaming clients (206). In some embodiments, the MANE may be implemented in the streaming system (200) separate from the streaming server (205).

[0057] Streaming clients (206) may include a video decoder (210) and a display (212). The video decoder (210) may, for example, decode the video bitstream (209) which is an incoming copy of the encoded video bitstream (204) and produce an outgoing video sample stream (211) that can be played back on a display (212) or other playback device. (not shown). In some streaming systems, the video bitstreams (204, 209) may be encoded according to certain video coding/compression standards. An example of such a standard, in particular, is ITU-T Rec. H.265. A video coding standard is currently being developed, unofficially called Versatile Video Coding (VVC). The described embodiments may be used in the context of the VVC standard.

[0058] FIG. 3 shows an example block diagram of a video decoder (210) connected to a display (212) in accordance with one embodiment of the present invention.

[0059] The video decoder (210) may include a channel (312), a receiver (310), a buffer memory (315), an entropy decoder/analyzer (320), a scaling/inverse transform block (351), an intra-prediction block (352), a motion compensated prediction (353), aggregator (355), loop filter block (356), reference picture memory (357), and current picture memory (358). In at least one embodiment, video decoder (210) may comprise an integrated circuit, a number of integrated circuits, and/or other electronic circuits. The video decoder (210) may also be partially or completely implemented in software running on one or more processors with appropriate memory.

[0060] In the present and other embodiments, the receiver (310) may receive one or more coded video sequences to be decoded by the decoder (210) one coded video sequence at a time, with each coded video sequence being decoded independently of other coded video sequences. The encoded video sequence may be obtained from a channel (312), which may be a hardware/software connection to a storage device that stores the encoded video data. The receiver (310) may receive encoded video data along with other data, such as encoded audio data and/or ancillary data, which may be transmitted to their respective entities (not shown) using them. The receiver (310) may separate the encoded video sequence from other data. To eliminate the influence of network jitter between the receiver (310) and the entropy decoder/analyzer (320) (hereinafter referred to as the analyzer), a buffer memory (315) can be installed. If the receiver (310) is receiving data from a storage/transmission device with sufficient bandwidth and control capability, or from an isosynchronous network, the buffer memory (315) may be unused or small. When used in packet networks with non-guaranteed data delivery, such as the Internet, buffer memory (415) may be required, which will be relatively large and may be adaptive.

[0061] The video decoder (210) may include an analyzer (320) for recovering symbols (321) from an entropy-encoded video sequence. The categories of these symbols contain, for example, information for controlling the operation of the decoder (210), and possibly also information for controlling a playback device, in particular a display (212), which may be connected to the decoder, as shown in FIG. 2. The control information for the playback device(s) can be represented, for example, as Supplementary Enhancement Information (SEI) messages or Video Usability Information (VUI) parameter set fragments (not shown). ). The analyzer (320) may perform analysis/entropy decoding of the received encoded video sequence. Encoding of the encoded video sequence may be in accordance with a video coding technology or standard, and may follow principles well known to those skilled in the art, such as variable length coding, Huffman coding, context sensitive or context sensitive arithmetic coding, etc. The analyzer (320) can extract from the encoded video sequence a set of subgroup parameters for at least one of the subgroups of pixels in the video decoder based on at least one parameter corresponding to the group. Subgroups may include groups of pictures (GOPs), pictures, tiles, slices, macroblocks, coding units (CUs), blocks, transform units (TUs), prediction units (PUs), and so on. The analyzer (320) may also extract information such as transform coefficients, quantizer parameter values, motion vectors, and so on, from the encoded video sequence.

[0062] The analyzer (320) may perform an entropy decoding/analysis operation of the video sequence obtained from the buffer memory (315) to create symbols (321).

[0063] Symbol recovery (321) may use many different units depending on the type of encoded video image or parts thereof (eg, inter-picture and intra-picture, inter-block and intra-block) and other factors. The composition and nature of the use of units can be regulated by the control information of the subgroup, which was extracted by the analyzer (320) from the encoded video sequence. The flow of such subgroup control information between the analyzer (320) and the various blocks is not shown below for simplicity.

[0064] In addition to the functional blocks already mentioned, the decoder 210 can be conceptually divided into a number of functional blocks, as described below. In practice, under commercial constraints, many of these blocks interact closely with each other and can be at least partially integrated with each other. However, when describing the present invention, it is advisable to use the conceptual division into the following functional units.

[0065] One block is a scaling/inverse transform block (351). The scaling/inverse transform block (351) can receive from the analyzer (320) the quantized transform coefficient, as well as control information in the form of symbol(s) (321), including data on the type of transform used, block size, quantization factor, quantization scaling matrix etc. At the output of the scaling / inverse transformation block (351), blocks containing sample values can be obtained, which can be fed to the aggregator (355).

[0066] In some cases, the output samples of the scaler/inverse transform block (351) may refer to an intra coding block, that is, a block that does not use prediction information based on previously restored images, but can use prediction information based on previously restored parts of the current image . Such prediction information may be provided by an intra prediction block (352). In some cases, the intra-prediction block (352) generates a block of the same size and shape as the restored block, using information about already restored neighboring blocks extracted from the current (partially restored) image in the current image memory (358). In some cases, the aggregator (355) adds, individually for each sample, the prediction information generated by the intra prediction block (352) to the output sample information from the scaler/inverse transform block (351).

[0067] In other cases, the output samples of the scaler/inverse transform block (351) may refer to an interframe coding block with possible motion compensation. In this case, the motion-compensated predictor (353) may access the reference picture memory (357) to obtain samples used for prediction. After compensating for the movement of the received samples according to the symbols (321) associated with the block, these samples can be added by the aggregator (355) to the output of the scaler / inverse transform (351) block (in this case called residual samples or residual signal) to form output information. The addresses in the reference picture memory (357) from which the motion-compensated predictor (353) retrieves prediction samples may be determined by the motion vectors. The motion vectors may be available to the motion compensated prediction block (353) as symbols (321), which may be represented by, for example, X, Y components, and a reference picture. Motion compensation may also involve interpolation of sample values obtained from the reference picture memory (357) when accurate sub-sample-level motion vectors, motion vector prediction mechanisms, etc. are used.

[0068] The output samples of the aggregator (355) may be subjected to various loop filtering techniques in the loop filter block (356). Video compression technologies may use loop filtering techniques controlled by parameters contained in the encoded video bitstream and available to the loop filter block (356) in the form of symbols (321) from the analyzer (320), but they may also be responsive to meta-information obtained in the process. decoding the previous (in the order of decoding) parts of the encoded image or encoded video sequence, as well as responding to previously restored and loop filtered sample values.

[0069] At the output of the loop filter block (356), there may be a stream of samples that can be fed to a playback device, such as a display (212), and also stored in reference picture memory (357) for later use for inter-picture prediction.

[0070] Some encoded pictures after full reconstruction may serve as reference pictures for future prediction. When the encoded picture is fully recovered and identified as a reference picture (e.g., by the analyzer (320)), the current reference picture may become part of the reference picture memory (357), whereby the current picture memory may be reallocated to a fresh picture before the next encoded picture is restored.

[0071] The video decoder (210) may perform decoding operations in accordance with a predetermined video compression technology, such as ITU-T Rec. H.265. The encoded video sequence may conform to the syntax defined by the compression technology or standard being used, in the sense that it conforms both to the syntax of the video compression technology or standard itself, as defined in the video compression technology or standard documentation, and to the specific profiles that are enshrined in those documents. Compliance with individual requirements of such a video compression technology or standard may also require that the complexity of the encoded video sequence does not go beyond the level of the video compression technology or standard. In some cases, levels limit the maximum image size, the maximum frame rate, the maximum recovery sample rate (measured in millions of samples per second, for example), the maximum reference image size, and so on. The limits defined by the levels may in some cases be further narrowed by the specifications of the Hypothetical Reference Decoder (HRD) and the metadata for managing the HRD buffer that is carried in the encoded video sequence.

[0072] In one embodiment, the receiver (310) may receive additional (redundant) data along with the encoded video. The additional data may be part of the encoded video sequence(s). The additional data may be used by the video decoder (210) to properly decode the data and/or recover the original video data more accurately. The additional data may be represented, for example, as temporal, spatial or SNR enhancement layers, redundant slices, redundant images, FEC codes, and so on.

[0073] FIG. 4 shows an example block diagram of a video encoder (203) connected to a video source (201), in accordance with one embodiment of the present invention.

[0074] The video encoder (203) may include, for example, an encoder representing a source encoder (430), an encoding engine (432), a (local) decoder (433), a reference picture memory (434), a predictor (435), a transmitter ( 440), entropy encoder (445), controller (450) and channel (460).

[0075] The video encoder (203) may receive video samples from a video source (201) (not part of the encoder) that may capture the video image(s) to be encoded by the encoder (203).

[0076] The video source (201) may provide the encoder (203) for encoding the source video sequence as a digital stream of video samples, which may have any suitable bit depth (e.g. 8 bit, 10 bit, 12 bit, ...), any color space (e.g. BT.601, Y CrCB, RGB, ...), and any suitable sampling structure (eg Y CrCb 4:2:0, Y CrCb 4:4:4). In media services systems, the video source (201) may be a storage device that stores pre-rendered video. In a videoconferencing system, the video source (203) may be a camera that captures a local image as a video sequence. The video data may be provided as a plurality of individual images that convey motion when viewed sequentially. The images themselves can be represented as a spatial array of pixels, where each pixel can consist of one or more samples depending on the sampling structure, color space, etc. used. It is easy for a person skilled in the art to understand the relationship between pixels and samples. In the description below, the readings appear.

[0077] According to one embodiment, the encoder (203) can encode and compress images of the original video sequence into the encoded video sequence (443) in real time or under any other time constraints as required by the application. Ensuring the proper coding rate is one of the functions of the controller (450). The controller (450) may also control other functional blocks, as described below, and be operatively linked to other functional blocks. For clarity, these relationships are not shown. The parameters set by the controller (450) may include parameters related to speed control (skipping images, quantizer, lambda value in rate-distortion optimization methods,…), image size, group of pictures (GOP) arrangement, maximum range motion vector search and so on. One of ordinary skill in the art can readily determine other functions of the controller (450) that may relate to a video encoder (203) optimized for a particular system design.

[0078] Some video encoders operate in what is known to those skilled in the art as a "coding loop". If the description is extremely simplified, then the encoding loop may consist of the encoding part of the source encoder (430) (responsible, for example, for creating symbols based on the input image to be encoded and the reference image(s)), and (local ) a decoder (433) built into the encoder (203) that decompresses the symbols to create samples that a (remote) decoder would also create in some video compression technologies when the compression between the symbols and the encoded video bitstream is lossless. The reconstructed sample stream may be entered into a reference picture memory (434). Since decoding a stream of symbols results in bit-accurate results regardless of the location of the decoder (local or remote), the contents of the reference picture memory of the local and remote encoder are also bit-accurate. In other words, the prediction part of the encoder "sees" as reference picture samples exactly the same sample values that the decoder "sees" when using prediction in the decoding process. This fundamental principle of reference picture synchronism (and the offset that occurs when synchronism cannot be maintained, for example due to channel errors) is known to those skilled in the art.

[0079] The "local" decoder (433) may function in the same way as the "remote" decoder (210), which has already been discussed in detail above in the description of FIG. 3. However, since the symbols are available and the entropy encoder (445) and parser (320) can be losslessly encoded and/or decoded into the encoded video sequence, in the local decoder (433) the entropy decoding related parts of the decoder (210) including channel (312), receiver (310), buffer memory (315) and analyzer (320), may not be fully implemented.

[0080] It should be noted here that any decoding technology, other than analysis and/or entropy decoding, that is applied in a decoder may be required in a corresponding encoder in a substantially identical functional form. For this reason, the description of the present invention is mainly devoted to the operation of the decoder. The description of encoding technologies can be abbreviated as they are inverse to the comprehensively described decoding technologies. Only a few aspects require a more detailed description, which is given below.

[0081] The source encoder (430) may perform motion-compensated predictive coding when an input picture is predictively encoded on one or more previously encoded pictures from a video sequence that have been defined as "reference pictures". In this case, the coding engine (432) encodes the differences between the pixel blocks of the input image and the pixel blocks of the reference image(s) that can be selected as the prediction source(s) for the input image.

[0082] The local video decoder (433) can decode the encoded

image video data that can be determined as reference images based on the symbols generated by the source encoder (430). The processes performed in the encoding engine (432) may advantageously be lossy processes. When the encoded video data can be decoded in a video decoder (not shown in FIG. 4), the reconstructed video sequence can typically be a copy of the original video sequence with some errors. The local video decoder (433) reproduces the decoding processes that may be performed by the video decoder for the reference pictures and may store the reconstructed reference pictures in the reference picture memory (434). Thus, the encoder (203) can locally store copies of the reconstructed reference pictures that are identical in content to the reconstructed reference pictures received by the remote video decoder (in the absence of transmission errors).

[0083] The predictor (435) may search for predictions for the encoding engine (432). That is, for a new encoded picture, the predictor (435) may search the reference picture memory (434) for sample data (in the form of pixel reference block candidates) or certain metadata such as reference picture motion vectors, block shapes, and so on, which can serve as suitable prediction source for new images. When searching for suitable sources of prediction, the predictor (435) may operate on the relationship between blocks of samples and blocks of pixels. In some cases, depending on the search results obtained by the predictor (435), prediction sources for the input picture may be obtained from multiple reference pictures stored in the reference picture memory (434).

[0084] The controller (450) may control the encoding operations of the video encoder (430), including, for example, setting parameters and subgroup parameters used to encode video data.

[0085] The output of all the functional blocks mentioned above can be entropy encoded in an entropy encoder (445).

An entropy encoder translates symbols generated by various functional blocks into an encoded video sequence, losslessly compressing these symbols using techniques known to those skilled in the art such as Huffman coding, variable length coding, arithmetic coding, and so on.

[0086] The transmitter (440) may buffer the encoded video sequence(s) created by the entropy encoder (445) in preparation for transmission over a communication channel (460), which may be a hardware/software connection to the storage device where the encoded video data will be stored. The transmitter (440) may combine the encoded video data from the video encoder (430) with other transmitted data, such as encoded audio data and/or auxiliary data streams (sources not shown).

[0087] The controller (450) may control the operation of the encoder (203). During the encoding process, the controller (450) may assign a certain type to each encoded image, which may affect the encoding methods that can be applied to the corresponding image. For example, images can often be referred to as an intraframe (type I image), a predictive image (type P image), or a bidirectional predictive image (type B image).

[0088] An intra frame (type I picture) can be encoded and decoded without using any other picture in the sequence as a prediction source. Some video codecs allow various types of intraframes to be used, including, for example, instant decoder update (IDR) pictures. Those skilled in the art will be aware of these Type I image variants and their respective applications and features.

[0089] A predictive picture (P-type picture) may be encoded and decoded using intra-frame or inter-frame prediction using at most one motion vector and a reference index to predict sample values of each block.

[0090] A picture with bidirectional prediction (picture type B) can be encoded and decoded using intra frame or inter frame prediction using no more than two motion vectors and reference indices to predict the sample values of each block. Similarly, multiple prediction pictures may use more than two reference pictures and their associated metadata to reconstruct one block.

[0091] Source images can typically be spatially partitioned into multiple blocks of samples (eg, blocks of 4x4, 8x8, 4x8, or 16x16 samples each) and encoded block by block. The blocks may be predictively encoded on other (already encoded) blocks according to the coding type applied to the corresponding block images. For example, type I image blocks may be encoded without prediction or with prediction on already encoded blocks of the same image (spatial prediction or intra prediction). Blocks of pixels of P-type images may be encoded without prediction, with spatial or temporal prediction on one previously encoded reference picture. Type B picture blocks may be encoded without prediction, with spatial or temporal prediction on one or two previously encoded reference pictures.

[0092] The video encoder (203) may perform encoding operations in accordance with a predetermined video coding standard or technology, such as ITU-T Rec. H.265. The video encoder (203) may perform various compression operations during its operation, including predictive coding operations that exploit temporal and spatial redundancy in the input video sequence. Thus, the encoded video data may follow the syntax defined by the video coding technology or standard used.

[0093] In one embodiment, the transmitter (440) may transmit additional data along with the encoded video. Video encoder (430) may include such data as part of an encoded video sequence. Additional data may include temporal, spatial, and SNR enhancement levels, other forms of redundant data such as redundant images and slices, incremental enhancement information (SEI) messages, usability information (VUI) parameter set fragments, and so on.

[0094] Embodiments of the present invention may modify the current VVC specification and implement the NAL unit type codes and classes defined in Table 2 above.

[0095] An "internal random access point picture" (or "IRAP picture") may be a picture that does not refer to any pictures other than itself for inter-frame prediction in the decoding process, and may be a pure random access (CRA) picture or an instantaneous decode updates (IDR). The first (in decoding order) picture in the bitstream may be an IRAP picture or a Gradual Decoding Update (GDR) picture. With the necessary parameter sets available at the right time, an IRAP picture and all following (in decoding order) non-RASL coded video sequence (CVS) pictures can be correctly decoded without decoding any of the pictures preceding it (in decoding order). ) to the IRAP image.

[0096] A "trailing picture" may be a picture that is not an IRAP picture and follows (in output order) its associated IRAP picture, while it is not a stepwise temporal sublayer access (STSA) picture.

[0097] A "Temporal Step Access Picture" (or "STSA Picture") may be a picture that does not use pictures with the same Temporalld ID as the STSA picture for inter-prediction. Pictures following (in decoding order) an STSA picture with the same Temporalld as the STSA picture cannot use pictures preceding (in decoding order) an STSA picture with the same Temporalld as the STSA picture for inter prediction. The STSA image may provide an upward switching to the sublayer containing the STSA image from the nearest lower sublayer. STSA images can have a Temporalld ID greater than 0.

[0098] The "Advance random access skip picture" (or "RASL picture") may be a picture that is a leading picture for the associated CRA picture. If the associated CRA image has the NoIncorrectPicOutputFlag flag set to 1, the RASL image cannot be output and cannot be correctly decoded because the RASL image may contain image references that are not present in the bitstream. RASL pictures cannot be used as reference pictures in a non-RASL picture decoding process. If RASL pictures are present, they may all precede (in decoding order) all subsequent pictures of the same associated CRA picture.

[0099] The "Advance random access decodable picture" (or "RADL picture") may be a leading picture that is not used as a reference picture in the process of decoding subsequent pictures of the same associated IRAP picture. If RADL pictures are present, they may all precede (in decoding order) all subsequent pictures of the same associated IRAP picture.

[0100] An "immediate decoding update picture" (or "IDR picture") may be a picture with which leading pictures present in the bitstream are not associated (for example, with a nal_unit_type parameter value of IDR_N_LP), or with which RASL pictures are not associated, present in the bitstream, but with which RADL images in the bitstream can be associated (eg, with a nal_unit_type parameter value of IDR_W_RADL).

[0101] A “Pure Random Access Picture” (or “CRA Picture”) may be a picture that does not refer to any pictures other than itself for inter prediction in the decoding process, and may be the first (in decoding order) picture in a bitstream, or may appear in the bitstream later. A CRA image may have RADL or RASL images associated with it. If the CRA picture has the NoIncorrectPicOutputFlag flag set to 1, the associated RASL pictures cannot be output by the decoder because they cannot be correctly decoded because they may contain references to pictures that are not in the bitstream.

[0102] According to one or more embodiments, if the mixed nalu_types_in_pic_flag syntax element of the PPS referenced by the encoded picture is 1, then the type of the encoded picture is determined (e.g., by a decoder) as follows:

[0103] (A) If the nal_unit_type parameter of an image NAL unit is TRAIL_NUT, and the nal_unit_type parameter of another image NAL unit is in the range of IDR_W_RADL to CRA_NUT, then the image is determined as a subsequent image.

[0104] (B) If the nal_unit_type parameter of an image NAL unit is RADLJNUT, and the nal_unit_type parameter of another image NAL unit is in the range of IDRWRADL to CRA_NUT, then the image is determined to be a RADL image.

[0105] (C) If the nal_unit_type parameter of an image NAL unit is STSA_NUT and the nal_unit_type parameter of another image NAL unit is IDR_W_RADL or IDR_N_LP, then the image is determined to be an STSA image.

[0106] (D) If the nal_unit_type parameter of an image NAL unit is STSA_NUT and the nal_unit_type parameter of another image NAL unit is CRA_NUT, then the image is determined as a subsequent image.

[0107] (E) If the nal_unit_type parameter of an image NAL unit is GDR_NUT, and the nal_unit_type parameter of another image NAL unit is in the range of IDR_W_RADL to CRA_NUT, then the image is determined as a subsequent image.

[0108] According to one or more embodiments, if the mixed_nalu_types_in_pic_flag syntax element of the PPS set referenced by the encoded picture is 1, then the type of the encoded picture is determined (eg, by a decoder) as a subsequent picture.

[0109] The aspect described above may provide a solution to "Problem 1" described above in the "Summary" section.

[0110] According to one or more embodiments, the mixing of STSA NAL units with IRAP NAL units may be prohibited.

[0111] For example, for VCL NAL units of any single picture, the following rules may apply:

[0112] If the value of the flag mixed_nalu_types_in_pic flag is 0, the value of the nal_unit_type parameter must be (they can be defined) the same for all NAL units of the encoded slices of the picture. The picture or PU is considered to be of the same NAL unit type as the encoded picture slice or PU NAL units.

[0113] Otherwise (if the value of the mixed_nalu_types_in_pic_flag flag is 1), then one or more VCL NAL units must all have (or can be determined to have) a specific nal_unit_type parameter value in the range from IDR_W_RADL to CRA_NUT inclusive, and all other NAL units VCLs must have (or may be defined to have) a specific nal_unit_type parameter value in the range RADL_NUT through RSV_VCL_6 inclusive, or equal to GDR_NUT or TRAIL_NUT.

[0114] According to one embodiment, the encoder may be configured to apply the above rules to prevent mixing of STSA_NAL blocks with IRAP_NAL blocks. According to one embodiment, the decoder may be configured to determine NAL unit values based on the above rules.

[0115] According to one or more embodiments, the Temporalld constraint on STSA_NUT of the current draft VVC JVET-P2001 specification may be removed.

[0116] That is, for example, in an embodiment of the present invention, it is possible not to implement the constraint that the Temporalld identifier must not be equal to 0 if the nal_unit_type parameter is set to STSA_NUT. However, an embodiment may still implement the constraint that the Temporalld identifier must be (or can be defined) equal to 0 if the nal_unit_type parameter has a value in the range of IDR_W_RADL through RSV_IRAP_12, inclusive.

[0117] According to one or more embodiments, a constraint may be implemented that the TemporallD of an image with a mixed nalu_types_in_pic flag of 1 must be 0. For example, an encoder or decoder of the present invention may determine the TemporallD of an image to be 0 based on that the mixed nalu_types_inj_pic flag is 1.

[0118] The aspects described above may provide a solution to "problem 2" described above in the "Summary" section.

[0119] According to one or more embodiments, the mixed_nalu_types_in_pic_flag syntax element may be present in the image header or slice header instead of being present in the PPS set. An example of the mixed nalu_types_in_pic flag syntax element in an image header is shown in Table 3 below.

[0120] If the value of the mixed_nalu_types_in_pic_flag syntax element is 1, then this may mean that each image associated with the PH has more than one NAL VCL unit, while the values of the nal_unit_type parameters of the VCL NAL units are different and the image is not an IRAP image. If the value of the mixed nalu_types_in_pic flag syntax element is 0, then this may mean that each picture associated with the PH has one or more NAL VCL units, and the NAL units of each picture associated with the PH have the same nal_unit_type parameter values.

[0121] If the value of the no_mixed_nalu_types_in_pic_constraint_flag syntax element is 1, the value of the mixed_nalu_types_in_pic_flag flag must be (or may be defined) equal to 0.

[0122] According to one or more embodiments, the mixed nalu_types_inj_pic flag syntax element may be present in an image header or slice header with an SPS presence flag.

[0123] An example of an SPS with a presence flag (sps_mixed_nalu_types_present_flag) is shown in Table 4 below.

[0124] An example of a picture header with a mixed_nalu_types_in_pic_flag syntax element is shown in Table 5 below.

[0125] If the value of the sps_mixed_nalu_types_present flag syntax element is 1, then this may mean that zero or more images referencing this SPS set have more than one NAL VCL unit, while the values of the nal_unit_type parameters of the VCL NAL units differ and the image is not IRAP image. If the value of the sps mixed nalu_types_jtresentflag syntax element is 0, then this means that each image referring to this SPS set has one or more VCL NAL units, and the VCL NAL units of each image referring to this PPS set have the same values parameter nal_unit_type.

[0126] If the value of the no mixed nalu_types_in_pic constraint flag syntax element is 1, the value of the sps_mixed_nalu_types_present_flag syntax element shall be (or may be defined) equal to 0.

[0127] If the value of the mixed_nalu_types_in_pic_flag syntax element is equal to 1, then this may mean that each image associated with the PH has more than one NAL VCL unit, while the values of the nal_unit_type parameters of the VCL NAL units are different and the image is not an IRAP image. If the value of the mixed nalu_types_in_pic flag syntax element is 0, then this may mean that each picture associated with the PH has one or more NAL VCL units, and the NAL units of each picture associated with the PH have the same nal_unit_type parameter values. If the mixed nalu_types_in_pic flag is absent, its value can be taken (for example, in the decoder) to be 0.

[0128] The aspects described above may provide a solution to "Problem 3" described above in the "Summary" section.

[0129] In one or more embodiments, the mixed nalu_types_injtic flag may be replaced by a mixed_nalu_types_in_pic_idc indicator.

[0130] An example of a picture parameter set with a mixed_nalu_types_in_pic_idc syntax element is shown in Table 6 below.

[0131] If the value of the mixed_nalu_types_in_pic_idc syntax element is 1 or 2, then this may mean that each image referring to this PPS set has more than one NAL VCL unit, while the values of the nal_unit_type parameters of the VCL NAL units differ and the image is not an image IRAP. If the value of the mixed_nalu_types_in_pic_idc syntax element is 0, then this may mean that each picture referring to this PPS set has one or more VCL NAL units, and the VCL NAL units of each picture referring to this PPS set have the same parameter values nal_unit_type. Other values of the mixed nalu_types_injtic idc syntax element may be reserved by ITU-T ISO/IEC for future use.

[0132] If the value of the no_mixed_nalu_types_in_pic_constraint_idc syntax element is equal to 1, the value of the mixed nalu_types_injtic_idc indicator should be (or may be determined, for example, by a decoder) equal to 0.

[0133] For each slice with a nal_unit_type parameter value of nalUnitTypeA ranging from IDR_W_RADL to CRA_NUT inclusive, in a picA image that also contains one or more slices with a different nal_unit_type parameter value (i.e., the mixed_nalu_types_in_pic_idc indicator value for the picA image is 1), the following rules may apply:

[0134] (A) The slice must belong to (or can be determined to belong to) a subpicture subpicA for which the value of the corresponding subpic_treated_as_pic_flag[i] flag is 1.

[0135] (B) A slice must not belong to (or may be determined not to belong to) a picA subimage containing VCL NAL units with a nal_unit_type syntax element not equal to nalUnitTypeA.

[0136] (C) For all subsequent (in decoding order) PUs in the CLVS sequence, neither RefPicListf 0] nor RefPicListf 1] of the slice in subpicture subpicA shall include in the active record any picture preceding (in decoding order ) picA.

[0137] RefPicList[0] may be a reference picture list that is used for inter prediction of slice P or a first reference picture list used for inter prediction of slice B.

The RefPicList[l] list may be the second reference picture list used for inter-picture prediction of slice B.

[0138] For VCL NAL units of any single picture, the following rules may apply:

[0139] (A) If the value of the mixed_nalu_types_in_pic_idc syntax element is 1, one or more VCL NAL units must all have (or can be determined to have) a specific nal_unit_type parameter value in the range IDR_W_RADL through CRA_NUT inclusive, and all other VCL NAL units must have (or may be defined as having) a specific nal_unit_type parameter value in the range TRAIL_NUT through RSV VCL 6, or equal to GDR_NUT.

[0140] (A) If the value of the mixed nalu_types_in_pic_idc syntax element is 2, one or more VCL NAL units must all have (or may be determined to have) a specific nal_unit_type parameter value equal to RASL_NUT or RADL_NUT inclusive or equal to GDR_NUT, and all other NAL units VCLs must have (or may be defined as having) a specific nal_unit_type parameter value ranging from TRAIL_NUT to RSV VCL 6 inclusive, or equal to GDR_NUT, with one nal_unit_type value different from another nal_unit_type value.

[0141] (C) Otherwise (if the value of mixed_nalu_types_in_pic_idc is 0), the value of the nal_unit_type parameter must be (they can be defined) the same for all NAL units of the encoded slices of the picture. The picture or PU is considered to be of the same NAL unit type as the encoded picture slice or PU NAL units.

[0142] If the value of the mixed_nalu_types_in_pic_idc indicator in the PPS set referenced by the encoded picture is 1 or 2, the picture may be determined (eg, by a decoder) as a subsequent picture.

[0143] The aspects described above may provide a solution to "Problem 4" described above in the "Summary" section.

[0144] According to one or more embodiments, if the mixed nalu_types_injpic flag syntax element of the image in layer A is 1, the mixed nalu_types_in jtic flag of the image in layer B, which is the reference layer for layer A, should be (or can be defined) equal to 1 in the same AU.

[0145] The aspect described above may provide a solution to "problem 5" described above in the "Summary" section.

[0146] According to one or more embodiments, one or more encoded video bitstreams and structures and syntax elements contained therein (such as VCL NAL units and parameter sets described above) may be received by the decoders described in the present invention to decode the received video data. . The decoders described in the present invention can decode an encoded video image based on VCL NAL units (e.g., VCL NAL units (500) shown in FIG. 5) of an encoded image with mixed VCL NAL unit types in accordance with embodiments of the present invention.

[0147] For example, in accordance with FIG. 6, the decoder (600) may comprise a decoding code (610) configured such that at least one decoder processor (600) decodes the encoded picture based on the VCL NAL units. According to one or more embodiments, the decoding code (610) may include a determination code (620) configured such that at least one decoder processor (600), as described in embodiments of the present invention, (a) determines or limits the NAL unit type of one or more VCL NAL units of the encoded picture based on the NAL unit type of the other one or more VCL VCL units of the encoded picture or based on an indicator (e.g. flag), (b) determined or restricted the encoded picture type based on the NAL unit type of one or more coded picture VCL NAL units or based on an indicator (eg, flag), (c) determine or restrict the coded picture identifier TemporallD based on the NAL unit type of one or more coded picture VCL NAL units or based on an indicator (eg, flag), and/ or (d) define or limit an indicator (such as a flag) that indicates whether it whether the encoded picture has a plurality of VCL NAL units of mixed NAL unit types, based on another indicator (eg, flag) that is received or determined.

[0148] Embodiments of the present invention may be used singly or combined in any order. In addition, each of the methods, encoders, and decoders of the present invention may be implemented by a processing circuit (eg, one or more processors or one or more integrated circuits). In one example, one or more processors execute a program that is stored on a computer-readable medium.

[0149] The methods described above may be implemented in computer software using computer-readable instructions and physically stored on one or more computer-readable media. In FIG. 7 shows an example of a computer system (900) that is suitable for implementing embodiments of the present invention.

[0150] Computer software may be encoded in any suitable machine code or computer language that may be assembled, compiled, combined, or similar mechanisms to produce code containing instructions that can be executed directly or through interpretation, execution of microcode, and similar actions by central processing units (CPUs), graphics processing units (GPUs) and other similar devices.

[0151] The instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, IoT devices, and other similar devices.

[0152] Shown in FIG. 7, the components of the computer system (900) are exemplary and are not intended to suggest any limitation as to the scope or functionality of the computer software implementing embodiments of the present invention. Also, do not interpret the configuration of the components in terms of any dependencies or requirements relating to any of the components shown in the exemplary computer system (900), or combinations thereof.

[0153] The computer system (900) may include certain human-machine interface input devices. Such a human-machine interface input device may respond to data input by one or more users through, for example, tactile input (eg, keystrokes, swipes, glove movements), audio input (eg, voice, claps), visual input (in in particular gestures), olfactory input (not shown). Human interface devices may also be used to capture information from certain media not necessarily directly related to conscious human input, such as audio (in particular speech, music, environmental sounds), images (in particular scanned images, photographic images taken camera), video (in particular 2D video, 3D video including stereoscopic video).

[0154] Human interface input devices may comprise one or more (only one shown) of the following: keyboard (901), mouse (902), trackpad (903), touch screen (910), glove, joystick ( 905), microphone (906), scanner (907), camera (908).

[0155] The computer system (900) may also include certain human interface output devices. Such human interface output devices can stimulate the senses of one or more users using, for example, tactile output, sound, light, and smell/taste. Such human interface output devices may comprise tactile output devices (particularly tactile feedback via touch screen (910), data glove, or joystick (905), but tactile feedback devices that do not serve as input devices may also be used). For example, such devices may be audio output devices (particularly speakers (909), headphones (not shown)), visual output devices (particularly screens (910), including CRT screens, LCD screens, plasma screens, OLED screens, each of which may or may not have touch input capability, each of which may or may not have tactile feedback capability, some of which may be capable of providing two-dimensional visual output or more than three-dimensional output by means such as stereographic output, virtual reality goggles (not shown), holographic displays and smoke tanks (not shown), and printers (not shown).

[0156] The computer system (900) may also include user-accessible storage devices and associated media, in particular optical media, including CD/DVD ROM/RW (920) with CD/DVD or similar media (921), flash drives (922), removable hard drives or solid state drives (923), legacy magnetic media such as tapes and floppy disks (not shown), dedicated ROM/ASIC/IUiy devices such as dongles (not shown), etc. similar devices.

[0157] Persons skilled in the art should also understand that, in the context of the present invention, the term "computer-readable medium" does not encompass transmission media, electromagnetic waves, or other transient signals.

[0158] The computer system (900) may also include an interface for connecting to one or more communication networks. Networks can be, for example, wireless, wired, optical. In addition, networks can be local, global, urban, transport and industrial, operating in real time, resistant to delays, etc. Examples of networks include local area networks, such as Ethernet, wireless local area networks, cellular networks, such as GSM, 3G, 4G, 5G, LTE, etc., wired or wireless digital wide area networks, including cable television, satellite television, and terrestrial broadcast television, transport and industrial networks, including CANBus, etc. Some networks typically require external network interface adapters that connect to certain general-purpose data ports or peripheral buses (949) (for example, computer system USB ports (900); others are usually integrated into the computer system core (900) by connecting to the system bus, as described below (e.g., an Ethernet interface on a PC computer system or a cellular network interface on a smartphone computer system.) Through any of these networks, the computer system (900) can communicate with other entities. Such communication can be unidirectional for receive only (e.g., broadcast television), unidirectional transmission-only (e.g. to certain CANbus devices over the CANbus) or bidirectional, such as with other computer systems using LANs or WANs. Such interactions may include interactions with a cloud computing environment (955). For each of described by you Several networks and network interfaces may use specific protocols and protocol stacks.

[0159] The human-machine interface devices mentioned above, user-accessible storage devices, and network interfaces (954) can be connected to the core (940) of the computer system (900).

[0160] The core (940) may include one or more central processing units (CPUs) (941), graphics processing units (GPUs) (942), specialized programmable processing units in the form of programmable logic integrated circuits (FPGAs) (943), hardware accelerators for certain tasks (944), etc. These devices, along with Read Only Memory (ROM) (945), Random Access Memory (RAM) (946), internal mass storage devices such as user-inaccessible internal hard drives, solid state drives, and the like (947), can be connected via the system bus (948). In some computer systems, the system bus (948) may be available through one or more physical connectors for expansion with additional CPUs, GPUs, or the like. Peripherals can be connected either directly to the kernel system bus (948) or via a peripheral bus (949). The peripheral bus can be built on the architecture of PCI, USB, etc. Graphics adapter 950 may be included with core 940.

[0161] CPUs (941), GPUs (942), FPGAs (943), and accelerators (944) may execute certain instructions, which together may form the aforementioned computer code. This computer code may be stored in ROM (945) or RAM (946). The RAM (946) may also store temporary data, while the permanent data may be stored, for example, in the internal storage device (947). Fast storage and retrieval in any of the storage devices can be achieved through the use of cache memory, which can be closely associated with one or more CPU (941), GPU (942), storage device (947), ROM (945), RAM ( 946), etc.

[0162] A computer-readable medium may store computer code for performing various computer-implemented operations. The medium and computer code may be specially designed and constructed for the purposes of the present invention, or may be well known and available to those skilled in the art of computer software.

[0163] By way of example and not limitation, the computer system architecture (900), and in particular the core (940), may provide functionality as a result of the processor(s) (including CPU, GPU, FPGA, accelerators, etc.) execute(s) software stored on one or more physical machine-readable media. Such computer-readable media can be media associated with user-accessible mass storage devices, as described above, as well as certain physical kernel storage devices (940), such as internal kernel storage (947) or ROM (945). Software implementing various embodiments of the present invention may be stored on such devices and executed by the kernel (940). The computer-readable medium may consist of one or more storage devices or microchips, depending on specific needs. Under the control of the software, the core (940) and, in particular, its constituent processors (including CPUs, GPUs, FPGAs, etc.) may execute certain processes or parts of certain processes described in this document, including the definition of data structures, stored in RAM (946), and modification of such data structures in accordance with the processes defined by the software. In addition, or alternatively, a computer system may provide functionality through hardware or otherwise circuitry-embedded logic (e.g., an accelerator (944)) that may operate in place of or in conjunction with software to perform separate processes or portions of separate processes described in this document. Software can mean logic and vice versa when appropriate. A computer-readable medium may refer to a circuit (eg, an integrated circuit (IC)) that stores software to execute, or a circuit that implements logic to execute, or both, as appropriate. The present invention extends to any suitable combination of hardware and software.

[0164] Although several non-limiting exemplary embodiments have been described above, the scope of the invention also includes possible changes, modifications, and various equivalents. Thus, it should be understood that those skilled in the art will be able to develop numerous systems and methods that, although not expressly represented or described herein, implement the principles of the present invention and, therefore, fall within its spirit and scope.

Claims (22)

1. A decoding method performed by at least one processor, in which: a first network abstraction layer (NAL) block of a video coding layer (VCL) of a first coded picture slice and a second NAL VCL of a second coded picture slice are received, wherein the first NAL VCL is of a first type of NAL VCLs and the second NAL of a VCL is of a second type VCL NAL units, which is different from the first VCL NAL unit type; and decoding the encoded image, wherein the decoding comprises determining the type of the encoded image based on the first type of the VCL NAL unit and the second type of the VCL NAL unit, or based on an indicator received by at least one processor and indicating that the encoded image contains mixed types of VCL NAL units, wherein the determination comprises determining that the encoded picture is a trailing picture, based on the first VCL NAL unit type indicating that the first VCL NAL unit contains a coded Gradual Decoding Update (GDR) picture slice, and the second VCL NAL unit type indicating that the second VCL NAL unit contains an instantaneous decoding update (IDR) encoded picture slice or a pure random access (CRA) encoded picture slice. 2. The method according to claim 1, in which the indicator is a flag, and said definition comprises determining that the encoded picture is a subsequent picture based on a flag that indicates that the encoded picture contains mixed type VCL NAL units. 3. The method according to claim 1, in which the indicator is a flag, and decoding the encoded picture also comprises determining that the TemporalID of the encoded picture is 0 based on a flag that indicates that the encoded picture contains mixed type VCL NAL units. 4. The method according to claim 1, in which the indicator is a flag, and the method also comprises receiving a flag in the image header or slice header. 5. The method according to claim 1, in which the indicator is a flag, and the encoded image is in the first level, wherein the method also contains receiving the flag; and determining that the additional encoded picture residing in the second layer, which is the reference layer of the first layer, contains mixed type VCL NAL units, based on a flag that indicates that the encoded picture contains mixed type VCL NAL units. 6. A decoding system, comprising: a memory configured to store computer program code; and at least one processor configured to receive at least one encoded video stream, access the computer program code, and perform operations in accordance with instructions from the computer program code to implement the method of any one of claims. 1-5. 7. A computer-readable medium that stores computer instructions, when executed by at least one processor, at least one processor performs the method according to any one of paragraphs. 1-5.

Images