[go: up one dir, main page]

WO2019129923A1 - Appareil, procédé et programme informatique pour vidéo volumétrique - Google Patents

Appareil, procédé et programme informatique pour vidéo volumétrique Download PDF

Info

Publication number
WO2019129923A1
WO2019129923A1 PCT/FI2018/050948 FI2018050948W WO2019129923A1 WO 2019129923 A1 WO2019129923 A1 WO 2019129923A1 FI 2018050948 W FI2018050948 W FI 2018050948W WO 2019129923 A1 WO2019129923 A1 WO 2019129923A1
Authority
WO
WIPO (PCT)
Prior art keywords
patches
stack
patch
projection plane
bitstream
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/FI2018/050948
Other languages
English (en)
Inventor
Payman AFLAKI
Sebastian Schwarz
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Nokia Technologies Oy
Original Assignee
Nokia Technologies Oy
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Nokia Technologies Oy filed Critical Nokia Technologies Oy
Publication of WO2019129923A1 publication Critical patent/WO2019129923A1/fr
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/282—Image signal generators for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T9/00—Image coding
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
    • H04N13/194—Transmission of image signals
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/204—Image signal generators using stereoscopic image cameras
    • H04N13/243—Image signal generators using stereoscopic image cameras using three or more 2D image sensors
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/271—Image signal generators wherein the generated image signals comprise depth maps or disparity maps
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
    • H04N19/132—Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
    • H04N19/136—Incoming video signal characteristics or properties
    • H04N19/14—Coding unit complexity, e.g. amount of activity or edge presence estimation
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
    • H04N19/17—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/30—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using hierarchical techniques, e.g. scalability
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/46—Embedding additional information in the video signal during the compression process
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
    • H04N19/597—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding specially adapted for multi-view video sequence encoding

Definitions

  • the present invention relates to an apparatus, a method and a computer program for volumetric video coding and decoding.
  • a video coding system may comprise an encoder that transforms an input video into a compressed representation suited for storage/transmission and a decoder that can uncompress the compressed video representation back into a viewable form.
  • the encoder may discard some information in the original video sequence in order to represent the video in a more compact form, for example, to enable the storage/transmission of the video information at a lower bitrate than otherwise might be needed.
  • Volumetric video data represents a three-dimensional scene or object and can be used as input for virtual reality (VR), augmented reality (AR) and mixed reality (MR) applications.
  • Such data describes the geometry attribute, e.g. shape, size, position in three-dimensional (3D) space, and respective attributes, e.g. colour, opacity, reflectance and any possible temporal changes of the geometry attribute and other attributes at given time instances.
  • Volumetric video is either generated from 3D models through computer-generated imagery (CGI), or captured from real-world scenes using a variety of capture solutions, e.g. multi-camera, laser scan, combination of video and dedicated depth sensors, and more. Also, a combination of CGI and real-world data is possible.
  • CGI computer-generated imagery
  • Typical representation formats for such volumetric data are triangle meshes, point clouds (PCs), or voxel arrays.
  • Temporal information about the scene can be included in the form of individual capture instances, i.e.“frames” in 2D video, or other means, e.g. position of an object as a function of time.
  • volumetric video describes a 3D scene (or object), such data can be viewed from any viewpoint. Therefore, volumetric video may be an important format for any AR, VR, or MR applications, especially for providing 6DOF viewing capabilities.
  • volumetric video representation formats suffer from poor spatial and temporal coding performance.
  • a volumetric video may comprise three-dimensional scenes represented as, for example, dynamic point clouds, arrays of voxels or mesh models or a combination of such.
  • the three- dimensional scenes may be projected onto a number of projection surfaces having simple geometries, for example sphere(s), cylinder(s), cube(s), polyhedron(s) and/or plane(s).
  • a projection surface may be a piece-wise continuous and smooth surface in three-dimensional space. Piece-wise smoothness may be understood so that there are regions of the surface where the direction of the surface normal does not change abruptly (i.e. the values of the coefficients of the surface normal’s coordinate components are continuous).
  • a projection surface may comprise pieces of simple geometric surfaces.
  • a projection surface may also evolve (change) over time.
  • the texture and geometry of point clouds, voxel arrays or mesh models may form pixel images, e.g. texture images and depth images (indicative of distance from the projection surface). These two images represent the same object projected onto the same geometry, therefore object boundaries are aligned in texture and depth image.
  • Such projection surfaces may be unfolded onto two-dimensional (2D) planes, e.g. resulting in a two-dimensional pixel image.
  • Standard 2D video coding may be applied for each projection to code the pixel information resulting from the texture data.
  • relevant projection geometry information comprising e.g. projection or projection surface type, location and orientation of the projection surface in 3D space, and/or size of the projection surface, may be transmitted either in the same bitstream or separately along with the bitstream.
  • the bitstream may be decoded and volumetric video may be reconstructed from decoded 2D projections and projection geometry information.
  • Two-dimensional images may be projected from different parts of scene objects to form several patches.
  • patches may be projections onto one of three orthogonal planes (front, side, top).
  • Patches may be derived by analysing surface normal and clustering related 3D data points.
  • the projection plane for each such patch may be the one of the above mentioned three planes with the closest surface normal to the average patch normal. All patches may be packed into a 2D grid for compression.
  • For each patch a 3D vector is signalled to specify the patch location in 3D space for reprojection at the decoder side.
  • a central point for the projection is the centre of a projected patch which is the same distance from sides of the patch (when the patch is of a rectangular shape) i.e. at a location of half height and half width of the patch. This is shown in Figures 9a and 9b where the centre of patches are marked with black circles.
  • the patch-to-2D grid mapping does not consider occlusions, as each patch has its individual 3D vector. However, by allowing occlusions in the patch-to-2D grip mapping to happen, one may reduce signalling at no cost in reconstruction quality. Furthermore, new, more content- adaptive compression features may be enabled.
  • multiple occluding projections are signalled with the same projection plane coordinates.
  • Overlapping (occluding) patches may be presented with aligned X and Y coordinates. Additional projection resolution adaption can be performed to put emphasis on more important regions of a 3D object, and/or on more important 3D objects in a 3D scene. Signalling only one set of aligned X and Y coordinates per stack of overlapping patches reduces signalling cost without impact on reconstruction quality. Assigning higher grid resolution to important patches allows for flexible region-of-interest and/or viewport dependent compression.
  • Some embodiments provide a method for encoding and decoding volumetric video information.
  • a method, apparatus and computer program product for volumetric video coding as well as decoding is provided.
  • each patch representing projection data of at least a part of an object to a projection plane
  • An apparatus comprises at least one processor and at least one memory, said at least one memory stored with code thereon, which when executed by said at least one processor, causes the apparatus to perform at least:
  • each patch representing projection data of at least a part of an object to a projection plane; identify overlapping patches from the two or more patches;
  • a computer readable storage medium comprises code for use by an apparatus, which when executed by a processor, causes the apparatus to perform:
  • each patch representing projection data of at least a part of an object to a projection plane; identify overlapping patches from the two or more patches;
  • An apparatus comprises:
  • each patch representing projection data of at least a part of an object to a projection plane
  • each patch representing projection data of at least a part of an object to a projection plane
  • An apparatus comprises at least one processor and at least one memory, said at least one memory stored with code thereon, which when executed by said at least one processor, causes the apparatus to perform at least:
  • each patch representing projection data of at least a part of an object to a projection plane
  • decode from the bitstream information identifying a stack of overlapping patches extract from the bitstream at least one projection plane coordinate associated with the stack of overlapping patches;
  • a computer readable storage medium comprises code for use by an apparatus, which when executed by a processor, causes the apparatus to perform:
  • each patch representing projection data of at least a part of an object to a projection plane
  • decode from the bitstream information identifying a stack of overlapping patches extract from the bitstream at least one projection plane coordinate associated with the stack of overlapping patches;
  • An apparatus comprises:
  • FIG. 1 shows an example of a system for capturing, encoding, decoding, reconstructing and viewing a three-dimensional scheme
  • FIGS. 2a and 2b show examples of a capture device and a viewing device
  • FIGs. 3a and 3b show examples of an encoder and decoder for encoding and decoding
  • Figs. 4a, 4b, 4c and 4d show an example of a setup for forming a stereo image of a scene to a user
  • Fig. 5 illustrates an example of projection of source volumes in a scene and parts of an object to projection surfaces, as well as determining depth information
  • FIGs. 6a and 6b show an example of flow charts for compressing, encoding and decoding of a scene of a volumetric video
  • Fig. 7a illustrates an example of an encoding element
  • Fig. 7b illustrates an example of a decoding element
  • Fig. 8a illustrates an example of different target grid resolutions for close and far objects
  • Fig. 8b illustrates an example of even allocation of grid resolution to different objects
  • Fig. 9a illustrates an example of aligning three arbitrary patches onto some central x,y coordinate
  • Fig. 9b illustrates an example of having more than one aligned stacked patches for an x, y projection plane
  • Fig. 9c illustrates an example of padding one or more of the patches in one or more
  • Point clouds are commonly used data structures for storing volumetric content. Compared to point clouds, sparse voxel octrees describe a recursive subdivision of a finite volume with solid voxels of varying sizes, while point clouds describe an unorganized set of separate points limited only by the precision of the used coordinate values.
  • a volumetric video frame is a sparse voxel octree or a point cloud that models the world at a specific point in time, similar to a frame in a 2D video sequence. Voxel or point attributes contain information like colour, opacity, surface normal vectors, and surface material properties. These are referenced in the sparse voxel octrees (e.g. colour of a solid voxel) or point clouds, but can also be stored separately.
  • each frame may produce several hundred megabytes or several gigabytes of voxel data which needs to be converted to a format that can be streamed to the viewer, and rendered in real-time.
  • the amount of data depends on the world complexity and the number of cameras. The larger impact comes in a multi-device recording setup with a number of separate locations where the cameras are recording. Such a setup produces more information than a camera at a single location.
  • Fig. 1 shows an example of a system for capturing, encoding, decoding, reconstructing and viewing a three-dimensional scheme, that is, for 3D video and/or 3D audio digital creation and playback.
  • the task of the system is that of capturing sufficient visual and/or auditory information from a specific scene to be able to create a scene model such that a convincing reproduction of the experience, or presence, of being in that location can be achieved by one or more viewers physically located in different locations and optionally at a time later in the future.
  • Such reproduction requires more information that can be captured by a single camera or microphone, in order that a viewer can determine the distance and location of objects within the scene using their eyes and/or their ears.
  • two camera sources may be used.
  • at least two microphones may be used (the commonly known stereo sound is created by recording two audio channels).
  • the human auditory system can detect the cues, e.g. in timing difference of the audio signals to detect the direction of sound.
  • the system of Fig. 1 may comprise three main parts: image sources, a server and a rendering device.
  • a video source SRC1 may comprise multiple cameras CAM1, CAM2, ..., CAMN with overlapping field of view so that regions of the view around the video capture device is captured from at least two cameras.
  • the video source SRC1 may comprise multiple microphones to capture the timing and phase differences of audio originating from different directions.
  • the video source SRC1 may comprise a high-resolution orientation sensor so that the orientation (direction of view) of the plurality of cameras CAM1, CAM2, ..., CAMN can be detected and recorded.
  • the cameras or the computers may also comprise or be functionally connected to means for forming distance information corresponding to the captured images, for example so that the pixels have corresponding depth data.
  • Such depth data may be formed by scanning the depth or it may be computed from the different images captured by the cameras.
  • the video source SRC1 comprises or is functionally connected to, or each of the plurality of cameras CAM1, CAM2, ..., CAMN comprises or is functionally connected to a computer processor and memory, the memory comprising computer program code configured to control the source and/or the plurality of cameras.
  • the image stream captured by the video source i.e. the plurality of the cameras, may be stored on a memory device for use in another device, e.g. a viewer, and/or transmitted to a server using a communication interface. It needs to be understood that although a video source comprising three cameras is described here as part of the system, another amount of camera devices may be used instead as part of the system.
  • one or more sources SRC2 of synthetic imagery may be present in the system, comprising a scene model.
  • Such sources may be used to create and transmit the scene model and its development over time, e.g. instantaneous states of the model.
  • the model can be created or provided by the source SRC1 and/or SRC2, or by the server SERVER.
  • Such sources may also use the model of the scene to compute various video bitstreams for transmission. Therefore, in some embodiments the system may not comprise any cameras.
  • One or more two-dimensional video bitstreams may be computed at the server SERVER or a device RENDERER used for rendering, or another device at the receiving end.
  • the devices SRC1 and SRC2 may comprise or be functionally connected to one or more computer processors (PROC2 shown) and memory (MEM2 shown), the memory comprising computer program (PROGR2 shown) code configured to control the source device SRC1/SRC2.
  • PROC2 computer processors
  • MEM2 memory
  • PROGR2 shown computer program code configured to control the source device SRC1/SRC2.
  • the image stream captured by the device and the scene model may be stored on a memory device for use in another device, e.g.
  • a viewer or transmitted to a server or the viewer using a communication interface (e.g. COMM2).
  • a communication interface e.g. COMM2
  • a server SERVER or a plurality of servers storing the output from the capture device SRC1 or device SRC2 and/or to form a scene model from the data from devices SRC1, SRC2.
  • the device SERVER comprises or is functionally connected to a computer processor PROC3 and memory MEM3, the memory comprising computer program PROGR3 code configured to control the server.
  • the device SERVER may be connected by a wired or wireless network connection, or both, to sources SRC1 and/or SRC2, as well as the viewer devices VIEWER1 and VIEWER2 over the communication interface COMM3.
  • the creation of a three-dimensional scene model may take place at the server SERVER or another device by using the images captured by the devices SRC1.
  • the scene model may be a model created from captured image data (a real-world model), or a synthetic model such as on device SRC2, or a combination of such.
  • the scene model may be encoded to reduce its size and transmitted to a decoder, for example viewer devices.
  • V1EWER1 and V1EWER2 For viewing the captured or created video content, there may be one or more viewer devices V1EWER1 and V1EWER2. These devices may have a rendering module and a display module, or these functionalities may be combined in a single device.
  • the devices may comprise or be functionally connected to a computer processor PROC4 and memory MEM4, the memory comprising computer program PROG4 code configured to control the viewing devices.
  • the viewer (playback) devices may comprise a data stream receiver for receiving a video data stream and for decoding the video data stream.
  • the video data stream may be received from the server SERVER or from some other entity, such as a proxy server, an edge server of a content delivery network, or a file available locally in the viewer device.
  • the data stream may be received over a network connection through communications interface COMM4, or from a memory device MEM6 like a memory card CARD2.
  • the viewer devices may have a graphics processing unit for processing of the data to a suitable format for viewing.
  • the viewer VIEWER1 may comprise a high-resolution stereo-image head-mounted display for viewing the rendered stereo video sequence.
  • the head-mounted display may have an orientation sensor DET1 and stereo audio headphones.
  • the viewer VIEWER2 may comprise a display (either two-dimensional or a display enabled with 3D technology for displaying stereo video), and the rendering device may have an orientation detector DET2 connected to it.
  • the viewer VIEWER2 may comprise a 2D display, since the volumetric video rendering can be done in 2D by rendering the viewpoint from a single eye instead of a stereo eye pair.
  • Fig. 1 depicts one SRC1 device and one SRC2 device, but generally the system may comprise more than one SRC1 device and/or SRC2 device.
  • Any of the devices may be a computer or a portable computing device, or be connected to such or configured to be connected to such. Moreover, even if the devices (SRC1, SRC2, SERVER, RENDERER, VIEWER1,
  • VIEWER2 are depicted as a single device in Fig. 1 , they may comprise multiple parts or may be comprised of multiple connected devices.
  • SERVER may comprise several devices, some of which may be used for editing the content produced by SRC1 and/or SRC2 devices, some others for compressing the edited content, and a third set of devices may be used for transmitting the compressed content.
  • Such devices may have computer program code configured to cause the device carry out methods according to various examples described in this text.
  • Processors PROC1, PROC2, PROC3, and PROC4 may comprise processor circuitry.
  • Processors, memories, and/or other structural elements illustrates in Fig. 1 represent means for performing one or more functions as described in the appended claims and throughout the specification.
  • Figs. 2a and 2b show examples of a capture device and a viewing device, respectively.
  • Fig. 2a illustrates a camera CAM1.
  • the camera has a camera detector CAMDET1, comprising a plurality of sensor elements for sensing intensity of the light hitting the sensor element.
  • the camera has a lens OBJ 1 (or a lens arrangement of a plurality of lenses), the lens being positioned so that the light hitting the sensor elements travels through the lens to the sensor elements.
  • the camera detector CAMDET1 has a nominal centre point CP1 that is a middle point of the plurality of sensor elements, for example for a rectangular sensor the crossing point of diagonals of the rectangular sensor.
  • the lens has a nominal centre point PP1, as well, lying for example on the axis of symmetry of the lens.
  • the direction of orientation of the camera is defined by the line passing through the centre point CP1 of the camera sensor and the centre point PP1 of the lens.
  • the direction of the camera is a vector along this line pointing in the direction from the camera sensor to the lens.
  • the optical axis of the camera is understood to be this line CP1-PP1.
  • the optical path from the lens to the camera detector need not always be a straight line but there may be mirrors and/or some other elements which may affect the optical path between the lens and the camera detector.
  • Fig. 2b shows a head-mounted display (HMD) for stereo viewing.
  • the head-mounted display comprises two screen sections or two screens DISP1 and DISP2 for displaying the left and right eye images.
  • the displays are close to the eyes, and therefore lenses are used to make the images easily viewable and for spreading the images to cover as much as possible of the eyes' field of view.
  • the user may put the device on her/his head so that it will be attached to the head of the user so that it stays in place even when the user turns his head.
  • the device may have an orientation detecting module ORDET1 for determining the head movements and direction of the head.
  • the head-mounted display gives a three-dimensional (3D) perception of the recorded/streamed content to a user.
  • Time-synchronized video and orientation data is first recorded with the capture devices. This can include multiple concurrent video streams as described above.
  • One or more time-synchronized audio streams may also be recorded with the capture devices.
  • the different capture devices may form image and geometry information of the scene from different directions. For example, there may be three, four, five, six or more cameras capturing the scene from different sides, like front, back, left and right, and/or at directions between these, as well as from the top or bottom, or any combination of these.
  • the cameras may be at different distances, for example some of the cameras may capture the whole scene and some of the cameras may be capturing one or more objects in the scene.
  • the cameras or the system may comprise means for determining geometry information, e.g. depth data, related to the captured video streams. From these concurrent video and audio streams, a computer model of a scene may be created. Alternatively or additionally, a synthetic computer model of a virtual scene may be used. The models (at successive time instances) are then transmitted immediately or later to the storage and processing network for processing and conversion into a format suitable for subsequent delivery to playback devices.
  • geometry information e.g. depth data
  • the conversion may involve processing and coding to improve the quality and/or reduce the quantity of the scene model data while preserving the quality at a desired level.
  • Each playback device receives a stream of the data (either computed video data or scene model data) from the network, and renders it into a viewing reproduction of the original location which can be experienced by a user.
  • the reproduction may be two-dimensional or three- dimensional (stereo image pairs).
  • Figs. 3a and 3b show examples of an encoder and decoder for encoding and decoding texture pictures, geometry pictures and/or auxiliary pictures.
  • a video codec may comprise an encoder that transforms an input video into a compressed representation suited for storage/transmission and a decoder that can uncompress the compressed video representation back into a viewable form.
  • Figure 3a illustrates an image to be encoded (I n ); a predicted representation of an image block (P 3 ⁇ 4 ); a prediction error signal (D n ); a reconstructed prediction error signal (D 3 ⁇ 4 ); a preliminary reconstructed image (I n ); a final reconstructed image (R 3 ⁇ 4 ); a transform (T) and inverse transform (T 1 ); a quantization (Q) and inverse quantization (Q 1 ); entropy encoding (E); a reference frame memory (RFM); inter prediction (P mter ); intra prediction (P mtra ); mode selection (MS) and filtering (F).
  • Figure 3b illustrates a predicted representation of an image block (P 3 ⁇ 4 ); a reconstructed prediction error signal ( D din ) ; a preliminary reconstructed image (F n ); a final reconstructed image (R 3 ⁇ 4 ); an inverse transform (T 1 ); an inverse quantization (Q 1 ); an entropy decoding (E 1 ); a reference frame memory (RFM); a prediction (either inter or intra) (P); and filtering (F).
  • P 3 ⁇ 4 a predicted representation of an image block (P 3 ⁇ 4 ); a reconstructed prediction error signal ( D din ) ; a preliminary reconstructed image (F n ); a final reconstructed image (R 3 ⁇ 4 ); an inverse transform (T 1 ); an inverse quantization (Q 1 ); an entropy decoding (E 1 ); a reference frame memory (RFM); a prediction (either inter or intra) (P); and filtering (F).
  • Figs. 4a, 4b, 4c and 4d show examples of a setup for forming a stereo image of a scene to a user, for example a video frame of a 3D video.
  • Fig. 4a a situation is shown where a human being is viewing two spheres Al and A2 using both eyes El and E2.
  • the sphere Al is closer to the viewer than the sphere A2, the respective distances to the first eye El being LEI,AI and LEI,A2 ⁇
  • the different objects reside in space at their respective (x,y,z) coordinates, defined by the coordinate system SX, SY and SZ.
  • the distance di 2 between the eyes of a human being may be approximately 62-64 mm on average, and varying from person to person between 55 and 74 mm. This distance is referred to as the parallax, on which stereoscopic view of the human vision is based on.
  • the viewing directions (optical axes) DIR1 and DIR2 are typically essentially parallel, possibly having a small deviation from being parallel, and define the field of view for the eyes.
  • the head of the user has an orientation (head orientation) in relation to the surroundings, most easily defined by the common direction of the eyes when the eyes are looking straight ahead. That is, the head orientation tells the yaw, pitch and roll of the head in respect of a coordinate system of the scene where the user is.
  • Fig. 4a In the setup of Fig. 4a, the spheres Al and A2 are in the field of view of both eyes. The centre-point O12 between the eyes and the spheres are on the same line. That is, from the centre-point, the sphere A2 is behind the sphere Al. However, each eye sees part of sphere A2 from behind Al, because the spheres are not on the same line of view from either of the eyes. [0063] In Fig. 4b, there is a setup shown, where the eyes have been replaced by cameras Cl and
  • the distances and directions of the setup are otherwise the same.
  • the purpose of the setup of Fig. 4b is to be able to take a stereo image of the spheres Al and A2.
  • the two images resulting from image capture are Fci and Fc2.
  • the "left eye” image Fci shows the image S A 2 of the sphere A2 partly visible on the left side of the image SAI of the sphere Al.
  • the "right eye” image Fc2 shows the image SA2 of the sphere A2 partly visible on the right side of the image SAI of the sphere Al .
  • This difference between the right and left images is called disparity, and this disparity, being the basic mechanism with which the human visual system (HVS) determines depth information and creates a 3D view of the scene, can be used to create an illusion of a 3D image.
  • HVS human visual system
  • the camera pair Cl and C2 has a natural parallax, that is, it has the property of creating natural disparity in the two images of the cameras. Natural disparity may be understood to be created even though the distance between the two cameras forming the stereo camera pair is somewhat smaller or larger than the normal distance (parallax) between the human eyes, e.g. essentially between 40 mm and 100 mm or even 30 mm and 120 mm.
  • the images Fci and Fc2 may be captured by cameras Cl and C2, where the cameras Cl and C2 may be real-world cameras or they may be virtual cameras.
  • the images Fci and Fc2 may be computed from a computer model of a scene by setting the direction, orientation and viewport of the cameras Cl and C2 appropriately such that a stereo image pair suitable for viewing by the human visual system (HVS) is created.
  • HVS human visual system
  • Fig. 4c the creating of this 3D illusion is shown.
  • the images Fci and Fc2 captured or computed by the cameras Cl and C2 are displayed to the eyes El and E2, using displays Dl and D2, respectively.
  • the disparity between the images is processed by the human visual system so that an understanding of depth is created. That is, when the left eye sees the image S A 2 of the sphere A2 on the left side of the image SAI of sphere Al, and respectively the right eye sees the image SA2 of the sphere A2 on the right side, the human visual system creates an understanding that there is a sphere V2 behind the sphere VI in a three-dimensional world.
  • the images Fci and F C 2 can also be synthetic, that is, created by a computer. If they carry the disparity information, synthetic images will also be seen as three-dimensional by the human visual system. That is, a pair of computer-generated images can be formed so that they can be used as a stereo image.
  • Fig. 4d illustrates how the principle of displaying stereo images to the eyes can be used to create 3D movies or virtual reality scenes having an illusion of being three-dimensional.
  • the images Fxi and Fx2 are either captured with a stereo camera or computed from a model so that the images have the appropriate disparity.
  • a large number e.g. 30
  • the field of view represented by the content may be greater than the displayed field of view e.g. in an arrangement depicted in Fig. 4d.
  • This direction of view that is, the head orientation
  • This direction of view may be determined as a real orientation of the head e.g. by an orientation detector mounted on the head, or as a virtual orientation determined by a control device such as a joystick or mouse that can be used to manipulate the direction of view without the user actually moving his head.
  • a control device such as a joystick or mouse that can be used to manipulate the direction of view without the user actually moving his head.
  • the term "head orientation" may be used to refer to the actual, physical orientation of the user's head and changes in the same, or it may be used to refer to the virtual direction of the user's view that is determined by a computer program or a computer input device.
  • the content may enable viewing from several viewing positions within the 3D space.
  • the texture picture(s), the geometry picture(s) and the geometry information may be used to synthesize the images Fxi and/or Fx2 as if the displayed content was captured by camera(s) located at the viewing position.
  • volumetric video describes a 3D scene or object at different (successive) time instances, such data can be viewed from any viewpoint. Therefore, volumetric video is an important format for any augmented reality, virtual reality and mixed reality applications, especially for providing viewing capabilities having six degrees of freedom (so-called 6DOF viewing).
  • FIG. 5 illustrates projection of source volumes in a digital scene model SCE and parts of an object model OBJ1, OBJ2, OBJ3, BG4 to projection surfaces Sl, S2, S3, S4, as well as determining depth information for the purpose of encoding volumetric video.
  • the projection of source volumes SV1, SV2, SV3, SV4 may result in texture pictures and geometry pictures, and there may be geometry information related to the projection source volumes and/or projection surfaces.
  • Texture pictures, geometry pictures and projection geometry information may be encoded into a bitstream.
  • a texture picture may comprise information on the colour data of the source of the projection. Through the projection, such colour data may result in pixel colour information in the texture picture. Pixels may be coded in groups, e.g. coding units of rectangular shape.
  • the projection geometry information may comprise but is not limited to one or more of the following:
  • a projection centre such as a projection centre point, axis, or plane location and/or orientation of a projection centre.
  • the projection may take place by projecting the geometry primitives (points of a point could, triangles of a triangle mesh or voxels of a voxel array) of a source volume SV1, SV2, SV3, SV4 (or an object OBJ1, OBJ2, OBJ3, BG4) onto a projection surface Sl, S2, S3, S4.
  • the geometry primitives may comprise information on the texture, for example a colour value or values of a point, a triangle or a voxel.
  • the projection surface may surround the source volume at least partially such that projection of the geometry primitives happens from the centre of the projection surface outwards to the surface.
  • a cylindrical surface has a centre axis and a spherical surface has a centre point.
  • a cubical or rectangular surface may have centre planes or a centre axis and the projection of the geometry primitives may take place either orthogonally to the sides of the surface or from the centre axis outwards to the surface.
  • the projection surfaces e.g. cylindrical and rectangular, may be open from the top and the bottom such that when the surface is cut and rolled out on a two- dimensional plane, it forms a rectangular shape.
  • projection surfaces need not be rectangular but may be arranged or located spatially on a rectangular picture. Such rectangular shape with pixel data can be encoded and decoded with a video codec.
  • the projection surface such as a planar surface or a sphere may be inside a group of geometry primitives, e.g. inside a point cloud that defines a surface.
  • the projection may take place from outside in towards the centre and may result in sub-sampling of the texture data of the source.
  • points may be represented with any floating point coordinates.
  • a quantized point cloud may be used to reduce the amount of data, whereby the coordinate values of the point cloud are represented e.g. with lO-bit, 12-bit or 16-bit integers. Integers may be used because hardware accelerators may be able to operate on integers more efficiently.
  • the points in the point cloud may have associated colour, reflectance, opacity etc. texture values.
  • the points in the point cloud may also have a size, or a size may be the same for all points.
  • the size of the points may be understood as indicating how large an object the point appears to be in the model in the projection.
  • the point cloud is projected by ray casting from the projection surface to find out the pixel values of the projection surface. In such a manner, the topmost point remains visible in the projection, while points closer to the centre of the projection surface may be occluded.
  • the original point cloud, meshes, voxels, or any other model is projected outwards to a simple geometrical shape, this simple geometrical shape being the projection surface.
  • Different projection surfaces may have different characteristics in terms of projection and reconstruction. In the sense of computational complexity, a projection to a cubical surface may be the most efficient, and a cylindrical projection surface may provide accurate results efficiently. Also cones, polyhedron-based parallelepipeds (hexagonal or octagonal, for example) and spheres or a simple plane may be used as projection surfaces. [0077]
  • the phrase along with the bitstream may be defined to refer to out-of-band transmission, signalling, or storage in a manner that the out-of-band data is associated with the bitstream.
  • decoding along with the bitstream or alike may refer to decoding the referred out-of-band data (which may be obtained from out-of-band transmission, signalling, or storage) that is associated with the bitstream.
  • an indication along with the bitstream may refer to metadata in a container file that encapsulates the bitstream.
  • Figure 8a illustrates an example of different target grid resolutions for close and far objects
  • Figure 8b illustrates an example of even allocation of grid resolution to different objects
  • Figure 9a illustrates an example of aligning three arbitrary patches onto some x,y coordinate
  • Figure 9b illustrates an example of having more than one aligned stacked patches for an x, y projection plane.
  • a grid is always referring to a two-dimensional grid on which the patches are located.
  • step 610 three-dimensional content is obtained and decomposed to several patches by a decomposer 701.
  • step 620 overlapping, i.e. occluding patches are identified and combined in a stack for further processing by a patch organizer 702.
  • the central point of the patches may or may not be the same.
  • step 630 the target grid resolution, or the target grid resolution relation between patches, is derived by a grid constructor 703 for each patch in a stack of overlapping patches based on one or more of the following criteria.
  • distance information between patches and between patches and projection plane is utilized, for example, so that more distant patches are assigned a lower grid resolution.
  • Resolution assigned to a patch may therefore be dependent on the distance between the patch and the projection plane.
  • the resolution is inversely proportional to the distance between the patch and the projection plane.
  • Figure 8a An example of this is illustrated in Figure 8a.
  • the farther distance part of the scene is allocated less number of pixels in the grid on the projection plane, while the closer part of the scene is allocated higher number of pixels in the grid.
  • the resolution ratio between far and close objects has been selected so that the closer object takes 75% of the grid resolution while the farther object takes 25% of the grid resolution.
  • such ratio selection may be accomplished based on, for example, a rate distortion optimization (RDO) algorithm or alternatively based on any subjective quality assessment metric to maximize the subjective quality of perceived patches.
  • RDO rate distortion optimization
  • object classifier is utilized to identify important regions, e.g. faces, hands, etc., and a higher grid resolution is assigned to the identified important regions.
  • saliency information is utilized to identify important regions, and assign them a higher grid resolution.
  • motion information is utilized to identify temporal active regions, and assign them a higher grid resolution (and/or a lower resolution to more static regions).
  • manual classifying information is taken into account, e.g. if an 3D object is marked as high interest then a higher grid resolution is assigned (and vice-versa).
  • illumination is taken into account, i.e. a brighter object is considered more important and assigned with a higher grid resolution
  • texture and/or geometry details are taken into account, i.e. a highly textured object is considered more important and assigned with a higher grid resolution.
  • the texture information may be achieved by using a DCT (discrete cosine transform) of the projected content and calculating the amount of HFCs (high frequency components) in that patch. The higher the amount of high frequency components, the higher the resolution to be allocated to that patch compared to other patches.
  • DCT discrete cosine transform
  • the (resolution adapted) patches of each stack are aligned by a patch aligner 704 to share at least one coordinate, e.g. a line in 3D space going through a point of all patches in the stack (orthogonal to the respective projection plane) ln
  • the shared coordinate comprises an X coordinate and an Y coordinate.
  • the shared coordinate may correspond to any location at the overlapping patches such as for example comers or central points. For simplicity, most examples have been described using central point as an example but it is to be understood that other shared coordinate locations may be used as well. An example illustration is depicted in Figure 9a.
  • the patches 90 la, 90 lb, 90 lc are of different size and they differ from the Z coordinate point of view ft should be noted that the patches in a stack may be located on either side of the coordinate plane, depending on the location of the projection planes 900. ln this embodiment, the stacked patches may all have the same central point 902. ln this embodiment, the stack can be presented with the same central X and Y coordinate. [0092] In another embodiment, the stacked patches may not all have the same central point. In this embodiment, one or more of the patches may be padded in one or more directions to align the central points to the same central X and Y coordinates.
  • the central X and Y coordinates may be defined so that the amount of padding in one or more directions of one or more of the patches is to be minimized.
  • the patches may be padded such that they share some other coordinate, for example one of the comers of a rectangle.
  • Figure 9c illustrates an example of padding with shared central point.
  • the minimization may be to minimize the added surface of patches due to the padding process.
  • the minimization may be to minimize the amount (area) of vertically padded patches.
  • the minimization may be to minimize the amount (area) of horizontally padded patches. In another embodiment, the minimization is performed per patch combining the previous embodiment e.g. minimizing the surface area of one patch, minimizing the amount of vertically padded area of another patch, and minimizing the amount of horizontally padded area of another patch.
  • more than one aligned stacked patches per projection plane may be defined. This may increase the freedom to select the physical location of patches in the coordinate system to better cover/project the scene objects.
  • An example of this is depicted in Figure 9b.
  • the patches 90 la— 90 li are aligned to three separate stacks each having its own central point 902a, 902b, 902b on the projection plane 900.
  • the patches are aligned before the resolution adaption i.e. the sequence of steps would be step 620, then step 640, then step 630, and then step 650.
  • the sequence of steps would be step 620, then step 640, then step 630, and then step 650.
  • it is selected what are the grid resolution ratios assigned to each patch.
  • the aligned stacked patches may belong to X,Y or Y,Z, or X,Z projection planes. All patches of a stack may belong to the same projection plane.
  • step 650 all patches are packed on the same two-dimensional grid by a patch packer 705.
  • patches belonging to the same stack may be allocated to the two- dimensional grid such that they are interlaced in a predefined or signalled manner around the location defined by the shared coordinate signalled for the stack.
  • a patch could either belong to a stack or not.
  • Each stack may have a stack ID, which may be also referred to as stack index or stack identifier. If the patch belongs to a stack, only its stack ID may be signalled. If it doesn't belong to a stack, its projection plane and X, Y and Z coordinates in 3D space are signalled.
  • each patch can have a resolution for the Z-direction defined by the bit depth of the geometry picture.
  • an optional Z-coordinate offset is signalled per stack.
  • the projection plane is also defined and used for all patched that belong to this stack. If the patches do not belong to a stack, the projection plane may be defined separately for each patch.
  • Each patch in the stack is defined with a patch ID.
  • Signalling information transmitted along with a video bitstream may comprise a signalling table or other data structure indicating a mapping between a stack ID and a plurality of patch IDs.
  • a binary flag 0, 1 in the signalling information may define whether or not the current patch belongs to a stack. For example, if the flag is 0, then the patch does not belong to a stack and if the flag is 1 the patch belongs to a stack and hence the stack index is signalled. Alternatively, an additional stack ID might be assigned to the“no stack” flag.
  • the codeword for the stack IDs may be optimised for entropy encoding, i.e. stack IDs with high frequency may be represented by a shorter codeword.
  • a patch belongs to a stack and the resolution adaption was performed, this can be signalled either individually on patch level or as a delta between patches.
  • Resolution of a patch may be signalled as a function of depth (e.g. distance to the projection plane as discussed above), as a function of stack level (e.g. an order of the patch in the stack of patches), or as a ratio between the resolution of patches belonging to the same stack.
  • the signalling information may include a common (shared) X and Y coordinate between patches in this stack.
  • the shared coordinate(s) may be associated with one of the patches associated with the stack ID.
  • the decoder can recreate the full set of 3D coordinates from the patch ID, the pixels in the patch, and the patch resolution adaption.
  • the decoder may use signalling information associated with a patch in combination with signalling information associated with a stack to recreate the set of 3D coordinates for the patch.
  • An encoding element 706 may encode the grid and patch information into a bitstream for transmission to a decoder and/or storage.
  • a flag can define whether or not the resolution adaptation has been applied on the current patch. For example, a flag value of 0 may indicate that the resolution adaptation has not been applied and a flag value of 1 may indicate that the resolution adaptation has been applied.
  • a two or three bit code may define the type and resolution of the downsampling which has been applied to the current patch.
  • a two or three bit code may define the type and resolution of the downsampling which has been applied to the current patch.
  • the following codes may be used for indicating the type and resolution:
  • the resolution of patches is a function of Z value (the distance of each patch from the projection plane).
  • the resolution of each path may not be required to be signalled.
  • the resolution of patches is a function of patch ID in the stack. For example, by changing the patch ID, the resolution changes according to a pre-defined function. In this embodiment, the resolution of each path may not be required to signalled. This function may define a delta (difference) between the resolution of consecutive patches or may define a ratio between the resolution of consecutive patches.
  • a patch ID map assigning patch IDs to 2D coordinates in the texture and geometry picture, the stack ID table, describing the projection plane, common X and Y coordinate, optional per- stack resolution adaption for each stack ID, and optional Z-coordinate offset, and a patch signalling sequence, is signalled.
  • the patch signalling sequence signals patch-by-patch, first if a patch belongs to a stack or not, either as a flag or as a part of the stack ID. E.g. a binary flag signals if the current patch belongs to a patch or not.
  • the flag is followed by the signalling of the respective stack ID and signalling for optional per-patch resolution adaption and optional per- patch Z-coordinate offset. If the current patch does not belong to a stack, the flag is followed by signalling for its projection plane, its 2D location in 3D space, and, optionally, its Z-coordinate offset.
  • a stack ID is signalled for every patch, with a certain ID, e.g.’O” assigned to patches which do not belong to any stack. If this ID is signalled, e.g. stack ID“0”, the stack ID is followed by the patch’s projection plane, and its 3D (optionally 2D) location in 3D space. If any other stack ID is signalled, e.g. current patch does belong to a stack, the stack ID is followed by optional per-patch resolution adaption and optional per-patch Z-coordinate offset. Then the respective information for the next patch is signalled until all patches have been handled.
  • the patches are sorted and ranked in a descending size order. For example, considering the introduced bit code it could be 010 (for half resolution in both directions for the first patch), 101 (for quarter resolution in both directions for the second patch), 101 (for quarter resolution in both directions for the third patch).
  • the occupancy of the pixels in the projection plane may happen according to the sorted and ranked patches i.e. the top left pixel is occupied with the highest ranked patch pixels and the rest of pixels are coming from other patches in the sorted list.
  • X and Y coordinate of a shared coordinate, for example a central point, per each stack will be signalled, if no resolution adaption is used, and stack index per each patch. In accordance with an embodiment, less than 8 bits may be sufficient to signal the stack index.
  • the Z coordinate is in the depth image. Even if Z coordinate were signalled here and assuming that there are at least two patches in a stack, it is still possible to reduce the number of bits for the signalling because the X and Y are signalled per stack. If there are more patches, the savings become larger.
  • the number of additional bits is 1— 5.
  • Allowing the patches to overlap may outperform by at least the two following aspects compared to the situation in which each patch is signalled separately: First, less signalling parameters to be transmitted are needed, and second, it is possible to have different resolution of patches visible for region of interest coding and viewport-adaptive functionality by stacking e.g. different qualities of the same region of interest or viewport onto the same stack. [0122] In one embodiment, at least one patch is presented with at least two different resolutions or qualities. This may be used in scalable content presentation i.e. providing the same content with different qualities and resolutions.
  • a decoder 720 receives 690 a bitstream and a decoding element 721 decodes 691 the bitstream to reconstruct the encoded information from the bitstream.
  • the decoded information may comprise information of the geographical shape of the grid, unless a predetermined shape is used, the size (e.g. the width and height) of the grid, information of the patches and stacks and their location within the grid., and/or flags related to the patches and stacks.
  • a patch parser 722 uses 692 this information to reconstruct 693 the patches from the grids. Reconstructed patches may then be converted 694 to point clouds and further to volumetric video by an image reconstructor 723.
  • the patch parser 722 may, for example, examine a flag to determine whether or not the current patch belongs to a stack. For example, if the flag is 1, the patch belongs to a stack and hence the patch parser 722 may further examine a stack index to determine the stack in which the current patch belongs. Alternatively, if no flag for indicating whether the patch belongs to a stack is present in the signalling information, the patch parser may examine a stack index for every patch. The patch parser 722 may also examine a value of another binary flag, which would define whether or not the resolution adaptation has been applied on the current patch. For example, a flag value of 0 may indicate that the resolution adaptation has not been applied and a flag value of 1 may indicate that the resolution adaptation has been applied.
  • the patch parser 722 may further examine the value of a two or three bit code to determine the type and resolution of the downsampling which has been applied to the current patch. Alternatively, if no flag for indicating whether resolution adaptation is applied to a patch is present in the signalling information, the patch parser may examine the type and resolution of the downsampling for every patch.
  • Entropy coding/decoding may be performed in many ways. For example, context-based coding/decoding may be applied, where in the encoder modifies the context state of a coding parameter based on previously coded coding parameters. Context-based coding may for example be context adaptive binary arithmetic coding (CABAC) or context-based variable length coding
  • CABAC context adaptive binary arithmetic coding
  • Entropy coding/decoding may alternatively or additionally be performed using a variable length coding scheme, such as Huffman coding/decoding or Exp- Golomb coding/decoding.
  • Decoding of coding parameters from an entropy-coded bitstream or codewords may be referred to as parsing.
  • the H.264/AVC standard was developed by the Joint Video Team (JVT) of the Video Coding Experts Group (VCEG) of the Telecommunications Standardization Sector of International Telecommunication Union (ITU-T) and the Moving Picture Experts Group (MPEG) of International Organisation for Standardization (ISO) / International Electrotechnical Commission (IEC).
  • JVT Joint Video Team
  • VCEG Video Coding Experts Group
  • MPEG Moving Picture Experts Group
  • ISO International Organisation for Standardization
  • ISO International Electrotechnical Commission
  • the H.264/AVC standard is published by both parent standardization organizations, and it is referred to as ITU-T Recommendation H.264 and ISO/IEC International Standard 14496-10, also known as MPEG- 4 Part 10 Advanced Video Coding (AVC).
  • ITU-T Recommendation H.264 and ISO/IEC International Standard 14496-10 also known as MPEG- 4 Part 10 Advanced Video Coding (AVC).
  • AVC MPEG- 4 Part 10 Advanced Video Coding
  • H.265/HEVC a.k.a. HEVC High Efficiency Video Coding
  • JCT-VC Joint Collaborative Team - Video Coding
  • the standard was published by both parent standardization organizations, and it is referred to as ITU-T Recommendation H.265 and ISO/IEC International Standard 23008-2, also known as MPEG-H Part 2 High Efficiency Video Coding (HEVC).
  • SHVC, MV-HEVC, and 3D-HEVC use a common basis specification, specified in Annex F of the version 2 of the HEVC standard.
  • This common basis comprises for example high-level syntax and semantics e.g. specifying some of the characteristics of the layers of the bitstream, such as inter layer dependencies, as well as decoding processes, such as reference picture list construction including inter-layer reference pictures and picture order count derivation for multi-layer bitstream.
  • Annex F may also be used in potential subsequent multi-layer extensions of HEVC.
  • a video encoder a video decoder, encoding methods, decoding methods, bitstream structures, and/or embodiments may be described in the following with reference to specific extensions, such as SHVC and/or MV-HEVC, they are generally applicable to any multi-layer extensions of HEVC, and even more generally to any multi-layer video coding scheme.
  • H.264/AVC and HEVC Some key definitions, bitstream and coding structures, and concepts of H.264/AVC and HEVC are described in this section as an example of a video encoder, decoder, encoding method, decoding method, and a bitstream structure, wherein the embodiments may be implemented.
  • Some of the key definitions, bitstream and coding structures, and concepts of H.264/AVC are the same as in HEVC - hence, they are described below jointly.
  • the aspects of the invention are not limited to H.264/AVC or HEVC, but rather the description is given for one possible basis on top of which the invention may be partly or fully realized.
  • bitstream syntax and semantics as well as the decoding process for error-free bitstreams are specified in H.264/AVC and HEVC.
  • the encoding process is not specified, but encoders must generate conforming bitstreams. Bitstream and decoder conformance can be verified with the Hypothetical Reference Decoder (HRD).
  • HRD Hypothetical Reference Decoder
  • the standards contain coding tools that help in coping with transmission errors and losses, but the use of the tools in encoding is optional and no decoding process has been specified for erroneous bitstreams.
  • the elementary unit for the input to an H.264/AVC or HEVC encoder and the output of an H.264/AVC or HEVC decoder, respectively, is a picture.
  • a picture given as an input to an encoder may also be referred to as a source picture, and a picture decoded by a decoded may be referred to as a decoded picture.
  • the source and decoded pictures are each comprised of one or more sample arrays, such as one of the following sets of sample arrays:
  • RGB Green, Blue and Red
  • YZX also known as XYZ
  • these arrays may be referred to as luma (or L or Y) and chroma, where the two chroma arrays may be referred to as Cb and Cr; regardless of the actual color representation method in use.
  • the actual color representation method in use can be indicated e.g. in a coded bitstream e.g. using the Video Usability Information (VUI) syntax of H.264/AVC and/or HEVC.
  • VUI Video Usability Information
  • a component may be defined as an array or single sample from one of the three sample arrays (luma and two chroma) or the array or a single sample of the array that compose a picture in monochrome format.
  • a picture may be defined to be either a frame or a field.
  • a frame may be defined to comprise a matrix of luma samples and possibly the corresponding chroma samples.
  • a field may be defined to be a set of alternate sample rows of a frame. Fields may be used as encoder input for example when the source signal is interlaced. Chroma sample arrays may be absent (and hence monochrome sampling may be in use) or may be subsampled when compared to luma sample arrays.
  • each of the two chroma arrays has half the height and half the width of the luma array.
  • each of the two chroma arrays has the same height and half the width of the luma array.
  • each of the two chroma arrays has the same height and width as the luma array.
  • a partitioning may be defined as a division of a set into subsets such that each element of the set is in exactly one of the subsets.
  • a coding block may be defined as an NxN block of samples for some value of N such that the division of a coding tree block into coding blocks is a partitioning.
  • a coding tree block (CTB) may be defined as an NxN block of samples for some value of N such that the division of a component into coding tree blocks is a partitioning.
  • a coding tree unit may be defined as a coding tree block of luma samples, two corresponding coding tree blocks of chroma samples of a picture that has three sample arrays, or a coding tree block of samples of a monochrome picture or a picture that is coded using three separate color planes and syntax structures used to code the samples.
  • a coding unit may be defined as a coding block of luma samples, two corresponding coding blocks of chroma samples of a picture that has three sample arrays, or a coding block of samples of a monochrome picture or a picture that is coded using three separate color planes and syntax structures used to code the samples.
  • a CU with the maximum allowed size may be named as LCU (largest coding unit) or coding tree unit (CTU) and the video picture is divided into non- overlapping LCUs.
  • a CU consists of one or more prediction units (PU) defining the prediction process for the samples within the CU and one or more transform units (TU) defining the prediction error coding process for the samples in the said CU.
  • PU prediction units
  • TU transform units
  • a CU consists of a square block of samples with a size selectable from a predefined set of possible CU sizes.
  • Each PU and TU can be further split into smaller PUs and TUs in order to increase granularity of the prediction and prediction error coding processes, respectively.
  • Each PU has prediction information associated with it defining what kind of a prediction is to be applied for the pixels within that PU (e.g. motion vector information for inter predicted PUs and intra prediction directionality information for intra predicted PUs).
  • Each TU can be associated with information describing the prediction error decoding process for the samples within the said TU (including e.g. DCT coefficient information). It is typically signalled at CU level whether prediction error coding is applied or not for each CU. In the case there is no prediction error residual associated with the CU, it can be considered there are no TUs for the said CU.
  • the division of the image into CUs, and division of CUs into PUs and TUs is typically signalled in the bitstream allowing the decoder to reproduce the intended structure of these units.
  • a picture can be partitioned in tiles, which are rectangular and contain an integer number of LCUs.
  • the partitioning to tiles forms a regular grid, where heights and widths of tiles differ from each other by one LCU at the maximum.
  • a slice is defined to be an integer number of coding tree units contained in one independent slice segment and all subsequent dependent slice segments (if any) that precede the next independent slice segment (if any) within the same access unit.
  • a slice segment is defined to be an integer number of coding tree units ordered consecutively in the tile scan and contained in a single NAL unit. The division of each picture into slice segments is a partitioning.
  • an independent slice segment is defined to be a slice segment for which the values of the syntax elements of the slice segment header are not inferred from the values for a preceding slice segment
  • a dependent slice segment is defined to be a slice segment for which the values of some syntax elements of the slice segment header are inferred from the values for the preceding independent slice segment in decoding order.
  • a slice header is defined to be the slice segment header of the independent slice segment that is a current slice segment or is the independent slice segment that precedes a current dependent slice segment
  • a slice segment header is defined to be a part of a coded slice segment containing the data elements pertaining to the first or all coding tree units represented in the slice segment.
  • the CUs are scanned in the raster scan order of LCUs within tiles or within a picture, if tiles are not in use. Within an LCU, the CUs have a specific scan order.
  • a motion-constrained tile set is such that the inter prediction process is constrained in encoding such that no sample value outside the motion-constrained tile set, and no sample value at a fractional sample position that is derived using one or more sample values outside the motion-constrained tile set, is used for inter prediction of any sample within the motion- constrained tile set. Additionally, the encoding of an MCTS is constrained in a manner that motion vector candidates are not derived from blocks outside the MCTS.
  • an MCTS may be defined to be a tile set that is independent of any sample values and coded data, such as motion vectors, that are outside the MCTS.
  • an MCTS may be required to form a rectangular area ft should be understood that depending on the context, an MCTS may refer to the tile set within a picture or to the respective tile set in a sequence of pictures. The respective tile set may be, but in general need not be, collocated in the sequence of pictures.
  • sample locations used in inter prediction may be saturated by the encoding and/or decoding process so that a location that would be outside the picture otherwise is saturated to point to the corresponding boundary sample of the picture.
  • encoders may allow motion vectors to effectively cross that boundary or a motion vector to effectively cause fractional sample interpolation that would refer to a location outside that boundary, since the sample locations are saturated onto the boundary.
  • encoders may constrain the motion vectors on picture boundaries similarly to any MCTS boundaries.
  • the temporal motion-constrained tile sets SEI message of HEVC can be used to indicate the presence of motion-constrained tile sets in the bitstream.
  • the decoder reconstructs the output video by applying prediction means similar to the encoder to form a predicted representation of the pixel blocks (using the motion or spatial information created by the encoder and stored in the compressed representation) and prediction error decoding (inverse operation of the prediction error coding recovering the quantized prediction error signal in spatial pixel domain). After applying prediction and prediction error decoding means the decoder sums up the prediction and prediction error signals (pixel values) to form the output video frame.
  • the decoder (and encoder) can also apply additional filtering means to improve the quality of the output video before passing it for display and/or storing it as prediction reference for the forthcoming frames in the video sequence.
  • the filtering may for example include one more of the following: deblocking, sample adaptive offset (SAO), and/or adaptive loop filtering (ALF).
  • deblocking sample adaptive offset (SAO), and/or adaptive loop filtering (ALF).
  • SAO sample adaptive offset
  • ALF adaptive loop filtering
  • motion information is indicated with motion vectors associated with each motion compensated image block, such as a prediction unit.
  • Each of these motion vectors represents the displacement of the image block in the picture to be coded (in the encoder side) or decoded (in the decoder side) and the prediction source block in one of the previously coded or decoded pictures.
  • motion vectors In order to represent motion vectors efficiently those are typically coded
  • the predicted motion vectors are created in a predefined way, for example calculating the median of the encoded or decoded motion vectors of the adjacent blocks.
  • Another way to create motion vector predictions is to generate a list of candidate predictions from adjacent blocks and/or co-located blocks in temporal reference pictures and signalling the chosen candidate as the motion vector predictor.
  • it can be predicted which reference picture(s) are used for motion-compensated prediction and this prediction information may be represented for example by a reference index of previously coded/decoded picture.
  • the reference index is typically predicted from adjacent blocks and/or co-located blocks in temporal reference picture.
  • typical high efficiency video codecs employ an additional motion information coding/decoding mechanism, often called merging/merge mode, where all the motion field information, which includes motion vector and corresponding reference picture index for each available reference picture list, is predicted and used without any modification/correction.
  • predicting the motion field information is carried out using the motion field information of adjacent blocks and/or co-located blocks in temporal reference pictures and the used motion field information is signalled among a list of motion field candidate list filled with motion field information of available adjacent/co-located blocks.
  • Typical video encoders utilize Lagrangian cost functions to find optimal coding modes, e.g. the desired coding mode for a block and associated motion vectors.
  • This kind of cost function uses a weighting factor l to tie together the (exact or estimated) image distortion due to lossy coding methods and the (exact or estimated) amount of information that is required to represent the pixel values in an image area:
  • C is the Lagrangian cost to be minimized
  • D is the image distortion (e.g. Mean Squared Error) with the mode and motion vectors considered
  • R the number of bits needed to represent the required data to reconstruct the image block in the decoder (including the amount of data to represent the candidate motion vectors).
  • Video coding standards and specifications may allow encoders to divide a coded picture to coded slices or alike. In-picture prediction is typically disabled across slice boundaries. Thus, slices can be regarded as a way to split a coded picture to independently decodable pieces. In H.264/AVC and HEVC, in-picture prediction may be disabled across slice boundaries. Thus, slices can be regarded as a way to split a coded picture into independently decodable pieces, and slices are therefore often regarded as elementary units for transmission. In many cases, encoders may indicate in the bitstream which types of in-picture prediction are turned off across slice boundaries, and the decoder operation takes this information into account for example when concluding which prediction sources are available. For example, samples from a neighboring CU may be regarded as unavailable for intra prediction, if the neighboring CU resides in a different slice.
  • NAL Network Abstraction Layer
  • H.264/AVC and HEVC For transport over packet-oriented networks or storage into structured files, NAL units may be encapsulated into packets or similar structures.
  • a bytestream format has been specified in H.264/AVC and HEVC for transmission or storage environments that do not provide framing structures. The bytestream format separates NAL units from each other by attaching a start code in front of each NAL unit.
  • a NAL unit may be defined as a syntax structure containing an indication of the type of data to follow and bytes containing that data in the form of an RBSP interspersed as necessary with emulation prevention bytes.
  • a raw byte sequence payload (RBSP) may be defined as a syntax structure containing an integer number of bytes that is encapsulated in a NAL unit.
  • NAL units consist of a header and payload.
  • the NAL unit header indicates the type of the NAL unit
  • NAL unit header contains one reserved bit, a six-bit NAL unit type indication, a three-bit
  • nuh_temporal_id_plusl indication for temporal level may be required to be greater than or equal to 1 and a six-bit nuh layer id syntax element.
  • TID may be used to interchangeably with the Temporalld variable.
  • Temporalld equal to 0 corresponds to the lowest temporal level.
  • the value of temporal_id_plusl is required to be non- zero in order to avoid start code emulation involving the two NAL unit header bytes.
  • NAL units can be categorized into Video Coding Layer (VCL) NAL units and non-VCL NAL units.
  • VCL NAL units are typically coded slice NAL units.
  • VCL NAL units contain syntax elements representing one or more CU.
  • a non-VCL NAL unit may be for example one of the following types: a sequence parameter set, a picture parameter set, a supplemental enhancement information (SEI) NAL unit, an access unit delimiter, an end of sequence NAL unit, an end of bitstream NAL unit, or a filler data NAL unit.
  • SEI Supplemental Enhancement Information
  • Parameter sets may be needed for the reconstruction of decoded pictures, whereas many of the other non-VCL NAL units are not necessary for the reconstruction of decoded sample values.
  • Parameters that remain unchanged through a coded video sequence may be included in a sequence parameter set.
  • the sequence parameter set may optionally contain video usability information (VUI), which includes parameters that may be important for buffering, picture output timing, rendering, and resource reservation.
  • VUI video usability information
  • a sequence parameter set RBSP includes parameters that can be referred to by one or more picture parameter set RBSPs or one or more SEI NAL units containing a buffering period SEI message.
  • a picture parameter set contains such parameters that are likely to be unchanged in several coded pictures.
  • a picture parameter set RBSP may include parameters that can be referred to by the coded slice NAL units of one or more coded pictures.
  • a video parameter set may be defined as a syntax structure containing syntax elements that apply to zero or more entire coded video sequences as determined by the content of a syntax element found in the SPS referred to by a syntax element found in the PPS referred to by a syntax element found in each slice segment header.
  • a video parameter set RBSP may include parameters that can be referred to by one or more sequence parameter set RBSPs.
  • Out-of-band transmission, signaling or storage can additionally or alternatively be used for other purposes than tolerance against transmission errors, such as ease of access or session negotiation.
  • a sample entry of a track in a file conforming to the ISO Base Media File Format may comprise parameter sets, while the coded data in the bitstream is stored elsewhere in the file or in another file.
  • the phrase along the bitstream e.g. indicating along the bitstream
  • a coded unit of a bitstream e.g.
  • indicating along a coded tile may be used in claims and described embodiments to refer to out-of-band transmission, signaling, or storage in a manner that the out-of-band data is associated with the bitstream or the coded unit, respectively.
  • decoding along the bitstream or along a coded unit of a bitstream or alike may refer to decoding the referred out-of-band data (which may be obtained from out-of-band transmission, signaling, or storage) that is associated with the bitstream or the coded unit, respectively.
  • a SEI NAL unit may contain one or more SEI messages, which are not required for the decoding of output pictures but may assist in related processes, such as picture output timing, rendering, error detection, error concealment, and resource reservation.
  • SEI messages are specified in H.264/AVC and HEVC, and the user data SEI messages enable organizations and companies to specify SEI messages for their own use.
  • H.264/AVC and HEVC contain the syntax and semantics for the specified SEI messages but no process for handling the messages in the recipient is defined.
  • encoders are required to follow the H.264/AVC standard or the HEVC standard when they create SEI messages, and decoders conforming to the H.264/AVC standard or the HEVC standard, respectively, are not required to process SEI messages for output order conformance.
  • One of the reasons to include the syntax and semantics of SEI messages in H.264/AVC and HEVC is to allow different system specifications to interpret the supplemental information identically and hence interoperate. It is intended that system specifications can require the use of particular SEI messages both in the encoding end and in the decoding end, and additionally the process for handling particular SEI messages in the recipient can be specified.
  • SEI NAL units there are two types, namely the suffix SEI NAL unit and the prefix SEI NAL unit, having a different nal unit type value from each other.
  • the SEI message(s) contained in a suffix SEI NAL unit are associated with the VCL NAL unit preceding, in decoding order, the suffix SEI NAL unit.
  • the SEI message(s) contained in a prefix SEI NAL unit are associated with the VCL NAL unit following, in decoding order, the prefix SEI NAL unit.
  • a coded picture is a coded representation of a picture.
  • a coded picture may be defined as a coded representation of a picture containing all coding tree units of the picture.
  • an access unit (AU) may be defined as a set of NAL units that are associated with each other according to a specified classification rule, are consecutive in decoding order, and contain at most one picture with any specific value of nuh layer id.
  • an access unit may also contain non- VCL NAL units. Said specified classification rule may for example associate pictures with the same output time or picture output count value into the same access unit.
  • a bitstream may be defined as a sequence of bits, in the form of a NAL unit stream or a byte stream, that forms the representation of coded pictures and associated data forming one or more coded video sequences.
  • a first bitstream may be followed by a second bitstream in the same logical channel, such as in the same file or in the same connection of a communication protocol.
  • An elementary stream (in the context of video coding) may be defined as a sequence of one or more bitstreams.
  • the end of the first bitstream may be indicated by a specific NAL unit, which may be referred to as the end of bitstream (EOB) NAL unit and which is the last NAL unit of the bitstream.
  • EOB NAL unit is required to have nuh layer id equal to 0.
  • a coded video sequence is defined to be a sequence of consecutive access units in decoding order from an IDR access unit, inclusive, to the next IDR access unit, exclusive, or to the end of the bitstream, whichever appears earlier.
  • a coded video sequence may be defined, for example, as a sequence of access units that consists, in decoding order, of an IRAP access unit with NoRaslOutputFlag equal to 1, followed by zero or more access units that are not IRAP access units with NoRaslOutputFlag equal to 1 , including all subsequent access units up to but not including any subsequent access unit that is an IRAP access unit with NoRaslOutputFlag equal to 1.
  • An IRAP access unit may be defined as an access unit in which the base layer picture is an IRAP picture.
  • NoRaslOutputFlag is equal to 1 for each IDR picture, each BLA picture, and each IRAP picture that is the first picture in that particular layer in the bitstream in decoding order, is the first IRAP picture that follows an end of sequence NAL unit having the same value of nuh layer id in decoding order.
  • HandleCraAsBlaFlag may be set to 1 for example by a player that seeks to a new position in a bitstream or tunes into a broadcast and starts decoding and then starts decoding from a CRA picture.
  • HandleCraAsBlaFlag is equal to 1 for a CRA picture, the CRA picture is handled and decoded as if it were a BLA picture.
  • a coded video sequence may additionally or alternatively (to the specification above) be specified to end, when a specific NAL unit, which may be referred to as an end of sequence (EOS) NAL unit, appears in the bitstream and has nuh layer id equal to 0.
  • EOS end of sequence
  • a group of pictures (GOP) and its characteristics may be defined as follows.
  • a GOP can be decoded regardless of whether any previous pictures were decoded.
  • An open GOP is such a group of pictures in which pictures preceding the initial intra picture in output order might not be correctly decodable when the decoding starts from the initial intra picture of the open GOP.
  • pictures of an open GOP may refer (in inter prediction) to pictures belonging to a previous GOP.
  • An HEVC decoder can recognize an intra picture starting an open GOP, because a specific NAL unit type, CRA NAL unit type, may be used for its coded slices.
  • a closed GOP is such a group of pictures in which all pictures can be correctly decoded when the decoding starts from the initial intra picture of the closed GOP. In other words, no picture in a closed GOP refers to any pictures in previous GOPs.
  • a closed GOP may start from an IDR picture.
  • a closed GOP may also start from a BLA W RADL or a BLA N LP picture.
  • An open GOP coding structure is potentially more efficient in the compression compared to a closed GOP coding structure, due to a larger flexibility in selection of reference pictures.
  • a Decoded Picture Buffer may be used in the encoder and/or in the decoder. There are two reasons to buffer decoded pictures, for references in inter prediction and for reordering decoded pictures into output order. As H.264/AVC and HEVC provide a great deal of flexibility for both reference picture marking and output reordering, separate buffers for reference picture buffering and output picture buffering may waste memory resources. Hence, the DPB may include a unified decoded picture buffering process for reference pictures and output reordering. A decoded picture may be removed from the DPB when it is no longer used as a reference and is not needed for output.
  • the reference picture for inter prediction is indicated with an index to a reference picture list.
  • the index may be coded with variable length coding, which usually causes a smaller index to have a shorter value for the corresponding syntax element.
  • two reference picture lists (reference picture list 0 and reference picture list 1) are generated for each bi-predictive (B) slice, and one reference picture list (reference picture list 0) is formed for each inter-coded (P) slice.
  • a reference picture index may be coded by an encoder into the bitstream is some inter coding modes or it may be derived (by an encoder and a decoder) for example using neighboring blocks in some other inter coding modes.
  • motion vector prediction HEVC includes two motion vector prediction schemes, namely the advanced motion vector prediction (AMVP) and the merge mode.
  • AMVP advanced motion vector prediction
  • merge mode a list of motion vector candidates is derived for a PU.
  • candidates spatial candidates and temporal candidates, where temporal candidates may also be referred to as TMVP candidates.
  • a candidate list derivation may be performed for example as follows, while it should be understood that other possibilities may exist for candidate list derivation. If the occupancy of the candidate list is not at maximum, the spatial candidates are included in the candidate list first if they are available and not already exist in the candidate list. After that, if occupancy of the candidate list is not yet at maximum, a temporal candidate is included in the candidate list. If the number of candidates still does not reach the maximum allowed number, the combined bi-predictive candidates (for B slices) and a zero motion vector are added in. After the candidate list has been constructed, the encoder decides the final motion information from candidates for example based on a rate- distortion optimization (RDO) decision and encodes the index of the selected candidate into the bitstream. Likewise, the decoder decodes the index of the selected candidate from the bitstream, constructs the candidate list, and uses the decoded index to select a motion vector predictor from the candidate list.
  • RDO rate- distortion optimization
  • AMVP and the merge mode may be characterized as follows.
  • the encoder indicates whether uni-prediction or bi-prediction is used and which reference pictures are used as well as encodes a motion vector difference.
  • the merge mode only the chosen candidate from the candidate list is encoded into the bitstream indicating the current prediction unit has the same motion information as that of the indicated predictor.
  • the merge mode creates regions composed of neighbouring prediction blocks sharing identical motion information, which is only signalled once for each region.
  • Scalable media is typically ordered into hierarchical layers of data.
  • a base layer contains an individual representation of a coded media stream such as a video sequence.
  • Enhancement layers contain refinement data relative to previous layers in the layer hierarchy. The quality of the decoded media stream progressively improves as enhancement layers are added to the base layer.
  • An enhancement layer enhances the temporal resolution (i.e., the frame rate), the spatial resolution, or simply the quality of the video content represented by another layer or part thereof.
  • Each layer, together with all of its dependent layers is one representation of the video signal at a certain spatial resolution, temporal resolution and quality level. Therefore, the term "scalable layer representation" is used herein to describe a scalable layer together with all of its dependent layers. The portion of a scalable bitstream corresponding to a scalable layer representation can be extracted and decoded to produce a representation of the original signal at a certain fidelity.
  • Each SVC layer is formed by NAL units, representing the coded video bits of the layer.
  • a Real Time Transport Protocol (RTP) stream carrying only one layer would carry NAL units belonging to that layer only.
  • An RTP stream carrying a complete scalable video bit stream would carry NAL units of a base layer and one or more enhancement layers.
  • SVC specifies the decoding order of these NAL units.
  • data in an enhancement layer can be truncated after a certain location, or at arbitrary positions, where each truncation position may include additional data representing increasingly enhanced visual quality.
  • the scalability is said to be "fine- grained,” hence the term “fine grained (granular) scalability” (FGS).
  • FGS fine grained (granular) scalability
  • CGS coarse-grained (granularity) scalability
  • an access unit comprises one primary coded picture.
  • detection of access unit boundaries can be simplified by inserting an access unit delimiter NAL unit into the bitstream hi SVC, an access unit may comprise multiple primary coded pictures, but at most one picture per each unique combination of dependency id, temporal level, and quality level.
  • Coded video bitstream may include extra information to enhance the use of the video for a wide variety purposes.
  • supplemental enhancement information SEI
  • VUI video usability information
  • H.264/AVC standard and its extensions include the support of SEI signalling through SEI messages.
  • SEI messages are not required by the decoding process to generate correct sample values in output pictures. Rather, they are helpful for other purposes, e.g., error resilience and display.
  • H.264/AVC contains the syntax and semantics for the specified SEI messages, but no process for handling the messages in the recipient is defined.
  • system specifications can require the use of particular SEI messages both in encoding end and in decoding end, and the process for handling SEI messages in the recipient may be specified for the application in a system specification.
  • SVC uses a similar mechanism as that used in H.264/AVC to provide hierarchical temporal scalability.
  • SVC a certain set of reference and non-reference pictures can be dropped from a coded bistream without affecting the decoding of the remaining bitstream.
  • Hierarchical temporal scalability requires multiple reference pictures for motion compensation, i e. there is a reference picture buffer containing multiple decoded pictures from which an encoder can select a reference picture for inter prediction.
  • a feature called sub-sequences enables hierarchical temporal scalability, where each enhancement layer contains sub-sequences and each sub-sequence contains a number of reference and/or non-reference pictures.
  • the subsequence is also comprised of a number of inter dependent pictures that can be disposed without any disturbance to any other sub-sequence in any lower subsequence layer.
  • the sub-sequence layers are hierarchically arranged based on their dependency on each other. Therefore, when a sub-sequence in the highest enhancement layer is disposed, the remaining bitstream remains valid.
  • SEI subsequence-related supplemental enhancement information
  • SVC the temporal level hierarchy is indicated in the header of Network Abstraction Layer (NAL) units.
  • SVC uses an inter-layer prediction mechanism, whereby certain information can be predicted from layers other than a currently reconstructed layer or a next lower layer.
  • Inter layer motion prediction also includes the prediction of block coding mode, header information, etc., where motion information from a lower layer may be used for predicting a higher layer. It is also possible to use intra coding in SVC, i.e., a prediction from surrounding macroblocks or from co located macroblocks of lower layers. Such prediction techniques do not employ motion information and hence, are referred to as intra prediction techniques Furthermore, residual data from lower layers can also be employed for predicting the current layer.
  • SVC involves the encoding of a "base layer” with some minimal quality, as well as the encoding of enhancement information that increases the quality up to a maximum level.
  • the base layer of SVC streams is typically advanced video coding (AVC)-compliant. ln other words, AVC decoders can decode the base layer of an SVC stream and ignore SVC-specific data. This feature has been realized by specifying coded slice NAL unit types that are specific to SVC, were reserved for future use in AVC, and must be skipped according to the AVC specification.
  • a video signal may be encoded into a base layer and one or more enhancement layers.
  • An enhancement layer may enhance, for example, the temporal resolution (i.e., the frame rate), the spatial resolution, or simply the quality of the video content represented by another layer or part thereof.
  • Each layer together with all its dependent layers is one representation of the video signal, for example, at a certain spatial resolution, temporal resolution and quality level, and can be extracted from the original bitstream and decoded to produce a representation of the original signal at certain fidelity.
  • Scalability modes or scalability dimensions may include but are not limited to the following:
  • Base layer pictures are coded at a lower quality than enhancement layer pictures, which may be achieved for example using a greater quantization parameter value (i.e., a greater quantization step size for transform coefficient quantization) in the base layer than in the enhancement layer.
  • a greater quantization parameter value i.e., a greater quantization step size for transform coefficient quantization
  • Spatial scalability Base layer pictures are coded at a lower resolution (i.e. have fewer samples) than enhancement layer pictures. Spatial scalability and quality scalability may sometimes be considered the same type of scalability.
  • Bit-depth scalability Base layer pictures are coded at lower bit-depth (e.g. 8 bits) than enhancement layer pictures (e.g. 10 or 12 bits).
  • Dynamic range scalability Scalable layers represent a different dynamic range and/or images obtained using a different tone mapping function and/or a different optical transfer function.
  • Chroma format scalability Base layer pictures provide lower spatial resolution in chroma sample arrays (e.g. coded in 4:2:0 chroma format) than enhancement layer pictures (e.g. 4:4:4 format).
  • enhancement layer pictures have a richer/broader color representation range than that of the base layer pictures - for example the enhancement layer may have UHDTV (ITU-R BT.2020) color gamut and the base layer may have the ITU-R BT.709 color gamut.
  • UHDTV ITU-R BT.2020
  • View scalability which may also be referred to as multiview coding.
  • the base layer represents a first view
  • an enhancement layer represents a second view.
  • a view may be defined as a sequence of pictures representing one camera or viewpoint. It may be considered that in stereoscopic or two-view video, one video sequence or view is presented for the left eye while a parallel view is presented for the right eye.
  • Depth scalability which may also be referred to as depth-enhanced coding.
  • a layer or some layers of a bitstream may represent texture view(s), while other layer or layers may represent depth view(s).
  • Hybrid codec scalability also known as coding standard scalability: In hybrid codec scalability, the bitstream syntax, semantics and decoding process of the base layer and the enhancement layer are specified in different video coding standards. Thus, base layer pictures are coded according to a different coding standard or format than enhancement layer pictures.
  • the term layer may be used in context of any type of scalability, including view scalability and depth enhancements.
  • An enhancement layer may refer to any type of an enhancement, such as SNR, spatial, multiview, depth, bit-depth, chroma format, and/or color gamut enhancement.
  • a base layer may refer to any type of a base video sequence, such as a base view, a base layer for SNR/spatial scalability, or a texture base view for depth-enhanced video coding.
  • Scalability may be enabled in two basic ways, either by introducing new coding modes for performing prediction of pixel values or syntax from lower layers of the scalable representation or by placing the lower layer pictures to a reference picture buffer (e.g. a decoded picture buffer, DPB) of the higher layer.
  • the first approach may be more flexible and thus may provide better coding efficiency in most cases.
  • the second, reference frame based scalability, approach may be implemented efficiently with minimal changes to single layer codecs while still achieving majority of the coding efficiency gains available.
  • a reference frame based scalability codec may be implemented by utilizing the same hardware or software implementation for all the layers, just taking care of the DPB management by external means.
  • a scalable video encoder for quality scalability also known as Signal-to-Noise or SNR
  • spatial scalability may be implemented as follows.
  • a base layer a conventional non- scalable video encoder and decoder may be used.
  • the reconstructed/decoded pictures of the base layer are included in the reference picture buffer and/or reference picture lists for an enhancement layer.
  • the reconstructed/decoded base-layer picture may be upsampled prior to its insertion into the reference picture lists for an enhancement- layer picture.
  • the base layer decoded pictures may be inserted into a reference picture list(s) for coding/decoding of an enhancement layer picture similarly to the decoded reference pictures of the enhancement layer.
  • the encoder may choose a base-layer reference picture as an inter prediction reference and indicate its use with a reference picture index in the coded bitstream.
  • the decoder decodes from the bitstream, for example from a reference picture index, that a base-layer picture is used as an inter prediction reference for the enhancement layer.
  • a decoded base-layer picture is used as the prediction reference for an enhancement layer, it is referred to as an inter-layer reference picture.
  • a second enhancement layer may depend on a first enhancement layer in encoding and/or decoding processes, and the first enhancement layer may therefore be regarded as the base layer for the encoding and/or decoding of the second enhancement layer.
  • inter-layer reference pictures from more than one layer in a reference picture buffer or reference picture lists of an enhancement layer, and each of these inter-layer reference pictures may be considered to reside in a base layer or a reference layer for the enhancement layer being encoded and/or decoded.
  • other types of inter-layer processing than reference-layer picture upsampling may take place instead or additionally.
  • the bit-depth of the samples of the reference-layer picture may be converted to the bit-depth of the enhancement layer and/or the sample values may undergo a mapping from the color space of the reference layer to the color space of the enhancement layer.
  • a scalable video coding and/or decoding scheme may use multi-loop coding and/or decoding, which may be characterized as follows.
  • a base layer picture may be reconstructed/decoded to be used as a motion-compensation reference picture for subsequent pictures, in coding/decoding order, within the same layer or as a reference for inter-layer (or inter view or inter-component) prediction.
  • the reconstructed/decoded base layer picture may be stored in the DPB.
  • An enhancement layer picture may likewise be reconstructed/decoded to be used as a motion-compensation reference picture for subsequent pictures, in coding/decoding order, within the same layer or as reference for inter-layer (or inter- view or inter-component) prediction for higher enhancement layers, if any.
  • syntax element values of the base/reference layer or variables derived from the syntax element values of the base/reference layer may be used in the inter-layer/inter-component/inter- view prediction.
  • Inter- layer prediction may be defined as prediction in a manner that is dependent on data elements (e.g., sample values or motion vectors) of reference pictures from a different layer than the layer of the current picture (being encoded or decoded). Many types of inter-layer prediction exist and may be applied in a scalable video encoder/decoder. [0191] T exture picture(s) and the respective geometry picture(s) may have the same or different chroma format.
  • a pixel may be defined to a be a sample of one of the sample arrays of the picture or may be defined to comprise the collocated samples of all the sample arrays of the picture.
  • Projecting 3D data onto 2D planes is independent from the 3D scene model representation format.
  • the decoder receives the coded bitstream of the volumetric video (or a part of it) and decodes the information carried by the bitstream or a part of it.
  • the decoder decodes the depth picture.
  • the decoder may then examine depth values of a block and compare them to the default value, for example to 0, a maximum depth value, or a minimum depth value. If the comparison reveals that the depth value of the current pixel is equal to the default value or deviates from the default value less than a threshold, the decoder may conclude that the depth pixel is non- valid. If all the depth pixels of a depth pixel are non- valid, the decoder may conclude that the depth block is non- valid. When the depth block is non- valid, the decoder can ignore that block and need not decode the corresponding texture block (i.e., all texture blocks that are fully covered by the non- valid depth block).
  • the usage of the threshold instead of the exact value in the comparisons may be beneficial at the decoder side because encoding, transmission and/or decoding may not be lossless, wherein a value of a parameter/variable after decoding may not be exactly the same than before encoding.
  • a depth picture which may also be called as a depth plane, is initialized to default values.
  • the default value may be, for example, a zero value, a maximum depth value, a minimum depth value, or some other predetermined value.
  • a texture picture which may also be called as a texture plane, may also be initialized to default values. The default values for the texture picture may be different from the default values for the depth picture.
  • the decoder may then decode the depth picture, wherein locations of the depth picture which contain projection data of the object, receive a corresponding depth value and locations which do not contain projection data of the object, remain in the default value. [0200] After decoding the depth picture, the decoded depth picture is examined to determine which pixels of the depth picture have received some value during the construction of the depth picture, meaning that a point of the object has been projected to that pixel (a point in the projection surface).
  • the examination may be performed, for example, so that each value in the depth picture is compared with the default value and if the comparison reveals that the value of the depth picture is equal to the default value, or only deviates from the default value less than a threshold, it is then determined that such a pixel does not contain any projection data and need not be decoded by the decoder. Moreover, such pixel will not be reconstructed into 3D space.
  • a decoder decodes certain syntax elements and concludes from a particular set of syntax element values that decoding of texture blocks corresponding to non- valid depth blocks is not required. For example, if a block- wise indication, such as the CBP, is used, the decoder may examine the CBP value of the block and if it is equal to the above mentioned specific value, the decoder can deduce that this block need not be decoded.
  • a block- wise indication such as the CBP
  • a decoder receives a bit stream containing at least one coded texture and one coded geometry picture, plus additional data in or along the bitstream, describing the 3D nature of the texture and geometry data required for reprojection into 3D space.
  • data could comprise: a patch ID map, representing a certain ID to each projected patch in the texture and geometry picture, a stack ID table, describing the projection plane, common X and Y coordinate, and optional per-stack resolution adaption for each stack, and a patch signalling sequence, describing the 3D reprojection of each patch.
  • a sequence will signal, for every patch in the texture/geometry picture, if it belongs to a stack or not, its location in 3D space and optional additional resolution adaption.
  • a decoder decodes texture, geometry and additional data as described above. Then, a patch is identified in the geometry and texture image based on the patch ID map. For each patch, the further processing is defined by its entry in the patch signalling sequence. For example, patch“0” is signaled to belong to the stack with ID“4” and has per-patch resolution adaption signaled (“1”) with option“100” (horizontal quarter resolution). Furthermore, stack“4” has a X/Y projection plane, X coordinate 100 and Y coordinate 150. The Z coordinate is given as 50 from the geometry picture.
  • the decoder now knows that it has to upsample the current patch by a factor of 4 along the horizontal axis and then project it at XYZ-position (100,150,50) into 3D space.
  • patch“1” is signalled to belong to the same stack“4” without resolution adaption and the Z-coordinate is given as 25 from the geometry picture. This patch will be projected at XYZ-position (100,150,25) into 3D space.
  • patch“2” is signalled to belong to no stack. It has Y/Z projection plane, X/Y coordinates are signalled as (200,100) and the Z coordinate from the geometry picture is 125. This patch would be projected at XYZ-position (125,200,100) in 3D space due to the different projection plane.
  • the decoder receives information if a patch belongs to a stack in form of a flag.
  • the decoder receives information if a patch belongs to a stack as part of the stack ID table, i.e. a certain stack ID (e.g.“0”) is assigned for patches which do not belong to a stack.
  • the decoder receives information if resolution adaption has been applied to a patch in form of a flag in the patch signaling sequence, followed by a codeword identifying the resolution adaption procedure.
  • the decoder receives information if resolution adaption has been applied to a stack in form of a flag in the stack ID table, followed by a codeword identifying the resolution adaption procedure.
  • the decoder receives Z-coordinate offset information as part of the patch signaling sequence.
  • Such an offset can be positive or negative.
  • patch“3” is signaled to belong to stack“4” without resolution adaption and Z-coordinate offset of 800. Its initial Z-coordinate is given as 50 from the geometry picture. This patch will be projected at XYZ-position (100,150,850) into 3D space.
  • the decoder receives Z-coordinate offset information as part of the stack ID table.
  • Such an offset can be positive or negative.
  • stack“5” is similar to stack“4” with a X/Y projection plane, X coordinate 100, Y coordinate 150, but additionally a Z coordinate offset of -5.
  • the respective coordinates for patch“0”,“1” and“3”, if assigned to stack “5” instead of“4” would be (100,150,45), (100,150,20) and (100,150,845).
  • said encoding may comprise one or more of the following: encoding source image data into a bitstream, encapsulating the encoded bitstream in a container file and/or in packet(s) or stream(s) of a communication protocol, and announcing or describing the bitstream in a content description, such as the Media Presentation Description (MPD) of ISO/IEC 23009-1 (known as MPEG-DASH) or the IETF Session Description Protocol (SDP).
  • MPD Media Presentation Description
  • SDP IETF Session Description Protocol
  • said decoding may comprise one or more of the following: decoding image data from a bitstream, decapsulating the bitstream from a container file and/or from packet(s) or stream(s) of a communication protocol, and parsing a content description of the bitstream,
  • the communication devices may communicate using various transmission technologies including, but not limited to, code division multiple access (CDMA), global systems for mobile communications (GSM), universal mobile telecommunications system (UMTS), time divisional multiple access (TDMA), frequency division multiple access (FDMA), transmission control protocol- internet protocol (TCP-IP), short messaging service (SMS), multimedia messaging service (MMS), email, instant messaging service (IMS), Bluetooth, IEEE 802.11, Long Term Evolution wireless communication technique (LTE) and any similar wireless communication technology.
  • CDMA code division multiple access
  • GSM global systems for mobile communications
  • UMTS universal mobile telecommunications system
  • TDMA time divisional multiple access
  • FDMA frequency division multiple access
  • TCP-IP transmission control protocol- internet protocol
  • SMS short messaging service
  • MMS multimedia messaging service
  • email instant messaging service
  • IMS instant messaging service
  • Bluetooth IEEE 802.11, Long Term Evolution wireless communication technique (LTE) and any similar wireless communication technology.
  • LTE Long Term Evolution wireless communication technique
  • communications device involved in implementing various embodiments of the present invention may communicate using various media including, but not limited to, radio, infrared, laser, cable connections, and any suitable connection.
  • embodiments of the invention operating within a wireless communication device
  • the invention as described above may be implemented as a part of any apparatus comprising a circuitry in which radio frequency signals are transmitted and received.
  • embodiments of the invention may be implemented in a mobile phone, in a base station, in a computer such as a desktop computer or a tablet computer comprising radio frequency communication means (e.g. wireless local area network, cellular radio, etc.).
  • radio frequency communication means e.g. wireless local area network, cellular radio, etc.
  • the various embodiments of the invention may be implemented in hardware or special purpose circuits or any combination thereof. While various aspects of the invention may be illustrated and described as block diagrams or using some other pictorial representation, it is well understood that these blocks, apparatus, systems, techniques or methods described herein may be implemented in, as non-limiting examples, hardware, software, firmware, special purpose circuits or logic, general purpose hardware or controller or other computing devices, or some combination thereof.
  • Embodiments of the inventions may be practiced in various components such as integrated circuit modules.
  • the design of integrated circuits is by and large a highly automated process.
  • Complex and powerful software tools are available for converting a logic level design into a semiconductor circuit design ready to be etched and formed on a semiconductor substrate.
  • Programs such as those provided by Synopsys, Inc. of Mountain View, California and Cadence Design, of San Jose, California automatically route conductors and locate components on a semiconductor chip using well established rules of design as well as libraries of pre stored design modules.
  • the resultant design in a standardized electronic format (e.g., Opus, GDSII, or the like) may be transmitted to a semiconductor fabrication facility or "fab" for fabrication.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Processing Or Creating Images (AREA)
  • Testing, Inspecting, Measuring Of Stereoscopic Televisions And Televisions (AREA)

Abstract

L'invention concerne divers procédés, appareils et produits-programmes informatiques destinés au codage et au décodage d'une vidéo volumétrique. Dans certains modes de réalisation, au moins deux blocs formés à partir d'informations d'image tridimensionnelles sont obtenus. Chaque bloc représente les données de projection d'au moins une partie d'un objet dans un plan de projection. Des blocs se chevauchant sont identifiés à partir des au moins deux blocs et combinés à une pile de blocs. Les blocs se chevauchant sont mappés pour avoir le même point central.
PCT/FI2018/050948 2017-12-29 2018-12-20 Appareil, procédé et programme informatique pour vidéo volumétrique Ceased WO2019129923A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
FI20176205 2017-12-29
FI20176205 2017-12-29

Publications (1)

Publication Number Publication Date
WO2019129923A1 true WO2019129923A1 (fr) 2019-07-04

Family

ID=67063267

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/FI2018/050948 Ceased WO2019129923A1 (fr) 2017-12-29 2018-12-20 Appareil, procédé et programme informatique pour vidéo volumétrique

Country Status (1)

Country Link
WO (1) WO2019129923A1 (fr)

Cited By (34)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2021011772A1 (fr) * 2019-07-16 2021-01-21 Apple Inc. Diffusion en continu de contenu de nuage de points volumétrique basée sur des protocoles de description de session et des protocoles en temps réel
WO2021053262A1 (fr) * 2019-09-20 2021-03-25 Nokia Technologies Oy Appareil, procédé et programme informatique pour vidéo volumétrique
US11044478B2 (en) 2018-07-02 2021-06-22 Apple Inc. Compression with multi-level encoding
US11132831B1 (en) 2020-03-02 2021-09-28 Qualcomm Incorporated Methods and apparatus for efficient multi-view rasterization
US11361471B2 (en) 2017-11-22 2022-06-14 Apple Inc. Point cloud occupancy map compression
US11367224B2 (en) 2018-10-02 2022-06-21 Apple Inc. Occupancy map block-to-patch information compression
US11386524B2 (en) 2018-09-28 2022-07-12 Apple Inc. Point cloud compression image padding
US11508094B2 (en) 2018-04-10 2022-11-22 Apple Inc. Point cloud compression
US11508095B2 (en) 2018-04-10 2022-11-22 Apple Inc. Hierarchical point cloud compression with smoothing
US11516394B2 (en) 2019-03-28 2022-11-29 Apple Inc. Multiple layer flexure for supporting a moving image sensor
US11514611B2 (en) 2017-11-22 2022-11-29 Apple Inc. Point cloud compression with closed-loop color conversion
US11527018B2 (en) 2017-09-18 2022-12-13 Apple Inc. Point cloud compression
US11533494B2 (en) 2018-04-10 2022-12-20 Apple Inc. Point cloud compression
US11538196B2 (en) 2019-10-02 2022-12-27 Apple Inc. Predictive coding for point cloud compression
US11552651B2 (en) 2017-09-14 2023-01-10 Apple Inc. Hierarchical point cloud compression
US11562507B2 (en) 2019-09-27 2023-01-24 Apple Inc. Point cloud compression using video encoding with time consistent patches
US11615557B2 (en) 2020-06-24 2023-03-28 Apple Inc. Point cloud compression using octrees with slicing
US11620768B2 (en) 2020-06-24 2023-04-04 Apple Inc. Point cloud geometry compression using octrees with multiple scan orders
US11627314B2 (en) 2019-09-27 2023-04-11 Apple Inc. Video-based point cloud compression with non-normative smoothing
US11625866B2 (en) 2020-01-09 2023-04-11 Apple Inc. Geometry encoding using octrees and predictive trees
US11647226B2 (en) 2018-07-12 2023-05-09 Apple Inc. Bit stream structure for compressed point cloud data
US11663744B2 (en) 2018-07-02 2023-05-30 Apple Inc. Point cloud compression with adaptive filtering
US11676309B2 (en) 2017-09-18 2023-06-13 Apple Inc Point cloud compression using masks
US11683525B2 (en) 2018-07-05 2023-06-20 Apple Inc. Point cloud compression with multi-resolution video encoding
CN116468992A (zh) * 2023-06-20 2023-07-21 北京天图万境科技有限公司 一种重复校正性监督空间识别与还原方法与装置
US11727603B2 (en) 2018-04-10 2023-08-15 Apple Inc. Adaptive distance based point cloud compression
US11798196B2 (en) 2020-01-08 2023-10-24 Apple Inc. Video-based point cloud compression with predicted patches
US11818401B2 (en) 2017-09-14 2023-11-14 Apple Inc. Point cloud geometry compression using octrees and binary arithmetic encoding with adaptive look-up tables
US11895307B2 (en) 2019-10-04 2024-02-06 Apple Inc. Block-based predictive coding for point cloud compression
US11935272B2 (en) 2017-09-14 2024-03-19 Apple Inc. Point cloud compression
US11948338B1 (en) 2021-03-29 2024-04-02 Apple Inc. 3D volumetric content encoding using 2D videos and simplified 3D meshes
US12047604B2 (en) 2019-03-19 2024-07-23 Nokia Technologies Oy Apparatus, a method and a computer program for volumetric video
US12100183B2 (en) 2018-04-10 2024-09-24 Apple Inc. Point cloud attribute transfer algorithm
US12439083B2 (en) 2019-07-02 2025-10-07 Apple Inc. Point cloud compression with supplemental information messages

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130162634A1 (en) * 2011-12-26 2013-06-27 Samsung Electronics Co., Ltd. Image processing method and apparatus using multi-layer representation
US20170256059A1 (en) * 2016-03-07 2017-09-07 Ricoh Company, Ltd. Object Segmentation from Light Field Data
US20170316602A1 (en) * 2014-10-31 2017-11-02 Nokia Technologies Oy Method for alignment of low-quality noisy depth map to the high-resolution colour image
US20170358092A1 (en) * 2016-06-09 2017-12-14 Lytro, Inc. Multi-view scene segmentation and propagation

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130162634A1 (en) * 2011-12-26 2013-06-27 Samsung Electronics Co., Ltd. Image processing method and apparatus using multi-layer representation
US20170316602A1 (en) * 2014-10-31 2017-11-02 Nokia Technologies Oy Method for alignment of low-quality noisy depth map to the high-resolution colour image
US20170256059A1 (en) * 2016-03-07 2017-09-07 Ricoh Company, Ltd. Object Segmentation from Light Field Data
US20170358092A1 (en) * 2016-06-09 2017-12-14 Lytro, Inc. Multi-view scene segmentation and propagation

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
MAMMOU, K.: "PCC Test Model Category 2 v0", ISO/IEC JTC1/SC29/ WG11, MPEG 120TH MEETING MACAU, OUTPUT DOCUMENT W17248 (N17248, 27 October 2017 (2017-10-27), pages 1 - 11, XP030023909, Retrieved from the Internet <URL:http://phenix.it-sudparis.eu/mpeg> [retrieved on 20180621] *

Cited By (44)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11935272B2 (en) 2017-09-14 2024-03-19 Apple Inc. Point cloud compression
US11552651B2 (en) 2017-09-14 2023-01-10 Apple Inc. Hierarchical point cloud compression
US11818401B2 (en) 2017-09-14 2023-11-14 Apple Inc. Point cloud geometry compression using octrees and binary arithmetic encoding with adaptive look-up tables
US11676309B2 (en) 2017-09-18 2023-06-13 Apple Inc Point cloud compression using masks
US11527018B2 (en) 2017-09-18 2022-12-13 Apple Inc. Point cloud compression
US11922665B2 (en) 2017-09-18 2024-03-05 Apple Inc. Point cloud compression
US11361471B2 (en) 2017-11-22 2022-06-14 Apple Inc. Point cloud occupancy map compression
US11514611B2 (en) 2017-11-22 2022-11-29 Apple Inc. Point cloud compression with closed-loop color conversion
US11533494B2 (en) 2018-04-10 2022-12-20 Apple Inc. Point cloud compression
US11727603B2 (en) 2018-04-10 2023-08-15 Apple Inc. Adaptive distance based point cloud compression
US11508094B2 (en) 2018-04-10 2022-11-22 Apple Inc. Point cloud compression
US11508095B2 (en) 2018-04-10 2022-11-22 Apple Inc. Hierarchical point cloud compression with smoothing
US12100183B2 (en) 2018-04-10 2024-09-24 Apple Inc. Point cloud attribute transfer algorithm
US11044478B2 (en) 2018-07-02 2021-06-22 Apple Inc. Compression with multi-level encoding
US11663744B2 (en) 2018-07-02 2023-05-30 Apple Inc. Point cloud compression with adaptive filtering
US11683525B2 (en) 2018-07-05 2023-06-20 Apple Inc. Point cloud compression with multi-resolution video encoding
US11647226B2 (en) 2018-07-12 2023-05-09 Apple Inc. Bit stream structure for compressed point cloud data
US12401822B2 (en) 2018-07-12 2025-08-26 Apple Inc. Bit stream structure for compressed point cloud data
US11386524B2 (en) 2018-09-28 2022-07-12 Apple Inc. Point cloud compression image padding
US11748916B2 (en) 2018-10-02 2023-09-05 Apple Inc. Occupancy map block-to-patch information compression
US11367224B2 (en) 2018-10-02 2022-06-21 Apple Inc. Occupancy map block-to-patch information compression
US12047604B2 (en) 2019-03-19 2024-07-23 Nokia Technologies Oy Apparatus, a method and a computer program for volumetric video
US11516394B2 (en) 2019-03-28 2022-11-29 Apple Inc. Multiple layer flexure for supporting a moving image sensor
US12439083B2 (en) 2019-07-02 2025-10-07 Apple Inc. Point cloud compression with supplemental information messages
WO2021011772A1 (fr) * 2019-07-16 2021-01-21 Apple Inc. Diffusion en continu de contenu de nuage de points volumétrique basée sur des protocoles de description de session et des protocoles en temps réel
CN114402571B (zh) * 2019-07-16 2025-03-11 苹果公司 基于会话描述协议和实时协议的体积点云内容的流式传输
US11588871B2 (en) 2019-07-16 2023-02-21 Apple Inc. Streaming of volumetric point cloud content based on session description protocols and real time protocols
CN114402571A (zh) * 2019-07-16 2022-04-26 苹果公司 基于会话描述协议和实时协议的体积点云内容的流式传输
WO2021053262A1 (fr) * 2019-09-20 2021-03-25 Nokia Technologies Oy Appareil, procédé et programme informatique pour vidéo volumétrique
US12069314B2 (en) 2019-09-20 2024-08-20 Nokia Technologies Oy Apparatus, a method and a computer program for volumetric video
US20220329871A1 (en) * 2019-09-20 2022-10-13 Nokia Technologies Oy An Apparatus, A Method and a Computer Program for Volumetric Video
US11627314B2 (en) 2019-09-27 2023-04-11 Apple Inc. Video-based point cloud compression with non-normative smoothing
US11562507B2 (en) 2019-09-27 2023-01-24 Apple Inc. Point cloud compression using video encoding with time consistent patches
US11538196B2 (en) 2019-10-02 2022-12-27 Apple Inc. Predictive coding for point cloud compression
US11895307B2 (en) 2019-10-04 2024-02-06 Apple Inc. Block-based predictive coding for point cloud compression
US11798196B2 (en) 2020-01-08 2023-10-24 Apple Inc. Video-based point cloud compression with predicted patches
US11625866B2 (en) 2020-01-09 2023-04-11 Apple Inc. Geometry encoding using octrees and predictive trees
US11631212B2 (en) 2020-03-02 2023-04-18 Qualcomm Incorporated Methods and apparatus for efficient multi-view rasterization
US11132831B1 (en) 2020-03-02 2021-09-28 Qualcomm Incorporated Methods and apparatus for efficient multi-view rasterization
US11615557B2 (en) 2020-06-24 2023-03-28 Apple Inc. Point cloud compression using octrees with slicing
US11620768B2 (en) 2020-06-24 2023-04-04 Apple Inc. Point cloud geometry compression using octrees with multiple scan orders
US11948338B1 (en) 2021-03-29 2024-04-02 Apple Inc. 3D volumetric content encoding using 2D videos and simplified 3D meshes
CN116468992B (zh) * 2023-06-20 2023-08-22 北京天图万境科技有限公司 一种重复校正性监督空间识别与还原方法与装置
CN116468992A (zh) * 2023-06-20 2023-07-21 北京天图万境科技有限公司 一种重复校正性监督空间识别与还原方法与装置

Similar Documents

Publication Publication Date Title
US12425571B2 (en) Apparatus, a method and a computer program for video coding and decoding
WO2019129923A1 (fr) Appareil, procédé et programme informatique pour vidéo volumétrique
US11523135B2 (en) Apparatus, a method and a computer program for volumetric video
US12088847B2 (en) Apparatus, a method and a computer program for video encoding and decoding
CN110870302B (zh) 用于全向视频的装置、方法和计算机程序
EP3614674A1 (fr) Appareil, procédé et programme informatique pour vidéo volumétrique
US11659151B2 (en) Apparatus, a method and a computer program for volumetric video
WO2020141260A1 (fr) Appareil, procédé et programme informatique de codage et de décodage vidéo
WO2019243663A1 (fr) Appareil, procédé et programme informatique pour vidéo volumétrique
US11651523B2 (en) Apparatus, a method and a computer program for volumetric video
JP2019513320A (ja) ビデオの符号化・復号装置、方法、およびコンピュータプログラム
WO2018087425A1 (fr) Appareil, procédé et programme informatique pour le codage et le décodage de vidéo
WO2019185985A1 (fr) Appareil, procédé et programme informatique pour vidéo volumétrique
WO2019234290A1 (fr) Appareil, procédé et programme d&#39;ordinateur pour vidéo volumétrique
WO2019197722A1 (fr) Appareil, procédé et programme informatique pour vidéo volumétrique
WO2020070378A1 (fr) Appareil, procédé et programme informatique pour vidéo volumétrique
EP3680859A1 (fr) Appareil, procédé et programme informatique pour vidéo volumétrique
HK40068430B (zh) 用於视频编码和解码的装置、方法
HK40061734A (en) An apparatus, a method and a computer program for video coding and decoding

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 18897747

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 18897747

Country of ref document: EP

Kind code of ref document: A1