WO2020122362A1 - Method for displaying 360-degree video including camera lens information and apparatus therefor - Google Patents

Abstract

A 360-degree image display method performed by a 360-degree video reception apparatus according to the present invention comprises the steps of: receiving 360-degree image data; acquiring information about an encoded picture and metadata from the 360-degree image data, wherein the metadata includes camera lens information; decoding a picture including a circular target region on the basis of the information about the encoded picture; displaying a display mode selection screen; processing the circular target region on the basis of a selected display mode and the camera lens information, and rendering the processed circular target region; and displaying a 360-degree video generated on the basis of the rendering, wherein the mode selection screen includes an icon indicating a PCI camera mode and an icon indicating a viewport video mode.

Description

Method and apparatus for displaying 360-degree video including camera lens information

The present invention relates to 360-degree video, and more particularly, to a method and apparatus for displaying a 360-degree video for 3DoF+ content including camera lens information and an interface for the 360-degree video.

The VR (Virtual Reality) system gives the user a sense of being in an electronically projected environment. The system for providing VR can be further improved to provide higher quality images and spatial sound. The VR system can allow a user to interactively consume VR contents.

3DoF+ (six Degrees of Freedom +) content can be used to provide a more diverse sensory experience by providing 3DoF or 360-degree videos that are newly formed according to the user's location movement through 360-degree videos of multiple viewpoints. have.

The technical problem of the present invention is to provide a method and apparatus for increasing the efficiency of VR video data transmission for providing a VR system.

Another technical problem of the present invention is to provide a method and apparatus for transmitting metadata about VR video data and VR video data.

Another technical problem of the present invention is to provide a method and apparatus for transmitting metadata about VR video data and camera lens information of VR video data.

The technical problem of the present invention is to provide a method and apparatus for increasing the efficiency of 360-degree video data transmission for providing a 3DoF+ system.

Another technical problem of the present invention is to provide a method and apparatus for transmitting metadata about a position and/or angle for a viewpoint/head position/eye view for providing a 3DoF+ system.

Another technical problem of the present invention is to provide a method and apparatus for correcting distortion by reflecting characteristics of a lens based on information related to distortion correction.

Another technical problem of the present invention is to provide a method and apparatus for correcting distortion by reflecting characteristics of a lens based on information indicating a radial distortion type.

Another technical problem of the present invention is to provide a method and apparatus for displaying a user interface for a display mode.

Another technical problem of the present invention is to provide a method and apparatus for displaying a user interface related to a plurality of viewpoints, head positions or eye views.

According to an embodiment of the present invention, a 360-degree video processing method performed by a 360-degree video transmission device is provided. The method includes obtaining a target circular area including a 360 degree image captured by a camera having at least one lens, mapping the target circular area to a picture, and encoding the picture to which the target circular area is mapped. Generating metadata for the 360-degree image, and performing processing for storing or transmitting the encoded picture and the metadata, wherein the metadata includes camera lens information. It is characterized by.

According to another embodiment of the present invention, a 360-degree video transmission apparatus for processing 360-degree video data is provided. The 360-degree video transmission apparatus includes a data input unit for obtaining a target circular region including a 360-degree image captured by a camera having at least one lens, a projection processing unit for mapping the target circular region to a picture, and the target circular region mapped A data encoder for encoding the picture, a metadata processing unit for generating metadata for the 360-degree image, and a transmission processing unit for performing processing for storing or transmitting the encoded picture and the metadata, wherein the The metadata is characterized by including camera lens information.

According to another embodiment of the present invention, a 360-degree image display method performed by a 360-degree video receiving apparatus is provided. The method includes receiving 360-degree image data, obtaining information and metadata about an encoded picture from the 360-degree image data, wherein the metadata includes camera lens information, and information on the encoded picture. Decoding a picture including a target circular region based on a display, displaying a display mode selection screen, processing and rendering the target circular region based on the selected display mode and the camera lens information, And displaying a 360-degree video generated based on the rendering, wherein the mode selection screen includes an icon representing a fisheye camera mode and an icon representing a viewport video mode.

According to another embodiment of the present invention, a 360-degree video receiving apparatus displaying a 360-degree image is provided. A receiver that receives 360-degree image data, obtains information and metadata about an encoded picture from the 360-degree image data, wherein the metadata is based on information about the encoded picture, a receiving processor comprising camera lens information, A data decoder for decoding a picture including a target circular region, a renderer for processing and rendering the target circular region based on the selected display mode and the camera lens information, and a display mode selection screen are displayed. It includes a display unit for displaying a 360-degree video generated based on the rendering, wherein the mode selection screen is characterized in that it includes an icon representing the fisheye camera mode and the icon representing the viewport video mode.

According to the present invention, 3DoF+ content can be efficiently transmitted in an environment that supports next-generation hybrid broadcasting using a terrestrial broadcasting network and an Internet network.

According to the present invention, a method for providing an interactive experience in a user's consumption of 3DoF+ content can be proposed.

According to the present invention, in a user's consumption of 3DoF+ content, it is possible to propose a method of signaling so that a 3DoF+ content creator's intention is accurately reflected.

According to the present invention, in the delivery of 3DoF+ content, it is possible to propose a method for efficiently increasing the transmission capacity and allowing necessary information to be delivered.

According to the present invention, it is possible to derive and render a circular area for a target viewpoint, target head position, and/or target eye view from 360-degree image data for 3DoF+ content based on camera lens information, through which the user's 3DoF+ It is possible to provide an interactive experience in content consumption.

According to the present invention, a polynomial function may be derived by reflecting a characteristic of a lens based on the projection function related information and/or the distortion correction function related information included in camera lens information, and the 360 degree based on the polynomial function It is possible to propose a method of more accurately mapping 360-degree image data into 3D space by correcting distortion generated in a picture in which an image is projected.

According to the present invention, a user interface for a display mode can be displayed, and through this, an interactive experience can be provided to a user who consumes 360 content.

According to the present invention, a user interface for a plurality of viewpoints, head positions or eye views can be displayed more efficiently.

1 is a diagram showing an overall architecture for providing 360-degree video according to the present invention.

2 and 3 are diagrams showing the structure of a media file according to an embodiment of the present invention.

4 shows an example of the overall operation of the DASH-based adaptive streaming model.

5 exemplarily shows the 3DoF+ VR system.

6 is a diagram showing the overall architecture for providing 3DoF+ video according to the present invention.

7 exemplarily shows an example of stitching a 360-degree video to a panoramic image based on camera lens information and/or additional camera lens information according to the present invention.

8A to 8B exemplarily illustrate an overall architecture for providing 360 content/3DoF+ content performed through a 360-degree video transmission device/360-degree video receiving device.

9 illustrates an example of processing a 360-degree video based on camera lens information in a 360-degree video receiving device.

10 is a diagram schematically illustrating a configuration of a 360-degree video transmission apparatus to which the present invention can be applied.

11 is a diagram schematically illustrating a configuration of a 360-degree video receiving apparatus to which the present invention can be applied.

12 exemplarily shows radial projection functions.

13 exemplarily shows various types of radial distortions.

14 shows an example of capturing a 360-degree video through a camera lens.

15 schematically shows a method of processing 360-degree video data by a 360-degree video transmission device according to the present invention.

16 schematically shows a 360-degree video transmission apparatus that performs a 360-degree image data processing method according to the present invention.

17 schematically shows a method of processing 360-degree video data by a 360-degree video receiving apparatus according to the present invention.

18 schematically shows a 360-degree video receiving apparatus for performing a 360-degree image data processing method according to the present invention.

19 schematically shows a method of displaying a 360-degree image by a 360-degree video receiving apparatus according to the present invention.

20 shows an example of a 360-degree video according to the display mode selection screen and display mode.

21 exemplarily shows a 360-degree video displayed.

22 schematically shows a 360-degree video receiving apparatus performing a 360-degree image display method according to the present invention.

The present invention can be applied to various changes and can have various embodiments, and specific embodiments will be illustrated in the drawings and described in detail. However, this is not intended to limit the invention to the specific examples. Terms used in this specification are only used to describe specific embodiments, and are not intended to limit the technical spirit of the present invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this specification, the terms “include” or “have” are intended to indicate that a feature, number, step, operation, component, part, or combination thereof described in the specification exists, or that one or more other features or It should be understood that numbers, steps, operations, components, parts, or combinations thereof do not preclude the presence or addition possibilities of those.

On the other hand, each component in the drawings described in the present invention is shown independently for convenience of description of different characteristic functions, does not mean that each component is implemented in separate hardware or separate software from each other. For example, two or more components of each component may be combined to form one component, or one component may be divided into a plurality of components. Embodiments in which each component is integrated and/or separated are also included in the scope of the present invention without departing from the essence of the present invention.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. Hereinafter, the same reference numerals are used for the same components in the drawings, and duplicate descriptions for the same components are omitted.

1 is a diagram showing an overall architecture for providing 360-degree video according to the present invention.

The present invention proposes a method of providing 360 content in order to provide a virtual reality (VR) to a user. VR may refer to a technology or environment for replicating a real or virtual environment. VR artificially provides the user with a sensuous experience, which allows the user to experience the same experience as being in an electronically projected environment.

360 content refers to overall content for implementing and providing VR, and may include 360-degree video and/or 360 audio. The 360-degree video may refer to video or image content necessary to provide VR, and simultaneously captured or played in all directions (360-degree). Hereinafter, a 360-degree video may mean a 360-degree video. The 360-degree video may mean a video or image displayed on various forms of 3D space according to a 3D model, and for example, the 360-degree video may be displayed on a spherical surface. 360 audio is also audio content for providing VR, and may mean spatial audio content that can be recognized as being located in a specific space in 3D. 360 content can be generated, processed, and transmitted to users, and users can consume the VR experience using the 360 content.

In particular, the present invention proposes a method for effectively providing 360-degree video. In order to provide 360-degree video, a 360-degree video may first be captured through one or more cameras. The captured 360-degree video is transmitted through a series of processes, and the receiving side can process the received data back into an original 360-degree video and render it. Through this, a 360-degree video can be provided to the user.

Specifically, the entire process for providing a 360-degree video may include a capture process, a preparation process, a transmission process, a processing process, a rendering process, and/or a feedback process.

The capturing process may refer to a process of capturing an image or video for each of a plurality of viewpoints through one or more cameras. Image/video data such as 110 of FIG. 1 shown by the capture process may be generated. Each plane of 110 shown in FIG. 1 may mean an image/video for each viewpoint. The captured multiple images/videos may be referred to as raw data. In the capture process, metadata related to capture may be generated.

A special camera for VR can be used for this capture. According to an embodiment, when a 360-degree video for a virtual space generated by a computer is to be provided, capture through a real camera may not be performed. In this case, the capturing process may be replaced by simply generating the relevant data.

The preparation process may be a process of processing the captured image/video and metadata generated during the capture process. The captured image/video may undergo a stitching process, a projection process, a region-wise packing process, and/or an encoding process in the preparation process.

First, each image/video may go through a stitching process. The stitching process may be a process of making a panoramic image/video or a spherical image/video by connecting each captured image/video.

Thereafter, the stitched image/video may be subjected to a projection process. In the projection process, stitched images/videos can be projected onto 2D images. This 2D image may be referred to as a 2D image frame depending on the context. Projecting a 2D image can also be expressed as mapping to a 2D image. The projected image/video data may be in the form of a 2D image as illustrated in FIG. 1 (120).

Video data projected on a 2D image may be subjected to region-wise packing in order to increase video coding efficiency and the like. Packing by region may mean a process of dividing and processing video data projected on a 2D image for each region. Here, the region may mean an area in which a 2D image in which 360-degree video data is projected is divided. Depending on the embodiment, these regions may be divided into evenly divided 2D images, or may be arbitrarily divided. Also, according to an embodiment, regions may be classified according to a projection scheme. The region-specific packing process is an optional process and may be omitted in the preparation process.

Depending on the embodiment, this processing may include rotating each region or rearranging on a 2D image to improve video coding efficiency. For example, by rotating regions so that specific sides of regions are positioned close to each other, efficiency in coding can be increased.

According to an embodiment, the process may include increasing or decreasing the resolution for a specific region in order to differentiate resolution for each region on a 360-degree video. For example, regions corresponding to regions that are relatively more important on a 360-degree video may have higher resolution than other regions. Video data projected on a 2D image or region-packed video data may be encoded through a video codec.

Depending on the embodiment, the preparation process may further include an editing process. In this editing process, editing of image/video data before and after projection may be further performed. Similarly in the preparation process, metadata about stitching/projection/encoding/editing, etc. can be generated. In addition, metadata regarding an initial viewpoint of a video data projected on a 2D image or a region of interest (ROI) may be generated.

The transmission process may be a process of processing and transmitting image/video data and metadata that have undergone a preparation process. For transmission, processing according to any transmission protocol may be performed. Data that has been processed for transmission may be transmitted through a broadcast network and/or broadband. These data may be delivered to the receiving side in an on demand manner. The receiving side can receive the corresponding data through various paths.

The processing process may mean a process of decoding the received data and re-projecting the projected image/video data on a 3D model. In this process, image/video data projected on 2D images may be re-projected onto 3D space. Depending on the context, this process can also be called mapping and projection. At this time, the 3D space to be mapped may have a different shape according to the 3D model. For example, a 3D model may have a sphere, cube, cylinder, or pyramid.

Depending on the embodiment, the processing process may further include an editing process, an up scaling process, and the like. In this editing process, editing of image/video data before and after re-projection may be further performed. When the image/video data is reduced, the size may be enlarged through upscaling of samples in the upscaling process. If necessary, the operation of reducing the size through downscaling may be performed.

The rendering process may refer to a process of rendering and displaying re-projected image/video data in 3D space. Depending on the expression, re-projection and rendering may be combined to render on a 3D model. The image/video re-projected onto the 3D model (or rendered onto the 3D model) may have the form shown in FIG. 1 130. The illustrated 130 of FIG. 1 is a case where the 3D model of a sphere is re-projected. The user can view some areas of the rendered image/video through a VR display or the like. At this time, the area viewed by the user may be in the form of (140) of FIG.

The feedback process may refer to a process of transmitting various feedback information that can be obtained in the display process to the transmitting side. Through the feedback process, interactivity may be provided in 360-degree video consumption. According to an embodiment, in the feedback process, head orientation information, viewport information indicating an area currently viewed by the user, and the like may be transmitted to the transmitting side. Depending on the embodiment, the user may interact with those implemented on the VR environment, in which case information related to the interaction may be transmitted from the sending side to the service provider side in the feedback process. Depending on the embodiment, the feedback process may not be performed.

The head orientation information may mean information about a user's head position, angle, and movement. Based on this information, information about an area that a user is currently viewing within a 360-degree video, that is, viewport information may be calculated.

The viewport information may be information about an area currently viewed by a user in a 360-degree video. Through this, Gaze Analysis may be performed to check how the user consumes the 360-degree video, which area of the 360-degree video, and how much gaze. Gaze analysis may be performed at the receiving side and transmitted to the transmitting side through a feedback channel. A device such as a VR display may extract a viewport area based on a user's head position/direction, vertical or horizontal FOV (Field Of View) information supported by the device, and the like.

Depending on the embodiment, the feedback information described above may not only be transmitted to the transmitting side, but may be consumed at the receiving side. That is, the decoding, re-projection, and rendering processes of the receiver may be performed using the above-described feedback information. For example, the head orientation information and/or viewport information may be used to preferentially decode and render only a 360-degree video for an area currently being viewed by a user.

Here, a viewport or a viewport area may mean an area that a user is viewing in a 360-degree video. A viewpoint is a point that a user is viewing in a 360-degree video, and may mean a center point of the viewport area. That is, the viewport is an area centered on a viewpoint, and a size shape occupied by the area may be determined by a field of view (FOV), which will be described later.

Within the entire architecture for providing the 360-degree video described above, image/video data that undergoes a series of processes of capture/projection/encoding/transfer/decoding/re-projection/rendering may be referred to as 360-degree video data. The term 360 degree video data may also be used as a concept including metadata or signaling information related to these image/video data.

In order to store and transmit media data such as audio or video described above, a standardized media file format may be defined. According to an embodiment, the media file may have a file format based on ISO BMFF (ISO base media file format).

2 and 3 are diagrams showing the structure of a media file according to an embodiment of the present invention.

The media file according to the present invention may include at least one box. Here, the box may be a data block or an object including media data or metadata related to media data. The boxes may form a hierarchical structure with each other, and accordingly, the data may be classified so that the media file has a form suitable for storage and/or transmission of a large amount of media data. In addition, the media file may have an easy structure for accessing media information, such as a user moving to a specific point in the media content.

Media files according to the present invention may include ftyp boxes, moov boxes and/or mdat boxes.

The ftyp box (file type box) may provide file type or compatibility information for a corresponding media file. The ftyp box may include configuration version information for media data of a corresponding media file. The decoder can identify the media file by referring to the ftyp box.

The moov box (movie box) may be a box including metadata about media data of a corresponding media file. The moov box can serve as a container for all metadata. The moov box may be a top-level box among metadata related boxes. Depending on the embodiment, there can be only one moov box in the media file.

The mdat box (media data box) may be a box containing actual media data of a corresponding media file. The media data can include audio samples and/or video samples, and the mdat box can serve as a container for these media samples.

According to an embodiment, the aforementioned moov box may further include a mvhd box, a trak box, and/or an mvex box as a sub box.

The mvhd box (movie header box) may include information related to media presentation of media data included in the corresponding media file. That is, the mvhd box may include information such as media creation time, change time, time specification, and duration of the media presentation.

The trak box (track box) may provide information related to tracks of corresponding media data. The trak box may include information such as stream-related information, presentation-related information, and access-related information for an audio track or video track. Multiple trak boxes may exist depending on the number of tracks.

The trak box may further include a tkhd box (track header box) as a sub box according to an embodiment. The tkhd box may include information on a corresponding track indicated by the trak box. The tkhd box may include information such as creation time, change time, and track identifier of the corresponding track.

The mvex box (movie extend box) may indicate that there may be a moof box to be described later in the corresponding media file. In order to know all the media samples of a particular track, moof boxes may have to be scanned.

The media file according to the present invention may be divided into a plurality of fragments according to an embodiment (200). Through this, the media file may be divided and stored or transmitted. The media data (mdat box) of the media file is divided into a plurality of fragments, and each fragment may include a moof box and a divided mdat box. According to an embodiment, information of a ftyp box and/or a moov box may be required to utilize fragments.

The moof box (movie fragment box) may provide metadata about media data of the corresponding fragment. The moof box may be a box of a top layer among metadata related boxes of a corresponding fragment.

The mdat box (media data box) may include actual media data as described above. The mdat box may include media samples of media data corresponding to each corresponding fragment.

According to an embodiment, the above-described moof box may further include an mfhd box and/or a traf box as a sub box.

The mfhd box (movie fragment header box) may include information related to association between a plurality of divided fragments. The mfhd box may include a sequence number to indicate the number of pieces of data in which the media data of the corresponding fragment is divided. In addition, it may be confirmed whether or not any of the divided data is missing using the mfhd box.

The traf box (track fragment box) may include information about the corresponding track fragment. The traf box may provide metadata for the segmented track fragment included in the corresponding fragment. The traf box may provide metadata so that media samples in a corresponding track fragment can be decoded/played. A plurality of traf boxes may exist depending on the number of track fragments.

According to an embodiment, the above-described traf box may further include a tfhd box and/or a trun box as a sub-box.

The tfhd box (track fragment header box) may include header information of a corresponding track fragment. The tfhd box may provide basic sample size, duration, offset, and identifier information for media samples of the track fragment indicated by the above-described traf box.

The trun box (track fragment run box) may include corresponding track fragment information. The trun box may include information such as duration, size, and playback time for each media sample.

The above-described media file or fragments of a media file may be processed as segments and transmitted. The segment may have an initialization segment and/or a media segment.

The file of the illustrated embodiment 210 may be a file including information related to the initialization of the media decoder, excluding media data. This file may, for example, correspond to the aforementioned initialization segment. The initialization segment may include the ftyp box and/or moov box described above.

The file of the illustrated embodiment 220 may be a file including the aforementioned fragment. This file may, for example, correspond to the media segment described above. The media segment may include the moof box and/or mdat box described above. In addition, the media segment may further include a styp box and/or a sidx box.

The styp box (segment type box) may provide information for identifying the media data of the segmented fragment. The styp box may perform the same function as the ftyp box described above for the segmented fragment. Depending on the embodiment, the styp box may have the same format as the ftyp box.

The sidx box (segment index box) may provide information indicating an index for a segmented fragment. Through this, it is possible to indicate how many fragments the corresponding fragment is.

According to an embodiment (230), an ssix box may be further included. The ssix box (sub-segment index box) may provide information indicating an index of the sub-segment when the segment is further divided into sub-segments.

The boxes in the media file may include more extended information based on a box or full box type, such as the illustrated embodiment 250. In this embodiment, the size field and the largesize field may indicate the length of the corresponding box in bytes. The version field may indicate the version of the corresponding box format. The Type field may indicate the type or identifier of the corresponding box. The flags field may indicate flags related to the corresponding box.

Meanwhile, fields (attributes) for 360-degree video of the present invention may be delivered by being included in a DASH-based adaptive streaming model.

4 shows an example of the overall operation of the DASH-based adaptive streaming model. The DASH-based adaptive streaming model according to the illustrated embodiment 400 describes an operation between an HTTP server and a DASH client. Here, DASH (Dynamic Adaptive Streaming over HTTP) is a protocol for supporting HTTP-based adaptive streaming, and can dynamically support streaming according to network conditions. Accordingly, AV content reproduction can be continuously provided.

First, the DASH client can acquire the MPD. MPD may be delivered from a service provider such as an HTTP server. The DASH client can request the corresponding segments to the server using access information to the segment described in the MPD. Here, this request can be performed by reflecting the network status.

After obtaining the corresponding segment, the DASH client can process it in the media engine and display it on the screen. The DASH client can request and acquire the required segment by reflecting the playback time and/or network conditions in real time (Adaptive Streaming). This allows content to be played seamlessly.

MPD (Media Presentation Description) is a file containing detailed information for allowing a DASH client to dynamically acquire a segment, and may be expressed in XML format.

The DASH client controller may generate a command for requesting an MPD and/or segment in response to a network condition. In addition, the controller can control the obtained information to be used in internal blocks of the media engine and the like.

The MPD parser can parse the acquired MPD in real time. Through this, the DASH client controller may be able to generate a command that can acquire the required segment.

The segment parser can parse the acquired segment in real time. Depending on the information included in the segment, internal blocks, such as a media engine, may perform a specific operation.

The HTTP client can request the required MPD and/or segment to the HTTP server. In addition, the HTTP client may deliver the MPD and/or segments obtained from the server to the MPD parser or segment parser.

The media engine may display content on the screen using media data included in the segment. At this time, the information of the MPD can be utilized.

The DASH data model may have a hierarchical structure 410. The media presentation can be described by MPD. MPD may describe a temporal sequence of a plurality of periods (Period) making a media presentation. The period may represent a section of media content.

In one section, data may be included in adaptation sets. The adaptation set may be a set of a plurality of media content components that can be exchanged with each other. The adaptation can include a set of representations. The representation may correspond to a media content component. Within one representation, the content can be divided temporally into a plurality of segments. This may be for proper accessibility and delivery. To access each segment, the URL of each segment can be provided.

The MPD may provide information related to the media presentation, and the period elements, adaptation set elements, and presentation elements may describe the corresponding period, adaptation set, and presentation, respectively. The representation may be divided into sub-representations, and the sub-representation element may describe the corresponding sub-representation.

Here, common properties/elements can be defined, and they can be applied (included) to adaptation sets, representations, sub-representations, and the like. Among the common properties/elements, there may be an essential property (EssentialProperty) and/or a supplemental property (SupplementalProperty).

The essential property may be information including elements that are considered essential in processing data related to the media presentation. The supplemental property may be information including elements that may be used in processing corresponding media presentation-related data. Descriptors to be described later according to an embodiment may be defined and delivered in an essential property and/or a supplemental property when delivered through an MPD.

Meanwhile, the present invention provides a method of providing 3 DoF+ (3 Degrees of Freedom+) contents in addition to the method of providing the above-described 360-degree content in order to provide the user with an immersive media/Immersive media experience. Suggest.

In the above-described existing 3DoF-based VR system (that is, the above-described existing 360-degree content-based system), the user was provided with a visual/aural experience of different viewing directions in the user's fixed position, whereas 3DoF+-based In the VR system, it aims to provide an extended visual/aural experience for different directions at different viewpoints and different viewing positions. That is, the 3DoF+ based VR system may represent a system that provides 360-degree content that is rendered to a plurality of viewing positions at a plurality of viewpoints.

The concept of location and viewpoint additionally defined in the 3DoF+ VR system can be explained as shown in the following figure.

5 exemplarily shows the 3DoF+ VR system.

Specifically, referring to (a) of FIG. 5, a plurality of viewpoints in which 360-degree content is provided in the 3DoF+ VR system is exemplarily shown. For example, as illustrated in (a) of FIG. 5, a plurality of specific locations in a specific space (such as a performance hall) may be considered as viewpoints provided with the 360-degree content. In this case, it may be assumed that the video/audio provided by each of the viewpoints existing in the same specific space has the same time flow.

Meanwhile, 360-degree content for a plurality of viewing positions may be rendered in a specific viewpoint. Accordingly, different visual/acoustic experiences may be provided according to a user's head motion at the specific viewpoint. Here, the 360-degree content may be referred to as 3DoF+ content, and the 3DoF+ content may include 3DoF+ video and 3DoF+ audio.

Meanwhile, for example, the value of the camera_id_type[i] field in the 3DoF+ content shown in FIG. 5(a) may be designated as 1. Here, the camera_id_type[i] field may indicate the purpose, embodiment, or type of the i-th camera. A detailed description of the camera_id_type[i] field will be described later.

When the value of the camera_id_type[i] field is 1, another head position supporting a 3DoF+ application at a specific viewpoint may be defined, through which a head motion parallax is generated. Can be supported.

Also, a value of the camera_id_type[i] field may be designated as 2. When the value of the camera_id_type[i] field is 2, a head position and a head direction may be provided. In addition, in supporting a binocular disparity in a 3DoF+ application, a left or right eye view subpicture may be displayed by using a value of the camera_id_type[i] field as 3. That is, when the value of the camera_id_type[i] field is 3, the camera_id_type[i] field may indicate a left or right eye view subpicture in which binocular disparity is supported.

Meanwhile, a post-processor such as a stitching or rendering processor processes the picture through camera lens information such as a camera_id[i] field and a corresponding camera_id_type[i] field for the 3DoF+ content. can do. The stitching or rendering processor may select a sub picture according to a specific purpose using the camera lens information described above. For example, one view for a monoscopic display may be displayed based on the camera lens information described above, and selectively processed according to a specific viewpoint or head position.

Another example of application of the camera lens information may be generating a panoramic image. The panoramic image may be generated based on images captured through a plurality of cameras at the same time or images captured at multiple locations with a single camera. Further, the position and lens characteristics of each camera among the plurality of cameras may be displayed based on the camera lens information in a light field video captured through a plurality of cameras in an array.

5B illustrates 3D spaces of a plurality of viewing positions at a specific viewpoint. The 3D space may be a sphere. Different visual and auditory experiences may be provided according to a user's point of view head motion at the specific viewpoint, so that video/voice/text information reflecting the relative position of each viewing position will be provided. Can be.

In addition, visual/aural information in various directions, such as existing 360-degree content, may be transmitted at a specific viewpoint of a specific viewpoint. That is, 360-degree content for the specific viewpoint of the specific viewpoint may be rendered in 3D space. In this case, not only a main source including video/voice/text, etc., but also various additional sources may be integrated and provided, and information about the additional source may be linked to or independent of the user's viewing orientation. Can be delivered to

FIG. 5( c) exemplarily shows a 3D space in which a 360-degree video of a viewing position is rendered. As shown in Fig. 5(c), each point on the spherical surface uses a spherical coordinate system, r (radius of the sphere), θ (rotation direction and degree based on the z axis), φ (rotation toward the z axis of the xy plane) Direction and degree). According to an embodiment, the spherical surface may match the world coordinate system, or the principal point of the front camera may be assumed to be the point (r, 0, 0) of the spherical surface.

On the other hand, the position of each point on the spherical surface can be represented based on the concept of the aircraft main axis (Aircraft Principal Axes). For example, the position of each point on the spherical surface can be expressed through pitch, yaw and roll.

The plane can rotate freely in three dimensions. The three-dimensional axes are called a pitch axis, a yaw axis, and a roll axis, respectively. In this specification, these can be abbreviated to be expressed as pitch, yaw, roll to pitch direction, yaw direction, roll direction. The pitch axis may mean an axis that is a reference for a direction in which the front nose of the airplane rotates up/down. In the illustrated airplane main axis concept, the pitch axis may mean an axis extending from the wing of the airplane to the wing. The yaw axis may mean an axis that serves as a reference for a direction in which the front nose of the airplane rotates left/right. In the illustrated airplane main axis concept, the yaw axis may mean an axis extending from the top to the bottom of the airplane. The roll axis is an axis from the nose of the airplane to the tail in the concept of the main axis of the airplane, and rotation in the roll direction may mean rotation based on the roll axis. As described above, the 3D space in the present invention can be described through the concept of pitch, yaw, and roll. Depending on the embodiment, the X, Y, Z axis concept or a method using a spherical coordinate system may be used.

6 is a diagram showing the overall architecture for providing 3DoF+ video according to the present invention.

Referring to FIG. 6, 3DoF+ video data and/or 3DoF+ audio data may be acquired (Acquisition). Specifically, a High Density Camera Array (HDCA), a Lenslet (microlens) camera, etc. may be used to capture the 3DoF+ content, or may be obtained through a new device designed for 3DoF+ video capture. A plurality of image/video data sets generated according to the captured camera position, such as the acquired image 610 illustrated in FIG. 6, may be generated. That is, a plurality of image/audio information according to head motions at a plurality of locations can be obtained. In this case, the image information may include depth information as well as visual information. As shown in the acquired image 610 shown in FIG. 6, a plurality of pieces of information of different viewing positions according to different viewing positions may be respectively obtained. In addition, a metadata indicating internal/external setting values of the camera may be generated in the process of capturing the 3DoF+ content. Meanwhile, in the case of a computer-generated image other than a camera, the capture process may be replaced.

When the image acquisition (Acquisition) process is performed, a composition (Composition) process may be performed. In the composition process, an image 610 obtained through an image/audio input device and an image (video/image, etc.), voice (audio/effect sound, etc.), text (subtitles, etc.) input through an external media are added to the user experience. It can be defined as the process of synthesis for inclusion.

The pre-processing process of the acquired image 610 may be a process of processing the captured image/video and metadata transferred in the capturing process. The pre-processing process is a stitching process, a color correction process, a projection process, and a separation of time points that are separated into a primary view and a secondary view to improve coding efficiency ( view segmenation) process and encoding process.

Specifically, the stitching process may be a process of creating an image/video connecting images captured in a 360 direction from a position of each camera in a panoramic or spherical shape centering on each camera position.

Thereafter, the stitched image/video may be subjected to a projection process. The projection process may refer to a process of deriving the projected picture 620 by projecting each stitched image as a 2D image. Here, the projection may be expressed as mapping to a 2D image. Images mapped at each camera position can be separated into a main view and a sub view, and different resolutions can be applied for each view to improve video coding efficiency. By varying, efficiency can be improved when coding. The image separation process according to the viewpoint may be referred to as a view segmentation process. Also, the secondary viewpoint may not exist depending on the capture environment. The secondary view may mean an image/video that needs to be played in the process of moving when the user moves from the main view to another main view, and may have a lower resolution than the main view, but may have the same resolution as necessary. have. Also, in some cases, the secondary view may be newly generated as virtual information in the receiver.

Depending on the embodiment, the pre-processing process may further include an editing process. The editing process may indicate a process of removing a boundary between regions of a 360-degree video, reducing color/brightness difference, or adding a visual effect of an image. In addition, the pre-processing process may include a packing process of rearranging images according to regions, and an encoding process of compressing image information. Like the projected picture 620, a projected picture may be generated based on a plurality of projected images of different viewing positions according to different viewing positions.

In addition, in the pre-processing process, editing of image/video data before and after projection may be further performed, and metadata may be generated. In addition, in the pre-processing process, when providing an image/video, metadata regarding an initial time point, an initial location of a user, and a region of interest (ROI) may be generated.

The media transmission process (Delivery) illustrated in FIG. 6 may represent a process of processing and transmitting image/video data and metadata obtained in the pre-processing process. For transmission of the image/video data and the metadata, processing according to any transmission protocol may be performed, and the pre-processed data may be transmitted through a broadcast network and/or broadband. In addition, the pre-processed data can be delivered to the receiving side on demand.

The processing process includes decoding the received image/video data and metadata and re-projection mapping or projecting the decoded projected picture image/video data into a three-dimensional (3D) model. It may include all processes prior to image creation to reproduce images/videos, such as the process, creation and synthesis of virtual viewpoints. The 3D model or projection map to be mapped may be a sphere, cube, cylinder, or pyramid like a conventional 360-degree video. Alternatively, the 3D model or the projection map may be a modified form of an existing 360-degree video projection map, or in some cases a free-form projection map.

Here, the process of generating and synthesizing a virtual view may represent a process of generating and synthesizing image/video data that should be reproduced when a user moves between the main view and the sub view or between the main view and the main view. In order to generate a virtual viewpoint, a process of processing the metadata transferred in the capture and preprocessing process may be necessary, and in some cases, only a part of the 360 images/videos in the virtual viewpoint may be generated/composited.

Meanwhile, according to an embodiment, the processing process may further include an editing process, an up scaling, a down scaling process, and the like. In the editing process, an additional editing process required before playback may be applied after the processing process. If necessary, an upscaling or downscaling of the transmitted image/video may be performed.

The rendering process may represent a process of rendering to display a reprojected image/video transmitted or generated. Sometimes the rendering and reprojection process is collectively called rendering. Therefore, a reprojection process may be included in the rendering process. The re-projection is formed in the form of (630) of FIG. 6 in which a user-oriented 360-degree video/image and a 360-degree video/image formed around each of the positions moved by the user according to the direction of movement are formed. There may be projection results. The user may view a partial area of the 360-degree video/image according to the device to be displayed, and the area that the user sees may be derived in the form of 640 of FIG. 6. In addition, when the user moves, the entire 360-degree video/image may not be rendered, but only the image corresponding to the position the user is viewing may be rendered. In addition, the 360-degree video receiving device may additionally render a video/image of a location to be moved (that is, a location predicted to move) by receiving metadata about the user's location and movement.

The feedback process may represent a process of transmitting various feedback information that can be obtained in the display process to the transmitting side. Through the feedback process, interactivity between 360-degree content and the user may occur, and according to an embodiment, the user's head and position position information (head/position orientation) and the area the user is currently viewing in the feedback process ) May be transmitted. The information may be transmitted to the transmitting side or the service provider in the feedback process, and depending on the embodiment, the feedback process may not be performed.

The user's location information may mean information about the user's head position, angle, movement, and moving distance, and the viewport information viewed by the user may be calculated based on the information.

7 exemplarily shows an example of stitching a 360-degree video to a panoramic image based on camera lens information and/or additional camera lens information according to the present invention.

8A to 8B exemplarily illustrate an overall architecture for providing 360 content/3DoF+ content performed through a 360-degree video transmission device/360-degree video receiving device.

360 content/3DoF+ content may be provided by an architecture as shown in FIGS. 8A to 8B. 3DoF+ content may be provided in the form of a file, or may be provided in the form of a segment-based download or streaming service such as DASH.

Specifically, referring to FIG. 8A, 360-degree video data and/or 360-degree audio data of 3DoF+ content may be obtained as described above (Acquisition). That is, a 360-degree video may be captured through a 360 camera, and a 360-degree video transmission device may acquire the 360-degree video data. In the part that performs the information acquisition of the 360-degree video transmission device, different information is simultaneously or continuously acquired according to the sensor orientation, the sensor acquisition point, and the sensor acquisition point. can do. In addition, in the case of an image, in the part where information is acquired by the 360-degree video transmission device, image information according to a viewing orientation, a viewing position, and a viewpoint can be simultaneously or continuously acquired. In this case, the video information may include video, image, audio, and location information.

In addition, the 360-degree audio data may go through an audio pre-processing process and an audio encoding process. In this process, audio-related metadata may be generated, and encoded audio and audio-related metadata may be processed for transmission (file/segment encapsulation).

The 360 degree video data may go through the same process as described above. 8A, a stitcher of a 360-degree video transmission device may perform stitching on 360-degree video data (Visual stitching). For example, the 360-degree video data may include texture information and depth information, and the 360-degree video transmission device may acquire the texture information and the depth information, respectively, and characteristics of each component According to this, different pre-processing may be performed on the texture information and the depth information. For example, in the case of texture information, the 360-degree video transmission apparatus uses 360 images of different orientations of the viewing position obtained at the same position using the image sensor position information. Also, an omnidirectional image may be configured, and for this, an image stitching process may be performed. This process may be omitted according to embodiments and may be performed at the receiving side.

In addition, referring to FIG. 8A, the projection processing unit of the 360-degree video transmission device may project 360-degree video data on a 2D image (Projection and mapping (packing)). The projection processing unit may receive 360-degree video data (Input Images), and in this case, perform stitching and projection processes. In the projection process, specifically, stitched 360-degree video data can be projected onto a 3D space, and the projected 360-degree video data can be viewed as being arranged on a 2D image. In the present specification, this process may be expressed as projecting 360-degree video data onto a 2D image. Here, the 3D space may be a sphere or a cube. This 3D space may be the same as the 3D space used for re-projection at the receiving side.

The 2D image may also be referred to as a projected frame or a projected picture. In addition, a region-wise packing process may be selectively performed on the 2D image. When the region-specific packing process is performed, regions on the 2D image may be mapped onto a packed frame by indicating the location, shape, and size of each region. The packed frame may be referred to as a packed picture. When the region-specific packing process is not performed on the projected frame, the projected frame may be the same as the packed frame. The region will be described later. The projection process and the region-specific packing process may be expressed as each region of 360-degree video data being projected on a 2D image. Depending on the design, 360-degree video data may be directly converted into a packed frame without intermediate processing.

In addition, in general, a depth image may be acquired through a depth camera, and in this case, a depth image may be generated in the same form as a texture image. Alternatively, depth data may be generated based on separately measured data. After the component-specific images are generated, the 360-degree video transmission apparatus may perform sub-picture generation by performing additional packaging to a video format for efficient compression or dividing it into parts that are actually required.

In addition, when additionally provided video/audio/text information is provided in addition to the acquired image data (or data for mainly servicing), information for synthesizing the additionally provided information at the time of final playback may be generated and provided Can be. For example, in the composition generation stage of the 360-degree video transmission device, media data generated externally based on the intention of the producer (in the case of additionally provided information is video, video/image information, voice, audio / Data for effect sound, text, and subtitles, etc.) may be generated for synthesis in the final reproduction stage, and the information may be transmitted as composition metadata.

Referring to FIG. 8A, a packed frame for 360-degree video data may be image-encoded or video-encoded. Meanwhile, even the same 3DoF+ content may have 360-degree video data for each viewpoint, and in this case, 360-degree video data for each viewpoint of the content may be encoded in different bit streams. The encoded 360-degree video data can be processed in a file format such as ISOBMFF by the above-described encapsulation processing unit. Alternatively, the encapsulation processor may process the encoded 360-degree video data into segments. Segments can be included in individual tracks for DASH based transmission.

With the processing of 360-degree video data, metadata related to 360-degree video can be generated as described above. This metadata can be delivered as part of a video stream or file format. This metadata can also be used in processes such as the encoding process, file format encapsulation, and processing for transmission.

The 360 audio/video data is processed for transmission according to a transmission protocol, and then transmitted. The above-described 360-degree video receiving device may receive it through a broadcast network or broadband.

Meanwhile, as illustrated in FIG. 8A, loudspeakers/headphones, displays, and head/eye tracking components may be performed by external devices or VR applications of a 360-degree video receiving device. Depending on the embodiment, the 360-degree video receiving device may include all of the speaker/headphone, the display, and the head/eye tracking component. According to an embodiment, the head/eye tracking component may correspond to the above-described feedback processing unit.

The 360-degree video receiving apparatus may perform processing (File/segment decapsulation) for receiving 360 audio/video data. The 360 audio data may be provided to a user through a speaker/headphone through an audio decoding and audio rendering process.

The 360-degree video data may be provided to a user through a display through an image decoding, video decoding, and rendering process. Here, the display may be a display supporting VR or a general display.

As described above, the rendering process may be specifically viewed as 360-degree video data being re-projected onto a 3D space, and re-projected 360-degree video data being rendered. It can also be expressed that the 360-degree video data is rendered in 3D space.

The head/eye tracking component may acquire and process a user's head orientation information, gaze information, viewport information, and the like. Content related to this may be as described above.

On the receiving side, there may be a VR application communicating with the above-described receiving-side processes.

In addition, FIG. 8B exemplarily shows a 2D image to which a packaging process for each region is applied according to a processing process and a projection scheme of 360-degree video. Referring to FIG. 8B, a process of processing input 360-degree video data may be illustrated. Specifically, referring to FIG. 8B, 360-degree video data of the input viewpoint can be stitched and projected into a 3D projection structure according to various projection schemes, and the 360-degree video data projected into the 3D projection structure can be represented as a 2D image. have. That is, the 360-degree video data can be stitched and projected into the 2D image. The 2D image in which the 360-degree video data is projected may be represented as a projected frame. Also, the above-described packing process for each region may be performed on the projected frame. That is, processing such as dividing an area including the projected 360-degree video data on the projected frame into regions, rotating or rearranging each region, or changing the resolution of each region may be performed. In other words, the region-specific packing process may indicate a process of mapping the projected frame to one or more packed frames. Performing the region-specific packing process may be optional, and when the region-specific packing process is not applied, the packed frame and the projected frame may be the same. When the region-specific packing process is applied, each region of the projected frame may be mapped to a region of the packed frame, and the location and shape of a region of the packed frame to which each region of the projected frame is mapped. And metadata indicating the size can be derived.

9 illustrates an example of processing a 360-degree video based on camera lens information in a 360-degree video receiving device.

Referring to FIG. 9, a spherical image or spherical images may be generated from subpictures extracted from the decoded picture. Is generated from sub-pictures extracted from the decoded picture.

For example, referring to FIG. 9, an active area corresponding to an intersection of a circular image and a rectangular area may be extracted as the spherical image or a sub-picture for generating spherical images. The spherical image and the rectangular region include the circular_region_center_x[i] field, circular_region_center_y[i] field, rect_region_top[i] field, rect_region_left[i] field, rect_region_width[i] field, rect_region_height[i] field, full_radius[i] field, It can be derived based on the scene_radius[i] field. The circular_region_center_x[i] field, the circular_region_center_y[i] field, the rect_region_top[i] field, the rect_region_left[i] field, the rect_region_width[i] field, the rect_region_height[i] field, the full_radius[i] field, the The detailed description of the scene_radius[i] field will be described later.

Also, referring to FIG. 9, redial distortion of the sub-picture due to a lens defect may be corrected based on a model derived based on a redial_distortion_type[i] field. Subsequently, the subpicture has a modeled projection equation or angle_projection[i][j] field and a polynomial_coeff_projection[i][j][k] field derived based on the lens_projection_type[i] field. It can be mapped through a projection equation derived by a polynomial coefficient that is a function of the angles shown. The redial_distortion_type[i] field, the lens_projection_type[i] field, the angle_projection[i][j] field, and the polynomial_coeff_projection[i][j][k] field will be described later in detail.

In addition, lens distortion may be corrected based on given parameters (angle_correction[i][j] field, polynomial_coeff_correction[i][j][k] field).

On the other hand, if there is rotation and offset for an image captured by a specific lens (ie, a decoded picture), the mapped spherical image has a local_sphere_rotation_azimuth[i] field, a local_sphere_rotation_elevation[i] field, and a local_sphere_rotation_tilt[i] field. Based on this, it can be rotated relatively in the global spherical coordinate system. In addition, the sphere center offset (sphere center offset) is based on the local_sphere_rotation_azimuth[i] field, local_sphere_rotation_elevation[i] field, local_sphere_rotation_tilt[i] field, the lens constituting the unit sphere of the eye view of the head position of the viewpoint (unit sphere) It can be corrected to match the sphere center corresponding to. A detailed description of the local_sphere_rotation_azimuth[i] field, the local_sphere_rotation_elevation[i] field, and the local_sphere_rotation_tilt[i] field will be described later.

Meanwhile, as described above, the stitching process may be performed as a spherical image. For example, if the decoded picture includes subpictures of different viewpoints and head positions, the camera_id[i] field and camera_id_type to extract subpictures for a specific viewpoint, head position and/or eye view The [i] field can be used. For example, in the case of viewport dependent processing, the camera_id[i] field, the camera_id_type[i] field, the field of view (FOV) information and/or reduction of processing time and delay based on rotation information From the viewpoint of sub-pictures that can increase the efficiency can be selected.

10 is a diagram schematically illustrating a configuration of a 360-degree video transmission apparatus to which the present invention can be applied.

The 360-degree video transmission apparatus according to the present invention may perform operations related to the above-described preparation process or transmission process. The 360-degree video transmission device includes a data input unit, a composition information processing unit, a stitcher, a projection processing unit, a (regional) packing processing unit, a sub-picture processing unit, a data encoder, an encapsulation processing unit, a file extraction unit, a transmission processing unit, and a transmission unit. , A viewpoint/viewing position/viewing orientation information and a metadata processing unit and/or a (transmission-side) feedback processing unit as internal/external elements. When the input data is a camera output image, the 360-degree video transmission device may perform stitching for constructing a sphere image (ie, a 360-degree image rendered in 3D space) for each viewpoint/viewpoint/component. The viewpoint/viewpoint/direction information and metadata processing unit may be referred to as a metadata processing unit.

The data input unit may receive images/videos for each captured viewpoint. These viewpoint-specific images/videos may be images/videos captured by one or more cameras. That is, the images/videos for each viewpoint may include images/videos for a plurality of viewpoints. Also, the data input unit may receive metadata generated during the capture process. The data input unit may transmit the input image/video for each viewpoint to the stitcher, and transmit metadata of the capture process to the signaling processing unit.

The stitcher may perform stitching on captured images/videos by viewpoint. The stitcher may deliver the stitched 360-degree video data to the projection processing unit. If necessary, the stitcher can receive necessary metadata from the metadata processing unit and use it for stitching. The stitcher may transmit metadata generated in the stitching process to the metadata processing unit. The metadata of the stitching process may include information such as whether stitching has been performed or not, and the stitching type.

The projection processing unit may project stitched 360-degree video data onto a 2D image. The projection processing unit may perform projection according to various schemes, which will be described later. The projection processing unit may perform mapping in consideration of a corresponding depth of 360-degree video data for each viewpoint. If necessary, the projection processing unit may receive metadata required for projection from the metadata processing unit and use it for projection work. The projection processing unit may transmit metadata generated in the projection process to the metadata processing unit. The metadata of the projection processing unit may include a type of projection scheme and the like.

(By region) The packing processing unit may perform the above-described packing process for each region. That is, the region-specific packing processing unit may perform processing such as dividing the projected 360-degree video data into regions, rotating and rearranging each region, or changing the resolution of each region. As described above, the packing process for each region is an optional process, and when the packing for each region is not performed, the packing process for each region may be omitted. If necessary, the region-specific packing processing unit may receive metadata required for region-specific packing from the metadata processing unit and use the region-specific packing operation. The region-specific packing processing unit may transmit metadata generated in the region-specific packing process to the metadata processing unit. The region-specific packing processing unit metadata may include a rotation degree and a size of each region.

The above-described stitcher, projection processing unit and/or region-specific packing processing unit may be performed in one hardware component according to an embodiment.

Depending on the application, the sub-picture processing unit may generate a sub-picture that divides a plurality of images into an image of a packing or a detailed area to make an integrated image. In addition, when the input data includes video/voice/text additional information, information on a method of adding and displaying the additional information to the central image may be generated, and the information may be transmitted together with the additional information. have.

The metadata processing unit may process metadata that may occur in a capture process, a stitching process, a projection process, a region-specific packing process, an encoding process, an encapsulation process, and/or a process for transmission. The metadata processing unit may generate 360-degree video-related metadata using these metadata. According to an embodiment, the metadata processor may generate 360-degree video-related metadata in the form of a signaling table. Depending on the signaling context, the 360-degree video-related metadata may be referred to as metadata or 360-degree video-related signaling information. In addition, the metadata processing unit may transmit the acquired or generated metadata to internal elements of the 360-degree video transmission device as needed. The metadata processing unit may transmit the 360-degree video-related metadata to the data encoder, the encapsulation processing unit and/or the transmission processing unit so that it can be transmitted to the receiving side.

The data encoder may encode 360-degree video data projected on a 2D image and/or packed-by-region 360-degree video data. 360-degree video data can be encoded in a variety of formats.

The encapsulation processing unit may encapsulate the encoded 360-degree video data and/or 360-degree video-related metadata in the form of a file. Here, the 360-degree video-related metadata may be received from the metadata processing unit described above. The encapsulation processing unit may encapsulate the data in a file format such as ISOBMFF, CFF, or other DASH segments. According to an embodiment, the encapsulation processing unit may include 360-degree video related metadata on a file format. The 360-related metadata may be included, for example, in various levels of boxes on the ISOBMFF file format or as data in separate tracks in the file. According to an embodiment, the encapsulation processing unit may encapsulate the 360-degree video related metadata itself into a file. The transmission processing unit may apply processing for transmission to 360-degree encapsulated video data according to a file format. The transmission processing unit may process 360-degree video data according to any transmission protocol. The processing for transmission may include processing for delivery through a broadcast network, and processing for delivery through a broadband. According to an embodiment, the transmission processing unit may receive not only 360-degree video data, but also 360-degree video-related metadata from the metadata processing unit, and may apply processing for transmission thereon.

The transmission unit may transmit the processed 360-degree video data and/or 360-degree video-related metadata through a broadcast network and/or broadband. The transmission unit may include an element for transmission over a broadcast network and/or an element for transmission over broadband.

According to an embodiment of the 360-degree video transmission device according to the present invention, the 360-degree video transmission device may further include a data storage unit (not shown) as an internal/external element. The data storage unit may store the encoded 360-degree video data and/or 360-degree video-related metadata before transmitting it to the transmission processing unit. The format in which these data are stored may be a file format such as ISOBMFF. When transmitting a 360-degree video in real time, a data storage unit may not be required, but when transmitting through on-demand, NRT (Non Real Time), broadband, etc., encapsulated 360 data is stored in the data storage unit for a certain period of time. It can also be stored and transmitted.

According to another embodiment of the 360-degree video transmission apparatus according to the present invention, the 360-degree video transmission apparatus may further include a (transmission-side) feedback processing unit and/or a network interface (not shown) as internal/external elements. The network interface may receive feedback information from the 360-degree video receiving apparatus according to the present invention, and transmit it to the transmitting-side feedback processing unit. The transmission-side feedback processing unit may transmit feedback information to the stitcher, projection processing unit, region-specific packing processing unit, data encoder, encapsulation processing unit, metadata processing unit, and/or transmission processing unit. According to an embodiment, the feedback information may be transmitted once to the metadata processing unit and then to each internal element again. Internal elements that receive feedback information may reflect the feedback information in subsequent processing of 360-degree video data.

According to another embodiment of the 360-degree video transmission device according to the present invention, the region-specific packing processing unit may rotate each region to map on a 2D image. At this time, each region may be rotated at different directions and at different angles and mapped on the 2D image. The rotation of the region may be performed in consideration of the portion where the 360-degree video data was adjacent to the spherical surface before projection, the stitched portion, and the like. Information about the rotation of the region, that is, the rotation direction, angle, etc., may be signaled by 360-degree video related metadata. According to another embodiment of the 360-degree video transmission apparatus according to the present invention, the data encoder may encode differently for each region. The data encoder may perform encoding in a specific region with high quality and in other regions with low quality. The transmitting-side feedback processing unit may transmit feedback information received from the 360-degree video receiving device to a data encoder, so that the data encoder uses a region-specific differential encoding method. For example, the transmitting-side feedback processing unit may transmit the viewport information received from the receiving side to the data encoder. The data encoder may perform encoding for regions including the region indicated by the viewport information with higher quality (UHD, etc.) than other regions.

According to another embodiment of the 360-degree video transmission apparatus according to the present invention, the transmission processing unit may perform processing for transmission differently for each region. The transmission processing unit may apply different transmission parameters (modulation order, code rate, etc.) for each region, so that the robustness of data transmitted for each region may be different.

At this time, the transmitting-side feedback processing unit may transmit the feedback information received from the 360-degree video receiving device to the transmitting processing unit, so that the transmitting processing unit performs the differentiated transmission processing for each region. For example, the transmission-side feedback processing unit may transmit viewport information received from the reception side to the transmission processing unit. The transmission processing unit may perform transmission processing for regions including a region indicated by the corresponding viewport information to have higher robustness than other regions.

The inner/outer elements of the 360-degree video transmission device according to the present invention described above may be hardware elements implemented in hardware. Depending on the embodiment, internal/external elements may be changed, omitted, or replaced with other elements. According to an embodiment, additional elements may be added to the 360-degree video transmission device.

11 is a diagram schematically illustrating a configuration of a 360-degree video receiving apparatus to which the present invention can be applied.

The 360-degree video receiving apparatus according to the present invention may perform operations related to the above-described processing process and/or rendering process. The 360-degree video receiving device includes a receiving unit, a receiving processing unit/file extraction unit, a decapsulation processing unit, a data decoder, a metadata parser, an inpacking/selection processing unit, a renderer, a composition processing unit, a (receiving side) feedback processing unit and/or re-projection The processing portion may be included as an inner/outer element. Meanwhile, the signaling parser may be referred to as a metadata parser.

The receiver may receive 360-degree video data transmitted by the 360-degree video transmitting apparatus according to the present invention. Depending on the channel being transmitted, the receiver may receive 360-degree video data through a broadcast network or 360-degree video data through broadband. The receiving unit may extract a necessary file after receiving the bitstream transmitted from the transmitting unit.

The reception processing unit may perform processing according to a transmission protocol on the received 360-degree video data. The reception processing unit may perform the reverse process of the above-described transmission processing unit so that the transmission side corresponds to the processing for transmission. The receiving processing unit may transmit the obtained 360-degree video data to the decapsulation processing unit, and transmit the obtained 360-degree video-related metadata to a metadata parser. The 360-degree video-related metadata obtained by the reception processing unit may be in the form of a signaling table.

The decapsulation processing unit may decapsulate 360-degree video data in a file format received from the reception processing unit. The decapsulation processing unit may decapsulate files according to ISOBMFF or the like to obtain 360-degree video data to 360-degree video-related metadata. The decapsulation processing unit may select the video stream in the generated file format using the viewpoint/viewpoint/direction information and video metadata transmitted from the feedback processing unit, and the selected bitstream may be reconstructed into video information through a decoder. Can be. The obtained 360-degree video data may be transmitted to a data decoder, and the obtained 360-degree video-related metadata may be transmitted to a metadata parser. The 360-degree video-related metadata obtained by the decapsulation processor may be in the form of a box or track in a file format. If necessary, the decapsulation processing unit may receive metadata required for decapsulation from a metadata parser.

The data decoder can decode 360-degree video data. The data decoder may receive metadata required for decoding from the metadata parser. The 360-degree video-related metadata obtained in the data decoding process may be transmitted to a metadata parser.

In the case of a packed image, the unpacking/selection processing unit may perform unpacking on the packed image based on the packing information transmitted through metadata. Also, if necessary, the unpacking/selection processing unit may perform a process of selecting an image and necessary components suitable for a viewpoint/viewpoint/direction transmitted from the feedback processing unit.

The metadata parser can perform parsing/decoding of 360-degree video related metadata. The metadata parser may transfer the obtained metadata to a data decapsulation processor, a data decoder, a re-projection processor, and/or a renderer.

The re-projection processing unit may perform re-projection on the decoded 360-degree video data. The re-projection processing unit may re-project 360-degree video data into 3D space. The 3D space may have a different shape depending on the 3D model used. The re-projection processing unit may receive metadata required for re-projection from the metadata parser. For example, the re-projection processing unit may receive information on the type of 3D model used and its detailed information from a metadata parser. According to an embodiment, the re-projection processing unit may re-project only 360-degree video data corresponding to a specific area in 3D space into 3D space using metadata required for re-projection.

The renderer can render the re-projected 360 degree video data. As described above, 360-degree video data may be expressed as being rendered in 3D space. In the case where both processes occur at once, the re-projection processing unit and the renderer are integrated, and all of these processes can be performed in the renderer. Depending on the embodiment, the renderer may render only the part the user is viewing according to the user's viewpoint information.

Also, the renderer may perform a rendering process of reconstructing the texture, depth, and overlay information of an image into a format suitable for reproduction. Before the final image is generated, a composition process in which information of different layers is integrated may be performed, and an image suitable for a display viewport may be generated and reproduced.

The user can view some areas of the 360-degree video rendered through a VR display or the like. The VR display is a device that plays 360-degree video, and may be included in a 360-degree video receiving device (tethered) or connected to a 360-degree video receiving device as a separate device (un-tethered).

According to an embodiment of the 360-degree video receiving apparatus according to the present invention, the 360-degree video receiving apparatus may further include a (receiving side) feedback processor and/or a network interface (not shown) as internal/external elements. The receiver feedback processing unit may obtain and process feedback information from a renderer, a re-projection processing unit, a data decoder, a decapsulation processing unit, and/or a VR display. The feedback information may include viewport information, head orientation information, gaze information, and the like. The network interface may receive feedback information from the feedback processing unit on the receiving side, and transmit the feedback information to a 360-degree video transmission device.

As described above, the feedback information may not only be delivered to the transmitting side, but may also be consumed at the receiving side. The receiving-side feedback processing unit may transmit the obtained feedback information to internal elements of the 360-degree video receiving device so that it can be reflected in a process such as rendering. The receiving-side feedback processing unit may transmit feedback information to a renderer, a re-projection processing unit, a data decoder and/or a decapsulation processing unit. For example, the renderer may preferentially render an area viewed by a user by using feedback information. In addition, the decapsulation processing unit, the data decoder, etc., may preferentially decapsulate and decode the area viewed by the user or the area to be viewed.

The inside/outside elements of the 360-degree video receiving apparatus according to the present invention described above may be hardware elements implemented in hardware. Depending on the embodiment, internal/external elements may be changed, omitted, or replaced with other elements. Depending on the embodiment, additional elements may be added to the 360-degree video receiving device.

Another aspect of the invention may relate to a method of transmitting 360-degree video and a method of receiving 360-degree video. The method for transmitting/receiving a 360-degree video according to the present invention may be performed by the above-described 360-degree video transmitting/receiving device according to the present invention or embodiments of the device.

Each embodiment of the above-described 360-degree video transmission/reception device, transmission/reception method according to the present invention, and respective embodiments of its internal/external elements may be combined with each other. For example, the embodiments of the projection processing unit and the embodiments of the data encoder may be combined with each other to produce embodiments of the 360-degree video transmission apparatus in the number of cases. Such combined embodiments are also included in the scope of the present invention.

Meanwhile, as described above, distortion may be generated according to characteristics of a camera and a lens for a 360-degree video, and in this case, a 360-degree video transmitting device and a 360-degree video receiving device to improve image quality, etc. May correct and process the distortion for the 360-degree video. For example, the 360-degree video transmitting device/360-degree video receiving device may project a 2D image by correcting distortion for the 360-degree video. Alternatively, the 360-degree video transmitting device/360-degree video receiving device may correct the distortion in the stitching process and/or rendering process for the 360-degree video.

In order to correct the distortion for the 360-degree video, information about the camera and/or lens for the 360-degree video is required. Since the characteristics may vary depending on the camera and/or lens, correction may be performed in consideration of distortion generated according to the characteristics.

Accordingly, the present invention proposes a method of defining and signaling camera lens information for camera and lens parameters. According to an embodiment, the camera lens information may be transmitted in the form of metadata of a video codec, and may be transmitted in an SEI message of a video codec such as High Efficiency Video Coding (HEVC) or Versatile Video Coding (VVC), or VPS, SPS, It can be delivered in the form of PPS. Also, according to an embodiment, the camera lens information may be transmitted through a digital wired/wireless interface, a system level file format, or the like.

For example, the camera lens information may be represented in a form included in the SEI message as shown in the following table.

Referring to Table 1, the SEI message may include camera_lens_information corresponding to the camera lens information and/or supplemental_camera_lens_information corresponding to supplemental camera lens information. The camera_lens_information may be indicated as a camera lens information SEI message, and the supplemental_camera_lens_information may be indicated as an additional camera lens information SEI message. Meanwhile, the camera lens information may be represented as viewpoint, head position, and/or eye view track information, and the additional camera lens information is supplemental viewpoint, head position And/or eye view track information.

On the other hand, the presence of the camera lens information SEI message in CLVS indicates that the coded video picture in CLVS includes a fisheye lens or a plurality of cameras, camera structures, or a combination of camera arrays. Indicates that it includes a plurality of subpictures captured by a camera structure or a camera array with lenses of.

The 360-degree video receiving device can be used to more appropriately render the omnidirectional camera output for applications such as 3DoF, 3DoF+, 6DoF or panoramic images of camera lens information SEI messages. The camera and lens information SEI message can be applied to the current CLVS including the SEI message. When a camera and lens information SEI message is present in CVLS, the camera lens information SEI message should be present in the first access unit of the CLVS, and may exist in other access units of the CLVS.

The camera_lens_information can be derived as shown in the following table.

Referring to Table 2, the camera_lens_information may include a camera_lens_info_id field and/or a camera_lens_info_cancel_flag field.

The camera_lens_info_id field may indicate an identifier that identifies the purpose of the corresponding camera lens information. For example, the value of the camera_lens_info_id field may indicate a use case of the camera lens information in the SEI message. Alternatively, the value of the camera_lens_info_id field may be used to support receiver performance, or the value of the camera_lens_info_id field may represent a sub-picture that may be composed of a single image. For example, the single image may be a sphere image or a panoramic image.

Meanwhile, as an example, when one or more camera lens information including the camera_lens_info_id field of the same value exists, fields included in the plurality of camera lens information may be the same. Also, a plurality of camera lens information including different values of the camera_lens_info_id field may exist. In this case, camera_lens_info_id fields may indicate that the plurality of camera lens information is information used for different purposes, or cascade of correction or projection sequentially based on the camera lens information. It may indicate that (cascading) is applied. Here, the order may be specified according to the application program. The value of the camera_lens_info_id field may be in the range of 0 to 2 ¹⁵ -1.

The camera_lens_info_cancel_flag field may indicate whether persistence for camera lens information before the camera lens information is canceled according to an output order applied to the current layer. For example, when the value of the camera_lens_info_cancel_flag field is 1, the camera_lens_info_cancel_flag field may indicate that the persistence of camera lens information before the camera lens information is canceled. In addition, when the value of the camera_lens_info_cancel_flag field is 0, the camera_lens_info_cancel_flag field may indicate that persistence with respect to camera lens information before the camera lens information is not canceled.

Referring to Table 2, the camera_lens_information may include a camera_lens_info_persistence_flag field, a supplemental_info_present_flag field, a view_dimension_idc_flag field, and/or a num_camera_id_minus1 field.

The camera_lens_info_persistence_flag field may indicate persistence of the camera lens information SEI message for the current layer. For example, when the value of the camera_lens_info_persistence_flag field is 0, the camera lens information may be applied only to the current decoded picture. In addition, when the value of the camera_lens_info_persistence_flag field is 1, the camera lens information may be maintained until any one of the conditions described below is satisfied in the output order in the current layer. That is, the camera lens information may be applied until any one of the conditions described below is satisfied in the output order in the current layer.

The conditions are as follows.

-When a new CLVS of the current layer starts

-When the bitstream ends

-When the picture picB of the current layer of the access unit including camera lens information applicable to the current layer is output, but PicOrderCnt(picB) is greater than PicOrderCnt(picA)

Here, picA may indicate the current picture, PicOrderCnt(picB) may indicate the PicOrderCntVal value of the picB immediately after an invocation of the decoding process for the picture order count of the picB, and PicOrderCnt( picA) may indicate the PicOrderCntVal value of the picA immediately after the invocation of the decoding process for the picture order count of the picB.

The supplemental_info_present_flag field may indicate whether supplemental information on the camera lens information exists outside the SEI message including the camera lens information. For example, when the value of the supplemental_info_present_flag field is 1, in the supplemental_info_present_flag field, camera lens information or additional information for the camera_lens_info_id field for which additional information for the camera lens information has a different value from the camera_lens_info_id field of the camera lens information It may indicate that it is included in the camera lens information. Further, when the value of the supplemental_info_present_flag field is 0, the supplemental_info_present_flag field may indicate that there is no additional information about the camera lens information.

The view_dimension_idc_flag field may indicate whether a view_dimension_idc field exists in the camera lens information. For example, when the value of the view_dimension_idc_flag field is 1, the view_dimension_idc_flag field may indicate that a view_dimension_idc field exists in the camera lens information, and when the view_dimension_idc_flag field has a value of 1, the view_dimension_idc_flag field is the camera lens It may indicate that the view_dimension_idc field does not exist in the information.

The view_dimension_idc field may indicate the alignment and viewing direction of fisheye lenses. For example, when the value of the view_dimension_idc field is 0, the value of the view_dimension_idc field is 2 and the value of the num_circular_images field is 2, and the camera_center_elevation field, camera_center_tilt field, camera_center_offset_x field, camera_center_offset_y field, and values of the camera_center_offset_z field are aligned images (aligned circular images are ) These are values that have optical axes to face in opposite directions, and may indicate that the sum of the values of the field_of_view field is 360*2 ¹⁶ or more. Here, the num_circular_images field may indicate the number of circular images.

In addition, for example, when the value of the view_dimension_idc field is 1, the value of the view_dimension_idc field is 2 and the value of the num_circular_images field is 2, and the camera_center_eltation field, camera_center_tilt field, camera_center_offset_x field, camera_center_offset_y field, and camera_center_offset_z field are parallel values of the circular images. Values that have parallel optical axes, but the optical axes are orthogonal to a line intersecting with the camera center points, and if i is 0, the i-th camera is a left-view ( left view). That is, when the value of the view_dimension_idc field is 1, the 0th camera may indicate a camera for the left view.

In addition, for example, when the value of the view_dimension_idc field is 2, the value of the view_dimension_idc field is 2, and the value of the num_circular_images field is 2, and the camera_center_eltation field, camera_center_tilt field, camera_center_offset_x field, camera_center_offset_y field and camera_center_offset_z field are parallel values of the circular images. Values that have parallel optical axes, but the orthogonal to the line where the optical axes intersecting the camera center points, and if i is 0, the i-th camera is a light view ( right view). That is, when the value of the view_dimension_idc field is 2, the 0th camera may indicate a camera for a light view.

In addition, for example, when the value of the view_dimension_idc field is 7, the view_dimension_idc field may indicate that additional constraints on syntax elements in the omnidirectional fisheye video SEI message are not implied. .

On the other hand, when the value of the view_dimension_idc field is 3 to 6, it is reserved for future use.

A value obtained by adding 1 to the value of the num_circular_images_minus1 field may indicate the number of camera ids.

Referring to Table 2, the camera_lens_information is a camera_id[i] field, camera_id_type[i] field, camera_location_per_viewpoint_x[i] field, camera_location_per_viewpoint_y[i] field, camera_location_per_viewpoint_z[i] field, camera_rotation_per_viewpoint_yaw[i] field, camera_rotation_per_view field camera_rotation_per_viewpoint_roll[i] field, camera_location_per_head_position_x[i] field, camera_location_per_head_position_y[i] field, camera_location_per_head_position_z[i] field, camera_rotation_per_head_position_yaw[i] field, camera_rotation_per_head_view_eye_field, camera field i] field, camera_location_per_eye_y[i] field, camera_location_per_eye_z[i] field, camera_rotation_per_eye_yaw[i] field, camera_rotation_per_eye_pitch[i] field and/or camera_rotation_per_eye_roll[i] field.

The camera_id[i] field may indicate an identification number used to identify the camera. That is, the camera_id[i] field may indicate the identifier of the i-th camera. The value of the camera_id[i] field may be used to indicate the i-th camera composed of one or more lenses. In addition, the camera_id[i] field may be used to indicate a single image or a plurality of images for a specific purpose. The one image or the plurality of images may be an image or images corresponding to the i-th camera. For example, the camera_id[i] field may indicate a sub-picture corresponding to a specific camera position, or the camera_id[i] field may be a sub-picture that supports binocular disparity of a viewpoint/head position. A sub-picture pair may be indicated, or the camera_id[i] field may indicate a wavelength and/or color filter of a sensor corresponding to a specific lens.

The camera_id_type[i] field may indicate the type, use case, or purpose of the camera_id[i] field. That is, the camera_id_type[i] field may indicate the type, usage example, or purpose of the image for the i-th camera indicated by the camera_id[i] field.

For example, when the value of the camera_id_type[i] field is 0, the camera_id_type[i] field is assigned to a subpicture corresponding to the camera_id[i] field (ie, a subpicture indicated by the camera_id[i] field). It may indicate that the type for the is not specified.

In addition, when the value of the camera_id_type[i] field is 1, the camera_id_type[i] field indicates that a subpicture corresponding to the camera_id[i] field should configure spheres of a viewpoint. Can be. In other words, when the value of the camera_id_type[i] field is 1, the camera_id_type[i] field is a picture in which a subpicture corresponding to the camera_id[i] field constitutes a sphere of a viewpoint (eg, the view) It can indicate that it is a picture rendered on the sphere of the point. Here, the sphere of the viewpoint may represent a 3D space in the viewpoint. In this case, additional information such as a camera location and orientation corresponding to a center anchor of a viewing sphere of the viewpoint can be processed to represent viewpoints different from the viewpoint. have.

In addition, when the value of the camera_id_type[i] field is 2, the camera_id_type[i] field indicates that the subpicture corresponding to the camera_id[i] field should constitute spheres of the head position. Can be represented. In other words, when the value of the camera_id_type[i] field is 2, the camera_id_type[i] field is a picture in which a sub-picture corresponding to the camera_id[i] field constitutes a sphere of a head position (for example, the head. It can indicate that the picture is rendered on the sphere of the position. Here, the sphere of the head position may represent a 3D space in the head position. In this case, additional information such as camera location and orientation corresponding to the center of the viewing sphere of the head position may be processed to indicate head positions different from the head position.

In addition, when the value of the camera_id_type[i] field is 3, the camera_id_type[i] field may indicate that a subpicture corresponding to the camera_id[i] field should constitute a stereoscopic video. In other words, when the value of the camera_id_type[i] field is 3, the camera_id_type[i] field may indicate that a subpicture corresponding to the camera_id[i] field is a picture constituting a stereoscopic video. In this case, additional information such as an eye view, camera location, and orientation corresponding to the center of the viewing sphere of the eye view is processed, and the eye view is different from the eye view. Can represent. Meanwhile, cases in which the value of the camera_id_type[i] field is 4 to 15 are reserved for future use.

The camera_location_per_viewpoint_x[i] field, the camera_location_per_viewpoint_y[i] field, and the camera_location_per_viewpoint_z[i] field may indicate the position of a viewpoint with respect to the camera_id[i] field in units of 2 ^-16 millimeters. That is, the camera_location_per_viewpoint_x[i] field, the camera_location_per_viewpoint_y[i] field, and the camera_location_per_viewpoint_z[i] field are the x component, y component, and z of the viewpoint for the camera_id[i] field in 2 ^-16 millimeters units. Ingredients can be indicated. The position of the viewpoint may correspond to the center position of the viewpoint anchor. For example, the center position of the viewpoint anchor may be a unit sphere center of a center head position. The camera_location_per_viewpoint_x [i] field, the camera_location_per_viewpoint_y [i] field and the camera_location_per_viewpoint_z [i] value of the field is -32768 * 2 ¹⁶ - present in the range of 1 (i.e., -2147483647) to 32768 * ²¹⁶ (i.e., 2147483648) Can be.

The camera_rotation_per_viewpoint_yaw[i] field, the camera_rotation_per_viewpoint_pitch[i] field, and the camera_rotation_per_viewpoint_roll[i] field are yaw angles and pitches for the viewpoint for the camera_id[i] field in 2 ^-16 degree units. pitch) and roll angle. The yaw angle, the pitch angle, and the roll angle can be applied to a unit sphere of a viewpoint anchor for the camera_id[i] field, and coordinates on a unit sphere of a viewpoint anchor are local coordinate axes. It can be converted from (local coordinate axes) to global coordinate axes. The value of the camera_rotation_per_viewpoint_yaw[i] field may be in the range of -180*2 ¹⁶ (ie, -11796480) to 180*2 ¹⁶ -1 (ie, 11796479). The value of the camera_rotation_per_viewpoint_pitch[i] field may be in the range of -90*2 ¹⁶ (ie, -5898240) to 90*2 ¹⁶ (ie, 5898240). The value of the camera_rotation_per_viewpoint_roll[i] field may be in the range of -180*2 ¹⁶ (ie, -11796480) to 180*2 ¹⁶ -1 (ie, 11796479). Meanwhile, the camera_rotation_per_viewpoint_yaw[i] field, the camera_rotation_per_viewpoint_pitch[i] field, and the camera_rotation_per_viewpoint_roll[i] field are azimuth angles with respect to the viewpoint for the camera_id[i] field in 2 ^-16 degrees units. , It may also indicate an elevation angle, a tilt angle.

The camera_location_per_head_position_x[i] field, the camera_location_per_head_position_y[i] field, and the camera_location_per_head_position_z[i] field may indicate the position of the head position with respect to the camera_id[i] field in 2 ^-16 millimeters. That is, the camera_location_per_head_position_x[i] field, the camera_location_per_head_position_y[i] field, and the camera_location_per_head_position_z[i] field are the x component, y component and z component of the head position for the camera_id[i] field in 2 ^-16 millimeters units. Ingredients can be indicated. The position of the head position may correspond to the center position of the head position anchor. For example, the center position of the head position anchor may be a unit sphere center of the center head position. The camera_location_per_head_position_x [i] field, the camera_location_per_head_position_y [i] field and the camera_location_per_head_position_z [i] value of the field is -32768 * 2 ¹⁶ - present in the range of 1 (i.e., -2147483647) to 32768 * ²¹⁶ (i.e., 2147483648) Can be.

The camera_rotation_per_head_position_yaw[i] field, the camera_rotation_per_head_position_pitch[i] field and the camera_rotation_per_head_position_roll[i] field are yaw angles and pitches for the head positions for the camera_id[i] field in 2 ^-16 degree units. pitch) and roll angle. The yaw angle, the pitch angle, and the roll angle may be applied to the unit sphere of the head position anchor for the camera_id[i] field, and the coordinates on the unit sphere of the head position anchor may be local coordinate axes. It can be converted from (local coordinate axes) to global coordinate axes. The value of the camera_location_per_head_position_yaw[i] field may be in a range of -180*2 ¹⁶ (ie, -11796480) to 180*2 ¹⁶ -1 (ie, 11796479). The value of the camera_location_head_position_pitch[i] field may be in the range of -90*2 ¹⁶ (ie, -5898240) to 90*2 ¹⁶ (ie, 5898240). The value of the camera_location_per_head_position_roll[i] field may be in the range of -180*2 ¹⁶ (ie, -11796480) to 180*2 ¹⁶ -1 (ie, 11796479). On the other hand, the camera_rotation_per_head_position_yaw[i] field, the camera_rotation_per_head_position_pitch[i] field and the camera_rotation_per_head_position_roll[i] field are azimuth angles with respect to the viewpoint for the camera_id[i] field in units of 2 ^-16 degrees. , It may also indicate an elevation angle, a tilt angle.

The left_eye_view_flag[i] field may indicate whether a sub-picture corresponding to the camera_id[i] field is a left eye view or a right eye view. For example, when the value of the left_eye_view_flag[i] field is 1, the left_eye_view_flag[i] field may indicate that a sub-picture corresponding to the camera_id[i] field is a left eye view, and the left_eye_view_flag[i] field When the value of is 0, the left_eye_view_flag[i] field may indicate that the sub-picture corresponding to the camera_id[i] field is a right eye view.

The camera_location_per_eye_x[i] field, the camera_location_per_eye_y[i] field, and the camera_location_per_eye_z[i] field may indicate the position of an eye view of the camera_id[i] field in 2 ^-16 millimeters units. . That is, the camera_location_per_eye_x[i] field, the camera_location_per_eye_y[i] field, and the camera_location_per_eye_z[i] field are the x component, y component, and z component of the eye view for the camera_id[i] field in 2 ^-16 millimeters units. Ingredients can be indicated. The eye view may correspond to a unit sphere of a left eye or a right eye. The camera_location_per_eye_x [i] field, the camera_location_per_eye_y [i] field and the camera_location_per_eye_z [i] value of the field is -32768 * 2 ¹⁶ - present in the range of 1 (i.e., -2147483647) to 32768 * ²¹⁶ (i.e., 2147483648) Can be.

The camera_rotation_per_eye_yaw[i] field, the camera_rotation_per_eye_pitch[i] field and the camera_rotation_per_eye_roll[i] field are yaw angles and pitches for the eye view for the camera_id[i] field in units of 2 ^-16 degrees. pitch) and roll angle. The yaw angle, the pitch angle, and the roll angle may be applied to a unit sphere of an eye view for the camera_id[i] field, and coordinates on a unit sphere of the eye view may be local coordinate axes (local). coordinate axes) to global coordinate axes. The value of the camera_rotation_per_eye_yaw[i] field may be in the range of -180*2 ¹⁶ (ie, -11796480) to 180*2 ¹⁶ -1 (ie, 11796479). The value of the camera_rotation_eye_pitch[i] field may be in the range of -90*2 ¹⁶ (ie, -5898240) to 90*2 ¹⁶ (ie, 5898240). The value of the camera_rotation_per_eye_roll[i] field may be in the range of -180*2 ¹⁶ (ie, -11796480) to 180*2 ¹⁶ -1 (ie, 11796479). On the other hand, the camera_rotation_per_eye_yaw[i] field, the camera_rotation_per_eye_pitch[i] field and the camera_rotation_per_eye_roll[i] field are azimuth angles with respect to the viewpoint for the camera_id[i] field in units of 2 ^-16 degrees. , It may also indicate an elevation angle, a tilt angle.

Referring to Table 2, the camera_lens_information may include a num_subpicture_minus1 field, a scene_radius_flag[i] field, a local_sphere_center_offset_flag[i] field, a local_sphere_rotation_flag[i] field, a lens_distortion_correction_flag[i] field, and/or a num_camera_idx_min1 field.

A value obtained by adding 1 to the value of the num_subpicture_minus1 field may indicate the number of subpictures of a coded picture.

The scene_radius_flag[i] field, the local_sphere_center_offset_flag[i] field, the local_sphere_rotation_flag[i] field, and the lens_distortion_correction_flag[i] field may indicate whether there are fields for each field in the camera lens information.

Specifically, the scene_radius_flag[i] field may indicate whether information about a circular region for the (i+1)-th subpicture exists in the camera lens information. In other words, the scene_radius_flag[i] field may indicate whether a scene_radius[i] field for the (i+1)-th subpicture exists in the camera lens information. Here, the (i+1)-th subpicture may be the (i+1)-th circular region in the coded picture. In addition, the circular region with respect to the (i+1)-th circular region may represent an area where an obstacle such as a camera body is not visible in the (i+1)-th circular region. For example, when the value of the scene_radius_flag[i] field is 1, the scene_radius_flag[i] field indicates that the scene_radius[i] field for the (i+1)-th circular region exists in the camera lens information. When the value of the scene_radius_flag[i] field is 0, the scene_radius_flag[i] field indicates that the scene_radius[i] field for the (i+1)-th circular region is not present in the camera lens information. You can. The detailed description of the scene_radius[i] field will be described later.

The local_sphere_center_offset_flag[i] field may indicate whether offset information of a focal center of a camera lens with respect to the i-th circular region exists in the camera lens information. In other words, the local_sphere_center_offset_flag[i] field may indicate whether the local_sphere_center_offset_x[i] field, local_sphere_center_offset_y[i] field and local_sphere_center_offset_z[i] field for the (i+1)-th circular region are present in the camera lens information. have. For example, when the value of the local_sphere_center_offset_flag[i] field is 1, the local_sphere_center_offset_flag[i] field is the local_sphere_center_offset_x[i] field, local_sphere_center_offset_y[i] for the (i+1)th circular region in the camera lens information. The field and the local_sphere_center_offset_z[i] field may be present, and when the value of the local_sphere_center_offset_flag[i] field is 0, the local_sphere_center_offset_flag[i] field is in the (i+1)-th circular region in the camera lens information. It may indicate that the local_sphere_center_offset_x[i] field, local_sphere_center_offset_y[i] field and local_sphere_center_offset_z[i] field do not exist. The detailed description of the local_sphere_center_offset_x[i] field, the local_sphere_center_offset_y[i] field, and the local_sphere_center_offset_z[i] field will be described later.

In addition, the local_sphere_rotation_flag[i] field is a spherical coordinate of a sphere region corresponding to a center point of the (i+1)-th circular region of an output picture cropped in the camera lens information. ). In other words, the local_sphere_rotation_flag[i] field may indicate whether a local_sphere_center_azimuth[i] field, a local_sphere_center_elevation[i] field and a local_sphere_center_tilt[i] field for the (i+1)-th circular region are present in the camera lens information. have. For example, when the value of the local_sphere_rotation_flag[i] field is 1, the local_sphere_rotation_flag[i] field is the local_sphere_center_azimuth[i] field for the (i+1)-th circular region in the camera lens information, local_sphere_center_elevation[i] The field and the local_sphere_center_tilt[i] field may be present, and when the value of the local_sphere_rotation_flag[i] field is 0, the local_sphere_rotation_flag[i] field is in the (i+1)-th circular region in the camera lens information. It may indicate that the local_sphere_center_azimuth[i] field, local_sphere_center_elevation[i] field and local_sphere_center_tilt[i] field do not exist. The detailed description of the local_sphere_center_azimuth[i] field, the local_sphere_center_elevation[i] field and the local_sphere_center_tilt[i] field will be described later.

In addition, the lens_distortion_correction_flag[i] field may indicate whether information related to distortion of the camera lens for the (i+1)-th circular region exists in the camera lens information. In other words, the lens_distortion_correction_flag[i] field includes the num_angle_correction_minus1[i] field, the angle_correction[i][j] field, the num_polynomial_coeff_correction_minus1[i][j] field and the (i+1)th circular region in the camera lens information. A polynomial_coeff_correction[i][j][k] field may be present. For example, when the value of the lens_distortion_correction_flag[i] field is 1, the lens_distortion_correction_flag[i] field is the num_angle_correction_minus1[i] field for the (i+1)th circular region in the camera lens information, angle_correction[i] [j] field, num_polynomial_coeff_correction_minus1[i][j] field and polynomial_coeff_correction[i][j][k] field may be present, and when the value of the lens_distortion_correction_flag[i] field is 0, the lens_distortion_correction_flag[i] Field is the num_angle_correction_minus1[i] field, the angle_correction[i][j] field, the num_polynomial_coeff_correction_minus1[i][j] field and the polynomial_coeff_correction[i][j] for the (i+1)th circular region in the camera lens information. It may indicate that the [k] field does not exist. The num_angle_correction_minus1[i] field, the angle_correction[i][j] field, the num_polynomial_coeff_correction_minus1[i][j] field and the polynomial_coeff_correction[i][j][k] field will be described later.

A value obtained by adding 1 to the value of the num_camera_idx_minus1 field may indicate the number of camera indicators referring to the i-th subpicture. Alternatively, a value obtained by adding 1 to the value of the num_camera_idx_minus1 field may indicate the number of camera indicators referring to the (i+1)th subpicture. Here, the (i+1)-th sub-picture may be the (i+1)-th circular region.

Referring to Table 2, the camera_lens_information is a camera_idx[i][j] field, circular_region_center_x[i] field, circular_region_center_y[i] field, rect_region_top[i] field, rect_region_left[i] field, rect_region_width[i] field, rect_region_height[i ] Field, full_radius[i] field, scene_radius[i] field, local_sphere_rotation_azimuth[i] field, local_sphere_rotation_elevation[i] field, local_sphere_center_offset_x[i] field, local_sphere_center_offset_y[i] field, local_sphere_center_offset_z[i] field, field_field_i field , lens_projection_type[i] field, scaling_factor[i] field, num_angle_projection_minus1[i] field, angle_projection[i][j] field, num_polynomial_coeff_projection_minus1[i][j] field, polynomial_coeff_projection[i][j][k] field, num_angle__cor The [i] field, angle_correction[i][j] field, num_polynomial_coeff_correction_minus1[i][j] field, polynomial_coeff_correction[i][j][k] field, and/or redial_distortion_type[i] field may be included.

The camera_idx[i][j] field may indicate the j-th camera indicator of the i-th subpicture for camera_id present in the camera lens information SEI message. Alternatively, the camera_idx[i][j] field may indicate the (j+1)th camera indicator of the (i+1)th subpicture for camera_id present in the camera lens information SEI message. Here, the (i+1)-th sub-picture may be the (i+1)-th circular region.

The circular_region_center_x[i] field and the circular_region_center_y[i] field may indicate the center point of the (i+1)-th circular region in a picture coded in units of 2 ^-16 luma samples. That is, the circular_region_center_x[i] field and the circular_region_center_y[i] field may indicate horizontal coordinates (x component) and vertical coordinates (y component) of the center point of the (i+1)-th circular region in ^2-16 luma sample units. . The circular_region_center_x [i] field, the circular_region_center_y [i] value of the field is from 0 to 65536 * ²¹⁶ - may be present in the range of 1 (i.e., 4294967295).

The rect_region_top[i] field, the rect_region_left[i] field, the rect_region_width[i] field, and the rect_region_height[i] field are (i+1)-th including the (i+1)-th circular region in units of luma samples. It can indicate the position of the upper left corner of the rectangular area (the position of the upper left corner), width and height. The rect_region_top[i] field, the rect_region_left[i] field, the rect_region_width[i] field, and the rect_region_height[i] field are horizontal coordinates (x) of the upper left corner position of the (i+1)-th square region in units of luma samples. Component) and vertical coordinates (y component), width and height.

The full_radius[i] field may indicate the radius of the (i+1)-th circular region. The radius of the (i+1)-th circular region may be defined as the length from the center point of the (i+1)-th circular region to the outermost sample boundary. The center point may be represented by the circular_region_center_x[i] field and the circular_region_center_y[i] field in ^2-16 luma sample units. Further, the radius of the (i+1)-th circular region may correspond to the maximum field of view of the (i+1)-th lens with respect to the (i+1)-th circular region indicated by the field_of_view[i] field. Alternatively, the radius may correspond to the maximum field of view of the i-th lens for the i-th circular region indicated by the field_of_view[i] field. The full_radius [i] value of the field is from 0 to 65536 * ²¹⁶ - may be present in the range of 1 (i.e., 4294967295).

Meanwhile, the actual sample region of the (i+1)-th circular region is derived based on the rect_region_top[i] field, the rect_region_left[i] field, the rect_region_width[i] field, and the rect_region_height[i] field. It can be defined as a region corresponding to the inner intersection of the circular region derived based on the rectangular region, the circular_region_center_x[i] field, the circular_region_center_y[i] field, and the full_radius[i] field.

The scene_radius[i] field may indicate the radius of a circular region in the (i+1)-th circular region. Here, the circular region in the (i+1)-th circular region may represent an area where an obstacle such as a camera body is not visible in the (i+1)-th circular region. The radius of the circular region in the (i+1)-th circular region may be represented by the scene_radius[i] field in units of 2 ^-16 luma samples. In addition, the scene_radius [i] is the value of the field full_radius [i] may be the same as or smaller than the value of the field, from 0 to 65 536 * 2 ¹⁶ - can be present in the range of 1 (i.e., 4294967295). The circular region may be a region in which a content provider is proposed to be used in the stitching process.

The local_sphere_rotation_azimuth[i] field and the local_sphere_rotation_elevation[i] field are spherical coordinates of a sphere region corresponding to a center point of the (i+1)th circular region of a cropped output picture. ). In other words, the local_sphere_rotation_azimuth[i] field may indicate an azimuth angle of spherical coordinates of a sphere region corresponding to the center point of the (i+1)-th circular region, and the local_sphere_rotation_elevation[i] field is the An elevation angle of spherical coordinates of the sphere region corresponding to the center point of the (i+1)-th circular region may be represented. Here, the local_sphere_rotation_azimuth[i] field and the local_sphere_rotation_elevation[i] field may represent an azimuth angle and an elevation angle of spherical coordinates of a sphere region corresponding to the center point in units of 2 ^-16 degrees. The value of the local_sphere_rotation_azimuth[i] field may be in the range of -180*2 ¹⁶ (ie, -11796480) to 180*2 ¹⁶ -1 (ie, 11796479). In addition, the value of the local_sphere_rotation_elevation[i] field may be in the range of -90*2 ¹⁶ (ie, -5898240) to 90*2 ¹⁶ (ie, 5898240).

The local_sphere_rotation_tilt[i] field may indicate a tilt angle of spherical coordinates of a sphere region corresponding to a center point of the (i+1)-th circular region. Here, the local_sphere_rotation_tilt[i] field may indicate a tilt angle of spherical coordinates of a sphere region corresponding to the center point in units of 2 ^-16 degrees. The value of the local_sphere_rotation_tilt[i] field may be in the range of -180*2 ¹⁶ (ie, -11796480) to 180*2 ¹⁶ -1 (ie, 11796479).

The local_sphere_center_offset_x[i] field, the local_sphere_center_offset_y[i] field, and the local_sphere_center_offset_z[i] field may indicate an offset of a focal center of the camera lens with respect to the (i+1)-th circular region. The focal center of the camera lens may be represented by the offset based on the focal center origin of the entire camera configuration. In other words, the local_sphere_center_offset_x[i] field may indicate the x offset of the focal center of the camera lens with respect to the (i+1)th circular region, and the local_sphere_center_offset_y[i] field may be the (i+1)th circular A y offset of a focal center of the camera lens with respect to a region may be indicated, and the local_sphere_center_offset_z[i] field may indicate a z offset of the focal center of the camera lens with respect to the (i+1)-th circular region. Here, the local_sphere_center_offset_x[i] field, the local_sphere_center_offset_y[i] field, and the local_sphere_center_offset_z[i] field may represent x offset, y offset, and z offset of the focal center of the camera lens in 2 ^-16 millimeters. have. The local_sphere_center_offset_x [i] field, the local_sphere_center_offset_y [i] field and the local_sphere_center_offset_z [i] value of the field is from 0 to 65536 * ²¹⁶ - may be present in the range of 1 (i.e., 4294967295).

Meanwhile, in generating a 360-degree video, the local_sphere_center_offset_x[i] field, the local_sphere_center_offset_y[i] field, and the local_sphere_center_offset_z[i] field may indicate the xyz position of the sphere center of the unit sphere. . The (i+1)-th sub-picture (the (i+1)-th circular region) may be mapped based on the sphere center of the unit sphere, and a 360-degree video stitched to the unit sphere may be provided. If the local_sphere_center_offset_x[i] field, the local_sphere_center_offset_y[i] field and the local_sphere_center_offset_z[i] field do not exist, the local_sphere_center_offset_x[i] field, the local_sphere_center_offset_y[i] field and the local_sphere_center_offset_z[i] field and the value of the local_sphere_center_offset_z[i] field Can be derived as

The field_of_view[i] field may indicate a spherical domain coverage of the (i+1)-th circular region of the coded picture. Here, the field_of_view [i] field may indicate the spherical domain range (degrees) units Fig. ^2-16. The value of the field_of_view[i] field may be in the range of 0 to 360*2 ¹⁶ .

The lens_projection_type[i] field may indicate the type of lens projection for the (i+1)-th circular region. For example, when the value of the lens_projection_type[i] field is 0, the lens_projection_type[i] field may indicate that the type of the lens projection for the (i+1)-th circular region is not specified, When the value of the lens_projection_type[i] field is 1, the lens_projection_type[i] field may indicate that the type of the lens projection for the (i+1)-th circular region is a perspective projection, and the lens_projection_type When the value of the [i] field is 2, the lens_projection_type[i] field may indicate that the type of the lens projection for the (i+1)-th circular region is a stereographic projection, and the lens_projection_type[ i] When the value of the field is 3, the lens_projection_type[i] field may indicate that the type of the lens projection for the (i+1)-th circular region is equality projection, and the lens_projection_type[i ] If the value of the field is 4, the lens_projection_type[i] field may indicate that the type of the lens projection for the (i+1)-th circular region is sine-law projection, and the lens_projection_type When the value of the [i] field is 5, the lens_projection_type[i] field may indicate that the type of the lens projection for the (i+1)-th circular region is an equi-solid projection, and the When the value of the lens_projection_type[i] field is 255, the lens_projection_type[i] field is the (i+1) number It may indicate that the type of the lens projection for the second circular region is an angular polynomial projection. Meanwhile, cases in which the value of the lens_projection_type[i] field is 6 to 254 are reserved for future use.

12 exemplarily shows radial projection functions. The radial projection functions may represent the types of lens projection described above.

The scaling_factor[i] field may indicate a scaling factor of the (i+1)-th lens projection type in 2 ^-8 units. The (i+1)-th lens projection type may indicate a lens projection type for the (i+1)-th circular region. It may be present in the range of 1 - the field_of_view [i] value of the field is from 0 to 256 * ^2-8.

The value obtained by adding 1 to the value of the num_angle_projection_minus1[i] field may represent the number of angular values indicating the angular direction of the projection function of the (i+1)-th circular region. When the value of the num_angle_projection_minus1[i] field is 0, the num_angle_projection_minus1[i] field has all angle values of the luma sample in the circular region for the sphere coordinate projection function of the (i+1)-th circular region. Can be assumed to be symmetrical with respect to.

The angle_projection[i][j] field may indicate an angular value indicating the direction of the (j+1)-th polynomial function of the (i+1)-th circular region. That is, the angle_projection[i][j] field may indicate the (j+1)th angle value of the (i+1)th circular region. Here, the angle_projection[i][j] field may indicate the angle value in units of 2 ^-7 degrees. The value of the angle_projection[i][j] field may be in the range of 0 to 360*2 ⁷ -1.

The value obtained by adding 1 to the value of the num_polynomial_coeff_projection_minus1[i][j] field is the (j+1)th lens distortion correction function (lens distortion) corresponding to the (j+1)th angle value in the (i+1)th circular region. correction function) polynomial coefficients. Here, the (j+1)-th lens distortion correction function may represent the (j+1)-th polynomial function. The polynomial function may also be referred to as an angular curve function transformation.

The polynomial_coeff_projection[i][j][k] field is the (j+1)th angler curve function transformation of the radial distance between the luma sample and the center point in the (i+1)th circular region. transformation) k-th polynomial coefficient value. Alternatively, the polynomial_coeff_projection[i][j][k] field transforms the (j+1)th angler curve function of the radial distance between the luma sample and the center point in the (i+1)th circular region. (k+1)th polynomial coefficient of curve function transformation. The (k+1)-th polynomial coefficient value is a normalized value based on the full_radius[i] field, and may be an angular value between the luma sample and vectors corresponding to the center point. Further, the luma sample and the center point may be represented by spherical coordinates having an origin corresponding to a focal point of the lens of the (i+1)-th circular region. The value of the polynomial_coeff_projection[i][j][k] field may be in the range of -128*2 ²⁴ (ie, 2147483648) to 128*2 ²⁴ -1 (ie, 2147483647).

The value obtained by adding 1 to the value of the num_angle_correction_minus1[i] field may represent the number of angle values indicating the angular direction of the lens distortion correction function of the (i+1)-th circular region. When the value of the num_angle_correction_minus1[i] field is 0, the num_angle_correction_minus1[i] field is symmetrical with respect to all angle values of the luma sample in the circular region for the lens distortion correction function of the (i+1)-th circular region. ).

The angle_correction[i][j] field is an angle indicating a direction from a center point of the (i+1)th circular region of the (j+1)th lens distortion correction function of the (i+1)th circular region to a boundary It can represent an angular value. That is, the angle_correction[i][j] field may indicate the (j+1)th angle value of the (i+1)th circular region. Here, the angle_correction[i][j] field may indicate the angle value in units of 2 ^-7 degrees. The value of the angle_correction[i][j] field may be in the range of 0 to 360*2 ⁷ -1.

The value obtained by adding 1 to the value of the num_polynomial_coeff_correction_minus1[i][j] field is the (j+1)th lens distortion correction function (lens distortion) corresponding to the (j+1)th angle value in the (i+1)th circular region. correction function) polynomial coefficients.

The polynomial_coeff_correction[i][j][k] field may represent the k-th polynomial coefficient value of the (j+1)-th lens distortion correction function of the (i+1)-th circular region in 2 ^-24 luma sample units. Alternatively, the polynomial_coeff_correction[i][j][k] field is the (k+1)th polynomial coefficient of the (j+1)th lens distortion correction function of the (i+1)th circular region in 2 ^-24 luma sample units. Can represent a value. The value of the polynomial_coeff_correction[i][j][k] field may be in the range of -128*2 ²⁴ (ie, 2147483648) to 128*2 ²⁴ -1 (ie, 2147483647).

The radial_distortion_type[i] field may indicate the type of lens radial distortion for the (i+1)-th circular region. For example, when the value of the radial_distortion_type[i] field is 0, the radial_distortion_type[i] field may indicate that the type of the lens radial distortion for the (i+1)-th circular region is not specified. , When the value of the radial_distortion_type[i] field is 1, the radial_distortion_type[i] field may indicate that the type of the lens radial distortion for the (i+1)-th circular region is barrel distortion, When the value of the radial_distortion_type[i] field is 2, the radial_distortion_type[i] field may indicate that the type of the lens radial distortion for the (i+1)-th circular region is pincushion distortion, and the When the value of the radial_distortion_type[i] field is 3, the radial_distortion_type[i] field may indicate that the type of the lens radial distortion for the (i+1)-th circular region is Mustache distortion. On the other hand, cases where the value of the radial_distortion_type[i] field is 4 to 254 are reserved for future use.

13 exemplarily shows various types of radial distortions. Different types of radial distortion may occur in the circular image depending on the lens to be photographed, and FIG. 10 may show various types of radial distortion. 13(a) may represent the barrel distortion, FIG. 13(b) may represent the pincushion distortion, and FIG. 13(c) illustrates the mustache distortion ( Mustache distortion).

Meanwhile, the radial distortion of the original image may be corrected based on Brown's distortion model or brow-conrady model. Also, based on the Brow-Conradi model, the tangential distortion caused by the physical distortion of the radial distortion and the lens may not be perfectly corrected. The tangential distortion may also be referred to as decentering distortion.

The process of correcting the radial distortion of the original image may be derived as follows.

Through the process of correcting the distortion, samples of (x _d , y _d ) coordinates of the original image may be moved to (x _u , y _u ) coordinates derived based on the equation. Through this, distortion of the original image may be removed.

Here, r may represent a distance between a distorted image point and a distortion center, and x _d and y _d are x of a distorted image point of a picture in which a 360-degree video is projected using a specified lens. Component, y component, x _u , y _u are the x component, y component of the undistorted image point of the picture in which the 360 degree video is projected using an ideal pinhole camera. Can represent Also, x _c and y _c may represent the x component and the y component of the distortion center. The distortion center may be assumed to be a principal point. K _n may represent the n-th radial distortion coefficient, and P _n may represent the n-th tentative distortion coefficient. The radial distortion coefficient and the tentative distortion coefficient may be derived based on the type of radial distortion of the original image.

The r can be derived by the following equation.

The above-described radial distortion-related information may be used to correct distortion of a 2D domain when an image not mapped to spherical coordinates exists.

The process of converting the sample position of the circular region to a sphere coordinate for global coordinate axes may be as described below. The conversion process described below may be sequentially applied from a circular region where i is 0 to a circular region where i is a value of the num_subpictures_minus1 field. Alternatively, the conversion process described below may be sequentially applied from a circular region where i is 0 to a circular region where i is a specific value, and the specific value is a set of circular regions corresponding to a subpicture indicated by a camera_idx [i][j] field. It can be a number.

The input of the conversion process may be as follows.

-Sample position (x, y) in luma sample units

-2 ^-16 center point position (x _c , y _c ) of the i-th image (or (i+1)-th circular region) derived based on the circular_region_center_x[i] field and the circular_region_center_y[i] field in units of luma samples And a radius r _c of the i-th circular image (or (i+1)-th circular region) derived based on the full_radius[i] field.

- ^2-16 also the field_of_view [i] i-th circular area is derived based on the fields of the unit (Fied Of View) FOV of the lens corresponding to (or the (i + 1) th circular area) θ _v

-2 rotation parameters derived from the local_sphere_center_azimuth[i] field, the local_sphere_center_elevation[i] field and the local_sphere_center_tilt[i] field in units of ^-16 degrees. _c , β _c , γ _c .

- the number of polynomial coefficients n, lens projection type m, the scale factor σ, the angle direction (angular direction) 2 ^-24 for ω _j diagram of units (i + 1) th circular area polynomial coefficients p _k (ω _j)

The output of the conversion process may be as follows.

-Spherical coordinate position (φ, θ) for global coordinate axes for sample position (x, y)

The process of converting the sample position in the (i+1)-th circular region to a position in the spherical coordinate system may be performed as shown in the following table. The conversion process may be represented as a process of mapping the (i+1)-th circular region to a spherical coordinate system.

Here, the formula for φ 'can be derived based on the lens_projection_type[i] field as shown in the following table.

Alternatively, the φ'may be derived from the angular directions θ'and the angular directions adjacent to the φ'based on an interpolation function F as follows:

ω _j and ω _{j +1} may indicate angular directions adjacent to the angular direction θ', and θ'may be greater than ω _j and ω _j+1 smaller.

Meanwhile, referring to Table 1 above, the SEI message may include supplemental_camera_lens_information corresponding to the additional camera lens information.

The supplemental_camera_lens_information can be derived as shown in the following table.

Referring to Table 5, the supplemental_camera_lens_information may include a num_circular_images field.

The num_circular_images field may indicate the number of circular images of a coded picture to which the supplemental_camera_lens_information is applied. For example, the value of the num_circular_images field may be 2. Alternatively, the value of the num_circular_images field may be a non-zero value excluding 2. Here, the num_circular_images field may be represented as a num_subpictures field, and the circular image may be represented as a circular region or a sub-picture.

Referring to Table 5, the num_camera_idx_minus1 field, camera_idx[i][j] field, image_flip[i] field, image_scale_axis_angle[i] field, image_scale_x[i] field, image_scale_y[i] field, num_angle_for_displaying_fov[i] field, displayed_fov[i ][j] field, overlapped_fov[i][j] field, num_local_fov_region[i] field, start_radius[i][j] field, end_radius[i][j] field, start_angle[i][j] field, end_angle[ i][j] field, radius_delta[i][j] field, angle_delta[i][j] field, local_fov_weight[i][j][k][l] field, num_polynomial_coefficients_lsc[i] field, polynomial_coefficient_K_lsc_R[i] [j] field, polynomial_coefficient_K_lsc_G[i][j] field, polynomial_coefficient_K_lsc_B[i][j] field, num_deadzones field, deadzone_left_horizontal_offset[i] field, deadzone_top_vertical_offset[i] field, deadzone_width[i]he/or dead field Field.

A value obtained by adding 1 to the value of the num_camera_idx_minus1 field may indicate the number of camera indicators referring to the i-th subpicture. Alternatively, a value obtained by adding 1 to the value of the num_camera_idx_minus1 field may indicate the number of camera indicators referring to the (i+1)th subpicture. Here, the (i+1)-th sub-picture may be the (i+1)-th circular region.

The camera_idx[i][j] field may indicate the j-th camera indicator of the i-th subpicture for camera_id present in the camera lens information SEI message. Alternatively, the camera_idx[i][j] field may indicate the (j+1)th camera indicator of the (i+1)th subpicture for camera_id present in the camera lens information SEI message. Here, the (i+1)-th sub-picture may be the (i+1)-th circular region.

The image_flip[i] field may indicate whether the (i+1)-th circular region is flipped and how it is flipped. Accordingly, the image_flip[i] field may indicate whether a reverse flipping operation should be applied to the (i+1)-th circular region in a stitching process and/or a rendering process. For example, when the value of the image_flip[i] field is 0, the image_flip[i] field may indicate that the (i+1)-th circular region is not flipped. Also, when the value of the image_flip[i] field is 1, the image_flip[i] field may indicate that the (i+1)-th circular region is flipped vertically. In addition, when the value of the image_flip[i] field is 2, the image_flip[i] field may indicate that the (i+1)-th circular region is flipped horizontally. In addition, when the value of the image_flip[i] field is 3, the image_flip[i] field may indicate that the (i+1)-th circular region is flipped in the vertical direction and the horizontal direction.

The image_scale_axis_angle[i] field, the image_scale_x[i] field, and the image_scale_y[i] field determine whether and how the size of the (i+1)-th circular region is scaled along a specific axis. Can be represented. The values of the image_scale_axis_angle[i] field, the image_scale_x[i] field, and the image_scale_y[i] field may be fixed-point 16. The image_scale_axis_angle[i] field, the image_scale_x[i] field, and the image_scale_y[i] field may be used to account for natural errors in camera-mirror settings. The specific axis may be defined as a single angle indicated by the value of the image_scale_axis_angle[i] field. The unit of the single angle may be a degree. For example, when the single angle is 0, the horizontal vector may be perfectly horizontal, and the vertical vector may be perfectly vertical. The image_scale_x[i] field and the image_scale_y[i] field may indicate scaling. Meanwhile, the image_scale_axis_angle[i] field, the image_scale_x[i] field, and the image_scale_y[i] field may be called affine parameters, and may satisfy the following equation (satisfy).

Equation 4 is a mathematical expression showing the relationship between actual sample coordinates (u, v) and ideal sample coordinates (u _N , v _N ), and c _x and c _y are the values of the image_center_x[i] field and image_center_y[i, respectively. ] Field value. In addition, c, d, and e indicate the value of the image_scale_x[i] field, the value of the image_scale_axis_angle[i] field, and the value of the image_scale_y[i] field, respectively.

The num_angle_for_displaying_fov[i] field may indicate the number of angles that define a displayed region and an overlapped region. Based on the value of the num_angle_for_displaying_fov[i] field, values of the displayed_fov[i][j] field and the overlapped_fov[i][j] field may be defined at the same interval, and the displayed_fov[i][j] The values of the field and overlapped_fov[i][j] fields can be defined in clockwise order from 12 o'clock.

The displayed_fov[i][j] field may indicate a region recommended to be displayed without a blending process with an adjacent circular region in the (i+1)th circular region.

The overlapped_fov[i][j] field may indicate a field of view (FOV) overlapping an adjacent circular area on a spherical surface of the (i+1)-th circular area. The FOV indicated by the overlapped_fov[i][j] field may indicate an area of the adjacent circular area and only one of the FOVs, or may indicate a recommended area to be displayed by applying a blending process with the adjacent circular area.

The num_local_fov_region[i] field may indicate the number of local fitting regions having different field of view (FOV) of the (i+1)-th circular region.

The start_radius[i][j] field, end_radius[i][j] field, start_angle[i][j] field and end_angle[i][j] field are for local fitting/warping. j+1)th region. The local fitting/warping may indicate transforming the actual FOV to display locally. The values of the start_radius[i][j] field and end_radius[i][j] field may be fixed-point 16. The start_radius[i][j] field may indicate the minimum radius value of the (j+1)th region, and the end_radius[i][j] field may indicate the maximum radius value of the (j+1)th region. Can be represented. The start_angle[i][j] field and the end_angle[i][j] field may indicate a minimum angle and a maximum angle value increasing clockwise starting at 12 o'clock in the (j+1)-th region. . Here, the start_angle[i][j] field and the end_angle[i][j] field may indicate the angle value in units of 2 ^-16 degrees. The values of the start_angle[i][j] field and the end_angle[i][j] fields may be in the range of -180*2 ¹⁶ to 180*2 ¹⁶ -1.

The radius_delta[i][j] field may indicate a delta radius value indicating a different FOV for each radius. The radius_delta[i][j] field may be a fixed-point 16.

The angle_delta[i][j] field may indicate a delta angle value indicating a different FOV for each angle. The start_angle[i][j] field and the angle_delta[i][j] field may indicate the delta angle value in units of 2 ^-16 degrees.

The local_fov_weight[i][j][k][l] field is the FOV of the position designated by the angle index i and the radius index j, that is, the start_radius[i][j] field, end_radius[i][j] field, The weight for the FOV of the position derived based on the start_angle[i][j] field and the end_angle[i][j] field may be indicated. The value of the local_fov_weight[i][j][k][l] field may be 8.24 fixed point format. A positive value of the local_fov_weight[i][j][k][l] field may indicate an expansion of the FOV, and the local_fov_weight[i][j][k][l] field The negative value of may indicate the contraction of the FOV.

The num_polynomial_coefficients_lsc[i] field may indicate the number of polynomial coefficients of lens shading compensation (LSC) parameters of the (i+1)-th circular region.

The polynomial_coefficient_K_lsc_R[i][j] field, the polynomial_coefficient_K_lsc_G[i][j] field and the polynomial_coefficient_K_lsc_B[i][j] field are lenses that reduce the color of a radial direction (eg, a fisheye lens) ) May indicate LSC parameters for correcting shading compensation. The values of the polynomial_coefficient_K_lsc_R[i][j] field, the polynomial_coefficient_K_lsc_G[i][j] field, and the polynomial_coefficient_K_lsc_B[i][j] field may be 8.24 fixed point format. The compensating weight multiplied by the original color may be approximated by a curve function, and the curve function representing the compensation weight may be derived as follows.

Here, r may represent a normalized radius. In other words, the radius at the center of the (i+1)-th circular region normalized based on the full_radius[i] field may be represented.

Also, P may indicate an LSC parameter. Here, the polynomial_coefficient_K_lsc_R[i][j] field, the polynomial_coefficient_K_lsc_G[i][j] field and the polynomial_coefficient_K_lsc_B[i][j] field are LSC parameters for red and LSC parameters for green, respectively. , LSC parameters may be represented in blue, and in this case, a weighting factor for red, a weighting factor for green, and a weighting factor for blue may be calculated, respectively.

Also, N may be derived as a value of the num_polynomial_coefficients_lsc[i] field. That is, N may represent the number of polynomial coefficients of the LSC parameter of the (i+1)-th circular region.

Meanwhile, the value of the displayed_fov[i][j] field and the overlapped_fov[i][j] field may be smaller than or equal to the value of the field_of_view[i] field. The value of the field_of_view[i] field is determined by the physical property of each lens, while the value of the displayed_fov[i][j] field and the overlapped_fov[i][j] field is determined by the configuration of a plurality of lenses Can be. For example, when the value of the num_circular_images field is 2 and two lenses are symmetrically positioned, the values of the displayed_fov[i][j] field and the overlapped_fov[i][j] field are each basically It can be set to 180 and 190. However, the values of the displayed_fov[i][j] field and the overlapped_fov[i][j] field may be changed according to the configuration of the lens and the characteristics of 360-degree video content. For example, stitching quality with values of displayed_fov (eg, 170 for the left camera lens, 190 for the right camera) and values of overlapped_fov (for example, 185 for the left camera and 190 for the right camera). If is better than the quality with default values of displayed_fov and overlapped_fov (i.e. 180 and 190), or if the physical configuration of the camera is asymmetric, the unequal values of the display_fov field and the overlapped_fov field can be derived. have. In addition, in the case of N (N>2) images (that is, circular areas), the exact area of each image cannot be represented by one displayed_fov field value. That is, the displayed_fov field may be different depending on the direction. Accordingly, the above-described num_angle_for_displaying_fov[i] field may be proposed to control the N images. For example, when the value of the num_angle_for_displaying_fov[i] field is 12, the image may be divided into 12 sectors, and each sector angle may be 30 degrees.

The num_deadzones field may indicate the number of dead zones of a coded picture. The dead zone may represent a rectangular area including unused samples of the coded picture, that is, a rectangular area to which the 360-degree video is not mapped.

The deadzone_left_horizontal_offset[i] field and the deadzone_top_vertical_offset[i] field may indicate the upper left position (the position of the upper left point) of the corresponding dead zone. The deadzone_width[i] field may indicate the width of the corresponding dead zone, and the deadzone_height[i] field may indicate the height of the corresponding dead zone. To reduce the bit amount for the 360-degree video, all samples in the dead zone may be set to the same sample value. For example, all samples in the dead zone may be set to a sample value indicating black.

In addition, camera intrinsic parameters such as focal length (f _x , f _y ), principal point (c _x , c _y ), skew coefficient skew_c of the camera or lens And camera extrinsic parameters such as rotation and translation parameters may also be defined in the above-mentioned camera lens information SEI message or additional camera lens information SEI message.

14 shows an example of capturing a 360-degree video through a camera lens. Referring to FIGS. 14A and 14B, a 360-degree video captured according to a focal length of a camera or lens may be derived. A sample of (Xc, Yc, Zc) coordinates can be captured at the (fxXc/Zc, fyYc/Zc) position of the image plane, and the standardized sample position is divided by the focal length at the position of the image plane. It can be derived as (Xc/Zc, Yc/Zc). It may represent the image plane at a focal length of 1 in the focus of the camera or lens of the standardized image plane.

Also, FIG. 14C can represent a sample of a 360-degree video captured obliquely by a specific angle. A specific angle representing an oblique degree may be derived based on the skew coefficient skew_c described above. For example, the skew coefficient may be derived as follows.

Meanwhile, the above-described camera lens information and/or additional camera lens information may be transmitted in a box form in the ISOBMFF file as described above. The 360 video data may be stored and transmitted based on an ISOBMFF file, and the camera lens information and/or the additional camera lens information may be delivered in a box form in the ISOBMFF file.

According to an embodiment, the box for the camera lens information and/or the additional camera lens information may be signaled for 360-degree video data stored/delivered through a corresponding video track (stream), sample, sample group, or the like. Also, according to an embodiment, a box for the camera lens information and/or the additional camera lens information may be located below a visual sample entry of a track in which the corresponding 360 degree video data is stored/transmitted. Also, according to an embodiment, the video information may be delivered through a format such as CFF.

Meanwhile, as an example, the box for the camera lens information and/or the additional camera lens information may include a SEI NAL unit. Further, as an example, the box for the camera lens information and/or the additional camera lens information may be included in VisualSampleEntry, AVCSampleEntry, MVCSampleEntry, SVCSampleEntry, HEVCSampleEntry, etc. associated with the corresponding 360 degree video information.

Also, according to an embodiment, a box for the camera lens information and/or the additional camera lens information may be included in SEI or Video Usability Information (VUI) that provides related information according to an area. Through this, different signaling information may be provided for each video frame included in the file format for each region.

Also, according to an embodiment, a box for the camera lens information and/or the additional camera lens information may be included in timed metadata and transmitted.

When the contents of the camera lens information and/or the additional camera lens information transmitted as timed metadata are equally applied to the entire video sample, a box for the camera lens information and/or the additional camera lens information is applicable. It can be included in a sample entry in the header of a timed metadata track (such as a moov or moof box).

Alternatively, if the contents of the camera lens information and/or the additional camera lens information transmitted as timed metadata need to be applied differently according to a video sample, a box for the camera lens information and/or the additional camera lens information Can be included in the timed metadata sample. In this case, fields of the box for the camera lens information and/or the additional camera lens information may be applied to a corresponding video sample.

Alternatively, when the contents of the camera lens information and/or the additional camera lens information transmitted as timed metadata should be applied to the entire video sequence, the box for the camera lens information and/or the additional camera lens information is As described above, it is included in the sample entry of the timed metadata track, but its meaning can be expanded so that the information (fields) of the box can be applied to the entire video sequence.

Meanwhile, the camera lens information and/or the additional camera lens information may be transmitted according to DASH.

The DASH-based descriptor may include @schemeIdUri field, @value field and/or @id field. The @schemeIdUri field may provide a URI for identifying the schema of the corresponding descriptor. The @value field may have values whose meaning is defined by the scheme indicated by the @schemeIdUri field. That is, the @value field may have values of descriptor elements according to a corresponding scheme, and these may be called parameters. These can be separated from each other by','. @id may indicate the identifier of the corresponding descriptor. If they have the same identifier, they can include the same scheme ID, value, and parameters.

When the camera lens information and/or the additional camera lens information is transmitted according to DASH, the camera lens information and/or the additional camera lens information is described in the form of a DASH descriptor, and may be included in an MPD or the like and transmitted to the receiving side . Descriptors for the camera lens information and/or the additional camera lens information may be delivered in the form of the above-described essential property descriptor and/or supplemental property descriptor. These descriptors can be delivered by being included in the adaptation set, representation, sub-representation, etc. of the MPD.

The camera lens information and/or the additional camera lens information according to all the above-described embodiments may also be described in the form of a DASH-based descriptor. That is, for all the above-described embodiments of the camera lens information and/or the additional camera lens information, each signaling field may be described by substituting a parameter of @value.

The above-described embodiments of the camera lens information and/or the additional camera lens information according to the present invention may be combined with each other. In embodiments of the 360-degree video transmitting device and/or the 360-degree video receiving device according to the present invention, camera lens information and/or additional camera lens information is the camera lens information and/or the addition according to the above-described embodiments It may be camera lens information.

15 schematically shows a method of processing 360-degree video data by a 360-degree video transmission device according to the present invention. The method disclosed in FIG. 15 may be performed by the 360 degree video transmission device disclosed in FIG. 10. Specifically, for example, S1500 in FIG. 15 may be performed by the data input unit of the 360-degree video transmission device, S1510 may be performed by the projection processing unit of the 360-degree video transmission device, and S1520 may be 360 degrees It may be performed by a data encoder of a video transmission device, S1530 may be performed by a metadata processing unit of the 360-degree video transmission device, and S1540 may be performed by a transmission processing unit of the 360-degree video transmission device. The transmission processing unit may be included in the transmission unit.

The 360-degree video transmission device acquires a target circular area including a 360-degree image captured by a camera having at least one lens (S1500). The 360-degree video transmission device may acquire a target circular area including a 360-degree image captured by a camera having at least one lens. For example, the lens may be a fish eye lens. Here, the 360-degree image may be a 360-degree image for 3DoF+ content, and the 360-degree image for 3DoF+ content may include a plurality of viewpoints, a plurality of head positions, and/or a plurality of eye views. Can represent

The 360-degree video transmission device maps the target circular area to a picture (S1510). The 360-degree video transmission device may map the target circular area to a rectangular area of the picture. The 360-degree video transmission apparatus may acquire a plurality of circular areas, and the picture may include at least one rectangular area. In this case, the 360-degree video transmission apparatus may map at least one circular area among the plurality of circular areas to the rectangular area.

In addition, the 360-degree video transmission apparatus may perform processing such as rotating, rearranging, or changing the resolution of the rectangular area of the picture. The process may be referred to as region-wise packing or frame packing.

Also, the 360-degree video transmission device may correct distortion of the target circular area of the picture. Through this, the 360-degree video transmission apparatus can derive the corrected picture.

The 360-degree video transmission apparatus encodes the picture to which the target circular region is mapped (S1520). The 360-degree video transmission device may encode the picture. In addition, the 360-degree video transmission device may encode metadata for the target circular region.

The 360-degree video transmission device generates metadata for the 360-degree image (S1530).

The metadata may include camera lens information.

The camera lens information includes the above-described camera_lens_info_id field, camera_lens_info_cancel_flag field, camera_lens_info_persistence_flag field, supplemental_info_present_flag field, view_dimension_idc_flag field, view_dimension_idc field, num_camera_id_minus1 field, camera_id_i_ field, camera_id_id Field, camera_location_per_viewpoint_z[i] field, camera_rotation_per_viewpoint_yaw[i] field, camera_rotation_per_viewpoint_pitch[i] field, camera_rotation_per_viewpoint_roll[i] field, camera_location_per_head_position_x[i] field, camera_location_per_head_position_i_ field, camera field camera_rotation_per_head_position_pitch[i] field, camera_rotation_per_head_position_roll[i] field, left_eye_view_flag[i] field, camera_location_per_eye_x[i] field, camera_location_per_eye_y[i] field, camera_location_per_eye_z[i] field, camera_location_per_eye_z[i] field, camera_ro_ i ] Field, num_subpicture_minus1 field, scene_radius_flag[i] field, local_sphere_center_offset_flag[i] field, local_sphere_rotation_flag[i] field, lens_distortion_correction_flag[i] field, num_camera_idx_minus1 field, camera_idx[i]_region_circle, circular field i] field, rect_region_top[i] field, rect_region_left[i] field, rect_region_width[i] field, rect_region_height[i] field, full_radius[i] field, scene_radius[i] field, local_sphere_rotation_azimuth[i] field, local_sphere_rotation_elevation[i] Field, local_sphere_center_offset_x[i] field, local_sphere_center_offset_y[i] field, local_sphere_center_offset_z[i] field, field_of_view[i] field, lens_projection_type[i] field, scaling_factor[i] field, num_angle_projection_minus1[i] field, angle_projection ] Field, num_polynomial_coeff_projection_minus1[i][j] field, polynomial_coeff_projection[i][j][k] field, num_angle__correction_minus1[i] field, angle_correction[i][j] field, num_polynomial_coeff_correction_minus1[i]cor i][j][k] field, and/or redial_distortion It may include the _type[i] field.

The meanings of the fields are as described above.

Specifically, as an example, the camera lens information may include information indicating a camera type for the target circular region. The camera type may be one of a viewpoint, a head position, and an eye view.

For example, when the value of the information indicating the camera type is 1, the camera type for the target circular region may be derived from the viewpoint, and when the value of the information indicating the camera type is 2, the target The camera type for the circular area may be derived from the head position, and when the information value indicating the camera type is 3, the camera type for the target circular area may be derived from the eye view.

The target circular area may be an image for a camera type indicated by information indicating the camera type, and the camera lens information may include camera type related information for the target circular area. Information indicating the camera type may indicate the camera_id_type[i] field.

For example, the camera lens information may include information indicating the x component, y component, and z component of the target viewpoint for the target circular region. That is, the camera lens information may include information indicating the position of the target viewpoint with respect to the target circular area. Information indicating the x component, y component, and z component of the target viewpoint for the target circular region may indicate the camera_location_per_viewpoint_x[i] field, the camera_location_per_viewpoint_y[i] field, and the camera_location_per_viewpoint_z[i] field. Further, the camera lens information may include information indicating a yaw angle, a pitch angle, and a roll angle of the target viewpoint with respect to the target circular area. That is, the camera lens information may include information indicating the rotation angle of the target viewpoint with respect to the target circular area. Information indicating the yaw angle, pitch angle, and roll angle of the target viewpoint for the target circular region may indicate the camera_rotation_per_viewpoint_yaw[i] field, the camera_rotation_per_viewpoint_pitch[i] field, and the camera_rotation_per_viewpoint_roll[i] field. Meanwhile, in this case, for example, the value of information indicating the camera type may be 1. That is, the camera type for the target circular area indicated by the information indicating the camera type may be the viewpoint.

Or, for example, the camera lens information may include information indicating an x component, a y component, and a z component of a target head position with respect to the target circular region. That is, the camera lens information may include information indicating the position of the target head position with respect to the target circular area. Information indicating the x component, y component, and z component of the target head position for the target circular region may indicate the camera_location_per_head_position_x[i] field, the camera_location_per_head_position_y[i] field, and the camera_location_per_head_position_z[i] field. In addition, the camera lens information may include information indicating a yaw angle, a pitch angle, and a roll angle of the target head position with respect to the target circular area. That is, the camera lens information may include information indicating the rotation angle of the target head position with respect to the target circular region. Information indicating the yaw angle, pitch angle, and roll angle of the target head position with respect to the target circular region includes the camera_rotation_per_head_position_yaw[i] field, the camera_rotation_per_head_position_pitch[i] field, and the camera_rotation_per_head_position_roll[i ] Field. Meanwhile, in this case, for example, the value of information indicating the camera type may be 2. That is, the camera type for the target circular area indicated by the information indicating the camera type may be the head position.

Or, for example, the camera lens information may include information indicating the x component, y component, and z component of the target eye view of the target circular region. That is, the camera lens information may include information indicating a position of a target eye view with respect to the target circular area. Information indicating the x component, y component, and z component of the target eye view for the target circular region may indicate the camera_location_per_eye_x[i] field, the camera_location_per_eye_y[i] field, and the camera_location_per_eye_z[i] field. Further, the camera lens information may include information indicating a yaw angle, a pitch angle, and a roll angle of the target eye view with respect to the target circular area. That is, the camera lens information may include information indicating a rotation angle of a target eye view with respect to the target circular area. Information indicating the yaw angle, pitch angle, and roll angle of the target eye view for the target circular area includes the camera_rotation_per_eye_yaw[i] field, the camera_rotation_per_eye_pitch[i] field, and the camera_rotation_per_eye_roll[i] Field. In addition, the camera lens information may include a flag indicating whether the target eye view of the target circular area is a left eye view. When the value of the flag is 1, the target eye view may be a left eye view, and when the value of the flag is 0, the target eye view may be a right eye view. have. The flag may indicate the left_eye_view_flag[i] field. Meanwhile, in this case, for example, the value of information indicating the camera type may be 3. That is, the eye type may be the camera type for the target circular area indicated by the information indicating the camera type.

Further, as another example, the camera lens information may include information indicating a lens projection type of the target circular region. The lens projection types include perspective projection, stereographic projection, equidistance projection, sine-law projection, equisolid projection and angler poly It may be one of angular polynomial projection.

For example, when the value of the information indicating the lens projection type is 1, the lens projection type of the target circular region may be derived as the perspective projection, and the value of information representing the lens projection type is 2 , The lens projection type of the target circular region may be derived from the stereographic projection, and when the value of the information indicating the lens projection type is 3, the lens projection type of the target circular region is the equidistance projection If it can be derived, and the value of the information indicating the lens projection type is 4, the lens projection type of the target circular region can be derived by the sine-ro projection, and the value of the information representing the lens projection type is When 5, the lens projection type of the target circular region may be derived from the equilisolid projection, and when the value of information indicating the lens projection type is 255, the lens projection type of the target circular region is the angle This can be derived from polynomial projection. Also, when the value of the information indicating the lens projection type is 0, the lens projection type of the target circular region may not be specified. Information indicating the lens projection type may indicate the lens_projection_type[i] field.

In addition, for example, when the lens projection type of the target circular area is the perspective projection, the stereographic projection, the equality projection, the sine-ro projection or the equality projection, the camera lens information is the target It may include information indicating a scaling factor for the circular region. That is, when the value of the information indicating the lens projection type is 1 or more and 5 or less, the camera lens information may include information indicating a scaling factor for the target circular region. Information indicating a scaling factor for the target circular region may indicate the scaling_factor[i] field.

Further, as another example, the camera lens information may include information related to a projection function. The projection function related information may include information indicating the number of projection functions for the target circular region. Information indicating the number of projection functions may indicate the num_angle_projection_minus1[i][j] field.

Further, the projection function related information may include information indicating an angle to which the projection function is applied to the target circular region. Information indicating an angle to which the projection function is applied may indicate the angle_projection[i][j] field.

Further, the projection function related information may include information indicating the number of coefficients of the projection function for the target circular region. Information indicating the number of coefficients of the projection function may indicate the num_polynomial_coeff_projection_minus1[i][j] field.

In addition, the projection function related information may include information indicating the coefficient of the projection function for the target circular region. Information indicating the coefficient of the projection function may indicate the polynomial_coeff_projection[i][j][k] field.

Further, as another example, the camera lens information may include a flag indicating whether information related to a distortion correction function for distortion correction of the target circular region exists. For example, when the value of the flag is 1, the camera lens information may include distortion correction function related information. In addition, when the value of the flag is 0, the camera lens information may not include distortion correction function related information. The flag may indicate the lens_distortion_correction_flag[i] field.

The distortion correction function related information may include information indicating the number of distortion correction functions for the target circular region. Information indicating the number of distortion correction functions may indicate the num_angle_correction_minus1[i][j] field.

Further, the distortion correction function related information may include information indicating an angle to which the distortion correction function is applied to the target circular region. Information indicating an angle to which the distortion correction function is applied may indicate the angle_correction[i][j] field.

In addition, the distortion correction function related information may include information indicating the number of coefficients of the distortion correction function for the target circular region. Information indicating the number of coefficients of the distortion correction function may indicate the num_polynomial_coeff_correction_minus1[i][j] field.

In addition, the distortion correction function related information may include information indicating a coefficient of the distortion correction function for the target circular region. Information indicating the coefficient of the distortion correction function may indicate the polynomial_coeff_correction[i][j][k] field.

A polynomial function for correcting distortion of the target circular region may be derived based on the projection function-related information and/or distortion correction function-related information, and the target circular region distortion may be derived based on the polynomial function. ) Can be modified.

Further, as another example, the camera lens information may include information indicating a type of radial distortion of the target circular region. The radial distortion type may be one of barrel distortion, pincushion distortion, and mustache distortion.

For example, when the value of the information indicating the radial distortion type is 1, the radial distortion type of the target circular region may be derived as the barrel distortion, and the value of information indicating the radial distortion type is 2 , The radial distortion type of the target circular region may be derived by the pincushion distortion, and when the value of the information indicating the radial distortion type is 3, the radial distortion type of the target circular region is derived by the mustache distortion. Can be.

The radial distortion of the target circular region may be modified based on Equation 1 described above. In this case, the radial distortion coefficient and the tangential distortion coefficient of Equation 1 may be derived based on the radial distortion type of the target circular region.

In addition, the metadata may include additional camera lens information.

The additional camera lens information includes the above-described num_subpictures field, num_camera_idx_minus1 field, camera_idx[i][j] field, image_flip[i] field, image_scale_axis_angle[i] field, image_scale_x[i] field, image_scale_y[i] field, num_angle_for_displaying_fov[i ] Field, displayed_fov[i][j] field, overlapped_fov[i][j] field, num_local_fov_region[i] field, start_radius[i][j] field, end_radius[i][j] field, start_angle[i][ j] field, end_angle[i][j] field, radius_delta[i][j] field, angle_delta[i][j] field, local_fov_weight[i][j][k][l] field, num_polynomial_coefficients_lsc[i] Field, polynomial_coefficient_K_lsc_R[i][j] field, polynomial_coefficient_K_lsc_G[i][j] field, polynomial_coefficient_K_lsc_B[i][j] field, num_deadzones field, deadzone_left_horizontal_offset[i] field, deadzone_top_i_width field, deadzone_top_vertical field /Or may include the deadzone_height[i] field.

The meanings of the fields are as described above.

Meanwhile, the metadata may be transmitted through an SEI message. In addition, the metadata may be included in an adaptation set, representation, or subrepresentation of a media presentation description (MPD). For example, the fisheye video information may be transmitted in the form of a Dynamic Adaptive Streaming over HTTP (DASH) descriptor included in the MPD. Here, the SEI message may be used for decoding a 2D image or assisting display of a 2D image into a 3D space.

The 360-degree video transmission apparatus performs processing for storing or transmitting the encoded picture and the metadata (S1540). The 360-degree video transmission device may encapsulate the encoded 360-degree video data and/or the metadata in the form of a file. The 360-degree video transmission device may encapsulate the encoded 360-degree video data and/or the metadata in a file format such as ISOBMFF, CFF, or other DASH segments. The 360 degree video transmission device may include the metadata on a file format. For example, the metadata may be included in various levels of boxes on the ISOBMFF file format, or may be included as data in separate tracks in the file. In addition, the 360-degree video transmission device may encapsulate the metadata itself as a file. The 360-degree video transmission apparatus may apply processing for transmission to the 360-degree video data encapsulated according to a file format. The 360-degree video transmission device can process the 360-degree video data according to any transmission protocol. The processing for transmission may include processing for transmission through a broadcasting network or processing for transmission through a communication network such as broadband. In addition, the 360-degree video transmission device may apply processing for transmission to the metadata. The 360-degree video transmission device may transmit the processed 360-degree video data and the metadata through a broadcast network and/or broadband.

According to the present invention, it is possible to derive and render a circular area for a target viewpoint, target head position, and/or target eye view from 360-degree image data for 3DoF+ content based on camera lens information, through which the user's 3DoF+ It is possible to provide an interactive experience in content consumption.

Further, according to the present invention, a polynomial function may be derived by reflecting characteristics of a lens based on the projection function related information and/or the distortion correction function related information included in camera lens information, and based on the polynomial function, It is possible to propose a method of more accurately mapping 360-degree image data into 3D space by correcting distortion generated in a picture in which a 360-degree image is projected.

16 schematically shows a 360-degree video transmission apparatus that performs a 360-degree image data processing method according to the present invention. The method disclosed in FIG. 15 may be performed by the 360 degree video transmission device disclosed in FIG. 16. Specifically, for example, the data input unit of the 360-degree video transmission device of FIG. 16 may perform S1500 of FIG. 15, and the projection processing unit of the 360-degree video transmission device of FIG. 16 may perform S1510 of FIG. 15. The data encoder of the 360-degree video transmission device of FIG. 16 may perform S1520 of FIG. 15, and the metadata processing unit of the 360-degree video transmission device of FIG. 16 may perform S1530 of FIG. 15, The transmission processing unit of the 360-degree video transmission device of 16 may perform S1540 of FIG. 15. The transmission processing unit may be included in the transmission unit.

17 schematically shows a method of processing 360-degree video data by a 360-degree video receiving apparatus according to the present invention. The method disclosed in FIG. 17 may be performed by the 360-degree video receiving device disclosed in FIG. 11. Specifically, for example, S1700 of FIG. 17 may be performed by the receiving unit of the 360-degree video receiving device, S1710 may be performed by the receiving processing unit of the 360-degree video receiving device, and S1720 of the 360-degree video It may be performed by the data decoder of the receiving device, and S1730 may be performed by the renderer of the 360-degree video receiving device.

The 360-degree video receiving apparatus receives 360-degree image data (S1700). The 360-degree video receiving device may receive the 360-degree video data signaled from the 360-degree video transmission device through a broadcasting network. In addition, the 360-degree video receiving device may receive the 360-degree video data through a communication network such as broadband or a storage medium. Here, the 360-degree video data may be 360-degree video data for 3DoF+ content, and the 360-degree video data for 3DoF+ content may include a plurality of viewpoints, a plurality of head positions, and/or a plurality of eye views. Can display 360-degree image data.

The 360-degree video receiving apparatus acquires information and metadata about an encoded picture from the 360-degree video data (S1710). The 360-degree video receiving apparatus may perform processing according to a transmission protocol on the received 360-degree image data, and obtain information about the encoded picture and the metadata from the 360-degree image data. In addition, the 360-degree video receiving device may perform a reverse process of processing for transmission of the above-described 360-degree video transmission device.

The metadata may include camera lens information.

The camera lens information includes the above-described camera_lens_info_id field, camera_lens_info_cancel_flag field, camera_lens_info_persistence_flag field, supplemental_info_present_flag field, view_dimension_idc_flag field, view_dimension_idc field, num_camera_id_minus1 field, camera_id_i_ field, camera_id_id Field, camera_location_per_viewpoint_z[i] field, camera_rotation_per_viewpoint_yaw[i] field, camera_rotation_per_viewpoint_pitch[i] field, camera_rotation_per_viewpoint_roll[i] field, camera_location_per_head_position_x[i] field, camera_location_per_head_position_i_ field, camera field camera_rotation_per_head_position_pitch[i] field, camera_rotation_per_head_position_roll[i] field, left_eye_view_flag[i] field, camera_location_per_eye_x[i] field, camera_location_per_eye_y[i] field, camera_location_per_eye_z[i] field, camera_location_per_eye_z[i] field, camera_ro_ i ] Field, num_subpicture_minus1 field, scene_radius_flag[i] field, local_sphere_center_offset_flag[i] field, local_sphere_rotation_flag[i] field, lens_distortion_correction_flag[i] field, num_camera_idx_minus1 field, camera_idx[i]_region_circle, circular field i] field, rect_region_top[i] field, rect_region_left[i] field, rect_region_width[i] field, rect_region_height[i] field, full_radius[i] field, scene_radius[i] field, local_sphere_rotation_azimuth[i] field, local_sphere_rotation_elevation[i] Field, local_sphere_center_offset_x[i] field, local_sphere_center_offset_y[i] field, local_sphere_center_offset_z[i] field, field_of_view[i] field, lens_projection_type[i] field, scaling_factor[i] field, num_angle_projection_minus1[i] field, angle_projection ] Field, num_polynomial_coeff_projection_minus1[i][j] field, polynomial_coeff_projection[i][j][k] field, num_angle__correction_minus1[i] field, angle_correction[i][j] field, num_polynomial_coeff_correction_minus1[i]cor i][j][k] field, and/or redial_distortion It may include the _type[i] field.

The meanings of the fields are as described above.

Specifically, as an example, the camera lens information may include information indicating a camera type for the target circular region. The camera type may be one of a viewpoint, a head position, and an eye view.

For example, when the value of the information indicating the camera type is 1, the camera type for the target circular region may be derived from the viewpoint, and when the value of the information indicating the camera type is 2, the target The camera type for the circular area may be derived from the head position, and when the information value indicating the camera type is 3, the camera type for the target circular area may be derived from the eye view.

The target circular area may be an image for a camera type indicated by information indicating the camera type, and the camera lens information may include camera type related information for the target circular area. Information indicating the camera type may indicate the camera_id_type[i] field.

For example, the camera lens information may include information indicating the x component, y component, and z component of the target viewpoint for the target circular region. That is, the camera lens information may include information indicating the position of the target viewpoint with respect to the target circular area. Information indicating the x component, y component, and z component of the target viewpoint for the target circular region may indicate the camera_location_per_viewpoint_x[i] field, the camera_location_per_viewpoint_y[i] field, and the camera_location_per_viewpoint_z[i] field. Further, the camera lens information may include information indicating a yaw angle, a pitch angle, and a roll angle of the target viewpoint with respect to the target circular area. That is, the camera lens information may include information indicating the rotation angle of the target viewpoint with respect to the target circular area. Information indicating the yaw angle, pitch angle, and roll angle of the target viewpoint for the target circular region may indicate the camera_rotation_per_viewpoint_yaw[i] field, the camera_rotation_per_viewpoint_pitch[i] field, and the camera_rotation_per_viewpoint_roll[i] field. Meanwhile, in this case, for example, the value of information indicating the camera type may be 1. That is, the camera type for the target circular area indicated by the information indicating the camera type may be the viewpoint.

Or, for example, the camera lens information may include information indicating an x component, a y component, and a z component of a target head position with respect to the target circular region. That is, the camera lens information may include information indicating the position of the target head position with respect to the target circular area. Information indicating the x component, y component, and z component of the target head position for the target circular region may indicate the camera_location_per_head_position_x[i] field, the camera_location_per_head_position_y[i] field, and the camera_location_per_head_position_z[i] field. In addition, the camera lens information may include information indicating a yaw angle, a pitch angle, and a roll angle of the target head position with respect to the target circular area. That is, the camera lens information may include information indicating the rotation angle of the target head position with respect to the target circular region. Information indicating the yaw angle, pitch angle, and roll angle of the target head position with respect to the target circular region includes the camera_rotation_per_head_position_yaw[i] field, the camera_rotation_per_head_position_pitch[i] field, and the camera_rotation_per_head_position_roll[i ] Field. Meanwhile, in this case, for example, the value of information indicating the camera type may be 2. That is, the camera type for the target circular area indicated by the information indicating the camera type may be the head position.

Or, for example, the camera lens information may include information indicating the x component, y component, and z component of the target eye view of the target circular region. That is, the camera lens information may include information indicating a position of a target eye view with respect to the target circular area. Information indicating the x component, y component, and z component of the target eye view for the target circular region may indicate the camera_location_per_eye_x[i] field, the camera_location_per_eye_y[i] field, and the camera_location_per_eye_z[i] field. Further, the camera lens information may include information indicating a yaw angle, a pitch angle, and a roll angle of the target eye view with respect to the target circular area. That is, the camera lens information may include information indicating a rotation angle of a target eye view with respect to the target circular area. Information indicating the yaw angle, pitch angle, and roll angle of the target eye view for the target circular area includes the camera_rotation_per_eye_yaw[i] field, the camera_rotation_per_eye_pitch[i] field, and the camera_rotation_per_eye_roll[i] Field. In addition, the camera lens information may include a flag indicating whether the target eye view of the target circular area is a left eye view. When the value of the flag is 1, the target eye view may be a left eye view, and when the value of the flag is 0, the target eye view may be a right eye view. have. The flag may indicate the left_eye_view_flag[i] field. Meanwhile, in this case, for example, the value of information indicating the camera type may be 3. That is, the eye type may be the camera type for the target circular area indicated by the information indicating the camera type.

Further, as another example, the camera lens information may include information indicating a lens projection type of the target circular region. The lens projection types include perspective projection, stereographic projection, equidistance projection, sine-law projection, equisolid projection and angler poly It may be one of angular polynomial projection.

For example, when the value of the information indicating the lens projection type is 1, the lens projection type of the target circular region may be derived as the perspective projection, and the value of information representing the lens projection type is 2 , The lens projection type of the target circular region may be derived from the stereographic projection, and when the value of the information indicating the lens projection type is 3, the lens projection type of the target circular region is the equidistance projection If it can be derived, and the value of the information indicating the lens projection type is 4, the lens projection type of the target circular region can be derived by the sine-ro projection, and the value of the information representing the lens projection type is When 5, the lens projection type of the target circular region may be derived from the equilisolid projection, and when the value of information indicating the lens projection type is 255, the lens projection type of the target circular region is the angle This can be derived from polynomial projection. Also, when the value of the information indicating the lens projection type is 0, the lens projection type of the target circular region may not be specified. Information indicating the lens projection type may indicate the lens_projection_type[i] field.

In addition, for example, when the lens projection type of the target circular area is the perspective projection, the stereographic projection, the equality projection, the sine-ro projection or the equality projection, the camera lens information is the target It may include information indicating a scaling factor for the circular region. That is, when the value of the information indicating the lens projection type is 1 or more and 5 or less, the camera lens information may include information indicating a scaling factor for the target circular region. Information indicating a scaling factor for the target circular region may indicate the scaling_factor[i] field.

Further, as another example, the camera lens information may include information related to a projection function. The projection function related information may include information indicating the number of projection functions for the target circular region. Information indicating the number of projection functions may indicate the num_angle_projection_minus1[i][j] field.

Further, the projection function related information may include information indicating an angle to which the projection function is applied to the target circular region. Information indicating an angle to which the projection function is applied may indicate the angle_projection[i][j] field.

Further, the projection function related information may include information indicating the number of coefficients of the projection function for the target circular region. Information indicating the number of coefficients of the projection function may indicate the num_polynomial_coeff_projection_minus1[i][j] field.

In addition, the projection function related information may include information indicating the coefficient of the projection function for the target circular region. Information indicating the coefficient of the projection function may indicate the polynomial_coeff_projection[i][j][k] field.

Further, as another example, the camera lens information may include a flag indicating whether information related to a distortion correction function for distortion correction of the target circular region exists. For example, when the value of the flag is 1, the camera lens information may include distortion correction function related information. In addition, when the value of the flag is 0, the camera lens information may not include distortion correction function related information. The flag may indicate the lens_distortion_correction_flag[i] field.

The distortion correction function related information may include information indicating the number of distortion correction functions for the target circular region. Information indicating the number of distortion correction functions may indicate the num_angle_correction_minus1[i][j] field.

Further, the distortion correction function related information may include information indicating an angle to which the distortion correction function is applied to the target circular region. Information indicating an angle to which the distortion correction function is applied may indicate the angle_correction[i][j] field.

In addition, the distortion correction function related information may include information indicating the number of coefficients of the distortion correction function for the target circular region. Information indicating the number of coefficients of the distortion correction function may indicate the num_polynomial_coeff_correction_minus1[i][j] field.

In addition, the distortion correction function related information may include information indicating a coefficient of the distortion correction function for the target circular region. Information indicating the coefficient of the distortion correction function may indicate the polynomial_coeff_correction[i][j][k] field.

The 360-degree video receiving device may derive a distortion correction function for correcting distortion of the target circular region based on the information related to the distortion correction function, and based on the distortion correction function, the distortion of the target circular region ( distortion).

Further, as another example, the camera lens information may include information indicating a type of radial distortion of the target circular region. The radial distortion type may be one of barrel distortion, pincushion distortion, and mustache distortion.

For example, when the value of the information indicating the radial distortion type is 1, the radial distortion type of the target circular region may be derived as the barrel distortion, and the value of information indicating the radial distortion type is 2 , The radial distortion type of the target circular region may be derived by the pincushion distortion, and when the value of the information indicating the radial distortion type is 3, the radial distortion type of the target circular region is derived by the mustache distortion. Can be.

The 360-degree video receiving apparatus may correct the radial distortion of the target circular region based on Equation 1 described above. In this case, the radial distortion coefficient and the tangential distortion coefficient of Equation 1 may be derived based on the radial distortion type of the target circular region.

In addition, the metadata may include additional camera lens information.

The additional camera lens information includes the above-described num_subpictures field, num_camera_idx_minus1 field, camera_idx[i][j] field, image_flip[i] field, image_scale_axis_angle[i] field, image_scale_x[i] field, image_scale_y[i] field, num_angle_for_displaying_fov[i ] Field, displayed_fov[i][j] field, overlapped_fov[i][j] field, num_local_fov_region[i] field, start_radius[i][j] field, end_radius[i][j] field, start_angle[i][ j] field, end_angle[i][j] field, radius_delta[i][j] field, angle_delta[i][j] field, local_fov_weight[i][j][k][l] field, num_polynomial_coefficients_lsc[i] Field, polynomial_coefficient_K_lsc_R[i][j] field, polynomial_coefficient_K_lsc_G[i][j] field, polynomial_coefficient_K_lsc_B[i][j] field, num_deadzones field, deadzone_left_horizontal_offset[i] field, deadzone_top_i_width field, deadzone_top_vertical field /Or may include the deadzone_height[i] field.

The meanings of the fields are as described above.

Meanwhile, the metadata may be received through an SEI message. In addition, the metadata may be included in an adaptation set, representation, or subrepresentation of a media presentation description (MPD). For example, the fisheye video information may be received in the form of a Dynamic Adaptive Streaming over HTTP (DASH) descriptor included in the MPD. Here, the SEI message may be used for decoding a 2D image or assisting display of a 2D image into a 3D space.

The 360-degree video receiving apparatus decodes a picture including a target circular region based on the information on the encoded picture (S1720). The 360-degree video receiving apparatus may decode a picture including the target circular region based on information about the encoded picture. The target circular area may be an area including the 360-degree image.

The 360-degree video receiving apparatus processes and renders the target circular area based on the camera lens information (S1730).

For example, the 360-degree video receiving apparatus may derive the target circular area from the picture based on the camera lens information. For example, the 360-degree video receiving apparatus may derive the target viewpoint for a specific location based on the camera lens information, and derive the target circular area for the target viewpoint. In addition, as another example, the 360-degree video receiving apparatus may derive the target head position for a specific position based on the camera lens information, and derive the target circular area for the target head position. Further, as another example, the 360-degree video receiving apparatus may derive the target eye view for a specific location based on the camera lens information, and derive the target circular area for the target eye view. Here, the specific location may be a location and/or rotation angle indicated through at least one of a local coordinate system and a global coordinate system, and the specific location is a location selected by a user and/or Or it may be a rotation angle. Further, the local coordinate system may be a coordinate system representing coordinates with x components, y components, and z components, and the global coordinate system may be a coordinate system representing coordinates with a yaw angle, a pitch angle, and a roll angle.

Further, as an example, the camera lens information of the metadata may include information describing the target circular area and information describing a rectangular area to which the target circular area is mapped. The 360-degree video receiving device may derive the rectangular area based on the information describing the rectangular area, and derive the target circular area mapped to the rectangular area based on the information describing the target circular area. . In this case, an area corresponding to the inner intersection of the area to which the rectangular area and the target circular area are mapped may be actual 360-degree image data. The remaining invalid areas can be identified by being marked with black. According to an embodiment, the 360-degree video receiving apparatus may derive an area corresponding to the intersection of the area to which the rectangular areas and the target circular area are mapped.

Further, as an example, the camera lens information may include information about a dead zone to which the 360 degree image data is not mapped. The 360-degree video receiving apparatus may derive a dead zone to which the 360-degree image data is not mapped based on information about the dead zone to which the 360-degree image data is not mapped.

In addition, the camera lens information may include information indicating the lens projection type, and the 360 video receiving device maps the target circular region to 3D space based on a spherical coordinate system mapping formula derived based on the lens projection type. can do. Specifically, for example, the 360 video receiving apparatus may project the target circular area into a plane based on a spherical coordinate system mapping formula derived based on the lens projection type (Projection). Here, the plane may be an ERP (Equirectangular Projection) plane. This projection process may be an intermediate step for re-projecting the target circular region into a 3D space such as a spherical coordinate system.

Further, the camera lens information may include information related to the projection function and/or information related to the distortion correction function. The 360 video reception device may derive a polynomial function for distortion correction of the target circular region based on the projection function related information and/or the distortion correction function related information, and based on the polynomial function The distortion of the circular region can be corrected.

In addition, the camera lens information may include information indicating a radial distortion type of the target circular region, and the 360 video receiving apparatus may perform the radial distortion of the target circular region based on Equation 1 described above. Can be modified.

The 360 video receiving device may perform rendering based on the finally synthesized ERP plane (picture) to generate a corresponding viewport.

According to the present invention, it is possible to derive and render a circular area for a target viewpoint, target head position, and/or target eye view from 360-degree image data for 3DoF+ content based on camera lens information, through which the user's 3DoF+ It is possible to provide an interactive experience in content consumption.

Further, according to the present invention, a polynomial function may be derived by reflecting characteristics of a lens based on the projection function related information and/or the distortion correction function related information included in camera lens information, and based on the polynomial function, It is possible to propose a method of more accurately mapping 360-degree image data into 3D space by correcting distortion generated in a picture in which a 360-degree image is projected.

18 schematically shows a 360-degree video receiving apparatus for performing a 360-degree image data processing method according to the present invention. The method disclosed in FIG. 17 may be performed by the 360-degree video receiving device disclosed in FIG. 18. Specifically, for example, the receiver of the 360-degree video receiving device of FIG. 18 may perform S1700 of FIG. 17, and the receiving processor of the 360-degree video receiving device of FIG. 18 may perform S1710 of FIG. 17. , The data decoder of the 360-degree video receiving apparatus of FIG. 18 may perform S1720 of FIG. 17, and the renderer of the 360-degree video receiving apparatus of FIG. 18 may perform S1730 of FIG. 17.

In addition, the present invention proposes a 360-degree video display method performed by a 360-degree video receiving device.

19 schematically shows a method of displaying a 360-degree image by a 360-degree video receiving apparatus according to the present invention. The method disclosed in FIG. 19 may be performed by the 360-degree video receiving apparatus disclosed in FIG. 11. Specifically, for example, S1900 of FIG. 19 may be performed by the receiving unit of the 360-degree video receiving device, S1910 may be performed by the receiving processing unit of the 360-degree video receiving device, and S1920 of the 360-degree video It may be performed by a data decoder of a receiving device, S1940 may be performed by a renderer of the 360-degree video receiving device, and S1930 and S1950 may be performed by a display unit of the 360-degree video receiving device.

The 360-degree video receiving apparatus receives 360-degree video data (S1900). The 360-degree video receiving device may receive the 360-degree video data signaled from the 360-degree video transmission device through a broadcasting network. In addition, the 360-degree video receiving device may receive the 360-degree video data through a communication network such as broadband or a storage medium. Here, the 360-degree video data may be 360-degree video data for 3DoF+ content, and the 360-degree video data for 3DoF+ content may include a plurality of viewpoints, a plurality of head positions, and/or a plurality of eye views. Can display 360-degree image data.

The 360-degree video receiving apparatus acquires information and metadata about an encoded picture from the 360-degree video data (S1910). The 360-degree video receiving apparatus may perform processing according to a transmission protocol on the received 360-degree image data, and obtain information about the encoded picture and the metadata from the 360-degree image data. In addition, the 360-degree video receiving device may perform a reverse process of processing for transmission of the above-described 360-degree video transmission device.

The metadata may include camera lens information.

The camera lens information includes the above-described camera_lens_info_id field, camera_lens_info_cancel_flag field, camera_lens_info_persistence_flag field, supplemental_info_present_flag field, view_dimension_idc_flag field, view_dimension_idc field, num_camera_id_minus1 field, camera_id_i_ field, camera_id_id Field, camera_location_per_viewpoint_z[i] field, camera_rotation_per_viewpoint_yaw[i] field, camera_rotation_per_viewpoint_pitch[i] field, camera_rotation_per_viewpoint_roll[i] field, camera_location_per_head_position_x[i] field, camera_location_per_head_position_i_ field, camera field camera_rotation_per_head_position_pitch[i] field, camera_rotation_per_head_position_roll[i] field, left_eye_view_flag[i] field, camera_location_per_eye_x[i] field, camera_location_per_eye_y[i] field, camera_location_per_eye_z[i] field, camera_location_per_eye_z[i] field, camera_ro_ i ] Field, num_subpicture_minus1 field, scene_radius_flag[i] field, local_sphere_center_offset_flag[i] field, local_sphere_rotation_flag[i] field, lens_distortion_correction_flag[i] field, num_camera_idx_minus1 field, camera_idx[i]_region_circle, circular field i] field, rect_region_top[i] field, rect_region_left[i] field, rect_region_width[i] field, rect_region_height[i] field, full_radius[i] field, scene_radius[i] field, local_sphere_rotation_azimuth[i] field, local_sphere_rotation_elevation[i] Field, local_sphere_center_offset_x[i] field, local_sphere_center_offset_y[i] field, local_sphere_center_offset_z[i] field, field_of_view[i] field, lens_projection_type[i] field, scaling_factor[i] field, num_angle_projection_minus1[i] field, angle_projection ] Field, num_polynomial_coeff_projection_minus1[i][j] field, polynomial_coeff_projection[i][j][k] field, num_angle__correction_minus1[i] field, angle_correction[i][j] field, num_polynomial_coeff_correction_minus1[i]cor i][j][k] field, and/or redial_distortion It may include the _type[i] field.

The meanings of the fields are as described above.

Specifically, as an example, the camera lens information may include information indicating a camera type for the target circular region. The camera type may be one of a viewpoint, a head position, and an eye view.

For example, when the value of the information indicating the camera type is 1, the camera type for the target circular region may be derived from the viewpoint, and when the value of the information indicating the camera type is 2, the target The camera type for the circular area may be derived from the head position, and when the information value indicating the camera type is 3, the camera type for the target circular area may be derived from the eye view.

The target circular area may be an image for a camera type indicated by information indicating the camera type, and the camera lens information may include camera type related information for the target circular area. Information indicating the camera type may indicate the camera_id_type[i] field.

For example, the camera lens information may include information indicating the x component, y component, and z component of the target viewpoint for the target circular region. That is, the camera lens information may include information indicating the position of the target viewpoint with respect to the target circular area. Information indicating the x component, y component, and z component of the target viewpoint for the target circular region may indicate the camera_location_per_viewpoint_x[i] field, the camera_location_per_viewpoint_y[i] field, and the camera_location_per_viewpoint_z[i] field. Further, the camera lens information may include information indicating a yaw angle, a pitch angle, and a roll angle of the target viewpoint with respect to the target circular area. That is, the camera lens information may include information indicating the rotation angle of the target viewpoint with respect to the target circular area. Information indicating the yaw angle, pitch angle, and roll angle of the target viewpoint for the target circular region may indicate the camera_rotation_per_viewpoint_yaw[i] field, the camera_rotation_per_viewpoint_pitch[i] field, and the camera_rotation_per_viewpoint_roll[i] field. Meanwhile, in this case, for example, the value of information indicating the camera type may be 1. That is, the camera type for the target circular area indicated by the information indicating the camera type may be the viewpoint.

Or, for example, the camera lens information may include information indicating an x component, a y component, and a z component of a target head position with respect to the target circular region. That is, the camera lens information may include information indicating the position of the target head position with respect to the target circular area. Information indicating the x component, y component, and z component of the target head position for the target circular region may indicate the camera_location_per_head_position_x[i] field, the camera_location_per_head_position_y[i] field, and the camera_location_per_head_position_z[i] field. In addition, the camera lens information may include information indicating a yaw angle, a pitch angle, and a roll angle of the target head position with respect to the target circular area. That is, the camera lens information may include information indicating the rotation angle of the target head position with respect to the target circular region. Information indicating the yaw angle, pitch angle, and roll angle of the target head position with respect to the target circular region includes the camera_rotation_per_head_position_yaw[i] field, the camera_rotation_per_head_position_pitch[i] field, and the camera_rotation_per_head_position_roll[i ] Field. Meanwhile, in this case, for example, the value of information indicating the camera type may be 2. That is, the camera type for the target circular area indicated by the information indicating the camera type may be the head position.

Or, for example, the camera lens information may include information indicating the x component, y component, and z component of the target eye view of the target circular region. That is, the camera lens information may include information indicating a position of a target eye view with respect to the target circular area. Information indicating the x component, y component, and z component of the target eye view for the target circular region may indicate the camera_location_per_eye_x[i] field, the camera_location_per_eye_y[i] field, and the camera_location_per_eye_z[i] field. Further, the camera lens information may include information indicating a yaw angle, a pitch angle, and a roll angle of the target eye view with respect to the target circular area. That is, the camera lens information may include information indicating a rotation angle of a target eye view with respect to the target circular area. Information indicating the yaw angle, pitch angle, and roll angle of the target eye view for the target circular area includes the camera_rotation_per_eye_yaw[i] field, the camera_rotation_per_eye_pitch[i] field, and the camera_rotation_per_eye_roll[i] Field. In addition, the camera lens information may include a flag indicating whether the target eye view of the target circular area is a left eye view. When the value of the flag is 1, the target eye view may be a left eye view, and when the value of the flag is 0, the target eye view may be a right eye view. have. The flag may indicate the left_eye_view_flag[i] field. Meanwhile, in this case, for example, the value of information indicating the camera type may be 3. That is, the eye type may be the camera type for the target circular area indicated by the information indicating the camera type.

Further, as another example, the camera lens information may include information indicating a lens projection type of the target circular region. The lens projection types include perspective projection, stereographic projection, equidistance projection, sine-law projection, equisolid projection and angler poly It may be one of angular polynomial projection.

For example, when the value of the information indicating the lens projection type is 1, the lens projection type of the target circular region may be derived as the perspective projection, and the value of information representing the lens projection type is 2 , The lens projection type of the target circular region may be derived from the stereographic projection, and when the value of the information indicating the lens projection type is 3, the lens projection type of the target circular region is the equidistance projection If it can be derived, and the value of the information indicating the lens projection type is 4, the lens projection type of the target circular region can be derived by the sine-ro projection, and the value of the information representing the lens projection type is When 5, the lens projection type of the target circular region may be derived from the equilisolid projection, and when the value of information indicating the lens projection type is 255, the lens projection type of the target circular region is the angle This can be derived from polynomial projection. Also, when the value of the information indicating the lens projection type is 0, the lens projection type of the target circular region may not be specified. Information indicating the lens projection type may indicate the lens_projection_type[i] field.

In addition, for example, when the lens projection type of the target circular area is the perspective projection, the stereographic projection, the equality projection, the sine-ro projection or the equality projection, the camera lens information is the target It may include information indicating a scaling factor for the circular region. That is, when the value of the information indicating the lens projection type is 1 or more and 5 or less, the camera lens information may include information indicating a scaling factor for the target circular region. Information indicating a scaling factor for the target circular region may indicate the scaling_factor[i] field.

Further, as another example, the camera lens information may include information related to a projection function. The projection function related information may include information indicating the number of projection functions for the target circular region. Information indicating the number of projection functions may indicate the num_angle_projection_minus1[i][j] field.

Further, the projection function related information may include information indicating an angle to which the projection function is applied to the target circular region. Information indicating an angle to which the projection function is applied may indicate the angle_projection[i][j] field.

Further, the projection function related information may include information indicating the number of coefficients of the projection function for the target circular region. Information indicating the number of coefficients of the projection function may indicate the num_polynomial_coeff_projection_minus1[i][j] field.

In addition, the projection function related information may include information indicating the coefficient of the projection function for the target circular region. Information indicating the coefficient of the projection function may indicate the polynomial_coeff_projection[i][j][k] field.

Further, as another example, the camera lens information may include a flag indicating whether information related to a distortion correction function for distortion correction of the target circular region exists. For example, when the value of the flag is 1, the camera lens information may include distortion correction function related information. In addition, when the value of the flag is 0, the camera lens information may not include distortion correction function related information. The flag may indicate the lens_distortion_correction_flag[i] field.

The distortion correction function related information may include information indicating the number of distortion correction functions for the target circular region. Information indicating the number of distortion correction functions may indicate the num_angle_correction_minus1[i][j] field.

Further, the distortion correction function related information may include information indicating an angle to which the distortion correction function is applied to the target circular region. Information indicating an angle to which the distortion correction function is applied may indicate the angle_correction[i][j] field.

In addition, the distortion correction function related information may include information indicating the number of coefficients of the distortion correction function for the target circular region. Information indicating the number of coefficients of the distortion correction function may indicate the num_polynomial_coeff_correction_minus1[i][j] field.

In addition, the distortion correction function related information may include information indicating a coefficient of the distortion correction function for the target circular region. Information indicating the coefficient of the distortion correction function may indicate the polynomial_coeff_correction[i][j][k] field.

The 360-degree video receiving device may derive a distortion correction function for correcting distortion of the target circular region based on the information related to the distortion correction function, and based on the distortion correction function, the distortion of the target circular region ( distortion).

Further, as another example, the camera lens information may include information indicating a type of radial distortion of the target circular region. The radial distortion type may be one of barrel distortion, pincushion distortion, and mustache distortion.

For example, when the value of the information indicating the radial distortion type is 1, the radial distortion type of the target circular region may be derived as the barrel distortion, and the value of information indicating the radial distortion type is 2 , The radial distortion type of the target circular region may be derived by the pincushion distortion, and when the value of the information indicating the radial distortion type is 3, the radial distortion type of the target circular region is derived by the mustache distortion. Can be.

The 360-degree video receiving apparatus may correct the radial distortion of the target circular region based on Equation 1 described above. In this case, the radial distortion coefficient and the tangential distortion coefficient of Equation 1 may be derived based on the radial distortion type of the target circular region.

In addition, the metadata may include additional camera lens information.

The additional camera lens information includes the above-described num_subpictures field, num_camera_idx_minus1 field, camera_idx[i][j] field, image_flip[i] field, image_scale_axis_angle[i] field, image_scale_x[i] field, image_scale_y[i] field, num_angle_for_displaying_fov[i ] Field, displayed_fov[i][j] field, overlapped_fov[i][j] field, num_local_fov_region[i] field, start_radius[i][j] field, end_radius[i][j] field, start_angle[i][ j] field, end_angle[i][j] field, radius_delta[i][j] field, angle_delta[i][j] field, local_fov_weight[i][j][k][l] field, num_polynomial_coefficients_lsc[i] Field, polynomial_coefficient_K_lsc_R[i][j] field, polynomial_coefficient_K_lsc_G[i][j] field, polynomial_coefficient_K_lsc_B[i][j] field, num_deadzones field, deadzone_left_horizontal_offset[i] field, deadzone_top_i_width field, deadzone_top_vertical field /Or may include the deadzone_height[i] field.

The meanings of the fields are as described above.

Meanwhile, the metadata may be received through an SEI message. In addition, the metadata may be included in an adaptation set, representation, or subrepresentation of a media presentation description (MPD). For example, the fisheye video information may be received in the form of a Dynamic Adaptive Streaming over HTTP (DASH) descriptor included in the MPD. Here, the SEI message may be used for decoding a 2D image or assisting display of a 2D image into a 3D space.

The 360-degree video receiving apparatus decodes a picture including a target circular region based on the information on the encoded picture (S1920). The 360-degree video receiving apparatus may decode a picture including the target circular region based on information about the encoded picture. The target circular area may be an area including the 360-degree image.

The 360-degree video receiving device displays a display mode selection screen (S1930). The 360-degree video receiving device may display a display mode selection screen. The display mode selection screen may include an icon representing a fisheye camera mode and an icon representing a viewport video mode.

20 shows an example of a 360-degree video according to the display mode selection screen and display mode. Referring to (a) of FIG. 20, the display mode selection screen may include an icon 2010 representing the fisheye camera mode and an icon 2020 representing the viewport video mode. In addition, the display mode selection screen may include text indicating mode selection. In addition, for example, the icon 2010 representing the fisheye camera mode may include the text "1) Fisheye camera mode, and the icon 2020 representing the viewport video mode is "2) Viewport video mode". May contain text. The user may select one of the icon 2010 representing the fisheye camera mode and the icon 2020 representing the viewport video mode, and a 360-degree video may be displayed according to the display mode selected by the user.

In addition, the left view 2030 of FIG. 20B can represent a 360-degree video displayed in the fisheye camera mode, and the right view 2040 of FIG. 20B is displayed in the viewport video mode. Can show a 360 degree video.

Referring to FIG. 19 again, the 360-degree video receiving apparatus processes and renders the target circular area based on the selected display mode and the camera lens information (S1940). Referring to FIG. 19, when the camera lens information SEI message is present, the 360-degree video receiving device may process and render a fisheye camera image, that is, a picture including the target circular region. In this case, the 360-degree video receiving device may guide the user to change the display mode.

For example, when the icon representing the fisheye camera mode is selected, the 360-degree video receiving device may derive the target circular region from the picture based on the camera lens information, and render without stitching the target circular region By doing so, the displayed 360-degree video can be generated. In this case, the 360-degree video receiving device may display a fisheye camera image suitable for a user's direction without stitching, or display a plurality of fisheye camera images at once.

For example, the 360-degree video receiving apparatus may derive the target circular area from the picture based on the camera lens information. For example, the 360-degree video receiving apparatus may derive the target viewpoint for a specific location based on the camera lens information, and derive the target circular area for the target viewpoint. In addition, as another example, the 360-degree video receiving apparatus may derive the target head position for a specific position based on the camera lens information, and derive the target circular area for the target head position. Further, as another example, the 360-degree video receiving apparatus may derive the target eye view for a specific location based on the camera lens information, and derive the target circular area for the target eye view. Here, the specific location may be a location and/or rotation angle indicated through at least one of a local coordinate system and a global coordinate system, and the specific location is a location selected by a user and/or Or it may be a rotation angle. Further, the local coordinate system may be a coordinate system representing coordinates with x components, y components, and z components, and the global coordinate system may be a coordinate system representing coordinates with a yaw angle, a pitch angle, and a roll angle.

Further, as an example, the camera lens information of the metadata may include information describing the target circular area and information describing a rectangular area to which the target circular area is mapped. The 360-degree video receiving device may derive the rectangular area based on the information describing the rectangular area, and derive the target circular area mapped to the rectangular area based on the information describing the target circular area. . In this case, an area corresponding to the inner intersection of the area to which the rectangular area and the target circular area are mapped may be actual 360-degree image data. The remaining invalid areas can be identified by being marked with black. According to an embodiment, the 360-degree video receiving apparatus may derive an area corresponding to the intersection of the area to which the rectangular areas and the target circular area are mapped.

Further, as an example, the camera lens information may include information about a dead zone to which the 360 degree image data is not mapped. The 360-degree video receiving apparatus may derive a dead zone to which the 360-degree image data is not mapped based on information about the dead zone to which the 360-degree image data is not mapped.

In addition, the camera lens information may include information indicating the lens projection type, and the 360 video receiving device maps the target circular region to 3D space based on a spherical coordinate system mapping formula derived based on the lens projection type. can do. Specifically, for example, the 360 video receiving apparatus may project the target circular area into a plane based on a spherical coordinate system mapping formula derived based on the lens projection type (Projection). Here, the plane may be an ERP (Equirectangular Projection) plane. This projection process may be an intermediate step for re-projecting the target circular region into a 3D space such as a spherical coordinate system.

Further, the camera lens information may include information related to the projection function and/or information related to the distortion correction function. The 360 video reception device may derive a polynomial function for distortion correction of the target circular region based on the projection function related information and/or the distortion correction function related information, and based on the polynomial function The distortion of the circular region can be corrected.

In addition, the camera lens information may include information indicating a radial distortion type of the target circular region, and the 360 video receiving apparatus may perform the radial distortion of the target circular region based on Equation 1 described above. Can be modified.

Further, as another example, when the icon representing the viewport video mode is selected, the 360 video receiving device may derive a target viewpoint for a specific location based on the camera lens information, and the user's viewport from the target viewpoint Areas for are derived, but the areas may include the target circular area. Thereafter, the 360 video receiving device may stitch the regions to generate a 360-degree video for the viewport. Here, the specific location may be a location indicated through at least one of a local coordinate system and a global coordinate system. That is, when the icon representing the viewport video mode is selected, the 360 video receiving device may provide a function of viewing a viewport suitable for user orientation. The 360 video receiving apparatus may select and stitch an image suitable for a viewport and a viewing direction.

For example, the camera lens information may include information indicating the x component, y component, and z component of the target viewpoint. Further, the camera lens information may include information indicating a yaw angle, a pitch angle, and a roll angle of the target viewpoint. The 360-degree video receiving device may derive the target viewpoint for a specific location based on the camera lens information, and derive regions for a user's viewport from the target viewpoint.

Further, for example, the camera lens information may include information indicating the x component, y component and z component of the target head position with respect to the target circular region. In addition, the camera lens information may include information indicating a yaw angle, a pitch angle, and a roll angle of the target head position with respect to the target circular area. The 360-degree video receiving apparatus may derive the target head position for a specific location based on the camera lens information, and derive regions for a user's viewport from the target head position.

Also, for example, the camera lens information may include information indicating the x component, y component, and z component of the target eye view of the target circular region. Further, the camera lens information may include information indicating a yaw angle, a pitch angle, and a roll angle of the target eye view with respect to the target circular area. The 360-degree video receiving device may derive the target eye view for a specific location based on the camera lens information, and derive regions for a user's viewport from the target eye view.

The 360 video receiving device may perform rendering based on the finally synthesized ERP plane (picture) to generate a corresponding viewport. The synthesized ERP plane may represent stitched regions.

The 360 video receiving device displays a 360-degree video generated based on the rendering (S1950). For example, when the icon representing the fisheye camera mode is selected, the 360 video receiving device may display a 360-degree video for the target circular area generated by rendering without stitching. Also, for example, when the icon representing the viewport video mode is selected, the 360 video receiving device may display a 360-degree video generated by stitching regions for a user's viewport.

Further, for example, the displayed 360-degree video may include a map including a plurality of viewpoints and a plurality of icons corresponding to each. In addition, the icons indicate a direction in which a 360-degree video providing service is possible, and the direction indicates information indicating the x component, the y component, and the z component of the viewpoints and the yaw angle, the pitch angle, and the roll angle. It can be derived based on the information it represents. That is, when the received 360-degree image data is data for a service composed of a plurality of viewpoints (for example, when the value of the camera_id_type[i] field is 1), the 360-video receiving device receives the viewpoints Information on the display can be displayed on the map so that the user can select it. For example, as shown in FIG. 21A, which will be described later, the entire map may be displayed on the screen, and a specific part capable of service may be represented. The specific part may be represented using information indicating the camera position and direction of the corresponding viewpoint. The information may include a camera_location_per_viewpoint_x[i] field, a camera_location_per_viewpoint_y[i] field, a camera_location_per_viewpoint_z[i] field, a camera_rotation_per_viewpoint_yaw[i] field, a camera_rotation_per_viewpoint_pitch[i] field, and a camera_rotation_per_viewpoint_roll[i] field. Through this, the user can select a preferred location among the provided viewpoints through the UI, and the 360 video receiving device may display additional information about the location.

In addition, for example, the displayed 360-degree video includes text indicating whether viewpoint movement is possible at a target head position with respect to the target circular area, and an indicator indicating a position where viewpoint movement is possible at the target position. can do. The camera lens information may include information indicating an x component, a y component, and a z component of the target head position with respect to the target circular region, and the camera lens information may require the target head position relative to the target circular region. It may include information indicating a (yaw) angle, a pitch angle, and a roll angle. Whether the viewpoint movement is possible and the position where the viewpoint movement is possible may be derived based on the camera lens information. That is, when the received 360-degree image data is data for a service composed of a plurality of head positions (for example, when the value of the camera_id_type[i] field is 2), a user within a limited range for the selected position Point of view movement may be allowed. In this case, it is possible to induce the user to enjoy the content sufficiently by notifying the user that the viewpoint movement is possible within a limited range. For example, as illustrated in the upper figure of FIG. 21B to be described later, it may indicate that viewpoint movement is possible through text, or may indicate a specific location signaled through a picture. The specific position is a camera_location_per_head_position_x[i] field, a camera_location_per_head_position_y[i] field, a camera_location_per_head_position_z[i] field, a camera_rotation_per_head_orientation_yaw[i] field, a camera_rotation_per_head_orientation_pitch[i]ro field, a number of deductions based on camera_rotation_per_head_orientation_p have. In addition, when the AR/VR content is serviced through a fixed display, an indicator indicating a position capable of moving a viewpoint and that shown in the lower figure of FIG. 21B to be described later may be displayed, and the user may be displayed. Can select one head position.

In addition, for example, the displayed 360-degree video may include a display screen for selecting a target eye view. The display screen may include an icon representing a stereoscopic, an icon representing a left view, and an icon representing a right view. That is, when the received 360-degree image supports stereoscopic, the 360-video receiving device may inform the user to select it. In this case, the user may prefer the monoscopic view, so a left view or a right view may be selected and used depending on the user's main viewpoint.

21 exemplarily shows a 360-degree video displayed. Referring to FIG. 21A, the displayed 360-degree video may include a map 2100 including a plurality of viewpoints and a plurality of icons 2110 respectively corresponding to the icons 2110. May indicate a direction in which a 360-degree video providing service is possible. Referring to FIG. 21B, the displayed 360-degree video includes text indicating whether viewpoint movement is possible at a target head position with respect to the target circular region and an indicator indicating a position where viewpoint movement is possible at the target position. Can be. The upper view of FIG. 21B may indicate text indicating whether a viewpoint movement is possible at a target head position with respect to the target circular region, and the lower view of FIG. 21B indicates a position where viewpoint movement is possible at the target position Indicator 2120. Referring to FIG. 21C, the displayed 360-degree video may include a display screen for selecting a target eye view.

22 schematically shows a 360-degree video receiving apparatus performing a 360-degree image display method according to the present invention. The method disclosed in FIG. 19 may be performed by the 360-degree video receiving device disclosed in FIG. 22. Specifically, for example, the receiver of the 360-degree video receiving device of FIG. 22 may perform S1900 of FIG. 19, and the receiving processor of the 360-degree video receiving device of FIG. 22 may perform S1910 of FIG. 19. , The data decoder of the 360-degree video receiving device of FIG. 22 may perform S1720 of FIG. 19, and the renderer of the 360-degree video receiving device of FIG. 22 may perform S1940 of FIG. 19, The display unit of the 360-degree video receiving device may perform S1930 and S1950 of FIG. 19.

According to the present invention described above, a user interface for a display mode can be displayed, and through this, an interactive experience can be provided to a user who consumes 360 content.

In addition, according to the present invention, a user interface for a plurality of viewpoints, head positions or eye views can be displayed more efficiently.

The above-described steps may be omitted or replaced by other steps performing similar/same operation according to an embodiment.

The 360-degree video transmission apparatus according to an embodiment of the present invention may include the above-described data input unit, stitcher, signaling processing unit, projection processing unit, data encoder, transmission processing unit, and/or transmission unit. Each internal component is as described above. The 360-degree video transmission apparatus and its internal components according to an embodiment of the present invention may perform the above-described embodiments of a method for transmitting a 360-degree video of the present invention.

The 360-degree video receiving apparatus according to an embodiment of the present invention may include the above-described receiving unit, receiving processing unit, data decoder, signaling parser, re-projection processing unit, renderer and/or display unit. Each internal component is as described above. The 360-degree video receiving apparatus and its internal components according to an embodiment of the present invention may perform the above-described embodiments of a method for receiving a 360-degree video of the present invention.

The internal components of the above-described apparatus may be processors that execute successive processes stored in memory, or may be hardware components composed of other hardware. These can be located inside/outside the device.

The above-described modules may be omitted or replaced by other modules performing similar/same operations according to embodiments.

Each of the above-described parts, modules, or units may be a processor or a hardware part that executes continuous execution processes stored in a memory (or storage unit). Each of the steps described in the above-described embodiment may be performed by a processor or hardware parts. Each module/block/unit described in the above-described embodiment can operate as a hardware/processor. Also, the methods proposed by the present invention can be executed as code. This code can be written to a storage medium that can be read by a processor, and thus can be read by a processor provided by an apparatus.

In the above-described embodiment, the methods are described based on a flowchart as a series of steps or blocks, but the present invention is not limited to the order of steps, and some steps may occur in a different order or simultaneously with other steps as described above. have. In addition, those skilled in the art will understand that the steps shown in the flowchart are not exclusive, other steps may be included or one or more steps in the flowchart may be deleted without affecting the scope of the present invention.

When embodiments are implemented in software in the present invention, the above-described method may be implemented as a module (process, function, etc.) that performs the above-described functions. Modules are stored in memory and can be executed by a processor. The memory may be internal or external to the processor, and may be connected to the processor by various well-known means. The processor may include an application-specific integrated circuit (ASIC), other chipsets, logic circuits, and/or data processing devices. The memory may include read-only memory (ROM), random access memory (RAM), flash memory, memory cards, storage media and/or other storage devices.

Claims (15)

The camera lens information is a 360-degree image display method performed by the target circular 360-degree video receiving device, Receiving 360 degree image data; Obtaining information and metadata about an encoded picture from the 360-degree image data, wherein the metadata includes camera lens information; Decoding a picture including a target circular region based on the information on the encoded picture; Displaying a display mode selection screen; Processing and rendering the target circular area based on the selected display mode and the camera lens information; And Displaying a 360-degree video generated based on the rendering, The display mode selection screen is a 360-degree video display method, characterized in that it comprises an icon representing a fisheye camera mode and an icon representing a viewport video mode. According to claim 1, When the icon representing the fisheye camera mode is selected, the rendering step may include: Deriving the target circular region from the picture based on the camera lens information; And And generating the displayed 360-degree video by rendering without stitching the target circular region. According to claim 1, When an icon representing the viewport video mode is selected, the rendering step may include: Deriving a target viewpoint for a specific location based on the camera lens information; And Deriving regions for the user's viewport from the target viewpoint, the regions including the target circular region; Generating a 360-degree video for the viewport by stitching the regions, The specific position is a local coordinate system (local coordinate system) and a global coordinate system (global coordinate system) 360-degree image display method, characterized in that the position indicated through at least one of. According to claim 3, The camera lens information includes information indicating the x component, y component, and z component of the target viewpoint. According to claim 3, The camera lens information includes information indicating yaw angle, pitch angle, and roll angle of the target viewpoint. According to claim 1, The displayed 360-degree video includes a map including a plurality of viewpoints and a plurality of icons respectively corresponding to the 360-degree image display method. The method of claim 6, The camera lens information includes information indicating the x component, y component, and z component of the viewpoints, and information representing yaw angle, pitch angle, and roll angle, The icons indicate a direction in which a 360-degree video providing service is possible, The direction is derived based on the information representing the x component, the y component and the z component of the viewpoints and the information representing the yaw angle, the pitch angle and the roll angle. . According to claim 1, The displayed 360-degree video includes a text indicating whether viewpoint movement is possible at a target head position with respect to the target circular area and an indicator indicating a position where viewpoint movement is possible at the target position. Figure video display method. The method of claim 8, The camera lens information includes information indicating an x component, a y component, and a z component of the target head position with respect to the target circular region. The method of claim 9, The camera lens information includes information indicating a yaw angle, a pitch angle and a roll angle of the target head position with respect to the target circular region. According to claim 1, The displayed 360-degree video includes a display screen for selecting a target eye view, The display screen includes a stereoscopic (stereoscopic) icon, a left view (left view) icon and a light view (right view) icon, characterized in that it comprises a 360-degree image display method. The method of claim 11, The camera lens information includes information indicating the x component, y component and z component of the target eye view. The method of claim 11, The camera lens information includes information indicating a yaw angle, a pitch angle, and a roll angle of the target eye view of the target circular area. According to claim 1, The camera lens information includes information indicating a camera type for the target circular region, The camera type is a viewpoint (viewpoint), head position (head position) and eye view (eye view), characterized in that the 360-degree image display method. The method of claim 14, When the value of the information indicating the camera type is 1, the camera type for the target circular area is derived from the viewpoint, When the value of the information indicating the camera type is 2, the camera type for the target circular area is derived as the head position, When the value of the information indicating the camera type is 3, a camera type for the target circular area is derived by the eye view.

Images