KR20170015938A - Methods and apparatus for delivering content and/or playing back content - Google Patents

Abstract

Content delivery and playback methods and apparatus are described. The methods and apparatus are suitable for delivery and playback of content in response to a 360 degree environment and are capable of streaming 3D content corresponding to events such as sports games while the content, e.g., And / or to support real-time delivery. Portions of the environment are captured by cameras located at different locations. The content captured from the different locations is encoded and made available for delivery. The playback device selects the content to be received based on the position of the user's head. The streams may be prioritized and selected for delivery based on the current view of the user and / or the direction of head rotation. Still images or composited images may be used and combined with content from one or more streams, for example, for background, sky and / or ground positions.

Description

[0001] METHODS AND APPARATUS FOR DELIVERING CONTENT AND OR PLAYING BACK CONTENT [0002]

The present invention relates to content delivery and / or playback, for example, playback of stereoscopic image content.

Display devices intended to provide an immersive experience typically cause the user to turn his head to experience a corresponding change in the scene being displayed. Head mount displays sometimes support 360 degree viewing where the scene being displayed changes as the user's head position changes, allowing the user to turn around with the head mount display.

With these devices, the user is provided with a scene captured in front of the camera position when viewed forward and a scene captured behind the camera position when the user turns completely. The user's field of view at any given time is generally limited to 120 degrees or less due to the nature of the person's ability to recognize the limited field of view at any given time while the user is turning his head backwards.

To support a 360 degree view, a 360 degree scene can be captured using multiple cameras together and the images are combined to create a 360 degree scene that will be made available for viewing.

The 360 degree view is typically captured for general television and many other video applications that do not have the opportunity to change the viewing angle used to determine the image to be displayed at a particular point in time so that much more than the encoded simple front view It should be understood that it includes image data.

Given the transmission constraints, for example, network data restrictions associated with streamed content, it may not be possible to stream the entire 360 degree view into full high definition video to all customers who want to receive and interact with the content . This is particularly the case where the content is stereoscopic video content that includes image content intended to correspond to left and right eye views to allow 3D viewing effects.

In view of the above discussion, a method for supporting the streaming and / or playback of content in a manner that allows an individual user to change his viewing position, for example, by turning his or her head, ≪ RTI ID = 0.0 > and / or < / RTI > It is desirable for the user to be provided with the option to change the viewing position of his head position accordingly while staying within data streaming limits that can be applied by bandwidth or other transfer related restrictions. At least some embodiments, although not necessarily for all embodiments, allow a plurality of users of different locations to receive streams at the same time, and to determine which portions or portions of the environment It is desirable to see any distinct parts.

Methods and apparatus for supporting delivery, e.g., streaming, of video or other content corresponding to a 360 degree viewing area are described. The methods and apparatus of the present invention are particularly advantageous in that the methods and apparatus of the present invention are particularly advantageous in that the data transmission constraints can make it difficult to deliver 360 degrees of transmission of content, e.g., using best quality coding and the highest supported frame rate, And / or streaming of other image content. However, methods are not limited to stereoscopic content.

In various embodiments, a 3D model of the environment in which the video content is obtained and / or corresponding 3D dimensional information is generated and / or accessed. Camera locations are recorded in the environment. A number of distinct camera positions may be provided in the environment. For example, separate final target camera positions and one or more intermediate field camera positions may be supported and used to capture real-time camera feeds.

The 3D model and / or other 3D information is stored in an image capture device or server used to stream video to one or more users.

The 3D model is provided to a user playback device, e.g., a customer premise device with image rendering and compositing capabilities. The customer premises device creates a 3D representation of the environment that is displayed to the user of the customer premises device, for example, via the head mounted display.

In various embodiments, less than a full 360 degrees of environment is streamed to an individual customer premise device at any given time. The customer premises device indicates that the camera feed is streamed based on user input. The user may select the compartment and / or the camera location via a portion of the customer premises device or an input device attached thereto.

In some embodiments, the 180 degree video stream is transmitted from the server and / or video cameras responsible for streaming the content to the customer playback device, for example, in real-time, real-time, or near-real-time stream. The playback device monitors the head position of the user, and thus the playback device recognizes the viewing area that the user of the playback device sees in the 3D environment created by the playback device. The customer premise device provides video when available for a portion of the 3D environment being viewed and the video content is replaced or displayed as an alternative to the simulated 3D environment to be provided in the absence of video content. When a user of the playback device turns his head, portions of the environment provided to the user may be supplied from the video content to the playback device, e.g., streamed, and other portions may be generated by compositing from the 3D model and / Lt; RTI ID = 0.0 > image content < / RTI >

Thus, the playback device can display video, for example, supplied via streaming, while a game, music concert, or other event is still pending, corresponding to, for example, a frontal 180 degree camera view, The rear and / or side portions are produced from the image content of the side or rear regions of the environment in complete synthesis or at different times.

A user may select camera locations by signaling a change in location to a server providing streaming content, but a server providing streaming content may provide information useful for creating a composite environment for parts of the non-streaming 3D environment .

 For example, in some embodiments, a plurality of back and side views are captured at different times, e.g., before streaming a portion of the content or from a point earlier in time. The images are buffered in the playback device. The server providing the content can signal to the playback device and signal it in some embodiments that either non-real-time scenes or a set of images are used for compositing environment portions that are not supplied to the video stream. For example, images of the sitting concert participants and other images of the concert participants behind the camera location may be supplied and stored on the playback device. The server can signal which of the set of stored image data should be used at a particular point in time. Thus, when the audience is standing, the server can signal that an image corresponding to the standing audience is to be used for background 180 degrees view during image composition, while when the audience is sitting, Or it may indicate to the customer premises device that image or image composition information corresponding to a sitting audience should be used when synthesizing the rear portions.

In at least some embodiments, the orientation of the cameras in each of the one or more locations in the 3D environment is tracked during image capture. Markers and / or identification points in the environment may facilitate the alignment and / or other mapping of images captured in a previously modeled and / or mapped 3D environment, e.g., live images, to be simulated by a customer premise device Lt; / RTI >

The combination of composite environment portions and real-time (streaming video) provides an immersive video experience. The environments may be measured or modeled using 3D photometry to generate 3D information used to simulate the environment when video is not available, e.g., if the environment has not previously been modeled, Measured or modeled.

The use of fiducial markers in the real-world space of determined positions helps with the calibration and alignment of the video with the previously created 3D model.

Location tracking of each camera is performed when the video is captured. The camera position information relates to the locations of X, Y, Z, and deviations to angles (so we know where each camera is pointed). This allows for easy detection of where the captured image corresponds to in the environment, and automatically transfers our video capture to the composite environment created by the playback device during image display, when delivered to the playback device with the captured video Playback for overlaying, for example, allowing playback to the user. The streamed content may be limited to a view of less than 360 degrees, for example, a captured 180 degree view of the area of the front of the camera location. When the viewer looks around, when turning to the back, the viewer will see the simulated background (not the black cavity), and the viewer will watch the video when turning to the front.

Composite environments can interact and interact in some environments. In some embodiments, a plurality of actual viewers, e.g., users of different customer premises devices, are included in the simulated environment, allowing the user to view games with his friends in a virtual 3D environment, To be seen.

The images of the users may be captured by the cameras included with or attached to the customer premise devices for use in creating the simulated environment, or may be captured in some embodiments, provided to the server, Users, for example, members of the group. User images may not be real-time images but may be real-time images.

Methods may be used to encode and provide content in real time or near real time, but are not limited to these real-time applications. If the ability to support real-time and near-real-time encoding and streaming to multiple users is provided, the methods and apparatus described herein allow individuals to view events, observe the stage or field, It is suitable for streaming scenes of sporting events, concerts, and / or other places that are recognizable by turning to the playground or the views of the audience. 360 degrees viewing and 3D support, the methods and apparatus of the present invention allow the user to view a scene from different viewing angles when the user is present in the environment and is turning the user's head left, right, And is intended for use with head-mounted displays intended to provide a 3D immersion experience through the degree of freedom to observe by turning.

Methods and apparatus for delivering image content, e.g., content corresponding to a 360 degree field of view, are described. In various embodiments, the field of view corresponds to different portions of the environment, such as the frontal portion, at least one rear portion, the top, and the bottom. In some embodiments, the left and right rear, e.g., rear, portions of the environment are separately generated and / or communicated. The playback device monitors the position of the user's head and generates stereoscopic images corresponding to images, e.g., a portion of the environment the user is viewing at a given time, which is then displayed to the user. In the case of stereoscopic image reproduction, individual left and right eye images are generated. The generated images may correspond to one or more scenes, e.g., environmental portions, and in some embodiments, corresponding thereto.

At the beginning of playback, the head level position looking forward of the user is set to correspond to the front scene portion by default. When the user turns his head and / or raises or lowers his head, other parts of the environment can come into view of the user.

The bandwidth and image decoding capabilities for many playback devices are limited by the bandwidth and / or throughput of the device to receive the image content. In some embodiments, the playback device determines which portion of the environment corresponds to the user's primary view. The device then selects the portion to be received at full resolution by the stream designated as a higher rate, e.g., from a priority view, as the primary stream. Content from one or more other streams providing content corresponding to other parts of the environment is also received, but is typically received at a lower data rate. Content delivery for a particular stream may be initiated by the playback device, for example, by sending a signal used to trigger content delivery. The signal may be used to provide content corresponding to a portion of the environment or to connect to a multicast group that initiates delivery of switched digital broadcasts. In the case of broadcast content that does not request a request or other signal, such as a multicast group connection signal, the device may initiate reception by tuning the content to an available channel.

Assuming that the users are primarily interested in the front view portion of the environment, especially since the main operation is generally performed when the content particularly corresponds to a sporting event, a rock concert fashion show or a number of different events, in some embodiments, The forward view portion of the environment is provided with a data transmission priority. In at least some embodiments, the images corresponding to the forward view position are streamed at a higher rate than one or more other portions of the 360 degree environment. Images corresponding to different parts of the environment are transmitted at lower data rates or transmitted as still images. For example, one or more still images of the top, e.g., sky and bottom, e.g., terrestrial, may be transmitted.

In some embodiments, a plurality of still captured images are transmitted for one or more portions of the environment, e.g., the rear view portion (s), or the sky portion. In some embodiments, control information is transmitted that indicates which of the still images should be used at a given time for a portion of the environment. When still images for a portion of the environment are transmitted, they may be transmitted in encoded form and then stored in memory in decoded form for use in combination with other image content. In this way, the decoding resources required during the event can be reduced since multiple streams need not be decoded simultaneously at the same frame rate. Still images can be sent before streaming the content of the main event. Alternatively, some images for different parts of the environment may be transferred and stored if they are needed during playback, given changes in the user's head position from the front viewing position. The stalled or uncommon images may be encoded and transmitted as part of the content stream that provides the main, e.g., forward, content for the viewing direction, or may be transmitted as a separate content stream.

The still images corresponding to the rear may be images captured before the event, sometimes such images, and the content corresponding to the front portion of the environment includes content that is captured and streamed, for example, in real time while the event is in progress And in many cases, including him.

For example, consider the case where two different rearview scenes are transmitted and stored in the playback device. One scene can correspond to a standing posture, and the other image can correspond to an audience in a sitting posture. The control information may indicate whether or not a user is sitting in a standing position or standing posture image at a given time when the user must turn his or her head back to a visible position, or, in some embodiments, indicate .

Similarly, multiple images in the sky may be delivered to the playback device and stored in the playback device in an encoded or decoded form. In some embodiments, which image in the sky portion is to be used at a given time is passed to the control information. In other embodiments, which scene in the sky is to be used is automatically determined based on the brightness of one or more images corresponding to a foreground scene area having a sky portion that is coincident with or close to the selected forward environment scene portion, , The bright forward scene area can be detected and used to control the selection of a bright sky image with fewer clouds. Similarly, in some embodiments detection of a dark front environmental zone will result in a dark cloudy sky image being used.

In instances where an image corresponding to a portion of the environment of the field of view is not available, the scene portion may be synthesized from, for example, content or information from other parts of the available environment. For example, if the rear image portion is not available, the content forming the left and / or right side of the front scene region may be copied and used to fill the missing rear portions of the environment. In addition to content duplication, blurring and / or other image processing operations may be used to fill in missing portions of the environment in some embodiments. Alternatively, in some embodiments, the creation information is provided in the content stream and the playback device completely generates the composite images for the missing portions. Such content, such as video game content, may be substantially realistic and may contain a wide variety of image effects and content generated from creation and / or other image generation rules stored in the playback device.

According to some embodiments, an exemplary method of operating a playback system includes determining a head position of a viewer corresponding to a current view; Receiving a first content stream that provides content corresponding to a first portion of the environment; At least some received content included in the first content stream, and i) stored content corresponding to a second portion of the environment, or ii) one corresponding to the current view, based on a composite image simulating a second portion of the environment Generating output images; And outputting or displaying a first output image, which is one of the one or more generated output images. According to some embodiments, an exemplary content playback system includes: a viewer head positioning module configured to determine a head position of a viewer, corresponding to a current view; A content stream receiving module configured to receive a first content stream providing content corresponding to a first portion of the environment; At least some received content included in the first content stream, and i) stored content corresponding to a second portion of the environment, or ii) one corresponding to the current view, based on a composite image simulating a second portion of the environment An output image content stream based generation module configured to generate the output images; And an output module configured to output the first output image or a display module configured to display the first output image.

Many variations and embodiments are possible and are discussed in the following detailed description.

The present invention provides methods and apparatus for delivering improved content and / or reproducing content.

1 illustrates an exemplary system implemented in accordance with some embodiments of the present invention that may be used to capture and stream content for subsequent display by one or more users with one or more synthesized portions of the environment FIG.
FIG. 2A illustrates an exemplary stereoscopic image scene, for example, a non-segmented full 360 degree stereoscopic scene; FIG.
FIG. 2B illustrates an exemplary stereoscopic image scene divided into three exemplary scenes in accordance with an exemplary embodiment; FIG.
FIG. 2C illustrates an exemplary stereoscopic image scene divided into four scenes in accordance with one exemplary embodiment; FIG.
Figure 3 illustrates one exemplary process for encoding an exemplary 360 degree stereoscopic scene in accordance with an exemplary embodiment;
4 illustrates an example of how an input image portion is encoded using various encoders to produce different encoded versions of the same input image portion;
Figure 5 shows stored encoded portions of an input stereoscopic scene divided into three parts;
Figure 6 is a flow diagram illustrating steps of one exemplary method for streaming content in accordance with one exemplary embodiment implemented using the system of Figure 1;
Figure 7 illustrates an exemplary content delivery system that includes an encoding capability that can be used to encode and stream content in accordance with aspects of the present invention.
Figure 8 illustrates an exemplary content playback system that may be used to receive, decode, and display content streamed by the system of Figure 7;
Figure 9 shows an exemplary camera rig having three camera pairs mounted at three different mounting locations with a calibration target that can be used to calibrate the camera rig.
Figure 10 shows a more focused view of a camera rig having three camera pairs mounted on the camera rig.
11 is a detailed view of an exemplary camera rig implemented in accordance with an exemplary embodiment.
12 illustrates an exemplary 360 scene scene, e.g., a 360 scene area that can be divided into different viewing areas / portions corresponding to different camera positions of each of the cameras capturing different portions of the 360 scene FIG.
FIG. 13 illustrates three exemplary 360 scene regions of FIG. 12 that can be captured by different cameras that correspond to and / or are positioned to include viewing regions / portions of an exemplary 360 scene region. BRIEF DESCRIPTION OF THE DRAWINGS Fig.
Figure 14A illustrates a first portion of a flowchart illustrating steps of an exemplary method of operating a playback device in accordance with one exemplary embodiment of the present invention.
Figure 14B illustrates a second portion of a flowchart illustrating steps of an exemplary method of operating a playback device in accordance with an exemplary embodiment of the present invention.
14 includes a combination of Figs. 14A and 14B. Fig.
15 is a flow chart illustrating the steps of a stream selection subroutine in accordance with one exemplary embodiment.
Figure 16 is a flow diagram illustrating the steps of a stream prioritization subroutine in accordance with one exemplary embodiment.
17 is a flow diagram illustrating steps of rendering a subroutine in accordance with one exemplary embodiment.
18 is an exemplary table that includes stream information corresponding to a plurality of content streams.
Figure 19 illustrates an exemplary playback system implemented in accordance with the present invention.
20A illustrates a first portion of a flowchart of one exemplary method of operating a content playback system in accordance with one exemplary embodiment.
Figure 20B illustrates a second portion of a flowchart of one exemplary method of operating a content playback system in accordance with an exemplary embodiment;
20C illustrates a third portion of a flowchart of one exemplary method of operating a content playback system in accordance with an exemplary embodiment.
20D illustrates a fourth portion of a flowchart of one exemplary method of operating a content playback system in accordance with an exemplary embodiment.
Figure 20E illustrates a fifth portion of a flowchart of one exemplary method of operating a content playback system in accordance with an exemplary embodiment;
20 is a view including a combination of Figs. 20A, 20B, 20C, 20D and 20E. Fig.
21 is an illustration of an exemplary content playback system, e.g., a content playback device or computer system coupled for display in accordance with one exemplary embodiment.
Figure 22 is an illustration of one exemplary assembly of modules that may be included in the exemplary content playback system of Figure 21;
Figure 23 illustrates an exemplary stream selection module that may be used in the playback system of Figure 19 in accordance with some embodiments.
24 depicts an exemplary stream prioritization module that may be implemented as part of the stream selection module of FIG. 23 or as a separate module.

Figure 1 illustrates an exemplary system 100 implemented in accordance with some embodiments of the present invention. The system 100 supports content delivery, e.g., imaging content delivery, to one or more customer devices, e.g., playback devices / content players located in customer premises. The system 100 includes an exemplary image capturing device 102, a content delivery system 104, a communications network 105, and a plurality of customer premises 106, ..., 110. The image capturing device 102 supports capturing a stereoscopic image. The image capturing device 102 captures and processes imaging content in accordance with the features of the present invention. The communication network 105 may be, for example, a hybrid fiber-coaxial (HFC) network, a satellite network, and / or the Internet.

The content delivery system 104 includes an encoding device 112 and a content streaming device / server 114. Encoding device 112 may comprise one or more encoders for encoding image data in accordance with the present invention, and in some embodiments thereof. The encoders can be used simultaneously to encode different parts of the scene and / or to encode the provided part of the scene to produce encoded versions with different data rates. Using multiple encoders at the same time can be particularly useful when real time or near real time streaming is supported.

The content streaming device 114 is configured to stream, e.g., transmit, the encoded content over the communication network 105, for example, to deliver the encoded image content to one or more customer devices. Through the network 105, the content delivery system 104 may be located in a customer premises 106, 110 shown in the figure by a link 120 that transmits information and / or traverses the communication network 105 You can exchange information with devices.

The encoding device 112 and the content delivery server 114 are shown as separate physical devices in the example of Figure 1, but in some embodiments they are implemented as a single device that encodes and streams the content. The encoding process may be a 3D, e.g., stereoscopic, image encoding process, encoded and embedded in the encoded image data so that the information corresponding to the left and right eye views of the scene portion may be supported by the 3D image viewing . The particular encoding method used is not critical to the present application and various encoders may be used as the encoding device 112 or to implement the encoding device 112. [

Each customer premise 106,110 may include a plurality of devices / players, e.g., playback systems, used to decode and play back / display the streamed imaging content by the content streaming device 114 have. Customer premise 1 106 includes a decoding device / playback device 122 coupled to display device 124 and customer premise N 110 includes a decoding device / playback device 126 coupled to display device 128. [ . In some embodiments, the display devices 124,128 are head-mounted stereoscopic image display devices. In some embodiments, the playback device 122/126 and head mount device 124/128 together form a playback system.

In various embodiments, the decoding device 122, 126 represents the imaging content on the corresponding display devices 124, 128. The decoding device / players 122, 126 may decode the received imaging content from the content delivery system 104 that generates the imaging content using the decoded content and may provide the imaging content 124, For example, devices capable of rendering 3D image content. Either decoding device / playback devices 122, 126 may be used as the decoding device / playback device 800 shown in FIG. A system / playback device such as that shown in Figures 8 and 19 may be used as either of the decoding device / playback devices 122, 126.

FIG. 2A illustrates an exemplary, non-segmented, stereoscopic image 200, for example, a full 360 degree stereoscopic view. A stereoscopic image scene may be the result of, and is generally the result of, combining a plurality of cameras, e.g., image data captured from cameras, often mounted on a single video capture platform or camera mount.

Figure 2B illustrates a partitioned version 250 of an exemplary stereoscopic image scene 200 in which the scene is divided into 3 (N = 3) exemplary portions, for example, 180 degrees front according to one exemplary embodiment The left rear 90 degree portion, and the right second half 90 degree portion.

FIG. 2C illustrates another divided version 280 of an exemplary stereoscopic image scene 200 divided into 4 (N = 4) portions according to one exemplary embodiment.

2B and 2C illustrate two exemplary portions, but it should be understood that other portions are possible. For example, the scene 200 may be divided into 12 (n = 12) 30 degree parts. In one such embodiment, rather than individually encoding each portion, multiple portions are grouped together and encoded as a group. Different groups of parts are corresponding to different parts of the image that can be streamed, depending on the viewing angle measured on the user's head position, e.g., a range of 0 degrees to 360 degrees, by the total angles of the scene And may be encoded and streamed to the user.

FIG. 3 illustrates an exemplary process for encoding an exemplary 360 degree stereoscopic scene in accordance with one exemplary embodiment. The input to the method 300 shown in Figure 3 includes, for example, 360 degree stereoscopic image data captured by a plurality of cameras configured to capture a 360 degree view of the scene. Stereoscopic image data, for example, stereoscopic video, can be any of a variety of known formats and, in most embodiments, includes left and right eye image data used to allow a 3D experience. Although the methods are particularly suitable for stereoscopic video, the techniques and methods described herein may also be applied to 2D images of, for example, a 360 degree or smaller scene area.

In step 304, the scene data 302 is divided into data corresponding to different scene regions, for example, N scene regions corresponding to different viewing directions. For example, in one embodiment, such as that shown in FIG. 2B, the 360 degree scene region is divided into three portions: a left rear portion corresponding to a 90 degree portion, a front 180 degree portion, and a right rear 90 degree portion . The different parts may have been captured by different cameras, but this is not necessarily the case, and indeed the 360 degree scene is edited from the data captured from multiple cameras before being divided into the N scene areas shown in Figures 2b and 2c .

In step 306, data corresponding to different scene portions is encoded in accordance with the present invention. In some embodiments, each scene portion is independently encoded by a plurality of encoders to support a plurality of possible bitrate streams for each portion. In step 308, the encoded scene portions are stored in the content delivery server 114 of the content delivery system 104 for streaming to, for example, customer playback devices.

4 is an illustration 400 illustrating an example of how an input image portion, e.g., a 180 degree frontal portion of a scene, is encoded using various encoders to produce different encoded versions of the same input image portion, to be.

As shown in the diagram 400, the input scene portion 402, e.g., the 180 degree frontal portion of the scene, is supplied to a plurality of encoders for encoding. In this example, there are K different encoders that encode input data using different encoding techniques with different resolutions to produce encoded data to support different data rate streams of image content. The plurality of K encoders may include a high definition (HD) encoder 1 404, a standard definition (SD) encoder 2 406, a reduced frame rate SD encoder 3 408, ..., K (410).

HD encoder 1 404 is configured to perform full high definition (HD) encoding to produce a high bit rate HD encoded image 412. SD encoder 2 406 is configured to perform low resolution standard image quality encoding to generate SD encoded version 2 414 of the input image. Reduced frame rate SD encoder 3 408 is configured to perform reduced frame rate low resolution SD encoding to produce a reduced rate SD encoded version 3 (416) of the input image. The reduced frame rate may be, for example, half the frame rate used by SD encoder 2 406 for encoding. The high compression reduction frame rate SD encoder K 410 is configured to perform reduced frame rate low resolution SD encoding by a high compression to produce a highly compressed reduced rate SD encoding version K 420 of the input image.

Thus, control of spatial and / or temporal resolution may be used to generate data streams of different data rates, and control of other encoder settings, such as level of data compression, may also be used alone, May be used in addition to spatial and / or temporal resolution control to produce data streams corresponding to the scene portion by the user.

FIG. 5 shows stored encoded portions 500 of an input stereoscopic scene divided into three exemplary portions. The stored encoded portions may be stored in the content delivery system 104, for example, as data / information in the memory. The stored encoded portions 500 of the stereoscopic image scene include three different sets of encoded portions, each portion corresponding to a different scene portion, each set comprising a plurality of different encoded < RTI ID = 0.0 > Versions. Each encoded version is a version of the encoded video data and thus represents a plurality of coded frames. When each encoded version 510, 512, 516 of the video corresponds to a plurality of time periods and when streaming, portions corresponding to time periods to be reproduced, for example, are used for transmission purposes Should be understood.

As shown and discussed above with respect to FIG. 4, each scene portion, e.g., front and back, may be encoded using a plurality of different encoders to produce K different versions of the same scene portion have. The outputs of the respective encoders corresponding to the input scene provided are grouped together and stored as one set together. The first set of encoded scene portions 502 corresponds to a frontal 180 degree scene portion and includes encoded version 1 510 of a frontal 180 degree scene, encoded version 2 512 of a frontal 180 degree scene, And an encoded version K 516 of a frontal 180 degree scene. The second set of encoded scene portions 504 corresponds to a scene portion 2, e.g., a 90 degree left rear scene portion, encoded version 1 520 of a 90 degree left rear scene portion, An encoded version 2 522 of the scene portion, ..., and an encoded version K 526 of the left rear scene portion of 90 degrees. The third set of similarly encoded scene portions 506 corresponds to the scene portion 3, e.g., a 90 degree right rear scene portion, and the encoded version 1 530, 90 degrees An encoded version 2 (532) of the right rear scene portion, and an encoded version K 536 of the right rear scene portion of 90 degrees.

 The various different stored encoded portions of the 360 degree scene may be used to generate various different bitrate streams for transmission to customer playback devices.

FIG. 6 is a flowchart 600 illustrating steps of an exemplary method of providing image content in accordance with an exemplary embodiment. The method of flowchart 600 is implemented in some embodiments using the capturing system shown in FIG.

The method starts in step 602, for example, by a delivery system that is powered and initialized. The method proceeds from start step 602 to steps 604. At step 604, the content delivery system 104, for example, the server 11 in the system 104, may request a request for content, e.g., a previously encoded program, or in some cases, And receives a request for a live event to be encoded and streamed in near real time, for example, while the event is still pending.

In response to the request, at step 606, the server 114 determines the data rate available for delivery. The data rate may be determined from information contained in the request indicating supported data rates and / or other information such as network information indicating the maximum bandwidth available for delivery to the device requesting the content. As will be appreciated, the available data rate may vary depending on the network load and may vary over a period of time during which the content is streamed. Changes are reported by the user device or are difficult to support the data rate at which the packets are dropped or the network is used and that the current available data rate is lower than the original data rate determined to be available for use. May be detected from messages or signals indicating that the amount of time is delayed beyond the amount of time.

Operation proceeds from step 606 to step 608 where the current head position of the user device from which the request for content is initiated, e.g., the current head position at the request time, is 0 degrees. The zero degree or forward facing position may be reinitialized in some embodiments by the user by the playback device signaling that reinitialization will occur. For example, with respect to the original head position, changes in the user's head position and / or the user's head position over time are reported to the content delivery system 104 and the update position is determined by the content delivery decisions It is used to perform.

Operation proceeds from step 608 to step 610, where portions of the 360 degree scene corresponding to the requested content are transmitted to initialize the playback device. In at least some embodiments, the initialization includes transmitting a full 360 degree set of scene data, e.g., N portions, and the 360 degree scene is divided into N portions.

As a result of the initialization in step 610, the playback device will have scene data corresponding to each of the different portions of the 360-viewable viewing area. Thus, if a user of the playback device suddenly turns rearward, at least some data will be available for display to the user, even if the user is not as recent as the portion he or she was watching before turning his head.

Operation proceeds from step 610 to steps 612 and 622. [ Step 622 corresponds to the full scene update path used to ensure that the playback device receives an updated version of the entire 360 degree scene in at least one full update period at least. Once initialized in step 610, the entire update process is delayed in the wait step 622 for a predetermined period of time. Thereafter, in step 624, a 360 degree scene update is performed. The dashed arrows 613 represent the transfer of information that the scene portions are transferred to the playback device during the support period corresponding to step 622. [ At step 624, a full 360 degree scene may be transmitted. However, in some embodiments, not all of the parts are transmitted in step 624. Portions of the scene that have been updated during the waiting period 622 are performed in step 624 in some embodiments because they are already being refreshed during normal streaming processes that transmit at least some portions of the scene based on the user & Is omitted from the updated update.

Operation proceeds from step 624 back to waiting step 622, and the wait is performed before the next full update. It should be appreciated that different overall refresh rates may be supported by adjusting the wait period used in step 622. [ In some embodiments, the content server selects a waiting period based on the type of scene content provided and a corresponding overall reference period is provided. In the case of sports events where the main operation is in the face-to-face area and one of the reasons for the refresh is possible changes in the outdoor lighting conditions, the waiting period may be relatively long, for example, Lt; / RTI > In the case of a rock concert in which the movement and activity of the audience can be changed frequently because different songs are played, the user may want to turn around and see the reaction of the audience and in addition to what is happening in the front stage viewing area, , The overall refresh rate may be higher than sports games, and sometimes higher.

In some embodiments, the entire baseline period is changed as a function of the portion of the representation being streamed. For example, during the game portion of a sporting event, the overall refresh rate may be relatively low, but the latter touch, where there is a greater possibility that the person in the arena or watching the game through the playback device will turn his head from the forward main area During a down moment, or during a timeout or intermission, the overall refresh rate may be increased by reducing the wait, e.g., the refresh period control used in step 622, and in some embodiments increased.

Although a full refresh process has been described with reference to steps 622 and 624, a regular supply of portions of a scene is described. As will be appreciated, the normal refresh of the scene or scene portion will occur at a rate of data allowed at a supported video frame rate, for at least one portion. Thus, with respect to at least one frame portion, for example, the portion indicated by the user's head facing will be supplied at a full video streaming frame rate assuming that the available data rate is sufficient.

At step 612, scene portions are selected to be provided based on the user's, indicated head position, e.g., viewing angle. The selected portions are periodically transmitted, e. G., Streamed, to the playback device. The rate at which the data corresponding to the portions are streamed depends on the video frame rate in some embodiments. For example, at least one selected portion may be streamed at a supported full frame rate. At least one scene portion is selected in step 612, but usually a plurality of scene portions, e.g., the scene portion that the user is facing, as well as the next closest scene portion, are selected. If the available data rate is sufficient to support delivery of multiple frame portions, additional scene portions may also be selected and supplied.

After the scene portions to be streamed are selected in step 612, the operation proceeds to step 614 and the encoded version of the selected stream portions is selected based on, for example, the viewing position of the user and the available data rate, do. For example, a full-rate high-resolution version of the scene portion facing the user represented by the currently reported head position may be streamed and generally streamed. One or more scene portions to the left and / or right of the current head position may use a lower resolution, another encoding device that reduces the amount of bandwidth required to transmit a scene region that is not currently viewed or at a lower time rate And may be selected to be streamed. The selection of the encoded version of the adjacent scene portion will depend on the amount of bandwidth reaming after the high quality version of the scene portion currently being viewed is transmitted. Although not currently visible scene portions can be transmitted as a lower resolution encoding version or as an encoded version with a greater temporal distance between frames, if the available bandwidth is sufficient, the full resolution high quality version may be transmitted periodically or It is often transmitted.

At step 616, the selected encoded versions of the selected scene portions are transmitted to the playback device that requested the content. Thus, at step 616, the encoded content corresponding to one or more portions, e.g., stereoscopic video content corresponding to a plurality of consecutive frames, is streamed to the playback device.

Operation proceeds from step 616 to step 618 and information indicating the current head position of the user is received. This information may be periodically and / or transmitted from the playback device in response to detecting a change in head position. In addition to changes in head position, changes in the available data rate may affect the content being streamed. Operation proceeds from step 618 to step 620 and a determination of the current data rate that may be used for content delivery to the playback device is performed. Thus, the content delivery system can detect changes in the amount of bandwidth available to support streaming to the requesting device.

Operation proceeds from step 620 to step 612 where a signal indicating that the session is terminated, for example, until the program or event is terminated, or from the playback device that requested the content, And continues to stream until a failure to receive the expected signal from the playback device, such as a head position update, indicating that the playback device is no longer communicating with the content server 114 is detected.

From the scene data delivered in the manner described above, the playback device will have at least some data corresponding to each scene portion available to display when the user quickly turns his head. It should be understood that since many users are uncomfortable changes in viewing position, users rarely turn their heads completely over a very short period of time. Thus, a full 360 degree scene may not be transmitted at all times, but a high quality version of the scene portion (s) with the greatest likelihood of being shown at any given time may be streamed and made available to the user.

Because the encoding process causes the N portions of the scene to be differently transmitted and processed to different users rather than having to separately encode the content for each individual user, the content delivery system 104 may include a plurality of concurrent users . Thus, multiple parallel encoders can be used to support real-time encoding to allow real-time or near-real-time streaming of sports events or other events, but the number of encoders used tends to be much less than the number of playback devices in which the content is streamed have.

It should be understood that while portions of content are described as portions corresponding to a 360 degree view, scenes may represent a flat version of the space also having a vertical size, and in some embodiments thereof. The playback device may map scene portions using a 3D environment, e.g., a model of space, and adjust for vertical viewing positions. Thus, the 360 degrees discussed in this application refers to the head position relative to the horizontal as if the user maintained his watch level while changing his viewing angle to left or right.

FIG. 7 illustrates an exemplary content delivery system 700 having an encoding capability that can be used to encode and stream content in accordance with aspects of the present invention.

The system may be used to perform encoding, storage, and transmission and / or content output according to aspects of the present invention. In some embodiments, the system 700 or elements within it perform operations corresponding to the process illustrated in FIG. The content delivery system 700 may be used as the system 104 of FIG. While the depicted system shown in FIG. 7 is used for encoding, processing, and streaming of content, system 700 may also include the ability to decode and display the processed and encoded image data to the operator Should be understood.

The system 700 includes a display 702, an input device 704, an input / output (I / O) interface 706, a processor 708, a network interface 710 and a memory 712. The various components of the system 700 are coupled together via a bus 709 that allows data to be transferred between the components of the system 700.

Memory 712 may be used to control system 700 to implement partitioning, encoding, storage, and streaming / transmission and / or output operations in accordance with the present invention, when executed by various modules, ≪ / RTI >

The memory 712 may be implemented in a computer system (e.g., a computer system) to implement the immersion stereoscopic video acquisition, encoding, storage, and transmission and / or output methods in accordance with the present invention, 700). The memory 712 may include control routines 714, a partitioning module 716, an encoder (s) 718, a streaming controller 720, received input images 732, Video scene, encoded scene portions 734, and timing information 736. In some embodiments, the modules are implemented as software modules. In other embodiments, the modules are implemented in hardware, for example, as individual circuits having respective modules implemented as circuits for performing the corresponding functions. In yet other embodiments, the modules are implemented using a combination of software and hardware.

Control routines 714 include device control routines and communication routines for controlling the operation of system 700. [ The segmentation module 716 is configured to segment the received stereoscopic 360-degree version of the scene into N scene portions according to aspects of the present invention.

The encoder (s) 718 may comprise a plurality of encoders configured to encode the received image content, e.g., a 360 degree version of the scene and / or one or more scene portions, in accordance with aspects of the present invention, In embodiments, it includes them. In some embodiments, the encoder (s) include a plurality of encoders configured to encode the stereoscopic video and / or segmented scene portions so that each encoder supports a given bitrate stream. Thus, in some embodiments, each scene portion may be encoded using multiple encoders to support a number of different bitrate streams for each scene. The output of encoder (s) 718 is encoded scene portions 734 that are stored in memory for streaming to customer devices, for example, playback devices. The encoded content may be streamed to one or more different devices via the network interface 710. [

Streaming controller 720 is configured to control the streaming of the encoded content, e.g., via communication network 105, to convey the encoded image content to one or more customer devices. In various embodiments, the plurality of steps of the flowchart 600 are implemented by elements of the streaming controller 720. The streaming controller 720 includes a request processing module 722, a data rate determination module 724, a current head positioning module 726, a selection module 728, and a streaming control module 730. The request processing module 722 is configured to process the received request for imaging content from the customer playback device. The request for content is received via the receiver at network interface 710 in various embodiments. In some embodiments, the request for content includes information indicating the identity of the requesting playback device. In some embodiments, the request for content may include the data rate supported by the customer playback device, the user's current head position, e.g., the location of the head mounted display. The request processing module 722 provides the retrieved information to the other elements of the streaming controller 720 to process the received request and take other actions. The request for content may include data rate information and current head position information, but in various embodiments the data rate supported by the playback device may include network checks and other network information between the system 700 and the playback device Can be determined from the exchange.

The data rate determination module 724 is configured to determine available data rates that may be used to stream imaging content to customer devices, for example, because a number of encoded scene portions are supported, 700 may support streaming of content at multiple data rates for the customer device. The data rate determination module 724 is also configured to determine a data rate supported by a playback device that requests content from the system 700. [ In some embodiments, data rate determination module 724 is configured to determine a data rate available for delivery of image content based on network measurements.

The current head positioning module 726 is configured to determine the current viewing angle and / or the current head position of the user, e.g., the head-mounted display, from information received from the playback device. In some embodiments, the playback device periodically transmits current head position information to the system 700, and the current head positioning module 726 receives and stores information to determine the current viewing angle and / .

Selection module 728 is configured to determine which of the portions of the 360 degree scene to stream to the playback device based on the user's current viewing angle / head position information. The selection module 728 is also configured to select the encoded versions of the determined scene portions based on the available data rate to support streaming of the content.

Streaming control module 730 is configured to control the streaming of multiple portions of a 360 degree stereoscopic image scene at various supported data rates in accordance with image content, e.g., features of the present invention. In some embodiments, the streaming control module 730 is configured to control streaming N portions of a 360 degree stereoscopic scene to a playback device that requests content to initialize the scene memory at the playback device. In various embodiments, the streaming control module 730 is configured to periodically transmit selected encoded versions of scene portions determined, for example, at a determined rate. In some embodiments, the streaming control module 730 is also configured to transmit a 360 degree scene update to the playback device in accordance with a time interval, e.g., once per minute. In some embodiments, transmitting the 360 degree scene update comprises transmitting N scene portions or NX scene portions of a full 360 degree stereoscopic image scene, where N is a full 360 degree stereoscopic scene, And X represents the selected scene portions currently transmitted to the playback device. In some embodiments, the streaming control module 730 waits for a predetermined amount of time after first transmitting N scene portions for initialization before transmitting a 360 degree scene update. In some embodiments, timing information for controlling transmission of a 360 degree scene update is included in timing information 736. [ In some embodiments, the streaming control module 730 also identifies portions of the scene that have not been transmitted to the playback device during the refresh interval; And to transmit an updated version of the identified scene portions that have not been transmitted to the playback device during the refresh interval.

In various embodiments, the streaming control module 730 may be configured to periodically deliver at least a sufficient number of N portions to the playback device so that the playback device refreshes the 360-degree version of the scene at least once for each refresh period .

Figure 8 illustrates a playback system 800 implemented in accordance with the present invention that may be used to receive, decode, store, and display imaging content received from a content delivery system such as that shown in Figures 1 and 7. A system 800 that may be implemented includes a single playback device 800 that includes a display 802 or a combination of elements such as an external display, e.g., a head mounted display 805 coupled to a computer system 800 ' ')to be.

The playback system 800, in at least some embodiments, includes a 3D head-mounted display. The head mount display may be implemented using a OCULUS RIFT ^TM VR (Virtual Reality) headset, which may include a head mount display 805. Other head mount displays may also be used. In some embodiments, one or more display screens in a head mount helmet or other head mounting device are used to display content in the left and right eyes of the user. By displaying different images for the left and right eyes on a single screen by a head mount configured to expose different portions of a single screen to different eyes, a single display can be displayed on the left and right sides Can be used to display eye images. In some embodiments, the mobile phone screen is used as a display of a head-mounted display device. In at least some such embodiments, the mobile phone is inserted into a head mounting device, and the mobile phone is used to display images.

The playback system 800 decodes the received encoded image data, e.g., left and right eye images and / or mono (single images) corresponding to different portions of the environment or screen, and for example, And has the ability to generate 3D image content for display to the customer by rendering and displaying different left and right eye views recognized as 3D images by the user. The playback system 800, in some embodiments, is located at a customer premise location, such as a home or office, but may also be located at an image capture site. The system 800 may perform signal reception, decoding, display, and / or other operations in accordance with the present invention.

System 800 includes a display 802, a display device interface 803, an input device 804, an input / output (I / O) interface 806, a processor 808, a network interface 810, 812). The various components of the system 800 are coupled together via a bus 809 that allows data to be transferred between components of the system 800 and / or by other connections or through an air interface. In some embodiments, although the display 802 is included as a selected element shown using a dash box, in some embodiments, the external display device 805, e.g., a head-mounted stereoscopic image display device, May be coupled to the playback device via interface 803.

For example, when a cellular phone processor is used as the processor 808 and the cellular phone generates and displays an image at the head mount, the system includes a processor 808, a display 802, and a memory 812). Processor 808, display 802, and memory 812 may all be part of a cellular telephone. In other embodiments of the system 800, the processor 808 may be part of a gaming system, such as an XBOX or PS4, having a display 805 mounted to the head mounting device and coupled to the gaming system. It is important to note that the processor 808 or memory 812 located in the device does not matter whether it is in the head or not, and, as can be appreciated, in some cases, While it may be convenient to locate, in at least some cases, it may be desirable to have a processor 808 and memory 812 coupled to a headgear that includes a display.

Although various embodiments contemplate the head mount display 805 or 802, the method and apparatus may also be used with non-head mounted displays capable of supporting 3D images. Thus, in many embodiments, the system 800 includes a head mounted display, but may also be implemented by a non-head mounted display.

Memory 812 includes routines that, when executed by various modules, e.g., processor 808, control playback device 800 to perform decoding and output operations in accordance with the present invention. The memory 812 may include control routines 814, a request to the content generation module 816, a head position and / or viewing angle determination module 818, a decoder module 820, a stereoscopic image Rendering module 822 and the data / information block 828 including the received encoded image content 824, decoded image content 826, a 360-degree decoded scene buffer 828, and generated stereoscopic image content 830 .

Control routines 814 include device control routines and communication routines for controlling the operation of device 800. [ The request generation module 816 is configured to generate a request for content to be sent to a content delivery system for providing content. The request for content is sent via network interface 810 in various embodiments. The head position and / or viewing angle determination module 818 determines the current viewing angle of the user and / or the current head position, e.g., the position of the head mounted display, and sends the determined position and / (700). In some embodiments, the playback device 800 periodically transmits current head position information to the system 700.

The decoder module 820 is configured to decode the encoded image content 824 received from the content delivery system 700 to generate the decoded image data 826. The decoded image data 826 may include decoded stereoscopic video scenes and / or decoded scene portions.

3D image rendering module 822 may use the decoded image content 826 for display on the 3D images, e.g., display 802 and / or display device 805, , It generates left and right eye images that are displayed in a manner that is perceived as a 3D image in accordance with the features of the present invention. The generated stereoscopic image image content 830 is an output of the 3D image generation module 822. Thus, the rendering module 822 renders the 3D image content 830 on the display. In some embodiments, the display device 805 may be part of a 3D display device such as Oculus Rift. The operator of the playback device 800 may select one or more parameters via the input device 804 and / or select the actions to be performed, for example, to display the 3D scene.

9 is also sometimes referred to as a camera rig or camera array having three camera pairs 902, 904, 906 mounted on three different mounting positions and a calibration target 915 that can be used to calibrate the camera assembly 800 There is shown a diagram illustrating an exemplary camera assembly 900. Camera rig 900 is used to capture image content in accordance with some embodiments of the present invention. In some embodiments, the camera rig 900 is used as the image capture device 102 of FIG. The camera rig 900 includes three pairs of cameras 902, 904 (901, 902), 904, 911 of stereoscopic cameras (901, 903, , 906). ≪ / RTI > The support structure supports the camera and includes a base 1120, also referred to herein as a mounting plate (see element 1120 shown in FIG. 11) on which the plates on which the cameras are mounted can be fixed. The support structure can be made of plastic, metal or synthetic material such as graphite or fiberglass, and is represented by lines that form a triangle that is also used to view the spacing and relationship between cameras. The center point at which the dashed lines intersect indicates a center node that camera pairs 902, 904, 906 can be rotated in some embodiments, but not necessarily in all embodiments. The central nodal point corresponds to, for example, a triangular base steel rod or threaded center mount to which the camera support frame 912, shown in triangular lines in some embodiments, can be rotated. The support frame may be a plastic housing or triangular structure with cameras mounted thereon.

In Figure 9, each pair of cameras 902, 904, 906 corresponds to a different camera pair position. The first camera pair 902 corresponds to a front position facing 0 degrees ahead. This position corresponds to a major scene area of interest, e.g., a field in which a normal sporting event is being played, a stage, or some other area where the main action is likely to occur. A second camera pair 904 corresponds to the 120 degree camera position and is used to capture the right rear viewing area. The third camera pair 906 corresponds to the 240 degree position (with respect to the 0 degree position) and the left rear viewing area. Note that the three camera positions are 120 degrees apart. Each camera viewing position includes one camera pair in the FIG. 9 embodiment, and each camera pair includes a left camera and a right camera that are used to capture images. The left camera sometimes captures what is called the left eye images, and the right camera captures what is sometimes called the right eye images. The images may be part of the view sequence or a still image captured more than once. Typically at least the front camera position corresponding to camera pair 902 will be occupied by high quality video cameras. Other camera locations may be occupied by high quality video cameras, low quality video cameras, or a single camera used to capture still or mono images. In some embodiments, the second and third camera embodiments are equipped with a camera that allows the first camera pair 902 to capture images corresponding to all three camera positions at different times, without the left occupied, The supported support plate is rotated. In some such embodiments, the left and right rear images are captured first, then a video of the forward camera position is captured during the event. The captured images may be encoded and streamed in real time, e.g., while the event is still in progress, to one or more playback devices.

The first camera pair shown in FIG. 9 includes a left camera 901 and a right camera 903. The left camera 901 has a first lens assembly 920 fixed to the first camera and the right camera 903 has a second lens assembly 920 'fixed to the right camera 903. [ The lens assemblies 920 and 920 'include lenses that allow a wide viewing angle to be captured. In some embodiments, each of the lens assemblies 920, 920 'includes a fisheye lens. Thus, each of the cameras 902, 903 can capture a 180 degree field of view or nearly 180 degrees. In some embodiments, less than 180 degrees are captured, but in some embodiments there is also at least some overlap in images captured from adjacent camera pairs. In the Figure 9 embodiment, the camera pair is positioned in each of the first (0 degree), second (120 degree), and third (240 degree) camera mount locations, Capture, but in many cases, each camera pair captures 180 degrees or nearly 180 degrees of the environment.

The second 904 and third 906 camera pairs are identical or similar to the first camera pair 902, but are located at 120 and 240 degree camera mount positions with respect to the front 0 degree position. The second camera pair 904 includes a left camera 905 and a left lens assembly 922 and a right camera 907 and a right camera lens assembly 922 '. The third camera pair 906 includes a left camera 909 and a left lens assembly 924 and a right camera 911 and a right camera lens assembly 924 '.

In Fig. 9, D represents the distance between the axes of the first pair of stereoscopic images of the cameras 901 and 903. In the example of FIG. 9, D is 117 mm, which is equal to or similar to the distance between the pupils of the left and right eyes of the average human. Dotted line 950 in FIG. 9 shows the distance from the center point of the panoramic array to the entrance pupil of the right camera lens 920 '(also known as a node offset). In one embodiment corresponding to the example of FIG. 9, the distance indicated by reference numeral 950 is 315 mm, but other distances are possible.

In one particular embodiment, the footprint of the camera rig 900 is relatively small, with a horizontal area of less than 640 mm 2. This small size allows the camera rig to be placed in the audience, for example, in a sitting position where the pan or attendee is normally located or placed. Thus, in some embodiments, the camera rig is placed in the audience area which allows the viewer to perceive a member of the audience that desires this effect. In some embodiments, the footprint, in some embodiments, corresponds to the size of the base that the support structure on which the center support rod is mounted or the support tower is located. As will be appreciated, in some embodiments, the camera rigs may rotate about the midpoint of the base corresponding to the center point between the three pairs of cameras. In other embodiments, the cameras are fixed and do not rotate about the center of the camera array.

Camera rigs can capture relatively close objects as well as distant objects. In one particular embodiment, the minimum imaging distance of the camera array is 649 mm, but other distances are possible, and such distances are never important.

The distance from the center of the camera assembly to the intersection 951 of the views of the second and third camera portions represents one exemplary calibration distance that can be used to correct images captured by the first and second camera pairs. Note that the target 915 may be located at a known distance from the camera pairs located in or slightly beyond the region of maximum distortion. The calibration target includes a known fixed calibration pattern. The calibration target may be to calibrate the size of the images captured by the cameras of the camera pairs and is used for that. This calibration is possible because the size and position of the calibration target is known with respect to the camera capturing the image of the calibration target 915. [

Figure 10 is a diagram 1000 of the camera array shown in more detail in Figure 9. Camera rigs are also shown with six cameras, but in some embodiments the camera leagues are occupied by only two cameras, e.g., camera pairs 902. There is a 120 degree gap between each pair of camera mount locations as shown. For example, when the center between each camera pair corresponds to the direction of the camera mount position, in this case, the first camera mount position corresponds to 0 degrees, the second camera mount position corresponds to 120 degrees, And the third camera mounting position corresponds to 240 degrees. Therefore, each camera mounting position is separated by 120 degrees. This can be seen when the angle between the centerline and the lines extending through the center of each camera pair 902, 904, 906 is measured.

In the example of FIG. 10, the pair of cameras 902, 904, 906 can rotate around the center of the camera rig, which allows different views to be captured at different times without having to change the location of the camera based basis, , It rotates around it. That is, the cameras can be rotated around the center support of the league and can be allowed to capture different scenes at different times that allow 360 degree scene capture using the league shown in Fig. 10, . This configuration is particularly advantageous from the perspective of cost given the cost of the stereoscopic cameras, and it is desirable to have a captured background at a different time from the time at which the frontal scene, including the main action during the sporting event or other event, It is suitable for many applications. For example, consider that objects that are desirable to be invisible during a key event during an event can be placed behind the camera. In such a scenario, the back images are captured before the main event and may be made available with the captured images in real time of the main event to provide a 360 degree set of image data, and are sometimes captured and available.

FIG. 11 shows a detail view of an exemplary camera rig 1100 implemented in accordance with an exemplary embodiment. As can be appreciated from Figure 11, camera rig 1100 includes three pairs of cameras 1102, 1104, and 1106, some embodiments of which are stereoscopic cameras, but not all embodiments. Each pair of cameras includes two cameras in some embodiments. Camera pairs 1102, 1104 and 1106 are identical or similar to the camera pairs 902, 904 and 906 discussed above with respect to Figs. In some embodiments, camera pairs 1102, 1104, 1106 are mounted on support structure 1120 of camera rig 1100. In some embodiments, cameras 1102, 1104, and 1106 of three pairs (six cameras) are mounted on support structure 1120 through respective camera pair mounting plates. Support structure 1120 includes three mounting locations for mounting stereoscopic camera pairs, with each mounting location corresponding to a different 120 degree viewing area. 11, a first pair of stereoscopic cameras 1102 is mounted in a first mounting position, e.g., a front position, of the three mounting positions, and corresponds to a 120-degree front viewing area . The second pair of stereoscopic cameras 1104 is mounted at a second mounting position of the three mounting positions, for example, at a background position that rotates 120 degrees clockwise with respect to the front position, and in a different 120 degree viewing area Respectively. The third pair of stereoscopic images cameras 1106 are mounted at a third mounting position of the three mounting positions, for example, at a background position that rotates 240 degrees clockwise with respect to the front position, Respectively. The three camera mount positions on the camera rig 1100 are offset by 120 degrees with respect to each other, but in some embodiments, each camera mounted on the camera rig has a field of view of about 180 degrees. In some embodiments, this extended view is achieved by using a fisheye lens on the camera devices.

Although not all mounting plates are shown in the drawing, the camera mounting plate 1110 used to mount the camera pair 1102 is shown. The mounting plates for the cameras have slots for screws that pass through the slots of the support base 1120 and from the bottom to the mounting plate through the threaded holes. This allows the mounting plate for the pair to be adjusted by unscrewing the accessible screws from the bottom and then tightening the screws to secure the camera pair mounting plate to the support structure. The individual camera positions can also be adjusted and then fixed after adjustments. In some embodiments, the individual cameras can be adjusted / fixed from the top to the mounting plate, and the camera mounting plates can be adjusted / fixed from the bottom.

In many embodiments, camera rig 1100 includes a base 1122 to which support structure 1120 is rotatably mounted. Thus, in various embodiments, the camera assembly on the support structure 1120 can be rotated 360 degrees around an axis passing through the center of the base. In some embodiments, the base 1122 may be part of a tripod or other mounting device. The support structure may be composed of plastic, metal, or synthetic material such as graphite or fiberglass. The camera pairs, in some embodiments, may be rotated around a central point, sometimes referred to as a central nodal point.

In addition to the above elements, in some embodiments, the camera rig 1100 further includes two simulated ears 1130 and 1132. [ These simulated ears 1130 and 1132 mimic human ears and in some embodiments are composed of silicone molded into the shape of a human ear. The simulated ears 1130 and 1132 include microphones whose two ears are spaced apart from each other by a distance equal to or approximately equal to the distance between the average human human ears. The microphones mounted on the simulated ears 1130 and 1132 are mounted on the face-facing camera pair 1102, but can alternatively be mounted on a support structure, for example, a platform 1120. The simulated ears 1130 and 1132 are positioned perpendicular to the front of camera pair 1102 in a manner similar to that human ears are positioned perpendicular to the front of the eyes on the human head. The holes on the sides of the simulated ears 1130 and 1132 serve as the audio input holes on the sides of the simulated ears and the simulated ears and holes are simulated as if the human ear were directing the audio sounds to the eardrum contained in the human ear Lt; RTI ID = 0.0 > microphone, < / RTI > In the left and right simulated ears 1130 and 1132, the microphones provide a stereo sound capture similar to that recognized through the left and right ears of a person when the human at the location of the camera rig 1100 is located at the position of the camera league .

11 illustrates one configuration of an exemplary camera rig 1100 with three stereoscopic camera pairs, it should be understood that other variations are possible within the scope. For example, in one implementation, the camera rig 1100 may include a pair of stereoscopic cameras capable of rotating about a central point of a camera rig that allows different portions of a single camera pair, e.g., a 360 degree scene, to be captured at different times Video cameras. Thus, a single pair of cameras can be mounted on the support structure and rotated around the center support of the league, and can be allowed to capture different scenes at different times that allow for 360 degree scene capture.

12 illustrates an exemplary 360 scene environment 1200 that may be divided into different viewing regions / portions corresponding to different camera positions of each camera capturing different portions of a 360 degree scene, for example, 360 < / RTI > In the depicted example, the 360 scene region 1200 includes three different camera / camera locations, such as cameras mounted and positioned on the camera rig 1100 shown in Figures 9, 10, and 11, Is divided into three portions corresponding to three 180 degree zones captured by the camera pairs. The zero degree marked on the 360 degree scene 1200 may be considered to be the center of the scene. In some embodiments where a fisheye lens is not used, the field of view of each camera is about 120 degrees, thus allowing the camera to capture a scene area of about 120 degrees. In these embodiments, the boundaries of the different 120 degree scene portions are shown in the figure using black solid lines that divide the 360 degree scene into three portions of 120 degrees each. In embodiments where cameras are equipped with fisheye lenses, the field of view of each camera extends to about 180 degrees (+ -5 degrees), thus allowing the camera to capture a scene area of nearly 180 degrees (+ -5 degrees).

A first zone (zone 1 corresponding to a 180 degree frontal scene area from 270 to 90) that includes 90 degrees to the left and right of the 0 degree mark indicates that the first camera, e.g., the camera device, May be captured by a camera pair 1102 located to capture a frontal scene area, with a fisheye lens to provide a field of view. The second zone (zone 2) is a 180 degree range from 30 to 210 that can be captured by a second camera, e. G., A pair of cameras 1104 positioned to capture a rear right scene region with a fisheye lens. (Zone 3) is captured by a third camera, e.g., camera pair 1106 located to capture a rear left scene region with a fisheye lens, corresponding to a rear right scene region, Which corresponds to a scene area from 180 to 150 degrees backward from 150 to 330. FIG. Legend 1250 includes information identifying different line patterns used to represent different zone boundaries, e.g., marking the start and end of scene regions included under different zones. It can be seen from the figure that there is substantial overlap between scene regions under three different zones included by different cameras. In the example shown in Fig. 12, the overlap between zone 1 and zone 2 is 60 degrees, i.e., 30 to 90 degrees below the scene area, and the overlap between zone 2 and zone 3 is also 60 degrees , That is, a scene area of 150 to 210 degrees, and the overlap between the zone 3 and the zone 1 is a scene area of 60 degrees, that is, 270 to 330 degrees. In the illustrated example, it is to be understood that although the overlap is 60 degrees, various different overlaps are possible. In some embodiments, the overlap between two different scene coverage areas is between 30 degrees and 60 degrees.

In some embodiments, the content provider publishes, e.g., multicasts, a content stream that includes content corresponding to different portions of the 360 degree scene regions captured by different cars, to the customer playback devices. In some embodiments, multiple versions of content corresponding to differently encoded different scene regions are multicasted by the content provider, supporting a particular version and the preferred playback device may select an appropriate content stream to decode and play have. According to one aspect of some embodiments, the playback device tracks the current head position representing the current view of the user and includes available content including content corresponding to a portion of the 360 degree scene area to choose to receive for playback One or more of the streams. For example, if the user's head position is looking ahead, the playback device will decode the stream carrying the 180 degrees frontal portion of the 360 degree scene, but when the user's head position and viewing angle change is detected, The device decodes the stream corresponding to the portion of the scene corresponding to the current viewing angle of the user, for example, the rear right, rear left, and back of the 360 degree scene region. In some embodiments, a stream that includes content corresponding to a first portion of a 360 degree scene area, e.g., a frontal 180 degree, is used to capture the front portion of a 360 degree scene, And scene regions captured by the left and right cameras.

FIG. 13 shows an example of an exemplary camera rig 1100, which may be located, for example, to include viewing regions / portions of an exemplary 360 scene region, ≪ RTI ID = 0.0 > 360 < / RTI > FIG. 13 example includes illustrations 1300, 1320, 1350 illustrating different portions of an exemplary 360 scene region. However, it can be seen from the drawings that at least some of each of the different parts of the scene captured by different cameras overlap in some embodiments. Different shading patterns were used for each of the figures 1300, 1320, 1350 to show portions of the scene area corresponding to different camera positions. In some embodiments, the different scene portions shown in Figures 1300, 1320 and 1350 are conveyed through different content streams that provide content corresponding to different viewing directions. In the example of FIG. 13, the scene portions are shown to include an approximately 180 degree viewing region, but in some embodiments, the scene portions may comprise between 120 and 180 degrees.

Drawing 1300 illustrates a first exemplary scene portion 1305 of a 360 degree scene 1200. The first exemplary scene portion 1305 corresponds to a front viewing direction and includes a 180 degree or approximately 180 degree viewing area in a 360 degree scene environment. The first scene portion 1305 may be captured by the first camera pair 902 or 1102, for example, located at a zero degree camera position. An area on the first exemplary scene portion 1305 is shown in the drawing 1300 using a diagonal pattern. A first exemplary scene portion 1305 may be conveyed by a first stream carrying frames of content corresponding to a first, e.g., frontal, viewing direction.

Drawing 1320 shows a second exemplary scene portion 1307 (including portions 1307 ', 1307 "shown in Figure 1320) of a 360 degree scene 1200. A second exemplary scene The portion 1307 corresponds to the rear right viewing direction and includes a 180 degree or 180 degree viewing area in a 360 degree scene environment extending from 30 to 210. The second scene portion 1307 is shown in Figures 9 and 10 May be captured by a second camera pair 904, for example located at the illustrated 120 degree camera position. An area under the second exemplary scene portion 1307 may be captured in the view 1300 using a horizontal line pattern Consider the figures 1300 and 1302. Note the overlapping portion 1308 of the scene region between the first and second scene portions 1305 and 1307. The overlapping portion 1308 includes a scene region 1308, 1200. In some embodiments, the portion of the captured scene region that is common to both the first and second scene portions 1305, , The overlap between the first and second scene portions 1305 and 1307 has a + -2 to 3 degree variation and is between 30 degrees and 60 degrees. In the example shown in FIG. 13, the overlap portion 1308 60 degrees, for example, from 30 to 90. Thus, in some embodiments, at least a portion of a scene region provided by different content streams and / or corresponding to different viewing directions captured by different cameras In some other embodiments, there is no overlap between scene regions corresponding to different viewing directions captured by different cameras. In some embodiments, the second exemplary scene portion < RTI ID = 0.0 > 1307 may be carried by a second stream carrying frames of content corresponding to a second viewing direction.

Drawing 1350 illustrates a third exemplary scene portion 1309 (including portions 1309 ', 1309 "shown in Figure 1350) of a 360 degree scene 1200. A third exemplary scene portion 1309 The scene portion 1309 corresponds to the rear left viewing direction and includes a 180 degree or approximately 180 degree viewing area of a 360 degree scene environment extending from 150 to 330. The third scene portion 1309 is shown in Figures 9 and 10 For example, by a third camera pair 906 located at a 240 degree camera position shown in Figure 13. An area under the third exemplary scene portion 1309 is shown in Figure 1300 using a vertical line pattern Consider figures 1320 and 1350. Note the overlapping parts 1310 and 1310 'that combine the overlapping area parts between the second and third scene parts 1307 and 1309. [ A common area under the overlapping portions 1310 and 1310 'is a cap that is common to both the second and third scene portions 1307 and 1309 of the scene region 1200. [ In some embodiments, the overlap between the second and third scene portions 1307 and 1309 is between < RTI ID = 0.0 > 30 < / RTI & The overlapping portions 1310 and 1310 'together include an area of about 60 degrees, for example, between 150 and 210. Let's consider further drawings 1300 and 1350 here. Note the overlapping portion 1312 that represents the portion of the overlapping region between the first and third scene portions 1305 and 1309. In some embodiments, the third exemplary scene portion 1309 is shown in a third viewing direction May be conveyed by a third stream conveying frames of corresponding content.

It should be understood that the drawings depicted in the example portion of FIG. 13 illustrate examples to facilitate understanding of some aspects of the present invention, but other variations are possible within the scope of the disclosure.

14, which includes the combination of FIGS. 14A and 14B, is a flowchart 1400 illustrating steps of an exemplary method of operating a playback system in accordance with embodiments of the present invention. The system may be the playback system 800 shown in FIG. 8 or it may be a playback system of any of the other views shown in this application.

An exemplary method begins at step 1402 and the playback device of any of the playback devices, e.g., playback device 1900 of FIG. 19 or other views, is turned on and started. For purposes of discussion, a playback system may include a head-mounted display device 1905 that includes, for example, a display in which the image content is displayed in the left and right eyes of the user represented by different images in the case of stereoscopic content, And a computer system 1900 'coupled to the computer system. Although the computer system 1900 'is shown externally to the head-mounted device including the display, the computer system 1900' may be integrated with the head-mounted display rather than external to it.

Operation proceeds from steps 1402 to step 1404. At step 1404, the playback system 1900 receives information about a plurality of content streams and / or initialization data, e.g., as part of a program guide. The received information may be in the form shown in Figure 18 and may be combined with information that may be used to receive streams such as multicast group identifiers or other identifiers that may be used by content streams to request content or to tune to the content Information that is available or available. For example, a multicast address associated with a content stream may be included in the received information or program identifier that may be used to request content when the content is provided via switched digital video. In the case of broadcast content, the received information may include, and sometimes includes, tuning information indicative of the channel and / or frequency with which the playback device should be tuned to receive the particular content stream.

The information received in step 1404 may include information about one or more programs. For a given program, for example a sporting event, a concert, etc., different streams may be available to provide content corresponding to different viewing directions with respect to camera position in the context in which the content corresponds. The camera position corresponds to the viewing position during playback. Thus, during playback, the user's viewing angles are correlated with portions of the environment represented in the content that can be received. Different parts of the environment may be communicated to different streams. For each portion of the environment, e.g., a portion of the 360 degree environment, one or more streams corresponding to different data rates may be listed in the provided information. Upper and lower portions of the environment may also be provided. In some embodiments, the content of the individual streams is a stereoscopic video content having different information provided for the left and right eye images, thereby allowing different images to be displayed to the user to provide the desired 3D effect. In some embodiments, the upper and lower portions of the spherical environment are provided with the same mono images as the left and right eye views, thus requiring only one image to be presented rather than two images.

The information about the program, the portion of the environment, and the data rate of the stream may be used by the playback system to prioritize which stream is received at a given limited bandwidth available for streaming the content, and in some embodiments Is used. Prioritization and selection of streams to be received at a given time may be based on, and in some embodiments based on, the user's head position and / or the direction of the user's current or previous head rotation.

In the case of streaming of stereoscopic video content, given limited bandwidth and / or data limits, selection of the appropriate stream to receive may be important for a satisfactory and high quality viewing experience. The information received in step 1404, which may be the same as or similar to the information shown in FIG. 18, for example, stream information 1405 is stored in memory and may be supplied to a playback device, for example, By connecting a multicast group corresponding to a selected stream tuned to a channel providing a requested delivery of the selected stream and / or stream by presenting the switched digital video channel to the network device, And is used to initiate the delivery of the selected stream.

Operation proceeds from step 1404 to step 1406. [ In step 1406, during initialization, the current head position of the user is detected. The user recognizing that the head position detected during the initialization phase is assumed to be the front facing position typically maintains his head in a position facing the front of the comfortable level during step 1406. [

Operation proceeds from step 1406 to step 1408. [ In step 1408, the user's head position 1407 detected in step 1406 is determined such that the user has captured his or her head in a zero degree environmental location, i.e., a content stream that corresponds to a particular portion of the environment, (0 degree) environment viewing position and view when displayed in this position corresponding to the forward position captured by the cameras used in the images. In the case of a sporting event, this location will correspond to the center of the field, if the main areas of operation, e.g., streams or streams, correspond to a concert in a normal environment, and the stage and streams correspond to a sporting event. Thus, at step 1408, the user's viewing position will be set to be interpreted as a 0 degree viewing position, e.g., a front / front position of the scene area. Note that the parts of the 360 degree view correspond to horizontal viewing positions with different parts visible when the user rotates his head. By moving the user up or down the head, the user can view the sky portion and / or the ground portion alone or in combination with one or more other portions. Since the main scene region is divided into portions along the 360 degree rotation assuming the level head position, the streams corresponding to these portions are usually allocated more bandwidth, and the top / bottom scene portions are still images or rarely changed . ≪ / RTI >

Operation proceeds from step 1408 to step 1406, where an environmental depth map is received. The depth map defines the surface of the 3D environment to which images of the content stream are to be mapped. In the absence of a depth map reception, the shape of the environment with the interior surfaces of the sphere being the surfaces mapped during rendering before the images of the environment is the assumed default. By providing and using a depth map, a more realistic experience is achieved because the shapes and surfaces of the environment in which the images in the content stream are to be simulated will be mapped to more realistic surfaces. Thus, the depth map received in step 1410 corresponds to the environment in which the content selected by the user to be received also corresponds. If the received environment map, or map, in step 1410 is not received, the default map is stored as environment map 1411 for subsequent use when rendering images.

Operation proceeds from step 1410 to step 1412. [ At step 1412, one or more UV maps to be used to map the 2D image content onto at least a portion of the 3D surface are received. In one embodiment, the at least one UV map is received for each portion of the environment that may be represented by different individual images. In some such embodiments, when the images carried by the content stream are captured by different cameras, e.g., left and right eye cameras, different UV maps may be provided for different cameras, , Is provided to him. Thus, in the example of FIG. 14A, a first UV map corresponding to a first portion of the environment, such as a front portion, is received at Step 1414, and at Step 1416 a second UV map corresponding to a second portion of the environment, A third UV map is received that corresponds to a third portion of the environment, such as the right rear portion. UV maps corresponding to the upper and lower portions of the environment are received in steps 1418 and 1420, respectively. If the portions are of equal size, the same UV map is used for multiple portions. However, in some embodiments, different UV maps are used for images captured by different cameras.

Thus, in one such embodiment, for each portion of the environment in which stereoscopic image data is provided, e.g., left and right eye images, UV maps are used to capture specific left or right eye image content Individual UV maps may be received and stored for each of the left and right eye images so that specific features of the camera assembly may be taken into account.

Each UV map provides a map that is used to map a two-dimensional image of the corresponding content stream to a corresponding portion of the surface of the 3D environment. Images captured by the camera in this manner can be transmitted as 2D images and then mapped as textures on a portion of the surface or surface defined by the 3D model.

By means of 3D model information and UV maps, if other content for the scene part can be received and stored, images that can be used as defaults are received. In step 1422, an image corresponding to one or more of the content, e.g., the first, second, third, and fifth scene portions, is received and stored in step 1422. In some embodiments, a number of alternative images, e.g., a background portion or a sky portion, corresponding to a portion of the environment are received and stored at Step 1422. Control information may be received indicating that the default image is to be used at a given time during an event for a portion of the plurality of default images being stored. For example, in some embodiments, a background image of a sitting audience and a background image of a standing audience are stored as two default images. The control information is used to indicate which background image is used for a given portion of the event. For example, during a portion of a game or concert that corresponds to a standing ovation, the spectator standing-up image is signaled that the background image to be displayed is facing the user in the background direction. However, during the main part of the case where the audience is usually sitting, the control information signals that a default image of the sitting audience should be used if the user turns his head to the background. The control information may be signaled separately from the content stream or may be included with the content stream for a portion of the environment that differs from the portion to which one or more of the default images are related. For example, the content stream corresponding to the forward direction may include images corresponding to the forward direction, e.g., left and right eye images, as well as default images at sky, ground, right background portion at various times during the event, It may provide control information indicating that it should be used for the left background portion. Alternatively, the playback device may determine, at a particular point in time, which of the background or sky portions to use based on the brightness of the similarity to one or more features of the foreground image. For example, when the foreground images are dark, this may be the detected cloudy sky image that can be detected and automatically detected, while when the foreground image is brighter, it may also be detected and the received and stored available default sky images Lt; RTI ID = 0.0 > a < / RTI > less cloudy, brighter sky image.

In step 1422, default images corresponding to different environment portions are received and stored in a generally encoded form. At step 1424, one or more of the received images are decoded, and the decoded content is stored in one or more image buffers at a later step 1426. In this way, the default images can be decoded and stored in decoded form so that they do not need to be decoded later when they are needed for rendering during playback. Because the default image may be used many times, decoding and storage may reduce the decoding requirements that are otherwise required to decode the image in the rendering or only before rendering. Assuming that the processing resources can provide pre-decoding and storage of default images in a < RTI ID = 0.0 > decoded form, < / RTI > images are decoded immediately prior to display and then decoded, Lt; RTI ID = 0.0 > embodiment < / RTI >

The same default decoded image may be used many times for, for example, sky, etc., but it may be combined with other received image content such that it closely matches other images in the environment in which it is combined to produce a viewable portion of the environment Can be processed. For example, in some embodiments, the decoded default images are subjected to brightness adjustment based on image portions that are combined or blurred along edges where at least the default image is to be combined with the image corresponding to the other portion of the environment. Thus, during use in at least some embodiments, image brightness and / or color features are filtered or modified to make them more similar to the same features of the ground image in which they are combined.

With the initialization data and default images stored for future use, operation proceeds to step 1428 where a set of one or more streams to be received, e.g., the currently selected set of streams, is initialized. Each stream may provide stereoscopic or monoimage data. Optionally, the corresponding audio may also be received in the selected streams, but is more common in one or more individual audio streams. The description will focus on the reception of video streams by the understanding that the audio stream will typically be received and decoded by the playback device and that the audio may include stereo audio.

In exemplary step 1428, the currently selected set of streams is set equal to the first stream carrying content corresponding to the front / front portion of the environment. This is because the initial position at the start is set to be the forward viewing position and therefore the user will expect to see the forward scene area at the start as a result of initialization.

Operation proceeds from step 1428 to step 1429. [ In step 1429, resource allocation information is received. The resource allocation information may be in the form of bandwidth and / or data rate allocation control information. The information received in step 1429 includes, in some embodiments, information as to how much bandwidth or data transfer capability should be allocated to one or more transport streams corresponding to different parts of the environment. The information can be expressed in terms of bandwidth or data rate by understanding that the data rate is usually consistent with the bandwidth. For example, the amount of data that can be received may be a function of the amount of bandwidth taking into account the type of data coding used to carry across the bandwidth.

The information received in step 1429 may represent a relative maximum amount of available transfer capability to be allocated to reception of images corresponding to a particular portion of the environment. For example, this means that up to 80% of the bandwidth or supportable data rate should be allocated to the primary data stream, for example, the forward data stream with the remaining 20% of the bandwidth is allocated to one or more other streams . The allocation of resources for different directions can be a function of the image content in a corresponding portion of the environment and / or detected crowd feedback, and in some embodiments is such a function. For example, in some embodiments, during intercommunications that occur during a corresponding event of content, the information received in step 1429 indicates that an increased amount of resources is present in the image corresponding to one or two posterior portions of the environment Lt; RTI ID = 0.0 > and / or < / RTI > This is because during the intermission there are more possibilities for the users to start turning their heads around and turn their eyes away from the main field or stage, May be desirable. For example, there may be images of a person purchasing or changing hot dogs in a baseball game, and in some embodiments, may be transmitted during interactions so that the background appears to be live, but still at other times. Similarly, images of billboards in the background may change during intercommunications for advertising and / or entertainment purposes. Thus, it may be desirable to trigger the playback device to allocate more resources for receiving background portions during intercommunications than during other portions of the game. The control information received in step 1429 may be different and sometimes different during other careful parts of the game or during intercommunications and major part of the game. In at least some embodiments, during the main event, the control information received in step 1429 causes more bandwidth and / or data rate to be allocated to a major, e.g., area of the environment, rather than the rear part. However, during intermissions or other separate portions, the data rate assigned to one or both of the backward sections may be forced to increase.

The separation between resources assigned to a given direction may be based on the content presented in the portion of the environment, the measure of the spectator's attention, and / or the portion of the event in progress, e.g., the main portion, the intermission portion, Lt; / RTI > In some embodiments, the control information provided in step 1429 may include, for example, the maximum amount of bandwidth or data rate to be allocated for reception of an image corresponding to one or more portions of the environment while the event is in progress, and / Specify the minimum amount. In some embodiments, the information indicates that the bandwidth or data may not be allocated for receiving the ground-to-sky image portions during the event, and thus these portions are filled in when it is necessary to use still images in this case.

The bandwidth / data rate allocation control information may vary over time, and different information is received at different times. The control information may be embedded into the forward content stream and / or transmitted separately, for example, as a separate set of control information.

Operation proceeds from step 1429 to step 1430 where the content delivery of the stream (s) in the selected set of streams that have not already been received is initiated. This may include connecting the selected stream to the corresponding multicast group, sending the message to the network device requesting delivery of the selected stream, and / or tuning to the broadcast channel on which the selected stream or streams are transmitted. In the case of the first pass through step 1430, this will include initiating delivery of the content stream corresponding to the forward portion of the environment since this was set as the initial viewing portion to be selected for delivery. However, because the head position of the user changes, for example, because the user turns his or her head to the left or right, the set of selected streams may change and usually change. For example, if the user turns his head to the left so that a portion of the front region and the right rear region are visible, the set of selected streams will be changed to receive content corresponding to the front region and the left rear region. If the left rear portion is not received and is selected, delivery of the content in the stream corresponding to the left rear portion will be initiated in step 1430. If the maximum data rate streams in both directions may not be supported, the lower data rate forward stream will be selected and therefore the start of both the lower data rate forward stream and the left backward content stream will begin. Streams outside the selected set are terminated prior to reception of the streams in the newly selected set. The ending and starting of the streams may be controlled by blurring and / or filtering used to dramatically reduce changes in the quality or source of images when the stream switching occurs, such that dramatic changes in the received content and / or temporal gaps occur In a flexible manner. For example, blurring may be performed over portions of images that are stitched together as part of a rendering or display process.

Operation proceeds from step 1430 to step 1432 by content delivery initiated for the current set of selected streams. In step 1432, the content is received from the streams in the selected set of content streams. This may include receiving content corresponding to the highest priority stream, and in various embodiments, receiving the content, e.g., the stream may include content corresponding to most of the view, and For example, the portions provide content corresponding to one or more other portions of the environment, providing a small portion of the image content corresponding to the field of view. A stream providing a small portion of the content for the current view may be described as a secondary stream. In one embodiment, where a single stream provides content for the entire view, less than 20% of the available bandwidth / supportable receive data rate is reserved, e.g., when the user turns away from the field of view, , ≪ / RTI > which are used to receive one or more secondary or lower priority streams. If the field of view is approximately evenly divided between regions corresponding to two different content streams, then each stream will be divided into regions of the same size as the regions of the region where the two streams are distributed, Since there is no possibility of changing its position quickly, approximately half of the available data rate / received data rate can be allocated.

By receiving the disclosed content, in step 1432, the content, e.g., images, is received from the selected set of streams. In the case of a primary, e. G., Highest priority content, the stream will typically be stereoscopic video content having both left and right eye image content received in the stream. For streams that are assigned low priority streams and / or low data rates, mono images may be received with a single image received for display on both the left and right eye images. The forward scene portion is typically received as stereoscopic video content, but one or more rear portions may be provided as mono image streams.

The encoded image content is typically received at 1432 in streams. In step 1434, in step 1438, where the received content is decoded and then reached via access node A 1436, the decoded content is stored in one or more image buffers. In some embodiments, the decoded image buffer is maintained for each portion of the environment. Only the only part of the received image can be finally displayed, but the entire received frame is usually decoded and buffered. The decoded buffered image may be kept in memory until it is replaced with a more recent image for the same scene portion, and so is maintained in some embodiments. Thus, at any given time, the decoded image is available for each portion of the environment that will be used to render the final output image needed based on the current view. Decoding of the images corresponding to each of the portions of the 360 degree environment need not occur for each frame time since the decoded image content is kept in memory until it is replaced with a more recent decoded image. Thus, although frame rates of 30 frames per second may be supported, 150 frames need to be decoded in each frame period, e.g., the frame period for each of the top, bottom, forward, left rear, right rear portions , Rather than the previously decoded still image portions that are updated at a lower rate than the portion corresponding to the main field of view, or a smaller number of frames that are equal to or slightly higher than the frame rate to be supported, It can be decoded with some parts.

Thus, after updating in step 1438, the currently decoded frames are available for use in rendering an image based on the current view of the user.

Operation proceeds from step 1438 to step 1440. [ In step 1440, the content is decoded using the available decoded content from the image buffer (s) to generate an environment map, in which one or more image portions define a surface to be applied, for example, as a texture, And is rendered to display the UV map (s) providing information on how to apply the decoded images. The 3D mesh model of the environment may be in the form of a grid with points of the UV map corresponding to the vertices of the mesh model defining the surface to which the images of the environment are applied.

As part of the rendering, the image portions corresponding to the content received from the different streams generate an image of the environment corresponding to the user's field of view, when the content from a single stream can not provide a portion that completely occupies the user's field of view . Filtering or blurring can be applied across the images being connected to form a composite image corresponding to the user's field of view, and is applied in some embodiments. This tends to reduce the seam significantly for the user. Also, the brightness of the image portions providing a portion of the composite image may, in some embodiments, be adjusted such that when making the brightness adjustments for the image portions being combined to form the composite image, the brightness of the back, top, Values are adjusted to reduce the luminance differences between the image portions to be combined to form a composite image having the luminance values of the front view that are beginning to be given a higher priority than the values of the front view.

The rendered image, which may be a composite of one or more image portions of images received from different content streams, is stored, displayed, or transmitted in step 1442. Which represents the creation and display of one or more frames of content based on the current set of selected content streams.

By changes in communication channel conditions or network problems over time, the maximum supportable data rate may vary. This may be detected at step 1443 and may be considered when selecting which content stream or streams to receive and process. At step 1443, the maximum supportable data rate and / or bandwidth is determined to be indicated by data 1444 with respect to that used in subsequent steps. In step 1445, the current head position of the user is detected. This can be done by using a position sensor on the headgear that includes a head mounted display.

Operation proceeds from step 1445 to step 1446, where a determination is made as to whether the user's head position changes. If the user ' s head position has not changed, then operation proceeds to step 1447 where it is determined whether the currently available maximum bandwidth or maximum supported data rate has changed, e.g., from the last time the stream selection was made Inspection is performed. If no change in maximum supported data rate or head position is detected, the previous stream selection remains valid and no changes are made to the set of selected content streams. Thus, the playback system will continue to receive content corresponding to the current view of the user, which remains unchanged. If no change is detected in step 1447, then operation proceeds back to step 1429 via access node B 1456. [

However, if a change is detected in one of steps 1446 or 1447, the operation proceeds to a stream selection step 1448 that includes a call to the stream selection subroutine. In this manner, detected changes in head position and / or supportable data rate may be considered, and selection of streams may be performed in support of the user ' s head position and / or data, Considering possible data rates are reconsidered.

Once the streams have been selected by the stream selection subroutine, operation proceeds to step 1450 and a check is made as to whether the selected streams are different from the current set of selected streams. If the newly selected set is the same as the currently selected set of streams in use, no changes need to be made to the received streams, and the current selected set of streams in step 1452 is passed to access node B 1456 via step 1429 ). ≪ / RTI > However, if the currently selected set of streams is different from the current set of selected streams, then the currently selected set of streams is updated in step 1454 to reflect changes in the selected set of streams, e.g., Is the same set as the newly selected set of streams selected by the selection subroutine.

Operation proceeds from step 1454 to step 1455 and the reception of streams that do not exist in the updated currently selected set of streams is terminated. This may be accomplished by receiving a stream that is terminated for another purpose, e.g., a multicast group corresponding to a stream that is no longer received or takes another action to use resources such as a tuner used for receiving a newly selected stream or streams Lt; RTI ID = 0.0 > a < / RTI >

Operation proceeds from step 1455 to step 1429 via connection note B (1456). Thereafter, in step 1430, the reception of any newly selected streams will be initiated and the received content will be used to render one or more images. Thus, the selected streams may also change since the user changes his head position over time and / or changes the data rate that can be supported.

FIG. 15 is a flow chart showing the steps of a stream selection subroutine in accordance with one exemplary embodiment that may be used as the stream selection subroutine called in step 1448 of the method of FIG. 14.

The stream selection subroutine 1500 begins at step 1502 when the subroutine is called by step 1448 of FIG. Operation proceeds from the start step 1502 to the stream selection step 1504. The inputs to the stream selection step 1504 include bandwidth and / or data rate constraints 1503. These may include information on a maximum portion of the available bandwidth that should be allocated to one or more scene portions, streams and / or stream directions, or the total received data rate. The constraints may also include minimum bandwidth or data assignments for scene portions, streams and / or viewing directions. Thus, constraints may limit the maximum amount of resources allocated to reception of content corresponding to a particular direction and / or when an action occurs at a portion of the environment, such as a rear region during an event intermission, Or a minimum amount of resources allocated for the direction to be updated at specific times.

The maximum supported data rate and / or the maximum available bandwidth 1444 are other inputs to the selection step 1504. This information 1444 represents the maximum supported data rate that can be used to support the receipt of one or a combination of the maximum bandwidth and / or the content streams available for receiving the content. Information about the user's current head position 1407 and available streams 1405, e.g., stream guide information as shown in FIG. 18, is also an input to step 1504.

In the stream selection step 1504, one or more content streams corresponding to a program or event, e. G., A live event proceeding in some instances, are selected. The selection of one or more streams is based on information about the user's current head position, stream information such as stream bandwidth or stream data rate requirements, and / or the maximum supportable data rate or maximum supportable bandwidth. Bandwidth and / or data rate limits 1503, which may be for viewing direction or stream priority based, may also be considered and used to perform stream selections in step 1504, and are sometimes considered and used.

An exemplary stream selection step 1504 includes step 1506, wherein the content streams are prioritized based on the current and / or past head position of the user. This may include a call to a stream prioritization subroutine as shown in FIG.

The streams corresponding to the same environment direction are assigned the same priority in some embodiments. Thus, multiple streams corresponding to the same portion of the environment and / or the content corresponding to the viewing direction can be assigned the same priority and are sometimes so assigned. The streams correspond to the same viewing direction, but in some cases the streams may be provided at a high frame rate in some cases to provide mono image content and / or low resolution images and / or to support low frame (image) And has different data rates from some streams that provide low resolution streams to effect content. Thus, all streams providing a content with a particular orientation that is considered a high priority and that correspond to a high priority orientation are the same, but in some embodiments thereafter, the amount of bandwidth available to receive content corresponding to a particular orientation Selection is made.

After prioritizing the content stream operation, operation proceeds from step 1506 to step 1508. [ At step 1508, the maximum bandwidth and / or data rate to be used for the stream with the highest priority is determined. This determination may be performed based on bandwidth or other constraints 1503 that may indicate the maximum, minimum, or fraction of available available resources to be allocated for the highest priority stream. In some embodiments, the minimum bandwidth / data rate allocation for the highest priority stream is greater than 50%, but other allocations are possible.

In step 1510, the maximum bandwidth and / or data rate to be used for each stream having a lower priority is determined. In some embodiments, at least 20% of the data rate or bandwidth is used for secondary or lower priority streams.

The data rates to be used for the streams of different priorities are determined in steps 1508 and 1510 and the operation proceeds to step 1512 where the highest priority stream is the maximum bandwidth for the highest priority stream and / Or based on a data rate. If any of the streams corresponding to the highest priority can be supported, then the decision in step 1512 will be affirmative and operation proceeds to step 1514, where the impersonation corresponding to the highest priority A high quality stream will be selected. This typically involves selecting the highest data rate stream from the set of streams with the highest priority assigned. For example, if the forward streams are assigned the highest priority, the highest data rate forward direction content stream that can be supported for a given data rate for the highest priority content stream will be selected.

Operation proceeds from step 1514 to step 1516. [ At step 1516, a determination is made as to whether the second highest priority stream can be supported. This is because, in some embodiments, how much bandwidth / data is available after the highest priority stream is selected and how much of the bandwidth / data is used for the second highest priority stream based on the received constraints And determining whether the data can be processed. If no constraints for the second highest priority stream are deployed, the total amount of bandwidth / data reception capability may be used for the second highest priority stream. If it is determined in step 1516 that the second highest priority stream can be supported, then operation proceeds to step 1518 where the second highest priority stream is transmitted, for example, to the second highest priority Is selected from the set of one or more streams assigned. Step 1518 may include selecting the highest data rate stream that can be supported with a second highest priority and, in some embodiments, including that. For example, if the second highest priority corresponds to the right rear portion of the environment, then step 1518 includes selecting the highest supported data rate stream that corresponds to the right rear portion of the environment.

In most cases, the highest priority and secondary streams will be supported, but will remain after the selection of the two highest priority streams to receive some other content of the environment, e. G., Some content corresponding to the invisible. There may be sufficient bandwidth available. Operation proceeds from step 1518 to step 1520 or directly from step 1516 to step 1520 if the second highest priority stream can not be supported.

At step 1520, it is determined whether the third highest priority stream can be supported, for example, using available bandwidth / data receiving resources after the first and / or second priority stream is selected Inspection is performed. In step 1520, if it is determined that the third highest priority stream can be supported, then operation proceeds to step 1522 and uses, for example, the remaining bandwidth / data rate that can be supported, The third highest priority stream is selected. Operation proceeds from step 1522 to step 1524, or if the third highest priority stream is not directly available in step (b) if the given available bandwidth and / or received data and / or bandwidth allocation constraints can not be supported 1520). ≪ / RTI >

At step 1524, a check is made to determine if there is an ability to receive any bandwidth, e. G., Remaining data that may be used to receive additional content, after other stream selections. If additional bandwidth remains, operation proceeds to step 1526 where one or more lower priority streams are selected to use the remaining available bandwidth / data rate. From step 1526 the operation proceeds to step 1530. [ Operation proceeds from step 1524 to return step 1530 if no additional bandwidth is available.

Return step 1530 causes the process to return to the point at which stream selection subroutine 1500 is called, for example, by the newly selected set of streams being determined by routine 1500.

16 is a flow chart showing the steps of a stream prioritization subroutine in accordance with one exemplary embodiment. 1600 of the drawing illustrates one exemplary stream prioritization routine that may be used, for example, one or more of the other routines or subroutines described herein. For example, step 1506 of Figure 15 may be implemented by a call to subroutine 1600. [ Stream prioritization subroutine 1600 begins at step 1602 when the routine is called. Operation proceeds from start step 1602 to step 1604. In step 1604, the current view of the user is determined based on the detected head position of the user. Considering that the user's field of view is less than 180 degrees, the current field of view may correspond to a single camera location, for example, a portion of the captured environment from a camera location facing forward. However, when the user moves his or her head, for example, turning his head left or right and / or looking up or down, the user's field of view is captured by cameras located at different camera positions It can respond to parts of the environment. For example, the user's view may correspond to the content to be delivered in two different streams when the images captured by different cameras are viewing a portion of the environment corresponding to the point where they meet or overlap. A stream that provides images corresponding to the largest portion of the view will usually be given the highest priority in some embodiments.

Depending on the view of the user determined at step 1604, the operation proceeds to step 1605 and a stream of delivering content, e.g., a mono image containing left and right eye images corresponding to the current view of the user Or stereoscopic image pairs are identified. Operation then proceeds to step 1606, where the size of the portion or portions, e.g., image portions of the scene region, corresponds to the current view of the user available from the identified streams. Thus, at the conclusion of step 1606, information is available as to which streams provide image content corresponding to the current view and the relative size of the parts and can be used to prioritize, e.g., stream have.

Operation proceeds from steps 1606 to step 1608. [ In step 1608, priorities are assigned to one or more streams that provide image content corresponding to the current view of the user. Prioritization, e.g., grading, is based on the size of the portion or portions of the user's field of view provided by the stream. For example, a stream providing image content corresponding to 80% of the user's field of view will be ranked higher than a stream providing image content corresponding to 15% of the user's field of view, and the remaining 5%, for example, , The stream providing the top or bottom will be assigned a third priority lower than the highest priority assigned to the stream providing the image (s) corresponding to the 80% portion of the field of view.

Step 1608 may include step 1610 and / or step 1612, and includes it in some embodiments. In step 1610, the highest priority is assigned to the stream providing the largest portion of the field of view. Step 1610 may include designating as a primary stream a stream providing the largest portion of the field of view. Step 1612 includes assigning the next highest priority or priorities to streams that provide portions for a view outside the portion provided by the highest priority stream. These streams are prioritized in step 1612 based on the size of the portion that they can provide to the current view by streams that can provide smaller portions that are assigned lower priorities.

For example, assuming that multiple streams with different data rates can provide the same portion to the view, although at potentially different resolutions or frame rates, multiple streams may be assigned the same priority For example, the streams corresponding to the forward view may be assigned the same priority, and the streams providing the left rear view may have the same priority, for example, different priorities allocated to the forward streams And the streams providing the right rear view may be given the same priority, e.g., different priorities than those assigned to the front or left rear view streams.

Thus, at step 1608, the streams that provide for the field of view will be ranked, or prioritized. Prioritization may be represented by listing streams in a ranked list, with the primary stream (s) assigned the highest priority and the other streams assigned the lower priority.

Not all streams are capable of responding to the field of view. For example, the top or two scene portions may be out of sight and therefore the streams providing these views may not have been prioritized at step 1608. [ Operation proceeds from step 1608 to step 1614. [ At step 1614, a determination is made as to whether any remaining streams to be prioritized are present. For example, if none of the remaining streams to be prioritized exist because they all correspond to the current view of the user, the operation proceeds to return step 1630. However, if at step 1614 it is determined that one or more streams are left to be prioritized, then operation proceeds to step 1616.

In step 1616, priorities are assigned to one or more additional streams carrying content, e.g., current out-of-view streams. The prioritization performed in step 161 may, in some embodiments, be based on the proximity of the content provided by the stream prioritized to the content seen in the current field of view and / Direction. For example, if the stream provides image content corresponding to a portion of the immediate immediate vicinity of the current view, then in some embodiments it may be higher than the stream providing the content corresponding to the portion of the image remote from the user ' Priority will be assigned. Similarly, a stream providing image content in the direction of the user's head rotation may be viewed by a user, with the assumption that the content in the head rotation direction is more likely to come into the user's view faster than the content in the opposite direction away from the detected direction of head rotation May be given a higher priority than the content in the direction of rotation away from the detected head rotation of the player.

In at least one embodiment, step 1616 includes step 1618 to determine whether the change in head position indicates a left or right turn as opposed to a head turn, e. G., A tilt above or below the head Is performed. If the head position is not detected in step 1618, then operation proceeds to step 1620 and the streams are prioritized based on the portion of image data they provide with respect to the current view of the user in some embodiments. Streams providing top and / or bottom portions may be assigned a default priority lower than streams providing left or right rear portions as well as left or right rear portions when the top and bottom portions are out of sight. Operation proceeds from step 1620 to return step 1630.

If it is assumed in step 1618 that the user's head rotation is detected, then operation proceeds to step 1622. [ In step 1622, the direction of the head rotation is determined, e.g., whether the user's head is turning left or right. This allows the orientation of the head rotation to be taken into account, since there is often more possibility in the direction of head rotation than the next part of the environment to be brought into the user's view will be away from it.

Operation proceeds from step 1622 to step 1624, where priorities are assigned to streams out of view based on one or more of the content streams, e.g., the direction of head rotation. In at least one embodiment, step 1624 includes step 1626, and the lowest unused priority is then assigned to the stream providing the content corresponding to a portion of the environment in the direction of head rotation. For example, if a user looking at the front portion of the environment turns his head to the right, the stream providing the right rear content out of view will be assigned a higher priority than the stream providing the left rear content that also exists outside the view will be. Operation proceeds from step 1624 to step 1628 where the remaining streams that have not been prioritized in advance are assigned lower priorities indicating lower priority than streams for which priorities have already been allocated.

Operation proceeds from step 1628 to step 1630. [ Until the return step 1630 is reached, the content streams will be prioritized, e.g., ranked or ordered, in priority order.

FIG. 17 is a flowchart illustrating steps 1700 of a rendering subroutine in accordance with one exemplary embodiment.

The rendering subroutine 1700 may be invoked by one or more routines when image rendering is desired. In the case of stereoscopic content, individual images are rendered for each of the user's left and right eyes. In the case of mono content, a single image is rendered and used for each of the user's left and right eyes. Rendering often involves a combination of image content from one or more streams. Thus, some portions of the environment may be provided as mono content, while other portions may be provided as stereoscopic content, in which case the different left and right eye images may have portions of the content being stereoscopic and other content being mono , A single image may be rendered that is generated for each of the left and right eye images when at least a portion of the environment is represented as stereoscopic image content.

Rendering routine 1700 begins at start step 1702 and proceeds to render step 1706. [ The input to the rendering step 1706 includes one or more decoded images or images on the surface defined by the environment map 1411, the decoded image content 1703 corresponding to the one or more views, And one or more UV maps 1704 used to map the portions. As discussed above, the environment map 1411 may default to spheres having images that are mapped into the surface of the sphere, if no more complex geometry is provided.

In some embodiments, the rendering step 1706 may include at least one UV map to generate at least one image corresponding to the current view of the user and the current view of the user, as well as one or more content streams corresponding to the environment map And using the generated content by decoding the embedded images (1708). In the case of stereoscopic images, e.g., 3D image content, rendering will cause left and right eye images to be generated in a format suitable for display. Differences in the rendered left and right eye images, in some embodiments, allow the user to recognize the images in 3D.

Operation proceeds from step 1706 to step 1710, where the rendered and / or outputted rendered images are returned to a program or routine to be fed to the display device.

The rendering subroutine 1700 may be called whenever an updated version or frame of view is displayed. Thus, rendering typically occurs at a rate that matches the image, e.g., frame, display rate.

18 shows an example table 1800 that includes stream information corresponding to a plurality of content streams. In some embodiments, the stream information included in the exemplary table 1800 is received as part of the guide information, e.g., as a program guide that provides information about the content streams that the playback system may choose to receive. Legend 1840 includes information indicating the meaning of the various letters used as abbreviations in the information contained in table 1800. [

The information contained in table 1800 may be used to access the corresponding content streams. As discussed in some embodiments, the stream information may include, for a plurality of available content streams, a multicast address of a multicast group that may be coupled to receive the provided corresponding content stream, providing the provided content stream Or one of the channel tuning information that can be used to control the tuner of the playback system to tune to the broadcast channel on which the provided content stream is broadcast .

In table 1800, each row corresponds to a separate content stream carrying content, and the corresponding content stream for the row is identified by a stream identifier shown in the corresponding input of column 1812. Each input of column 1804 identifies the program content delivered by the respective individual content stream for which the row corresponds. As can be appreciated from table 1800, a first group 1820 of rows corresponds to program content "soccer" which may represent program / event titles shown in corresponding inputs of column 1804. [ There are a plurality of such groups corresponding to a variety of different programs / events. Each group includes content streams, each of which corresponds to a viewing direction, and supports a provided data rate as discussed. For simplicity, only two groups are shown in the figure, and the rows corresponding to the second group 1822 are only partially shown for illustrative purposes only. The second group of rows 1822 corresponds to the program content "HI" represented by the corresponding inputs in column 1804. [

In column 1806, each input represents a section of the scene area, e.g., a 360-degree scene area 1200 conveyed by a corresponding content stream. Thus, the first three rows of the group 1820, while each corresponding to a different content stream is divided into a front scene part (e.g., zone 1 shown in FIG. 12, which includes a 270 ° to 90 ° viewing area) Which is mapped to < / RTI > Each of the three rows of the group 1820, corresponding to the different content streams, is mapped to zone 1 shown in FIG. 12, including a rear right scene part, for example, a 30 ° to 210 ° viewing area . The last three rows of group 1820, each corresponding to a different content stream is mapped to a region 3 shown in FIG. 12, including a rear left scene portion (e.g., a 150 ° to 330 ° viewing region) .

Each input of column 1808 represents the data rate supported by the corresponding content stream. Each input of column 1810 represents a multicast group identifier / address of a multicast group that can be connected to receive the corresponding content stream identified by the stream identifier shown in the corresponding input of column 1812. Each input of column 1814 includes a stream descriptor for the corresponding content stream identified by the stream identifier shown in the corresponding input of column 1812. Each input of column 1816 includes access information, e.g., tuning parameters and / or other access parameters that may be used to access or request a corresponding content stream.

As can be appreciated from exemplary table 1800, in the depicted example, multiple, e.g., three, different versions of each of the content streams corresponding to the provided viewing direction available for use during playback And each version of the content stream supports a different data rate. Thus, the playback system may be configured to play back, based on one or more factors such as, for example, supported bandwidth, data rate, user head position, etc., in accordance with aspects of the invention discussed in detail with respect to FIGS. One or more streams may be selected for use.

In order to more clearly understand how the information in the table 1800 can be used by the playback system to select and / or access one or more content streams, the first row and columns 1804, 1806, 1808, 1810, 1812, 1814, 1816). The first input of the column 1804 indicates that the event / program "soccer" delivered by the first content stream identified by the stream identifier (S ₁ D ₁ ) is included in the column 1812. A corresponding input of column 1806 indicates that the first stream carries content corresponding to a preceding scene portion (e.g., a 270 ° to 90 ° viewing region). This viewing area information is used by the playback system to identify one or more streams carrying content corresponding to the current head position of the user / viewer corresponding to the current view. Continuing with the example above, the corresponding first input of column 1808 indicates that the first content stream supports and / or requires data rate D1. The corresponding input of column 1810 indicates that the first content stream can be accessed by concatenating the multicast group Ml and Ml represents the multicast group address and / or identifier that maps to the address. The corresponding input of column 1814 is the input of the first stream having a corresponding camera viewing angle V ₁ , codec type C ₁ , supported data rate D ₁ and frame rate F ₁₎ representing the stream descriptor and a "V ₁ C ₁ D _1, F _1," corresponding to the first content stream. The corresponding entry in the last column 1816 indicates access tuning parameters and / or other access parameters (shown as A123) that may be used to access or request the first content stream.

With this information discussed above with respect to the available content streams that may be used, a playback system, such as playback system 1900, may be used to select and access one or more content streams for use in playback according to aspects of the present invention . For better understanding, consider a simple example in which the playback system determines that the user's head position indicates that the user is looking at the front portion of the 360 degree scene. In this case, in one embodiment, the playback system selects at least one content stream that carries the front scene portion. Depending on various other factors discussed with respect to Figures 14-17, such as available bandwidth, supportable data rates, stream bandwidth, and / or data rate limits, the playback system may use three different uses One of the possible streams (S ₁ D ₁ , S ₁ D ₂ , S ₁ D ₃ ) can be selected. If the restrictions allow, the playback system will select the highest quality stream, e.g., stream S ₁ D ₁ , of the three corresponding from the plurality of content streams corresponding to the front scene portion. The information provided in table 1800 facilitates the selection of the appropriate stream for use in playback, since at least some of the information that may be used for selection is provided by stream information 1800. [ After stream selection, the playback system may initiate content delivery, e.g., content reception, by connecting the multicast group corresponding to the selected stream, e.g., M1, or by using the access information to obtain the content stream The stream information 1800 can be used again.

19 illustrates a playback system 1900 implemented in accordance with the present invention that may be used to receive, decode, store and display imaging content received from a content delivery system. The system 1900 may be implemented as a single playback device 1900 'including a display 1902 or as a combination of elements such as an external display, e.g., a head mounted display 1905 coupled to a computer system 1900' .

The playback system 1900, in at least some embodiments, includes a 3D head-mounted display. The head mount display may be implemented using a OCULUS RIFT ^TM VR (Virtual Reality) headset, which may include a head mount display 1905. Other head mount displays may also be used. In some embodiments, a head mount helmet or other head mounting device is used as the device 1905, in which one or more display screens are used to display content in the user's left and right eyes. By displaying different images on the left and right eyes on a single screen with a head mount configured to expose different portions of a single screen to different eyes, a single display can be displayed on the left and right eye images Lt; / RTI > In some embodiments, the mobile phone screen is used as a display of a head-mounted display device. In at least some such embodiments, the mobile phone is inserted into the head mounting device and the mobile phone is used to display images. In some embodiments, the display device 1905 may be part of a 3D display device such as Oculus Rift.

The reproduction system 1900 decodes the received encoded image data, e.g., left and right eye images and / or mono (single images) corresponding to different parts of the environment or scene and, for example, Has the ability to generate 3D image content for display to the customer by rendering and displaying different left and right eye views recognized as 3D images by the user. The playback system 1900, in some embodiments, is located at a customer premise location, such as a home or office, but may also be located at an image capture site. The system 1900 may perform signal reception, decoding, display, and / or other operations in accordance with the present invention.

The system 1900 includes a display 1902, a display device interface 1903, an input device 1904, an input / output (I / O) interface 1906, a processor 1908, a network interface 1910, ). The various components of system 1900 are coupled together via bus 1909 and / or via other connections or via an air interface to allow data to be transferred between the components of system 1900. In some embodiments, although the display 1902 is included as an optional element shown using a dash box, in some embodiments, the external display device 1905, e.g., a head-mounted stereoscopic image display device, May be coupled to the playback device via a device interface 1903.

For example, if a cellular telephone processor is used as a processor and the cellular phone generates and displays images on the head mount, the system includes a processor 1908, a display 1906, and a memory 1912 as part of the head mount device can do. Both processor 1908, display 1902, and memory 1912 may be part of a cellular telephone. In other embodiments of system 1900, processor 1908 may be part of a gaming system such as XBOX or PS4, and display 1905 is mounted to a head mounting device and coupled to a gaming system. Processor 1908 or memory 1912 may be located in a device that is not critical to whether it is overhead or not and may be recognized in some cases from a power, It may be convenient to have a memory and processor 1908 coupled to the headgear that includes a display in at least some instances.

While various embodiments contemplate a head mount display 1905 or 1902, methods and apparatus can also be used with non-head mounted displays capable of supporting 3D images. Thus, in many embodiments, system 1900 includes a head mounted display, but it may also be implemented with a non-head mounted display.

The operator of the playback system 1900 may control one or more parameters via the input device 1904 and / or may select to select the actions to be performed, e.g., to display the 3D scene. Through the I / O interface 1906, the system 1900 can be coupled to external devices and / or exchange signals and / or information with other devices. In some embodiments, through the I / O interface 1906, the system 1900 can receive images captured by various cameras that may be part of a camera rig, such as the camera rig 900.

The processor 1908, e.g., the CPU, uses various modules to execute the routines 1914 and to control the playback system 1900 to operate in accordance with the present invention. The processor 1908 is responsible for controlling all the general operations of the playback system 1900. In various embodiments, the processor 1908 is configured to perform the functions discussed as being performed by the playback device.

Via network interface 1610, system 1900 can send signals and / or information (e.g., including images and / or video content) to various external devices, for example, And / or receive over the same communication network. The network interface 1910 includes a receiver 1911 and a transmitter 1913 in which receive and / or transmit operations are performed. In some embodiments, the system receives one or more selected content streams from a content provider via a network interface 1910. In some embodiments, the system 1900 receives one or more selected content streams to be used for playback via the receiver 1911 of the interface 1910. The received content stream may be received as encoded data, e.g., encoded scene portions 1952. Receiver 1911 is also configured to receive stream information 1946 and / or initialization data, e.g., as part of a program guide. The system 1900 also receives data rate allocation control information 1952 that includes bandwidth limitations for bandwidth and / or different viewing directions, for example via a receiver 1911, And specifies the maximum bandwidth to be used to receive one or more content streams that provide content corresponding to the viewing direction corresponding to the bandwidth limitation. In some embodiments, the receiver 1911 may also include at least one environment map, e.g., a 3D depth map defining a 3D surface, and at least a portion of the 3D depth map, e.g., And to receive one or more UV maps to be used to map the image content. In some embodiments, the receiver 1911 includes a first UV map corresponding to the first portion of the scene environment, a second UV map corresponding to the second portion of the scene environment, a third UV map corresponding to the third portion, A fourth UV corresponding to the fourth portion, and a fifth UV map corresponding to the fifth portion of the scene environment. In some embodiments, during initialization, the system 1900 may transmit content, e.g., through the receiver of the interface 1910, to the first, second, third, fourth, And receives an image corresponding to one or more of the five portions.

Memory 1912 includes routines that, when executed by various modules, such as processor 1908, control playback system 1900 to perform decoding and output operations in accordance with the present invention. The memory 1912 includes control routines 1914, a head positioning module 1916, a current viewing position initialization module 1918, a decoder module 1920, a currently selected stream initialization module 1922, a content delivery initiation module 1924 ), Image buffer (s) 1926, an image buffer update module 1928, an image rendering module 1930, also referred to as an image generation module, an available bandwidth and / or supported data rate determination module 1932, A determination module 1934, an available bandwidth and / or supported data rate change determination module 1936, a stream selection module 1938, a selected stream set change determination module 1940, a selected stream set update module 1942, (S) termination module 1944, data / information including received stream information 1946, received bandwidth and / or data rate allocation information 1948, determined current maximum available bandwidth, and / The received encoded image content 1952, the received environment map 1954, the received UV map (s) 1956, the decoded image content 1958, and the generated 3D content 1960 ).

Control routines 1914 include device control routines and communication routines for controlling the operation of system 1900. [ The head positioning module 1916 is configured to determine the current head position of the user, e.g., the position of the head mounted display. The head positioning module 1916 may incorporate and / or operate in conjunction with, for example, a position sensor that may be present on the head gear including the head mounted display. The current viewing position initialization module 1918 sets the current user's current viewing position to a forward (0 degree) viewing position, for example, during the initialization phase, Position.

Decoder module 1920 is configured to decode the encoded image content 1952 received from content delivery system 700 to generate decoded image data 1958. The decoded image data 1958 may include decoded stereoscopic video scenes and / or decoded scene portions. The decoded content is stored in one or more image buffer (s) 1926 in some embodiments. The currently selected stream initialization module 1922 is configured to initialize the current set of one or more selected content streams to be received. The currently selected stream initialization module 1922 is configured to set the currently selected set of streams to be the first stream carrying content corresponding to the front / front portion of the environment / scene.

The content delivery initiation module 1924 is configured to initiate delivery of the selected content stream (s). In some embodiments, the content delivery initiation module 1924 initiates delivery of the content streams in the selected set that has not already been received. In some embodiments, the content delivery initiation module 1924 may include a multicast group corresponding to the selected content stream, e.g., a multicast group corresponding to the content streams delivering content corresponding to the currently selected set of streams (e.g., Lt; RTI ID = 0.0 > a < / RTI > In some other embodiments, the content delivery initiation module 1924 is configured to generate and send a request to a device in the network requesting delivery of a switched digital channel over which the selected content stream is delivered.

Image buffer update module 1928 is configured to update image buffer (s) 1926 with updated content, for example, when updated content delivered by a selected set of content streams is received and decoded.

Image rendering module 1930 may use the decoded image content 1958 to display to a 3D image, e.g., display 1902 and / or display device 1905, For example, left and right eye images, which are displayed in a manner that is perceived as a 3D image. In some embodiments, the image rendering module 1930 is configured to render content for display using the decoded image content 1958, environment map 1954, and UV map corresponding to the user's current viewing area . Thus, in some embodiments, the image rendering module 1930 is configured to perform the functions discussed with respect to the steps shown in Fig. The generated image content 1960 is the output of the 3D image generation module 1930. Thus, the rendering module 1930 renders the 3D image content 1960 on the display. In some embodiments, the image rendering module 1930 is configured to output one or more generated images to, for example, a display device or other device. The generated images may be output via the network interface 1910 and / or the display device interface 1903.

The available bandwidth and / or supported data rate determination module 1932 determines, for example, the current maximum available bandwidth and / or the current maximum supported data rate that may be available at any given time to receive the content streams . In some embodiments, the determination module 1932 may determine that the available bandwidth and / or supported data rates are not available because the available bandwidth and / or supportable data rates may vary over time due to changes in communication channel conditions or network problems. And continually monitors and / or makes decisions to detect changes in the data rate. The determined current maximum supportable data rate and / or bandwidth 1950 is the output of decision module 1932 and may be updated as needed.

The head position change determination module 1934 is configured to determine whether there has been a change in the user's head position, for example, by examining and comparing the changes in the output of the head position determination module 1916. [ The available bandwidth and / or supported data rate change determination module 1936 may determine the available bandwidth compared to the current maximum available bandwidth determined by decision module 1932 and / or the current maximum supported data rate and / And to detect if there is any change in the data rate.

The stream selection module 1938 is configured to select a stream to be received for use in playback at a given time from a plurality of content streams based on the current head position of the user. Based on changes in the user's current head position and / or other factors, the stream selection module 1938 may select different stream (s) at different times. The currently selected set of streams 1961 is the output of the stream selection module 1938 and represents the set of currently selected content streams for reception. In some embodiments, the stream selection module 1938 includes a plurality of sub-modules that are configured to perform various functions as part of the stream selection operation. Figure 23 shows the stream selection module 1938 and the various modules included therein in more detail and will be discussed later.

The selected stream set change determination module 1940 may determine the current stream set 1961 of streams 1961 by, for example, the selection module selecting one or more additional content streams and / or by one or more streams received / And to determine if a change has been made in the selected set. The selected stream set update module 1942 may update the selected set of streams 1961 by adding or terminating the content streams for a selected set of streams 1961, And to update the currently selected stream set 1961 to reflect changes. The stream (s) termination module 1944 may, for example, store the current selected set of streams 1961, which may have been previously received, by the currently selected set of streams 1961 to be updated due to changes in the selection of streams Stopping / stopping receiving one or more content streams that no longer exist.

Stream information 1946 includes information about a plurality of content streams that may be available for reception and use during playback. The information contained in the stream information 1946 is the same as or similar to that shown in FIG. 18 and discussed previously. The received bandwidth and / or data rate allocation control information 1948 includes information on the bandwidth limitations for different viewing directions and / or limits on data rates corresponding to content streams providing content for various different viewing directions Lt; / RTI > The determined current maximum supportable data rate and / or bandwidth 1950 represents the maximum supported data rate and / or bandwidth determined by the playback system 1900 at a given time.

The received environment map 1954 includes a 3D depth map of the environment that defines the 3D surface. One or more of these depth maps corresponding to the environment of interest may be received by the playback system 1900 in some embodiments. The received UV map (s) 1956 includes one or more UV maps corresponding to portions of the environment / scene of interest. The decoded data 1958 includes data decoded by the decoder 1920 according to the present invention. Decoded data 1958 includes content that includes scene or scene portions of the environment conveyed by a selected set of streams.

In some embodiments, the various modules discussed above are implemented as software modules. In other embodiments, the modules are implemented in hardware, for example, as individual circuits, and each module is implemented as a circuit for the module to perform the corresponding function. In yet other embodiments, the modules are implemented using a combination of software and hardware.

Although shown as being included in the memory 1912 in the example of FIG. 19, the modules shown as included in the playback device 1900 may be fully implemented in hardware within the processor 1908, e.g., as separate circuits, It is so implemented in the examples. The modules may be fully implemented in hardware, for example, as separate circuits corresponding to different modules, and so implemented in some embodiments. In other embodiments, some of the modules are implemented as circuits, for example, in a processor 1908 having other modules implemented, e.g., as external to and coupled to processor 1908. [ As can be appreciated, the level of integration with modules on the processor and / or with some modules external to the processor may be one of the design choices. Alternatively, rather than being implemented as circuits, all or a portion of the modules may be implemented in software and stored in the memory 1912 of the system 1900, wherein the modules include modules such as processors 1908 ) Controls the operation of the system 1900 to perform functions corresponding to the modules. In yet other embodiments, multiple modules are implemented as a combination of hardware and software, for example, other circuitry external to the processor may be coupled to a processor 1908 ≪ / RTI >

23 shows the stream selection module 1938 used in the playback system 1900 and the various modules contained therein in more detail. The stream selection module is configured to select one or more content streams in accordance with the inventive methods discussed in detail with respect to Figures 14-16. In some embodiments, the stream selection module is configured to select which of a plurality of content streams to receive, based on the user's head location, stream information 1946, and / or the maximum supported data rate. In some embodiments, the stream selection module 1938 includes a stream prioritization module 2306 configured to prioritize content streams based on a user's head location. The output of stream prioritization module 2306 is, for example, a prioritized list of content streams having assigned priorities. Stream prioritization module 2306 is discussed in more detail below with respect to FIG.

The stream selection module 1938 is configured to determine the maximum bandwidth and / or data rate to be used for the stream having the highest priority based on, for example, bandwidth and / or data rate restrictions, (S) maximum bandwidth and / or data rate determination module 2308 configured to determine a maximum bandwidth and / or data rate to be used for each stream having a lower priority and / Module < / RTI > The decision modules 2308 and 2310 use the bandwidth control information 1948 and the output of the stream prioritization module 2306 to perform the respective decisions in some embodiments. Thus, the stream selection module 1938 includes one or more stream bandwidth determination modules configured to determine, for example, the bandwidth for at least one content stream based on bandwidth limitations passed from the network device / server to the playback system can do.

The stream selection module 1938 may determine whether the highest priority stream is supported based on the determined maximum bandwidth and / or data rate for the highest priority stream (s) and on available bandwidth and / or supportable data rates And a module 2314 configured to select the highest-priority highest data rate stream that can be supported. In some embodiments, the selection module 2314 is configured to select one content stream from a plurality of content streams assigned the highest priority, and each content stream with the highest priority assigned thereto is selected in the same viewing direction As a portion configured to select from among a plurality of content streams having the same priority. In some embodiments, module 2314 is configured to select from a plurality of content streams having the same priority, e.g., the highest priority based on a determined amount of available bandwidth. Thus, in some embodiments, when a plurality of streams having the same priority are available, for example, some have a higher data rate requirement compared to others having lower data rate requirements, Selects a higher data rate stream if the highest quality stream, e.g., available bandwidth and / or supportable data rate and bandwidth limits, is acceptable for this selection.

The stream selection module 1938 may be configured to determine the available bandwidth and / or bandwidth based on the maximum bandwidth and / or data rate determined for the second highest priority stream (s) A module 2316 configured to determine whether a second highest priority stream can be supported based on a supportable data rate, a module configured to select a second highest highest priority data rate stream that can be supported ( 2318), based on the determined maximum bandwidth and / or data rate for the third highest priority stream (s), and based on available bandwidth (e.g., all available or remaining available) and / A module configured to determine if a third highest priority stream can be supported (2320), and a module (2322) configured to select the third highest highest priority data rate stream that may be supported. Thus, in some embodiments, the stream selection module 1938 is configured, for example, by the prioritization module 2306 to select one or more content streams assigned the highest priorities.

The stream selection module 1938 may include additional capabilities configured to determine whether there is any remaining or additional available bandwidth to receive additional content streams, e.g., after one or more higher priority streams have been selected for reception / Bandwidth availability determination module 2324. < / RTI > In some embodiments, stream selection module 1938 may be supported based on the maximum determined maximum bandwidth and / or data rate for one or more lower priority streams and based on available bandwidth and / or supportable data rates And a module (2326) configured to select one or more lower priority streams that can be selected.

24 illustrates a stream prioritization module 2306 that may be implemented as part of the stream selection module 1938, e.g., as its sub-modules or as individual modules. Stream prioritization module 2306 is configured to prioritize the content streams based on the user ' s head location. If the content streams are prioritized, the stream selection module 1938 may perform stream selection from the prioritized content streams. In some embodiments, the stream prioritization module 2306 includes a current field of view identification module 2404 configured to identify a user's current field of view based on a user's current head position indicating a portion of a scene area the user is viewing, And current scene stream (s) identification module 2404 configured to identify stream (s) carrying content corresponding to a portion of the scene region corresponding to the current view of the user. The output of the current field of view stream (s) identification module 2404 is a list of identified streams that may be stored in the memory 1912 in some embodiments, and the list may be updated as the user's head position, . Thus, in various embodiments, to prioritize the various available content streams, the user's current view is first identified corresponding to the head position, and the streams conveying the content corresponding to the view are identified.

In some embodiments, the stream prioritization module 2306 includes a module 2406 configured to determine the size of portions of the scene region corresponding to the current view of the user available from the identified stream (s) Allocation module 2408 configured to assign priorities to one or more streams that provide portions of the scene region corresponding to the current view of the user based on the size of the portions it provides. In some embodiments, the priority assigning / assigning module 2408 assigns the highest priority to the stream providing the largest portion of the view, and provides, for example, the largest portion of the scene corresponding to the current view And a module 2410 configured to designate the stream as a primary stream. In some embodiments, the priority allocation / allocation module 2408 may determine, based on the size of the viewport portion provided by each of the remaining streams, that a different portion of the scene corresponding to the current view, for example, To assign the next highest priority (s) to the stream that provides a higher priority than the stream and a larger portion of the view provided for the designation and to specify the remaining streams (e.g., as secondary, tertiary, etc.) And further comprises a configured module 2412.

In some embodiments, the stream prioritization module 2306 includes a module 2414 configured to determine whether there are streams that provide content corresponding to the remaining streams to be prioritized, e.g., the current out-of- .

In some embodiments, the stream prioritization module 2306 may be configured to determine, for one or more additional streams that provide the user's current out-of-view content based on one of the proximity of the image content to the current view or the orientation of the head position And further includes a module 2416 configured to prioritize, e.g., prioritize. In some embodiments, module 2416 is configured to prioritize one or more additional stream delivery content corresponding to portions of the current out-of-view based on the proximity of the image content delivered for the current view, The content stream delivering close image content for the region of the current view is assigned a higher priority than the current out-of-sight and away from the content stream.

In some embodiments, the module 2416 includes a head rotation determination module 2418 configured to determine whether a user's head rotation is detected, for example, as part of a change in the user's head position. In some but not all embodiments, there is a change in the head position when the user looks up or gazes down toward the sky or roof, but such head movement is not considered a head rotation. In some embodiments, the module 2416 is configured to prioritize one or more additional content streams based on the direction of the user ' s head rotation, and may include a content stream < RTI ID = 0.0 > Is assigned a higher priority than other content streams that provide image content that is currently out of view in a direction away from the direction of head rotation. In some such embodiments, the module 2416 may then select a next lower priority (e.g., after higher priorities are assigned to the streams providing content corresponding to the field of view) and stream designations, e.g., , And a module 2420 configured to assign tertiary to a portion of the current out-of-view scene, e.g., a stream providing content corresponding to the top or bottom of the scene environment. In some embodiments, when it is determined that no head rotation exists, the head rotation determination module provides a control input to module 2420 to assign priorities to additional streams.

In some embodiments, the module 2416 further includes a head rotational direction determination module 2422 configured to determine the direction of the user's head rotation, e.g., left or right with respect to the previous head position. In some embodiments, the module 2416 further includes a module 2424 configured to assign priorities to one or more additional streams conveying content corresponding to portions of the current out-of-sight, taking into account the direction of head rotation . In some embodiments, the module 2424 then selects a lower priority (e.g., the next available priority from a higher priority) and a designation, e.g., a tertiary stream, And to a stream providing content corresponding to the portion. Thus, when head rotation is detected, it should be understood that the assignment of priorities to the streams is performed based on the direction of head rotation in some embodiments. In some embodiments, module 2416 further includes an additional module 2428 configured to assign lower priorities to any of the remaining streams considered.

It is to be understood that the processor 1908 may be implemented as one or more processors, e.g., computers, although it is shown as a single processor, e.g., a computer, in the example of FIG. When implemented in software, the modules include code that, when executed by the processor 1908, configures the processor 1908 to perform a function corresponding to the module. In embodiments in which the various modules shown in Figures 19, 23 and 24 are stored in memory 1912, memory 1912 may be configured so that at least one computer, e.g., processor 1908, For example, a computer readable medium containing individual code for each module.

Completely hardware-based or completely software-based modules can be used. It should be understood, however, that any combination of software and hardware, for example, circuit implemented modules, may be used to implement the functions. As will be appreciated, the modules shown in Figures 19, 23, and 24 perform the functions of the corresponding steps shown and / or described in the method of flowchart 1400 of Figure 14, And / or configure elements therein, such as system 1900 or processor 1908, to perform the functions of the corresponding steps.

FIG. 20, which includes a combination of FIGS. 20A, 20B, 20C, 20D, and 20E, is an exemplary methodology flowchart 2000 of operating a content playback system in accordance with various exemplary embodiments. A content playback system is, for example, a content playback device or computer system coupled to a display in accordance with various embodiments.

The operation of the exemplary method begins at step 2002, where the content playback system is powered up or started. Operation proceeds from step 2002 to step 2004 where the content playback system receives a first image corresponding to a first rear view portion of the environment. Operation proceeds from step 2004 to step 2006, where the content reproduction system stores the received first image corresponding to the first rear view portion of the environment. Operation proceeds from step 2006 to step 2008, wherein the content reproduction system includes at least a second image corresponding to the first rearview portion of the environment, including at least a second image corresponding to the first rearview portion of the environment And receives one or more additional images. Operation proceeds from step 2008 to step 2010, where the content reproduction system stores the received one or more additional images corresponding to the first rear view portion of the environment. Operation proceeds from step 2010 to step 2012.

In step 2012, the content playback system receives a first image corresponding to a second rear view portion of the environment. Operation proceeds from step 2012 to step 2014, where the content reproduction system stores the received first image corresponding to the second rear view portion of the environment. Operation proceeds from step 2014 to step 2016, wherein the content reproduction system includes a second rear view portion corresponding to the second rear view portion of the environment, And receives corresponding one or more additional images. Operation proceeds from step 2016 to step 2018, where the content reproduction system stores the received one or more additional images corresponding to the second rear view portion of the environment. Operation proceeds from step 2018 to step 2020. [

In step 2020, the content playback system receives one or more images corresponding to the sky view portion of the environment. Operation proceeds from step 2020 to step 2022, where the content playback system stores the received one or more images corresponding to the sky view portion of the environment. Operation proceeds from step 2022 to step 2024, where the content playback system receives one or more images corresponding to the ground view portion of the environment. Operation proceeds from step 2024 to step 2026, where the content reproduction system stores the received one or more images corresponding to the above-ground view portion of the environment. In some embodiments, the sky and ground views refer to the up and down directions with respect to the nominal head direction view of the viewer and are used to apply to indoor environments as well as to outdoor environments.

In some embodiments, the images may be received, for example, according to a particular embodiment, corresponding to a portion of the first rearview portion, the second rearview portion, the sky view portion, and the ground view portion, but not necessarily all .

Operation proceeds from step 2026 via access node A 2028 to step 2030 to step 2034 and through connecting node B 2036 to steps 2028, 2040, 2042, 2044, 2046 , 2048, 2050, 2052). Returning to step 2030, in step 2030, the content playback system determines the head position of the viewer, and the head position corresponds to the current view. Operation proceeds from step 2030 to step 2032, where the content reproduction system determines the current view for the viewer based on the determined head position. Operation proceeds from step 2032 to step 2030. [ Steps 2032 and 2030 are performed periodically, for example, by refreshing the current view, for example, repeatedly. The determined current view is available for use in generating output images.

In step 2034, the content playback system receives a first content stream that provides a content corresponding to the first portion, e.g., a forward partial view of the environment.

In step 2038, the content playback system may be further configured to determine which of the plurality of previously delivered images corresponding to the first back-view portion of the environment is a playback time measured with respect to the playback time indicated in the first content stream Lt; RTI ID = 0.0 > a < / RTI > In step 2040, the content playback system receives image selection information indicating which of a plurality of images corresponding to the first rearview portion of the environment should be used during a portion of the event.

In step 2042, the content playback system may be configured to display a plurality of previously delivered images corresponding to the second back view portion of the environment during a playback time, which is measured with respect to the playback time indicated in the first content stream, The control information indicating whether or not the control information should be transmitted. In step 2044, the content playback system receives image selection information indicating which of a plurality of images corresponding to the second rear view portion of the environment should be used during a portion of the event.

In step 2046, the content playback device determines whether any of the plurality of previously delivered images corresponding to the sky view portion of the environment is to be displayed during a playback time measured with respect to the playback time indicated in the first content stream Lt; / RTI > In step 2048, the content playback system receives image selection information indicating which of a plurality of images corresponding to the sky view portion of the environment should be used during a portion of the event.

In step 2050, the content playback system is to be displayed during a playback time, which of the plurality of previously delivered images corresponding to the above-ground view portion of the environment is measured with respect to the playback time indicated in the first content stream Or the like. In step 2052, the content reproduction device receives image selection information indicating which of a plurality of images corresponding to the above-ground view portion of the environment should be used during a portion of the event.

Operation proceeds from access node C 2054 through steps 2032, 2040, 2042, 2044, 2046, 2048, 2050, 2052 that may be performed concurrently, step 2032, step 2034, Via node D 2056, proceeds to step 2058. [

In some embodiments, the control information from steps 2038, 2040, 2042, 2044, 2046, 2048, 2050, 2052 may be, for example, piecemeal, And is transmitted slightly before the first stream content. In some other embodiments, a block of control information is received before or simultaneously with the start of reception of the first content stream.

In step 2058, the content playback system receives a first partial view of the environment, e.g., content received from a first content stream corresponding to a front partial view, a stored received image corresponding to a first rear view portion of the environment, A stored received image corresponding to the second back view portion of the environment, a stored received image corresponding to the sky view portion of the environment, a stored received image corresponding to the ground view portion of the environment, or a portion of the current view where no image is available Based on at least one of the synthesized image corresponding to the current field of view. Step 2058 includes steps 2060, 2062, 2064, 2066, 2068, 2076,

In step 2060, the content reproduction system determines a set of view portions based on the current view, for example, where there is available data to be used to generate one or more output images. Some exemplary determined sets include, for example, {}, {front view portion}, {first rear view portion}, {second rear view portion}, {sky view portion}, {ground view portion} A forward view portion, a first rear view portion, a forward view portion, a second rear view portion, a forward view portion, a first rear view portion, A first rear view portion, a ground view portion, a forward view portion, a second rear view portion, and a second rear view portion. , A first rear view portion, a second rear view portion}, a first rear view portion, a first rear view portion, a second rear view portion, Second back view portion, sky view portion}, {first rearview portion, second rearview portion, ground view portion}, {second rearview portion, skyview portion} Ground view portion} The.

Operation proceeds from step 2060 to step 2062. [ In step 2062, the content reproduction system determines whether the following two conditions are satisfied: (i) the determined set of view portions from step 2060 includes only the first view portion, and (Ii) there are no parts of the current field of view outside the first view part; If the determined set only includes the first view portion and it is determined that there are no portions of the current view outside the first view portion, then operation proceeds from step 2062 to step 2064; Otherwise, operation proceeds from step 2062 to step 2066. [

In step 2064, the content playback system generates one or more output images corresponding to the current view, based on the content received from the first content stream.

In step 2066, the content playback system determines if any portion of the current view is not available for the image. If the content playback system determines that at least a portion of the current view is not available, the operation proceeds from step 2066 to step 2076; Otherwise, operation proceeds from step 2066 to step 2078. [

In step 2068, the content reproduction system generates one or more output images corresponding to the current view, based on the determined set of view portions to be used when generating one or more output images. Step 2068 may include, but is not limited to, step 2070, wherein the current view is based on at least some received content included in the first content stream and the stored content corresponding to the second portion of the environment Lt; RTI ID = 0.0 > images. ≪ / RTI > In some embodiments, step 2070 includes one or both of steps 2072,2074. In step 2072, the content reproduction system selects an image corresponding to a second partial view of the environment based on the received image selection information. Operation proceeds from step 2072 to step 2074. [ In step 2074, the content reproduction system combines the content obtained from the first content stream captured at the second time point with a first image corresponding to the first time point, and the first and second time points are different .

In some embodiments, the first image is a first image of a second portion of the environment, and the second portion is one of a first rearview portion of the environment and a second rearview portion of the environment. In some such embodiments, the first time point corresponds to a time preceding the second time point. In some such embodiments, the first point in time precedes the time of the real-time event while images of the first content streams are captured.

Returning to step 2076, step 2076 is performed for each part for which an image is not available. In step 2076, the content reproduction system synthesizes an image for a portion of the current view that an image is not available. Operation proceeds from step 2076 to step 2078. [ At step 2078, the content reproduction system generates one or more output images and / or one or more output images corresponding to the current view, based on the determined set of view portions of the environment to be used when generating the one or more composite images . The output images generated by step 2078 may include: a fully synthesized image; An image comprising content from the synthesized image and content from the first content stream; An image comprising content from the synthesized image, content from the first content stream, and content from the stored image; And an image comprising content from the synthesized image and content from the stored image. In various embodiments, step 2078 can include one or both of steps 2080 and 2082, and sometimes includes it.

In step 2080, the content playback system may simulate one or more output images corresponding to the current view and a portion of the environment, e.g., a second portion, based on at least some received content included in the first content stream Lt; / RTI > In step 2080, the content reproduction device combines the combined image with at least a portion of the received image to produce an image corresponding to the current field of view.

It should be understood that the current view can vary over time and is generally so. In response to changes in the current field of view, different sets of view portions used to generate an output image may be determined at step 2060, different images may be required to be present, for example, In step 2076, corresponding to different portions of the field of view that do not. In addition, at different times based on the received control information, different stored images can be identified for use in generating an output image combined at different times.

Operation proceeds from step 2058, via access node E 2084, to step 2086, where the content playback system outputs and / or displays the generated one or more output images. Step 2086 includes the step 2088 of the content playback system outputting and / or displaying a first output image, wherein the first output image is one of one or more generated output images.

In some embodiments, the generated output image, corresponding to the current view, e.g., generated in step 2070, includes information from a first portion of the environment, a second portion of the environment, and a third portion of the environment And sometimes includes it. In some embodiments, a first portion of the environment corresponding to the first content stream is a front view portion, a second portion of the environment includes a first rear view portion, e.g., a right rear view portion, The second rear view portion, e.g., the left rear view portion, and the third portion of the environment is one of the sky view portion and the ground view portion of the environment. In some such embodiments, the content corresponding to the first portion includes real-time content captured and streamed to the playback system while the event is in progress, and the content corresponding to the second and third portions is a non- admit.

In various embodiments, a combination of content for generating an output image corresponding to a current view may be stored in one or more of the boundary areas, for example, an image obtained from a first content stream corresponding to a front view portion of the environment, A stored image corresponding to the first rearview portion of the environment, a stored image corresponding to the second rearview portion of the environment, a stored image corresponding to the sky view portion of the environment, a stored image corresponding to the ground view portion of the environment, Blurring, luminance variations, and / or color variations in a boundary region between any two of the combined images corresponding to a non-existing current in-view area.

In some embodiments, there are first, e.g., some portions of the environment not covered by the front view portion corresponding to the first content stream, and one or more additional portions where the received images are stored. In various embodiments, for these non-included portions of the environment, the images are composited. For example, in one embodiment, the stored image corresponding to the sky view portion may not be present, and when the current view includes a portion of the sky view, the image is composited. In another example, there may be a dead spot, e.g., an unincorporated region between the first rearview portion and the second rearview portion. In some embodiments, compositing the image includes repeating a portion of the image, a portion of the image obtained from the received first content stream corresponding to the contiguous region in the environment, or a portion of the stored image.

In some embodiments, the first image content received in step 2034 is a stereoscopic image content including, for example, a left eye image and a right eye image, wherein pairs of left and right eyes are received. In some such embodiments, the received and stored images corresponding to the first rearview portion, the second rearview portion, the sky view portion, and the ground view portions may include left eye images and right eye images, e.g., Pairs. Thus, for example, in step 2070, when generating one or more output images corresponding to the current view, the left eye image from the first content stream corresponding to the first view portion, e.g., the front view portion, Are combined with stored left eye images corresponding to one or more different parts of the environment and the right eye images from the first content stream corresponding to the first view part, e.g., the front view part, are combined with one or more other Lt; RTI ID = 0.0 > images, < / RTI >

In some other embodiments, the received and stored images corresponding to the first rearview portion, the second rearview portion, the sky view portion, and the ground view portions are moved from the original pair of images to the left eye image or the right eye image (Mono) image from a single camera that contains one or, for example, operates solely to capture the view portion. Thus, in one such embodiment, for example, at step 2070, when generating one or more output images corresponding to the current field of view, a left eye image from the first content stream and corresponding, for example, Both of the corresponding right eye images from the first content stream are combined with the same stored image from the other view portions.

In still other embodiments, some of the stored images include both left and right eye image pairs, and others include one stored image. For example, the received stored stored images corresponding to the first rearview portion may include pairs of left and right eye images, while the stored received images corresponding to the sky view portion may contain single images, e.g., For example, you can include mono images rather than pairs.

In some embodiments, for example, the first portion of the environment corresponding to the received first content stream is a front view portion and the second portion of the environment corresponds to a rear view portion, e.g., right rear of the viewer The second rear view portion corresponding to the left rear of the viewer and the third portion of the environment is the sky view portion or the ground view portion. In some such embodiments, images are received at different rates corresponding to the first, second, and third portions, and more images are received for events corresponding to the first portion than the second portion do. In some such embodiments, more images are received corresponding to the second portion than the third portion.

According to some embodiments, an exemplary method of operating a content playback system includes: determining a head position of a viewer corresponding to a current view; Receiving a first content stream that provides content corresponding to a first portion of the environment; At least some received content included in the first content stream and i) a stored content corresponding to a second portion of the environment or ii) a second portion corresponding to the current view based on a composite image simulating a second portion of the environment Generating output images; And outputting or displaying a first output image, wherein the first output image is one of one or more generated output images. In some embodiments, the content playback system is a content playback device. In some embodiments, the content playback system is a computer system coupled to a display.

In various embodiments, the method includes: receiving a first image corresponding to a second portion of the environment; And storing the first image corresponding to the second portion of the environment.

In some embodiments, the first image of the second portion of the environment corresponds to a first viewpoint; Generating one or more output images corresponding to a current view includes combining content obtained from the first content stream captured at a second viewpoint with the first image corresponding to the first viewpoint, And the second time points are different. In some such embodiments, the first viewpoint corresponds to a time preceding the second viewpoint. In some such embodiments, the first viewpoint precedes the time of the real-time event while images included in the first content stream are captured.

In various embodiments, the method further comprises receiving one or more additional images corresponding to the second portion of the environment, wherein the one or more additional images corresponding to the second portion of the environment At least a second image.

In some embodiments, the method further comprises determining which of a plurality of previously delivered images corresponding to the second portion of the environment should be displayed during a playback time measured with respect to the playback time indicated in the first content stream And receiving control information.

In some embodiments, the second portion of the environment is one of a first rearview portion, a second rearview portion, a sky view portion, or a ground view portion. In some such embodiments, the method further comprises receiving one or more images corresponding to a third portion of the environment.

In various embodiments, the first portion of the environment is a front view portion; Wherein the third portion is one of a sky view or a ground view portion, the images are received at different rates corresponding to the first, second, and third portions, and more images are viewed with respect to the second portion And is received for an event corresponding to the first portion.

In various embodiments, the content corresponding to the first portion includes real-time content captured and streamed to the playback device while the event is in progress; The content corresponding to the images corresponding to the second and third portions are non real-time images. In some such embodiments, the method includes receiving image selection information indicating which of a plurality of images corresponding to the second portion of the environment should be used during a portion of the event, and determining, based on at least some received content, Generating the one or more output images corresponding to the second portion of the environment comprises selecting an image corresponding to the second portion of the environment based on the received image selection information.

In various embodiments, the exemplary method includes determining that an image is not available for a portion of the current view; Synthesizing an image to be used for the portion of the current field of view where no image is available; And combining the synthesized image with at least a portion of the received image to produce an image corresponding to the current view.

In various embodiments, the first image content is a stereoscopic image content comprising a left eye image and a right eye image.

21 illustrates an exemplary content playback system (e.g., a content playback system) implemented in accordance with the present invention that can be used to receive, decode, store, process, and display imaging content received from a content delivery system such as the one shown in FIGS. 2100). The system 2100 may be implemented as a single playback device 2100 'including a display 2102 or as a combination of elements such as an external display, e.g., a head mounted display 2105 coupled to a computer system 2100' .

Content playback system 2100, in at least some embodiments, is a 3D head-mounted display. The head mount display may be implemented using a OCULUS RIFT ^TM VR (Virtual Reality) headset, which may include a head mount display 2105. In various embodiments, the head-mounted display 2105 is identical to the head-mounted display 805. Other head mount displays may also be used. In some embodiments, one or more display screens of the head mount helmet or other head mounting device are used to display content on the left and right eyes of the user. By displaying different images on the left and right eyes on a single screen by means of a head mount configured to expose different parts of a single screen to different eyes, a single display can be displayed on the left and right eyes Can be used to display images. In some embodiments, the mobile phone screen is used as a display of a head-mounted display device. In at least some such embodiments, the mobile phone is inserted into a head mounting device, and the mobile phone is used to display images.

The content playback system 2100 may decode the received encoded image data, e.g., left and right eye images and / or mono (single images) corresponding to different portions of the environment or scene, , And has the ability to generate a 3D image for display to a customer by rendering and displaying different left and right eye views recognized by the user as a 3D image. The content playback system 2100 is located in a customer premise location, such as a home or office in some embodiments, but may also be located at an image capture site. The content playback system 2100 may perform signal reception, decoding, processing, generation, display and / or other operations in accordance with the present invention.

The system 2100 includes a display 2102, a display device interface 2013, an input device 2104, an input / output (I / O) interface 2106, a processor 2108, a network interface 2110, 2112). Memory 2112 includes an assembly of modules 2114, e.g., an assembly of software modules, and data / information 2116. In some embodiments, system 2100 includes an assembly of modules 2115, e.g., assemblies of hardware modules, e.g., circuits. The various components of system 2100 are coupled together via bus 2109 and / or via other connections or via an air interface to allow data to be communicated between the components of system 2100. In some embodiments, the display 2102 is included as a selectable element shown using a dash box, but in some embodiments, the external dicingplay device 2105, e.g., a headmount stereoscopic image display device, May be coupled to the playback device via the device interface 2103.

For example, if a cellular phone processor is used as the processor 2108 and the cellular phone generates and displays images on the head mount, the system may include a processor 2108, a display 2102, and a memory 2112). Processor 2108, display 2102, and memory 2112 may all be part of a cellular telephone. In other embodiments of the system 2100, the processor 2108 may be part of a gaming system, such as an XBOX or PS4, having a display 2105 mounted to the head mounting device and coupled to the gaming system. Whether or not the processor 2108 and / or the memory 2112 is located in a head-writing device is not as important as can be appreciated, while in some cases it is important to note that from a power, heat, It may be convenient to place the processor 2108 in a common location, and in at least some cases, it may be desirable to have the processor 2108 and the memory 2112 coupled to the headgear that includes the display.

Although various embodiments contemplate the head mount display 2105 or 2102, the method and apparatus may also be used with non-head mounted displays capable of supporting 3D images. Thus, in many embodiments, the system 2100 includes a head mounted display, but this may also be implemented by a non-head mounted display.

Memory 21112 includes routines that, when executed by various modules, e.g., processor 2108, control content playback system 2100 to perform operations in accordance with the present invention. Memory 2112 includes an assembly of modules 2114, e.g., an assembly of software modules, and data / information 2116.

Data information 2116 includes: received images corresponding to the first back view portion 2118, received images corresponding to the second rearview portion 2120, received images corresponding to the sky view portion 2112 Images, and received images corresponding to the ground view portion 2124, all of which are shown in FIG. Exemplary received images corresponding to the rear view portion 2118 or 2120 may include, for example, images of a standing audience or audience, images of a sitting audience or audience, images with different visible advertisements, images of a cheering crowd And the like. Exemplary received images corresponding to sky views include, for example, clear skies, different cloud patterns, degrees of different darkness corresponding to different times, and the like. Data information 2116 includes: control information corresponding to first back view portion 2130, received control information corresponding to second rear view portion 2132, received control information corresponding to sky view portion 2134, , And received control information corresponding to the ground view portion 2136, all or one or more. The data information 2116 includes a determined current viewer head position 2126, a determined current view 2128, a received right first eye image corresponding to the front view portion, and a left received first content stream (S) 2140 and corresponding output images 2138 corresponding to the current view in which at least some of the received content available to combine the generated output images 2142 is present, Lt; RTI ID = 0.0 > a < / RTI > The generated output images 2142. The generated output images include a first, e.g., combined image content from a first content stream corresponding to a front view portion, and may include, for example, a back view portion, a sky portion, And / or may comprise a synthesized image or a generated output image comprising a portion of the synthesized image, and sometimes includes the generated output image.

22 is a diagram of an assembly of modules 2200 that may be included in system 800 of FIG. 8 or an exemplary content playback system 2100 of FIG. 21, in accordance with one exemplary embodiment. The modules in the assemblies of the modules 2200 may be fully implemented in hardware within the processor 2108, e.g., as separate circuits, and in some embodiments, so implemented. The modules in the assemblies of modules 2200 may be fully implemented in hardware within the assembly of modules 2115, e.g., as separate circuits corresponding to different modules, and so implemented in some embodiments. In other embodiments, some of the modules are implemented as, for example, circuits in a processor 2108 having other modules implemented as circuits outside of the processor 2108 and in the assemblies 2115 of the modules coupled thereto . As will be appreciated, the level of integration with the modules on the processor and / or some modules external to the processor may be one of the design choices.

Alternatively, rather than as implemented as circuits, all or a portion of the modules may be implemented in software and stored in memory 2112 of system 2100, and modules may be coupled to a processor, e. The modules control the operation of the system 2100 to perform functions corresponding to the modules. In some such embodiments, the assembly of modules 2200 is included in the memory 2112 as an assembly of modules 2114. In other embodiments, the various modules of the assembly 2200 of modules are implemented as a combination of hardware and software, for example, other circuitry external to the processor may be operable to perform a portion of the functionality of the modules under software control Lt; / RTI > It is to be understood that the processor 2108 may be implemented as one or more processors, e.g., computers, although it is shown in the FIG. 21 embodiment as a single processor, e.g., a computer.

When implemented in software, the modules include code that, when executed by the processor 2108, configures the processor 2108 to perform a function corresponding to the module. In embodiments in which the assembly 2200 of modules is stored in a memory 2112, the memory 2112 may include code for causing at least one computer, e.g., processor 2108, For example, a computer program product comprising a computer readable medium comprising individual code for each module.

Completely hardware-based or completely software-based modules can be used. However, it should be understood that any combination of software and hardware, for example, circuit implemented modules, may be used to implement the functions. As will be appreciated, the modules shown in FIG. 22 may be implemented as a system 2100 or processor 2108 to perform the functions of corresponding steps shown and / or described in the method of flowchart 2000 of FIG. Controls and / or configures the elements. Thus, the assembly of modules 2200 includes various modules that perform the functions corresponding to one or more of the steps of FIG.

The assembly of modules 2200 includes a viewer head positioning module 2202, a current view determination module 2204, a content stream selection module 2206, a content stream reception module 2208, an image reception module 2210, A storage module 2212, a control information receiving module 2214, an output image generating module 2216, an output module 2242, a display module 2244, and control routines 2246.

The viewer head positioning module 2202 is configured to determine the head position of the viewer, and the head position corresponds to the current field of view. The current field of view determination module 2204 is configured to determine a current field of view of the viewer based on the determined head position.

The content stream selection module 2206 is configured to select a content stream from among a plurality of alternative content streams based on, for example, a user, e.g., a viewer, input. The different content streams may correspond to different events. In various embodiments, the different content streams corresponding to the same event correspond, for example, to different cameras directed to different directions to provide alternative front view views to the viewer. In some embodiments, at least some of the selectable image streams include stereoscopic image content that includes left eye image and right eye image pairs.

The content stream receiving module 2208 is configured to receive a first content stream that provides content corresponding to a first location, e. G., A forward partial view, of an embodiment. In various embodiments, the first image content received in the first content stream is stereoscopic image content comprising pairs of left eye image and right eye image.

The image receiving module 2210 is configured to receive images corresponding to one or more different view portions, e.g., a first rearview portion, a second rearview portion, a sky view portion, and a ground view portion. An image receiving module 2210 is configured to receive a first image corresponding to a second portion of the environment. In some such embodiments, the image receiving module is also configured to receive one or more additional images corresponding to the second portion of the environment, wherein the one or more additional images include at least a second portion of the environment . In some embodiments, the second portion of the environment is one of a first rearview portion, a second rearview portion, a sky view portion, or a ground view portion. In some embodiments, the second portion of the environment is one of a first rearview portion or a second rearview portion. In some embodiments, the image receiving module 2210 is configured to receive one or more images corresponding to a third portion of the environment. In some embodiments, the first portion of the environment is the front view portion and the third portion of the environment is the sky view or the ground view portion.

In some embodiments, the image receiving module 2210 is configured to receive a first image corresponding to a first rearview portion of the environment, and to receive one or more additional images corresponding to the first rearview portion of the environment Wherein the one or more additional images corresponding to the first rearview portion include at least one second image corresponding to the first rearview portion. In some embodiments, the image receiving module 2210 is configured to receive a first image corresponding to a second rear view portion of the environment and to receive one or more additional images corresponding to the second rearview portion of the environment And wherein the one or more additional images corresponding to the second rearview portion include at least one second image corresponding to the second rearview portion. In some embodiments, the image receiving module 2210 is configured to receive one or more images corresponding to the sky view portion of the environment. In some embodiments, the image receiving module 2210 is configured to receive one or more images corresponding to a ground view portion of the environment.

The received image storage module 2212 is configured to store the images received by the received image storage module 2212. The received image storage module 2212 is configured to store a first image corresponding to the second portion of the environment. The received image storage module 2212 is configured to store one or more additional images corresponding to a second portion of the environment. The received image storage module 2212 is configured to store the received one or more images corresponding to a third portion of the environment. In various embodiments, the received image storage module 2212 stores a first image corresponding to the first rear view portion of the environment, and one or more additional images corresponding to the first rearview portion of the environment . In some embodiments, the received image storage module 2212 is configured to store one or more additional images corresponding to the sky view portion of the environment. In some embodiments, the received image storage module 2212 is configured to store one or more images corresponding to the above-ground view portion of the environment.

Control information receiving module 2214 receives control information indicating which of a plurality of previously delivered images corresponding to a second portion of the environment should be displayed for a playback time measured with respect to playback times indicated in the first content stream Information. In various embodiments, the control information receiving module 2214 may determine which of the plurality of previously delivered images corresponding to the third portion of the environment is the playback time measured with respect to the playback times represented in the first content stream Lt; RTI ID = 0.0 > a < / RTI > In some embodiments, control information receiving module 2214 is configured to receive image selection information that indicates which of a plurality of images corresponding to a second portion of the environment should be used during a portion of an event. In some embodiments, the control information receiving module 2214 is configured to receive image selection information indicating which of a plurality of images corresponding to a third portion of the environment should be used during a portion of the event.

In some embodiments, the control information receiving module 2214 may be configured to determine which of the plurality of previously delivered images corresponding to the first rear view portion of the environment is a playback Lt; / RTI > is to be displayed for a period of time. In some embodiments, the control information receiving module 2214 receives image selection information indicating which of a plurality of previously delivered images corresponding to the first rear view portion of the environment should be used during a portion of an event . In some embodiments, the control information receiving module 2214 may determine which of the plurality of previously delivered images corresponding to the second rear view portion of the environment is a playback Lt; / RTI > is to be displayed for a period of time. In some embodiments, the control information receiving module 2214 receives image selection information indicating which of a plurality of previously delivered images corresponding to the second rear view portion of the environment should be used during a portion of an event .

In some embodiments, the control information receiving module 2214 may be configured to determine which of a plurality of previously delivered images corresponding to the sky view portion of the environment is to be played during a playback time measured in terms of playback time in the first content stream And to receive control information indicating whether it should be displayed. In some embodiments, the control information receiving module 2214 is configured to receive image selection information indicating which of a plurality of images corresponding to the sky view portion of the environment should be used during a portion of an event. In some embodiments, the control information receiving module 2214 may be configured to determine which of the plurality of previously delivered images corresponding to the terrestrial view portion of the environment is to be played during a playback time measured in terms of playback time in the first content stream And to receive control information indicating whether it should be displayed. In some embodiments, the control information receiving module 2214 is configured to receive image selection information that indicates which of a plurality of images corresponding to the above-ground view portion of the environment should be used during a portion of an event.

Output image generation module 2216 includes: content received from a first content stream corresponding to a first partial view of the environment, a stored received image corresponding to a first rear view portion of the environment, A stored received image corresponding to the sky view portion of the environment, a stored received image corresponding to the ground view portion of the environment, or a composite image corresponding to a portion of the current view in which the image is not available, Image, based on at least one of the plurality of output images. Output image generation module 2216 includes a view portion set determination module 2218, a content stream only determination module 2220, a missing portion determination module 2222, an image synthesizer module 2224, and a content stream output image generation module 2226 A synthesized output image generation module 2228, an output image content stream based generation module 2230, and an output image non-stream based generation module 2236.

View subset determination module 2218 is configured to determine a set of view portions, based on the current view, for example, where at least some of the available image content of the environment to be used when generating one or more output images exists . Some exemplary determined sets include, for example, {}, {front view portion}, {first rear view portion}, {second rear view portion}, {sky view portion}, {ground view portion} A forward view portion, a first rear view portion, a forward view portion, a second rear view portion, a forward view portion, a first rear view portion, A first rear view portion, a ground view portion, a forward view portion, a second rear view portion, and a second rear view portion. , A first rear view portion, a second rear view portion}, a first rear view portion, a first rear view portion, a second rear view portion, Second back view portion, sky view portion}, {first rearview portion, second rearview portion, ground view portion}, {second rearview portion, skyview portion} Ground view portion} The.

The content stream-only determination module 2220 determines whether the one or more output images are received images that are stored from other partial views, e.g., the first back view portion, the second back view portion, the sky view portion, And is based on the content in the first content stream without having to rely on the synthesized image. Content stream only determination module 2220 determines whether the determined set includes a single element that is a first, e.g., a forward view portion, corresponding to the first content stream, and if the view is a first, e.g., Or the like.

Missing portion determination module 2222 determines whether an image is not available for a portion of the current view, for example, if the image corresponds to a first content stream that corresponds to a front view of the environment or to another portion of the environment And to determine that an image is not available from the received stored image. The image synthesizer module 2224 is configured to synthesize the image to be used for the portion of the current field of view where no image is available. In various embodiments, the image synthesizer module 2224 generates a composite image that is slightly larger than needed to fill the missing portion, for example, to allow for the entire portion on the boundary.

The content stream output image generation module 2226 determines whether the determined set includes only the first view portion, e.g., only the front view portion, And to generate one or more output images corresponding to the current field of view based on the content received from the first content stream when portions of the current field of view are outside the view region. In some embodiments, the content stream output image generation module 2226 performs a cropping operation on the image obtained from the first content stream.

Output image content stream based generation module 2230 may include at least some received content included in the first content stream, e.g., corresponding to a front view, and i) stored content corresponding to a second portion of the environment, e.g., A stored image corresponding to a first rear view, a second rear view, a sky view or a ground view, or ii) a composite image that simulates a second portion of the environment, And to generate one or more output images corresponding to the current field of view based on the synthesized image. The stored content corresponding to the second portion of the environment is, for example, a stored image stored by the received image storage module 2212. The composite image is, for example, an image generated by module 2224. In some embodiments, the output image content stream based generation module 2230 is configured to generate an image corresponding to the second portion of the environment based on the received image selection information, As a part.

Output image content stream based generation module 2230 includes a synthesized image integration module 2232 configured to integrate one or more synthesized images into an output image to be generated. The combined image integration module 2232 may be configured to receive from the received image, e.g., a first, e.g., a front view portion obtained from the received first content stream to generate an image corresponding to the current view Or combined with at least a portion of the stored received image corresponding to one of the first back view portion, the second back view portion, the sky view portion, or the ground view portion. The stored image integration module 2134 is configured to integrate a portion of one or more stored images into an output image to be generated. As part of generating an output image, module 2230, module 2232, and / or module 2234 perform mixing in the border regions. In various embodiments, the blending step includes filtering, blurring, luminance variations and / or color variations.

Output image non-stream based generation module 2236 may include: i) a stored image corresponding to a portion of the environment, e.g., a first back view, a second rear view, a sky view, or a stored image corresponding to a ground view, Or ii) a composite image that simulates a portion of the environment, e.g., a composite image that simulates a portion of the field of view where an image is not available, to generate one or more output images corresponding to the current field of view based on at least one of . Module 2236 generates one or more images when the current view is outside the area corresponding to the first, e.g., the front view portion corresponding to the first content stream. In some embodiments, the output image non-content stream based generation module 2236 generates an image corresponding to the second portion of the environment based on the received image selection information, to generate one or more output images corresponding to the current field of view As a constituent part.

Output image non-stream based generation module 2236 includes a composite image integration module 2238 configured to integrate one or more composite images into an output image to be generated. Composite image integration module 2238 may be configured to receive a received image from a received image, e.g., a first back view portion, a second rear view portion, a sky view portion, or a ground view portion, And to combine the synthesized image with at least a portion of the image. The stored image integration module 2240 is configured to integrate a portion of one or more stored images into an output image to be generated. As part of generating an output image, module 2236, module 2232, and / or module 2234 perform mixing in the border regions. In various embodiments, the blending includes filtering, blurring, luminance variations and / or color variations.

Output module 2242 may be one or more of the output image content stream based generation module 2130, the content stream output image generation module 2130, and the output image non-content stream based generation module 2236, And output the generated output images, wherein the one or more output images comprise a first output image. The output module is configured to output the first output image, for example, via the network interface 2110 and / or the display device interface 2103.

Display module 2244 may include one or more of the output image content stream based generation module 2230, the content stream output image generation module 2230, and the output image non content stream based generation module 2136, for example, And display the generated output images, wherein the one or more output images comprise a first output image. Display module 2244 is configured to display a first output image, e.g., via display 2102 and / or display 2105. [

Control routines 2246 include device control routines and communication routines for controlling the operation of system 2100. [

An exemplary content playback system, e.g., system 2100 of FIG. 21, according to the present invention, includes: a viewer head positioning module 2202 configured to determine a viewer's head position, Corresponding viewer head positioning module 2202; A content stream receiving module configured to receive a first content content that provides content corresponding to a first portion of the environment; At least some received content included in the first content stream and i) stored content corresponding to a second portion of the environment, or ii) a second portion corresponding to the current view based on the composite image simulating the second portion of the environment An output image content stream based generation module (2330) configured to generate one or more output images; And a display module (2244) configured to display a first output image, wherein the first output image is one of one or more generated output images , The output module 2242, or the display module 2244.

In some embodiments, the content playback system 2100 is a content device 2100 '. In some embodiments, the content playback system 2100 is a computer system 2100 'coupled to the display 2105. [

In some embodiments, the system comprises: an image receiving module (2210) configured to receive a first image corresponding to the second portion of the environment; And a received image storage module (2212) configured to store the first image corresponding to the second portion of the environment.

In various embodiments, the first image of the second portion of the environment corresponds to a first viewpoint, and the output image content stream-based generation module 2239 generates a second image stream from the first content stream captured at a second viewpoint And combine the acquired content with the first image corresponding to the first viewpoint, wherein the first and second viewpoints are different. In some such embodiments, the first viewpoint corresponds to a time preceding the second viewpoint. In various embodiments, the first viewpoint precedes the time of the real-time event while the images in which the first content stream is captured are included.

In some embodiments, the image receiving module 2210 is further configured to receive one or more additional images corresponding to the second portion of the environment, wherein the one or more additional images corresponding to the second portion of the environment At least a second image.

In various embodiments, the system may be configured to: display any of a plurality of previously delivered images corresponding to the second portion of the environment during a playback time measured with respect to playback times indicated in the first content stream And a control information receiving module 2214 configured to receive control information indicating whether the control information is received.

In some embodiments, the second portion of the environment is one of a first rearview portion, a second rearview portion, a sky view portion, or a ground view portion. In some such embodiments, the image receiving module 2210 is also configured to receive one or more images corresponding to a third portion of the environment.

In some embodiments, the first portion of the environment is a front view portion; Wherein the third portion is one of a sky view or a ground view portion, images are received at different rates corresponding to the first, second, and third portions, and more images are received at different rates than the second portion 1 < / RTI >

In some embodiments, the content corresponding to the first portion includes real-time content captured and streamed to the playback device while the event is in progress; The content corresponding to the images corresponding to the second and third portions are non real-time images. In some such embodiments, the control information receiving module 2214 is also configured to receive image selection information indicating which of a plurality of images corresponding to the second portion of the environment should be used during a portion of the event Being; The output image content stream based generation module 2230 is configured to select an image corresponding to the second portion of the environment based on the received image selection information as a portion configured to generate one or more output images corresponding to the current field of view .

In various embodiments, the system includes a missing portion determination module 2222 configured to determine that an image is not available for a portion of the field of view; An image synthesizer module (2224) configured to synthesize an image to be used for the portion of the field of view where an image is not available; And a composite image integration module 2232 configured to combine the synthesized image with at least a portion of the received image to produce an image corresponding to the current view.

In various embodiments, the first image content is a stereoscopic image content that includes a left eye image and a right eye image.

In some embodiments, the assembly 2200 of modules is included in the content playback system 800 of FIG. 8, for example, in addition to the other modules shown in FIG. The modules in the assembly 2200 of modules may be included as a hardware module external to the processor 808 coupled to the memory 812, to the processor 808, and / or to the bus 809 in the system 800.

23 is a drawing 2300 illustrating an exemplary stream selection module 1938 that may be used in the playback system 1900 in accordance with some embodiments.

FIG. 24 is a drawing 2400 illustrating an exemplary stream prioritization module 2306 that may be implemented as part of the stream selection module of FIG. 23 or as a separate module.

Some embodiments relate to a non-transitory computer readable medium embodying a set of software instructions, e.g., computer-executable instructions, for controlling a computer or other device to encode and compress stereoscopic video. Other embodiments are directed to a computer readable medium embodying a set of software instructions, e.g., computer executable instructions, for controlling a computer or other device to decode and decompress video on a player end. Although referred to as separate operations capable of encoding and compressing, it should be understood that the encoding may be used to perform compression, and therefore the encoding may include compression in some cases. Similarly, decoding may include decompression.

The techniques of various embodiments may be implemented using software, hardware, and / or a combination of software and hardware. Various embodiments are directed to an apparatus, e.g., an image data processing system. The various embodiments also relate to methods, e.g., methods of processing image data. The various embodiments may also be implemented in a non-volatile machine, e.g., a computer, a readable medium, such as a ROM, a RAM, a CD, a CD, Hard disks, and the like.

Various features of the present invention are implemented using modules. Such modules may be implemented as software modules, and so implemented in some embodiments. In other embodiments, the modules are implemented in hardware. In yet other embodiments, the modules are implemented using a combination of software and hardware. In some embodiments, the modules are implemented as separate circuits, and each module is implemented as a circuit for the module to perform the corresponding function. A wide variety of embodiments are contemplated including some embodiments in which different modules are implemented differently, e.g., using hardware, some using software, and some using hardware and software combinations. It should also be noted that routines and / or subroutines, or some of the steps performed by such routines, may be executed in dedicated hardware as opposed to software executed on a general purpose processor. These embodiments are within the scope of the present invention. Many of the above-described methods or method steps may be implemented in a memory device, such as a memory device, for controlling a general purpose computer with or without additional hardware, e.g., to implement all or a portion of the methods described above. For example, it may be implemented using machine executable instructions, such as software included in a machine-readable medium, such as RAM, floppy disk, and the like. Thus, among other things, the present invention relates to a machine-readable medium comprising machine-executable instructions for causing a machine, e.g., a processor and associated hardware, to perform one or more of the steps of the described method (s) will be.

Numerous additional variations on the methods and apparatus of the various embodiments described above will be apparent to those skilled in the art in view of the foregoing description. These variations are considered within the scope.

700: Content Delivery System 702: Display
704: input device 706: I / O interface
708: Processor 710: Network Interface
712: Memory 714: Control routines
716: Split module 718: Encoder (s)
720: streaming controller 722: request processing module
724: data rate determination module 726: current head positioning module
728: Selection module 730: Streaming control module
1900 ': Computer system / content playback device
1902: Display
1903: Display device interface
1904: input device
1905: Display, for example, a head-mounted stereoscopic display
1906: I / O interface 1908: Processor
1910: Network interface 1911: Receiver
1912: Memory 1913: Transmitter
1914: Control routines 1916: Head positioning module
1918: Current viewing location initialization module
1920: Decoder module
1922: Currently selected stream initialization module
1924: Content Delivery Initiation Module 1926: Image Buffer (s)
1928: Image Buffer Update Module
1930: an image rendering module, for example an image generation module
1932: Available bandwidth and / or supportable data rate determination module
1934: Head position change detection module
1936: Available bandwidth and / or supportable data rate change determination module
1938: Stream selection module
1940: Selected stream set change determination module
1942: Selected stream set update module
1944: Stream (s) determination module 1946: received stream information
1948: received bandwidth and / or data rate allocation control information
1950: determined current maximum available bandwidth and / or supported data rate
1952: Received encoded data, for example, encoded scene portions
1954: received environment map 1956: received UV map (s)
1958: Decoded data, e.g., decoded scene portions
1960: Generated 3D content 1961: Currently selected stream set
2200: assembly of modules 2202: viewer head positioning module
2204: Current view determination module 2206: Content stream selection module
2208: Content stream receiving module
2210: image receiving module 2212: received image storing module
2214: Control information receiving module 2216: Output image generating module
2218: View subset determination module
2220: Decision module dedicated to content stream
2222: missing part determination module 2224: image synthesizer module
2226: Content stream output image generation module
2228: Composite output image generation module
2230: Output image content stream based generation module
2232: synthesized image integration module 2234: stored image integration module
2236: Output image non-stream based generation module
2238: synthesized image integration module 2240: stored image integration module
2242: Output module 2244: Display module
2246: Control routines

Claims (68)

A method of operating a playback system,
Selecting, based on the user's head position, which of the plurality of content streams to receive for initial playback use; And
And receiving one or more selected content streams for use for playback. The method according to claim 1,
Wherein said selecting includes prioritizing content streams based on the head position of the user. 3. The method of claim 2,
Wherein the prioritizing step comprises:
Identifying one or more streams conveying content corresponding to the current view of the user; And
Prioritizing one or more content streams that provide portions of the scene corresponding to the current view, based on the size of portions of the view that each stream provides. The method of claim 3,
Wherein the prioritizing of the content streams comprises the step of providing one or more additional images for delivering content corresponding to portions outside the current view based on at least one of proximity to the current view of the transferred image portions or direction of head rotation ≪ / RTI > further comprising prioritizing the streams. 5. The method of claim 4,
Wherein prioritizing the content streams comprises prioritizing one or more additional streams delivering content corresponding to portions outside the current view based on proximity of the delivered image content to the current view Including,
Wherein a content stream delivering image content in close proximity to a region of the current view is assigned a higher priority than the current view field and a content stream far away therefrom. 5. The method of claim 4,
Wherein prioritizing the one or more additional content streams is based on a direction of the head rotation of the user and the content stream providing image content that is outside of the current view but in the direction of head rotation is moved away from the direction of the head rotation Wherein a higher priority is assigned to another content stream that provides image content outside of the current view in the direction of the current view. 3. The method of claim 2,
Wherein the selecting includes selecting one or more content streams to which the highest priority is assigned. 3. The method of claim 2,
Wherein the selecting comprises:
Determining an available bandwidth to receive the content streams; And
Further comprising selecting from a plurality of content streams having the same priority based on the determined amount of available bandwidth. 9. The method of claim 8,
Further comprising determining a bandwidth for at least one content stream based on a bandwidth limitation delivered to the playback system,
Wherein selecting from the plurality of content streams having the same priority is also based on a bandwidth determined for the at least one content stream. 10. The method of claim 9,
Further comprising receiving bandwidth constraints for different viewing directions,
Wherein the individual bandwidth limitations specify a maximum bandwidth to be used to receive one or more content streams that provide content corresponding to a viewing direction corresponding to the individual bandwidth limitations. 9. The method of claim 8,
Wherein selecting from the plurality of content streams having the same priority comprises selecting one content stream from the plurality of content streams to which the highest priority is assigned, Wherein the content stream provides content corresponding to the same viewing direction. 12. The method of claim 11,
Wherein selecting from among the plurality of content streams having the same priority comprises selecting one of the plurality of content streams to which a second highest priority is assigned, Wherein each content stream assigned a priority provides content corresponding to the same viewing direction. The method according to claim 1,
Further comprising receiving guide information that provides information about content streams that the playback system may choose to receive. 14. The method of claim 13,
Wherein the guide information includes information that can be used to access the content streams for which the guide information is provided. 15. The method of claim 14,
The guide information comprising, for a first content stream, a multicast address of a multicast group that can be connected to receive the first content stream, a multicast address of a multicast group that is to be connected to receive the first content stream for a switched digital video channel used to provide the first content stream A playback system comprising one of the information that can be used to request access or channel tuning information that can be used to tune the tuner of the playback system to tune to the broadcast channel on which the first content stream is broadcast How to operate. 16. The method of claim 15,
Further comprising initiating delivery of the selected content stream,
Wherein initiating delivery of the selected content stream comprises transmitting a signal to connect to a multicast group corresponding to the selected content stream. 16. The method of claim 15,
Further comprising initiating delivery of the selected content stream,
Wherein initiating delivery of the selected content stream comprises forwarding a request to a device in the network requesting delivery of a switched digital channel over which the selected content stream is delivered. In the reproducing system,
A stream selection module configured to select, based on the user's head position, which of a plurality of content streams is to be initially received for use in playback; And
And a receiver configured to receive one or more selected content streams for use for playback. 19. The method of claim 18,
Wherein the stream selection module comprises a stream prioritization module configured to prioritize content streams based on the head position of the user. 20. The method of claim 19,
The stream prioritization module comprising:
An identification module configured to identify one or more streams conveying content corresponding to a current view of the user; And
And a priority assignment module configured to prioritize one or more content streams providing a portion of the scene corresponding to the current view based on the size of the portions of the view provided by each stream. 21. The method of claim 20,
Wherein the stream prioritization module prioritizes one or more additional streams delivering content corresponding to portions outside the current view based on at least one of a proximity of the delivered image portions for the current view or a direction of head rotation Further comprising an additional stream prioritization module configured to rank. 22. The method of claim 21,
Wherein the additional stream prioritization module is configured to prioritize one or more additional streams delivering content corresponding to portions outside the current view based on proximity of the delivered image content for the current view, Wherein a content stream carrying image content that is very close to the current view area is assigned a higher priority than the current view field and a content stream farther away therefrom. 22. The method of claim 21,
The additional stream prioritization module prioritizes one or more additional content streams based on the direction of the user's head rotation and provides a content stream that is external to the current view but provides image content that is in the direction of head rotation Wherein a higher priority is assigned than another content stream providing image content outside the current view in a direction away from the direction of head rotation. 20. The method of claim 19,
Wherein the stream selection module is further configured to select one or more content streams assigned the highest priority. 20. The method of claim 19,
Further comprising an available bandwidth and data rate determination module configured to determine an available bandwidth to receive the content streams,
Wherein the stream selection module comprises a selection module configured to select from a plurality of content streams having the same priority based on the determined amount of available bandwidth. 26. The method of claim 25,
Wherein the stream selection module further comprises a stream bandwidth determination module configured to determine a bandwidth for at least one content stream based on a bandwidth limitation delivered to the playback system,
Wherein the selection module is configured to select from the plurality of content streams having the same priority based also on the determined bandwidth for the at least one content stream. 27. The method of claim 26,
The receiver is also configured to receive bandwidth constraints for different viewing directions, and the individual bandwidth constraints comprise a maximum bandwidth to be used to receive one or more content streams that provide content corresponding to the viewing direction corresponding to the individual bandwidth constraints Playback system. 26. The method of claim 25,
Wherein the selection module is configured to select one content stream from among a plurality of content streams assigned the highest priority, and wherein each content stream to which the highest priority is assigned selects content corresponding to the same viewing direction As portions that are configured to select from a plurality of content streams having the same priority. 29. The method of claim 28,
Wherein the selection module is configured to select from a plurality of content streams having the same priority and to select one content stream from the plurality of streams to which a second highest priority is assigned, Wherein each content stream assigned a highest priority of 2 provides content corresponding to the same viewing direction. 19. The method of claim 18,
Wherein the receiver is further configured to receive guide information that provides information about content streams that the playback system may choose to receive. 31. The method of claim 30,
Wherein the guide information comprises information that can be used to access the content streams for which the guide information is provided. 32. The method of claim 31,
The guide information comprising, for a first content stream, a multicast address of a multicast group that can be connected to receive the first content stream, a multicast address of a multicast group that can be connected to receive the first content stream for a switched digital video channel used to provide the first content stream Information that can be used to request access or channel tuning information that can be used to control the tuner of the playback system to tune to a broadcast channel on which the first content stream is broadcast. 33. The method of claim 32,
Further comprising a content delivery initiation module configured to initiate delivery of the selected content stream, wherein the content delivery initiation module is further configured to send a signal to concatenate the multicast group corresponding to the selected content stream. 33. The method of claim 32,
Further comprising a content delivery initiation module configured to initiate delivery of the selected content stream, wherein the content delivery initiation module is further configured to generate a request for delivery of the switched digital channel over which the selected content stream is delivered, To the playback system. 17. A non-transitory computer readable medium comprising processor executable instructions,
The processor executable instructions, when executed by a processor,
Selecting which of a plurality of content streams is to be used for initial playback based on the head position of the user,
And to receive one or more selected content streams for use for playback. A method of operating a content playback system,
Determining a head position of a viewer corresponding to a current visual field;
Receiving a first content stream that provides content corresponding to a first portion of the environment;
I) at least some received content included in the first content stream, and i) stored content corresponding to a second portion of the environment, or ii) corresponding to the current view based on a composite image simulating a second portion of the environment Generating one or more output images; And
Outputting or displaying a first output image that is one of said one or more generated output images. 37. The method of claim 36,
Wherein the content playback system is a content playback device. 37. The method of claim 36,
Wherein the content playback system is a computer system coupled to a display. 37. The method of claim 36,
Receiving a first image corresponding to the second portion of the environment; And
Further comprising storing the first image corresponding to the second portion of the environment. ≪ Desc / Clms Page number 22 > 40. The method of claim 39,
Wherein the first image of the second portion of the environment corresponds to a first viewpoint,
Wherein generating one or more output images corresponding to the current view comprises combining content obtained from the first content stream captured at a second viewpoint with the first image corresponding to the first viewpoint, Wherein the first and second views are different. 41. The method of claim 40,
Wherein the first time point corresponds to a time preceding the second time point. 42. The method of claim 41,
Wherein the first viewpoint precedes the time of a real-time event in which images included in the first content stream are captured. 41. The method of claim 40,
Further comprising receiving one or more additional images corresponding to the second portion of the environment,
Wherein the one or more additional images corresponding to the second portion of the environment include at least one second image. 44. The method of claim 43,
Receiving control information indicating which of a plurality of previously delivered images corresponding to the second portion of the environment should be displayed during a playback time measured relative to playback times indicated in the first content stream, ≪ / RTI > 45. The method of claim 44,
Wherein the second portion of the environment is one of a first rearview portion, a second rearview portion, a sky view portion, or a ground view portion. 46. The method of claim 45,
The method further comprising receiving one or more images corresponding to a third portion of the environment. 47. The method of claim 46,
Wherein the first portion of the environment is a front view portion,
The third portion being one of a sky view or a ground view portion,
Images are received at different rates corresponding to the first, second, and third portions, and more images are received for events corresponding to the first portion than for the second portion A method for operating a system. 49. The method of claim 47,
Wherein the content corresponding to the first portion comprises real-time content that is captured and streamed to the playback device during an event progress,
Wherein the content corresponding to the images corresponding to the second and third portions is non real-time images. 37. The method of claim 36,
Further comprising receiving image selection information indicating which of a plurality of images corresponding to the second portion of the environment should be used during a portion of the event,
Wherein generating at least one output images corresponding to the current view based on at least some received content comprises selecting an image corresponding to the second portion of the environment based on the received image selection information And operating the content playback system. 50. The method of claim 49,
Determining that an image is not available for a portion of the current view;
Synthesizing an image to be used for the portion of the current field of view where no image is available; And
Further comprising combining the combined image with at least a portion of the received image to produce an image corresponding to the current view. ≪ Desc / Clms Page number 21 > 37. The method of claim 36,
Wherein the first image content is a stereoscopic image content comprising a left eye image and a right eye image. In the content playback system,
A viewer head positioning module configured to determine a head position of a viewer, the head position corresponding to a current field of view;
A content stream receiving module configured to receive a first content stream providing content corresponding to a first portion of the environment;
At least some received content included in the first content stream, and i) stored content corresponding to a second portion of the environment, or ii) corresponding to the current view based on a composite image simulating a second portion of the environment An output image content stream based generation module configured to generate one or more output images; And
An output module configured to output a first output image or a display module configured to display the first output image, wherein the output module is one or more generated output images. 53. The method of claim 52,
Wherein the content reproduction system is a content reproduction device. 53. The method of claim 52,
Wherein the content playback system is a computer system coupled to a display. 53. The method of claim 52,
An image receiving module configured to receive a first image corresponding to the second portion of the environment; And
Further comprising a received image storage module configured to store the first image corresponding to the second portion of the environment. 56. The method of claim 55,
Wherein the first image of the second portion of the environment corresponds to a first viewpoint,
Wherein the output image content stream based generation module is configured to combine the content obtained from the first content stream captured at the second time point with the first image corresponding to the first time point, A different, content playback system. 57. The method of claim 56,
Wherein the first time point corresponds to a time preceding the second time point. 58. The method of claim 57,
Wherein the first viewpoint precedes the time of the real-time event in which the images contained in the first content stream are captured. 57. The method of claim 56,
Wherein the image receiving module is further configured to receive one or more additional images corresponding to the second portion of the environment and wherein the one or more additional images corresponding to the second portion of the environment include at least one second image The content playback system comprising: 60. The method of claim 59,
Control information configured to receive control information indicating which of a plurality of previously delivered images corresponding to the second portion of the environment should be displayed for a playback time measured relative to the playback time indicated in the first content stream Further comprising a module. 64. The method of claim 60,
Wherein the second portion of the environment is one of a first rearview portion, a second rearview portion, a sky view portion, or a ground view portion. 62. The method of claim 61,
Wherein the image receiving module is further configured to receive one or more images corresponding to a third portion of the environment. 63. The method of claim 62,
Wherein the first portion of the environment is a front view portion
The third portion being one of a sky view or a ground view portion,
Images are received at different rates corresponding to the first, second, and third portions, and more images are received for events corresponding to the first portion than for the second portion system. 64. The method of claim 63,
Wherein the content corresponding to the first portion includes real-time content captured during streaming and streamed to the playback device,
And wherein the content corresponding to the images corresponding to the second and third portions is non real-time images. 65. The method of claim 64,
The control information receiving module is further configured to receive image selection information indicating which of a plurality of images corresponding to the second portion of the environment should be used during a portion of the event,
Wherein the output image content stream based generation module is configured to select an image corresponding to the second portion of the environment based on the received image selection information as a portion that is configured to generate one or more output images corresponding to the current field of view Lt; / RTI > 66. The method of claim 65,
A missing portion determination module configured to determine that an image is not available for a portion of the field of view;
An image synthesizer module configured to synthesize an image to be used for the portion of the field of view where an image is not available; And
Further comprising a synthesized image integration module configured to combine the synthesized image with at least a portion of the received image to produce an image corresponding to the current view. 53. The method of claim 52,
Wherein the first image content is a stereoscopic image content including a left eye image and a right eye image. 17. A non-transitory machine-readable medium comprising processor executable instructions,
Wherein the processor executable instructions, when executed as a processor of a content playback system,
Determining a head position of a viewer corresponding to a current visual field;
Receiving a first content stream that provides content corresponding to a first portion of the environment;
At least some received content included in the first content stream and i) stored content corresponding to a second portion of the environment or ii) a second portion corresponding to the current view based on a composite image simulating a second portion of the environment Generating one or more output images; And
To output or display a first output image that is one of the one or more generated output images.

Images