TW202536802A - A tile selection method for tile-based 360-degree video streaming and computer-readable storage media - Google Patents

Abstract

A tile selection method for tile-based 360-degree video streaming and computer- readable storage media is provided, the method includes cutting a 360-Degree video into a plurality of tiles, and each tile encoded to a plurality of qualities; obtaining and storing importance of each tile; estimating a current bandwidth used for tile transmission according to past bandwidth record, bandwidth variation range and player buffer length; and obtaining a final tile quality combination according to a tile quality selection algorithm, outputting and storing the tiles in the final tile quality combination to a player buffer.

Description

A method for selecting tiles in a tile-based panoramic video streaming system and a computer-readable storage medium.

This invention relates to tile selection based on tile panoramic video streaming, and more particularly to a method for tile selection based on tile panoramic video streaming and a computer-readable storage medium.

With the development of 5G networks and AI artificial intelligence, the application of immersive technology combined with VR virtual reality is booming, subverting people's traditional thinking about watching videos. In immersive applications, people can use head-mounted devices (HMDs) to freely choose the viewing angle and experience a 360-degree surround interactive virtual experience.

Currently, the industry mostly uses the full-view transmission method when transmitting 360-degree panoramic video streaming content. This method transmits the entire 360-degree panoramic image to the user's terminal. The terminal decodes the content from all views and presents the corresponding image content according to the current view. The advantage of this method is low latency; when the user changes viewpoints, the video frame can be switched immediately. However, the disadvantage is that it easily leads to a waste of network bandwidth and terminal computing resources, as a large amount of bandwidth and computing power is required for images that are not being played.

To address the issue of bandwidth waste, the Moving Picture Experts Group (MPEG) proposed the OMAF (Omnidirectional Media Application Format) standard in 2015, which introduced the FOV (Field-of-View) transmission method to reduce data transmission volume. The so-called FOV transmission method means that the video server only transmits the content from the perspective of the user's current viewing angle. Considering that when users view panoramic images through terminal devices, their field of vision often only falls on a small part of the panoramic image, OMAF defines a tile-based encoding specification based on this characteristic: the panoramic image is cut into multiple modules that can be independently encoded into multiple resolutions. According to the user's viewing behavior, the appropriate module to be transmitted is dynamically adjusted (the image within the viewpoint is transmitted in high resolution, and the image outside the viewpoint is transmitted in low resolution). The terminal receives the mixed image quality and then decodes and displays the image. The advantage of this method is that it can significantly reduce bandwidth and improve transmission performance. The disadvantage is that when the viewpoint changes, it will take extra time to re-download the corresponding high-definition video after the viewpoint change, which will cause the image quality to change from bad to good or the image to freeze, thus causing dizziness and discomfort to the user.

To ensure a good viewing experience for users of 360-degree panoramic videos, MTP (Motion To Photon) latency is usually used as the evaluation standard. MTP refers to the time delay between when the user moves their viewpoint and when the image is displayed on the screen. The shorter the MTP latency, the better the user's immersive experience in the VR device; the longer the MTP latency, the more intense the user's discomfort/dizziness may be. Therefore, when watching panoramic images, once the user changes their viewpoint, they will see a low-quality image for a short time before switching to a high-quality image. However, if the switching time is too long, the user may switch to another viewpoint or even give up watching, which will seriously affect the user's viewing experience.

To reduce MTP latency, viewpoint prediction is an effective approach. When a user views a panoramic image and moves their viewpoint, the terminal needs to obtain the user's current viewpoint before the server downloads the corresponding audio/video module for playback. However, each download involves requesting, downloading, decoding, and playback, all of which require additional processing time, increasing MTP latency. By combining viewpoint prediction with the ability to predict user viewpoint movement with a high probability, and pre-downloading potentially playable audio/video modules and storing them in the terminal's buffer, the terminal can directly access the buffered content when the user switches to that viewpoint, reducing MTP latency and ensuring viewing quality.

Although the FOV (Field of View) transmission method for panoramic video, combined with view prediction, can pre-download the predicted combination of high and low quality tiles to try to reduce MTP latency and maintain viewing quality, this approach still faces practical challenges. For example, if the network quality is unstable and the predetermined combination of tiles cannot be downloaded completely when pre-downloading the corresponding tile combination based on the view prediction results, some tiles may not be downloaded completely, resulting in a black screen in the user's head-mounted display field of view, thus affecting the user experience.

This invention provides a method for selecting tiles based on tile-based panoramic video streaming and a computer-readable storage medium, which can reduce MTP latency and ensure that there will be no black screen due to incorrect prediction of the user's viewing angle, thus guaranteeing the quality of panoramic video viewing.

The present invention provides a tile selection method based on tile-based panoramic video streaming, comprising: cutting the panoramic video into multiple tiles, each tile being coded into multiple qualities; acquiring and storing the importance of each tile; estimating the current bandwidth used for tile transmission based on past bandwidth records, bandwidth variation range, and terminal video storage length; and acquiring the final tile quality combination according to a tile quality selection algorithm, outputting and storing the tiles in the final tile quality combination in the terminal video storage area.

The present invention provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store computer programs, and when the computer programs are executed, methods such as the tile selection method based on tile panoramic video streaming are executed.

Based on the above, this invention provides a method for selecting tiles based on tile-based panoramic video streaming and a computer-readable storage medium. It can select the optimal combination of tile quality indicators for user experience based on the estimated bandwidth currently available for tile transmission and the predicted view angle of the head-mounted display. This combination is pre-downloaded to the terminal device's video buffer. Subsequently, when the user switches to that view angle, the terminal device can directly access and play the buffered content in the terminal video buffer, reducing MTP latency and ensuring that there are no black screens due to incorrect prediction of the user's head-turning view angle, thus guaranteeing the quality of the panoramic video viewing experience.

To make the above features and advantages of this invention more apparent and understandable, specific examples are given below, and detailed explanations are provided in conjunction with the accompanying drawings.

Some embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Component symbols used in the following description are considered to be the same or similar components when the same component symbol appears in different accompanying drawings. These embodiments are only a part of the present invention and do not disclose all possible embodiments of the present invention.

The use of ordinal numbers such as "first" and "second" in the specification and scope of the invention application to modify components does not imply or represent any prior ordinal number for that component, nor does it represent the order of one component with another, or the order of manufacturing methods. The use of these ordinal numbers is solely for the purpose of clearly distinguishing one component with a given name from another component with the same name. The invention application and the specification may not use the same terminology; therefore, a first component in the specification may be a second component in the scope of the invention application. It should be understood that the following embodiments can be modified by substituting, recombining, or mixing technical features in several different embodiments to complete other embodiments without departing from the spirit of this disclosure.

Figure 1 is a flowchart of a tile selection method based on a tile panoramic video stream according to an embodiment of the present invention.

The tile selection method disclosed herein for tile-based panoramic video streaming selects the optimal combination of tile quality indicators based on the estimated bandwidth currently available for tile transmission and the predicted field of view of the head-mounted display. This combination is pre-downloaded to the terminal device's video buffer. Subsequently, when the user switches to that viewpoint, the terminal device can directly access and play the cached content in the terminal video buffer.

Figure 2 is a flowchart of obtaining the final combination of image quality based on the image quality selection algorithm.

Referring to Figures 1 and 2, in step S101, the panoramic video is cut into multiple tiles, and each tile is encoded into multiple qualities, such as high quality and low quality. This invention is not limited to this.

In step S102, the importance of each tile (i.e. the probability or likelihood of it being seen by the user) is obtained and stored according to the importance formula.

In step S103, the bandwidth currently used for tile transmission is estimated based on past bandwidth records, bandwidth variation range, and terminal video storage length.

In step S104, the final tile quality combination is obtained based on the tile quality selection algorithm. The tiles in the final tile quality combination are output and stored in the terminal video storage area to ensure that there will be no black screen due to incorrect prediction of the user's head turning angle. At the same time, the quality of high-importance tiles is improved based on three user experience indicators (i.e. average tile quality, spatial quality difference, and temporal quality difference), ultimately achieving the goal of saving bandwidth and maintaining user experience.

Please refer to Figure 2 for a detailed flowchart of step S104. Step S104 includes steps S1041 to S1045.

Specifically, in step S1041, multiple low-quality tiles are selected to form an initial quality combination.

In step S1042, construct multiple new first quality combinations with the same total number of tiles as the initial quality combination. Adjust the quality of the first tile in the initial quality combination from low to high, and combine all tiles except the first tile with the first tile to form a new first quality combination. Adjust the quality of the second tile in the initial quality combination from low to high, and combine all tiles except the second tile with the second tile to form a new first quality combination. The first tile and the second tile are different tiles from the initial quality combination.

In step S1043, user experience indicators of the initial quality combination and the first new quality combination are obtained, and the quality combination with the optimized user experience indicators is selected from the initial quality combination and the first new quality combination as the updated first quality combination.

In step S1044, it is determined whether changing the quality of any tile in the initial quality combination from low to high quality will cause the bandwidth required for tile transmission to exceed the current bandwidth used for tile transmission, or whether changing the quality of any tile in the initial quality combination from low to high quality will cause the user experience index of the first new quality combination to decrease.

If changing the quality of any tile in the initial quality combination from low to high would cause the bandwidth required for tile transmission to exceed the current bandwidth used for tile transmission, or if changing the quality of any tile in the initial quality combination from low to high would cause the user experience index of the first new quality combination to decrease, then in step S1045, the updated first quality combination is used as the final tile quality combination, and the tiles in the final tile quality combination are output and stored in the terminal video buffer.

If changing the quality of any tile in the initial quality combination from low to high will not cause the bandwidth required to transmit the tile to exceed the current bandwidth used for tile transmission, or if changing the quality of any tile in the initial quality combination from low to high will not cause a decrease in the user experience index of the first new quality combination, then return to step S1042.

In one embodiment, user experience metrics may include average tile quality, spatial quality variation, and temporal quality variation. Average tile quality may be the average tile quality calculated by weighting tile importance, and the quality metric is represented using peak signal-to-noise ratio (PSNR). Spatial quality variation may be the standard deviation of tile quality calculated by weighting tile importance. Temporal quality variation is the magnitude of the difference between the current average tile quality and the average tile quality of the previous time period. By combining average tile quality, spatial quality variation, and temporal quality variation, a user experience index can be obtained. Average tile quality, spatial quality variation, and temporal quality variation must be optimized simultaneously, as they can affect each other (for example, reducing spatial quality variation may lead to a decrease in average tile quality).

The method described in the embodiments of Figures 1 and 2 of this invention will be explained below with reference to Figures 3 to 8. The various processes of this method may be adjusted according to the implementation situation, and are not limited thereto.

Figure 3 is a schematic diagram of the center of the head-mounted display's field of view and the horizontal line predicted by a user's head-turning angle prediction model at a certain moment according to an embodiment of the present invention. Figure 4 is a schematic diagram of the great circle distance according to an embodiment of the present invention.

Referring to Figures 3 and 4, in one embodiment, since the tile 301 closer to the center Q of the field of view 302 of the head-mounted display and the horizontal line 303 is more likely to be viewed by the user, the curvature difference between the tile center P of tile 301 and the horizontal line 303 is... The smaller the distance between the center P of the tile and the center Q of the field of view of the head-mounted display 302, the greater the great circle distance. Smaller tiles are more important, and more important tiles are more likely to be presented in high quality, thus improving the user experience.

As shown in Figure 3, the panoramic video 30 is divided into 18 tiles, each 6 pixels wide and 3 pixels high. Each tile can be set to high quality or low quality. The high-quality tile consumes 2 Mbps of bandwidth and has a PSNR of 50; the low-quality tile consumes 1 Mbps of bandwidth and has a PSNR of 40. In this embodiment, the center Q of the field of view 302 of the head-mounted display, predicted by a user's head-turning angle prediction model at a certain moment, is shown as the position indicated by the asterisk. The field of view 302 of the head-mounted display covers 4 tiles. The steps of the method described in Figures 1 to 8 are explained in practice based on the above conditions.

In step S102, specifically, the importance of each tile is calculated based on the great circle distance between the center 3011 of each tile and the field of view center of the head-mounted display, the difference in curvature between the center of each tile and the horizontal line, and the importance formula, wherein the importance formula is: …Formula 1

in This is the difference in radians between the center of the tile and the horizontal line (i.e., the latitude of the tile center on the sphere). Let be the great circle distance between the center of the tile and the center of the field of view of the head-mounted display (the radius of sphere 40 is 1). The great circle distance is the length of the shortest path (from P to Q in Figure 4) from one point on the sphere to another point on the sphere. This represents the importance of the tiles. In this embodiment, there are a total of 18 tiles, therefore... (The tile numbering order is shown in Table 1. Figure 3 shows the tile numbering order.) ).

Table 1: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18

In this embodiment, the tiles in Figure 3 are... , As shown in Tables 2 and 3 below, the tiles can be... , The importance of each tile in Figure 3 can be calculated by plugging in Formula 1. (The results are shown in Table 4 below).

Table 2: Each tile 0.73 0.73 0.73 0.73 0.73 0.73 0 0 0 0 0 0 0.73 0.73 0.73 0.73 0.73 0.73

Table 3: Each tile 2.517 2.407 1.652 1.036 1.115 1.793 2.660 2.487 1.477 0.482 0.655 1.664 2.106 2.027 1.349 0.624 0.735 1.489

Table 4: Importance of Each Piece 0.334 0.349 0.509 0.812 0.754 0.469 0.591 0.632 1.063 3.262 2.398 0.944 0.399 0.415 0.624 1.348 1.145 0.565

Of the four tile areas covered by the head-mounted display's field of view (as indicated by the dashed boxes in Table 4), the tile in the upper left corner of the dashed box has the highest importance (3.262) because it is closest to the horizontal line 303 and the center Q of the head-mounted display's field of view 302. Similarly, the tile in the upper right corner of the dashed box is the second most important (importance 2.398), followed by the tile in the lower left corner (importance 1.348) and the tile in the lower right corner (importance 1.145). The importance of each tile will be used in the calculation of user experience indicators in step S104.

In step S103, specifically, the current bandwidth used for tile transmission is obtained based on past bandwidth records, bandwidth variation range, terminal video temporary storage length, and Formula 2, where Formula 2 is: …Formula 2

in To estimate network bandwidth, To extend the video's length on the terminal, To temporarily store the target video length on the terminal, For the length of a single tile video, It is an indicator of network stability.

Obtain network stability indicators according to Formula 3. Formula 3 is: …Formula 3

in For the past The average bandwidth recorded over a period of time. For the past Standard deviation of bandwidth records over a period of time. A network stability indicator. The higher the value, the more stable the network.

In this embodiment, it is possible to set Second, Second, , , Then, according to Formula 3, the result can be calculated. According to Formula 2, the following can be calculated: This means that all tiles must not use a total bandwidth exceeding 22 Mbps.

In step S104, specifically, the average tile quality is obtained according to Formula 4, and the spatial quality difference is obtained according to Formula 5. …Formula 4 …Formula 5

in For average tile quality, Due to differences in space quality, Number the blocks In quality number Peak signal-to-noise ratio (PSNR) at that time. Quality number to indicate the importance of tile i A larger value indicates a higher quality tile. In this embodiment, the tile can be set to high quality or low quality, i.e. ; Low quality; For high quality.

According to Formula 6, the difference in quality over time is obtained. …Formula 6

in Due to time and quality differences, The average tile quality from the previous time. This represents the average tile quality.

User experience indicators are obtained based on average tile quality, spatial quality variation, temporal quality variation, and Formula 7. …Formula 7

in, As a user experience indicator, and These represent the weights of spatial quality differences and temporal quality differences in user experience indicators.

The following, in conjunction with Figures 5 to 8, will specifically explain the first to fourth rounds of tile quality selection in step S104.

Figure 5 is a schematic diagram of the first round of tile quality selection operation according to an embodiment of the present invention.

Please refer to Figure 5. In the first round of tile quality selection, you can set the initial settings. , If step S1041 should generate 18 new quality combinations, in this embodiment, it is simplified to an initial quality combination 501 consisting of 4 tiles covered by the head-mounted display's field of view 302, while the other 14 uncovered tiles are all of low quality and consume 1 Mbps of bandwidth. The quality of the 4 tiles in the initial quality combination 501 is sequentially increased from low to high quality, thus constructing or generating 4 first new quality combinations 502. Each of these 4 first new quality combinations 502 differs slightly from the quality of one high-quality tile in the initial quality combination 501. The user experience indicators for each tile in these 4 first new quality combinations 502 and the initial quality combination 501 are calculated.

The top-left corner of the head-mounted display's field of view (the first new quality combination 503 within the dashed box in Figure 5) changes from low quality to high quality, and the user experience index generated by the first new quality combination 503 within the dashed box is the highest. At this point, the importance values of each tile as shown in Table 4 above are considered. and the quality of the tiles as shown in Table 5 .

Table 5: PSNR of each tile quality in the first round 40 40 40 40 40 40 40 40 40 50 40 40 40 40 40 40 40 40

The average tile quality of the first new quality combination 503 within the dashed box can be obtained using Formula 4. The difference in spatial quality can be calculated using formula 5. The time-quality difference can be calculated using formula 6. User experience metrics can be calculated using Formula 7. .

Since the first new quality combination 503 within the dashed frame contains one high-quality tile using 2Mbps bandwidth and 17 low-quality tiles using 1Mbps bandwidth (as shown in Table 6), the total bandwidth used by the first new quality combination 503 within the dashed frame is 2*1 + 1*17 = 19Mbps. Since 19Mbps is still less than the bandwidth available for tile transmission calculated in step S103 (…),… In other words, in the first round of tile quality selection, setting the quality of any tile in the initial quality combination to high quality will not cause the bandwidth required for tile transmission to exceed the current bandwidth used for tile transmission. Therefore, the updated first new quality combination 504 selected in the first round of tile quality selection can be used as the initial quality combination 601 for the second round of tile quality selection.

Table 6: Bandwidth used by each tile in the first round (unit: Mbps) 1 1 1 1 1 1 1 1 1 2 1 1 1 1 1 1 1 1

Figure 6 is a schematic diagram of the second round of tile quality selection operation according to an embodiment of the present invention.

Please refer to Figure 6. In the second round of tile quality selection, the three low-quality tiles in the initial quality combination 601 are upgraded to high quality in turn. As a result, three first new quality combinations 602 will be constructed or generated respectively. Each of these three first new quality combinations 602 has a slight difference in quality from the tiles in the initial quality combination 601, which is one high-quality tile.

The top right corner of the head-mounted display's field of view (the first new quality combination 603 within the dashed box in Figure 6) changes from low quality to high quality, and the user experience index generated by the first new quality combination 603 within the dashed box is the highest. At this point, the importance values of each tile as shown in Table 4 above are considered. and the quality of the tiles as shown in Table 7 .

Table 7: PSNR of each tile quality in the second round 40 40 40 40 40 40 40 40 40 50 50 40 40 40 40 40 40 40

The average tile quality of the first new quality combination 603 within the dashed box can be obtained using Formula 4. The difference in spatial quality can be calculated using formula 5. The time-quality difference can be calculated using formula 6. User experience metrics can be calculated using Formula 7. .

Since the first new quality combination 603 within the dashed frame contains 2 high-quality tiles using 2Mbps bandwidth and 16 low-quality tiles using 1Mbps bandwidth (as shown in Table 8), the total bandwidth used by the first new quality combination 603 within the dashed frame is 2*2 + 1*16 = 20Mbps. This is still smaller than the bandwidth available for tile transmission calculated in step S103. In other words, during the second round of tile quality selection, setting the quality of any tile in the initial quality combination to high quality will not cause the bandwidth required for tile transmission to exceed the current bandwidth used for tile transmission, and user experience indicators will remain unchanged. Compared to the first round of improvements ( Therefore, the updated first new quality combination 604 selected in the second round of tile quality selection can be used as the initial quality combination 701 in the third round of tile quality selection.

Table 8: Bandwidth used by each tile in the second round (unit: Mbps) 1 1 1 1 1 1 1 1 1 2 2 1 1 1 1 1 1 1

Figure 7 is a schematic diagram of the third round of tile quality selection operation according to an embodiment of the present invention.

Please refer to Figure 7. In the third round of tile quality selection, the two low-quality tiles in the initial quality combination 701 are upgraded to high quality in turn. As a result, two first new quality combinations 702 will be constructed or generated respectively. The quality of these two first new quality combinations 702 is slightly different from that of the high-quality tiles in the initial quality combination 701.

The lower left corner of the head-mounted display's field of view (the first new quality combination 703 within the dashed box in Figure 7) changes from low quality to high quality, and the user experience index generated by the first new quality combination 703 within the dashed box is the highest. At this point, the importance values of each tile as shown in Table 4 above are considered. and the quality of the tiles as shown in Table 9 .

Table 9: PSNR of each tile quality in the third round 40 40 40 40 40 40 40 40 40 50 50 40 40 40 40 50 40 40

The average tile quality of the first new quality combination 703 within the dashed box can be obtained using Formula 4. The difference in spatial quality can be calculated using formula 5. The time-quality difference can be calculated using formula 6. User experience metrics can be calculated using Formula 7. .

Since the first new quality combination 703 within the dashed frame contains 3 high-quality tiles using 2Mbps bandwidth and 15 low-quality tiles using 1Mbps bandwidth (as shown in Table 10), the total bandwidth used by the first new quality combination 703 within the dashed frame is 2*3 + 1*15 = 21Mbps. This is still less than the bandwidth available for tile transmission calculated in step S103. In other words, during the third round of tile quality selection, setting the quality of any tile in the initial quality combination to high quality will not cause the bandwidth required for tile transmission to exceed the current bandwidth used for tile transmission, and user experience indicators will remain unchanged. Compared to the second round of improvements ( Therefore, the updated first new quality combination 704 selected in the third round of tile quality selection can be used as the initial quality combination 801 in the fourth round of tile quality selection.

Table 10: Bandwidth used by each tile in the third round (unit: Mbps) 1 1 1 1 1 1 1 1 1 2 2 1 1 1 1 2 1 1

Figure 8 is a schematic diagram of the fourth round of tile quality selection operation according to an embodiment of the present invention.

Please refer to Figure 8. In the fourth round of tile quality selection, one low-quality tile in the initial quality combination 801 is upgraded to high quality. This will construct or generate a first new quality combination 802. The quality of this first new quality combination 802 is slightly different from that of the high-quality tile in the initial quality combination 801.

The tile in the lower right corner of the head-mounted display's field of view (the first new quality combination 803 within the dashed box in Figure 8) changes from low quality to high quality, and the user experience index generated by the first new quality combination 803 within the dashed box is the highest. At this point, the importance values of each tile as shown in Table 4 above are considered. and the quality of the tiles as shown in Table 11 .

Table 11: PSNR of each tile quality in the fourth round 40 40 40 40 40 40 40 40 40 50 50 40 40 40 40 50 50 40

The average tile quality of the first new quality combination 803 within the dashed box can be obtained using Formula 4. The difference in spatial quality can be calculated using formula 5. The time-quality difference can be calculated using formula 6. User experience metrics can be calculated using Formula 7. .

Since the first new quality combination 803 within the dashed box contains 4 high-quality tiles using 2Mbps bandwidth and 14 low-quality tiles using 1Mbps bandwidth (as shown in Table XII), the total bandwidth used by the first new quality combination 803 within the dashed box is 2*4 + 1*14 = 22Mbps. Although user experience indicators... Compared to the third round of improvements ( However, since the bandwidth available for tile transmission has been exhausted ( Therefore, the judgment condition in step S1044 is no longer met, so the tile selection operation is stopped and the updated first quality combination 804 is used as the final tile quality combination. The tiles in the updated first quality combination 804 are output and stored in the terminal video temporary storage area.

Table 12: Bandwidth used by each tile in the fourth round (unit: Mbps) 1 1 1 1 1 1 1 1 1 2 2 1 1 1 1 2 2 1

This application embodiment also provides a computer-readable storage medium storing computer program code, wherein when the processor executes the computer program code, the electronic device executes the method in any of the foregoing embodiments.

Those skilled in the art will appreciate that, in one or more of the examples above, the functions described in this application can be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium. Computer-readable media include computer-readable storage media and communication media, wherein communication media include any medium that facilitates the transmission of computer programs from one location to another. Storage media can be any available medium accessible to general-purpose or special-purpose computers.

Based on the above, this invention provides a method for selecting tiles based on tile-based panoramic video streaming and a computer-readable storage medium. It can select the optimal combination of tile quality indicators for user experience based on the estimated bandwidth currently available for tile transmission and the predicted view angle of the head-mounted display. This combination is pre-downloaded to the terminal device's video buffer. Subsequently, when the user switches to that view angle, the terminal device can directly access and play the buffered content in the terminal video buffer, reducing MTP latency and ensuring that there are no black screens due to incorrect prediction of the user's head-turning view angle, thus guaranteeing the quality of the panoramic video viewing experience.

Although the present invention has been disclosed above by way of embodiments, it is not intended to limit the present invention. Anyone with ordinary skill in the art may make some modifications and refinements without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended patent application.

30: Panoramic Video 301: Tile 302: Head-Mounted Display Field of View 303: Horizon Line 3011: Tile Center 40: Sphere 501, 601, 701, 801: Initial Quality Combination 502, 503, 602, 603, 702, 703, 802, 803: First New Quality Combination 504, 604, 704, 804: Updated First New Quality Combination S101, S102, S103, S104, S1041, S1042, S1043, S1044, S1045: Steps

Figure 1 is a flowchart of a tile selection method based on a tile panoramic video stream according to an embodiment of the present invention. Figure 2 is a flowchart of obtaining the final tile quality combination according to a tile quality selection algorithm. Figure 3 is a schematic diagram of the center of the head-mounted display's field of view and the horizontal line predicted by a user's head-turning view angle prediction model at a certain moment according to an embodiment of the present invention. Figure 4 is a schematic diagram of the great circle distance according to an embodiment of the present invention. Figure 5 is a schematic diagram of the first round of tile quality selection operation according to an embodiment of the present invention. Figure 6 is a schematic diagram of the second round of tile quality selection operation according to an embodiment of the present invention. Figure 7 is a schematic diagram of the third round of tile quality selection operation according to an embodiment of the present invention. Figure 8 is a schematic diagram of the fourth round of tile quality selection according to an embodiment of the present invention.

S101, S102, S103, S104: Steps

Claims (6)

A method for tile selection based on tile-based panoramic video streaming includes: segmenting the panoramic video into multiple tiles, and encoding each tile into multiple qualities; acquiring and storing the importance of each tile; estimating the bandwidth currently used for transmitting the tiles based on past bandwidth records, bandwidth variation, and terminal video storage length; and obtaining a final tile quality combination according to a tile quality selection algorithm, outputting and storing the tiles in the final tile quality combination in the terminal video storage area. The tile selection method as described in claim 1, wherein the step of acquiring and storing the importance of each tile includes: acquiring the importance of each tile based on the great circle distance between the center of each tile and the center of the field of view of the head-mounted display, and the difference in curvature between the center of each tile and the horizontal line. The tile selection method as described in claim 1, wherein the step of estimating the current bandwidth used for tile transmission based on the past bandwidth records, the bandwidth variation range, and the terminal video storage length includes: Obtaining a network stability index based on the average value of bandwidth records within past time intervals and the standard deviation of bandwidth records within the past time intervals; Obtaining the current bandwidth used for tile transmission based on the past bandwidth records, the bandwidth variation range, the terminal video storage length, the terminal target video storage length, the length of a single tile video, and the network stability index. The tile selection method as described in claim 2, wherein the step of obtaining the final tile quality combination according to the tile quality selection algorithm, outputting and storing the tiles in the final tile quality combination to the terminal video temporary storage area includes: Selecting multiple low-quality tiles from the selected tiles to form an initial quality combination; Construct multiple new first quality combinations with the same total number of tiles as the initial quality combination. Adjust the quality of the first tile in the initial quality combination from low to high quality, and combine the tiles from these new combinations (excluding the first tile) with the first tile to form the new first quality combination. Adjust the quality of the second tile in the initial quality combination from low to high quality, and combine the tiles from these new combinations (excluding the second tile) with the second tile to form the new first quality combination. The first tile and the second tile are different tiles from the initial quality combination. Obtain user experience metrics for the initial quality mix and the first new quality mixes; select the quality mix with optimized user experience metrics from the initial quality mix and the first new quality mixes to form the updated first quality mix; and If upgrading the quality of any tile in the initial quality combination to high quality would cause the bandwidth required to transmit those tiles to exceed the current bandwidth used for tile transmission, or if upgrading the quality of any tile in the initial quality combination from low quality to high quality would cause a decrease in the user experience index of those first new quality combinations, then the updated first quality combination is designated as the final tile quality combination, and the tiles in the final tile quality combination are output and stored in the terminal video storage area. The tile selection method as described in claim 4, wherein the step of obtaining the user experience index of the initial quality combination and the first new quality combinations, and selecting the quality combination with the optimized user experience index from the initial quality combination and the first new quality combinations as the updated first quality combination, includes: Based on Number In quality number The average tile quality is obtained by using the peak signal-to-noise ratio (PSNR) and the importance of the tile at that time. Number In this quality number The spatial quality difference is obtained based on the PSNR, the importance of the tile, and the average tile quality. The temporal quality difference is obtained based on the previous average tile quality and the average tile quality. The user experience index is obtained based on the average tile quality, the spatial quality difference, and the temporal quality difference. A computer-readable storage medium, wherein the computer-readable storage medium is used to store a computer program, which, when executed, performs the method described in any one of claims 1-5.