WO2015131935A1

WO2015131935A1 - System and method for controlling video resolution depending on an upload transfer rate

Info

Publication number: WO2015131935A1
Application number: PCT/EP2014/054236
Authority: WO
Inventors: Marcel SCHNELLE; E. Servet MUTLU
Original assignee: 2KB BETEILIGUNGS GmbH
Current assignee: 2KB BETEILIGUNGS GmbH
Priority date: 2014-03-05
Filing date: 2014-03-05
Publication date: 2015-09-11
Anticipated expiration: 2016-09-05

Abstract

A video film is generated as a sequence of film segments, wherein each segment is uploaded to a server, while a consecutive segment is recorded. A method and corresponding system for controlling the video resolution of a segment depending on an available upload rate of a communicatively coupled wireless network is described in order to enable uploading of the segments in time.

Description

System and method for controlling video resolution depending on an upload transfer rate

The invention relates to a device and a corresponding method for controlling the video resolution of a media segment of a segmented video film depending on an estimated upload rate of a communicatively coupled network. Similarly the audio resolution, i.e. the audio sampling rate of a media segment may be controlled depending on the estimated upload rate.

Since modern smartphones and so-called tablet computers not only are equipped with a camera for taking still images but also with video functionality for producing high-quality video clips and video films extending to nearly endless duration, it has become popular to upload the videos to a network server in order to publish them. A plurality of internet services is known that allow uploading of a video. Once the video is uploaded it is accessible by a plurality of users, wherein the video may be downloaded either as a stream for immediate viewing or as a file thus enabling storing of the video file for offline viewing.

Furthermore there are services that enable so-called live-streaming of videos. These services forward provided video data immediately to a viewer. In one example TV live transmissions can be received via smartphones or handheld devices that are communicatively coupled to the internet using software applications that allow reception of a TV program via a radio connection, i.e. a data connection according to a well-known telecommunications standard such as UMTS or LTE or via wireless LAN. The video camera deployed for generating the video data is typically coupled to the network exhibiting a fast and reliable upload connection, thus enabling the transmission of generated video data with a sufficient upload transmission rate. By providing a sufficiently fast upload connection to the network any transmission delay in the upload channel is anticipated.

However, when using a smartphone or tablet computer or any other device coupled via a wireless LAN or a cellular telecommunications network, e.g. a cellular telephone network, there is a problem of varying network conditions causing varying uplink transfer rates. Besides being limited by intentionally limiting the bandwidth it is known that radio connections may suffer from various effects, i.e. interference from other radio sources or weak signals due to long distance to a service base station or shadowing caused by obstacles that reduce the uplink transfer rate. Thus there is a problem of varying radio conditions when uploading a video via a radio connection.

This problem is solved by a method and corresponding device as described in more detail and with reference to the accompanying figures wherein

Fig. 1 depicts an arrangement for uploading a segmented video;

Fig. 2 depicts a sequence of media segments forming a video film.

Figure 1 depicts an arrangement 100 of an electronic device 110 coupled to a network server 130 via a radio connection 140 provided by a radio access network 160 for uploading segments of a video film. A viewing client 150 is communicatively coupled to server 130 via network 170, which is configured for downloading the video film generated by device 110.

Electronic device 110 comprises a camera means for generating video films wherein the camera means may be configured for generating video films in at least two pixel resolutions. In one embodiment the camera means may be configured for generating for a video resolution of 1080p, i.e. so-called full HD characterized by 1080 horizontal pixel lines/vertical resolution implying a horizontal resolution of 1920 pixel lines assuming a 16:9 format, or a video resolution of 720p, i.e. so-called HD characterized by 720 horizontal pixel lines/ vertical resolution implying a horizontal resolution of 1280 pixel lines in a 16:9 format, or a video resolution of 480p, i.e. 480 horizontal pixel lines/vertical resolution implying a horizontal resolution of 640 pixel lines in a 4:3 format or 854 lines in a 16:9 format. Note that the invention shall not be limited to these exemplary values, but any arbitrary video resolution may be considered. The pixel resolution of the camera means of device 110 may be controlled by a control means preferably comprised in device 110. According to the selected video resolution the generated files are of varying size. As a consequence of the varying file sizes the upload of a video film of a low resolution requires fewer resources than an upload of a higher resolution video.

Similarly the file size of a video film depends on the sampling rate of the audio information provided in the video film, i.e. a video film providing audio information at a higher sampling rate typically comes at the cost of an increased file size compared to a video film providing audio information at a lower audio sampling rate. Furthermore electronic device 110 is capable of uploading a video film via a radio connection 140, wherein the radio connection can be established via WLAN, a cellular telecommunication network, a wide area network WAN, WIMAX protocol or any other suitable radio access system, wherein the cellular communication system can be a system according to the GSM/CSD/HSCSD/GPRS or UMTS or LTE or PDC or any other system allowing data file transmission. Thus radio access system 160 may comply with any of the above exemplifying standards.

Radio access system 160 is communicatively coupled to network system 170 and in particular to a server 130 comprised in network system 170. The network system may comprise a plurality of networks. In one embodiment network system 170 may comprise the internet and a smaller network, e.g. the network of a service provider, wherein said small network comprises server 130.

Server 130 is configured and adapted for receiving, storing and enabling download of files, particularly video films or segmented video films including additional files associated with the video film or film segments as a streamable video.

As insinuated in figure 1, a user may generate a film of an arbitrary object using electronic device 110, which in one embodiment may be a smartphone comprising a camera means capable of generating a video frames. Similarly the electronic device may comprise a microphone means for recording audio samples. The electronic device further comprises a control means, e.g. a software application, enabling the method steps and control of the smartphone as described in the following.

When the user starts filming the object using the control means, i.e. the application, the means controls the pixel resolution of the camera means and produces the film as a temporal sequence of segments. Each segment typically exhibits a predefined finite duration, which in one embodiment may be around 5 seconds. However, other durations may be chosen and the described system shall not be limited by the predefined duration of the segments, though the duration of a segment typically is chosen to a value of a few seconds. A video film that lasts a couple of minutes thus is segmented into a plurality of segments called media segments.

Note that in one embodiment the control means may also control the duration of the media segments, wherein the control means not only defines the end of the last segment of a segmented video film but may also control the length of a segment before the last segment. Accordingly the control means may extend or reduce the length, i.e. the duration, of a segment that is currently generated or that is generated next depending on the radio link conditions.

Furthermore the control means, i.e. the software application, controls the pixel resolution of the camera, wherein the pixel resolution may be adapted for each segment individually. Consequently, the generated video film may consist of at least two media segments that exhibit different pixel resolutions, which optionally may be of different duration and further optionally may have different audio sampling rates.

Optionally the control means, i.e. the software application, may control the audio sampling rate of the microphone means, wherein the sampling rate may be adapted for each segment individually. Media segments of a video film accordingly may exhibit different audio sampling rates.

Figure 2 illustrates the generation of media segments 210a-210 of a video film. In one embodiment the first segment is generated using a predefined pixel resolution, i.e. a default pixel resolution, while the pixel resolution deployed for generating any consecutive segment may be calculated based on an available upload transfer rate, i.e. if a high upload transfer rate is expected, then the application may set the video resolution for this segment to a high value. Note that the available upload transfer rate may be calculated by well-known algorithms based on a previously measured signal-to-noise ratio or the availability of transmission resources allocated by the radio access network. In one embodiment electronic device 110 may determine the available upload transfer rate just before starting the segment generation, thus the device may consider the allocated upload transfer rate. In an alternative

embodiment, for example when using a WLAN network, the electronic device may listen to traffic in the WLAN network in order to estimate a potential upload transfer rate. Similarly, if the electronic device is communicatively coupled by a cellular telecommunications network, the device may request a bandwidth allocation and may determine an upload transfer rate based on the communicated allocated bandwidth. In consideration of the fact that a higher pixel resolution causes an increasing data size of the segment, the electronic device 100 may set pixel resolution that allows to reliably upload the generated segment during the generation of the following segment. Furthermore and optionally, the device may control the audio sampling of a media segment to adapt the file size of a media segment in order to enable a reliable upload. The duration of the first and each subsequently generated segment can be a predefined value, i.e. a default value. In an alternative embodiment the duration of the first and also any subsequent segment may be selected based on a determined reliability of the radio connection.

When the first segment 210a or any other segment has been completed, either because the segment has reached its duration or because the user intentionally has stopped the generation, the completed segment is uploaded from the electronic device 110 via radio link 140 and radio access network 160 to the destined server 130, wherein the upload is performed as fast as possible. As soon as a media segment of a video film has been successfully uploaded to server 130, it is checked and made available for download as fast as possible. Note that before enabling the downloading a plurality of optional checks may be performed in order to check integrity of the segment file or to prevent the unlawful publication of any prohibited content.

As soon as the first segment is provided on server 130 for downloading, a user may start to download that first segment, wherein downloading may either mean a full download or streaming the segment for instant viewing without storing the file first. In order to provide a seamless continuation in viewing the film, the next segment of that film shall be provided for downloading before the current segment ends, i.e. there is a time interval having the duration of the currently downloaded film segment for uploading and checking the second film segment. Hence, there is the problem of providing a next film segment for downloading from server 130 before the previous segment ends. So if the file size of the next segment to be uploaded extends the upload capacity of the radio link in a time interval having the duration of the previous, currently viewed segment, then the downloading user will have to wait until the upload of the segment is completed. That is the viewing user has to wait for the next segment. However, if the video resolution of the segment is low and the file size of the next segment is small, then the upload within the available time interval requires less resources but at the cost of a low pixel resolution. If the available radio link provides for a high upload transfer rate, then a larger file, i.e. a segment having a high pixel resolution, can be uploaded within the time interval. Accordingly, the size of the files containing the film segments must be adapted according to the expected available radio link.

The first media segment 210a exhibits a comparatively high pixel resolution as indicated on the y-axis by the height of the block and a duration of T as indicated on the x-axis. For illustration purposes the high resolution may be 1080p and the duration of the segment may be T₀«5 seconds.

Note that in contrast to the illustration in figure 1, there is no gap in time between the segments of the video. Instead, the segments seamlessly follow up.

While a segment is uploaded, the uplink transfer rate V of said segment is measured, wherein V_n shall denote the average uplink transfer rate of uploading the n-th media segment, wherein index n shall denote the index of a media segment. In electronic device 110 the average uplink transfer rate can be determined by evaluating the time required for uploading the file containing the n-th media segment. The related raw information can be retrieved from the hard-/software module that actually handles the upload.

Based on the measured uplink transfer rate V_n an expected upload rate μ_η+1 , i.e. an upload transfer rate that presumably is available for uploading the (n + l)-th segment, is calculated as

/½₊i = ^Vn+^^n→, with n = 1, 2, 3 . .. and μ₀ = 0 wherein μ_η--1 is the expected upload transfer rate of the (n - l)-th segment. Note that the expected upload rate for the very first segment is set to zero, i.e. μ₀ = 0. The calculated upload rate for uploading the (n + l)-th segment thus is an average of the actual upload transfer rate V_n of the previously uploaded segment and the expected rate of the (n - l)-th segment.

Based on the calculated expected upload transfer rate μ a current tendency value L is calculated according to

Ln = v_n - μ_η+ι , with L_n=1 = 0

Current tendency value L thus is the rate difference between the measured upload transfer rate V_n and the transfer rate expected for the next segment μ_η+1 . This calculated value, i.e. the difference of the measured and expected upload transfer rate thus is of unit bytes per second, and may have values greater than or less than zero.

Based on all of the current tendency values L_n as determined for uploading segments of one film a long-term tendency value is calculated as an average according to GN — _N∑n=l L_N wherein in this formula n is the running index and N denotes the number of media segments uploaded so far. Note that this averaged value may be greater than or less than zero, thus a change in the algebraic sign is possible. A change of the algebraic sign from + to - indicates a significant decline of the radio link quality, i.e. a smaller upload transfer rate is expected. Vice versa a change in the algebraic sign of long- term tendency value G_N from - to + indicates an improvement of the radio link quality, i.e. an increase in the expected uplink transfer rate.

Long-term tendency value G_N is calculated right before a new media segment begins. Based on a change in the algebraic sign of the long-term tendency value the pixel resolution of the camera means for the next segment is adapted, i.e. the pixel resolution of the camera means is set to a next higher value if the algebraic sign changes from - to + , unless the highest resolution has been set already. In this way the resolution of the camera means comprised in the electronic device is adapted, i.e. in this case increased, based on the calculated expected upload transfer rate. As a consequence the file size of the next media segment is larger than that of the previously generated segment, under the assumption of unchanged segment duration. Due to the expected higher upload transfer rate, the upload of the larger segment file is expected to be successfully completed in the time interval of the previous segment, thus enabling downloading the segment seamlessly to the previous segment.

Similarly the pixel resolution is reduced to a next lower value in case the algebraic sign of the long-term tendency changes from + to -, thus reducing the file size of next media segment, unless the lowest pixel resolution has been already set in the electronic device. The reduced file size of the media segment thus requires a lower upload transfer rate in order to enable a complete upload within the scheduled time interval.

Note that in an alternative embodiment the camera means comprised in the electronic device may provide raw video frames having the highest video resolution possible by that camera means. The video frames are buffered in a memory means comprised in the electronic device before a media segment is generated based on provided high-resolution raw video frames, but wherein the video resolution of the media segment is adapted according to a configuration of a segmenter means. Said segmenter means may be implemented by hard or software and may be configured to generate a media segment based on provided audio and video data, i.e. audio samples and video frames, wherein the segmenter means downscales the pixel resolution of the video frames and/or the audio resolution in samples when producing a media segment. Similarly raw audio samples may be provided in with high sampling rate to the segmenter. In this way the video resolution; i.e. the pixel resolution of the video frames, and/or the audio resolution, i.e. the audio sample resolution, of a media segment can be controlled by configuring the media segmenter. In this way the video resolution and/or the audio resolution of a media segment may be downscaled, i.e. reduced, by using a correspondingly configured segmenter based on raw video frames having a high pixel resolution and raw audio samples having a high sampling rate. Accordingly the camera means comprised in the electronic device may be configured by the application, i.e. above mentioned control means, to output raw video frames having the highest possible pixel resolution and the microphone means may be configured to output audio samples at the highest possible sampling rate, while the video frames and audio samples of a media segment exhibit a downscaled resolution.

So in addition to controlling the video resolution of a media segment, the audio sampling rate of the audio information comprised in a media segment may be controlled. In one embodiment this may be achieved by controlling the output audio sampling rate of the microphone means. In an alternative embodiment, the microphone means may be controlled to output audio samples at the highest possible sampling rate and the segmenter may be controlled to generate audio frames exhibiting an adapted audio sampling rate, i.e. the segmenter downscales the audio sampling rate according to a provided configuration.

The following table of exemplifying values demonstrates varying upload transfer rates and corresponding calculated values. However the invention shall not be limited by the values of the table. n /½+l

segment measured expected

index rate rate

1 2000 1000 0 0

2 4000 2500 1500 750

3 500 1500 -1000 166

4 750 1125 -375 31 5 400 762 -362 -47

6 200 481 -281 -86

7 300 390 -90 -86

8 1425 907 518 -11

9 1860 1383 477 43

10 1300 1341 -41 34

11 500 1420 80 38

12 160 1530 110 44

Note that the initial values of L_n and G_n are set to zero at the beginning of uploading a sequence of media segments. Though the measured upload transfer rate V_n sharply falls in the upload of the 3^rd segment from 4000 to 500, the algebraic sign of G_n does not switch immediately, but in the 5^th upload. Accordingly, as illustrated in the figure, the pixel resolution of the media segments remains unchanged for media segments 210a-210e and is then reduced by one step for generating the 6^th media segment. Due to the reduced pixel resolution the size of the files containing segments 6 to 9, 210f-210i, is smaller than that of segments 1-5.

The algebraic sign of the long-term tendency value G_n then switches from minus to plus in the ninth segment. Consequently the pixel resolution for the next media segment is set to the next higher value, i.e. for generating the tenth segment 210K. The size of the file containing the tenth segment accordingly is larger when compared to a file of same duration but a lower pixel resolution. Consequently a higher upload transfer rate is required for uploading that file. However, the change of the arithmetic sign indicates an expected higher upload transfer rate that enables the upload on time in order to provide the segment for downloading from server 130.

In this way the video resolution of the next media segment may be set to a higher or lower pixel resolution depending on a change of the algebraic sign of the long-term tendency value, i.e. the pixel resolution of a media segment is set to a higher value when the value of the long-term tendency changes from a negative to a positive value, i.e. the algebraic sign changes from - to +, and the pixel resolution is set to a lower value when the long-term tendency changes from a positive to a negative value, i.e. the algebraic sign changes from + to -. Note that similar to adapting the pixel resolution of a media segment, the audio sampling rate of a media segment optionally may be set to a higher or lower sampling rate. In this way the size of the file containing the next media segment to be generated is adapted to the conditions of the radio link 140 that is used for uploading the segment.

In one embodiment the duration, i.e. the time length, of video film segments may be predefined to a fixed duration. Consequently, all segments of a video film except for the last segment have the same duration. Said predefined duration typically may be set to a value between 1/2 and 60 seconds, more preferably to a value between 1 and 10 seconds, and most preferably to a value of around 5 seconds. Since the last segment of the film ends when the user stops the generation, e.g. by pressing a stop button or releasing a record button on the electronic device, that segment typically will have a duration shorter than predefined.

Optionally and in addition to changing the video and or audio resolution of the camera comprised in electronic device 110, the duration, i.e. the time length, of a segment can be adapted to the channel conditions of radio link 140 by control means in the electronic device.

Varying uplink transfer rates can be detected by monitoring the uplink transfer rates that actually occur when uploading segments. In one embodiment and as described above the uplink transfer rate can be determined from the time span required for uploading a file containing a film segment and the known size of the uploaded file which results in an averaged uplink transfer rate. Alternatively, the maximum and minimum uplink transfer rate for a predefined time span can be measured, wherein said predefined time span may be shorter than the duration of a segment.

Based on a plurality of measured uplink transfer rates, the rate of change of the uplink transfer rate can be determined. In case the rate of change heavily varies, then the replay duration of a segment may be reduced thus enabling a more flexible adaptation of the video resolution and/or audio sampling rate to the varying uplink transfer rates. If the monitoring of the uplink transfer rates indicates almost constant transfer rates, then the duration of a segment may be increased thus increasing efficiency by reducing processing overhead caused by handling many but short segments

Though a decrease in duration causes an increase in overhead data, i.e. an increase in managing the generation and uploading the film segments that in turn requires more radio resources, the adaptation of the segment duration may improve the achievable overall quality, because the video resolution may be adapted more flexible thus supporting uploading of segments on time.

If monitoring of the uplink transfer rates indicates fast and significant changes of the uplink transfer rate, the duration of a media segment may be set to a shorter duration. Conversely, if the monitoring of the uplink transfer rates indicates a stable, reliable radio link and small changes in the transfer rate, then the duration of the next segment to be generated may be increased. However, considering that said increase in duration causes a corresponding increase of the file size, the duration of the next segment must be limited in order to enable an upload on time.

The handheld electronic device in this way is adapted to control the pixel resolution and/or the audio sampling rate of a media segment depending on an expected upload transfer rate. If a low upload transfer rate is expected, the pixel resolution of the video frames and/or the audio sampling rate may be decreased by one step to reduce the file size of the media segment to be uploaded next. Vice versa, the pixel resolution of the video frames and/or the audio sampling rate may be increased by one step to allow a bigger file size of the media segment to be uploaded next.

Claims

Claims:

1. A method for controlling the video resolution of a segmented video film

comprising

- generating at least a first media segment of a temporal sequence of media segments, and

- uploading the at least first media segment to a server via a radio link and measuring the upload transfer rate while uploading, and

- determining an expected averaged upload transfer rate based at least on the measured upload transfer rate, and

- adapting the video resolution of the media segment based on the

determined expected averaged upload transfer rate, and

- generating a second media segment of the temporal sequence of media segments deploying the adapted video resolution, and

- uploading the second media segment to the server.

2. The method of any preceding claim wherein the step of adapting the video resolution of the media segment comprises

- determining a current expected upload rate based on an averaged upload transfer rate as measured for the latest upload and the expected upload rate determined for the latest upload,

- determining a current tendency value as the difference between the current expected upload rate and the upload rate measured for the latest upload, and

- determining a long term tendency value as the average of all previously determined current tendency values, and

reducing or increasing the video resolution of the media segment based on the determined averaged tendency value.

3. The method of claim 2 wherein the video resolution is reduced if the value of the determined long-term tendency changes from a positive value to a negative value.

4. The method of claim 2 wherein the video resolution is increased if the value of the determined long term resolution tendency changes from a negative to a positive value.

5. The method of any preceding claim further comprising adapting the audio sampling rate of the media segment based on the determined expected averaged upload transfer rate and generating the second media segment deploying the adapted audio sampling rate.

6. The method of any preceding claim wherein the method steps are performed in one of a cell phone or a tablet computer or a personal digital assistant.

7. The method of any preceding claim wherein the radio link is one of a WLAN or a cellular telecommunications link.

8. The method of any preceding claim further comprising the step of

monitoring the speed of change of the upload transfer rate and increasing or decreasing the duration of a media segment depending on the monitored speed of change of the upload transfer rate.

9. An electronic device comprising a camera device and a radio interface, said device being adapted for

- adapting the video resolution of the media segment based on the

determined expected averaged upload transfer rate, and

- generating a second media segment on the temporal sequence of video segments deploying the adapted camera device, and

- uploading the second media segment to the server.

10. The electronic device of claim 8 wherein the stop of adapting the video

resolution of the media segment comprises

- determining a current expected upload rate based on a n averaged upload transfer rate as measured for the latest upload and the expected upload rate determined for the latest upload, and

- determining a current tendency value as the difference between the current expected upload rate and the upload rate measured for the latest upload, and - determining a long term tendency value as the average of all previously determined current tendency values, and

- reducing or increasing the video resolution of the media segment based on the determined averaged tendency value.

11. The electronic device of any claim 8-9 wherein the video resolution is reduced if the value of the determined long term tendency changes from a positive to a negative value.

12. The electronic device of any claim 8-10 wherein the video resolution is

increased if the value of the determined long term resolution tendency changes from a negative to a positive value.

13. The electronic device of any claim 8-12 wherein the device is furthermore adapted for adapting the audio sampling rate of the media segment based on the determined expected averaged upload transfer rate and generating the second media segment deploying the adapted audio sampling rate.

14. The electronic device of any claim 8-11 wherein the method steps are

performed in one of a cell phone or a tablet computer or a personal digital assistant.

15. The electronic device of any claim 8-12 wherein the radio link is one of a WLAN or a cellular communications link.

16. The electronic device of any claim 8-13 wherein the device is further adapted to monitor the speed of change of the upload transfer rate and increasing or decreasing the duration of a media segment depending on the monitored speed of change of the upload transfer rate.