WO2015180853A1 - Device and method for displaying and editing videos by means of a mobile telephone - Google Patents

Abstract

The invention relates to a method for displaying and editing video files on a mobile telephone, wherein the function of the individual components of the mobile telephone is controlled by an operating system, and wherein the mobile telephone comprises a central processing unit (CPU) as a computing unit and a graphics processing unit as a further computing unit and additionally comprises a memory system, wherein the central processing unit is connected to the graphics processing unit in such a way that data exchange can occur. The method comprises the following steps: decoding the video data by means of one of the two computing units, editing the decoded video data by means of the other of the two computing units, displaying the video data by means of the aforementioned one of the two computing units. The invention further relates to a device which is suitable for carrying out said method.

Description

 Apparatus and method for the presentation and processing of films by means of

 mobile phones

The invention relates to a device and a method for the presentation and processing of films by means of mobile phones, which are equipped in particular with the Windows Phone operating system.

As mobile phone performance increases and social networks become more popular, the demand for video editing programs continues to grow. For example, the need of users to cut their (especially self-recorded) films and combine them with other video clips, images, music or other sound effects is increasing.

These films (also often referred to as "videos" or "video files") include video data that includes the image information and audio data that includes information about acoustic information (e.g., sounds, music, or speech).

In general, films are in the form of encoded data, otherwise the storage capacities of electronic devices would quickly reach their limits. So far, in the context of video editing, therefore, the classic procedure decoding - filters / edit - represent / En encode is always applied, in which a video file is first decoded, then filtered and edited and last displayed or re-encoded to the system again as a video file to make available.

Although it is easy to have these tasks done by powerful computer systems, many users wish to be able to edit and watch movies on their mobile phones while on the go. However, due to the limited computing power of mobile phones, movie editing applications are fast approaching their limits. In particular, the processing of high-definition films (for example in H D quality) on many mobile phones is not, or at least not without problems, possible.

However, as the demand for high definition films or at least acceptable video quality is steadily increasing, the limited ability to process them by mobile phones is a disadvantage.

The object of the present invention is to overcome the disadvantages of the prior art and to provide an apparatus and a method capable of processing and optimally displaying high-resolution computer-readable films on mobile telephones.

This object is achieved by a device and a method for the presentation and editing of videos on mobile phones according to the respective independent claims. Among other things, the inventors found the difficulty that for a fluent reproduction of a film the video images have to be processed with high frequency and thereby generate considerable amounts of data that exceed the computing capacity of a mobile telephone, in particular when playing high-resolution films.

Due to the size of such films, it is not possible a whole video

un compressed to keep in memory.

Moreover, in addition to the filtering of the video frames and the display of the same, a computationally intensive encoding or decoding of the video data must be carried out in parallel. It also adds that some of the filters involve such a high computational effort that the rate at which images can be filtered is less than the frame rate of the video, which prevents the film from flowing.

Lastly, when playing filtered video, often user input needs to be processed and the user interface refreshed. In this case, a high reaction speed of the user interface is to ensure program changes and user input.

The inventive method for the presentation and editing of video files on a mobile phone, in which the function of the individual components is controlled by an operating system, wherein the device comprises a central processing unit as a computing unit and as a further processing unit comprises a graphics processing unit and additionally a memory system wherein the central processor unit is connected to the computer unit in such a way that a data exchange can take place based on the steps:

Decoding the film data, in particular the video data, by one of the two arithmetic units (central processing unit or graphics processing unit),

Editing the decoded film data, in particular the video data, by the other of the two arithmetic units (graphics processing unit or central processing unit),

 - Darstel len of the movie data, in particular the video data, by the above one of the two processing units (central processing unit or graphics processing unit).

Between the steps "decode" and "edit" and / or between the steps "edit" and "display" len the appropriate data are preferably transferred in each case in a separate step between the two computing units (from one to the other or from the other to the one). The device according to the invention comprises a mobile telephone in which the function of the individual components is controlled by an operating system, wherein the device comprises a central processing unit as a computing unit and as a further computing unit comprises a graphics processing unit and additionally a memory system, wherein the central processing unit with the The security unit is so connected that a data exchange can take place. In addition, this device is so designed and the internal components interconnected so that the inventive method can be performed on the device.

Mobile phones are known in the art. Likewise, the memory system and the central processing unit (possibly under the name "CPU") .This central processor unit serves to operate this device, but not directly the graphic output on the display of the device in question and can a single processor, one divided into several cores Processor or multiple processors.

The Grafikr echeneinheit is the skilled person ebenfal ls known and used for the depiction of graphics on the display of the device in question. It may also include a single processor, a multi-core processor, or multiple processors. In general, the graphics processing unit is part of the graphics unit of the device and controls its function. Preferred frame units include graphics processors (GPUs) optimized for this purpose. In addition, the graphics processing unit preferably includes a graphics memory. The mobile telephone as a technical device with all its hardware components comprises a control structure which controls the functioning of the individual components. and their interaction. This tax structure is usually implemented by an operating system. Since the composition of the components does not change over the lifetime of a mobile telephone, the essential structures of the operating system (for the method according to the invention) likewise do not change over this period, in spite of updates and patches. A definition of the operating system is thus possible through the brand names of the operating system, at least with the limitation to the state at the time of filing the present invention or the functions and programming interfaces relating to the invention.

In a preferred embodiment, the mobile is designed for the Windows Phone 7, 8, or 8.1 operating system, and the components of the mobile are controlled by this operating system. In this embodiment, the mode of operation of the central processor unit and that of the graphics computing unit also depend on the architecture of the "Windows Phone" operating system.

Moreover, the method is characterized in particular by the fact that the operating system of the mobile telephone does not provide an API (interface for application programming) for decoding the video data. With regard to such mobile telephones, the present method represents a special possibility To speed up the data processing of these mobile phones with regard to the processing of movie files, and thus to control the mobile phone in such a way that an optimized use of the hardware resources is achieved.

High-definition films are films with an image resolution greater than 920 kilopixels, in particular greater than 1.5 megapixels or even greater than 2 megapixels. The lowest resolution for high-resolution films in particular the standard "720p" is considered, which corresponds to an H D resolution of 1 280 pixels x 720 pixels (about 921 kilopixels). Another preferred standard for H D films is "Fill HD" (Fill High

Definition), which corresponds to a H D resolution of 1 920 pixels x 1 080 pixels (about 2.07 megapixels). The method described here is particularly designed to process high-resolution films, in particular Fül l H D films. In a preferred embodiment, the method described is suitable for processing and displaying high-resolution video formats, in particular formats within the framework of the "High Definition Video" standard (for example AVCH D, H DV, 4K2K, UH DV).

The step of decoding the video data is dependent on the encoding of the video data. A preferred standard for encoding the video data is Mpeg-4, in particular with the visual codecs DivX, Xvid, H DX4, 3ivx, FFmpeg, Advanced Video Coding (AVC) or Quicktime container formats.

Methods for decoding video data are known to those skilled in the art.

 In a preferred embodiment, the step of decoding corresponds to d m

Step: Decode the video data by the graphics processing unit. The processing of the decoded film data comprises in particular its modification by filters and / or the modification or addition of additional data. The processing may extend to the video data and / or audio data. Preferably, video data is changed by filters for image processing, additional video data is inserted (and in particular modified by filters) and / or video data is deleted (ie the video track of the film is cut).

Preferably, audio data are changed by filters for sound processing, additional audio data is inserted (and in particular changed by filters) and / or audio data is deleted (ie the audio track of the film is cut). In a preferred embodiment, the processing step comprises the step of: - outsourcing the filtering process into its own thread, which is executed in parallel for decoding, in particular by the central processing unit. Preferably, the processing of a film image takes place at a time when the subsequent image is decoded.

The calculations necessary for the depiction of the film are carried out in particular by that computing unit which has also carried out the decoding of the film data.

In a preferred embodiment, the presentation of the video data is carried out by means of the graphics function. In addition, the method preferably has the additional step of "encoding the film data." This encoding is preferably achieved by means of the graphics processing unit.

The separate processing of the video data has the additional advantage of maintaining responsiveness of the user interface and allowing the user to operate the device well during processing of the movie data (particularly video data).

In a preferred embodiment, the different rates at which video images are captured and filtered will be canceled out by using a "proguiser / cousin model 1." To do this, the method additionally provides the step that the unfiltered images are queued, the filter component of that queue extracts images and filters them. Preferably, the filtered images are queued in a second queue from which they are taken for the presen- tation or the encoding.

With this procedure, first of all a complete decoupling of the decoding, filtering and display / encoding is achieved, so that none of these processes can block other program sequences.

In a preferred embodiment, which further improves the fluid presentation of videos, in the case that new images are decoded faster (eg, from the computational unit) than they can be filtered, they are decoded from the past Discarded images and processed only the latest image, this is done in particular when queuing the new images and / or f iltern and / or when viewing the images. This compensates for the high computation time, depending on the type of filter, and it can, to a certain extent, ensure a synchronized and smooth presentation of the filtered video in relation to the original video.

In a preferred embodiment, the decoded image is added to a queue, preferably deleting all the previously existing filtered images, and images are read out of that queue for processing and / or the edited / filtered image is added to a queue, preferably al the previously existing filtered images are deleted. In a preferred embodiment, the videos are not stored back on the device with applied filters, but the video is filtered in real-time to the preview images to preview the videos.

In a further preferred embodiment, the final videos are produced after decoding or previewing by a new rendering of the filters and then displayed and / or encoded. A program for editing videos, which can be equipped with the method shown here, is offered for example under the name "shakem" (= brand).

Incidentally, the invention also encompasses the copier pr ogram m itself, data carrier on which the method is stored in the form of a computer program, or devices which include this com puter program in a volatile or non-volatile memory.

Examples of preferred embodiments of the device according to the invention and of the method according to the invention are shown in the figures.

FIG. 1 shows schematically the procedure of the method;

 FIG. 2 schematically shows a preferred filtering process.

Figure 1 shows schematically the flow of the method in the form of a flow chart. The dashed lines show the arithmetic unit which is important in the respective steps or executes the corresponding step. The processing units are the central processing unit (CPU) or the G rat ^"i kr ec Henein hey it (GPU).

The dot at the entrance to the flow chart indicates the reading of the movie file, especially its video data.

As a first step, the next video image is decoded by the GPU. Of course, at the beginning of the process, this next video image is the first video image of the movie. The next step is optional, but beneficial for the process. It is to check if a new image is available or not. In the event that one If the image is available, the procedure continues; if not, the termination condition is read in and then checked.

The decoded image is then copied to the CPU, often involving the GPU at this step as it usually controls access to the graphics memory.

However, in the next step, which is optionally advantageous for the process, the decoded image is added to a VVartesch for a long time (for decoded images) and is also read from it during the subsequent processing. In the next step, the decoded image is read from the decoded image queue and processed, e.g. filtered.

However, in the next step, which is optional for the operation of the method, the filtered image is added to a queue (for filtered images), with preferably all the previously existing filtered images being deleted. The last optional step has the advantage that, if the subsequent display is slowed down, the flow of the pictures is not interrupted, but the most current picture is displayed.

The filtered image is subsequently (in particular read out from the VVartesch for a long time) and copied to the GPU, whereby the CPU is often involved in this step as it usually controls the access to the memory of the mobile telephone.

In a next step, the image is displayed on the display of the mobile phone.

Finally, an abort condition is optionally read in and checked and returned to the first step of the method, decoding the next video frame, as it progresses. II

FIG. 2 schematically shows a preferred filtering process in the form of a flow chart, which in this example is completely controlled by the CPU. In the first step, the decoded image is read from the decoded image queue.

It then checks to see if there is any new image in the decoded image queue. If not, then a jump to the step "read abort condition" takes place, if so, it is continued.

In the next step, all images except for the latest decoded image from VVartesch will be deleted long for decoded images.

This is followed by filtering this latest decoded image.

In the next step, the filtered image is added to a VVartesch for a long time for filtered images, wherein preferably in a subsequent step all the previously existing filtered images are deleted. Finally, an abort condition is optionally read in and checked and returned to the first step of the method, reading in a decoded image, of the next video image as it progresses.

Claims

claims 1 . A method for displaying and editing video files on a mobile phone, wherein the function of the individual components of the mobile phone is controlled by an operating system, and wherein the mobile as a computing unit comprises a central processing unit (CPU) and as a further processing unit comprises a graphics processing unit and additionally a memory system, wherein the central processing unit is connected to the graphics processing unit so that a data exchange can take place, characterized by the steps of:  Decoding the film data by one of the two arithmetic units (CPU or graphics re g e e n e),  Editing of the decoded film data by the other of the two arithmetic units (graphics processing unit or CPU), - Representation of Filmdaten by the above one of the two arithmetic units (CPU or Grafikrecheneinheit). 2. The method according to claim 1, characterized in that the step of decoding the step: - decoding the video data by the Graf i krechenei n unit, equivalent. 3. The method according to any one of the preceding claims, characterized in that the processing of the decoded Filmdaten their change by filters and / or the change or addition of additional data includes and preferably video data is changed by filters for image processing, inserted additional video data and / or video data is deleted, and / or preferably changes audio data through filters for the sound processing, inserts additional audio data and / or deletes audio data. 4. The method according to any one of the preceding claims, characterized in that the processing step comprises the step:  - Outsourcing the filtering process into its own thread, the paral lel is executed for decoding, in particular by the central processing unit. Preferably, the processing of a film image takes place at a time when the subsequent image is decoded. 5. The method according to any one of the preceding claims, characterized in that the operating system of the mobile phone no API for Decoding the video data is available, and the mobile phone is designed especially for the operating system Windows Phone 7, 8 or 8.1 and the components of the mobile phone are preferably controlled by this operating system. 6. The method according to any one of the preceding claims, characterized in that the calculations for the presentation of the film necessary calculations are performed by that arithmetic unit, which has also carried out the decoding of the film data, in particular by the G r af i krec h e n h e i t. A method according to any one of the preceding claims, characterized in that the unfiltered images are long queued in a VVartesch, the filter component of this VVartesch extracts and filters long images, and preferably the filtered images are queued to a second queue from which they are used the presentation or the encoding be removed. 8. Method according to one of the preceding claims, characterized in that in the case that new images are decoded more quickly, as they can be filtered, past decoded images are discarded and only the most recent image is processed, particularly as the new images are queued and / or filtered and / or displayed. 9. The method according to any one of the preceding claims, characterized in that the videos are not stored on the device again with applied filters, but for a preview of the videos, the filtering is applied in real time to the preview images. 1 0. Device comprising a mobile phone, in which the function of the individual components is controlled by an operating system, wherein the mobile as a computing unit comprises a central processing unit and as a further processing unit comprises a graphics processing unit and additionally a memory system, wherein the central processing unit with the graphics processing unit is so connected that a data exchange can take place, characterized in that this device is designed in such a way and the internal components are interconnected so that the inventive method is carried out according to one of the preceding claims on the device can.