WO2021218751A1 - Cloud phone-based media data processing method and terminal device - Google Patents

Abstract

The present application provides a cloud phone-based media data processing method. The method is applied to a terminal device, the terminal device comprises a browser and a processor, the browser is loaded with a script, and the processor runs the script to perform the following steps: receiving first media data sent by a cloud phone, the first media data being generated when the cloud phone runs a cloud application; decoding the first media data to obtain a first audio and a first video; playing the first audio and the first video in the browser; receiving an operation command inputted for the cloud application by a user in the browser, and sending the operation command to the cloud phone. By loading the script in the browser of the terminal device to implement the decoding of the media data of a game sent by the cloud phone, cross-platform cloud phone game audio and video display can be achieved without being limited to a personal computer.

Description

Media data processing method based on cloud mobile phone and terminal equipment Technical field

This application relates to the field of cloud technology, and in particular to a media data processing method and terminal equipment based on a cloud mobile phone.

Background technique

In the process of running cloud applications, the cloud phone will send media data to the browser of the personal computer via the network. The browser of the personal computer is equipped with a flash plug-in. The flash plug-in is used to process the media data and generate audio and video. The picture and sound of the cloud application are presented to the user in the browser.

The Flash plug-in needs to be installed in the browser of the personal computer, and the Flash plug-in only supports the operating system of the personal computer, not applicable to the operating system of the mobile phone.

Summary of the invention

This application provides a cloud mobile phone-based media data processing method and a terminal device. By loading a script in the browser of the terminal device to decode the media data of the cloud application sent by the cloud mobile phone, it does not need to be limited to a personal computer. Realize cross-platform audio and video display of cloud mobile phone applications.

In the first aspect, a cloud-based mobile phone-based media data processing method is provided. The method is applied to a terminal device. The terminal device includes a browser and a processor, the browser is loaded with a script, and the processor runs all Describe the script to perform the following steps:

Receiving first media data sent by a cloud mobile phone, the first media data being generated when the cloud mobile phone runs a cloud application;

Decoding the first media data to obtain first audio and first video;

Playing the first audio and the first video in the browser;

An operation command for the cloud application input by a user in the browser is received, and the operation command is sent to the cloud mobile phone.

In the above solution, the decoding of the media data of the application sent by the cloud phone is realized by loading the script in the browser of the terminal device, without being limited to a personal computer, and cross-platform audio and video display of the cloud phone application can be realized.

In some possible designs, after sending the operation command to the cloud phone, the processor runs the script to perform the following steps:

Receiving second media data sent by the cloud mobile phone, where the second media data is generated when the cloud mobile phone runs the cloud application according to the operation command;

Decoding the second media data to obtain second audio and second video;

Playing the second audio and the second video in the browser.

In the above solution, the user's operation commands on the terminal device will also be sent to the cloud phone for operation. The media data generated by the cloud phone is then decoded by loading a script in the browser of the terminal device, which can further reduce the number of terminal devices. Without being limited to personal computers, it can achieve cross-platform audio and video display of cloud mobile phone applications.

In some possible designs, the processor runs the script to generate a first browser thread, a second browser thread, and a third browser thread;

The first processing core of the processor runs the first browser thread to receive the first media data;

The second processing core of the processor runs the third browser thread to decode the first media data and obtain the first audio and the first video;

The first processing core of the processor runs the second browser thread to play the first audio and the first video in the browser; and receives the user’s input in the browser for the game Operation command, sending the operation command to the cloud mobile phone.

In the above solution, the third browser thread used for decoding can separately occupy the second processing core, thereby increasing the decoding speed.

In some possible designs, decoding the first media data includes:

When the data volume of the media data in the media consumption queue is less than the threshold, the second processing core of the processor runs the third browser thread to force the decoding of the first media data, wherein the media consumption The queue is used to store the first media data.

In the above solution, the third browser thread can force the decoding of the first media data when the data volume of the media data in the media consumption queue is less than the threshold, without slowly waiting for the media data in the media consumption queue. Decoding is performed only when the amount of data is greater than or equal to the threshold, which speeds up the decoding process of the third browser thread and effectively reduces the time delay.

In some possible designs, decoding the first media data includes:

When the data volume of the media data in the media consumption queue is less than the threshold, the second processing core of the processor runs the third browser thread to add pseudo frames to the media consumption queue, so that the media consumption The data amount of the media data including the pseudo frame in the queue is greater than or equal to a threshold, and the first media data is decoded when the data amount of the media data is greater than or equal to the threshold.

In the above solution, the method of adding pseudo frames accelerates the increase in the amount of data in the media data in the media consumption queue to reach the threshold as soon as possible for decoding, which speeds up the decoding process of the third browser thread and effectively reduces the time delay.

In some possible designs, when the second processing core of the processor runs the third browser thread, a binary code written in a low-level language is called to decode the first media data.

In the above solution, the decoding of the first media data is accelerated by calling a binary code written in a low-level language.

In a second aspect, a terminal device is provided, the terminal device includes a browser and a processor, the browser is loaded with a script, and the processor runs the script to: receive first media data sent by a cloud phone; Decode the first media data to obtain the first audio and the first video; play the first audio and the first video in the browser; receive the operation of the cloud application input by the user in the browser Command to send the operation command to the cloud mobile phone, wherein the first media data is generated when the cloud mobile phone runs the cloud application.

In some possible designs, the processor runs the script to: receive the second media data sent by the cloud phone; decode the second media data to obtain the second audio and the second video; The second audio and second video are played in the browser, wherein the second media data is generated when the cloud mobile phone runs the cloud application according to the operation command.

In some possible designs, the processor runs the script to generate a first browser thread, a second browser thread, and a third browser thread;

The first processing core of the processor runs the first browser thread to receive the first media data sent by the cloud mobile phone;

The second processing core of the processor runs the third browser thread to decode the first media data to obtain the first audio and the first video;

The first processing core of the processor runs the second browser thread to play the first audio and first video in the browser; and receives input from the user in the browser for the cloud application Send the operation command to the cloud phone.

In some possible designs, decoding the first media data includes:

When the data volume of the media data in the media consumption queue is less than the threshold, the second processing core of the processor runs the third browser thread to force the decoding of the first media data, wherein the media consumption The queue is used to store the first media data.

In some possible designs, decoding the first media data includes:

When the data volume of the media data in the media consumption queue is less than the threshold, the second processing core of the processor runs the third browser thread to add pseudo frames to the media consumption queue, so that the media consumption The data amount of the media data including the pseudo frame in the queue is greater than or equal to a threshold, and the first media data is decoded when the data amount of the media data is greater than or equal to the threshold.

In some possible designs, when the second processing core of the processor runs the third browser thread, a binary code written in a low-level language is called to decode the first media data.

In a third aspect, a computer-readable storage medium is provided, including instructions, which when run on a terminal device, cause the terminal device to execute the method according to any one of the first aspect.

In a fourth aspect, a terminal device is provided, including a processor and a memory, and the processor executes the code in the memory to execute the method according to any one of the first aspect.

In a fifth aspect, a computer program product is provided, including a computer program, when the computer program is read and executed by a computing device, the computing device is caused to execute the method according to any one of the first aspect.

In the above solution, the decoding of the media data of the application sent by the cloud phone is realized by loading the script in the browser of the terminal device, without being limited to a personal computer, and cross-platform audio and video display of the cloud phone application can be realized.

Description of the drawings

FIG. 1 is a schematic structural diagram of a media data processing system of a cloud mobile phone involved in this application;

2A-2B exemplarily show an exemplary user interface in which two different terminal devices open a game page in a browser to play a game;

3A-3B exemplarily show a comparison diagram of the original browser structure and the modified browser structure;

FIG. 4 is a flow interaction diagram of a media data processing method based on a cloud mobile phone provided by the present application;

Figure 5 is a schematic diagram of the structure of the WebSocket protocol message involved in this application;

Fig. 6 is a schematic structural diagram of a terminal device provided by the present application;

Fig. 7 is a schematic structural diagram of another terminal device provided by the present application.

Detailed ways

Refer to Fig. 1, which is a schematic structural diagram of a media data processing system of a cloud mobile phone involved in the present application. The media data processing system of the cloud mobile phone in this embodiment includes a terminal device 110 and a cloud data center 120. Wherein, the terminal device 110 may communicate with the cloud data center 120 through a network.

The terminal device 110 may be a smart phone, a cloud terminal, a thin terminal, or the like. In a specific embodiment, the terminal device 110 may be a terminal device used in conjunction with a cloud mobile phone, that is, the terminal device 110 has a strong network communication capability, but may weaken the computing capability and storage capability. A browser is installed in the terminal device 110, and the browser engine used by the browser may be Trident, Gecko, Presto, or Webkit, etc., which is not specifically limited here. The operating system supported by the terminal device 110 may be an IOS system, an Android system (Android), a macOS system, a Windows system, a Linux system, and so on. In the game system, different terminal devices 110 can respectively support different operating systems. For example, terminal device 1 supports Android system, terminal 2 supports macOS system, terminal device 3 supports IOS system, terminal device 4 supports Windows system and terminal 5 supports Linux system. Of course, in the game system, different terminal devices 110 may also support the same operating system, which is not specifically limited here.

The cloud data center 120 may include a service node 121 and a cloud mobile phone management node 125. The service node 121 may be a general physical server, for example, a physical server, such as an X86 server or an ARM server. The cloud mobile phone management node 125 may be a server used to manage cloud mobile phones in a public cloud, and may be a general physical server, or a virtual machine (Virtual Machine, VM) based on a general physical server combined with NFV technology. A virtual machine refers to a complete computer system with complete hardware system functions simulated by software and running in a completely isolated environment, and this application does not specifically limit it.

In a specific embodiment, the service node 121 includes a cloud phone management agent node 124, hardware resources, an operating system (Operating System, OS) 122, and at least one cloud phone 123, and each cloud phone 123 is in the form of a container. Share hardware resources and operating systems with other cloud phones on the server. Among them, a virtualized cloud server with an operating system and a virtual mobile phone function is essentially to transfer applications on the mobile phone to a virtual mobile phone on the public cloud to run.

Hardware resources may include computing resources, storage resources, and network resources, such as processors, memory, network cards, routers, databases, etc., and may also include hardware resources that may be required by other cloud phones, which are not specifically limited in this application.

The operating system 122 may be an operating system applicable to cloud phones, such as an Android operating system, which is not specifically limited in this application. It should be noted that the operating system 122 may be an official complete operating system, or it may be an operating system modified to individual driver modules of the official complete operating system in order to adapt to the operating mode of the service node 121, and this application does not specifically limit it. .

Cloud phone 123 shares hardware resources and operating systems with other cloud phones on the server in the form of a container. Each container can include the cloud applications required by the cloud phone and the dependent resources required by each cloud application. Dependent resources can be operating system core libraries and system libraries, such as functional function libraries, three-dimensional graphics processing libraries (such as OpenGLES), media libraries (Media Libraries), input managers (Input Manager), and so on. In an embodiment, the container may include a game application and dependent resources required by the game application. Among them, the container in which each cloud mobile phone 123 is located uses hardware resources and operating system 122, but does not monopolize the hardware resources, but shares hardware resources and operating system 122 with the containers in which other cloud mobile phones are located. In other words, each container does not have its own kernel, and the application process in the container runs directly on the kernel of the service node 121.

The cloud mobile phone management agent node 124 can be used to monitor and collect the service information of the service node 121, and report the service information to the cloud mobile phone management node 125 in real time, and is also used to according to the received cloud mobile phone specification information, then, Create a container on the service node 121, and divide the resources from the hardware resources to set the hardware and software required by the specification information of the cloud phone for the container.

The process of creating a cloud phone in the cloud data center 120 may be: the terminal device 110 sends cloud phone creation information to the cloud phone management node 125. According to the specification information of the cloud phone 123 in the cloud phone creation information and the service information sent by each cloud phone management agent node 124, the cloud phone management node 125 determines that the server 1 creates the cloud phone 1, and the server 2 creates the cloud phone 2, then the server 1 The cloud phone management agent node 1 can receive the specification information of the cloud phone 1 sent by the cloud phone management node 125, and complete the creation of the cloud phone 1 according to the specification information, and the cloud phone management agent node 2 on the server 2 can receive the cloud phone management node The specification information of the cloud phone 2 sent by 125, and the creation of the cloud phone 2 is completed according to the specification information. It can be understood that the cloud phone management node 125 and the cloud phone management agent node 124 deployed on each service node 121 can be implemented based on container management systems such as Kubernetes (K8S) and Docker, which are not specifically limited in this application.

Based on the media data processing system of the cloud mobile phone shown in FIG. 1, users can play games with low latency through the browser in the terminal device without installing the cloud application in the terminal device. A detailed description will be given below by taking the terminal devices as an example of a smart phone and a thin terminal respectively. For the sake of simplicity, the following examples all use cloud applications as game applications as an example for description, but in other examples, cloud applications may also be other applications, which should not be specifically limited here.

The terminal device is a smart phone. Fig. 2A exemplarily shows an exemplary user interface for opening a game page in a browser to play a game. As shown in the upper left corner of FIG. 2A: the user interface may include: a status bar, a calendar indicator, a weather indicator, a tray with commonly used application icons, a navigation bar, and other application icons. in:

The status bar may include: one or more signal strength indicators for mobile communication signals (also called cellular signals), one or more signal strength indicators for wireless fidelity (Wi-Fi) signals, and battery status Indicator, time indicator.

The calendar indicator can be used to indicate the current time, such as date, day of the week, hour and minute information, etc.

The weather indicator can be used to indicate the type of weather, such as cloudy to clear, light rain, etc., and can also be used to indicate information such as temperature.

A tray with icons of commonly used applications can display: phone icons, contact icons, text messages, and camera icons.

The navigation bar may include system navigation keys such as the return key, the home screen key, and the multitasking key. When it is detected that the user clicks the return button, the smartphone can display the previous page of the current page. When it is detected that the user taps the home button, the smart phone displays the home interface. When it is detected that the user clicks on the multitasking button, the smart phone can display the task that the user has recently opened. The naming of each navigation key can also be other, which is not limited in this application. Not limited to virtual keys, each navigation key in the navigation bar can also be implemented as a physical key.

Other application icons may be, for example, icons for mutual transfer, icons for gallery, icons for music, icons for applications, icons for mailboxes, icons 212 for browsers, icons for memos, and icons for settings. The user interface 210 may also include a page indicator. Other application icons can be distributed on multiple pages, and the page indicator can be used to indicate which application the user is currently browsing. The user can swipe the area of other application icons left and right to browse application icons in other pages.

In some embodiments, the user interface 210 exemplarily shown in FIG. 2A may be a home screen (Home screen). It is understandable that FIG. 2A only exemplarily shows the user interface on the smart phone, and should not constitute a limitation to the embodiment of the present application.

Exemplarily, the user may click the icon 212 of the browser on the user interface 210, the smart phone detects the above-mentioned user operation, and in response to the above-mentioned user operation, the smart phone opens the browser 220.

As shown in the upper right corner of FIG. 2A: the browser 220 may include: a search box 222, a scan control, a tab bar, multiple recommended directories, and a tray of commonly used controls, among which:

The search box 222 can be used to search for a game webpage matching the character according to the character input by the user.

The scan control can be used to scan the QR code or barcode, so as to quickly open the game webpage corresponding to the QR code or barcode.

The title bar can be used to indicate various types of topics, such as recommendations, games, hotspots, and entertainment. The presentation form of the title bar can be text information, icons or other forms.

The control 302 may receive a user operation (for example, a touch operation), and in response to the detected user operation, the anchor terminal 120 may display a page for logging in or switching a cloud live broadcast account.

The tray of commonly used controls can display: previous page control icons, next page control icons, home page control icons, opened webpage icons, and so on. Among them, the aforementioned control icons can all accept user operations (such as touch operations). Corresponding to the detected user operations, the smart phone can display a response page. Specifically, the previous page control icon can be used to display the previous page display. For web pages, the previous page control icon can be used to display the web page displayed on the next page, and the home page control icon can be used for the home page of the browser. The browser can be user-defined, or the navigation page preset by the browser , The icon of the opened webpage can be used to display multiple pages that are currently opened.

A plurality of recommended directories may, for example, display a title and a display picture 224, the display title shows game related information, and the display picture 224 shows game related pictures. The display picture 224 is associated with a hyperlink of the game page. When the user clicks the display picture 224, the browser can jump to the game page 230.

As shown in the lower diagram of FIG. 2A, the game page 230 is used to display the screen content of the game and play the sound of the game. The game page can include game titles, game screens, and game controls.

The game title is optional, that is, the user can choose to display the game title or not to display the game title, or the game page can display the game tag by default, or the game title is not displayed by default.

The image quality, definition, resolution, size, and color of the game screen can be set according to the user's needs. For example, the user can set the game screen to be full screen or non-full screen. In addition, the game screen can also be set on top of other applications in a floating form, so that users can handle other things while playing the game. The game screen can be a two-dimensional screen or a three-dimensional screen.

The specific process of the user opening the game page in the browser to play the game is: the user clicks the browser icon 212 on the user interface 210 (UI) of the smart phone with a finger to open the browser 220. The user enters the Uniform Resource Locator (URL) of the game webpage 230 in the search bar 222 of the browser 120, or clicks the hyperlink 224 recommended by the browser 220 (the hyperlink points to the game webpage 230 and includes the The URL of the game web page 230), thereby opening the game web page 230. Then, the user can play the game on the game page 230.

It is understandable that FIG. 2A only exemplarily shows various user interfaces for opening the game page on the smart phone, and should not constitute a limitation to the embodiment of the present application.

The terminal device is a thin terminal. Fig. 2B exemplarily shows an exemplary desktop where a game page in a browser is opened to play a game. As shown in the upper left corner of Figure 2B: the desktop may include: a calendar indicator, an application tray has been opened, and a start button. in:

The calendar indicator, located in the lower right corner of the desktop, can be used to indicate the current time, such as date, day of the week, hour and minute information, etc.

The opened application tray, located at the bottom of the desktop, is used to indicate the applications that have been opened. For example, when the browser is opened, the browser icon will be displayed in the opened application tray, and when the browser is closed When the browser icon disappears from the open application tray. When the browser is opened multiple times, multiple overlapping browser icons may be displayed to indicate that the browser has been opened multiple times.

The start button, located in the lower left corner of the desktop, is used to display one-level or multi-level menus. Each level of the menu can display the entrances of multiple applications, such as control panels, devices and printers, default programs, security, and help.

Multiple application icons may be displayed on the desktop, such as cloud application icons, camera icons, time icons, calendar icons, application icons, browser icons 310, settings icons, and so on.

Exemplarily, as shown in the upper left diagram of FIG. 2B, the user can click on the icon 310 of the browser on the desktop, the thin terminal detects the above-mentioned user operation, and in response to the above-mentioned user operation, the thin terminal opens the browser 320.

As shown in the upper right corner of FIG. 2B, the browser 320 may include: a search box 322, a previous page control, a next page control, a reload control, a title bar, and multiple recommended directories. Among them:

The search box 322 can be used to search for a game webpage matching the character according to the character input by the user.

The previous page control can be used to display the web page displayed on the previous page, the previous page control icon can be used to display the web page displayed on the next page, and the reload control can be used to refresh the current page to reload the current page .

The title bar can be used to indicate various types of topics, such as recommendations, games, hotspots, and entertainment. The presentation form of the title bar can be text information, icons or other forms.

A plurality of recommended directories may, for example, display a title and a display picture 324, the display title shows game related information, and the display picture 324 shows game related pictures. The display picture 324 is associated with a hyperlink of the game page. When the user clicks the display picture 324, the browser can jump to the game page 330.

As shown in the lower diagram of FIG. 2B, the game page 330 is used to display the screen content of the game and play the game sound. The game page can include game titles, game screens, and game controls.

The game title is optional, that is, the user can choose to display the game title or not to display the game title, or the game page can display the game tag by default, or the game title is not displayed by default.

The picture quality, definition, resolution, size, and color of the game screen can be set according to the needs of the user. For example, the user can set the game screen to be full screen or non-full screen. In addition, the game screen can also be set on top of other applications in a floating form, so that users can handle other things while playing the game. The game screen can be a two-dimensional screen or a three-dimensional screen.

The specific process of the user opening the game page in the browser to play the game is: the user clicks the browser icon 312 on the interface 310 of the display with the mouse, thereby opening the browser 320. The user enters the Uniform Resource Locator (URL) of the game webpage 330 in the search bar 322 of the browser 320, or clicks the hyperlink 324 recommended by the browser 320 (the hyperlink points to the game webpage 330 and includes the The URL of the game web page 330), thereby opening the game web page 330. Then, the user can play the game on the game page 330. Wherein, the interface 310 can be used to display not only browser icons, but also music application icons, mail application icons, gallery application icons, settings icons, etc., which are not specifically limited here.

It can be understood that FIG. 2B only exemplarily shows various user interfaces for opening the game page on the thin terminal, and should not constitute a limitation to the embodiment of the present application.

In addition to the above examples, in actual applications, the terminal device may also be a virtual device, etc., which is not specifically limited here.

The internal structure of the browser involved in this application will be specifically introduced below.

As shown in Figure 3A, taking the browser as WebKit as an example, the browser includes a rendering engine and a JavaScript (JS) engine, where the rendering engine is used to convert resource files (HTML/CSS/JavaScript text and corresponding resource files) into User-visible results. The rendering engine usually includes CSS, SVG, layout, render tree, HTML, DOM, inspector, and so on. The JS engine is used to read the JavaScript in the file, process and execute it, and the JS engine is single-threaded. In addition, Webkit also includes WebKit embedded interface and corresponding WebKit binding, WebKit2 embedded interface and corresponding WebKit2 binding, WebKitPorts. Among them, WebKitPorts can include modules such as grid segment, video, text, hardware acceleration, and image decoding. In addition, the database and operating system that WebKit depends on can also be included. Among them, the database may include a 2D graphics library, a 3D graphics library, a network library, a storage, an audio library, a video library, and so on.

As shown in the dashed box part of FIG. 3B, multiple browser threads including a first browser thread, a second browser thread, and a third browser thread may be added to the JS engine of the existing browser. Among them, the browser thread can be WebWorker. WebWorker is used to enable the browser to run a script operation in a background thread (worker) separate from the main execution thread, thereby creating a multi-threaded runtime environment for the JS engine, that is, the main thread of the browser creates a browser thread and assigns tasks For the browser thread, the browser thread is also running while the main thread is running, without interfering with each other, and the result is returned to the main thread after the browser thread runs. Among them, the first browser thread may also be referred to as a network thread. The network thread adopts the WebSocket protocol, which can realize two-way communication of game data (including media data and control data) between the terminal device and the cloud phone. The second browser thread may also be referred to as the main control thread, which is responsible for the overall operation of the game. The third browser thread may also be referred to as a decoding thread, which is responsible for decoding the media data of the game.

In the following, in conjunction with the browser shown in FIG. 3B, the implementation process that the user can play the game with low latency through the browser in the terminal device will be introduced in detail. As shown in Fig. 4, Fig. 4 is a flow interaction diagram of a game running method provided by the present application. The game running method in this embodiment includes the following steps:

S101: When the user enters the URL of the game webpage in the search bar of the browser, or clicks the hyperlink recommended by the browser, the browser of the terminal device generates a media data request, wherein the media data request is used to generate For the media data of the game webpage, the media data request includes a URL, and the URL corresponds to the game.

In a specific embodiment, the browser engine used by the browser may be Trident, Gecko, Presto, or Webkit, etc., which is not specifically limited here.

In a specific embodiment, the terminal device detects whether the user enters the URL of the game webpage in the search bar through the main control thread in the browser, or whether the user clicks the hyperlink recommended by the browser. Enter the URL of the game webpage in the search bar, or click the hyperlink recommended by the browser, the main control thread generates a media data request, and sends the media data request to the network thread through shared memory or pipeline communication, etc. .

S102: The browser of the terminal device sends the media data request to the cloud mobile phone. Correspondingly, the cloud mobile phone receives the media data request sent by the terminal device.

In a specific embodiment, the network thread in the browser sends the media data request to the cloud phone through the WebSocket protocol. Among them, the network thread is an exclusive Webwork. Therefore, the network thread can realize the data transmission and reception between the browser and the cloud phone.

S103: The cloud mobile phone runs a game application to generate first media data.

In a specific embodiment, after receiving the media data request sent by the browser of the terminal device, the cloud mobile phone obtains the URL of the game webpage from the media data request, determines the corresponding game application according to the URL of the game webpage, and runs the game application , And generate the first media data. Then, the cloud mobile phone encodes the generated first media data through an encoding method, thereby reducing the amount of data transmission.

S104: The cloud mobile phone sends the first media data to the terminal device. Correspondingly, the terminal device receives the first media data sent by the cloud mobile phone.

In a specific embodiment, the network thread in the browser of the terminal device receives the first media data sent by the cloud phone through the WebSocket protocol. After the main control thread in the browser monitors the first media data sent by the network thread, it pushes the first media data to the media data queue. The main control thread in the browser sends the data in the media data queue to the decoding thread for decoding in a real-time manner through shared memory or pipeline communication, etc., thereby effectively reducing the delay of the game.

S105: The terminal device decodes the first media data, so as to obtain the first audio and/or the first video.

In a specific embodiment, the decoding thread in the browser in the terminal device decodes the first media data, thereby obtaining the first audio and/or the first video. Then, the decoding thread in the browser sends the first media data to the main control thread through methods such as shared memory or pipe communication. Among them, the decoding method adopted by the decoding thread corresponds to the encoding method adopted by the cloud phone. When the encoding method adopted by the cloud phone is lossless compression, the decoding method adopted by the terminal device is also lossless compression. When the method is lossy compression, the decoding method adopted by the terminal device is also lossy compression.

S106: The terminal device plays the first audio and/or the first video in the first game page generated by the browser.

S107: When the user operates the game on the game page of the browser, receive the game operation command input by the user on the game page of the browser, and convert it into a game operation that can be recognized by the cloud mobile phone.

In a specific embodiment, the main control thread in the browser of the terminal device receives the game operation command input by the user through the external input device on the game page of the browser. Among them, the external input device can be a touch screen, a mouse, a keyboard, a camera, a scanner, a light pen, a handwriting input board, a joystick, a microphone, and so on. The operation commands of the game can include forward, backward, turn left, turn right, move left, move right, jump, sit down, stand up, retract/take arms, walk/run, etc. Terminal devices and cloud phones have different definitions of operations. Therefore, the browser needs to convert the operation commands input by the user into game operations that the cloud phone can recognize. Then, the main control thread in the browser sends game operations to the network thread through shared memory or pipe communication.

S108: The terminal device sends a game operation to the cloud mobile phone. Correspondingly, the cloud mobile phone receives the game operation sent by the terminal device.

In a specific embodiment, the terminal device sends the game operation to the cloud phone through the WebSocket protocol.

S109: The cloud mobile phone runs the game application according to the game operation, thereby generating second media data.

In a specific embodiment, the cloud mobile phone continues to run the game application, and generates the second media data according to the game operation. Then, the cloud mobile phone encodes the generated second media data through an encoding method, thereby reducing the amount of data transmission.

S110: The cloud mobile phone sends the second media data to the terminal device. Correspondingly, the terminal device receives the second media data sent by the cloud mobile phone.

In a specific embodiment, the network thread in the browser of the terminal device receives the second media data sent by the cloud phone through the WebSocket protocol. After the main control thread in the browser monitors the second media data sent by the network thread, it pushes the second media data to the media data queue. The main control thread in the browser sends the data in the media data queue to the decoding thread for decoding in a real-time manner through shared memory or pipeline communication, etc., thereby effectively reducing the delay of the game.

S111: The terminal device decodes the second media data, so as to obtain the second audio and/or the second video.

In a specific embodiment, the decoding thread in the browser in the terminal device decodes the second media data, thereby obtaining the second audio and/or the second video. Then, the decoding thread in the browser sends the second media data to the main control thread through methods such as shared memory or pipe communication.

S112: The terminal device plays the second audio and/or the second video in the second game page generated by the browser.

In a specific embodiment, the media data request, the first audio and/or the first video, the game operation, and the second audio and/or the second video are all encapsulated in the WebSocket protocol format. As shown in Figure 5, the WebSocket protocol format includes 1-bit FIN, 1-bit RSV1, RSV2, RSV3, 4-bit Opcode, 1-bit mask (MASK), and 7-bit data payload length ( payload len), a mask key value (Masking-key) of 0 to 4 bytes, and a multi-byte payload data (payload). Here, the media data request, the first audio and/or the first video, the game operation, and the second audio and/or the second video load are in the payload.

In a specific embodiment, the terminal device receives the first media data sent by the cloud phone, and needs to decode the first media data into the first audio and/or the first video as soon as possible, and display it in the first game page generated by the browser. Play the first audio and/or the first video. According to different browser capabilities, terminal devices can be divided into at least the following two methods:

In the first way, the terminal device uses a video component to decode the first media data into the first audio and/or the first video as soon as possible, and play the first video in the first game page generated by the browser. Audio and/or first video. Here, the video (video) component can be implemented by using modules in WebKitPorts, for example, video, hardware acceleration, and so on. The terminal device generally plays a video in the browser through a video component, so the video component is not sensitive to time delay. After the network thread receives the media data, the video component will cache the media data first, and will not decode the media data until the number of cached media data exceeds the threshold, and play it out in the browser. Ensure the smoothness of the playback, but this will cause a very large delay. However, the game is very sensitive to delay and cannot accept such a large delay. In order to solve the problem of large time delay, the video component can be improved to reduce the time delay. Specifically, improvements to the video component include at least the following aspects: (1) The video component issues a mandatory command, which instructs the decoding thread to immediately decode the received first media data, thereby obtaining The first video and the first audio, and the main control thread will immediately play the first video and the first audio in the browser without buffering the first media data. (2) The video component instructs the main control thread to add pseudo frames to the media consumption queue, so that the media data in the buffered media consumption queue reaches the threshold as soon as possible. When the threshold is reached, the video component instructs the decoding thread to treat the first media data Perform decoding to obtain the first video and the first audio, and the video component instructs the main control thread to play the first video and the first audio in the browser. Among them, the dummy frame may be pre-stored, the dummy frame may be a data frame completely unrelated to the first media data, and the dummy frame may also be referred to as a transparent frame. (3) After decoding the multi-frame video and audio of the first media data, the video component instructs the main control thread to forcefully play the video and audio of the last frame of the decoded first video and first audio to Reduce time delay. In order to further reduce the time delay, the video component can also process the media data by skipping frames. When the media data is processed by skipping frames, the media data can be sampled, and the closer the data is The denser the frame sampling, the sparser the sampling of the farther data frame.

In the second way, the terminal device uses the Canvas component to decode the first media data into the first audio and/or the first video as soon as possible, and play the first audio and/or in the first game page generated by the browser. Or the first video. Here, the Canvas component can be implemented using modules in WebKitPorts, for example, video, hardware acceleration, and so on. The decoding thread is decoded by files written in JavaScript semantics, and the decoding efficiency is very low. For this reason, the Canvas component can instruct the decoding thread to call the binary code written in a low-level language, for example, WebAssembly for decoding, thereby increasing the decoding rate. In addition, decoding A single thread can independently occupy the core of a processor for decoding, and it can also significantly increase the decoding rate.

The foregoing describes the methods of the embodiments of the present application in detail. In order to facilitate better implementation of the above-mentioned solutions of the embodiments of the present application, correspondingly, the following provides related equipment for cooperating with the implementation of the foregoing solutions.

The terminal device provided in this application may include a processor and a browser as shown in FIG. 3B. The browser includes a JS engine, the browser is loaded with a script, and the processor runs the script to make the JS engine generate the first The browser thread, the second browser thread, and the third browser thread.

The first processing core of the processor runs the first browser thread to receive first media data sent by the cloud mobile phone, the first media data being generated when the cloud mobile phone runs a game;

The second processing core of the processor runs the third browser thread to decode the first media data to obtain the first audio and the first video;

The first processing core of the processor runs the second browser thread to play the first audio and the first video in the browser;

The first processing core of the processor runs the second browser thread to receive an operation command for the game input by a user in the browser, and send the operation command to the cloud mobile phone.

In some possible designs, the first processing core of the processor runs the first browser thread to receive the second media data sent by the cloud mobile phone, and the second media data is executed on the cloud mobile phone according to the operation Generated when the command is running the game;

The second processing core of the processor runs the third browser thread to decode the second media data to obtain second audio and second video;

The second processing core of the processor runs the third browser thread to play the second audio and second video in the browser.

In some possible designs, when the data volume of the media data in the media consumption queue is less than the threshold, the second processing core of the processor runs the third browser thread to force the decoding of the first media data , Wherein the media consumption queue is used to store the first media data.

In some possible designs, when the data volume of the media data in the media consumption queue is less than the threshold, the second processing core of the processor runs the third browser thread to add pseudo frames to the media consumption queue , So that the data amount of the media data including the pseudo frame in the media consumption queue is greater than or equal to a threshold, and when the data amount of the media data is greater than or equal to the threshold, the first media data To decode.

In some possible designs, when the second processing core of the processor runs the third browser thread, a binary code written in a low-level language is called to decode the first media data.

It should be understood that the internal unit modules of the terminal device shown in FIG. 3B may also be divided into various types. Each module may be a software module, a hardware module, or a part of a software module and a part of a hardware module. limit. FIG. 3B is an exemplary division method, which is not specifically limited in this application.

The following describes an implementation manner of the terminal device involved in the embodiment of the present invention. The terminal device supports browsers running multiple webpage tags. The browser can use different input and output devices, for example, the input device can be a keyboard and a mouse, and the output device can be a display screen. Or, the browser uses a common input and output device: touch screen. The touch screen is used to receive the user's touch operation and display the output content of the browser.

Fig. 6 is a structural block diagram of an implementation manner of a terminal device. As shown in FIG. 6, the terminal device may be a thin terminal, including a host 410, an output device 420, and an input device 430.

The host 410 may integrate: one or more processors, a clock module, and a power management module. The clock module integrated in the host 410 is mainly used to generate a clock required for data transmission and timing control for the processor. The power management module integrated in the host 410 is mainly used to provide a stable, high-precision voltage for the processor, the output device 420, and the input device 430. The host 410 also integrates a memory for storing various software programs and/or multiple sets of instructions. In a specific implementation, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory does not need to install an operating system and a large number of application programs. The memory only needs to store some necessary application programs, such as an input and output program, which can be used to communicate with one or more input devices or output devices. The communication program can be used to communicate with cloud phones. The memory can also store a user interface program, which can vividly display the content of the browser through a graphical operation interface, and receive user control operations on the browser through input controls such as menus, dialog boxes, and buttons.

The output device 420 mainly includes a display, and the display may include a cathode ray tube display (Cathode Ray Tube, CRT), a plasma display (Plasma Display Panel, PDP), a liquid crystal display (Liquid Crystal Display, LCD), and so on. Taking the display as LCD as an example, the liquid crystal display includes a liquid crystal panel and a backlight module. The liquid crystal display panel includes a polarizing film, a glass substrate, a black matrix, a color filter, a protective film, a common electrode, a calibration layer, and a liquid crystal layer (liquid crystal , Spacer, sealant), capacitor, display electrode, prism layer, light scattering layer. The backlight module includes: illuminating light source, reflecting plate, light guide plate, diffuser, brightness enhancement film (prism sheet), frame and so on.

The input device 430 may include a keyboard and a mouse. Keyboard and mouse are the most commonly used and most important input devices. Through the keyboard, English letters, numbers, punctuation marks, etc. can be input into the computer, so as to issue commands to the computer, input data, etc., and quickly locate the vertical and horizontal coordinates through the mouse. This simplifies the operation. The keyboard may include a mechanical keyboard, a plastic film keyboard (Mechanical), a conductive rubber keyboard (Membrane), a contactless electrostatic capacitor keyboard (Capacitives), etc. The mouse may include a roller ball mouse, an optical mouse, and a wireless mouse.

The input device 430 is configured to receive first media data sent by a cloud mobile phone, the first media data being generated when the cloud mobile phone runs a cloud application;

The host 410 is configured to decode the first media data to obtain the first audio and the first video;

The output device 420 is used to play the first audio and the first video in the browser;

The input device 430 is configured to receive an operation command for the cloud application input by a user in the browser, so as to send the operation command to the cloud phone through a communication module.

For the sake of brevity, the terminal device shown in FIG. 6 has not been introduced in detail. For details, please refer to FIGS. 1 to 5 and related descriptions, which will not be described again this time.

Fig. 7 is a structural block diagram of another implementation manner of a terminal device. As shown in FIG. 7, the terminal device may be a smart phone, including: a baseband chip 510, a memory 515, including one or more computer-readable storage media, a radio frequency (RF) module 516, and a peripheral system 517. These components may communicate on one or more communication buses 514.

The peripheral system 517 is mainly used to implement the interactive function between the terminal device 510 and the user/external environment, and mainly includes the input and output devices of the terminal device. In a specific implementation, the peripheral system 517 may include: a touch screen controller 518, a camera controller 519, an audio controller 520, and a sensor management module 521. Among them, each controller can be coupled with its corresponding peripheral devices, such as a touch screen 523, a camera 524, an audio circuit 525, and a sensor 526. In some embodiments, the gesture sensor in the sensor 526 may be used to receive gesture control operations input by the user. The pressure sensor in the sensor 526 can be disposed under the touch screen 523, and can be used to collect the touch pressure acting on the touch screen 523 when the user inputs a touch operation through the touch screen 523. It should be noted that the peripheral system 517 may also include other I/O peripherals.

The baseband chip 510 may integrate: one or more processors 511, a clock module 512, and a power management module 513. The clock module 512 integrated in the baseband chip 510 is mainly used to generate a clock required for data transmission and timing control for the processor 511. The power management module 513 integrated in the baseband chip 510 is mainly used to provide a stable and high-precision voltage for the processor 511, the radio frequency module 516, and peripheral systems.

The memory 515 is used to store various software programs and/or multiple sets of instructions. In a specific implementation, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory does not need to install an operating system and a large number of application programs. The memory only needs to store some necessary application programs, such as an input and output program, which can be used to communicate with one or more input devices or output devices. The communication program can be used to communicate with cloud phones. The memory can also store desktop programs.

The radio frequency (RF) module 516 is used to receive and transmit radio frequency signals, and mainly integrates the receiver and transmitter of the terminal device. The radio frequency (RF) module 516 communicates with the communication network and other communication devices through radio frequency signals. In a specific implementation, the radio frequency (RF) module 516 may include, but is not limited to: an antenna system, an RF transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a CODEC chip, a SIM card 5161 And storage media. In addition, the radio frequency module can also include WIFI 5162, Bluetooth and other short-range wireless communication modules. In some embodiments, the radio frequency (RF) module 516 may be implemented on a separate chip.

The radio frequency (RF) module 516 is configured to receive first media data sent by a cloud mobile phone, and the first media data is generated when the cloud mobile phone runs a cloud application;

The baseband chip 510 is configured to decode the first media data to obtain the first audio and the first video;

The touch screen 523 is used to play the first video in the browser, and the audio circuit 525 is used to play the first audio;

The touch screen 523 is configured to receive an operation command for the cloud application input by a user in the browser, so as to send the operation command to the cloud phone through a radio frequency (RF) module 516.

For the sake of brevity, the terminal device shown in FIG. 7 has not been introduced in detail. For details, please refer to FIGS. 1 to 5 and related descriptions, which will not be further described this time.

In the above solution, the decoding of the media data of the game sent by the cloud mobile phone is realized by loading a script in the browser of the terminal device, without being limited to a personal computer, and a cross-platform cloud mobile phone game audio and video display can be realized.

Through the description of the above embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general hardware. Of course, it can also be implemented by dedicated hardware including dedicated integrated circuits, dedicated CPUs, dedicated memory, Dedicated components and so on to achieve. Under normal circumstances, all functions completed by computer programs can be easily implemented with corresponding hardware. Moreover, the specific hardware structures used to achieve the same function can also be diverse, such as analog circuits, digital circuits or special-purpose circuits. Circuit etc. However, for this application, software program implementation is a better implementation in more cases. Based on this understanding, the technical solution of this application essentially or the part that contributes to the existing technology can be embodied in the form of a software product, and the computer software product is stored in a readable storage medium, such as a computer floppy disk. , U disk, mobile hard disk, ROM, RAM, magnetic disk or optical disk, etc., including several instructions to make a computer device (which can be a personal computer, training device, or network device, etc.) execute the various embodiments described in this application method.

In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented by software, it can be implemented in the form of a computer program product in whole or in part.

The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website, computer, training device, or data. The center transmits to another website, computer, training equipment, or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.). The computer-readable storage medium may be any available medium that can be stored by a computer or a data storage device such as a training device or a data center integrated with one or more available media. The usable medium may be a magnetic medium (for example, a floppy disk, a hard disk, and a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk (SSD)).

Claims (13)

A cloud mobile phone-based media data processing method, characterized in that the method is applied to a terminal device, the terminal device includes a browser and a processor, the browser is loaded with a script, and the processor runs the script To perform the following steps: Receiving first media data sent by a cloud mobile phone, the first media data being generated when the cloud mobile phone runs a cloud application; Decoding the first media data to obtain first audio and first video; Playing the first audio and the first video in the browser; An operation command for the cloud application input by a user in the browser is received, and the operation command is sent to the cloud mobile phone. The method according to claim 1, wherein after sending the operation command to the cloud phone, the processor runs the script to perform the following steps: Receiving second media data sent by the cloud mobile phone, where the second media data is generated when the cloud mobile phone runs the cloud application according to the operation command; Decoding the second media data to obtain second audio and second video; Playing the second audio and the second video in the browser. The method according to claim 1 or 2, wherein the processor runs the script to generate a first browser thread, a second browser thread, and a third browser thread; The first processing core of the processor runs the first browser thread to receive the first media data; The second processing core of the processor runs the third browser thread to decode the first media data and obtain the first audio and the first video; The first processing core of the processor runs the second browser thread to play the first audio and the first video in the browser; and receives the user’s input in the browser for the game Operation command, sending the operation command to the cloud mobile phone. The method according to claim 3, wherein decoding the first media data comprises: When the data volume of the media data in the media consumption queue is less than the threshold, the second processing core of the processor runs the third browser thread to force the decoding of the first media data, wherein the media consumption The queue is used to store the first media data. The method according to claim 3, wherein decoding the first media data comprises: When the data volume of the media data in the media consumption queue is less than the threshold, the second processing core of the processor runs the third browser thread to add pseudo frames to the media consumption queue, so that the media consumption The data amount of the media data including the pseudo frame in the queue is greater than or equal to a threshold, and the first media data is decoded when the data amount of the media data is greater than or equal to the threshold. The method according to any one of claims 3 to 5, wherein when the second processing core of the processor is running the third browser thread, a binary code written in a low-level language is used to transfer the second processing core to the third browser thread. One media data is decoded. A terminal device, characterized in that the terminal device includes a browser and a processor, the browser is loaded with a script, and the processor runs the script to: receive first media data sent by a cloud mobile phone; The first media data is decoded to obtain the first audio and the first video; the first audio and the first video are played in the browser; the operation command for the cloud application input by the user in the browser is received, The operation command is sent to the cloud mobile phone, wherein the first media data is generated when the cloud mobile phone runs the cloud application. The terminal device according to claim 7, wherein the processor runs the script to: receive the second media data sent by the cloud mobile phone; decode the second media data to obtain the second media data Audio and second video; playing the second audio and second video in the browser, wherein the second media data is generated when the cloud mobile phone runs the cloud application according to the operation command. The terminal device according to claim 7 or 8, wherein the processor runs the script to generate a first browser thread, a second browser thread, and a third browser thread; The first processing core of the processor runs the first browser thread to receive the first media data sent by the cloud mobile phone; The second processing core of the processor runs the third browser thread to decode the first media data to obtain the first audio and the first video; The first processing core of the processor runs the second browser thread to play the first audio and first video in the browser; and receives input from the user in the browser for the cloud application Send the operation command to the cloud phone. The terminal device according to claim 9, wherein decoding the first media data comprises: When the data volume of the media data in the media consumption queue is less than the threshold, the second processing core of the processor runs the third browser thread to force the decoding of the first media data, wherein the media consumption The queue is used to store the first media data. The terminal device according to claim 9, wherein decoding the first media data comprises: When the data volume of the media data in the media consumption queue is less than the threshold, the second processing core of the processor runs the third browser thread to add pseudo frames to the media consumption queue, so that the media consumption The data amount of the media data including the pseudo frame in the queue is greater than or equal to a threshold, and the first media data is decoded when the data amount of the media data is greater than or equal to the threshold. The terminal device according to any one of claims 9 to 11, wherein when the second processing core of the processor runs the third browser thread, it calls a binary code written in a low-level language to the The first media data is decoded. A computer-readable storage medium, characterized by comprising instructions, which when run on a terminal device, cause the terminal device to execute the method according to any one of claims 1 to 6.

Images