WO2025227883A1 - Graphics rendering method and apparatus and electronic device - Google Patents

Abstract

The present application provides a graphics rendering method applied to a graphics rendering apparatus. The apparatus comprises a decoupling vertical synchronization module and a rendering module, and the method comprises: in a vertical synchronization period, in response to the input of a first event, the decoupling vertical synchronization module sends a first instruction to the rendering module at a first opportunity, to instruct the rendering module to render a frame following a current rendered frame, wherein the first opportunity is previous to the moment when a following vertical synchronization signal is received, and the first instruction is also used for representing a display time of the frame following the current rendered frame; and upon receiving the first instruction, the rendering module starts rendering the frame following the current rendered frame. The method can decouple the vertical synchronization dependence between rendering and display, and subsequent frames can be pre-rendered by using the residual execution time of a short frame, without waiting the execution of a vertical synchronization signal triggering frame, realizing asynchronous parallelism of rendering and sending for display, and reducing the frame loss rate and the rendering delay.

Description

Methods, apparatus and electronic devices for graphics rendering

This application claims priority to Chinese Patent Application No. 202410539029.0, filed on April 30, 2024, entitled "Method, Apparatus and Electronic Device for Graphic Rendering", the entire contents of which are incorporated herein by reference.

Technical Field

This application relates to the field of graphics rendering, and more specifically, to a method, apparatus, and electronic device for graphics rendering.

Background Technology

Currently, electronic devices use vertical synchronization (VSYNC) technology to avoid screen tearing during display, thereby improving the smoothness and stability of the screen display.

However, VSYNC works by establishing a synchronization signal between the graphics processor (GPU) and the monitor. When the monitor begins its next refresh, it sends a VSYNC signal to the GPU, which then begins rendering the next frame upon receiving the signal. In other words, the GPU must wait for the monitor to finish refreshing before it can render the next frame. This can potentially lead to a lower frame rate and rendering delays, especially under heavy load, which can result in dropped frames and rendering latency.

Summary of the Invention

This application provides a method, apparatus, and electronic device for graphics rendering. This method, apparatus, and electronic device decouple the vertical synchronization dependency between rendering and display. Instead of waiting for the vertical synchronization signal to trigger frame execution, the remaining execution time of short frames can be used to pre-render subsequent frames, achieving asynchronous parallelism between frame rendering and display. This allows for the reuse of CPU time that would otherwise be wasted rendering short frames, reducing frame drop rate and rendering latency.

A first aspect provides a method for graphics rendering, applied to a graphics rendering apparatus including a decoupled vertical synchronization module and a rendering module. The method includes: within a vertical synchronization cycle, in response to the input of a first event, the decoupled vertical synchronization module sends a first instruction to the rendering module at a first opportune moment, the first request message being used to request rendering of the next frame of the current rendering frame, the first instruction being used to instruct the rendering module to render the next frame of the current rendering frame, wherein the first opportune moment is before the moment of receiving the next vertical synchronization signal, and the first instruction is also used to indicate the display time of the next frame of the current rendering frame; after receiving the first instruction, the rendering module begins rendering the next frame of the current rendering frame.

In some embodiments, the vertical synchronization period is the time interval between the display of two adjacent frames; the vertical synchronization period can also be understood as the time interval between two consecutive refreshes of the display screen; the vertical synchronization period can also be understood as the time interval between the consecutive reception of two vertical synchronization signals.

The next frame of the current rendering frame can refer to the next frame of the frame that the rendering module has completed rendering. The current rendering frame can be the next frame of the current display frame or the previous pre-rendered frame; the current rendering frame has not yet been displayed on the device screen.

In some embodiments, the first event may be a user input event at the application layer, such as a user swipe event or an event of opening the APP; the first event may also be an event in which the rendering module finishes rendering the current frame and requests to render the next frame.

It should be noted that the number of first events input within a vertical synchronization cycle is not limited; it can be one or more. Correspondingly, the number of frames rendered by the rendering module within a vertical synchronization cycle can be zero, one, or more.

In one example, within one vertical synchronization cycle:

In response to the input of the first first event, the decoupled vertical synchronization module sends the first first instruction to the rendering module at the first moment within the vertical synchronization cycle. Assuming that the rendering module has just finished rendering the first frame, the first first instruction is used to instruct the rendering module to render the second frame. The first first instruction is also used to indicate the display time of the second frame.

In response to the input of the second first event, the decoupled vertical synchronization module sends a second first instruction to the rendering module at the second moment within the vertical synchronization cycle. The second first instruction is used to instruct the rendering module to render the third frame, and the second first instruction is also used to indicate the display time of the third frame.

Similarly, only one frame can be displayed within a vertical synchronization cycle, but the rendering module can render multiple frames, i.e., it can pre-render frames. The number of frames rendered within a vertical synchronization cycle is not fixed and is affected by factors such as the length of the frame and the length of the vertical synchronization cycle. It can also be understood that the frame rendering process in the solution provided in this application embodiment is not constrained by the vertical synchronization cycle and the vertical synchronization signal.

In some embodiments, the first instruction indicates the display time of the next frame of the current rendered frame in the following way: the first instruction carries a first virtual timestamp, which is used to indicate the display time of the next frame of the current rendered frame.

Virtual timestamps refer to a virtual future time.

The vertical synchronization signal is used to indicate the next frame to be displayed. In this embodiment, by pre-rendering the frame and assigning a virtual display time to the pre-rendered frame, the vertical synchronization dependency between rendering and display can be decoupled. It is not necessary to wait for the vertical synchronization signal to trigger the frame execution. The remaining execution time of the short frame can be used to pre-render subsequent frames, realizing asynchronous parallelism between frame rendering and display. The CPU time that would otherwise be wasted rendering short frames can be reused, reducing frame drop rate and rendering latency.

In conjunction with the first aspect, in one possible implementation, the decoupled vertical synchronization module sends a first instruction to the rendering module at a first opportune moment, including: the decoupled vertical synchronization module receiving a first request message, the first request message being used to request rendering the next frame of the current rendering frame; the decoupled vertical synchronization module sending the first instruction to the rendering module at a first opportune moment according to the current scene.

The current scene includes one of the following: an animated scene (which can also be described as a continuous animated scene), a non-animated scene (which can also be described as a continuous non-animated scene), an animated-to-non-animated scene, or a non-animated-to-animated scene.

In conjunction with the first aspect, in one possible implementation, the decoupled vertical synchronization module sends a first instruction to the rendering module at a first opportune moment based on the current scene, including: when the current scene is an animated scene or a non-animated scene, the decoupled vertical synchronization module sends the first instruction to the rendering module.

In conjunction with the first aspect, in one possible implementation, the decoupled vertical synchronization module sends a first instruction to the rendering module at a first opportune moment based on the current scene. This includes: when the current scene is transitioning from an animation to a non-animation scene, the decoupled vertical synchronization module waits to receive a non-animation request; if a non-animation request is received within m vertical synchronization cycles, the decoupled vertical synchronization module sends a first instruction to the rendering module upon receiving the non-animation request; if no non-animation request is received within the m vertical synchronization cycles, the decoupled vertical synchronization module sends a first instruction to the rendering module at the end of the m vertical synchronization cycles. Here, the starting point of the m vertical synchronization cycles is the moment when the decoupled vertical synchronization module receives the first request message, and m is any value greater than 1 and less than or equal to 2.

In conjunction with the first aspect, in one possible implementation, the decoupled vertical synchronization module sends a first instruction to the rendering module at a first opportune moment based on the current scene. This includes: when the current scene is a non-animation-to-animation scene, the decoupled vertical synchronization module waits to receive an animation request; if an animation request is received within the m vertical synchronization cycles, the decoupled vertical synchronization module sends a first instruction to the rendering module upon receiving the animation request; if no animation request is received within the m vertical synchronization cycles, the decoupled vertical synchronization module sends a first instruction to the rendering module at the end of the m vertical synchronization cycles.

In some examples, the value of m is any one of 1.1, 1.2, 1.3, 1.5, and 1.8.

This implementation method can also be described as:

When the current scene is either an animated or non-animated scene, the decoupled vertical synchronization module sends the first instruction to the rendering module upon receiving the first request message. When the current scene transitions from an animated to a non-animated scene, the decoupled vertical synchronization module starts waiting upon receiving the first request message, with a maximum waiting time set to m vertical synchronization cycles. If a non-animated request is received before the current waiting time is less than m vertical synchronization cycles, the decoupled vertical synchronization module sends the first instruction to the rendering module upon receiving the non-animated request. If no non-animated request is received before the current waiting time reaches m vertical synchronization cycles, the decoupled vertical synchronization module... When the waiting time reaches m vertical synchronization cycles, a first instruction is sent to the rendering module. When the current scene is a non-animation to animation transition scene, the decoupled vertical synchronization module starts waiting upon receiving the first request message, with a maximum waiting time set to m vertical synchronization cycles. If an animation request is received before the current waiting time is less than m vertical synchronization cycles, the decoupled vertical synchronization module sends a first instruction to the rendering module upon receiving the animation request. If no animation request is received before the current waiting time reaches m vertical synchronization cycles, the decoupled vertical synchronization module sends a first instruction to the rendering module after the current waiting time reaches m vertical synchronization cycles.

It can be understood that if the current scene is an animated scene or a non-animated scene, the decoupled vertical synchronization module immediately sends the first instruction to the rendering module. Determining whether the current scene is an animated scene or a non-animated scene can be understood as the trigger condition for sending the first instruction to the rendering module. When the trigger condition is met, the first instruction is sent to the rendering module. There is no time interval or the time interval is extremely short between the two actions of "determining whether the current scene is an animated scene or a non-animated scene" and "sending the first instruction to the rendering module".

In this embodiment, the timing of the decoupled vertical synchronization module sending the first instruction to the rendering module may differ in different scenarios. It is not necessary to wait for the vertical synchronization signal to trigger the execution of the frame. The remaining execution time of the short frame can be used to pre-render subsequent frames, and a virtual timestamp can be assigned to the pre-rendered frame to indicate its display time. This can decouple the vertical synchronization dependency between rendering and display, realize asynchronous parallelism between frame rendering and display, and reuse the CPU time that was originally wasted on rendering short frames, thereby reducing frame drop rate and rendering latency.

In conjunction with the first aspect, in one possible implementation, the method further includes: the decoupled vertical synchronization module determines the current scene based on a first parameter carried by the first request message, the first parameter being used to indicate the type of the first request message, the type of the first request message including an animation request or a non-animation request, the animation request coming from the rendering module, and the non-animation request coming from the application module.

Specifically, the decoupled vertical synchronization module determines the current scenario based on the first parameter carried in the first request message and the parameter carried in the request message received in the previous vertical synchronization cycle.

In this embodiment of the application, the request message for requesting the execution of the next frame carries a parameter indicating the type of the request message, so that the decoupled vertical synchronization module can determine the first timing for sending the first instruction to the rendering module based on the parameter, thus providing a basis for the pre-rendering of the frame.

In conjunction with the first aspect, in one possible implementation, the decoupled vertical synchronization module determines the current scene based on the first parameter carried by the first request message, including: when the first request message is an animation request and no non-animation request was received in the previous vertical synchronization cycle, the decoupled vertical synchronization module determines the current scene as an animation scene, where the vertical synchronization cycle is the time interval between the display of two adjacent frames; when the first request message is an animation request and a non-animation request was received in the previous vertical synchronization cycle, the decoupled vertical synchronization module determines the current scene as an animation-to-non-animation scene; when the first request message is a non-animation request and all request messages received in the previous vertical synchronization cycle are non-animation requests, or when the first request message is a non-animation request and no request message was received in the previous vertical synchronization cycle, the decoupled vertical synchronization module determines the current scene as a non-animation scene; when the first request message is a non-animation request and an animation request was received in the previous vertical synchronization cycle, the decoupled vertical synchronization module determines the current scene as a non-animation-to-animation scene.

In conjunction with the first aspect, in one possible implementation, the method further includes: the decoupled vertical synchronization module determines the display time of the next frame of the current rendering frame based on the display time of the current display frame and the vertical synchronization period.

The display time of the next frame can be a virtual timestamp.

In this embodiment of the application, by assigning a virtual timestamp to the pre-rendered frame to represent its display time, the vertical synchronization dependency between rendering and display can be decoupled, so that the execution of the frame can be triggered without waiting for the vertical synchronization signal.

In conjunction with the first aspect, in one possible implementation, the decoupled vertical synchronization module sends a first instruction to the rendering module at a first opportune moment, including: when the difference between the display time of the next frame of the current rendered frame and the current real time is less than or equal to X vertical synchronization cycles, the decoupled vertical synchronization module sends the first instruction to the rendering module at the first opportune moment, where X is a positive integer greater than or equal to 1.

In some embodiments, when the difference between the first virtual timestamp and the current real timestamp is greater than X vertical synchronization cycles, the system waits for the next vertical synchronization signal from the compositing and display module, and after receiving the next vertical synchronization signal, sends a first instruction to the rendering module.

In this embodiment of the application, by controlling the difference between the virtual timestamp corresponding to the pre-rendered frame and the current real timestamp to be less than a preset value, the number of pre-rendered frames can be controlled to be less than a preset number. In this way, the waste of storage resources caused by too many pre-rendered frames can be avoided. Furthermore, when the number of pre-rendered frames is too large, the accuracy of the pre-rendered frames may decrease. Controlling the number of pre-rendered frames to be less than a preset number can also improve the accuracy of the pre-rendered frames.

In conjunction with the first aspect, in one possible implementation, the method further includes: the rendering module adding the next frame of the currently rendered frame to a buffer queue as a buffer frame in the buffer queue, so that the compositing and display module obtains the next frame of the current display frame from multiple buffer frames in the buffer queue in each vertical synchronization cycle.

In some embodiments, the rendering module adds the next frame of the rendered current frame to the buffer queue in a first-in-first-out order. Correspondingly, the compositing and display module retrieves the next frame of the current display frame from multiple buffer frames in the buffer queue in a first-in-first-out order during each vertical synchronization cycle, for example, at the beginning of each vertical synchronization cycle.

In conjunction with the first aspect, in one possible implementation, the decoupled vertical synchronization module sends a first instruction to the rendering module at a first opportune moment, including: the decoupled vertical synchronization module queries the number of buffers in the buffer queue; when the number of buffers in the buffer queue is less than the maximum number of buffers, the decoupled vertical synchronization module sends the first instruction to the rendering module at the first opportune moment.

In some embodiments, the decoupled vertical synchronization module sends a first query message to the buffer queue, the first query message being used to query the number of buffers in the buffer queue.

In conjunction with the first aspect, in one possible implementation, the method further includes: when the number of buffers in the buffer queue is greater than or equal to the maximum number of buffers, the decoupled vertical synchronization module sends the first instruction to the rendering module upon receiving the next vertical synchronization signal.

Specifically, when the number of buffers in the buffer queue is greater than or equal to the maximum number of buffers, the decoupled vertical synchronization module waits for the next vertical synchronization signal from the compositing and display module; when the decoupled vertical synchronization module receives the next vertical synchronization signal, it sends the first instruction to the rendering module.

In this embodiment of the application, by controlling the number of pre-rendered frames in the buffer queue to be less than a preset number, the waste of storage resources caused by too many pre-rendered frames can be avoided. Furthermore, when the number of pre-rendered frames is too large, the accuracy of the pre-rendered frames may decrease. Controlling the number of pre-rendered frames to be less than a preset number can also improve the accuracy of the pre-rendered frames.

In conjunction with the first aspect, in one possible implementation, the device further includes an event state prediction module. After the first event is input, the method further includes: the event state prediction module predicts the state of the first event to update the state of the first event to an event state corresponding to the display time of the next frame.

In this embodiment of the application, when pre-rendering a frame in response to an event, the event state is predicted before the event is transmitted, and the current state of the event is modified to the state corresponding to the display time of the pre-rendered frame, which can improve the accuracy of the frame pre-rendering.

In conjunction with the first aspect, in one possible implementation, the decoupling vertical synchronization module sends a first instruction to the rendering module at a first opportune moment, including: when the scene corresponding to the first event is a first scene, the decoupling vertical synchronization module starts the decoupling vertical synchronization process; when the decoupling vertical synchronization process is started, the decoupling vertical synchronization module sends a first instruction to the rendering module at the first opportune moment.

The first scenario refers to a scenario with load fluctuations or a scenario with heavy loads.

In some embodiments, the first scenario includes one or more of the following: list scrolling, opening an application, opening a folder, opening the control center, opening the notification center, unlocking, screen rotation, and transitions between applications.

In this embodiment, the decoupled vertical synchronization module is activated to execute the graphics rendering process only when the current scenario involves load fluctuations or heavy loads. The remaining execution time of short frames can be used to pre-render subsequent frames, achieving asynchronous parallelism between frame rendering and display delivery. This reuses the CPU time that would otherwise be wasted rendering short frames, reducing frame drop rate and rendering latency.

In conjunction with the first aspect, in one possible implementation, the device further includes a vertical synchronization module, and the method further includes: when the current scene changes to a non-first scene, the decoupling vertical synchronization module shuts down the decoupling vertical synchronization process; when the decoupling vertical synchronization process is shut down, the vertical synchronization module sends a second instruction to the rendering module upon receiving the next vertical synchronization signal, the second instruction being used to instruct the rendering module to render the next frame of the current display frame; after receiving the second instruction, the rendering module begins rendering the next frame of the current display frame.

In this embodiment, when the current scene does not require frame pre-rendering (i.e., there is no load fluctuation or heavy load), the decoupled vertical synchronization process is turned off, and the graphics rendering process is executed by the vertical synchronization module. Each time the vertical synchronization module receives a vertical synchronization signal from the compositing and display thread, it instructs the rendering module to start rendering the next frame, without pre-rendering the frame. This enables the decoupling of the vertical synchronization module and scene-based cooperation between them, allowing the solution to be applied to more scenarios.

Secondly, a graphics rendering apparatus is provided, comprising a decoupled vertical synchronization module and a rendering module. The decoupled vertical synchronization module includes a pre-execution module. Specifically, the pre-execution module is configured to, within a vertical synchronization cycle, in response to the input of a first event, send a first instruction to the rendering module at a first opportune moment. The first request message is used to request the rendering of the next frame of the current rendering frame, and the first instruction is used to instruct the rendering module to render the next frame of the current rendering frame. The first opportune moment is before the moment when the next vertical synchronization signal is received, and the first instruction is also used to indicate the display time of the next frame of the current rendering frame. The rendering module is configured to start rendering the next frame of the current rendering frame after receiving the first instruction.

The next frame of the current rendering frame can refer to the next frame of the frame that the rendering module has completed rendering. The current rendering frame can be the next frame of the current display frame or the previous pre-rendered frame; the current rendering frame has not yet been displayed on the device screen.

In some embodiments, the first event may be a user input event at the application layer, such as a user swipe event or an event of opening the APP; the first event may also be an event in which the rendering module finishes rendering the current frame and requests to render the next frame.

It should be noted that the number of first events input within a vertical synchronization cycle is not limited; it can be one or more. Correspondingly, the number of frames rendered by the rendering module within a vertical synchronization cycle can be zero, one, or more.

In one example, within one vertical synchronization cycle:

In response to the input of the first first event, the pre-execution module sends the first first instruction to the rendering module at the first moment within the vertical synchronization cycle. Assuming that the rendering module has just finished rendering the first frame, the first first instruction is used to instruct the rendering module to render the second frame. The first first instruction is also used to indicate the display time of the second frame.

In response to the input of the second first event, the pre-execution module sends a second first instruction to the rendering module at the second moment within the vertical synchronization cycle. The second first instruction is used to instruct the rendering module to render the third frame, and the second first instruction is also used to indicate the display time of the third frame.

Similarly, only one frame can be displayed within a vertical synchronization cycle, but the rendering module can render multiple frames, i.e., it can pre-render frames. The number of frames rendered within a vertical synchronization cycle is not fixed and is affected by factors such as the length of the frame and the length of the vertical synchronization cycle. It can also be understood that the frame rendering process in the solution provided in this application embodiment is not constrained by the vertical synchronization cycle and the vertical synchronization signal.

In some embodiments, the first instruction indicates the display time of the next frame of the current rendered frame in the following way: the first instruction carries a first virtual timestamp, which is used to indicate the display time of the next frame of the current rendered frame.

Virtual timestamps refer to a virtual future time.

The vertical synchronization signal is used to indicate that the next frame of the current display frame will be displayed.

In this embodiment, by pre-rendering the frame and assigning a virtual display time to the pre-rendered frame, the vertical synchronization dependency between rendering and display can be decoupled. There is no need to wait for the vertical synchronization signal to trigger the execution of the frame. The remaining execution time of the short frame can be used to pre-render subsequent frames, realizing asynchronous parallelism between frame rendering and display. The CPU time that was originally wasted on rendering short frames can be reused, reducing the frame drop rate and rendering latency.

In conjunction with the second aspect, in one possible implementation, the pre-execution module is specifically used to: receive a first request message, which requests the rendering of the next frame of the current rendering frame; and send the first instruction to the rendering module at a first opportune moment, based on the current scene.

The current scene includes one of the following: an animated scene (which can also be described as a continuous animated scene), a non-animated scene (which can also be described as a continuous non-animated scene), an animated-to-non-animated scene, or a non-animated-to-animated scene.

In conjunction with the second aspect, in one possible implementation, the pre-execution module is also specifically used to send the first instruction to the rendering module when the current scene is an animated scene or a non-animated scene.

In conjunction with the second aspect, in one possible implementation, the pre-execution module is further specifically used for: when the current scene is transitioning from animation to non-animation, waiting to receive a non-animation request; if a non-animation request is received within m vertical synchronization cycles, then a first instruction is sent to the rendering module upon receiving the non-animation request; if no non-animation request is received within the m vertical synchronization cycles, then a first instruction is sent to the rendering module at the end of the m vertical synchronization cycles, wherein the timing start point of the m vertical synchronization cycles is the moment when the first request message is received, and m is any value greater than 1 and less than or equal to 2.

In conjunction with the second aspect, in one possible implementation, the pre-execution module is also specifically used for: when the current scene is a non-animation to animation scene, waiting to receive an animation request; if an animation request is received within the m vertical synchronization cycles, then sending a first instruction to the rendering module upon receiving the animation request; if no animation request is received within the m vertical synchronization cycles, then sending a first instruction to the rendering module at the end of the m vertical synchronization cycles.

In some examples, the value of m is any one of 1.1, 1.2, 1.3, 1.5, and 1.8.

It can be understood that if the current scene is an animated scene or a non-animated scene, the pre-execution module immediately sends the first instruction to the rendering module. Determining whether the current scene is an animated scene or a non-animated scene can be understood as the trigger condition for sending the first instruction to the rendering module. When the trigger condition is met, the first instruction is sent to the rendering module. There is no time interval between the two actions of "determining whether the current scene is an animated scene or a non-animated scene" and "sending the first instruction to the rendering module".

In this embodiment, the timing of the decoupled vertical synchronization module sending the first instruction to the rendering module may differ in different scenarios. It is not necessary to wait for the vertical synchronization signal to trigger the execution of the frame. The remaining execution time of the short frame can be used to pre-render subsequent frames, and a virtual timestamp can be assigned to the pre-rendered frame to indicate its display time. This can decouple the vertical synchronization dependency between rendering and display, realize asynchronous parallelism between frame rendering and display, and reuse the CPU time that was originally wasted on rendering short frames, thereby reducing frame drop rate and rendering latency.

In conjunction with the second aspect, in one possible implementation, the pre-execution module is further specifically used to: determine the current scene based on the first parameter carried by the first request message, the first parameter being used to indicate the type of the first request message, the type of the first request message including animation request or non-animation request, animation request coming from the rendering module, and non-animation request coming from the application module.

Specifically, the decoupled vertical synchronization module determines the current scenario based on the first parameter carried in the first request message and the parameter carried in the request message received in the previous vertical synchronization cycle.

In this embodiment of the application, the request message for requesting the execution of the next frame carries a parameter indicating the type of the request message, so that the decoupled vertical synchronization module can determine the first timing for sending the first instruction to the rendering module based on the parameter, thus providing a basis for the pre-rendering of the frame.

In conjunction with the second aspect, in one possible implementation, the pre-execution module is specifically used to: determine that the current scene is an animation scene when the first request message is an animation request and no non-animation request was received in the previous vertical synchronization cycle, where the vertical synchronization cycle is the time interval between the display of two adjacent frames; determine that the current scene is an animation-to-non-animation scene when the first request message is an animation request and a non-animation request was received in the previous vertical synchronization cycle; determine that the current scene is a non-animation scene when the first request message is a non-animation request and all request messages received in the previous vertical synchronization cycle are non-animation requests, or when the first request message is a non-animation request and no request message was received in the previous vertical synchronization cycle; and determine that the current scene is a non-animation-to-animation scene when the first request message is a non-animation request and an animation request was received in the previous vertical synchronization cycle.

In conjunction with the second aspect, in one possible implementation, the decoupled vertical synchronization module further includes a virtual time module, used to determine the display time of the next frame of the current rendering frame based on the display time of the current display frame and the vertical synchronization period.

The display time of the next frame can be a virtual timestamp.

In this embodiment of the application, by assigning a virtual timestamp to the pre-rendered frame to represent its display time, the vertical synchronization dependency between rendering and display can be decoupled, so that the execution of the frame can be triggered without waiting for the vertical synchronization signal.

In conjunction with the second aspect, in one possible implementation, the pre-execution module is further specifically used to: send the first instruction to the rendering module at a first opportune moment when the difference between the display time of the current rendered frame and the current real time is less than or equal to X vertical synchronization cycles, where X is a positive integer greater than or equal to 1.

In some embodiments, the pre-execution module is further configured to: wait for the next vertical synchronization signal from the compositing and display module when the difference between the display time of the next frame of the current rendered frame and the current real time is greater than X vertical synchronization cycles, and send a first instruction to the rendering module after receiving the next vertical synchronization signal.

In this embodiment of the application, by controlling the difference between the virtual timestamp corresponding to the pre-rendered frame and the current real timestamp to be less than a preset value, the number of pre-rendered frames can be controlled to be less than a preset number. In this way, the waste of storage resources caused by too many pre-rendered frames can be avoided. Furthermore, when the number of pre-rendered frames is too large, the accuracy of the pre-rendered frames may decrease. Controlling the number of pre-rendered frames to be less than a preset number can also improve the accuracy of the pre-rendered frames.

In conjunction with the second aspect, in one possible implementation, the rendering module is further configured to: add the next frame of the currently rendered frame that has been rendered to a buffer queue as a buffer frame in the buffer queue, so that the compositing and display module can obtain the next frame of the current display frame from multiple buffer frames in the buffer queue in each vertical synchronization cycle.

In some embodiments, the rendering module adds the next frame of the rendered current frame to the buffer queue in a first-in-first-out order. Correspondingly, the compositing and display module retrieves the next frame of the current display frame from multiple buffer frames in the buffer queue in a first-in-first-out order during each vertical synchronization cycle, for example, at the beginning of each vertical synchronization cycle.

In conjunction with the second aspect, in one possible implementation, the pre-execution module is further specifically used to: query the number of buffers in the buffer queue; and when the number of buffers in the buffer queue is less than the maximum number of buffers, send the first instruction to the rendering module at the first opportune moment.

In some embodiments, the pre-execution module sends a first query message to the buffer queue, the first query message being used to query the number of buffers in the buffer queue.

In conjunction with the second aspect, in one possible implementation, the pre-execution module is further configured to: send the first instruction to the rendering module upon receiving the next vertical synchronization signal when the number of buffers in the buffer queue is greater than or equal to the maximum number of buffers.

Specifically, when the number of buffers in the buffer queue is greater than or equal to the maximum number of buffers, the pre-execution module waits for the next vertical synchronization signal from the compositing and display module; when the pre-execution module receives the next vertical synchronization signal, it sends the first instruction to the rendering module.

In this embodiment of the application, by controlling the number of pre-rendered frames in the buffer queue to be less than a preset number, the waste of storage resources caused by too many pre-rendered frames can be avoided. Furthermore, when the number of pre-rendered frames is too large, the accuracy of the pre-rendered frames may decrease. Controlling the number of pre-rendered frames to be less than a preset number can also improve the accuracy of the pre-rendered frames.

In conjunction with the second aspect, in one possible implementation, the device further includes: an event state prediction module, used to predict the state of the first event after the first event is input, so as to update the state of the first event to the event state corresponding to the display time of the next frame.

In this embodiment of the application, when pre-rendering a frame in response to an event, the event state is predicted before the event is transmitted, and the current state of the event is modified to the state corresponding to the display time of the pre-rendered frame, which can improve the accuracy of the frame pre-rendering.

In conjunction with the second aspect, in one possible implementation, the decoupled vertical synchronization module further includes a control module, which is used to initiate the decoupled vertical synchronization process when the scene corresponding to the first event is the first scene; the pre-execution module is specifically used to send a first instruction to the rendering module at a first opportune moment when the decoupled vertical synchronization process is initiated.

The first scenario refers to a scenario with load fluctuations or a scenario with heavy loads.

In some embodiments, the first scenario includes one or more of the following: list scrolling, opening an application, opening a folder, opening the control center, opening the notification center, unlocking, screen rotation, and transitions between applications.

In this embodiment, the decoupled vertical synchronization module is activated to execute the graphics rendering process only when the current scenario is a scenario with load fluctuations or a scenario with heavy load. It can use the remaining execution time of the short frame to pre-render subsequent frames, realize asynchronous parallelism of frame rendering and display, and reuse the CPU time that was originally wasted on rendering short frames, thereby reducing frame drop rate and rendering latency.

In conjunction with the second aspect, in one possible implementation, the control module is further configured to: shut down the decoupled vertical synchronization process when the current scene changes to a non-first scene; the device further includes: a vertical synchronization module, configured to send a second instruction to the rendering module upon receiving the vertical synchronization signal when the decoupled vertical synchronization process is shut down, the second instruction being used to instruct the rendering module to render the next frame of the current display frame; the rendering module is further configured to: start rendering the next frame of the current display frame after receiving the second instruction.

In this embodiment, when the current scene does not require frame pre-rendering (i.e., there is no load fluctuation or heavy load), the decoupled vertical synchronization process is turned off, and the graphics rendering process is executed by the vertical synchronization module. Each time the vertical synchronization module receives a vertical synchronization signal from the compositing and display thread, it instructs the rendering module to start rendering the next frame, without pre-rendering the frame. This decouples the vertical synchronization module and the scene-based cooperation between them, enabling the graphics rendering apparatus provided in this application to be used in more scenarios.

Thirdly, an electronic device is provided, comprising a memory and a processor, wherein the memory is used to store computer program code, and the processor is used to execute the computer program code stored in the memory to implement the method in the first aspect or any possible implementation thereof.

Fourthly, a computer-readable storage medium is provided, which stores a computer program or instructions that, when executed, implement the method described in the first aspect or any possible implementation thereof.

Fifthly, a chip is provided, wherein instructions are stored that, when executed on a device, cause the chip to perform the methods of the first aspect or any possible implementation thereof.

In a sixth aspect, a computer program product is provided, which stores a computer program or instructions that, when executed, implement the method in the first aspect or any possible implementation of the first aspect.

Attached Figure Description

Figure 1 is a schematic diagram of the structure of the electronic device provided in an embodiment of this application;

Figure 2 is a software structure block diagram of an electronic device provided in an embodiment of this application;

Figure 3 is a schematic diagram of a VSYNC architecture;

Figure 4 is a schematic diagram of the rendering architecture corresponding to the VSYNC architecture;

Figure 5 is a schematic diagram of the rendering architecture corresponding to another VSYNC architecture;

Figure 6 is a schematic flowchart of a graphics rendering method provided in an embodiment of this application;

Figure 7 is a schematic interactive diagram of a graphics rendering method provided in an embodiment of this application;

Figure 8 is a schematic interactive diagram of another graphics rendering method provided in an embodiment of this application;

Figure 9 is a schematic diagram of a graphics rendering system architecture provided in an embodiment of this application;

Figure 10 is a schematic diagram of another graphics rendering system architecture provided in an embodiment of this application;

Figure 11 is a schematic diagram illustrating the principle of controller control of decoupled vertical synchronous opening and closing provided in an embodiment of this application;

Figure 12 is a schematic diagram illustrating the principle of a pre-executor sending a first instruction to a rendering module according to an embodiment of this application;

Figure 13 is a schematic diagram illustrating the principle of a virtual timer determining a virtual timestamp according to an embodiment of this application;

Figure 14 is a schematic diagram illustrating the principle of an input event prediction module predicting the state of an input event according to an embodiment of this application;

Figure 15 is a comparison diagram of a VSYNC rendering pipeline and a D-VSYNC rendering pipeline provided in an embodiment of this application;

Figure 16 is a test result diagram of the average pixel difference in a first scenario provided by an embodiment of this application, namely continuous fast scrolling, alternating fast scrolling and slow scrolling.

Detailed Implementation

The technical solutions of this application will now be described with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments.

The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. In the description of the embodiments of this application, unless otherwise stated, "/" means "or," for example, A/B can mean A or B; "and/or" in this text is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and/or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, in the description of the embodiments of this application, "plural" or "multiple" refers to two or more than two.

Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this embodiment, unless otherwise stated, "a plurality of" means two or more.

The terminology used in the following embodiments is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. As used in the specification and appended claims of this application, the singular expressions “a,” “an,” “the,” “the,” “the,” and “this” are intended to also include expressions such as “one or more,” unless the context clearly indicates otherwise. It should also be understood that in the following embodiments of this application, “at least one” and “one or more” refer to one, two, or more than two. The term “and/or” is used to describe the relationship between related objects, indicating that three relationships may exist; for example, A and/or B can indicate: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character “/” generally indicates that the preceding and following related objects are in an “or” relationship.

References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "one embodiment," "some embodiments," "another embodiment," "other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

The method provided in this application can be applied to electronic devices with display functions, such as mobile phones, tablets, wearable devices, in-vehicle devices, augmented reality (AR)/virtual reality (VR) devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), smart home devices, and other electronic devices. This application does not impose any restrictions on the specific type of electronic device.

For example, Figure 1 shows a schematic diagram of the structure of an electronic device 100. The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

Processor 110 may include one or more processing units, such as an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, memory, a video codec, a digital signal processor (DSP), a baseband processor, and/or a neural network processing unit (NPU). These different processing units may be independent devices or integrated into one or more processors.

The controller can be the nerve center and command center of the electronic device 100. The controller can generate operation control signals according to the instruction opcode and timing signals to complete the control of fetching and executing instructions.

The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

In some embodiments, the processor 110 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, a universal asynchronous receiver/transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input/output (GPIO) interface, and/or a universal serial bus (USB) interface, etc.

USB port 130 is a USB standard compliant interface, specifically a Mini USB port, Micro USB port, or USB Type-C port. USB port 130 can be used to connect a charger to charge electronic device 100, and can also be used for data transfer between electronic device 100 and peripheral devices. It can also be used to connect headphones for audio playback. This interface can also be used to connect other electronic devices, such as AR devices.

It is understood that the interface connection relationships between the modules illustrated in the embodiments of this application are merely illustrative and do not constitute a structural limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

Electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a miniature LED, a microLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, electronic device 100 may include one or N displays 194, where N is a positive integer greater than 1.

The external storage interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external storage interface 120 to perform data storage functions. For example, music, video, and other files can be saved on the external memory card.

Internal memory 121 can be used to store computer executable program code, which includes instructions. Processor 110 executes various functional applications and data processing of electronic device 100 by running the instructions stored in internal memory 121. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of electronic device 100 (such as audio data, phonebook, etc.). Furthermore, internal memory 121 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

Buttons 190 include a power button, volume buttons, etc. Buttons 190 can be mechanical buttons or touch-sensitive buttons. Electronic device 100 can receive button input and generate key signal inputs related to user settings and function control of electronic device 100.

Indicator 192 can be an indicator light, used to indicate charging status, power changes, or to indicate messages, missed calls, notifications, etc.

The software system of electronic device 100 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application embodiment uses the layered architecture Android system as an example to exemplify the software structure of electronic device 100.

Figure 2 is a software structure block diagram of an electronic device 100 according to an embodiment of this application. The layered architecture divides the software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the Android runtime and system libraries, and the kernel layer. The application layer may include a series of application packages.

As shown in Figure 2, the application package may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, and SMS.

The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions.

As shown in Figure 2, the application framework layer may include a window manager, content provider, view system, phone manager, resource manager, notification manager, etc.

The window manager is used to manage windowed applications. It can retrieve screen size, determine the presence of a status bar, lock the screen, and capture screenshots, among other things.

Content providers store and retrieve data, making that data accessible to applications. This data may include videos, images, audio, made and received phone calls, browsing history and bookmarks, phone books, etc.

A view system includes visual controls, such as controls for displaying text and controls for displaying images. View systems can be used to build applications. A display interface can consist of one or more views. For example, a display interface including a text notification icon could include views for displaying text and views for displaying images.

The phone manager is used to provide communication functions for electronic device 100. For example, it manages call status (including connection and disconnection).

The file explorer provides applications with various resources, such as localized strings, icons, images, layout files, video files, and more.

The notification manager allows applications to display notifications in the status bar. These notifications can be used to deliver informational messages and can disappear automatically after a short pause, requiring no user interaction. For example, the notification manager can be used to notify users of completed downloads or message alerts. The notification manager can also display notifications as icons or scrolling text in the top status bar, such as notifications from background applications, or as dialog boxes on the screen. Examples include displaying text messages in the status bar, emitting sounds, vibrating electronic devices, and flashing indicator lights.

The Android Runtime consists of core libraries and a virtual machine. The Android Runtime is responsible for the scheduling and management of the Android system.

The core library consists of two parts: one part is the functionalities that need to be called by the Java language, and the other part is the Android core library.

The application layer and application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

System libraries can include multiple functional modules. For example: surface manager, media libraries, 3D graphics processing libraries (e.g., OpenGL ES), 2D graphics engines (e.g., SGL), etc.

The Surface Manager is used to manage the display subsystem and provides the blending of 2D and 3D layers for multiple applications.

The media library supports playback and recording of various common audio and video formats, as well as still image files. It supports multiple audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG.

The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

A 2D graphics engine is a graphics engine for 2D drawing.

The kernel layer is the layer between hardware and software. The kernel layer contains at least the display driver, camera driver, audio driver, and sensor driver.

It should be understood that the technical solutions in the embodiments of this application can be used in systems such as Android, iOS, and HarmonyOS.

The technical solutions of this application embodiment can be applied to the graphics rendering process of electronic devices to improve the smoothness and stability of the screen display of electronic devices.

Among them, electronic devices can be televisions, desktop computers, laptops, or portable electronic devices such as mobile phones, foldable screens, tablets, cameras, camcorders, and video recorders. They can also be smart home devices such as refrigerators, washing machines, robot vacuums, and any other electronic devices with display functions. They can also be electronic devices in 5G networks or in future evolved public land mobile networks (PLMNs).

The following section will introduce the relevant terms used in the embodiments of this application.

1. Vertical Synchronization (VSYNC)

Vertical sync (V-Sync) is a graphics card setting used in games to address the "screen tearing" issue. Screen tearing occurs when a game's frame rate exceeds the monitor's refresh rate, causing the monitor to display only a portion of the frames at a time, resulting in a discontinuous vertical display that appears as if the image is "ripped." When V-Sync is enabled, the graphics card limits the game's frame rate based on the monitor's refresh rate. For example, if the monitor's refresh rate is 60Hz, the graphics card will limit the game's frame rate to no more than 60 frames per second. This ensures that the graphics card outputs a complete frame each time the monitor refreshes, thus preventing screen tearing.

2. Decoupled Vertical Synchronization (D-VSYNC)

Decoupling vertical synchronization decouples the VSYNC dependency between frame rendering and display. For example, it can decouple the VSYNC dependency between the rendering thread and the compositing/display thread, or between the rendering thread and the UI thread. Currently, electronic devices use vertical synchronization (VSYNC) technology to avoid screen tearing during use, thereby improving the smoothness and stability of screen display.

For example, Figure 3 shows a schematic diagram of a VSYNC architecture.

As shown in Figure 3, in the VSYNC architecture, the UI thread maintains frame synchronization with the rendering thread. At the beginning of the first VSYNC cycle of app rendering (i.e., VSYNC-app shown in Figure 3), the VSYNC signal wakes up Choreographer to trigger the UI thread to perform the drawing operation of the first frame; after the drawing of the first frame is completed, the rendering thread immediately enters to render the first frame; then at the beginning of the second VSYNC cycle of compositing display (i.e., VSYNC-sf shown in Figure 3), the compositing display thread is triggered to composite and display the first frame, and at the beginning of the third VSYNC cycle of display (i.e., HW-VSYNC shown in Figure 3), the first frame is displayed on the screen of the electronic device.

One crucial point to note is that when the compositing thread is triggered to composite the second frame at the start of the third VSYNC cycle of the compositing display, the rendering thread has not yet completed rendering the second frame. Since the rendering thread for the second frame occupies two VSYNC cycles, the second frame cannot be composited during the third VSYNC cycle. It must wait until the second frame is rendered before the compositing thread can composite it at the start of the fourth VSYNC cycle. Consequently, the second frame only appears on the electronic device screen at the start of the fifth VSYNC cycle, resulting in no frame being displayed during the fourth VSYNC cycle. In other words, because the second frame is a long frame, a frame drop occurs in the fourth VSYNC cycle. And this pattern continues, with the third frame only appearing on the screen at the start of the sixth VSYNC cycle. If multiple long frames like the second frame subsequently appear, it will exacerbate the rendering frame drops and latency.

Choreographer's primary role is to work in conjunction with VSYNC to provide a stable information processing opportunity for the upper-layer app's rendering. Specifically, when VSYNC arrives, the system controls the timing of each frame's rendering operation by adjusting the VSYNC signal period. For example, if the phone's refresh rate is 60Hz (16.6ms), the system can set the VSYNC period to 16.6ms to match the screen's refresh rate. Every 16.6ms, the VSYNC signal wakes up Choreographer to perform the app's rendering operations; this is the main function of introducing Choreographer.

For example, taking the Android system as an example, Figure 4 shows a schematic diagram of the rendering architecture corresponding to the VSYNC architecture.

As shown in Figure 4, the system adopts a separate rendering architecture, with the UI thread and the compositing and display thread running according to the VSYNC beat. When the Nth frame is displayed on the screen, VSYNC-sf can drive the compositing and display thread to compose the N+1th frame, and VSYNC-app can drive the UI thread to execute the N+2th frame.

For example, the compositing and display thread can be SurfaceFlinger. SurfaceFlinger performs compositing, scaling, or blending on the rendered image and then hands it over to the CPU or GPU for rendering. In other words, after receiving the vertical synchronization signal, SurfaceFlinger retrieves the rendered frames from the frame buffer queue, performs compositing and other processing, and then displays them.

For example, Figure 5 shows a schematic diagram of another rendering architecture corresponding to the VSYNC architecture.

As shown in Figure 5, the system adopts a unified rendering architecture, with the UI thread and rendering thread running according to the VSYNC timing. When frame N is displayed on the screen, VSYNC-rs can drive the rendering thread to render frame N+1, and VSYNC-app can drive the UI thread to execute frame N+2. Here, VSYNC-rs refers to the vertical synchronization signal received by the rendering thread, used to indicate the rendering of the next frame; VSYNC-sf refers to the vertical synchronization signal received by the compositing and display thread, used to indicate the compositing of the next frame; and VSYNC-app refers to the vertical synchronization signal received by the UI thread, used to indicate the execution of the next frame.

VSYNC works by establishing a synchronization signal (VSYNC signal) between the graphics processor (GPU) and the monitor. When the monitor begins its next refresh, it sends a VSYNC signal to the GPU. Upon receiving the VSYNC signal, the GPU begins rendering the next frame. In other words, the GPU must wait for the monitor to finish refreshing before it can render the next frame. This can potentially lead to a decrease in the rendering frame rate and rendering delays, especially under conditions of high load fluctuations, which can easily result in dropped frames and rendering latency.

In order to clearly understand the solution of this application, the rendering and display trajectories of several scenarios will be analyzed below.

(1) For swipe reading scenarios, after adopting the VSYNC architecture, the rendering thread needs to wait for the next VSYNC signal to arrive before starting to render the next frame. That is: when the i-th VSYNC signal is generated, the i-th frame is displayed on the screen of the electronic device, the rendering thread renders the i+1-th frame, and the UI thread executes the i+2-th frame.

In this process, after the UI thread sends a signal to VSYNC requesting the execution of the next frame, the main loop of VSYNC blocks and waits for the VSYNC signal. Upon receiving the VSYNC signal, it sends a trigger command to the rendering thread, which instructs the rendering thread to start rendering the next frame. When the first frame of the slide is a long frame, the rendering thread cannot complete the rendering of the long frame within one VSYNC cycle, resulting in a dropped frame.

(2) For the scenario of opening a large folder on the desktop, the specific operation process is as follows: the user clicks the large folder icon, the large folder expands, and then clicks any non-icon location on the expanded page, the large folder collapses.

During this process, the UI thread is idle, and the rendering thread executes the animation. The animation (animate) in the main loop of the rendering thread updates the state of the unified rendering tree. At the same time, animate sends a signal to VSYNC to request the execution of the next frame's animation.

When the workload of each frame is uneven, some short frames may not be able to fill the entire VSYNC cycle, while some long frames may take too long to execute, which can easily lead to frame drops.

In some examples, visualization tests were conducted on the responsiveness of swiping scenarios. Using the VSYNC architecture, a red ball was constructed to follow the finger's touch, and its rendering trajectory was tested. Ideally, without any latency, the red ball should strictly follow the touch position and be covered by the fingertip. However, when swiping upwards, the red ball clearly appears to fall behind the fingertip. When the latency reaches 45ms, with rapid finger swiping, the maximum distance difference between the fingertip and the red ball reaches approximately 400 pixels, with an average distance difference of around 200 pixels.

Therefore, under the current VSYNC architecture, due to the fluctuating load of applications, long frames are often not rendered within a single VSYNC cycle, easily resulting in dropped frames. Furthermore, since the VSYNC signals of the rendering thread and the compositing and display thread are forcibly aligned and synchronized, there is at least a rendering delay of two VSYNC cycles from the start of frame rendering to its display on the screen. In addition, since short frames cannot fill the entire VSYNC cycle, the CPU/GPU computing power of electronic devices cannot be fully utilized, leading to a waste of CPU/GPU computing power.

While current graphics rendering methods can solve image tearing and other adverse effects, they suffer from problems such as frame dropping, rendering latency, and low parallelism between rendering and display.

In view of this, embodiments of this application provide a graphics rendering method that can decouple the vertical synchronization dependency between rendering and display. Instead of waiting for the vertical synchronization signal to trigger the execution of the frame, it can use the remaining execution time of the short frame to pre-render subsequent frames, realizing asynchronous parallelism between frame rendering and display. This can reuse the CPU time that was originally wasted on rendering short frames, reducing frame drop rate and rendering latency.

For example, FIG6 shows a schematic flowchart of a graphics rendering method 600 provided in an embodiment of this application. As shown in FIG6, the method 600 includes:

S601: In the first scenario, the Decoupled Vertical Synchronization (D-VSYNC) module initiates the decoupled vertical synchronization process.

In some embodiments, the first scenario is the scenario corresponding to the input event, the first scenario is the scenario where load fluctuations or large loads may occur, and the first scenario may include, for example, a list scrolling scenario, an app opening scenario, a folder opening scenario, a map zooming scenario, etc.

In some embodiments, when not in the first scenario, the decoupled vertical synchronization process is not enabled, and graphics rendering is performed based on the VSYNC architecture. When the i-th VSYNC signal is generated, the rendering module renders the (i+1)-th frame, and the application module executes the (i+2)-th frame.

In some embodiments, the application layer monitors changes in the current scene and then notifies the controller in the decoupling vertical synchronization module via an API interface. The controller then controls the start and stop of the decoupling vertical synchronization process according to the current scene.

In some embodiments, when there is no event input (i.e. when the application module is not involved), the performance monitoring module in the rendering module monitors the changes in the current scene based on the rendering frame rate and the rendering duration per frame, and then notifies the controller in the decoupling vertical synchronization module, which controls the opening and closing of the decoupling vertical synchronization process according to the current scene.

For example, when the current rendering frame rate is lower than the first frame rate threshold, and the rendering duration of each or part of the consecutive frames is greater than the first time threshold, the current scene is determined to be the first scene, and the controller starts the decoupled vertical synchronization process.

Optionally, after the decoupled vertical synchronization process is initiated, changes in the current scene are continuously monitored. If the current scene changes to a different scenario than the first scenario, the controller shuts down the decoupled vertical synchronization process. The rendering duration of each frame in a series of consecutive frames corresponding to the non-first scenario is less than a second time threshold, which is less than a first time threshold. In this way, the decoupled vertical synchronization process can be turned on and off in a timely manner according to the scene.

S602: After receiving the first request message, the decoupled vertical synchronization module sends a first instruction to the rendering module at a first opportune moment. The first instruction is used to instruct the rendering module to start rendering the next frame of the current rendering frame. The first instruction carries a first virtual timestamp. The first opportune moment is determined according to the first parameter carried in the first request message.

In some embodiments, when the first request message is a non-animation request (a request initiated in response to an input event, i.e., a user interface animation, which can be regarded as an external call request), the first request message comes from the application module; when the first request message is an animation request (i.e., a rendering thread property animation, which can be regarded as a self-call request), the first request message comes from the rendering module (the current rendering frame of the rendering module requests to render the next frame), and the first request message is used to request to render the next frame of the current rendering frame. The first request message may be, for example, RequestNextVSync; the first parameter carried by the first request message may be, for example, fromWhom, where RequestNextVSync is used to indicate that the request is to execute the next frame; fromWhom is used to indicate the source of the first request message, i.e., whether it comes from the application module or the rendering module.

The first virtual timestamp is determined based on the display time of the current frame and the VSYNC period. The VSYNC period is the time interval between two adjacent frames displayed on the screen, which can also be understood as the time interval of the display refresh. Alternatively, the first virtual timestamp is determined based on the current time and the VSYNC period.

In some embodiments, the first parameter can be used to distinguish whether the first request message is an animation request from the rendering module or a non-animation request from the application module. An animation request refers to the rendering module requesting rendering of the next frame after rendering the previous frame; scenarios involving animation requests include opening a folder, opening an app, expanding the notification bar, expanding the status bar, etc. A non-animation request refers to a rendering request from the app, typically involving user input processing or application logic; scenarios involving non-animation requests include swiping the desktop, swiping a list, etc.

Based on the type of the first request message received in the previous VSYNC cycle (whether it is an animation request or a non-animation request, which is determined by the first parameter carried by the first request message received in the previous VSYNC cycle) and the type of the first request message received in the current VSYNC cycle (whether it is an animation request or a non-animation request, which is determined by the first parameter carried by the first request message received in the current VSYNC cycle), it can be determined whether the current scene is an animation scene, a non-animation scene, or a mixed scene (including animation and events).

In some embodiments, the hybrid scenario could be, for example, an app opening scenario where the pop-up window receives continuous animation requests from the rendering module, while the application module might synchronize the rendered app data to the rendering module during the window pop-up and request the next frame via RequestNextVSync. In this case, the user might see a white screen for the first half of the app opening, with content appearing only in the second half. Another example is a sliding list scenario where the first half is a non-animation request when the finger is on the screen, with the app synchronizing data to the rendering module in each frame. When the finger leaves the screen, an inertial animation occurs, and the rendering module makes continuous auto-draw requests.

In some embodiments, the specific method for determining the scenario is as follows:

Non-animation scene (can be understood as continuous non-animation scene): If a first request message of type non-animation is received in the current VSYNC cycle, and the first request message of type non-animation was received in the previous VSYNC cycle, or if no first request message was received in the previous VSYNC cycle (the situation that will occur at the beginning of rendering), then the current scene is a non-animation scene.

In some embodiments, determining a first timing based on a first parameter carried in a first request message includes:

When the current scene is a non-animation scene, the decoupled vertical synchronization module immediately sends the first instruction to the rendering module after receiving the first request message. In other words, in a non-animation scene, the rendering module can immediately respond to the drawing request from the application module, avoiding the occurrence of dropped frames in the first frame and reducing rendering latency.

Animated scene (which can be understood as a continuous animation scene): If a first request message of type animation is received in the current VSYNC cycle, and no first request message of type non-animation was received in the previous VSYNC cycle, then the current scene is an animated scene.

In some other embodiments, determining the first timing based on the first parameter carried in the first request message includes:

When the current scene is an animation scene, the decoupled vertical synchronization module immediately sends the first instruction to the rendering module after receiving the first request message. In other words, in the animation scene, frames will be piled up, achieving the effect of pre-rendering frames.

Hybrid Scene (which can be understood as including animation-to-non-animation scenes and non-animation-to-animation scenes): If a first request message of type animation is received in the current VSYNC cycle, and a first request message of type non-animation was received in the previous VSYNC cycle, then the current scene is a hybrid scene; or, if a first request message of type non-animation is received in the current VSYNC cycle, and a first request message of type animation was received in the previous VSYNC cycle, then the current scene is a hybrid scene.

In some other embodiments, determining the first timing based on the first parameter carried in the first request message includes:

When the current scene transitions from animation to non-animation, the decoupled vertical synchronization module, upon receiving a first request message of type animation request, waits for the corresponding non-animation request for that frame. Upon receiving the non-animation request, it immediately sends a first instruction to the rendering module. The waiting time can be 1.2 VSYNC cycles. If the wait times out, it indicates the input event stream has ended. In response to the animation request, the decoupled vertical synchronization module immediately sends the first instruction to the rendering module after the timeout.

When the current scene is transitioning from non-animation to animation, after receiving a first request message of type non-animation, the decoupled vertical synchronization module waits for the animation request corresponding to that frame. Upon receiving the animation request, it immediately sends a first instruction to the rendering module. The waiting time can be 1.2 VSYNC cycles. If the wait times out, it indicates that the property animation has ended, and in response to the non-animation request, the decoupled vertical synchronization module immediately sends a first instruction to the rendering module.

In animated scenes, where the current frame does not involve APP event responses, D-VSYNC, upon receiving an animation request from the rendering module, immediately sends a first instruction to the rendering module to pre-execute the current frame if it is an animated scene. Similarly, upon receiving a non-animation request from the application module, D-VSYNC, upon receiving a non-animation request, immediately sends a first instruction to the rendering module to pre-execute the current frame if it is a non-animation scene. In mixed scenes, upon receiving both a non-animation request from the application module and an animation request from the rendering module, D-VSYNC immediately sends a first instruction to the rendering module to execute the current frame.

In some embodiments, after receiving the first request message, the decoupled vertical synchronization module first determines a first virtual timestamp. When the difference between the first virtual timestamp and the current real timestamp is less than or equal to X VSYNC cycles, it sends a first instruction to the rendering module at a first opportune moment.

In some embodiments, the number of pre-rendered frames buffered in the frame buffer queue may have an upper limit X, which can be configured by the controller in the decoupled vertical synchronization module; for example, X can be 1, 2, 3, 4, or 5, etc. Based on this, the electronic device can control the difference between the first virtual timestamp and the current real timestamp to be less than or equal to X VSYNC cycles.

In some examples, the number of frames buffered in the frame buffer queue can have an upper limit of X+2 (i.e., the maximum allowed number of buffers in the frame buffer queue), which can be configured by the controller in the decoupled vertical synchronization module; for example, X can be 1, 2, 3, 4, or 5, etc. Based on this, the electronic device can control the difference between the first virtual timestamp and the current real timestamp to be less than or equal to X VSYNC cycles.

In some embodiments, after receiving the first request message, the decoupled vertical synchronization module first determines a first virtual timestamp. If the difference between the first virtual timestamp and the current real timestamp is greater than X VSYNC cycles, it indicates that if the first instruction is sent to the rendering module in the current VSYNC cycle in response to the first request message, the number of pre-rendered frames buffered in the frame buffer queue will exceed the upper limit X. In this case, after receiving the first request message, the decoupled vertical synchronization module will wait until the next VSYNC cycle (i.e., when the VSYNC signal is received) before sending the first instruction to the rendering module.

Another implementation method is as follows: After receiving the first request message, the decoupled vertical synchronization module queries the number of frames buffered in the frame buffer queue. If the number of frames buffered in the frame buffer queue is less than X+2, the decoupled vertical synchronization module sends the first instruction to the rendering module at the first opportune moment.

S603: After receiving the first instruction, the rendering module begins to render the next frame of the current rendering frame and uses the first virtual timestamp to represent the display time corresponding to the next frame of the current rendering frame.

In this embodiment, a virtual timestamp is used to represent the future display time of the pre-rendered frame, thereby decoupling the VSYNC dependency between the rendering module and the compositing and display module. Utilizing the CPU idle time of the electronic device to pre-render subsequent frames enables asynchronous parallel fast display of the two stages of frame rendering and display. This reuses the CPU time that would otherwise be wasted rendering short frames, reducing frame drop rate, reducing rendering latency, and improving responsiveness.

For example, FIG7 shows a schematic interactive diagram of a graphics rendering method 700 provided in an embodiment of this application. As shown in FIG7, the method 700 includes:

S701 to S703: After the first event is input, if the scenario corresponding to the first event is the first scenario, the decoupling vertical synchronization module starts the decoupling vertical synchronization process and receives the first request message.

The explanation of the first scenario and the first request message is described in detail in the embodiment shown in Figure 6, and will not be repeated here for the sake of brevity.

Note that there are two S703s in Figure 7: one indicating that the first request message comes from the application module, and the other indicating that the first request message comes from the rendering module. When there is an event input at the application layer, the application module will issue a first request message, i.e., a non-animation request. In addition, in the rendering module, the current frame can also request the rendering of the next frame, i.e., an animation request. In the same scene, there may only be animation requests (i.e., an animated scene), there may only be non-animation requests (i.e., a non-animation scene), or there may be both animation requests and non-animation requests (i.e., a mixed scene). Specifically, this is described in detail in the embodiment shown in Figure 6.

In one implementation, the decoupling vertical synchronization module includes a controller that controls the activation and deactivation of the decoupling vertical synchronization based on the scenario.

S704: After receiving the first request message, the decoupled vertical synchronization module determines the first timing based on the first parameter carried in the first request message.

The explanation of the first parameter is detailed in the embodiment shown in Figure 6, and will not be repeated here for the sake of brevity.

In some embodiments, the current scene (animated scene, non-animated scene, or mixed scene) is determined based on the first parameter carried in the first request message received in the previous VSYNC cycle and the first parameter carried in the first request message received in the current VSYNC cycle, and then the first timing is determined based on the current scene. For a description of how the current scene is determined based on the first parameter and how the first timing is determined based on the current scene, please refer to the description of S602 in the embodiment shown in Figure 6; for brevity, it will not be repeated here.

S705: The decoupled vertical synchronization module sends the first instruction to the rendering module at the first opportune moment.

In one implementation, the decoupled vertical synchronization module includes a pre-executor that receives a first request message and sends a first instruction to the rendering module at a first opportune moment based on the first parameters carried in the first request message.

In some embodiments, before sending the first instruction to the rendering module, the decoupling vertical synchronization module determines the first virtual timestamp based on the display time of the current display frame and the VSYNC period, or the decoupling vertical synchronization module determines the first virtual timestamp based on the current time and the VSYNC period.

In one implementation, the decoupled vertical synchronization module includes a virtual timer. This virtual timer determines a first virtual timestamp based on the display time of the current display frame and the VSYNC period, or based on the current time and the VSYNC period.

It should be understood that each pre-rendered frame has a corresponding virtual timestamp. In one example, if the pre-rendered frame is the second pre-rendered frame, and the frame just before the one before the one before the one before the one is displayed, then the virtual timestamp corresponding to the pre-rendered frame can be the sum of the display time of the frame before the one before the one before the one and two VSYNC cycles.

The explanation of the first instruction is detailed in the embodiment shown in Figure 6, and will not be repeated here for the sake of brevity.

S706: After receiving the first instruction, the rendering module begins rendering the first target frame. The first target frame refers to the frame following the current frame.

Specifically, after rendering the first target frame, the rendering module associates the first virtual timestamp carried by the first instruction with the first target frame. That is, the first virtual timestamp represents the display time of the first target frame.

S707: After the rendering module finishes rendering the first target frame, it sends the first target frame to the frame buffer queue.

In some embodiments, the rendering module may request a buffer from the frame buffer queue before sending the first target frame to the frame buffer queue.

S708: After receiving the first target frame, the frame buffer queue adds the first target frame to the frame buffer queue in a first-in-first-out order.

The composite display module can retrieve the frame to be displayed from the frame buffer queue at the beginning of each VSYNC cycle, that is, the next frame after the current display frame, and display it on the screen of the electronic device.

In this embodiment, a virtual timestamp is used to represent the future display time of the pre-rendered frame, thereby decoupling the VSYNC dependency between the rendering module and the compositing and display module. Utilizing the CPU idle time of the electronic device to pre-render subsequent frames enables asynchronous parallel fast display of the two stages of frame rendering and display. This reuses the CPU time that would otherwise be wasted rendering short frames, reducing frame drop rate, reducing rendering latency, and improving responsiveness.

Specifically, in the VSYNC architecture, the frame rendering time recorded by the choreographer (VSYNC-app timestamp) and other system clocks are used to represent the frame display time. Therefore, in the VSYNC architecture, frame rendering must be performed shortly before the frame is displayed. However, in the D-VSYNC architecture provided in this application embodiment, since the state and behavior of the graphics system are deterministic, the virtual timer is merged into the modified choreographer, enabling the display time of the pre-rendered frame to be represented by a virtual timestamp instead of the current time. This allows the D-VSYNC architecture to support frame pre-rendering. In other words, the virtual timer calculates when the currently rendered frame will be displayed and associates the calculated virtual timestamp with the rendered frame, ensuring that the rendered frame is displayed at the time corresponding to the virtual timestamp. Therefore, by applying the virtual timer, D-VSYNC can decouple the frame content display time from the frame rendering code execution time, achieving frame pre-rendering and thus improving the performance of electronic devices.

For example, FIG8 shows a schematic interactive diagram of another graphics rendering method 800 provided in an embodiment of this application. As shown in FIG8, the method 800 includes:

S801 to S803 are the same as S701 to S703 in the embodiment shown in Figure 7, and will not be described again here for the sake of simplicity.

S804: The decoupled vertical synchronization module sends a first query message to the frame buffer queue. This first query message is used to query the number of frames currently existing in the frame buffer queue.

In some embodiments, of all frames in the frame buffer queue, two are buffer frames used for frame caching and frame acquisition between the rendering module and the composition and display module, and are non-pre-rendered frames. All other frames are pre-rendered frames.

In some embodiments, the number of frames included in the frame buffer queue must be less than or equal to M. The value of M can be any one of 3, 4, or 5, and can also be other values greater than or equal to 3. This application does not limit the value of M, where M can be understood as the maximum number of buffered frames allowed.

S805: After receiving the first query message, the frame buffer queue sends a first number N to the decoupled vertical synchronization module. The first number N represents the number of frames currently existing in the frame buffer queue.

S806: Determine if the number N of frames currently existing in the frame buffer queue is less than M. If so, it means that the frame buffer queue can still hold more pre-rendered frames, and further execute S807.

S807 to S811 are the same as S704 to S708 in the embodiment shown in FIG7, and will not be described again here for the sake of simplicity.

Specifically, when the number of frames N currently existing in the frame buffer queue is greater than or equal to M, it waits for the VSYNC signal from the compositing and display module, and sends the first instruction to the rendering module after receiving the VSYNC signal.

It can be understood that S804 to S806 are an alternative implementation of the "controlling the difference between the first virtual timestamp and the current real timestamp to be less than or equal to X VSYNC cycles" described in the embodiment shown in Figure 6. When the decoupled vertical synchronization module performs the judgment of "whether the difference between the first virtual timestamp and the current real timestamp is less than or equal to X VSYNC cycles", S804 to S806 are not executed; when the decoupled vertical synchronization module executes S804 to S806, the judgment of "whether the difference between the first virtual timestamp and the current real timestamp is less than or equal to X VSYNC cycles" is not executed. The purpose of both is the same. Therefore, if it is necessary to control the number of pre-rendered frames in the buffer to not exceed a preset number, only one of the two needs to be executed.

S812: After the decoupling vertical synchronization module receives a second request message from the application module or the rendering module, it repeats the above steps S804 to S811, wherein the second request message is used to request the execution of the next frame. Similarly, in the first scenario, each time the decoupling vertical synchronization module receives a request message from the application module or the rendering module to request the execution of the next frame, it repeats the above steps S804 to S811.

Optionally, when the current scene changes to a different scene than the first scene, the decoupled vertical synchronization process is turned off, and a vertical synchronization architecture is used for graphics rendering.

The following sections S813 to S817 describe the process by which the composite display module retrieves and displays frames from the frame buffer queue in a first-in-first-out (FIFO) order at the beginning of each VSYNC cycle. This process can be considered a relatively independent flow and is not directly related to the execution logic of the above-mentioned S802 to S812.

S813: The composite display module retrieves the next frame of the current display frame from the frame buffer queue.

S814: When the next VSYNC cycle begins, the composite display module composites and displays the next frame of the current display frame.

S815: After completing the display of the next frame of the current display frame, the composite display module sends a VSYNC signal to the decoupled vertical synchronization module.

S816: After completing the display of the next frame of the current display frame, the composite display module sends a release command to the frame buffer queue, which is used to instruct the frame buffer queue to release the next frame of the current display frame.

S817: After receiving a release command, the frame buffer queue removes the next frame from the frame buffer queue of the currently displayed frame.

In some embodiments, the composite display module consumes the frame buffer queue in FIFO order during each VSYNC cycle; and the number of buffers queued in the frame buffer queue and other VSYNC configuration parameters (such as VSYNC cycle or phase) are always available for querying, for example, through the first query message mentioned above.

For example, Figure 9 shows a schematic diagram of a system architecture for graphics rendering provided in an embodiment of this application.

As shown in Figure 9, the system architecture includes an application module, a rendering module, a frame buffer queue, a compositing and display module, and a frame rendering timing control module 900. The frame rendering timing control module 900 includes a vertical synchronization module 910 and a decoupled vertical synchronization module 920. The decoupled vertical synchronization module 920 includes a controller 921, a pre-executor 922, and a virtual timer 923. Specifically:

The vertical synchronization module 910 in the frame rendering timing control module 900 is used to control the graphics rendering according to the rules of vertical synchronization when the decoupled vertical synchronization module 920 is in the off state. That is: after receiving the first request message, it waits for the VSYNC signal sent by the composite display module; after receiving the VSYNC signal, it sends a first instruction to the rendering module to instruct the rendering module to start rendering the next frame. Thus, when the i-th VSYNC signal is generated, the i-th frame is displayed, the rendering module renders the (i+1)-th frame, and the application module executes the (i+2)-th frame.

The following focuses on describing the coordination process between the modules when the decoupling vertical synchronization module 920 is in the enabled state.

S901 to S902: After the first event is input, when the application module detects that the scene corresponding to the first event is the first scene, it sends an enable command to the controller 921 through the API interface configured on the controller 921, so that the controller 921 enables decoupled vertical synchronization. Furthermore, the application module sends a first request message to the pre-executor 922, which is used to request the execution of the next frame.

In some embodiments, after the graphics rendering begins, the performance monitoring module in the rendering module monitors the rendering frame rate and the duration of each frame. When it detects that the current scene is the first scene, it directly enables decoupled vertical synchronization through the controller 921. Furthermore, the rendering module sends a first request message to the pre-executor 922, which requests the execution of the next frame.

In some other embodiments, the first request message may be sent by the rendering module to the pre-executor 922 after rendering the current frame, to request the execution of the next frame.

S903: The decoupled vertical synchronization module 920 sends a first query message to the frame buffer queue, which is used to query the number of frames currently existing in the frame buffer queue.

The explanation of this step is the same as that of S804 in the embodiment shown in Figure 8, and will not be repeated here for the sake of brevity.

S904: After receiving the first query message, the frame buffer queue sends a first quantity N to the decoupled vertical synchronization module 920. The first quantity N represents the number of frames currently existing in the frame buffer queue.

S905: When N is less than M, the decoupled vertical synchronization module 920 sends the first instruction to the rendering module at the first opportune moment.

For an explanation of this step, please refer to the explanation of S806 to S808 in the embodiment shown in Figure 8. For the sake of brevity, it will not be repeated here.

S906: In response to a request from the rendering module, the framebuffer queue allocates a buffer to the rendering module.

The explanations of S907 to S910 are similar to those of S808 to S817 in the embodiment shown in Figure 8, and will not be repeated here for the sake of brevity.

In some embodiments, to ensure compatibility with existing applications, D-VSYNC is developed based on the traditional VSYNC framework. D-VSYNC adds new API interfaces while retaining the semantics of the original API interfaces.

In some embodiments, D-VSYNC can be partially or completely turned off when the rendering process depends on unpredictable data or incompatible applications.

For example, Figure 10 shows a schematic diagram of another system architecture for graphics rendering provided in an embodiment of this application.

As shown in Figure 10, the system architecture includes an application module, a rendering module, a frame buffer queue, a compositing and display module, and a frame rendering timing control module 900. The frame rendering timing control module 900 includes a vertical synchronization module 910 and a decoupled vertical synchronization module 920. The decoupled vertical synchronization module 920 includes a controller 921, a pre-executor 922, and a virtual timer 923. Additionally, the system architecture also includes an input event prediction module 1000, which may include, for example, a list sliding predictor 1100, a zoom event predictor 1200, and other predictors. Specifically:

The vertical synchronization module 910 in the frame rendering timing control module 900 is used to control the graphics rendering to proceed according to the rules of vertical synchronization when the decoupled vertical synchronization module 920 is in the off state. That is: after receiving the first request message, it waits for the VSYNC signal sent by the composite display module; after receiving the VSYNC signal, it sends a first instruction to the rendering module to instruct the rendering module to start rendering the next frame. Thus, when the i-th VSYNC signal is generated, the i-th frame is displayed, the rendering module renders the (i+1)-th frame, and the application module executes the (i+2)-th frame.

The following focuses on describing the coordination process between the modules when the decoupling vertical synchronization module 920 is in the enabled state.

It can be understood that the embodiment shown in Figure 10 adds an input event prediction module 1000 to the embodiment shown in Figure 9, so that the state of the input event is the event state corresponding to the virtual time calculated by the virtual timer. That is, the state of the input event is updated to the event state corresponding to the display time of the pre-rendered frame (e.g., the display position of the pre-rendered frame). This can avoid rendering errors caused by the loss of the latest event state between rendering and display due to the frame being executed before some VSYNC cycles in the D-VSYNC pre-rendering architecture, and can improve the accuracy of real-time rendering.

The embodiment shown in Figure 10 differs from the embodiment shown in Figure 9 in steps S1001 to S1003, specifically:

S1001 to S1002: After the first event is input, the input event prediction module 1000 obtains the first virtual timestamp from the virtual timer 923. This first timestamp is the expected display time corresponding to the next frame. Based on the input first event, a predictor (list sliding predictor 1100, zoom event predictor 1200, or other predictor) is determined. The determined predictor corresponding to the first event predicts the state of the first event and modifies the state of the first event to the expected state corresponding to the first virtual timestamp, thereby improving the accuracy of the pre-rendering of the next frame.

In one example, when the application component displays a list of items that the user can scroll through (the first input event is the list scroll event), the list scroll predictor 1100 built into the input event prediction module 1000 will be activated and take effect. The list scroll predictor 110 uses an internal heuristic model and a virtual timestamp output by the virtual timer 923 to modify the current coordinates of the touch to the expected coordinates corresponding to the virtual timestamp.

In some embodiments, the input event prediction module 1000 provides an API interface to the application for registering/unregistering predictors, which allows custom predictors to be registered to the input event prediction module 1000 so that the input event prediction module 1000 can be applied to more scenarios.

S1003: The input event prediction module 1000 updates the status of the first predicted event to the application module.

The input event prediction module 1000 can also be described as an event state prediction layer (ESPL), and the list scrolling predictor 1100 can also be described as a list scrolling position predictor (LSPP).

S1004 to S1012 are the same as S902 to S910 in the embodiment shown in Figure 9, and will not be described again here for the sake of simplicity.

To better understand the architecture of graphics rendering provided in this application embodiment, the controller 921, pre-executor 922, virtual timer 923, and input event prediction module 1000 in the decoupled vertical synchronization module 900 will be described in detail below with reference to Figures 11 to 14.

For example, Figure 11 shows a schematic diagram of the principle of controller control of decoupled vertical synchronization on and off according to an embodiment of this application.

As shown in Figure 11, the controller is used to control the start and stop of D-VSYNC. In addition, the controller is also used to configure D-VSYNC parameters. The D-VSYNC parameters may include the maximum number of buffers in the frame buffer queue, and may also include other VSYNC configuration parameters, such as the VSYNC period.

There are two ways to control the D-VSYNC to turn on and off:

1. The controller in D-VSYNC exposes API interfaces for starting, stopping, and configuring to the APP layer. In the first scenario, the APP layer (or OS) instructs the controller to start D-VSYNC through these API interfaces.

Optionally, when the scene changes to a different scene than the first scene, the controller can be instructed to disable D-VSYNC via this API interface.

This implementation method can take full advantage of the APP's familiarity with the scene and avoid dropping frames in the first frame.

2. After the graphics start rendering, the rendering module's performance monitoring module performs rendering frame rate and per-frame execution time statistics. When the current scene is determined to be the first scene based on the rendering frame rate and per-frame execution time statistics, D-VSYNC is enabled through the controller.

Optionally, the performance monitoring module can disable D-VSYNC via the controller when the scene changes to a different scenario.

This implementation method enables automatic start and stop of D-VSYNC without requiring any changes at the APP level.

For example, Figure 12 shows a schematic diagram of the principle of a pre-executor sending a first instruction to a rendering module according to an embodiment of this application.

As shown in Figure 12(a), after receiving the first request message for requesting the execution of the next frame, the frame rendering timing control module uses the VSYNC architecture to execute the frame rendering logic if D-VSYNC is not enabled. That is, it continues to wait for the VSYNC signal sent by the compositing and display module. After receiving the VSYNC signal, it sends a first instruction to the rendering module to instruct the rendering module to execute the next frame. If D-VSYNC is enabled, the pre-executor determines the first timing based on the first parameter carried in the first request message, and sends a first instruction to the rendering module at the first timing to instruct the rendering module to execute the next frame, and uses a first virtual timestamp to represent the display time of the rendered frame.

Figure 12(b) shows a schematic diagram of a rule for a pre-executor to determine a first timing based on a first parameter, according to an embodiment of this application.

As shown in Figure 12(b), after receiving the first request message for requesting the execution of the next frame, if D-VSYNC is enabled, the pre-executor determines the first timing based on the first parameter.

During the graphics rendering process, four scenarios may occur: continuous animated scenes, continuous non-animated scenes, animated scene transitioning to non-animated scenes, and non-animated scene transitioning to animated scenes. The transition from animated to non-animated scenes and from non-animated to animated scenes involves alternating animation and non-animation requests. Based on the first parameter, it can be determined which of the four scenarios the current scene falls into, and based on this determination, the first timing is identified.

In some embodiments, the pre-executor determines the current scene (animated scene, non-animated scene, or mixed scene) based on the first parameter carried by the first request message received in the previous VSYNC cycle and the first parameter carried by the first request message received in the current VSYNC cycle, and then determines the first timing based on the current scene.

In some embodiments, the pre-executor determines a first timing based on a first parameter, including:

If a first request message of type non-animation is received in the current VSYNC cycle, and the first request messages received in the previous VSYNC cycle were all of type non-animation, or no first request message was received in the previous VSYNC cycle (the situation that occurs at the very beginning of rendering), then the current scene is a non-animation scene (which can be understood as a continuous non-animation scene); when the current scene is a non-animation scene, the pre-executor immediately sends the first instruction to the rendering module after receiving the first request message.

If a first request message of type animation is received in the current VSYNC cycle, and no first request message of type non-animation was received in the previous VSYNC cycle, then the current scene is an animation scene (which can be understood as a continuous animation scene). When the current scene is an animation scene, the pre-executor sends the first instruction to the rendering module immediately after receiving the first request message.

When a first request message of type "animation request" is received within the current VSYNC cycle, if a first request message of type "non-animation request" was received in the previous VSYNC cycle, then the current scene is a mixed scene, specifically, an animation-to-non-animation scene within a mixed scene. When the current scene is an animation-to-non-animation scene, after receiving a first request message of type "animation request," the pre-executor waits for the corresponding non-animation request for that frame. Upon receiving the corresponding non-animation request, it immediately sends a first instruction to the rendering module. The waiting time can be m VSYNC cycles. If the wait times out, it indicates the end of the input event stream. In response to the animation request, the pre-executor immediately sends a first instruction to the rendering module, where m is any value greater than 1 and less than or equal to 2, such as 1.1, 1.2, 1.3, 1.5, or 1.8.

Alternatively, if a first request message of type non-animation is received within the current VSYNC cycle, and a first request message of type animation was received in the previous VSYNC cycle, then the current scene is a mixed scene, specifically a non-animation-to-animation scene within a mixed scene. When the current scene is a non-animation-to-animation scene, after receiving a first request message of type non-animation, the pre-executor waits for the animation request corresponding to that frame. Upon receiving the animation request, it immediately sends a first instruction to the rendering module. The waiting time can be m VSYNC cycles. If the wait times out, it indicates that the property animation has ended. In response to the non-animation request, the pre-executor immediately sends a first instruction to the rendering module. This first instruction carries a first virtual timestamp. After receiving the first instruction, the rendering module begins executing the next frame and uses the first virtual timestamp to represent the display time of the next frame.

In this embodiment, the pre-executor enables the frame rendering module to perform frame rendering in advance without having to align with the VSYNC signal from the composite display module, and assigns a virtual timestamp to the frame to represent the future display time of the frame.

For example, Figure 13 shows a schematic diagram of the principle of a virtual timer for determining a virtual timestamp according to an embodiment of this application.

As shown in Figure 13, the virtual timer provides a corresponding virtual timestamp for each pre-rendered frame. The virtual timestamp can be the time when the pre-rendered frame will be displayed on the screen in the future, which is virtualized based on the timestamp of the previous frame and the VSYNC period.

Specifically, the first request message carries a second parameter, which is used to carry the timestamp of the current frame (i.e., the previous frame of the current pre-rendered frame). This second parameter can be lastVSyncTS. When the first request message is a request from the rendering module to execute the autodraw request for the next frame, the second parameter carries the timestamp of the previous frame; when the first request message is a non-animation request, the second parameter has a default value of 0. Referring to Figure 13, when the first request message is a request from the rendering module to execute the autodraw request for the next frame, the second parameter carries the timestamp of the previous frame. The calculated virtual timestamp of the next frame is the sum of the timestamp of the previous frame (lastVSyncTS) and one VSYNC cycle. Therefore, in the frame stacking state, the animation can be executed as smoothly as a traditional VSYNC, where lastVSyncTS represents the timestamp of the current frame (i.e., the previous frame of the current pre-rendered frame).

Note that the virtual timestamp for the next frame calculated here must be greater than the current time (now). If it is less than the current time, it means that a frame drop has occurred even though D-VSYNC was used. In this case, the first expected VSYNC timestamp after the current time is used as the virtual timestamp for the next frame. For example, the sum of the actual timestamp of the current VSYNC cycle and one VSYNC cycle can be used as the virtual timestamp for the next frame.

As shown in Figure 13, when the first request message is a non-animation request, the first VSYNC timestamp expected after the current time (now) is used as the virtual timestamp of the next frame. For example, the sum of the actual timestamp of the current VSYNC cycle and one VSYNC cycle is used as the virtual timestamp of the next frame. Furthermore, the virtual timestamp of the next frame must be greater than or equal to the virtual timestamp of the previous event. If the virtual timestamp of the next frame is less than the virtual timestamp of the previous event, the virtual timestamp of the previous event is used as the virtual timestamp of the next frame. This ensures that in the case of frame stacking, time rollback will not occur due to a sudden receipt of a non-animation request.

The real VSYNC signal records the real timestamp (realVSyncTS) and the real VSYNC period (realPeriod) for each VSYNC period.

In some embodiments, after calculating the virtual timestamp, each frame calibrates the virtual timestamp using the real timestamp (realVSyncTS) and the real VSYNC period (realPeriod) recorded by the real VSYNC signal. The virtual timestamp (expectedVSyncTS) is calibrated to realVSyncTS + k * realPeriod, where k is an integer.

This calibration helps prevent the accumulation of errors in virtual timestamps.

In some embodiments, the virtual timer also provides an API interface for obtaining the display time, which the application can use in its own frame logic when needed.

For example, Figure 14 shows a schematic diagram of the principle of an input event prediction module predicting the state of an input event according to an embodiment of this application.

Figure 14(a) shows a schematic diagram of the principle of the List Scroll Position Predictor (LSPP) in the input event prediction module predicting the state of the sliding event.

As shown in Figure 14(a), when the application component displays a scrollable list of items (i.e., the input event is a list scrolling event), LSPP is activated and takes effect. After LSPP is activated, a model is selected based on the scrolling of the item list. Specifically, when the item list scrolls upwards quickly, the upward curve model shown in Figure 14(a) is selected; when the item list scrolls downwards quickly, the downward curve model shown in Figure 14(a) is selected; and when the item list scrolls slowly, the straight line model shown in Figure 14(a) is selected. After the model selection is completed, the model is updated, that is, the current model is modified to the latest selected model; then, based on the updated model and the virtual timestamp from the virtual timer, the current coordinates of the touch are modified to the coordinates corresponding to the virtual timestamp, and the coordinates corresponding to the virtual timestamp are output. In this way, the current state of the input event can be modified and corrected to the expected state corresponding to the expected display time of the frame calculated by the virtual timer.

Figure 14(b) shows a schematic diagram of the principle of the Map Zoom Distance Predictor (MZDP) in the input event prediction module predicting the state of zoom events.

As shown in Figure 14(b), after an input event, when two touch points are detected (i.e., the input event is a zoom event), MZDP is activated and takes effect. After MZDP is activated, the distance between the two touch points is calculated. The coordinates of the two touch points are ( _x1 , _y1 ) and ( _x2 , _y2 ), and the distance between the two touch points is... Then, the model is updated based on the calculated distance between the two touch points. Following this, the distance between the two touch points is modified to match the distance between the two touch points corresponding to the virtual timestamp, based on the updated model and the virtual timestamp from the virtual timer. The distance between the two touch points corresponding to the virtual timestamp is then output. This allows the current state of the input event to be modified and corrected to the expected state corresponding to the expected display time of the frame calculated by the virtual timer.

In addition, the input event prediction module may also include a drawing point predictor or a game angle predictor, as well as other custom predictors, to predict the state of different events.

In some embodiments, since almost all applications use some form of list to display pages such as news, contacts, or settings, the input event prediction module includes LSPP by default for list scrolling prediction. LSPP and D-VSYNC are activated when an application uses a list and detects dropped frames, providing a default performance boost for most applications.

In some embodiments, the D-VSYNC architecture provides rollback and fault tolerance mechanisms for the frame buffer queue, specifically:

During the pre-rendering of frames in the D-VSYNC architecture, when an input event passes through the corresponding predictor in the input event prediction module, the input event prediction module records the event state prediction data. In other words, the input event prediction module records the most recent historical event state prediction data. This historical data can be used to evaluate the pixel difference between the predicted state and the actual data, thus determining the accuracy of the event state prediction.

D-VSYNC provides an API interface for discarding buffered frames. For any pre-rendered frame in the framebuffer queue, if its content is inaccurate due to incorrect prediction, and the inaccuracy exceeds a set threshold, the buffered pre-rendered data can be discarded through this API interface.

In some embodiments, when discarding pre-rendered data in the buffer via this API interface, the specific discarding method is as follows: release all buffered frames in the frame buffer queue before the currently rendered frame, and release all buffered frames after the last frame synthesized by the compositing and display module. The currently rendered frame will be the next frame to be displayed.

It should be understood that the rollback and fault tolerance mechanisms set up for the frame buffer queue in the D-VSYNC architecture are optional. In some embodiments, application logic can determine the inaccuracy of the prediction and explicitly call the API interface for discarding buffered frames.

Since incorrect predictions of event states only result in minor pixel differences, in some embodiments, an API interface for discarding buffered frames may not be configured.

To more intuitively demonstrate the beneficial effects of D-VSYNC pre-rendering in reducing frame drops and lowering rendering latency, Figure 15 shows a comparison diagram of a VSYNC rendering pipeline and a D-VSYNC rendering pipeline provided in an embodiment of this application.

As shown in Figure 15, the VSYNC rendering pipeline maintains frame synchronization between the UI thread and the rendering thread. At the start of the first VSYNC cycle of the app (i.e., VSYNC-app as shown in Figure 15), the VSYNC signal triggers the UI thread to draw the first frame. After the UI thread finishes rendering the first frame, the rendering thread immediately follows to render the first frame. Then, at the start of the second VSYNC cycle for compositing and display (i.e., VSYNC-sf as shown in Figure 15), the compositing and display thread is triggered to composite and display the first frame. Finally, at the start of the third VSYNC cycle for display (i.e., HW-VSYNC as shown in Figure 15), the first frame is displayed on the screen of the electronic device. That is, when the i-th VSYNC signal is generated, the i-th frame is displayed, the rendering thread renders the (i+1)-th frame, and the UI thread executes the (i+2)-th frame.

One point to note is that when the UI thread draws the 11th frame, because the 11th frame is an extremely long frame, drawing the 11th frame takes more than 3 VSYNC cycles. This means that the rendering thread cannot perform rendering operations within 3 cycles. Consequently, no frame is displayed on the screen within 3 cycles, which results in dropped frames.

As shown in Figure 15, the D-VSYNC rendering pipeline pre-renders frames and adds them to the frame buffer queue, accumulating frames to be displayed—this is the accumulation stage in the D-VSYNC rendering pipeline shown in Figure 15. When the number of frames in the frame buffer queue reaches the maximum allowed number, D-VSYNC uses a frame synchronization method between the UI thread and the rendering thread for pre-rendering—this is the synchronization stage in the D-VSYNC rendering pipeline shown in Figure 15. When the UI thread draws the 11th frame, since the 11th frame is a very long frame, drawing the 11th frame takes more than 3 VSYNC cycles. At this time, the compositing and display thread consumes the pre-rendered frames in the frame buffer queue in a first-in-first-out order. In this way, even if the rendering thread cannot perform rendering operations within 3 cycles, frames can still be displayed on the screen during those 3 cycles, thus avoiding frame drops. After the 11th frame is rendered, since the number of buffers in the frame buffer queue is less than the maximum allowed number of buffers, D-VSYNC will start pre-rendering frames again to accumulate the frames to be displayed, which is the long frame & accumulation stage in the D-VSYNC rendering pipeline in Figure 15; when the number of buffers in the frame buffer queue reaches the maximum allowed number of buffers, D-VSYNC switches to pre-rendering by keeping the UI thread and the rendering thread synchronized, which is the synchronization stage in the D-VSYNC rendering pipeline in Figure 15.

Among them, the execution entity of the UI thread can be the application module, the execution entity of the rendering thread can be the rendering module, and the execution entity of the compositing and display thread can be the compositing and display module.

In this embodiment, D-VSYNC extends the VSYNC framework, dividing the rendering framework into an accumulation phase and a synchronization phase. During the accumulation phase, D-VSYNC decouples rendering time from display time. When there are still spare frame buffers, some frames can be executed ahead of schedule based on an abstraction of the calculated display time, without waiting for the VSYNC signal. Therefore, D-VSYNC can fully utilize the time and computing power saved by short frames during load fluctuations, allowing time for the execution of long frames, thereby avoiding dropped frames.

In addition, in order to clearly demonstrate the beneficial effects of the embodiments of this application, based on the above-mentioned sliding reading scenario (1) and the scenario of opening a large folder on the desktop (2), the rendering display trajectory of these two scenarios after using the graphics rendering method provided in the embodiments of this application is analyzed as follows.

(1) For sliding reading scenarios, after adopting the D-VSYNC architecture, D-VSYNC can immediately instruct the rendering module to execute the next frame when it receives a request to execute the next frame. The display time of the frame executed in the rendering module is represented by a virtual timestamp from the virtual timer. The rendering module does not need to wait for the next VSYNC signal to arrive before it starts rendering the next frame, which can avoid the waiting caused by VSYNC signal alignment, thus having more rendering time and avoiding frame drops.

(2) For scenarios where a large folder is opened on the desktop, after adopting the D-VSYNC architecture, after the current frame requests the execution of the next frame, D-VSYNC can instruct the rendering module to render the next frame in advance according to the virtual timestamp corresponding to the next frame from the virtual timer. In this way, short frames will be executed one after another, and when a long frame appears, the execution time of the long frame can occupy the time saved by the short frames, avoiding frame drops and rendering delays.

In some examples, in short-frame scenarios, compared to the rendering latency of VSYNC (at least 2 VSYNC cycles), D-VSYNC allows the UI thread and the rendering thread to complete within the same VSYNC cycle, reducing the rendering latency to 1 VSYNC cycle.

In one example, when the first scene was a fast, continuous sliding scene, the D-VSYNC architecture was used to test frame drop performance. The frame drop situation for different applications was tested with different numbers of buffers set in the frame buffer queue. When the number of buffers in the D-VSYNC frame buffer queue was set to 4, it eliminated 71.6% of frame drops, and the average frame drops per second (FDPS) for 25 applications decreased from 2.04 to 0.58. When the number of buffers in the D-VSYNC frame buffer queue was 5, it eliminated 87.7% of frame drops, and the average FDPS for 25 applications decreased from 2.04 to 0.25.

In one example, Figure 16 shows the test results of the average pixel difference in a first scenario provided by an embodiment of this application, namely continuous fast scrolling, alternating fast scrolling, and slow scrolling.

As shown in Figure 16, D-VSYNC with 5 buffers has higher responsiveness. For continuous fast scrolling scenes, it can reduce the average pixel difference (APD) by 61.6%, and for slow scrolling scenes, it can reduce the APD by 55.9%.

In some other examples, performance model verification was performed based on SkiaGL. As shown in Table 1 below, SkiaGL improved the frame rate by 4.9 frames in scenes with less than full frames and achieved full frames in 10 scenes.

Table 1

Similarly, in some other examples, performance models were validated based on Skia VK. As shown in Table 2 below, Skia VK improved the frame rate by an average of 3.1 frames in scenarios with less than full frames, and achieved full frames in 20 scenarios with the same load as the baseline.

Table 2

The graphics rendering method provided in this application can be applied not only to the operating system graphics field, but also to other VSYNC-driven scenarios, such as web scenarios based on VSYNC.

One or more modules or units described herein can be implemented in software, hardware, or a combination of both. When any of the above modules or units are implemented in software, the software exists as computer program instructions and is stored in memory. A processor can be used to execute the program instructions and implement the above method flow. The processor can include, but is not limited to, at least one of the following: a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), a microcontroller unit (MCU), or an artificial intelligence processor, etc., and various computing devices that run software. Each computing device may include one or more cores for executing software instructions to perform calculations or processing. The processor can be built into a SoC (System-on-a-Chip) or an application-specific integrated circuit (ASIC), or it can be a separate semiconductor chip. In addition to the cores for executing software instructions to perform calculations or processing, the processor may further include necessary hardware accelerators, such as field-programmable gate arrays (FPGAs), PLDs (programmable logic devices), or logic circuits that implement dedicated logic operations.

When the modules or units described herein are implemented in hardware, the hardware may be any one or any combination of a CPU, microprocessor, DSP, MCU, artificial intelligence processor, ASIC, SoC, FPGA, PLD, application-specific digital circuit, hardware accelerator, or non-integrated discrete device, which may run the necessary software or perform the above method flow independently of software.

When the modules or units described herein are implemented using software, they can be implemented, in whole or in part, in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims (18)

A method for graphics rendering, characterized in that the method is applied to a graphics rendering apparatus, the apparatus comprising a decoupled vertical synchronization module and a rendering module, the method comprising: Within a vertical synchronization cycle, in response to the input of a first event, the decoupled vertical synchronization module sends a first instruction to the rendering module at a first opportune moment; wherein, the first instruction is used to instruct the rendering module to render the next frame of the current rendering frame, and the first opportune moment is before the moment of receiving the next vertical synchronization signal, the first instruction is also used to indicate the display time of the next frame of the current rendering frame. After receiving the first instruction, the rendering module begins rendering the next frame of the current rendering frame. According to the method of claim 1, the first instruction carries a first virtual timestamp, which is used to indicate the display time of the next frame of the current rendering frame. According to the method of claim 1 or 2, the decoupled vertical synchronization module sends a first instruction to the rendering module at a first opportune moment, comprising: The decoupled vertical synchronization module receives a first request message, which is used to request the rendering of the next frame of the current rendering frame; The decoupled vertical synchronization module sends the first instruction to the rendering module at a first opportune moment based on the current scene. The current scene includes one of the following: an animated scene, a non-animated scene, an animated-to-non-animated scene, and a non-animated-to-animated scene. According to the method of claim 3, the decoupled vertical synchronization module sends a first instruction to the rendering module at a first opportune moment based on the current scene, including: When the current scene is an animated scene or a non-animated scene, the decoupled vertical synchronization module sends the first instruction to the rendering module; When the current scene transitions from animation to non-animation, the decoupled vertical synchronization module waits to receive a non-animation request. If the non-animation request is received within m vertical synchronization cycles, the decoupled vertical synchronization module sends the first instruction to the rendering module upon receiving the non-animation request; or, if the non-animation request is not received within the m vertical synchronization cycles, the decoupled vertical synchronization module sends the first instruction to the rendering module at the end of the m vertical synchronization cycles. The starting point of the m vertical synchronization cycles is the moment when the decoupled vertical synchronization module receives the first request message, and m is any value greater than 1 and less than or equal to 2. When the current scene is a non-animation to animation scene, the decoupled vertical synchronization module waits to receive an animation request. If the animation request is received within the m vertical synchronization cycles, the decoupled vertical synchronization module sends the first instruction to the rendering module upon receiving the animation request. If the animation request is not received within the m vertical synchronization cycles, the decoupled vertical synchronization module sends the first instruction to the rendering module at the end of the m vertical synchronization cycles. The method according to claim 3 or 4, characterized in that the method further comprises: The decoupled vertical synchronization module determines the current scene based on the first parameter carried in the first request message. The first parameter is used to indicate the type of the first request message. The type of the first request message includes an animation request or a non-animation request. The animation request comes from the rendering module, and the non-animation request comes from the application module. According to the method of claim 5, the decoupling vertical synchronization module determines the current scene based on the first parameter carried in the first request message, including: When the first request message is an animation request, and no non-animation request was received in the previous vertical synchronization cycle, the decoupled vertical synchronization module determines that the current scene is an animation scene, and the vertical synchronization cycle is the time interval between the display of two adjacent frames. When the first request message is an animation request, and a non-animation request was received in the previous vertical synchronization cycle, the decoupled vertical synchronization module determines that the current scene is an animation-to-non-animation scene. When the first request message is a non-animation request, and all request messages received in the previous vertical synchronization cycle are non-animation requests, the decoupled vertical synchronization module determines that the current scene is a non-animation scene. When the first request message is a non-animation request, and no request message was received in the previous vertical synchronization cycle, the decoupled vertical synchronization module determines that the current scene is a non-animation scene; or When the first request message is a non-animation request, and an animation request was received in the previous vertical synchronization cycle, the decoupled vertical synchronization module determines that the current scene is a non-animation to animation scene. The method according to any one of claims 1 to 6, characterized in that the method further comprises: The decoupled vertical synchronization module determines the display time of the next frame of the current rendering frame based on the display time of the current display frame and the vertical synchronization period. The method according to any one of claims 1 to 7, characterized in that, the decoupling vertical synchronization module sends a first instruction to the rendering module at a first opportune moment, comprising: When the difference between the display time of the next frame of the current rendered frame and the current real time is less than or equal to X vertical synchronization cycles, the decoupled vertical synchronization module sends the first instruction to the rendering module at the first opportune moment, where X is a positive integer greater than or equal to 1. The method according to any one of claims 1 to 8, characterized in that the method further comprises: The rendering module adds the next frame of the rendered current frame to the buffer queue as a buffer frame in the buffer queue, so that the compositing and display module can obtain the next frame of the current display frame from multiple buffer frames in the buffer queue in each vertical synchronization cycle. According to the method of claim 9, the decoupling vertical synchronization module sends a first instruction to the rendering module at a first opportune moment, comprising: The decoupled vertical synchronization module queries the number of buffers in the buffer queue; When the number of buffers in the buffer queue is less than the maximum number of buffers, the decoupled vertical synchronization module sends the first instruction to the rendering module at the first opportune moment. The method according to claim 10, characterized in that the method further comprises: When the number of buffers in the buffer queue is greater than or equal to the maximum number of buffers, the decoupled vertical synchronization module sends the first instruction to the rendering module upon receiving the next vertical synchronization signal. The method according to any one of claims 1 to 11, characterized in that the apparatus further comprises an event state prediction module, and after the first event input, the method further comprises: The event state prediction module predicts the state of the first event and updates the state of the first event to the event state corresponding to the display time of the next frame of the current rendering frame. The method according to any one of claims 1 to 12, characterized in that, the decoupling vertical synchronization module sends a first instruction to the rendering module at a first opportune moment, comprising: When the scenario corresponding to the first event is the first scenario, the decoupling vertical synchronization module starts the decoupling vertical synchronization process. The first scenario includes one or more of the following: list scrolling scenario, application opening scenario, folder opening scenario, control center opening scenario, notification center opening scenario, unlocking scenario, screen rotation scenario, and application transition scenario. When the decoupled vertical synchronization process is initiated, the decoupled vertical synchronization module sends a first instruction to the rendering module at the first opportune moment. The method according to claim 13, wherein the device further comprises a vertical synchronization module, and the method further comprises: When the current scenario changes to a different scenario than the first scenario, the decoupling vertical synchronization module shuts down the decoupling vertical synchronization process; When the decoupled vertical synchronization process is turned off, the vertical synchronization module sends a second instruction to the rendering module when it receives the next vertical synchronization signal. The second instruction is used to instruct the rendering module to render the next frame of the current display frame. After receiving the second instruction, the rendering module begins rendering the next frame of the currently displayed frame. An electronic device, characterized in that it comprises: One or more processors; One or more memory units; And one or more computer programs, wherein the one or more computer programs are stored in the one or more memories, the one or more computer programs including instructions that, when executed by the one or more processors, cause the electronic device to perform the method as described in any one of claims 1 to 14. A computer-readable storage medium, characterized in that the storage medium stores a program or instructions that, when the program or instructions are executed, implement the method as described in any one of claims 1 to 14. A chip, characterized in that the chip stores instructions that, when executed, implement the method as described in any one of claims 1 to 14. A computer program product, characterized in that the computer program product stores a program or instructions, which, when the program or instructions are run, implement the method as described in any one of claims 1 to 14.