CN117710180A - Image rendering method and related equipment - Google Patents

Abstract

The application relates to the technical field of image processing, aims to solve the problem of high power consumption expenditure in a rendering process, and provides an image rendering method and related equipment. The image rendering method is applied to the electronic equipment, an application program is installed in the electronic equipment, and the application program transmits a rendering instruction stream to render a first model in a first image, and the image rendering method comprises the following steps: intercepting a specific instruction in a rendering instruction stream; acquiring a central position of the first model according to a specific instruction, wherein the central position is a position of a central point of the first model in the first image; determining a coloring rate of the first model according to the central position of the first model and the target area of the first image, wherein the coloring rate of the first model is lower than or equal to the coloring rate of the target area; rendering the first model according to the shading rate of the first model. The method has the beneficial effects that the corresponding coloring rate is set for the model, so that the coloring rate of partial areas in the image is reduced, and the power consumption is further reduced.

Description

Image rendering methods and related equipment

Technical field

The present application relates to the field of image processing technology, specifically, to an image rendering method and related equipment.

Background technique

When an electronic device renders an image, it includes coloring of the image. For example, after the program developer sets the shading rate through software algorithms and image post-processing technology, the shading rate is applied to the entire image. The graphics processing unit (GPU) of the electronic device can color each pixel of the image separately, thereby completing the coloring process of the entire image. However, as image pixels increase and rendering scenes in images become more and more complex, image shading processing will produce higher rendering loads on electronic devices, such as increasing computing power and power consumption during the rendering process.

Contents of the invention

This application provides image rendering methods and related equipment to solve the problems of increased computing power and increased power consumption during the rendering process.

In a first aspect, embodiments of the present application provide an image rendering method, which is applied to an electronic device. An application program is installed in the electronic device, and the application program issues a rendering instruction stream to render the first model in the first image. The method includes: Intercept specific instructions in the rendering instruction stream; obtain the center position of the first model according to the specific instruction, where the center position is the position of the center point of the first model in the first image; according to the center position of the first model and the first image The target area determines the shading rate of the first model, wherein the shading rate of the first model is lower than or equal to the shading rate of the target area; the first model is rendered according to the shading rate of the first model.

The image rendering method provided by the embodiment of the present application can set different shading rates for different areas in the image to be displayed, so that the electronic device can reduce the rendering workload of the electronic device by reducing the shading rate of some areas in the image. , memory and bandwidth. By reasonably setting different shading rates for different areas in the image to be displayed, while reducing power consumption and heat generation during the rendering process, the rendered image will not have a significant impact on the user's perception. This can reduce the power consumption and heat generation of electronic devices while improving user experience. Specifically, for the target area in the image that the user is concerned about, changes in the color accuracy of the model are easily noticed by the user, so a higher coloring rate can be used for the target area to obtain a high-precision coloring effect. For areas where user perception is not strong (such as areas outside the target area), changes in model color accuracy are not easily noticed by users, so a lower coloring rate can be used for rapid coloring in this part of the area to reduce power consumption. Furthermore, by improving the software framework of the electronic device, the electronic device obtains specific instructions in the rendering instruction stream, and can quickly and accurately determine the center position of the model according to the specific instructions, thereby quickly and accurately determining the center position of the model. and the relationship between the target area in the image, and then adaptively set the corresponding shading rate for the model based on the relationship between the model and the target area, providing guarantee for subsequent reduction of computing power and power consumption in the rendering process.

In one possible implementation, the specific instruction includes a first specific instruction, and obtaining the center position of the first model according to the specific instruction includes: obtaining the vertex data of each vertex of the first model and the MVP corresponding to the first model according to the first specific instruction. Matrix; determine the position of the center point of the first model in the first image based on the vertex data and MVP matrix.

By intercepting specific instructions, the vertex data and MVP matrix of the first model are obtained, and then the position of the center point of the first model in the first image is determined based on the vertex data and MVP matrix of the first model.

In one possible implementation, determining the shading rate of the first model based on the center position of the first model and the target area of the first image includes: obtaining the first distance based on the center position of the first model and the target area of the first image. , where the first distance is the distance between the center position of the first model and the target area in the observation space or clipping space; the shading rate corresponding to the first model is determined according to the first distance. The smaller the first distance is, the higher the shading rate of the first model is.

The relationship between the first model and the target area is determined by calculating a first distance between the center position of the first model and the target area. When the first distance is smaller, that is, the closer the first model is to the target area, the higher the possibility that it will receive the user's attention, and the corresponding shading rate can be set higher to obtain a high-precision shading effect. When the first distance is larger, that is, the first model is further away from the target area, the less likely it is to receive the user's attention, and its corresponding shading rate can be set lower to reduce power consumption.

In one possible implementation, determining the shading rate of the first model based on the center position of the first model and the target area of the first image includes: obtaining the category of the first model; obtaining the first distance, where the first distance is In the observation space or clipping space, the distance between the center position of the first model and the target area; the shading rate of the first model is determined according to the category of the first model and the first distance.

By referring to the category and the first distance of the first model, the shading rate of the first model can be set more reasonably, thereby improving the user experience.

In one possible implementation, determining the shading rate corresponding to the first model based on the category of the first model and the first distance includes: obtaining a specific object category corresponding to the application; when the category of the first model is a specific object category, The coloring rate corresponding to the specific object category is used as the coloring rate corresponding to the first model; when the category of the first model is not a specific object category, the coloring rate corresponding to the first model is determined based on the first distance.

The shading rate corresponding to the model category can be set according to different application scenarios or different game applications to improve the user experience.

In one of the possible implementations, the specific instruction includes a second specific instruction, and obtaining the category of the first model includes: obtaining the model information of the first model according to the second specific instruction; determining the first model based on the model information of the first model. category.

By intercepting the second specific instruction, the model information of the first model is obtained according to the second specific instruction, thereby determining the category of the first model.

In one of the possible implementations, the method further includes: determining the range of the target area according to the type of the application; determining the target area in the first image according to the range of the target area and the position of the target object in the first image, where the target Objects are relevant to the user's focus.

By setting the corresponding target area according to the user's focus, the shading rate of each model can be determined based on the target area. In this way, while reducing power consumption and heat generation during the rendering process, the rendered image will not affect the user's perception. Obvious impact, improving user experience.

In one possible implementation, when the shading rate of the first model is lower than the shading rate of the target area, rendering the first model according to the shading rate of the first model includes: rendering according to the highest shading rate and image quality enhancement algorithm target area and render the first model according to the first model's shading rate.

By increasing the highest shading rate and image quality enhancement algorithm to render the target area, it provides users with high-precision shading effects and image rendering effects. At the same time, rendering the first model based on a low shading rate can reduce power consumption.

In one possible implementation, the electronic device includes a central processing unit and a graphics processor, and rendering the first model according to the shading rate of the first model includes: the central processor calling a first variable speed corresponding to the shading rate of the first model. rate shading API, and issues a first model rendering instruction to the graphics processor according to the first variable rate shading API; the graphics processor renders the first model according to the shading rate of the first model in response to the first model rendering instruction.

By calling the first variable rate shading API corresponding to the shading rate of the first model, it is ensured that the first model is rendered according to the shading rate of the first model.

In one possible implementation, the first image further includes a second model, and the method further includes: obtaining a second distance according to the center position of the second model and the target area of the first image, where the second distance is In space or clipping space, the distance between the center position of the second model and the target area; determine the shading rate corresponding to the second model based on the second distance, where the second distance is greater than the first distance, and the shading rate of the second model is low Renders the second model based on the shading rate of the first model; renders the second model based on the shading rate of the second model.

The second distance is smaller than the first distance. Correspondingly, the set shading rate of the second model is higher than the shading rate of the first model. That is, the model closer to the target area has a higher shading rate, and the model farther away from the target area has a lower shading rate.

In one possible implementation manner, rendering the second model according to the shading rate of the second model includes: the central processor calls a second variable rate shading API corresponding to the shading rate of the second model, and renders the second model according to the second variable rate. The shading API issues a second model rendering instruction to the graphics processor; the graphics processor responds to the second model rendering instruction and renders the second model according to the shading rate of the second model.

Different shading rates correspond to different shading rate APIs. For example, when the first distance and the second distance distribution belong to different distance ranges, their corresponding shading rates are different, thus calling different shading rate APIs.

In a second aspect, embodiments of the present application provide an electronic device. The electronic device includes one or more processors and one or more memories; the one or more memories are coupled to the one or more processors, and the one or more memories Computer instructions are stored; when one or more processors execute the computer instructions, the electronic device is caused to perform any of the above image rendering methods.

In a third aspect, embodiments of the present application provide a computer-readable storage medium. The computer-readable storage medium includes computer instructions. When the computer instructions are run, any one of the above image rendering methods is executed.

In the fourth aspect, embodiments of the present application provide a chip system. The chip system includes a processor and a communication interface; the processor is configured to call and run the computer program stored in the storage medium from the storage medium, and execute any of the above image rendering methods. .

In addition, the technical effects brought about by the second to fourth aspects can be found in the descriptions related to the respective designed methods in the above method section, and will not be described again here.

Description of the drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present application and therefore should not be viewed as The drawings only limit the scope. For those of ordinary skill in the art, other relevant drawings can be obtained based on these drawings without exerting creative efforts.

Figure 1 is a schematic diagram of the process of image rendering by an electronic device according to the rendering instruction stream of an application provided by an embodiment of the present application;

Figure 2A is a schematic diagram of the main scene of the game provided by the embodiment of the present application;

Figure 2B is a schematic diagram of the game character after movement in the main scene of the game in Figure 2A;

Figure 3 is a schematic diagram of the software structure of the electronic device provided by the embodiment of the present application;

Figure 4 is a schematic flow chart of an image rendering method provided by an embodiment of the present application;

Figure 5 is a schematic diagram of the interception rendering instruction flow provided by the embodiment of the present application;

Figure 6 is a schematic diagram of calculating the center position of the first model provided by the embodiment of the present application;

Figure 7 is a schematic flow chart of another image rendering method provided by an embodiment of the present application;

Figure 8 is a schematic flow chart of another image rendering method provided by an embodiment of the present application;

Figure 9 is a schematic structural diagram of a chip system provided by an embodiment of the present application.

Detailed ways

As used herein, the terms "comprises," "comprises," or any other variation thereof are intended to cover a non-exclusive inclusion such that a process, method, article, or apparatus that includes a list of elements includes not only those elements, but also includes not expressly Other elements listed may also include elements inherent to such process, method, article or apparatus. Without further limitation, an element defined by the statement "comprises a" does not exclude the presence of additional identical elements in a process, method, article, or apparatus that includes that element.

"And/or" in this application describes the relationship between associated objects, indicating that there can be three relationships. For example, A and/or B can mean: A exists alone, A and B exist simultaneously, and B exists alone, where A and B can be singular or plural. The terms "first", "second", "third", "fourth", etc. (if present) in the description, claims and drawings of this application are used to distinguish similar objects, rather than to Describe a specific order or sequence.

The image rendering method provided by the embodiment of the present application can be executed by an electronic device. The electronic device may be a terminal device, and the terminal device may also be called a terminal (terminal), user equipment (UE), mobile station (MS), mobile terminal (MT), etc. Terminal devices can be smartphones, computers, smart TVs, personal digital assistants (PDAs), wearable devices, augmented reality (AR)\virtual reality (VR) devices, media players, etc. Portable mobile devices. The electronic device may also be a vehicle-mounted device, an Internet of Things device, or other device capable of image rendering processing. The electronic device may be a device running Android system, IOS system, Windows system or other operating systems. The embodiments of this application do not specifically limit the type of electronic device and the operating system run by the electronic device.

Please refer to Figure 1 to exemplarily introduce the process of image rendering by an electronic device according to the rendering instruction flow of the application program. The electronic device 100 includes a central processing unit (CPU) 101, a graphics processor 102, a memory 103, and a display 104. The central processing unit 101 is used to run application programs 111 and operating system 112. The application program 111 may be a game application, a video application (video player), etc. Operating system 112 provides graphics libraries and image synthesis modules. The image synthesis module is used for the synthesis of two-dimensional or three-dimensional images. The rendering pipeline (Renderingpipeline) in the graphics processor 102 is a series of operations performed sequentially by the graphics processor 102 in the process of rendering graphics or images. The rendering pipeline includes but is not limited to the following operations: vertex processing, primitive processing (primitive processing), rasterization and fragment processing. The memory 103 includes one or more of internal memory (ie, memory), cache, graphics card memory (Video Memory), and external memory.

When the application program 111 needs to render an image, it issues a rendering instruction stream. The central processing unit 101 calls an application programming interface (API) in the graphics library according to the rendering instruction stream issued by the application program 111, so as to instruct the graphics processor 102 to perform the corresponding rendering operation. The graphics library generates a stream of instructions that the rendering pipeline can recognize based on the API it calls. The graphics processor 102 receives the instruction stream sent by the graphics library and performs rendering through the rendering pipeline. The rendering result after the graphics processor 102 performs rendering can be stored in the memory 103 (such as a video memory) in the electronic device 100 . The image synthesis module of the operating system 112 obtains the rendering results from the memory 103 , synthesizes the rendering results, and displays them on the display 104 .

More and more applications, such as game applications or video applications, need to display scene-rich images on electronic devices. These images are usually rendered by electronic devices based on models. Taking the application as a game application as an example, the main scene displayed on the monitor by the game application is as shown in Figure 2A. The main scene includes various models, such as character models, stone models, grass models, etc. To provide beautiful images, images are usually rendered at a higher shading rate. A higher shading rate corresponds to a smaller number of pixels being shaded simultaneously than a lower shading rate. For example, comparing the shading rate of 1×1 and the shading rate of 2×2, then the shading rate of 2×2 can be a lower shading rate, the shading rate of 1×1 can be a higher shading rate, and the shading rate of 1×1 can be a higher shading rate. The shading rate means that the pixel color is calculated once for each pixel, and the 2×2 shading rate means that the pixel color is calculated once for 4 pixels. In other words, when rendering an image at a higher shading rate, the pixel color will be calculated multiple times, and then the shading process will be performed multiple times, thereby increasing the computing power of the electronic device during the rendering process, increasing the memory overhead, and causing greater harm to the electronic device. Large power consumption causes electronic devices to heat up or drop frames, affecting user experience.

When implementing the embodiments of the present application, the inventor found that in some scenarios, such as when the user uses a game application or watches a video, the area that the user pays attention to will be relatively concentrated. The character model shown in Figure 2A is a game character controlled by the player, and the player often pays attention to the area where the game character is located. As shown in Figure 2B, as the game character moves, the content of the image will also change, but the player's focus is always on the game character, which is the center area of the image. On the contrary, the player focuses on the edges of the image. The area perception is not strong, such as the stone model and grass model in the image.

In view of this, embodiments of the present application provide an image rendering method and related equipment, which can set different shading rates for different areas in the image to be displayed, so that the electronic device can reduce the shading rate of some areas in the image. Rendering workload, memory, and bandwidth for this area. By reasonably setting different shading rates for different areas in the image to be displayed, while reducing power consumption and heat generation during the rendering process, the rendered image will not have a significant impact on the user's perception. This can reduce the power consumption and heat generation of electronic devices while improving user experience. Specifically, for the target area in the image that the user is concerned about, changes in the color accuracy of the model are easily noticed by the user, so a higher coloring rate can be used for the target area to obtain a high-precision coloring effect. For areas where user perception is not strong (such as areas outside the target area), changes in model color accuracy are not easily noticed by users, so a lower coloring rate can be used for rapid coloring in this part of the area to reduce power consumption. Furthermore, by improving the software framework of the electronic device, the electronic device obtains specific instructions in the rendering instruction stream, and can quickly and accurately determine the center position of the model according to the specific instructions, thereby quickly and accurately determining the center position of the model. and the relationship between the target area in the image, and then adaptively set the corresponding shading rate for the model based on the relationship between the model and the target area, providing guarantee for subsequent reduction of computing power and power consumption in the rendering process.

The solutions provided by the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

The software system of the above-mentioned electronic device can adopt a layered architecture, an event-driven architecture, a microkernel architecture, a microservice architecture or a cloud architecture. The following embodiments of the present application take the Android system with a layered architecture as an example to illustrate the software structure of the electronic device. Of course, in other operating systems, as long as the functions implemented by each functional module are similar to those of the embodiments of the present application, the embodiments of the present application can also be implemented.

Please refer to Figure 3 for an exemplary introduction to the software structure of the electronic device provided by the embodiment of the present application.

The layered architecture divides the software into several layers, and each layer has clear roles and division of labor. The layers communicate through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: application layer, application framework layer, system library and hardware layer.

The application layer can include a series of application packages. Application packages can include game applications, video applications, and other applications that need to display pictures or videos to users by rendering images.

The application framework layer can provide application programming interfaces and programming frameworks for applications in the application layer. The application framework layer includes some predefined functions.

For example, the application framework layer may include a window manager, a content provider, a view system, a resource manager, an activity manager, an input manager, etc. The window manager provides a window management service (Window Manager Service, WMS), which can be used for window management, window animation management, surface management, and as a transfer station for the input system. Content providers are used to store and retrieve data and make this data accessible to applications. This data can include videos, images, audio, etc. The view system includes visual controls, such as controls that display text, controls that display pictures, etc. A view system can be used to build applications. The display interface can be composed of one or more views. For example, a display interface including a text message notification icon may include a view for displaying text and a view for displaying pictures. The resource manager provides various resources to applications, such as localized strings, icons, pictures, layout files, video files, etc. The activity manager can provide activity management service (AMS) to manage the life cycle of each application and the navigation fallback function. AMS can be used for the startup, switching, and scheduling of system components (such as activities, services, content providers, and broadcast receivers) as well as the management and scheduling of application processes. The input manager can provide input management service (Input Manager Service, IMS), and IMS can be used to manage system input, such as touch screen input, key input, sensor input, etc. IMS takes out events from the input device node and distributes the events to appropriate windows through interaction with WMS.

In this embodiment of the present application, one or more functional modules may be set in the application framework layer to implement the image rendering method provided by the embodiment of the present application. For example, an interception module and a determination module can be set in the application framework layer. Among them, the interception module is used to intercept specific instructions in the rendering instruction stream issued by the application, where the rendering instruction stream is used to instruct the model in the rendered image. The determine module is used to determine the shading rate of the model. In other embodiments, the application framework layer may also include a computing module. The calculation module is used to calculate the center position of the model. In other embodiments, the application framework layer may also include an identification module. The identification module is used to identify the category of the model. The determining module may also be used to determine the shading rate corresponding to the model according to one or both of the center position of the model and the category of the model. In subsequent examples, the functions of each of the above modules will be explained in detail.

The interception module, calculation module, determination module and identification module may each be a program run on the electronic device to implement corresponding functions. In other embodiments, the identification module may identify the category of the current rendering model. For example, perform offline analysis on the game model categories in advance and record the corresponding characteristics. The identification module performs feature analysis on the data intercepted during the real-time running of the game and matches it with the features obtained from offline analysis to determine the category of the model.

System libraries can include multiple functional modules. For example, graphics library, surface manager, etc.

The graphics library is also called the drawing library. The graphics library is used to define cross-programming language and cross-platform application programming interfaces, which contains many functions for processing graphics (images). Take the Open Graphics Library (OpenGL) as an example. The API defined by OpenGL includes interfaces for implementing corresponding functions, for example, an interface for drawing two-dimensional images or three-dimensional images. The interface includes drawing functions, such as Drawing function glDrawElements(). As another example, an interface is used to present an image drawn by a drawing function to a display interface (display). The interface includes a rendering function, such as the function eglSwapBuffers()). The embodiments of this application will not give examples one by one here.

Among them, functions in OpenGL can be called through instructions. For example, drawing functions can be called through instructions in the rendering instruction stream to draw two-dimensional images or three-dimensional images. The instructions in the rendering instruction stream are instructions written by developers based on functions in the graphics library when developing game applications, and are used to call the interface of the graphics library corresponding to the instructions.

Among them, graphics libraries include but are not limited to: open graphics library forembedded system (OpenGL ES), the khronos platform graphics interface (the khronos platform graphics interface) or Vulkan (a cross-platform graphics application program interface) and other graphics libraries.

The surface manager is used to manage the display subsystem and provides the fusion of 2D and 3D layers for multiple applications.

In this embodiment of the present application, the graphics library (such as OpenGL ES) in the system library can provide extended functions of variable rate shading. Then, when the electronic device needs to perform variable rate shading for a certain Drawcall, it can call the variable rate shading API in OpenGL ES to implement variable rate shading of the current Drawcall. For example, the electronic device can use a lower rate (such as 2×1, 2×2, 4×2, 4×4, etc.) to shade the current Drawcall, thereby reducing the overhead caused by shading the current Drawcall.

The hardware layer may include a processor (such as a central processing unit, a graphics processor, etc.), components with storage functions (such as memory), and components with display functions (such as a display). For relevant contents of the central processing unit, graphics processor and memory, please refer to Figure 1 above. The central processing unit and graphics processor in electronic equipment can be located on the same chip, or they can be separate chips.

In some embodiments, the central processing unit can be used to control each module in the application framework layer to implement their respective functions, and the graphics processor can be used to control the graphics library called according to the instructions processed by each module in the application framework layer. (such as OpenGL ES) performs the corresponding rendering processing.

In order to explain more clearly the functions of each layer in the software architecture provided by the embodiments of this application, the following takes image rendering in a game application as an example to illustrate the functional implementation of each component with the software components shown in Figure 3.

Please refer to Figure 4 for an exemplary introduction to the image rendering method flow provided by the embodiment of the present application.

Step S40: The game application issues a rendering instruction stream.

In this embodiment of the present application, the application program installed in the electronic device can issue a rendering instruction stream in response to the user's operation. For example, a game application can start a game, load a game, or run a game in response to a user's operation. When the game starts, loads, or runs, the game application sends a rendering instruction stream to the central processor of the electronic device to instruct the electronic device to render an image.

One rendering instruction stream corresponds to one frame of image, and the rendering instruction stream can be used to instruct one or more models in one frame of image to be rendered. Rendering for a model in an image can be issued through one Drawcall, and the rendering instruction stream can include one or more Drawcalls. For example, when a game application needs to render a main scene image, the game application issues a rendering instruction stream for rendering the main scene image to the central processor. The rendering instruction stream may include character models, house models, and tree models for the main scene image. One or more renderings, that is, the rendering instruction stream includes one or more of a Drawcall instructing to render a character model, a Drawcall instructing to render a house model, and a Drawcall instructing to render a tree model.

In this embodiment of the present application, the rendering instruction stream issued by the application program is used to instruct the rendering of the first model in the first image as an example. That is, the rendering instruction stream includes a Drawcall instructing the rendering of the first model in the first image. In other embodiments, the rendering instruction stream may be used to instruct rendering of two or more models (such as a first model and a second model) in the first image, for other objects other than the first model in the first image. For the rendering processing of the model, please refer to the first model and will not be described in detail here.

In this embodiment of the present application, the rendering instruction stream may include one or more instructions (ie, functions), and the Drawcall indicating rendering of the first model may include one or more instructions. Taking the graphics library in an electronic device as OpenGL as an example, the rendering instruction stream may include one or more of the following instructions (functions): glGenBuffer instruction, glBindBuffer instruction, glBufferData instruction, glBufferSubData instruction, glMapBuffer instruction, glMapBufferRange instruction, glUseProgram instruction and the glDrawElement directive.

Among them, the glGenbuffer instruction can be used to create a buffer. That is, one or more storage spaces are divided into the memory (such as memory) of the electronic device, and each storage space can have an identification (ID). The divided storage space can be used to store various related data during the rendering process. For example, some buffers can be used to store the vertex coordinates of the model to be rendered (such as the first model), and some buffers can be used to store the model-view-projection (MVP) matrix corresponding to the model to be rendered, etc.

Among them, the MVP matrix includes three matrices: Model matrix (M matrix), View matrix (V matrix), and Projection matrix (P matrix). The M matrix is used to convert local coordinates in local space to world coordinates in world space. The V matrix is used to convert world coordinates in world space to observation coordinates in observation space. The P matrix is used to convert observation coordinates in observation space into clipping coordinates in clipping space.

Among them, the glBindbuffer instruction can be used to bind the buffer. By binding the buffer with this instruction, subsequent rendering operations can use the corresponding buffer. As an example, the created buffer includes buffer A and buffer B. Through glBindbuffer(A), when the electronic device performs subsequent rendering operations such as writing data (such as vertex coordinates), the electronic device can write the vertex coordinates to buffer A for storage.

Among them, the glBufferdata instruction can be used to transfer data. For example, if the data carried by the glBufferdata instruction is not empty (NULL), the electronic device can store the data (or data pointer) carried by the glBufferdata instruction into the bound buffer. For example, when the glBufferdata instruction carries vertex coordinates, the electronic device can store the vertex coordinates in the bound frame buffer. For another example, when the glBufferdata instruction carries the index of the vertex coordinate, the electronic device can store the vertex coordinate index into the bound frame buffer. Similar to the glBufferdata instruction, the glMapBuffer instruction and glMapBufferRange instruction can also be used to transfer data. For example, the glMapBuffer instruction can be used to map data in a buffer object to the address space in an electronic device. In this way, when the GPU needs to use this data, it can be read directly from the address space in the electronic device. Different from the glMapBuffer instruction, the glMapBufferRange instruction can map a part of specific data to the address space of the electronic device to facilitate subsequent GPU calls.

Among them, the glBufferSubData instruction can be used to update data. For example, a game application can use the glBufferSubData instruction to update part or all of the vertex coordinates, thereby instructing the electronic device (such as the GPU of the electronic device) to draw and render based on the new vertex coordinates.

Among them, the glUseProgram instruction uses the program object program as part of the current rendering state.

Among them, the glDrawElement instruction can be used to draw three-dimensional images, which can draw a triangle composed of four vertices. This instruction includes four parameters: mode (used to specify the drawing mode), count (used to specify the number of vertices to draw), type (data type used to indicate the vertex index) and indices (used to point to the index array pointer) , that is, it uses count elements to define a geometric sequence, and the index values of these elements are stored in the indices array.

In some embodiments, the rendering instruction stream may also include shader-related instructions, such as glLinkProgram (link shader object instruction), glShaderSource (associate shader instruction), glCompileShader (shader compilation instruction), and glUseProgram (shader call instructions) and other instructions. The embodiments of this application do not specifically limit the instructions included in the rendering instruction stream.

In this embodiment of the present application, the rendering instruction stream may include different contents, which is not specifically limited. For example, when the video application in the application layer needs to render a certain image (such as the Nth frame image, that is, the first image), the rendering instruction stream can be initiated. The rendering instruction stream may include various relevant data in the rendering process, such as the vertex data of the model that needs to be rendered in the first image, the MVP matrix (i.e., M matrix, V matrix, and P matrix) and one or more drawing elements ( Drawelement), the vertex data may be used to indicate the vertex coordinates of the vertices of the model to be rendered, and the vertex coordinates may be coordinates based on local space. In some implementations, the rendering instruction stream may also include the frame buffer that needs to be called to render the Nth frame image, and the rendering operations that need to be performed in each frame buffer. For example, the Nth frame image needs to use FB0 and FB1 as frame buffers. The rendering instruction stream may include the bindFrameBuffer(1) instruction, thereby binding the current rendering operation to FB1. After binding FB1, you can use the glDrawElement instruction to implement rendering operations on the FB1.

Step S41: The interception module intercepts the first specific instruction in the rendering instruction stream to obtain vertex data of each vertex of the first model and the MVP matrix corresponding to the first model.

In the embodiment of the present application, the interception module can intercept specific instructions in the rendering instruction stream based on a hook function to obtain the center position of the first model in the first image based on the specific instructions. Hook functions refer to various technologies that modify or extend the behavior of operating systems, applications or other software components by intercepting function calls, message passing, and event passing between software modules. In other words, the hook function is the code used to handle intercepted function calls, events, and messages.

In this embodiment of the present application, the above-mentioned specific instructions include the first specific instructions. The first specific instruction is used to obtain relevant data that can be used to calculate the center position of the first model in the first image. The relevant data may include, for example, vertex data of each vertex of the first model and the MVP matrix corresponding to the first model.

In this embodiment of the present application, the vertex data at least includes the vertex coordinates of each vertex and the index data of the vertex coordinates. The vertex coordinates may be local coordinates based on the local space.

In this embodiment of the present application, the MVP matrix corresponding to the first model includes: three matrices: the M matrix, the V matrix, and the P matrix required to convert the local coordinates of the first model into cropping coordinates (or screen coordinates).

Specifically, the first specific instruction includes a first instruction carrying a first parameter and/or a second instruction carrying a second parameter.

Wherein, the first parameter may be a parameter indicating a parameter used to transmit vertex data related to the vertex. The first parameter may include targets such as GL_ELEMENT_ARRAY_BUFFER, GL_ARRAY_BUFFER, and corresponding buffer objects.

It can be understood that the buffer in OpenGL is a memory area. The buffer is bound to a specific buffer object, which means that the buffer is given a corresponding specific meaning. For example, when a buffer object is bound to GL_ARRAY_BUFFER, the buffer corresponding to the buffer object is used to store vertex data, such as vertex coordinates, normals, colors, texture coordinates, etc. When a buffer object is bound to GL_ELEMENT_ARRAY_BUFFER, the buffer corresponding to the buffer object is the index data used to store vertex data.

Taking the first instruction as an example, the first instruction includes but is not limited to the following instructions: glGenBuffers instruction, which is used to generate one or more buffer objects and return their identifiers. glBindBuffer instruction, which is used to bind a buffer object to the specified target. The glBufferData directive allocates data to a buffer bound to a specified target and sets how it is used. glBufferSubData directive, which is used to update a portion of the data in the buffer bound to the specified target. glVertexAttribPointer instruction, which is used to define vertex attributes, such as vertex coordinates, normals, colors, etc. glEnableVertexAttribArray directive, which is used to enable specified vertex attributes.

The second parameter may be a parameter used for MVP matrix transmission. The second parameter may include targets such as GL_UNIFORM_BUFFER and corresponding buffer objects. The second instructions include but are not limited to the following instructions: glBufferSubData instruction, glGenbuffer instruction, glBindbuffer instruction, glBufferdata instruction, etc.

In some embodiments, the first instruction and the second instruction may be the same instruction, that is, the first specific instruction may be an instruction carrying both the first parameter and the second parameter, such as the glBufferSubData instruction.

In the embodiment of the present application, the interception module intercepts the target and buffer object in the first specific instruction to implement the interception of vertex data and MVP matrix. Specifically, when intercepting vertex data, the interception module Hook first specific instruction intercepts the target GL_ELEMENT_ARRAY_BUFFER, GL_ARRAY_BUFFER and the buffer object, and further intercepts and obtains data (data) based on the target and buffer object and buffers them. When intercepting the MVP matrix: The interception module first analyzes the vertex shader (VS shader) corresponding to the first model to determine the variable (Uniform) name of the MVP matrix corresponding to the first model. For example, the M matrix is Primitive_LocalToWorldTranslated and the VP matrix is View_TranslatedWorldToClip. Secondly, determine the value transfer method of the MVP matrix. Finally, obtain the value of the corresponding MVP matrix.

Please refer to Figure 5 together, taking the rendering instruction stream issued by the application including glGenBuffer, glBindBuffer, glBufferData, glBufferSubData, glUseProgram and glDrawElement as an example. The first specific instruction includes a first instruction and a second instruction.

Take the interception module intercepting the following first instruction as an example:

Intercepting glGenbuffers(1, buffer111) means creating a buffer object buffer111 for vertex data, and buffer111 points to a buffer.

Intercepting glBindbuffer(GL_ARRAY_BUFFER, buffer111) means binding the buffer object buffer111 to the target GL_ARRAY_BUFFER.

Intercepting glBufferdata(GL_ARRAY_BUFFER, data1) means writing data1 to the bound buffer. That is, after the buffer object buffer111 and the target GL_ARRAY_BUFFER have been bound, the buffer corresponding to buffer111 can be accessed through the GL_ARRAY_BUFFER target, and then data1 is written. Enter the buffer corresponding to Buffer111;

Intercepting glBufferSubData(GL_ARRAY_BUFFER, data2) means updating the data in the bound buffer to data2.

Among them, data1 and data2 can include vertex data, such as vertex coordinates, normal vectors of vertices, etc.

Take the interception module intercepting the following second command as an example:

Intercepting glGenbuffers(1, buffer111) means creating a buffer object buffer111 for a unified variable (such as MVP matrix).

Intercepting glBindbuffer(GL_UNIFORM_BUFFER, buffer111) means binding the buffer object buffer111 to the target GL_UNIFORM_BUFFER.

Intercepting glBufferdata(GL_UNIFORM_BUFFER, data3) means writing data3 to the bound buffer. That is, after the buffer object buffer111 and the target GL_UNIFORM_BUFFER have been bound, the buffer corresponding to buffer111 can be accessed through the GL_UNIFORM_BUFFER target, and then data3 is written. The buffer corresponding to buffer111;

Intercepting glBufferSubData(GL_UNIFORM_BUFFER, data4) means updating the data in the bound buffer to data4. Among them, data3 and data4 can include MVP matrices.

For example, use a GPU debugging tool (such as renderDoc) to intercept a frame of image in the game. Taking the little yellow duck model (that is, the first model) in this frame of image (that is, the first image) as an example, we will introduce in detail how to intercept the small yellow duck. The vertex data and MVP matrix of the yellow duck model.

Intercept vertex data:

As shown in Table 1, the drawing model corresponding to Draw No. 56 is a little yellow duck model.

Table 1

Instruction ID(EID) Draw# Name … … … 978 55 glDrawElements(4329) 990 56 glDrawElements(1560) 1002 57 glDrawElements(1563) … … …

For example, the instruction sequence corresponding to the little yellow duck model is the instruction sequence with ID 979 to ID 990 shown in Table 2.

Table 2

The interception module obtains the BufferID corresponding to the little yellow duck model by intercepting the glBindBuffer instruction, and intercepts the glBufferrData or glBufferSubData instruction based on the BufferID. Obtain the vertex data of the little yellow duck model by intercepting the glBufferrData or glBufferSubData command. The interception module intercepts the glVertexAttribPointer instruction to obtain the corresponding parameters, and parses the vertex data data in the buffer based on the parameters to obtain the vertex data. Among them, the parameters in the glVertexAttribPointer instruction represent the organization form of the vertex data in the buffer. The interception module parses the vertex data in the buffer through the intercepted parameters such as index, step size, and data offset.

Interception MVP matrix:

For example, the vertex shader corresponding to the little yellow duck is shader 2039. By analyzing shader 2039, it can be determined that the variable name of the M matrix is Primitive_LocalToWorldTranslatedvb1, and its corresponding variable is stored in Buffer 10626. The variable name of the VP matrix is View_TranslatedWorldToClipvb0, and its corresponding variable is stored in Buffer 2832. The interception module intercepts Buffer10626 and Buffer 2832 and parses them to obtain MVP data.

Step S42: The interception module stores the obtained vertex data and MVP matrix into the memory.

In this embodiment of the present application, the interception module is also used to back up and store intercepted specific data. For example, the interception module stores the obtained vertex coordinates and MVP matrix into a memory, which can be a memory, etc. For example, the electronic device may store these data in an area of memory that the CPU can recall. As a result, the memory of the electronic device can store vertex data (such as vertex coordinates) and MVP matrices that may be used in the subsequent rendering process. It can be understood that the rendering instruction stream issued by the native application is used to transmit data to the storage area that the GPU can call. Therefore, through the backup storage in this example, the CPU can also have an interception module to intercept The ability to call the data (such as vertex data and MVP matrix) ensures the implementation of subsequent solutions, such as calculating the center position of the first model.

Step S43: The calculation module obtains the vertex data and MVP matrix, and calculates the center position of the first model based on the vertex data and MVP matrix.

After the interception module backs up and stores the intercepted data in step S42, the computing module can obtain the data intercepted by the interception module from the storage (such as memory), such as vertex data and MVP matrix.

In the embodiment of the present application, the center position of the first model may be: in the observation space or the cropping space, the position of the center point of the first model in the first image. The center position may be the observation coordinates, cropping coordinates, or screen coordinate.

In the embodiment of this application, the calculation module obtains the vertex data and MVP matrix intercepted by the interception module, and then calculates the position of the first model in the local space based on the vertex coordinates in the vertex data and the idea of AABB (Axis-aligned bounding box) bounding box. The center coordinate V1: {X, Y, Z}, where X=(x_min+x_max)/2, Y=(y_min+y_max)/2, Z=(z_min+z_max)/2, where, y_min, y_max, z_min and z_max are all the local coordinates of the first model in the local space obtained according to the vertex data. x_min is the minimum value of the local coordinates of the first model in the x direction. x_max is the maximum value of the local coordinates of the first model in the x direction. y_min is the minimum value of the local coordinates of the first model in the y direction. y_max is the maximum value of the local coordinates of the first model in the y direction. z_min is the minimum value of the local coordinates of the first model in the z direction. z_max is the maximum value of the local coordinates of the first model in the z direction. Finally, the calculation module changes the center coordinate V1 of the first model in the local space through the MVP matrix to obtain the position of the center point of the first model in the first image. The calculation process is: P*V*M*V1.

The following takes the center position as the screen coordinates as an example to explain in detail the process of the calculation module calculating the center position of the first model based on the vertex coordinates and the MVP matrix.

Referring to FIG. 6 , the first image 60 presented on the display of the electronic device includes objects 61 , 62 , 63 and 64 . The objects 61 , 62 , 63 and 64 correspond to the rendering instructions issued by the game application. The models indicated for rendering in the stream are the first model 611, the second model 621, the third model 631 and the fourth model 641 as examples. The following uses the calculation of the center position of the first model 611 as an example. The calculation of the center positions of other models can refer to the first model 611 and will not be described again here.

The game application may carry the vertex coordinates of each vertex of the first model 611 in the coordinate system of the local space in the rendering instruction stream issued to the first model 611 of the first image 60. The vertex coordinates of each vertex are the local coordinates. As mentioned above, the calculation module calculates the center coordinate V1 of the first model 611 based on the vertex coordinates of each vertex of the first model 611 and the bounding box in the coordinate system of the local space. The center coordinate V1 is the local coordinate of the local space.

The calculation module can convert the center coordinate V1 of the first model 611 in the local space into the coordinate in the world space through the M matrix corresponding to the first model 611 issued by the game application to obtain the center coordinate V2. The center coordinate V2 is the world coordinate of the center point of the first model in world space. As shown in FIG. 6 , after the first model 611 is converted to the coordinates of the world space, the positions of the second model 621 , the third model 631 and the fourth model 641 in the world space can be seen.

After the calculation module obtains the center coordinate V2 of the center point of the first model 611 in the world space, the calculation module can calculate the center point of the first model 611 in the world space according to the V matrix corresponding to the first model 611 issued by the game application. The center coordinate V2 in is converted to the coordinate in the observation space to obtain the center coordinate V3. The center coordinate V3 is the observation coordinate of the center point of the first model 611 in the observation space. As shown in FIG. 6 , after the first model 611 is converted to the coordinates of the observation space, the positions of the second model 621 , the third model 631 and the fourth model 641 in the observation space can be seen.

In this example, the coordinate space corresponding to the camera position can be called the observation space. As an example, take the camera set in the positive direction of the y-axis in world space. As shown in FIG. 6 , since the camera is located in the positive direction of the y-axis and shoots downward, the first to fourth models 611 to 641 corresponding to the observation space can be presented as a bird's-eye view.

After the calculation module obtains the center coordinate V3 of the center point of the first model 611 in the observation space, the calculation module can calculate the center point of the first model 611 in the observation space according to the P matrix corresponding to the first model 611 issued by the game application. The center coordinate V3 in is converted to the coordinate in the clipping space to obtain the center coordinate V4. The center coordinate V4 is the clipping coordinate of the center point of the first model in the clipping space. As shown in FIG. 6 , after the first model 611 is converted to the coordinates of the clipping space, the positions of the second model 621 , the third model 631 and the fourth model 641 in the clipping space can be seen.

Through the above-mentioned MVP matrix transformation (ie, M matrix transformation, V matrix transformation, and P matrix transformation), the electronic device can obtain the coordinates of the center points of each model displayed on the display (ie, cropping coordinates). Then, the electronic device can also transform the cropping coordinates into screen coordinates. For example, use the viewport transformation (Viewport Transform) to transform the center coordinate V4 located in the range of -1.0 to 1.0 to the coordinates within the coordinate range defined by the glViewport command, and obtain Center coordinate V5. The center coordinate V5 is the screen coordinate of the center point of the first image projected onto the screen of the electronic device. The finally transformed center coordinate V5 is the center position of the first model. The center coordinate V5 will be sent to the rasterizer to obtain display data corresponding to each pixel. Based on these display data, the electronic device can control the display to perform corresponding display. As shown in FIG. 6 , the first model 611 is displayed as the object 61 on the display, and the center coordinate V5 of the first model 611 corresponds to the coordinate P1 of the center of the object 61 .

Similarly, when the game application issues the rendering instruction stream for the second model 621 to the fourth model 641 in the first image 60, through the above-mentioned MVP transformation and viewport transformation, the electronic device can also obtain the second The screen coordinates of the respective center points of the model to the fourth model on the screen. The second model 621 to the fourth model 641 are displayed on the display as objects 62 to 64. The center positions of the second model to the fourth model respectively correspond to the object 62. The coordinates P2 of the center, the coordinates P3 of the center of the object 63 , and the coordinates P4 of the center of the object 64 . In some embodiments, the calculation module hooks instructions related to graphics drawing to perform calculation of the center position of the model before the instructions call the GPU driver. As shown in Figure 5, the interception module also intercepts the DrawElements instruction to calculate the center position of the first model through the calculation module before the DrawElements instruction calls the GPU driver.

In other embodiments, after intercepting data related to a specific instruction (such as a first specific instruction), the interception module stores the intercepted data into the memory and stores a state data. This status data is used to indicate status information related to the data intercepted by the interception module. Taking the first specific instruction as an example, if the status data is "ok", it indicates that the interception module has intercepted relevant data in the first specific instruction, such as vertex data and MVP matrix, and the intercepted data has been stored in the memory. When the computing module reads the status data as "ok" from the memory, the computing module can continue to read the vertex data and MVP matrix intercepted by the interception module from the memory. When the computing module reads the status data from the memory that is not "ok", the computing module does not read the data intercepted by the interception module from the memory.

In other embodiments, the interception module can trigger the computing module to obtain status data from the memory before the DrawElements instruction calls the GPU driver. When the status data is "ok", the computing module obtains the data intercepted by the interception module from the memory.

It can be understood that in different implementations of the present application, the triggering mechanism for the calculation module to calculate the center position of each model may be different. As long as the module can know the center position corresponding to the drawelement before sending the corresponding drawelement to the GPU. For example, in this example, the calculation module's calculation of the center position of each model can be triggered by the interception module intercepting drawelement.

Step S44: The calculation module sends the center position of the first model to the determination module.

As shown in the example shown in Figure 6, the electronic device determines the center position as the screen coordinates, the calculation module uses the calculated center coordinate V5 as the center position of the first model, and transmits the center coordinate V5 to the determination module. It can be understood that when the electronic device determines that the center position is the cropping coordinate, the calculation module uses the calculated center coordinate V4 as the center position of the first model, and transmits the center coordinate V4 to the determination module.

In this embodiment of the present application, during the image rendering process, in addition to determining the vertex coordinates of each model, the electronic device also needs to color each pixel in the current first image, that is, determine the color data of each pixel. Thus, according to the color data of each pixel, the display is controlled to display the corresponding color at the corresponding pixel position.

In the embodiment of the present application, when determining the color data of each pixel in the first image, the coloring rate of the area where the model is located is determined based on the model in the first image. The calculation module sends the calculated center position of the first model to the determination module, so that the determination module decides the shading rate of the first model according to the center position of the first model. Correspondingly, the calculation module may also send the center position of the second model in the first image to the determination module, so that the determination module determines the shading rate of the second model based on the center position of the second model.

Step S45: The determining module determines the coloring rate of the first model according to the center position of the first model.

In this embodiment of the present application, the determination module obtains the first distance based on the center position of the first model and the target area of the first image, and determines the shading rate corresponding to the first model based on the first distance. Wherein, the smaller the first distance is, that is, the closer the center position of the first model is to the target area, the higher the shading rate of the first model is. Correspondingly, the larger the first distance is, that is, the further away the center position of the first model is from the target area, the lower the shading rate of the first model is. The shading rate of the first model is lower than or equal to the shading rate of the target area.

In this embodiment of the present application, the target area of the first image may be the center area of the first image, or may be the area where the target model is located in the first image, such as the area where the game character controlled by the player is located. In other embodiments, the user's focus on the first image can be determined by mining the user's click, touch, or hover selection behaviors, and then the area where the user's focus on the first image is located is determined as the target area. , the embodiment of the present application does not specifically limit the setting method of the target area of the first image.

In the embodiment of the present application, the coordinates corresponding to the target area of the first model may be: in the observation space or the cropping space, the coordinates of the target area in the first image, that is, the coordinates of the target area may be the observation coordinates, Clipping coordinates or screen coordinates.

In the embodiment of the present application, the first distance is the distance between the center position of the first model and the target area in the observation space or the clipping space, and the distance between the center position of the first model and the target area may be The distance between the center point of a model and the center point of the target area can also be the distance between the center position of the first model and the edge of the target area. The edge of the target area can be the edge closest to the first model among the edges of the target area. . The definition of the distance between the center position of the first model and the target area can be set according to the actual situation, and the embodiment of the present application does not specifically limit this.

When obtaining the first distance, compare the distance between the central position and the target area using the same space or the same type of coordinates. Taking the first distance as the distance between the center point of the first model and the center point of the target area as an example, obtain the clipping coordinate A of the center point of the target area in the clipping space, and obtain the center point of the first model in the clipping space. The cropping coordinate B is calculated as the distance between the cropping coordinate A and the cropping coordinate B to obtain the first distance.

In this embodiment of the present application, the determination module may preset the corresponding relationship between distance and coloring rate, such as setting different coloring rates according to different distance ranges. For example, the corresponding relationship is as shown in Table 3 below:

table 3

distance Shading rate first distance range 1X1 Second distance range 1X2 Third distance range 2X2

Among them, when the first distance is within the first distance range, it indicates that the center position of the first model is within the target area. At this time, there is no need to use the variable rate shading mechanism. The determination module can determine that the current first model uses 1X1 according to Table 3. Shading rate. That is, for a model whose center is located in the target area, its corresponding shading rate is 1X1. When the first distance is within the second distance range, it indicates that the first model is not located within the target area. At this time, a variable rate shading mechanism needs to be used. The determination module can determine that the current first model uses a 1X2 shading rate according to Table 3. When the first distance is within the third distance range, it indicates that the first model is located in the edge area of the first image. At this time, a variable rate shading mechanism needs to be used. The determination module can determine that the current first model uses a 2X2 shading rate according to Table 3.

For example, the first distance range may be greater than or equal to 1 cm and less than or equal to 3 cm, the second distance range may be greater than 3 cm and less than or equal to 5 cm, and the third distance may be greater than 5 cm and less than the edge of the first image. The first distance range, the second distance range, and the third distance range can be set according to the size of the first image and relevant parameters of the display, and are not specifically limited here.

It can be seen that the larger the first distance is, the lower the corresponding shading rate is, which correspondingly reduces the power consumption overhead in the shading process. Correspondingly, the smaller the first distance is, the higher the coloring rate is and the clearer the corresponding coloring effect is.

It should be noted that the corresponding relationship between distance and coloring rate shown in Table 3 above is only an example. In other embodiments of the present application, the corresponding relationship between distance and shading rate can also be different from the corresponding relationship shown in Table 3. The setting of the corresponding relationship can be flexibly based on the specific application and the scenarios used in the implementation process. Adjustment.

Taking the above Table 3 as an example, as shown in FIG. 6 , the central area of the first image 60 is the target area, the area enclosed by the frame 601 is the target area, and the target area corresponds to the first distance range. That is, when the center position of the model in the image is within the area enclosed by the border 601, the first distance between the center position of the model and the target area is within the first distance range.

Within the area enclosed by the frame 602, the area except the target area corresponds to the second distance range. That is, when the center position of the model in the image is within the annular area enclosed by the frame 601 and the frame 602, the first distance between the center position of the model and the target area is within the second distance range.

The frame 603 is the edge of the first image 60 . Within the area enclosed by the frame 603 , the area other than the area enclosed by the frame 602 corresponds to the third distance range. That is, when the center position of the model in the image is within the annular area enclosed by the frame 602 and the frame 603, the first distance between the center position of the model and the target area is within the third distance range.

When converted into screen coordinates, the center position P1 of the first object 61 and the center position P2 of the second object 62 are both located in the target area, so the shading rates of the first object 61 and the second object 62 correspond to the first distance range. The shading rate is 1X1. The center position P3 of the third object 63 is located outside the target area and within the border 602, then the coloring rate of the third object 63 is the coloring rate 1X2 corresponding to the second distance range. The center position P4 of the fourth object 64 is located outside the target area and within the border 603, then the coloring rate of the fourth object 64 is the coloring rate 2X2 corresponding to the third distance range.

Step S46: The determination module outputs the corresponding calling instruction to the graphics library to call the variable rate shading API corresponding to the shading rate of the first model.

In the embodiment of the present application, after the determination module determines the shading rate of the first model according to the above solution, the variable rate shading API corresponding to the shading rate of the first model in the graphics library can be called. The determining module outputs a corresponding calling instruction to the graphics library to call the variable rate shading API corresponding to the shading rate of the first model.

For example, taking the shading rate 1X1 corresponding to the first variable rate shading API and the shading rate 1X2 corresponding to the second variable rate shading API as an example, the determination module obtains the first distance between the first model and the target area, the first distance It belongs to the first distance range, and the determination module determines that the shading rate of the first model is 1X1 based on the first distance. The determining module may determine that the shading rate when executing the drawelement included in the Drawcall of the first model is 1X1, and the determining module outputs a calling instruction to the graphics library, which instructs to call the first variable rate shading API corresponding to the shading rate 1X1. The determination module obtains the second distance between the second model and the target area, the second distance belongs to the second distance range, and the determination module determines the shading rate of the second model to be 1X2 based on the second distance. The determination module may determine that the shading rate when executing the drawelement included in the Drawcall of the second model is 1X2, and the determination module outputs a calling instruction to the graphics library, which instructs to call the second variable rate shading API corresponding to the shading rate 1X2.

As an example, the determination module can issue the calling instructions and drawing instructions to the graphics library, so that the graphics library can send corresponding model rendering instructions to the GPU. The drawing instruction can be the glDrawElecments instruction in OpenGL or the DrawIndexedPrimitive instruction in DirectX. Please refer to Figure 5 together. The determination module can send the calling instruction of the variable rate shading API corresponding to the shading rate corresponding to the first model and the drawelement in the current Drawcall to the graphics library.

Step S47: The graphics library calls the corresponding variable rate shading API in response to the calling instruction, and sends the first model rendering instruction to the graphics processor.

For example, if the module determines that the shading rate corresponding to the first model is 2X2, then, after passing through the graphics library, the first model rendering instructions passed to the GPU can include: glVRS (2X2) and glDrawElements. Among them, glVRS(2X2) is used to indicate that the current model is shaded using a shading rate of 2×2. That is, the first model rendering instruction is used to indicate that shading glVRS (2X2) is enabled before calling glDrawElements.

In this way, using Drawcall as the unit, the electronic device can color the first model using the corresponding coloring rate according to the center position of the first model, thereby achieving a reasonable distribution of the coloring rate, reducing the power consumption of the coloring operation, and at the same time improving the coloring effect. the goal of.

In step S48, the graphics processor responds to the first model rendering instruction and performs a shading operation on the first model according to the shading rate corresponding to the first model.

The first model rendering instruction may carry code corresponding to the variable shading rate API corresponding to the first model. The GPU responds to the first model rendering command and executes Drawelement using the shading rate indicated by the corresponding parameter. As in the above example, the code corresponding to the variable shading rate API corresponding to the first model includes glVRS (2X2).

For the specific content of the shading operation performed by the graphics processor in response to the first model rendering instruction, reference can be made to related technologies and will not be described again here.

In the embodiment of this application, the user's focus is taken as the core, the target area is determined based on the user's focus, and the load reduction operation (that is, the coloring rate is reduced) is performed on the screen content outside the user's focus (target area). For the content in the target area The coloring effect can be guaranteed, thus ensuring the image quality experience and improving the performance of electronic devices. By improving the operating system of the electronic device, the interception module intercepts specific instructions in the rendering instruction stream issued by the application. Based on the specific instructions, the center position of the first model in the first image can be quickly and accurately obtained. Based on the center Position can quickly and accurately describe the relationship between the first model and the target area. Then, a corresponding shading rate is reasonably set for the first model based on the relationship between the first model and the target area.

Please refer to FIG. 7 , which is a flowchart of another image rendering method provided by an embodiment of the present application. The difference between Figure 7 and Figure 4 is that the specific instructions intercepted by the interception module are also used to obtain the category of the first model. Specifically, the interception module also intercepts the second specific instructions to obtain the model that can be used to identify the model based on the second specific instructions. Model information of the category to which it belongs. The application framework layer also includes an identification module, and the determination module determines the shading rate of the first model based on the center position of the first model calculated by the calculation module and the category of the first model identified by the identification module.

Step S700: The game application issues a rendering instruction stream.

For step S700, please refer to the relevant content in Figure 4.

Step S701: The interception module intercepts the first specific instruction and the second specific instruction in the rendering instruction stream to obtain vertex data of each vertex of the first model, the MVP matrix corresponding to the first model, and model information of the first model.

Among them, the relevant contents of the first specific instruction, vertex data and MVP matrix can be referred to the above, and will not be described again here.

In this embodiment of the present application, the above-mentioned specific instructions include a second specific instruction, and the second specific instruction may be an instruction carrying a category parameter. Category parameters can be any parameters used to identify the category to which the model belongs. For example, category parameters include parameters that indicate the iconic characteristics of specific objects. Take specific objects such as smoke, gunfire, spray and other translucent particles as an example, and their corresponding rendering instructions. It carries the rendering effect parameters for adding translucent particles. In other embodiments, category parameters may also include size, texture information, coordinate variables in the shader, data buffer characteristics, etc. Texture information can include floor model textures, building model textures, etc.

In this embodiment of the present application, the categories to which models can be classified include but are not limited to: translucent particles, character models, architectural models, etc.

For example, the first model is smoke, which corresponds to the instruction segment A. The instruction segment A starts with the glEnable() instruction and ends with the glDisable() instruction. The glEnable instruction enables the color mixing state, that is, the glEnable instruction instructs the electronic device to start rendering translucent particles. The glDisable directive instructs the electronics to completely turn off the current color mixing operation for translucent particles. When the interception module intercepts the glEnable instruction and glDisable instruction, it can obtain the second specific instruction including the glEnable() instruction and glDisable() instruction. The model information that the interception module can obtain based on the glEnable() and glDisable() instructions is the rendering effect of translucent particles.

In other embodiments, the first model is used as a character model, and the interception module intercepts the second specific instruction to obtain model information including but not limited to: human skeleton node coordinate information, facial skin map information, hairstyle map information, etc.

In some embodiments, the first specific instruction and the second specific instruction may be the same instruction. For example, the vertex data can be obtained through the above-mentioned first instruction, and then the number of vertices of the model can be obtained. Model information, such as the size of the model, can also be determined based on the number of vertices, and then the model category is determined based on the size of the model. For example, the model size of an architectural model is generally larger than a grass model.

Step S702: The interception module stores the obtained vertex data, MVP matrix and model information into the memory.

Step S703: The calculation module obtains the vertex data and MVP matrix, and calculates the center position of the first model based on the vertex data and MVP matrix.

For steps S702 to S703, please refer to the relevant content in Figure 4 .

Step S704: The identification module obtains model information and identifies the category of the first model based on the model information.

In this embodiment of the present application, the recognition module obtains the model information of the first model intercepted by the interception module from the memory. The recognition module can analyze the model information of the first model offline, and then determine the category of the first model.

In some embodiments, the electronic device can pre-store model information corresponding to a specific object category, such as storing model information A corresponding to a character model, and storing model information B corresponding to a transparent particle model. The identification module matches the obtained model information of the first model with the above-mentioned model information A and model information B to determine whether the category of the first model belongs to a specific object category. If so, determine whether the category of the first model belongs to a character model or a transparent particle model.

The trigger mechanism for the recognition module to obtain model information from the memory can refer to the relevant content in Figure 4 above. For example, refer to the trigger mechanism for the calculation module to calculate the center position of the first model in Figure 4, which will not be described again here.

Step S705: The calculation module sends the center position of the first model to the determination module.

For step S705, please refer to the relevant content in Figure 4.

Step S706: The recognition module sends the category of the first model to the determination module.

It can be understood that the execution order of the above steps S703 to S706 is not specifically limited. For example, the calculation module and the recognition module can obtain corresponding data from the memory at the same time, or the calculation module and the recognition module can send the processed data to the determination module at the same time.

Step S707: The determining module determines the coloring rate of the first model according to the center position and category of the first model.

In the embodiment of the present application, the determination module obtains the specific object category corresponding to the application program. When the category of the first model is the specific object category, the coloring rate corresponding to the specific object category is used as the coloring rate corresponding to the first model. When the first model When the category of the model is not a specific object category, the shading rate corresponding to the first model is determined based on the first distance.

The memory of the electronic device can pre-store specific object categories corresponding to each application program, and the coloring rate corresponding to each specific object category. Among them, the specific object category corresponding to each application programmer and the coloring rate corresponding to the specific object category can be set according to the actual situation, which is not specifically limited in the embodiment of the present application.

For example, taking the corresponding specific object categories in the first game application as including character models and translucent particle models, the shading rate corresponding to the character model is 1X2, and the shading rate corresponding to the translucent particle model is 2X2. The first game application instructs to render the first model. When the first model is a character model, the shading rate of the first model is determined to be 1X2. When the first model is a translucent particle model, the shading rate of the first model is determined to be 2X2. When the first model is a grass model, that is, the category of the first model does not belong to the specific object category corresponding to the first game application, the determining module determines the shading rate of the first model based on the center position of the first model.

In the embodiment of the present application, the corresponding shading rate can be determined for the model based on the actual use of each application, and then a personalized shading rate can be provided for the model during user use. For example, if the first game application is a shooting game, in order to improve the user experience, the coloring rate corresponding to the character model can be increased. The shading rate of the character model can be increased even if the character model is at the edge of the image (i.e. outside the target area). For translucent particle models, the effect is translucent. Even if it is within the target area, its shading rate can be reduced.

Step S708: The determination module outputs the corresponding calling instruction to the graphics library to call the variable rate shading API corresponding to the shading rate of the first model.

Step S709: The graphics library calls the corresponding variable rate shading API in response to the calling instruction, and sends the first model rendering instruction to the graphics processor.

In step S710, the graphics processor responds to the first model rendering instruction and performs a shading operation on the first model according to the shading rate corresponding to the first model.

For steps S708 to S710, please refer to the relevant content in Figure 4 .

In this embodiment of the present application, the determination module may also consider the category of the first model when determining the shading rate of the first model. The shading rate corresponding to the model category can be set according to different application scenarios or different game applications to improve the user experience.

In other embodiments, in FIGS. 4 and 7 , the electronic device determines the range of the target area according to the application type of the application program. For example, taking the first game application as a shooting application and the second game application as having an overall darker picture with a smaller brightness area, based on the information and game types presented in the game images of the two games, the game can be The target area range of the first game application is set larger than the target area range of the second game application.

In other embodiments, in Figures 4 and 7, the electronic device can determine the range of the target area according to the type of the application, and then determine the range of the target area in the first image according to the range of the target area and the position of the target object in the first image. Target area, where the target object is related to the user's focus.

In other embodiments, in FIG. 4 and FIG. 7 , the interception module may intercept the third specific instruction, and the third specific instruction carries center position related parameters. When the rendering instruction stream includes the third specific instruction, the third specific instruction can be directly intercepted to obtain the center position of the first model.

In other embodiments, in Figures 4 and 7, when the shading rate of the first model is lower than the shading rate of the target area, then when the graphics processor renders the first model according to the shading rate of the first model, the graphics processor The processor renders the target area according to the highest shading rate and image quality enhancement algorithm, and renders the first model according to the first model's shading rate. When the shading rate of the target area is already the highest, the target area is rendered at the highest shading rate, and the image quality enhancement algorithm is used to render the target area. When the shading rate of the target area is not the highest, the target area is rendered with the highest shading rate and image quality enhancement algorithm. It can be seen that the image quality of the target area is improved while maintaining the same power consumption. The image quality enhancement algorithm can be implemented based on the shader language.

The following uses an electronic device as the execution subject to introduce the image rendering method provided by the embodiment of the present application.

Please refer to FIG. 8 , which is a schematic flowchart of another image rendering method provided by an embodiment of the present application.

Step S81: The application installed on the electronic device issues a rendering instruction stream, where the rendering instruction stream is used to instruct rendering of the first model in the first image.

Step S82: The electronic device intercepts specific instructions in the rendering instruction stream.

Step S83: The electronic device obtains the center position of the first model according to the specific instruction, where the center position is the position of the center point of the first model in the first image.

Step S84: The electronic device determines the coloring rate of the first model based on the center position of the first model and the target area of the first image.

Step S85: The electronic device renders the first model according to the shading rate of the first model.

For the relevant content in Figure 8, please refer to the relevant content in Figures 4 and 7 above, and will not be described again here.

An embodiment of the present application also provides an electronic device, which includes the hardware and software shown in Figure 1 . In other embodiments, the electronic device includes at least one or more processors and one or more memories. Processors include central processing units and graphics processors. One or more memories are coupled to one or more processors, and the one or more memories store computer instructions; when the one or more processors execute the computer instructions, the electronic device executes the images shown in Figures 4 and 7 above. Rendering method.

Please refer to FIG. 9 , which is a schematic structural diagram of a chip system provided by an embodiment of the present application.

Figure 9 shows a schematic diagram of the composition of a chip system 1600. The chip system 1600 may include: a processor 1601 and a communication interface 1602, used to support related devices to implement the functions involved in the above embodiments. In one possible design, the chip system also includes a memory for storing necessary program instructions and data for the electronic device. The chip system may be composed of chips, or may include chips and other discrete devices. It should be noted that in some implementations of this application, the communication interface 1602 may also be called an interface circuit.

It should be noted that all relevant content of each step involved in the above method embodiment can be quoted from the functional description of the corresponding functional module, and will not be described again here.

Embodiments of the present application also provide a computer program product. When the computer program product is run on a computer, it causes the computer to perform the above related steps to implement the image rendering method in the above method embodiments.

Embodiments of the present application also provide a computer storage medium that includes computer instructions. When the computer instructions are run on an electronic device, the electronic device causes the electronic device to execute the image rendering method as in the above embodiment.

Among them, the electronic devices, computer storage media, computer program products or chip systems provided by the embodiments of the present application are all used to execute the corresponding methods provided above. Therefore, the beneficial effects they can achieve can be referred to the above provided The beneficial effects of the corresponding methods will not be described again here.

Through the above description of the embodiments, those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above functional modules is used as an example. In actual applications, the above functions can be allocated as needed. It is completed by different functional modules, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above.

In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods, for example, multiple units or components may be combined. Either it can be integrated into another device, or some features can be ignored, or not implemented. On the other hand, the coupling or direct coupling or communication connection between each other shown or discussed may be through some interfaces, and the indirect coupling or communication connection of the devices or units may be in electrical, mechanical or other forms.

The unit described as a separate component may or may not be physically separate. The component shown as a unit may be one physical unit or multiple physical units, that is, it may be located in one place, or it may be distributed to multiple different places. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present application can be integrated into one processing unit, each unit can exist physically alone, or two or more units can be integrated into one unit. The above integrated units can be implemented in the form of hardware or software functional units.

If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of the present application are essentially or contribute to the existing technology, or all or part of the technical solution can be embodied in the form of a software product, and the software product is stored in a storage medium , including several instructions to cause a device (which can be a microcontroller, a chip, etc.) or a processor to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage media include: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk and other media that can store program code. .

The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application shall be covered by the protection scope of the present application.

Claims (14)

1. An image rendering method, applied to an electronic device, in which an application program is installed, the application program issuing a rendering instruction stream to render a first model in a first image, the method comprising:

intercepting a specific instruction in the rendering instruction stream;

acquiring a central position of the first model according to the specific instruction, wherein the central position is a position of a central point of the first model in the first image;

determining a coloring rate of the first model according to the central position of the first model and a target area of the first image, wherein the coloring rate of the first model is lower than or equal to the coloring rate of the target area;

rendering the first model according to a shading rate of the first model.

2. The image rendering method according to claim 1, wherein the specific instruction includes a first specific instruction, and the acquiring the center position of the first model according to the specific instruction includes:

Obtaining vertex data of each vertex of the first model and an MVP matrix corresponding to the first model according to the first specific instruction;

and determining the position of the center point of the first model in the first image according to the vertex data and the MVP matrix.

3. The image rendering method according to claim 1 or 2, wherein the determining the coloring rate of the first model from the center position of the first model and the target area of the first image includes:

acquiring a first distance according to the central position of the first model and a target area of the first image, wherein the first distance is the distance between the central position of the first model and the target area in an observation space or a clipping space;

and determining the coloring rate corresponding to the first model according to the first distance.

4. The image rendering method according to claim 1 or 2, wherein the determining the coloring rate of the first model from the center position of the first model and the target area of the first image includes:

acquiring the category of the first model;

acquiring a first distance, wherein the first distance is a distance between the central position of the first model and the target area in an observation space or a clipping space;

Determining a coloring rate of the first model according to the category of the first model and the first distance.

5. The image rendering method of claim 4, wherein the determining a shading rate corresponding to the first model according to the class of the first model and the first distance comprises:

acquiring a specific object category corresponding to the application program;

when the category of the first model is the specific object category, the coloring rate corresponding to the specific object category is used as the coloring rate corresponding to the first model;

and when the category of the first model is not the specific object category, determining the coloring rate corresponding to the first model according to the first distance.

6. The image rendering method of claim 5, wherein the specific instruction comprises a second specific instruction, and wherein the obtaining the class of the first model comprises:

obtaining model information of the first model according to the second specific instruction;

and determining the category of the first model according to the model information of the first model.

7. The image rendering method according to any one of claims 1 to 6, characterized in that the method further comprises:

Determining the range of the target area according to the type of the application program;

and determining the target area in the first image according to the range of the target area and the position of a target object in the first image, wherein the target object is related to the attention point of the user.

8. The image rendering method according to any one of claims 1 to 7, wherein when a coloring rate of the first model is lower than a coloring rate of the target region, then the rendering the first model according to the coloring rate of the first model includes:

and rendering the target area according to the highest coloring rate and an image quality enhancement algorithm, and rendering the first model according to the coloring rate of the first model.

9. The image rendering method according to claim 3 or 4, wherein the electronic device includes a central processor and a graphics processor, and wherein the rendering the first model according to the coloring rate of the first model includes:

the central processing unit calls a first variable rate coloring API corresponding to the coloring rate of the first model, and issues a first model rendering instruction to the graphics processor according to the first variable rate coloring API;

The graphics processor renders the first model according to a shading rate of the first model in response to the first model rendering instructions.

10. The image rendering method of claim 9, wherein the first image further comprises a second model, the method further comprising:

acquiring a second distance according to the central position of the second model and a target area of the first image, wherein the second distance is the distance between the central position of the second model and the target area in an observation space or a clipping space;

determining a coloring rate corresponding to the second model according to the second distance, wherein the second distance is larger than the first distance, and the coloring rate of the second model is lower than that of the first model;

rendering the second model according to a coloring rate of the second model.

11. The image rendering method of claim 10, wherein the rendering the second model according to the shading rate of the second model comprises:

the central processing unit calls a second variable rate coloring API corresponding to the coloring rate of the second model, and issues a second model rendering instruction to the graphics processor according to the second variable rate coloring API;

The graphics processor renders the second model according to a shading rate of the second model in response to the second model rendering instructions.

12. An electronic device comprising one or more processors and one or more memories; the one or more memories coupled to the one or more processors, the one or more memories storing computer instructions; the computer instructions, when executed by the one or more processors, cause the electronic device to perform the image rendering method of any one of claims 1 to 11.

13. A computer readable storage medium, characterized in that the computer readable storage medium comprises computer instructions which, when run, perform the image rendering method of any one of claims 1 to 11.

14. A chip system, wherein the chip system comprises a processor and a communication interface; the processor is configured to call up and execute a computer program stored in a storage medium from the storage medium, to perform the image rendering method according to any one of claims 1 to 11.