CN114419229A

CN114419229A - Image rendering method and device, computer equipment and storage medium

Info

Publication number: CN114419229A
Application number: CN202210069302.9A
Authority: CN
Inventors: 陈兴; 丁河
Original assignee: Shenzhen TCL New Technology Co Ltd
Current assignee: Shenzhen TCL New Technology Co Ltd
Priority date: 2022-01-21
Filing date: 2022-01-21
Publication date: 2022-04-29

Abstract

The embodiment of the application discloses an image rendering method, an image rendering device, computer equipment and a storage medium; acquiring a to-be-processed rendering instruction corresponding to a to-be-processed frame; determining a predicted load state of the graphics processor according to the rendering instruction to be processed; updating the rendering instruction to be processed into a target rendering instruction when the predicted load state of the graphics processor is a high load state; and rendering the frame to be processed according to the target rendering instruction through the graphics processor to obtain a rendered image. The scheme can adjust the load state of the graphics processor when rendering the frame to be processed, optimize the performance of the graphics processor, prevent the graphics processor from being blocked when rendering the frame to be processed, and improve the smoothness of the image picture.

Description

Image rendering method and device, computer equipment and storage medium

Technical Field

The present application relates to the field of image processing technologies, and in particular, to an image rendering method, an image rendering apparatus, a computer device, and a storage medium.

Background

The graphic rendering technology is a hotspot of research in the field of intelligent terminals, and most of graphic images are rendered by adopting a graphic processor at present. Under heavy-duty scenes rendered by a graphics processor, such as 3D large games, the performance of the graphics processor tends to be inadequate.

Therefore, in order to solve the problem of insufficient performance of the graphics processor, hardware upgrading is usually adopted to solve the problem, and the cost is high. Therefore, how to optimize the performance of the graphics processor is a significant problem currently facing.

Disclosure of Invention

The embodiment of the application provides an image rendering method, an image rendering device, computer equipment and a storage medium, which can adjust the load state of a graphics processor when rendering a frame to be processed and optimize the performance of the graphics processor.

An embodiment of the present application provides an image rendering, including: acquiring a to-be-processed rendering instruction corresponding to a to-be-processed frame; determining a predicted load state of the graphics processor according to the rendering instruction to be processed; updating the rendering instruction to be processed into a target rendering instruction when the predicted load state of the graphics processor is a high load state; and rendering the frame to be processed according to the target rendering instruction through the graphics processor to obtain a rendered image.

An embodiment of the present application further provides an image rendering apparatus, including: the system comprises an acquisition unit, a processing unit and a processing unit, wherein the acquisition unit is used for acquiring a to-be-processed rendering instruction corresponding to a to-be-processed frame; a determining unit, configured to determine a predicted load state of the graphics processor according to the to-be-processed rendering instruction; the updating unit is used for updating the rendering instruction to be processed into a target rendering instruction when the predicted load state of the graphics processor is a high load state; and the rendering unit is used for rendering the frame to be processed according to the target rendering instruction through the graphics processor to obtain a rendered image.

The embodiment of the application also provides computer equipment, which comprises a memory, a storage and a control unit, wherein the memory stores a plurality of instructions; the processor loads instructions from the memory to execute the steps of any image rendering method provided by the embodiment of the application.

The embodiment of the present application further provides a computer-readable storage medium, where a plurality of instructions are stored, and the instructions are suitable for being loaded by a processor to perform the steps in any one of the image rendering methods provided in the embodiment of the present application.

The method and the device for processing the frame can obtain a to-be-processed rendering instruction corresponding to the to-be-processed frame; determining a predicted load state of the graphics processor according to the rendering instruction to be processed; updating the rendering instruction to be processed into a target rendering instruction when the predicted load state of the graphics processor is a high load state; and rendering the frame to be processed according to the target rendering instruction through the graphics processor to obtain a rendered image. According to the method and the device, the load state of the to-be-processed frame processed by the graphics processor is predicted, the to-be-processed rendering instruction is updated in the high load state, and the target rendering instruction is used for rendering, so that the load state of the graphics processor in the process of rendering the to-be-processed frame is adjusted, the performance of the graphics processor is optimized, the graphics processor cannot be jammed in the process of rendering the to-be-processed frame, and the fluency of an image picture is improved.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.

Fig. 1 is a scene schematic diagram of an image rendering method according to an embodiment of the present application;

fig. 2 is a schematic flowchart of an image rendering method according to an embodiment of the present disclosure;

fig. 3 is a schematic rendering flow applied to a terminal according to an embodiment of the present disclosure;

FIG. 4 is a flowchart illustrating a further image rendering method according to an embodiment of the present disclosure;

fig. 5 is a schematic structural diagram of an image rendering apparatus according to an embodiment of the present application;

fig. 6 is a schematic structural diagram of a computer device provided in an embodiment of the present application.

Detailed Description

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.

The embodiment of the application provides an image rendering method, an image rendering device, computer equipment and a storage medium.

The image rendering apparatus may be specifically integrated in an electronic device, and the electronic device may be a terminal, a server, or the like. The terminal can be a mobile phone, a tablet Computer, an intelligent bluetooth device, a notebook Computer, or a Personal Computer (PC), and the like; the server may be a single server or a server cluster composed of a plurality of servers.

In some embodiments, the image rendering apparatus may be further integrated in a plurality of electronic devices, for example, the image rendering apparatus may be integrated in a plurality of servers, and the image rendering method of the present application is implemented by the plurality of servers.

In some embodiments, the server may also be implemented in the form of a terminal.

For example, referring to fig. 1, in some embodiments, a scene schematic of an image rendering system is provided, which may include a terminal 1000, a server 2000, and a network 3000. The terminal is integrated with an image rendering device, the server stores image data to be rendered, the image data to be rendered comprise frames to be processed, and the server and the terminal can perform data interaction through a network.

The terminal can acquire image data to be rendered from the server through the network, acquire a frame to be processed from the image data to be rendered, and acquire a rendering instruction to be processed corresponding to the frame to be processed after the frame to be processed is acquired; determining a predicted load state of the graphics processor according to the rendering instruction to be processed; when the predicted load state of the graphics processor is a high load state, updating the rendering instruction to be processed into a target rendering instruction; and rendering the frame to be processed according to the target rendering instruction through the graphics processor to obtain a rendered image for sending the frame to be processed to the terminal.

The network is used for data transmission between the server and the target device, and may be a wireless network or a wired network, for example, the wireless network is a Wireless Local Area Network (WLAN), a Local Area Network (LAN), a cellular network, a 2G network, a 3G network, a 4G network, a 5G network, or the like.

The following are detailed below. The numbers in the following examples are not intended to limit the order of preference of the examples.

In this embodiment, an image rendering method is provided, as shown in fig. 2, a specific flow of the image rendering method may be as follows:

110. and acquiring a to-be-processed rendering instruction corresponding to the to-be-processed frame.

The frame to be processed refers to a frame to be rendered by the graphics processor, and may be one or more frames to be rendered by the graphics processor.

In some embodiments, in order to reduce the amount of data processed in a single time, simplify the rendering instruction processing process, reduce the resource overhead of the graphics processor, and avoid the stutter and data error, the frame to be processed is the current frame. The current frame refers to a frame currently to be rendered by the graphics processor.

The rendering is a process of generating an image from a model in a computer drawing process, and is generally performed by a Graphics Processing Unit (GPU), which is a microprocessor dedicated to image operation and is often used for Graphics rendering. For example, in this embodiment, a Central Processing Unit (CPU) may be deployed in the terminal, and when the terminal runs a game application program, the game program instructs the Central Processing Unit to package game resources in rendering instructions and send the rendering instructions to the graphics processor, so as to process the rendering instructions through the graphics processor and finally generate a game screen. For example, as shown in fig. 3, on one hand, in the software drawing process, the central processing unit performs software drawing through Measure, Layout, Draw, and Skia, and on the other hand, in the hardware drawing process, the central processing unit generates rendering commands through Measure, Layout, and Draw, and submits the generated rendering commands to the graphics processor through an OpenGL ES (OpenGL for Embedded Systems) module, where each rendering command corresponds to an OpenGL ES API, and the graphics processor renders graphics according to the rendering commands, and displays a screen on the screen after rasterizing the software and hardware drawing results, and the rendering process in this embodiment of the application is the hardware drawing process in fig. 3.

The to-be-processed rendering instruction is an instruction used for instructing the graphics processor to render a picture according to a to-be-processed frame, and the to-be-processed rendering instruction embodies the workload of the GPU for processing the to-be-processed frame. And when the CPU generates rendering instructions according to the program resources, collecting the rendering instructions by taking a frame as a unit so as to perform rendering through the GPU. In some embodiments, before step 110, the method may further include the steps of: and generating a rendering instruction to be processed corresponding to the frame to be processed.

120. A predicted load state of the graphics processor is determined based on the pending rendering instructions.

The load state refers to a GPU load working state, and may include, but is not limited to, at least one of a high load state, a low load state, or a normal load state. For example, a high load state may be used to characterize a GPU load above a first load threshold, a normal load state may be used to characterize a GPU load between the first load threshold and a second load threshold, and a low load state may be used to characterize a GPU load below the second load threshold, where the first load threshold is greater than the second load threshold, and the first load threshold and the second load threshold may be set according to application scenarios or experience. In actual application, the GPU load may be represented by a feature value related to the GPU load, such as a GPU occupancy, a GPU operating frequency, or the number of APIs, and the first load threshold and the second load threshold may be also set to a preset GPU occupancy, the number of APIs, or the like.

The predicted load state refers to a load state when the graphics processor processes rendering instructions according to the rendering instructions corresponding to the frame to be processed, and may be predicted based on current or historical load information of the GPU, or based on parameters of the GPU, or may be predicted by combining the two.

In some embodiments, the predicted load condition may include a high load condition and a low load condition. When the graphics processor is in a high load state, the graphics processor is overloaded and is easy to have abnormal situations such as pause or frame dropping, and when the graphics processor is in a low load state, the graphics processor is normal in load and is not easy to have abnormal situations such as pause or frame dropping. For example, a high load state may be used to characterize a GPU load above a first load threshold, and a low load state may be used to characterize a GPU load below the first load threshold.

In some embodiments, to facilitate determining the predicted load status of the graphics processor and improve the processing efficiency, step 120 may include steps 1.1-1.2, as follows:

1.1, acquiring a first parameter corresponding to a rendering instruction to be processed;

1.2, determining the predicted load state of the graphics processor according to the first parameter and the threshold condition.

The first parameter refers to a parameter when an API (Application Programming Interface) corresponding to the rendering instruction is called, and may be obtained through an API of a frame to be processed by a HOOK function (HOOK function), and the first parameter may include a function name and other parameters.

The threshold condition is a condition for determining a load status of the graphics processor, and for example, the load status of the graphics processor may be determined according to the type, the number, the type and the number of the first parameter, or the like, or may be preset according to needs or experience. For example, since there are generally a plurality of first parameters corresponding to the image frame, the first parameters may include two types, i.e., type one and type two, and all the first parameters are classified according to the two types, and m first parameters of type one and n first parameters of type two are obtained, the threshold condition may be that the load state of the graphics processor is determined to be a high load state when there are more than a first parameters of type one, or the load state of the graphics processor is determined to be a high load state when there are first parameters of type two, and so on.

By setting the threshold condition, the first parameter corresponding to the frame to be processed can be compared with the threshold condition to determine whether the workload of the graphics processor exceeds the workload determined by the threshold condition when the rendering instruction of the frame to be processed is executed.

In some embodiments, the amount of the first parameter is used to reflect the workload of API interface call when executing rendering instruction, so as to more intuitively characterize the workload of the graphics processor by using the first parameter and a threshold condition, where the threshold condition may include determining the predicted load status of the graphics processor as a high load status when the amount of the first parameter is greater than a preset value, and determining the predicted load status of the graphics processor as a low load status when the amount of the first parameter is less than or equal to the preset value, and step 1.2 may include the steps of: a predicted load state of the graphics processor is determined based on the number of first parameters and a threshold condition.

In some embodiments, steps 2.1-2.4 may be included before step 120, as follows:

2.1, acquiring a preset frame;

2.2, generating a preset rendering instruction corresponding to the preset frame;

2.3, acquiring a second parameter corresponding to the preset rendering instruction;

and 2.4, determining a threshold condition according to the second parameter.

The preset frame is a frame used for determining the threshold condition. The preset frame may be preset according to actual needs or experience, for example, the preset frame is determined according to the data size of the frame, and the frame with the data size equal to the preset data size is used as the preset frame, where the preset data size may be determined according to the operating frequency of the graphics processor, may also be determined according to a preset application scene and a frame rate threshold corresponding to the scene, and may also be preset, and the like. The preset frame may comprise one or more frames, and in some embodiments, the number of the preset frame is the same as the number of the frame to be processed, for example, when the frame to be processed is one frame, the preset frame is also one frame.

The preset rendering instruction is an instruction for instructing the graphics processor to render the picture according to a preset frame. The second parameter refers to a parameter when an API (Application Programming Interface) corresponding to the preset rendering instruction is called, and may be obtained through an API of a HOOK function preset frame, and the second parameter may include a function name and other parameters. The method for determining the preset rendering instruction and the second parameter is the same as the instruction to be processed and the first parameter.

Before a frame to be processed is rendered, a threshold condition is determined through a preset frame, the frame to be processed is judged according to the threshold condition, the load state of a graphics processor when the frame to be processed is determined, different preset frames can be selected according to an application scene, the flexible judgment of the load state of the graphics processor is achieved, the obtained threshold condition is strong in frame relevance, the matching degree of the threshold condition and the characteristics of the frame to be processed is improved, and the judgment result of the load state of the graphics processor is more reasonable.

In step 2.4, the second parameter may be used as a threshold reference to determine the threshold condition, for example, the number of the second parameters is used as a preset value in the threshold condition described above, or a plurality of second parameters corresponding to a plurality of preset frames may be obtained, and the threshold reference is obtained according to the plurality of second parameters to determine the threshold condition, for example, an average value of the number of the plurality of second parameters is used as a preset value in the threshold condition described above.

In some embodiments, to simplify the processing procedure and improve the processing efficiency, the preset value in the threshold condition may be the number of the second parameters.

In some embodiments, the preset frame is a frame that meets the preset operating condition, and the obtaining of the candidate frame and the determining of the preset frame from the candidate frame can increase the rationality of the process of determining the candidate frame, so that the result of determining the load state of the graphics processor is more reasonable, and step 2.1 may include steps 2.1.1 to 2.1.3, as follows:

2.1.1, acquiring a plurality of candidate frames;

2.1.2, rendering each candidate frame respectively to obtain the graphic processor operation information aiming at each candidate frame;

and 2.1.3, determining a preset frame from a plurality of candidate frames according to the graphics processor running information of all the candidate frames and preset running conditions.

The candidate frame may be preset according to actual needs or experience, or may be randomly selected, for example, a plurality of frames having the same scene as the frame to be processed are randomly selected as the candidate frame, and if the frame to be processed is a frame of a 3D game screen, a plurality of frames may be randomly selected from the 3D game screen as the candidate frame.

The gpu running information refers to information of the gpu processing each candidate frame, and may include at least one of a frequency, a stuck condition, an occupancy, a processing duration, and the like.

The preset operation condition is a preset condition for determining the operation information of the graphics processor, and may include a condition for determining whether the frequency of the graphics processor when rendering the candidate frame is higher than a preset frequency, whether the graphics processor is stuck when rendering the candidate frame, whether the occupancy of the graphics processor when rendering the candidate frame is higher than a preset occupancy, whether the processing duration of the graphics processor when rendering the candidate frame is higher than a preset duration, and the like.

In some embodiments, in order to facilitate determining a predicted load status of the graphics processor and improve processing efficiency, the graphics processor operation information may include a stuck condition and an operation parameter, the operation parameter may include at least one of a frequency, an occupancy, a processing duration, and the like, the preset operation condition may include determining a candidate frame that is stuck during rendering and satisfies a preset parameter condition as a preset frame, and step 2.1.3 may include the steps of: determining the candidate frames which are blocked during rendering as high-load frames according to the graphic processor operation information of all the candidate frames; according to the operation parameters and preset parameter conditions corresponding to the high-load frames, a preset frame is determined from all the high-load frames, for example, the high-load frame with the operation parameter value being the minimum value among the parameter values corresponding to all the high-load frames is determined as the preset frame, and if the high-load frame with the shortest processing time length is determined as the preset frame, at least two of the processing time length, the processing frequency and the processing occupancy rate can be subjected to weight calculation, and the high-load frame with the minimum value obtained through calculation is determined as the preset frame.

130. When the predicted load state of the graphics processor is a high load state, the pending rendering instruction is updated to the target rendering instruction.

The target rendering instruction is an instruction which is updated according to the rendering instruction to be processed and is used for the graphic processor to render pictures according to the frame to be processed. The method for updating the to-be-processed instruction can update the to-be-processed rendering instruction into the target rendering instruction by reducing the rendering resolution. And updating the rendering instruction to be processed, and rendering by using the target rendering instruction so as to adjust the load state of the graphics processor when rendering the frame to be processed and optimize the performance of the graphics processor.

In some embodiments, in order to reduce the amount of data to be processed by the image processor and improve the processing efficiency while the image quality of the rendered image is not changed, the to-be-processed rendering instruction includes a plurality of to-be-processed sub-instructions, and step 130 may include steps 3.1 to 3.4, as follows:

3.1, determining a first target object and a second target object in a frame to be processed;

3.2, performing instruction cutting on the to-be-processed sub instruction corresponding to the first target object to obtain a first target to-be-processed sub instruction;

3.3, carrying out instruction combination on the to-be-processed sub-instructions corresponding to the second target object to obtain second target to-be-processed sub-instructions;

and 3.4, updating the rendering instruction to be processed into a target rendering instruction according to the first target sub-instruction to be processed and the second target sub-instruction to be processed.

The first target object is a non-key object in the picture of the frame to be processed, a key object used for representing the image scene is arranged in the picture of the frame to be processed, the key object can be preset according to the scene, objects except the key object in the picture of the frame to be processed are non-key objects, and the non-key objects do not influence the expression effect of the picture of the frame to be processed. For example, when the frame to be processed is a frame of a 3D game screen, a player character in the screen, a building in a near view, and the like are key objects, and a building, a plant, and the like in a distant view are non-key objects.

The second target object is the same object in the picture of the frame to be processed, for example, a plurality of static models of the same material need to be rendered in the picture of the frame to be processed, the models are in fixed positions in the application running process and do not undergo scale transformation, and the static models are the second objects. The to-be-processed rendering instruction comprises to-be-processed sub-instructions corresponding to the first target object and the second target object, and also comprises to-be-processed sub-instructions corresponding to the key objects and the like. It should be noted that there may be a plurality of second target objects in the frame to be processed, each second target object may include a plurality of identical second target sub-objects, each second target object is different, for example, there may be a plurality of identical houses and a plurality of identical trees in the 3D game frame, the plurality of identical houses are one second target sub-object, and the plurality of identical trees are another second target sub-object.

Wherein the instruction clipping refers to an operation of reducing an amount of data to be processed when rendering the first target object. For example, the number of vertices may be reduced, or the pending sub-instruction corresponding to the non-critical object may be deleted or used as a null instruction. If the number of the vertices is reduced to process the second target object, a new rendering instruction corresponding to the second target object is obtained, the new rendering instruction is used as a second target to-be-processed sub-instruction, and when the to-be-processed sub-instruction corresponding to the non-key object is deleted or used as a null instruction, the second target to-be-processed sub-instruction is a null instruction.

The instruction merging refers to merging the same instructions corresponding to the second target object. For example, the same instructions for a plurality of the same second target sub-objects are merged, or the same instructions of one second target sub-object are merged to obtain a second target pending sub-instruction, and so on. The second target pending sub-instruction may be a batch instruction for the second target object.

The method includes the steps that a first target object is determined through a non-key object in a picture of a frame to be processed, a second target object is determined through the same object, the first target object and a to-be-processed sub-instruction corresponding to the second target object are processed respectively to obtain a first target to-be-processed sub-instruction and a second target to-be-processed sub-instruction, the to-be-processed sub-instruction corresponding to the first target object in the to-be-processed rendering instruction can be replaced by the first target to-be-processed sub-instruction, and the to-be-processed sub-instruction corresponding to the second target object in the to-be-processed rendering instruction is replaced by the second target to-be-processed sub-instruction, so that the target rendering instruction is obtained.

In some embodiments, the vertex data of the first target object is simplified to obtain the first target to-be-processed sub-instruction, so as to reduce the data amount to be processed of the image processor and improve the processing efficiency, and step 3.2 may include steps 3.2.1 to 3.2.3, as follows:

3.2.1, determining initial vertex data corresponding to the first target object;

3.2.2, determining target vertex data from the initial vertex data through a clustering algorithm;

and 3.2.3, updating the to-be-processed sub-instruction corresponding to the first target object according to the target vertex data to obtain a first target to-be-processed sub-instruction.

The initial vertex data refers to the position information of all the vertexes of the first target object in the picture of the frame to be processed, and the initial vertex data comprises the position information of a plurality of vertexes. The graphics processor sends the initial vertex data into the graphics processor, and the image of the first object can be obtained by rendering the initial vertex data.

The clustering algorithm can be a K-Means clustering algorithm or a Rosssignac vertex clustering algorithm, and the like. For example, the network is simplified by detecting and merging clusters of adjacent vertexes by adopting a Rosssignac vertex clustering algorithm, each cluster is replaced by a representative vertex which can be the center of a vertex cluster or the vertex with the maximum weight in the cluster, and all the representative vertexes form target vertex data.

And updating the to-be-processed sub-instruction corresponding to the first target object into the first target to-be-processed sub-instruction by acquiring the vertex data of the first target object, obtaining target vertex data consisting of representative vertices by a clustering algorithm and generating the first target to-be-processed sub-instruction according to the target vertex data.

In some embodiments, in order to simplify the rendering process for multiple same objects, reduce the amount of data to be processed by the image processor, and improve the processing efficiency, the second target object includes multiple same second target sub-objects, and step 3.3 may include steps 3.3.1 to 3.3.3, as follows:

3.3.1, acquiring a to-be-processed sub-instruction corresponding to any one second target sub-object;

3.3.2, generating multiple instance rendering instructions corresponding to the multiple second target sub-objects according to the sub-instructions to be processed and the multiple same second target sub-objects;

3.3.3, determining the multiple instance rendering instruction as a second target pending sub-instruction.

The multi-instance rendering instruction is an instruction for executing a plurality of same rendering instructions, and the plurality of same rendering instructions are in one-to-one correspondence with a plurality of same second target sub-objects.

When a plurality of identical objects to be rendered exist in a picture of a frame to be processed, a plurality of data values need to be transmitted to a graphics processor during original rendering so as to respectively render each object, and a multi-instance rendering instruction can be adopted to transmit more vertex data as much as possible in one-time rendering instruction, so that the calling of the rendering instruction is reduced, namely, a plurality of objects can be rendered by transmitting data once.

It should be noted that steps 3.3.1 to 3.3.2 are processes executed for one second target object, and when there are a plurality of second target objects, steps 3.3.1 to 3.3.2 are respectively executed for the plurality of second target objects.

In some embodiments, in order to adjust the operating status of the graphics processor and reduce the load of the graphics processor, after step 130, the method may further include the steps of: and adjusting the working frequency of the graphics processor to the target working frequency.

The target operating frequency refers to a preset operating frequency of the graphics processor, and may be preset according to actual needs or experience, and the target operating frequency may be a frequency higher than that of the graphics processor at the current time, or may be determined according to a frequency when the graphics processor processes a preset frame, for example, the frequency is used as the target operating frequency, or the frequency is used as the target operating frequency after weight calculation is performed on the frequency, and so on. In some embodiments, the target operating frequency is the highest frequency of the graphics processor in order to maximize utilization of the graphics processor's resources.

140. And rendering the frame to be processed according to the target rendering instruction through the graphics processor to obtain a rendered image.

The rendered image is an image of a picture containing a frame to be processed.

After obtaining the target rendering instruction, the graphics processor processes the target rendering instruction to generate a picture of the frame to be processed. It should be noted that the embodiment of the present application may be used to render an image with multiple consecutive frames, and when a rendering instruction corresponding to each frame is obtained, a target rendering instruction may be obtained by the method of the embodiment of the present application, and after the target rendering instruction corresponding to each frame of the image with multiple consecutive frames is processed, a picture of the image is obtained.

The image rendering scheme provided by the embodiment of the application can be applied to various image rendering scenes. For example, taking game rendering as an example, a to-be-processed rendering instruction corresponding to a to-be-processed frame may be obtained; determining a predicted load state of the graphics processor according to the rendering instruction to be processed; when the predicted load state of the graphics processor is a high load state, updating the rendering instruction to be processed into a target rendering instruction; and rendering the frame to be processed according to the target rendering instruction through the graphics processor to obtain a rendered image.

According to the method provided by the embodiment of the application, the load state of the to-be-processed frame processed by the graphics processor can be predicted, the to-be-processed rendering instruction is updated in a high-load state, the target rendering instruction is used for rendering, so that the load state of the graphics processor in the process of rendering the to-be-processed frame is adjusted, the performance of the graphics processor is optimized, the graphics processor cannot be jammed in the process of rendering the to-be-processed frame, and the fluency of an image picture is improved.

The method described in the above embodiments is further described in detail below.

In this embodiment, the method of the embodiment of the present application will be described in detail by taking an example of applying the image rendering of the present application to television screen rendering.

As shown in fig. 4, a specific flow of an image rendering method is as follows:

210. and acquiring a to-be-processed rendering instruction corresponding to the to-be-processed frame, wherein the to-be-processed rendering instruction comprises a plurality of to-be-processed sub-instructions.

The frame to be processed is a frame, and may be a frame to be currently rendered, a rendering command (HOOK API) of the frame to be processed is first obtained, that is, marking is performed before and after each frame is rendered, and a function name and a parameter of each frame of the rendering command are obtained through the HOOK API. When the to-be-processed rendering instruction of the to-be-processed frame is obtained, the to-be-processed rendering instruction may be counted, and data support may be performed for subsequent optimization, for example, the number or the type of APIs corresponding to the to-be-processed sub-instructions in the to-be-processed rendering instruction may be counted, so that the statistical result may be determined according to a threshold condition, and the load state of the to-be-processed rendering instruction may be determined.

It should be noted that the terms and some specific implementations of steps in this embodiment may refer to corresponding parts of the foregoing embodiments, and are not described herein again.

220. And determining the predicted load state of the graphics processor according to the rendering instruction to be processed, executing steps 230-270 when the predicted load state of the graphics processor is a high load state, and executing step 280 when the predicted load state of the graphics processor is a low load state.

The predicted load state of the graphics processor may be determined by:

1. dividing the load level of the graphics processor: firstly, a frame rate detection tool is used for detecting the frame rate condition of each scene, and the load level (including a frame rate threshold value for each application scene) of the graphics processor is preset aiming at the application scene, wherein the load is normal and indicates that the system does not have a stuck frame, and the load is overweight and indicates that the system has a stuck frame. For example, when the corresponding terminal is a television, when the frame rate of the picture rendered by the graphics processor is greater than or equal to the frame rate threshold, the load is determined to be normal, and when the frame rate is less than the preset frame rate, the load is determined to be too heavy, and the frame corresponding to the rendered picture is determined to be a candidate frame. At this time, the frame rate of the graphics processor rendering the television pictures is the GPU load used to distinguish between the high load state and the low load state in the foregoing embodiment, and the frame rate threshold is the first load threshold.

2. And obtaining a corresponding threshold value T by utilizing the divided load grade condition, wherein the threshold value is determined by counting the number of candidate frame rendering command APIs respectively corresponding to the load normal scene and the load overweight scene, for example, the maximum value of the number of the APIs corresponding to the load normal scene is used as the threshold value, and the corresponding frame is a preset frame, or the minimum value of the number of the APIs corresponding to the load overweight scene is used as the threshold value, and the corresponding frame is the preset frame.

3. And judging whether the rendering load of the frame to be processed is overweight or not according to the threshold value T, if so, determining the frame to be processed to be in a high load state, and otherwise, determining the frame to be processed to be in a low load state. Therefore, the predicted load state of the frame to be processed which is larger than the threshold value T is determined to be a high load state, and the predicted load state of the frame to be processed which is smaller than or equal to the threshold value T is determined to be a low load state, so that the load state when the frame to be processed is processed by the graphics processor is predicted. That is, the frame rate of the graphics processor is less than the preset frame rate (i.e., in a high load state) when the frame to be processed greater than the threshold T is predicted to be processed, and the frame rate of the graphics processor is greater than or equal to the preset frame rate (i.e., in a low load state) when the frame to be processed less than the threshold T is processed.

By judging the predicted load state of the graphics processor, if the load of the frame to be processed is normal, the frame does not need to be processed, and if the load is too heavy, frequency modulation, instruction clipping, instruction combination and the like are performed.

230. When the predicted load state of the graphics processor is a high load state, the operating frequency of the graphics processor is adjusted to a highest frequency.

When the predicted load state of the graphics processor is a high load state, the frame to be processed is in a stuck state, the frequency of the graphics processor needs to be adjusted, and the television does not need to consider the power consumption problem too much, so that the frequency of the graphics processor is adjusted to be the highest and the frame to be processed is continuously rendered until the rendering of the frame to be processed is finished. One frame rendering starts with dequeueBuffer as a starting point and ends with the QueueBuffer method as a flag.

240. And when the predicted load state of the graphics processor is a high load state, performing instruction clipping on the to-be-processed sub-instruction corresponding to the first target object to obtain a first target to-be-processed sub-instruction.

The method can be used for cutting the instruction aiming at the small objects with far visual distance in the 3D scene, and can reduce the number of triangles (namely the number of vertexes) as much as possible on the premise of not reducing the expression effect so as to improve the drawing efficiency of the graph. And according to the size entering rule of 3D imaging, taking the minimum value of the Z coordinate of the object in the scene to judge whether the object belongs to the long shot or not in the scene. The vertex reduction method adopts a merging mechanism, for example, a vertex clustering algorithm such as Rossignac is used to detect and merge clusters of adjacent vertices to simplify the mesh, each cluster is replaced by a representative vertex, which may be the center of the vertex cluster or the vertex with the maximum weight in the cluster (the weight value of the vertex is defined to emphasize relative visual generality), and finally, overlapping or degenerated edges or triangles caused by clustering operation are processed to obtain the first target pending sub-instruction.

In some embodiments, the manner of instruction clipping may also be replaced by selecting a null instruction or a manner of reducing the number of vertices.

250. And when the predicted load state of the graphics processor is a high load state, performing instruction merging on the to-be-processed sub-instructions corresponding to the second target object to obtain a second target to-be-processed sub-instruction.

The instruction combination is that static objects in a scene are processed in batch, if a plurality of static models made of the same material need to be drawn uniformly in the scene, the models are in fixed positions in the application running process and do not need scale transformation, and the models are rendered by replacing glElements with gldrawelementsInstance, so that only one instance needs to be transmitted to a graphics processor, and a plurality of models can be rendered after the number parameter of rendering is set.

Batch processing may render using gldraweementslnstance instead of gldraweements for static objects in the scene, and adjust the graphics processor frequency to be highest before rendering, for a duration of time from before rendering to the end of rendering.

260. And updating the rendering instruction to be processed into a target rendering instruction according to the first target sub-instruction to be processed and the second target sub-instruction to be processed.

And replacing the to-be-processed sub-instruction corresponding to the first target object in the to-be-processed rendering instruction with a first target to-be-processed sub-instruction, and replacing the to-be-processed sub-instruction corresponding to the second target object in the to-be-processed rendering instruction with a second target to-be-processed sub-instruction to obtain a target rendering instruction.

270. And after the graphic processor is adjusted to the highest frequency, rendering the frame to be processed according to the target rendering instruction to obtain a rendered image.

280. And when the predicted load state of the graphics processor is a low load state, rendering the frame to be processed through the graphics processor according to the rendering instruction to be processed to obtain a rendered image.

As can be seen from the above, in the embodiment of the present application, the load state of the graphics processor is predicted, and the hiton detection is performed in units of frames, so that when the load state of the graphics processor is predicted to be the high load state, and a frequency raising request is initiated, the operating frequency of the graphics processor is adjusted to the highest frequency in real time, so as to schedule resources in real time. According to the obtained rendering command, the rendering command is cut and processed in batch, so that the workload and the load of the graphics processor can be reduced, the performance of the graphics processor is optimized, and the fluency of an image picture is improved.

In order to better implement the method, an embodiment of the present application further provides an image rendering apparatus, where the image rendering apparatus may be specifically integrated in an electronic device, and the electronic device may be a terminal, a server, or other devices. The terminal can be a mobile phone, a tablet computer, an intelligent Bluetooth device, a notebook computer, a personal computer and other devices; the server may be a single server or a server cluster composed of a plurality of servers.

For example, in the present embodiment, the method of the present embodiment will be described in detail by taking an example in which an image rendering device is specifically integrated in a terminal.

For example, as shown in fig. 5, the image rendering apparatus may include an acquisition unit 310, a determination unit 320, an update unit 330, and a rendering unit 340, as follows:

acquisition unit 310

And the method is used for acquiring the rendering instruction to be processed corresponding to the frame to be processed.

(II) determination unit 320

For determining a predicted load state of the graphics processor based on the pending rendering instructions.

In some embodiments, the to-be-processed rendering instruction includes a plurality of to-be-processed sub-instructions, and the determining unit 320 may be specifically used in steps 4.1 to 4.2, as follows:

4.1, acquiring a first parameter corresponding to the rendering instruction to be processed;

4.2, determining the predicted load state of the graphics processor according to the first parameter and the threshold condition.

In some embodiments, determination unit 320 may also be used in steps 5.1-5.4, as follows:

5.1, acquiring a preset frame;

5.2, generating a preset rendering instruction corresponding to the preset frame;

5.3, acquiring a second parameter corresponding to the preset rendering instruction;

and 5.4, determining a threshold condition according to the second parameter.

(III) update Unit 330

And the processor is used for updating the rendering instruction to be processed into the target rendering instruction when the predicted load state of the graphics processor is a high load state.

In some embodiments, the to-be-processed rendering instruction includes a plurality of to-be-processed sub-instructions, and the updating unit 330 may be specifically configured to perform steps 6.1 to 6.4 as follows:

6.1, determining a first target object and a second target object in a frame to be processed;

6.2, performing instruction cutting on the to-be-processed sub instruction corresponding to the first target object to obtain a first target to-be-processed sub instruction;

6.3, carrying out instruction combination on the to-be-processed sub-instructions corresponding to the second target object to obtain second target to-be-processed sub-instructions;

and 6.4, updating the rendering instruction to be processed into a target rendering instruction according to the first target sub-instruction to be processed and the second target sub-instruction to be processed.

In some embodiments, step 6.2 may include steps 6.2.1 to 6.2.3, as follows:

6.2.1, determining initial vertex data corresponding to the first target object;

6.2, determining target vertex data from the initial vertex data through a clustering algorithm;

6.2.3, updating the to-be-processed sub-instruction corresponding to the first target object according to the target vertex data to obtain a first target to-be-processed sub-instruction.

In some embodiments, the second target object comprises a plurality of identical second target sub-objects, and step 6.3 may comprise steps 6.3.1 to 6.3.2, as follows:

6.3.1, acquiring a to-be-processed sub-instruction corresponding to any one second target sub-object;

6.3.2, generating multiple instance rendering instructions corresponding to the multiple second target sub-objects according to the sub-instructions to be processed and the multiple same second target sub-objects;

6.3.3, determining the multiple instance rendering instruction as a second target pending sub-instruction.

In some embodiments, after step 130, the method may further include the step of: and adjusting the working frequency of the graphics processor to the target working frequency.

(IV) rendering Unit 340

And the image processing device is used for rendering the frame to be processed according to the target rendering instruction through the graphics processor to obtain a rendered image.

In a specific implementation, the above units may be implemented as independent entities, or may be combined arbitrarily to be implemented as the same or several entities, and the specific implementation of the above units may refer to the foregoing method embodiments, which are not described herein again.

Therefore, the load state of the graphics processor during rendering of the frame to be processed can be adjusted, the performance of the graphics processor is optimized, the graphics processor cannot be jammed during rendering of the frame to be processed, and the fluency of an image picture is improved.

Correspondingly, the embodiment of the present application further provides a computer device, where the computer device may be a terminal or a server, and the terminal may be a terminal device such as a smart phone, a tablet computer, a notebook computer, a touch screen, a game machine, a Personal computer, and a Personal Digital Assistant (PDA).

As shown in fig. 6, fig. 6 is a schematic structural diagram of a computer device 400 according to an embodiment of the present application, where the computer device 400 includes a processor 410 having one or more processing cores, a memory 420 having one or more computer-readable storage media, and a computer program stored in the memory 420 and running on the processor. The processor 410 is electrically connected to the memory 420. Those skilled in the art will appreciate that the computer device configurations illustrated in the figures are not meant to be limiting of computer devices and may include more or fewer components than those illustrated, or some components may be combined, or a different arrangement of components.

The processor 410 is a control center of the computer device 400, connects various parts of the entire computer device 400 using various interfaces and lines, performs various functions of the computer device 400 and processes data by running or loading software programs and/or modules stored in the memory 420 and calling data stored in the memory 420, thereby monitoring the computer device 400 as a whole.

In the embodiment of the present application, the processor 410 in the computer device 400 loads instructions corresponding to processes of one or more applications into the memory 420, and the processor 410 executes the applications stored in the memory 420 according to the following steps, so as to implement various functions:

acquiring a to-be-processed rendering instruction corresponding to a to-be-processed frame; determining a predicted load state of the graphics processor according to the rendering instruction to be processed; when the predicted load state of the graphics processor is a high load state, updating the rendering instruction to be processed into a target rendering instruction; and rendering the frame to be processed according to the target rendering instruction through the graphics processor to obtain a rendered image.

The above operations can be implemented in the foregoing embodiments, and are not described in detail herein.

Optionally, as shown in fig. 6, the computer device 400 further includes: touch display 430, radio frequency circuit 440, audio circuit 450, input unit 460 and power supply 470. The processor 410 is electrically connected to the touch display 430, the radio frequency circuit 440, the audio circuit 450, the input unit 460 and the power supply 470, respectively. Those skilled in the art will appreciate that the computer device configuration illustrated in FIG. 6 does not constitute a limitation of computer devices, and may include more or fewer components than those illustrated, or some components may be combined, or a different arrangement of components.

The touch display 430 can be used for displaying a graphical user interface and receiving operation instructions generated by a user acting on the graphical user interface. The touch display screen 430 may include a display panel and a touch panel. The display panel may be used, among other things, to display information entered by or provided to a user and various graphical user interfaces of the computer device, which may be made up of graphics, text, icons, video, and any combination thereof. Alternatively, the Display panel may be configured in the form of a Liquid Crystal Display (LCD), an Organic Light-Emitting Diode (OLED), or the like. The touch panel may be used to collect touch operations of a user on or near the touch panel (for example, operations of the user on or near the touch panel using any suitable object or accessory such as a finger, a stylus pen, and the like), and generate corresponding operation instructions, and the operation instructions execute corresponding programs. Alternatively, the touch panel may include two parts, a touch detection device and a touch controller. The touch detection device detects the touch direction of a user, detects a signal brought by touch operation and transmits the signal to the touch controller; the touch controller receives touch information from the touch sensing device, converts the touch information into touch point coordinates, sends the touch point coordinates to the processor 410, and can receive and execute commands sent by the processor 410. The touch panel may overlay the display panel, and when the touch panel detects a touch operation thereon or nearby, the touch panel transmits the touch operation to the processor 410 to determine the type of the touch event, and then the processor 410 provides a corresponding visual output on the display panel according to the type of the touch event. In the embodiment of the present application, the touch panel and the display panel may be integrated into the touch display screen 430 to implement input and output functions. However, in some embodiments, the touch panel and the display panel may be implemented as two separate components to perform the input and output functions. That is, the touch display 430 can also be used as a part of the input unit 460 to implement an input function.

In the embodiment of the present application, a game application is executed by the processor 410 to generate a graphical user interface on the touch display 430, where a virtual scene on the graphical user interface includes at least one skill control area, and the skill control area includes at least one skill control. The touch display 430 is used for presenting a graphical user interface and receiving an operation instruction generated by a user acting on the graphical user interface.

The rf circuit 440 may be used for transceiving rf signals to establish wireless communication with a network device or other computer device via wireless communication, and for transceiving signals with the network device or other computer device.

The audio circuit 450 may be used to provide an audio interface between a user and a computer device through a speaker, microphone. The audio circuit 450 may transmit the electrical signal converted from the received audio data to a speaker, and convert the electrical signal into a sound signal for output; on the other hand, the microphone converts the collected sound signal into an electrical signal, which is received by the audio circuit 450 and converted into audio data, which is then processed by the audio data output processor 410, and then passed through the radio frequency circuit 440 to be sent to, for example, another computer device, or output to the memory 420 for further processing. The audio circuit 450 may also include an earbud jack to provide communication of peripheral headphones with the computer device.

The input unit 460 may be used to receive input numbers, character information, or user characteristic information (e.g., fingerprint, iris, facial information, etc.), and generate keyboard, mouse, joystick, optical, or trackball signal inputs related to user settings and function control.

The power supply 470 is used to power the various components of the computer device 400. Optionally, the power supply 470 may be logically connected to the processor 410 through a power management system, so as to implement functions of managing charging, discharging, power consumption management, and the like through the power management system. The power supply 470 may also include any component of one or more dc or ac power sources, recharging systems, power failure detection circuitry, power converters or inverters, power status indicators, and the like.

Although not shown in fig. 6, the computer device 400 may further include a camera, a sensor, a wireless fidelity module, a bluetooth module, etc., which are not described in detail herein.

In the foregoing embodiments, the descriptions of the respective embodiments have respective emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.

Therefore, the computer device provided by the embodiment can adjust the load state of the graphics processor when rendering the frame to be processed, optimize the performance of the graphics processor, prevent the graphics processor from being blocked when rendering the frame to be processed, and improve the fluency of the image.

It will be understood by those skilled in the art that all or part of the steps of the methods of the above embodiments may be performed by instructions or by associated hardware controlled by the instructions, which may be stored in a computer readable storage medium and loaded and executed by a processor.

To this end, the present application provides a computer-readable storage medium, in which a plurality of computer programs are stored, and the computer programs can be loaded by a processor to execute the steps in any image rendering method provided by the present application. For example, the computer program may perform the steps of:

Wherein the storage medium may include: read Only Memory (ROM), Random Access Memory (RAM), magnetic or optical disks, and the like.

Since the computer program stored in the storage medium can execute the steps in any image rendering method provided in the embodiment of the present application, beneficial effects that can be achieved by any image rendering method provided in the embodiment of the present application can be achieved, which are detailed in the foregoing embodiments and will not be described herein again.

The image rendering method, the image rendering device, the storage medium, and the computer device provided in the embodiments of the present application are described in detail above, and a specific example is applied in the present application to explain the principle and the implementation of the present application, and the description of the above embodiments is only used to help understand the method and the core idea of the present application; meanwhile, for those skilled in the art, according to the idea of the present application, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present application.

Claims

1. An image rendering method, comprising:

acquiring a to-be-processed rendering instruction corresponding to a to-be-processed frame;

determining a predicted load state of the graphics processor according to the rendering instruction to be processed;

updating the rendering instruction to be processed into a target rendering instruction when the predicted load state of the graphics processor is a high load state;

and rendering the frame to be processed according to the target rendering instruction through the graphics processor to obtain a rendered image.

2. The image rendering method of claim 1, wherein determining a predicted load state of a graphics processor from the pending rendering instructions comprises:

acquiring a first parameter corresponding to the rendering instruction to be processed;

and determining the predicted load state of the graphics processor according to the first parameter and the threshold condition.

3. The image rendering method of claim 2, prior to said determining a predicted load state of a graphics processor from the pending rendering instructions, further comprising:

acquiring a preset frame;

generating a preset rendering instruction corresponding to the preset frame;

acquiring a second parameter corresponding to the preset rendering instruction;

and determining a threshold condition according to the second parameter.

4. The image rendering method of claim 1, wherein the pending rendering instruction comprises a plurality of pending sub-instructions, the updating the pending rendering instruction to a target rendering instruction comprising:

determining a first target object and a second target object in the frame to be processed;

performing instruction cutting on the to-be-processed sub-instruction corresponding to the first target object to obtain a first target to-be-processed sub-instruction;

performing instruction combination on the to-be-processed sub-instruction corresponding to the second target object to obtain a second target to-be-processed sub-instruction;

and updating the rendering instruction to be processed into a target rendering instruction according to the first target sub-instruction to be processed and the second target sub-instruction to be processed.

5. The image rendering method of claim 4, wherein the performing instruction clipping on the to-be-processed sub-instruction corresponding to the first target object to obtain a first target to-be-processed sub-instruction comprises:

determining initial vertex data corresponding to the first target object;

determining target vertex data from the initial vertex data through a clustering algorithm;

and updating the to-be-processed sub-instruction corresponding to the first target object according to the target vertex data to obtain a first target to-be-processed sub-instruction.

6. The image rendering method of claim 4, wherein the performing instruction merging on the to-be-processed sub-instruction corresponding to the second target object to obtain a second target to-be-processed sub-instruction comprises:

acquiring the to-be-processed sub-instruction corresponding to any one second target sub-object;

generating multiple instance rendering instructions corresponding to the multiple second target sub-objects according to the to-be-processed sub-instruction and the multiple same second target sub-objects;

determining the multi-instance rendering instruction as a second target pending sub-instruction.

7. The image rendering method of claim 1, further comprising, after the updating the pending rendering instruction to a target rendering instruction when the predicted load state of the graphics processor is a high load state, further:

and adjusting the working frequency of the graphics processor to be a target working frequency.

8. An image rendering apparatus, comprising:

the system comprises an acquisition unit, a processing unit and a processing unit, wherein the acquisition unit is used for acquiring a to-be-processed rendering instruction corresponding to a to-be-processed frame;

a determining unit, configured to determine a predicted load state of the graphics processor according to the to-be-processed rendering instruction;

the updating unit is used for updating the rendering instruction to be processed into a target rendering instruction when the predicted load state of the graphics processor is a high load state;

and the rendering unit is used for rendering the frame to be processed according to the target rendering instruction through the graphics processor to obtain a rendered image.

9. A computer device comprising a processor and a memory, the memory storing a plurality of instructions; the processor loads instructions from the memory to perform the steps in the image rendering method according to any of claims 1 to 7.

10. A computer readable storage medium storing instructions adapted to be loaded by a processor to perform the steps of the image rendering method according to any one of claims 1 to 7.