WO2024051471A1 - 一种图像处理方法和电子设备 - Google Patents
一种图像处理方法和电子设备 Download PDFInfo
- Publication number
- WO2024051471A1 WO2024051471A1 PCT/CN2023/113746 CN2023113746W WO2024051471A1 WO 2024051471 A1 WO2024051471 A1 WO 2024051471A1 CN 2023113746 W CN2023113746 W CN 2023113746W WO 2024051471 A1 WO2024051471 A1 WO 2024051471A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- coordinate data
- model
- frame image
- electronic device
- rendering
- Prior art date
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 21
- 238000009877 rendering Methods 0.000 claims abstract description 186
- 239000013598 vector Substances 0.000 claims abstract description 145
- 230000033001 locomotion Effects 0.000 claims abstract description 133
- 238000000034 method Methods 0.000 claims abstract description 86
- 230000003068 static effect Effects 0.000 claims abstract description 80
- 230000015654 memory Effects 0.000 claims description 64
- 239000011159 matrix material Substances 0.000 claims description 61
- 230000006870 function Effects 0.000 claims description 54
- 230000009466 transformation Effects 0.000 claims description 23
- 238000004590 computer program Methods 0.000 claims description 7
- 238000004891 communication Methods 0.000 claims description 4
- 238000004364 calculation method Methods 0.000 abstract description 44
- 230000008569 process Effects 0.000 abstract description 36
- 238000012545 processing Methods 0.000 abstract description 15
- 238000010586 diagram Methods 0.000 description 30
- 239000000872 buffer Substances 0.000 description 19
- 238000012546 transfer Methods 0.000 description 12
- 230000003993 interaction Effects 0.000 description 11
- 238000007726 management method Methods 0.000 description 9
- 230000007246 mechanism Effects 0.000 description 9
- 238000013461 design Methods 0.000 description 7
- 238000005516 engineering process Methods 0.000 description 6
- 238000012360 testing method Methods 0.000 description 6
- 230000000694 effects Effects 0.000 description 5
- 238000012986 modification Methods 0.000 description 5
- 230000004048 modification Effects 0.000 description 5
- 230000005540 biological transmission Effects 0.000 description 4
- 238000013473 artificial intelligence Methods 0.000 description 2
- 230000015572 biosynthetic process Effects 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 238000013500 data storage Methods 0.000 description 2
- 238000003786 synthesis reaction Methods 0.000 description 2
- 230000001133 acceleration Effects 0.000 description 1
- 230000003044 adaptive effect Effects 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 230000003190 augmentative effect Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 210000000988 bone and bone Anatomy 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 239000003086 colorant Substances 0.000 description 1
- 238000006073 displacement reaction Methods 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 238000003780 insertion Methods 0.000 description 1
- 230000037431 insertion Effects 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
- G06T7/207—Analysis of motion for motion estimation over a hierarchy of resolutions
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T1/00—General purpose image data processing
- G06T1/20—Processor architectures; Processor configuration, e.g. pipelining
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T1/00—General purpose image data processing
- G06T1/60—Memory management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
- G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Definitions
- Embodiments of the present application relate to the field of image processing, and in particular, to an image processing method and electronic device.
- Frame prediction technology can predict the position of each object in the next frame of image through the relevant data of the rendered frame image. This technology is widely used in display solutions that require frame insertion.
- Current frame prediction technology can determine the position of the same object (model) in different frame images based on the color or brightness matching of each pixel. Then, the motion vector of each pixel is calculated and obtained based on the change in position. Based on the continuity of image display, prediction of future frame images can be achieved based on the motion vector.
- this solution also puts forward higher requirements for computing power and power consumption.
- Embodiments of the present application provide an image processing method and an electronic device that can accurately and quickly calculate motion vectors for static grids and dynamic grids respectively, thereby accurately predicting future frame images. Through separate calculations of static grids and dynamic grids, the computing power overhead in the vector calculation process is saved and the prediction efficiency is improved.
- an image processing method is provided.
- the method is applied to an electronic device.
- the method includes: the electronic device obtains at least two frames of images through image rendering, and the at least two frames of the images include a dynamic grid and a static grid, and the The dynamic grid corresponding model has different coordinates in the world coordinate system in different frame images, and the static grid corresponding model has the same coordinates in the world coordinate system in different frame images.
- the rendering intermediate variables include the model-observation-projection MVP matrix and depth data of the corresponding frame image.
- the first model corresponding grid is a dynamic grid in the image
- the coordinate data includes the normalized device NDC coordinates and drawing parameters of the first model in the corresponding frame image
- the NDC coordinates include at least one vertex coordinate of the drawing Parameters are used by the electronic device to draw the first model. Therefore, the electronic device can determine the next frame of image based on the position of the static grid of the next frame of image and the position of the first model in the next frame of image.
- the electronic device can determine the motion model based on the coordinate data of the corresponding model using a processing mechanism similar to the first model. position in the next image frame. Combining the positions of different moving objects in the next frame of images and the positions of static objects in the next frame of images, the electronic device can comprehensively obtain the next frame of images and achieve prediction of future frame images.
- the electronic device can determine the motion vector of the static mesh through the MVP matrix and depth data.
- the MVP matrix and depth data can be obtained directly through the instruction stream issued by the application, avoiding a large amount of data calculation overhead in the process of determining the static mesh.
- electronic devices can determine each dynamic The motion vector of the mesh. For example, electronic devices can match dynamic grids based on coordinate data, thereby avoiding the computational overhead of pixel-by-pixel brightness/color matching. From this, the calculation of motion vectors of static meshes and dynamic meshes can be realized through the above solution. At the same time, it can significantly reduce computing power overhead and improve computing efficiency.
- the image includes multiple models corresponding to dynamic grids
- determining the position of the first model in the next frame of images based on the coordinate data of the first model in at least two completed images includes: According to the completed coordinate data of the first model in at least two frames of images, the coordinate data of the first model in different frame images is determined based on feature hash value matching. Among them, the feature hash values of the first model in different frame images are the same, and the feature hash values of different models in the same frame image are different. According to the coordinate data of the first model in different frame images, the position of the first model in the next frame image is determined. In this way, in more complex images with multiple dynamic grids, the coordinate correspondence of the same model in different frame images can be determined without matching one by one at the pixel level. This significantly saves the corresponding computing power overhead and time overhead.
- the completed frame image includes the Nth frame image and the N-1th frame image
- the next frame image is the N+1th frame image.
- Determining the position of the static grid of the next frame of image based on the rendering intermediate variables of at least two completed frames of images includes: based on the first MVP matrix and the first depth data of the N-1th frame of image, the Nth The second MVP matrix and the second depth data of the frame image determine the motion vector of the static grid in the Nth frame image. According to the position of the static grid in the N-th frame image and the motion vector of the static grid, the position of the static grid in the N+1-th frame image is determined.
- the electronic device can obtain the relevant data in the Nth frame image after the current frame image is rendered, and combine it with the data in the N-1th frame image to perform the N+th Prediction of 1 frame image.
- the completed frame image may also be two discontinuous frame images, such as the Nth frame image and the N-2th frame image, etc.
- the memory of the electronic device is configured with a rendering intermediate variable cache.
- the method further includes: The first MVP matrix, the first depth data, the second MVP matrix and the second depth data are obtained, and the obtained data are stored in the rendering intermediate variable cache.
- Determining the position of the static grid of the next frame of image based on the rendering intermediate variables of at least two completed frames of images includes: reading the first MVP matrix, the first depth data, and the first depth data from the rendering intermediate variable cache.
- the second MVP matrix and the second depth data are used to determine the position of the static grid in the N+1th frame image.
- the application program issues a first instruction stream to instruct the electronic device to render the N-1th frame image.
- the rendering intermediate variable cache includes a first rendering intermediate variable cache.
- the obtaining and storing the first MVP matrix includes: the electronic device intercepts the first instruction segment used to transmit the first MVP matrix in the first instruction stream, and converts the first MVP matrix according to the first instruction segment. Stored in the first rendering intermediate variable cache.
- the electronic device intercepts the first instruction segment of the first instruction stream according to the first preset identifier.
- the first preset identifier is a uniform parameter.
- the electronic device can use the above solution to back up the first MVP matrix that needs to be used later and store it in a preset location in the memory, such as in the rendering intermediate variable cache. for subsequent calls.
- the application program issues a first instruction stream to instruct the electronic device to render the N-1th frame image.
- the rendering intermediate variable cache includes a second rendering intermediate variable cache.
- the obtaining and storing the first depth data includes: the electronic device intercepts the second instruction segment related to the first depth data in the first instruction stream, and executes the second instruction segment according to the second instruction. Let the first depth data be stored in the second rendering intermediate variable cache.
- the second instruction segment related to the first depth data is used to instruct the electronic device to perform multi-object rendering MRT.
- the electronic device can use the above solution to back up the first depth data that needs to be used later and store it in a preset location in the memory, such as a rendering intermediate variable cache. for subsequent calls.
- the application program issues a second instruction stream to instruct the electronic device to render the Nth frame image.
- the rendering intermediate variable cache includes a third rendering intermediate variable cache.
- the obtaining and storing the second MVP matrix includes: the electronic device intercepts the third instruction segment used to transmit the second MVP matrix in the second instruction stream, and converts the second MVP matrix according to the third instruction segment. Stored in the third rendering intermediate variable cache.
- the application program issues a second instruction stream to instruct the electronic device to render the Nth frame image.
- the rendering intermediate variable cache includes a fourth rendering intermediate variable cache.
- the obtaining and storing the second depth data includes: the electronic device intercepts a fourth instruction segment related to the second depth data in the second instruction stream, and stores the second depth data according to the fourth instruction segment. In this fourth render intermediate variable cache.
- the electronic device can perform the backup storage of the above-mentioned MVP matrix and depth data during the rendering process of each frame of image, for example, perform the above-mentioned steps respectively during the rendering process of the N-1th frame image and the Nth frame image.
- This solution enables the above-mentioned backup stored MVP matrix and depth data to be successfully called when subsequent prediction of future frames is performed to determine the position of the static grid in future frames.
- the completed frame image includes the Nth frame image and the N-1th frame image
- the next frame image is the N+1th frame image.
- Determining the position of the first model in the next frame of images based on the coordinate data of the first model in at least two completed frames of images includes: based on the position of the first model in the N-1th frame of images.
- a coordinate data, and the second coordinate data of the first model in the N-th frame image determine the motion vector of the first model.
- the position of the first model in the N+1-th frame image is determined.
- the frame image may include multiple dynamic grids, corresponding to multiple moving objects. Then, for each dynamic grid, the electronic device can execute this solution, thereby determining the motion vector of each dynamic grid.
- the first model is taken as an example.
- the electronic device is configured with an NDC cache.
- the method further includes: obtaining the first coordinate data of the first model and the The second coordinate data stores the first coordinate data and the second coordinate data in the NDC cache.
- Determining the motion vector of the first model based on the coordinate data of the first model includes: reading the first coordinate data and the second coordinate data of the first model from the NDC cache, and based on the first coordinate data and The second coordinate data determines the motion vector of the first model.
- coordinate data backup is stored in the pre-set NDC cache. To facilitate prediction of subsequent future frames.
- the application program issues a first instruction stream to instruct the electronic device to render the N-1th frame image.
- the NDC cache includes a first NDC cache.
- Obtaining the first coordinate data of the first model includes: before starting to draw the first model in the N-1th frame image, enabling a transformation feedback function, and the GPU of the electronic device is based on the transformation feedback function , when executing the drawing of the first model, the first coordinate data is fed back to the electronic device, the first coordinate data includes the first NDC coordinate data of the first model in the N-1th frame image, and the First The first rendering parameter corresponding to the model in the N-1th frame image.
- the electronic device stores the first coordinate data in the first NDC cache.
- This example provides a specific solution for obtaining coordinate data. For example, by enabling the transformation feedback function, the GPU can feed back the coordinate data obtained by rendering to the electronic device, so that the electronic device can back up and store the coordinate data.
- the method also includes: turning off the transformation feedback function. It can be understood that after completing the storage of the coordinate data of the dynamic grid corresponding to a Drawcall, the electronic device can turn off the transformation feedback function. In this way, if the next Drawcall corresponds to drawing a static grid, there is no need to call back the corresponding coordinate data.
- the application program issues a second instruction stream to instruct the electronic device to render the Nth frame image.
- the NDC cache includes a second NDC cache.
- Obtaining the second coordinate data of the first model includes: before starting to draw the first model in the Nth frame image, enabling a transformation feedback function. Based on the transformation feedback function, the GPU of the electronic device When executing the drawing of the first model, the second coordinate data is fed back to the electronic device.
- the second coordinate data includes the second NDC coordinate data of the first model in the N-th frame image, and the first model in the N-th frame image. The corresponding second drawing parameter in the Nth frame image.
- the electronic device stores the second coordinate data in the second NDC cache. Similar to the acquisition of MVP matrix and depth data, for the rendering process of each frame image, the electronic device can execute this solution for each Drawcall, so that the coordinate data of all dynamic grids in the frame image can be acquired and storage.
- the method before acquiring the first coordinate data and the second coordinate data of the first model, the method further includes: determining that the grid of the first model is a dynamic grid.
- determining that the grid of the first model is a dynamic grid includes: determining that the grid of the first model is a dynamic grid when the coordinate data of the first model in the current frame image is updated. grid.
- the electronic device when receiving a rendering instruction for a model, can determine whether the data in the frame buffer storing coordinate data indicated by the rendering instruction has been updated in the frame image. If so, it indicates that the corresponding model is a motion model. , corresponding to the dynamic grid. On the contrary, if it has not been updated, it means that the corresponding model is a static model and corresponds to a static mesh.
- the Nth frame image includes at least two models with dynamic meshes
- the first model is included in the model with at least two meshes as dynamic meshes
- the NDC cache stores each The model corresponds to coordinate data of different frame images
- the method further includes: determining two coordinate data corresponding to the first model in different frame images in the NDC cache.
- motion vectors can be different for different dynamic meshes.
- two sets of coordinate data can be stored in the NDC cache. For example, the coordinate data of all dynamic grids of the N-1th frame image in the first NDC cache. Another example is the coordinate data of all dynamic grids of the Nth frame image in the second NDC cache. Then, before calculating the motion vector of each model, it is necessary to first match and determine the coordinate data of the same model in different frame images. Continuing with the first model as an example.
- determining the two coordinate data corresponding to the first model in different frame images in the NDC cache includes: determining each coordinate data according to the drawing parameters included in each coordinate data stored in the NDC cache. The corresponding feature hash value. Coordinate data in the NDC cache that has the same feature hash value corresponding to the first coordinate data is determined as the second coordinate data.
- This example provides a simple coordinate matching mechanism for the same model in different frame images. For example, for any coordinate data in the first NDC cache and the second NDC cache, the drawing parameters therein are mapped to a unique feature hash value. In this way, only the first NDC cache and the second NDC In the cache, look for coordinate data with the same feature hash value, which can be used as matching coordinate data for the same model in different frame images.
- determining the two coordinate data corresponding to the first model in different frame images in the NDC cache includes: determining each coordinate data according to the drawing parameters included in each coordinate data stored in the NDC cache.
- the corresponding feature hash value The coordinate data in the NDC cache that has the same feature hash value corresponding to the first coordinate data and the distance between the first vertex coordinates in the two coordinate data is less than the preset distance is determined as the second coordinate data .
- This example provides a more accurate and gradual coordinate matching mechanism.
- the distance in different frame images can be smaller than the preset distance. This can further improve the accuracy of coordinate matching. This improves the calculation accuracy of motion vectors based on coordinate data.
- the drawing parameters include at least one of the following: vertex identification ID, index ID, drawing number, and offset.
- an electronic device in a second aspect, includes one or more processors and one or more memories; the one or more memories are coupled to the one or more processors, and the one or more memories store computer instructions; When the one or more processors execute the computer instructions, the electronic device is caused to perform the method of any one of the above-mentioned first aspect and various possible designs.
- a chip system in a third aspect, includes an interface circuit and a processor; the interface circuit and the processor are interconnected through lines; the interface circuit is used to receive signals from the memory and send signals to the processor, and the signals include data stored in the memory.
- Computer instructions when the processor executes the computer instructions, the chip system performs the method of the first aspect and any of various possible designs.
- a computer-readable storage medium includes computer instructions. When the computer instructions are executed, the method of any one of the above-mentioned first aspect and various possible designs is executed.
- a computer program product includes instructions.
- the computer program product When the computer program product is run on a computer, the computer can execute any one of the above-mentioned first aspect and various possible designs according to the instructions. method.
- Figure 1 is a logical schematic diagram of instruction stream transmission in the image rendering process
- Figure 2 is a schematic diagram of a multi-frame image
- Figure 3 is a schematic diagram of a prediction scheme for future frame images
- Figure 4 is a schematic diagram of the software composition of an electronic device provided by an embodiment of the present application.
- Figure 5 is a schematic diagram of module interaction of an image processing method provided by an embodiment of the present application.
- Figure 6 is a schematic diagram of the composition of coordinate data provided by an embodiment of the present application.
- Figure 7 is a schematic diagram of an NDC cache provided by an embodiment of the present application.
- Figure 8 is a schematic diagram of coordinate data storage provided by an embodiment of the present application.
- Figure 9 is a schematic diagram of module interaction of an image processing method provided by an embodiment of the present application.
- Figure 10 is a schematic diagram of module interaction of an image processing method provided by an embodiment of the present application.
- Figure 11 is a schematic diagram of a memory storing data provided by an embodiment of the present application.
- Figure 12 is a schematic diagram of a memory storing data provided by an embodiment of the present application.
- Figure 13 is a schematic diagram of module interaction of an image processing method provided by an embodiment of the present application.
- Figure 14 is a schematic diagram of module interaction of an image processing method provided by an embodiment of the present application.
- Figure 15 is a schematic diagram of the correspondence between coordinate data and hash values provided by the embodiment of the present application.
- Figure 16 is a schematic diagram of coordinate data matching provided by an embodiment of the present application.
- Figure 17 is a schematic diagram of coordinate data matching provided by an embodiment of the present application.
- Figure 18 is a schematic diagram of module interaction of an image processing method provided by an embodiment of the present application.
- Figure 19 is a schematic flowchart of an image processing method provided by an embodiment of the present application.
- Figure 20 is a schematic diagram of the composition of an electronic device provided by an embodiment of the present application.
- Figure 21 is a schematic diagram of the composition of a chip system provided by an embodiment of the present application.
- Electronic devices can run various applications to provide users with rich functions.
- an application program in the electronic device may instruct the display of the electronic device to provide a display function to the user.
- the display function may include functions such as displaying video streams and image streams.
- a video stream can be composed of multiple frames of images.
- the electronic device can quickly and sequentially play frame images, so that the user can see a dynamic picture composed of continuously played frame images through the display of the electronic device.
- the number of frame images played by an electronic device per unit time can be identified by the frame rate. The higher the frame rate, the more frame images the electronic device plays per unit time, and the corresponding dynamic images will be clearer and more realistic.
- the application can display the electronic device to perform corresponding drawing through the rendering instruction stream, thereby obtaining the display information and displaying it through the monitor.
- the application when a frame image needs to be displayed, the application can issue a rendering instruction stream to instruct the electronic device to perform a rendering operation according to the rendering instruction stream.
- the central processing unit (Central Processing Unit, CPU) of the electronic device can receive the rendering instruction stream, and call the corresponding application programming interface (application programming interface, API) in the rendering environment installed in the electronic device according to the rendering instruction stream.
- the CPU can instruct the graphics processor (Graphic Processing Unit, GPU) in the electronic device with image rendering function to perform the corresponding rendering operation by calling the API.
- the rendering results obtained after the GPU performs a series of rendering operations can correspond to the display information.
- the display information may include color information, brightness information, depth data, normal information, etc. corresponding to each pixel in the current frame image.
- the display can obtain the rendering result and display it accordingly.
- the acquisition process of display information is similar to Figure 1. That is to say, when multiple frame images need to be displayed quickly, the electronic device needs to perform corresponding rendering operations according to each rendering instruction stream issued by the application program in order to obtain the display information of the corresponding frame image.
- frame image prediction technology can be used to predict the display situation of the next frame image based on the frame image that has been rendered.
- the display information of the next frame of image can be obtained without performing the rendering operation of the next frame of image.
- the electronic device can predict and obtain the A+1-th frame image based on the A-1th frame image and the A-th frame image. .
- Figure 3 is an example of a frame image prediction scheme.
- the object 21 and the object 22 may be included in the A-1th frame image.
- the A-th frame image may also include object 21 and object 22.
- the position of the object 21 in the A-1th frame and the A-th frame image is the same. That is, the object 21 does not move.
- the position of object 22 in the A-1 and A-th frame images is different. That is, the object 22 is displaced.
- the electronic device may divide the frame image into multiple tiles. Each tile contains multiple pixels.
- the object 21 may be located in the area of the block 23 .
- Object 22 may be located in the area of tile 24 .
- object 21 is still located in the area of block 23.
- Object 22 then moves to tile 25.
- the electronic device can determine the motion vector of each block in two adjacent frame images in pixel units.
- the electronic device can determine, for each pixel in block 23 , that in the two frames of image, block 23 Whether the color information of each pixel included in has changed.
- the electronic device can determine that the motion vector of the block 23 is 0, that is, In these two frames of images, block 23 does not move. In this way, the electronic device can predict that in the A+1th frame image, the block 23 will still use the previous moving state, that is, remain stationary.
- the electronic device can determine, for each pixel in the block 24 , that in the two frame images, the block 24 Whether the color information of each pixel included in has changed.
- the object 22 moves from the block 24 in the A-1th frame image to the block 25 in the A-th frame image.
- the electronic device performs the above color comparison for the block 24, it can be determined that the corresponding color information of the block 24 has changed in the two frames of images.
- the electronic device can use the color information of each pixel in the block 24 in the A-1th frame image as a reference to find blocks corresponding to the reference in other blocks adjacent to the block 24 in the A-th frame image.
- the electronic device may determine that in the A-th frame image, the color information of the block 25 corresponds to the color information of the block 24 in the A-1-th frame image. This also means that during the switching from the A-1th frame image to the A-th frame image, the block 24 moves from the position of the block 24 in the A-1th frame image to the position of the block 25 in the A-th frame image. . In this way, the electronic device can determine that the motion vector of block 24 between the A-1th frame image and the A-th frame image is from block 24 to block 25 .
- the absolute length of the motion vector can be determined based on the distance between the corresponding color information before and after the movement of the block 24.
- the motion vector moving block 24 to block 25 is denoted as motion vector 26 .
- object 22 will also move along motion vector 26.
- the electronic device can predict that in the A+1th frame image, the block 25 in the Ath frame image can inherit the previous movement state along the motion vector 26, then in the A+1th frame image , the block 25 will move outside the frame image. Therefore, based on the frame image prediction mechanism, the electronic device can determine that the object 22 in the block 24 of the A-1 frame image and the object 22 in the block 25 of the A-th frame image will be in the A+1-th frame. The image moves outside the frame and no longer needs to be displayed on the frame image.
- the electronic device can predict and determine the content to be displayed in the A+1th frame image based on the motion vectors of each block in the A-1th frame image and the Ath frame image.
- the electronic device can calculate the motion vector in each block through a color matching mechanism based on a two-dimensional image (such as a two-dimensional image corresponding to the observation space).
- a two-dimensional image such as a two-dimensional image corresponding to the observation space.
- the block division is relatively rough. Although it can reduce the amount of calculation, the prediction results will be greatly affected.
- the calculation process based on the color matching mechanism will also introduce large computing power and power consumption overhead.
- embodiments of the present application provide an image processing method that enables an electronic device to calculate motion vectors for dynamic objects and stationary objects respectively. For example, combined with the depth data in the image rendering process, the full-screen motion vector corresponding to the stationary object is calculated from the perspective of three-dimensional space.
- Another example is to use hash matching to simplify the matching process of different models and reduce the movement of dynamic objects through the normalized device coordinates (NDC) spatial coordinates and related rendering parameters corresponding to each drawing command (Drawcall). Operational overhead during vector calculations.
- NDC normalized device coordinates
- the technical solutions provided by the embodiments of the present application can be applied to electronic devices with image display functions.
- the electronic device may include a mobile phone, a foldable electronic device, a tablet computer, a desktop computer, a laptop computer, a handheld computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a cellular phone, a personal computer Digital assistant (personal digital assistant, PDA), augmented reality (AR) device, virtual reality (VR) device, artificial intelligence (artificial intelligence, AI) device, wearable device, vehicle-mounted device, smart home equipment, or at least one of smart city equipment.
- PDA personal digital assistant
- AR augmented reality
- VR virtual reality
- AI artificial intelligence
- wearable device wearable device
- vehicle-mounted device smart home equipment
- smart home equipment smart home equipment
- the electronic device involved in the embodiments of the present application may include a processor, an external memory interface, an internal memory, a universal serial bus (USB) interface, a charging Management module, power management module, battery, antenna 1, antenna 2, mobile communication module, wireless communication module, audio module, speaker, receiver, microphone, headphone interface, sensor module, buttons, motor, indicator, camera, display, As well as subscriber identification module (SIM) card interface, etc.
- a processor an external memory interface
- an internal memory a universal serial bus (USB) interface
- a charging Management module power management module
- battery antenna 1, antenna 2
- mobile communication module wireless communication module
- audio module audio module
- speaker speaker
- receiver microphone
- headphone interface headphone interface
- sensor module buttons, motor, indicator, camera, display
- SIM subscriber identification module
- the sensor module can include a pressure sensor, a gyroscope sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, etc.
- the above hardware composition does not constitute a specific limitation on electronic equipment.
- the electronic device may include more or fewer components, some components may be combined, some components may be separated, or different components may be arranged.
- FIG. 4 is a schematic diagram of the software composition of an electronic device provided by an embodiment of the present application.
- the electronic device may include an application (Application, APP) layer, a framework (Framework) layer, a system library, and a hardware (HardWare) layer. wait.
- application Application, APP
- framework Framework
- HardWare hardware
- the application layer can also be called the application layer.
- the application layer may include a series of application packages.
- Application packages can include camera, gallery, calendar, calling, map, navigation, WLAN, Bluetooth, music, video, SMS and other applications.
- the application package may also include applications that need to display images or videos to users by rendering images.
- video can be understood as the continuous playback of multiple frames of images.
- the application that needs to render an image may include a game application, for example wait.
- the framework layer can also be called the application framework layer.
- This framework layer can provide API and programming framework for applications in the application layer.
- the framework layer includes some predefined functions.
- the framework layer may include a window manager, a content provider, a view system, a resource manager, a notification manager, an activity manager, an input manager, etc.
- the window manager provides window management service (Window Manager Service, WMS).
- WMS can be used for window management, window animation management, surface management, and as a transfer station for the input system.
- Content providers are used to store and retrieve data and make this data accessible to applications. This data can include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.
- the view system includes visual controls, such as controls that display text, controls that display pictures, etc.
- a view system can be used to build applications.
- the display interface can be composed of one or more views.
- a display interface including a text message notification icon may include a view for displaying text and a view for displaying pictures.
- the resource manager provides various resources to applications, such as localized strings, icons, pictures, layout files, video files, etc.
- the notification manager allows applications to display notification information in the status bar, which can be used to convey notification-type messages and can automatically disappear after a short stay without user interaction.
- the notification manager is used to notify download completion, message reminders, etc.
- the notification manager can also be notifications that appear in the status bar at the top of the system in the form of charts or scroll bar text, such as notifications for applications running in the background, or notifications that appear on the screen in the form of conversation windows.
- the Activity Manager can provide Activity Management Service (AMS), which can be used for the startup, switching, and scheduling of system components (such as activities, services, content providers, and broadcast receivers) as well as the management and scheduling of application processes.
- AMS Activity Management Service
- the input manager can provide input management service (Input Manager Service, IMS).
- IMS can be used to manage system input, such as touch screen input, key input, sensor input, etc. IMS takes out events from the input device node and distributes the events to appropriate windows through interaction with WMS.
- one or more functional modules may be provided in the framework layer to implement the solution provided by the embodiment of the present application.
- the framework layer may be provided with an interception module, a data dump module, a vector calculation module, etc. These module settings can be used to support electronic devices to implement the image processing method provided by the embodiments of this application. Its specific functions and implementation will be described in detail later.
- the electronic device may also be provided with a system library including a graphics library.
- the graphics library may include at least one of the following: Open Graphics Library (Open GL), Open GL for Embedded Systems (OpenGL ES), Vulkan, etc.
- other modules may also be included in the system library. For example: surface manager (surface manager), media framework (Media Framework), standard C library (Standard C library, libc), SQLite, Webkit, etc.
- the surface manager is used to manage the display subsystem and provides two-dimensional (2D) and fusion of three-dimensional (3D) layers.
- the media framework supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc.
- the media library can support a variety of audio and video encoding formats, such as: Moving Pictures Experts Group (MPEG4), H.264, Moving Picture Experts Group Audio Layer3 (MP3), Advanced Audio Coding (AAC), Adaptive Multi-Rate (AMR), Joint Photographic Experts Group (JPEG, or JPG), Portable Network Graphics , PNG) etc.
- MPEG4 Moving Pictures Experts Group
- MP3 Moving Picture Experts Group Audio Layer3
- AAC Advanced Audio Coding
- AMR Adaptive Multi-Rate
- JPEG Joint Photographic Experts Group
- PNG Portable Network Graphics
- OpenGL ES and/or Vulkan provide the drawing and manipulation of 2D graphics and 3D graphics in applications.
- SQLite provides a lightweight relational
- each module in the framework layer can call the corresponding API in the graphics library to instruct the GPU to perform the corresponding rendering operation.
- a hardware layer may also be included in the electronic device.
- This hardware layer can include CPU, GPU, and memory with storage function (such as memory).
- the CPU can be used to control each module in the framework layer to implement their respective functions
- the GPU can be used to execute the API in the graphics library (such as OpenGL ES) called according to the instructions processed by each module in the framework layer. Corresponding rendering processing.
- the solutions provided by the embodiments of this application can be applied to the electronic device as shown in FIG. 4 . It should be noted that the example in Figure 4 does not constitute a restriction on electronic equipment. In other embodiments, the electronic device may include more or fewer components. The embodiments of this application do not limit the specific composition of the electronic device.
- the technical solution provided by the embodiment of the present application can collect and cache corresponding data when rendering operations are performed on the current frame image and the previous frame image. For example, take the current frame image as the Nth frame image, and the electronic device also collects data of the N-1th frame image.
- the interception module provided in the framework layer can be used to intercept the required instruction flow during the execution of the rendering operation of each frame image.
- the interception module can be used to intercept the flow of rendering instructions issued by the application.
- the interception module may also have simple judgment capabilities.
- the interception module can determine whether the instruction stream is used to draw static objects or dynamic objects based on whether the model information corresponding to the rendering instruction stream is updated.
- the model information may include coordinate data of the model to be drawn, etc.
- the interception module can also identify that the current instruction stream includes the depth (Depth) information of the current frame image and/or the Model-View-Projection (MVP) matrix based on the preset function or the parameters carried by the function. relevant data.
- the depth data and MVP matrix can be collectively referred to as rendering intermediate variables.
- the interception module can also be used to send the instruction stream instructing the rendering of dynamic objects to the data dump module for subsequent processing.
- the interception module can also be used to send an instruction stream instructing rendering of a static object to the GPU of the electronic device for subsequent processing.
- the interception module can also be used to send the instruction stream including rendering intermediate variables to the data dump module for subsequent processing.
- the data dump module can be used to perform corresponding data dump operations according to the instruction flow from the interception module.
- the data dump module can enable the transform feedback function when receiving a stream of rendering instructions for a dynamic object, and send the stream of rendering instructions for a pair of dynamic objects to the GPU. So that the GPU can perform corresponding rendering operations.
- the data dump module obtains some data generated by the GPU during rendering operations.
- the data dump module can obtain the coordinate data corresponding to the dynamic object in the NDC space based on the transformation feedback function.
- the data transfer module can transfer the coordinate data corresponding to the dynamic object in the NDC space to the memory of the electronic device for subsequent use.
- the data dump module can store rendering intermediate variables in a rendering intermediate variable cache created in advance in memory.
- the instruction stream intercepted by the above interception module will eventually be sent to the GPU to perform related rendering operations.
- Each data in the rendering process is invisible to the electronic device (such as the CPU of the electronic device). Then, in order to facilitate the subsequent calculation of motion vectors of adjacent frame images, in this application, the electronic device can back up and store the data that needs to be used later into the memory of the electronic device through the data transfer module.
- the MVP matrix of each frame image, the depth data of each frame image, and each frame can be stored in the memory of the electronic device at a specific location Coordinate data corresponding to dynamic objects included in the image.
- the above-mentioned data stored in a specific location can be used to support the electronic device in predicting the N+1th frame image.
- the vector calculation module in the electronic device can calculate and obtain the static vector corresponding to the N-1th frame image to the Nth frame image based on the MVP matrix of each frame image and the depth data of each frame image.
- the motion vector of the static object can also correspond to the full-screen motion vector.
- the vector calculation module in the electronic device can calculate and obtain the motion vector of the dynamic object corresponding to the N-1th frame image to the Nth image based on the coordinate data corresponding to the dynamic object included in each frame image.
- the electronic device can predict and determine the specific position of each object in the N+1th frame image based on the rendering result of the Nth frame image, as well as the motion vector of the static object and the motion vector of the dynamic object, thereby realizing the N+1th frame image. Prediction of frame images.
- the electronic device can calculate motion vectors for static objects and dynamic objects respectively.
- the full-screen motion vector calculated based on three-dimensional information of the MVP matrix and depth data is obviously more accurate than the current motion vector calculated based on two-digit information.
- the calculation of the motion vector of the dynamic object is decoupled from the calculation of the full-screen motion vector. The calculation can obtain a more accurate motion vector of the dynamic object, and then more accurate calculation of the more precise operation can be obtained to obtain the dynamic object. More accurate prediction effect in the N+1th frame image.
- FIG. 5 a schematic diagram of module interaction is provided for an image processing method provided by an embodiment of the present application.
- This solution can be applied to the rendering process of any frame image (such as the N-1th frame image, the Nth frame image, etc.). This enables the electronic device to back up and store the coordinate data of dynamic objects.
- this solution can include:
- the application program can issue an instruction stream including multiple Drawcalls during the process of instructing the electronic device to render a frame of image.
- the instruction stream 511 may correspond to a Drawcall and be included in the instruction stream issued by the application program for the rendering process of a certain frame of image.
- the Drawcall corresponding to the instruction stream 511 can be used to instruct the electronic device to draw the dynamic object of the current frame image.
- the instruction stream 511 may correspond to a drawing instruction of a dynamic object.
- the application will issue corresponding Drawcalls for the multiple dynamic objects.
- the electronic device can execute the process shown in Figure 5 for the Drawcall corresponding to each dynamic object, so as to realize backup storage of the coordinate data of each dynamic object.
- the interception module determines that the instruction stream 511 is used to draw a dynamic grid.
- the dynamic mesh corresponds to the mesh (Mesh) of the dynamic object.
- the mesh of a static object can be called a static mesh.
- the interception module may have the ability to determine whether the current instruction stream is used to instruct drawing of a dynamic grid or a static grid. For example, the interception module may determine that the instruction stream 511 is used to draw a dynamic mesh when the instruction stream 511 indicates that the drawn model information is updated. Correspondingly, the interception module may determine that the instruction stream 511 is used to draw a static mesh when the instruction stream 511 indicates that the drawn model information has not been updated.
- the interception module can compare whether the corresponding model information (such as coordinate information, etc.) of the same model in the current Drawcall and the previous Drawcall has changed, and determine whether the model to be drawn by the current Drawcall is a dynamic mesh or a static mesh. grid. For example, the interception module can compare whether the coordinate information stored in the frame buffer of the corresponding model after the current Drawcall is issued is the same as the coordinate information in the same frame buffer in the previous frame image to implement the above judgment mechanism.
- the corresponding model information such as coordinate information, etc.
- the instruction stream 511 is used to draw a dynamic grid as an example.
- the interception module calls back the instruction flow 511 to the GPU.
- the interception module can intercept the instruction flow from the application and determine the subsequent strategy, and can also call back the intercepted instruction flow to the GPU to implement native logic.
- the interception module can intercept the instruction stream 511 and determine that the instruction stream 511 is used to draw a dynamic grid, and can also call back the instruction stream 511 to the GPU so that the GPU can respond accordingly.
- the interception module sends the dynamic identification to the data transfer module.
- the dynamic identifier can be used to indicate that the current Drawcall is used to draw dynamic grids.
- the interception module can notify the data dump module through the dynamic identifier to perform corresponding data backup storage. For example, perform backup storage of the coordinate data of the dynamic object.
- the data transfer module instructs the GPU to enable the transformation feedback function.
- the transformation feedback function can be used to collect data during the GPU's subsequent rendering operations.
- the GPU can feed back the coordinate data of the current model (that is, the dynamic object corresponding to the instruction stream 511) generated during the rendering operation performed according to the instruction stream 511 to the data dump module.
- the transform feedback function may include a tramsform feedback function.
- the data dump module can instruct the GPU to enable the transformation feedback function by calling the function used to enable the tramsform feedback function.
- the GPU executes the corresponding rendering operation according to the instruction stream 511 and obtains the rendering result 521.
- the interception module can intercept the instruction stream 511 and call back the instruction stream 511 to the GPU, thereby realizing native rendering logic.
- the GPU can receive the instruction stream 511 from the interception module, perform the corresponding rendering operation, and obtain the rendering result 521.
- the rendering operation performed by the GPU according to the instruction stream 511 can be referred to the process shown in Figure 1 for the specific implementation process. That is, the interception module can call the corresponding API according to the instruction flow 511, so that The GPU performs corresponding operations according to this API.
- the rendering operation performed by the GPU according to the instruction stream can be implemented using the above process, which will not be described again.
- the electronic device can implement the rendering response to the instruction stream 511 and obtain the corresponding dynamic grid rendering result.
- the GPU sends the rendering result 521 to the memory.
- the memory stores the rendering result 521.
- the electronic device can store the rendering result 521 in the memory, so that based on the rendering result 521 and other rendering results of the current frame image, the display information corresponding to the current frame image can be obtained through operations such as synthesis and denoising.
- the GPU calls back the coordinate data 531 according to the transformation feedback function.
- the data transfer module is used to obtain the coordinate data of the current Drawcall instruction to draw the dynamic grid by enabling the transformation feedback function.
- step S509 the GPU can perform the rendering operation according to the instruction stream 511 and, after acquiring the coordinate data 531 corresponding to the current Drawcall, call back the coordinate data 531 to the GPU through the transformation feedback function.
- the coordinate data may include the specific coordinates of each vertex of the model corresponding to the Drawcall in the NDC space.
- the specific coordinates in the NDC space may be referred to as NDC coordinates for short.
- the application program when it instructs the electronic device to draw a model, it can carry the vertex coordinates of the model in the instruction stream.
- the vertex coordinates can be based on the local space established by the model itself.
- the application program can also send the MVP matrix to the electronic device.
- the electronic device can convert the fixed-point coordinates based on the local space into the coordinates of the world space, the coordinates of the observation space, and the coordinates of the clipping space.
- the clipping space coordinates can correspond to the coordinates on the display screen.
- the NDC coordinates can be the normalized device coordinates after the local space has been transformed by the MVP matrix.
- the electronic device can also restore the coordinates of each vertex in the world space. For example, by multiplying the NDC coordinates by the inverse matrix of the VP matrix, and then normalizing and restoring according to the w component, the corresponding coordinates in the world space can be obtained.
- the coordinate data called back by the GPU to the data dump module may also include the drawing parameters corresponding to the current Drawcall.
- the drawing parameters may include at least one of the following: vertex ID (Vertex Id), index ID (Index Id), draw count (Draw Count), offset (Draw Offset), etc.
- drawing parameters can be necessary parameters when the GPU performs rendering operations, and can be carried in the instruction stream 511 and sent to the GPU.
- different Drawcalls correspond to different drawing parameters. That is, drawing parameters can be used to identify different Drawcalls. That is, drawing parameters can be used to mark different dynamic meshes. In this way, the electronic device can identify and match corresponding dynamic grids in different frame images based on the drawing parameters in the coordinate data.
- the drawing parameters may also be sent by the interception module to the data dump module.
- the interception module can send the drawing parameters therein to the data dump module. Then, the GPU does not need to carry the drawing parameters when calling back the coordinate data.
- the data transfer module sends the coordinate data 531 to the memory.
- the memory stores coordinate data 531 in the NDC cache.
- the data dump module can back up and store the data required for subsequent prediction of future frames (such as the N+1th frame) during the image rendering process of each frame.
- the data dump module can obtain the coordinate data 531 of the dynamic grid from the GPU, and store the coordinate data 531 in a specific location in the memory of the electronic device.
- the specific location where the coordinate data is stored may be created in advance.
- the specific location may be an NDC cache that is pre-created in the memory.
- Figure 7 shows a schematic diagram of an NDC cache.
- the memory in the electronic device may create an NDC cache including a plurality of sub-caches in advance.
- the plurality of sub-caches may respectively constitute two cache groups, such as a first NDC cache and a second NDC cache.
- Each cache group can be used to store the coordinate data of the dynamic grid of a frame image.
- the first NDC buffer can be used to store the coordinate data of the dynamic grid of the N-1th frame image.
- the second NDC cache may be used to store coordinate data of the dynamic grid of the Nth frame image.
- the first NDC cache may include NDC cache A1 to NDC cache An.
- each of NDC cache A1 to NDC cache An is a sub-cache in the NDC cache.
- This sub-cache can correspondingly store the coordinate data of a dynamic grid of the N-1th frame image.
- the second NDC cache may include NDC cache B1 to NDC cache Bm.
- each of the NDC caches B1 to NDC caches Bm may also correspond to a sub-cache. This sub-cache can correspondingly store the coordinate data of a dynamic grid of the Nth frame image.
- the coordinate data of the n dynamic grids are: coordinate data A1 to coordinate data An. Then, based on the above steps S501-S511, for each coordinate data, the data dump module can store it in an NDC cache.
- the data transfer module can store the coordinate data A1 in a sub-cache of the first NDC cache.
- the coordinate data A1 is stored in the NDC cache A1.
- the data transfer module can store the coordinate data B1 in a sub-cache of the second NDC cache.
- the coordinate data B1 is stored in the NDC buffer B1.
- the coordinate data corresponding to each dynamic grid in the N-1th frame image can be stored in the first NDC cache.
- the coordinate data corresponding to each dynamic grid in the Nth frame image can be stored in the second NDC cache.
- the coordinate data of all dynamic grids can be backed up and cached in the NDC cache.
- the data dump module can also execute S512, that is, turn off the transformation feedback function.
- S512 that is, turn off the transformation feedback function.
- this solution can include:
- the application issues an instruction flow 512.
- the Drawcall corresponding to the instruction stream 51 can be used to instruct the electronic device to draw the dynamic object of the current frame image. Combined with the description in S501 in Figure 5.
- the instruction stream 512 may be one of multiple instruction streams sent by the application program when instructing the electronic device to render the N-1th frame image or the Nth frame image.
- the interception module can identify that the instruction stream 512 is used to instruct static mesh drawing according to the following steps of S902, and execute the corresponding policy.
- the interception module intercepts the instruction flow 512 and determines that the instruction flow 512 is used to draw a static grid.
- the interception module may determine whether the instruction stream 512 is used to draw a static mesh based on whether the model information of the drawing model indicated by the instruction stream 512 has been updated. For example, after the current Drawcall is issued, the interception module can determine that the data in the frame buffer storing coordinate information of the corresponding model in the instruction stream 512 is the same as the coordinate data in the same frame buffer in the previous frame image and has not changed, then it indicates that The coordinate data of the model corresponding to this instruction stream 512 has not been updated, that is, it is a static grid.
- the electronic device does not need to back up the coordinate data for storing the static grid. Then, when it is determined that the instruction stream 512 is used to instruct drawing of a static mesh, execution of the following S903 may be triggered.
- the interception module calls back the instruction stream 512 to the GPU.
- the GPU executes the corresponding rendering operation according to the instruction stream 512 and obtains the rendering result 522.
- the GPU sends the rendering result 522 to the memory.
- the memory stores the rendering result 522.
- the electronic device can perform a rendering operation on the static grid according to the native logic and obtain the corresponding rendering result 522. Then, by combining the rendering result 522 with other rendering results in the current frame image, through operations such as synthesis and denoising, the display data of the current frame image can be obtained.
- the application instructs the electronic device to draw a certain frame image
- it also needs to issue other rendering instruction flows to the electronic device.
- the application can deliver rendering intermediate variables such as the MVP matrix and depth data used for depth rendering.
- the electronic device can also back up and store the rendering intermediate variable to facilitate subsequent determination of the full-screen motion vector.
- FIG. 10 is a schematic diagram of interaction between modules of yet another image processing method provided by an embodiment of the present application.
- This solution can be applied to the rendering process of any frame image. As shown in Figure 10, this solution can include:
- the instruction stream 513 may carry rendering intermediate variables corresponding to the current frame image.
- the application when starting the rendering of a certain frame of image, can send various data needed in the process of rendering the frame of image to the electronic device.
- the application program can send rendering intermediate variables including the MVP matrix and depth data to the electronic device through the instruction stream 513 .
- the interception module determines that the instruction stream 513 includes the rendering intermediate variable of the current frame image.
- the interception module may determine that the instruction flow 513 includes rendering intermediate variables based on the preset function or the parameters carried by the function.
- the method of transmitting the MVP matrix is generally relatively fixed.
- the application can pass the MVP matrix through a function carrying uniform parameters.
- the interception module can determine that the instruction stream 513 includes the MVP function when the instruction stream 513 includes a function carrying a uniform parameter.
- the interception module can be obtained based on the corresponding information of Multiple Render Targets (MRT).
- MRT Multiple Render Targets
- electronic devices can output RGBA colors, normals, depth data or texture coordinates to multiple buffers through one rendering.
- the output buffer corresponding to MRT can be instructed by the application program through the instruction stream.
- the interception module may determine that the instruction stream 513 includes a command instructing MRT rendering when the instruction stream 513 instructs to output multiple rendering results to different buffers at one time.
- the command instructing MRT rendering includes a frame buffer for storing depth data.
- the interception module can determine that when the instruction stream 513 instructs MRT rendering, it can determine that the depth data of the current frame image can be obtained through the instruction stream 513 .
- the depth data of the current frame image may also be called a full-screen depth map.
- the interception module can determine the transmission instructions of the MVP matrix based on the function carrying uniform parameters.
- the interception module can also determine the depth data transmission instructions based on the MRT rendering commands.
- the MRT rendering command may correspond to an instruction to output multiple rendering results to different buffers at one time.
- the MVP matrix transfer instruction and the depth data transfer instruction may be included in the instruction stream 513 .
- the interception module sends the instruction stream 513 to the data dump module.
- the data dump module sends the rendering intermediate variables in the instruction stream 513 to the memory.
- the memory stores the rendering intermediate variables of the current frame image in the rendering intermediate variable cache.
- the data dump module can obtain the MVP matrix of the current frame image according to the function carrying the uniform parameter, and send the MVP matrix to the memory for storage.
- the data dump module can determine the frame buffer ID that stores depth data according to the MRT rendering command. Then, the data dump module can read the depth data from the corresponding frame buffer and send it to the memory for storage.
- the electronic device can create the corresponding storage space in the memory before executing S1004 and S1005 (ie, storing the intermediate variable cache).
- an electronic device may create a cache of intermediate variables in memory.
- the rendering intermediate variables of the corresponding frame image can be backed up and stored in the memory (such as the intermediate variable cache).
- the coordinate data of all dynamic grids corresponding to the frame image can also be backed up and stored in the memory (such as the NDC cache).
- the coordinate data 1111 of all the dynamic grids of the N-1th frame image can be backed up and stored in the NDC cache in the memory.
- the MVP matrix 1121 of the N-1th frame image and the depth data 1131 can be backed up and stored in the intermediate variable cache in the memory.
- the rendering result corresponding to the N-1th frame image may also be stored in the memory.
- the electronic device can perform rendering of the Nth frame image according to the instructions of the application program.
- all dynamic meshes of the N-1th frame image can be backed up and stored in the NDC cache in the memory. Coordinate data 1111, and coordinate data 1112 of all dynamic grids of the Nth frame image.
- the rendering intermediate variables of the N-1th frame image and the rendering intermediate variables of the Nth frame image can be backed up and stored.
- the rendering intermediate variables of the N-1th frame image may include MVP matrix 1121 and depth data 1131.
- the rendering intermediate variables of the Nth frame image may include MVP matrix 1122 and depth data 1132.
- the rendering result corresponding to the N-1th frame image can also be overwritten in the memory, and the rendering result corresponding to the Nth frame image is stored.
- the electronic device can calculate the motion vector from the N-1th frame image to the Nth frame image based on various data backed up and stored in the memory as shown in Figure 12 , thereby achieving the purpose of predicting the N+1th frame image based on the motion vector.
- the electronic device can calculate the motion vector of a static object between two frames of images and the motion vector of a dynamic object between two frames of images through different solutions.
- FIG. 13 it is a schematic diagram of another image processing method provided by an embodiment of the present application.
- the motion vector of a static object can correspond to the motion vector of all other display elements in the frame image except the dynamic object. That is, the motion vector of a static object can correspond to a full-screen motion vector.
- this solution can include:
- the vector calculation module reads the depth data and MVP matrix from the rendering intermediate variable cache.
- the vector calculation module can read the corresponding depth data and MVP matrix of the N-1th frame image and the Nth frame image respectively from the intermediate variable cache.
- the vector calculation module may read the MVP matrix 1121, depth data 1131, MVP matrix 1122 and depth data 1132 from the rendering intermediate variable cache.
- each frame of image can correspond to an MVP matrix.
- the depth data of each frame image may include: depth data corresponding to each pixel of the frame image.
- the vector calculation module calculates and obtains the motion vector of each pixel on the screen.
- the vector calculation module can be preset with formula (1), which is used to calculate the motion vector of each pixel on the screen between the N-1th frame image and the Nth frame image based on the data read in S1301. .
- VP p and VP c are the motion vector of the pixel.
- P c and P p are the three-dimensional coordinates of the pixel in the camera coordinate system in the two frames of images respectively.
- VP p and VP c correspond to the VP matrices of the two frames before and after respectively.
- the three-dimensional coordinates of the pixel in the camera coordinate system in the N-1th frame image or the Nth frame image can be obtained through the following formula (2).
- (u, v) is any pixel on the screen.
- z is the depth of the pixel in the current frame image.
- P x is the three-dimensional coordinate of the pixel in the camera coordinate system of the current frame image (such as the N-1th frame image or the Nth frame image).
- the motion vector of each pixel can be calculated and obtained based on the three-dimensional coordinates of the pixel including depth data.
- the set of motion vectors of each point on the screen can correspond to the full-screen motion vector.
- the vector calculation module sends the full-screen motion vector including the motion vectors of all pixels to the memory.
- the memory stores the full-screen motion vector.
- the electronic device can calculate and obtain the motion vector of the static object based on the depth data of the two frames before and after completing the rendering of the Nth frame image. It can be understood that compared with the existing motion vector calculation method based on two-dimensional information, the solution shown in Figure 13 combines depth data to obtain motion vectors, so the calculated full-screen motion vector is more accurate.
- the full-screen motion vector calculated by the above formula (1) and formula (2) can be used for Realize the prediction of the position of a static object (that is, a stationary object in the world coordinate system) in the N+1th frame image.
- a static object that is, a stationary object in the world coordinate system
- the position of the static object in the N+1th frame image can be obtained by calculating the displacement along the full-screen motion vector.
- the motion trends can be different in different frame images.
- the electronic device can separately calculate and obtain corresponding motion vectors for each dynamic object. Then predicting the position of the dynamic object in the N+1th frame image based on the customized motion vector can obviously be more accurate.
- the motion vector calculation scheme of dynamic objects in the embodiment of the present application is illustrated below with reference to Figures 14 to 18.
- the corresponding motion vector can be calculated and determined for each dynamic object. This solution can be implemented in two steps: grid matching and motion vector calculation. Each is explained below.
- this solution can include:
- the matching module reads coordinate data from the memory.
- the matching module may read the coordinate data of the N-1th frame image and the Nth frame image from the NDC cache of the memory.
- the matching module can read the coordinate data corresponding to each motion grid in the N-1th frame image from the first NDC cache. For example, the matching module may read coordinate data A to coordinate data An from the first NDC cache. The matching module can also read the coordinate data corresponding to each motion grid in the Nth frame image from the second NDC buffer. For example, the matching module may read coordinate data B to coordinate data Bm from the second NDC cache.
- one coordinate data can correspond to one Drawcall, that is, to one model.
- the coordinate data may include the specific coordinates of each vertex of the model in the NDC space in a certain frame of image (such as the N-1th frame of image or the Nth frame of image).
- the coordinate data may also include drawing parameters corresponding to the model.
- the matching module may determine two matching coordinate data in two frames of images based on the drawing parameters included in different coordinate data.
- the matching module can convert any drawing parameter into a corresponding hash value. Different drawing parameters correspond to different hash values.
- the matching module can determine the feature hash value corresponding to the coordinate data 531 based on the vertex ID, index ID, drawing number, and offset included therein. Similarly, for each other coordinate data, the matching module can also determine the corresponding feature hash value based on the drawing parameters of each coordinate data.
- each coordinate data in the N-1th frame image and the Nth frame image can each correspond to a feature hash value.
- the coordinate data A1 may correspond to the feature hash value C1
- the coordinate data A2 may correspond to the feature hash value C2, ...
- the coordinate data An may correspond to the feature hash value Cn
- the coordinate data B1 may correspond to the feature hash value D1
- the coordinate data B2 may correspond to the feature hash value D2, ...
- the coordinate data Bm may correspond to the feature hash value Dm.
- the matching module can match the coordinate data in different frame images based on the feature hash value corresponding to each coordinate data.
- the matching module can search for items matching the characteristic hash value C1 between the characteristic hash value D1 and the characteristic hash value Dm according to the characteristic hash value C1.
- the matching module may use the coordinate data corresponding to the feature hash value matching the feature hash value C1 as the coordinate data in the N-th frame image that matches the coordinate data A1 in the N-1-th frame image.
- the matching between feature hash values may include: when two feature hash values are the same, the two feature hash values are considered to match each other.
- the matching module may determine that the coordinate data A1 in the N-1th frame image matches the coordinate data B1 in the Nth frame image. That is, the model corresponding to the coordinate data A1 moves from the position indicated by the coordinate data A1 to the position indicated by the coordinate data B1 between the N-1th frame image and the Nth frame image.
- the matching module may determine that the coordinate data A2 in the N-1th frame image matches the coordinate data B2 in the Nth frame image. That is, the model corresponding to the coordinate data A2 moves from the position indicated by the coordinate data A2 to the position indicated by the coordinate data B2 between the N-1th frame image and the Nth frame image.
- the matching module can respectively include according to the two coordinate data.
- the first vertex coordinate of verify the matching relationship between the two coordinate data.
- the matching module can also extract the coordinate value of the first vertex coordinate in each coordinate data and use it as a matching factor to perform the matching operation.
- the coordinate value of the first vertex coordinate of coordinate data A1 can be the vertex coordinate E1
- the coordinate value of the first vertex coordinate of the coordinate data A2 can be the vertex coordinate E2,...
- coordinates The coordinate value of the first vertex coordinate of the data An may be the vertex coordinate En.
- the coordinate value of the first vertex coordinate of coordinate data B1 can be vertex coordinate F1
- the coordinate value of the first vertex coordinate of coordinate data B2 can be vertex coordinate F2,...
- coordinate data The coordinate value of the first vertex coordinate of Bm can be the vertex coordinate Fm.
- the matching module can search for items matching the coordinate data in the feature hash values and vertex coordinates corresponding to the coordinate data B1 to coordinate data Bm for each coordinate data in the coordinate data A1 to the coordinate data An, as The coordinate data in the Nth frame image corresponding to this coordinate data.
- the matching of feature hash values may include: two feature hash values are the same.
- Vertex coordinate matching may include: the Euclidean distance between two vertex coordinates is less than a preset distance.
- the feature hash value C1 is the same as the feature hash value D1, and the Euclidean distance between the vertex coordinate E1 and the vertex coordinate F1 is less than the preset distance. Then the coordinate data A1 matches the coordinate data B1.
- the feature hash value C2 is the same as the feature hash value D2, and the Euclidean distance between the vertex coordinate E2 and the vertex coordinate F2 is less than the preset distance. Then the coordinate data A2 matches the coordinate data B2.
- the matching module can also determine two coordinate data that match each other based on the Euclidean distance between multiple vertex coordinates, or in combination with feature hash values.
- the matching module can obtain the coordinate data corresponding to each moving grid in the N-1th frame image and the Nth The corresponding relationship between the matched coordinate data in the frame image.
- the matching module sends the corresponding relationship between the two matched coordinate data to the memory.
- the memory stores the corresponding relationship between the two matching coordinate data.
- the storage method of the correspondence between the coordinate data may be different in different implementations.
- corresponding entries may be stored in the memory.
- Each table entry can be used to store two coordinate data that match each other, or to store the storage addresses of two coordinate data that match each other.
- the memory can set an identifier in the storage area of the two coordinate data in the NDC cache. So that the same identifier can be searched in the first NDC cache and the second NDC cache later to determine the two coordinate data that match each other.
- the electronic device can complete the grid matching process. Then, the vector calculation module of the electronic device can calculate and obtain the motion vector corresponding to each dynamic grid based on the grid matching results.
- the motion vector of each dynamic mesh can be used to identify the motion of each dynamic object from the N-1th frame image to the Nth frame image.
- FIG. 18 is a schematic diagram of another image processing method provided by an embodiment of the present application.
- This solution can be applied after completing grid matching as shown in Figure 14. According to the grid matching results, the motion vector corresponding to each dynamic grid is calculated and obtained. As shown in Figure 18, this solution can include:
- the vector calculation module reads two coordinate data that match each other.
- each set of coordinate data can correspond to the position of a dynamic grid (model) in two frames of images.
- the vector calculation module can read any one of the at least one set of coordinate data, and calculate the motion vector of the dynamic grid corresponding to the set of coordinate data according to subsequent steps.
- the vector calculation module calculates the motion vector corresponding to the two coordinate data that match each other.
- the vector calculation module can calculate and obtain the motion vector corresponding to the two coordinate data according to the preset formula (3).
- ⁇ V c ⁇ and ⁇ V P ⁇ are two coordinate data that match each other in the preceding and following frames respectively. It can be understood that two coordinate data that match each other can respectively correspond to multiple vertex coordinates of a dynamic object (model). Therefore, ⁇ V c ⁇ and ⁇ V P ⁇ can respectively include a set of vertex coordinates composed of multiple vertex coordinates of the same dynamic object in the previous and next frame images.
- the vector calculation module can also enable depth testing when calculating the motion vector of a moving object.
- the electronic device can turn on the depth test, compare the source depth value and the target depth value, and determine whether the depth test passes by referring to the preset rules.
- the source depth value and the target depth value can respectively identify two different depth values with the same coordinate information in the same frame image. It is understandable that during the image rendering process, objects will be compressed from three dimensions to two dimensions, so there will inevitably be some three-dimensional space points that cover each other on the two-dimensional image. Then, through this depth test, the location of the point that needs to be displayed can be determined.
- the vector calculation module can selectively calculate the motion vector of points that pass the depth test. For points that fail the depth test, the calculation of the motion vector can be skipped. This saves the calculation overhead of motion vectors.
- the vector calculation module sends the motion vector corresponding to the two matching coordinate data to the memory.
- the memory stores the motion vectors corresponding to the two coordinate data that match each other.
- the vector calculation module can store the motion vector in the memory.
- the electronic device may store the corresponding relationship between the motion vector and the corresponding coordinate data (or moving object). So that it can be called accurately during subsequent use.
- the electronic device can perform prediction of the N+1th frame image.
- the electronic device can predict the N+1th frame image based on the motion vectors of static objects and dynamic objects.
- the electronic device can use the Nth frame image as a reference to move the static object in the Nth frame image based on the motion vector of the static object obtained by the above calculation, thereby predicting and obtaining the motion vector of the N+1th frame image.
- the position of the static object can be used as a reference to move the static object in the Nth frame image based on the motion vector of the static object obtained by the above calculation, thereby predicting and obtaining the motion vector of the N+1th frame image. The position of the static object.
- the electronic device can use the Nth frame image as a reference to move any dynamic object in the Nth frame image according to the corresponding motion vector obtained by the above calculation, thereby predicting and acquiring the dynamic object.
- the position in the N+1th frame image can be used.
- the electronic device can also perform similar predictions to obtain the positions of other dynamic objects in the N+1th frame image.
- FIG. 19 shows a logical schematic diagram during the implementation of the image processing method provided by the embodiment of this application.
- the electronic device may intercept and store the NDC coordinate data corresponding to the dynamic grid of the N-1th frame.
- the electronic device can also intercept and store the rendering intermediate variables of the N-1th frame.
- the rendering intermediate variables may include the MVP matrix and depth data of the N-1th frame image, etc.
- the electronic device can intercept and store the NDC coordinate data corresponding to the dynamic grid of the Nth frame.
- the electronic device can also intercept and store the rendering intermediate variables of the Nth frame.
- the rendering intermediate variables may include the MVP matrix and depth data of the Nth frame image, etc.
- the electronic device can calculate the motion vector based on the above backup stored data. For example, the electronic device can determine the motion vector of the static object based on the intermediate rendering variables. The electronic device can determine the motion vector of the dynamic object based on the NDC coordinate data. In this way, the motion vectors of the static objects from the N-1th frame image to the Nth frame image can be obtained, as well as the corresponding motion vectors of each dynamic object.
- the electronic device can predict the N+1th frame image based on the motion vectors of static objects and dynamic objects.
- the motion vector calculation process for static objects combines depth data, Compared with the motion vector determined based on two-dimensional coordinate calculation, it is more accurate.
- the corresponding motion vectors of the dynamic objects are calculated to obtain more accurate motion vectors of the dynamic objects.
- the prediction of future frame images can be accurately performed.
- the matching method used is hash matching based on rendering parameters. Compared with matching methods based on brightness or color, it can significantly reduce the need for computing power and power consumption.
- Figure 20 shows a schematic diagram of the composition of an electronic device 2000.
- the electronic device 2000 may include: a processor 2001 and a memory 2002.
- the memory 2002 is used to store computer execution instructions.
- the processor 2001 executes instructions stored in the memory 2002
- the electronic device 2000 can be caused to execute the image processing method shown in any of the above embodiments.
- FIG 21 shows a schematic diagram of the composition of a chip system 2100.
- the chip system 2100 may include: a processor 2101 and a communication interface 2102, used to support related devices to implement the functions involved in the above embodiments.
- the chip system also includes a memory for saving necessary program instructions and data for the terminal.
- the chip system may be composed of chips, or may include chips and other discrete devices.
- the communication interface 2102 may also be called an interface circuit.
- the functions, actions, operations, steps, etc. in the above embodiments may be implemented in whole or in part by software, hardware, firmware, or any combination thereof.
- a software program When implemented using a software program, it may be implemented in whole or in part in the form of a computer program product.
- the computer program product includes one or more computer instructions. When computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part.
- the computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable device.
- the computer instructions may be stored in or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transferred from a website, computer, server, or data center Transmission to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) means.
- the computer-readable storage medium can be any available medium that can be accessed by a computer or include one or more data storage devices such as servers and data centers that can be integrated with the medium.
- the available media may be magnetic media (eg, floppy disk, hard disk, magnetic tape), optical media (eg, DVD), or semiconductor media. media (such as solid state disk (SSD)), etc.
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Multimedia (AREA)
- Image Generation (AREA)
Abstract
一种图像处理方法和电子设备,涉及图像处理领域,可以分别准确快速地对静态网格以及动态网格进行运动矢量的计算,节省矢量计算过程中的算力开销,提升预测效率。方法包括:根据已完成的至少两帧图像的渲染中间变量,确定下一帧图像的静态网格的位置。该渲染中间变量包括对应帧图像的MVP矩阵和深度数据。根据已完成的至少两帧图像中,第一模型的坐标数据,确定下一帧图像中,该第一模型的位置。该第一模型对应网格为该图像中的动态网格,该坐标数据包括该第一模型在对应帧图像中的NDC坐标以及绘制参数,该NDC坐标包括至少一个顶点坐标该绘制参数用于该电子设备绘制该第一模型。
Description
本申请要求于2022年9月7日提交国家知识产权局、申请号为202211105947.X、发明名称为“一种图像处理方法和电子设备”的中国专利申请的优先权,其全部内容通过引用结合在本申请中。
本申请实施例涉及图像处理领域,尤其涉及一种图像处理方法和电子设备。
帧预测技术能够通过已经完成渲染的帧图像的相关数据,预测下一帧图像中各个物体的位置。该技术在有插帧需求的显示方案中被广泛应用。
目前的帧预测技术可以基于各个像素点的颜色或亮度匹配,确定同一个物体(模型)在不同帧图像中的位置。进而根据位置的变化计算获取各个像素点的运动矢量。基于图像显示的连续性,根据该运动矢量就能够实现对未来帧图像的预测。
该方案在实施过程中,对于算力以及功耗也提出了较高的要求。
发明内容
本申请实施例提供一种图像处理方法和电子设备,可以分别准确快速地对静态网格以及动态网格进行运动矢量的计算,进而准确进行未来帧图像的预测。通过静态网格和动态网格的分别计算,节省矢量计算过程中的算力开销,提升预测效率。
为了达到上述目的,本申请实施例采用如下技术方案:
第一方面,提供一种图像处理方法,该方法应用于电子设备,该方法包括:该电子设备通过图像渲染得到至少两帧图像,至少两帧该图像中包括动态网格和静态网格,该动态网格对应模型在不同帧图像中的世界坐标系下的坐标不同,该静态网格对应模型在不同帧图像中的世界坐标系下的坐标相同。根据已完成的至少两帧图像的渲染中间变量,确定下一帧图像的静态网格的位置。该渲染中间变量包括对应帧图像的模型-观察-投影MVP矩阵和深度数据。根据已完成的至少两帧图像中,第一模型的坐标数据,确定下一帧图像中,该第一模型的位置。该第一模型对应网格为该图像中的动态网格,该坐标数据包括该第一模型在对应帧图像中的归一化设备NDC坐标以及绘制参数,该NDC坐标包括至少一个顶点坐标该绘制参数用于该电子设备绘制该第一模型。由此,电子设备可以基于下一帧图像的静态网格的位置以及下一帧图像中第一模型的位置,确定所述下一帧图像。可以理解的是,在图像中包括第一模型之外的其他更多动态网格的情况下,则电子设备可以根据对应模型的坐标数据,采用类似于第一模型的处理机制,确定该运动模型在下一帧图像中的位置。结合不同运动物体在下一帧图像中的位置以及静态物体在下一帧图像中的位置,电子设备就可以综合获取下一帧图像,实现未来帧图像的预测。
这样,电子设备可以通过MVP矩阵以及深度数据确定静态网格的运动矢量。其中,MVP矩阵以及深度数据可以直接通过应用程序下发的指令流获取,避免在确定静态网格的过程中产生大量的数据计算开销。此外,电子设备还可以通过坐标数据确定每个动态
网格的运动矢量。例如,电子设备可以基于坐标数据对动态网格进行匹配,从而避免逐个像素的亮度/颜色匹配产生的算力开销。由此即可通过上述方案实现静态网格以及动态网格的运动矢量的计算。同时能够显著降低算力开销,提升计算效率。
可选的,在该图像中包括多个动态网格对应的模型,该根据已完成的至少两帧图像中第一模型的坐标数据,确定下一帧图像中该第一模型的位置,包括:根据已完成的至少两帧图像中第一模型的坐标数据,基于特征哈希值匹配,确定该第一模型在不同帧图像中的坐标数据。其中,该第一模型在不同帧图像中的特征哈希值相同,不同模型在同一帧图像中的特征哈希值不同。根据该第一模型在不同帧图像中的坐标数据,确定下一帧图像中该第一模型的位置。这样,在多个动态网格的较为复杂的图像中,不需要通过像素级别的逐个匹配,即可确定同一个模型在不同帧图像的坐标对应关系。由此显著节省对应的算力开销以及时间开销。
可选的,该已完成帧图像包括第N帧图像以及第N-1帧图像,该下一帧图像为第N+1帧图像。该根据已完成的至少两帧图像的渲染中间变量,确定下一帧图像的静态网格的位置,包括:根据该第N-1帧图像的第一MVP矩阵以及第一深度数据,该第N帧图像的第二MVP矩阵以及第二深度数据,确定该第N帧图像中的静态网格的运动矢量。根据该静态网格在该第N帧图像中的位置,以及该静态网格的运动矢量,确定该第N+1帧图像中该静态网格的位置。这样,以当前帧图像为第N帧图像为例,电子设备可以在当前帧图像完成渲染后,获取第N帧图像中的相关数据,结合第N-1帧图像中的数据,进行第N+1帧图像的预测。应当理解的是,在另一些实现中,已经完成的帧图像也可以是不连续的两帧图像,如第N帧图像以及第N-2帧图像等。
可选的,该电子设备的存储器中配置有渲染中间变量缓存,在该根据已完成的至少两帧图像的渲染中间变量,确定下一帧图像的静态网格的位置之前,该方法还包括:获取该第一MVP矩阵、该第一深度数据、该第二MVP矩阵以及该第二深度数据,将获取的数据存储在该渲染中间变量缓存中。该根据已完成的至少两帧图像的渲染中间变量,确定下一帧图像的静态网格的位置,包括:从该渲染中间变量缓存中读取该第一MVP矩阵、该第一深度数据、该第二MVP矩阵以及该第二深度数据,并确定第N+1帧图像中该静态网格的位置。
可选的,该应用程序下发第一指令流指示该电子设备进行第N-1帧图像的渲染。该渲染中间变量缓存包括第一渲染中间变量缓存。该获取并存储该第一MVP矩阵,包括:该电子设备拦截该第一指令流中,用于传输该第一MVP矩阵的第一指令段,根据该第一指令段,将该第一MVP矩阵存储在该第一渲染中间变量缓存中。
可选的,该电子设备根据第一预设标识,拦截该第一指令流的该第一指令段。
可选的,该第一预设标识为uniform参数。
这样,电子设备可以在进行第N-1帧图像的渲染过程中,通过上述方案将后续需要使用的第一MVP矩阵备份存在存储器中的预设位置,例如渲染中间变量缓存中。以便后续调用。
可选的,该应用程序下发第一指令流指示该电子设备进行第N-1帧图像的渲染。该渲染中间变量缓存包括第二渲染中间变量缓存。该获取并存储该第一深度数据,包括:该电子设备拦截该第一指令流中,与该第一深度数据相关的第二指令段,根据该第二指
令段,将该第一深度数据存储在该第二渲染中间变量缓存中。
可选的,该与该第一深度数据相关的第二指令段用于指示该电子设备进行多目标渲染MRT。
这样,电子设备可以在进行第N-1帧图像的渲染过程中,通过上述方案将后续需要使用的第一深度数据备份存在存储器中的预设位置,例如渲染中间变量缓存中。以便后续调用。
可选的,该应用程序下发第二指令流指示该电子设备进行第N帧图像的渲染。该渲染中间变量缓存包括第三渲染中间变量缓存。该获取并存储该第二MVP矩阵,包括:该电子设备拦截该第二指令流中,用于传输该第二MVP矩阵的第三指令段,根据该第三指令段,将该第二MVP矩阵存储在该第三渲染中间变量缓存中。
可选的,该应用程序下发第二指令流指示该电子设备进行第N帧图像的渲染。该渲染中间变量缓存包括第四渲染中间变量缓存。该获取并存储该第二深度数据,包括:该电子设备拦截该第二指令流中,与该第二深度数据相关的第四指令段,根据该第四指令段,将该第二深度数据存储在该第四渲染中间变量缓存中。
应当理解的是,电子设备可以在执行每一帧图像的渲染过程中,执行上述MVP矩阵以及深度数据的备份存储,例如在第N-1帧图像以及第N帧图像的渲染过程中分别执行上述方案,由此使得后续进行未来帧预测时,能够顺利调用到上述备份存储的MVP矩阵和深度数据,用来确定未来帧中的静态网格的位置。
可选的,该已完成帧图像包括第N帧图像以及第N-1帧图像,该下一帧图像为第N+1帧图像。该根据已完成的至少两帧图像中,第一模型的坐标数据,确定下一帧图像中,该第一模型的位置,包括:根据该第一模型在该第N-1帧图像中的第一坐标数据,以及该第一模型在该第N帧图像中的第二坐标数据,确定该第一模型的运动矢量。根据该静态网格在该第N帧图像中的位置,以及该第一模型的运动矢量,确定该第N+1帧图像中该第一模型的位置。由此,该方案提供了一种确定动态网格的运动矢量的实现。可以理解的是,在帧图像中可以包括多个动态网格,对应多个运动物体。那么,针对每个动态网格,电子设备都可以执行该方案,由此确定各个动态网格的运动矢量。以下示例中,以第一模型为例进行说明。
可选的,该电子设备中配置有NDC缓存,在根据该第一模型的坐标数据确定该第一模型的运动矢量之前,该方法还包括:获取该第一模型的该第一坐标数据以及该第二坐标数据,将该第一坐标数据以及该第二坐标数据存储在该NDC缓存中。该根据该第一模型的坐标数据确定该第一模型的运动矢量,包括:从该NDC缓存中读取该第一模型的第一坐标数据以及该第二坐标数据,根据该第一坐标数据以及该第二坐标数据,确定该第一模型的运动矢量。通过动态网格渲染过程中,将坐标数据备份存储在预先设置的NDC缓存中。以便于后续未来帧的预测使用。
可选的,该应用程序下发第一指令流指示该电子设备进行第N-1帧图像的渲染。该述NDC缓存包括第一NDC缓存。该获取该第一模型的该第一坐标数据,包括:在开始该第N-1帧图像中该第一模型的绘制之前,使能enable变换反馈功能,该电子设备的GPU基于该变换反馈功能,在执行该第一模型的绘制时,向该电子设备反馈该第一坐标数据,该第一坐标数据包括该第一模型在该第N-1帧图像中的第一NDC坐标数据,以及该第一
模型在该第N-1帧图像中对应的第一绘制参数。该电子设备将该第一坐标数据存储在该第一NDC缓存中。该示例提供了一种具体的获取坐标数据的方案。比如,通过使能变换反馈功能,使得GPU可以将渲染获取的坐标数据反馈给电子设备,以便于电子设备对该坐标数据进行备份存储。
可选的,该方法还包括:关闭该变换反馈功能。可以理解的是,在完成一个Drawcall对应的动态网格的坐标数据的存储后,电子设备可以关闭该变换反馈功能。这样如果下一个Drawcall对应绘制静态网格时,就不需要回调对应的坐标数据。
可选的,该应用程序下发第二指令流指示该电子设备进行第N帧图像的渲染。该述NDC缓存包括第二NDC缓存。该获取该第一模型的该第二坐标数据,包括:在开始该第N帧图像中该第一模型的绘制之前,使能enable变换反馈功能,该电子设备的GPU基于该变换反馈功能,在执行该第一模型的绘制时,向该电子设备反馈该第二坐标数据,该第二坐标数据包括该第一模型在该第N帧图像中的第二NDC坐标数据,以及该第一模型在该第N帧图像中对应的第二绘制参数。该电子设备将该第二坐标数据存储在该第二NDC缓存中。类似于MVP矩阵以及深度数据的获取,针对每一帧图像的渲染过程,电子设备都可以针对其中各个Drawcall执行该方案,由此使得帧图像中的所有动态网格的坐标数据都能够被获取并存储。
可选的,在该获取该第一模型的该第一坐标数据以及该第二坐标数据之前,该方法还包括:确定该第一模型的网格为动态网格。
可选的,该确定该第一模型的网格为动态网格,包括:在该第一模型在当前帧图像中的坐标数据被更新的情况下,确定该第一模型的网格为动态网格。
示例性的,电子设备可以在接收到对一个模型的渲染指令时,判断该渲染指令指示的存放坐标数据的帧缓冲中的数据是否在该帧图像被更新,如果是则表明对应模型为运动模型,对应动态网格。反之,如果未被更新,则表明对应模型为静态模型,对应静态网格。
可选的,该第N帧图像中包括至少两个网格为动态网格的模型,该第一模型包括在该至少两个网格为动态网格的模型中,该NDC缓存中存储有各个模型对应与不同帧图像的坐标数据,该方法还包括:确定该NDC缓存中,该第一模型在不同帧图像中对应的两个坐标数据。
可以理解的是,在一个帧图像中可以包括多个动态网格。不同动态网格的运动矢量可以不同。而通过前述的数据备份存储后,在NDC缓存中可以分别存储有两组坐标数据。如第一NDC缓存中的第N-1帧图像的所有动态网格的坐标数据。又如第二NDC缓存中的第N帧图像的所有动态网格的坐标数据。那么,在计算每个模型的运动矢量之前,就需要先匹配确定同一个模型在不同帧图像中的坐标数据。继续以第一模型为例。
可选的,该确定该NDC缓存中,该第一模型在不同帧图像中对应的两个坐标数据,包括:根据该NDC缓存中,存储的各个坐标数据中包括的绘制参数,确定各个坐标数据对应的特征哈希值。将该NDC缓存中,与该第一坐标数据对应的特征哈希值相同的坐标数据,确定为该第二坐标数据。本示例提供了一种简便的相同模型在不同帧图像中的坐标匹配机制。比如,针对第一NDC缓存和第二NDC缓存中的任一个坐标数据,将其中的绘制参数对应到一个唯一的特征哈希值。这样,只需要在第一NDC缓存中和第二NDC
缓存中,查找具有相同特征哈希值的坐标数据,即可作为同一个模型在不同帧图像中的匹配坐标数据。
可选的,该确定该NDC缓存中,该第一模型在不同帧图像中对应的两个坐标数据,包括:根据该NDC缓存中,存储的各个坐标数据中包括的绘制参数,确定各个坐标数据对应的特征哈希值。将该NDC缓存中与该第一坐标数据对应的特征哈希值相同,并且两个坐标数据中的第一个顶点坐标之间的距离小于预设距离的坐标数据,确定为该第二坐标数据。本示例提供了一种更加准确并且渐变的坐标匹配机制。在该示例中,通过特征哈希值匹配后,还可以结合两个坐标数据中携带的多个顶点坐标之间的欧式距离,进一步验证两个坐标数据的匹配度。可以理解的是,对于同一个模型,在相邻或相近帧图像中的运动距离是有限的。因此,对于同一个模型的同一个顶点,在不同帧图像中的距离可以小于预设距离。这样能够进一步提升坐标匹配的准确度。由此提升基于坐标数据的运动矢量的计算准确度。
可选的,该绘制参数包括以下中的至少一项:顶点标识ID,索引ID,绘制数量,偏移量。
第二方面,提供一种电子设备,电子设备包括一个或多个处理器和一个或多个存储器;一个或多个存储器与一个或多个处理器耦合,一个或多个存储器存储有计算机指令;当一个或多个处理器执行计算机指令时,使得电子设备执行如上述第一方面以及各种可能的设计中任一种的方法。
第三方面,提供一种芯片***,芯片***包括接口电路和处理器;接口电路和处理器通过线路互联;接口电路用于从存储器接收信号,并向处理器发送信号,信号包括存储器中存储的计算机指令;当处理器执行计算机指令时,芯片***执行如上述第一方面以及各种可能的设计中任一种的方法。
第四方面,提供一种计算机可读存储介质,计算机可读存储介质包括计算机指令,当计算机指令运行时,执行如上述第一方面以及各种可能的设计中任一种的方法。
第五方面,提供一种计算机程序产品,计算机程序产品中包括指令,当计算机程序产品在计算机上运行时,使得计算机可以根据指令执行如上述第一方面以及各种可能的设计中任一种的方法。
应当理解的是,上述第二方面,第三方面,第四方面,以及第五方面提供的技术方案,其技术特征均可对应到第一方面及其可能的设计中提供的技术方案,因此能够达到的有益效果类似,此处不再赘述。
图1为一种图像渲染过程中的指令流传输的逻辑示意图;
图2为一种多帧图像的示意图;
图3为一种未来帧图像的预测方案示意图;
图4为本申请实施例提供的一种电子设备的软件组成示意图;
图5为本申请实施例提供的一种图像处理方法的模块交互示意图;
图6为本申请实施例提供的一种坐标数据的组成示意图;
图7为本申请实施例提供的一种NDC缓存的示意图;
图8为本申请实施例提供的一种坐标数据存储的示意图;
图9为本申请实施例提供的一种图像处理方法的模块交互示意图;
图10为本申请实施例提供的一种图像处理方法的模块交互示意图;
图11为本申请实施例提供的一种存储器存储数据的示意图;
图12为本申请实施例提供的一种存储器存储数据的示意图;
图13为本申请实施例提供的一种图像处理方法的模块交互示意图;
图14为本申请实施例提供的一种图像处理方法的模块交互示意图;
图15为本申请实施例提供的一种坐标数据与哈希值的对应示意图;
图16为本申请实施例提供的一种坐标数据匹配的示意图;
图17为本申请实施例提供的一种坐标数据匹配的示意图;
图18为本申请实施例提供的一种图像处理方法的模块交互示意图;
图19为本申请实施例提供的一种图像处理方法的流程示意图;
图20为本申请实施例提供的一种电子设备的组成示意图;
图21为本申请实施例提供的一种芯片***的组成示意图。
电子设备中可以运行有各类应用程序,用于向用户提供丰富的功能。例如,电子设备中的应用程序可以指示电子设备的显示器向用户提供显示功能。该显示功能可以包括显示视频流、图像流等功能。
以显示视频流为例。视频流可以由多个帧图像构成。电子设备可以快速地顺序播放帧图像,使得用户可以通过电子设备的显示器看到连续播放的帧图像构成的动态画面。电子设备在单位时间内播放帧图像的数量可以通过帧率标识。帧率越高,电子设备在单位时间内播放的帧图像越多,对应的动态画面也就更加清晰真实。
对于每个帧图像,应用程序都可以通过渲染指令流,展示电子设备进行相应的绘制,由此获取显示信息,并通过显示器进行显示。
示例性的,结合图1,在需要显示一个帧图像时,应用程序可以下发渲染指令流,以便指示电子设备根据该渲染指令流进行渲染操作。电子设备的中央处理器(Central Processing Unit,CPU)可以接收渲染指令流,根据该渲染指令流调取电子设备中安装的渲染环境中相应的应用编程接口(application programming interface,API)。CPU可以通过调用API,指示电子设备中具有图像渲染功能的图形处理器(Graphic Processing Unit,GPU)进行对应的渲染操作。GPU执行一些列渲染操作后获取的渲染结果,即可对应到显示信息。在一些实现中,该显示信息可以包括当前帧图像中,各个像素点对应的颜色信息、亮度信息、深度数据、法线信息等。在需要显示该帧图像时,显示器就可以获取该渲染结果,并据此进行相应的显示。
对于视频流中的其他帧图像,其显示信息的获取过程与图1类似。也就是说,在需要快速显示多个帧图像时,电子设备就需要根据应用程序下发的各个渲染指令流进行对应的渲染操作,才能获取相应帧图像的显示信息。
目前,为了提供更好的显示体验,对电子设备的图像处理能力提出了更高的要求。例如,随着电子设备所显示图像的分辨率越来越高,那么对应的每帧图像的渲染操作也变得更加复杂。此外,电子设备显示视频流时的帧率(或屏幕刷新率)也在不断提升,这又要求电子设备能够更加快速地渲染获取各个帧图像。如果对每一帧图像都执行如图
1所示的根据应用程序下发的渲染指令流执行的实时的渲染操作,就可能出现渲染结果的获取无法满足显示速度的情况,从而造成画面卡顿等问题。此外,还会导致电子设备在图像处理过程中产生较高的功耗。
为了能够应对上述问题,可以通过帧图像预测技术,基于已经完成渲染的帧图像,预测下一帧图像的显示情况。由此就可以在不需要执行下一帧图像的渲染操作的情况下,获取下一帧图像的显示信息。例如,如图2所示,以当前帧图像为第A帧图像为例。在第A帧图像以及前一帧图像(如第A-1帧图像)均已经完成渲染后,电子设备就可以根据第A-1帧图像以及第A帧图像,预测获取第A+1帧图像。
作为一种示例,请参考图3,为一种帧图像预测方案的示例。如图3所示,在第A-1帧图像中可以包括物体21以及物体22。在第A帧图像中也可以包括物体21以及物体22。其中,物体21在第A-1帧与第A帧图像中的位置相同。也即,物体21没有发生移动。物体22则在第A-1与第A帧图像中的位置不同。也即,物体22发生了位移。
在本示例中,电子设备可以将帧图像划分为多个图块。每个图块中包括多个像素点。例如,如图3所示,在第A-1帧图像中,物体21可以位于图块23的区域中。物体22可以位于图块24的区域中。在第A帧图像中,物体21依然位于图块23的区域中。物体22则移动到图块25中。电子设备可以像素为单位,分别确定各个图块在相邻两个帧图像中的运动矢量。
示例性的,以图块23为例。电子设备可以针对图块23中的每个像素点,通过对比该图块23在第A-1帧图像以及第A帧图像中对应位置的颜色信息,确定在该两帧图像中,图块23中包括的各个像素的颜色信息是否有变化。在如图3的示例中,在确定该两帧图像中,图块23中包括的各个像素的颜色信息是没有变化的情况下,则电子设备就可以确定图块23的运动矢量为0,即在该两帧图像中,图块23没有发生移动。这样,电子设备就可以预测在第A+1帧图像中,该图块23依然沿用之前的移动状态,即静止不动。
以图块24为例。电子设备可以针对图块24中的每个像素点,通过对比该图块24在第A-1帧图像以及第A帧图像中对应位置的颜色信息,确定在该两帧图像中,图块24中包括的各个像素的颜色信息是否有变化。在如图3的示例中,物体22从第A-1帧图像中的图块24移动到了第A帧图像中的图块25。那么,电子设备执行上述针对图块24的颜色对比中,可以确定在该两帧图像中图块24相应的颜色信息发生了变化。接着,电子设备可以第A-1帧图像中图块24中各个像素的颜色信息为基准,寻找第A帧图像中,与图块24相邻的其他图块中,与基准相应的图块。例如,电子设备可以确定在第A帧图像中,图块25的颜色信息与第A-1帧图像中的图块24的颜色信息相应。由此也就表明图块24在第A-1帧图像到第A帧图像的切换中,由第A-1帧图像中图块24的位置,移动到了第A帧图像中图块25的位置。这样,电子设备就可以确定图块24在第A-1帧图像与第A帧图像之间的运动矢量为由图块24指向图块25。该运动矢量的绝对长度可以根据图块24移动前后,相应的颜色信息之间的距离确定。将该图块24移动到图块25的运动矢量记为运动矢量26。显而易见的,图块24移动到图块25的过程中,物体22也会沿着运动矢量26移动。这样,电子设备就可以预测在第A+1帧图像中,第A帧图像中的图块25可以沿用之前的沿运动矢量26的移动状态,那么在第A+1帧图像
中,该图块25就会移动到帧图像之外。因此,电子设备就可以基于该帧图像预测机制,确定在A-1帧图像的图块24中的物体22、在第A帧图像的图块25中的物体22,会在第A+1帧图像移动到图像外,不再需要在帧图像上显示。
依次类推,电子设备就可以根据各个图块在第A-1帧图像以及第A帧图像中的运动矢量,预测确定第A+1帧图像中需要显示的内容。
可以看到,上述如图3所示的帧图像预测方案中,电子设备可以基于二维图像(如观察空间对应的二维图像),通过颜色匹配的机制计算每个图块中的运动矢量。其中,图块划分越细致,计算量越大,对应预测结果越准确。而在需要节省算力的情况下,图块划分较为粗糙,虽然能够降低计算量,但预测结果会收到大幅影响。此外,基于颜色匹配机制的计算过程也会引入较大的算力和功耗开销。
为了解决上述问题,本申请实施例提供一种图像处理方法,能够使得电子设备可以分别针对动态物体以及静止物体进行运动矢量的计算。例如,结合图像渲染过程中的深度数据,从三维空间的角度计算获取静止物体对应的全屏运动矢量。又如通过各个绘制指令(Drawcall)对应的归一化的设备坐标(Normalize Device Coordinates,NDC)空间坐标以及相关的渲染参数,通过哈希匹配的方式,简化不同模型的匹配过程,降低动态物体运动矢量计算过程中的运算开销。
以下将结合附图对本申请实施例提供的方案进行详细说明。
本申请实施例提供的技术方案,可以应用于具有图像显示功能的电子设备中。该电子设备可以包括手机、可折叠电子设备、平板电脑、桌面型计算机、膝上型计算机、手持计算机、笔记本电脑、超级移动个人计算机(ultra-mobile personal computer,UMPC)、上网本、蜂窝电话、个人数字助理(personal digital assistant,PDA)、增强现实(augmented reality,AR)设备、虚拟现实(virtual reality,VR)设备、人工智能(artificial intelligence,AI)设备、可穿戴式设备、车载设备、智能家居设备、或智慧城市设备中的至少一种。本申请实施例对该电子设备的具体类型不作特殊限制。
示例性的,在一些实施例中,从硬件组成的角度,本申请实施例涉及的电子设备可以包括处理器,外部存储器接口,内部存储器,通用串行总线(universal serial bus,USB)接口,充电管理模块,电源管理模块,电池,天线1,天线2,移动通信模块,无线通信模块,音频模块,扬声器,受话器,麦克风,耳机接口,传感器模块,按键,马达,指示器,摄像头,显示屏,以及用户标识模块(subscriber identification module,SIM)卡接口等。其中,传感器模块可以包括压力传感器,陀螺仪传感器,气压传感器,磁传感器,加速度传感器,距离传感器,接近光传感器,指纹传感器,温度传感器,触摸传感器,环境光传感器,骨传导传感器等。
需要说明的是,上述硬件组成并不构成对电子设备的具体限定。在另一些实施例中,电子设备可以包括更多或更少的部件,或者组合某些部件,或者拆分某些部件,或者不同的部件布置。
在另一些实施例中,本申请实施例涉及的电子设备还可以具有软件划分。以电子设备中运行有操作***为例。示例性的,图4为本申请实施例提供的一种电子设备的软件组成的示意图。如图4所示,该电子设备可以包括应用(Application,APP)层,框架(Framework)层,***库,以及硬件(HardWare)层
等。
其中,应用层也可以称为应用程序层。在一些实现中,应用程序层可以包括一系列应用程序包。应用程序包可以包括相机,图库,日历,通话,地图,导航,WLAN,蓝牙,音乐,视频,短信息等应用程序。在本申请实施例中,应用程序包还可以包括需要通过渲染图像向用户展示图像或者视频的应用程序。其中,视频可以理解为多帧图像的连续播放。示例性的,该需要渲染图像的应用程序可以包括游戏类应用程序,例如
等。
框架层也可以称为应用程序框架层。该框架层可以为应用层的应用程序提供API和编程框架。框架层包括一些预先定义的函数。示例性的,框架层可以包括窗口管理器,内容提供器,视图***,资源管理器,通知管理器,活动管理器,输入管理器等。窗口管理器提供窗口管理服务(Window Manager Service,WMS),WMS可以用于窗口管理、窗口动画管理、surface管理以及作为输入***的中转站。内容提供器用来存放和获取数据,并使这些数据可以被应用程序访问。该数据可以包括视频,图像,音频,拨打和接听的电话,浏览历史和书签,电话簿等。视图***包括可视控件,例如显示文字的控件,显示图片的控件等。视图***可用于构建应用程序。显示界面可以由一个或多个视图组成的。例如,包括短信通知图标的显示界面,可以包括显示文字的视图以及显示图片的视图。资源管理器为应用程序提供各种资源,比如本地化字符串,图标,图片,布局文件,视频文件等等。通知管理器使应用程序可以在状态栏中显示通知信息,可以用于传达告知类型的消息,可以短暂停留后自动消失,无需用户交互。比如通知管理器被用于告知下载完成,消息提醒等。通知管理器还可以是以图表或者滚动条文本形式出现在***顶部状态栏的通知,例如后台运行的应用程序的通知,还可以是以对话窗口形式出现在屏幕上的通知。例如在状态栏提示文本信息,发出提示音,电子设备振动,指示灯闪烁等。活动管理器可以提供活动管理服务(Activity Manager Service,AMS),AMS可以用于***组件(例如活动、服务、内容提供者、广播接收器)的启动、切换、调度以及应用进程的管理和调度工作。输入管理器可以提供输入管理服务(Input Manager Service,IMS),IMS可以用于管理***的输入,例如触摸屏输入、按键输入、传感器输入等。IMS从输入设备节点取出事件,通过和WMS的交互,将事件分配至合适的窗口。
在本申请实施例中,在框架层中可以设置有一个或多个功能模块,用于实现本申请实施例提供的方案。示例性的,框架层中可以设置有拦截模块,数据转存模块,矢量计算模块等。这些模块设置可以用于支持电子设备实现本申请实施例提供的图像处理方法。其具体功能以及实现将在后续详细说明。
如图4所示,电子设备中还可以设置有包括图形库的***库。在不同实现中,图形库可以包括如下中的至少一种:开放图形库(Open Graphics Library,OpenGL)、嵌入式***的开放图形库(OpenGL for Embedded Systems,OpenGL ES)、Vulkan等。在一些实施例中,***库中还可以包括其他模块。例如:表面管理器(surface manager),媒体框架(Media Framework)、标准C库(Standard C library,libc)、SQLite、Webkit等。
其中,表面管理器用于对显示子***进行管理,并且为多个应用程序提供了二维(2D)
和三维(3D)图层的融合。媒体框架支持多种常用的音频,视频格式回放和录制,以及静态图像文件等。媒体库可以支持多种音视频编码格式,例如:动态图像专家组4(Moving Pictures Experts Group,MPEG4),H.264,动态影像专家压缩标准音频层面3(Moving Picture Experts Group Audio Layer3,MP3),高级音频编码(Advanced Audio Coding,AAC),自适应多码解码(Adaptive Multi-Rate,AMR),联合图像专家组(Joint Photographic Experts Group,JPEG,或称为JPG),便携式网络图形(Portable Network Graphics,PNG)等。OpenGL ES和/或Vulkan提供应用程序中2D图形和3D图形的绘制和操作。SQLite为电子设备的应用程序提供轻量级关系型数据库。
在应用程序下发渲染命令之后,框架层中的各个模块就可以调用图形库中对应的API,指示GPU执行相应的渲染操作。
在如图4的示例中,电子设备中还可以包括硬件层。该硬件层中可以包括CPU,GPU,以及具有存储功能的存储器(如内存)。在一些实现中,CPU可以用于控制框架层中的各个模块实现其各自的功能,GPU可以用于根据框架层中各个模块处理后的指令所调用的图形库(如OpenGL ES)中的API执行相应的渲染处理。
本申请实施例提供的方案均能够应用于如图4所示的电子设备中。需要说明的是,如图4的示例,并不构成对电子设备的限制。在另一些实施例中,电子设备还可以包括更多或更少的组件。本申请实施例对于电子设备的具体组成不做限制。
作为一种示例,本申请实施例提供的技术方案,可以当前帧图像以及之前的帧图像进行渲染操作时,收集并缓存相应的数据。示例性的,以当前帧图像为第N帧图像,电子设备还收集第N-1帧图像的数据为例。
结合图4,设置在框架层中的拦截模块可以用于在执行各个帧图像的渲染操作的过程中,拦截所需指令流。例如,拦截模块可以用于拦截应用程序下发的渲染指令流。在一些实施例中,拦截模块还可以具有简单的判断能力。例如,拦截模块可以根据渲染指令流对应的模型信息是否被更新,确定该指令流用于绘制静态物体或动态物体。其中,该模型信息可以包括待绘制模型的坐标数据等。又如,拦截模块还可以根据预设的函数或函数携带的参数,识别当前指令流包括当前帧图像的深度(Depth)信息和/或模型-观察-投影(Model-View-Projection,MVP)矩阵的相关数据。在以下说明中,该深度数据以及MVP矩阵可以统称为渲染中间变量。
拦截模块还可以用于将指示进行动态物体的渲染的指令流发送给数据转存模块以便后续处理。拦截模块还可以用于将指示进行静态物体的渲染的指令流发送给电子设备的GPU以便后续处理。拦截模块还可以用于将包括渲染中间变量的指令流发送给数据转存模块以便后续处理。
数据转存模块可以用于根据来自拦截模块的指令流,执行对应的数据转存操作。示例性的,在一些实现中,数据转存模块可以在接收到对动态物体的渲染指令流时,使能变换反馈(transform feedback)功能,并将该对动态物体的渲染指令流下发给GPU,以便GPU执行相应的渲染操作。基于该变换反馈功能,数据转存模块获取GPU执行渲染操作过程中生成的一些数据。例如,数据转存模块可以基于变换反馈功能,获取动态物体对应在NDC空间下的坐标数据。在本示例中,数据转存模块可以将该动态物体对应在NDC空间下的坐标数据转存到电子设备的存储器中,以便后续使用。在另一些实现中,
数据转存模块可以将渲染中间变量存储在预先在存储器中创建的渲染中间变量缓存中。
应当理解的是,基于原生的图像渲染逻辑,上述拦截模块拦截的指令流最终会发送给GPU执行相关的渲染操作。该渲染流程中的各项数据对于电子设备(如电子设备的CPU)而言是不可见的。那么,为了便于后续计算相邻帧图像的运动矢量,本申请中,电子设备可以通过数据转存模块将后续需要使用的数据备份存储到电子设备的存储器中。
这样,在完成第N-1帧图像以及第N帧图像的渲染之后,在电子设备的存储器中就可以在特定位置存储有各个帧图像的MVP矩阵、各个帧图像的深度数据、以及每个帧图像中包括的动态物体对应的坐标数据。
上述存储在特定位置的数据可以用于支持电子设备预测第N+1帧图像。示例性的,在一些实施例中,电子设备中的矢量计算模块,可以根据各个帧图像的MVP矩阵、各个帧图像的深度数据,计算获取第N-1帧图像到第N帧图像对应的静态物体的运动矢量。该静态物体的运动矢量也可对应到全屏运动矢量。在另一些实施例中,电子设备中的矢量计算模块,可以根据每个帧图像包括的动态物体对应的坐标数据,计算获取第N-1帧图像到第N图像对应的动态物体的运动矢量。由此,电子设备可以基于第N帧图像的渲染结果,以及静态物体的运动矢量和动态物体的运动矢量,预测确定第N+1帧图像中各个物体的具***置,从而实现对第N+1帧图像的预测。
这样,电子设备可以分别对静态物体以及动态物体进行运动矢量的计算。基于MVP矩阵以及深度数据的三维信息计算获取的全屏运动矢量显然比目前的基于二位信息计算获取的运动矢量更加精确。此外,对于动态物体的运动矢量的计算与全屏运动矢量的计算解耦,计算获取更加准确的动态物体的运动矢量,进而更加准确的计算的更精确化操作,由此获取从而能够获取对动态物体在第N+1帧图像中更加准确的预测效果。
以下将结合各个模块之间的交互,对上述方案的具体实现进行示例性的说明。
示例性的,结合图5,为本申请实施例提供的一种图像处理方法的模块交互示意图。该方案可以应用于任一个帧图像(如第N-1帧图像,第N帧图像等)的渲染过程中。由此实现电子设备对动态物体的坐标数据的备份存储。
如图5所示,该方案可以包括:
S501、应用程序下发指令流511。
可以理解的是,应用程序在指示电子设备进行一帧图像的渲染过程中,可以下发包括多个Drawcall的指令流。本示例中,该指令流511可以对应一个Drawcall,包括在应用程序针对某一帧图像的渲染过程下发的指令流中。
其中,该指令流511对应的Drawcall可以用于指示电子设备对当前帧图像的动态物体进行绘制。
需要说明的是,在本示例中,该指令流511可以对应一个动态物体的绘制指令。在帧图像中包括多个动态物体需要分别绘制的情况下,那么应用程序就会分别下发多个动态物体各自对应的Drawcall。这样,电子设备就可以针对每个动态物体对应的Drawcall,执行该如图5所示的流程,以便实现对各个动态物体的坐标数据的备份存储。
S502、拦截模块确定指令流511用于绘制动态网格。其中,动态网格对应动态物体的网格(Mesh)。对应的,静态物体的网格可以称为静态网格。
在本示例中,拦截模块可以具有判断当前指令流用于指示绘制动态网格或静态网格的能力。示例性的,拦截模块可以在该指令流511指示绘制的模型信息被更新的情况下,确定该指令流511用于绘制动态网格。对应的,拦截模块可以在该指令流511指示绘制的模型信息未更新的情况下,确定该指令流511用于绘制静态网格。
作为一种可能的实现,拦截模块可以对比同一个模型在当前Drawcall以及前一个Drawcall中对应的模型信息(如坐标信息等)是否有变化,确定当前Drawcall所要绘制的模型为动态网格还是静态网格。例如,拦截模块可以对比当前Drawcall下发后,存储对应模型的帧缓冲中的坐标信息是否与前一帧图像中,同一个帧缓冲中的坐标信息是否相同,实现上述判断机制。
在当前Drawcall下发后,该存储坐标信息的帧缓冲中的数据与前一个帧图像中相同帧缓冲中的坐标数据相同没有变更时,则表明该模型的坐标数据没有更新,即为静态网格。
在当前Drawcall下发后,该存储坐标信息的帧缓冲中的数据与前一个帧图像中相同帧缓冲中的坐标数据有至少部分不同时,则表明该模型的坐标数据被更新,即为动态网格。
本示例中,以指令流511用于绘制动态网格为例。
S503、拦截模块向GPU回调指令流511。
示例性的,拦截模块可以在拦截来自应用程序的指令流,确定后续策略的同时,还可以将该拦截的指令流回调给GPU,实现原生逻辑。例如,拦截模块可以在拦截指令流511,确定该指令流511用于绘制动态网格的同时,还可以将该指令流511回调给GPU,以便GPU进行相应的响应。
S504、拦截模块向数据转存模块发送动态标识。
其中,动态标识可以用于标示当前Drawcall用于绘制动态网格。
在本实例中,拦截模块在确定当前Drawcall(即指令流511)对应于动态物体的绘制后,可以通过动态标识,通知数据转存模块,执行对应的数据备份存储。例如,执行对该动态物体的坐标数据的备份存储。
S505、数据转存模块指示GPU使能变换反馈功能。
其中,变换反馈功能可以用于收集GPU执行后续渲染操作的过程中的数据。示例性的,通过使能变换反馈功能,GPU可以将根据指令流511执行的渲染操作过程中,生成的当前模型(即指令流511对应的动态物体)的坐标数据反馈给数据转存模块。
在一些实施例中,该变换反馈功能可以包括tramsform feedback功能。那么,数据转存模块可以通过调用用于使能tramsform feedback功能的函数,实现指示GPU使能变换反馈功能的目的。
S506、GPU根据指令流511,执行对应的渲染操作,获取渲染结果521。
结合前述说明,拦截模块可以在拦截指令流511的同时,向GPU回调指令流511,由此实现原生的渲染逻辑。
示例性的,GPU可以接收来自拦截模块的指令流511,并执行对应的渲染操作,获取渲染结果521。需要说明的是,GPU根据指令流511执行的渲染操作,具体实现过程可以参考图1所示的流程。也即,拦截模块可以根据指令流511调用相应的API,以便
于GPU根据该API执行对应的操作。在本申请实施例的说明中,GPU根据指令流执行的渲染操作的实现可以沿用上述流程,后续不再赘述。
这样,电子设备就可以实现对指令流511的渲染响应,获取对应的动态网格的渲染结果。
S507、GPU将渲染结果521发送给存储器。
S508、存储器存储渲染结果521。
通过S507-S508,电子设备可以将渲染结果521存储在存储器中,以便于基于该渲染结果521以及当前帧图像的其他渲染结果,通过合成、去噪等操作,获取当前帧图像对应的显示信息。
S509、GPU根据变换反馈功能回调坐标数据531。
本示例中,在S505中,数据转存模块通过使能变换反馈功能,用于获取当前Drawcall指示绘制动态网格的坐标数据。
那么,在该步骤S509中,GPU可以在根据指令流511执行渲染操作的同时,在获取当前Drawcall对应的坐标数据531后,通过变换反馈功能将该坐标数据531回调给GPU。
示例性的,如图6所示,该坐标数据可以包括该Drawcall对应模型的各个顶点在NDC空间下的具体坐标。该NDC空间下的具体坐标可以简称为NDC坐标。
应当理解的是,应用程序在指示电子设备绘制一个模型时,可以在指令流中携带该模型的顶点坐标。该顶点坐标可以是基于模型本身建立的局部空间的。为了能够使得电子设备可以将不同模型分别绘制到帧图像所展示的世界空间中,应用程序还可以向电子设备发送MVP矩阵。根据MPV矩阵,电子设备就可以将基于局部空间的定点坐标,分别转换到世界空间的坐标,观察空间的坐标,以及裁剪空间的坐标。裁剪空间坐标即可对应到显示屏上的坐标。本示例中,NDC坐标可以是局部空间经过MVP矩阵变换后,归一化的设备坐标。基于NDC坐标,以及MVP矩阵,电子设备也可以还原各个顶点在世界空间中的坐标。例如,NDC坐标乘以VP矩阵的逆矩阵,然后根据w分量进行归一化还原,就可以获取对应的世界空间中的坐标。
此外,在本申请实施例中,GPU回调给数据转存模块的坐标数据还可以包括当前Drawcall对应的绘制参数。
其中,该绘制参数可以包括以下中的至少一项:顶点ID(Vertex Id),索引ID(Index Id),绘制数量(Draw Count),偏移量(Draw Offset)等。
可以理解的是,绘制参数可以为GPU执行渲染操作时的必备参数,可以携带在指令流511中下发给GPU。同一帧图像中,不同Drawcall对应的绘制参数不同。也就是说,绘制参数可以用于标识不同的Drawcall。也即,绘制参数可以用于标示不同的动态网格。这样,电子设备就可以根据该坐标数据中的绘制参数,识别匹配不同帧图像中相应的动态网格。
在本申请的另一些实施例中,该绘制参数也可以是由拦截模块发送给数据转存模块的。示例性的,拦截模块可以在识别指令流511后,将其中的绘制参数发送给数据转存模块。那么,GPU在回调坐标数据时,就可以不携带该绘制参数。
S510、数据转存模块向存储器发送坐标数据531。
S511、存储器在NDC缓存中存储坐标数据531。
在本申请实施例中,数据转存模块可以在各个帧图像渲染过程中,备份存储后续预测未来帧(如第N+1帧)时所需数据。
示例性的,数据转存模块可以从GPU获取动态网格的坐标数据531,并将该坐标数据531存储在电子设备的存储器中的特定位置。
其中,该存储坐标数据的特定位置可以是预先创建的。结合图5,该特定位置可以为在存储器中预先创建的NDC缓存。
作为一种示例,图7示出了一种NDC缓存的示意。在图7的示例中,在电子设备中的存储器可以预先创建包括多个子缓存的NDC缓存。其中,该多个子缓存可以分别构成两个缓存组,如第一NDC缓存以及第二NDC缓存。每个缓存组可以分别用于存储一个帧图像的动态网格的坐标数据。比如,第一NDC缓存可以用于存储第N-1帧图像的动态网格的坐标数据。第二NDC缓存可以用于存储第N帧图像的动态网格的坐标数据。
如图7所示,第一NDC缓存可以包括NDC缓存A1到NDC缓存An。其中,NDC缓存A1到NDC缓存An中的每个即为NDC缓存中的一个子缓存。该一个子缓存可以对应存储一个第N-1帧图像的动态网格的坐标数据。对应的,第二NDC缓存可以包括NDC缓存B1到NDC缓存Bm。其中,NDC缓存B1到NDC缓存Bm中的每个也可对应到一个子缓存。该一个子缓存可以对应存储一个第N帧图像的动态网格的坐标数据。
以第N-1帧图像包括n个动态网格为例。该n个动态网格的坐标数据分别为:坐标数据A1到坐标数据An。那么,基于上述S501-S511的步骤,对于每个坐标数据,数据转存模块都可以将其存储到一个NDC缓存中。
示例性的,结合图8,以指令流511指示对第N-1帧图像中的坐标数据A1对应动态网格进行渲染为例。根据上述方案实现,数据转存模块可以将该坐标数据A1存储在第一NDC缓存的一个子缓存中。如,将该坐标数据A1存储到NDC缓存A1中。
以指令流511指示对第N帧图像中的坐标数据B1对应动态网格进行渲染为例。根据上述方案实现,数据转存模块可以将该坐标数据B1存储在第二NDC缓存的一个子缓存中。如,将该坐标数据B1存储到NDC缓存B1中。
这样,在完成第N-1帧图像的渲染后,在第一NDC缓存中就可以存储有该第N-1帧图像中各个动态网格对应的坐标数据。类似的,在完成第N帧图像的渲染后,在第二NDC缓存中就可以存储有该第N帧图像中各个动态网格对应的坐标数据。
由此,在第N-1帧图像以及第N帧图像的渲染过程中,通过上述S501-S511,即可实现对所有动态网格的坐标数据向NDC缓存中的备份缓存。
需要说明的是,继续结合图5。在每个动态网格的Drawcall结束时,数据转存模块还可以执行S512,即关闭变换反馈功能。由此使得在拦截模块接收到下一个Drawcall时,可以重新按照如图5所示的流程,仅针对动态网格使能变换反馈功能,备份存储相应的坐标数据。
上述如图5-图8的示例中,对应用程序下发针对动态网格的渲染指令流的处理机制进行了说明。以下结合图9,对应用程序下发针对静态网格的渲染指令流时,电子设备的处理机制进行示例性说明。
如图9所示,该方案可以包括:
S901、应用程序下发指令流512。
其中,该指令流51对应的Drawcall可以用于指示电子设备对当前帧图像的动态物体进行绘制。结合图5中的S501中的说明。该指令流512可以是应用程序在指示电子设备进行第N-1帧图像或第N帧图像的渲染时,发送的多个指令流中的一个。本示例中,拦截模块可以根据如下S902的步骤识别该指令流512用于指示进行静态网格的绘制,并执行相应的策略。
S902、拦截模块拦截指令流512,确定指令流512用于绘制静态网格。
结合图5中的S502,拦截模块可以根据该指令流512所指示绘制模型的模型信息是否被更新,判断该指令流512是否用于绘制静态网格。示例性的,在当前Drawcall下发后,拦截模块可以确定指令流512对应模型的存储坐标信息的帧缓冲中的数据与前一个帧图像中相同帧缓冲中的坐标数据相同没有变更时,则表明该指令流512对应模型的坐标数据没有更新,即为静态网格。
在本示例中,电子设备不需要备份存储静态网格的坐标数据。那么,在确定指令流512用于指示进行静态网格的绘制的情况下,可以触发执行以下S903。
S903、拦截模块将指令流512回调给GPU。
S904、GPU根据指令流512,执行对应的渲染操作,获取渲染结果522。
S905、GPU将渲染结果522发送给存储器。
S906、存储器存储渲染结果522。
通过上述S903-S906,电子设备可以按照原生逻辑,对静态网格进行渲染操作,获取对应的渲染结果522。那么,结合该渲染结果522,以及当前帧图像中的其他渲染结果,通过合成、去噪等操作,即可获取当前帧图像的显示数据。
需要说明的是,应用程序在指示电子设备绘制某一个帧图像时,除了上述用于指示绘制静态物体或动态物体的指令流之外,还需要向电子设备下发其他渲染指令流。例如,应用程序可以下发MVP矩阵、用于进行深度渲染的深度数据等渲染中间变量。在本示例中,电子设备还可以备份存储该渲染中间变量,以便于后续确定全屏运动矢量。
作为一种示例,请参考图10,为本申请实施例提供的又一种图像处理方法的模块间交互示意图。通过该方案,能够实现对渲染中间变量的备份存储。该方案可以应用于对任一帧图像的渲染过程中。如图10所示,该方案可以包括:
S1001、应用程序下发指令流513。
其中,该指令流513可以携带有当前帧图像对应的渲染中间变量。
示例性的,应用程序可以在开始某一帧图像的渲染时,向电子设备发送该帧图像渲染过程中需要用到的各项数据。例如,应用程序可以通过该指令流513向电子设备发送包括MVP矩阵以及深度数据的渲染中间变量。
S1002、拦截模块确定指令流513包括当前帧图像的渲染中间变量。
本示例中,拦截模块可以根据预设的函数或函数携带的参数,确定该指令流513包括渲染中间变量。
应当理解的是,对于确定的渲染平台,传递MVP矩阵的方式一般也相对固定。比如,应用程序可以通过携带uniform参数的函数,传递MVP矩阵。那么,拦截模块就可以在指令流513中包括携带uniform参数的函数时,确定该指令流513包括MVP函数。
对于深度数据的识别,拦截模块可以根据多目标渲染(Multiple Render Targets,MRT)对应信息获取。其中,基于MRT技术,电子设备可以通过一次渲染,将RGBA颜色、法线、深度数据或者纹理坐标输出到多个缓冲区。MRT对应的输出缓冲可以是应用程序通过指令流指示的。本申请中,拦截模块可以在指令流513指示一次性将多个渲染结果输出到不同缓冲区时,确定该指令流513中包括指示进行MRT渲染的命令。而在该指示进行MRT渲染的命令中包括用于存储深度数据的帧缓冲。也就是说,拦截模块可以确定指令流513指示MRT渲染的情况下,确定通过该指令流513即可获取当前帧图像的深度数据。在一些实施例中,该当前帧图像的深度数据也可以称为全屏深度图。
这样,拦截模块可以根据携带有uniform参数的函数,确定MVP矩阵的传输指令。拦截模块还可以根据MRT渲染的命令,确定深度数据的传输指令。其中,该MRT渲染的命令,可以对应指示一次性将多个渲染结果输出到不同缓冲区的指令。
本示例中,该MVP矩阵的传输指令以及深度数据的传输指令可以包括在指令流513中。
S1003、拦截模块向数据转存模块发送指令流513。
S1004、数据转存模块将指令流513中的渲染中间变量发送给存储器。
S1005、存储器在渲染中间变量缓存中存储当前帧图像的渲染中间变量。
结合S1002中的说明,数据转存模块可以根据携带有uniform参数的函数,获取当前帧图像的MVP矩阵,并将该MVP矩阵发送给存储器进行存储。
另外,数据转存模块可以根据MRT渲染的命令,确定存储深度数据的帧缓冲ID。那么,数据转存模块就可以从对应的帧缓冲中读取深度数据,发送给存储器进行存储。
在本示例中,如图10所示,电子设备可以在执行S1004以及S1005之前(即存储中间变量缓存)之前,在存储器中创建对应的存储空间。例如,电子设备可以在存储器中创建中间变量缓存。
这样,通过如图10所示的方案实现,就能够在任一个帧图像完成渲染时,在存储器(如中间变量缓存)中备份存储对应帧图像的渲染中间变量。结合上述图5的示例,在任一个帧图像完成时,在存储器(如NDC缓存)中还可以备份存储有对应帧图像的所有动态网格的坐标数据。
示例性的,在完成第N-1帧图像的渲染时,如图11所示,在存储器中的NDC缓存中可以备份存储有第N-1帧图像所有的动态网格的坐标数据1111。在存储器中的中间变量缓存中可以备份存储有第N-1帧图像的MVP矩阵1121,以及深度数据1131。此外,在存储器中还可以存储有第N-1帧图像对应的渲染结果。
接着,电子设备可以根据应用程序的指令,执行第N帧图像的渲染。结合前述图5-图10的方案说明,在完成第N帧图像的渲染时,如图12所示,在存储器中的NDC缓存中可以备份存储有第N-1帧图像所有的动态网格的坐标数据1111,以及第N帧图像所有的动态网格的坐标数据1112。在存储器中的中间变量缓存中可以备份存储有第N-1帧图像的渲染中间变量,以及第N帧图像的渲染中间变量。其中,第N-1帧图像的渲染中间变量可以包括MVP矩阵1121,深度数据1131。第N帧图像的渲染中间变量可以包括MVP矩阵1122,深度数据1132。此外,在存储器中还可以覆盖第N-1帧图像对应的渲染结果,存储有第N帧图像对应的渲染结果。
在本申请中,电子设备可以在完成第N帧图像的渲染操作后,根据如图12所示的存储器中备份存储的各项数据,计算第N-1帧图像到第N帧图像的运动矢量,从而达到根据运动矢量预测第N+1帧图像的目的。
示例性的,电子设备可以通过不同的方案,计算静态物体在两帧图像之间的运动矢量,以及动态物体在两帧图像之间的运动矢量。
以下分别进行说明。
如图13所示,为本申请实施例提供的又一种图像处理方法的示意图。通过该方案,能够实现静态物体在两帧图像之间的运动矢量的计算。应当理解的是,在不同帧图像中,静态物体在世界坐标系下的坐标并未变化。因此,静态物体的运动矢量即可对应到帧图像中除动态物体之外的其他所有显示元素的运动矢量。也即,静态物体的运动矢量可以对应到全屏运动矢量。如图13所示,该方案可以包括:
S1301、矢量计算模块从渲染中间变量缓存读取深度数据以及MVP矩阵。
本示例中,矢量计算模块可以从中间变量缓存分别读取第N-1帧图像以及第N帧图像各自对应的深度数据以及MVP矩阵。
作为一种示例,结合图12的示意。矢量计算模块可以从渲染中间变量缓存中读取MVP矩阵1121、深度数据1131、MVP矩阵1122以及深度数据1132。
其中,每一帧图像分别可以对应一个MVP矩阵。每帧图像的深度数据可以包括:对应帧图像的每个像素点的深度数据。
S1302、矢量计算模块计算获取屏幕上每个像素的运动矢量。
示例性的,矢量计算模块中可以预设有公式(1),用于根据S1301中读取的数据,分别屏幕上各个像素点在第N-1帧图像到第N帧图像之间的运动矢量。
公式(1):
其中,为像素点的运动矢量。Pc和Pp分别为前后两帧图像中,该像素点在相机坐标系下的三维坐标。VPp和VPc分别前后两帧图像分别对应的VP矩阵。
在上述公式(1)中,像素点在第N-1帧图像或第N帧图像中的相机坐标系下的三维坐标可以通过如下公式(2)获取。
公式(2):Px=<(u*2.0+1.0)*z|v*2.0+1.0)*z|z>T。
其中,(u,v)为屏幕上任一个像素点。z为该像素点在当前帧图像中的深度。Px为该像素点在当前帧图像(如第N-1帧图像或第N帧图像)的相机坐标系下的三维坐标。
这样,通过上述公式(1)以及公式(2),即可基于像素点的包括深度数据在内的三维坐标,计算获取各个像素点的运动矢量。
本示例中,屏幕上各个点的运动矢量的集合即可对应到全屏运动矢量。
S1303、矢量计算模块将包括所有像素点运动矢量的全屏运动矢量发送给存储器。
S1304、存储器存储全屏运动矢量。
这样,通过如图13所示的方案,电子设备就能够在完成第N帧图像的渲染之后,根据前后两帧图像的深度数据,计算获取静态物体的运动矢量。可以理解的是,相比于现有的基于二维信息的运动矢量计算方式,该如图13所示的方案结合了深度数据进行运动矢量的获取,因此计算获取的全屏运动矢量更加准确。
应当理解的是,上述公式(1)以及公式(2)计算获取的全屏运动矢量,可以用于
实现对静态物体(即世界坐标系下静止的物体)在第N+1帧图像中位置的预测。例如,以第N帧图像为基准,通过沿全屏运动矢量的位移计算,即可获取第N+1帧图像中静态物体的位置。
不同于静态物体,对于动态物体,在不同帧图像中运动趋势可以各不相同。在本申请实施例中,电子设备可以针对各个动态物体,分别计算获取其各自对应的运动矢量。那么根据该定制化的运动矢量预测该动态物体在第N+1帧图像中的位置,显然可以更加准确。
示例性的,以下结合图14-图18,对本申请实施例中动态物体的运动矢量计算方案进行举例说明。通过本申请实施例提供的方案,能够针对每个动态物体分别计算确定其对应的运动矢量。该方案可以通过两个步实现:网格匹配,运动矢量计算。以下分别进行说明。
在本方案的实现中,可以通过网格匹配,确定同一个动态物体(运动网格)在不同帧图像中的位置。如图14所示,该方案可以包括:
S1401、匹配模块从存储器读取坐标数据。
示例性的,匹配模块可以从存储器的NDC缓存中,读取第N-1帧图像以及第N帧图像的坐标数据。
作为一种实现,结合图12以及图8的说明,匹配模块可以从第一NDC缓存中读取第N-1帧图像中,各个运动网格对应的坐标数据。例如,匹配模块可以从第一NDC缓存中读取坐标数据A到坐标数据An。匹配模块还可以从第二NDC缓存中读取第N帧图像中,各个运动网格对应的坐标数据。例如,匹配模块可以从第二NDC缓存中读取坐标数据B到坐标数据Bm。
S1402、针对一帧图像中的任一个坐标数据,确定另一帧图像中与该坐标数据匹配的坐标数据。
结合前述图6的相关说明,一个坐标数据可以对应一个Drawcall,也即对应一个模型。该坐标数据中可以包括该模型在某一帧图像(如第N-1帧图像或第N帧图像)中,各个顶点在NDC空间下的具体坐标。该坐标数据还可以包括该模型对应的绘制参数。
在一些实施例中,匹配模块可以根据不同坐标数据中包括的绘制参数,确定两帧图像中,匹配的两个坐标数据。
示例性的,匹配模块可以将任一个绘制参数,转换为相应的哈希值。不同的绘制参数对应的哈希值不同。
例如,参考图15,以绘制参数包括顶点ID(Vertex Id),索引ID(Index Id),绘制数量(Draw Count),偏移量(Draw Offset)为例。对于坐标数据531,匹配模块可以根据其中包括的顶点ID、索引ID、绘制数量以及偏移量,确定该坐标数据531对应的特征哈希值。类似的,对于其他各个坐标数据,匹配模块也可以根据各个坐标数据的绘制参数,确定相应的特征哈希值。
这样,第N-1帧图像以及第N帧图像中的各个坐标数据就都可以分别对应到一个特征哈希值。
例如,参考图16。对于第N-1帧图像,坐标数据A1可以对应于特征哈希值C1,坐标数据A2可以对应于特征哈希值C2,……,坐标数据An可以对应于特征哈希值Cn。
类似的,对于第N帧图像,坐标数据B1可以对应于特征哈希值D1,坐标数据B2可以对应于特征哈希值D2,……,坐标数据Bm可以对应于特征哈希值Dm。
匹配模块可以基于各个坐标数据对应的特征哈希值,实现对不同帧图像中坐标数据的匹配。
示例性的,匹配模块可以根据特征哈希值C1,在特征哈希值D1到特征哈希值Dm之间,查找与该特征哈希值C1匹配的项。匹配模块可以将该与特征哈希值C1匹配的特征哈希值对应的坐标数据,作为第N帧图像中与第N-1帧图像中的坐标数据A1向匹配的坐标数据。其中,特征哈希值之间的匹配可以包括:两个特征哈希值相同时,则认为两个特征哈希值互相匹配。
例如,以特征哈希值D1与特征哈希值C1匹配为例。匹配模块可以确定第N-1帧图像中的坐标数据A1与第N帧图像中的坐标数据B1匹配。也即,该坐标数据A1对应的模型,在第N-1帧图像到第N帧图像之间,由坐标数据A1指示的位置移动到了坐标数据B1所指示的位置。
以特征哈希值D2与特征哈希值C2匹配为例。匹配模块可以确定第N-1帧图像中的坐标数据A2与第N帧图像中的坐标数据B2匹配。也即,该坐标数据A2对应的模型,在第N-1帧图像到第N帧图像之间,由坐标数据A2指示的位置移动到了坐标数据B2所指示的位置。
在另一些实施例中,在根据如图16所示的方案初步确定第N-1帧图像以及第N帧图像互相匹配的两个坐标数据后,匹配模块可以根据该两个坐标数据中分别包括的第一个顶点坐标,验证该两个坐标数据的匹配关系。
示例性的,结合图17,匹配模块可以确定每个坐标数据对应的特征哈希值之外,还可以提取每个坐标数据中第一个顶点坐标的坐标值,作为匹配因子执行匹配操作。例如,第N-1帧图像中,坐标数据A1的第一个顶点坐标的坐标值可以为顶点坐标E1,坐标数据A2的第一个顶点坐标的坐标值可以为顶点坐标E2,……,坐标数据An的第一个顶点坐标的坐标值可以为顶点坐标En。对应的,第N帧图像中,坐标数据B1的第一个顶点坐标的坐标值可以为顶点坐标F1,坐标数据B2的第一个顶点坐标的坐标值可以为顶点坐标F2,……,坐标数据Bm的第一个顶点坐标的坐标值可以为顶点坐标Fm。
那么,匹配模块可以针对坐标数据A1到坐标数据An中的每个坐标数据,在坐标数据B1到坐标数据Bm对应的特征哈希值以及顶点坐标中,查找与该坐标数据相匹配的项,作为该坐标数据对应的第N帧图像中的坐标数据。其中,特征哈希值的匹配可以包括:两个特征哈希值相同。顶点坐标匹配可以包括:两个顶点坐标之间的欧氏距离小于预设距离。
例如,特征哈希值C1与特征哈希值D1相同,并且,顶点坐标E1与顶点坐标F1之间的欧氏距离小于预设距离。那么坐标数据A1与坐标数据B1相匹配。又如,特征哈希值C2与特征哈希值D2相同,并且,顶点坐标E2与顶点坐标F2之间的欧氏距离小于预设距离。那么坐标数据A2与坐标数据B2相匹配。
在另一些实施例中,匹配模块还可以根据多个顶点坐标之间的欧式距离,或者结合特征哈希值确定互相匹配的两个坐标数据。
由此,匹配模块就可以获取第N-1帧图像中,各个运动网格对应的坐标数据与第N
帧图像中匹配的坐标数据的对应关系。
S1403、匹配模块向存储器发送匹配的两个坐标数据之间对应关系。
S1404、存储器存储匹配的两个坐标数据之间对应关系。
示例性的,该坐标数据之间的对应关系的存储方式,在不同实现中可以是不同的。
在一些实施例中,存储器中可以存储有对应表项。每个表项可以用于存储两个互相匹配的坐标数据,或者存储两个互相匹配的坐标数据的存储地址。
在另一些实施例中,存储器在获取匹配模块发送的对应关系后,可以该两个坐标数据在NDC缓存中的存储区域设置标识。以便后续可以在第一NDC缓存中和第二NDC缓存中查找相同标识,确定互相匹配的两个坐标数据。
由此,电子设备就可以完成网格匹配的过程。接着,电子设备的矢量计算模块可以根据网格匹配结果,计算获取各个动态网格对应的运动矢量。该每个动态网格的运动矢量可以用于标识每个动态物体在第N-1帧图像到第N帧图像的运动情况。
作为一种示例,请参考图18,为本申请实施例提供的又一种图像处理方法的示意图。该方案可以应用于如图14完成网格匹配之后,根据网格匹配结果,计算获取每一个动态网格对应的运动矢量。如图18所示,该方案可以包括:
S1801、矢量计算模块读取互相匹配的两个坐标数据。
结合图14所示的方案说明,在完成网格匹配之后,在存储器中可以存储有互相匹配的至少一组坐标数据的对应关系。其中,每组坐标数据可以对应到一个动态网格(模型)在两帧图像中的位置。
那么,本示例中,矢量计算模块可以读取该至少一组坐标数据中的任意一组,并依据后续的步骤计算该组坐标数据对应动态网格的运动矢量。
S1802、矢量计算模块计算该互相匹配的两个坐标数据对应的运动矢量。
在本示例中,矢量计算模块可以根据预设的公式(3),计算获取该两个坐标数据对应的运动矢量。
公式(3):
其中,为运动矢量。{Vc}和{VP}分别为前后帧互相匹配的两个坐标数据。可以理解的是,互相匹配的两个坐标数据可以分别对应一个动态物体(模型)的多个顶点坐标。因此,{Vc}和{VP}分别可以分别包括前后帧图像中,同一个动态物体的多个顶点坐标构成的顶点坐标集合。
在本申请的一些实施例中,在通过公式(3)计算获取后,还可以对该进行光栅化处理,进而获取该两个坐标数据对应的动态物体在第N-1帧到第N帧之间的运动矢量。
需要说明的是,在本申请的另一些实施例中,矢量计算模块在计算运动物体的运动矢量时,还可以开启深度测试。
在本示例中,电子设备可以开启深度测试,比较源深度值与目标深度值,并参考预设规则判断深度测试是否通过。其中,源深度值和目标深度值可以分别标识同一个帧图像中具有相同坐标信息的两个不同深度值。可以理解的是,在图像渲染过程中,物体会从三维被压缩到二维,那么就不可避免地存在一些在二维图像上互相覆盖的三维空间点。那么,通过该深度测试,就可以确定需要显示的点的位置。
由此,矢量计算模块可以在计算一个运动物体相应的运动矢量时,选择性地对通过深度测试的点进行运动矢量的计算。对于未通过深度测试的点则可以跳过不执行运动矢量的计算。由此节省运动矢量的计算开销。
S1803、矢量计算模块向存储器发送该互相匹配的两个坐标数据对应的运动矢量。
S1804、存储器存储该互相匹配的两个坐标数据对应的运动矢量。
在获取互相匹配的两个坐标数据对应的运动矢量后,矢量计算模块可以将该运动矢量存储在存储器中。
这样,针对第N-1帧图像或第N帧图像中,每一组互相匹配的坐标数据执行上述S1801-S1804,即可获取所有运动物体各自对应的运动矢量。
需要说明的是,在一些情况下,在互相匹配的坐标数据包括多组时,也即在图像中存在多个运动物体需要计算运动矢量时,为了使得计算获取运动矢量能够与运动物体向对应,电子设备可以在每获取并存储一个运动矢量时,存储该运动矢量与对应的坐标数据(或运动物体)的对应关系。以便在后续使用时能够准确调用。
基于上述图5-图18的方案实现,即可分别完成对静态物体以及动态物体的运动矢量的计算确定。那么基于此,电子设备就可以执行对第N+1帧图像的预测。
示例性的,电子设备可以根据静态物体和动态物体的运动矢量预测第N+1帧图像。
作为一种可能的实现,电子设备可以第N帧图像为基准,对第N帧图像中的静态物体根据上述计算获取的静态物体的运动矢量进行矢量移动,从而预测获取第N+1帧图像中的静态物体的位置。
在另一种实现中,电子设备可以第N帧图像为基准,对第N帧图像中的任一个动态物体,根据上述计算获取的与之对应的运动矢量进行矢量移动,从而预测获取该动态物体在第N+1帧图像中的位置。相应的,针对其他动态物体,电子设备也可以执行类似的预测,从而获取其他动态物体在第N+1帧图像中的位置。
为了使得本领域技术人员能够更加清楚地了解本申请提供方案的全貌,图19示出了本申请实施例提供的图像处理方法在实施过程中的一种逻辑示意图。
如图19所示,以当前帧图像为第N帧图像为例。
在执行第N-1帧图像的渲染过程中,电子设备可以拦截并存储第N-1帧的动态网格对应的NDC坐标数据。电子设备还可以拦截并存储第N-1帧的渲染中间变量。该渲染中间变量可以包括该第N-1帧图像的MVP矩阵以及深度数据等。
类似的,在执行第N帧图像的渲染过程中,电子设备可以拦截并存储第N帧的动态网格对应的NDC坐标数据。电子设备还可以拦截并存储第N帧的渲染中间变量。该渲染中间变量可以包括该第N帧图像的MVP矩阵以及深度数据等。
这样,在完成第N帧图像的渲染之后,电子设备就可以根据上述备份存储的数据,计算运动矢量。示例性的,电子设备可以根据渲染中间变量,确定静态物体的运动矢量。电子设备可以根据NDC坐标数据,确定动态物体的运动矢量。由此即可获取第N-1帧图像到第N帧图像的静态物体的运动矢量,以及各个动态物体各自对应的运动矢量。
那么基于此,电子设备就可以根据静态物体和动态物体的运动矢量预测第N+1帧图像。
本申请实施例提供的方案中,对于静态物体的运动矢量计算过程结合了深度数据,
相较于基于二维坐标计算确定的运动矢量更加准确。通过动态物体和静态物体的分离,对动态物体进行各自对应的运动矢量的计算,从而获取更加准确的动态物体的运动矢量。基于上述更加准确的静态物体以及动态物体的运动矢量,就可以准确地执行对未来帧图像的预测。
此外,在对动态物体进行运动矢量计算时,所采用的匹配方式为基于绘制参数的哈希匹配,相较于基于亮度或颜色的匹配方式,能够显著降低对算力的需求以及功耗开销。
上述主要从各个服务模块的角度对本申请实施例提供的方案进行了介绍。为了实现上述功能,其包含了执行各个功能相应的硬件结构和/或软件模块。本领域技术人员应该很容易意识到,结合本文中所公开的实施例描述的各示例的单元及算法步骤,本申请能够以硬件或硬件和计算机软件的结合形式来实现。某个功能究竟以硬件还是计算机软件驱动硬件的方式来执行,取决于技术方案的特定应用和设计约束条件。专业技术人员可以对每个特定的应用来使用不同方法来实现所描述的功能,但是这种实现不应认为超出本申请的范围。需要说明的是,本申请实施例中对模块的划分是示意性的,仅仅为一种逻辑功能划分,实际实现时可以有另外的划分方式。
图20示出了的一种电子设备2000的组成示意图。如图20所示,该电子设备2000可以包括:处理器2001和存储器2002。该存储器2002用于存储计算机执行指令。示例性的,在一些实施例中,当该处理器2001执行该存储器2002存储的指令时,可以使得该电子设备2000执行上述实施例中任一种所示的图像处理方法。
需要说明的是,上述方法实施例涉及的各步骤的所有相关内容均可以援引到对应功能模块的功能描述,在此不再赘述。
图21示出了的一种芯片***2100的组成示意图。该芯片***2100可以包括:处理器2101和通信接口2102,用于支持相关设备实现上述实施例中所涉及的功能。在一种可能的设计中,芯片***还包括存储器,用于保存终端必要的程序指令和数据。该芯片***,可以由芯片构成,也可以包含芯片和其他分立器件。需要说明的是,在本申请的一些实现方式中,该通信接口2102也可称为接口电路。
需要说明的是,上述方法实施例涉及的各步骤的所有相关内容均可以援引到对应功能模块的功能描述,在此不再赘述。
在上述实施例中的功能或动作或操作或步骤等,可以全部或部分地通过软件、硬件、固件或者其任意组合来实现。当使用软件程序实现时,可以全部或部分地以计算机程序产品的形式来实现。该计算机程序产品包括一个或多个计算机指令。在计算机上加载和执行计算机程序指令时,全部或部分地产生按照本申请实施例所述的流程或功能。所述计算机可以是通用计算机、专用计算机、计算机网络、或者其他可编程装置。所述计算机指令可以存储在计算机可读存储介质中,或者从一个计算机可读存储介质向另一个计算机可读存储介质传输,例如,所述计算机指令可以从一个网站站点、计算机、服务器或者数据中心通过有线(例如同轴电缆、光纤、数字用户线(digital subscriber line,DSL))或无线(例如红外、无线、微波等)方式向另一个网站站点、计算机、服务器或数据中心进行传输。所述计算机可读存储介质可以是计算机能够存取的任何可用介质或者是包括一个或多个可以用介质集成的服务器、数据中心等数据存储设备。所述可用介质可以是磁性介质(例如,软盘、硬盘、磁带),光介质(例如,DVD)、或者半导
体介质(例如固态硬盘(solid state disk,SSD))等。
尽管结合具体特征及其实施例对本申请进行了描述,显而易见的,在不脱离本申请的精神和范围的情况下,可对其进行各种修改和组合。相应地,本说明书和附图仅仅是所附权利要求所界定的本申请的示例性说明,且视为已覆盖本申请范围内的任意和所有修改、变化、组合或等同物。显然,本领域的技术人员可以对本申请进行各种改动和变型而不脱离本申请的精神和范围。这样,倘若本申请的这些修改和变型属于本申请权利要求及其等同技术的范围之内,则本申请也意图包括这些改动和变型在内。
Claims (24)
- 一种图像处理方法,其特征在于,所述方法应用于电子设备,所述方法包括:所述电子设备通过图像渲染得到至少两帧图像,所述至少两帧图像中的任一帧图像包括动态网格和静态网格,所述动态网格对应模型在不同帧图像中的世界坐标系下的坐标不同,所述静态网格对应模型在不同帧图像中的世界坐标系下的坐标相同;根据所述至少两帧图像的渲染中间变量,确定下一帧图像的静态网格的位置;所述渲染中间变量包括对应帧图像的MVP矩阵和深度数据;根据所述至少两帧图像中第一模型的坐标数据,确定下一帧图像中所述第一模型的位置;所述第一模型对应网格为动态网格,所述坐标数据包括所述第一模型在对应帧图像中的NDC坐标以及绘制参数,所述NDC坐标包括至少一个顶点坐标,所述绘制参数用于指示所述电子设备绘制所述第一模型;基于所述下一帧图像的静态网格的位置以及所述下一帧图像中所述第一模型的位置,确定所述下一帧图像。
- 根据权利要求1所述的方法,其特征在于,在所述图像中包括多个动态网格对应的模型;所述根据所述至少两帧图像中第一模型的坐标数据,确定下一帧图像中所述第一模型的位置,包括:根据所述至少两帧图像中第一模型的坐标数据,基于特征哈希值匹配,确定所述第一模型在不同帧图像中的坐标数据;其中,所述第一模型在不同帧图像中的特征哈希值相同,不同模型在同一帧图像中的特征哈希值不同;根据所述第一模型在不同帧图像中的坐标数据,确定下一帧图像中所述第一模型的位置。
- 根据权利要求1或2所述的方法,其特征在于,所述至少两帧图像包括第N帧图像以及第N-1帧图像,所述下一帧图像为第N+1帧图像;所述根据所述至少两帧图像中第一模型的坐标数据,确定下一帧图像中所述第一模型的位置,包括:根据所述第一模型在所述第N-1帧图像中的第一坐标数据,以及所述第一模型在所述第N帧图像中的第二坐标数据,确定所述第一模型的运动矢量;根据所述静态网格在所述第N帧图像中的位置,以及所述第一模型的运动矢量,确定所述第N+1帧图像中所述第一模型的位置。
- 根据权利要求3所述的方法,其特征在于,所述电子设备中配置有NDC缓存,在根据所述第一模型的坐标数据确定所述第一模型的运动矢量之前,所述方法还包括:获取所述第一坐标数据以及所述第二坐标数据,将所述第一坐标数据以及所述第二坐标数据存储在所述NDC缓存中;所述根据所述第一模型的坐标数据确定所述第一模型的运动矢量,包括:从所述NDC缓存中读取所述第一模型的第一坐标数据以及所述第二坐标数据,根据所述第一坐标数据以及所述第二坐标数据,确定所述第一模型的运动矢量。
- 根据权利要求4所述的方法,其特征在于,所述获取所述第一模型的所述第一坐标数据,包括:在开始所述第N-1帧图像中所述第一模型的绘制之前,打开变换反馈功能,所述电子设备的图形处理器GPU基于所述变换反馈功能,在执行所述第一模型的绘制时,向所述电子设备反馈所述第一坐标数据,所述第一坐标数据包括所述第一模型在所述第N-1帧图像中的第一NDC坐标数据,以及所述第一模型在所述第N-1帧图像中对应的第一绘制参数;所述电子设备从所述GPU获取所述第一坐标数据,并存储在NDC缓存的第一NDC缓存中。
- 根据权利要求5所述的方法,其特征在于,所述方法还包括:关闭所述变换反馈功能。
- 根据权利要求4-6中任一项所述的方法,其特征在于,所述获取所述第一模型的所述第二坐标数据,包括:在开始所述第N帧图像中所述第一模型的绘制之前,打开变换反馈功能,所述电子设备的GPU基于所述变换反馈功能,在执行所述第一模型的绘制时,向所述电子设备反馈所述第二坐标数据,所述第二坐标数据包括所述第一模型在所述第N帧图像中的第二NDC坐标数据,以及所述第一模型在所述第N帧图像中对应的第二绘制参数;所述电子设备将所述第二坐标数据存储在所述NDC缓存的第二NDC缓存中。
- 根据权利要求4-7中任一项所述的方法,其特征在于,在所述获取所述第一模型的所述第一坐标数据以及所述第二坐标数据之前,所述方法还包括:确定所述第一模型的网格为动态网格。
- 根据权利要求8所述的方法,其特征在于,所述确定所述第一模型的网格为动态网格,包括:在所述第一模型在当前帧图像中的坐标数据被更新的情况下,确定所述第一模型的网格为动态网格。
- 根据权利要求3-9中任一项所述的方法,其特征在于,所述图像中包括至少两个网格为动态网格的模型,所述第一模型包括在所述至少两个网格为动态网格的模型中,所述NDC缓存中存储有各个模型对应与不同帧图像的坐标数据,所述方法还包括:确定所述NDC缓存中,所述第一模型在不同帧图像中对应的两个坐标数据。
- 根据权利要求10所述的方法,其特征在于,所述确定所述NDC缓存中,所述第一模型在不同帧图像中对应的两个坐标数据,包括:根据所述NDC缓存中,存储的各个坐标数据中包括的绘制参数,确定各个坐标数据对应的特征哈希值;将所述NDC缓存中,与所述第一坐标数据对应的特征哈希值相同的坐标数据,确定为所述第二坐标数据。
- 根据权利要求10所述的方法,其特征在于,所述确定所述NDC缓存中,所述第一模型在不同帧图像中对应的两个坐标数据,包括:根据所述NDC缓存中,存储的各个坐标数据中包括的绘制参数,确定各个坐标数据 对应的特征哈希值;将所述NDC缓存中与所述第一坐标数据对应的特征哈希值相同,并且两个坐标数据中的第一个顶点坐标之间的距离小于预设距离的坐标数据,确定为所述第二坐标数据。
- 根据权利要求1-12中任一项所述的方法,其特征在于,所述绘制参数包括以下中的至少一项:顶点标识ID,索引ID,绘制数量,偏移量。
- 根据权利要求1-13中任一项所述的方法,其特征在于,所述至少两帧图像包括第N帧图像以及第N-1帧图像,所述下一帧图像为第N+1帧图像;所述根据所述至少两帧图像的渲染中间变量,确定下一帧图像的静态网格的位置,包括:根据所述第N-1帧图像的第一MVP矩阵以及第一深度数据,所述第N帧图像的第二MVP矩阵以及第二深度数据,确定所述第N帧图像中的静态网格的运动矢量;根据所述静态网格在所述第N帧图像中的位置,以及所述静态网格的运动矢量,确定所述第N+1帧图像中所述静态网格的位置。
- 根据权利要求14所述的方法,其特征在于,所述电子设备的存储器中配置有渲染中间变量缓存,在根据所述至少两帧图像的渲染中间变量,确定所述第N帧图像中的静态网格的运动矢量之前,所述方法还包括:获取所述第一MVP矩阵、所述第一深度数据、所述第二MVP矩阵以及所述第二深度数据,将获取的数据存储在所述渲染中间变量缓存中;所述根据所述至少两帧图像的渲染中间变量,确定所述第N帧图像中的静态网格的运动矢量,包括:从所述渲染中间变量缓存中读取所述第一MVP矩阵、所述第一深度数据、所述第二MVP矩阵以及所述第二深度数据,并确定所述第N帧图像中的静态网格的运动矢量。
- 根据权利要求15所述的方法,其特征在于,所述获取并存储所述第一MVP矩阵,包括:所述电子设备拦截第一指令流中,用于传输所述第一MVP矩阵的第一指令段,根据所述第一指令段,将所述第一MVP矩阵存储在所述渲染中间变量缓存的第一渲染中间变量缓存中;其中,所述第一指令流用于指示所述电子设备进行第N-1帧图像的渲染。
- 根据权利要求16所述的方法,其特征在于,所述电子设备拦截所述第一指令段,包括:所述电子设备拦截所述第一指令流中,携带有uniform参数的第一指令段。
- 根据权利要求15-17中任一项所述的方法,其特征在于,所述获取并存储所述第一深度数据,包括:所述电子设备拦截第一指令流中,与所述第一深度数据相关的第二指令段,根据所述第二指令段,将所述第一深度数据存储在所述渲染中间变量缓存的第二渲染中间变量缓存中;其中,所述第一指令流用于指示所述电子设备进行第N-1帧图像的渲染。
- 根据权利要求18所述的方法,其特征在于,所述与所述第一深度数据相关的第 二指令段用于指示所述电子设备进行多目标渲染MRT。
- 根据权利要求15-19中任一项所述的方法,其特征在于,所述获取并存储所述第二MVP矩阵,包括:所述电子设备拦截所述第二指令流中,用于传输所述第二MVP矩阵的第三指令段,根据所述第三指令段,将所述第二MVP矩阵存储在所述渲染中间变量缓存的第三渲染中间变量缓存中;其中,所述第二指令流用于指示所述电子设备进行第N帧图像的渲染。
- 根据权利要求15-20中任一项所述的方法,其特征在于,所述获取并存储所述第二深度数据,包括:所述电子设备拦截所述第二指令流中,与所述第二深度数据相关的第四指令段,根据所述第四指令段,将所述第二深度数据存储在所述渲染中间变量缓存的第四渲染中间变量缓存中;其中,所述第二指令流用于指示所述电子设备进行第N帧图像的渲染。
- 一种电子设备,其特征在于,所述电子设备包括一个或多个处理器和一个或多个存储器;所述一个或多个存储器与所述一个或多个处理器耦合,所述一个或多个存储器存储有计算机指令;当所述一个或多个处理器执行所述计算机指令时,使得所述电子设备执行如权利要求1-21中任一项所述的方法。
- 一种计算机可读存储介质,其特征在于,所述计算机可读存储介质包括计算机指令,当所述计算机指令运行时,执行如权利要求1-21中任一项所述的方法。
- 一种芯片***,其特征在于,所述芯片***包括处理器和通信接口;所述处理器用于从存储介质中调用并运行所述存储介质中存储的计算机程序,执行如权利要求1-21中任一项所述的方法。
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202211105947.X | 2022-09-07 | ||
CN202211105947.XA CN117710404A (zh) | 2022-09-07 | 2022-09-07 | 一种图像处理方法和电子设备 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2024051471A1 true WO2024051471A1 (zh) | 2024-03-14 |
Family
ID=90143019
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/CN2023/113746 WO2024051471A1 (zh) | 2022-09-07 | 2023-08-18 | 一种图像处理方法和电子设备 |
Country Status (2)
Country | Link |
---|---|
CN (1) | CN117710404A (zh) |
WO (1) | WO2024051471A1 (zh) |
Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
KR20160147408A (ko) * | 2015-06-15 | 2016-12-23 | 엘지전자 주식회사 | 프레임 간 변화량을 이용한 가변 해상도 처리 장치 및 그 제어 방법 |
CN113362442A (zh) * | 2021-01-22 | 2021-09-07 | 惠州Tcl移动通信有限公司 | 一种虚拟现实图像的渲染方法、存储介质及虚拟现实设备 |
CN113837920A (zh) * | 2021-08-18 | 2021-12-24 | 荣耀终端有限公司 | 一种图像渲染方法及电子设备 |
CN114862997A (zh) * | 2022-04-08 | 2022-08-05 | 北京大甜绵白糖科技有限公司 | 图像渲染方法和装置、介质和计算机设备 |
-
2022
- 2022-09-07 CN CN202211105947.XA patent/CN117710404A/zh active Pending
-
2023
- 2023-08-18 WO PCT/CN2023/113746 patent/WO2024051471A1/zh unknown
Patent Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
KR20160147408A (ko) * | 2015-06-15 | 2016-12-23 | 엘지전자 주식회사 | 프레임 간 변화량을 이용한 가변 해상도 처리 장치 및 그 제어 방법 |
CN113362442A (zh) * | 2021-01-22 | 2021-09-07 | 惠州Tcl移动通信有限公司 | 一种虚拟现实图像的渲染方法、存储介质及虚拟现实设备 |
CN113837920A (zh) * | 2021-08-18 | 2021-12-24 | 荣耀终端有限公司 | 一种图像渲染方法及电子设备 |
CN114862997A (zh) * | 2022-04-08 | 2022-08-05 | 北京大甜绵白糖科技有限公司 | 图像渲染方法和装置、介质和计算机设备 |
Also Published As
Publication number | Publication date |
---|---|
CN117710404A (zh) | 2024-03-15 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN106504185B (zh) | 一种渲染优化方法和装置 | |
JP2023517917A (ja) | 仮想シーンの表示方法、装置、機器、及びコンピュータープログラム | |
KR101952983B1 (ko) | 콘텐트의 타일-기반 렌더링을 위한 방법 및 콘텐트를 렌더링하기 위한 시스템 | |
CN110573224A (zh) | 三维环境创作和生成 | |
CN113244614B (zh) | 图像画面展示方法、装置、设备及存储介质 | |
CN114210055B (zh) | 一种图像渲染方法和电子设备 | |
US11558562B2 (en) | Apparatus and method for providing 360-degree panoramic background during video call | |
CN110020300B (zh) | 一种浏览器页面合成方法及终端 | |
CN113368492A (zh) | 渲染的方法、装置 | |
CN111209422A (zh) | 图像显示方法、装置、电子设备及存储介质 | |
CN112053370A (zh) | 基于增强现实的显示方法、设备及存储介质 | |
CN115018692B (zh) | 一种图像渲染方法及电子设备 | |
CN114443189B (zh) | 一种图像处理方法和电子设备 | |
CN114570020A (zh) | 数据处理方法以及*** | |
WO2024027231A1 (zh) | 一种图像渲染方法和电子设备 | |
CN110045958A (zh) | 纹理数据生成方法、装置、存储介质及设备 | |
WO2024051471A1 (zh) | 一种图像处理方法和电子设备 | |
Lluch et al. | Interactive three-dimensional rendering on mobile computer devices | |
US20220253807A1 (en) | Context aware annotations for collaborative applications | |
CN115690269B (zh) | 一种视图对象的处理方法及电子设备 | |
WO2023170726A1 (ja) | 点群データ表示装置、点群データ記憶装置、点群データ表示方法及び点群データ表示プログラム | |
CN111125571B (zh) | 图片显示方法及装置 | |
WO2024045701A9 (zh) | 数据处理方法、装置、设备及存储介质 | |
WO2023284625A1 (zh) | 应用的跨平台显示方法、可读介质和电子设备 | |
WO2024045787A1 (zh) | 拾取对象的检测方法、装置、计算机设备、可读存储介质和计算机程序产品 |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 23862168 Country of ref document: EP Kind code of ref document: A1 |