WO2024061292A1 - Interface generation method and electronic device - Google Patents

Interface generation method and electronic device Download PDF

Info

Publication number
WO2024061292A1
WO2024061292A1 PCT/CN2023/120200 CN2023120200W WO2024061292A1 WO 2024061292 A1 WO2024061292 A1 WO 2024061292A1 CN 2023120200 W CN2023120200 W CN 2023120200W WO 2024061292 A1 WO2024061292 A1 WO 2024061292A1
Authority
WO
WIPO (PCT)
Prior art keywords
rendering
application
interface
view
tree
Prior art date
Application number
PCT/CN2023/120200
Other languages
French (fr)
Chinese (zh)
Inventor
陈路路
李俊杰
李煜
李蔚
金成铭
高彬彬
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Publication of WO2024061292A1 publication Critical patent/WO2024061292A1/en

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0481—Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44—Arrangements for executing specific programs
    • G06F9/451—Execution arrangements for user interfaces
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T11/00—2D [Two Dimensional] image generation

Definitions

  • the present application relates to the field of electronic technology, and in particular to interface generation methods and electronic devices.
  • the resolution and refresh rate of screens of electronic devices are getting higher and higher.
  • the resolution of the screen affects the pixels included in one frame of the interface, and the refresh rate affects the time to generate one frame of the interface.
  • the rendering of views is based on the painter’s algorithm, that is, the views in the interface are rendered sequentially from the bottom layer to the upper layer. According to this rendering order, electronic devices can achieve effects such as transparency mixing, shadowing, and linear overlay of views or layers.
  • this rendering order results in parts of the view being completely obscured.
  • the pixels in the area corresponding to the completely occluded view will be covered multiple times.
  • the pixels obtained by the last rendering will overwrite the pixels obtained by the previous rendering. That is, the application will call the GPU to generate pixels that are not displayed on the interface, thereby increasing the probability of interface freezes and reducing the energy efficiency of interface generation.
  • Embodiments of the present application provide interface generation methods and electronic devices, and relate to the field of electronic technology.
  • the interface generation method provided by this application can first identify a completely obscured control, and by deleting or not generating the rendering node corresponding to the control, it can reduce overdrawing during the interface generation process and improve the energy efficiency of interface generation.
  • embodiments of the present application provide an interface generation method, applied to an electronic device including a screen.
  • the method includes: determining that views that are completely obscured by other views in the first application include the first view; generating a first rendering tree, the first rendering tree does not include a first rendering node, the first rendering node is a rendering node corresponding to the first view, and the rendering node stores the drawing instructions of the corresponding view; the first interface is generated based on the first rendering tree , the first interface includes the interface of the first application.
  • the electronic device first determines the completely occluded view. Since the completely occluded view has nothing to do with the display content, a rendering tree that does not include the rendering node corresponding to the fully prepared view can be generated, and then based on the The rendering tree generates the interface. Since the rendering tree does not contain the rendering node corresponding to the fully prepared view, the view will not be drawn during the process of generating the interface.
  • determining that the view that is completely occluded by other views in the first application includes the first view specifically includes: in the process of traversing the views of the first application, determining The position of the second view is the second position; when traversing to the first view, the position of the first view is determined to be the first position, and the first view is determined to be the first position based on the second position and the first position.
  • a view that is completely obscured by other views within an application specifically includes: in the process of traversing the views of the first application, determining The position of the second view is the second position; when traversing to the first view, the position of the first view is determined to be the first position, and the first view is determined to be the first position based on the second position and the first position.
  • the application program can determine the positions of different views, and then determine the mutual shielding relationship between the views, and further determine whether any one view is completely shielded.
  • the method further includes: not saving the first view draw instructions to the first rendering tree; skip the subviews of the first view to continue traversing, and the first rendering tree does not include rendering nodes corresponding to the subviews of the first view.
  • the drawing method call of the first view can be skipped, and the first view and the first view can be skipped.
  • the drawing instructions of the view's subviews are stored in the rendering tree, and a rendering tree that does not contain useless drawing instructions can be directly generated.
  • the view determined to be completely obscured by other views within the first application includes the first view, specifically including: in the process of traversing the views of the first application, after determining that the position of the first view is the first position, determining that the position of the second view is the second position; based on the first position and the second position, determining that the first view is completely obscured by the second view, and recording the identifier of the first view in the first parameter.
  • the application when the application traverses the views, after first determining the position of the first view and then determining the position of the second view, Since the first view has been traversed at this time, the identifier of the first view needs to be saved.
  • the identifier of the first view is used to determine the rendering node corresponding to the completely blocked view in subsequent steps.
  • generating a first rendering tree specifically includes: after traversing the views of the first application, generating a second rendering tree, where the second rendering tree includes a first rendering node. ;Delete the first rendering node in the second rendering tree based on the first parameter to obtain the first rendering tree.
  • the first parameter records the identifier of the completely occluded view
  • the application program can delete the rendering node corresponding to the completely occluded view based on the first parameter.
  • determining that the view completely obscured by other views in the first application includes the first view specifically includes: after traversing the views of the first application, generating a second rendering tree, the second rendering tree including the first rendering node and the second rendering node; determining, based on rendering attributes of the first rendering node and the rendering attributes of the second rendering node, that the first view is completely obscured by the second view, and the second view corresponds to the second rendering node.
  • the application can also determine the rendering node corresponding to the completely occluded view after generating the rendering tree.
  • generating the first rendering tree specifically includes: deleting the second rendering node in the second rendering tree to obtain the first rendering tree.
  • the electronic device can delete the rendering node corresponding to the completely occluded view in the first rendering tree.
  • the method further includes: merging the first rendering tree and the second rendering tree to obtain a target rendering tree, the second rendering tree is generated by the second application, and the third rendering tree is generated by the second application.
  • the second rendering tree is used to generate the interface of the second application
  • the first interface also includes the interface of the second application
  • the target rendering tree does not include the third rendering node
  • the third rendering node corresponds to the third view
  • the The third view is the view of the first application, and the third view is completely blocked by the interface of the second application
  • generating the first interface based on the first rendering tree specifically includes: generating the first interface based on the target rendering tree .
  • views that are completely occluded by the interfaces of other applications can also be deleted, thereby generating a rendering tree corresponding to the interfaces of multiple applications, and the rendering tree does not include rendering nodes of the views that are completely occluded.
  • the method before generating the first interface based on the target rendering tree, the method further includes: determining that the rendering node corresponding to the view completely occluded by the interface of the second application includes: The third rendering node; merging the first rendering tree and the second rendering tree to obtain the target rendering tree specifically includes: in the process of merging the first rendering tree and the second rendering tree to obtain the target rendering tree, from the third The third rendering node is deleted from a rendering tree or the target rendering tree.
  • the rendering nodes corresponding to the completely occluded views can be deleted.
  • the method before generating the first interface based on the target rendering tree, the method further includes: determining that the rendering node corresponding to the view completely occluded by the interface of the second application includes: The third rendering node; merging the first rendering tree and the second rendering tree to obtain a target rendering tree, specifically including: merging the first rendering tree and the second rendering tree to obtain a third rendering tree; Delete the third rendering node to obtain the target rendering tree.
  • the rendering node corresponding to the completely occluded view can also be deleted.
  • the rendering node corresponding to the completely occluded view includes the third rendering node, specifically including: the window Z order based on the first application, the second application
  • the window Z order, the first rendering tree and the second rendering tree determine that the rendering node corresponding to the view of the first application that is completely obscured by the interface of the second application includes the third rendering node; or, determine the completely obscured area,
  • the rendering node corresponding to the view of the first application that is completely blocked by the interface of the second application includes the third rendering node, and the completely blocked area is the interface of the second application. Covers the area of the first application's interface.
  • the display shielding relationship of different application program interfaces can be determined through the window Z order of different applications, and then the completely blocked view can be determined.
  • the method further includes: the first interface further includes an interface of a second application, and the interface of the second application completely blocks the first area of the interface of the first application. , determine the first area; determine based on the first area that the view completely blocked by the interface of the second application includes a fourth view, the first rendering tree does not include a fourth rendering node, and the fourth rendering node is related to the fourth View correspondence.
  • the electronic device can first determine the occluded area in the two applications, so that the application program does not generate rendering nodes corresponding to the views in the area during the process of generating the rendering tree.
  • the method further includes: determining that the partially occluded view includes a fifth view; adding a cropping instruction to the drawing instruction list in the fifth rendering node, where the cropping instruction is used In order to indicate not to render the partially occluded content in the fifth view, the fifth rendering node corresponds to the fifth view.
  • a clipping instruction can be added so that the partially occluded content is not rendered during the bitmap generation process, thereby reducing the load on the GPU.
  • an embodiment of the present application provides an interface generation method, which is applied to an electronic device including a screen, on which a first application, a second application and a first process are running, the method comprising: the first application sends a first rendering tree to the first process, the first rendering tree is used to generate an interface of the first application; the second application sends a second rendering tree to the first process, the second rendering tree is used to generate an interface of the second application; the first process generates a target rendering tree based on the first rendering tree and the second rendering tree, the target rendering tree is used to generate a first interface, the first interface includes an interface of the first application and an interface of the second application; after the first process determines a first area, the first rendering node is deleted based on the first area, the interface of the second application completely covers the first area of the interface of the first application, the target rendering tree does not include the first rendering node, the first rendering node is a rendering node in the first rendering tree, a first view corresponding to the first rendering node is
  • the first process can receive rendering trees generated by different applications, determine fully occluded views across applications, and then delete the rendering nodes corresponding to the completely occluded views to reduce the time spent in the process of generating bitmaps. Overdraw, reducing GPU load.
  • the method before the first application sends the first rendering tree, the method further includes: the first application determines a third rendering tree by traversing the view, and the third rendering The tree is used to generate the interface of the first application; the first application determines that views that are completely obscured by other views of the first application include the second view; the first application deletes the second rendering node in the third rendering tree The first rendering tree is obtained, and the second rendering node corresponds to the second view.
  • the application before sending the rendering tree to the first process, the application can first delete the rendering node corresponding to the completely occluded view, thereby reducing the load on the GPU during the subsequent bitmap generation process.
  • the method further includes: the first process deletes a second rendering node, the second rendering node is a rendering node in the first rendering tree, and the second rendering Nodes correspond to the second view, which is completely obscured by other views of the first application.
  • the first process can delete the rendering node of a view that is completely occluded by other views in the same application, thereby reducing the load on the GPU during subsequent bitmap generation.
  • embodiments of the present application provide an interface generation method, applied to an electronic device including a screen.
  • the method includes: the electronic device displays a first interface on the screen, and in the first interface, the first application
  • the interface of the program displays the first content in full screen
  • the interface of the second application program displays the second content suspended on the interface of the first application program
  • the interface of the second application program is located in the first area in the first interface; in During the process of displaying the first interface, the electronic device generates a first rendering tree, which is used to generate the interface of the first application;
  • the electronic device displays a second interface on the screen, and in the second interface , the interface of the first application program displays the first content in full screen, the interface of the second application program displays the second content in a floating manner on the first application program, and the second application program is located in the second part of the first interface.
  • area the first area is different from the area; in the process of displaying the second interface, the electronic device generates a second rendering tree, the second rendering tree is used to
  • the interface of the second application is suspended and displayed on the interface of the first application.
  • the position of the interface of the second application changes and the interface of the first application does not change, because the first application
  • the rendering node corresponding to the completely occluded view is deleted, so the rendering tree generated by the first application is different and does not contain redundant information, thereby reducing the load on the GPU during subsequent bitmap generation.
  • the first rendering tree is used to generate an interface of the first application that does not include the first area; the second rendering tree is used to generate the first The interface of the application does not include the interface of the second area.
  • the load of the GPU in the subsequent process of generating bitmaps can be reduced.
  • the electronic device generates a first rendering tree, specifically including: the electronic device determines, based on the first area, the view of the first application that is obscured by the interface of the second application. Including a first view; the electronic device generates a third rendering tree; the electronic device deletes the first rendering node in the third rendering tree to obtain the first rendering tree, the first rendering node corresponds to the first view, and the rendering node Save the drawing instructions of the corresponding view.
  • the electronic device can delete the view of the first application that is blocked by the interface of the second application, thereby reducing the load of the GPU in the subsequent bitmap generation process and improving the energy efficiency of interface generation.
  • inventions of the present application provide an electronic device.
  • the electronic device includes: one or more processors, a memory, and a screen; the memory is coupled to the one or more processors, and the memory is used to store a computer program. code, the computer program code includes computer Instructions, the one or more processors invoke the computer instructions to cause the electronic device to perform: determine that a view that is completely occluded by other views in the first application includes the first view; generate a first rendering tree, the first rendering tree does not It includes a first rendering node, the first rendering node is a rendering node corresponding to the first view, and the rendering node stores a drawing instruction of the corresponding view; a first interface is generated based on the first rendering tree, and the first interface includes the third An application interface.
  • the one or more processors are specifically configured to invoke the computer instructions to cause the electronic device to execute: in the process of traversing the view of the first application, determine The position of the second view is the second position; when traversing to the first view, the position of the first view is determined to be the first position, and the first view is determined to be the first position based on the second position and the first position.
  • the one or more processors are also configured to call the computer instructions to cause the electronic device to execute: not saving the drawing instructions of the first view to the first rendering. In the tree; skip the subviews of the first view to continue traversing, and the first rendering tree does not include rendering nodes corresponding to the subviews of the first view.
  • the one or more processors are specifically configured to invoke the computer instructions to cause the electronic device to execute: in the process of traversing the view of the first application, determine After the position of the first view is the first position, the position of the second view is determined to be the second position; based on the first position and the second position, it is determined that the first view is completely blocked by the second view, and the first view is recorded. The ID of the view is passed to the first parameter.
  • the one or more processors are specifically configured to call the computer instructions to cause the electronic device to execute: after traversing the views of the first application, generate a second rendering tree, the second rendering tree includes a first rendering node; the first rendering node in the second rendering tree is deleted based on the first parameter to obtain the first rendering tree.
  • the one or more processors are specifically configured to call the computer instructions to cause the electronic device to execute: after traversing the views of the first application, generate a second rendering tree, the second rendering tree includes the first rendering node and the second rendering node; it is determined based on the rendering attributes of the first rendering node and the rendering attributes of the second rendering node that the first view is completely obscured by the second view, the The second view corresponds to the second rendering node.
  • the one or more processors are specifically configured to call the computer instructions to cause the electronic device to execute: delete the second rendering node in the second rendering tree to obtain The first render tree.
  • the one or more processors are further used to call the computer instructions to cause the electronic device to execute: merging the first rendering tree and the second rendering tree to obtain a target rendering tree, the second rendering tree is generated by the second application, the second rendering tree is used to generate an interface of the second application, the first interface also includes the interface of the second application, the target rendering tree does not include the third rendering node, the third rendering node corresponds to a third view, the third view is a view of the first application, and the third view is completely obscured by the interface of the second application; generating the first interface based on the first rendering tree specifically includes: generating the first interface based on the target rendering tree.
  • the one or more processors are also configured to call the computer instructions to cause the electronic device to execute: determine the corresponding view that is completely obscured by the interface of the second application.
  • the rendering nodes include the third rendering node; merging the first rendering tree and the second rendering tree to obtain the target rendering tree specifically includes: in the process of merging the first rendering tree and the second rendering tree to obtain the target rendering tree. , delete the third rendering node from the first rendering tree or the target rendering tree.
  • the one or more processors are also configured to call the computer instructions to cause the electronic device to execute: determine the corresponding view that is completely obscured by the interface of the second application.
  • the rendering nodes include the third rendering node; merging the first rendering tree and the second rendering tree to obtain a target rendering tree specifically includes: merging the first rendering tree and the second rendering tree to obtain a third rendering tree; The third rendering node is deleted from the third rendering tree to obtain the target rendering tree.
  • the one or more processors are specifically configured to call the computer instructions to cause the electronic device to execute: the window sequence based on the first application, the second The window Z order of the application, the first rendering tree and the second rendering tree determine that the rendering node corresponding to the view of the first application that is completely obscured by the interface of the second application includes the third rendering node; or, determine the completely obscured area. , and determine based on the first rendering tree and the complete occlusion area that the rendering node corresponding to the view of the first application that is completely occluded by the interface of the second application includes the third rendering node, and the completely occlusion area is of the second application.
  • the interface covers an area of the interface of the first application.
  • the one or more processors are also used to call the computer instructions to cause the electronic device to execute: the first interface also includes an interface of the second application, the The interface of the second application completely blocks the first area of the interface of the first application, and the first area is determined; it is determined based on the first area that the view completely blocked by the interface of the second application includes a fourth view, and the first rendering The tree does not include a fourth rendering node corresponding to the fourth view.
  • the one or more processors are further configured to call the computer instructions to cause the electronic device to perform: determining that the partially occluded view includes a fifth view; A cropping instruction is added to the drawing instruction list in the fifth rendering node. The cropping instruction is used to instruct not to render the partially obscured content in the fifth view.
  • the fifth rendering node corresponds to the fifth view.
  • inventions of the present application provide an electronic device.
  • the electronic device includes: one or more processors, a memory, and a screen; the memory is coupled to the one or more processors, and the memory is used to store computer programs.
  • Code the computer program code includes computer instructions, the one or more processors invoke the computer instructions to cause the electronic device to execute: the first application sends a first rendering tree to the first process, the first rendering tree is used to Generate the interface of the first application; the second application sends a second rendering tree to the first process, and the second rendering tree is used to generate the interface of the second application; the first process is based on the first rendering tree and the The second rendering tree generates a target rendering tree, and the target rendering tree is used to generate a first interface.
  • the first interface includes the interface of the first application and the interface of the second application; after the first process determines the first area, based on The first region deletes the first rendering node, the interface of the second application completely blocks the first region of the interface of the first application, the target rendering tree does not include the first rendering node, and the first rendering node is the first rendering node.
  • a rendering node in the rendering tree, the first view corresponding to the first rendering node is located in the first area, and the rendering node stores the drawing instructions of the corresponding view; the first process generates the first interface based on the target rendering tree .
  • the one or more processors are also configured to call the computer instructions to cause the electronic device to execute: the first application determines the third rendering tree by traversing the view, The third rendering tree is used to generate the interface of the first application; the first application determines that views that are completely obscured by other views of the first application include the second view; the first application deletes the views in the third rendering tree.
  • the second rendering node obtains the first rendering tree, and the second rendering node corresponds to the second view.
  • the one or more processors are also used to call the computer instructions to cause the electronic device to execute: the first process deletes the second rendering node, the second The rendering node is a rendering node in the first rendering tree, the second rendering node corresponds to the second view, and the second view is completely obscured by other views of the first application.
  • embodiments of the present application provide an electronic device.
  • the electronic device includes: one or more processors, a memory, and a screen; the memory is coupled to the one or more processors, and the memory is used to store computer programs.
  • Code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to cause the electronic device to execute: the electronic device displays a first interface on the screen, and in the first interface, the first application
  • the interface of the program displays the first content in full screen
  • the interface of the second application program displays the second content suspended on the interface of the first application program, and the interface of the second application program is located in the first area in the first interface; in During the process of displaying the first interface, the electronic device generates a first rendering tree, which is used to generate the interface of the first application;
  • the electronic device displays a second interface on the screen, and in the second interface , the interface of the first application program displays the first content in full screen, the interface of the second application program displays the second content in a floating manner on the first application
  • the first rendering tree is used to generate an interface of the first application that does not include the first area; the second rendering tree is used to generate the first The interface of the application does not include the interface of the second area.
  • the one or more processors are specifically configured to invoke the computer instructions to cause the electronic device to execute: the electronic device determines the second application based on the first area The view of the first application blocked by the interface includes the first view; the electronic device generates a third rendering tree; the electronic device deletes the first rendering node in the third rendering tree to obtain the first rendering tree, and the first rendering The node corresponds to the first view, and the rendering node stores the drawing instructions of the corresponding view.
  • embodiments of the present application provide a chip system, which is applied to an electronic device.
  • the chip system includes one or more processors, and the processor is used to call computer instructions to cause the electronic device to execute the first step.
  • embodiments of the present application provide a computer program product containing instructions.
  • the electronic device causes the electronic device to execute the first aspect, the second aspect, the third aspect, and the first aspect. Any possible implementation method in the aspect, any possible implementation method in the second aspect, and any possible implementation method in the third aspect.
  • embodiments of the present application provide a computer-readable storage medium, including instructions.
  • the electronic device When the instructions are run on an electronic device, the electronic device causes the electronic device to execute the first aspect, the second aspect, the third aspect, and the first aspect. Any possible implementation method in the aspect, any possible implementation method in the second aspect, and any possible implementation method in the third aspect.
  • the electronic device provided by the fourth aspect the electronic device provided by the fifth aspect, the electronic device provided by the sixth aspect, the chip system provided by the seventh aspect, the computer program product provided by the eighth aspect, and the ninth aspect provide computer storage media are used to execute the methods provided by the embodiments of this application. Therefore, the beneficial effects it can achieve can be referred to the beneficial effects in the corresponding methods, and will not be described again here.
  • Figure 1 is an exemplary schematic diagram of a bitmap generated by an application program according to an embodiment of the present application.
  • FIG. 2 is an exemplary schematic diagram of a rendering thread generating a bitmap based on a rendering tree provided by an embodiment of the present application.
  • 3A-3C are an exemplary schematic diagram of the interface provided by the embodiment of the present application when views are mutually obscured.
  • FIG. 4 is another exemplary schematic diagram of an interface in a case where views obscure each other provided in an embodiment of the present application.
  • FIG. 5 is an exemplary schematic diagram of an electronic device generation interface provided in an embodiment of the present application.
  • Figure 6 is an exemplary schematic diagram of the process of the interface generation method provided by the embodiment of the present application.
  • FIG. 7 is an exemplary schematic diagram for determining a completely occluded view provided by an embodiment of the present application.
  • FIG. 8 is an exemplary schematic diagram for determining a completely occluded view provided by an embodiment of the present application.
  • FIG. 9 is another exemplary schematic diagram of determining a completely blocked view provided by an embodiment of the present application.
  • Figures 10A-10D provide an exemplary schematic diagram of a rendering thread or unified rendering process deleting a rendering node according to the embodiment of the present application.
  • Figure 11 is an exemplary schematic diagram of the data flow of the interface generation method provided by the embodiment of the present application.
  • Figure 12 is another exemplary schematic diagram of the data flow of the interface generation method provided by the embodiment of the present application.
  • Figures 13 and 14 are another exemplary schematic diagram of the data flow of the interface generation method provided by the embodiment of the present application.
  • FIG. 15 is an exemplary schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present application.
  • 16A and 16B are another exemplary schematic diagram of the software architecture of the electronic device provided in an embodiment of the present application.
  • first and second are used for descriptive purposes only and shall not be understood as implying or implying relative importance or implicitly specifying the quantity of indicated technical features. Therefore, the features defined as “first” and “second” may explicitly or implicitly include one or more of the features. In the description of the embodiments of this application, unless otherwise specified, “plurality” The meaning is two or more.
  • GUI graphical user interface
  • the interface serves as a media interface for interaction and information exchange between the application and the user. Every time a vertical synchronization signal (vsync) arrives, the electronic device needs to generate the interface of the application for the foreground application.
  • the frequency of the vertical synchronization signal is related to the refresh rate of the screen of the electronic device. For example, the frequency of the vertical synchronization signal is the same as the refresh rate of the screen of the electronic device.
  • each time the electronic device refreshes the content displayed on the screen it needs to generate the interface of the application for the foreground application, so as to present the newly generated interface of the application to the user when the screen is refreshed.
  • the electronic device generates an application interface that requires the application to render and generate a bitmap (bitmap) by itself, and pass its own bitmap to the surface compositor (SurfaceFlinger). That is, the application performs rendering as a producer to generate a bitmap, and stores the bitmap in the buffer queue (BufferQueue) provided by the surface compositor; the surface compositor, as a consumer, continuously obtains the bitmap generated by the application from the BufferQueue. Among them, the bitmap is located on the surface generated by the application, and the surface will be filled into the BufferQueue.
  • the surface compositor After the surface compositor obtains the visible application bitmap, the surface compositor interacts with the hardware composition strategy module (Hardware Composer, HWC) determines the way bitmaps are layered as layers.
  • HWC hardware Composer
  • the surface compositor and/or hardware composition strategy module After the surface compositor and/or hardware composition strategy module performs bitmap composition, the surface compositor and/or hardware composition strategy module fills in the synthesized bitmap (the bitmap can also be called a layer on the surface compositor). It is passed into the frame buffer (Frame Buffer) and passed to the display subsystem (Display Subsystem, DSS). After DSS gets the synthesized bitmap, it can display the synthesized bitmap on the screen.
  • the frame buffer can be an on-screen buffer.
  • Figure 1 is an exemplary schematic diagram of a bitmap generated by an application program according to an embodiment of the present application.
  • the application After receiving the vertical synchronization signal, the application begins to generate a bitmap.
  • the specific steps can be divided into three steps, namely 1, 2, and 3.
  • views are the basic elements that constitute the application interface.
  • a control on the interface can correspond to one or more views.
  • the UI thread of the application will also read the content carried on the view into the memory.
  • the picture view (imageview) carries the picture
  • the text view (imageview) carries the text.
  • the application's UI thread determines the operation of reading the content carried on the view into memory and recording it into the draw instruction list.
  • the drawing operation structure is a data structure used to draw graphics, such as drawing lines, drawing rectangles, drawing text, etc.
  • the drawing operation structure will be converted into API calls of the image processing library at the rendering node, such as OpenGLES interface calls.
  • DrawLineOp is a data structure, which contains drawing data such as line length, width and other information.
  • the drawing instruction list may be a buffer, which records all drawing operation structures or identifiers of all drawing operations included in one frame of the application program interface, such as addresses, serial numbers, etc.
  • the drawing instruction list may be a buffer, which records all drawing operation structures or identifiers of all drawing operations included in one frame of the application program interface, such as addresses, serial numbers, etc.
  • the rendering tree is generated by the UI thread and is used to generate a data structure of the application interface.
  • the rendering tree can include multiple rendering nodes, and each rendering node includes rendering attributes and a drawing instruction list.
  • the rendering tree records some or all of the information that generates a frame of the application's interface.
  • the rendering node in the rendering tree includes a drawing instruction list and rendering attributes.
  • the rendering attributes are used to determine the position, size, transparency and other attributes of the view to be rendered by the rendering node in the surface (which can also be called the canvas on the surface).
  • the drawing instruction list is used to determine the content of the view to be rendered by the rendering node, such as lines, rectangles, circles, etc.
  • the interface that the application needs to display is composed of multiple nested views, and different views have a parent-child relationship, so the parent-child relationship between the rendering nodes of the rendering tree generated by traversing the view is the same as the parent-child relationship of the view. That is, the parent-child relationship between views determines the nested relationship between different rendering nodes (hereinafter, the nested relationship between rendering nodes can also be called parent-child relationship), and then the rendering thread generates bitmaps based on the rendering tree.
  • the application interface can be rendered correctly.
  • a view can correspond to one or more rendering nodes, and the root view (DecorView) corresponds to the root rendering node (RootRenderNode).
  • the rendering node also includes rendering properties (properties), which are used to determine the position, size, transparency, etc. of the view corresponding to the rendering node on the surface when rendering and generating the bitmap.
  • the main thread may only perform traversal of views in dirty areas (which may also be referred to as areas that need to be redrawn) to generate a differential rendering tree.
  • the rendering thread can determine the rendering tree that needs to be used for interface rendering in this frame through the differential rendering tree and the rendering tree used for the previous frame rendering.
  • the UI thread passes/synchronizes the rendering tree to the rendering thread (Render Thread).
  • the rendering tree is located in the stack of the process corresponding to the application, and may not be continuously distributed on the physical address.
  • the rendering thread first obtains a hardware canvas (HardwareCanvas), and performs drawing operations in the rendering tree on the hardware canvas to generate a bitmap.
  • the hardware canvas is located in a surface held by the application, and the surface carries bitmap or other format data used to save image information.
  • 1 is the construction phase, which is mainly responsible for determining the size, position, transparency and other attributes of each view in the application.
  • drawLine in the view can be encapsulated into a DrawLineOp during construction, which contains drawing data such as line length, width, etc., and can also include interface calls corresponding to the DrawLineOp of the underlying graphics processing library, which is used in the rendering phase. Call the underlying graphics library to generate bitmaps.
  • 3 can be considered as the rendering stage, which is mainly responsible for traversing the rendering nodes of the rendering tree and performing the drawing operation of each rendering node, thereby generating a bitmap on the hardware canvas.
  • the rendering thread calls the underlying graphics processing Libraries, such as OpenGL ES library, Skia library, etc., then call the GPU to complete rendering to generate bitmaps.
  • the construction phase requires the use of CPU computing resources
  • the rendering phase requires the use of GPU resources.
  • FIG. 2 is an exemplary schematic diagram of a rendering thread generating a bitmap based on a rendering tree provided by an embodiment of the present application.
  • the generated rendering tree includes a root rendering node, rendering node 1, rendering node 2, and rendering node 3.
  • rendering node 1 is the left child node of the root rendering node
  • rendering node 2 is the right child node of the root rendering node
  • rendering node 3 is a child node of rendering node 2.
  • the rendering content corresponding to rendering node 1 includes rendering bitmap 1
  • the rendering content corresponding to rendering node 2 includes rendering bitmap 2.
  • the rendering thread traverses the rendering tree in the process of generating bitmaps, the order from front to back is: root rendering node, rendering node 1, rendering node 2, rendering node 3. Then, the rendering thread first performs the drawing operation in rendering node 1, and then performs the drawing operation in rendering node 2. That is, the rendering thread calls the GPU to first render view 1 and then render view 2 on the surface provided by the surface compositor. Since view 2 completely obscures view 1, after the rendering thread renders view 2, view 1 becomes invisible.
  • 3A-3C are an exemplary schematic diagram of the interface provided by the embodiment of the present application when views are mutually obscured.
  • the interface displayed by the application program on the electronic device is interface 3001.
  • the interface 3001 includes multiple views corresponding to controls, where control 301 is view 3 and control 302 is view 8.
  • control 301 is view 3
  • control 302 is view 8.
  • the interface displayed by the application program changes from interface 3001 to interface 3002.
  • control 301 In the process of the interface 3001 changing to the interface 3002, the control 301 continues to become larger. In the process of becoming larger, the control 301 completely blocks the control 302, that is, the view 3 completely blocks the view 8.
  • Interface 3003 includes multiple controls, such as control 303, control 304, control 305, and control 306.
  • control 303 controls
  • control 304 controls
  • control 305 controls
  • the application displays a fragment 307, which completely covers the controls 304, 305, and 306.
  • the interface displayed by the application program on the electronic device changes from interface 3005 to interface 3006.
  • the interface 3005 includes a control 308, and the interface 3006 includes a floating window 309.
  • the floating window 309 is a window used for playing videos, and the floating window 309 completely blocks the control 308.
  • Figure 4 is another exemplary schematic diagram of the interface provided by the embodiment of the present application when views are mutually obscured.
  • interface 3007 is the interface of application 1
  • interface 3008 is the interface of application 1 and application 2.
  • the interface of the application is displayed in the floating window 311, and in interface 3008, the icon control 310 is fully prepared by the floating window 311 .
  • Figure 5 is an exemplary schematic diagram of an electronic device generation interface provided by an embodiment of the present application.
  • application 1 generates bitmap 1 and passes bitmap 1 to the surface compositor and/or hardware compositing strategy module;
  • application 2 generates bitmap 2 and passes bitmap 2 to the surface compositor. and/or a hardware compositing strategy module;
  • the surface compositor and/or a hardware compositing strategy module performs layer composition on bitmap 1 and bitmap 2 to obtain bitmap 3.
  • bitmap 1 is the bitmap corresponding to the interface 3007
  • bitmap 2 is the bitmap corresponding to the floating window 311
  • bitmap 3 is the bitmap of the interface 3008.
  • the operation of rendering a completely obscured view onto the surface by the rendering thread is an invalid operation, wasting rendering threads.
  • the UI thread since the UI thread needs to traverse the view before the rendering thread performs rendering, the UI thread determines and saves the drawing operation of the completely obscured view as an invalid operation, which wastes the resources of the UI thread and reduces the energy efficiency of interface generation.
  • inventions of the present application provide an interface generation method.
  • the interface generation method provided by the embodiment of the present application includes: during the process of the UI thread traversing the views, after determining that any view is a completely occluded view, skip the view to continue executing the traversal; secondly, after determining that any view is a completely occluded view, skip the measurement method calls and layout method calls of the subviews of the view; finally, the rendering tree generated by the UI thread of the application does not contain the The corresponding render node for a fully occluded view.
  • the calculation amount of the UI thread is reduced; secondly, because the UI thread skips the traversal operation of the child nodes of the completely occluded view, further reducing the The calculation amount of the UI thread is reduced; thirdly, the rendering tree does not contain the corresponding rendering node of the completely occluded view, which reduces the calculation amount of the bitmap generated by the rendering thread, thereby improving the energy efficiency ratio of the interface generation.
  • the UI thread or the rendering thread performs a traversal of the rendering tree to determine and trim the corresponding rendering nodes of the completely occluded views, thereby reducing The amount of calculation required by the rendering thread to render the generated bitmap.
  • the unified rendering process (unirender) merges the rendering trees of one or more applications to generate a target rendering tree, or in the process of merging and generating the target rendering tree, the unified rendering process determines and clips the corresponding rendering nodes of the completely obscured view, thereby reducing the amount of calculation for the rendering thread to render and generate the bitmap.
  • the unified rendering process is a process independent of the application, responsible for bitmap rendering, layer synthesis, etc. After generating the bitmap, the unified rendering process will pass the bitmap to the display subsystem for display.
  • the unified rendering process or the window management service sends a notification to the application, and the notification is used to indicate the application's The location of the area of the window that is completely obscured by other applications.
  • the UI thread of the application determines that any view is a completely occluded view, and then skips the view to continue the traversal; after generating the rendering tree, the UI thread of the application determines and clips the view that is completely occluded. The corresponding rendering node.
  • the following exemplarily introduces the process of the interface generation method provided by the embodiment of the present application in conjunction with the content shown in FIG. 6 .
  • Figure 6 is an exemplary schematic diagram of the process of the interface generation method provided by the embodiment of the present application.
  • the electronic device can determine the completely occluded view in a variety of ways. Several methods of determining the completely occluded view are illustratively introduced below, which are not limited here.
  • the UI thread of the application may determine a completely obscured view during the process of traversing the view (which may be a traversal of only dirty areas).
  • the UI thread of the application can determine the size and position of view 3 and the size and position of view 8 through measurement method calls and layout method calls. According to the elevation attribute of the view or the default sequence of the views, it is determined that view 3 completely blocks view 8, that is, view 8 is a completely blocked view.
  • the default order of views may refer to the order of views in the view structure.
  • the order of the views in the view structure is that the left view node is higher than the right view node among the sibling view nodes. That is, view 8 can be considered as the right view node of view 3. In the case where view 8 is the left view node of view 3, view 8 will not be completely obscured by view 3.
  • the traversal of the subviews of the first view is skipped and the traversal continues, as shown in Figure 7 .
  • FIG. 7 is an exemplary schematic diagram for determining a completely occluded view provided by an embodiment of the present application.
  • the application's UI thread traversing views, after traversing to the first view and determining the position of the first view, it is determined that the first view is completely obscured, such as by view 2 or view 3. After the application's UI thread determines that the first view is completely obscured, it will not traverse View 4 and View 5, nor will it determine and save the drawing operations of the first view, View 4, and View 5, thereby saving the application's UI thread's time. computing resources, thereby improving the energy efficiency of interface generation.
  • UI thread of the application can also traverse the views in other orders, which is not limited here.
  • the UI thread of the application can determine the completely occluded view after traversing the views.
  • the UI thread of the application prunes the rendering tree, maintaining the original logic of view traversal of the UI thread of the application, and making less modifications to the view system; on the other hand, the UI thread of the application prunes the view traversal.
  • the order is not necessarily consistent with the mutual shielding relationship of the views. As a result, when traversing view container 2, it is determined that the first view that has been traversed is completely obscured by view container 2, and the first view cannot be skipped during the traversal process. As shown in Figure 8.
  • FIG. 8 is an exemplary schematic diagram for determining a completely occluded view provided by an embodiment of the present application.
  • a parameter is recorded separately. This parameter is used to point to the obscured view.
  • the UI thread of the application program can determine the first view, View 2 and View 3 are completely obscured.
  • the UI thread, rendering thread or unified rendering process of the application can determine that the first view, view 2, and view 3 are completely obscured based on the rendering tree.
  • the rendering thread or unified rendering process traverses the rendering tree and determines the occlusion relationship between views through the rendering attributes of the rendering nodes.
  • the application when the interface displayed by the application includes at least two windows, the application can also determine the completely obscured view based on the window Z-order.
  • the UI thread of the application after receiving the vertical synchronization signal, the UI thread of the application will generate multiple rendering trees (the number of rendering trees can be the same as the number of windows), and the UI thread can determine the completely obscured view in the process of traversing any rendering tree, and then further determine the completely obscured view based on the window Z-order.
  • the UI thread, the rendering thread or the unified rendering process determines the completely obscured view from the inside of the rendering tree based on the rendering tree, and then further determines the completely obscured view based on the window Z-order.
  • the window Z-order can be specified by the application or the window management service to indicate the upper and lower relationship (mutual obscuration relationship) between different windows, or the Z-order of the window represents the height of the window on the Z-axis, and the Z-axis is perpendicular to the normal of the surface.
  • the control 308 completely obscured by the floating window 309 is a completely obscured view further determined based on the window Z order.
  • the window Z-order of the window where the floating window 309 is located is higher than the window Z-order of the window where the control 308 is located.
  • the unified rendering process determines the views corresponding to the rendering nodes in the rendering trees of different applications based on the window Z order. mutual occlusion relationship, and then the unified rendering process determines the completely occluded view. Among them, the unified rendering process will receive the window Z order of different windows determined and sent by the window management service.
  • the unified rendering process can determine the icon control 310 Completely obscured.
  • the window management service determines the completely blocked areas in different windows based on the window Z order, so that the application program
  • the UI thread and rendering thread can determine views that are completely obscured by other windows.
  • the UI thread of the application can determine the view during the traversal, or, after the UI thread of the application traverses the view, the UI thread or the rendering thread determines the view that is completely obscured by other windows.
  • FIG. 9 is another exemplary schematic diagram of determining a completely blocked view provided by an embodiment of the present application.
  • the window management service After determining that a completely obscured area exists in the window of any application, the window management service sends a message to the application.
  • the message is used to notify the application of the location of the completely obscured area in the window.
  • the window management service determines that the window of application 1 will be completely blocked by the window of application 2, it sends a message to application 1. This message is used to notify application 1 that the completely blocked area is in the window. location in.
  • Application 1's UI thread, rendering thread, or unified rendering process may have views that are determined to be fully occluded based on this message.
  • the rendering thread or unified rendering process will not render a fully occluded view when rendering the resulting bitmap.
  • the corresponding rendering nodes will still appear in the rendering tree and be rendered on the surface by the rendering thread or the unified rendering process.
  • the rendering node corresponding to the incomplete occlusion will still appear in the rendering tree, and during the rendering process, the incomplete occlusion will be stopped by clipping instructions such as Clip() to render on the completely occluded area.
  • the UI thread of the application can pause or stop the playback of the video. Pausing or stopping the playback of the video can be done through Media codec (mediacodec), media player (mediaplayer) and other functional modules are implemented.
  • mediacodec Media codec
  • mediaplayer media player
  • the input buffer of the media codec can be deleted, and the input buffer of the media codec can be deleted.
  • the decoder pauses the decoding of the video and deletes the output buffer of the media codec to pause or stop the video playback.
  • the UI thread of the application program can convert the video to background playback, that is, only play the audio of the video.
  • host videos are not limited to video views, but can also be views provided by other third-party libraries, custom views, etc.
  • the UI thread of the application program when it is determined that the completely occluded view is an image view, the UI thread of the application program will not load the image or decode the image.
  • control hosting the video is not limited to image views, but can also be views provided by other third-party libraries, custom views, etc.
  • S602 The electronic device generates a rendering tree that does not include the rendering node corresponding to the first view.
  • step S601 the UI thread, rendering thread and/or unified rendering process of the application program have determined the completely occluded view, and then the target rendering tree can be generated.
  • the target rendering tree does not include rendering nodes corresponding to the first view and other completely occluded views.
  • the UI thread, rendering thread or unified rendering process of the application can add a clipping instruction such as Clip() in the drawing instruction list of the rendering node corresponding to the incomplete occlusion, so that the incomplete occlusion is not It will render to the completely occluded area, reducing invalid rendering and improving the energy efficiency of interface generation.
  • a clipping instruction such as Clip() in the drawing instruction list of the rendering node corresponding to the incomplete occlusion
  • the UI thread of the application does not generate the rendering node corresponding to the completely occluded view, nor does it determine and save the drawing of the completely occluded view.
  • the rendering tree generated by the UI thread of the application does not include the rendering nodes corresponding to the first view and other completely occluded views.
  • the UI thread of the application in the case where the UI thread of the application determines a completely occluded view after traversing the views, the UI thread of the application can delete the corresponding rendering node of the completely occluded view. , and then generate a rendering tree that does not include rendering nodes corresponding to the first view and other completely occluded views.
  • the UI thread of the application in the case where the UI thread of the application determines a completely occluded view after traversing the views, the UI thread of the application can send the identification of the completely occluded view to the renderer.
  • the rendering thread or the unified rendering process then deletes the rendering node corresponding to the completely occluded view, and then generates a rendering tree that does not include the rendering nodes corresponding to the first view and other completely occluded views.
  • the UI thread of the application in the case where the UI thread of the application determines the view that is completely occluded after traversing the views, the UI thread of the application can determine the view that is completely occluded in the rendering node corresponding to the view that will be completely occluded. Add a parameter to the rendering attribute or drawing instruction list, which is used to identify the view corresponding to the rendering node as a fully occluded view.
  • the rendering thread or the unified rendering process receives the rendering tree, the rendering thread or the unified rendering process deletes the corresponding rendering node of the completely occluded view, and then generates a rendering node that does not include the first view and other completely occluded views. Render tree.
  • the generated rendering tree is passed to the rendering thread or the unified rendering process.
  • the rendering thread or the unified rendering process determines the view that is completely occluded, The rendering thread or the unified rendering process deletes the corresponding rendering node of the completely occluded view, thereby generating a rendering tree that does not include the first view and the rendering nodes corresponding to other completely occluded views.
  • the UI thread, rendering thread, and unified rendering process of the application program uniformly delete the rendering node corresponding to the completely obscured view, and skip the rendering node and the child rendering nodes of the rendering node during the rendering process.
  • the UI thread, rendering thread, or unified rendering process of the application program needs to add parameters to the rendering node so that the rendering thread or unified rendering process determines that the rendering node is the rendering node corresponding to the completely obscured view when generating a bitmap.
  • Figures 10A-10D provide an exemplary schematic diagram of a rendering thread or unified rendering process deleting a rendering node according to the embodiment of the present application.
  • the rendering thread deletes the rendering node as shown in Figure 10A.
  • Rendering tree 1 includes rendering node 1, rendering node 1, rendering node 2, and rendering node 3.
  • the rendering thread can determine that rendering node 3 corresponds to a completely occluded view, such as the first view.
  • the rendering thread deletes render node 3.
  • the bitmap obtained by the rendering thread performing rendering based on rendering tree 1 (the rendering tree before deleting rendering node 3) is bitmap 1
  • the rendering thread performing rendering based on the target rendering tree (the rendering tree after deleting rendering node 3) obtains the bitmap Same for Bitmap 1.
  • the unified rendering process deletes rendering nodes as shown in Figure 10B and Figure 10C. Unlike rendering threads, the unified rendering process receives rendering trees generated by the UI threads of different applications.
  • the unified rendering process receives the rendering tree 1 generated by the UI thread of application 1 and the UI of application 2 Thread-generated rendering tree 2.
  • Rendering tree 1 includes root rendering node 1, rendering node 1, rendering node 2 and rendering node 3.
  • Rendering node 3 is a child rendering node of rendering node 2, and rendering node 1 and rendering node 2 are child nodes of root rendering node 1.
  • Rendering tree 2 includes root rendering node 2, rendering node 4, rendering node 5 and rendering node 6.
  • Rendering node 6 is a child rendering node of rendering node 5, and rendering node 5 and rendering node 4 are child rendering nodes of root rendering node 2. node.
  • Render node 6 corresponds to a fully occluded view.
  • the unified rendering process deletes rendering node 6 during the process of merging rendering trees to obtain a target rendering tree.
  • the process of merging rendering trees includes creating a new root rendering node and using root rendering node 1 and root rendering node 2 as child rendering nodes of the new root rendering node.
  • root rendering node 2 is the right child rendering node of the new root rendering node
  • root rendering node 1 is the right child rendering node of the new root rendering node
  • the root rendering node 2 if the window Z-order of the window of application 2 is greater than the window Z-order of the window of application 1, and the view in application 1 does not have incomplete occlusion, then the root rendering node 2 The sibling order of root render node 1 is interchangeable.
  • the sibling order of root rendering node 2 and root rendering node 1 can be interchanged.
  • the rendering tree can be changed during rendering generation.
  • the traversal order of the bitmap traversal rendering nodes does not change the traversal order determined by the parent-child relationship).
  • the rendering instructions of the rendering node in the unified rendering process are The list is decapsulated to obtain a series of drawing operation structures, and then the target rendering tree is used as a whole to perform drawing operation structure operations for sub-batch (Batch) and merge (Merge). In step S603, it can be obtained with less calculation amount. Generate bitmap.
  • the batch and merge ranges can be larger than the range supported by the native Skia library.
  • the native Skia library cannot batch and merge the drawing operation structures that depend on subsequent results (such as the sibling views rendered later) with the drawing operation results that occurred before the subsequent results.
  • the drawing operation structures that depend on subsequent results can be batched and merged with the drawing operation results that occurred before the subsequent results.
  • the merging and batching supported by the original Skia library only perform merging and batching on the drawing operation structure corresponding to the interface of an application (or the interface in a window in an application), so there is optimization potential.
  • the upper limit is low.
  • the target rendering tree since the target rendering tree includes rendering trees of different applications, the energy efficiency ratio of interface generation can be maximized by merging and batch improvement, and the upper limit of priority potential is high.
  • the unified rendering process can adopt multi-threading (such as Multi-threaded rendering supported by the vlukan library) renders different subtrees of the target rendering tree separately, that is, parallel rendering generates bitmaps, which can reduce the time spent on interface generation and improve the energy efficiency of interface generation.
  • multi-threading such as Multi-threaded rendering supported by the vlukan library
  • rendering tree 1 includes the root rendering node
  • the child rendering nodes of root rendering node 1 include rendering node 1
  • rendering node 2 includes rendering node 5
  • rendering nodes of rendering node 2 are rendering node 6 and rendering node 7.
  • the rendering tree corresponding to interface 3009 is the rendering tree obtained by deleting rendering node 1, rendering node 4, and rendering node 5 from rendering tree 1. Among them, the views corresponding to rendering node 1, rendering node 4 and rendering node 5 are in the completely obscured area.
  • the completely occluded area changes
  • the completely occluded view changes
  • the rendering tree generated by the application also changes.
  • the view that is not completely in the completely occluded area is the incomplete occluded view.
  • S603 The electronic device generates a bitmap based on the rendering tree and sends it to the display.
  • the rendering thread After the rendering thread generates the bitmap, it passes the bitmap to the surface compositor and/or hardware compositing strategy module. After performing layer compositing, the surface compositor and/or hardware compositing strategy module sends the composited bitmap to the display subroutine. The system sends the display.
  • the bitmap can be directly sent to the display subsystem and display.
  • the unified rendering process can send the bitmap to the surface compositor and/or hardware compositing strategy module. After performing layer compositing, the surface compositor and/or hardware compositing strategy module will The image is sent to the display subsystem for display.
  • the interface generation method can reduce invalid measurement method calls, layout method calls, and drawing method calls caused by completely occluded views, as well as reduce the number of invalid measurement method calls, layout method calls, and drawing method calls caused by rendering nodes corresponding to completely occluded views. Ineffective rendering, thereby improving the energy efficiency of interface generation.
  • Figure 11 is an exemplary schematic diagram of the data flow of the interface generation method provided by the embodiment of the present application.
  • the data flow of the interface generation method includes the following steps.
  • the window management server may send the location of the completely blocked area to the blocked application. For example, the window management service sends the location of the fully obscured area to Application 1's UI thread.
  • step S1101 is not executed, and in steps S1102 and S1105, the application can only determine the views that are completely obscured by other views of the same application, and generate a rendering tree that does not include the completely obscured views.
  • the UI thread of Application 1 builds a rendering tree 1 that does not contain the rendering node corresponding to the completely occluded view.
  • the completely occluded view includes: the view is completely located in the completely occluded area and the view is occluded by the views of other applications 1.
  • Application 1's UI thread passes or synchronizes render tree 1 to Application 1's rendering thread.
  • S1104 Generate bitmap 1 based on rendering tree 1.
  • the rendering thread of application 1 generates bitmap 1 based on rendering tree 1. During the process of rendering and generating bitmap 1, since rendering tree 1 does not contain the rendering node corresponding to the completely occluded view, it will not be on the surface. Render the fully occluded view.
  • S1105 After receiving the vertical synchronization signal, construct and generate a rendering tree 2 that does not include the rendering node corresponding to the completely occluded view.
  • the UI thread of Application 2 builds a rendering tree 2 that does not contain the rendering node corresponding to the completely occluded view.
  • the views that are completely blocked include: views that are blocked by the views of other applications 2.
  • Application 2's UI thread passes or synchronizes Render Tree 2 to Application 2's rendering thread.
  • the rendering thread of application 2 generates bitmap 2 based on rendering tree 2. During the process of rendering and generating bitmap 2, since rendering tree 2 does not contain the rendering node corresponding to the completely occluded view, it will not be on the surface. Render the fully occluded view.
  • Application 1's rendering thread sends Bitmap 1 to the surface compositor and/or hardware compositing strategy module.
  • Application 2's rendering thread sends Bitmap 2 to the surface compositor and/or the hardware compositing strategy module.
  • the surface compositor and/or hardware compositing strategy module generates bitmap 3 after performing layer compositing on bitmap 1 and bitmap 2.
  • the surface compositor and/or hardware compositing strategy module may send bitmap 3 to the display subsystem for display.
  • Figure 12 is another exemplary schematic diagram of the data flow of the interface generation method provided by the embodiment of the present application.
  • the UI thread of application 1 generates rendering tree 1 and deletes the rendering nodes in rendering tree 1.
  • the deleted rendering nodes correspond to completely occluded views, where the completely occluded views are application A view in Application 1 that is completely obscured by other views in Application 1.
  • the UI thread of Application 1 generates Rendering Tree 2 and deletes the rendering nodes in Rendering Tree 2.
  • the deleted rendering nodes correspond to the completely occluded views.
  • the completely occluded views are the ones being applied in Application 2. 2 other views Fully occluded view.
  • Application 1's UI thread passes or synchronizes rendering tree 1 to the unified rendering process
  • application 2 passes or synchronizes rendering tree 2 to the unified rendering process.
  • the unified rendering process generates the target rendering tree based on rendering tree 1 and rendering tree 2, and deletes the rendering nodes.
  • the deleted rendering node corresponds to a completely occluded graph, where a completely occluded view is a view that is completely occluded by the interface of other applications.
  • the unified rendering process generates a bitmap based on the target rendering tree, which is equivalent to bitmap 3 in Figure 11.
  • Figures 13 and 14 are another exemplary schematic diagram of the data flow of the interface generation method provided by the embodiment of the present application.
  • the data flow of the interface generation method includes the following steps.
  • the UI threads of different applications build and generate rendering trees after receiving vertical synchronization signals.
  • Different applications such as Application 1, Application 2, ..., Application N.
  • the UI threads of different applications pass cross-process or synchronize the rendering tree to the unified rendering process.
  • the window management service provides window control information to the unified rendering process.
  • the window control information can include information about windows of different applications, such as size, position, etc.
  • the unified rendering process can determine the application whose interface is occluded and the completely occluded area based on the window control information, and then determine the view that is completely in the completely occluded area.
  • S1304 Merge the rendering trees, delete the rendering nodes corresponding to the completely occluded views, and generate the target rendering tree.
  • the unified rendering process merges multiple rendering trees to generate a target rendering tree, and deletes the rendering nodes corresponding to the completely occluded views.
  • the fully occluded views include views that are occluded by other views in the same application, and also include views that are completely in a completely occluded area.
  • S1305 Generate bitmap 1 based on the target rendering tree.
  • Bitmap 1 is equivalent to bitmap 3 in Figure 11.
  • FIG. 13 The content shown in Figure 13 is different from that shown in Figure 14.
  • an application such as Application 1
  • FIG. 13 The content shown in Figure 13 is different from that shown in Figure 14.
  • the UI thread of an application such as Application 1
  • builds and generates a rendering tree it can delete a completely occluded view.
  • a completely occluded view includes a view within the same application.
  • step S1304 the unified rendering process deletes the rendering node corresponding to the completely occluded view, where the completely occluded view includes a view in a completely occluded area, that is, there is no need for the unified rendering process to delete the view that is used by the same application.
  • the rendering node corresponding to the view obscured by other views in the program.
  • FIG. 15 is an exemplary schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application.
  • the electronic device may be a mobile phone, tablet computer, desktop computer, laptop computer, handheld computer, notebook computer, ultra-mobile personal computer (UMPC), netbook, as well as cellular phone, personal digital assistant (personal digital assistant) assistant (PDA), augmented reality (AR) devices, virtual reality (VR) devices, artificial intelligence (AI) devices, wearable devices, vehicle-mounted devices, smart home devices and/or smart devices City equipment, the embodiment of this application does not place special restrictions on the specific type of electronic equipment.
  • PDA personal digital assistant
  • AR augmented reality
  • VR virtual reality
  • AI artificial intelligence
  • wearable devices wearable devices
  • vehicle-mounted devices smart home devices and/or smart devices City equipment
  • the electronic device may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc.
  • SIM subscriber identification module
  • the sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.
  • the structures illustrated in the embodiments of the present invention do not constitute specific limitations on the electronic equipment.
  • the electronic device may include more or less components than shown in the figures, or some components may be combined, some components may be separated, or some components may be arranged differently.
  • the components illustrated may be implemented in hardware, software, or a combination of software and hardware.
  • the processor 110 may include one or more processing units.
  • the processor 110 may include an application processor (application processor, AP), a modem processor, a graphics processing unit (GPU), and an image signal processor. (image signal processor, ISP), controller, video codec, digital signal processor (digital signal processor, DSP), baseband processor, and/or neural network processor (neural-network processing unit, NPU), etc.
  • application processor application processor, AP
  • modem processor graphics processing unit
  • GPU graphics processing unit
  • image signal processor image signal processor
  • ISP image signal processor
  • controller video codec
  • digital signal processor digital signal processor
  • DSP digital signal processor
  • baseband processor baseband processor
  • neural network processor neural-network processing unit
  • the controller can generate operation control signals based on the instruction operation code and timing signals to complete the control of fetching and executing instructions.
  • the processor 110 may also be provided with a memory for storing instructions and data.
  • the memory in processor 110 is cache memory. This memory may hold instructions or data that have been recently used or recycled by processor 110 . If the processor 110 needs to use the instructions or data again, it can be called directly from the memory. Repeated access is avoided and the waiting time of the processor 110 is reduced, thus improving the efficiency of the system.
  • processor 110 may include one or more interfaces.
  • Interfaces may include integrated circuit (inter-integrated circuit, I2C) interface, integrated circuit built-in audio (inter-integrated circuit sound, I2S) interface, pulse code modulation (pulse code modulation, PCM) interface, universal asynchronous receiver and transmitter (universal asynchronous receiver/transmitter (UART) interface, mobile industry processor interface (MIPI), general-purpose input/output (GPIO) interface, subscriber identity module (SIM) interface, and /or universal serial bus (USB) interface, etc.
  • I2C integrated circuit
  • I2S integrated circuit built-in audio
  • PCM pulse code modulation
  • UART universal asynchronous receiver and transmitter
  • MIPI mobile industry processor interface
  • GPIO general-purpose input/output
  • SIM subscriber identity module
  • USB universal serial bus
  • the I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (derail clock line, SCL).
  • processor 110 may include multiple sets of I2C buses.
  • the processor 110 can separately couple the touch sensor 180K, charger, flash, camera 193, etc. through different I2C bus interfaces.
  • the processor 110 can be coupled to the touch sensor 180K through an I2C interface, so that the processor 110 and the touch sensor 180K communicate through the I2C bus interface to implement the touch function of the electronic device.
  • the I2S interface can be used for audio communication.
  • processor 110 may include multiple sets of I2S buses.
  • the processor 110 can be coupled with the audio module 170 through the I2S bus to implement communication between the processor 110 and the audio module 170 .
  • the audio module 170 can transmit audio signals to the wireless communication module 160 through the I2S interface to implement the function of answering calls through a Bluetooth headset.
  • the PCM interface can also be used for audio communications to sample, quantize and encode analog signals.
  • the audio module 170 and the wireless communication module 160 may be coupled through a PCM bus interface.
  • the audio module 170 can also transmit audio signals to the wireless communication module 160 through the PCM interface to implement the function of answering calls through a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.
  • the UART interface is a universal serial data bus used for asynchronous communication.
  • the bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication.
  • a UART interface is generally used to connect the processor 110 and the wireless communication module 160 .
  • the processor 110 communicates with the Bluetooth module in the wireless communication module 160 through the UART interface to implement the Bluetooth function.
  • the audio module 170 can transmit audio signals to the wireless communication module 160 through the UART interface to implement the function of playing music through a Bluetooth headset.
  • the MIPI interface can be used to connect the processor 110 with peripheral devices such as the display screen 194 and the camera 193 .
  • MIPI interfaces include camera serial interface (CSI), display serial interface (DSI), etc.
  • the processor 110 and the camera 193 communicate through a CSI interface to implement the shooting function of the electronic device.
  • the processor 110 and the display screen 194 communicate through the DSI interface to implement the display function of the electronic device.
  • the GPIO interface can be configured through software.
  • the GPIO interface can be configured as a control signal or as a data signal.
  • the GPIO interface can be used to connect the processor 110 with the camera 193, display screen 194, wireless communication module 160, audio module 170, sensor module 180, etc.
  • the GPIO interface can also be configured as an I2C interface, I2S interface, UART interface, MIPI interface, etc.
  • the USB interface 130 is an interface that complies with the USB standard specification, and may be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc.
  • the USB interface 130 can be used to connect a charger to charge the electronic device, and can also be used to transmit data between the electronic device and peripheral devices. It can also be used to connect headphones to play audio through them. This interface can also be used to connect other electronic devices, such as AR devices, etc.
  • the interface connection relationships between the modules illustrated in the embodiments of the present invention are only schematic illustrations and do not constitute structural limitations on the electronic equipment.
  • the electronic device may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.
  • the charging management module 140 is used to receive charging input from the charger.
  • the charger can be a wireless charger or a wired charger.
  • the charging management module 140 may receive charging input from the wired charger through the USB interface 130 .
  • the charging management module 140 may receive wireless charging input through a wireless charging coil of the electronic device. While the charging management module 140 charges the battery 142, it can also provide power to the electronic device through the power management module 141.
  • the power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110.
  • the power management module 141 receives input from the battery 142 and/or the charging management module 140, and supplies power to the processor 110, the internal memory 121, the display screen 194, the camera 193, the wireless communication module 160, and the like.
  • the power management module 141 can also be used to monitor battery capacity, battery cycle times, battery health status (leakage, impedance) and other parameters.
  • the power management module 141 may also be provided in the processor 110 .
  • the power management module 141 and the charging management module 140 may also be provided in the same device.
  • the wireless communication function of the electronic device can be through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem Processor and baseband processor implementation.
  • Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals.
  • Each antenna in an electronic device can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization.
  • Antenna 1 can be reused as a diversity antenna for a wireless LAN.
  • antennas may be used in conjunction with tuning switches.
  • the mobile communication module 150 can provide wireless communication solutions including 2G/3G/4G/5G applied to electronic devices.
  • the mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc.
  • the mobile communication module 150 can receive electromagnetic waves through the antenna 1, perform filtering, amplification and other processing on the received electromagnetic waves, and transmit them to the modem processor for demodulation.
  • the mobile communication module 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves through the antenna 1 for radiation.
  • at least part of the functional modules of the mobile communication module 150 may be disposed in the processor 110 .
  • at least part of the functional modules of the mobile communication module 150 and at least part of the modules of the processor 110 may be provided in the same device.
  • a modem processor may include a modulator and a demodulator.
  • the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal.
  • the demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal.
  • the demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing.
  • the application processor outputs sound signals through audio devices (not limited to speaker 170A, receiver 170B, etc.), or displays images or videos through display screen 194.
  • the modem processor may be a stand-alone device.
  • the modem processor may be independent of the processor 110 and may be provided in the same device as the mobile communication module 150 or other functional modules.
  • the wireless communication module 160 can provide wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) network), Bluetooth (BT), and global navigation satellite systems for use in electronic devices. (global navigation satellite system, GNSS), frequency modulation (frequency modulation, FM), near field communication technology (near field communication, NFC), infrared technology (infrared, IR) and other wireless communication solutions.
  • WLAN wireless local area networks
  • BT Bluetooth
  • GNSS global navigation satellite system
  • frequency modulation frequency modulation, FM
  • NFC near field communication technology
  • infrared technology infrared, IR
  • the wireless communication module 160 may be one or more devices integrating at least one communication processing module.
  • the wireless communication module 160 receives electromagnetic waves via the antenna 2 , frequency modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110 .
  • the wireless communication module 160 can also receive the signal to be sent from the processor 110, frequency modulate it, amplify it, and convert it into electromagnetic waves through the antenna 2 for
  • the antenna 1 of the electronic device is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device can communicate with the network and other devices through wireless communication technology.
  • the wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), broadband Code division multiple access (wideband code division multiple access, WCDMA), time division code division multiple access (time-division code division multiple access, TD-SCDMA), long term evolution (long term evolution, LTE), BT, GNSS, WLAN, NFC , FM, and/or IR technology, etc.
  • the GNSS may include global positioning system (GPS), global navigation satellite system (GLONASS), Beidou navigation satellite system (BDS), quasi-zenith satellite system (quasi) -zenith satellite system (QZSS) and/or satellite based augmentation systems (SBAS).
  • GPS global positioning system
  • GLONASS global navigation satellite system
  • BDS Beidou navigation satellite system
  • QZSS quasi-zenith satellite system
  • SBAS satellite based augmentation systems
  • the electronic device implements display functions through the GPU, display screen 194, and application processor.
  • the GPU is an image processing microprocessor and is connected to the display screen 194 and the application processor. GPUs are used to perform mathematical and geometric calculations for graphics rendering.
  • Processor 110 may include one or more GPUs that execute program instructions to generate or alter display information.
  • the display screen 194 is used to display images, videos, etc.
  • Display 194 includes a display panel.
  • the display panel can use a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active matrix organic light emitting diode or an active matrix organic light emitting diode (active-matrix organic light emitting diode).
  • LCD liquid crystal display
  • OLED organic light-emitting diode
  • AMOLED organic light-emitting diode
  • FLED flexible light-emitting diode
  • Miniled MicroLed, Micro-oLed, quantum dot light emitting diode (QLED), etc.
  • the electronic device may include 1 or N display screens 194, where N is a positive integer greater than 1.
  • the electronic device can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.
  • the ISP is used to process the data fed back by the camera 193. For example, when taking a photo, the shutter is opened, the light is transmitted to the camera sensor through the lens, the optical signal is converted into an electrical signal, and the camera sensor passes the electrical signal to the ISP for processing, and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on image noise and brightness. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, the ISP may be provided in the camera 193.
  • Camera 193 is used to capture still images or video.
  • the object passes through the lens to produce an optical image that is projected onto the photosensitive element.
  • the photosensitive element may be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor.
  • CMOS complementary metal-oxide-semiconductor
  • the photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP Convert into digital image signal.
  • ISP outputs digital image signals to DSP for processing.
  • DSP converts digital image signals into standard RGB, YUV and other format image signals.
  • the electronic device may include 1 or N cameras 193, where N is a positive integer greater than 1.
  • Digital signal processors are used to process digital signals. In addition to digital image signals, they can also process other digital signals. For example, when the electronic device selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.
  • Video codecs are used to compress or decompress digital video.
  • Electronic devices may support one or more video codecs. In this way, electronic devices can play or record videos in multiple encoding formats, such as: Moving Picture Experts Group (MPEG)1, MPEG2, MPEG3, MPEG4, etc.
  • MPEG Moving Picture Experts Group
  • MPEG2 MPEG2, MPEG3, MPEG4, etc.
  • NPU is a neural network (NN) computing processor.
  • NN neural network
  • Intelligent cognitive applications of electronic devices can be realized through NPU, such as image recognition, face recognition, speech recognition, text understanding, etc.
  • the internal memory 121 may include one or more random access memories (RAM) and one or more non-volatile memories (NVM).
  • RAM random access memories
  • NVM non-volatile memories
  • Random access memory may include static random-access memory (SRAM), dynamic random-access memory (DRAM), synchronous dynamic random-access memory (SDRAM), double data rate synchronous dynamic random-access memory (DDR SDRAM, for example, the fifth generation DDR SDRAM is generally referred to as DDR5 SDRAM), etc.
  • SRAM static random-access memory
  • DRAM dynamic random-access memory
  • SDRAM synchronous dynamic random-access memory
  • DDR SDRAM double data rate synchronous dynamic random-access memory
  • DDR5 SDRAM double data rate synchronous dynamic random-access memory
  • Non-volatile memory can include disk storage devices and flash memory.
  • Flash memory can be divided according to the operating principle to include NOR FLASH, NAND FLASH, 3D NAND FLASH, etc.
  • the storage unit potential level it can include single-level storage cells (single-level cell, SLC), multi-level storage cells (multi-level cell, MLC), third-level storage unit (triple-level cell, TLC), fourth-level storage unit (quad-level cell, QLC), etc., which can include universal flash storage (English: universal flash storage, UFS) according to storage specifications. , embedded multi media card (embedded multi media Card, eMMC), etc.
  • the random access memory can be directly read and written by the processor 110, can be used to store executable programs (such as machine instructions) of the operating system or other running programs, and can also be used to store user and application data, etc.
  • the non-volatile memory can also store executable programs and user and application program data, etc., and can be loaded into the random access memory in advance for direct reading and writing by the processor 110.
  • the external memory interface 120 can be used to connect an external non-volatile memory to expand the storage capacity of the electronic device.
  • the external non-volatile memory communicates with the processor 110 through the external memory interface 120 to implement the data storage function. For example, save music, video and other files in external non-volatile memory.
  • the electronic device can implement audio functions through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone interface 170D, and the application processor. Such as music playback, recording, etc.
  • the audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. Audio module 170 may also be used to encode and decode audio signals. In some embodiments, the audio module 170 may be provided in the processor 110 , or some functional modules of the audio module 170 may be provided in the processor 110 .
  • Speaker 170A also called “speaker” is used to convert audio electrical signals into sound signals.
  • the electronic device can listen to music through speaker 170A, or listen to hands-free calls.
  • Receiver 170B also called “earpiece” is used to convert audio electrical signals into sound signals.
  • the electronic device answers a call or a voice message, the voice can be heard by bringing the receiver 170B close to the human ear.
  • Microphone 170C also called “microphone” or “microphone” is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak close to the microphone 170C with the human mouth and input the sound signal to the microphone 170C.
  • the electronic device may be provided with at least one microphone 170C. In other embodiments, the electronic device may be provided with two microphones 170C, which in addition to collecting sound signals, may also implement a noise reduction function. In other embodiments, the electronic device can also be equipped with three, four or more microphones 170C to collect sound signals, reduce noise, identify sound sources, and implement directional recording functions, etc.
  • the headphone interface 170D is used to connect wired headphones.
  • the headphone interface 170D may be a USB interface 130, or may be a 3.5mm open mobile terminal platform (OMTP) standard interface, or a Cellular Telecommunications Industry Association of the USA (CTIA) standard interface.
  • OMTP open mobile terminal platform
  • CTIA Cellular Telecommunications Industry Association of the USA
  • the pressure sensor 180A is used to sense pressure signals and can convert the pressure signals into electrical signals.
  • pressure sensor 180A may be disposed on display screen 194 .
  • pressure sensors 180A such as resistive pressure sensors, inductive pressure sensors, Capacitive pressure sensor, etc.
  • a capacitive pressure sensor may include at least two parallel plates of conductive material. When a force is applied to pressure sensor 180A, the capacitance between the electrodes changes. Electronics determine the intensity of the pressure based on changes in capacitance.
  • the electronic device detects the strength of the touch operation according to the pressure sensor 180A. The electronic device may also calculate the touched position based on the detection signal of the pressure sensor 180A.
  • touch operations acting on the same touch location but with different touch operation intensities may correspond to different operation instructions. For example: when a touch operation with a touch operation intensity less than the first pressure threshold is applied to the short message application icon, an instruction to view the short message is executed. When a touch operation with a touch operation intensity greater than or equal to the first pressure threshold is applied to the short message application icon, an instruction to create a new short message is executed.
  • the gyro sensor 180B can be used to determine the motion posture of the electronic device. In some embodiments, the angular velocity of the electronic device about three axes (ie, x, y, and z axes) may be determined by gyro sensor 180B.
  • the gyro sensor 180B can be used for image stabilization. For example, when the shutter is pressed, the gyro sensor 180B detects the angle at which the electronic device shakes, and calculates the distance that the lens module needs to compensate based on the angle, so that the lens can offset the shake of the electronic device through reverse movement to achieve anti-shake.
  • the gyro sensor 180B can also be used for navigation and somatosensory game scenes.
  • the air pressure sensor 180C is used to measure air pressure.
  • the electronic device calculates the altitude through the air pressure value measured by the air pressure sensor 180C to assist positioning and navigation.
  • Magnetic sensor 180D includes a Hall sensor.
  • the electronic device can use the magnetic sensor 180D to detect the opening and closing of the flip holster.
  • the electronic device may detect opening and closing of the flip according to the magnetic sensor 180D. Then, based on the detected opening and closing status of the leather case or the opening and closing status of the flip cover, features such as automatic unlocking of the flip cover are set.
  • the acceleration sensor 180E can detect the magnitude of the acceleration of the electronic device in all directions (generally three axes). When the electronic device is stationary, it can detect the magnitude and direction of gravity. It can also be used to identify the posture of the electronic device and is applied to applications such as horizontal and vertical screen switching and pedometers.
  • Distance sensor 180F for measuring distance.
  • Electronic devices can measure distance via infrared or laser. In some embodiments, when shooting a scene, the electronic device can utilize the distance sensor 180F to measure distance to achieve fast focusing.
  • Proximity light sensor 180G may include, for example, a light emitting diode (LED) and a light detector, such as a photodiode.
  • the light emitting diode may be an infrared light emitting diode.
  • Electronic devices emit infrared light through light-emitting diodes.
  • Electronic devices use photodiodes to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the electronic device. When insufficient reflected light is detected, the electronic device may determine that there is no object near the electronic device.
  • Electronic devices can use the proximity light sensor 180G to detect when the user holds the electronic device close to the ear and talk, so that the screen can be automatically turned off to save power.
  • the proximity light sensor 180G can also be used in holster mode, and pocket mode automatically unlocks and locks the screen.
  • the ambient light sensor 180L is used to sense ambient light brightness.
  • the electronic device can adaptively adjust the brightness of the display screen 194 based on perceived ambient light brightness.
  • the ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures.
  • the ambient light sensor 180L can also cooperate with the proximity light sensor 180G to detect whether the electronic device is in the pocket to prevent accidental touching.
  • Fingerprint sensor 180H is used to collect fingerprints. Electronic devices can use the collected fingerprint characteristics to unlock fingerprints, access application locks, take photos with fingerprints, answer incoming calls with fingerprints, etc.
  • Temperature sensor 180J is used to detect temperature.
  • the electronic device uses the temperature detected by the temperature sensor 180J to execute the temperature processing strategy. For example, when the temperature reported by the temperature sensor 180J exceeds a threshold, the electronic device reduces the performance of a processor located near the temperature sensor 180J in order to reduce power consumption and implement thermal protection. In other embodiments, when the temperature is lower than another threshold, the electronic device heats the battery 142 to prevent the low temperature from causing abnormal shutdown of the electronic device. In some other embodiments, when the temperature is lower than another threshold, the electronic device performs boosting on the output voltage of the battery 142 to avoid abnormal shutdown caused by low temperature.
  • Touch sensor 180K also known as "touch device”.
  • the touch sensor 180K can be disposed on the display screen 194.
  • the touch sensor 180K and the display screen 194 form a touch screen, which is also called a "touch screen”.
  • the touch sensor 180K is used to detect a touch operation on or near the touch sensor 180K.
  • the touch sensor can pass the detected touch operation to the application processor to determine the touch event type.
  • Visual output related to the touch operation may be provided through display screen 194 .
  • the touch sensor 180K may also be disposed on the surface of the electronic device at a location different from that of the display screen 194 .
  • Bone conduction sensor 180M can acquire vibration signals.
  • the bone conduction sensor 180M can acquire the vibration signal of the vibrating bone mass of the human body's vocal part.
  • the bone conduction sensor 180M can also contact the human body's pulse and receive blood pressure beating signals.
  • the bone conduction sensor 180M can also be provided in an earphone and combined into a bone conduction earphone.
  • the audio module 170 can analyze the voice signal based on the vibration signal of the vocal vibrating bone obtained by the bone conduction sensor 180M to implement the voice function.
  • the application processor can analyze the heart rate information based on the blood pressure beating signal acquired by the bone conduction sensor 180M to implement the heart rate detection function.
  • the buttons 190 include a power button, a volume button, etc.
  • Key 190 may be a mechanical key. It can also be a touch button.
  • the electronic device can receive key input and generate key signal input related to user settings and function control of the electronic device.
  • the motor 191 can generate vibration prompts.
  • the motor 191 can be used for vibration prompts for incoming calls and can also be used for touch vibration feedback.
  • touch operations for different applications can correspond to different vibration feedback effects.
  • the motor 191 can also respond to different vibration feedback effects for touch operations in different areas of the display screen 194 .
  • Different application scenarios such as time reminders, receiving information, alarm clocks, games, etc.
  • the touch vibration feedback effect can also be customized.
  • the indicator 192 may be an indicator light, which may be used to indicate the charging status, power changes, messages, missed calls, notifications, etc.
  • the SIM card interface 195 is used to connect a SIM card.
  • the SIM card can be inserted into the SIM card interface 195 or pulled out from the SIM card interface 195 to realize contact and separation from the electronic device.
  • the electronic device can support 1 or N SIM card interfaces, where N is a positive integer greater than 1.
  • SIM card interface 195 can support Nano SIM card, Micro SIM card, SIM card, etc. Multiple cards can be inserted into the same SIM card interface 195 at the same time. The types of the plurality of cards may be the same or different.
  • the SIM card interface 195 is also compatible with different types of SIM cards.
  • the SIM card interface 195 is also compatible with external memory cards.
  • Electronic devices interact with the network through SIM cards to implement functions such as calls and data communications.
  • the electronic device uses an eSIM, that is, an embedded SIM card.
  • the eSIM card can be embedded in the electronic device and cannot be separated from the electronic device.
  • FIG. 16A and FIG. 16B are another exemplary schematic diagram of the software architecture of the electronic device provided by the embodiment of the present application.
  • Software systems of electronic devices can adopt layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture.
  • the embodiment of the present invention takes the Android system with a layered architecture as an example to illustrate the software structure of the electronic device.
  • the layered architecture divides the software into several layers, and each layer has clear roles and division of labor.
  • the layers communicate through software interfaces.
  • the Android system is divided into four layers, from top to bottom: application layer, application framework layer, Android runtime and system libraries, and kernel layer.
  • the application layer can include a series of application packages.
  • the application package can include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message, etc.
  • the application framework layer provides application programming interface (API) and programming framework for the applications in the application layer.
  • API application programming interface
  • the application framework layer includes some predefined functions.
  • the application framework layer may include window management services, activity management services, content providers, view systems, phone managers, resource managers, notification managers, etc.
  • the window management service is used to manage window programs and can determine the window order of windows of different applications.
  • the unified rendering process may obtain the window Z order of windows of different applications from the window management server, and then determine the rendering nodes corresponding to the views obscured by the interfaces of other applications in different rendering trees.
  • the window management service may determine the completely obscured area and send the completely obscured area to the UI thread of the application.
  • Content providers are used to store and retrieve data and make this data accessible to applications.
  • Said data can include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.
  • the view system includes visual controls, such as controls that display text, controls that display pictures, etc.
  • a view system can be used to build applications.
  • the display interface can be composed of one or more views.
  • a display interface including a text message notification icon may include a view for displaying text and a view for displaying pictures.
  • the view system also includes an occlusion module, which is used to determine views that are completely occluded, that is, views that are occluded by other views of the same application, and views that are completely occluded by the interfaces of other applications.
  • an occlusion module which is used to determine views that are completely occluded, that is, views that are occluded by other views of the same application, and views that are completely occluded by the interfaces of other applications.
  • Telephone managers are used to provide communication functions of electronic devices. For example, call status management (including connected, hung up, etc.).
  • the resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.
  • the notification manager allows applications to display notification information in the status bar, which can be used to convey notification-type messages and can automatically disappear after a short stay without user interaction.
  • the notification manager is used to notify download completion, message reminders, etc.
  • the notification manager can also be notifications that appear in the status bar at the top of the system in the form of charts or scroll bar text, such as notifications for applications running in the background, or notifications that appear on the screen in the form of conversation windows. For example, text information is prompted in the status bar, a beep sounds, the electronic device vibrates, the indicator light flashes, etc.
  • Android Runtime includes core libraries and virtual machines. Android runtime is responsible for the scheduling and management of the Android system.
  • the core library contains two parts: one is the functional functions that need to be called by the Java language, and the other is the core library of Android.
  • the application layer and application framework layer run in virtual machines.
  • the virtual machine executes the java files of the application layer and application framework layer as binary files.
  • the virtual machine is used to perform object life cycle management, stack management, thread management, security and exception management, and Garbage collection and other functions.
  • System libraries can include multiple functional modules. For example: browser engine (webkit), surface compositor, hardware synthesis strategy module, media libraries (Media Libraries), 3D graphics processing library (for example: OpenGL ES), 2D graphics engine (for example: SGL), etc.
  • browser engine webkit
  • surface compositor hardware synthesis strategy module
  • media libraries Media Libraries
  • 3D graphics processing library for example: OpenGL ES
  • 2D graphics engine for example: SGL
  • the media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc.
  • the media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.
  • the 3D graphics processing library is used to implement 3D graphics drawing, image rendering, composition, and layer processing.
  • 2D Graphics Engine is a drawing engine for 2D drawing.
  • the kernel layer is the layer between hardware and software.
  • the kernel layer includes display driver, camera driver, audio driver, sensor driver, etc.
  • the view system may further include a unified rendering process, which is used to receive rendering trees of different applications, merge the rendering trees into a target rendering tree, and then generate a bitmap based on the target rendering tree.
  • a unified rendering process which is used to receive rendering trees of different applications, merge the rendering trees into a target rendering tree, and then generate a bitmap based on the target rendering tree.
  • the system library may not include a surface compositor, or it may be generated when The surface compositor does not perform layer compositing during the interface.
  • the display subsystem can be considered as a functional module including part of the display driver and part of the display.
  • the term “when” may be interpreted to mean “if" or “after” or “in response to determining" or “in response to detecting" depending on the context.
  • the phrase “when determining" or “if (stated condition or event) is detected” may be interpreted to mean “if it is determined" or “in response to determining" or “on detecting (stated condition or event)” or “in response to detecting (stated condition or event)”.
  • the computer program product includes one or more computer instructions.
  • the computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable device.
  • the computer instructions may be stored in or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted over a wired connection from a website, computer, server, or data center (such as coaxial cable, optical fiber, digital subscriber line) or wireless (such as infrared, wireless, microwave, etc.) means to transmit to another website, computer, server or data center.
  • the computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center integrated with one or more available media.
  • the available media may be magnetic media (eg, floppy disk, hard disk, magnetic tape), optical media (eg, DVD), or semiconductor media (eg, solid state drive), etc.
  • the program can be stored in a computer-readable storage medium.
  • the aforementioned storage media include: ROM, random access memory (RAM), magnetic disks, optical disks and other media that can store program codes.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Software Systems (AREA)
  • Human Computer Interaction (AREA)
  • Image Generation (AREA)

Abstract

Embodiments of the present invention provide an interface generation method and an electronic device, which relate to the technical field of electronics. The interface generation method provided in the present invention can first identify a completely shielded control, and then reduce overdrawing in an interface generation process by means of deleting or not generating rendering nodes corresponding to the control, thereby improving an energy efficiency ratio of interface generation. Herein, the completely shielded view can be a view that is shielded by other views in the same application or a view that is shielded by an interface of other applications.

Description

界面生成方法及电子设备Interface generation method and electronic device
本申请要求于2022年09月22日提交中国国家知识产权局、申请号为202211155586.X、申请名称为“界面生成方法及电子设备”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on September 22, 2022, with application number 202211155586.X and application name "Interface Generation Method and Electronic Device", the entire content of which is incorporated by reference in in this application.
技术领域Technical field
本申请涉及电子技术领域，尤其涉及界面生成方法及电子设备。The present application relates to the field of electronic technology, and in particular to interface generation methods and electronic devices.
背景技术Background technique
随着电子技术的发展，电子设备的屏幕的分辨率和刷新率越来越高，其中，屏幕的分辨率影响一帧界面中包含的像素，刷新率影响生成一帧界面的时间。With the development of electronic technology, the resolution and refresh rate of screens of electronic devices are getting higher and higher. The resolution of the screen affects the pixels included in one frame of the interface, and the refresh rate affects the time to generate one frame of the interface.
界面的生成过程中，视图(view)的渲染是基于画家算法(painter’s algorithm)实现的，即按照从底层到上层的顺序依次渲染界面中的视图。按照这种渲染顺序，电子设备可以实现视图或图层(layer)的透明度混合、阴影、线性叠加等效果。During the interface generation process, the rendering of views is based on the painter’s algorithm, that is, the views in the interface are rendered sequentially from the bottom layer to the upper layer. According to this rendering order, electronic devices can achieve effects such as transparency mixing, shadowing, and linear overlay of views or layers.
但是，这种渲染顺序会导致有部分视图被完全遮蔽。在渲染过程中，被完全遮蔽的视图对应的区域内的像素会被多次覆盖，其中，最后一次渲染得到的像素会覆盖之前渲染得到的像素。即，应用程序会调用GPU来生成界面上不显示的像素，进而增加界面卡顿的概率，降低界面生成的能效比。However, this rendering order results in parts of the view being completely obscured. During the rendering process, the pixels in the area corresponding to the completely occluded view will be covered multiple times. Among them, the pixels obtained by the last rendering will overwrite the pixels obtained by the previous rendering. That is, the application will call the GPU to generate pixels that are not displayed on the interface, thereby increasing the probability of interface freezes and reducing the energy efficiency of interface generation.
发明内容Contents of the invention
本申请实施例提供了界面生成方法及电子设备，涉及电子技术领域。本申请提供的界面生成方法首先可以识别被完全遮蔽的控件，通过删除或者不生成该控件对应的渲染节点，降低界面生成过程中的过度绘制，提高界面生成的能效比。Embodiments of the present application provide interface generation methods and electronic devices, and relate to the field of electronic technology. The interface generation method provided by this application can first identify a completely obscured control, and by deleting or not generating the rendering node corresponding to the control, it can reduce overdrawing during the interface generation process and improve the energy efficiency of interface generation.
第一方面，本申请实施例提供了一种界面生成方法，应用于包括屏幕的电子设备，该方法包括：确定被第一应用内的其他视图完全遮蔽的视图包括第一视图；生成第一渲染树，该第一渲染树不包括第一渲染节点，该第一渲染节点为该第一视图对应的渲染节点，渲染节点保存有对应的视图的绘制指令；基于该第一渲染树生成第一界面，该第一界面包括该第一应用的界面。In a first aspect, embodiments of the present application provide an interface generation method, applied to an electronic device including a screen. The method includes: determining that views that are completely obscured by other views in the first application include the first view; generating a first rendering tree, the first rendering tree does not include a first rendering node, the first rendering node is a rendering node corresponding to the first view, and the rendering node stores the drawing instructions of the corresponding view; the first interface is generated based on the first rendering tree , the first interface includes the interface of the first application.
在上述实施例中，电子设备先确定被完全遮蔽的视图，由于被完全遮蔽的视图与显示内容无关，故，可以生成不包含该被完全准备的视图对应的渲染节点的渲染树，进而基于该渲染树生成界面，由于该渲染树不包含该被完全准备的视图对应的渲染节点，在生成该界面的过程中，不会绘制该视图。In the above embodiment, the electronic device first determines the completely occluded view. Since the completely occluded view has nothing to do with the display content, a rendering tree that does not include the rendering node corresponding to the fully prepared view can be generated, and then based on the The rendering tree generates the interface. Since the rendering tree does not contain the rendering node corresponding to the fully prepared view, the view will not be drawn during the process of generating the interface.
结合第一方面的一些实施例，在一些实施例中，该确定被第一应用内的其他视图完全遮蔽的视图包括第一视图，具体包括：在遍历该第一应用的视图的过程中，确定第二视图的位置为第二位置；在遍历到该第一视图时，确定该第一视图的位置为第一位置，基于该第二位置和该第一位置确定该第一视图为被该第一应用内的其他视图完全遮蔽的视图。With reference to some embodiments of the first aspect, in some embodiments, determining that the view that is completely occluded by other views in the first application includes the first view, specifically includes: in the process of traversing the views of the first application, determining The position of the second view is the second position; when traversing to the first view, the position of the first view is determined to be the first position, and the first view is determined to be the first position based on the second position and the first position. A view that is completely obscured by other views within an application.
在上述实施例中，应用程序遍历视图的过程中，可以确定不同视图的位置，进而确定视图之间的互相遮蔽关系，再进而可以确定任意一个视图是否被完全遮蔽。In the above embodiment, during the process of traversing the views, the application program can determine the positions of different views, and then determine the mutual shielding relationship between the views, and further determine whether any one view is completely shielded.
结合第一方面的一些实施例，在一些实施例中，在该基于该第二位置和该第一位置确定该第一视图为被完全遮蔽的视图后，该方法还包括：不保存第一视图的绘制指令至该第一渲染树中；跳过该第一视图的子视图以继续遍历，该第一渲染树不包括该第一视图的子视图对应的渲染节点。In conjunction with some embodiments of the first aspect, in some embodiments, after determining that the first view is a completely occluded view based on the second position and the first position, the method further includes: not saving the first view draw instructions to the first rendering tree; skip the subviews of the first view to continue traversing, and the first rendering tree does not include rendering nodes corresponding to the subviews of the first view.
在上述实施例中，在应用程序遍历视图的过程中，当先确定第二视图的位置后确定第一视图的位置后，可以跳过第一视图的绘制方法调用，不将第一视图和第一视图的子视图的绘制指令保存在渲染树中，就可以直接生成不包含无用的绘制指令的渲染树。In the above embodiment, when the application program traverses the views, after first determining the position of the second view and then determining the position of the first view, the drawing method call of the first view can be skipped, and the first view and the first view can be skipped. The drawing instructions of the view's subviews are stored in the rendering tree, and a rendering tree that does not contain useless drawing instructions can be directly generated.
结合第一方面的一些实施例，在一些实施例中，该确定被第一应用内的其他视图完全遮蔽的视图包括第一视图，具体包括：在遍历该第一应用的视图的过程中，确定该第一视图的位置为第一位置后，确定第二视图的位置为第二位置；基于该第一位置和该第二位置确定该第一视图被该第二视图完全遮蔽，记录该第一视图的标识至第一参数中。In combination with some embodiments of the first aspect, in some embodiments, the view determined to be completely obscured by other views within the first application includes the first view, specifically including: in the process of traversing the views of the first application, after determining that the position of the first view is the first position, determining that the position of the second view is the second position; based on the first position and the second position, determining that the first view is completely obscured by the second view, and recording the identifier of the first view in the first parameter.
在上述实施例中，在应用程序遍历视图的过程中，当先确定第一视图的位置后确定第二视图的位置后， 由于此时已经遍历过第一视图，则需要保存第一视图的标识，该第一视图的标识用于在后续步骤中确定被完全遮蔽视图对应的渲染节点。In the above embodiment, when the application traverses the views, after first determining the position of the first view and then determining the position of the second view, Since the first view has been traversed at this time, the identifier of the first view needs to be saved. The identifier of the first view is used to determine the rendering node corresponding to the completely blocked view in subsequent steps.
结合第一方面的一些实施例，在一些实施例中，该生成第一渲染树，具体包括：遍历该第一应用的视图后，生成第二渲染树，该第二渲染树包括第一渲染节点；基于该第一参数删除该第二渲染树中的第一渲染节点得到该第一渲染树。In conjunction with some embodiments of the first aspect, in some embodiments, generating a first rendering tree specifically includes: after traversing the views of the first application, generating a second rendering tree, where the second rendering tree includes a first rendering node. ;Delete the first rendering node in the second rendering tree based on the first parameter to obtain the first rendering tree.
在上述实施例中，第一参数记录有完全被遮蔽视图的标识，则应用程序可以基于第一参数删除被被完全遮蔽视图对应的渲染节点。In the above embodiment, the first parameter records the identifier of the completely occluded view, and the application program can delete the rendering node corresponding to the completely occluded view based on the first parameter.
结合第一方面的一些实施例，在一些实施例中，该确定被第一应用内的其他视图完全遮蔽的视图包括第一视图，具体包括：遍历该第一应用的视图后，生成第二渲染树，该第二渲染树包括该第一渲染节点和第二渲染节点；基于该第一渲染节点的渲染属性和该第二渲染节点的渲染属性确定该第一视图被第二视图完全遮蔽，该第二视图与该第二渲染节点对应。In combination with some embodiments of the first aspect, in some embodiments, determining that the view completely obscured by other views in the first application includes the first view specifically includes: after traversing the views of the first application, generating a second rendering tree, the second rendering tree including the first rendering node and the second rendering node; determining, based on rendering attributes of the first rendering node and the rendering attributes of the second rendering node, that the first view is completely obscured by the second view, and the second view corresponds to the second rendering node.
在上述实施例中，应用程序也可以在生成渲染树后，才去确定被完全遮蔽的视图对应的渲染节点。In the above embodiment, the application can also determine the rendering node corresponding to the completely occluded view after generating the rendering tree.
结合第一方面的一些实施例，在一些实施例中，该生成第一渲染树，具体包括：删除该第二渲染树中的第二渲染节点得到该第一渲染树。With reference to some embodiments of the first aspect, in some embodiments, generating the first rendering tree specifically includes: deleting the second rendering node in the second rendering tree to obtain the first rendering tree.
在上述实施例中，电子设备可以删除第一渲染树中的被完全遮蔽的视图对应的渲染节点。In the above embodiment, the electronic device can delete the rendering node corresponding to the completely occluded view in the first rendering tree.
结合第一方面的一些实施例，在一些实施例中，该方法还包括：合并该第一渲染树与第二渲染树得到目标渲染树，该第二渲染树是第二应用生成的，该第二渲染树用于生成该第二应用的界面，该第一界面还包括该第二应用的界面，该目标渲染树不包括该第三渲染节点，该第三渲染节点与第三视图对应，该第三视图为该第一应用的视图，该第三视图被该第二应用的界面完全遮蔽；该基于该第一渲染树生成第一界面，具体包括：基于该目标渲染树生成该第一界面。In conjunction with some embodiments of the first aspect, in some embodiments, the method further includes: merging the first rendering tree and the second rendering tree to obtain a target rendering tree, the second rendering tree is generated by the second application, and the third rendering tree is generated by the second application. The second rendering tree is used to generate the interface of the second application, the first interface also includes the interface of the second application, the target rendering tree does not include the third rendering node, the third rendering node corresponds to the third view, and the The third view is the view of the first application, and the third view is completely blocked by the interface of the second application; generating the first interface based on the first rendering tree specifically includes: generating the first interface based on the target rendering tree .
在上述实施例中，还可以删除被其他应用程序的界面完全遮蔽的视图，进而生成一个对应于多个应用程序的界面的渲染树，并且该渲染树不包括被完全遮蔽的视图的渲染节点。In the above embodiment, views that are completely occluded by the interfaces of other applications can also be deleted, thereby generating a rendering tree corresponding to the interfaces of multiple applications, and the rendering tree does not include rendering nodes of the views that are completely occluded.
结合第一方面的一些实施例，在一些实施例中，在该基于该目标渲染树生成第一界面前，该方法还包括：确定被该第二应用的界面完全遮蔽的视图对应的渲染节点包括该第三渲染节点；该合并该第一渲染树与第二渲染树得到目标渲染树，具体包括：在合并该第一渲染树与该第二渲染树得到目标渲染树的过程中，从该第一渲染树或该目标渲染树中删除该第三渲染节点。In conjunction with some embodiments of the first aspect, in some embodiments, before generating the first interface based on the target rendering tree, the method further includes: determining that the rendering node corresponding to the view completely occluded by the interface of the second application includes: The third rendering node; merging the first rendering tree and the second rendering tree to obtain the target rendering tree specifically includes: in the process of merging the first rendering tree and the second rendering tree to obtain the target rendering tree, from the third The third rendering node is deleted from a rendering tree or the target rendering tree.
在上述实施例中，可以在合并不同应用程序生成的渲染树的过程中，删除被完全遮蔽的视图对应的渲染节点。In the above embodiment, in the process of merging rendering trees generated by different applications, the rendering nodes corresponding to the completely occluded views can be deleted.
结合第一方面的一些实施例，在一些实施例中，在该基于该目标渲染树生成第一界面前，该方法还包括：确定被该第二应用的界面完全遮蔽的视图对应的渲染节点包括该第三渲染节点；该合并该第一渲染树与第二渲染树得到目标渲染树，具体包括：合并该第一渲染树与该第二渲染树得到第三渲染树；从该第三渲染树中删除该第三渲染节点得到该目标渲染树。In conjunction with some embodiments of the first aspect, in some embodiments, before generating the first interface based on the target rendering tree, the method further includes: determining that the rendering node corresponding to the view completely occluded by the interface of the second application includes: The third rendering node; merging the first rendering tree and the second rendering tree to obtain a target rendering tree, specifically including: merging the first rendering tree and the second rendering tree to obtain a third rendering tree; Delete the third rendering node to obtain the target rendering tree.
在上述实施例中，也可以在合并渲染树得到目标渲染树后，删除被完全遮蔽视图对应的渲染节点。In the above embodiment, after merging the rendering trees to obtain the target rendering tree, the rendering node corresponding to the completely occluded view can also be deleted.
结合第一方面的一些实施例，在一些实施例中，该确定被完全遮蔽的视图对应的渲染节点包括该第三渲染节点，具体包括：基于该第一应用的窗口Z序、该第二应用的窗口Z序、该第一渲染树和该第二渲染树确定被该第二应用的界面完全遮蔽的第一应用的视图对应的渲染节点包括该第三渲染节点；或者，确定完全遮蔽区域，并基于该第一渲染树和该完全遮蔽区域确定被该第二应用的界面完全遮蔽的第一应用的视图对应的渲染节点包括该第三渲染节点，该完全遮蔽区域为该第二应用的界面遮蔽第一应用的界面的区域。With reference to some embodiments of the first aspect, in some embodiments, the rendering node corresponding to the completely occluded view includes the third rendering node, specifically including: the window Z order based on the first application, the second application The window Z order, the first rendering tree and the second rendering tree determine that the rendering node corresponding to the view of the first application that is completely obscured by the interface of the second application includes the third rendering node; or, determine the completely obscured area, And based on the first rendering tree and the completely blocked area, it is determined that the rendering node corresponding to the view of the first application that is completely blocked by the interface of the second application includes the third rendering node, and the completely blocked area is the interface of the second application. Covers the area of the first application's interface.
在上述实施例中，可以通过不同应用程序的窗口Z序确定不同应用程序界面的显示遮蔽关系，进而确定被完全遮蔽的视图。In the above embodiment, the display shielding relationship of different application program interfaces can be determined through the window Z order of different applications, and then the completely blocked view can be determined.
结合第一方面的一些实施例，在一些实施例中，该方法还包括：该第一界面还包括第二应用的界面，该第二应用的界面完全遮蔽该第一应用的界面的第一区域，确定该第一区域；基于该第一区域确定被该第二应用的界面完全遮蔽的视图包括第四视图，该第一渲染树不包括第四渲染节点，该第四渲染节点与该第四视图对应。In conjunction with some embodiments of the first aspect, in some embodiments, the method further includes: the first interface further includes an interface of a second application, and the interface of the second application completely blocks the first area of the interface of the first application. , determine the first area; determine based on the first area that the view completely blocked by the interface of the second application includes a fourth view, the first rendering tree does not include a fourth rendering node, and the fourth rendering node is related to the fourth View correspondence.
在上述实施例中，电子设备可以先确定两个应用中遮蔽的区域，进而使得应用程序在生成渲染树的过程中，不生成该区域中的视图对应的渲染节点。In the above embodiment, the electronic device can first determine the occluded area in the two applications, so that the application program does not generate rendering nodes corresponding to the views in the area during the process of generating the rendering tree.
结合第一方面的一些实施例，在一些实施例中，该方法还包括：确定被部分遮蔽的视图包括第五视图；在第五渲染节点中的绘制指令列表中增加裁剪指令，该裁剪指令用于指示不渲染该第五视图中被部分遮蔽的内容，该第五渲染节点与该第五视图对应。 Combined with some embodiments of the first aspect, in some embodiments, the method further includes: determining that the partially occluded view includes a fifth view; adding a cropping instruction to the drawing instruction list in the fifth rendering node, where the cropping instruction is used In order to indicate not to render the partially occluded content in the fifth view, the fifth rendering node corresponds to the fifth view.
在上述实施例中，对于被部分遮蔽的视图来说，可以通过增加剪裁指令，在生成位图的过程中，不渲染被部分遮蔽的内容，可以降低GPU的负载。In the above embodiment, for a partially occluded view, a clipping instruction can be added so that the partially occluded content is not rendered during the bitmap generation process, thereby reducing the load on the GPU.
第二方面，本申请实施例提供了一种界面生成方法，应用于包括屏幕的电子设备，该电子设备上运行有第一应用、第二应用和第一进程，该方法包括：该第一应用向第一进程发送第一渲染树，该第一渲染树用于生成该第一应用的界面；该第二应用向该第一进程发送第二渲染树，该第二渲染树用于生成该第二应用的界面；该第一进程基于该第一渲染树和该第二渲染树生成目标渲染树，该目标渲染树用于生成第一界面，该第一界面包括该第一应用的界面和该第二应用的界面；该第一进程确定第一区域后，基于该第一区域删除第一渲染节点，该第二应用的界面完全遮蔽该第一应用的界面的该第一区域，该目标渲染树不包括该第一渲染节点，该第一渲染节点为该第一渲染树中的渲染节点，该第一渲染节点对应的第一视图位于该第一区域中，渲染节点保存有对应的视图的绘制指令；该第一进程基于该目标渲染树生成该第一界面。In a second aspect, an embodiment of the present application provides an interface generation method, which is applied to an electronic device including a screen, on which a first application, a second application and a first process are running, the method comprising: the first application sends a first rendering tree to the first process, the first rendering tree is used to generate an interface of the first application; the second application sends a second rendering tree to the first process, the second rendering tree is used to generate an interface of the second application; the first process generates a target rendering tree based on the first rendering tree and the second rendering tree, the target rendering tree is used to generate a first interface, the first interface includes an interface of the first application and an interface of the second application; after the first process determines a first area, the first rendering node is deleted based on the first area, the interface of the second application completely covers the first area of the interface of the first application, the target rendering tree does not include the first rendering node, the first rendering node is a rendering node in the first rendering tree, a first view corresponding to the first rendering node is located in the first area, and the rendering node stores a drawing instruction of the corresponding view; the first process generates the first interface based on the target rendering tree.
在上述实施例中，第一进程可以接收不同应用程序生成的渲染树，并确定跨应用程序的被完全遮蔽的视图，进而删除被完全遮蔽的视图对应的渲染节点，降低生成位图过程中的过度绘制，降低GPU的负载。In the above embodiment, the first process can receive rendering trees generated by different applications, determine fully occluded views across applications, and then delete the rendering nodes corresponding to the completely occluded views to reduce the time spent in the process of generating bitmaps. Overdraw, reducing GPU load.
结合第二方面的一些实施例，在一些实施例中，在该第一应用程序发送第一渲染树前，该方法还包括：该第一应用通过遍历视图确定第三渲染树，该第三渲染树用于生成该第一应用的界面；该第一应用确定被该第一应用程序的其他视图完全遮蔽的视图包括第二视图；该第一应用删除该第三渲染树中的第二渲染节点得到该第一渲染树，该第二渲染节点与该第二视图对应。Combined with some embodiments of the second aspect, in some embodiments, before the first application sends the first rendering tree, the method further includes: the first application determines a third rendering tree by traversing the view, and the third rendering The tree is used to generate the interface of the first application; the first application determines that views that are completely obscured by other views of the first application include the second view; the first application deletes the second rendering node in the third rendering tree The first rendering tree is obtained, and the second rendering node corresponds to the second view.
在上述实施例中，应用程序在向第一进程发送渲染树前，可以先删除被完全遮蔽的视图对应的渲染节点，进而降低后续在生成位图过程中GPU的负载。In the above embodiment, before sending the rendering tree to the first process, the application can first delete the rendering node corresponding to the completely occluded view, thereby reducing the load on the GPU during the subsequent bitmap generation process.
结合第二方面的一些实施例，在一些实施例中，该方法还包括：该第一进程删除第二渲染节点，该第二渲染节点为该第一渲染树中的渲染节点，该第二渲染节点与该第二视图对应，该第二视图被该第一应用的其他视图完全遮蔽。In conjunction with some embodiments of the second aspect, in some embodiments, the method further includes: the first process deletes a second rendering node, the second rendering node is a rendering node in the first rendering tree, and the second rendering Nodes correspond to the second view, which is completely obscured by other views of the first application.
在上述实施例中，第一进程可以删除被同一应用程序内的其他视图完全遮蔽的视图的渲染节点，进而降低后续在生成位图过程中GPU的负载。In the above embodiment, the first process can delete the rendering node of a view that is completely occluded by other views in the same application, thereby reducing the load on the GPU during subsequent bitmap generation.
第三方面，本申请实施例提供了一种界面生成方法，应用于包括屏幕的电子设备，该方法包括：该电子设备在该屏幕显示第一界面，在该第一界面中，该第一应用程序的界面全屏显示第一内容，该第二应用程序的界面在该第一应用程序的界面上悬浮显示第二内容，该第二应用程序的界面在该第一界面中位于第一区域；在显示该第一界面的过程中，该电子设备生成第一渲染树，该第一渲染树用于生成该第一应用的界面；该电子设备在该屏幕显示第二界面，在该第二界面中，该第一应用程序的界面全屏显示该第一内容，该第二应用程序的界面在该第一应用程序上悬浮显示该第二内容，该第二应用程序在该第一界面中位于第二区域，该第一区域与该区域不同；在显示该第二界面的过程中，该电子设备生成第二渲染树，该第二渲染树用于生成该第一应用的界面，该第一渲染树与该第二渲染树不同。In a third aspect, embodiments of the present application provide an interface generation method, applied to an electronic device including a screen. The method includes: the electronic device displays a first interface on the screen, and in the first interface, the first application The interface of the program displays the first content in full screen, the interface of the second application program displays the second content suspended on the interface of the first application program, and the interface of the second application program is located in the first area in the first interface; in During the process of displaying the first interface, the electronic device generates a first rendering tree, which is used to generate the interface of the first application; the electronic device displays a second interface on the screen, and in the second interface , the interface of the first application program displays the first content in full screen, the interface of the second application program displays the second content in a floating manner on the first application program, and the second application program is located in the second part of the first interface. area, the first area is different from the area; in the process of displaying the second interface, the electronic device generates a second rendering tree, the second rendering tree is used to generate the interface of the first application, the first rendering tree Different from this second render tree.
在上述实施例中，第二应用的界面悬浮显示在第一应用的界面上，在第二应用的界面的位置发生变化前后且第一应用的界面没有发生变化的情况下，由于第一应用程序删除了被完全遮蔽视图对应的渲染节点，所以第一应用程序生成的渲染树不一样，不包含冗余的信息，进而降低后续在生成位图过程中GPU的负载。In the above embodiment, the interface of the second application is suspended and displayed on the interface of the first application. Before and after the position of the interface of the second application changes and the interface of the first application does not change, because the first application The rendering node corresponding to the completely occluded view is deleted, so the rendering tree generated by the first application is different and does not contain redundant information, thereby reducing the load on the GPU during subsequent bitmap generation.
结合第三方面的一些实施例，在一些实施例中，该第一渲染树用于生成该第一应用的界面中不包括该第一区域的界面；该第二渲染树用于生成该第一应用的界面中不包括该第二区域的界面。In conjunction with some embodiments of the third aspect, in some embodiments, the first rendering tree is used to generate an interface of the first application that does not include the first area; the second rendering tree is used to generate the first The interface of the application does not include the interface of the second area.
在上述实施例中，由于第一渲染树和第二渲染树中不包括被完全遮蔽视图对应的渲染节点，可以降低后续在生成位图过程中GPU的负载。In the above embodiment, since the first rendering tree and the second rendering tree do not include rendering nodes corresponding to the completely occluded views, the load of the GPU in the subsequent process of generating bitmaps can be reduced.
结合第三方面的一些实施例，在一些实施例中，该电子设备生成第一渲染树，具体包括：该电子设备基于该第一区域确定该第二应用的界面遮蔽的该第一应用的视图包括第一视图；该电子设备生成第三渲染树；该电子设备删除该第三渲染树中的第一渲染节点得到该第一渲染树，该第一渲染节点与该第一视图对应，渲染节点保存有对应的视图的绘制指令。With reference to some embodiments of the third aspect, in some embodiments, the electronic device generates a first rendering tree, specifically including: the electronic device determines, based on the first area, the view of the first application that is obscured by the interface of the second application. Including a first view; the electronic device generates a third rendering tree; the electronic device deletes the first rendering node in the third rendering tree to obtain the first rendering tree, the first rendering node corresponds to the first view, and the rendering node Save the drawing instructions of the corresponding view.
在上述实施例中，电子设备可以删除被第二应用程序的界面遮蔽的第一应用程序的视图，进而降低后续在生成位图过程中GPU的负载，提高界面生成的能效比。In the above embodiment, the electronic device can delete the view of the first application that is blocked by the interface of the second application, thereby reducing the load of the GPU in the subsequent bitmap generation process and improving the energy efficiency of interface generation.
第四方面，本申请实施例提供了一种电子设备，该电子设备包括：一个或多个处理器、存储器和屏幕；该存储器与该一个或多个处理器耦合，该存储器用于存储计算机程序代码，该计算机程序代码包括计算机 指令，该一个或多个处理器调用该计算机指令以使得该电子设备执行：确定被第一应用内的其他视图完全遮蔽的视图包括第一视图；生成第一渲染树，该第一渲染树不包括第一渲染节点，该第一渲染节点为该第一视图对应的渲染节点，渲染节点保存有对应的视图的绘制指令；基于该第一渲染树生成第一界面，该第一界面包括该第一应用的界面。In a fourth aspect, embodiments of the present application provide an electronic device. The electronic device includes: one or more processors, a memory, and a screen; the memory is coupled to the one or more processors, and the memory is used to store a computer program. code, the computer program code includes computer Instructions, the one or more processors invoke the computer instructions to cause the electronic device to perform: determine that a view that is completely occluded by other views in the first application includes the first view; generate a first rendering tree, the first rendering tree does not It includes a first rendering node, the first rendering node is a rendering node corresponding to the first view, and the rendering node stores a drawing instruction of the corresponding view; a first interface is generated based on the first rendering tree, and the first interface includes the third An application interface.
结合第四方面的一些实施例，在一些实施例中，该一个或多个处理器，具体用于调用该计算机指令以使得该电子设备执行：在遍历该第一应用的视图的过程中，确定第二视图的位置为第二位置；在遍历到该第一视图时，确定该第一视图的位置为第一位置，基于该第二位置和该第一位置确定该第一视图为被该第一应用内的其他视图完全遮蔽的视图。In conjunction with some embodiments of the fourth aspect, in some embodiments, the one or more processors are specifically configured to invoke the computer instructions to cause the electronic device to execute: in the process of traversing the view of the first application, determine The position of the second view is the second position; when traversing to the first view, the position of the first view is determined to be the first position, and the first view is determined to be the first position based on the second position and the first position. A view that is completely obscured by other views within an application.
结合第四方面的一些实施例，在一些实施例中，该一个或多个处理器，还用于调用该计算机指令以使得该电子设备执行：不保存第一视图的绘制指令至该第一渲染树中；跳过该第一视图的子视图以继续遍历，该第一渲染树不包括该第一视图的子视图对应的渲染节点。In conjunction with some embodiments of the fourth aspect, in some embodiments, the one or more processors are also configured to call the computer instructions to cause the electronic device to execute: not saving the drawing instructions of the first view to the first rendering. In the tree; skip the subviews of the first view to continue traversing, and the first rendering tree does not include rendering nodes corresponding to the subviews of the first view.
结合第四方面的一些实施例，在一些实施例中，该一个或多个处理器，具体用于调用该计算机指令以使得该电子设备执行：在遍历该第一应用的视图的过程中，确定该第一视图的位置为第一位置后，确定第二视图的位置为第二位置；基于该第一位置和该第二位置确定该第一视图被该第二视图完全遮蔽，记录该第一视图的标识至第一参数中。In conjunction with some embodiments of the fourth aspect, in some embodiments, the one or more processors are specifically configured to invoke the computer instructions to cause the electronic device to execute: in the process of traversing the view of the first application, determine After the position of the first view is the first position, the position of the second view is determined to be the second position; based on the first position and the second position, it is determined that the first view is completely blocked by the second view, and the first view is recorded. The ID of the view is passed to the first parameter.
结合第四方面的一些实施例，在一些实施例中，该一个或多个处理器，具体用于调用该计算机指令以使得该电子设备执行：遍历该第一应用的视图后，生成第二渲染树，该第二渲染树包括第一渲染节点；基于该第一参数删除该第二渲染树中的第一渲染节点得到该第一渲染树。In conjunction with some embodiments of the fourth aspect, in some embodiments, the one or more processors are specifically configured to call the computer instructions to cause the electronic device to execute: after traversing the views of the first application, generate a second rendering tree, the second rendering tree includes a first rendering node; the first rendering node in the second rendering tree is deleted based on the first parameter to obtain the first rendering tree.
结合第四方面的一些实施例，在一些实施例中，该一个或多个处理器，具体用于调用该计算机指令以使得该电子设备执行：遍历该第一应用的视图后，生成第二渲染树，该第二渲染树包括该第一渲染节点和第二渲染节点；基于该第一渲染节点的渲染属性和该第二渲染节点的渲染属性确定该第一视图被第二视图完全遮蔽，该第二视图与该第二渲染节点对应。In conjunction with some embodiments of the fourth aspect, in some embodiments, the one or more processors are specifically configured to call the computer instructions to cause the electronic device to execute: after traversing the views of the first application, generate a second rendering tree, the second rendering tree includes the first rendering node and the second rendering node; it is determined based on the rendering attributes of the first rendering node and the rendering attributes of the second rendering node that the first view is completely obscured by the second view, the The second view corresponds to the second rendering node.
结合第四方面的一些实施例，在一些实施例中，该一个或多个处理器，具体用于调用该计算机指令以使得该电子设备执行：删除该第二渲染树中的第二渲染节点得到该第一渲染树。In conjunction with some embodiments of the fourth aspect, in some embodiments, the one or more processors are specifically configured to call the computer instructions to cause the electronic device to execute: delete the second rendering node in the second rendering tree to obtain The first render tree.
结合第四方面的一些实施例，在一些实施例中，该一个或多个处理器，还用于调用该计算机指令以使得该电子设备执行：合并该第一渲染树与第二渲染树得到目标渲染树，该第二渲染树是第二应用生成的，该第二渲染树用于生成该第二应用的界面，该第一界面还包括该第二应用的界面，该目标渲染树不包括该第三渲染节点，该第三渲染节点与第三视图对应，该第三视图为该第一应用的视图，该第三视图被该第二应用的界面完全遮蔽；该基于该第一渲染树生成第一界面，具体包括：基于该目标渲染树生成该第一界面。In combination with some embodiments of the fourth aspect, in some embodiments, the one or more processors are further used to call the computer instructions to cause the electronic device to execute: merging the first rendering tree and the second rendering tree to obtain a target rendering tree, the second rendering tree is generated by the second application, the second rendering tree is used to generate an interface of the second application, the first interface also includes the interface of the second application, the target rendering tree does not include the third rendering node, the third rendering node corresponds to a third view, the third view is a view of the first application, and the third view is completely obscured by the interface of the second application; generating the first interface based on the first rendering tree specifically includes: generating the first interface based on the target rendering tree.
结合第四方面的一些实施例，在一些实施例中，该一个或多个处理器，还用于调用该计算机指令以使得该电子设备执行：确定被该第二应用的界面完全遮蔽的视图对应的渲染节点包括该第三渲染节点；该合并该第一渲染树与第二渲染树得到目标渲染树，具体包括：在合并该第一渲染树与该第二渲染树得到目标渲染树的过程中，从该第一渲染树或该目标渲染树中删除该第三渲染节点。In conjunction with some embodiments of the fourth aspect, in some embodiments, the one or more processors are also configured to call the computer instructions to cause the electronic device to execute: determine the corresponding view that is completely obscured by the interface of the second application. The rendering nodes include the third rendering node; merging the first rendering tree and the second rendering tree to obtain the target rendering tree specifically includes: in the process of merging the first rendering tree and the second rendering tree to obtain the target rendering tree. , delete the third rendering node from the first rendering tree or the target rendering tree.
结合第四方面的一些实施例，在一些实施例中，该一个或多个处理器，还用于调用该计算机指令以使得该电子设备执行：确定被该第二应用的界面完全遮蔽的视图对应的渲染节点包括该第三渲染节点；该合并该第一渲染树与第二渲染树得到目标渲染树，具体包括：合并该第一渲染树与该第二渲染树得到第三渲染树；从该第三渲染树中删除该第三渲染节点得到该目标渲染树。In conjunction with some embodiments of the fourth aspect, in some embodiments, the one or more processors are also configured to call the computer instructions to cause the electronic device to execute: determine the corresponding view that is completely obscured by the interface of the second application. The rendering nodes include the third rendering node; merging the first rendering tree and the second rendering tree to obtain a target rendering tree specifically includes: merging the first rendering tree and the second rendering tree to obtain a third rendering tree; The third rendering node is deleted from the third rendering tree to obtain the target rendering tree.
结合第四方面的一些实施例，在一些实施例中，该一个或多个处理器，具体用于调用该计算机指令以使得该电子设备执行：基于该第一应用的窗口Z序、该第二应用的窗口Z序、该第一渲染树和该第二渲染树确定被该第二应用的界面完全遮蔽的第一应用的视图对应的渲染节点包括该第三渲染节点；或者，确定完全遮蔽区域，并基于该第一渲染树和该完全遮蔽区域确定被该第二应用的界面完全遮蔽的第一应用的视图对应的渲染节点包括该第三渲染节点，该完全遮蔽区域为该第二应用的界面遮蔽第一应用的界面的区域。In conjunction with some embodiments of the fourth aspect, in some embodiments, the one or more processors are specifically configured to call the computer instructions to cause the electronic device to execute: the window sequence based on the first application, the second The window Z order of the application, the first rendering tree and the second rendering tree determine that the rendering node corresponding to the view of the first application that is completely obscured by the interface of the second application includes the third rendering node; or, determine the completely obscured area. , and determine based on the first rendering tree and the complete occlusion area that the rendering node corresponding to the view of the first application that is completely occluded by the interface of the second application includes the third rendering node, and the completely occlusion area is of the second application. The interface covers an area of the interface of the first application.
结合第四方面的一些实施例，在一些实施例中，该一个或多个处理器，还用于调用该计算机指令以使得该电子设备执行：该第一界面还包括第二应用的界面，该第二应用的界面完全遮蔽该第一应用的界面的第一区域，确定该第一区域；基于该第一区域确定被该第二应用的界面完全遮蔽的视图包括第四视图，该第一渲染树不包括第四渲染节点，该第四渲染节点与该第四视图对应。With reference to some embodiments of the fourth aspect, in some embodiments, the one or more processors are also used to call the computer instructions to cause the electronic device to execute: the first interface also includes an interface of the second application, the The interface of the second application completely blocks the first area of the interface of the first application, and the first area is determined; it is determined based on the first area that the view completely blocked by the interface of the second application includes a fourth view, and the first rendering The tree does not include a fourth rendering node corresponding to the fourth view.
结合第四方面的一些实施例，在一些实施例中，该一个或多个处理器，还用于调用该计算机指令以使得该电子设备执行：确定被部分遮蔽的视图包括第五视图；在第五渲染节点中的绘制指令列表中增加裁剪指令，该裁剪指令用于指示不渲染该第五视图中被部分遮蔽的内容，该第五渲染节点与该第五视图对应。 In conjunction with some embodiments of the fourth aspect, in some embodiments, the one or more processors are further configured to call the computer instructions to cause the electronic device to perform: determining that the partially occluded view includes a fifth view; A cropping instruction is added to the drawing instruction list in the fifth rendering node. The cropping instruction is used to instruct not to render the partially obscured content in the fifth view. The fifth rendering node corresponds to the fifth view.
第五方面，本申请实施例提供了一种电子设备，该电子设备包括：一个或多个处理器、存储器和屏幕；该存储器与该一个或多个处理器耦合，该存储器用于存储计算机程序代码，该计算机程序代码包括计算机指令，该一个或多个处理器调用该计算机指令以使得该电子设备执行：该第一应用向该第一进程发送第一渲染树，该第一渲染树用于生成该第一应用的界面；该第二应用向该第一进程发送第二渲染树，该第二渲染树用于生成该第二应用的界面；该第一进程基于该第一渲染树和该第二渲染树生成目标渲染树，该目标渲染树用于生成第一界面，该第一界面包括该第一应用的界面和该第二应用的界面；该第一进程确定第一区域后，基于该第一区域删除第一渲染节点，该第二应用的界面完全遮蔽该第一应用的界面的该第一区域，该目标渲染树不包括该第一渲染节点，该第一渲染节点为该第一渲染树中的渲染节点，该第一渲染节点对应的第一视图位于该第一区域中，渲染节点保存有对应的视图的绘制指令；该第一进程基于该目标渲染树生成该第一界面。In a fifth aspect, embodiments of the present application provide an electronic device. The electronic device includes: one or more processors, a memory, and a screen; the memory is coupled to the one or more processors, and the memory is used to store computer programs. Code, the computer program code includes computer instructions, the one or more processors invoke the computer instructions to cause the electronic device to execute: the first application sends a first rendering tree to the first process, the first rendering tree is used to Generate the interface of the first application; the second application sends a second rendering tree to the first process, and the second rendering tree is used to generate the interface of the second application; the first process is based on the first rendering tree and the The second rendering tree generates a target rendering tree, and the target rendering tree is used to generate a first interface. The first interface includes the interface of the first application and the interface of the second application; after the first process determines the first area, based on The first region deletes the first rendering node, the interface of the second application completely blocks the first region of the interface of the first application, the target rendering tree does not include the first rendering node, and the first rendering node is the first rendering node. A rendering node in the rendering tree, the first view corresponding to the first rendering node is located in the first area, and the rendering node stores the drawing instructions of the corresponding view; the first process generates the first interface based on the target rendering tree .
结合第五方面的一些实施例，在一些实施例中，该一个或多个处理器，还用于调用该计算机指令以使得该电子设备执行：该第一应用通过遍历视图确定第三渲染树，该第三渲染树用于生成该第一应用的界面；该第一应用确定被该第一应用程序的其他视图完全遮蔽的视图包括第二视图；该第一应用删除该第三渲染树中的第二渲染节点得到该第一渲染树，该第二渲染节点与该第二视图对应。In conjunction with some embodiments of the fifth aspect, in some embodiments, the one or more processors are also configured to call the computer instructions to cause the electronic device to execute: the first application determines the third rendering tree by traversing the view, The third rendering tree is used to generate the interface of the first application; the first application determines that views that are completely obscured by other views of the first application include the second view; the first application deletes the views in the third rendering tree. The second rendering node obtains the first rendering tree, and the second rendering node corresponds to the second view.
结合第五方面的一些实施例，在一些实施例中，该一个或多个处理器，还用于调用该计算机指令以使得该电子设备执行：该第一进程删除第二渲染节点，该第二渲染节点为该第一渲染树中的渲染节点，该第二渲染节点与该第二视图对应，该第二视图被该第一应用的其他视图完全遮蔽。In conjunction with some embodiments of the fifth aspect, in some embodiments, the one or more processors are also used to call the computer instructions to cause the electronic device to execute: the first process deletes the second rendering node, the second The rendering node is a rendering node in the first rendering tree, the second rendering node corresponds to the second view, and the second view is completely obscured by other views of the first application.
第六方面，本申请实施例提供了一种电子设备，该电子设备包括：一个或多个处理器、存储器和屏幕；该存储器与该一个或多个处理器耦合，该存储器用于存储计算机程序代码，该计算机程序代码包括计算机指令，该一个或多个处理器调用该计算机指令以使得该电子设备执行：该电子设备在该屏幕显示第一界面，在该第一界面中，该第一应用程序的界面全屏显示第一内容，该第二应用程序的界面在该第一应用程序的界面上悬浮显示第二内容，该第二应用程序的界面在该第一界面中位于第一区域；在显示该第一界面的过程中，该电子设备生成第一渲染树，该第一渲染树用于生成该第一应用的界面；该电子设备在该屏幕显示第二界面，在该第二界面中，该第一应用程序的界面全屏显示该第一内容，该第二应用程序的界面在该第一应用程序上悬浮显示该第二内容，该第二应用程序在该第一界面中位于第二区域，该第一区域与该区域不同；在显示该第二界面的过程中，该电子设备生成第二渲染树，该第二渲染树用于生成该第一应用的界面，该第一渲染树与该第二渲染树不同。In a sixth aspect, embodiments of the present application provide an electronic device. The electronic device includes: one or more processors, a memory, and a screen; the memory is coupled to the one or more processors, and the memory is used to store computer programs. Code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to cause the electronic device to execute: the electronic device displays a first interface on the screen, and in the first interface, the first application The interface of the program displays the first content in full screen, the interface of the second application program displays the second content suspended on the interface of the first application program, and the interface of the second application program is located in the first area in the first interface; in During the process of displaying the first interface, the electronic device generates a first rendering tree, which is used to generate the interface of the first application; the electronic device displays a second interface on the screen, and in the second interface , the interface of the first application program displays the first content in full screen, the interface of the second application program displays the second content in a floating manner on the first application program, and the second application program is located in the second part of the first interface. area, the first area is different from the area; in the process of displaying the second interface, the electronic device generates a second rendering tree, the second rendering tree is used to generate the interface of the first application, the first rendering tree Different from this second render tree.
结合第六方面的一些实施例，在一些实施例中，该第一渲染树用于生成该第一应用的界面中不包括该第一区域的界面；该第二渲染树用于生成该第一应用的界面中不包括该第二区域的界面。Combined with some embodiments of the sixth aspect, in some embodiments, the first rendering tree is used to generate an interface of the first application that does not include the first area; the second rendering tree is used to generate the first The interface of the application does not include the interface of the second area.
结合第六方面的一些实施例，在一些实施例中，该一个或多个处理器，具体用于调用该计算机指令以使得该电子设备执行：该电子设备基于该第一区域确定该第二应用的界面遮蔽的该第一应用的视图包括第一视图；该电子设备生成第三渲染树；该电子设备删除该第三渲染树中的第一渲染节点得到该第一渲染树，该第一渲染节点与该第一视图对应，渲染节点保存有对应的视图的绘制指令。In conjunction with some embodiments of the sixth aspect, in some embodiments, the one or more processors are specifically configured to invoke the computer instructions to cause the electronic device to execute: the electronic device determines the second application based on the first area The view of the first application blocked by the interface includes the first view; the electronic device generates a third rendering tree; the electronic device deletes the first rendering node in the third rendering tree to obtain the first rendering tree, and the first rendering The node corresponds to the first view, and the rendering node stores the drawing instructions of the corresponding view.
第七方面，本申请实施例提供了一种芯片***，该芯片***应用于电子设备，该芯片***包括一个或多个处理器，该处理器用于调用计算机指令以使得该电子设备执行如第一方面、第二方面、第三方面、第一方面中任一可能的实现方式、第二方面中任一可能的方式以及第三方面中任一可能的实现方式描述的方法。In a seventh aspect, embodiments of the present application provide a chip system, which is applied to an electronic device. The chip system includes one or more processors, and the processor is used to call computer instructions to cause the electronic device to execute the first step. The method described in the aspects, the second aspect, the third aspect, any possible implementation method in the first aspect, any possible implementation method in the second aspect, and any possible implementation method in the third aspect.
第八方面，本申请实施例提供一种包含指令的计算机程序产品，当上述计算机程序产品在电子设备上运行时，使得上述电子设备执行如第一方面、第二方面、第三方面、第一方面中任一可能的实现方式、第二方面中任一可能的方式以及第三方面中任一可能的实现方式描述的方法。In an eighth aspect, embodiments of the present application provide a computer program product containing instructions. When the computer program product is run on an electronic device, the electronic device causes the electronic device to execute the first aspect, the second aspect, the third aspect, and the first aspect. Any possible implementation method in the aspect, any possible implementation method in the second aspect, and any possible implementation method in the third aspect.
第九方面，本申请实施例提供一种计算机可读存储介质，包括指令，当上述指令在电子设备上运行时，使得上述电子设备执行如第一方面、第二方面、第三方面、第一方面中任一可能的实现方式、第二方面中任一可能的方式以及第三方面中任一可能的实现方式描述的方法。 In a ninth aspect, embodiments of the present application provide a computer-readable storage medium, including instructions. When the instructions are run on an electronic device, the electronic device causes the electronic device to execute the first aspect, the second aspect, the third aspect, and the first aspect. Any possible implementation method in the aspect, any possible implementation method in the second aspect, and any possible implementation method in the third aspect.
可以理解地，上述第四方面提供的电子设备、第五方面提供的电子设备、第六方面提供的电子设备、第七方面提供的芯片***、第八方面提供的计算机程序产品和第九方面提供的计算机存储介质均用于执行本申请实施例所提供的方法。因此，其所能达到的有益效果可参考对应方法中的有益效果，此处不再赘述。It can be understood that the electronic device provided by the fourth aspect, the electronic device provided by the fifth aspect, the electronic device provided by the sixth aspect, the chip system provided by the seventh aspect, the computer program product provided by the eighth aspect, and the ninth aspect provide computer storage media are used to execute the methods provided by the embodiments of this application. Therefore, the beneficial effects it can achieve can be referred to the beneficial effects in the corresponding methods, and will not be described again here.
附图说明BRIEF DESCRIPTION OF THE DRAWINGS
图1为本申请实施例提供的应用程序生成位图的一个示例性示意图。Figure 1 is an exemplary schematic diagram of a bitmap generated by an application program according to an embodiment of the present application.
图2为本申请实施例提供的渲染线程基于渲染树生成位图的一个示例性示意图。FIG. 2 is an exemplary schematic diagram of a rendering thread generating a bitmap based on a rendering tree provided by an embodiment of the present application.
图3A-图3C为本申请实施例提供的视图互相遮蔽情况下界面的一个示例性示意图。3A-3C are an exemplary schematic diagram of the interface provided by the embodiment of the present application when views are mutually obscured.
图4为本申请实施例提供的视图互相遮蔽情况下界面的另一个示例性示意图。FIG. 4 is another exemplary schematic diagram of an interface in a case where views obscure each other provided in an embodiment of the present application.
图5为本申请实施例提供的电子设备生成界面的一个示例性示意图。FIG. 5 is an exemplary schematic diagram of an electronic device generation interface provided in an embodiment of the present application.
图6为本申请实施例提供的界面生成方法的流程的一个示例性示意图。Figure 6 is an exemplary schematic diagram of the process of the interface generation method provided by the embodiment of the present application.
图7为本申请实施例提供的确定被完全遮蔽视图的一个示例性示意图。FIG. 7 is an exemplary schematic diagram for determining a completely occluded view provided by an embodiment of the present application.
图8为本申请实施例提供的确定被完全遮蔽视图的一个示例性示意图。FIG. 8 is an exemplary schematic diagram for determining a completely occluded view provided by an embodiment of the present application.
图9为本申请实施例提供的确定被完全遮蔽的视图的另一个示例性示意图。FIG. 9 is another exemplary schematic diagram of determining a completely blocked view provided by an embodiment of the present application.
图10A-图10D本申请实施例提供的渲染线程或统一渲染进程删除渲染节点的一个示例性示意图。Figures 10A-10D provide an exemplary schematic diagram of a rendering thread or unified rendering process deleting a rendering node according to the embodiment of the present application.
图11为本申请实施例提供的界面生成方法的数据流动的一个示例性示意图。Figure 11 is an exemplary schematic diagram of the data flow of the interface generation method provided by the embodiment of the present application.
图12为本申请实施例提供的界面生成方法的数据流动的另一个示例性示意图。Figure 12 is another exemplary schematic diagram of the data flow of the interface generation method provided by the embodiment of the present application.
图13和图14为本申请实施例提供的界面生成方法的数据流动的另一个示例性示意图。Figures 13 and 14 are another exemplary schematic diagram of the data flow of the interface generation method provided by the embodiment of the present application.
图15为本申请实施例提供的电子设备的硬件结构的一个示例性示意图。FIG. 15 is an exemplary schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present application.
图16A和图16B为本申请实施例提供的电子设备的软件架构的另一个示例性示意图。16A and 16B are another exemplary schematic diagram of the software architecture of the electronic device provided in an embodiment of the present application.
具体实施方式Detailed ways
本申请以下实施例中所使用的术语只是为了描述特定实施例的目的，而并非旨在作为对本申请的限制。如在本申请的说明书和所附权利要求书中所使用的那样，单数表达形式“一个”、“一种”、“该”、“上述”、“该”和“这一”旨在也包括复数表达形式，除非其上下文中明确地有相反指示。还应当理解，本申请中使用的术语“和/或”是指并包含一个或多个所列出项目的任何或所有可能组合。The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in the specification and appended claims of this application, the singular expressions "a", "an", "the", "above", "the" and "the" are intended to also include Plural expressions unless the context clearly indicates otherwise. It will also be understood that the term "and/or" as used in this application refers to and includes any and all possible combinations of one or more of the listed items.
以下，术语“第一”、“第二”仅用于描述目的，而不能理解为暗示或暗示相对重要性或者隐含指明所指示的技术特征的数量。由此，限定有“第一”、“第二”的特征可以明示或者隐含地包括一个或者更多个该特征，在本申请实施例的描述中，除非另有说明，“多个”的含义是两个或两个以上。Hereinafter, the terms “first” and “second” are used for descriptive purposes only and shall not be understood as implying or implying relative importance or implicitly specifying the quantity of indicated technical features. Therefore, the features defined as “first” and “second” may explicitly or implicitly include one or more of the features. In the description of the embodiments of this application, unless otherwise specified, “plurality” The meaning is two or more.
本申请以下实施例中的术语“用户界面(user interface，UI)”，是应用程序或操作***与用户之间进行交互和信息交换的介质接口，它实现信息的内部形式与用户可以接受形式之间的转换。用户界面是通过java、可扩展标记语言(extensible markup language，XML)等特定计算机语言编写的源代码，界面源代码在电子设备上经过解析，渲染，最终呈现为用户可以识别的内容。用户界面常用的表现形式是图形用户界面(graphic user interface，GUI)，是指采用图形方式显示的与计算机操作相关的用户界面。它可以是在电子设备的显示屏中显示的文本、图标、按钮、菜单、选项卡、文本框、对话框、状态栏、导航栏、Widget等可视的界面元素。The term "user interface (UI)" in the following embodiments of this application is a media interface for interaction and information exchange between an application or operating system and a user. It realizes the difference between the internal form of information and the form acceptable to the user. conversion between. The user interface is source code written in specific computer languages such as Java and extensible markup language (XML). The interface source code is parsed and rendered on the electronic device, and finally presented as content that can be recognized by the user. The commonly used form of user interface is graphical user interface (GUI), which refers to a user interface related to computer operations that is displayed graphically. It can be text, icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, widgets and other visual interface elements displayed on the display screen of an electronic device.
为了便于理解，下面先对本申请实施例涉及的相关术语及相关概念进行介绍。本发明的实施方式部分使用的术语仅用于对本发明的具体实施例进行解释，而非旨在限定本发明。In order to facilitate understanding, the relevant terms and concepts involved in the embodiments of this application are first introduced below. The terms used in the embodiments of the present invention are only used to explain specific embodiments of the present invention and are not intended to limit the present invention.
界面作为应用程序与用户之间的交互和信息交互的介质接口，在每一次垂直同步信号(vsync)到来时，电子设备需要为前台的应用程序生成该应用程序的界面。其中，垂直同步信号的频率与电子设备的屏幕的刷新率有关，例如垂直同步信号的频率与电子设备的屏幕的刷新率相同。The interface serves as a media interface for interaction and information exchange between the application and the user. Every time a vertical synchronization signal (vsync) arrives, the electronic device needs to generate the interface of the application for the foreground application. The frequency of the vertical synchronization signal is related to the refresh rate of the screen of the electronic device. For example, the frequency of the vertical synchronization signal is the same as the refresh rate of the screen of the electronic device.
即每次电子设备刷新屏幕上显示的内容前，都需要为前台应用生成该应用程序的界面，以在屏幕刷新时向用户展现应用程序的新生成的界面。That is, each time the electronic device refreshes the content displayed on the screen, it needs to generate the interface of the application for the foreground application, so as to present the newly generated interface of the application to the user when the screen is refreshed.
其中，电子设备生成应用程序的界面需要应用程序自己渲染生成位图(bitmap)，将自己的位图传递给表面合成器(SurfaceFlinger)。即，应用程序作为生产者执行渲染生成位图，将该位图存入表面合成器提供的缓冲队列(BufferQueue)中；表面合成器作为消费者不断的从BufferQueue获取应用程序生成的位图。其中，位图位于应用程序生成的surface上，该surface会被填入BufferQueue中。Among them, the electronic device generates an application interface that requires the application to render and generate a bitmap (bitmap) by itself, and pass its own bitmap to the surface compositor (SurfaceFlinger). That is, the application performs rendering as a producer to generate a bitmap, and stores the bitmap in the buffer queue (BufferQueue) provided by the surface compositor; the surface compositor, as a consumer, continuously obtains the bitmap generated by the application from the BufferQueue. Among them, the bitmap is located on the surface generated by the application, and the surface will be filled into the BufferQueue.
在表面合成器获得可见的应用程序的位图后，表面合成器与硬件合成策略模块(Hardware Composer, HWC)确定位图作为图层(layer)的图层合成的方式。After the surface compositor obtains the visible application bitmap, the surface compositor interacts with the hardware composition strategy module (Hardware Composer, HWC) determines the way bitmaps are layered as layers.
在表面合成器和/或硬件合成策略模块执行位图合成后，由表面合成器和/或硬件合成策略模块将合成后的位图(位图在表面合成器上也可以称为图层)填入帧缓冲(Frame Buffer)中传递给显示子***(Display Subsystem,DSS)，DSS在拿到合成后的位图可以将该合成后的位图显示到屏幕上。该帧缓冲可以是在屏缓冲(on-screen buffer)。After the surface compositor and/or hardware composition strategy module performs bitmap composition, the surface compositor and/or hardware composition strategy module fills in the synthesized bitmap (the bitmap can also be called a layer on the surface compositor). It is passed into the frame buffer (Frame Buffer) and passed to the display subsystem (Display Subsystem, DSS). After DSS gets the synthesized bitmap, it can display the synthesized bitmap on the screen. The frame buffer can be an on-screen buffer.
其中，应用程序生成位图的过程如下图1所示。Among them, the process of generating bitmaps by the application is shown in Figure 1 below.
图1为本申请实施例提供的应用程序生成位图的一个示例性示意图。Figure 1 is an exemplary schematic diagram of a bitmap generated by an application program according to an embodiment of the present application.
如图1所示，应用程序在接收到垂直同步信号后，开始生成位图，具体的步骤可以分为三步，分别为①、②、③。As shown in Figure 1, after receiving the vertical synchronization signal, the application begins to generate a bitmap. The specific steps can be divided into three steps, namely ①, ②, and ③.
①让视图结构(view hierarchy)失效，主线程(UI线程，UI Thread)通过测量方法调用(measure())、布局方法调用(layout())、绘制方法调用(draw()，或绘制录制方法调用)遍历该应用程序的视图(view)(后文不作特殊说明，通过测量方法调用、布局方法调用和绘制方法调用生成渲染输的操作统称为遍历)，确定并保存每个视图的绘制操作，并将视图和该视图涉及的绘制操作结构体(Draw Operation Struct，如DrawOP)或绘制操作(绘制指令)录制到渲染树的渲染节点(RenderNode)的绘制指令列表(displaylist)中。① Invalidate the view hierarchy (view hierarchy), and the main thread (UI thread, UI Thread) calls the measurement method (measure()), the layout method (layout()), the drawing method (draw()), or the drawing recording method Call) to traverse the views of the application (no special explanation will be given later, the operations of generating rendering output through measurement method calls, layout method calls and drawing method calls are collectively referred to as traversal), determine and save the drawing operation of each view, And record the view and the drawing operation structure (Draw Operation Struct, such as DrawOP) or drawing operations (drawing instructions) involved in the view into the drawing instruction list (displaylist) of the rendering node (RenderNode) of the rendering tree.
其中，视图是构成应用程序界面的基本元素，界面上的一个控件可以对应于一个或多个视图。Among them, views are the basic elements that constitute the application interface. A control on the interface can correspond to one or more views.
可选的，在本申请一些实施方式中，在绘制方法调用内，应用程序的UI线程还会读取视图上承载的内容至内存中。例如，图片视图(imageview)承载的图片，文本视图(imageview)承载的文本。或者，在绘制方法调用内，应用程序的UI线程确定读取视图上承载的内容至内存中的操作，并录制到绘制指令列表中。Optionally, in some implementations of the present application, within the drawing method call, the UI thread of the application will also read the content carried on the view into the memory. For example, the picture view (imageview) carries the picture, and the text view (imageview) carries the text. Alternatively, within a draw method call, the application's UI thread determines the operation of reading the content carried on the view into memory and recording it into the draw instruction list.
其中，绘制操作结构体为一个数据结构体，用于绘制图形，例如绘制线条、绘制矩形、绘制文本等。绘制操作结构体在渲染节点会被转换为图像处理库的API调用，如OpenGLES的接口调用。例如DrawLineOp是一个数据结构体，数据结构体里面包含有绘制的数据如线的长度、宽度等信息。Among them, the drawing operation structure is a data structure used to draw graphics, such as drawing lines, drawing rectangles, drawing text, etc. The drawing operation structure will be converted into API calls of the image processing library at the rendering node, such as OpenGLES interface calls. For example, DrawLineOp is a data structure, which contains drawing data such as line length, width and other information.
其中，绘制指令列表可以是一个缓冲区，该缓冲区中记录有应用程序一帧界面所包括的所有绘制操作结构体或是所有绘制操作的标识，如地址、序号等。当应用程序有多个窗口、或者在不同的显示区域(display)上显示时，需要独立的生成多个渲染树，其中，会独立的生成多个对应不同窗口、显示区域的绘制指令列表。The drawing instruction list may be a buffer, which records all drawing operation structures or identifiers of all drawing operations included in one frame of the application program interface, such as addresses, serial numbers, etc. When an application has multiple windows or is displayed on different display areas, multiple rendering trees need to be generated independently, in which multiple drawing instruction lists corresponding to different windows and display areas will be generated independently.
其中，渲染树是UI线程生成的，用于生成应用程序界面的一个数据结构体，渲染树可以包括多个渲染节点，每个渲染节点包括渲染属性和绘制指令列表。渲染树记录有生成应用程序一帧界面的部分或所有信息。渲染树中的渲染节点包括绘制指令列表和渲染属性，其中，渲染属性用于决定该渲染节点要渲染的视图在表面(也可以称为表面上的画布)中的位置、大小、透明度等属性，绘制指令列表用于决定该渲染节点要渲染的视图的内容，如线条、矩形、圆形等。Among them, the rendering tree is generated by the UI thread and is used to generate a data structure of the application interface. The rendering tree can include multiple rendering nodes, and each rendering node includes rendering attributes and a drawing instruction list. The rendering tree records some or all of the information that generates a frame of the application's interface. The rendering node in the rendering tree includes a drawing instruction list and rendering attributes. The rendering attributes are used to determine the position, size, transparency and other attributes of the view to be rendered by the rendering node in the surface (which can also be called the canvas on the surface). The drawing instruction list is used to determine the content of the view to be rendered by the rendering node, such as lines, rectangles, circles, etc.
其中，应用程序需要显示的界面由多个视图嵌套组成，不同的视图之间具有父子关系，所以遍历视图生成的渲染树的渲染节点之间的父子关系与视图的父子关系相同。即视图之间的父子关系决定了不同渲染节点的之间的嵌套关系(后文中，渲染节点之间的嵌套关系也可以称为父子关系)，进而渲染线程在依据渲染树生成位图时可以正确渲染出应用程序的界面。一个视图可以对应一个或多个渲染节点，根视图(DecorView)对应根渲染节点(RootRenderNode)。其中，渲染节点中还包括有渲染属性(properties)，用与在渲染生成位图时确定被该渲染节点对应的view在surface上的位置、大小、透明度等。Among them, the interface that the application needs to display is composed of multiple nested views, and different views have a parent-child relationship, so the parent-child relationship between the rendering nodes of the rendering tree generated by traversing the view is the same as the parent-child relationship of the view. That is, the parent-child relationship between views determines the nested relationship between different rendering nodes (hereinafter, the nested relationship between rendering nodes can also be called parent-child relationship), and then the rendering thread generates bitmaps based on the rendering tree. The application interface can be rendered correctly. A view can correspond to one or more rendering nodes, and the root view (DecorView) corresponds to the root rendering node (RootRenderNode). Among them, the rendering node also includes rendering properties (properties), which are used to determine the position, size, transparency, etc. of the view corresponding to the rendering node on the surface when rendering and generating the bitmap.
可选地，在本申请一些实施方式中，主线程可以只对脏区域(也可以称为需要被重绘的区域)的视图执行遍历，生成差分渲染树。其中，差分渲染树在被传递/同步到渲染线程后，渲染线程可以差分渲染树和上一帧渲染使用的渲染树确定本帧界面渲染需要使用的渲染树。Optionally, in some implementations of the present application, the main thread may only perform traversal of views in dirty areas (which may also be referred to as areas that need to be redrawn) to generate a differential rendering tree. Among them, after the differential rendering tree is passed/synchronized to the rendering thread, the rendering thread can determine the rendering tree that needs to be used for interface rendering in this frame through the differential rendering tree and the rendering tree used for the previous frame rendering.
②UI线程将渲染树传递/同步给渲染线程(Render Thread)，其中，渲染树位于应用程序对应的进程的栈(stack)中，并且在物理地址上可以不是连续分布的。②The UI thread passes/synchronizes the rendering tree to the rendering thread (Render Thread). The rendering tree is located in the stack of the process corresponding to the application, and may not be continuously distributed on the physical address.
③渲染线程首先获取一个硬件画布(HardwareCanvas)，并在该硬件画布上执行渲染树中的绘制操作，进而生成位图。其中，该硬件画布位于该应用程序持有的surface中，该surface中承载有位图或者其他格式的用于保存图像信息的数据。③The rendering thread first obtains a hardware canvas (HardwareCanvas), and performs drawing operations in the rendering tree on the hardware canvas to generate a bitmap. Among them, the hardware canvas is located in a surface held by the application, and the surface carries bitmap or other format data used to save image information.
可以认为①为构建阶段，主要负责确定该应用程序中每个view的大小、位置、透明度等属性。例如，视图中的drawLine，构建中可以被封装成一个DrawLineOp，里面包含有绘制的数据如线的长度、宽度等，还可以包含有底层图形处理库的DrawLineOp对应的接口调用，用于在渲染阶段调用底层图形库生成位图。 It can be considered that ① is the construction phase, which is mainly responsible for determining the size, position, transparency and other attributes of each view in the application. For example, drawLine in the view can be encapsulated into a DrawLineOp during construction, which contains drawing data such as line length, width, etc., and can also include interface calls corresponding to the DrawLineOp of the underlying graphics processing library, which is used in the rendering phase. Call the underlying graphics library to generate bitmaps.
类似的，可以认为③为渲染阶段，主要负责遍历渲染树的渲染节点，并执行每个渲染节点的绘制操作，进而在硬件画布上生成位图，在该过程中，渲染线程通过调用底层图形处理库，如OpenGL ES库、Skia库等，进而调用GPU完成渲染以生成位图。Similarly, ③ can be considered as the rendering stage, which is mainly responsible for traversing the rendering nodes of the rendering tree and performing the drawing operation of each rendering node, thereby generating a bitmap on the hardware canvas. In this process, the rendering thread calls the underlying graphics processing Libraries, such as OpenGL ES library, Skia library, etc., then call the GPU to complete rendering to generate bitmaps.
值得说明的是，构建阶段需要占用CPU的计算资源，渲染阶段需要占用GPU的资源。It is worth mentioning that the construction phase requires the use of CPU computing resources, and the rendering phase requires the use of GPU resources.
值得说明的是，若不开启硬件加速，则应用程序通过UI线程完成构建阶段、渲染阶段的所有操作，并且不需要封装绘制操作为渲染节点，在遍历应用程序的视图后以及确定视图的绘制操作后，向表面合成器申请一块匿名共享内存，在该内存上直接调用底层图形库(如二维图像引擎)生成一个位图。It is worth mentioning that if hardware acceleration is not turned on, the application completes all operations in the construction phase and rendering phase through the UI thread, and there is no need to encapsulate the drawing operations as rendering nodes. After traversing the application's views and determining the drawing operations of the views, Finally, apply for an anonymous shared memory from the surface synthesizer, and directly call the underlying graphics library (such as a two-dimensional image engine) on this memory to generate a bitmap.
其中，在渲染线程基于渲染树生成位图的过程如图2所示。Among them, the process of generating bitmaps based on the rendering tree in the rendering thread is shown in Figure 2.
图2为本申请实施例提供的渲染线程基于渲染树生成位图的一个示例性示意图。FIG. 2 is an exemplary schematic diagram of a rendering thread generating a bitmap based on a rendering tree provided by an embodiment of the present application.
如图2所示，应用程序对视图执行遍历后，生成的渲染树包括根渲染节点、渲染节点1、渲染节点2和渲染节点3。其中，渲染节点1为根渲染节点的左子节点和渲染节点2为根渲染节点的右子节点，渲染节点3为渲染节点2的子节点。As shown in FIG2 , after the application traverses the view, the generated rendering tree includes a root rendering node, rendering node 1, rendering node 2, and rendering node 3. Among them, rendering node 1 is the left child node of the root rendering node, rendering node 2 is the right child node of the root rendering node, and rendering node 3 is a child node of rendering node 2.
其中，渲染节点1对应的渲染内容包括渲染位图1，渲染节点2对应的渲染内容包括渲染位图2。Among them, the rendering content corresponding to rendering node 1 includes rendering bitmap 1, and the rendering content corresponding to rendering node 2 includes rendering bitmap 2.
若渲染线程在生成位图的过程中遍历渲染树的顺序从前到后依次为：根渲染节点、渲染节点1、渲染节点2、渲染节点3。则，渲染线程先执行渲染节点1中的绘制操作、然后执行渲染节点2中的绘制操作，即渲染线程调用GPU在表面合成器提供的表面上首先渲染视图1、然后渲染视图2。由于视图2完全遮蔽视图1，在渲染线程渲染视图2后，视图1变为不可见。If the rendering thread traverses the rendering tree in the process of generating bitmaps, the order from front to back is: root rendering node, rendering node 1, rendering node 2, rendering node 3. Then, the rendering thread first performs the drawing operation in rendering node 1, and then performs the drawing operation in rendering node 2. That is, the rendering thread calls the GPU to first render view 1 and then render view 2 on the surface provided by the surface compositor. Since view 2 completely obscures view 1, after the rendering thread renders view 2, view 1 becomes invisible.
其中，应用程序内视图的互相遮蔽的情况如图3A-图3C所示。Among them, the mutual occlusion of views in the application is shown in Figure 3A-Figure 3C.
图3A-图3C为本申请实施例提供的视图互相遮蔽情况下界面的一个示例性示意图。3A-3C are an exemplary schematic diagram of the interface provided by the embodiment of the present application when views are mutually obscured.
如图3A所示，电子设备上应用程序显示的界面为界面3001，界面3001包括多个与控件对应的视图，其中，控件301为视图3，控件302视图8。在接收到用户对视图3的点击操作后，应用程序显示的界面从界面3001变为界面3002。As shown in Figure 3A, the interface displayed by the application program on the electronic device is interface 3001. The interface 3001 includes multiple views corresponding to controls, where control 301 is view 3 and control 302 is view 8. After receiving the user's click operation on view 3, the interface displayed by the application program changes from interface 3001 to interface 3002.
在界面3001变化为界面3002的过程中，控件301不断变大，在变大的过程中控件301完全遮蔽控件302，即视图3完全遮蔽视图8。In the process of the interface 3001 changing to the interface 3002, the control 301 continues to become larger. In the process of becoming larger, the control 301 completely blocks the control 302, that is, the view 3 completely blocks the view 8.
如图3B所示，电子设备上应用程序显示的界面为界面3003，界面3003包括多个控件，如控件303、控件304、控件305、控件306。在接收到用户点击控件303后，应用程序显示切片(fragment)307，切片307完全遮蔽控件304、控件305、控件306。As shown in Figure 3B, the interface displayed by the application program on the electronic device is interface 3003. Interface 3003 includes multiple controls, such as control 303, control 304, control 305, and control 306. After receiving the user's click on the control 303, the application displays a fragment 307, which completely covers the controls 304, 305, and 306.
如图3C所示，当视频以悬浮窗(或画中画、小窗)的形式播放的情况下，电子设备上应用程序显示的界面从界面3005变为界面3006。其中，界面3005包括控件308、界面3006包括悬浮窗309，悬浮窗309为用于播放视频的窗口，并且悬浮窗309完全遮蔽控件308。As shown in Figure 3C, when the video is played in the form of a floating window (or picture-in-picture, or small window), the interface displayed by the application program on the electronic device changes from interface 3005 to interface 3006. Among them, the interface 3005 includes a control 308, and the interface 3006 includes a floating window 309. The floating window 309 is a window used for playing videos, and the floating window 309 completely blocks the control 308.
结合图2和图3A所示的内容，当电子设备显示的界面变为界面3002，渲染视图8或渲染控件302的操作为无效的操作；结合图2和图3B所示的内容，当电子设备显示的界面为界面3004，渲染控件304、控件305和控件306的操作为无效的操作；结合图2和图3C所示的内容，当电子设备显示的界面为界面3006，渲染控件308的操作为无效的操作。Combining the content shown in Figure 2 and Figure 3A, when the interface displayed by the electronic device changes to interface 3002, the operation of rendering view 8 or rendering control 302 is an invalid operation; combining the content shown in Figure 2 and Figure 3B, when the electronic device The displayed interface is interface 3004, and the operations of rendering control 304, control 305 and control 306 are invalid operations; combined with the content shown in Figure 2 and Figure 3C, when the interface displayed by the electronic device is interface 3006, the operation of rendering control 308 is Invalid operation.
除了上文所示应用程序内视图的互相遮蔽的情况，应用程序间视图的互相遮蔽的情况如图4和图5所示。In addition to the mutual occlusion of views within an application as shown above, the mutual occlusion of views between applications is shown in Figure 4 and Figure 5.
图4为本申请实施例提供的视图互相遮蔽情况下界面的另一个示例性示意图。Figure 4 is another exemplary schematic diagram of the interface provided by the embodiment of the present application when views are mutually obscured.
如图4所示，当应用程序2以悬浮窗的形式显示的情况下，电子设备上显示的界面从界面3007变为界面3008。其中，界面3007为应用程序1的界面，界面3008为应用程序1和应用程序2的界面，应用程序的界面显示在悬浮窗311中，并且在界面3008中，图标控件310被悬浮窗311完全准备。As shown in Figure 4, when the application program 2 is displayed in the form of a floating window, the interface displayed on the electronic device changes from interface 3007 to interface 3008. Among them, interface 3007 is the interface of application 1, and interface 3008 is the interface of application 1 and application 2. The interface of the application is displayed in the floating window 311, and in interface 3008, the icon control 310 is fully prepared by the floating window 311 .
其中，电子设备生成界面3008的过程如图5所示。Among them, the process of the electronic device generating the interface 3008 is shown in Figure 5.
图5为本申请实施例提供的电子设备生成界面的一个示例性示意图。Figure 5 is an exemplary schematic diagram of an electronic device generation interface provided by an embodiment of the present application.
如图5所示，应用程序1生成位图1，并将位图1传递给表面合成器和/或硬件合成策略模块；应用程序2生成位图2，并将位图2传递给表面合成器和/或硬件合成策略模块；表面合成器和/或硬件合成策略模块对位图1和位图2执行图层合成，进而得到位图3。在图4所示的内容中，位图1即为界面3007对应的位图，位图2即为悬浮窗311对应的位图，位图3即为界面3008的位图。As shown in Figure 5, application 1 generates bitmap 1 and passes bitmap 1 to the surface compositor and/or hardware compositing strategy module; application 2 generates bitmap 2 and passes bitmap 2 to the surface compositor. and/or a hardware compositing strategy module; the surface compositor and/or a hardware compositing strategy module performs layer composition on bitmap 1 and bitmap 2 to obtain bitmap 3. In the content shown in Figure 4, bitmap 1 is the bitmap corresponding to the interface 3007, bitmap 2 is the bitmap corresponding to the floating window 311, and bitmap 3 is the bitmap of the interface 3008.
结合图5和图4所示的内容，在电子设备显示如界面3008所示的界面的情况下，渲染图标控件310的操作为无效的操作。Combining the contents shown in FIG. 5 and FIG. 4 , when the electronic device displays an interface as shown in interface 3008 , the operation of rendering the icon control 310 is an invalid operation.
综上，可以理解的是，渲染线程渲染被完全遮蔽的视图至表面上的操作为无效的操作，浪费了渲染线 程和GPU的资源，降低了界面生成的能效比。除此之外，由于渲染线程执行渲染前还需要UI线程对视图执行遍历，则UI线程确定并保存被完全遮蔽的视图的绘制操作为无效的操作，浪费了UI线程的资源，降低了界面生成的能效比。In summary, it can be understood that the operation of rendering a completely obscured view onto the surface by the rendering thread is an invalid operation, wasting rendering threads. In addition, since the UI thread needs to traverse the view before the rendering thread performs rendering, the UI thread determines and saves the drawing operation of the completely obscured view as an invalid operation, which wastes the resources of the UI thread and reduces the energy efficiency of interface generation.
基于此，本申请实施例提供了一种界面生成方法，本申请实施例提供的界面生成方法包括：在UI线程遍历视图的过程中，确定任意视图为被完全遮蔽的视图后，跳过该视图以继续执行遍历；其次，在确定任意视图为被被完全遮蔽的视图后，跳过该视图的子视图的测量方法调用、布局方法调用；最后，应用程序的UI线程生成的渲染树不包含被完全遮蔽的视图的对应的渲染节点。Based on this, embodiments of the present application provide an interface generation method. The interface generation method provided by the embodiment of the present application includes: during the process of the UI thread traversing the views, after determining that any view is a completely occluded view, skip the view to continue executing the traversal; secondly, after determining that any view is a completely occluded view, skip the measurement method calls and layout method calls of the subviews of the view; finally, the rendering tree generated by the UI thread of the application does not contain the The corresponding render node for a fully occluded view.
可以理解的是，由于UI线程没有确定并保存被完全遮蔽的视图的绘制操作，降低UI线程的计算量；其次，由于UI线程跳过了该完全遮蔽的视图的子节点的遍历操作，进一步降低了UI线程的计算量；再次，渲染树不包含被完全遮蔽的视图的对应的渲染节点，降低了渲染线程渲染生成位图的计算量，进而提高了界面生成的能效比。It is understandable that because the UI thread does not determine and save the drawing operation of the completely occluded view, the calculation amount of the UI thread is reduced; secondly, because the UI thread skips the traversal operation of the child nodes of the completely occluded view, further reducing the The calculation amount of the UI thread is reduced; thirdly, the rendering tree does not contain the corresponding rendering node of the completely occluded view, which reduces the calculation amount of the bitmap generated by the rendering thread, thereby improving the energy efficiency ratio of the interface generation.
可选地，在本申请一些实施方式中，应用程序的UI线程在生成渲染树后，UI线程或渲染线程对渲染树执行遍历，确定并剪裁被完全遮蔽的视图的对应的渲染节点，进而降低渲染线程渲染生成位图的计算量。Optionally, in some embodiments of the present application, after the UI thread of the application generates the rendering tree, the UI thread or the rendering thread performs a traversal of the rendering tree to determine and trim the corresponding rendering nodes of the completely occluded views, thereby reducing The amount of calculation required by the rendering thread to render the generated bitmap.
可选地，在本申请一些实施方式中，应用程序的UI线程在生成渲染树后，由统一渲染进程(unirender)将一个或多个应用程序的渲染树合并生成目标渲染树后，或者在合并生成目标渲染树的过程中，统一渲染进程确定并剪裁被完全遮蔽的视图的对应的渲染节点，进而降低渲染线程渲染生成位图的计算量。其中，统一渲染进程为与应用程序独立的进程，负责位图的渲染、图层合成等。统一渲染进程在生成位图后，会将位图传递到显示子***以送显。Optionally, in some embodiments of the present application, after the UI thread of the application generates the rendering tree, the unified rendering process (unirender) merges the rendering trees of one or more applications to generate a target rendering tree, or in the process of merging and generating the target rendering tree, the unified rendering process determines and clips the corresponding rendering nodes of the completely obscured view, thereby reducing the amount of calculation for the rendering thread to render and generate the bitmap. Among them, the unified rendering process is a process independent of the application, responsible for bitmap rendering, layer synthesis, etc. After generating the bitmap, the unified rendering process will pass the bitmap to the display subsystem for display.
可选地，在本申请一些实施方式中，在应用程序开始生成下一帧界面前，由统一渲染进程或窗口管理服务(WindowManagerService，WMS)向应用程序发送通知，该通知用于指示应用程序的窗口中被其他应用程序完全遮蔽的区域的位置。应用程序的UI线程遍历视图的过程中，确定任意视图为被完全遮蔽的视图后，跳过该视图以继续执行遍历；应用程序的UI线程在生成渲染树后，确定并剪裁被完全遮蔽的视图的对应的渲染节点。Optionally, in some embodiments of the present application, before the application starts to generate the next frame of the interface, the unified rendering process or the window management service (WindowManagerService, WMS) sends a notification to the application, and the notification is used to indicate the application's The location of the area of the window that is completely obscured by other applications. During the process of traversing views, the UI thread of the application determines that any view is a completely occluded view, and then skips the view to continue the traversal; after generating the rendering tree, the UI thread of the application determines and clips the view that is completely occluded. The corresponding rendering node.
下面结合图6所示的内容，示例性的介绍本申请实施例提供的界面生成方法的流程。The following exemplarily introduces the process of the interface generation method provided by the embodiment of the present application in conjunction with the content shown in FIG. 6 .
图6为本申请实施例提供的界面生成方法的流程的一个示例性示意图。Figure 6 is an exemplary schematic diagram of the process of the interface generation method provided by the embodiment of the present application.
S601：接收到垂直同步信号后，电子设备确定被完全遮蔽的视图包括第一视图。S601: After receiving the vertical synchronization signal, the electronic device determines that the completely blocked view includes the first view.
其中，电子设备可以通过多种方式确定被完全遮蔽的视图，下面示例性的介绍几种确定被完全遮蔽的视图的几种方法，在此不作限定。The electronic device can determine the completely occluded view in a variety of ways. Several methods of determining the completely occluded view are illustratively introduced below, which are not limited here.
可选地，在本申请一些实施例中，应用程序的UI线程可以在遍历视图(可以是仅对脏区域的遍历)的过程中，确定被完全遮蔽的视图。Optionally, in some embodiments of the present application, the UI thread of the application may determine a completely obscured view during the process of traversing the view (which may be a traversal of only dirty areas).
例如，在图3A所示的界面3002中，应用程序的UI线程可以通过测量方法调用和布局方法调用，确定视图3的大小和位置以及视图8的大小和位置。根据视图的海拔属性(elevation)或视图默认的先后顺序，确定视图3完全遮蔽视图8，即视图8为被完全遮蔽的视图。For example, in interface 3002 shown in Figure 3A, the UI thread of the application can determine the size and position of view 3 and the size and position of view 8 through measurement method calls and layout method calls. According to the elevation attribute of the view or the default sequence of the views, it is determined that view 3 completely blocks view 8, that is, view 8 is a completely blocked view.
其中，视图默认的先后顺序，可以是指视图在视图结构中的顺序。其中，视图结构中的视图的顺序为同级视图节点中左视图节点高于右视图节点。即视图8可以认为是视图3的右视图节点。在视图8为视图3的左视图节点的情况下，视图8不会被视图3完全遮蔽。Among them, the default order of views may refer to the order of views in the view structure. Among them, the order of the views in the view structure is that the left view node is higher than the right view node among the sibling view nodes. That is, view 8 can be considered as the right view node of view 3. In the case where view 8 is the left view node of view 3, view 8 will not be completely obscured by view 3.
若是在遍历的过程中，确定被完全遮蔽的视图包括第一视图，则跳过第一视图的子视图的遍历，继续遍历，如图7所示。If during the traversal process, it is determined that the completely occluded view includes the first view, the traversal of the subviews of the first view is skipped and the traversal continues, as shown in Figure 7 .
图7为本申请实施例提供的确定被完全遮蔽视图的一个示例性示意图。FIG. 7 is an exemplary schematic diagram for determining a completely occluded view provided by an embodiment of the present application.
若应用程序的UI线程遍历视图的顺序从前至后为：Decorview、根视图(视图容器1)、视图容器2、视图2、视图3、第一视图、视图4、视图5、视图容器3。其中，视图2和视图3为视图容器2的子节点，视图4和视图5为第一视图的子节点，视图容器2、第一视图和视同容器3为根视图的子节点。If the application's UI thread traverses the views in order from front to back: Decorview, root view (view container 1), view container 2, view 2, view 3, first view, view 4, view 5, view container 3. Among them, view 2 and view 3 are child nodes of view container 2, view 4 and view 5 are child nodes of the first view, and view container 2, first view and view container 3 are child nodes of the root view.
在应用程序的UI线程遍历视图的过程中，在遍历到第一视图后，确定第一视图的位置后，确定第一视图被完全遮蔽，如被视图2或视图3完全遮蔽。应用程序的UI线程确定第一视图被完全遮蔽后，不会遍历视图4和视图5，也不会确定并保存第一视图、视图4和视图5的绘制操作，进而节约应用程序的UI线程的计算资源，进而提高界面生成的能效比。During the process of the application's UI thread traversing views, after traversing to the first view and determining the position of the first view, it is determined that the first view is completely obscured, such as by view 2 or view 3. After the application's UI thread determines that the first view is completely obscured, it will not traverse View 4 and View 5, nor will it determine and save the drawing operations of the first view, View 4, and View 5, thereby saving the application's UI thread's time. computing resources, thereby improving the energy efficiency of interface generation.
值得说明的是，应用程序的UI线程也可以以其他顺序遍历视图，在此不作限定。 It is worth noting that the UI thread of the application can also traverse the views in other orders, which is not limited here.
可选地，在本申请一些实施例中，应用程序的UI线程可以在遍历视图后，确定被完全遮蔽的视图。Optionally, in some embodiments of the present application, the UI thread of the application can determine the completely occluded view after traversing the views.
一方面，应用程序的UI线程对渲染树进行剪裁，保持应用程序的UI线程的原始的对视图遍历的逻辑，对视图***的修改较少；另一方面，应用程序的UI线程对视图遍历的顺序不一定与视图的互相遮蔽关系一致，进而导致在遍历视图容器2的时候，才会确定已经遍历过的第一视图被视图容器2完全遮蔽，无法在遍历的过程中跳过第一视图，如图8所示。On the one hand, the UI thread of the application prunes the rendering tree, maintaining the original logic of view traversal of the UI thread of the application, and making less modifications to the view system; on the other hand, the UI thread of the application prunes the view traversal. The order is not necessarily consistent with the mutual shielding relationship of the views. As a result, when traversing view container 2, it is determined that the first view that has been traversed is completely obscured by view container 2, and the first view cannot be skipped during the traversal process. As shown in Figure 8.
图8为本申请实施例提供的确定被完全遮蔽视图的一个示例性示意图。FIG. 8 is an exemplary schematic diagram for determining a completely occluded view provided by an embodiment of the present application.
如图8所示，若应用程序的UI线程遍历视图的顺序从前至后为：Decorview、根视图(视图容器1)、第一视图、视图2、视图3、视图容器2、视图4、视图5、视图容器3。其中，视图2和视图3为第一视图的子节点，视图4和视图5为视图容器2的子节点，视图容器2、第一视图和视同容器3为根视图的子节点。As shown in Figure 8, if the UI thread of the application traverses the views in the following order from front to back: Decorview, root view (view container 1), first view, view 2, view 3, view container 2, view 4, view 5, view container 3. Among them, view 2 and view 3 are child nodes of the first view, view 4 and view 5 are child nodes of view container 2, and view container 2, first view and view container 3 are child nodes of the root view.
可选地，在本申请一些实施方式中，在应用程序的UI线程遍历视图的过程中，单独记录一个参数，该参数用于指向被遮蔽的视图，应用程序的UI线程可以确定第一视图、视图2、视图3被完全遮蔽。Optionally, in some embodiments of the present application, during the process of the UI thread of the application program traversing the views, a parameter is recorded separately. This parameter is used to point to the obscured view. The UI thread of the application program can determine the first view, View 2 and View 3 are completely obscured.
可选地，在本申请一些实施方式中，应用程序的UI线程、渲染线程或统一渲染进程基于渲染树可以确定第一视图、视图2、视图3被完全遮蔽，例如，应用程序的UI线程、渲染线程或统一渲染进程遍历渲染树，进而通过渲染节点的渲染属性确定视图之间的遮蔽关系。Optionally, in some embodiments of the present application, the UI thread, rendering thread or unified rendering process of the application can determine that the first view, view 2, and view 3 are completely obscured based on the rendering tree. For example, the UI thread of the application, The rendering thread or unified rendering process traverses the rendering tree and determines the occlusion relationship between views through the rendering attributes of the rendering nodes.
可选地，在本申请一些实施方式中，在应用程序显示的界面包括至少两个窗口的情况下，应用程序还可以基于窗口Z序(Z-order)确定被完全遮蔽的视图。在该情况下，应用程序的UI线程在收到垂直同步信号后，会生成多个渲染树(渲染树的数量可以与窗口的数量相同)，UI线程可以在遍历任意一个渲染树的过程中确定被完全遮蔽的视图，然后基于窗口Z序进一步确定被完全遮蔽的视图。或者，UI线程、渲染线程或统一渲染进程基于渲染树，确定完全遮蔽关系来自渲染树内部的被完全遮蔽的视图，然后基于窗口Z序进一步确定被被完全遮蔽的视图。其中，窗口Z序可以由应用程序或窗口管理服务指定，用于指示不同窗口之间的上下关系(互相遮蔽关系)，或者，窗口的Z序表示窗口在Z轴上的高度，Z轴为垂直与表面的法线。Optionally, in some embodiments of the present application, when the interface displayed by the application includes at least two windows, the application can also determine the completely obscured view based on the window Z-order. In this case, after receiving the vertical synchronization signal, the UI thread of the application will generate multiple rendering trees (the number of rendering trees can be the same as the number of windows), and the UI thread can determine the completely obscured view in the process of traversing any rendering tree, and then further determine the completely obscured view based on the window Z-order. Alternatively, the UI thread, the rendering thread or the unified rendering process determines the completely obscured view from the inside of the rendering tree based on the rendering tree, and then further determines the completely obscured view based on the window Z-order. Among them, the window Z-order can be specified by the application or the window management service to indicate the upper and lower relationship (mutual obscuration relationship) between different windows, or the Z-order of the window represents the height of the window on the Z-axis, and the Z-axis is perpendicular to the normal of the surface.
例如，在上文中的图3C所示的界面3006中，悬浮窗309完全遮蔽的控件308为基于窗口Z序进一步确定的被完全遮蔽的视图。其中，悬浮窗309所在窗口的窗口Z序高于控件308所在窗口的窗口Z序。For example, in the interface 3006 shown in FIG. 3C above, the control 308 completely obscured by the floating window 309 is a completely obscured view further determined based on the window Z order. Among them, the window Z-order of the window where the floating window 309 is located is higher than the window Z-order of the window where the control 308 is located.
可选地，在本申请一些实施方式中，在电子设备显示的界面包括多个应用程序的界面的情况下，由统一渲染进程基于窗口Z序确定不同应用程序的渲染树中渲染节点对应的视图的互相遮蔽关系，进而由统一渲染进程确定被完全遮蔽的视图。其中，统一渲染进程会接收窗口管理服务确定并发送的不同窗口的窗口Z序。Optionally, in some embodiments of the present application, when the interface displayed by the electronic device includes interfaces of multiple applications, the unified rendering process determines the views corresponding to the rendering nodes in the rendering trees of different applications based on the window Z order. mutual occlusion relationship, and then the unified rendering process determines the completely occluded view. Among them, the unified rendering process will receive the window Z order of different windows determined and sent by the window management service.
例如，在上文中的图4所示的界面3008中，统一渲染进程在确定悬浮窗311所在窗口的窗口Z序高于图标控件310所在窗口的窗口Z序后，统一渲染进程可以确定图标控件310被完全遮蔽。For example, in the interface 3008 shown in Figure 4 above, after the unified rendering process determines that the window Z order of the window where the floating window 311 is located is higher than the window Z order of the window where the icon control 310 is located, the unified rendering process can determine the icon control 310 Completely obscured.
可选地，在本申请一些实施方式中，电子设备显示的界面包括多个应用程序的界面的情况下，窗口管理服务基于窗口Z序，确定不同窗口中被完全遮蔽的区域，进而使得应用程序的UI线程、渲染线程可以确定被其他窗口完全遮蔽的视图。其中，应用程序的UI线程可以在遍历的过程中确定，或者，在应用程序的UI线程遍历视图后，由UI线程或渲染线程确定被其他窗口完全遮蔽的视图。Optionally, in some embodiments of the present application, when the interface displayed by the electronic device includes interfaces of multiple applications, the window management service determines the completely blocked areas in different windows based on the window Z order, so that the application program The UI thread and rendering thread can determine views that are completely obscured by other windows. Among them, the UI thread of the application can determine the view during the traversal, or, after the UI thread of the application traverses the view, the UI thread or the rendering thread determines the view that is completely obscured by other windows.
图9为本申请实施例提供的确定被完全遮蔽的视图的另一个示例性示意图。FIG. 9 is another exemplary schematic diagram of determining a completely blocked view provided by an embodiment of the present application.
窗口管理服务在确定任意应用程序的窗口中存在被完全遮蔽区域后，向该应用程序发送消息，该消息用于通知该应用程序被完全遮蔽的区域在窗口中的位置。After determining that a completely obscured area exists in the window of any application, the window management service sends a message to the application. The message is used to notify the application of the location of the completely obscured area in the window.
如图9所示，窗口管理服务在确定应用程序1的窗口会被应用程序2的窗口完全遮蔽一部分后，向应用程序1发送消息，该消息用于通知应用程序1被完全遮蔽的区域在窗口中的位置。As shown in Figure 9, after the window management service determines that the window of application 1 will be completely blocked by the window of application 2, it sends a message to application 1. This message is used to notify application 1 that the completely blocked area is in the window. location in.
应用程序1的UI线程、渲染线程或统一渲染进程可有基于该消息确定被完全遮蔽的视图。渲染线程或统一渲染进程在渲染生成位图时，不会渲染被完全遮蔽的视图。Application 1's UI thread, rendering thread, or unified rendering process may have views that are determined to be fully occluded based on this message. The rendering thread or unified rendering process will not render a fully occluded view when rendering the resulting bitmap.
其中，没有被完全遮蔽(残缺遮蔽或被部分遮蔽)的视图，例如被部分遮蔽的视图，对应的渲染节点还是会出现在渲染树中，并被渲染线程或统一渲染进程渲染在表面上。或者，残缺遮蔽对应的渲染节点还是会出现在渲染树中，并在渲染的过程中通过裁剪指令如Clip()中止该残缺遮蔽在被完全遮蔽的区域上进行渲染。Among them, for views that are not completely occluded (incompletely occluded or partially occluded), such as partially occluded views, the corresponding rendering nodes will still appear in the rendering tree and be rendered on the surface by the rendering thread or the unified rendering process. Alternatively, the rendering node corresponding to the incomplete occlusion will still appear in the rendering tree, and during the rendering process, the incomplete occlusion will be stopped by clipping instructions such as Clip() to render on the completely occluded area.
可选地，在本申请一些实施方式中，在确定被完全遮蔽的视图为视频视图(videoview)的情况下，应用程序的UI线程可以暂停或停止视频的播放，暂停或停止视频的播放可以通过媒体编解码器(mediacodec)、媒体播放器(mediaplayer)等多种功能模块实现，例如，可以删除媒体编解码器的输入缓冲、通过媒体编 解码器暂停视频的解码、删除媒体编解码器的输出缓冲实现暂停或停止视频的播放。Optionally, in some embodiments of the present application, when it is determined that the completely occluded view is a video view (videoview), the UI thread of the application can pause or stop the playback of the video. Pausing or stopping the playback of the video can be done through Media codec (mediacodec), media player (mediaplayer) and other functional modules are implemented. For example, the input buffer of the media codec can be deleted, and the input buffer of the media codec can be deleted. The decoder pauses the decoding of the video and deletes the output buffer of the media codec to pause or stop the video playback.
可选地，在本申请一些实施方式中，在确定被完全遮蔽的视图为视频视图(videoview)的情况下，应用程序的UI线程可以将视频转为后台播放，即只播放视频的音频。Optionally, in some embodiments of the present application, when it is determined that the completely occluded view is a video view (videoview), the UI thread of the application program can convert the video to background playback, that is, only play the audio of the video.
值得说明的是，承载视频的控件并不局限于视频视图，还可以是其他第三方库提供的视图、自定义视图等。It is worth noting that the controls that host videos are not limited to video views, but can also be views provided by other third-party libraries, custom views, etc.
可选地，在本申请一些实施方式中，在确定被完全遮蔽的视图为图像视图的情况下，应用程序的UI线程也不会执图像的加载，不会对图像执行解码。Optionally, in some embodiments of the present application, when it is determined that the completely occluded view is an image view, the UI thread of the application program will not load the image or decode the image.
值得说明的是，承载视频的控件并不局限于图像视图，还可以是其他第三方库提供的视图、自定义视图等。It is worth mentioning that the control hosting the video is not limited to image views, but can also be views provided by other third-party libraries, custom views, etc.
S602：电子设备生成不包含第一视图对应的渲染节点的渲染树。S602: The electronic device generates a rendering tree that does not include the rendering node corresponding to the first view.
在步骤S601中，应用程序的UI线程、渲染线程和/或统一渲染进程已经确定被完全遮蔽的视图，进而可以生成目标渲染树。其中，目标渲染树不包括第一视图以及其他被完全遮蔽的视图对应的渲染节点。In step S601, the UI thread, rendering thread and/or unified rendering process of the application program have determined the completely occluded view, and then the target rendering tree can be generated. The target rendering tree does not include rendering nodes corresponding to the first view and other completely occluded views.
可选地，在本申请一些实施例中，应用程序的UI线程、渲染线程或统一渲染进程可以在残缺遮蔽对应的渲染节点的绘制指令列表中增加裁剪指令如Clip()，进而使得残缺遮蔽不会渲染到被完全遮蔽区域，降低无效的渲染，提升界面生成的能效比。Optionally, in some embodiments of the present application, the UI thread, rendering thread or unified rendering process of the application can add a clipping instruction such as Clip() in the drawing instruction list of the rendering node corresponding to the incomplete occlusion, so that the incomplete occlusion is not It will render to the completely occluded area, reducing invalid rendering and improving the energy efficiency of interface generation.
可选地，在本申请一些实施方式中，应用程序的UI线程在遍历视图的过程中，不生成被完全遮蔽的视图对应的渲染节点，也不会确定并保存该被完全遮蔽的视图的绘制操作，在该情况下，应用程序的UI线程生成的渲染树不包括第一视图及其它被完全遮蔽视图对应的渲染节点。Optionally, in some embodiments of the present application, during the process of traversing views, the UI thread of the application does not generate the rendering node corresponding to the completely occluded view, nor does it determine and save the drawing of the completely occluded view. In this case, the rendering tree generated by the UI thread of the application does not include the rendering nodes corresponding to the first view and other completely occluded views.
可选地，在本申请一些实施方式中，在应用程序的UI线程在遍历视图后才确定被完全遮蔽的视图的情况下，应用程序的UI线程可以删除该被完全遮蔽视图的对应的渲染节点，进而生成不包括第一视图及其它被完全遮蔽视图对应的渲染节点的渲染树。Optionally, in some embodiments of the present application, in the case where the UI thread of the application determines a completely occluded view after traversing the views, the UI thread of the application can delete the corresponding rendering node of the completely occluded view. , and then generate a rendering tree that does not include rendering nodes corresponding to the first view and other completely occluded views.
可选地，在本申请一些实施方式中，在应用程序的UI线程在遍历视图后才确定被完全遮蔽的视图的情况下，应用程序的UI线程可以将被完全遮蔽的视图的标识发送到渲染线程或统一渲染进程，进而由渲染线程或统一渲染进程删除该被完全遮蔽视图的对应的渲染节点，再进而生成不包括第一视图及其它被完全遮蔽视图对应的渲染节点的渲染树。Optionally, in some embodiments of the present application, in the case where the UI thread of the application determines a completely occluded view after traversing the views, the UI thread of the application can send the identification of the completely occluded view to the renderer. The rendering thread or the unified rendering process then deletes the rendering node corresponding to the completely occluded view, and then generates a rendering tree that does not include the rendering nodes corresponding to the first view and other completely occluded views.
可选地，在本申请一些实施方式中，在应用程序的UI线程在遍历视图后才确定被完全遮蔽的视图的情况下，应用程序的UI线程可以在将被完全遮蔽的视图对应的渲染节点的渲染属性或绘制指令列表中增加参数，该参数用于标识该渲染节点对应的视图为被完全遮蔽的视图。渲染线程或统一渲染进程在接收到渲染树后，由渲染线程或统一渲染进程删除该被完全遮蔽视图的对应的渲染节点，进而生成不包括第一视图及其它被完全遮蔽视图对应的渲染节点的渲染树。Optionally, in some embodiments of the present application, in the case where the UI thread of the application determines the view that is completely occluded after traversing the views, the UI thread of the application can determine the view that is completely occluded in the rendering node corresponding to the view that will be completely occluded. Add a parameter to the rendering attribute or drawing instruction list, which is used to identify the view corresponding to the rendering node as a fully occluded view. After the rendering thread or the unified rendering process receives the rendering tree, the rendering thread or the unified rendering process deletes the corresponding rendering node of the completely occluded view, and then generates a rendering node that does not include the first view and other completely occluded views. Render tree.
可选地，在本申请一些实施方式中，应用程序的UI线程执行遍历视图后，将生成的渲染树传递给渲染线程或统一渲染进程，渲染线程或统一渲染进程确定被完全遮蔽的视图后，由渲染线程或统一渲染进程删除该被完全遮蔽视图的对应的渲染节点，进而生成不包括第一视图及其它被完全遮蔽视图对应的渲染节点的渲染树。Optionally, in some embodiments of the present application, after the UI thread of the application executes the traversal of views, the generated rendering tree is passed to the rendering thread or the unified rendering process. After the rendering thread or the unified rendering process determines the view that is completely occluded, The rendering thread or the unified rendering process deletes the corresponding rendering node of the completely occluded view, thereby generating a rendering tree that does not include the first view and the rendering nodes corresponding to other completely occluded views.
可选地，在本申请一些实施方式中，应用程序的UI线程、渲染线程和统一渲染进程均布删除被完全遮蔽的视图对应的渲染节点，而是在渲染的过程中跳过该渲染节点以及该渲染节点的子渲染节点。其中，应用程序的UI线程、渲染线程或统一渲染进程需要在渲染节点添加参数以使得渲染线程或统一渲染进程在生成位图时确定该渲染节点为被完全遮蔽视图对应的渲染节点。Optionally, in some embodiments of the present application, the UI thread, rendering thread, and unified rendering process of the application program uniformly delete the rendering node corresponding to the completely obscured view, and skip the rendering node and the child rendering nodes of the rendering node during the rendering process. The UI thread, rendering thread, or unified rendering process of the application program needs to add parameters to the rendering node so that the rendering thread or unified rendering process determines that the rendering node is the rendering node corresponding to the completely obscured view when generating a bitmap.
其中，上文中，渲染线程或统一渲染进程删除渲染节点的过程如图10A和图10B所示。Among them, in the above, the process of deleting rendering nodes by the rendering thread or the unified rendering process is as shown in Figure 10A and Figure 10B.
图10A-图10D本申请实施例提供的渲染线程或统一渲染进程删除渲染节点的一个示例性示意图。Figures 10A-10D provide an exemplary schematic diagram of a rendering thread or unified rendering process deleting a rendering node according to the embodiment of the present application.
渲染线程删除渲染节点如图10A所示。渲染树1包括跟渲染节点1、渲染节点1、渲染节点2、渲染节点3，其中，渲染线程可以确定渲染节点3对应被完全遮蔽的视图，如第一视图。渲染线程删除渲染节点3。The rendering thread deletes the rendering node as shown in Figure 10A. Rendering tree 1 includes rendering node 1, rendering node 1, rendering node 2, and rendering node 3. The rendering thread can determine that rendering node 3 corresponds to a completely occluded view, such as the first view. The rendering thread deletes render node 3.
其中，渲染线程基于渲染树1(删除渲染节点3前的渲染树)执行渲染得到的位图为位图1，渲染线程基于目标渲染树(删除渲染节点3后的渲染树)执行渲染得到位图同样为位图1。Among them, the bitmap obtained by the rendering thread performing rendering based on rendering tree 1 (the rendering tree before deleting rendering node 3) is bitmap 1, and the rendering thread performing rendering based on the target rendering tree (the rendering tree after deleting rendering node 3) obtains the bitmap Same for Bitmap 1.
统一渲染进程删除渲染节点如图10B和图10C所示。与渲染线程不同，统一渲染进程会接收不同应用程序的UI线程生成的渲染树。The unified rendering process deletes rendering nodes as shown in Figure 10B and Figure 10C. Unlike rendering threads, the unified rendering process receives rendering trees generated by the UI threads of different applications.
如图10B所示，统一渲染进程接收到应用程序1的UI线程生成的渲染树1以及应用程序性2的UI 线程生成的渲染树2。As shown in Figure 10B, the unified rendering process receives the rendering tree 1 generated by the UI thread of application 1 and the UI of application 2 Thread-generated rendering tree 2.
渲染树1包括根渲染节点1，渲染节点1、渲染节点2和渲染节点3，其中，渲染节点3是渲染节点2的子渲染节点，渲染节点1和渲染节点2为根渲染节点1的子节点。渲染树2包括根渲染节点2、渲染节点4、渲染节点5和渲染节点6，其中，渲染节点6是渲染节点5的子渲染节点，渲染节点5和渲染节点4是根渲染节点2的子渲染节点。渲染节点6对应被完全遮蔽的视图。Rendering tree 1 includes root rendering node 1, rendering node 1, rendering node 2 and rendering node 3. Rendering node 3 is a child rendering node of rendering node 2, and rendering node 1 and rendering node 2 are child nodes of root rendering node 1. . Rendering tree 2 includes root rendering node 2, rendering node 4, rendering node 5 and rendering node 6. Rendering node 6 is a child rendering node of rendering node 5, and rendering node 5 and rendering node 4 are child rendering nodes of root rendering node 2. node. Render node 6 corresponds to a fully occluded view.
统一渲染进程在合并渲染树的过程中，删除渲染节点6，得到目标渲染树。其中，合并渲染树的过程，包括新建新根渲染节点，并将根渲染节点1和根渲染节点2作为新根渲染节点的子渲染节点。The unified rendering process deletes rendering node 6 during the process of merging rendering trees to obtain a target rendering tree. The process of merging rendering trees includes creating a new root rendering node and using root rendering node 1 and root rendering node 2 as child rendering nodes of the new root rendering node.
可选地，在本申请一些实施方式中，若统一渲染进程在渲染生成位图的过程中，从左到右遍历，且应用程序2的窗口的窗口Z序大于应用程序1的窗口的窗口Z序，则根渲染节点2为新根渲染节点的右子渲染节点，根渲染节点1为新根渲染节点的右子渲染节点。Optionally, in some embodiments of the present application, if the unified rendering process traverses from left to right during the process of rendering and generating the bitmap, and the window Z order of the window of application 2 is greater than the window Z order of the window of application 1 order, then root rendering node 2 is the right child rendering node of the new root rendering node, and root rendering node 1 is the right child rendering node of the new root rendering node.
可选地，在本申请一些实施方式中，若应用程序2的窗口的窗口Z序大于应用程序1的窗口的窗口Z序，且应用程序1中的视图不存在残缺遮蔽，则根渲染节点2和根渲染节点1的同级顺序可以互换。Optionally, in some embodiments of the present application, if the window Z-order of the window of application 2 is greater than the window Z-order of the window of application 1, and the view in application 1 does not have incomplete occlusion, then the root rendering node 2 The sibling order of root render node 1 is interchangeable.
可选地，在本申请一些实施方式中，若目标渲染树中的残缺遮蔽对应的渲染节点被添加了裁剪指令，则根渲染节点2和根渲染节点1的同级顺序可以互换。Optionally, in some embodiments of the present application, if a clipping instruction is added to the rendering node corresponding to the incomplete occlusion in the target rendering tree, the sibling order of root rendering node 2 and root rendering node 1 can be interchanged.
可选地，在本申请一些实施方式中，若目标渲染树中的残缺遮蔽对应的渲染节点被添加了裁剪指令，或者，若目标渲染树中不存在残缺遮蔽，则可以改变渲染树在渲染生成位图遍历渲染节点的遍历顺序(不改变父子关系决定的遍历顺序)。Optionally, in some embodiments of the present application, if the rendering node corresponding to the incomplete occlusion in the target rendering tree is added with a clipping instruction, or if there is no incomplete occlusion in the target rendering tree, the rendering tree can be changed during rendering generation. The traversal order of the bitmap traversal rendering nodes (does not change the traversal order determined by the parent-child relationship).
可选地，在本申请一些实施方式中，若目标渲染树中的残缺遮蔽对应的渲染节点被添加了裁剪指令，或若目标渲染树中不存在残缺遮蔽，统一渲染进程对渲染节点的绘制指令列表进行解封装得到一系列绘制操作结构体，进而将目标渲染树作为整体进行绘制操作结构体操作进行次批次(Batch)和合并(Merge)，在步骤S603中得到使用更少计算量就可以生成位图。Optionally, in some embodiments of the present application, if a clipping instruction is added to the rendering node corresponding to the incomplete occlusion in the target rendering tree, or if there is no incomplete occlusion in the target rendering tree, the rendering instructions of the rendering node in the unified rendering process are The list is decapsulated to obtain a series of drawing operation structures, and then the target rendering tree is used as a whole to perform drawing operation structure operations for sub-batch (Batch) and merge (Merge). In step S603, it can be obtained with less calculation amount. Generate bitmap.
其中，批次和合并的范围可以大于原生的Skia库所支持的范围。Among them, the batch and merge ranges can be larger than the range supported by the native Skia library.
一方面，原生的Skia库并不能将依赖于后续结果(如在后渲染的同级视图)的绘制操作结构体与发生在后续结果前的绘制操作结果提执行批次和合并。但是，在本申请实施例提供的界面生成方法中，在一些情况下，由于目标渲染树的同级渲染节点的遍历顺序可以互换，可以将依赖于后续结果(如在后渲染的同级视图)的绘制操作结构体与发生在后续结果前的绘制操作结果提执行批次和合并。On the one hand, the native Skia library cannot batch and merge the drawing operation structures that depend on subsequent results (such as the sibling views rendered later) with the drawing operation results that occurred before the subsequent results. However, in the interface generation method provided in the embodiment of the present application, in some cases, since the traversal order of the sibling rendering nodes of the target rendering tree can be interchanged, the drawing operation structures that depend on subsequent results (such as the sibling views rendered later) can be batched and merged with the drawing operation results that occurred before the subsequent results.
另一方面，原始的Skia库所支持的合并和批次仅针对一个应用程序的界面(或者是一个应用程序中的一个窗口中的界面)对应的绘制操作结构体执行合并和批次，优化潜力上限低。但是，在本申请实施例提供的界面生成方法中，由于目标渲染树包括不同应用程序的渲染树，能最大程度的利用合并和批次提升界面生成的能效比，优先潜力上限高。On the other hand, the merging and batching supported by the original Skia library only perform merging and batching on the drawing operation structure corresponding to the interface of an application (or the interface in a window in an application), so there is optimization potential. The upper limit is low. However, in the interface generation method provided by the embodiment of the present application, since the target rendering tree includes rendering trees of different applications, the energy efficiency ratio of interface generation can be maximized by merging and batch improvement, and the upper limit of priority potential is high.
可选地，在本申请一些实施方式中，若目标渲染树中的残缺遮蔽对应的渲染节点被添加了裁剪指令，或若目标渲染树中不存在残缺遮蔽，统一渲染进程可以采取多线程(如vlukan库所支持的多线程渲染)分别渲染目标渲染树的不同子树，即并行的渲染生成位图，可以降低界面生成花费的时间，提升界面生成的能效比。Optionally, in some embodiments of the present application, if the rendering node corresponding to the incomplete occlusion in the target rendering tree is added with a clipping instruction, or if there is no incomplete occlusion in the target rendering tree, the unified rendering process can adopt multi-threading (such as Multi-threaded rendering supported by the vlukan library) renders different subtrees of the target rendering tree separately, that is, parallel rendering generates bitmaps, which can reduce the time spent on interface generation and improve the energy efficiency of interface generation.
如图10C所示，与图10B所示内容不同的是，统一渲染进程在合并渲染树后，删除渲染节点6，进而得到目标渲染树。与图10A所示内容相同的是，合并后的渲染树对应的位图与目标渲染树对应的位图的显示内容相同。As shown in Figure 10C, what is different from what is shown in Figure 10B is that the unified rendering process deletes the rendering node 6 after merging the rendering trees, and then obtains the target rendering tree. The same content as shown in FIG. 10A is that the bitmap corresponding to the merged rendering tree has the same display content as the bitmap corresponding to the target rendering tree.
如图10D所示，结合图4所示的内容，当电子设备显示界面3008时，应用程序1生成的渲染树对应于界面3009。对比界面3009和界面3007，界面3009不包括处于完全遮蔽区域内的视图，其余与界面3007相同。As shown in Figure 10D, combined with the content shown in Figure 4, when the electronic device displays interface 3008, the rendering tree generated by application 1 corresponds to interface 3009. Comparing interface 3009 and interface 3007, interface 3009 does not include views in the completely obscured area, and the rest is the same as interface 3007.
即，若界面3007对应的渲染树为渲染树1，渲染树1包括根渲染节点，根渲染节点1的子渲染节点包括渲染节点1、渲染节点2、渲染节点3，渲染节点1的子渲染节点为渲染节点4和渲染节点5，渲染节点2的子渲染节点为渲染节点6和渲染节点7。That is, if the rendering tree corresponding to interface 3007 is rendering tree 1, rendering tree 1 includes the root rendering node, the child rendering nodes of root rendering node 1 include rendering node 1, rendering node 2, and rendering node 3, and the child rendering nodes of rendering node 1 include are rendering node 4 and rendering node 5, and the child rendering nodes of rendering node 2 are rendering node 6 and rendering node 7.
则界面3009对应的渲染树为从渲染树1删除渲染节点1、渲染节点4、渲染节点5后得到的渲染树。其中，渲染节点1、渲染节点4和渲染节点5对应的视图处于完全遮蔽区域中。Then the rendering tree corresponding to interface 3009 is the rendering tree obtained by deleting rendering node 1, rendering node 4, and rendering node 5 from rendering tree 1. Among them, the views corresponding to rendering node 1, rendering node 4 and rendering node 5 are in the completely obscured area.
对应的，当应用程序2的窗口变化后，完全遮蔽区域变化，被完全遮蔽的视图发生变化，应用程序生成的渲染树也会发生变化。Correspondingly, when the window of application 2 changes, the completely occluded area changes, the completely occluded view changes, and the rendering tree generated by the application also changes.
其中，不完全处于完全遮蔽区域的视图为残缺遮蔽视图。 Among them, the view that is not completely in the completely occluded area is the incomplete occluded view.
S603：电子设备基于渲染树生成位图，并送显。S603: The electronic device generates a bitmap based on the rendering tree and sends it to the display.
渲染线程生成位图后，将位图传递到表面合成器和/或硬件合成策略模块，表面合成器和/或硬件合成策略模块在执行图层合成后，将合成后的位图发送到显示子***以送显。After the rendering thread generates the bitmap, it passes the bitmap to the surface compositor and/or hardware compositing strategy module. After performing layer compositing, the surface compositor and/or hardware compositing strategy module sends the composited bitmap to the display subroutine. The system sends the display.
统一渲染进程生成位图后，可以直接将位图发送到显示子***与送显。或者，统一渲染进程在生成位图后，可以将位图发送到表面合成器和/或硬件合成策略模块，表面合成器和/或硬件合成策略模块在执行图层合成后，将合成后的位图发送到显示子***以送显。After the unified rendering process generates the bitmap, the bitmap can be directly sent to the display subsystem and display. Alternatively, after generating the bitmap, the unified rendering process can send the bitmap to the surface compositor and/or hardware compositing strategy module. After performing layer compositing, the surface compositor and/or hardware compositing strategy module will The image is sent to the display subsystem for display.
综上，本申请实施例提供的界面生成方法，可以降低由于被完全遮蔽的视图引起的无效的测量方法调用、布局方法调用、绘制方法调用，以及降低由于被完全遮蔽的视图对应的渲染节点引起的无效的渲染，进而提高了界面生成的能效比。In summary, the interface generation method provided by the embodiments of the present application can reduce invalid measurement method calls, layout method calls, and drawing method calls caused by completely occluded views, as well as reduce the number of invalid measurement method calls, layout method calls, and drawing method calls caused by rendering nodes corresponding to completely occluded views. Ineffective rendering, thereby improving the energy efficiency of interface generation.
图11为本申请实施例提供的界面生成方法的数据流动的一个示例性示意图。Figure 11 is an exemplary schematic diagram of the data flow of the interface generation method provided by the embodiment of the present application.
如图11所示，界面生成方法的数据流动包括如下步骤。As shown in Figure 11, the data flow of the interface generation method includes the following steps.
S1101：发送被完全遮蔽区域的位置。S1101: Send the position of the completely obscured area.
窗口管理服务器在确定有应用程序的界面被其他应用程序的界面遮蔽后，可以向被遮蔽的应用程序发送被完全遮蔽区域的位置。例如，窗口管理服务向应用程序1的UI线程发送被完全遮蔽区域的位置。After determining that the interface of an application is blocked by the interface of other applications, the window management server may send the location of the completely blocked area to the blocked application. For example, the window management service sends the location of the fully obscured area to Application 1's UI thread.
可选地，在本申请一些实施方式中，步骤S1101不被执行，在步骤S1102和步骤S1105中，应用程序只能分别确定被同应用程序的其他视图完全遮蔽的视图，并生成不包括该完全遮蔽的视图的渲染树。Optionally, in some implementations of the present application, step S1101 is not executed, and in steps S1102 and S1105, the application can only determine the views that are completely obscured by other views of the same application, and generate a rendering tree that does not include the completely obscured views.
S1102：接收到垂直同步信号后，构建生成不包含被完全遮蔽视图对应的渲染节点的渲染树1。S1102: After receiving the vertical synchronization signal, construct and generate a rendering tree 1 that does not include the rendering node corresponding to the completely occluded view.
应用程序1的UI线程构建生成不包含被完全遮蔽视图对应的渲染节点的渲染树1。其中，被完全遮蔽的视图包括：视图完整地处于被完全遮蔽区域的位置和被其他应用程序1的视图遮蔽的视图。The UI thread of Application 1 builds a rendering tree 1 that does not contain the rendering node corresponding to the completely occluded view. Among them, the completely occluded view includes: the view is completely located in the completely occluded area and the view is occluded by the views of other applications 1.
S1103：传递或同步渲染树1。S1103: Pass or synchronize rendering tree 1.
应用程序1的UI线程向应用程序1的渲染线程传递或同步渲染树1。Application 1's UI thread passes or synchronizes render tree 1 to Application 1's rendering thread.
S1104：基于渲染树1生成位图1。S1104: Generate bitmap 1 based on rendering tree 1.
应用程序1的渲染线程基于渲染树1生成位图1，其中，在渲染生成位图1的过程中，由于渲染树1中不包含被完全遮蔽的视图对应的渲染节点，故也不会在表面上渲染该被完全遮蔽的视图。The rendering thread of application 1 generates bitmap 1 based on rendering tree 1. During the process of rendering and generating bitmap 1, since rendering tree 1 does not contain the rendering node corresponding to the completely occluded view, it will not be on the surface. Render the fully occluded view.
S1105：接收到垂直同步信号后，构建生成不包含被完全遮蔽视图对应的渲染节点的渲染树2。S1105: After receiving the vertical synchronization signal, construct and generate a rendering tree 2 that does not include the rendering node corresponding to the completely occluded view.
类似的，应用程序2的UI线程构建生成不包含被完全遮蔽视图对应的渲染节点的渲染树2。其中，被完全遮蔽的视图包括：被其他应用程序2的视图遮蔽的视图。Similarly, the UI thread of Application 2 builds a rendering tree 2 that does not contain the rendering node corresponding to the completely occluded view. Among them, the views that are completely blocked include: views that are blocked by the views of other applications 2.
S1106：传递或同步渲染树2。S1106: Pass or synchronize render tree 2.
应用程序2的UI线程向应用程序2的渲染线程传递或同步渲染树2。Application 2's UI thread passes or synchronizes Render Tree 2 to Application 2's rendering thread.
S1107：基于渲染树2生成位图2。S1107: Generate bitmap 2 based on rendering tree 2.
应用程序2的渲染线程基于渲染树2生成位图2，其中，在渲染生成位图2的过程中，由于渲染树2中不包含被完全遮蔽的视图对应的渲染节点，故也不会在表面上渲染该被完全遮蔽的视图。The rendering thread of application 2 generates bitmap 2 based on rendering tree 2. During the process of rendering and generating bitmap 2, since rendering tree 2 does not contain the rendering node corresponding to the completely occluded view, it will not be on the surface. Render the fully occluded view.
S1108：发送位图1。S1108: Send bitmap 1.
应用程序1的渲染线程向表面合成器和/或硬件合成策略模块发送位图1。Application 1's rendering thread sends Bitmap 1 to the surface compositor and/or hardware compositing strategy module.
S1109：发送位图2。S1109: Send bitmap 2.
应用程序2的渲染线程向表面合成器和/或硬件合成策略模块发送位图2。Application 2's rendering thread sends Bitmap 2 to the surface compositor and/or the hardware compositing strategy module.
S1110：对位图1和位图2执行图层合成以获得位图3。S1110: Perform layer composition on Bitmap 1 and Bitmap 2 to obtain Bitmap 3.
表面合成器和/或硬件合成策略模块在对位图1和位图2执行图层合成后，生成位图3。表面合成器和/或硬件合成策略模块可以将为位图3发送到显示子***以送显。The surface compositor and/or hardware compositing strategy module generates bitmap 3 after performing layer compositing on bitmap 1 and bitmap 2. The surface compositor and/or hardware compositing strategy module may send bitmap 3 to the display subsystem for display.
图12为本申请实施例提供的界面生成方法的数据流动的另一个示例性示意图。Figure 12 is another exemplary schematic diagram of the data flow of the interface generation method provided by the embodiment of the present application.
如图12所示，应用程序1的UI线程生成渲染树1，会删除渲染树1中的渲染节点，其中被删除的渲染节点对应于被完全遮蔽的视图，其中，被完全遮蔽的视图为应用程序1中被应用程序1的其他视图完全遮蔽的视图。As shown in Figure 12, the UI thread of application 1 generates rendering tree 1 and deletes the rendering nodes in rendering tree 1. The deleted rendering nodes correspond to completely occluded views, where the completely occluded views are application A view in Application 1 that is completely obscured by other views in Application 1.
应用程序1的UI线程生成渲染树2，会删除渲染树2中的渲染节点，其中被删除的渲染节点对应于被完全遮蔽的视图，其中，被完全遮蔽的视图为应用程序2中被应用程序2的其他视图完全遮蔽的视图。The UI thread of Application 1 generates Rendering Tree 2 and deletes the rendering nodes in Rendering Tree 2. The deleted rendering nodes correspond to the completely occluded views. The completely occluded views are the ones being applied in Application 2. 2 other views Fully occluded view.
应用程序1的UI线程将渲染树1传递或同步到统一渲染进程，应用程序2将渲染树2传递或同步到统一渲染进程。统一渲染进程基于渲染树1和渲染树2生成目标渲染树，并删除渲染节点。被删除的渲染节点对应于被完全遮蔽的图，其中，被完全遮蔽的视图为被其他应用程序的界面完全遮蔽的视图。 Application 1's UI thread passes or synchronizes rendering tree 1 to the unified rendering process, and application 2 passes or synchronizes rendering tree 2 to the unified rendering process. The unified rendering process generates the target rendering tree based on rendering tree 1 and rendering tree 2, and deletes the rendering nodes. The deleted rendering node corresponds to a completely occluded graph, where a completely occluded view is a view that is completely occluded by the interface of other applications.
统一渲染进程基于目标渲染树生成位图，该位图等价与图11中的位图3。The unified rendering process generates a bitmap based on the target rendering tree, which is equivalent to bitmap 3 in Figure 11.
图13和图14为本申请实施例提供的界面生成方法的数据流动的另一个示例性示意图。Figures 13 and 14 are another exemplary schematic diagram of the data flow of the interface generation method provided by the embodiment of the present application.
如图13所示，界面生成方法的数据流动包括如下步骤。As shown in FIG. 13 , the data flow of the interface generation method includes the following steps.
S1301：接收到垂直同步信号后，构建生成渲染树。S1301: After receiving the vertical synchronization signal, build a rendering tree.
不同的应用程序的UI线程在接收到垂直同步信号后，构建生成渲染树。不同的应用程序如应用程序1、应用程序2、…、应用程序N。The UI threads of different applications build and generate rendering trees after receiving vertical synchronization signals. Different applications such as Application 1, Application 2, ..., Application N.
S1302：跨进程传递或同步渲染树。S1302: Pass or synchronize rendering trees across processes.
不同的应用程序的UI线程向统一渲染进程跨进程传递或同步渲染树。The UI threads of different applications pass cross-process or synchronize the rendering tree to the unified rendering process.
S1303：发送应用程序的窗口控制信息。S1303: Sending the application's window control information.
窗口管理服务向统一渲染进程窗口控制信息，窗口控制信息可以包括不同应用程序的窗口的信息，如尺寸、位置等。统一渲染进程可以基于窗口控制信息确定界面被遮蔽的应用程序，以及被完全遮蔽的区域，进而确定完全处于被完全遮蔽的区域的视图。The window management service provides window control information to the unified rendering process. The window control information can include information about windows of different applications, such as size, position, etc. The unified rendering process can determine the application whose interface is occluded and the completely occluded area based on the window control information, and then determine the view that is completely in the completely occluded area.
S1304：合并渲染树，并删除被完全遮蔽的视图对应的渲染节点，生成目标渲染树。S1304: Merge the rendering trees, delete the rendering nodes corresponding to the completely occluded views, and generate the target rendering tree.
统一渲染进程在接收到不同应用程序发送的渲染树，合并多个渲染树生成目标渲染树，并且删除被完全遮蔽的视图对应的渲染节点。其中，被完全遮蔽的视图包括被同应用程序内的其他视图遮蔽的视图，也包括完全处于被完全遮蔽的区域的视图。After receiving the rendering trees sent by different applications, the unified rendering process merges multiple rendering trees to generate a target rendering tree, and deletes the rendering nodes corresponding to the completely occluded views. The fully occluded views include views that are occluded by other views in the same application, and also include views that are completely in a completely occluded area.
S1305：基于目标渲染树生成位图1。S1305: Generate bitmap 1 based on the target rendering tree.
统一渲染进程基于目标渲染树生成位图1，位图1等价于图11中的位图3。The unified rendering process generates bitmap 1 based on the target rendering tree. Bitmap 1 is equivalent to bitmap 3 in Figure 11.
图13与图14所示内容不同的，应用程序(如应用程序1)的UI线程在构建生成渲染树时，可以删除完全被遮蔽的视图，其中，被完全遮蔽的视图包括被同应用程序内的其他视图遮蔽的视图，如步骤S1306所示。The content shown in Figure 13 is different from that shown in Figure 14. When the UI thread of an application (such as Application 1) builds and generates a rendering tree, it can delete a completely occluded view. Among them, a completely occluded view includes a view within the same application. The view blocked by other views, as shown in step S1306.
S1306：接收到垂直同步信号后，构建生成不包含被完全遮蔽的视图对应的渲染节点的渲染树。S1306: After receiving the vertical synchronization signal, construct and generate a rendering tree that does not include the rendering node corresponding to the completely occluded view.
对应的，在步骤S1304中，统一渲染进程删除的被完全遮蔽的视图对应的渲染节点，其中，被完全遮蔽的视图包括处于被完全遮蔽的区域的视图，即不需要统一渲染进程删除被同应用程序内的其他视图遮蔽的视图对应的渲染节点。Correspondingly, in step S1304, the unified rendering process deletes the rendering node corresponding to the completely occluded view, where the completely occluded view includes a view in a completely occluded area, that is, there is no need for the unified rendering process to delete the view that is used by the same application. The rendering node corresponding to the view obscured by other views in the program.
最后，介绍本申请实施例提供的电子设备的硬件结构和软件架构。Finally, the hardware structure and software architecture of the electronic device provided by the embodiments of the present application are introduced.
图15为本申请实施例提供的电子设备的硬件结构的一个示例性示意图。FIG. 15 is an exemplary schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application.
电子设备可以是手机、平板电脑、桌面型计算机、膝上型计算机、手持计算机、笔记本电脑、超级移动个人计算机(ultra-mobile personal computer，UMPC)、上网本，以及蜂窝电话、个人数字助理(personal digital assistant，PDA)、增强现实(augmented reality，AR)设备、虚拟现实(virtual reality，VR)设备、人工智能(artificial intelligence,AI)设备、可穿戴式设备、车载设备、智能家居设备和/或智慧城市设备，本申请实施例对该电子设备的具体类型不作特殊限制。The electronic device may be a mobile phone, tablet computer, desktop computer, laptop computer, handheld computer, notebook computer, ultra-mobile personal computer (UMPC), netbook, as well as cellular phone, personal digital assistant (personal digital assistant) assistant (PDA), augmented reality (AR) devices, virtual reality (VR) devices, artificial intelligence (AI) devices, wearable devices, vehicle-mounted devices, smart home devices and/or smart devices City equipment, the embodiment of this application does not place special restrictions on the specific type of electronic equipment.
电子设备可以包括处理器110，外部存储器接口120，内部存储器121，通用串行总线(universal serial bus，USB)接口130，充电管理模块140，电源管理模块141，电池142，天线1，天线2，移动通信模块150，无线通信模块160，音频模块170，扬声器170A，受话器170B，麦克风170C，耳机接口170D，传感器模块180，按键190，马达191，指示器192，摄像头193，显示屏194，以及用户标识模块(subscriber identification module，SIM)卡接口195等。其中传感器模块180可以包括压力传感器180A，陀螺仪传感器180B，气压传感器180C，磁传感器180D，加速度传感器180E，距离传感器180F，接近光传感器180G，指纹传感器180H，温度传感器180J，触摸传感器180K，环境光传感器180L，骨传导传感器180M等。The electronic device may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.
可以理解的是，本发明实施例示意的结构并不构成对电子设备的具体限定。在本申请另一些实施例中，电子设备可以包括比图示更多或更少的部件，或者组合某些部件，或者拆分某些部件，或者不同的部件布置。图示的部件可以以硬件，软件或软件和硬件的组合实现。It can be understood that the structures illustrated in the embodiments of the present invention do not constitute specific limitations on the electronic equipment. In other embodiments of the present application, the electronic device may include more or less components than shown in the figures, or some components may be combined, some components may be separated, or some components may be arranged differently. The components illustrated may be implemented in hardware, software, or a combination of software and hardware.
处理器110可以包括一个或多个处理单元，例如：处理器110可以包括应用处理器(application processor，AP)，调制解调处理器，图形处理器(graphics processing unit，GPU)，图像信号处理器(image signal processor，ISP)，控制器，视频编解码器，数字信号处理器(digital signal processor，DSP)，基带处理器，和/或神经网络处理器(neural-network processing unit，NPU)等。其中，不同的处理单元可以是独立的器件，也可以集成在一个或多个处理器中。The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (application processor, AP), a modem processor, a graphics processing unit (GPU), and an image signal processor. (image signal processor, ISP), controller, video codec, digital signal processor (digital signal processor, DSP), baseband processor, and/or neural network processor (neural-network processing unit, NPU), etc. Among them, different processing units can be independent devices or integrated in one or more processors.
控制器可以根据指令操作码和时序信号，产生操作控制信号，完成取指令和执行指令的控制。 The controller can generate operation control signals based on the instruction operation code and timing signals to complete the control of fetching and executing instructions.
处理器110中还可以设置存储器，用于存储指令和数据。在一些实施例中，处理器110中的存储器为高速缓冲存储器。该存储器可以保存处理器110刚用过或循环使用的指令或数据。如果处理器110需要再次使用该指令或数据，可从所述存储器中直接调用。避免了重复存取，减少了处理器110的等待时间，因而提高了***的效率。The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in processor 110 is cache memory. This memory may hold instructions or data that have been recently used or recycled by processor 110 . If the processor 110 needs to use the instructions or data again, it can be called directly from the memory. Repeated access is avoided and the waiting time of the processor 110 is reduced, thus improving the efficiency of the system.
在一些实施例中，处理器110可以包括一个或多个接口。接口可以包括集成电路(inter-integrated circuit，I2C)接口，集成电路内置音频(inter-integrated circuit sound，I2S)接口，脉冲编码调制(pulse code modulation，PCM)接口，通用异步收发传输器(universal asynchronous receiver/transmitter，UART)接口，移动产业处理器接口(mobile industry processor interface，MIPI)，通用输入输出(general-purpose input/output，GPIO)接口，用户标识模块(subscriber identity module，SIM)接口，和/或通用串行总线(universal serial bus，USB)接口等。In some embodiments, processor 110 may include one or more interfaces. Interfaces may include integrated circuit (inter-integrated circuit, I2C) interface, integrated circuit built-in audio (inter-integrated circuit sound, I2S) interface, pulse code modulation (pulse code modulation, PCM) interface, universal asynchronous receiver and transmitter (universal asynchronous receiver/transmitter (UART) interface, mobile industry processor interface (MIPI), general-purpose input/output (GPIO) interface, subscriber identity module (SIM) interface, and /or universal serial bus (USB) interface, etc.
I2C接口是一种双向同步串行总线，包括一根串行数据线(serial data line，SDA)和一根串行时钟线(derail clock line，SCL)。在一些实施例中，处理器110可以包含多组I2C总线。处理器110可以通过不同的I2C总线接口分别耦合触摸传感器180K，充电器，闪光灯，摄像头193等。例如：处理器110可以通过I2C接口耦合触摸传感器180K，使处理器110与触摸传感器180K通过I2C总线接口通信，实现电子设备的触摸功能。The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (derail clock line, SCL). In some embodiments, processor 110 may include multiple sets of I2C buses. The processor 110 can separately couple the touch sensor 180K, charger, flash, camera 193, etc. through different I2C bus interfaces. For example, the processor 110 can be coupled to the touch sensor 180K through an I2C interface, so that the processor 110 and the touch sensor 180K communicate through the I2C bus interface to implement the touch function of the electronic device.
I2S接口可以用于音频通信。在一些实施例中，处理器110可以包含多组I2S总线。处理器110可以通过I2S总线与音频模块170耦合，实现处理器110与音频模块170之间的通信。在一些实施例中，音频模块170可以通过I2S接口向无线通信模块160传递音频信号，实现通过蓝牙耳机接听电话的功能。The I2S interface can be used for audio communication. In some embodiments, processor 110 may include multiple sets of I2S buses. The processor 110 can be coupled with the audio module 170 through the I2S bus to implement communication between the processor 110 and the audio module 170 . In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 through the I2S interface to implement the function of answering calls through a Bluetooth headset.
PCM接口也可以用于音频通信，将模拟信号抽样，量化和编码。在一些实施例中，音频模块170与无线通信模块160可以通过PCM总线接口耦合。在一些实施例中，音频模块170也可以通过PCM接口向无线通信模块160传递音频信号，实现通过蓝牙耳机接听电话的功能。所述I2S接口和所述PCM接口都可以用于音频通信。The PCM interface can also be used for audio communications to sample, quantize and encode analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 may be coupled through a PCM bus interface. In some embodiments, the audio module 170 can also transmit audio signals to the wireless communication module 160 through the PCM interface to implement the function of answering calls through a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.
UART接口是一种通用串行数据总线，用于异步通信。该总线可以为双向通信总线。它将要传输的数据在串行通信与并行通信之间转换。在一些实施例中，UART接口通常被用于连接处理器110与无线通信模块160。例如：处理器110通过UART接口与无线通信模块160中的蓝牙模块通信，实现蓝牙功能。在一些实施例中，音频模块170可以通过UART接口向无线通信模块160传递音频信号，实现通过蓝牙耳机播放音乐的功能。The UART interface is a universal serial data bus used for asynchronous communication. The bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, a UART interface is generally used to connect the processor 110 and the wireless communication module 160 . For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 through the UART interface to implement the Bluetooth function. In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 through the UART interface to implement the function of playing music through a Bluetooth headset.
MIPI接口可以被用于连接处理器110与显示屏194，摄像头193等***器件。MIPI接口包括摄像头串行接口(camera serial interface，CSI)，显示屏串行接口(display serial interface，DSI)等。在一些实施例中，处理器110和摄像头193通过CSI接口通信，实现电子设备的拍摄功能。处理器110和显示屏194通过DSI接口通信，实现电子设备的显示功能。The MIPI interface can be used to connect the processor 110 with peripheral devices such as the display screen 194 and the camera 193 . MIPI interfaces include camera serial interface (CSI), display serial interface (DSI), etc. In some embodiments, the processor 110 and the camera 193 communicate through a CSI interface to implement the shooting function of the electronic device. The processor 110 and the display screen 194 communicate through the DSI interface to implement the display function of the electronic device.
GPIO接口可以通过软件配置。GPIO接口可以被配置为控制信号，也可被配置为数据信号。在一些实施例中，GPIO接口可以用于连接处理器110与摄像头193，显示屏194，无线通信模块160，音频模块170，传感器模块180等。GPIO接口还可以被配置为I2C接口，I2S接口，UART接口，MIPI接口等。The GPIO interface can be configured through software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 with the camera 193, display screen 194, wireless communication module 160, audio module 170, sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, I2S interface, UART interface, MIPI interface, etc.
USB接口130是符合USB标准规范的接口，具体可以是Mini USB接口，Micro USB接口，USB Type C接口等。USB接口130可以用于连接充电器为电子设备充电，也可以用于电子设备与***设备之间传输数据。也可以用于连接耳机，通过耳机播放音频。该接口还可以用于连接其他电子设备，例如AR设备等。The USB interface 130 is an interface that complies with the USB standard specification, and may be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 can be used to connect a charger to charge the electronic device, and can also be used to transmit data between the electronic device and peripheral devices. It can also be used to connect headphones to play audio through them. This interface can also be used to connect other electronic devices, such as AR devices, etc.
可以理解的是，本发明实施例示意的各模块间的接口连接关系，只是示意性说明，并不构成对电子设备的结构限定。在本申请另一些实施例中，电子设备也可以采用上述实施例中不同的接口连接方式，或多种接口连接方式的组合。It can be understood that the interface connection relationships between the modules illustrated in the embodiments of the present invention are only schematic illustrations and do not constitute structural limitations on the electronic equipment. In other embodiments of the present application, the electronic device may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.
充电管理模块140用于从充电器接收充电输入。其中，充电器可以是无线充电器，也可以是有线充电器。在一些有线充电的实施例中，充电管理模块140可以通过USB接口130接收有线充电器的充电输入。在一些无线充电的实施例中，充电管理模块140可以通过电子设备的无线充电线圈接收无线充电输入。充电管理模块140为电池142充电的同时，还可以通过电源管理模块141为电子设备供电。The charging management module 140 is used to receive charging input from the charger. Among them, the charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive charging input from the wired charger through the USB interface 130 . In some wireless charging embodiments, the charging management module 140 may receive wireless charging input through a wireless charging coil of the electronic device. While the charging management module 140 charges the battery 142, it can also provide power to the electronic device through the power management module 141.
电源管理模块141用于连接电池142，充电管理模块140与处理器110。电源管理模块141接收电池142和/或充电管理模块140的输入，为处理器110，内部存储器121，显示屏194，摄像头193，和无线通信模块160等供电。电源管理模块141还可以用于监测电池容量，电池循环次数，电池健康状态(漏电，阻抗)等参数。在其他一些实施例中，电源管理模块141也可以设置于处理器110中。在另一些实施例中，电源管理模块141和充电管理模块140也可以设置于同一个器件中。The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and/or the charging management module 140, and supplies power to the processor 110, the internal memory 121, the display screen 194, the camera 193, the wireless communication module 160, and the like. The power management module 141 can also be used to monitor battery capacity, battery cycle times, battery health status (leakage, impedance) and other parameters. In some other embodiments, the power management module 141 may also be provided in the processor 110 . In other embodiments, the power management module 141 and the charging management module 140 may also be provided in the same device.
电子设备的无线通信功能可以通过天线1，天线2，移动通信模块150，无线通信模块160，调制解调 处理器以及基带处理器等实现。The wireless communication function of the electronic device can be through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem Processor and baseband processor implementation.
天线1和天线2用于发射和接收电磁波信号。电子设备中的每个天线可用于覆盖单个或多个通信频带。不同的天线还可以复用，以提高天线的利用率。例如：可以将天线1复用为无线局域网的分集天线。在另外一些实施例中，天线可以和调谐开关结合使用。Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in an electronic device can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example: Antenna 1 can be reused as a diversity antenna for a wireless LAN. In other embodiments, antennas may be used in conjunction with tuning switches.
移动通信模块150可以提供应用在电子设备上的包括2G/3G/4G/5G等无线通信的解决方案。移动通信模块150可以包括至少一个滤波器，开关，功率放大器，低噪声放大器(low noise amplifier，LNA)等。移动通信模块150可以由天线1接收电磁波，并对接收的电磁波进行滤波，放大等处理，传送至调制解调处理器进行解调。移动通信模块150还可以对经调制解调处理器调制后的信号放大，经天线1转为电磁波辐射出去。在一些实施例中，移动通信模块150的至少部分功能模块可以被设置于处理器110中。在一些实施例中，移动通信模块150的至少部分功能模块可以与处理器110的至少部分模块被设置在同一个器件中。The mobile communication module 150 can provide wireless communication solutions including 2G/3G/4G/5G applied to electronic devices. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves through the antenna 1, perform filtering, amplification and other processing on the received electromagnetic waves, and transmit them to the modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves through the antenna 1 for radiation. In some embodiments, at least part of the functional modules of the mobile communication module 150 may be disposed in the processor 110 . In some embodiments, at least part of the functional modules of the mobile communication module 150 and at least part of the modules of the processor 110 may be provided in the same device.
调制解调处理器可以包括调制器和解调器。其中，调制器用于将待发送的低频基带信号调制成中高频信号。解调器用于将接收的电磁波信号解调为低频基带信号。随后解调器将解调得到的低频基带信号传送至基带处理器处理。低频基带信号经基带处理器处理后，被传递给应用处理器。应用处理器通过音频设备(不限于扬声器170A，受话器170B等)输出声音信号，或通过显示屏194显示图像或视频。在一些实施例中，调制解调处理器可以是独立的器件。在另一些实施例中，调制解调处理器可以独立于处理器110，与移动通信模块150或其他功能模块设置在同一个器件中。A modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs sound signals through audio devices (not limited to speaker 170A, receiver 170B, etc.), or displays images or videos through display screen 194. In some embodiments, the modem processor may be a stand-alone device. In other embodiments, the modem processor may be independent of the processor 110 and may be provided in the same device as the mobile communication module 150 or other functional modules.
无线通信模块160可以提供应用在电子设备上的包括无线局域网(wireless local area networks，WLAN)(如无线保真(wireless fidelity，Wi-Fi)网络)，蓝牙(bluetooth，BT)，全球导航卫星***(global navigation satellite system，GNSS)，调频(frequency modulation，FM)，近距离无线通信技术(near field communication，NFC)，红外技术(infrared，IR)等无线通信的解决方案。无线通信模块160可以是集成至少一个通信处理模块的一个或多个器件。无线通信模块160经由天线2接收电磁波，将电磁波信号调频以及滤波处理，将处理后的信号发送到处理器110。无线通信模块160还可以从处理器110接收待发送的信号，对其进行调频，放大，经天线2转为电磁波辐射出去。The wireless communication module 160 can provide wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) network), Bluetooth (BT), and global navigation satellite systems for use in electronic devices. (global navigation satellite system, GNSS), frequency modulation (frequency modulation, FM), near field communication technology (near field communication, NFC), infrared technology (infrared, IR) and other wireless communication solutions. The wireless communication module 160 may be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2 , frequency modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110 . The wireless communication module 160 can also receive the signal to be sent from the processor 110, frequency modulate it, amplify it, and convert it into electromagnetic waves through the antenna 2 for radiation.
在一些实施例中，电子设备的天线1和移动通信模块150耦合，天线2和无线通信模块160耦合，使得电子设备可以通过无线通信技术与网络以及其他设备通信。所述无线通信技术可以包括全球移动通讯***(global system for mobile communications，GSM)，通用分组无线服务(general packet radio service，GPRS)，码分多址接入(code division multiple access，CDMA)，宽带码分多址(wideband code division multiple access，WCDMA)，时分码分多址(time-division code division multiple access，TD-SCDMA)，长期演进(long term evolution，LTE)，BT，GNSS，WLAN，NFC，FM，和/或IR技术等。所述GNSS可以包括全球卫星定位***(global positioning system，GPS)，全球导航卫星***(global navigation satellite system，GLONASS)，北斗卫星导航***(beidou navigation satellite system，BDS)，准天顶卫星***(quasi-zenith satellite system，QZSS)和/或星基增强***(satellite based augmentation systems，SBAS)。In some embodiments, the antenna 1 of the electronic device is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), broadband Code division multiple access (wideband code division multiple access, WCDMA), time division code division multiple access (time-division code division multiple access, TD-SCDMA), long term evolution (long term evolution, LTE), BT, GNSS, WLAN, NFC , FM, and/or IR technology, etc. The GNSS may include global positioning system (GPS), global navigation satellite system (GLONASS), Beidou navigation satellite system (BDS), quasi-zenith satellite system (quasi) -zenith satellite system (QZSS) and/or satellite based augmentation systems (SBAS).
电子设备通过GPU，显示屏194，以及应用处理器等实现显示功能。GPU为图像处理的微处理器，连接显示屏194和应用处理器。GPU用于执行数学和几何计算，用于图形渲染。处理器110可包括一个或多个GPU，其执行程序指令以生成或改变显示信息。The electronic device implements display functions through the GPU, display screen 194, and application processor. The GPU is an image processing microprocessor and is connected to the display screen 194 and the application processor. GPUs are used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or alter display information.
显示屏194用于显示图像，视频等。显示屏194包括显示面板。显示面板可以采用液晶显示屏(liquid crystal display，LCD)，有机发光二极管(organic light-emitting diode，OLED)，有源矩阵有机发光二极体或主动矩阵有机发光二极体(active-matrix organic light emitting diode的，AMOLED)，柔性发光二极管(flex light-emitting diode，FLED)，Miniled，MicroLed，Micro-oLed，量子点发光二极管(quantum dot light emitting diodes，QLED)等。在一些实施例中，电子设备可以包括1个或N个显示屏194，N为大于1的正整数。The display screen 194 is used to display images, videos, etc. Display 194 includes a display panel. The display panel can use a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active matrix organic light emitting diode or an active matrix organic light emitting diode (active-matrix organic light emitting diode). emitting diode (AMOLED), flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, quantum dot light emitting diode (QLED), etc. In some embodiments, the electronic device may include 1 or N display screens 194, where N is a positive integer greater than 1.
电子设备可以通过ISP，摄像头193，视频编解码器，GPU，显示屏194以及应用处理器等实现拍摄功能。The electronic device can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.
ISP用于处理摄像头193反馈的数据。例如，拍照时，打开快门，光线通过镜头被传递到摄像头感光元件上，光信号转换为电信号，摄像头感光元件将所述电信号传递给ISP处理，转化为肉眼可见的图像。ISP还可以对图像的噪点，亮度进行算法优化。ISP还可以对拍摄场景的曝光，色温等参数优化。在一些实施例中，ISP可以设置在摄像头193中。The ISP is used to process the data fed back by the camera 193. For example, when taking a photo, the shutter is opened, the light is transmitted to the camera sensor through the lens, the optical signal is converted into an electrical signal, and the camera sensor passes the electrical signal to the ISP for processing, and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on image noise and brightness. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, the ISP may be provided in the camera 193.
摄像头193用于捕获静态图像或视频。物体通过镜头生成光学图像投射到感光元件。感光元件可以是电荷耦合器件(charge coupled device，CCD)或互补金属氧化物半导体(complementary metal-oxide-semiconductor，CMOS)光电晶体管。感光元件把光信号转换成电信号，之后将电信号传递给ISP 转换成数字图像信号。ISP将数字图像信号输出到DSP加工处理。DSP将数字图像信号转换成标准的RGB，YUV等格式的图像信号。在一些实施例中，电子设备可以包括1个或N个摄像头193，N为大于1的正整数。Camera 193 is used to capture still images or video. The object passes through the lens to produce an optical image that is projected onto the photosensitive element. The photosensitive element may be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP Convert into digital image signal. ISP outputs digital image signals to DSP for processing. DSP converts digital image signals into standard RGB, YUV and other format image signals. In some embodiments, the electronic device may include 1 or N cameras 193, where N is a positive integer greater than 1.
数字信号处理器用于处理数字信号，除了可以处理数字图像信号，还可以处理其他数字信号。例如，当电子设备在频点选择时，数字信号处理器用于对频点能量进行傅里叶变换等。Digital signal processors are used to process digital signals. In addition to digital image signals, they can also process other digital signals. For example, when the electronic device selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.
视频编解码器用于对数字视频压缩或解压缩。电子设备可以支持一种或多种视频编解码器。这样，电子设备可以播放或录制多种编码格式的视频，例如：动态图像专家组(moving picture experts group，MPEG)1，MPEG2，MPEG3，MPEG4等。Video codecs are used to compress or decompress digital video. Electronic devices may support one or more video codecs. In this way, electronic devices can play or record videos in multiple encoding formats, such as: Moving Picture Experts Group (MPEG)1, MPEG2, MPEG3, MPEG4, etc.
NPU为神经网络(neural-network，NN)计算处理器，通过借鉴生物神经网络结构，例如借鉴人脑神经元之间传递模式，对输入信息快速处理，还可以不断的自学习。通过NPU可以实现电子设备的智能认知等应用，例如：图像识别，人脸识别，语音识别，文本理解等。NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can continuously learn by itself. Intelligent cognitive applications of electronic devices can be realized through NPU, such as image recognition, face recognition, speech recognition, text understanding, etc.
内部存储器121可以包括一个或多个随机存取存储器(random access memory，RAM)和一个或多个非易失性存储器(non-volatile memory，NVM)。The internal memory 121 may include one or more random access memories (RAM) and one or more non-volatile memories (NVM).
随机存取存储器可以包括静态随机存储器(static random-access memory，SRAM)、动态随机存储器(dynamic random access memory，DRAM)、同步动态随机存储器(synchronous dynamic random access memory,SDRAM)、双倍资料率同步动态随机存取存储器(double data rate synchronous dynamic random access memory,DDR SDRAM，例如第五代DDR SDRAM一般称为DDR5SDRAM)等；Random access memory may include static random-access memory (SRAM), dynamic random-access memory (DRAM), synchronous dynamic random-access memory (SDRAM), double data rate synchronous dynamic random-access memory (DDR SDRAM, for example, the fifth generation DDR SDRAM is generally referred to as DDR5 SDRAM), etc.
非易失性存储器可以包括磁盘存储器件、快闪存储器(flash memory)。Non-volatile memory can include disk storage devices and flash memory.
快闪存储器按照运作原理划分可以包括NOR FLASH、NAND FLASH、3D NAND FLASH等，按照存储单元电位阶数划分可以包括单阶存储单元(single-level cell,SLC)、多阶存储单元(multi-level cell,MLC)、三阶储存单元(triple-level cell,TLC)、四阶储存单元(quad-level cell,QLC)等，按照存储规范划分可以包括通用闪存存储(英文：universal flash storage，UFS)、嵌入式多媒体存储卡(embedded multi media Card，eMMC)等。Flash memory can be divided according to the operating principle to include NOR FLASH, NAND FLASH, 3D NAND FLASH, etc. According to the storage unit potential level, it can include single-level storage cells (single-level cell, SLC), multi-level storage cells (multi-level cell, MLC), third-level storage unit (triple-level cell, TLC), fourth-level storage unit (quad-level cell, QLC), etc., which can include universal flash storage (English: universal flash storage, UFS) according to storage specifications. , embedded multi media card (embedded multi media Card, eMMC), etc.
随机存取存储器可以由处理器110直接进行读写，可以用于存储操作***或其他正在运行中的程序的可执行程序(例如机器指令)，还可以用于存储用户及应用程序的数据等。The random access memory can be directly read and written by the processor 110, can be used to store executable programs (such as machine instructions) of the operating system or other running programs, and can also be used to store user and application data, etc.
非易失性存储器也可以存储可执行程序和存储用户及应用程序的数据等，可以提前加载到随机存取存储器中，用于处理器110直接进行读写。The non-volatile memory can also store executable programs and user and application program data, etc., and can be loaded into the random access memory in advance for direct reading and writing by the processor 110.
外部存储器接口120可以用于连接外部的非易失性存储器，实现扩展电子设备的存储能力。外部的非易失性存储器通过外部存储器接口120与处理器110通信，实现数据存储功能。例如将音乐，视频等文件保存在外部的非易失性存储器中。The external memory interface 120 can be used to connect an external non-volatile memory to expand the storage capacity of the electronic device. The external non-volatile memory communicates with the processor 110 through the external memory interface 120 to implement the data storage function. For example, save music, video and other files in external non-volatile memory.
电子设备可以通过音频模块170，扬声器170A，受话器170B，麦克风170C，耳机接口170D，以及应用处理器等实现音频功能。例如音乐播放，录音等。The electronic device can implement audio functions through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone interface 170D, and the application processor. Such as music playback, recording, etc.
音频模块170用于将数字音频信息转换成模拟音频信号输出，也用于将模拟音频输入转换为数字音频信号。音频模块170还可以用于对音频信号编码和解码。在一些实施例中，音频模块170可以设置于处理器110中，或将音频模块170的部分功能模块设置于处理器110中。The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. Audio module 170 may also be used to encode and decode audio signals. In some embodiments, the audio module 170 may be provided in the processor 110 , or some functional modules of the audio module 170 may be provided in the processor 110 .
扬声器170A，也称“喇叭”，用于将音频电信号转换为声音信号。电子设备可以通过扬声器170A收听音乐，或收听免提通话。Speaker 170A, also called "speaker", is used to convert audio electrical signals into sound signals. The electronic device can listen to music through speaker 170A, or listen to hands-free calls.
受话器170B，也称“听筒”，用于将音频电信号转换成声音信号。当电子设备接听电话或语音信息时，可以通过将受话器170B靠近人耳接听语音。Receiver 170B, also called "earpiece", is used to convert audio electrical signals into sound signals. When the electronic device answers a call or a voice message, the voice can be heard by bringing the receiver 170B close to the human ear.
麦克风170C，也称“话筒”，“传声器”，用于将声音信号转换为电信号。当拨打电话或发送语音信息时，用户可以通过人嘴靠近麦克风170C发声，将声音信号输入到麦克风170C。电子设备可以设置至少一个麦克风170C。在另一些实施例中，电子设备可以设置两个麦克风170C，除了采集声音信号，还可以实现降噪功能。在另一些实施例中，电子设备还可以设置三个，四个或更多麦克风170C，实现采集声音信号，降噪，还可以识别声音来源，实现定向录音功能等。Microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak close to the microphone 170C with the human mouth and input the sound signal to the microphone 170C. The electronic device may be provided with at least one microphone 170C. In other embodiments, the electronic device may be provided with two microphones 170C, which in addition to collecting sound signals, may also implement a noise reduction function. In other embodiments, the electronic device can also be equipped with three, four or more microphones 170C to collect sound signals, reduce noise, identify sound sources, and implement directional recording functions, etc.
耳机接口170D用于连接有线耳机。耳机接口170D可以是USB接口130，也可以是3.5mm的开放移动电子设备平台(open mobile terminal platform，OMTP)标准接口，美国蜂窝电信工业协会(cellular telecommunications industry association of the USA，CTIA)标准接口。The headphone interface 170D is used to connect wired headphones. The headphone interface 170D may be a USB interface 130, or may be a 3.5mm open mobile terminal platform (OMTP) standard interface, or a Cellular Telecommunications Industry Association of the USA (CTIA) standard interface.
压力传感器180A用于感受压力信号，可以将压力信号转换成电信号。在一些实施例中，压力传感器180A可以设置于显示屏194。压力传感器180A的种类很多，如电阻式压力传感器，电感式压力传感器， 电容式压力传感器等。电容式压力传感器可以是包括至少两个具有导电材料的平行板。当有力作用于压力传感器180A，电极之间的电容改变。电子设备根据电容的变化确定压力的强度。当有触摸操作作用于显示屏194，电子设备根据压力传感器180A检测所述触摸操作强度。电子设备也可以根据压力传感器180A的检测信号计算触摸的位置。在一些实施例中，作用于相同触摸位置，但不同触摸操作强度的触摸操作，可以对应不同的操作指令。例如：当有触摸操作强度小于第一压力阈值的触摸操作作用于短消息应用图标时，执行查看短消息的指令。当有触摸操作强度大于或等于第一压力阈值的触摸操作作用于短消息应用图标时，执行新建短消息的指令。The pressure sensor 180A is used to sense pressure signals and can convert the pressure signals into electrical signals. In some embodiments, pressure sensor 180A may be disposed on display screen 194 . There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, Capacitive pressure sensor, etc. A capacitive pressure sensor may include at least two parallel plates of conductive material. When a force is applied to pressure sensor 180A, the capacitance between the electrodes changes. Electronics determine the intensity of the pressure based on changes in capacitance. When a touch operation is performed on the display screen 194, the electronic device detects the strength of the touch operation according to the pressure sensor 180A. The electronic device may also calculate the touched position based on the detection signal of the pressure sensor 180A. In some embodiments, touch operations acting on the same touch location but with different touch operation intensities may correspond to different operation instructions. For example: when a touch operation with a touch operation intensity less than the first pressure threshold is applied to the short message application icon, an instruction to view the short message is executed. When a touch operation with a touch operation intensity greater than or equal to the first pressure threshold is applied to the short message application icon, an instruction to create a new short message is executed.
陀螺仪传感器180B可以用于确定电子设备的运动姿态。在一些实施例中，可以通过陀螺仪传感器180B确定电子设备围绕三个轴(即，x，y和z轴)的角速度。陀螺仪传感器180B可以用于拍摄防抖。示例性的，当按下快门，陀螺仪传感器180B检测电子设备抖动的角度，根据角度计算出镜头模组需要补偿的距离，让镜头通过反向运动抵消电子设备的抖动，实现防抖。陀螺仪传感器180B还可以用于导航，体感游戏场景。The gyro sensor 180B can be used to determine the motion posture of the electronic device. In some embodiments, the angular velocity of the electronic device about three axes (ie, x, y, and z axes) may be determined by gyro sensor 180B. The gyro sensor 180B can be used for image stabilization. For example, when the shutter is pressed, the gyro sensor 180B detects the angle at which the electronic device shakes, and calculates the distance that the lens module needs to compensate based on the angle, so that the lens can offset the shake of the electronic device through reverse movement to achieve anti-shake. The gyro sensor 180B can also be used for navigation and somatosensory game scenes.
气压传感器180C用于测量气压。在一些实施例中，电子设备通过气压传感器180C测得的气压值计算海拔高度，辅助定位和导航。The air pressure sensor 180C is used to measure air pressure. In some embodiments, the electronic device calculates the altitude through the air pressure value measured by the air pressure sensor 180C to assist positioning and navigation.
磁传感器180D包括霍尔传感器。电子设备可以利用磁传感器180D检测翻盖皮套的开合。在一些实施例中，当电子设备是翻盖机时，电子设备可以根据磁传感器180D检测翻盖的开合。进而根据检测到的皮套的开合状态或翻盖的开合状态，设置翻盖自动解锁等特性。Magnetic sensor 180D includes a Hall sensor. The electronic device can use the magnetic sensor 180D to detect the opening and closing of the flip holster. In some embodiments, when the electronic device is a flip machine, the electronic device may detect opening and closing of the flip according to the magnetic sensor 180D. Then, based on the detected opening and closing status of the leather case or the opening and closing status of the flip cover, features such as automatic unlocking of the flip cover are set.
加速度传感器180E可检测电子设备在各个方向上(一般为三轴)加速度的大小。当电子设备静止时可检测出重力的大小及方向。还可以用于识别电子设备姿态，应用于横竖屏切换，计步器等应用。The acceleration sensor 180E can detect the magnitude of the acceleration of the electronic device in all directions (generally three axes). When the electronic device is stationary, it can detect the magnitude and direction of gravity. It can also be used to identify the posture of the electronic device and is applied to applications such as horizontal and vertical screen switching and pedometers.
距离传感器180F，用于测量距离。电子设备可以通过红外或激光测量距离。在一些实施例中，拍摄场景，电子设备可以利用距离传感器180F测距以实现快速对焦。Distance sensor 180F for measuring distance. Electronic devices can measure distance via infrared or laser. In some embodiments, when shooting a scene, the electronic device can utilize the distance sensor 180F to measure distance to achieve fast focusing.
接近光传感器180G可以包括例如发光二极管(LED)和光检测器，例如光电二极管。发光二极管可以是红外发光二极管。电子设备通过发光二极管向外发射红外光。电子设备使用光电二极管检测来自附近物体的红外反射光。当检测到充分的反射光时，可以确定电子设备附近有物体。当检测到不充分的反射光时，电子设备可以确定电子设备附近没有物体。电子设备可以利用接近光传感器180G检测用户手持电子设备贴近耳朵通话，以便自动熄灭屏幕达到省电的目的。接近光传感器180G也可用于皮套模式，口袋模式自动解锁与锁屏。Proximity light sensor 180G may include, for example, a light emitting diode (LED) and a light detector, such as a photodiode. The light emitting diode may be an infrared light emitting diode. Electronic devices emit infrared light through light-emitting diodes. Electronic devices use photodiodes to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the electronic device. When insufficient reflected light is detected, the electronic device may determine that there is no object near the electronic device. Electronic devices can use the proximity light sensor 180G to detect when the user holds the electronic device close to the ear and talk, so that the screen can be automatically turned off to save power. The proximity light sensor 180G can also be used in holster mode, and pocket mode automatically unlocks and locks the screen.
环境光传感器180L用于感知环境光亮度。电子设备可以根据感知的环境光亮度自适应调节显示屏194亮度。环境光传感器180L也可用于拍照时自动调节白平衡。环境光传感器180L还可以与接近光传感器180G配合，检测电子设备是否在口袋里，以防误触。The ambient light sensor 180L is used to sense ambient light brightness. The electronic device can adaptively adjust the brightness of the display screen 194 based on perceived ambient light brightness. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 180L can also cooperate with the proximity light sensor 180G to detect whether the electronic device is in the pocket to prevent accidental touching.
指纹传感器180H用于采集指纹。电子设备可以利用采集的指纹特性实现指纹解锁，访问应用锁，指纹拍照，指纹接听来电等。Fingerprint sensor 180H is used to collect fingerprints. Electronic devices can use the collected fingerprint characteristics to unlock fingerprints, access application locks, take photos with fingerprints, answer incoming calls with fingerprints, etc.
温度传感器180J用于检测温度。在一些实施例中，电子设备利用温度传感器180J检测的温度，执行温度处理策略。例如，当温度传感器180J上报的温度超过阈值，电子设备执行降低位于温度传感器180J附近的处理器的性能，以便降低功耗实施热保护。在另一些实施例中，当温度低于另一阈值时，电子设备对电池142加热，以避免低温导致电子设备异常关机。在其他一些实施例中，当温度低于又一阈值时，电子设备对电池142的输出电压执行升压，以避免低温导致的异常关机。Temperature sensor 180J is used to detect temperature. In some embodiments, the electronic device uses the temperature detected by the temperature sensor 180J to execute the temperature processing strategy. For example, when the temperature reported by the temperature sensor 180J exceeds a threshold, the electronic device reduces the performance of a processor located near the temperature sensor 180J in order to reduce power consumption and implement thermal protection. In other embodiments, when the temperature is lower than another threshold, the electronic device heats the battery 142 to prevent the low temperature from causing abnormal shutdown of the electronic device. In some other embodiments, when the temperature is lower than another threshold, the electronic device performs boosting on the output voltage of the battery 142 to avoid abnormal shutdown caused by low temperature.
触摸传感器180K，也称“触控器件”。触摸传感器180K可以设置于显示屏194，由触摸传感器180K与显示屏194组成触摸屏，也称“触控屏”。触摸传感器180K用于检测作用于其上或附近的触摸操作。触摸传感器可以将检测到的触摸操作传递给应用处理器，以确定触摸事件类型。可以通过显示屏194提供与触摸操作相关的视觉输出。在另一些实施例中，触摸传感器180K也可以设置于电子设备的表面，与显示屏194所处的位置不同。Touch sensor 180K, also known as "touch device". The touch sensor 180K can be disposed on the display screen 194. The touch sensor 180K and the display screen 194 form a touch screen, which is also called a "touch screen". The touch sensor 180K is used to detect a touch operation on or near the touch sensor 180K. The touch sensor can pass the detected touch operation to the application processor to determine the touch event type. Visual output related to the touch operation may be provided through display screen 194 . In other embodiments, the touch sensor 180K may also be disposed on the surface of the electronic device at a location different from that of the display screen 194 .
骨传导传感器180M可以获取振动信号。在一些实施例中，骨传导传感器180M可以获取人体声部振动骨块的振动信号。骨传导传感器180M也可以接触人体脉搏，接收血压跳动信号。在一些实施例中，骨传导传感器180M也可以设置于耳机中，结合成骨传导耳机。音频模块170可以基于所述骨传导传感器180M获取的声部振动骨块的振动信号，解析出语音信号，实现语音功能。应用处理器可以基于所述骨传导传感器180M获取的血压跳动信号解析心率信息，实现心率检测功能。Bone conduction sensor 180M can acquire vibration signals. In some embodiments, the bone conduction sensor 180M can acquire the vibration signal of the vibrating bone mass of the human body's vocal part. The bone conduction sensor 180M can also contact the human body's pulse and receive blood pressure beating signals. In some embodiments, the bone conduction sensor 180M can also be provided in an earphone and combined into a bone conduction earphone. The audio module 170 can analyze the voice signal based on the vibration signal of the vocal vibrating bone obtained by the bone conduction sensor 180M to implement the voice function. The application processor can analyze the heart rate information based on the blood pressure beating signal acquired by the bone conduction sensor 180M to implement the heart rate detection function.
按键190包括开机键，音量键等。按键190可以是机械按键。也可以是触摸式按键。电子设备可以接收按键输入，产生与电子设备的用户设置以及功能控制有关的键信号输入。 The buttons 190 include a power button, a volume button, etc. Key 190 may be a mechanical key. It can also be a touch button. The electronic device can receive key input and generate key signal input related to user settings and function control of the electronic device.
马达191可以产生振动提示。马达191可以用于来电振动提示，也可以用于触摸振动反馈。例如，作用于不同应用(例如拍照，音频播放等)的触摸操作，可以对应不同的振动反馈效果。作用于显示屏194不同区域的触摸操作，马达191也可对应不同的振动反馈效果。不同的应用场景(例如：时间提醒，接收信息，闹钟，游戏等)也可以对应不同的振动反馈效果。触摸振动反馈效果还可以支持自定义。The motor 191 can generate vibration prompts. The motor 191 can be used for vibration prompts for incoming calls and can also be used for touch vibration feedback. For example, touch operations for different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. The motor 191 can also respond to different vibration feedback effects for touch operations in different areas of the display screen 194 . Different application scenarios (such as time reminders, receiving information, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also be customized.
指示器192可以是指示灯，可以用于指示充电状态，电量变化，也可以用于指示消息，未接来电，通知等。The indicator 192 may be an indicator light, which may be used to indicate the charging status, power changes, messages, missed calls, notifications, etc.
SIM卡接口195用于连接SIM卡。SIM卡可以通过***SIM卡接口195，或从SIM卡接口195拔出，实现和电子设备的接触和分离。电子设备可以支持1个或N个SIM卡接口，N为大于1的正整数。SIM卡接口195可以支持Nano SIM卡，Micro SIM卡，SIM卡等。同一个SIM卡接口195可以同时***多张卡。所述多张卡的类型可以相同，也可以不同。SIM卡接口195也可以兼容不同类型的SIM卡。SIM卡接口195也可以兼容外部存储卡。电子设备通过SIM卡和网络交互，实现通话以及数据通信等功能。在一些实施例中，电子设备采用eSIM，即：嵌入式SIM卡。eSIM卡可以嵌在电子设备中，不能和电子设备分离。The SIM card interface 195 is used to connect a SIM card. The SIM card can be inserted into the SIM card interface 195 or pulled out from the SIM card interface 195 to realize contact and separation from the electronic device. The electronic device can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. SIM card interface 195 can support Nano SIM card, Micro SIM card, SIM card, etc. Multiple cards can be inserted into the same SIM card interface 195 at the same time. The types of the plurality of cards may be the same or different. The SIM card interface 195 is also compatible with different types of SIM cards. The SIM card interface 195 is also compatible with external memory cards. Electronic devices interact with the network through SIM cards to implement functions such as calls and data communications. In some embodiments, the electronic device uses an eSIM, that is, an embedded SIM card. The eSIM card can be embedded in the electronic device and cannot be separated from the electronic device.
图16A和图16B为本申请实施例提供的电子设备的软件架构的另一个示例性示意图。FIG. 16A and FIG. 16B are another exemplary schematic diagram of the software architecture of the electronic device provided by the embodiment of the present application.
电子设备的软件***可以采用分层架构，事件驱动架构，微核架构，微服务架构，或云架构。本发明实施例以分层架构的Android***为例，示例性说明电子设备的软件结构。Software systems of electronic devices can adopt layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. The embodiment of the present invention takes the Android system with a layered architecture as an example to illustrate the software structure of the electronic device.
分层架构将软件分成若干个层，每一层都有清晰的角色和分工。层与层之间通过软件接口通信。在一些实施例中，将Android***分为四层，从上至下分别为应用程序层，应用程序框架层，安卓运行时(Android runtime)和***库，以及内核层。The layered architecture divides the software into several layers, and each layer has clear roles and division of labor. The layers communicate through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: application layer, application framework layer, Android runtime and system libraries, and kernel layer.
应用程序层可以包括一系列应用程序包。The application layer can include a series of application packages.
如图16A所示，应用程序包可以包括相机，图库，日历，通话，地图，导航，WLAN，蓝牙，音乐，视频，短信息等应用程序。As shown in Figure 16A, the application package can include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message, etc.
应用程序框架层为应用程序层的应用程序提供应用编程接口(application programming interface，API)和编程框架。应用程序框架层包括一些预先定义的函数。The application framework layer provides application programming interface (API) and programming framework for the applications in the application layer. The application framework layer includes some predefined functions.
如图16A所示，应用程序框架层可以包括窗口管理服务，活动管理服务、内容提供器，视图***，电话管理器，资源管理器，通知管理器等。As shown in Figure 16A, the application framework layer may include window management services, activity management services, content providers, view systems, phone managers, resource managers, notification managers, etc.
窗口管理服务用于管理窗口程序，可以确定不同应用程序的窗口的窗口Z序。The window management service is used to manage window programs and can determine the window order of windows of different applications.
可选地，在本申请的一些实施方式中，统一渲染进程可以从窗口管理服务器获取不同应用程序的窗口的窗口Z序，进而确定不同渲染树中的被其他应用程序的界面遮蔽的视图对应的渲染节点。Optionally, in some embodiments of the present application, the unified rendering process may obtain the window Z order of windows of different applications from the window management server, and then determine the rendering nodes corresponding to the views obscured by the interfaces of other applications in different rendering trees.
可选地，在本申请一些实施方式中，窗口管理服务可以确定被完全遮蔽区域，并将被完全遮蔽区域发送到应用程序的UI线程。Optionally, in some embodiments of the present application, the window management service may determine the completely obscured area and send the completely obscured area to the UI thread of the application.
内容提供器用来存放和获取数据，并使这些数据可以被应用程序访问。所述数据可以包括视频，图像，音频，拨打和接听的电话，浏览历史和书签，电话簿等。Content providers are used to store and retrieve data and make this data accessible to applications. Said data can include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.
视图***包括可视控件，例如显示文字的控件，显示图片的控件等。视图***可用于构建应用程序。显示界面可以由一个或多个视图组成的。例如，包括短信通知图标的显示界面，可以包括显示文字的视图以及显示图片的视图。The view system includes visual controls, such as controls that display text, controls that display pictures, etc. A view system can be used to build applications. The display interface can be composed of one or more views. For example, a display interface including a text message notification icon may include a view for displaying text and a view for displaying pictures.
视图***中还包括遮蔽模块，遮蔽模块用于确定被完全遮蔽的视图，即被同应用程序的其他视图遮蔽的视图，以及被其他应用程序的界面完全遮蔽的视图。The view system also includes an occlusion module, which is used to determine views that are completely occluded, that is, views that are occluded by other views of the same application, and views that are completely occluded by the interfaces of other applications.
电话管理器用于提供电子设备的通信功能。例如通话状态的管理(包括接通，挂断等)。Telephone managers are used to provide communication functions of electronic devices. For example, call status management (including connected, hung up, etc.).
资源管理器为应用程序提供各种资源，比如本地化字符串，图标，图片，布局文件，视频文件等等。The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.
通知管理器使应用程序可以在状态栏中显示通知信息，可以用于传达告知类型的消息，可以短暂停留后自动消失，无需用户交互。比如通知管理器被用于告知下载完成，消息提醒等。通知管理器还可以是以图表或者滚动条文本形式出现在***顶部状态栏的通知，例如后台运行的应用程序的通知，还可以是以对话窗口形式出现在屏幕上的通知。例如在状态栏提示文本信息，发出提示音，电子设备振动，指示灯闪烁等。The notification manager allows applications to display notification information in the status bar, which can be used to convey notification-type messages and can automatically disappear after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be notifications that appear in the status bar at the top of the system in the form of charts or scroll bar text, such as notifications for applications running in the background, or notifications that appear on the screen in the form of conversation windows. For example, text information is prompted in the status bar, a beep sounds, the electronic device vibrates, the indicator light flashes, etc.
Android Runtime包括核心库和虚拟机。Android runtime负责安卓***的调度和管理。Android Runtime includes core libraries and virtual machines. Android runtime is responsible for the scheduling and management of the Android system.
核心库包含两部分：一部分是java语言需要调用的功能函数，另一部分是安卓的核心库。The core library contains two parts: one is the functional functions that need to be called by the Java language, and the other is the core library of Android.
应用程序层和应用程序框架层运行在虚拟机中。虚拟机将应用程序层和应用程序框架层的java文件执行为二进制文件。虚拟机用于执行对象生命周期的管理，堆栈管理，线程管理，安全和异常的管理，以及 垃圾回收等功能。The application layer and application framework layer run in virtual machines. The virtual machine executes the java files of the application layer and application framework layer as binary files. The virtual machine is used to perform object life cycle management, stack management, thread management, security and exception management, and Garbage collection and other functions.
***库可以包括多个功能模块。例如：浏览器引擎(webkit)、表面合成器，硬件合成策略模块，媒体库(Media Libraries)，三维图形处理库(例如：OpenGL ES)，2D图形引擎(例如：SGL)等。System libraries can include multiple functional modules. For example: browser engine (webkit), surface compositor, hardware synthesis strategy module, media libraries (Media Libraries), 3D graphics processing library (for example: OpenGL ES), 2D graphics engine (for example: SGL), etc.
媒体库支持多种常用的音频，视频格式回放和录制，以及静态图像文件等。媒体库可以支持多种音视频编码格式，例如:MPEG4，H.264，MP3，AAC，AMR，JPG，PNG等。The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.
三维图形处理库用于实现三维图形绘图，图像渲染，合成，和图层处理等。The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, composition, and layer processing.
2D图形引擎是2D绘图的绘图引擎。2D Graphics Engine is a drawing engine for 2D drawing.
内核层是硬件和软件之间的层。内核层包括显示驱动，摄像头驱动，音频驱动，传感器驱动等。The kernel layer is the layer between hardware and software. The kernel layer includes display driver, camera driver, audio driver, sensor driver, etc.
如图16B所示，与图16A所示内容不同的是，视图***还可以包括统一渲染进程，统一渲染进程用于接收不同应用程序的渲染树，并将渲染树合并为目标渲染树后，基于目标渲染树生成位图。As shown in FIG. 16B , different from the content shown in FIG. 16A , the view system may further include a unified rendering process, which is used to receive rendering trees of different applications, merge the rendering trees into a target rendering tree, and then generate a bitmap based on the target rendering tree.
可选地，在本申请一些实施方式中，由于统一渲染进程直接是基于目标渲染树生成位图的，故在图16B所示的软件架构中，***库可以不包括表面合成器，或者在生成界面的过程中，表面合成器不执行图层合成。Optionally, in some embodiments of the present application, since the unified rendering process directly generates bitmaps based on the target rendering tree, in the software architecture shown in Figure 16B, the system library may not include a surface compositor, or it may be generated when The surface compositor does not perform layer compositing during the interface.
可选地，在本申请一些实施方式中，显示子***可以认为是包括部分显示驱动、部分显示的功能模块。Optionally, in some embodiments of the present application, the display subsystem can be considered as a functional module including part of the display driver and part of the display.
上述实施例中所用，根据上下文，术语“当…时”可以被解释为意思是“如果…”或“在…后”或“响应于确定…”或“响应于检测到…”。类似地，根据上下文，短语“在确定…时”或“如果检测到(所陈述的条件或事件)”可以被解释为意思是“如果确定…”或“响应于确定…”或“在检测到(所陈述的条件或事件)时”或“响应于检测到(所陈述的条件或事件)”。As used in the above embodiments, the term "when" may be interpreted to mean "if..." or "after" or "in response to determining..." or "in response to detecting..." depending on the context. Similarly, depending on the context, the phrase "when determining..." or "if (stated condition or event) is detected" may be interpreted to mean "if it is determined..." or "in response to determining..." or "on detecting (stated condition or event)” or “in response to detecting (stated condition or event)”.
在上述实施例中，可以全部或部分地通过软件、硬件、固件或者其任意组合来实现。当使用软件实现时，可以全部或部分地以计算机程序产品的形式实现。该计算机程序产品包括一个或多个计算机指令。在计算机上加载和执行该计算机程序指令时，全部或部分地产生按照本申请实施例该的流程或功能。该计算机可以是通用计算机、专用计算机、计算机网络、或者其他可编程装置。该计算机指令可以存储在计算机可读存储介质中，或者从一个计算机可读存储介质向另一个计算机可读存储介质传输，例如，该计算机指令可以从一个网站站点、计算机、服务器或数据中心通过有线(例如同轴电缆、光纤、数字用户线)或无线(例如红外、无线、微波等)方式向另一个网站站点、计算机、服务器或数据中心进行传输。该计算机可读存储介质可以是计算机能够存取的任何可用介质或者是包含一个或多个可用介质集成的服务器、数据中心等数据存储设备。该可用介质可以是磁性介质，(例如，软盘、硬盘、磁带)、光介质(例如DVD)、或者半导体介质(例如固态硬盘)等。In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on the computer, the processes or functions according to the embodiments of the present application are generated in whole or in part. The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable device. The computer instructions may be stored in or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted over a wired connection from a website, computer, server, or data center (such as coaxial cable, optical fiber, digital subscriber line) or wireless (such as infrared, wireless, microwave, etc.) means to transmit to another website, computer, server or data center. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center integrated with one or more available media. The available media may be magnetic media (eg, floppy disk, hard disk, magnetic tape), optical media (eg, DVD), or semiconductor media (eg, solid state drive), etc.
本领域普通技术人员可以理解实现上述实施例方法中的全部或部分流程，该流程可以由计算机程序来指令相关的硬件完成，该程序可存储于计算机可读取存储介质中，该程序在执行时，可包括如上述各方法实施例的流程。而前述的存储介质包括：ROM或随机存储记忆体RAM、磁碟或者光盘等各种可存储程序代码的介质。 Those of ordinary skill in the art can understand that all or part of the process of implementing the method of the above embodiments can be completed by instructing relevant hardware through a computer program. The program can be stored in a computer-readable storage medium. When the program is executed, , may include the processes of the above method embodiments. The aforementioned storage media include: ROM, random access memory (RAM), magnetic disks, optical disks and other media that can store program codes.

Claims (23)

  1. 一种界面生成方法，其特征在于，应用于包括屏幕的电子设备，所述方法包括：A method for generating an interface, characterized in that it is applied to an electronic device including a screen, and the method comprises:
    确定被第一应用内的其他视图完全遮蔽的视图包括第一视图；Determining that views that are completely obscured by other views in the first application include the first view;
    生成第一渲染树，所述第一渲染树不包括第一渲染节点，所述第一渲染节点为所述第一视图对应的渲染节点，渲染节点保存有对应的视图的绘制指令；Generate a first rendering tree, the first rendering tree does not include a first rendering node, the first rendering node is a rendering node corresponding to the first view, and the rendering node stores a drawing instruction of the corresponding view;
    基于所述第一渲染树生成第一界面，所述第一界面包括所述第一应用的界面。A first interface is generated based on the first rendering tree, and the first interface includes an interface of the first application.
  2. 根据权利要求1所述的方法，其特征在于，所述确定被第一应用内的其他视图完全遮蔽的视图包括第一视图，具体包括：The method of claim 1, wherein determining the view that is completely obscured by other views in the first application includes the first view, specifically including:
    在遍历所述第一应用的视图的过程中，确定第二视图的位置为第二位置；In the process of traversing the views of the first application, determine the position of the second view as the second position;
    在遍历到所述第一视图时，确定所述第一视图的位置为第一位置，基于所述第二位置和所述第一位置确定所述第一视图为被所述第一应用内的其他视图完全遮蔽的视图。When traversing to the first view, the position of the first view is determined to be the first position, and the first view is determined to be the view within the first application based on the second position and the first position. A view that is completely obscured by other views.
  3. 根据权利要求2所述的方法，其特征在于，在所述基于所述第二位置和所述第一位置确定所述第一视图为被完全遮蔽的视图后，所述方法还包括：The method of claim 2, wherein after determining that the first view is a completely occluded view based on the second position and the first position, the method further includes:
    不保存第一视图的绘制指令至所述第一渲染树中；Do not save the drawing instructions of the first view to the first rendering tree;
    跳过所述第一视图的子视图以继续遍历，所述第一渲染树不包括所述第一视图的子视图对应的渲染节点。Skip the subviews of the first view to continue traversing, and the first rendering tree does not include rendering nodes corresponding to the subviews of the first view.
  4. 根据权利要求1所述的方法，其特征在于，所述确定被第一应用内的其他视图完全遮蔽的视图包括第一视图，具体包括：The method of claim 1, wherein determining the view that is completely obscured by other views in the first application includes the first view, specifically including:
    在遍历所述第一应用的视图的过程中，确定所述第一视图的位置为第一位置后，确定第二视图的位置为第二位置；In the process of traversing the views of the first application, after determining the position of the first view as the first position, determine the position of the second view as the second position;
    基于所述第一位置和所述第二位置确定所述第一视图被所述第二视图完全遮蔽，记录所述第一视图的标识至第一参数中。Based on the first position and the second position, it is determined that the first view is completely blocked by the second view, and the identification of the first view is recorded into a first parameter.
  5. 根据权利要求4所述的方法，其特征在于，所述生成第一渲染树，具体包括：The method according to claim 4, characterized in that generating the first rendering tree specifically includes:
    遍历所述第一应用的视图后，生成第二渲染树，所述第二渲染树包括第一渲染节点；After traversing the views of the first application, generate a second rendering tree, where the second rendering tree includes a first rendering node;
    基于所述第一参数删除所述第二渲染树中的第一渲染节点得到所述第一渲染树。Delete the first rendering node in the second rendering tree based on the first parameter to obtain the first rendering tree.
  6. 根据权利要求1所述的方法，其特征在于，所述确定被第一应用内的其他视图完全遮蔽的视图包括第一视图，具体包括：The method of claim 1, wherein determining the view that is completely obscured by other views in the first application includes the first view, specifically including:
    遍历所述第一应用的视图后，生成第二渲染树，所述第二渲染树包括所述第一渲染节点和第二渲染节点；After traversing the views of the first application, generate a second rendering tree, where the second rendering tree includes the first rendering node and a second rendering node;
    基于所述第一渲染节点的渲染属性和所述第二渲染节点的渲染属性确定所述第一视图被第二视图完全遮蔽，所述第二视图与所述第二渲染节点对应。It is determined based on the rendering attributes of the first rendering node and the rendering attributes of the second rendering node that the first view is completely obscured by a second view corresponding to the second rendering node.
  7. 根据权利要求6所述的方法，其特征在于，所述生成第一渲染树，具体包括：The method according to claim 6, characterized in that generating the first rendering tree specifically includes:
    删除所述第二渲染树中的第二渲染节点得到所述第一渲染树。Delete the second rendering node in the second rendering tree to obtain the first rendering tree.
  8. 根据权利要求1-7中任一项所述的方法，其特征在于，所述方法还包括：The method according to any one of claims 1-7, characterized in that the method further includes:
    合并所述第一渲染树与第二渲染树得到目标渲染树，所述第二渲染树是第二应用生成的，所述第二渲染树用于生成所述第二应用的界面，所述第一界面还包括所述第二应用的界面，所述目标渲染树不包括所述第三渲染节点，所述第三渲染节点与第三视图对应，所述第三视图为所述第一应用的视图，所述第三视图被所述第二应用的界面完全遮蔽；Merging the first rendering tree and the second rendering tree to obtain a target rendering tree, the second rendering tree is generated by a second application, the second rendering tree is used to generate an interface of the second application, and the third An interface further includes an interface of the second application, the target rendering tree does not include the third rendering node, the third rendering node corresponds to a third view, and the third view is the first application's interface. View, the third view is completely obscured by the interface of the second application;
    所述基于所述第一渲染树生成第一界面，具体包括：基于所述目标渲染树生成所述第一界面。Generating the first interface based on the first rendering tree specifically includes: generating the first interface based on the target rendering tree.
  9. 根据权利要求8所述的方法，其特征在于，在所述基于所述目标渲染树生成第一界面前，所述方法还包括： The method according to claim 8, characterized in that before generating the first interface based on the target rendering tree, the method further includes:
    确定被所述第二应用的界面完全遮蔽的视图对应的渲染节点包括所述第三渲染节点；Determine that the rendering node corresponding to the view completely obscured by the interface of the second application includes the third rendering node;
    所述合并所述第一渲染树与第二渲染树得到目标渲染树，具体包括：在合并所述第一渲染树与所述第二渲染树得到目标渲染树的过程中，从所述第一渲染树或所述目标渲染树中删除所述第三渲染节点。The merging of the first rendering tree and the second rendering tree to obtain a target rendering tree specifically includes: in the process of merging the first rendering tree and the second rendering tree to obtain the target rendering tree, from the first The third rendering node is deleted from the rendering tree or the target rendering tree.
  10. 根据权利要求9所述的方法，其特征在于，在所述基于所述目标渲染树生成第一界面前，所述方法还包括：The method according to claim 9, characterized in that before generating the first interface based on the target rendering tree, the method further includes:
    确定被所述第二应用的界面完全遮蔽的视图对应的渲染节点包括所述第三渲染节点；Determine that the rendering node corresponding to the view completely obscured by the interface of the second application includes the third rendering node;
    所述合并所述第一渲染树与第二渲染树得到目标渲染树，具体包括：合并所述第一渲染树与所述第二渲染树得到第三渲染树；从所述第三渲染树中删除所述第三渲染节点得到所述目标渲染树。The merging of the first rendering tree and the second rendering tree to obtain a target rendering tree specifically includes: merging the first rendering tree and the second rendering tree to obtain a third rendering tree; Delete the third rendering node to obtain the target rendering tree.
  11. 根据权利要求9或10所述的方法，其特征在于，所述确定被完全遮蔽的视图对应的渲染节点包括所述第三渲染节点，具体包括：The method according to claim 9 or 10, wherein determining the rendering node corresponding to the completely occluded view includes the third rendering node, specifically including:
    基于所述第一应用的窗口Z序、所述第二应用的窗口Z序、所述第一渲染树和所述第二渲染树确定被所述第二应用的界面完全遮蔽的第一应用的视图对应的渲染节点包括所述第三渲染节点；Determine the first application that is completely obscured by the interface of the second application based on the window Z-order of the first application, the window Z-order of the second application, the first rendering tree, and the second rendering tree. The rendering node corresponding to the view includes the third rendering node;
    或者，确定完全遮蔽区域，并基于所述第一渲染树和所述完全遮蔽区域确定被所述第二应用的界面完全遮蔽的第一应用的视图对应的渲染节点包括所述第三渲染节点，所述完全遮蔽区域为所述第二应用的界面遮蔽第一应用的界面的区域。Or, determine a completely blocked area, and determine based on the first rendering tree and the completely blocked area that the rendering node corresponding to the view of the first application that is completely blocked by the interface of the second application includes the third rendering node, The completely shielded area is an area in which the interface of the second application blocks the interface of the first application.
  12. 根据权利要求1-7中任一项所述的方法，其特征在于，所述方法还包括：The method according to any one of claims 1-7, characterized in that the method further includes:
    所述第一界面还包括第二应用的界面，所述第二应用的界面完全遮蔽所述第一应用的界面的第一区域，确定所述第一区域；The first interface also includes an interface of a second application, the interface of the second application completely blocks the first area of the interface of the first application, and the first area is determined;
    基于所述第一区域确定被所述第二应用的界面完全遮蔽的视图包括第四视图，所述第一渲染树不包括第四渲染节点，所述第四渲染节点与所述第四视图对应。It is determined based on the first area that the view completely occluded by the interface of the second application includes a fourth view, the first rendering tree does not include a fourth rendering node, and the fourth rendering node corresponds to the fourth view. .
  13. 根据权利要求1-12所述的方法，其特征在于，所述方法还包括：The method according to claims 1-12, characterized in that the method further includes:
    确定被部分遮蔽的视图包括第五视图；determining that the partially occluded views include a fifth view;
    在第五渲染节点中的绘制指令列表中增加裁剪指令，所述裁剪指令用于指示不渲染所述第五视图中被部分遮蔽的内容，所述第五渲染节点与所述第五视图对应。A cropping instruction is added to the drawing instruction list in the fifth rendering node, where the cropping instruction is used to instruct not to render the partially occluded content in the fifth view. The fifth rendering node corresponds to the fifth view.
  14. 一种界面生成方法，其特征在于，应用于包括屏幕的电子设备，所述电子设备上运行有第一应用、第二应用和第一进程，所述方法包括：An interface generation method, characterized in that it is applied to an electronic device including a screen, and a first application, a second application and a first process are run on the electronic device, and the method includes:
    所述第一应用向所述第一进程发送第一渲染树，所述第一渲染树用于生成所述第一应用的界面；The first application sends a first rendering tree to the first process, and the first rendering tree is used to generate an interface of the first application;
    所述第二应用向所述第一进程发送第二渲染树，所述第二渲染树用于生成所述第二应用的界面；The second application sends a second rendering tree to the first process, and the second rendering tree is used to generate an interface of the second application;
    所述第一进程基于所述第一渲染树和所述第二渲染树生成目标渲染树，所述目标渲染树用于生成第一界面，所述第一界面包括所述第一应用的界面和所述第二应用的界面；The first process generates a target rendering tree based on the first rendering tree and the second rendering tree. The target rendering tree is used to generate a first interface. The first interface includes the interface of the first application and The interface of the second application;
    所述第一进程确定第一区域后，基于所述第一区域删除第一渲染节点，所述第二应用的界面完全遮蔽所述第一应用的界面的所述第一区域，所述目标渲染树不包括所述第一渲染节点，所述第一渲染节点为所述第一渲染树中的渲染节点，所述第一渲染节点对应的第一视图位于所述第一区域中，渲染节点保存有对应的视图的绘制指令；After the first process determines the first area, the first rendering node is deleted based on the first area. The interface of the second application completely blocks the first area of the interface of the first application. The target rendering The tree does not include the first rendering node, the first rendering node is a rendering node in the first rendering tree, the first view corresponding to the first rendering node is located in the first area, and the rendering node saves There are drawing instructions for corresponding views;
    所述第一进程基于所述目标渲染树生成所述第一界面。The first process generates the first interface based on the target rendering tree.
  15. 根据权利要求14所述的方法，其特征在于，在所述第一应用程序发送第一渲染树前，所述方法还包括：The method according to claim 14, characterized in that, before the first application sends the first rendering tree, the method further includes:
    所述第一应用通过遍历视图确定第三渲染树，所述第三渲染树用于生成所述第一应用的界面；The first application determines a third rendering tree by traversing views, and the third rendering tree is used to generate an interface of the first application;
    所述第一应用确定被所述第一应用程序的其他视图完全遮蔽的视图包括第二视图；the first application determines that views that are completely occluded by other views of the first application include a second view;
    所述第一应用删除所述第三渲染树中的第二渲染节点得到所述第一渲染树，所述第二渲染节点与所述第二视图对应。The first application deletes the second rendering node in the third rendering tree to obtain the first rendering tree, and the second rendering node corresponds to the second view.
  16. 根据权利要求14所述的方法，其特征在于，所述方法还包括： The method of claim 14, further comprising:
    所述第一进程删除第二渲染节点，所述第二渲染节点为所述第一渲染树中的渲染节点，所述第二渲染节点与所述第二视图对应，所述第二视图被所述第一应用的其他视图完全遮蔽。The first process deletes a second rendering node, where the second rendering node is a rendering node in the first rendering tree, the second rendering node corresponds to the second view, and the second view is completely obscured by other views of the first application.
  17. 一种界面生成方法，其特征在于，应用于包括屏幕的电子设备，所述方法包括：An interface generation method, characterized in that it is applied to an electronic device including a screen, and the method includes:
    所述电子设备在所述屏幕显示第一界面，在所述第一界面中，所述第一应用程序的界面全屏显示第一内容，所述第二应用程序的界面在所述第一应用程序的界面上悬浮显示第二内容，所述第二应用程序的界面在所述第一界面中位于第一区域；The electronic device displays a first interface on the screen. In the first interface, the interface of the first application program displays the first content in full screen. The interface of the second application program displays the first content on the screen. The second content is displayed suspended on the interface, and the interface of the second application is located in the first area of the first interface;
    在显示所述第一界面的过程中，所述电子设备生成第一渲染树，所述第一渲染树用于生成所述第一应用的界面；In the process of displaying the first interface, the electronic device generates a first rendering tree, where the first rendering tree is used to generate an interface of the first application;
    所述电子设备在所述屏幕显示第二界面，在所述第二界面中，所述第一应用程序的界面全屏显示所述第一内容，所述第二应用程序的界面在所述第一应用程序上悬浮显示所述第二内容，所述第二应用程序在所述第一界面中位于第二区域，所述第一区域与所述区域不同；The electronic device displays a second interface on the screen. In the second interface, the interface of the first application program displays the first content in full screen. The interface of the second application program displays the first content on the first interface. The second content is displayed suspended on the application, the second application is located in a second area in the first interface, and the first area is different from the area;
    在显示所述第二界面的过程中，所述电子设备生成第二渲染树，所述第二渲染树用于生成所述第一应用的界面，所述第一渲染树与所述第二渲染树不同。In the process of displaying the second interface, the electronic device generates a second rendering tree, the second rendering tree is used to generate the interface of the first application, the first rendering tree and the second rendering Trees are different.
  18. 根据权利要求17所述的方法，其特征在于，The method according to claim 17, characterized in that:
    所述第一渲染树用于生成所述第一应用的界面中不包括所述第一区域的界面；The first rendering tree is used to generate an interface of the first application that does not include the first area;
    所述第二渲染树用于生成所述第一应用的界面中不包括所述第二区域的界面。The second rendering tree is used to generate an interface of the first application that does not include the second area.
  19. 根据权利要求17或18所述的方法，其特征在于，所述电子设备生成第一渲染树，具体包括：The method according to claim 17 or 18, characterized in that the electronic device generates a first rendering tree, specifically including:
    所述电子设备基于所述第一区域确定所述第二应用的界面遮蔽的所述第一应用的视图包括第一视图；The electronic device determines, based on the first area, that the view of the first application obscured by the interface of the second application includes a first view;
    所述电子设备生成第三渲染树；The electronic device generates a third rendering tree;
    所述电子设备删除所述第三渲染树中的第一渲染节点得到所述第一渲染树，所述第一渲染节点与所述第一视图对应，渲染节点保存有对应的视图的绘制指令。The electronic device deletes the first rendering node in the third rendering tree to obtain the first rendering tree. The first rendering node corresponds to the first view, and the rendering node stores a drawing instruction of the corresponding view.
  20. 一种电子设备，其特征在于，所述电子设备包括：一个或多个处理器、存储器和屏幕；An electronic device, characterized in that the electronic device includes: one or more processors, memory and screens;
    所述存储器与所述一个或多个处理器耦合，所述存储器用于存储计算机程序代码，所述计算机程序代码包括计算机指令，所述一个或多个处理器调用所述计算机指令以使得所述电子设备执行：如权利要求1-19中任一项所述的方法。The memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors invoke the computer instructions to cause the The electronic device performs: the method according to any one of claims 1-19.
  21. 一种芯片***，其特征在于，所述芯片***应用于电子设备，所述芯片***包括一个或多个处理器，所述处理器用于调用计算机指令以使得所述电子设备执行如权利要求1-19中任一项所述的方法。A chip system, characterized in that the chip system is applied to electronic equipment, and the chip system includes one or more processors, and the processor is used to call computer instructions to cause the electronic equipment to execute claims 1- The method described in any one of 19.
  22. 一种计算机可读存储介质，包括指令，其特征在于，当所述指令在电子设备上运行时，使得所述电子设备执行如权利要求1-19中任一项所述的方法。A computer-readable storage medium comprising instructions, characterized in that when the instructions are run on an electronic device, the electronic device is caused to perform the method according to any one of claims 1-19.
  23. 一种包含指令的计算机程序产品，其特征在于，当所述计算机程序产品在电子设备上运行时，使得所述电子设备执行如权利要求1-19中任一项所述的方法。 A computer program product comprising instructions, characterized in that when the computer program product is run on an electronic device, the electronic device is caused to execute the method as claimed in any one of claims 1 to 19.
PCT/CN2023/120200 2022-09-22 2023-09-20 Interface generation method and electronic device WO2024061292A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202211155586.X 2022-09-22
CN202211155586.XA CN117785343A (en) 2022-09-22 2022-09-22 Interface generation method and electronic equipment

Publications (1)

Publication Number Publication Date
WO2024061292A1 true WO2024061292A1 (en) 2024-03-28

Family

ID=90391563

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2023/120200 WO2024061292A1 (en) 2022-09-22 2023-09-20 Interface generation method and electronic device

Country Status (2)

Country Link
CN (1) CN117785343A (en)
WO (1) WO2024061292A1 (en)

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20140282176A1 (en) * 2013-03-14 2014-09-18 Adobe Systems Incorporated Method and system of visualizing rendering data
US20170221242A1 (en) * 2016-02-01 2017-08-03 Facebook, Inc. Automatic overdraw reduction before rendering
US20180286116A1 (en) * 2017-03-30 2018-10-04 Magic Leap, Inc. Centralized rendering
CN113791706A (en) * 2020-09-04 2021-12-14 荣耀终端有限公司 Display processing method and electronic equipment

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20140282176A1 (en) * 2013-03-14 2014-09-18 Adobe Systems Incorporated Method and system of visualizing rendering data
US20170221242A1 (en) * 2016-02-01 2017-08-03 Facebook, Inc. Automatic overdraw reduction before rendering
US20180286116A1 (en) * 2017-03-30 2018-10-04 Magic Leap, Inc. Centralized rendering
CN113791706A (en) * 2020-09-04 2021-12-14 荣耀终端有限公司 Display processing method and electronic equipment

Also Published As

Publication number Publication date
CN117785343A (en) 2024-03-29

Similar Documents

Publication Publication Date Title
WO2021027747A1 (en) Interface display method and device
CN115473957B (en) Image processing method and electronic equipment
WO2020093988A1 (en) Image processing method and electronic device
US20230418696A1 (en) Method for performing drawing operation by application and electronic device
US20230254550A1 (en) Video Synthesis Method and Apparatus, Electronic Device, and Storage Medium
WO2021135838A1 (en) Page rendering method and related apparatus
CN116048933B (en) Fluency detection method
WO2022206681A1 (en) Window display method and related apparatus
WO2023066165A1 (en) Animation effect display method and electronic device
WO2023179123A1 (en) Bluetooth audio playback method, electronic device, and storage medium
WO2023016014A1 (en) Video editing method and electronic device
WO2023071482A1 (en) Video editing method and electronic device
WO2024061292A1 (en) Interface generation method and electronic device
WO2024083009A1 (en) Interface generation method and electronic device
WO2024083014A1 (en) Interface generation method and electronic device
WO2024082987A1 (en) Interface generation method and electronic device
WO2024046010A1 (en) Interface display method, and device and system
WO2023066177A1 (en) Animation effect display method and electronic device
WO2024067551A1 (en) Interface display method and electronic device
WO2022262291A1 (en) Image data calling method and system for application, and electronic device and storage medium
WO2023246783A1 (en) Method for adjusting device power consumption and electronic device
WO2024109573A1 (en) Method for floating window display and electronic device
WO2023169542A1 (en) Display method and electronic device
CN116193275B (en) Video processing method and related equipment
WO2024045841A1 (en) Storage method and apparatus, and electronic device

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23867578

Country of ref document: EP

Kind code of ref document: A1