CN117519860A

CN117519860A - Application window display method and device and related equipment

Info

Publication number: CN117519860A
Application number: CN202210912641.9A
Authority: CN
Inventors: 李洋
Original assignee: Huawei Technologies Co Ltd
Current assignee: Huawei Technologies Co Ltd
Priority date: 2022-07-30
Filing date: 2022-07-30
Publication date: 2024-02-06

Abstract

An application window display method, comprising: and acquiring image data of an application window in a GuestOS of the client machine and region description information of the image data, wherein the region description information is used for indicating the display position of the application window, so that image processing is executed according to the association relation between the image data and the region description information to obtain an image processing result, and the image processing result is output and is used for displaying the application window. In the process of displaying the application window, image processing is performed according to the association relation between the image data of the application window and the region description information of the image data, and an image processing result is obtained, so that in the process of dragging the application window, when each frame of image data of the application window is displayed according to the image processing result, only one display position can be corresponding to the image data, and a plurality of display positions can not be corresponding to the image data, thereby avoiding the problem of smear of the application window in the dragging process as much as possible, and improving user experience.

Description

Application window display method and device and related equipment

Technical Field

The present disclosure relates to the field of virtualization technologies, and in particular, to an application window display method, apparatus, and related devices.

Background

With the development of virtualization technology, a Virtual Machine (VM) may generally run in a physical machine, where an operating system installed in the physical machine may be referred to as a host operating system (host operating system, hostOS), and an operating system of the virtual machine may be referred to as a guest operating system (guest operating system, gueastos).

At present, description information related to one or more application windows in Guest OS can be obtained based on a window area obtaining technology, and the description information and an image of the application window in Guest OS are combined for display, so that the window applied in Guest OS is displayed in HoastOS.

However, during the dragging of the application window in gueastos, there is a tendency for smear to exist, as shown in fig. 1. Therefore, how to avoid the application window in Guest OS from generating smear during the dragging process as much as possible becomes an important issue to be solved.

Disclosure of Invention

The application window display method, device, computing equipment, computer readable storage medium and computer program product are used for avoiding the occurrence of smear of an application window in Guest OS in the dragging process as far as possible and improving user experience.

In a first aspect, the present application provides an application window display method, including: and acquiring image data of an application window in a GuestOS of the client machine and region description information of the image data, wherein the region description information is used for indicating the display position of the application window, so that image processing is executed according to the association relation between the image data and the region description information to obtain an image processing result, and the image processing result is output and is used for displaying the application window.

Because the image processing is executed according to the association relation between the image data of the application window and the region description information of the image data in the process of displaying the application window, the image processing result for displaying the application window is obtained, so that in the process of dragging the application window, when each frame of image data of the application window is displayed according to the image processing result, only one display position can be corresponding to the image processing result, and a plurality of display positions can not be corresponding to the image processing result, the problem that the application window is smeared in the dragging process can be avoided as much as possible, and the user experience is improved.

In one possible embodiment, the associating operation for associating the image data with the region description information may include: saving the region description information and the image in the same storage space; or, saving the region description information as a data structure, and establishing an association relationship between the data structure and the image data; or, the same identifier is added to the image data and the region description information, and the identifier is used for associating the region description information and the image data. In this way, in the process of displaying the application window, the image data of the application window and the region description information can be associated one to one, so that when the application window is displayed later, the application window is displayed only at one position, and is not displayed at a plurality of positions at the same time.

In one possible implementation, the region description information of the image data includes region description information corresponding to the image data of the windows of the plurality of applications, respectively, and then the windows of different applications are not overlapped when displayed. In this way, the region description information corresponding to the image data of the windows is divided into the sets based on the application granularity, and each set corresponds to one application, so that the windows of each application can be independently displayed in the finally presented image, that is, the windows of different applications can not have overlapping of display regions, and the experience of the user interacting with the application through the windows can be further improved.

In a possible implementation manner, when image processing is performed, the area description information corresponding to the image data of the windows of the plurality of applications is specifically divided to obtain a plurality of sets, and each set includes the area description information corresponding to the image data of at least one window of the same application; and then executing image processing according to the association relation between the plurality of sets and the image data of the windows of the plurality of applications to obtain an image processing result. Therefore, when the windows of each application are finally displayed, the windows of different applications can not overlap with each other, and the experience of interaction between the user and the application through the windows can be further improved.

In a possible implementation manner, the application window display method is applied to a terminal; alternatively, the application window display method is applied to a server for providing a virtualized service, for example, a server for providing a desktop cloud service, and the server has one or more virtual machines running therein.

In a second aspect, the application window display device further provides an application window display device, where the application window display device includes modules for executing the application window display method in the first aspect or any possible implementation manner of the first aspect.

In a third aspect, the present application also provides a computing device comprising: a processor and a memory; the memory is configured to store computer instructions, and the processor is configured to execute the window display method according to the computer instructions stored in the memory, or any implementation method of the first aspect. It should be noted that the memory may be integrated into the processor or may be independent of the processor. The computing device may also include a bus. The processor is connected with the memory through a bus. The memory may include a readable memory and a random access memory, among others.

Further, the computing device may further include an input-output device, such that image processing results from execution of the computer instructions by the processor may be output to the input-output device and utilized to display windows of one or more applications.

In a fourth aspect, the present application provides a computer readable storage medium having instructions stored therein, which when run on a computer, cause the computer to perform the steps of the method of any one of the embodiments of the first aspect and the first aspect described above.

In a fifth aspect, the present application provides a computer program product comprising instructions which, when run on a computer, cause the computer to perform the steps of the method of the first aspect and any implementation of the first aspect.

Further combinations of the present application may be made to provide further implementations based on the implementations provided in the above aspects.

Drawings

FIG. 1 is a schematic diagram of an exemplary drag application window provided herein;

FIG. 2 is a schematic diagram of an exemplary data processing system 200 provided herein;

FIG. 3 is a schematic diagram of another exemplary data processing system 200 provided herein;

fig. 4 is a flow chart of an application window display method provided in the present application;

FIG. 5 is a flowchart of another method for displaying application windows according to the present application;

fig. 6 is a schematic structural diagram of an application window display device provided in the present application;

fig. 7 is a schematic hardware structure of a computing device provided in the present application.

Detailed Description

In the unified Linux-based operating system (unified operating system, UOS) and the galenical operating system (KylinOS), existing windows applications are usually required to be compatible. At present, application in a windows-compatible system in a Linux system is generally realized by adopting a virtual machine mode. In the process, a window area acquisition technology is generally adopted to realize window fusion switching of applications in a windows system under a Linux system. However, when a window is dragged by an application in a windows system, a problem of smear is likely to occur.

In order to solve the problem of smear generated in the process of dragging an application window in Guest OS, the application window display method is provided, and image processing is performed according to the association relation by associating image data required for displaying the application window with region description information of the image data, so that even if the application window is in a dragged state, the problem of smear cannot occur when the application window is displayed according to an image processing result.

The technical solutions in the present application will be described below with reference to the drawings in the embodiments of the present application.

With reference to FIG. 2, a schematic architecture diagram of an exemplary data processing system is provided. As shown in FIG. 2, data processing system 200 includes a host 201 (also referred to as a physical machine), and one or more virtual machines 202 may be created on host 201, illustrated in FIG. 2 by way of example as creating one virtual machine 202.

The host 201 and the virtual machine 202 may operate based on the same type of operating system, for example, the operating systems of the host 201 and the virtual machine 202 are Linux operating systems, or are Windows operating systems, etc. Alternatively, host 201 and virtual machine 202 may run based on different types of operating systems, such as host 201 running based on a Linux operating system, virtual machine 202 running based on a Windows operating system, etc., so that ecological applications compatible with other operating systems may be implemented on host 201 using virtual machine 202.

One or more applications (applications) may be running in the operating system (also referred to as guetos os) of the virtual machine 202, and during the running process, the application typically creates one or more windows (windows), which is an interface for the application to interact with data of the user, and may be displayed to the user by the host 201 or a display device connected to the host 201 after the image rendering is completed.

By way of example, host 201 may be implemented by a device having computing capabilities, such as a server or the like. One or more processors may be provided in a device with computing capabilities, such as with a central processing unit (central processing unit, CPU) or application-specific integrated circuit (ASIC) or programmable logic device (programmable logic device, PLD), which may be a complex program logic device (complex programmable logical device, CPLD), FPGA, general array logic (generic array logic, GAL) or any combination thereof.

In actual applications, host 201 may be a terminal, such as a personal computer (personal computer, PC) or the like, configured with a display device (e.g., a display, etc.), such that virtual machine 202 may present one or more windows of the application on host 201. Alternatively, host 201 may be a server, such as a server that provides a virtualization service, such as a desktop cloud (closed desktop) service. At this time, after the virtual machine 202 finishes rendering the image data of the application window, the rendered data may be sent to a display device (such as a thin terminal) on the user side through a network, so that one or more windows of the application are displayed on the display device.

For ease of understanding, the data processing system 200 will be further described below using the host 201 as an example of a terminal equipped with a display device.

Referring to FIG. 3, a schematic diagram of a data processing system 200 is shown. In the data processing system 200 shown in fig. 3, the host 201 runs based on the Linux operating system, the virtual machine 202 runs based on the Windows operating system, and two applications, such as application 1 and application 2 shown in fig. 3, may run in the gueastos of the virtual machine 202. In this embodiment, application 1 and application 2 are taken as examples for illustration, and are not used to limit the number of applications that can be executed by the gueastos. Wherein, the application 1 can create the window 1 in the running time and generate the image data of the window 1 by means of image rendering. Similarly, application 2 may create window 2 and window 3 at runtime and render image data for window 2 and window 3, respectively.

Also, virtual machine 202 includes a direct expansion runtime environment (direct extension runtime) 2021, a user-mode display driver (user mode display driver, UMD) 2022, a direct expansion graphics kernel subsystem (direct extension graphics kernel subsystem) 2023, a kernel-mode display driver (kernel mode display driver, KMD) 2024, a graphics processor driver 2025, and a virtual display 2026.

The direct extension running environment 2021 is configured to respond to image rendering of the application 1 or the application 2 for the window, call the UMD2022 to cache image data rendered by each application, and record region description information (such as information of a window size) of the image data of each window, where the region description information can be used to indicate a display position of the image data of the window.

UMD2022 may create one or more graphical surfaces (surfaces) that may act as buffers, where a surface is associated with at least one buffer, each buffer for buffering image data for one or more windows. Further, the UMD2022 may also buffer the region description information of the image data of each window using the graphics surface.

The direct expansion graphics kernel subsystem 2023 is configured to control the KMD2024 to synthesize the buffered image data of one or more windows after buffering the image data of the windows and recording the region description information, and instruct the KMD2024 to apply for a corresponding memory space as a buffer for buffering the region description information and the synthesized image data.

KMD2024 is configured to synthesize image data of one or more windows under the control of direct expansion graphics kernel subsystem 2023, perform a rendering operation again on the synthesized image data, and store the rendered image data in a buffer applied in advance. Then, KMD2024 supplies the rendered image data and the region description information of the image data of each window to graphic processor driver 2025; alternatively, KMD2024 may provide the rendered image data and corresponding region description information to virtual display 2026 via a virtual machine communication channel with virtual display 2026.

The graphic processor driver 2025 is configured to output the rendered image data and the region description information of the image data of each window to the virtual display 2026.

A virtual display 2026 for displaying images including contents of windows 1 to 3 on the host 201 according to the received region description information and image data.

In an actual application scenario, a user may drag one or more windows presented on host 102. Accordingly, data processing system 200 may update and display the position of the dragged window in real time based on the user's continued drag operation. However, there is a problem in that the time delay of obtaining the image data by the virtual display 2026 is not consistent with the time delay of obtaining the region description information. For example, when the computing power of the host 201 is low, the time taken to render the image data of the window may be too long, resulting in a higher latency for the virtual display 2026 to acquire the image data than for the region description information. This makes it possible that the virtual display 2026, upon receiving a frame of image data of an application window, may have received a plurality of region description information corresponding to the application window (the plurality of region description information is generated based on a continuous drag operation by a user), so that when displaying image data of an application window according to the plurality of region description information, the application window may be presented at a plurality of positions along a drag path of the user in a display image, thereby causing a problem of drag by the user in dragging the application window, for example, as shown in fig. 1.

Therefore, in the application window display method provided in the embodiment of the present application, in the process of displaying an application window, an image processing operation is performed according to an association relationship between image data of the application window and region description information of the image data, so as to obtain an image processing result, where the image processing result may be, for example, an image to be finally displayed or data for generating a final display image, which makes the virtual display 2026 correspond to only one display position and not to multiple display positions when displaying each frame of image data of the application window according to the image processing result in the process of dragging the application window, so that a problem of drag of the application window in the dragging process can be avoided as much as possible, and user experience is improved.

It should be noted that the data processing system 200 shown in fig. 3 is merely an example, and the application window display method may be applied to other possible data processing systems, for example, in other possible data processing systems, more functional units may be included, which is not limited in this embodiment.

The process of displaying an application window in Guest OS provided in the present application is further described below with reference to the accompanying drawings.

With reference to fig. 4, fig. 4 is a schematic flow chart of an application window display method according to an embodiment of the present application, where the method may be applied to the data processing system 200 shown in fig. 2 or fig. 3, or may be applied to other data processing systems, and the embodiment is not limited thereto. For ease of description, the following is exemplary in connection with data processing system 200 shown in FIG. 3. As shown in fig. 4, the method specifically may include:

s401: image data of an application window in Guest OS and region description information of the image data are acquired, wherein the region description information is used for describing the display position of the application window.

In this embodiment, application 1 or application 2 in the gueastos of the virtual machine 202 may create one or more application windows during the running process, so as to implement data interaction with the user. For ease of understanding, the present embodiment is exemplified by displaying one window created by the application 1 (i.e., window 1).

When the application 1 creates the window 1, rendering processing may be performed according to the content included in the window 1, so as to generate a corresponding frame of image data, and generate region description information of the frame of image data, where the region description information can be used to indicate a display position where the application window is located at the time of final display. As some examples, the region description information of the image data may be, for example, one or more of the size of the window 1, the transparent frame size, the title description, and the like, or may be other types of information, for example, when the same application creates a plurality of windows, the region description information of the image data of each window may further include parent-child relationship information between the window and other windows, and the like.

After rendering the image data, the application 1 may output the image data and the region description information.

S402: and executing image processing according to the association relation between the image data and the region description information to obtain an image processing result.

Since there may be a difference in the time delay of processing the output image data and the output region description information in the data processing system 200 in some application scenarios, in this embodiment, the image data of the application window and the region description information of the image data may be associated (or referred to as binding) one to one, so that the image data and the region description information when finally displayed (i.e. sent to the virtual display 2026 for display) can correspond to the same time, i.e. the real display region of the image data is consistent with the display region indicated by the region description information.

Illustratively, the present embodiment provides the following several implementation examples in which image data of an application window and region description information are associated one-to-one.

In a first implementation example, the application 1 may call a direct extension (multimedia programming interface) interface at rendering time and enumerate handle information of the window 1, which is used to identify the window 1. The direct expansion execution environment 2021, in response to the invocation of the application 1 for the DX interface, specifies the UMD2022 by a windows system graphics driver model (windows display driver model, WDDM) to save the image data of the window 1 and the region description information corresponding to the image data to the same storage space, so as to realize one-to-one association between the image data and the region description information. In particular, UMD2022 may create a graphical surface (surface) for the window 1, such as graphical surface 1 in FIG. 3, and apply for a corresponding memory space for the graphical surface. The graphics surface 1 may be used as a part of the storage space in the same storage space to buffer the image data of the window 1, so after the application 1 renders a frame of image data of the window 1, the UMD2022 may store the image data in the graphics surface 1 corresponding to the window 1 to buffer the frame of image data. In practice, application 1 may continuously render multiple frames of image data for window 1, and UMD2022 may buffer each frame of image data for window 1 using the surface. In addition, the graphics surface 1 may be further configured with an expansion buffer corresponding to the graphics surface 1, such as the expansion buffer 1 in fig. 3, where the expansion buffer 1 is a part of the storage space in the same storage space, and a logical address of the expansion buffer 1 may be continuous with a logical address of the graphics surface 1, so as to store the region description information corresponding to the image data. In this way, by buffering the image data and the region description data with the graphics surface 1 and the expansion buffer 1, respectively, it is possible to realize one-to-one association of the image data and the region description information of the application window.

In actual application, one graphics surface and expansion buffer may be created in UMD2022 for each window created by the application separately, for storing image data and region description information corresponding to the window, such as graphics surface 2 and expansion buffer 2 created for window 2, graphics surface 3 created for window 3, and expansion buffer 3 in fig. 3.

In a second implementation example, UMD2022 may store image data for window 1 to graphical surface 1 created for window 1 and save the region description information for window 1 as a data structure. The data structure refers to a set including a plurality of data elements, and the plurality of data elements may be, for example, a window size, a transparent frame size, a header description, and the like. The UMD2022 may store the data structure in any storage area, for example, the UMD2022 may store, in a section of memory space of a separate application, a data structure corresponding to one or more area description information, and so on. The UMD2022 may then establish an association between the identity of the graphical surface 1 and the name of the data structure and record. The identification of the graphic surface 1 may be, for example, one or more of a name, a number, an ID, and a head address of the graphic surface 1, or may be other information that may be used to identify the graphic surface 1. Thus, by establishing an association relationship between the identification of the graphic surface 1 and the data structure, one-to-one association of the image data of the window 1 with the region description information can be achieved.

In a third implementation example, the UMD2022 may store the image data of the window 1 to the graphics surface 1 created for the window 1, and store the region description information corresponding to the image data in a section of the memory space applied separately, and then the UMD2022 may add the same identifier to the image data of the application window and the region description information, respectively, so as to one-to-one associate the region description information of the window 1 and the image data of the window 1 with the identifier. The added identifier may specifically be a frame identifier when the application 1 renders to obtain a frame of image data, such as a frame number; alternatively, the identification may specifically be a time stamp that may be used to indicate when the application 1 rendered one frame of image data. In this way, with the frame identification or the time stamp, the real display area of the image data can be made coincident with the display area indicated by the area description information.

After saving the image data and the region description information of window 1, UMD2022 may inform direct extended graphics kernel subsystem 2023 to perform further rendering operations on the image data in UMD2022 to generate image data that can be conveniently displayed by a display.

In this embodiment, before performing a rendering operation based on the image data, the direct expansion graphics kernel subsystem 2023 may instruct the KMD2024 to create a display graphics surface (display surface), such as the display graphics surface 1 in fig. 3, and apply a corresponding memory space for the display graphics surface 1. Wherein the display graphics surface 1 may buffer image data generated by the KMD2024 performing the rendering operation as an image buffer. The direct extended graphics kernel subsystem 2023 may then access the image data for window 1 from the UMD2022 and control the KMD2024 to perform rendering operations based on the image data for window 1. After the rendering operation is completed, KMD2024 may save the rendered image data on display graphics surface 1.

Furthermore, direct extended graphics kernel subsystem 2023 may also save the region description information for window 1 to memory space in KMD 2024. Illustratively, the direct extended graphics kernel subsystem 2023 may control the KMD2024 to create an extended buffer A1, as shown in fig. 3, and the direct extended graphics kernel subsystem 2023 may save the region description information of the window 1 into the extended buffer A1. In actual application, KMD2024 may create a plurality of extension buffers for a plurality of applications, such as extension buffer A1 and extension buffer A2 in fig. 3, where extension buffer A1 is used to store region description information of image data of one or more windows belonging to application 1, and extension buffer A2 is used to store region description information of image data of one or more windows belonging to application 2. Alternatively, KMD2024 may store the region description information of window 1 in another storage space, which is not limited in this embodiment.

Note that, when the KMD2024 stores the region description information of the window 1 and the image data obtained by rendering, a one-to-one association relationship is established between the region description information and the image data obtained by rendering of the KMD2024, and a specific implementation manner of implementing synchronous association may be referred to the description of the related parts, which is not repeated herein.

KMD2024 may then send the rendered image data with region description information to virtual display 2026. Wherein KMD2024 may forward it to virtual display 2026 by graphics processor driver 2025; alternatively, KMD2024 may establish a virtual machine communication channel, and send the rendered image data and region description information to virtual display 2026 through the virtual machine communication channel. Because the image data rendered by the KMD2024 and the region description information have a one-to-one association relationship, the region description information can participate in the display of the application window along with the image data of the window 1, so that the unique display position of the window 1 during display can be determined according to the associated region description information later, and the problem of smear caused by a plurality of display positions during display of the window 1 is avoided.

After obtaining the region description information of the window 1 and displaying the image data in the graphic surface 1, the virtual display 2026 may perform image processing according to the association relationship between the region description information and the image data, resulting in an image processing result, which may be, for example, generating an image that is finally used for display to the user. In general, the final displayed image may include a window display area and a non-window display area, where the image content in the window display area is the content in the window 1. The image content of the non-window display area may be, for example, the content of a background image, and the present embodiment is not limited thereto. Alternatively, the image processing result obtained by the virtual display 2026 performing the image processing may be data for generating a final displayed image, and the expression form of the image processing result is not limited in this embodiment.

As an implementation example, the virtual display 2026 may determine the imaging area of the window 1 in the finally displayed image according to the area description information, that is, determine the window display area described above. Then, the virtual display 2026 may populate a window display area in a preset image (e.g., background image, etc.) with the image data provided by the KMD2024, so that an image may be generated that is ultimately displayed to the user. The image content of the non-window display area in the image is the content in the preset image.

S403: and outputting an image processing result, wherein the image processing result is used for displaying an application window.

In this embodiment, after obtaining the image processing result, the virtual display 2026 may display the image including the window 1 with the display device of the host 201 according to the image processing result.

When the host 201 is not configured with a display device, such as a server in a desktop cloud scenario, the virtual display 2026 may send the image processing result to a display device (typically located on the user side) connected to the host 201, so that the display device displays an image including the window 1 according to the image processing result.

In actual application, the application 1 may continuously generate multi-frame image data for the window 1, so that each frame of image data for the window 1 may be displayed in sequence based on the above steps S401 to S403.

In addition, the process of displaying the application window shown in the steps S401 to S403 is described by taking the image data of displaying one window as an example, and when the application in the gueastos creates a plurality of windows, the image data corresponding to the plurality of windows may be displayed respectively according to the similar process, which is not described herein in detail.

Since the UMD2022 and KMD2024 perform one-to-one association between the image data of the window 1 and the region description information of the window 1 in the process of displaying the application window, each frame of image data of the window 1 received by the virtual display 2026 has the region description information uniquely associated with the frame of image data, so that the image data of the window 1 can only correspond to one display position and not correspond to a plurality of display positions when being displayed in the process of dragging the window 1, the problem that the window 1 should have a smear in the dragging process can be avoided as much as possible, and the user experience is improved.

In an actual application scenario, multiple applications may be running in the gueastos at the same time, and each application in the multiple applications may create one or multiple windows, where when the windows of the multiple applications are displayed, there may be a problem that the windows of different applications overlap each other, for example, a display area of the window 1 of the application 1 and the window 2 of the application 2 in a final displayed image may overlap, so that a window of a part of applications may cover part or all of contents of a window of another application. For this reason, another application window display method is provided in the embodiments of the present application, so as to avoid the display area overlapping between the displayed windows of different applications as much as possible. Next, an exemplary description will be given of a window showing the application 1 and the application 2 in fig. 3.

Referring to fig. 5, a flowchart of another application window display method provided in an embodiment of the present application is shown, and as shown in fig. 5, the method may specifically include:

s501: image data respectively corresponding to a plurality of windows of applications in Guest OS and region description information of the image data of each window are acquired, and the region description information of the image data of each window is used for describing the display position of the window.

The specific implementation of step S501 may be described with reference to the relevant point of step S401 in the foregoing embodiment, which is not described herein.

S502: dividing the region description information respectively corresponding to the image data of the windows of the plurality of applications to obtain a plurality of sets, wherein each set comprises the region description information respectively corresponding to the image data of at least one window of the same application.

S503: and establishing association relations between the plurality of sets and the image data of the windows of the plurality of applications.

In this embodiment, the UMD2022 may buffer image data and region description information of a plurality of windows of the application, and when the UMD2022 buffers the data, a one-to-one association relationship may be established between the image data of each of the plurality of windows and the region description information of the image data. Illustratively, the present embodiment provides the following three implementations for establishing the association relationship.

In a first implementation, the UMD2022 may apply for a storage space for each of the windows 1 to 3, each storage space being used to store image data of one window and region description information of the image data. In particular implementations, UMD2022 may create respective graphical surfaces 1-3, each corresponding to a window, as shown in fig. 3, for storing image data for the window. Also, the UMD2022 creates an extension buffer for each graphics surface, such as extension buffer 1 to extension buffer 3 shown in fig. 3, and each extension buffer may have a logically consecutive storage address with the storage address of the graphics surface corresponding to the extension buffer for storing the region description information. Thus, the image data of each window and the region description information of the image data can be stored in the same storage space. As shown in fig. 3, in a section of storage space applied for the window 1, the line surface 1 is used for caching image data of the window 1, and the expansion buffer area 1 is used for caching area description information of the window 1; in a section of storage space applied for the window 2, the line surface 2 is used for caching image data of the window 2, and the expansion buffer area 2 is used for caching area description information of the window 2; in a section of the storage space applied for the window 3, the line surface 3 is used for caching image data of the window 3, and the extended buffer area 3 is used for caching area description information of the window 3. In this way, the image data and the region description data of each window can be associated one to one by using the graphics surface and the expansion buffer to buffer the image data and the region description data respectively. In practice, the UMD2022 caches the graphical surface of the image data for multiple windows, which may form a rectangular buffer (or circular buffer).

In a second implementation manner, the UMD2022 caches the image data of the windows 1 to 3 by using the graphics surfaces 1 to 3, respectively, and may store the region description information of the windows 1 to 3 as data structures, respectively, and establish association relationships between the graphics surfaces and the data structures for the same window, for example, establish association relationships between the identifiers of the graphics surfaces and the names of the data structures, so as to implement one-to-one association between the image data of each window and the region description information of the image data.

In a third implementation, the UMD2022 caches the image data of the windows 1 to 3 by using the graphics surfaces 1 to 3, and caches the region description information corresponding to the windows 1 to 3 to the storage regions of the separate application, and then the UMD2022 may add the same identifier to the image data of the application window and the region description information, respectively, so as to one-to-one associate the image data of each window and the region description information of the image data by using the identifier. In this way, for the image data of each window and the region description information corresponding to the image data, the one-to-one association of the image data and the region description information can be achieved by using the same identifier (such as a frame identifier or a timestamp).

In this embodiment, after associating the image data of the plurality of windows with the region description information of the image data, the region information data of each window may follow the image data of the window, and may be read and written by the KMD 2024. For windows of multiple applications, KMD2024 may continue to render image data of each window, and after the rendering operation is completed, may perform a compositing operation on the image data rendered by each window, and the image data obtained after performing the compositing operation may be cached on the display graphics surface.

Furthermore, KMD2024 may divide the region description information corresponding to each of windows 1 to 3 according to the application to which each window belongs to obtain 2 sets, which are set 1 and set 2 respectively, where set 1 includes the region description information corresponding to window 1 created by application 1, and set 2 includes the region description information corresponding to window 2 and window 3 created by application 2. Thus, the application can be granularity, and the division of the plurality of region description information is realized.

Then, the KMD2024 may store the multiple divided sets in an extension buffer A1 and an extension buffer A2 created by the KMD2024, where the extension buffer A1 is used to store region description information in the set 1, that is, region description information corresponding to image data of all windows created by the application 1 (i.e., window 1); the extension buffer A2 is used for storing the region description information in the set 2, that is, the region description information corresponding to the image data of all the windows (i.e., the window 2 and the window 3) created by the application 2.

Furthermore, when caching the image data and the plurality of sets, the KMD2024 establishes a one-to-one association relationship between the plurality of sets and the image data, so that the region description information of the image data of each window can complete transmission and display along with the image data of the window. The specific implementation process of establishing the association between each set and the synthesized image data by KMD2024 may refer to the description of the association region description information and the related parts of the image data, which will not be described herein.

S504: and executing image processing according to the association relation between the plurality of sets and the image data of the windows of the plurality of applications to obtain an image processing result.

S505: and outputting an image processing result, wherein the image processing result is used for displaying an application window.

In this embodiment, the virtual display 2026 may acquire image data in the display graphics surface 1, region description information in the plurality of sets stored in the extension buffers (A1 and A2) through the graphics processor driver 2025, or may directly establish a virtual machine communication channel with the KMD2024, and acquire the image data in the display graphics surface 1, region description information in the plurality of sets in the extension buffers (A1 and A2) through the virtual machine communication channel. Then, the virtual display 2026, after obtaining the region description information in the plurality of sets and displaying the image data in the graphics surface 1, may perform image processing on the region description information in the plurality of sets and the image data, obtain an image processing result, and display windows of the application 1 and the application 2 according to the image processing result.

In particular implementations, the virtual display 2026 may create 2 sets of window display regions, window display region 1 and window display region 2, respectively, in the image to be finally displayed based on the region description information in the 2 sets, and the positions of the two window display regions on the image do not overlap. Then, the virtual display 2026 may intercept and display the image data belonging to the application 1 corresponding to the set 1 in the graphic surface 1, and fill the intercepted image data into the window display area 1; and intercepting the image data of the application 2 corresponding to the set 2 in the display graph surface 1, and filling the intercepted image data into the window display area 2, namely, displaying the windows created by the application 1 and the application 2 in different window display areas, so that when the application window is displayed, the window of the application 1 is not covered or hidden by the window of the application 2, and meanwhile, the window of the application 2 is not covered or hidden. The image also includes a non-window display area, and the image content of the window display area may be filled with specified image data, for example, so that an image that is finally displayed to the user may be generated. In this way, in the finally displayed image, the windows of different applications can be independently displayed, so that the problem that part of the applied windows are covered or hidden by other applied windows can be avoided.

In a further embodiment, when the image content of the window display area includes image data of a plurality of windows, such as the window display area 2 includes image data of the window 2 and the window 3, different windows may be independently displayed in the window display area, that is, there may be no coverage or masking between the plurality of windows in the window display area; or there may be an overlay or a mask between different windows, for example, when window 2 and window 3 satisfy a parent-child relationship (e.g., window 3 may be derived based on window 2), the image content of window 3 may overlay or mask the image content of window 2 within window display area 2, etc. In the present embodiment, the display manner between the plurality of windows in each window display area is not limited.

In this embodiment, after the image is generated, the virtual display 2026 may use the image as an image processing result and display an image including the window 1 and the window 3 using the display device of the host 201. When the host 201 is not configured with a display device, such as a server or the like where the host 201 is providing a virtualization service, the virtual display 2026 may send the image processing result to a display device (typically located on the user side) connected to the host 201, so that the display device displays the images including the windows 1 to 3 according to the image processing result. In practical applications, the image processing result output by the virtual display 2026 may be data indicating an image to be finally displayed, such as position information including the window display area and the non-window display area, image data in each window display area, and data of the non-window display area, so that the display device of the host 201 or the display device connected to the pre-host 201 may generate an image including windows 1 to 3 according to the image processing result.

In actual application, the application 1 and the application 2 may continuously generate the multi-frame image data for the respective created windows, so that each of the multi-frame image data for the plurality of windows may be sequentially displayed based on the above steps S501 to S505.

In the process of displaying the application window, the image data of each window is associated with the area description information of the image data one to one, so that in the process of dragging part of the window, the image data of the dragged window can only correspond to one display position and cannot correspond to a plurality of display positions, the problem of dragging of the window in the dragging process can be avoided as much as possible, and the user experience is improved.

And the region description information corresponding to the image data of the windows is divided into a plurality of sets based on application granularity, and each set corresponds to one application, so that the windows of each application can be independently displayed in the finally presented image, namely, the windows of different applications can not have overlapping of display regions, and the experience of interaction between the user and the application through the windows can be further improved.

It is noted that other reasonable combinations of steps can be envisaged by the person skilled in the art from the above description, and are also within the scope of protection of the present application. Further, those skilled in the art will also be familiar with the preferred embodiments, and the description of the embodiments is not necessarily a requirement of the present application.

The application window display method provided by the present application is described in detail above with reference to fig. 1 to 5, and the application window display device and the computing apparatus provided according to the present application will be described below with reference to fig. 6 to 7.

Fig. 6 is a schematic structural diagram of an application window display device provided in the present application. The application window display device 600 is not limited to this, and may be applied to a terminal, a server providing a virtualization service (a virtual machine is run on the server), or the like. As shown in fig. 6, the application window display apparatus 600 may include:

an obtaining module 601, configured to obtain image data of an application window in a guest operating system gueastos and region description information of the image data, where the region description information is used to indicate a display position of the application window;

an image processing module 602, configured to perform image processing according to the association relationship between the image data and the region description information, so as to obtain an image processing result;

and the output module 603 is configured to output the image processing result, where the image processing result is used to display the application window.

In a possible implementation manner, the association relationship between the image data and the region description information is established through an association operation, and the association operation includes:

Storing the region description information and the image in the same storage space; or alternatively, the first and second heat exchangers may be,

storing the region description information into a data structure, and establishing an association relationship between the data structure and the image data of the application window; or alternatively, the first and second heat exchangers may be,

and adding the same identifier to the image data of the application window and the region description information respectively, wherein the identifier is associated with the region description information and the image data of the application window.

In one possible implementation manner, the region description information of the image data includes region description information corresponding to the image data of the windows of the plurality of applications, and then the windows of different applications are not overlapped when displayed.

In a possible implementation manner, the image processing module 602 is configured to:

dividing the region description information respectively corresponding to the image data of the windows of the plurality of applications to obtain a plurality of sets, wherein each set comprises the region description information respectively corresponding to the image data of at least one window of the same application;

and executing image processing according to the association relation between the plurality of sets and the image data of the windows of the plurality of applications to obtain an image processing result.

In one possible implementation, the apparatus 600 is applied to a terminal, or the apparatus 600 is applied to a server providing a virtualization service.

It should be understood that, when the application window display device 600 according to the embodiment of the present application implements the application window display method shown in fig. 1 to 5 through software, the application window display device 600 and its respective modules may also be software modules.

The application window display apparatus 600 according to the embodiment of the present application may correspond to performing the method described in the embodiment of the present application, and the above and other operations and/or functions of each module of the application window display apparatus 600 are for implementing the respective flows of each method in fig. 3 and fig. 4, and are not described herein for brevity.

In this embodiment, in the process of displaying an application window, image data of each window is associated with the region description information of the image data in one-to-one manner, so that in the process of dragging a part of windows, the image data of the dragged window can only correspond to one display position and not correspond to a plurality of display positions, thereby avoiding the problem of smearing of the window in the dragging process as much as possible and improving user experience.

Fig. 7 is a schematic diagram of a computing device 700 provided herein, where the computing device 700 includes a processor 701, a memory 702, a communication interface 703, and a bus 704 as shown. The processor 701, the memory 702, and the communication interface 703 communicate via the bus 704, or may communicate via other means such as wireless transmission. The bus 704 may include a power bus, a control bus, a status signal bus, and the like in addition to a data bus. But for clarity of illustration, the various buses are labeled as bus 704 in the figures.

The processor 701 may be a CPU, and the processor 701 may also be other general purpose processors, digital Signal Processors (DSPs), application Specific Integrated Circuits (ASICs), field Programmable Gate Arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete device components, etc.

Memory 702 may include read-only memory and random access memory, and provides computer instructions and data to processor 701. Also, the memory 702 may be volatile memory or nonvolatile memory, or may include both volatile and nonvolatile memory. The nonvolatile memory may be a read-only memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an electrically Erasable EPROM (EEPROM), or a flash memory. The volatile memory may be random access memory (random access memory, RAM) which acts as an external cache. By way of example, and not limitation, many forms of RAM are available, such as Static RAM (SRAM), dynamic Random Access Memory (DRAM), synchronous Dynamic Random Access Memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous Link DRAM (SLDRAM), and direct memory bus RAM (DR RAM).

Further, the computing device 700 may further include an input/output (IO) device 705, where the input/output device 705 communicates with the processor 701 and the memory 702 through the bus 704, and the input/output device 705 may be, for example, a device with a display function, such as a display, etc., so that the processor 701 may send the image processing result obtained by the processing to the input/output device 705 to display a window of one or more windows.

As one possible implementation, the computing device 700 may communicate with an input-output device 706 through a communication interface 703. The input/output device 706 is similar to the input/output device 705 in specific form, and includes a device having a display function, such as a display, so that the processor 701 may send the image processing result obtained by the processing to the input/output device 706 through the communication interface 703 to display one or more windows.

It should be understood that the computing device 700 according to the embodiments of the present application may correspond to the application window display apparatus 600 in the embodiments of the present application, and may correspond to performing the methods shown in fig. 3 and 4 according to the embodiments of the present application, and that the foregoing and other operations and/or functions implemented by the computing device 700 are, respectively, for implementing the respective flows of the methods in fig. 3 and 4, and are not repeated herein for brevity.

The above embodiments may be implemented in whole or in part by software, hardware, firmware, or any other combination. When implemented in software, the above-described embodiments may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded or executed on a computer, produces a flow or function in accordance with embodiments of the present application, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions may be stored in or transmitted from one computer-readable storage medium to another, for example, by wired (e.g., coaxial cable, optical fiber, digital Subscriber Line (DSL)), or wireless (e.g., infrared, wireless, microwave, etc.). The computer readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that contains one or more sets of available media. The usable medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. The semiconductor medium may be a solid state disk (solid state drive, SSD)

While the invention has been described with reference to certain preferred embodiments, it will be understood by those skilled in the art that various changes and substitutions of equivalents may be made and equivalents will be apparent to those skilled in the art without departing from the scope of the invention. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. An application window display method, the method comprising:

acquiring image data of an application window in a GuestOS of a client operating system and region description information of the image data, wherein the region description information is used for indicating the display position of the application window;

executing image processing according to the association relation between the image data and the region description information to obtain an image processing result;

and outputting the image processing result, wherein the image processing result is used for displaying the application window.

2. The method according to claim 1, wherein the association relationship of the image data and the region description information is established by an association operation comprising:

Storing the region description information into a data structure, and establishing an association relationship between the data structure and the image data; or alternatively, the first and second heat exchangers may be,

and adding the same identification to the image data and the region description information respectively, wherein the identification is associated with the region description information and the image data.

3. The method according to claim 1 or 2, wherein the region description information of the image data includes region description information corresponding to image data of windows of a plurality of applications, respectively, and windows of different applications are not overlapped when displayed.

4. A method according to claim 3, wherein said performing image processing according to the association relationship of the image data and the region description information comprises:

5. The method according to any one of claims 1 to 4, wherein the method is applied to a terminal or the method is applied to a server providing a virtualization service.

6. An application window display apparatus, the apparatus comprising:

the acquisition module is used for acquiring image data of an application window in a client operating system Guest OS and region description information of the image data, wherein the region description information is used for indicating the display position of the application window;

the image processing module is used for executing image processing according to the association relation between the image data and the region description information to obtain an image processing result;

and the output module is used for outputting the image processing result, and the image processing result is used for displaying the application window.

7. The apparatus according to claim 6, wherein the association relationship of the image data and the region description information is established by an association operation comprising:

8. The apparatus according to claim 6 or 7, wherein the region description information of the image data includes region description information corresponding to image data of windows of a plurality of applications, respectively, and windows of different applications are not overlapped when displayed.

9. The apparatus of claim 8, wherein the image processing module is configured to:

10. The apparatus according to any one of claims 6 to 9, wherein the apparatus is applied to a terminal or the apparatus is applied to a server providing a virtualization service.

11. A computing device comprising a processor and a memory for storing computer instructions; the processor being adapted to perform the operational steps of the method according to any one of claims 1 to 5 according to the computer instructions.

12. A computer readable storage medium comprising instructions which, when run on a computing device, cause the computing device to perform the operational steps of the method of any one of claims 1 to 5.