CN115767092A

CN115767092A - Image transmission method, cloud desktop, image transmission system, image transmission equipment and product

Info

Publication number: CN115767092A
Application number: CN202211364308.5A
Authority: CN
Inventors: 张子鹏; 闵洪波; 田巍; 任晋奎; 张献涛
Original assignee: Alibaba China Co Ltd
Current assignee: Alibaba China Co Ltd
Priority date: 2022-11-02
Filing date: 2022-11-02
Publication date: 2023-03-07

Abstract

The embodiment of the application provides an image transmission method, a cloud desktop, an image transmission system, equipment and a product. The method comprises the following steps: acquiring a desktop image; according to the network service quality information and the image content of the desktop image, carrying out block division on the desktop image to obtain a plurality of blocks; determining corresponding coding algorithms for the plurality of blocks respectively; respectively coding the plurality of blocks by utilizing the coding algorithms corresponding to the plurality of blocks to obtain a plurality of coding information corresponding to the desktop image; and respectively sending the plurality of coded information corresponding to the desktop image to a terminal so that the terminal can display the desktop image after decoding the plurality of coded information for user operation. By adopting the scheme provided by the embodiment, the data volume sent by the network at each time can be balanced and controllable, and the occurrence probability of burst traffic is reduced, so that network congestion is avoided, and the use experience of cloud desktop users is promoted.

Description

Image transmission method, cloud desktop, image transmission system, image transmission equipment and product

Technical Field

The application relates to the technical field of computers, in particular to an image transmission method, a cloud desktop, an image transmission system, image transmission equipment and an image transmission product.

Background

Cloud desktop is a remote desktop service in which a cloud provides computing and storage capabilities, that is, a technology in which a user runs a remote desktop using a terminal through a network. A user using the cloud desktop can access documents and applications that are needed to be used in their cloud desktop from any device, including windows and Mac computers, iPad/iPhone and Android smart devices, desktop terminals. The operating system operated by the cloud desktop can be installed on a remote host, data processing and storage are also completed at a remote end, and the local terminal only needs to execute control commands such as a keyboard and a mouse and transmit display pictures.

The actual use experience of the cloud desktop is closely related to the fluency of network data transmission. If congestion and packet loss occur on the network, a user feels high response time delay, unsmooth operation and the like when using the cloud desktop.

Disclosure of Invention

In view of the above problems, embodiments of the present application provide a technical solution that can improve the problems of network congestion, packet loss, and the like.

In one embodiment of the application, a cloud desktop image transmission method is provided and is suitable for a cloud desktop server. Specifically, the method comprises the following steps:

acquiring a desktop image;

according to the network service quality information and the image content of the desktop image, carrying out block division on the desktop image to obtain a plurality of blocks;

determining corresponding coding algorithms for the plurality of blocks respectively;

respectively coding the plurality of blocks by utilizing the coding algorithms corresponding to the plurality of blocks to obtain a plurality of coding information corresponding to the desktop image;

and respectively sending the plurality of coded information corresponding to the desktop image to a terminal so that the terminal can display the desktop image after decoding the plurality of coded information for user operation.

In another embodiment of the application, a cloud desktop image transmission method is provided and is suitable for a terminal. Specifically, the method comprises the following steps:

respectively receiving a plurality of coded messages sent by a cloud desktop server; the coding information contains desktop image identification, belonging block identification and the total number of blocks corresponding to the desktop image;

after determining that the number of the received coded information with the same desktop image identifier is the total number of the blocks, decoding the coded information according to the decoding algorithms corresponding to the coded information respectively to obtain a plurality of blocks;

splicing the blocks according to the block identifiers corresponding to the blocks to obtain the desktop image;

and displaying the desktop image.

In yet another embodiment of the present application, a cloud desktop system is provided. The system comprises:

the cloud desktop server is used for acquiring a desktop image; according to the network service quality information and the image content of the desktop image, carrying out block division on the desktop image to obtain a plurality of blocks; determining corresponding coding algorithms for the blocks respectively; respectively coding the plurality of blocks by utilizing the coding algorithms corresponding to the plurality of blocks to obtain a plurality of coding information corresponding to the desktop image; respectively sending a plurality of coded information corresponding to the desktop images to a terminal;

the terminal is in communication connection with the cloud desktop server and is used for receiving the plurality of coded information respectively sent by the cloud desktop server and decoding the plurality of coded information respectively to obtain a plurality of blocks; and splicing the plurality of blocks to obtain the desktop image, and displaying the desktop image.

In yet another embodiment of the present application, an image transmission method is provided, adapted to a first device. The method comprises the following steps:

acquiring a target image;

according to the network service quality information and the image content of the target image, carrying out block division on the target image to obtain a plurality of blocks;

respectively coding the blocks by utilizing the coding algorithms corresponding to the blocks to obtain a plurality of coding information corresponding to the target image;

and respectively sending a plurality of pieces of coding information corresponding to the target image to second equipment so that the second equipment can display the target image after decoding the plurality of pieces of coding information.

In yet another embodiment of the present application, an image transmission method is provided, which is adapted to a second device. Specifically, the method comprises the following steps:

respectively receiving a plurality of pieces of coding information sent by first equipment; the coding information contains a target image identifier, an affiliated block identifier and the total number of blocks corresponding to the target image;

after determining that a plurality of pieces of coding information containing the same target image identifier are received in the total number of the blocks, decoding the plurality of pieces of coding information according to the decoding algorithms corresponding to the plurality of pieces of coding information respectively to obtain a plurality of blocks;

splicing the blocks according to the block identifications corresponding to the blocks to obtain the target image;

and displaying the target image.

In still another embodiment of the present application, there is provided an image transmission system including:

a first device for acquiring a target image; according to the network service quality information and the image content of the target image, carrying out block division on the target image to obtain a plurality of blocks; determining corresponding coding algorithms for the plurality of blocks respectively; respectively coding the blocks by utilizing the coding algorithms corresponding to the blocks to obtain a plurality of coding information corresponding to the target image; respectively sending a plurality of pieces of coding information corresponding to the target image to second equipment so that the second equipment can display the target image after decoding the plurality of pieces of coding information;

the second device is in communication connection with the first device and is used for receiving the plurality of coded information sent by the first device and decoding the plurality of coded information respectively to obtain a plurality of blocks; splicing the blocks to obtain the target image; and displaying the target image.

The present application further provides an embodiment of a computing device comprising a memory storing one or more computer instructions and a processor; the processor, coupled to the memory, is configured to execute the one or more computer instructions to implement the steps of the cloud desktop image transmission method or the steps of the image transmission method.

Embodiments of the present application also provide a computer-readable storage medium storing computer instructions, which, when executed by one or more processors, cause the one or more processors to perform the steps in the cloud desktop image transmission method or the steps in the image transmission method.

Yet another embodiment of the present application provides a computer program product, which includes a computer program or instructions, and when the computer program or instructions are executed by a processor, the processor executes the steps in the cloud desktop image transmission method or the steps in the image transmission method.

In the technical scheme provided by each embodiment of the application, the desktop image at the cloud desktop service end side is divided into a plurality of blocks based on the network service quality and the image content; and then, coding the plurality of blocks by utilizing the coding algorithms corresponding to the plurality of blocks respectively to obtain a plurality of coding information corresponding to the desktop image. And then respectively sending a plurality of coded information corresponding to the desktop images to the terminals. Therefore, the data volume sent by the network at each time is balanced and controllable, and the occurrence probability of burst flow is reduced, so that network congestion is avoided, and the use experience of cloud desktop users is improved.

Drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required to be utilized in the description of the embodiments or the prior art are briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present application, and other drawings can be obtained according to the drawings without creative efforts for those skilled in the art.

Fig. 1 shows a schematic diagram of a screen capture encoding process of a cloud desktop server in the prior art;

FIG. 2 illustrates a diagram of network traffic bursts as occurs in the prior art;

fig. 3 is a schematic diagram illustrating a cloud desktop system provided by an embodiment of the present application;

fig. 4 is a flowchart illustrating a cloud desktop image transmission method according to an embodiment of the present application;

fig. 5 illustrates a flow diagram of dividing a desktop image into codes and sending the codes in the cloud desktop image transmission method according to an embodiment of the present application;

FIG. 6 is a graph illustrating a comparison of a network traffic curve with a prior art curve using the solution provided by an embodiment of the present application;

fig. 7 is a schematic flowchart illustrating a cloud desktop image transmission method according to an embodiment of the present application;

FIG. 8 is a flowchart illustrating an image transmission method according to an embodiment of the present application;

fig. 9 is a schematic flowchart illustrating a cloud desktop image transmission method according to another embodiment of the present application;

fig. 10 is a schematic structural diagram illustrating a cloud desktop image transmission apparatus according to an embodiment of the present application;

fig. 11 is a schematic structural diagram of a cloud desktop image transmission apparatus according to another embodiment of the present application;

fig. 12 shows a schematic structural diagram of a computer device provided in an embodiment of the present application.

Detailed Description

The cloud desktop processing mode is mainly that pictures are transmitted through picture command streams, full-screen video streams and picture/video mixed streams, and then the pictures and the videos are rendered at a client side. The cloud desktop processing mode needs to occupy a large amount of bandwidth resources. Therefore, in the prior art, a screen of a cloud desktop is subjected to screenshot, then the intercepted desktop image is encoded, and then the encoded desktop image is sent to a terminal; and the terminal decodes and displays the data.

The terminal and the cloud desktop server establish connection based on a specified protocol (which can be called a streaming protocol). The terminal can establish connection with the cloud desktop server side by using a specified protocol. The specific protocol may be a specific cloud desktop protocol, which is a set of rules for specifying how "control information" and "data information" are exchanged between the cloud desktop server and the terminal. That is, after the connection based on the streaming protocol is established, the terminal can transmit the desktop operation data to the cloud desktop server based on the streaming protocol; the cloud desktop server can also transmit the desktop data to the terminal based on the streaming protocol. Accordingly, the connection may be considered a specific connection dedicated to transferring desktop data and other related information.

The cloud desktop is a typical cloud computing application, and is mainly used for deploying an operating system and application software required by a user through a virtual machine at a cloud end and then transmitting a desktop of a server virtual machine to a client in a remote cloud desktop mode. The client may be software installed on a hardware device (i.e., a terminal), hardware, or a browser.

When the desktop of the server virtual machine is remotely transmitted to the terminal, the common practice is to transmit the update of the screen to the terminal through a frame of picture coded by a frame, and the transmission data can be compressed by adopting a compression algorithm with a higher compression ratio. Meanwhile, the operation of external input devices such as a keyboard, a mouse, a touch pen, a touch screen and the like of the terminal can be redirected to the server virtual machine through the network, and the server virtual machine receives input to update screen information. It is a common practice to acquire operations of a mouse, a keyboard, and the like at a terminal, and then send these operation messages to a virtual machine via a network, and the virtual machine then transmits an updated desktop image to the terminal for display.

Fig. 1 illustrates a process of encoding and retransmitting a desktop image captured by a cloud desktop screen in the prior art. The process is as follows:

1. the method comprises the following steps of (1) carrying out cloud desktop screen capture to obtain a desktop image (also called screen frame data);

2. encoding the desktop image to obtain encoding information;

3. the encoded information is transmitted.

One disadvantage of the above procedure is: when the screen resolution is high and the screen change is severe, the data generated after encoding may be very large, and obvious burst traffic may be generated when the network transmits the data. As shown in fig. 2, when the coded information of the first frame of desktop image is transmitted, and when the coded information of the second frame of desktop image is transmitted, a significant traffic peak occurs. The burst-type network traffic is not beneficial to the underlying network management, and is easy to cause serious network packet loss, cause network congestion and finally affect user experience.

Accordingly, the present application provides the following embodiments to improve the problems in the prior art. In order to make the technical solutions better understood by those skilled in the art, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application.

In some of the flows described in the specification, claims, and above-described figures of the present application, a number of operations are included that occur in a particular order, which operations may be performed out of order or in parallel as they occur herein. The sequence numbers of the operations, e.g., 101, 102, etc., are used merely to distinguish between the various operations, and do not represent any order of execution per se. Additionally, the flows may include more or fewer operations, and the operations may be performed sequentially or in parallel. It should be noted that, the descriptions of "first", "second", etc. in this document are used for distinguishing different messages, devices, modules, etc., and do not represent a sequential order, nor do they limit the types of "first" and "second". In the present application, the term "or/and" is only one kind of association relationship describing the associated object, and means that three relationships may exist, for example: a or/and B, which means that A can exist independently, A and B exist simultaneously, and B exists independently; the "/" character in this application generally indicates that the objects associated with each other are in an "or" relationship. It is also noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a good or system that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such good or system. Without further limitation, an element defined by the phrases "comprising one of 8230; \8230;" 8230; "does not exclude the presence of additional like elements in articles of commerce or systems including such elements. In addition, the embodiments described below are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments obtained by a person skilled in the art based on the embodiments in the present application without making any creative effort belong to the protection scope of the present application.

Before introducing the embodiments of the method of the present application, a description will be given of a hardware system on which the embodiments of the method of the present application are based. In the embodiment shown in fig. 3, the present application provides a cloud desktop system. This cloud desktop system includes: cloud desktop server 11 and terminal 12. The cloud desktop server 11 is configured to provide a remote desktop. And the terminal 12 is in communication connection with the cloud desktop server 11 and is used for displaying the remote desktop for user operation.

The cloud desktop server 11 is used for acquiring a desktop image; according to the network service quality information and the image content of the desktop image, carrying out block division on the desktop image to obtain a plurality of blocks; determining corresponding coding algorithms for the plurality of blocks respectively; respectively coding the plurality of blocks by utilizing the coding algorithms corresponding to the plurality of blocks to obtain a plurality of coding information corresponding to the desktop image; respectively sending a plurality of coded information corresponding to the desktop images to a terminal;

the terminal 12 is configured to receive the multiple pieces of encoded information respectively sent by the cloud desktop server, and decode the multiple pieces of encoded information respectively to obtain multiple blocks; and splicing the plurality of blocks to obtain the desktop image, and displaying the desktop image.

The cloud desktop server 11 is a server device providing a cloud desktop service, and the server device may be an entity device or a virtual device. For example, the server device may be a single server, a server cluster composed of a plurality of servers, or a Virtual Machine (VM) running in a single server or a server cluster, and so on. The cloud desktop server 11 may respond to the interrupted cloud desktop service request, and provide the service related to the cloud desktop for the user. Cloud desktop services that the cloud desktop service 11 can deploy include, but are not limited to: the cloud desktop management and control system comprises the cloud desktop, an operating system of the cloud desktop, management and control services of the cloud desktop, protocol services of the cloud desktop and the like.

The terminal 12 and the cloud desktop server 11 are connected in a communication manner, for example, a mobile phone, a notebook, a tablet computer, an intelligent wearable device, a thin host, and the like. Alternatively, the terminal 12 may also be a soft terminal corresponding to a cloud desktop. A client program corresponding to the cloud desktop can be installed on the terminal 12, and a corresponding virtual machine on the cloud desktop server 11 is accessed through a desktop transport protocol to implement an interactive operation, as shown in fig. 2, so that a use experience consistent with that of an existing computer is achieved.

Further, when the terminal 12 receives the multiple pieces of encoding information respectively sent by the cloud desktop server 11 and decodes the multiple pieces of encoding information respectively to obtain multiple blocks, the terminal is specifically configured to:

after determining that a plurality of pieces of coding information containing the same desktop image identifier are received in the total number of the blocks, decoding the plurality of pieces of coding information according to the decoding algorithms corresponding to the plurality of pieces of coding information respectively to obtain a plurality of blocks;

and splicing the plurality of blocks according to the block identifications corresponding to the plurality of blocks to obtain the desktop image.

In the technical scheme provided by the embodiment, the desktop image at the cloud desktop service end side is divided into a plurality of blocks based on the network service quality and the image content; and then, coding the plurality of blocks by utilizing the coding algorithms corresponding to the plurality of blocks respectively to obtain a plurality of coding information corresponding to the desktop image. And then respectively sending a plurality of coded information corresponding to the desktop images to the terminals. Therefore, the data volume sent by the network at each time is balanced and controllable, and the occurrence probability of burst flow is reduced, so that network congestion is avoided, and the use experience of cloud desktop users is promoted.

For specific implementation of the cloud desktop server and the corresponding function of the terminal in the cloud desktop system, reference may be made to the following related contents.

Fig. 4 and fig. 5 show a flow chart of a cloud desktop image transmission method provided in an embodiment of the present application. The execution subject of the method provided by the embodiment may be a cloud desktop server. The cloud desktop server is used for providing a far-end desktop for the terminal to be operated by a terminal user. Correspondingly, the method comprises the following steps:

101. acquiring a desktop image;

102. according to the network service quality information and the image content of the desktop image, carrying out block division on the desktop image to obtain a plurality of blocks;

103. determining corresponding coding algorithms for the plurality of blocks respectively;

104. respectively coding the plurality of blocks by utilizing the coding algorithms corresponding to the plurality of blocks to obtain a plurality of coding information corresponding to the desktop image;

105. and respectively sending the plurality of coded information corresponding to the desktop image to a terminal so that the terminal can display the desktop image after decoding the plurality of coded information for user operation.

In the foregoing 101, the desktop image may be obtained in a screenshot manner. For example, when the desktop is refreshed, screenshot is performed, or screenshot may be performed according to a set frequency, which is not limited in this embodiment.

What needs to be added here is: in this embodiment, the desktop image may be a captured image of a full screen, or may be an image of one or more cloud application windows on the desktop, or the like. The cloud application runs at a cloud service end, and the application at the cloud service end can be used by the user terminal through a streaming protocol without installing the application at the user terminal.

In the above 102, the network Quality of Service (QoS) information includes at least one parameter selected from the following parameters:

bandwidth and Throughput, when the system load is serious and various features are opened, the Throughput of the system can still be ensured;

packet Loss Rate, packet Loss Rate;

delay, packet forwarding Delay;

jitter of the forwarding Delay, jitter, also called Delay Variance;

bit Error Rate, bit Error Rate.

The image content of the desktop image may include, according to the foreground and the background: application icons, desktop backgrounds, application windows, etc. on the desktop; the pattern differentiation may include: various patterns identified by image recognition techniques. For example, if the image content of the desktop image is divided according to the foreground and background, in a specific embodiment, the application icon and the desktop background may be used as the background area, and the application window may be used as the foreground area. The foreground region may be divided into one or more blocks. The background area may be divided into one or more blocks.

The sizes of the plurality of blocks divided from the desktop image may be the same or different.

In an implementation scheme, the foreground region should be a region in which the user focuses attention, and therefore, the type of the encoding algorithm corresponding to the partition divided by the foreground region may be of the first type. The background region may not be a region focused by the user, and the type of the encoding algorithm corresponding to the partition divided by the background region may be a second type. Wherein the first type of compressed image quality is higher than the second type. For example, the first type is a lossless coding type, and the second type is a lossy coding type.

Lossy coding, also called irreversible coding, refers to lossy compression of an image, which has a high compression ratio, but the decoded and reconstructed image has certain distortion with the original image.

Lossless coding is also called reversible coding, and means that a decompressed restored image is the same as an original image, but the compression ratio is not high.

Alternatively, the types of coding algorithms can be classified into predictive coding, transform coding, and statistical coding according to the coding technique. Predictive coding is to predict the current pixel value by using the high correlation of the image signal in local space and time range and taking the already-transmitted neighboring pixel value as a reference, and then quantize and code the prediction error. The transform coding is to convert the image data described in the space domain into another transform domain (frequency domain) through some orthogonal transform (such as discrete fourier transform DFT, discrete cosine transform DCT, discrete wavelet transform DWT, etc.) for description, the result after the transform is a batch of transform coefficients, and then the transform coefficients are coded, thereby achieving the purpose of compressing the image data. Statistical coding, also called entropy coding, is a type of information-preserving variable-length coding that operates on the principle of information entropy. During encoding, events (encoded symbols) having a high occurrence probability are represented by short codes, and events having a low occurrence probability are represented by long codes. In the current international standard for image coding, the common entropy coding methods are Huffman (Huffman) coding and arithmetic coding.

In the above 103, a corresponding coding algorithm may be determined for each tile based on the QoS information. For example, when the QoS information represents that the current network state is good, a coding algorithm with high compressed image quality (i.e., a low compression ratio) may be used to code each block; when the QoS information represents that the current network state is poor, an encoding algorithm with low compressed image quality (i.e., a high compression ratio) may be used to encode each block.

Here, it should be noted that: in this embodiment, the encoding algorithm corresponding to each block is not limited. In actual implementation, the machine learning model technology can be used for learning the network QoS information and the block contents of the blocks and determining the coding algorithm corresponding to each block, so as to obtain a calculation model. The input of the calculation model may be the current QoS information of the network and a plurality of blocks, and the output may be the coding algorithm corresponding to each block.

In the foregoing 105, a plurality of pieces of encoding information corresponding to the desktop image may be added to the data queue, and wait for the forwarding module to forward to the terminal.

In an implementation technical solution, the aforementioned 102 "dividing the desktop image into blocks according to the network quality of service information and the image content of the desktop image to obtain a plurality of blocks" may include:

1021. dividing the desktop image into a plurality of blocks based on image content of the desktop image;

1022. determining a coding algorithm type adapted to each block for the plurality of blocks;

1023. and combining part of the blocks according to the network service quality information and the coding algorithm types corresponding to the blocks to obtain combined blocks.

In an implementation manner, the aforementioned step 1021 "dividing the desktop image into a plurality of blocks based on the image content of the desktop image" may include:

analyzing the image content of the desktop image; and dividing the desktop image based on the image content analysis result to obtain a plurality of blocks.

As mentioned above, the foreground region and the background region may be identified by analyzing the image content of the desktop image; or a user attention area and a non-attention area; or a rectangular region corresponding to a pattern (such as sky, window, animal, person, etc.) identified by image recognition technology, etc. Then, according to the analysis result, the foreground area is divided into one or more blocks, and the background area is divided into one or more blocks. Or, the user attention area is divided into one or more blocks, and the non-attention area is divided into one or more blocks. Alternatively, the rectangular region corresponding to the recognized pattern may be divided into one or more blocks, and the remaining region other than the rectangular region corresponding to each pattern may be divided into one or more blocks.

The step 1022 "determining the type of encoding algorithm adapted to each partition for the plurality of partitions" may include:

analyzing the block contents of the blocks respectively to obtain image characteristics corresponding to the blocks; and determining the adaptive coding algorithm types for the blocks respectively according to the image characteristics corresponding to the blocks.

Further, the aforementioned step 1023 of "merging some of the multiple blocks according to the network qos information and the coding algorithm types corresponding to the multiple blocks" includes:

s1, determining a block threshold according to the network service quality information;

s2, searching at least two adjacent blocks with the same coding algorithm type based on the coding algorithm types corresponding to the blocks;

and S3, if the coding algorithm types are the same and the size of the combined block of at least two adjacent blocks is smaller than or equal to the threshold value, combining the at least two adjacent blocks with the same coding algorithm types to obtain a block.

It should be noted that: if the number of blocks generated in the above steps is too large, the area of each block is too small, which results in a low compression rate of encoding. Therefore, it is necessary to determine a block threshold according to the network QoS information, and determine the size of each area.

In specific implementation, a block threshold value corresponding to the current QoS information of the network may be calculated based on a preset algorithm. Or, based on big data experience summary, determining a block threshold value corresponding to the current QoS information. Alternatively, a deterministic model is trained using machine learning model techniques.

What needs to be supplemented is: the machine learning model mentioned here and the machine learning model mentioned above may be the same type of model, or may not be, for example, a deep learning model, a convolutional neural network model, and the like, which is not limited in this embodiment.

Further, if the coding algorithm types are the same and the size of the merged block of two adjacent blocks is larger than the threshold, the two adjacent blocks are not merged and are respectively used as a block.

In a specific implementation, when the coding algorithm type is the same and the number of the adjacent blocks is three or more, and the size of the merged block of the three or more adjacent blocks is larger than the threshold, correspondingly, the step 1023 "merge partial blocks of the blocks according to the network quality of service information and the coding algorithm types corresponding to the blocks" may further include:

grouping and merging three or more adjacent blocks with the same coding algorithm type by referring to the threshold value to obtain at least two blocks;

wherein the block size is less than the threshold.

For example, there are three partitions: segment 1, segment 2, and segment 3. Wherein, block 1 is adjacent to block 2, and block 2 is adjacent to block 3. If the block where the partition 1 and the partition 2 are merged is less than or equal to the threshold, the partition 1 and the partition 2 may be merged into one block. Partition 3 serves as another partition. Since partition 1 and partition 3 are not adjacent, partition 1 and partition 3 cannot be merged.

Further, the method provided by the embodiment of the present application may further include the following steps:

106. determining whether a trigger condition is satisfied based on the network quality of service information;

107. if the triggering condition is met, triggering and executing the step of carrying out block division on the desktop image according to the network service quality information and the image content of the desktop image;

108. and if the triggering condition is not met, the desktop image is coded, and the coded desktop image is sent to a terminal.

Fig. 7 shows a flowchart of a cloud table image transmission method provided by the embodiment of the application. With reference to fig. 7 and fig. 1, the execution subject of the embodiment may be a terminal in the cloud desktop system. Specifically, the method comprises the following steps:

201. respectively receiving a plurality of coded messages sent by a cloud desktop server; the coding information contains desktop image identification, belonging block identification and the total number of blocks corresponding to the desktop image;

202. after determining that the number of the received coded information with the same desktop image identifier is the total number of the blocks, decoding the coded information according to the decoding algorithms corresponding to the coded information respectively to obtain a plurality of blocks;

203. splicing the plurality of blocks according to the block identifications corresponding to the plurality of blocks to obtain the desktop image;

204. and displaying the desktop image.

The technical scheme provided by the embodiment of the application can be suitable for a cloud desktop scene and can also be suitable for a scene of screen sharing between any two devices. That is, the present embodiment provides an image transmission system including a first device and a second device. The first device and the second device may be the same type of device in the network, such as terminals; or different types of devices in the network, such as one of the devices being a server device and the other being a client device. The server device may be a server, a server cluster, or a virtual server deployed on the server, which is not specifically limited in this embodiment. In particular, the method comprises the following steps of,

In the technical scheme provided by the embodiment of the application, the desktop image at the first equipment side is divided into a plurality of blocks based on the network service quality and the image content; and then, coding the plurality of blocks by utilizing the coding algorithms corresponding to the plurality of blocks respectively to obtain a plurality of coding information corresponding to the desktop image. And then respectively sending a plurality of pieces of coded information corresponding to the desktop images to the second equipment. Therefore, the data volume sent by the network at each time is balanced and controllable, and the occurrence probability of burst flow is reduced, so that network congestion is avoided, and the use experience of cloud desktop users is promoted.

Further, the second device receives the plurality of encoded information sent by the first device, and decodes the plurality of encoded information to obtain a plurality of blocks; when the target image is obtained by splicing the plurality of blocks, the method is specifically configured to:

respectively receiving a plurality of pieces of coding information sent by first equipment; the coding information contains a target image identifier, a block identifier of the target image and the total number of blocks corresponding to the target image;

and splicing the blocks according to the block identifications corresponding to the blocks to obtain the target image.

As shown in fig. 8, the present embodiment provides a flowchart of an image transmission method. The execution subject of the method of the embodiment may be the first device in the image transmission system. Specifically, the method comprises the following steps:

301. acquiring a target image;

302. according to the network service quality information and the image content of the target image, carrying out block division on the target image to obtain a plurality of blocks;

303. determining corresponding coding algorithms for the plurality of blocks respectively;

304. respectively coding the blocks by utilizing the coding algorithms corresponding to the blocks to obtain a plurality of coding information corresponding to the target image;

305. and respectively sending the plurality of pieces of coding information corresponding to the target image to second equipment so that the second equipment can display the target image after decoding the plurality of pieces of coding information.

In 301, the target image may be a screenshot of the first device, or an application window screenshot on the first device, and the like, which is not specifically limited in this embodiment. Similarly, the screenshot can be performed every time the desktop is refreshed, and can also be captured according to a set frequency.

For the above 302-305, reference may be made to the description of the embodiment corresponding to fig. 4, which is not repeated herein.

Correspondingly, as shown in fig. 9, a further embodiment of the present application provides an image transmission method. The method is applicable to the second device in the image transmission system. Specifically, the method comprises the following steps:

401. respectively receiving a plurality of coded messages sent by first equipment; the coding information contains a target image identifier, a block identifier of the target image and the total number of blocks corresponding to the target image;

402. after determining that the multiple pieces of coding information containing the same target image identifier are received in the total number of the blocks, decoding the multiple pieces of coding information according to the decoding algorithms respectively corresponding to the multiple pieces of coding information to obtain multiple blocks;

403. splicing the blocks according to the block identifications corresponding to the blocks to obtain the target image;

404. and displaying the target image.

With the development of cloud computing data center technology, virtualization technology, which is an important technology in cloud computing, is widely used. By using virtualization technology, multiple Virtual Machines (VMs) can be virtualized on the physical device to virtualize the network, so that the computing resources on the physical device are fully utilized. A Virtual Machine (VM) refers to a complete computer system with complete hardware system functionality, emulated by software, running in a completely isolated environment. The work that can be done in a physical computer can be implemented in a virtual machine. When a virtual machine is created in a computer, a part of hard disk and memory capacity of a physical machine needs to be used as virtual hard disk and memory capacity. Each virtual machine may operate as if it were a physical machine. Between virtual machines, communication between the virtual machines and an external network depends on a virtual switch.

Under the cloud computing environment, computing resources are segmented, virtual machines run on a server, network cards of the virtual machines are also required to be interconnected and intercommunicated, the network cards of the virtual machines are not physical and are connected to a virtual switch through the virtual network cards, and the virtual switch forwards traffic among the virtual machines on the same server. If the virtual switch is then connected to the hardware network card of the server, the virtual machine can communicate outside the server. The virtual network card is simulated by the CPU, and is provided with one or more data queues (or receiving and transmitting queues), and the data queues are also simulated, namely a memory on the server.

In a network virtualization scenario, a Virtual Switch (VS) is a key component for implementing network virtualization in a physical device, and may provide services such as data forwarding and processing (e.g., error checking, address mapping maintenance) for a virtual network card and the physical network card, so that a communication connection may be established between each virtual machine or virtual machine in the virtual network and an external device (e.g., an external PC (Personal Computer)) outside the virtual network to implement data transmission.

The virtual machine switch is a logic and virtual switch which is presented externally by combining two physically connected switches by using a virtual switching technology. The virtual machines VM1, VM2 and VW3 and external devices (not shown in the figure) communicate with the external devices through physical network cards by connecting to the virtual switch VS through respective virtual network cards. According to the actual use requirement, one or more than two virtual network cards can be configured on the virtual machine. Generally, a virtual network card has one or more data queues for buffering data, that is, all data received by the virtual network card enters the one or more data queues, and then the virtual switch uses a processor (e.g., a Central Processing Unit (CPU)) to fetch data from the one or more data queues for Processing.

The technical scheme provided by each embodiment of the application can be realized on the premise that: the method comprises the steps that based on a first virtual network card on a cloud desktop server and a second virtual network card on a terminal, the cloud desktop server divides an intercepted desktop image into a plurality of blocks based on network service quality information and image content of the desktop image; then, coding each block by using a coding algorithm corresponding to each block; and finally, respectively sending the plurality of coded information corresponding to the desktop images to a second virtual network card of the terminal by using the first virtual network card.

In addition, each terminal can correspond to one virtual machine and is shared by the terminal. The terminal may include a keyboard, a display, a mouse, etc. The virtual machine may provide all services required by the terminal, such as desktop services, data processing services, data storage services, and so on. For example, the cloud desktop system provides a virtual machine for each user through virtualization, and the enterprise legacy application system can be accessed through an interface of the cloud desktop system. Each virtual machine can enjoy the service of the original application system of the enterprise through the cloud desktop system, such as some professional applications, mail systems, office systems, EPRs and the like of the enterprise.

In summary, embodiments of the present application provide a traffic shaping scheme for improving cloud desktop usage experience, divide a screen into a plurality of region codes according to network QoS information, and send the region codes respectively through a network, thereby avoiding sending too large coded data at one time, avoiding network congestion caused by burst traffic, and finally achieving the purpose of improving user experience.

According to the scheme, the screen frame data is divided into a plurality of areas to be coded and sent, the data volume sent by the network at each time is uniform and controllable, and burst type flow is avoided, so that network congestion can be avoided, and finally user experience can be improved.

The scheme has the function of flow shaping objectively. Fig. 6 is a schematic comparison diagram of the flow shapes corresponding to the present solution and the flow shapes not adopted in the present solution, and it can be seen that the network flow output by the present solution is relatively uniform, and there is no obvious flow peak.

Fig. 10 shows a schematic structural diagram of a cloud desktop image transmission apparatus according to an embodiment of the present application. As shown in fig. 10, the data processing apparatus in the cloud desktop system includes: the device comprises an acquisition module 21, a division module 22, a determination module 23, an encoding module 24 and a sending module 25. The obtaining module 21 is configured to obtain a desktop image. The dividing module 22 is configured to perform block division on the desktop image according to the network quality of service information and the image content of the desktop image, so as to obtain a plurality of blocks. The determining module 23 is configured to determine corresponding encoding algorithms for the blocks respectively. The encoding module 24 is configured to encode the multiple blocks by using the respective corresponding encoding algorithms of the multiple blocks, so as to obtain multiple pieces of encoding information corresponding to the desktop image. The sending module 25 is configured to send the plurality of coded information corresponding to the desktop image to a terminal, so that the terminal can display the desktop image after decoding the plurality of coded information, for a user to operate.

Further, when the partitioning module 22 performs block partitioning on the desktop image according to the network service quality information and the image content of the desktop image, the partitioning module is specifically configured to:

dividing the desktop image into a plurality of blocks based on image content of the desktop image; determining a coding algorithm type adapted to each block for the plurality of blocks; and combining part of the plurality of blocks according to the network service quality information and the coding algorithm types corresponding to the plurality of blocks to obtain a combined block.

Further, when merging the partial blocks of the multiple blocks according to the network quality of service information and the coding algorithm types corresponding to the multiple blocks, the partitioning module 22 is specifically configured to:

determining a block threshold according to the network service quality information; searching at least two adjacent blocks with the same coding algorithm type based on the coding algorithm types corresponding to the blocks; and if the coding algorithm types are the same and the size of the block after the merging of the at least two adjacent blocks is smaller than or equal to the threshold value, merging the at least two adjacent blocks with the same coding algorithm types to obtain a block.

In a specific embodiment, when the coding algorithm types are the same and the size of the block after merging the adjacent three or more blocks is larger than the threshold, correspondingly, the partitioning module 22 is further configured to, when merging some of the blocks according to the network qos information and the coding algorithm types corresponding to the blocks, further:

grouping and combining three or more adjacent blocks with the same coding algorithm type by referring to the threshold value to obtain at least two blocks;

wherein the block size is less than the threshold.

Further, when the dividing module 22 divides the desktop image into a plurality of blocks based on the image content of the desktop image, it is specifically configured to:

Further, when determining the type of the encoding algorithm adapted to each partition for the plurality of partitions, the partitioning module 22 is specifically configured to:

analyzing the block contents of the blocks respectively to obtain image characteristics corresponding to the blocks; and determining the adaptive coding algorithm types for the blocks according to the image characteristics corresponding to the blocks.

Further, the method provided by the embodiment of the present application may further include a triggering module. The trigger module is used for:

determining whether a trigger condition is satisfied based on the network quality of service information; if the triggering condition is met, triggering and executing the step of carrying out block division on the desktop image according to the network service quality information and the image content of the desktop image;

and if the triggering condition is not met, encoding the desktop image to obtain the encoded desktop image and sending the encoded desktop image to a terminal.

Here, it should be noted that: the cloud desktop image transmission apparatus provided in this embodiment may implement the technical solutions described in the foregoing cloud desktop image transmission method embodiments, and the specific implementation principles of the modules or units may refer to the corresponding contents in the foregoing method embodiments, which are not described herein again.

Fig. 11 shows a schematic structural diagram of a cloud desktop image transmission apparatus according to an embodiment of the present application. As shown in fig. 11, the data processing apparatus in the cloud desktop system includes: a receiving module 31, a determining module 32, a splicing module 33 and a display module 34. The receiving module 31 is configured to receive a plurality of pieces of encoding information sent by the cloud desktop server respectively; the coding information contains desktop image identification, belonging block identification and the total number of blocks corresponding to the desktop image. The determining module 32 is configured to determine that a plurality of pieces of coding information containing the same desktop image identifier are received in the total number of blocks, and then decode the plurality of pieces of coding information according to the decoding algorithms corresponding to the plurality of pieces of coding information, respectively, to obtain a plurality of blocks. The splicing module 33 is configured to splice the multiple blocks according to the block identifiers corresponding to the multiple blocks, so as to obtain the desktop image. The display module 34 is used for displaying the desktop image.

An embodiment of the application also provides an image transmission device. The image transmission device may include: the device comprises an acquisition module, a division module, a determination module, a coding module and a sending module. The acquisition module is used for acquiring a target image. The dividing module is used for dividing the target image into a plurality of blocks according to the network service quality information and the image content of the target image. The determining module is used for determining corresponding coding algorithms for the blocks respectively. The encoding module is used for encoding the blocks respectively by using the encoding algorithms corresponding to the blocks respectively to obtain a plurality of encoding information corresponding to the target image. The sending module is used for sending the plurality of pieces of coding information corresponding to the target image to a second device respectively, so that the second device can display the target image after decoding the plurality of pieces of coding information.

Correspondingly, another embodiment of the application provides an image transmission device. The image transmission device comprises a receiving module, a determining module, a splicing module and a display module. The receiving module is used for respectively receiving a plurality of coded messages sent by the first equipment; the coding information contains a target image identifier, a block identifier of the target image and the total number of blocks corresponding to the target image. And the determining module is used for determining that the received multiple pieces of coding information containing the same target image identifier in the total number of the blocks are decoded according to the decoding algorithms corresponding to the multiple pieces of coding information respectively to obtain multiple blocks. And the splicing module is used for splicing the blocks according to the block identifiers corresponding to the blocks to obtain the target image. The display module is used for displaying the target image.

Here, it should be noted that: the image transmission apparatus provided in this embodiment may implement the technical solutions described in the above embodiments of the image transmission method, and the specific implementation principles of the modules or units may refer to the corresponding contents in the above embodiments of the method, which are not described herein again.

The schematic structural diagram of the computing device provided by one embodiment of the application is provided. The schematic diagram of the principle structure is shown in fig. 12. In particular, the computing device includes a memory 51 and a processor 52. Wherein the memory 51 is configured to store one or more computer instructions; the processor 52 is coupled to the memory 51, and is configured to implement the at least one or more computer instructions (e.g., computer instructions for implementing data storage logic) to implement the steps in the cloud desktop image transmission method or the steps in the image transmission method provided in the embodiment of the present application.

In the above, the memory 51 may be implemented by any type of volatile or non-volatile memory device or combination thereof, such as Static Random Access Memory (SRAM), electrically Erasable Programmable Read Only Memory (EEPROM), erasable Programmable Read Only Memory (EPROM), programmable Read Only Memory (PROM), read Only Memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

The illustration of FIG. 12 is merely illustrative of some of the components that a computing device may contain and does not imply that a computing device includes only the components shown in FIG. 12, such as audio component 56, display 54, power component 55, communication component 53, and the like.

Yet another embodiment of the present application provides a computer program product (not shown in any figure of the drawings). The computer program product comprises a computer program or instructions which, when executed by a processor, causes the processor to carry out the steps in the above-described method embodiments.

Accordingly, the present application further provides a computer-readable storage medium storing a computer program, where the computer program can implement the method steps or functions provided by the foregoing embodiments when executed by a computer.

Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware. Based on the understanding, the above technical solutions substantially or otherwise contributing to the prior art may be embodied in the form of a software product, which may be stored in a computer-readable storage medium, such as ROM/RAM, magnetic disk, optical disk, etc., and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the method according to the various embodiments or some parts of the embodiments.

Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present application.

Claims

1. A cloud desktop image transmission method is suitable for a cloud desktop server, and comprises the following steps:

acquiring a desktop image;

2. The method of claim 1, wherein the block dividing the desktop image according to the network quality of service information and the image content of the desktop image to obtain a plurality of blocks comprises:

dividing the desktop image into a plurality of blocks based on image content of the desktop image;

determining a coding algorithm type adapted to each block for the plurality of blocks;

and combining part of the blocks according to the network service quality information and the coding algorithm types corresponding to the blocks to obtain combined blocks.

3. The method of claim 2, wherein merging some of the plurality of partitions according to the network quality of service information and the coding algorithm types corresponding to the plurality of partitions comprises:

determining a block threshold according to the network service quality information;

searching at least two adjacent blocks with the same coding algorithm type based on the coding algorithm types corresponding to the blocks;

and if the coding algorithm types are the same and the size of the block after the merging of the at least two adjacent blocks is smaller than or equal to the threshold value, merging the at least two adjacent blocks with the same coding algorithm types to obtain a block.

4. The method of claim 3, wherein when the coding algorithm type is the same and the number of adjacent partitions is three or more, and the block size after merging of the three or more adjacent partitions is greater than the threshold,

merging part of the plurality of blocks according to the network service quality information and the coding algorithm types corresponding to the plurality of blocks, further comprising:

wherein the block size is less than the threshold.

5. The method of any of claims 2 to 4, wherein dividing the desktop image into a plurality of partitions based on image content of the desktop image comprises:

analyzing the image content of the desktop image;

and dividing the desktop image based on the image content analysis result to obtain a plurality of blocks.

6. The method according to any of claims 2 to 4, wherein determining the type of coding algorithm adapted to each partition for the plurality of partitions comprises:

analyzing the block contents of the blocks respectively to obtain image characteristics corresponding to the blocks;

and determining the adaptive coding algorithm types for the blocks according to the image characteristics corresponding to the blocks.

7. The method of claim 1, further comprising:

determining whether a trigger condition is satisfied based on the network quality of service information;

if the triggering condition is met, triggering to execute the step of carrying out block division on the desktop image according to the network service quality information and the image content of the desktop image;

8. A cloud table image transmission method is suitable for a terminal, and comprises the following steps:

splicing the plurality of blocks according to the block identifications corresponding to the plurality of blocks to obtain the desktop image;

and displaying the desktop image.

9. A cloud desktop system, comprising:

the cloud desktop server is used for acquiring a desktop image; according to the network service quality information and the image content of the desktop image, carrying out block division on the desktop image to obtain a plurality of blocks; determining corresponding coding algorithms for the plurality of blocks respectively; respectively coding the plurality of blocks by utilizing the coding algorithms corresponding to the plurality of blocks to obtain a plurality of coding information corresponding to the desktop image; respectively sending a plurality of coded information corresponding to the desktop images to a terminal;

10. An image transmission method adapted to a first device, the method comprising:

acquiring a target image;

determining corresponding coding algorithms for the blocks respectively;

11. An image transmission method adapted to a second device, the method comprising:

respectively receiving a plurality of coded messages sent by first equipment; the coding information contains a target image identifier, an affiliated block identifier and the total number of blocks corresponding to the target image;

after determining that the multiple pieces of coding information containing the same target image identifier are received in the total number of the blocks, decoding the multiple pieces of coding information according to the decoding algorithms respectively corresponding to the multiple pieces of coding information to obtain multiple blocks;

and displaying the target image.

12. An image transmission system, comprising:

a first device for acquiring a target image; according to the network service quality information and the image content of the target image, carrying out block division on the target image to obtain a plurality of blocks; determining corresponding coding algorithms for the plurality of blocks respectively; respectively coding the plurality of blocks by utilizing the coding algorithms corresponding to the plurality of blocks to obtain a plurality of coding information corresponding to the target image; respectively sending a plurality of pieces of coding information corresponding to the target image to second equipment so that the second equipment can display the target image after decoding the plurality of pieces of coding information;

13. A computing device, comprising: a memory and a processor, wherein,

the memory storing one or more computer instructions;

the processor, coupled to the memory, configured to execute the one or more computer instructions for implementing the steps in the method of any one of claims 1 to 7, or the steps in the method of claims 8, 10 or 11.

14. A computer program product comprising a computer program or instructions which, when executed by a processor, causes the processor to carry out the steps of the method of any one of claims 1 to 7 or the steps of the method of claim 8, 10 or 11.