GB2574380A

GB2574380A - Processing image data

Info

Publication number: GB2574380A
Application number: GB1808814.6A
Authority: GB
Inventors: Charles Wilson Nicholas
Original assignee: RealVNC Ltd
Current assignee: RealVNC Ltd
Priority date: 2018-05-30
Filing date: 2018-05-30
Publication date: 2019-12-11
Also published as: GB201808814D0

Abstract

Image processing method comprising: tiling RGB image data; for each tile: determining whether applying an intra-pixel filter before compression will improve compression and if so, applying the intra-pixel filter, subsequently determining, for each of multiple inter-pixel filters, whether applying the inter-pixel filter before compression will improve compression and if so, applying the inter-pixel filter that provides the smallest compressed output; subsequently, compressing the tiles to generate and output compressed image data. Filtered and unfiltered scores may be determined and compared for each tile. Scoring may comprise computing an intermediate representation of a tile on which an arithmetic operation corresponding to filtering is performed. Intra pixel filtering may comprise colour transformation of each pixel by subtracting green channel values from red and blue channel values. Inter pixel filtering may comprise subtracting from the pixel value a predictor value based on pixels that are adjacent, above, left eg. using standard filters sub, up, paeth, average. Weighted average or gradient descent average of neighbouring pixels may be used. Saturating arithmetic may clamp predictor values. Scan lines for each colour channel may be extracted from tiles and sent for compression. May be applied in pre-processing to optimise compression.

Description

Processing Image Data

TECHNICAL FIELD

The present disclosure relates to processing image data on a computing device.

BACKGROUND

It is known to use a local computer or similar device to interact with a physically remote computer. For the individual user, one common scenario is to help troubleshoot the computer of a distant less-technically-savvy relative. Sitting at your desk in Baltimore, remote computing allows a user to take control of a relative's PC in California and show them how to use a new software package by actually doing it from your PC in Baltimore. A very common business application is in remote system administration, where it is used to allow administrators to take control of employee machines to diagnose and fix problems. In all scenarios, key presses, mouse movements, and mouse clicks are sent from the local computer to the remote computer to effect changes on the remote computer. A continually updated copy of the remote computer’s display is shown in a window on the local computer. Accordingly, it appears to a user that they are sitting in front of the remote computer even though it is at a different location. The continually updated copy is achieved by encoding pixel data describing the contents of the remote computer’s display and transmitting this data, or any changes in this data, to the local computer. There are many ways of achieving this encoding. At one end of the range, there is Raw encoding which has no data compression and thus requires minimal CPU overhead at the remote computer but is only suitable for fast networks because of the large volume of data to be transmitted. Higher compression encoding (e.g. Run Length Encoding RLE or variations thereof) offers better compression and are suitable for slower networks but require higher CPU overhead. These standard encodings are all lossless and thus the display on the local computer is an exact representation of the display on the remote computer.

The use of filters to pre-condition input image data to make it more compressible is well-known, and is used for example in the Portable Network Graphics (PNG) image format which makes use of the Sub, Up, Average and Paeth filter types before the DEFLATE (zlib) compression stage. Another known image format which uses filters is the Multiple-image Network Graphics (MNG) image format.

SUMMARY

The inventors have identified techniques to improve the compression of image data which results in smaller compressed image data than compared to known techniques. Dependent on the context, embodiments of the present disclosure provide numerous advantages. For example, the compressed image data may be output to local storage of a computing device (prior to decoding), in this scenario embodiments of the present disclosure advantageously reduce the storage requirements of the computing device. In a remote computing scenario whereby the compressed image data is output for transmission via a network to a remote computing device where it is decoded for output on a display device associated with the remote computing device, the reduced size of the compressed image data advantageously reduces the transmission time and incurs less network bandwidth.

According to one aspect of the present disclosure there is provided a method of processing image data on a computing device, the method comprising: receiving image data having a red channel, green channel and a blue channel; apportioning the image data into at least one tile, each tile comprising a portion of said image data; preconditioning the at least one tile prior to compression by a compression module of the computing device, by, for each tile: processing said tile by determining if applying a predetermined intra-pixel filter to the tile prior to the compression module compressing the tile will improve compression by providing a compressed output from the compression module that is smaller in size as compared to a compressed output generated from the compression module based on compressing the tile in unfiltered form, and if so, applying the predetermined intra-pixel filter to the tile; after said processing, determining for each of a plurality of predetermined inter-pixel filters whether applying the predetermined inter-pixel filter to the tile prior to the compression module compressing the tile will improve compression by providing a compressed output from the compression module that is smaller in size as compared to the compressed output generated from the compression module based on compressing the tile in unfiltered or intra-pixel-filtered form; if it is determined that at least one inter-pixel filter of the plurality of predetermined inter-pixel filters provides a compressed output from the compression module that is smaller in size as compared to a compressed output generated from the compression module based on compressing the tile in unfiltered or intra-pixel-filtered form, applying an inter-pixel filter of the at least one inter-pixel filter to the tile prior to the compression module compressing the tile, wherein said inter-pixel filter provides the smallest compressed output from the compression module; after said preconditioning, the compression module compressing each of the at least one tile to generate compressed image data, and outputting the compressed image data.

Determining if applying a predetermined intra-pixel filter to the tile prior to the compression module compressing the tile will improve compression may comprise: determining a score of the tile in unfiltered form; determining a score of the tile filtered using said predetermined intra-pixel filter; and determining that applying the predetermined intra-pixel filter to the tile prior to the compression module compressing the tile is likely to improve compression based on a comparison of the score of the tile filtered using said predetermined intra-pixel filter and the score of the tile in unfiltered form.

The determining a score of the tile filtered using said predetermined intra-pixel filter may comprise filtering the tile using said predetermined intra-pixel filter to generate an intra-pixel filtered tile, and scoring said intra-pixel filtered tile to determined said score.

The determining a score of the tile filtered using said predetermined intra-pixel filter may comprise computing a representation of the tile, and performing an arithmetic operation on said representation that corresponds to filtering the tile using said predetermined intra-pixel filter to determine said score without filtering the tile using said predetermined intra-pixel filter

The determining for each of a plurality of the predetermined inter-pixel filters whether applying the predetermined inter-pixel filter to the tile prior to the compression module compressing the tile will improve compression may comprise: determining a score of the tile filtered using the predetermined inter-pixel filter; and determining that applying the predetermined inter-pixel filter to the tile prior to the compression module compressing the tile is likely to improve compression based on a comparison of the score of the tile filtered using the predetermined inter-pixel filter exceeding the score of the tile in unfiltered form.

The determining a score of the tile filtered using the predetermined inter-pixel filter may comprise filtering the tile using said predetermined inter-pixel filter to generate an inter-pixel filtered tile, and scoring said inter-pixel filtered tile to determined said score.

The determining a score of the tile filtered using said predetermined inter-pixel filter may comprise computing a representation of the tile, and performing an arithmetic operation on said representation that corresponds to filtering the tile using said predetermined inter-pixel filter to determine said score without filtering the tile using said predetermined inter-pixel filter.

The intra-pixel filter may apply a colour transformation to each pixel of the tile.

For each pixel said colour transformation may comprise subtracting a green channel value of the pixel from a red channel value of the pixel, and subtracting the green channel value of the pixel from a blue channel value of the pixel.

The plurality of predetermined inter-pixel filters may each operate by, for each of at least one pixel in the tile, subtracting a predictor value from a pixel value of said pixel being filtered to generate a filtered output, said predictor value determined based on a pixel value of at least one neighbouring pixel in the tile that neighbours said pixel being filtered.

The plurality of predetermined inter-pixel filters may comprise an inter-pixel filter wherein said predictor value is determined based on a pixel value of a pixel that is adjacent to said pixel being filtered, and is positioned at a left-side of said pixel being filtered.

The plurality of predetermined inter-pixel filters may comprise an inter-pixel filter wherein said predictor value is determined based on a pixel value of a pixel that is adjacent to said pixel being filtered, and is positioned at an upper-side of said pixel being filtered

The plurality of predetermined inter-pixel filters may comprise an inter-pixel filter wherein said predictor value is determined based on: a pixel value of a pixel that is adjacent to said pixel being filtered, and is positioned at a left-side of said pixel being filtered; a pixel value of a second pixel that is adjacent to said pixel being filtered, and is positioned at an upper-side of said pixel being filtered; and a pixel value of a third pixel that is adjacent to said pixel being filtered, and is positioned at an upper left-side of said pixel being filtered.

The predictor value may be determined based on a weighted average of (i) the pixel value of said pixel, (ii) the pixel value of said second pixel, and (iii) the pixel value of said third pixel.

The predictor value may be determined based on a gradient descent average of (i) the pixel value of said pixel, (ii) the pixel value of said second pixel, and (iii) the pixel value of said third pixel.

The inter-pixel filter may apply saturating arithmetic to clamp the predictor value between a minimum and maximum level.

The plurality of predetermined inter-pixel filters may consist of three predetermined inter-pixel filters, wherein the three predetermined inter-pixel filters each operate by, for each pixel in the tile, subtracting a predictor value from a pixel value of said pixel being filtered to generate a filtered output.

The three predetermined inter-pixel filters may include: a first inter-pixel filter wherein said predictor value is determined based on a pixel value of a pixel that is adjacent to said pixel being filtered, and is positioned at a left-side of said pixel being filtered; a second inter-pixel filter wherein said predictor value is determined based on a pixel value of a pixel that is adjacent to said pixel being filtered, and is positioned at an upper-side of said pixel being filtered; and a third inter-pixel filter wherein said predictor value is determined based on: a pixel value of a pixel that is adjacent to said pixel being filtered, and is positioned at a left-side of said pixel being filtered; a pixel value of a second pixel that is adjacent to said pixel being filtered, and is positioned at an upper-side of said pixel being filtered; and a pixel value of a third pixel that is adjacent to said pixel being filtered, and is positioned at an upper left-side of said pixel being filtered, wherein the third inter-pixel filter applies saturating arithmetic to clamp the predictor value between a minimum and maximum level.

The method may further comprise: after said preconditioning, extracting scan lines of image data from the at least one tile and supplying the extracted scan lines of image data to the compression module for said compressing.

The extracted scan lines of image data may each comprise pixel data stored in a row or column of pixels, and the pixel data of all of the scan lines is in an order associated with a single direction across the at least one tile.

The extracted scan lines of image data may each comprise pixel data stored in a row or column of pixels, and the pixel data of a first set of the scan lines is in an order associated with a first direction across the at least one tile, and the pixel data of a second set of the scan lines is in an order associated with a second direction across the at least one tile, wherein the second direction is the opposite of the first direction.

The extracted scan lines of image data may comprise: a first set of scan lines comprising red channel pixel values of the image data from the at least one tile; a second set of scan lines comprising blue channel pixel values of the image data from the at least one tile; and a third set of scan lines comprising red channel pixel values of the image data from the at least one tile.

One or more tile of said at least one tile may have a tile size of 64x64 pixels.

The compression module may output the compressed image data to local storage on said computing device for output on a display device associated with said computing device.

The compression module may output the compressed image data for transmission via a network to a remote computing device for output on a display device associated with said remote computing device.

The compression module may compress one or more of the at least one tile to generate compressed image data using a lossless compression. The lossless compression may be Zstandard.

The compression module may compress one or more of the at least one tile to generate compressed image data using lossy compression.

According to another aspect of the present disclosure there is provided a computer program product for processing image data, the computer program product comprising code embodied on a non-transient computer-readable medium and configured so as when executed on a processor of a computing device to perform the method steps described herein.

The code may be provided on a carrier such as a disk, CD- or DVD-ROM, programmed memory such as read-only memory (Firmware), or on a data carrier such as an optical or electrical signal carrier. Code (and/or data) to implement embodiments of the present disclosure may comprise source, object or executable code in a conventional programming language (interpreted or compiled) such as C, or assembly code, code for setting up or controlling an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array), or code for a hardware description language.

According to another aspect of the present disclosure there is provided a computing device for processing image data, the computing device comprising: a tile generator configured to (i) receive image data having a red channel, green channel and a blue channel; and (ii) apportion the image data into at least one tile, each tile comprising a portion of said image data; a compression module; and a preconditioning stage configured to precondition the at least one tile prior to compression by the compression module, wherein the preconditioning stage is configured, for each tile, to: process said tile by determining if applying a predetermined intrapixel filter to the tile prior to the compression module compressing the tile will improve compression by providing a compressed output from the compression module that is smaller in size as compared to a compressed output generated from the compression module based on compressing the tile in unfiltered form, and if so, apply the predetermined intra-pixel filter to the tile; after said processing, determine for each of a plurality of predetermined inter-pixel filters whether applying the predetermined inter-pixel filter to the tile prior to the compression module compressing the tile will improve compression by providing a compressed output from the compression module that is smaller in size as compared to the compressed output generated from the compression module based on compressing the tile in unfiltered or intra-pixel-filtered form; if it is determined that at least one inter-pixel filter of the plurality predetermined inter-pixel filters provides a compressed output from the compression module that is smaller in size as compared to a compressed output generated from the compression module based on compressing the tile in unfiltered or intra-pixel-filtered form, apply an inter-pixel filter of the at least one inter-pixel filter to the tile prior to the compression module compressing the tile, wherein said inter-pixel filter provides the smallest compressed output from the compression module; wherein the compression module is configured, after said preconditioning, to compress each of the at least one tile to generate compressed image data, and output the compressed image data.

These and other aspects will be apparent from the embodiments described in the following. The scope of the present disclosure is not intended to be limited by this summary nor to implementations that necessarily solve any or all of the disadvantages noted.

BRIEF DESCRIPTION OF THE DRAWINGS

For a better understanding of the present disclosure and to show how embodiments may be put into effect, reference is made to the accompanying drawings in which: Figure 1 is schematic block diagram of a remote computing system;

Figure 2 is a schematic block diagram of an application executed on a processor of a computing device;

Figure 3 is a flow chart of a process performed by the application in accordance with embodiments described herein;

Figures 4a and 4b illustrate image data for processing by the application;

Figures 5a-c illustrates examples of how scan lines may be extracted for supply to a compression stage of the application; and

Figure 6 shows the labelling of neighbouring pixels adjacent to a pixel to be filtered.

DETAILED DESCRIPTION

Embodiments will now be described by way of example only. In particular, to illustrate the concepts of the present disclosure, embodiments are described below with processing image data in the context of a remote computing system. However this is merely an example and embodiments may extend to processing other types of image data which does not correspond to image data that has been displayed on a local computing device and is to be transmitted to a remote computing device. For example, the image data processed on a computing device in accordance with embodiments of the present disclosure may have been captured by a webcam coupled to the computing device.

Figure 1 shows the components of a remote computing system 100 comprising a computing device 102 connected via a data link 106 to a remote computing device 104.

The computing device 102 comprises central processing unit (CPU) 110, memory 120 in the form of random access memory (RAM), and a graphics processor unit (GPU) 130 which may perform vector processing such that it is capable of performing the same operation on multiple pieces of data simultaneously. As shown in Figure 1, the computing device 102 may be coupled to an external display 140 via a wired or wireless connection. In other embodiments, the display 140 is an internal component of the computing device 102. It will be appreciated that the computing device 102 may include various other components not shown in Figure 1 for reasons of clarity.

As explained in more detail below, a VNC Server application 112 is running on the CPU 110 to initiate a capture of an image displayed on the display 140 and send it via the data link 106 to the remote computing device 104, thus the computing device 102 may be termed a VNC Server. It will be appreciated that the image data displayed on the display 140 will change over time as a user of the computing device operates the computing device 102.

The remote computing device 104 comprises a CPU 150 connected to the data link 106 and a display 160. It will be appreciated that the remote computing device 104 may include various other components not shown in Figure 1 for reasons of clarity. A corresponding VNC Viewer application 152 is running on CPU 150 to receive image data displayed on the display 140 via the data link 106 and output it on the remote computing device display 160, thus the remote computing device 104 may be termed a VNC Viewer.

The data link 106 may be a wired connection (e.g. Ethernet, USB connection) or a wireless connection (e.g. Wi-Fi®, Bluetooth®, Zigbee®, Cellular).

Referring back to the computing device 102, as shown in Figure 1 the RAM 120 comprises a screenshot buffer 122. The GPU 130 comprises a frame buffer 134 which is coupled to the display 140. When an image is to be output to the display 140, the image is rendered into the frame buffer 134 in the GPU 130. The image data in the frame buffer 134 is then sent to the display 140 for output. The GPU 130 further comprises a screen capture module 132 coupled to the frame buffer 134.

In one embodiment, the VNC Server application 112 is configured to receive a screen update request that has been transmitted by the remote computing device 104 over the data link 106. The screen update request is received at the computing device 102 at an interface and supplied to the CPU 110 on which the VNC Server application 112 is running. It will be appreciated that the type of interface will depend on the type of data link 106 that is used, such interfaces are well known to persons skilled in the art and are therefore not discussed in detail herein.

As shown in Figure 1 the CPU 110 and GPU 130 are able to communicate via a suitable connection. In response to receiving the screen update request, the VNC Server application 112 instructs the GPU 130 to capture a screenshot of the image data that is being displayed on display 140. The communication between the VNC Server application 112 and the screen capture module 132 may be implemented using an API.

Upon receiving the instruction from the VNC Server application 112, the screen capture module 132 communicates with the frame buffer 134 to capture the image data (otherwise referred to herein as display data) that is displayed on display 140 and places the captured image data into the screenshot buffer 122 (replacing any image data currently stored in the screenshot buffer 122). That is, the screen capture module 132 obtains a screenshot from the frame buffer 134.

The VNC Server application 112 may retrieve and transmit all of the captured image data in the screenshot buffer 122. Alternatively, the VNC Server application 112 may determine (using methods that are outside the scope of the present disclosure) that the screenshot buffer 122 comprises image data of a region of the display 140 that has changed since the previous screen capture (this image data is termed a “rectangle” in the present disclosure) and only rectangles are retrieved from the screenshot buffer 122 for transmission. The term “rectangle” is used herein to refer to n by m pixels of image data, where m and n could be equal (i.e. ‘rectangle’ includes the possibility of ‘square’) or non-equal. The term “rectangle” is discussed in more detail below.

In another embodiment, the VNC Server application 112 is configured to periodically check the frame buffer 134 for screen updates, without needing a screen update request message to arrive from the VNC Viewer 152.

Thus it will be apparent from the above that the VNC Server application 112 receives image data. We now describe how this received image data is processed by the VNC Server application 112 with reference to Figures 2 and 3.

As shown in Figure 2 the VNC Server application 112 comprises a plurality of modules. The plurality of modules comprise a tile generator module 202; a preconditioning stage comprising: an intra-pixel filter selection module 204, a scoring module 206, and an inter-pixel filter selection module 208; a scan line arrangement module 210 and a data compression module 212.

The tile generator module 202 is configured to receive input image data. In the context of a remote computing session, this image data may correspond to at least one rectangle of image data that has been displayed on display 140. The term “rectangle” is described in more detail below with reference to Figures 3 and 4.

The tile generator module 202 is configured to apportion the rectangle into at least one tile. The term “tile” is described in more detail below with reference to Figures 3 and 4.

The tile generator module 202 is configured to output the at least one tile to the intra-pixel filter selection module 204.

The intra-pixel filter selection module 204 is coupled to the scoring module 206. For each of the received tiles, the intra-pixel filter selection module 204 is configured to communicate with the scoring module to determine whether or not the tile should be filtered using the intra-pixel filter selection module 204. If the intra-pixel filter selection module 204 determines that a tile is to be filtered using the intra-pixel filter selection module 204, then the intra-pixel filter selection module 204 performs intra-pixel filtering on the tile and then outputs the intra-pixel filtered tile to the inter-pixel filter selection module 208.

If the intra-pixel filter selection module 204 determines that a tile is not to be filtered using the intra-pixel filter selection module 204, then the intra-pixel filter selection module 204 does not perform intra-pixel filtering on the tile and outputs the un-filtered tile to the inter-pixel filter selection module 208.

The inter-pixel filter selection module 208 is configured to receive the tiles that are output from the intra-pixel filter selection module 204. It will be appreciated from the above that each tile received at the inter-pixel filter selection module 208 may have been previously intra-pixel filtered using the intra-pixel filter selection module 204 or no intra-pixel filtering has been applied to the tile.

The inter-pixel filter selection module 208 is coupled to the scoring module 206. For each of the received tiles, the inter-pixel filter selection module 208 is configured to communicate with the scoring module to determine whether or not the tile should be filtered using the inter-pixel filter selection module 208. If the inter-pixel filter selection module 208 determines that a tile is to be filtered using the inter-pixel filter selection module 208, then the inter-pixel filter selection module 208 performs inter-pixel filtering on the tile and then outputs the inter-pixel filtered tile to storage e.g. RAM 120, or to the scan line arrangement module 210. If the inter-pixel filter selection module 208 determines that a tile is not to be filtered using the inter-pixel filter selection module 208, then the inter-pixel filter selection module 208 does not perform interpixel filtering on the tile and outputs the tile to storage.

The inventor has observed that per-tile filtering provides improved compression over known techniques such as the PNG format which performs per-line inter-pixel filtering only. Furthermore performing the intra-pixel filtering before the inter-pixel filtering improves the prediction quality.

Once all of the tiles of a rectangle have been processed by the intra-pixel filter selection module 204 and the inter-pixel filter selection module 208, the scan line arrangement module 210 extracts scan lines of image data from the plurality of tiles and supplies the extracted scan lines of image data to the data compression module 212 for compressing.

The data compression module 212 performs compression on the received scan lines and outputs compressed image data. As noted above, the compressed image data may be output to local storage on the computing device 102 (prior to decoding) or the data compression module 212 may output the compressed image data for transmission via a network on data link 106 to the remote computing device 104 where it is decoded for output on the display 160.

The data compression module 212 may perform lossless compression. For example the data compression module 212 may implement Zstandard lossless compression. The choice of Zstandard provides slightly improved compression ratio, and much higher compression speed than the DEFLATE compression in the PNG format. Alternatively, the data compression module 212 may perform lossy compression, for example by permitting error in the encoding of the residuals to reduce compressed size at the expense of some quality.

Reference is now made to Figure 3 which is a flow chart for a process 300 performed by the VNC Server application 112 when executed on the CPU 110.

The functionality of the VNC Server application 112 described herein may be implemented in code (software) stored on a memory (of the computing device 102) comprising one or more storage media, and arranged for execution on the CPU 110 comprising one or more processing units. The code is configured so as when fetched from the memory and executed on the CPU 110 to perform operations in line with embodiments discussed herein. The code may be provided on a carrier such as a disk, CD or DVD-ROM, programmed memory such as nonvolatile memory (e.g. Flash) or read-only memory (Firmware).

At step S302, the VNC Server application 112, in particular the tile generator module 202, receives image data (for example from the screenshot buffer 122) in the form of a rectangle.

The term “rectangle” is used herein to refer to n by m pixels of image data, and m and n could be equal (i.e. includes the possibility of the rectangle being ‘square’) ornon-equal.

The rectangle may comprise (i) image data of a first frame of image data, wherein the frame is to be, but has not yet been encoded by the VNC Server application 112; (ii) all changed image data (that has changed since the previous screen capture); or (iii) a mix of changed image data and unchanged image data (that has not changed since the previous screen capture).

Figure 4a illustrates a first rectangle 402 of image data of an image 400 and a second rectangle 404 of image data of the image 400. It will be appreciated from Figure 4a that the tile generator module 202 may receive rectangles of different sizes (in terms of width/height of pixels). That is, a rectangle does not have a determinate size.

At step S304, the tile generator module 202 apportions a received rectangle into at least one tile which entirely covers the rectangle.

The term “tile” is used herein to refer to n by m pixels of image data, and m and n could be equal (i.e. includes the possibility of the tile being ‘square’) or non-equal.

A tile comprises a portion of image data of its associated rectangle. This portion may be the whole of the image data of the rectangle in the example where the rectangle is apportioned into (e.g. assigned as) a single tile. Alternatively this portion may be a sub-portion of the image data of the rectangle in the example where the rectangle is apportioned (e.g. divided into) multiple tiles. Expressed another way, the size of a tile may be less than or equal to its associated rectangle. The step of apportioning the received rectangle may comprise dividing the rectangle into a plurality of non-overlapping tiles however it is possible that a tile may have the same size as a rectangle.

A tile is an arrangement of pixels that preserves spatial and colour information and is used as input to intra-pixel and inter-pixel filtering phases described in more detail below. It is preferably similar to the pixel arrangement used in the rectangle, to avoid the cost of re-arranging pixels in memory.

In contrast, a scanline is an arrangement of pixel values that may have been transformed by an intra-pixel filter and/or an inter-pixel filter and is used as input to the data compression module 212.

The tile generator module 202 may be configured, upon receipt of a received rectangle, to attempt to apportion the rectangle into at least one tile comprising tiles having a predetermined size (a predetermined height and a predetermined width). This predetermined size may be 64x64 pixels however it will be appreciated that this is merely an example and other predetermined sizes may be used. There is a trade-off between using larger tiles which eventually reduce the prediction quality and using smaller tiles which can slightly harm compression quality.

Figure 4b shows how, upon receipt of the rectangle 402, the tile generator module 202 apportions the rectangle 402 into a plurality of tiles 406. Using the top-left hand corner of the rectangle 402 as a starting point, the tile generator module 202 attempts to apportion the rectangle 402 into tiles of the predetermined size (e.g. 64x64 pixel). As shown in the example of Figure 4b this is possible for tiles 406a, 406b, 406c. As shown in Figure 4b, the remaining area of the rectangle 402 cannot be apportioned into tiles of 64x64 pixels. The tile generator module 202 then attempts to apportion the remaining area of the rectangle 402 into tiles having either a height equalling the predetermined height or a width equalling the predetermined width. As shown in the example of Figure 4b this is possible for tile 406d (having a height equalling the predetermined height), and for tiles 406e, 406f, 406g (having a width equalling the predetermined width). As shown in the example of Figure 4b this not possible for the tile 406h.

Similarly, Figure 4b shows how, upon receipt of the rectangle 404, the tile generator module 202 apportions the rectangle 404 into a plurality of tiles 408. Using the top-left hand corner of the rectangle 404 as a starting point, the tile generator module 202 attempts to apportion the rectangle 404 into tiles of the predetermined size (e.g. 64x64 pixel). As shown in Figure 4b, the rectangle 404 cannot be apportioned into tiles of 64x64 pixels so the tile generator module 202 then attempts to apportion the rectangle 404 into tiles having either a height equalling the predetermined height or a width equalling the predetermined width. As shown in the example of Figure 4b this is possible for tile 408a, 408b, and 408c (having a width equalling the predetermined width). As shown in the example of Figure 4b this not possible for the tile 408d.

To illustrate the concepts above, tile 406a is shown in Figure 4b as having a width of 64 pixels and a height of 64 pixels, and tile 408a is shown in Figure 4b as having a width of 64 pixels and a height of 50 pixels.

It will be appreciated that a tile of a first rectangle of image data may actually be larger than a second rectangle of image data.

For all rectangles having a size that is larger than one scan line, the tile generator module 202 will output at least one tile that is larger than a scan line for these rectangles.

After step S304, the process 300 enters a loop at step S306 where the intra-pixel filter selection module 204 and the inter-pixel filter selection module 208 process each of the plurality of tiles. Whilst Figure 3 shows the processing of each of the tiles being in series for simplicity, it will be appreciated that the intra-pixel filter selection module 204 and the inter-pixel filter selection module 208 may process the plurality of tiles in parallel.

At step S306, the intra-pixel filter selection module 204 determines a score for a first tile of the plurality of tiles by communicating with the scoring module 206. The score provides an indication of the size of the compressed tile when compressed by the data compression module 212.

The score for a tile may be the mathematical sum of the score of the component bytes of each pixel in the tile, where the score of a component byte is the absolute value of the byte when interpreted as an 8-bit signed (twos-complement) integer. Thus the score for the tile is calculated byte-by-byte by the scoring module 206. So however a pixel is represented, the score simply uses those bytes (although typically each byte contains one colour channel, bytes may contain mixed bits e.g. in an RGB565 image with the scoring not taking into consideration pixel representation).

Taking a simple example to illustrate the concept, given a pixel which is represented by the four byte values of: 100, 200, 10, 250, (which are typically RGBA components taking a byte each) these byte values are interpreted as 8-bit signed integers to have the mathematical values 100, -56, 10, -6. The scoring module 206 then takes the absolute values 100, 56, 10, 6, which sum to 172. So if the tile consisted of just this pixel, the score for the tile would be 172.

The scoring procedure above is described in the PNG specification, RFC 2083, section 9.6. It will be appreciated that embodiments of the present disclosure are not limited to this scoring method and other alternative tile scoring methods may be used.

In the above described scoring method, lower scores are “better” in that they indicate a smaller size of the compressed tile when compressed by the data compression module 212. This is because a low score in this scoring method indicates a pixel distribution with a high probability of repeated sequences which are likely to be compressed to produce a smaller compressed output. However it will be appreciated in alternative scoring methods this may not be the case e.g. a higher score may indicate a smaller size of the compressed tile when compressed by the data compression module 212.

The process 300 then proceeds to steps S308 and S310 to determine whether or not to filter the tile at the intra-pixel filter selection module 204 using a predetermined intra-pixel filter which applies a colour-space transformation to pixels of a tile. The reference to “intra-pixel” is used to indicate that the colour-space transformation operates on values of an individual pixel without using values of any neighbouring pixels which neighbour (are adjacent to) the individual pixel.

The purpose of an intra-pixel filter is to remove redundancy inside individual pixel values. The observation is that the RGB pixel values are correlated, and in areas with high luminosity (high green), the distribution of RB values is heavily weighted. Therefore, applying a colour-space transformation ought to reduce the redundancy inherent in the data. When an intra-pixel filter intra-pixel filters a pixel, the pixel is filtered by the intra-pixel filter applying a function to each colour channel.

In some embodiments the intra-pixel filter selection module 204 comprises a single intra-pixel filter.

The single intra-pixel filter may apply a colour-space transformation which is described by the LOCO-I format: RgGBg, in which the colour components are (R - G, G, B - G). That is, the intra-pixel filter operating on each pixel of a tile subtracts the green channel pixel value from the red channel pixel value and also subtracts the green channel pixel value from the blue channel pixel value.

The single intra-pixel filter may alternatively apply a colour-space transformation: YCoCg which involves a transformation of an associated RGB colour space into a luma value (denoted as Y) and two chroma values called chrominance green (Cg) and chrominance orange (Co). The YCoCg transformation may be used in the H.264 format.

In the context of image data captured during a remote computing session, it was found that although the YCoCg model provides higher-quality separation of the colour components, it was found to provide no better compression than RgGBg, which is faster to compute, thus in embodiments where the tiles correspond to image data captured during a remote computing session, the intra-pixel filter is preferably the RgGBg intra-pixel filter.

It will be appreciated that the single intra-pixel filter may apply an alternative colour-space transformation to those described above.

At step S308 the intra-pixel filter selection module 204 determines a score for the tile filtered using the predetermined intra-pixel filter of the intra-pixel filter selection module 204 by communicating with the scoring module 206.

At step S310 the intra-pixel filter selection module 204 determines whether applying the predetermined intra-pixel filter to the tile prior to the data compression module 212 compressing the tile is likely to improve compression based on a comparison of the score of the tile filtered using the predetermined intra-pixel filter (determined at step S308) and the score of the tile in unfiltered form (determined at step S306).

If it is determined that applying the predetermined intra-pixel filter to the tile prior to the data compression module 212 compressing the tile is likely to improve compression, then at step S310, the intra-pixel filter selection module 204 filters the tile using the single intra-pixel filter.

If it is determined that applying the predetermined intra-pixel filter to the tile prior to the data compression module 212 compressing the tile is not likely to improve compression, then no intra-pixel filtering is performed by the intra-pixel filter selection module 204.

At step S308, the intra-pixel filter selection module 204 may perform intra-pixel filtering on the tile to output an intra-pixel filtered tile to RAM 120 and score the intra-pixel filtered tile to determine the score. In this example, if is determined at step S310 that the predetermined intrapixel filter is to be applied, the intra-pixel filter selection module 204 need not recompute the filtered values for the tile at step S310.

However the inventor has identified that it is not necessary that the predetermined intra-pixel filter is applied to the tile to generate an intra-pixel filtered tile in order to make the determination at step S308. For example, in an alternative method the intra-pixel filter selection module 204 may compute an intermediate representation of the tile, and perform an arithmetic operation on the intermediate representation that corresponds to filtering the tile using the predetermined intra-pixel filter to determine the score without filtering the tile using the predetermined intrapixel filter. This alternative method advantageously incurs less memory resource of the computing device 102 as the filtered tile is not written out to RAM 120 during the scoring procedure. The intermediate representation may or may not be re-used at step S310 to obtain the intra-pixel filtered tile if it is determined that applying the predetermined intra-pixel filter to the tile prior to the data compression module 212 compressing the tile is likely to improve compression

Whilst Figure 3 shows that the intra-pixel filter is applied if the score of the tile filtered using the predetermined intra-pixel filter (determined at step S308) is less than the score of the tile in unfiltered form (determined at step S306) this is related to a scoring method whereby lower scores are “better” in that they indicate a smaller size of the compressed tile when compressed by the data compression module 212, and as noted above alternative scoring methods may be used which may affect the decision logic implemented at step S310.

The process 300 then proceeds to steps S312 to S314 to determine whether or not to filter the tile at the inter-pixel filter selection module 208 using one of a plurality of predetermined interpixel filters. It will be appreciated that the tile received at the inter-pixel filter selection module 208 will either have been filtered using the predetermined intra-pixel filter of the intra-pixel filter selection module 204 or will be in unfiltered form. The reference to an “inter-pixel” filter is used to indicate that the inter-pixel filter filters an individual pixel using the value of the individual pixel and also the values of at least one neighbouring pixel which neighbours (is adjacent to) the individual pixel.

In particular, with reference to Figure 6 an inter-pixel filter filters an individual pixel P 602 by subtracting a predictor value from the pixel value of the individual pixel P 602. The predictor value may be based on one or more of: (i) a pixel value of a pixel ‘a’ 604 that is adjacent to pixel P 602, and is positioned to the left of pixel P 602, (ii) a pixel value of a pixel ‘b’ that is adjacent to pixel P 602, and is positioned above pixel P 602, and a pixel value of a pixel ‘c’ that is adjacent to pixel P 602, and is positioned above-left of pixel P 602.

In PNG, there are four inter-pixel filters which operate by subtracting from each pixel, a predictor value calculated from its neighbours. The four filters are described in the PNG specification (PNG specification, RFC 2083, March 1997 accessible at https://tools.ietf.org/html/rfc2083) and are summarised below:

• sub, which uses the adjacent pixel to the left as the predictor • up, which uses the pixel above • Average, which uses the arithmetic mean of the left and above pixels • Paeth, which uses the exact value of the left, above, or above-left pixel, based on a linear interpolation. The “Paeth principle” is to choose the closest adjacent pixel value based on the linear predictor (1,1, -1), where the vector (x, y, z) represents the weights assigned to the (left, above, above-left) pixel values i.e. Predictor(a,b,c) = value in {a,b,c} closest to a + b - c.

The known LOCO-I compression scheme incorporates the following inter-pixel filter:

• Given values (a, b, c) for the adjacent pixel values in the (left, above, above-left) positions, then:

If c > max(a, ti) then predict min(a, b);

otherwise if c < min(a, b) then predict max (a, b);

otherwise predict a + b - c.

Further details on the LOCO-I compression scheme can be found in The LOCO-I Lossless Image Compression Algorithm: Principles and Standardization into JPEG-LS M.Weinberger, G.Seroussi, and G.Sapiro, Hewlett Packard Report HPL-98-193R1, November 1998, revised October 1999, accessible at http://www.hpl.hp.com/loco/.

A predictor may return an identical value to an existing pixel, for example in the Paeth filter and two of the branches of the loco-i filter. The advantage of this is that the predictor does not introduce any new pixel values, so for solid regions the residual pixel values (the pixel values after subtraction of the predictor) are more likely to have the same values as other residuals, so that the distribution of the residuals may comprise fewer distinct values, which may improve compression. The disadvantage is that for many common pixel distributions (such as photographic data), this advantage is outweighed by another effect of these branches on the distribution of residuals, namely that the mean size of the residuals is larger, which may hinder compression.

The inventor has identified a selection of inter-pixel filters that provides an improvement to the compression as compared to the known compression schemes.

In the context of a remote computing session it was found that configuring the inter-pixel filter selection module 208 with exactly three predetermined inter-pixel filters was found to provide an optimum trade-off between improving the compression of the data compression module 212 and time incurred in order to analyse the possible compression benefits of filtering the tile using each of the plurality of predetermined inter-pixel filters. It will be appreciated that in other implementations two, or more than three, predetermined inter-pixel filters may be used.

The three predetermined inter-pixel filters may comprise a first inter-pixel filter termed “Left” which uses the adjacent pixel to the left of the pixel being filtered as the predictor, and a second inter-pixel filter termed “up which uses the adjacent pixel above the pixel being filtered as the predictor. These inter-pixel filters correspond to “sub and Up respectively found in PNG. The first inter-pixel filter and second inter-pixel filter are simple to implement and are computationally cheap to implement.

The inventor has found that the Paeth principle is disadvantageous, namely that using the exact adjacent pixels values is disadvantageous in that the compressed output is larger than compared to using the arithmetic output of the linear filter.

Thus a third inter-pixel filter termed “Wav” is used which uses an arithmetically-interpolated predictor value as the predictor based on the linear predictor (1, 1, -1), where the vector (x, y, z) represents the weights assigned to the (left, above, above-left) pixel values i.e. Predictor(a, b, c) = a + b - c.

The inventor has further identified that using saturating arithmetic provides a clear (and principled) improvement to overall compression. Thus the third inter-pixel filter may use saturating arithmetic such that Predictor(a, b, c) = saturate(a + b - c).

As an example, consider the linear predictor (1, 1, -1) when applied to the pixel values (250, 250, 220): the arithmetic value is 250 + 250 - 220 = 280, which must be reduced to a single byte in the range [0,255], Both PNG and LOCO-I use wrapping arithmetic (which in this case would predict 24), but a more accurate predictor value is possible using saturating arithmetic (and hence predict 255).

It will be appreciated that for the pixels on an outer edge of a tile, for some of these pixels (e.g. pixels in the top left of a tile) an inter-pixel filter wouldn’t be able to apply one or more of the three inter-pixel filters, e.g. the “Up” inter-pixel filter does not apply to the top row of pixels in a tile, and the “Left” filter does not apply to the left-most column of pixels in a tile. Thus in embodiments certain inter-pixel filters skip the filtering of certain pixels in a tile (no calculation is performed), or if there is no pixel at a position referenced by the inter-pixel filter, the calculation is performed as if the adjacent pixel had the value zero.

As an alternative to one of the inter-pixel filters described above or in addition to the plurality of predetermined inter-pixel filters described above, the plurality of predetermined inter-pixel filters may comprise an inter-pixel filter which uses a predictor value that is determined based on a weighted average of the left, above, and above-left pixel values of pixels(corresponding to pixels 604, 606, 608 in Figure 6) neighbouring the pixel to be filtered. For example weighted average schemes such as (3, 3, 2) + 8, (2, 2, 1)-^5, where the vector (x, y, z) represents the weights assigned to the (left, above, above-left) pixel values, may be used as the predictor. These schemes average the adjacent values but with heavier weight on the nearer left/above values. Division by 8 is a fast operation.

As an alternative to one of the inter-pixel filters described above or in addition to the plurality of predetermined inter-pixel filters described above, the plurality of predetermined inter-pixel filters may comprise an inter-pixel filter which uses a predictor value that is determined based on a gradient descent average of the left, above, and above-left pixel values of pixels(corresponding to pixels 604, 606, 608 in Figure 6) neighbouring the pixel to be filtered. For example gradientdescent average schemes, such as (3, 3, -1) + 5, (5, 5, -2) + 8, where the vector (x, y, z) represents the weights assigned to the (left, above, above-left) pixel values, may be used as the predictor. These are based on the visualisation of the image as a two-dimensional surface where we take a first-order gradient approximation to provide the predictor.

The weighted average inter-pixel filter and gradient-descent average inter-pixel filter described above can be clamped to neighbouring values as with Paeth, or saturated (as with Wav), or clamped using LOCO-l's more complex scheme.

Referring back to Figure 3, at step S312 for each of the plurality of the predetermined inter-pixel filters, the inter-pixel filter selection module 208 determines a score for the tile filtered using the respective predetermined inter-pixel filter of the inter-pixel filter selection module 208 by communicating with the scoring module 206. Taking the example whereby the inter-pixel filter selection module 208 includes the three predetermined inter-pixel filters “Left”, “up and “Wav”, at step S312 the inter-pixel filter selection module 208 determines a first score for the tile filtered using the “Left” inter-pixel filter, a second score for the tile filtered using the “Up” interpixel filter, and a third score for the tile filtered using the “Wav” inter-pixel filter.

At step S314 the inter-pixel filter selection module 208 determines whether applying one of the plurality of predetermined inter-pixel filters to the tile prior to the data compression module 212 compressing the tile is likely to improve compression based on a comparison of the scores of the tile filtered using the predetermined inter-pixel filters (determined at step S312) and the score of the tile received at the inter-pixel filter selection module 208.

The score of the tile received at the inter-pixel filter selection module 208 is the score determined at step S308 if the tile was intra-pixel filtered at the intra-pixel filter selection module 204, or is the score determined at step S306 if the tile was not intra-pixel filtered at the intrapixel filter selection module 204.

If it is determined that applying one of the plurality of predetermined inter-pixel filters to the tile prior to the data compression module 212 compressing the tile is likely to improve compression, then at step S316, the inter-pixel filter selection module 208 filters the tile using the inter-pixel filter with the “best” score, and the process 300 then proceeds to S318.

That is, of the inter-pixel filters which provide a better score than the score of the tile received at the inter-pixel filter selection module 208, the inter-pixel filter selection module 208 selects (from these) the inter-pixel filter with the “best” score. What is regarded as the “best score” is dependent on the scoring method used, lower scores may be “better” in that they indicate a smaller size of the compressed tile when compressed by the data compression module 212 however this is merely an example.

If it is determined that applying one of the plurality of predetermined inter-pixel filters to the tile prior to the data compression module 212 compressing the tile is not likely to improve compression, then no inter-pixel filtering is performed by the inter-pixel filter selection module 208 and the process 300 then proceeds to S318.

At step S312, for each of the plurality of predetermined inter-pixel filters, the inter-pixel filter selection module 208 may perform inter-pixel filtering on the tile to output an inter-pixel filtered tile to RAM 120 and score the inter-pixel filtered tile to determine the score for the tile associated with the particular inter-pixel filter. In this example, if is determined at step S314 that a predetermined inter-pixel filter is to be applied, the inter-pixel filter selection module 208 need not recompute the filtered values for the tile at step S316.

However the inventor has identified that it is not necessary that a predetermined inter-pixel filter is applied to the tile to generate an inter-pixel filtered tile in order to make the determination at step S312. For example, in an alternative method the inter-pixel filter selection module 208 may compute an intermediate representation of the tile, and perform an arithmetic operation on the intermediate representation that corresponds to filtering the tile using the predetermined interpixel filter to determine the score without filtering the tile using the predetermined inter-pixel filter. This alternative method advantageously incurs less memory resource of the computing device 102 as the filtered tile is not written out to RAM 120 during the scoring procedure. The intermediate representation may or may not be re-used at step S316 to obtain the inter-pixel filtered tile if it is determined that applying one of the plurality of predetermined inter-pixel filters to the tile prior to the data compression module 212 compressing the tile is likely to improve compression.

To give an example of an inter-pixel filter that would benefit: the Wav filter uses saturation, so it may be advantageous to represent pixel values of OxOORRGGBB in registers as 0X000000RR00GG00BB (i.e. unpack each byte to 16-bit depth), then perform the arithmetic on the unpacked representation (e.g. using vector processing) and sum the components to obtain the score - thus obtaining the score without ever having re-packed the intermediate 16-bit representation.

Whilst Figure 3 shows that an inter-pixel filter is applied if the score of the tile filtered using one of the plurality of predetermined inter-pixel filters (determined at step S312) is less than the score of the tile received at the inter-pixel filter selection module 208, this is related to a scoring method whereby lower scores are “better” in that they indicate a smaller size of the compressed tile when compressed by the data compression module 212, and as noted above alternative scoring methods may be used which may affect the decision logic implemented at step S314.

The scoring described above is used to determine which inter-pixel filter (if any) to use, and which intra-pixel filter (if any). This avoids compressing each tile multiple times, allowing an estimate to be obtained, before compression, of which filter is likely to provide the smallest compressed output.

At step S318, the processed tile is stored in memory (e.g. RAM 120) on the computing device 102. It will be appreciated that the processed tile may be unfiltered, filtered by the intra-pixel filter selection module 204 only, filtered by the inter-pixel filter selection module 208 only, or filtered by both the intra-pixel filter selection module 204 and the inter-pixel filter selection module 208.

As indicated by step S320, the loop of steps S306 to S320 is repeated until all tiles of the input image data have been processed.

At step S322, the scan line arrangement module 210 is configured to arrange the image data of each of the processed tiles into scan lines and supply the scan lines to the data compression module 212.

The inter-pixel filter selection module 208 may supply the processed tiles to the scan line arrangement module 210. Alternatively, the inter-pixel filter selection module 208 may instruct the scan line arrangement module 210 to retrieve the processed tiles from memory.

Figure 5a illustrates a tile 406,408 comprising 24 pixels 416 (this is merely an example to illustrate the concept). For the tile, the scan line arrangement module 210 may extract a first scan line 502 corresponding to a top row of pixels, a second scan line 504 corresponding to a further row of pixels, a third scan line 506 corresponding to a further row of pixels, and a fourth scan line 508 corresponding to a bottom row of pixels. As shown in Figure 5a, the pixel data of all of the scan lines is in an order associated with a single scan direction (left-to-right) across the plurality of tiles. As an alternative, the pixel data of all of the scan lines may be in an order associated with a scan direction (right-to-left) across the plurality of tiles.

The scan line arrangement module 210 may implement a pixel value interlacing technique to improve the size of the compressed output, by creating more uniform runs of pixel values which are processed by the data compression module 212 more efficiently.

For example, the scan line arrangement module 210 may reverse the odd-numbered scanlines, so that the scan direction is right-to-left then left-to-right for alternating rows, to bring together like pixel values. This is shown in Figure 5b whereby the first scan line 502 has been reversed to generate a new first scan line 510, and the third scan line 506 has been reversed to generate a new third scan line 512.

In another example, as shown in Figure 5c the scan line arrangement module 210 may deinterlace the R/G/B components of the pixels in the tile, so that the tile is represented by an array 520 of R scanlines, an array 522 of G scanlines, and an array 524 of B scanlines, to bring together like pixel values.

The scan line arrangement module 210 may perform other interlacing or pixel reordering techniques than those described herein to bring together like pixel values

Whilst the scan lines have been described above as corresponding to rows of pixels in a tile, the scan lines may alternatively correspond to columns of pixels in a tile.

At step S324, the data compression module 212 receives the scan lines of each of the tiles and performs compression on the scan lines to generate compressed image data which is output by the data compression module 212.

As noted above, the data compression module 212 may perform lossless compression. For example the data compression module 212 may implement Zstandard lossless compression. The choice of Zstandard provides slightly improved compression ratio, and much higher compression speed than the DEFLATE compression in the PNG format. Alternatively, the data compression module 212 may perform lossy compression, for example by permitting error in the encoding of the residuals to reduce compressed size at the expense of some quality.

At a corresponding decoder (which may be on computing device 102 or on a remote computing device 104, the decoder reads in a filter specification (for each compressed tile), then the compressed image data (e.g. a Zstandard stream created by the encoder). The compressed image data is decompressed, then the decoder un-applies the filters by reversing the filter transformation.

The process 300 allows at most one intra-pixel filter and one inter-pixel filter to be chosen, and the filters are chosen independently for each tile of the received image data. The intra-pixel filter (if any) is applied first by the encoder, followed by the inter-pixel filter (if any); the decoder unapplies any inter-pixel filter first, then any intra-pixel filter.

The process 300 operates by pre-conditioning the image data using filters, which remove arithmetic redundancy, after which the image data is passed to the data compression module 212.

Whilst embodiments have been described above with reference to the data compression module 212 performing lossless compression, it will be appreciated that as an alternative the data compression module 212 may perform lossy compression.

The intra-pixel filter selection module 204 may comprise a chroma sub-sampling intra-pixel filter to replace, or be used as an addition to, the RgGBg intra-pixel filter described above. In operation, the chroma sub-sampling intra-pixel filter performs chroma subsampling after doing a colour-space transform to luminance/chroma colour space.

In embodiments, when the data compression module 212 is operating in a lossy mode (the data compression module 212 performs lossy compression) to improve the compression the chroma sub-sampling intra-pixel filter would be used as the intra-pixel filter in the process 300 described above, and when the data compression module 212 is operating in a lossless mode (the data compression module 212 performs lossless compression) the RgGBg intra-pixel filter would be used as the intra-pixel filter in the process 300 described above. The data compression module 212 may be configured to operate in a single mode or switch between the lossy and lossless modes. The switching between modes may be implemented on a per-tile or per-frame basis.

Generally, any of the functions described herein can be implemented using software, firmware, hardware (e.g., fixed logic circuitry), or a combination of these implementations. The terms “functionality”, “module” and “application” as used herein generally represent software, firmware, hardware, or a combination thereof. In the case of a software implementation, the functionality or application represents program code that performs specified tasks when executed on a processor (e.g. CPU or CPUs). The program code can be stored in one or more computer readable memory devices. The features of the techniques described below are 5 platform-independent, meaning that the techniques may be implemented on a variety of commercial computing platforms having a variety of processors.

Whilst embodiments have been described with reference to steps performed by the VNC Server application 112 it will be appreciated that embodiments are not limited to processing image data 10 in the context of a remote computing system, thus the VNC Server application 112 can be considered more generally as an encoder.

While the present disclosure has been particularly shown and described with reference to preferred embodiments, it will be understood to those skilled in the art that various changes in 15 form and detail may be made without departing from the scope of the present disclosure as defined by the appendant claims.

Claims

CLAIMS:

1. A method of processing image data on a computing device, the method comprising: receiving image data having a red channel, green channel and a blue channel; apportioning the image data into at least one tile, each tile comprising a portion of said image data;

preconditioning the at least one tile prior to compression by a compression module of the computing device, by, for each tile:

processing said tile by determining if applying a predetermined intra-pixel filter to the tile prior to the compression module compressing the tile will improve compression by providing a compressed output from the compression module that is smaller in size as compared to a compressed output generated from the compression module based on compressing the tile in unfiltered form, and if so, applying the predetermined intra-pixel filter to the tile;

after said processing, determining for each of a plurality of predetermined inter-pixel filters whether applying the predetermined inter-pixel filter to the tile prior to the compression module compressing the tile will improve compression by providing a compressed output from the compression module that is smaller in size as compared to the compressed output generated from the compression module based on compressing the tile in unfiltered or intra-pixel-filtered form;

if it is determined that at least one inter-pixel filter of the plurality of predetermined inter-pixel filters provides a compressed output from the compression module that is smaller in size as compared to a compressed output generated from the compression module based on compressing the tile in unfiltered or intra-pixel-filtered form, applying an inter-pixel filter of the at least one inter-pixel filter to the tile prior to the compression module compressing the tile, wherein said inter-pixel filter provides the smallest compressed output from the compression module;

after said preconditioning, the compression module compressing each of the at least one tile to generate compressed image data, and outputting the compressed image data.

2. The method of claim 1, wherein determining if applying a predetermined intra-pixel filter to the tile prior to the compression module compressing the tile will improve compression comprises:

determining a score of the tile in unfiltered form;

determining a score of the tile filtered using said predetermined intra-pixel filter; and determining that applying the predetermined intra-pixel filter to the tile prior to the compression module compressing the tile is likely to improve compression based on a comparison of the score of the tile filtered using said predetermined intra-pixel filter and the score of the tile in unfiltered form.

3. The method of claim 2, wherein said determining a score of the tile filtered using said predetermined intra-pixel filter comprises filtering the tile using said predetermined intra-pixel filter to generate an intra-pixel filtered tile, and scoring said intra-pixel filtered tile to determined said score.

4. The method of claim 2, wherein said determining a score of the tile filtered using said predetermined intra-pixel filter comprises computing a representation of the tile, and performing an arithmetic operation on said representation that corresponds to filtering the tile using said predetermined intra-pixel filter to determine said score without filtering the tile using said predetermined intra-pixel filter

5. The method of any of claims 2 to 4, wherein determining for each of a plurality of the predetermined inter-pixel filters whether applying the predetermined inter-pixel filter to the tile prior to the compression module compressing the tile will improve compression comprises:

determining a score of the tile filtered using the predetermined inter-pixel filter; and determining that applying the predetermined inter-pixel filter to the tile prior to the compression module compressing the tile is likely to improve compression based on a comparison of the score of the tile filtered using the predetermined inter-pixel filter exceeding the score of the tile in unfiltered form.

6. The method of claim 5, wherein said determining a score of the tile filtered using the predetermined inter-pixel filter comprises filtering the tile using said predetermined inter-pixel filter to generate an inter-pixel filtered tile, and scoring said inter-pixel filtered tile to determined said score.

7. The method of claim 5, wherein said determining a score of the tile filtered using said predetermined inter-pixel filter comprises computing a representation of the tile, and performing an arithmetic operation on said representation that corresponds to filtering the tile using said predetermined inter-pixel filter to determine said score without filtering the tile using said predetermined inter-pixel filter

8. The method of any preceding claim, wherein the intra-pixel filter applies a colour transformation to each pixel of the tile.

9. The method of claim 8, wherein for each pixel said colour transformation comprises subtracting a green channel value of the pixel from a red channel value of the pixel, and subtracting the green channel value of the pixel from a blue channel value of the pixel.

10. The method of any preceding claim, wherein the plurality of predetermined inter-pixel filters each operate by, for each of at least one pixel in the tile, subtracting a predictor value from a pixel value of said pixel being filtered to generate a filtered output, said predictor value determined based on a pixel value of at least one neighbouring pixel in the tile that neighbours said pixel being filtered.

11. The method of claim 10, wherein the plurality of predetermined inter-pixel filters comprise an inter-pixel filter wherein said predictor value is determined based on a pixel value of a pixel that is adjacent to said pixel being filtered, and is positioned at a left-side of said pixel being filtered.

12. The method of claim 10 or 11, wherein the plurality of predetermined inter-pixel filters comprise an inter-pixel filter wherein said predictor value is determined based on a pixel value of a pixel that is adjacent to said pixel being filtered, and is positioned at an upper-side of said pixel being filtered

13. The method of any of claims 10 to 12, wherein the plurality of predetermined inter-pixel filters comprise an inter-pixel filter wherein said predictor value is determined based on:

a pixel value of a pixel that is adjacent to said pixel being filtered, and is positioned at a left-side of said pixel being filtered;

a pixel value of a second pixel that is adjacent to said pixel being filtered, and is positioned at an upper-side of said pixel being filtered; and a pixel value of a third pixel that is adjacent to said pixel being filtered, and is positioned at an upper left-side of said pixel being filtered.

14. The method of claim 13, wherein said predictor value is determined based on a weighted average of (i) the pixel value of said pixel, (ii) the pixel value of said second pixel, and (iii) the pixel value of said third pixel.

15. The method of claim 13, wherein said predictor value is determined based on a gradient descent average of (i) the pixel value of said pixel, (ii) the pixel value of said second pixel, and (iii) the pixel value of said third pixel.

16. The method of any of claims 11 to 15, wherein inter-pixel filter applies saturating arithmetic to clamp the predictor value between a minimum and maximum level.

17. The method of any preceding claim, wherein the plurality of predetermined inter-pixel filters consist of three predetermined inter-pixel filters, wherein the three predetermined interpixel filters each operate by, for each pixel in the tile, subtracting a predictor value from a pixel value of said pixel being filtered to generate a filtered output.

18. The method of claim 17, wherein the three predetermined inter-pixel filters include:

a first inter-pixel filter wherein said predictor value is determined based on a pixel value of a pixel that is adjacent to said pixel being filtered, and is positioned at a left-side of said pixel being filtered;

a second inter-pixel filter wherein said predictor value is determined based on a pixel value of a pixel that is adjacent to said pixel being filtered, and is positioned at an upper-side of said pixel being filtered; and a third inter-pixel filter wherein said predictor value is determined based on:

a pixel value of a second pixel that is adjacent to said pixel being filtered, and is positioned at an upper-side of said pixel being filtered; and a pixel value of a third pixel that is adjacent to said pixel being filtered, and is positioned at an upper left-side of said pixel being filtered, wherein the third inter-pixel filter applies saturating arithmetic to clamp the predictor value between a minimum and maximum level.

19. The method of any preceding claim, further comprising: after said preconditioning, extracting scan lines of image data from the at least one tile and supplying the extracted scan lines of image data to the compression module for said compressing.

20. The method of claim 19 wherein the extracted scan lines of image data each comprise pixel data stored in a row or column of pixels, and the pixel data of all of the scan lines is in an order associated with a single direction across the at least one tile.

21. The method of claim 19 wherein the extracted scan lines of image data each comprise pixel data stored in a row or column of pixels, and the pixel data of a first set of the scan lines is in an order associated with a first direction across the at least one tile, and the pixel data of a second set of the scan lines is in an order associated with a second direction across the at least one tile, wherein the second direction is the opposite of the first direction.

22. The method of claim 19 wherein the extracted scan lines of image data comprise:

a first set of scan lines comprising red channel pixel values of the image data from the at least one tile;

a second set of scan lines comprising blue channel pixel values of the image data from the at least one tile; and a third set of scan lines comprising red channel pixel values of the image data from the at least one tile.

23. The method of any preceding claim, wherein one or more tiles of said at least one tile has a tile size of 64x64 pixels.

24. The method of any preceding claim, wherein the compression module outputs the compressed image data to local storage on said computing device for output on a display device associated with said computing device.

25. The method of any preceding claim, wherein the compression module outputs the compressed image data for transmission via a network to a remote computing device for output on a display device associated with said remote computing device.

26. The method of any preceding claim, wherein the compression module compresses one or more of the at least one tile to generate compressed image data using a lossless compression.

27. The method of claim 26, wherein the lossless compression is Zstandard

28. The method of any preceding claim, wherein the compression module compresses one or more of the at least one tile to generate compressed image data using lossy compression.

29. A computer program product for processing image data, the computer program product comprising code embodied on a non-transient computer-readable medium and configured so as when executed on a processor of a computing device to perform the method of any of claims 1 to 28.

30. A computing device for processing image data, the computing device comprising:

a tile generator configured to (i) receive image data having a red channel, green channel and a blue channel; and (ii) apportion the image data into at least one tile, each tile comprising a portion of said image data;

a compression module; and a preconditioning stage configured to precondition the at least one tile prior to compression by the compression module, wherein the preconditioning stage is configured, for each tile, to:

process said tile by determining if applying a predetermined intra-pixel filter to the tile prior to the compression module compressing the tile will improve compression by providing a compressed output from the compression module that is smaller in size as compared to a compressed output generated from the compression module based on compressing the tile in unfiltered form, and if so, apply the predetermined intra-pixel filter to the tile;

after said processing, determine for each of a plurality of predetermined interpixel filters whether applying the predetermined inter-pixel filter to the tile prior to the compression module compressing the tile will improve compression by providing a compressed output from the compression module that is smaller in size as compared to the compressed output generated from the compression module based on compressing the tile in unfiltered or intra-pixel-filtered form;

if it is determined that at least one inter-pixel filter of the plurality predetermined inter-pixel filters provides a compressed output from the compression module that is smaller in size as compared to a compressed output generated from the compression module based on compressing the tile in unfiltered or intra-pixel-filtered form, apply an inter-pixel filter of the at least one inter-pixel filter to the tile prior to the compression module compressing the tile, wherein said inter-pixel filter provides the smallest compressed output from the compression module;

wherein the compression module is configured, after said preconditioning, to compress each of the at least one tile to generate compressed image data, and output the compressed image data.