GB2429890A

GB2429890A - Display processing system using a tree to represent windows

Info

Publication number: GB2429890A
Application number: GB0517683A
Authority: GB
Inventors: Neil Turton
Original assignee: Ant Software Ltd
Current assignee: Ant Software Ltd
Priority date: 2005-08-31
Filing date: 2005-08-31
Publication date: 2007-03-07
Also published as: GB0517683D0

Abstract

A method of rendering a plurality of overlapping image display elements, the method comprising: processing said image display elements according to a hierarchy defined by a tree, nodes of the tree representing said image display elements and links between said nodes defining said hierarchy of said elements, a state of said tree defining processed elements of said tree, said processing rendering said processed display elements into a screen frame buffer for display; wherein the method further comprises: identifying an image display element for offscreen processing; processing a portion of said identified image display element using a temporary buffer; writing a result of said element portion processing into said screen frame buffer; and repeating said element portion processing and result writing to process said identified element portion-by-portion.

Description

jplay Processing Systems This invention generally relates to display

processing methods, apparatus and computer program code, and more particularly to thiproved techniques for displaying overlapping screen elements such as document parts or windows.

To illustrate one of the problems which embodiments of the invention address we consider the CSS3 Color Module Specification, for example the W3C draft of 14th May 2003 Oip://www.w3.orgfrpJ2oo3/553 color'00305 14). Cascading Style Sheets (CSS) is a language for describing the rendering of HTML and XML documents on screen and the specification contains, among other things, a definition for an opacity property which governs how opacity processing is performed on an element. A style sheet is generally part of a Document Object Model (DOM), which provides an interface to facilitate access to and modification of the content, stnicture and/or style of a document.

When opacity processing is active, an element and its descendents are rendered into an RGBA (Red, Green, Blue, Alpha) off-screen image before being blended into the current composite rendering; the opacity settings specifies how the blending is fanned.

Typically the resulting colour of the blended combination is a function of the two (or more) blended layers. The alpha value ranges from zero (fully transparent) to one (fully opaque), and in the composite image maybe employed hardware support, for example to combine the composite image with a TV or video backplane. As described later, in some preferred embodiments the elements comprise HTML document elements or windows of a browser forming part of a television system, and the composite rendering is displayed on the screen of a television or other multimedia entertainment device,

optionally over a background moving picture.

Conventionally the opacity processing is applied recursively to any applicable elements, but this approach is problematic, particularly in an embedded system, since the size of the off-screen image is bounded only by the size of the (display) screen. Similar problems are encountered with other techniques which require a portion of a document to be rendered into a off-screen buffer, for example those used to produce visual effects such as blun-ing, chroma keying, brightness adjustment, rotation, and other image modifications, According to the present invention there is therefore provided a method of rendering a plurality of overlapping image display elements, the method comprising: processing said image display elements according to a hierarchy defined by a tree, nodes of the tree representing said image display elements and links between said nodes defining said hierarchy of said elements, a state of said tree defining processed elements of said tree, said processing rendering said processed display elements into a screen frame buffer for display; wherein the method further comprises: identifying an image display element for offscreen processing; processing a portion of said identified image display element using a temporary buffer; writing a result of said element portion processing into said screen frame buffer; and repeating said element portion processing and result writing to process said identified element portion-by-portion.

Broadly speaking, in embodiments of the context of a hierarchy of windows or other document elements defined by a tree, when off-screen processing is needed only a small section or portion of the element is rendered at a time to enable re-use of the off-screen image (buffer) for each portion. This enables the size of the off-screen image to be restricted, for example to a reasonable size for a system in which the method is implemented. Preferably, prior to performing the off-screen processing the state of the tree is pushed on to a stack or otherwise temporarily saved, and afterwards popped or restored. This facilitates implementation of embodiments of the method iteratively so that if, for example, a processed element has a descendent which also uses the temporary frame buffer then this descendent can be partitioned into portions and processed similarly. The process may be applied iteratively for multiple descendents, if needed.

The elements may comprise HTML (Hyper Text Mark up Language) or XML (eXtended Mark up Language) documents or document (screen) elements, or windows in a windowed display system, or other GUI (Graphical User Interface) components. In embodiments the image display elements are described by a document, window structure or other data structure which defines how the elements are to be processed (redrawn) into a screen frame buffer, typically forming part of a video display system.

Embodiments of the method are particularly suitable for embedded systems where available memory may be limited.

In preferred embodiments the temporary is smaller than the screen frame buffer but large enough (in terms of memory) to hold the processed portion of the image display element identified for off-screen processing. The size of a portion for processing may either be determined during the processing of the image display elements or a memory area of the expected size may be allocated in advance; the size of a portion is a balance between the size of the temporary buffer and the number of portions needed to process an image display element.

Preferably the method further comprises saving a state of the tree prior to the portion- by-portion processing of the element identified for off-screen processing, and restoring the saved tree state after this processing. This facilitates iterative implementation of embodiments of the method and therefore preferred embodiments further comprise processing a sub-tree of elements rooted at the identified element for each processed portion of the identified element. Thus where it is identified that a said element portion overlaps another element for off-screen processing the current state of the element portion processing is saved and halted whilst the second element is processed (portion- by-portion if needed, depending upon the size), after which the saved state is restored and the portion-by-portion processing is continued. The temporary frame buffer may be re-used (for example, if it is saved) or another temporary frame buffer may be allocated.

In preferred embodiments the off-screen processing includes copying each portion of the identified element into the temporary buffer and then processing the copied element portions off-screen in the temporary buffer (depending upon the complexity of the processing operation), and the result writing may then comprise blending off-screen image data in the temporary buffer with data in the screen frame buffer, for example to performed opacity processing. The skilled person will recognise, however, that embodiments of these techniques may also be used to produce other visual effects which require a portion of a document or window to be rendered into an off-screen buffer, including but not limited to one or more of sharpening, blurring, chroma keying, brightness adjustment, contrast adjustment, gamma adjustment, embossing, engraving, shadow effects, radiance, distortion, rippling, rotation and other image modification or enhancement operations.

The invention further provides processor control code to implement the above-described systems and methods, in particular on a data carrier such as a disk, CD- or DVD-ROM, programmed memory such as read-only memory (Firmware), or on a data carrier such as an optical or electrical signal carrier. Code (and/or data) to implement embodiments of the invention may comprise source, object or executable code in a conventional programming language (interpreted or compiled) such as C, or assembly code, code for setting up or controlling an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array), or code for a hardware description language such as Verilog (Trade Mark) or VHDL (Very high speed integrated circuit Hardware Description Language). As the skilled person will appreciate such code and/or data may be distributed between a plurality of coupled components in communication with one another.

The invention further provides a computer system for rendering a plurality of overlapping image display elements by processing said image display elements according to a hierarchy defined by a tree, nodes of the tree representing said image display elements and links between said nodes defining said hierarchy of said elements, a state of said tree defining processed elements of said tree, said processing rendering said processed display elements into a screen frame buffer for display, the system comprising: data memory for storing data for said image display elements data representing said tree, said state of said tree, said screen frame buffer, and a temporary buffer; program memory storing processor control code; and a processor coupled to said data memory, and to said program memory to load and implement said processor control code, said code comprising code for controlling the processor to: identify an image display element for offscreen processing; process a portion of said identified image display element using a temporary buffer; write a result of said element portion processing into said screen frame buffer; and repeat said element portion processing and result writing to process said identified element portion-by-portion.

Some particular embodiments of the system may be implemented, for example, on a set- top box (STB) or as part of a computer system associated with or incorporated within a television or other home multimedia entertainment system. In such embodiments the above described methods may be implemented as part of a mark-up language browser such as an HTML or XML browser, and the screen display may comprise a television screen display, the data in the screen frame buffer overlying a video display. In other similar embodiments the system comprises part of a video or DVD (digital versatile disc) player or home computer. The system (and embodiments of the above described method) may also be implemented in a portable device such as a mobile communications device or portable audio and/or video player.

These and other aspects of the invention will now be further described, by way of example only, with reference to the accompanying figures in which: Figure 1 shows an example set of windows and the corresponding tree; Figure 2 shows a flow diagram of a procedure embodying an aspect of the present invention; and Figure 3 shows a block diagram of a system within which the procedure of figure 2 may be implemented.

The method we describe applies in particular to a redraw algorithm which is non- recursive. This is a tree traversal algorithm, where every window which is encountered is redrawn if it intersects with the invalid region. (As the skilled person will understand, generally speaking in the context of a windowed display system invalidation of a region causes a re-draw an invalid region specifies an area to redraw, and a valid flag is set of the re-draw.) When a window (call it window A) which requires opacity processing is encountered, the current state is pushed onto a stack and a temporary frame buffer is allocated. The region of the sub-tree rooted at window A which needs redrawing is determined by taking the intersection between the sub-tree's bounding box with the invalid region.

A portion of this region which is no larger than the size of the temporary buffer is chosen to be processed. This portion is removed from the region of window A which need processing. The affected part of the frarnebuffer is copied into the temporary framebuffer. The sub-tree rooted at window A is then redrawn, either into the temporary buffer or into the framebuffer depending on the implementation. When the tree traversal algorithm arrives back at window A, the temporary buffer is blended into the framebuffer. The region which needs redrawing is then checked and if any more needs redrawing, another portion is chosen and the process is repeated.

When the sub-tree rooted at window A has been redrawn, the state is removed (popped) from the stack and the remainder of the window tree/hierarchy is redrawn as normal.

The procedure also works when there is a window requiring opacity processing which is nested inside another which requires opacity processing. When the outer window is processed, the state is pushed onto the stack and traversal continues into the sub-tree.

When the inner window is encountered, the updated state is pushed onto the stack and that window and its descendants are redrawn. When that is complete, the state from the inner window is removed from the stack and the remainder of the outer window's descendants are redrawn. The result is blended and another portion of the outer window is chosen. The process may then be repeated as necessary.

Referring now to figure 1, this shows an example screen display 10 comprising three windows A, B, C and a corresponding tree describing the display and illustrating the hierarchy of the windows within the display (the root of the tree). The tree shows that window A contains the visible part of B (which is clipped), generally referred to using the shorthand "A contains B". It can be seen that although A is in front of both B and C, A contains B but does not contain C. As described further later, window A is partitioned in to a number of regions 12 labelled 1,..n.

Referring now to figure 2, this shows a flow diagram of a window processing procedure. Conceptually the redrawing begins at the front and proceeds towards the back, but in practice a two-pass procedure is implemented, at step S200 performing a first pass traversing the windows from front to back to identify what to redraw (that is, to identify what is visible), and then at step S202 performing a second pass from back to front to perform the redrawing. A state of this procedure is represented by a pointer to a node of the tree identifying a window currently being processed, and a list of redrawn items.

As part of the redrawing procedure, at step S204 it is determined whether the processing for a window requires a temporary frame buffer, opacity processing being one example.

If so (at S206) the procedure pushes a state of the tree and allocates the temporary frame buffer. The processing is then performed (at S208) by partitioning the window into a plurality of portions or partitions, and performing the processing portion-by-portion using the temporary frame buffer. After the complete result has been determined the state of the tree is popped (S2 10) and the temporary frame buffer is free, the redrawing procedure then continuing (S202).

The portion-by-portion processing comprises redrawing each portion in turn in to the temporary frame buffer (S2OSA), performing the required processing the temporary frame buffer (S208B) when such processing is needed, and then providing the results to the screen frame buffer (S208C), for example in the case of opacity processing blending the contents of the temporary frame buffer in to the screen frame buffer.

Referring again to the example of figure I, because the redrawing proceeds from back to front window C is redrawn and then the sub-tree rooted at window A is processed.

Let us say, for example, that window A requires opacity processing. In this case the state of the tree is pushed and window A is partitioned as shown and then each partition in turn is redrawn to the temporary frame buffer and blended into the screen frame buffer. When partition n is reached, because this has a portion of window B behind it the procedure determines whether the processing of window B requires use of a temporary frame buffer and if so iteratively implements partitioning and processing of window B as previously described. Thus, again, the state of the tree is pushed, window B is processed portion-by-portion and then the state of the tree is popped and the portion-by-portion processing of window A is continued. It will be appreciated that, depending upon how the tree is implemented as a data structure, only a portion of the tree (or sub-tree) may be saved and restored.

Figure 3 shows a block diagram of a system 300 configured to implement the above described methods. This comprises a processor 302 coupled to working memory 304, and non-volatile memory 306 storing tree-traversal window processing code, browser code, user interface code and operating system code for processor 302 to load and implement as described above. This code may be provided on a removable storage medium, by way of example illustrated by disk 307 and/or downloaded from a remote source. Thus as illustrated system 300 includes a network interface 310 for a cable, terrestrial, satellite or other feed, and a data store 308 for storing screen display and related web page data such as electronic program guide data, with which aspects of the invention may be employed. System 300 also includes a TV system 314 although in alternative arrangements the system 300 and in particular processor 302 and the program and data memory may be implemented within the TV system with additional stored code for one or more existing processors of the system. A remote control interface 312 for receiving browser and display coniniands from a remote control 316 is also provided, for example to control the loading of screen data pages and display of a plurality of overlapping screen display elements as illustrated above.

No doubt many other effective alternatives will occur to the skilled person. It will be understood that the invention is not limited to the described embodiments and encompasses modifications apparent to those skilled in the art lying within the spirit and scope of the claims appended hereto.

Claims

CLAIMS: 1. A method of rendering a plurality of overlapping image display

elements, the method comprising: processing said image display elements according to a hierarchy defined by a tree, nodes of the tree representing said image display elements and links between said nodes defining said hierarchy of said elements, a state of said tree defining processed elements of said tree, said processing rendering said processed display elements into a screen frame buffer for display; wherein the method fbrther comprises: identifying an image display element for offscreen processing; processing a portion of said identified image display element using a temporary buffer; writing a result of said element portion processing into said screen frame buffer; and repeating said element portion processing and result writing to process said identified element portion-by-portion.
2. A method as claimed in claim 1 wherein said temporary buffer is smaller than said screen frame buffer, and wherein each said portion is no larger than said temporary buffer.
3. A method as claimed in claim I or 2 further comprising partitioning said identified element into said portions for said portion-by-portion processing.
4. A method as claimed in claim 1, 2 or 3 further comprising saving a said state of said tree priorto said portion-by-portion processing of said identified element and restoring said saved tree state after said portion-by-portion processing.
5. A method as claimed in any preceding claim further comprising processing a sub-tree of said elements rooted at said identified element for each processed portion of said identified element.
6. A method as claimed in any one of claims I to 5 further comprising: identiing a said element portion overlapping a second said element for offscreen processing; saving a current state of said element portion processing; halting said portion-by-portion processing; processing said second element; restoring said saved state of said element portion processing; and continuing said portion-by-portion processing.
7. A method as claimed in any one of claims 1 to 5 wherein said method is applied iteratively to one or more descendants of elements in said tree of said identified element when processing for a said descendent element employs offscreen processing.
8. A method as claimed in any preceding claim wherein said offscreen processing includes copying each said portion of said identified element into said temporary buffer, and processing said copied element portions offscreen in said temporary buffer.
9. A method as claimed in claim 8 wherein said offscreen processing includes an image modification or enhancement operation.
10. A method as claimed in any preceding claim wherein said result writing comprises blending offscreen image data in said temporary buffer with data in said screen frame buffer to perform opacity processing.
II. A carrier carrying processor control code to, when running, implement the method of any preceding claim.
12. A computer system for rendering a plurality of overlapping image display elements by processing said image display elements according to a hierarchy defined by a tree, nodes of the tree representing said image display elements and links between said nodes defining said hierarchy of said elements, a state of said tree defining processed elements of said tree, said processing rendering said processed display elements into a screen frame buffer for display, the system comprising: data memory for storing data for said image display elements data representing said tree, said state of said tree, said screen frame buffer, and a temporary buffer; program memory storing processor control code; and a processor coupled to said data memory, and to said program memory to load and implement said processor control code, said code comprising code for controlling the processor to: identify an image display element for offscreen processing; process a portion of said identified image display element using a temporary buffer; write a result of said element portion processing into said screen frame buffer; and repeat said element portion processing and result writing to process said identified element portion- by-portion.