GB2458813A

GB2458813A - Age-dependent pixel modification in composite images

Info

Publication number: GB2458813A
Application number: GB0905743A
Authority: GB
Inventors: Holger Vogel; Jan Zaussinger
Original assignee: Carl Zeiss Optronics GmbH
Current assignee: Hensoldt Optronics GmbH
Priority date: 2005-03-03
Filing date: 2009-04-02
Publication date: 2009-10-07
Also published as: GB0905743D0

Abstract

A composite image (eg 100' or 1') is constructed from a plurality of component images 101' to 105'. The component images are captured using acquisition equipment, such as a camera 2, which may be movably mounted upon a support 3. The camera acquires component images as it is directed towards different parts of a scene. Component images are supplied to a computer 4 which processes them to form the composite image. An image processing/display system, such as computer 4, processes each of the component images such that a distinction can be made between composite image regions of different age, when displayed upon display equipment, eg screens 5a,5b. For example, older regions may be subject to age-dependent pixel value adjustment to yield darkening (eg image 101',102' etc), brightening or colour effects. Alternative methods of processing include image framing (eg 11', 12'), or resolution alteration. Accordingly, the user is instantly able to identify the relative age of processed regions of the displayed composite image.

Description

Device and a method for displaying image mosaics The invention relates to a device and a method for displaying image mosaics.

An image mosaic is generally understood to be a number of individual images which together produce an overall image. In so doing the individual images may be displayed alongside one another or overlapping one another in output equipment.

In map production and in photography it is generally known to produce image mosaics from a tapestry of individual images taken one after another.

A disadvantage of conventional image mosaics with live individual images is that in surveillance tasks a state is displayed to an observer which conveys apparent security as the observer is not informed about events now taking place in areas of the mosaic image that are not currently refreshed.

The object of the invention is to provide a device and a method for displaying image mosaics in which an observer is given information about how up to date the image mosaic or parts of the image mosaic is/are.

The object of the invention is achieved by a device with the features of Claim 1 and by a method with the features of Claim 8.

One or more cameras in the visible and/or infrared regions, e.g. LWIR, MWIR, VLWIR, FIR, as well as in the SWIR, NIR and TJV region may serve as acquisition equipment. Other imaging representations are also conceivable.

The acquisition equipment may be attached either fixedly or rnoveably, e.g. on a height-adjustable

I

support, a wall, on a vehicle or on a flying object.

Moveable acquisition equipment may, for example, be able to be panned, tiltable, adjustable and/or movable in its height and/or lateral orientation.

In the method according to the invention and with the device according to the invention image areas of the image mosaic are displayed differently in output equipment (image generator) of an electronic processing and display system depending on their age. One display or a plurality of displays may be provided as the image generator.

With the method according to the invention and the device according to the invention the advantages of conventional mosaicking and of panning a camera with a precise viewing direction are combined. The overall overview through the mosaic is retained, the current viewing direction is visible, and the age of the image areas displayed is likewise recognizable.

Updates of the mosaic image can be carried out. These may take place automatically and/or by user stipulation. In doing this, new individual images of the observed scene may, for example, be inserted one after another into the mosaic image. A new mosaic image of another observed scene may also be built up.

An image mosaic displayed in output equipment, for example a display, may consist of a defined number of pixels. An individual image consists of a defined number of pixels and usually pixe1s�fldVdU51 image<PiXelSimage mosaic* Individual images of different sizes, i.e. pixel numbers, may be provided.

By zooming it is possible to change between different

fields of view. The image area of the subsequently

taken current individual images can then be adapted to

the new field of view.

The image mosaic is composed from the individual images in the form of digital raster graphics made up of pixels. When an image area of an image mosaic is replaced by a more recent individual image, the pixels for this image area are updated, i.e. overwritten. This may involve one or more older individual images being completely or partly superimposed by the more recent individual image. The more recent individual image may also be larger or smaller than an older individual image. With mosaics made of live images which are continually being refreshed the overlap often amounts to almost the entire image area of the overwritten older individual image.

It is also possible to provide for producing more than one individual image for an image mosaic simultaneously or essentially simultaneously, e.g. with a plurality of cameras. The pixels of a plurality of image areas, e.g. the pixels of two new individual images, may then be updated in parallel.

The electronic processing and display system, e.g. a computer or a computer network, notes the age for each pixel of an image mosaic. On the image generator of the electronic processing and display system, e.g. a display, the pixels for the respective image mosaic can then be altered according to age, e.g. be displayed in darkened fashion. In live mode a darkening of older image areas occurs while new, lighter individual images are added.

This age-dependent function can be chosen optionally by the user. That is to say, the user can choose between a conventional and the time-dependent display of the image mosaic. With the conventional display of the image mosaic the observer cannot recognize which image areas are older and which are more recent.

The age can be displayed individually for each pixel, e.g. through an individual brightness value. However, it is also conceivable to undertake a grouping into different age levels. The age gradation may take place for pixel groups defined by their age, so that, for example, on one image generator an age-dependent gradation, e.g. of brightness, takes place for 2-15 pixel groups. The most recent defined pixel group may correspond to the most recent individual image or alternatively correspond to the image area occupied by the 2, 3, 4 or more most recent individual images. In this way an overall image can be divided into a plurality of image areas.

The age display may be carried out in various ways. The age display may, for example, occur through an adaptation of the image brightness over time. Older image areas slowly become darker and it is immediately clear to the observer that this cannot be an up-to-date image area. In addition, or alternatively, the up-to-date image area may be marked.

So as not to allow very old image areas to disappear completely, the adaptation of the image brightness can also be restricted to a certain brightness range. By virtue of the fact that the older image areas become darker but can still be perceived, a good overall overview of the observed scene is retained. The behaviour over time, through which ageing becomes visible, may be predefined or individually set by the user.

Other adaptations over time are also conceivable that may be carried out alternatively or in addition.

Instead of being darkened, the older image areas may be brightened. A colouring, e.g. with thermal images, is also possible. For example, green or blue may be provided as a colouring. A lowering of image definition and a reduction in resolution are further possibilities for adaptation. It is also possible to provide for enframing the most recent individual image or a current image area.

Particularly with live recordings it is possible to provide for a current image area to be particularly highlighted through a marking. The marking can be arranged depending on pixel age and be provided, for example, for the most current individual image. Thus a frame may be put around the now current image area of the live image, which may be an optimized display of an individual image. By this means the observer receives additional information about the current orientation of the camera lens with respect to the scene. In the case where the pixels are grouped in age levels it is possible to provide for providing a marking for one or more age levels. Thus, for example, the two or three most recent age levels may be provided with a common frame. Other markings are also conceivable.

An FIR filter (Finite Impulse Response Filter), for example, may be provided for the optical change depending on the age of the recording.

Advantageous embodiments of the invention are reproduced in the subclaims.

The method according to the invention and the device according to the invention will be described with reference to the figures.

These show: -Figure 1: a conventional image mosaic display

according to the prior art;

-Figure 2: an image mosaic display according to the invention; and -Figure 3: the device according to the invention.

The image mosaics 1 and 100 from Figure 1 and Figure 2 were shot with a film camera. In the method according to the invention and the device according to the invention, image mosaics of high resolution and also in colour, in particular in colour photo quality, can generally be produced with a camera. In order to improve reproducibility for publication, the image mosaics here have been strongly coarsened and reduced to a few grey shades.

Figure 1 shows a conventional image mosaic display 1 on an image generator according to the prior art. This image mosaic 1 appears to an observer as made up from a plurality of juxtaposed and simultaneously overlapping individual images 11-22. Only the image areas lying in the uppermost image plane are respectively visible, since the pixels of the older image areas are updated, i.e. overwritten, by the pixels of the more recent image acquisitions. The brightness and image definition is at least essentially the same for each of the individual images. The make up of the image from individual images can be surmised only on the basis of the various lower and upper edges of the individual images. The number of individual images from which the image mosaic 1 is composed can hardly be determined. It is not possible to recognize whether image acquisition equipment, e.g. a camera, has panned from left to right or from right to left and how old the individual images 11-22 are. The current viewing direction of the camera cannot be recognized either.

Figure 2 shows an image mosaic display 100 according to the invention of the same scene as in Figure 1. The image mosaic 100 is made up of twelve individual images 101 to 112 from left to right. Here too only the image areas lying in the uppermost image plane are respectively visible, since the pixels of the older image areas are updated, i.e. overwritten, by the pixels of the more recent image acquisitions. A rural space with buildings is shown, with forest and free plots of land in the background, residential houses in the left foreground and a car park in the right foreground, along with a factory building on the far right. The older individual images are displayed darker than the more recent individual images. A further car park in the bottom left is therefore difficult for the observer to recognize. To preserve the overall impression, maximum darkening of the older individual images is provided, with which the observer retains a residual impression of the scene. The darkest image areas may have, for example, 5% to 50%, in particular 10% to 301, of their original brightness.

It can easily be recognized that the image acquisition equipment 2 was panned from left to right to shoot the image mosaic 100; the older, i.e. darker, individual images are hence found on the left-hand side of the image mosaic 100. It is also easy to recognize that the respectively more recent individual image overlaps the previous individual image. Alongside a main panning direction from left to right, a panning of the camera 2 and hence a shift in the location of individual images additionally occurred first upwards and then downwards again.

Figure 3 shows an embodiment of the device according to the invention. Acquisition equipment 2, e.g. a camera, is provided. This acquisition equipment 2 may be attached either fixedly or moveably, e.g. on a height-adjustable support 3, a wall, on a vehicle or on a flying object. With the present acquisition equipment 2 panning and tilting in the x, y and z directions along with height adjustment in the z direction are provided.

These movements of the acquisition equipment 2 may take place automatically or following an instruction by a user. While being moved, this acquisition equipment 2 takes a number of individual images, e.g. the schematically represented individual images 101' to 105', which are transferred, for example, to a computer 4 -or a computer cluster -with monitor 5a and input equipment 6a. In live acquisition, for example, 25 frames per second may be taken.

The image mosaic 100' built up from the images 101' to 105' can be displayed on output equipment, e.g. the monitor 5a. In the present embodiment two monitors 5a and 5b are provided. It is, however, conceivable to work with only one monitor or more than two monitors.

Via the input equipment 6a and 6b, e.g. keyboard or joystick, or alternatively in automated fashion after certain time intervals, the computer 4 can be caused to change back and forth between the conventional mode of time-independent image mosaic display and the mode according to the invention of time-dependent image mosaic display. Alternatively, it is also possible to provide only the mode of time-dependent image mosaic display according to the invention. By means of the input equipment Ga and Gb it is possible for the user to carry out updates of the image mosaic 100. Instead of two pieces of input equipment 6a and 6b it is also possible to provide only one piece of input equipment or else more than two pieces of input equipment.

The acquisition equipment 2, computer 4, output equipment 5a and 5b, and input equipment 6a and 6b may be formed as separate elements. It is also conceivable to combine all or certain of these elements structurally. For example, the output equipment 5a, 5b may be formed as the display of a Head Mounted Display (11DM) and/or the computer 4 and the acquisition equipment 2 may form one physical unit.

The individual images 101' to 105' and 11' to 15' are represented as shaded for better illustration. The outlinings of the individual images 11' and 12', carried out as an example, are represented in black only to enhance understanding, so that it can be recognized that the superposition of the images progresses from left to right.

On the image generator represented on the right, here the monitor 5b, a conventional image mosaic display 1' with an image generator according to the prior art is reproduced. Such a display is also called a panorama.

This image mosaic 1' appears to an observer as made up from a plurality of juxtaposed and simultaneously overlapping individual images 11' to 15'. Only the image areas lying in the uppermost image plane are respectively visible, since the pixels of the older image areas are updated, i.e. overwritten, by the pixels of the more recent image acquisitions. In the present case only the individual image 15' is completely visible. The brightness and image definition is at least essentially the same for each of the individual images. Contrast, brightness and image definition can be set for the image mosaic as a whole.

In a conventional panorama display the individual images appear to merge into one another. The make up of the image from individual images can be surmised there only on the basis of the various lower and upper edges of the individual images that might be present. The number of individual images from which such an image mosaic 1 is composed can then often hardly, or not at all, be determined. With a multitude of individual images that form an overall mosaic it is then not possible to recognize whether image acquisition equipment, e.g. a camera 2, has panned from left to right or from right to left, or has jumped in one go from one place to a more distant place, and how old the individual images are. The current viewing direction of the camera 2 cannot be recognized either.

The provision of a common image generator for an alternating conventional image mosaic display and an -10 -image mosaic display according to the invention may be provided. It is then possible to change between the ways of displaying by means of input equipment. Such a change may also be provided automatically, e.g. after certain time intervals, by the electronic processing and display system.

On the left monitor 5a an image mosaic display 100' according to the invention is reproduced, which display might show the same scene as on the right monitor. For presentational reasons only shadings are reproduced here instead of a genuine scene. The image mosaic 100' is made up of five individual images 101' to 105' from left to right. Here toO only the image areas lying in the uppermost image plane are respectively visible, since the pixels of the older image areas are updated, i.e. overwritten, by the pixels of the more recent image acquisitions. Instead of the grey shadings the following scene -corresponding to Figure 2 -could also be imagined: "a rural space with buildings is shown, with forest and free plots of land in the background, residential houses in the left foreground and a car park in the right foreground, along with a factory building on the far right". The older individual images are displayed darker than the more recent individual images. An element of the scene "a further car park in the bottom left" would therefore be difficult for the observer to recognize. To preserve the overall impression, maximum darkening of the older individual images is provided, with which the observer retains a residual impression of the scene. The darkest image areas may have, for example, 5% to 50%, in particular 101 to 30%, of their original brightness.

It can easily be recognized that the image acquisition equipment 2 was panned from left to right to shoot the image mosaic 100'; the older, i.e. darker, individual images are hence found on the left-hand side of the image mosaic 100'. It is also easy to recognize that -11 -the respectively more recent individual image overlaps the previous individual image. For example, individual image 103' is overlapped by individual image 104' and itself overlaps individual image 102'. Alongside a main panning direction from left to right, a panning of the camera 2 and hence a shift in the location of individual images additionally occurred first upwards and then downwards again.

The individual images may have different image sizes.

Zooming in and out is also conceivable, so that the single individual images reproduce different sized sections of the scene.

With a further update older image areas of the image mosaic 100' might be replaced by new individual images.

In the present example this may mean a new panning of the camera from left to right. It is also conceivable to place new individual images which do not correspond to the location of the old individual images 101' to 105' in the image mosaic 100'. In this way the observer can view current image areas of interest to him.

If the observer also wants to observe the entire image mosaic without age-dependent changes, in so far as only a common image generator is available for both modes, he can change back and forth between the conventional image mode and the update mode of the invention.

Of course, an image mosaic can be built up in many ways. A linear configuration of individual images, e.g. from one side to the other, from top to bottom or diagonally, is conceivable. A configuration in the x and y directions as a plane is also conceivable. The acquisition equipment 2 may record individual images of areas lying next to each other or else jump from one area of the scene to another, distant area of the scene. It is also conceivable for image areas of an image mosaic that are defined as important to be

S

-12 -refreshed more frequently than less important areas, e.g. through a programmed scan line.

i image mosaic may, in particular, be produced from S the live image of a moving, e.g. a panned and optionally also still moving, camera, e.g. during manual or automatic panning operation.

An image mosaic can be built up from a multitude of images. In live acquisition new individual images can be continually inserted and old individual images overwritten. An image mosaic might, for example, be built up from 5 to 50 individual images. There may also be more or fewer individual images. With live recordings, between 1 and 50 new individual images per minute are, for example, conceivable.

-13 -List of references symbols 1 Image mosaic with age-independent display of the individual images 11-22 Individual images with age-independent display Image mosaic with age-dependent display of the individual images 101-112 Individual images with age-dependent display 2 Acquisition equipment 3 Support 4 Computer 5a, 5b Output equipment, e.g. display 6a, 6b Input equipment 1' Image mosaic with age-independent display of the individual images 11'-15' Individual images with age-independent display 100' Image mosaic with age-dependent display of the individual images ioi' Individual images with age-dependent display -105'

Claims

-14 -CLAIMS1. Device for displaying image mosaics, comprising: -optical acquisition equipment that can be moved in at least one direction for taking a temporal succession of individual images; and -an electronic processing and display system (4) for producing an image mosaic in the form of digital raster graphics made up of pixels from the individual images in output equipment (Sa); characterized in that -the pixels can be displayed with optical modification in the output equipment (5a) depending on their age since acquisition.
2. Device according to Claim 1, characterized in that in the display of the digital raster graphics (100, 100') in the output equipment (5a) the age since acquisition of the pixels is displayed as a function of the image brightness, in particular as an image brightness that declines with the age since acquisition.
3. Device according to Claim 1 or 2, characterized in that in the display of the digital raster graphics (100, 100') in the output equipment (5a) the age since acquisition of the pixels is displayed as a function of the image definition, in particular as an image definition that declines with the age since acquisition.
4. Device according to one of the preceding claims, characterized in that in the display of the digital raster graphics (100, 100') in the output equipment (5a) the age since acquisition of the pixels is displayed as a function of the resolution, in particular as a resolution that declines with the age since acquisition.S

-15 -
5. Device according to one of the preceding claims, characterized in that in the display of the digital raster graphics (100, 100') in the output equipment (5a) the age since acquisition of the pixels is displayed as a function of a colouring, in particular as a colouring that increases with the age since acquisition.
6. Device according to one of the preceding claims, io characterized in that in the display of the digital raster graphics (100, 100') the pixels are grouped by their age since acquisition and in the output equipment (5a) either the respectively newest group of pixels, or at least the two newest groups of pixels, are marked by a marking, in particular by enframing the pixel group or a common enframing of the pixel groups.
7. Device according to one of the preceding claims, characterized in that input equipment (6a, 6b), in particular a keyboard or a joystick, is provided for setting between a first mode with age-dependent display of the pixels (101-112, 101'-105') and a second mode with pixel display irrespective of age (11-22, 11'-15')
8. Method for displaying image mosaics, in which a temporal succession of individual images recorded is displayed together, in particular juxtaposed or overlapping, in output equipment, in particular a display, as an image mosaic in the form of digital raster graphics made up of pixels, characterized in that -the pixels are displayed with optical modification in the output equipment (5a) depending on their age since acquisition.
9. Method according to Claim 8, characterized in that an existing mosaic image (100) is updated by one or more new individual images.-16 -
10. Method according to Claim 8 or 9, characteriZed in that in acquiring a current individual image the older image area that this individual image overlaps is overwritten with this current individual image.
11. Method according to one of Claims 8 to 10, characterized in that for the optical modification of pixels (101-112, l01'-105') depending on the age since acquisition at least one of the following measures is carried out: a) image brightness changing with the age since acquisition; b) image definition changing with the age since acquisition; c) resolution changing with the age since acquisition; d) colouring changing with the age since acquisition; and e) electronic filtering changing with the age since acquisition.
12. Method according to one of Claims 8 to 11, characterized in that for the display of the individual images (101-112, 101'-105') as an image mosaic (100) in the output equipment (5a) either the respectively newest individual image or a defined number of the most recent individual images is/are marked, in particular by enframing the newest individual image or the group of the most recent individual images.
13. Method according to one of Claims 8 to 12, characterized in that a choice is made between a first mode with age-dependent display of the pixels (101-112, lOl'-105') and a second mode with pixel display irrespective of age (11-22, 11'-15')
14. Method according to one of Claims 8 to 13, characterized in that for the age-dependent display the -17 -individual images (101-112, 101'-105') are combined in age-dependent groups, with each group having at least one individual image, and the optical modification taking place in groups.
15. Method according to one of Claims 8 to 14, characterized in that for the optical modification in the age-dependent display limiting values are provided in order to preserve a residual visibility for the observer.