WO1999026199A1

WO1999026199A1 - Wide-band image enhancement

Info

Publication number: WO1999026199A1
Application number: PCT/US1998/023933
Authority: WO
Inventors: Eliezer Peli
Original assignee: Schepens Eye Research Institute, Inc.
Priority date: 1997-11-13
Filing date: 1998-11-10
Publication date: 1999-05-27
Also published as: AU1315599A

Abstract

An image processing technique produces modified images by extracting strong features of the original image, i.e., bars and edges, and superimposing such extracted features onto the original image. The invention combines the Hilbert transform of the image data with the image data in a pre-defined manner to produce the so-called energy function whose maxima correspond to the strong features of the image. Addition of these extracted features to the original image results in obtaining an enhanced image. In addition, the invention provides techniques for enhancing the real-world view of natural scenes.

Description

WIDE-BAND IMAGE ENHANCEMENT

Background

This invention relates to methods and apparatus for image processing and more particularly to image enhancement. In particular, the invention relates to methods and apparatus for the enhancement of both video images of natural scenes that contain a wide range of spatial frequencies and of real-world views of natural scenes.

Traditional image enhancement methods suffer from a number of drawbacks. Many traditional image enhancement methods can not effectively enhance images over a wide band of spatial frequencies. For example, one traditional technique, enhances an image by changing its spatial frequency content through manipulation of the coefficients of a discrete cosine transform ("DCT") of the image. The method segments the image into 8 x 8 pixel sections and obtains the cosine transform of each section. See E. Peli, Limitations of image enhancement for the visually impaired, Optometry and Vision Science, vol. 6, pp. 15-24 (1992); E. Peli, E. Lee, C. L. Trempe, S. Buzney, Image enhancement for the visually impaired: the effects of enhancement on face recognition, Journal of Optical Society of America, vol. 11 pp. 1929-1939 (1994). This technique fails to capture the low frequency components that arise as a result of features that have significant variations in luminance mainly over an area larger than an 8 x 8 section. Therefore, such traditional image enhancement techniques are not suitable for enhancing the images of many natural scenes that contain a wide range of spatial frequencies. Further, human observers detect moving objects that contain a wide band of frequencies more readily than those with a narrow band of frequencies. Thus, the traditional techniques are not appropriate in systems for assisting detection of moving objects, or in systems that provide real-time viewing enhancement of natural scenes.

Traditional methods also can not readily enhance an image while the size of the image changes. For example, the viewer of a digital television display could desire to follow the image of an object that undergoes a large change in its size while maintaining a selected degree of enhancement. The ability to enhance a wide range of frequencies is crucial in such applications. Traditional techniques, such as a DCT method, are not appropriate for such applications because they provide a limited range of spatial frequencies of the image.

In addition, traditional enhancement methods, both in the spatial domain and in the frequency domain, typically manipulate a large fraction of pixels. As a result, their use in the enhancement of color pictures requires tracking the color content of many pixels as the computation changes the luminance of those pixels. Accordingly, it is an object of this invention to provide methods and apparatus for enhancing images over a wide band of spatial frequencies.

It is another object of the invention to provide methods and apparatus that can readily enhance such images over a reasonable range of image sizes. It is yet another object of the invention to provide methods and apparatus for real-time viewing enhancement of natural scenes.

It is a further object of the invention to provide methods and apparatus for efficient enhancement of color pictures.

The invention is next described in connection with illustrated embodiments. It will, however, be obvious to those skilled in the art that various modifications can be made in the embodiments without departing from the spirit or scope of this invention.

Summary of the invention

The methods and apparatus according to this invention modify an image by 1) locating certain features of the image, such as the boundaries of objects in the image, 2) manipulating such located features to obtain modified features, and 3) adding the modified features to the original image. In particular, such practices of the invention employ a two-dimensional Hilbert transform of the image data to create a two- dimensional function, the so-called energy function, whose local maxima correspond to points lying on the boundaries between regions of marked difference in luminance, i.e., edges, or to points corresponding to peaks or troughs in luminance, i.e., bars. The invention further provides techniques to interconnect these maxima, thus delineating the desired features.

An application of this invention is to improve the visibility of video images for people with visual impairment, e.g., cataracts or macular degeneration. In particular, one embodiment of the present invention allows real-time image processing and enhancement of the real -world view for the visually impaired. This embodiment incorporates a dedicated microprocessor, programmed to extract the boundaries of objects in the field of view, according to the methods of the invention from the data inputted from a digital camera. This embodiment also incorporates video equipment to project the extracted features onto screens. These screens can be integrated in a wearable real-time image enhancement apparatus. Another application enhances the real-world view, under reduced visibility conditions such as fog, by projecting the enhanced features, obtained from non-visual sensors, e.g., infrared or radar, on heads-up displays (HUD) of an airplane or of a car windshield. Another application of this invention is to improve the visibility of television images for individuals with visual impairment. Yet, other applications relate to the enhancement of satellite and reconnaissance pictures or other military imaging devices, and to the delineation of features of interest in such pictures. The invention is typically practiced on a digital image that consists of a discrete two-dimensional map of luminance. Some embodiments of the invention represent such images by two dimensional matrices. The invention employs an extension of the well known methods for calculating the Hilbert transform of a function in one dimension to obtain a discrete two-dimensional Hilbert transform of a function of the image data. It is well understood that the one-dimensional Hilbert transform of a function of a single variable can be calculated by 1) obtaining the Fourier transform of the function, 2) obtaining a modified transform function whose values are zero at points where its independent variable is less than zero, and whose values are those of the Fourier transform at points where its independent variable is larger than zero. A third step is to obtain the inverse transform of this modified transform function.

One preferred embodiment of the invention obtains the two-dimensional Hilbert transform of the image data by 1) computing the two-dimensional Fourier transform of the image, 2) obtaining a new two-dimensional transform function whose values in a selected arbitrary contiguous half of the two-dimensional Fourier plane are zero, and whose values correspond to those of the two-dimensional Fourier transform of the image in the other half, and 3) obtaining the inverse Fourier transform of the modified transform function. The real part of the complex inverse Fourier transform of the modified transform function corresponds to the original image and the imaginary part corresponds to the Hilbert transform of the image. A preferred embodiment of the invention combines the image data with the

Hilbert transform of the image data to obtain a new two-dimensional function, a so- called energy function. In particular, the procedure for forming the energy function calls for obtaining the square root of the Pythagorean sum of the image data and of the values of the Hilbert transform at each point, e.g., at each pixel of a digital image. One embodiment of the invention utilizes the positions of the peaks of the energy function to locate the strong luminance features of the image. It is understood that such peaks correspond to peaks or troughs in luminance, or to those locations in the original image where changes in image intensity profile occur because of the existence of maximal phase congruency among the various Fourier components of the image. The local maxima of the energy function correspond to points of both minimum and of maximum intensity in the original image data, and also to the boundaries between regions of low and of high luminance. It is not reasonably feasible to classify the maxima of the energy function with respect to the polarity of the corresponding points in the image data based purely on the energy function. Thus, some embodiments of the invention implement a further examination of the image data at each point that corresponds to a maximum of the energy function to label the polarity of each such maximum.

One aspect of the present invention relates to the creation of a map of dots corresponding to the points designated as the maxima of the energy function. The invention employs methods known in the art to connect these dots to produce lines corresponding to the desired features. In addition, the invention provides the capability of manipulating these lines by widening them through convolution with an appropriate windowing function, e.g., a Gaussian with a selected width, or enhancing their intensities, to improve the contrast of the image.

Some embodiments of the invention employ only one arbitrarily selected polarity, i.e., either dark or bright, to display the dots or the contour lines at edges, whereas other embodiments utilize two polarities. A bipolar representation displays an edge with two dots, one dark and the other bright, next to each other. Some embodiments that utilize a bipolar representation examine the unmodified image to select a choice for juxtaposition of the dark and bright dots that corresponds to the sense of the transition of luminance at the corresponding location of the image. Both embodiments represent the polarity of bars in accordance with the polarity in the original image. Other embodiments of the invention use only a single polarity of dots, i.e., light or dark, to represent all bars or edges.

A preferred embodiment of the invention superimposes these modified contour lines onto the original image to obtain a new image in which certain features have been modified, e.g., the boundaries of the objects in the image have been enhanced.

The invention can also enhance color images. Because the invention manipulates only a limited number of pixels, i.e., those corresponding to the strong features of the image, only a few pixels change color due to the enhancement. Thus, the methods of the invention are more efficient in enhancing color pictures than traditional techniques. Thus, the invention attains the objectives set forth above by extracting strong features of an image, manipulating these features to obtain modified features, and superimposing such modified features onto the original image to obtain a modified image.

These and other features of the invention are more fully set forth below with reference to the detailed description of illustrated embodiments, and the accompanying drawings. Description of the drawings

FIGURE 1 is a flow chart depicting steps according to one embodiment of the invention for enhancing a wide-band image,

FIGURE 2 illustrates examples of the application of the methods depicted in figure 1 to two images with both the unipolar and bipolar representations of edges,

FIGURE 3 provides examples of the application of two alternative embodiments of the invention, where one embodiment employs two Hilbert transforms and the other employs four such transforms,

FIGURE 4 shows a flow chart depicting an apparatus according to an embodiment of the invention,

FIGURE 5 shows a human observer employing an apparatus according to an embodiment of the invention for the real-time viewing enhancement of natural scenes,

FIGURE 6 shows an original image and three enhanced versions of the original image obtained according to an embodiment of the invention, where the image labeled "enhanced" employs both dark and bright lines, and the other two modified images employ only bright lines,

FIGURE 7 shows an one embodiment of the invention for illuminating the features of an object in a natural scene,

FIGURE 8 illustrates enhancement of images with different sizes according to the invention,

FIGURE 9, similar to figure 8, shows the enhancement of two images with different sizes according to an embodiment of the invention, and

FIGURE 10 depicts various steps according to one embodiment of the invention for enhancing broadcast television images. Illustrated embodiments

The flow chart of FIGURE 1 shows various steps that an illustrated embodiment of the invention employs to modify an image represented by Image Data. This particular illustrated embodiment in step 10 applies a high pass filter in the spatial frequency domain to the image data to eliminate selected frequency components of the image. The high pass filter is typically constructed to retain frequency components that correspond to a few cycles per image, e.g., 16 cycles per image or higher, and to discard components that correspond to lower frequencies.

The illustrated embodiment of FIGURE 1 obtains the two-dimensional Hilbert transform of the filtered image data in step 12 by performing a sequence of three operations. The first operation is to calculate the two-dimensional Fourier transform of the filtered image data to obtain a transform function. The second operation is to create a modified transform function that vanishes over a selected contiguous half of the two- dimensional Fourier space of the transformed filtered image data, and has values identical to those of the transform function of the previous operation in the other half. The third operation is to apply an inverse Fourier transform to the modified transform function to obtain a complex function whose imaginary part corresponds to the Hilbert transform of the filtered image data.

An alternative practice of the operations of step 12 of the figure 1 sequence, suited for manipulating an image data that is represented by a two-dimensional matrix, obtains a discrete two-dimensional Hilbert transform of the image data by performing three operations. The first operation is to calculate a discrete two-dimensional Fourier transform of the image matrix to obtain a transform matrix. The second operation is to set the values of a selected half of the components of the transform matrix to zero to obtain a modified transform matrix, and the third operation is to obtain the discrete inverse Fourier transform of the modified matrix to obtain a matrix whose imaginary part corresponds to the discrete Hilbert transform. There exists an inherent arbitrariness in the choice of the half of the modified transform matrix that is set to zero. One preferred embodiment of the invention sets the lower half of the transform matrix to zero to obtain the modified transform matrix. Another embodiment sets the upper half of the transform matrix to zero to obtain the modified transform matrix. Yet, another embodiment sets the components below the diagonal of the matrix to zero and retains the rest. Subsequently, application of a discrete inverse Fourier transform to the modified transform matrix results in obtaining a matrix of complex numbers, the inverse modified transform matrix, whose imaginary part corresponds to the discrete Hilbert transform of the filtered image data. Those skilled in the art appreciate that a similar arbitrariness exists in creating a two-dimensional Hilbert transform if a continuous function rather than a matrix represents the image data. This arbitrariness stems from the choice of the half of the two-dimensional plane on which the values of a modified transform function, described above, are zero.

Referring to FIGURE 1 shows that the step 14 of the illustrated embodiment constructs a so-called energy matrix by performing four operations that combine the image matrix with the discrete Hilbert transform, represented by the imaginary part of the modified transform matrix. The first operation is to obtain the square of the image matrix. The second operation is to obtain the square of the discrete Hilbert transform matrix. The third operation is to add the square of each matrix to the square of the other, and the fourth operation is to compute the square root of the summation to obtain the energy matrix. It is clear to those skilled in the art that the same sequence of operations provides an energy function when applied to continuous functions rather than to discrete representations of such functions by matrices. Those skilled in the art also appreciate that there are methods, other than the Fourier method described above, for obtaining two-dimensional Hilbert transforms for use in the practice of the invention.

The peakfinder step 16 of the illustrated embodiment, shown in FIGURE 1, provides a number of maximum points of the energy matrix to subsequent steps of the illustrated embodiment by performing three operations. The first operation locates the local extrema of the energy matrix, i.e., local maxima and minima, by computing a two- dimensional gradient of the energy matrix and finding points at which the gradient vanishes, according to known methods in the art. The second operation obtains the second derivative of the energy matrix at each located extremum, to determine whether such a point corresponds to a maximum or a minimum of the energy matrix, and retains the maximum points and discards the minimum points. The third operation compares the intensity of the maxima of the energy matrix or the intensity of the second derivative of the energy matrix at each selected maximum with a pre-defined threshold value, and retains only points whose intensities exceed the threshold value. The maxima that the peakfinder step selects correspond to three types of features in the original image. They can either indicate the locations of minimum or maximum intensities, i.e., bars, or transitions between regions of varying intensities, i.e., edges. In the case of edges, the polarity of the transitions for a pre-defined direction, e.g., left to right and top to bottom, can not be readily gleaned from the energy matrix. One implementation of the illustrated embodiment chooses an arbitrary unipolar representation of the located maxima, i.e., it represents the maxima as bright or dark dots regardless of whether they correspond to bars or edges and also regardless of their actual polarities. Another implementation that opts for a bipolar representation employs dark and bright dots, symmetrically disposed with respect to the locations of the maxima, to display the maxima. One such implementation chooses an arbitrary polarity for displaying the dark and bright dots that represent edges, whereas a different implementation examines the image data to choose a polarity that corresponds to that in the image.

FIGURE 1 shows that step 18 of the illustrated embodiment, the phase detector step, provides the option of examining the image data in a selected neighborhood of each point corresponding to a maximum of the energy matrix to determine whether such a point corresponds to a bar or an edge in the image. In addition, this step determines polarities of the transitions in luminance at points corresponding to edges in the image.

Further reference to FIGURE 1 illustrates that step 20 of the illustrated embodiment utilizes the information that the step 16 supplies, and also in some implementations the information that the step 18 supplies, to create contour lines corresponding to the selected strong luminance features of the image by performing three operations. The first operation creates a two-dimensional map of dots corresponding to the selected maxima. The second operation, which is optional, can alter the widths of the dots through convolution with a tapered window, e.g., a Gaussian function with a pre-determined width, or alternatively enhance the dots by changing the degree of their luminance. The third operation is to join the dots to create contour lines in a manner known in the art.

Referring to FIGURE 1 shows that step 22 superimposes a display of the contour lines onto the original image. The display can be unipolar or bipolar, and it can have an arbitrary polarity or a polarity that corresponds to that of the feature in the actual image. Thus, the illustrated embodiment, employing steps 10 through 22 of FIGURE 1 , produces an enhanced version of the original image by accentuating the strong luminance features of the image.

FIGURE 2 illustrates the results of the application of the methods of the embodiment illustrated in FIGURE 1 to two images. In particular, the different views in FIGURE 2 allow the comparison of unipolar and bipolar edge representations of the modified images. For example, reference to views 2E and 2F shows that both the unipolar and bipolar displays represent the wrinkles on the forehead, i.e., bars, as dark lines. The views 2B and 2E show that the unipolar displays represent edges, such as transitions in luminance at the boundary of the jacket and the face, as dark lines whereas the views 2C and 2F show that bipolar displays represent such transitions as dark and bright line pairs disposed symmetrically with respect to the center of the transition. Furthermore, a comparison of the bipolar edge representations in views 2C and 2F with the views 2 A and 2D of the unmodified images readily illustrates that the chosen polarities of the edges correspond to the actual polarities in the unmodified images. Another embodiment of the invention combines multiple Hilbert transforms, obtained in the manner described in the illustrated embodiment of FIGURE 1, to produce a modified energy function of the filtered image. For example, with reference to the second operation of the step 1 of FIGURE 1, one implementation of this embodiment employs two sets of Hilbert transforms corresponding to two different orientations of axes in the Fourier plane to obtain a modified energy matrix. The embodiment employs this modified energy matrix to delineate the selected features of the image in the same manner as described in the previous embodiment. Further reference to FIGURE 1 shows that the contour construction step 20 creates contours of all features obtained through multiple Hilbert transforms. The image constructor step 22 superimposes all these contours onto the original image to produce an enhanced image. One advantage of employing multiple Hilbert transforms is that each transform results in a preferential delineation of the luminance features that substantially lie in the direction of the selected axes in the two-dimensional Fourier plane, utilized to obtain the transform. Thus, superposition of luminance features obtained from a set of Hilbert transforms results in better enhancement of the image than superposition of features obtained from only one such transform. FIGURE 3 provides a comparison of two enhanced versions of an image obtained by employing multiple Hilbert transforms. In particular, the view 2B of the original image 2A was obtained by employing two Hilbert transforms, whereas the view 2D of the original image 2C was obtained by employing four Hilbert transforms.

The flow chart of FIGURE 4 depicts an apparatus according to an embodiment of the invention. A digitizer 24 supplies a digitized image data corresponding to an input image to a microprocessor or a programmed digital computer 26. The microprocessor or the computer performs a sequence of operations corresponding to the steps of the illustrated embodiment of FIGURE 1 on the digitized inputted image data to obtain data corresponding to an enhanced version of the input image. A display unit 28, e.g., a monitor or a viewer, presents an enhanced version of the input image to an observer. FIGURE 5 shows an illustrated embodiment of the invention according to the apparatus of FIGURE 4 that allows real-time image enhancement of real- world scenery. Reference to FIGURE 4 shows a human observer wearing an apparatus according to the invention that includes a video camera, preferably a digital camera, that provides image data corresponding to the natural scene. Subsequently, the apparatus transfers the digital image data to a dedicated microprocessor, programmed to extract the bars and edges in the image and to provide a contour map of the extracted bars and edges according to the methods of the present invention. The processor transfers the contour map to a video display module that projects the map on two partially transparent screens positioned in the front of the observer's eyes, known in the art as a see-through head-mounted display. This allows the observer to view the natural scene with an enhancement of its distinctive features.

The apparatus of FIGURE 5 can also be designed to enhance a portion of an observer's field of view, e.g., the central portion. Such an apparatus continuously enhances the central portion of the observer's field of view as the observer turns turn his eyes from one part of a natural scene to another, FIGURE 6 depicts various modified versions of a natural scene, employing different polarities, according to the methods of the present invention. The image labeled enhanced uses a bipolar representation. The bottom images use only positive polarities, i.e., bright lines. The use of bright lines is the practical method for enhancing the real world view. While the image labeled "positive" only uses bright lines to represent only features that correspond to positive polarity in the original image, the image labeled "all positive" uses bright lines to represent all features of interest.

FIGURE 7 shows an apparatus according to the invention that illuminates the features of objects in a natural scene that correspond to the luminance features in an image of such objects, e.g., bars and edges. Referring to FIGURE 7, a digitizer 30 digitizes an image of a natural scene. A microprocessor or a programmed digital computer 32 obtains the locations of the bars and edges in the image and supplies this data to a light source guidance system 34. The guidance system directs the light source to illuminate the locations in the natural scene corresponding to the bars and edges in the image of the scene. One implementation of this embodiment that can find its utility in laser shows and similar applications employs laser beams scanned over the locations of the bars and edges to illuminate these features.

One advantage of the methods of this invention is the ability to change the size of an image while retaining a selected degree of enhancement. FIGURES 8 and 9 illustrate this aspect of the invention by presenting images of different sizes and their enhanced counterparts. In particular, these figures show images that differ in their respective areas by a scale factor of sixteen. An examination of these two figures clearly illustrates that application of the methods of the invention to the smaller images produces acceptable results.

One application of the present invention relates to enhancing broadcast television images. FIGURE 10 depicts various steps according to the invention for optional enhancement of such images. Referring to FIGURE 10 shows an encoder 36 at a central broadcasting station that forms an enhancement signal by supplanting an image signal with the information needed for enhancing the image, i.e., the pixels that need to be modified and the degree of modification of each pixel. The broadcast of the enhancement signal is manageable because the methods of the invention modify only a small fraction of the pixels that comprise the image, thus requiring minimal expansion of the transmission bandwidth. A transmitter 38 send the original image data and the enhancing information to a receiver 40 that optionally uses the information for enhancing the received image. In an analog television system, the enhancing information is transmitted to the receiver during the so-called "blank time," in a manner similar to that utilized for producing captions for the hearing impaired, to produce an enhanced image for viewers with visual impairments. Further reference to FIGUREIO shows a switch 42 that controls whether the original image data or an enhanced image is sent from the receiver to a display unit 44.

Those skilled in the art appreciate that application of the methods of the invention for enhancing images is not limited to gray scale pictures, but can also be employed to enhance color pictures.

It will thus be seen that the invention attains the objectives set forth above. Because certain changes in the above processes can be made without departing from the spirit or scope of the invention, the above description is intended to be interpreted as illustrative and not in a limiting sense.

Claims

What is claimed is

1) A method for modifying an image comprising the steps

(a) extracting features of said image selected form the group of maximum luminance, minimum luminance, and transitions between regions of varying luminance, and

(b) superimposing said extracted features onto said image to create a modified image.

2) A method for modifying an image comprising of the steps

(a) forming a two-dimensional mathematical representation of said image,

(b) obtaining the Hilbert transform of said mathematical representation,

(c) creating a two-dimensional function whose points of local maxima correspond to locations of local minimum luminance, maximum luminance, or a transition in the intensity of luminance in said image by combining said Hilbert transform with said mathematical representation of the image,

(d) obtaining the locations of local maxima of said function,

(e) producing a two-dimensional map of dots corresponding to said locations of local maxima of said function, and (f) superimposing said map onto said image to obtain a modified image.

3) The method according to claim 2, wherein the step of obtaining said Hilbert transform comprises

(a) obtaining the Fourier transform of said mathematical representation of the image,

(b) creating a mathematical function whose value is zero in an arbitrarily chosen contiguous half of a plane, defined by the two independent variables of said function, and whose values at points in the other half of said plane are equal to those of said Fourier transform, and (c) obtaining the real part of the inverse Fourier transform of said function, whereby creating said Hilbert transform. 4) The method according to claim 2, wherein the step of combining said mathematical representation of said image with said Hilbert transform is further characterized by

(a) obtaining the square of said mathematical representation, i (b) obtaining the square of said Hilbert transform,

(c) summing the square of said image data at each point to the square of said Hilbert transform at the corresponding point, and

(d) obtaining the square root of said sum to produce said two-dimensional function.

5) The method according to claim 2, wherein the step of locating the maxima of said two-dimensional function is further characterized by

(a) obtaining a two-dimensional gradient of said function, and

(b) locating zero crossings of said two-dimensional gradient.

6) Apparatus for modifying an image comprising

(a) means for digitizing said image,

(b) means for creating contour lines corresponding to selected features of said image, and (c) means for superimposing said contour lines onto said image.

7) The apparatus of claim 6, wherein said means for extracting selected features comprises a microprocessor receiving said digital image, obtaining the Hilbert transform of said digital image, computing a function whose values are the Pythagorean sum of said Hilbert transform and said digital image, and computing zero crossings of said function.

8) The apparatus of claim 6, wherein said means for creating said contour lines comprises a graphics software receiving data corresponding to locations of extracted features from said means of extracting selected features and displaying such locations as dark or bright dots. 9) An apparatus for real-time enhancement of view of a natural scene for a human observer comprising

(a) means for obtaining an image of said natural scene,

(b) means for extracting features of said image corresponding to maximum, minimum luminance, and transitions between regions of varying luminance, and

(c) means for projecting said features onto a pair of semi-transparent screens disposed in front of the eyes of said observer.

10) An apparatus for illuminating parts of selected objects in a natural environment, said parts giving rise to luminance features in an image of said objects, said apparatus comprising:

(a) means for obtaining an image of said objects,

(b) means for obtaining locations of said parts by locating features of said image corresponding to maximum, minimum luminance, and transitions between regions of varying luminance, and

(c) means for projecting light onto locations corresponding to said parts of selected objects.