CA2562480A1 - System for assessing images - Google Patents

Description

Application number / numero de demande: .2 S'Q
Figures: ~

Pages:

Unscannable item(s) received with this application To inquire if you can order a copy of the unscannable items, please visit the CIPO WebSite at HTTP://CIPO.GC.CA

Item(s) ne pouvant etre balayes Documents regus avec cette demande ne pouvant etre balayes.
Pour vous renseigner si vous pouvez commander une copie des items ne pouvant etre balayes, veuillez visiter le site web de l'OPIC au HTTP://CIPO.GC.CA

Patent Application ImageChecker System for Assessing Images INTRODUCTION

With the advent of the digital camera, consumers are faced with the task of searching through hundreds and even thousands of images to select those images suitable for printing, display or for sharing electronically in databases or slideshows via websites, e-mail or other means.
At present, the consumer is faced with not only selecting images based on the desired content (e.g.
vacation photos last year at the beach) but also with selecting images of suitable quality. A suitable image is an image that does not exhibit undesirable characteristics which may include but are not limited to:

= Over or under exposure = Blurring = Poor color reproduction = Poor white balance = Repetitiveness - Similarity to other images = Jpeg artefacts The present invention is an "ImageChecker" - a software algorithm to detect photo images that are inferior or unsuitable for printing or display due to the presence of one or more undesirable characteristics. The invention is also used to detect images that are suitable for printing or display.
While the above characteristics are representative of the type of characteristic that may be undesirable, it is understood that various other characteristics may be used to determine that an image is either suitable or unsuitable. An overall decision maker algorithm decides if an image is suitable or not based on the output from a group of sub-algorithms, where each sub-algorithm detects the presence or absence of at least one undesirable characteristic.

The applications for the software include but are not limited to:
= The preparation of photobooks online, via kiosks or at home = Online image management services = Consumer image management software = Onboard camera operation At present photo book preparation and the management of image databases is largely left to the consumer, who works diligently from his or her PC to select suitable images for printing or sharing. The typical means of selecting images efficiently is to scroll through a database of images, observing on the computer monitor a number of small thumbnail images simultaneously and tagging those that contain the desired content. Unfortunately, the resolution of thumbnail images is insufficient to allow the consumer to determine the quality of the image. In fact, it is often difficult to determine the quality of images that occupy a significant proportion of the monitor of a PC. The result is that unsuitable images are often printed, either individually or in photobooks. A photobook is a bound book with pages containing one or more images printed on each page, sometimes on both sides of the page. Captions and templates are used to enhance the pages. Photobooks are now produced by many photo finishers and kits are available to print and bind photobooks at home. While unsuitable individual prints can be simply discarded (but with the wasted cost of the print), an unsuitable image is particularly annoying in a photobook because it cannot be discarded without defacing the book. In addition, while unsuitable images can be simply deleted from a slideshow or database, they must first be detected. It is assumed that most consumers wish to display only those images that they consider suitable for viewing by others.
With the advent of digital cameras, many camera users engage in the practice of "bracketing" i.e.
taking numerous photos of the same image with different camera seffings. In this case, it would be helpful for a consumer to be able to pick the best image out of a group of bracketed images. The ImageChecker software can be used to automatically select the most suitable image from a group of bracketed images based on the presence or lack of the aforementioned characteristics. It would also be useful for consumers if a digital camera could provide a waming if the captured image exhibited undesirable characteristics so that the image could be deleted and, perhaps with some adjustments to camera settings, another picture could be taken. The ImageChecker software provides guidance to consumers so that 1) undesirable images can avoided, discarded or enhanced and

2) suitable images can be selected.

The automation of 1) the detecfion of defecfive images; and 2) the detection of suitable images; is content-based. By content-based, it is meant that the image is classified based on the content of the image that is detected by the ImageChecker software. The image content includes the presence of objects in the image including but not limited to:

- sky - people - skin - eyes - teeth - foliage - grass - an indoor or outdoor environment - jpg artifacts - etc.

The image content also includes the condition of the image, including but not limited to the brightness of the colours and the degree of exposure, sharpness, saturation and white balance.

PRIOR ART

Previous work has been completed by iSYS on the development of the automated image orientation (US Patent Application 20060067591) and automated red-eye removal software (US
Patent Application 20040114829) for the photo imaging industry. This work has led to the development of the new ImageChecker software to detect photo images that are inferior or unsuitable for printing or display.
With regards to competing technologies, the existing technologies involving the characteristics iSYS
proposes to detect have been assessed to a limited extent. No technology was found regarding the detection of over/under exposure or over/under saturation in the manner planned by iSYS.

Blur detection is currently offered on some higher end digital cameras. The related patents are based on detecting camera motion or vibration, not image analysis. The iSYS
algorithm is based on image analysis.

Some digital cameras also feature a white balance feature whereby software in the camera automatically adjusts the white balance when a white reference is displayed in front of the camera.
iSYS, on the other hand, assesses the degree of exposure and/or white balance by examining the content of the image, histograms related to the image. For blur detection, iSYS performs transforms of the image, including but not limited to Fast Fourier Transforms.

There are various patents describing software applications that automatically "enhance" the color of images prior to printing. These software enhancements are typically applied globally in varying degree to all images examined, with no provision for deciding whether or not enhancement is required in the first place.

There are various existing means of assessing the similarity of images and jpeg artefacts in images, some of which are patented.

There is no known commercially available software that provides an overall assessment of image quality for the purpose of screening out images that are unsatisfactory for the purpose of printing or sharing.

DESCRIPTION
The ImageChecker software detects the characteristics of an image to determine the image quality and to determine the likelihood that image quality will be acceptable to an individual or individuals. The quality of an image may be inferior or unsuitable for various reasons. An unsuitable image may exhibit characteristics including but not limited to:

= Over or under exposure = Blurring = Poor color reproduction = Poor white balance = Repetitiveness - Similarity to other images = Jpeg artefacts While the above characteristics are representative of the type of characteristic that may be undesirable, it is understood that other characteristics may be used to determine that an image is suitable or unsuitable.

An overall decision maker algorithm decides if an image is suitable or not based on the output from a group of sub-algorithms, where each sub-algorithm detects the presence or absence of at least one undesirable characteristic.

For each sub-algorithm for a undesirable characteristic, a database of random consumer images that exhibit that characteristic is collected. A second database of images that do not exhibit the characteristic are also compiled. Software is developed to detect the characteristic based on features in the image that are indicative of the characteristic. The databases are used to train the software to detect only images that exhibit the characteristic, and to avoid those that do not.

The automation of 1) the detection of unsuitable images; and 2) the detection of suitable images; is content-based. By content-based, it is meant that the image is classified based on the content of the image that is detected The image content includes the presence of objects in the image (including but not limited to people, skin, eyes, teeth, animals, sky, clouds, foliage, beach, water, snow and jpeg artifacts), as well as the condition of the image, including but not limited to the quality of the colours (including brightness) and factors indicative of the degree of exposure, sharpness, saturation and white balance. The degree or presence of a characteristic of an image may be indicated by histograms of, for example, image intensity/brightness or saturation. Measures other than histograms, including but not limited to fast fourier transforms (FFTs) or other transforms may also be used to determine the presence or degree of a characteristic. Histograms, FFTs, other transforms and/or other measures may also be applied to portions of an image rather than the entire image. The detection of image content includes but is not limited to the steps of segmentation, feature extraction and classifier development.
The location of the portion of an image to which the histogram, FFT, other transform or other measure is applied may be specified or it may be identified by the process of segmentation, feature extraction and classification.

SEGMENTATION

In the segmentation step objects or regions of interest are isolated in an image based on a number of visual cues including but not limited to color and texture. A number of segmentation techniques are used to deal with variations in resolu6on and lighting. For example, blue regions in an image can be segmented as potential sky regions.

FEATURE EXTRACTION

For each characteristic, information about segmented regions or objects is extracted from the image.
Functions are written to extract information describing the color, texture, shape and other features of each segmented region or object. These features may emulate the types of features used by humans to detect the characteristic. For example blue regions surrounding white and grey regions (like clouds) are more likely to be sky than blue regions completely surrounding red regions.

CLASSIFIER DEVELOPMENT

Feature information is used to decide which segmented objects/regions are actually the objects of interest. Functions are written to determine an appropriate sub-set of features which adequately describe the object of interest. The classifier algorithm determines if the value of each feature in the sub-set of features is within a range of values that have been determined through a training process to be indicative of the object of interest. The range of acceptable values of each feature used by the classifier algorithm is determined through training and testing using the development database.
EXPOSURE

The exposure of an image refers to the quantity of light allowed to act on a photographic material (a CCD array or film). It is a product of the intensity (controlled by the lens opening) and the duration (controlled by the shutter speed or enlarging time) of light striking the array or film.

When an image is overexposed, too much light reaches the array or film, producing a very brightRight image. When an image is underexposed, too little light reaches the array or film, producing a dark image, or a muddy-looking print.

It is not unusual for images to be acquired with incorrect exposure. There is no exact definition of what a"correcf' or "best" exposure should be. It can be defined generally as the exposure that enables one to reproduce the most important regions (according to contextual or perceptive criteria) with a level of gray or brightness, more or less in the middle of the possible range. The dynamic range of the exposure in an image is measured in EV (exposure value), or stops of exposure. A dynamic range of 5 EV means that the lightest object that shows detail in the image is 5 stops brighter than the darkest area with detail.
Daylight images typically range from an EV of 11 to an EV of 15. An EV of 5 is indicative of night home interiors with average lighting, while an EV of 15 corresponds to subjects in bright or hazy sun. An EV of 11 corresponds to subjects in open shade.

wil'"'I='a, r.,,; p,en v: "r~'i ~ ::.
~~ "~ ; ~=.~~', t r ~

i =~lx,'"~,,r~'a' i i11='.'' .

~fbl~iT/~Y~Figure 1. Seaside Image corresponding to histogram in Figure 2.

The seaside image shown above is an example of a correctly exposed image with a"good" brightness (or intensity) histogram as shown in the figure below.

. ~A"ik Figure 2. Intensity Histogram corresponding to the correctly exposed image of Fig.1.
The smooth curve of the histogram in Fig. 2 as it moves downwards ending in 255 shows that the subtle highlight detail in the clouds and waves is preserved. Likewise, the shadow area at the "toe" of the histogram starts at 0 (black) and builds up gradually. On this histogram each "stack" or "bar" is one pixel wide. The 256 bars are stacked side by side without any space between them.

0 z55 Figure 3. Histogram of an underexposed version of seaside image in Fig. 1.

The histogram of Fig.3 indicates there are a lot of pixels with value 0 or close to 0, which is an indication of "clipped shadows". Unless there is a lot of pure black in the image, there should not be that many pure black pixels. There are also very few pixels in the highlight area.

-A MOA

Figure 4. Histogram of overexposed version of seaside image from Fig 1.

The histogram in Fig. 4 indicates overexposure. There are a lot of pixels with vaiue 255 (white) or close to 255, which is an indication of "clipped highlights". Subtle highlight detail in the clouds and waves is lost. There are also very few pixels in the shadow area, from 0 to about 75.

It can be seen from the examples above that by examining the intensity/brightness histogram of an image, an indication of the degree of exposure can be obtained. This is the starting point for the assessment of image exposure. Other image features that may indicate the degree of exposure include but are not limited to the examination of EXIF header file information. EXIF
header files provide information regarding, among other things, whether or not a flash was used when the image was captured. If no flash is used for example, the probability that the image will be over-exposed is reduced, and the probability of under-exposure is increased.

The time/date stamp inserted in some digital images must be removed before analyzing the image to determine exposure, since the stamp would cause an artificially high number of bright or intense pixels to show up in the histogram.

BLUR DETECTION

Blur refers to an image or portion of an image that lacks sharpness. Sharpness refers to the amount of detail that can be perceived in an image. It is defined as an image's degree of clarity in terms of focus and contrast. In a sharp image objects are easily distinguished from each other, with well-defined edges and distinct bright and shadowed areas. A blurry image has poorly defined edges and indistinct transitions between bright and dark areas. Blurring in digital images can be produced through a number of causes:

= The camera is out of focus, or mistakenly focused on the background or foreground. When a camera lens, or any lens, is set to best focus for a specific object distance then at other distances (either near or far) the image is out of focus.

= The camera or subject moves while the picture is taken. The blur is only in one direction and is called motion blur.

= During the digitizing process in digital cameras continuous gradations of color are transformed into points on a regular sampling grid. Detail finer than the sampling frequency is averaged into a single pixel producing a softening effect.

= When an image is scanned the scanner interpolates pixels to produce greater resolution in dpi (dots per inch) than the scanner can detect.
Where blur is introduced, it is necessary to determine how much blur is present and whether the image should be sharpened, discarded or saved for printing, display or sharing.
Out of focus images are characterized by either the complete lack of edges or the lack of sharp edges.
An image with no edges would be very uninteresting and would probably be very rare. Images without sharp edges but with smooth edges are more common. Edge detection at different frequencies is performed in different regions of the image and a decision is made regarding blurriness. Things that complicate the decision include the presence of jpg artifacts, which can produce artificial vertical and horizontal edges in an unfocused image. Images taken of a subject in focus while the background or foreground is out of focus (shallow depth of field) may be problematic especially if the subject is small or very off centre. Images of a subject taken in front of a flat background must be accounted for.
Motion blur usually produces a sub-standard image. This type of blurring is more difficult to detect than out of focus blurring because the blurring is only be in one direction, leaving sharp edges in the perpendicular direction. For this reason directional edge information is gathered from different regions of the image. A lack of sharp edges in a certain direction indicates motion blur. Parallel lines in the perpendicular direction are a further indication of motion blur. Motion blur caused by the motion of something in the image is more difficult to detect if the object is small and not in the centre of the image.
It is further complicated by the fact that motion blur caused by the movement of an object in the image does not always make the image unacceptable. In fact the image may have been taken intentionally with the object moving.
The types of blur caused by digitization using a camera or scanner are less problematic than out of focus and motion blur. These are related to the resolution of the camera and scanner. The resulting images will be uniformly blurry, so if a consumer is trying to print or save the images, it is not feasible to inform them that all of their images are inferior. It may be useful to indicate that the images are low resolution, and to ask if the consumer wishes to proceed with printing or storage.

COLOR REPRODUCTION

The color reproduction of an image captured by a camera may be quite faithful to the original image, or it may be quite different. The reasons a captured image may differ from the original image are varied, including but not limited to the camera settings (either automatic or manual), internal camera color correction software and the camera hardware itself. "Color correction" is a phrase that is often used loosely to describe several different things. For the purposes of this proposal, color correction is the adjustment of color in a photographic image in an attempt to get the most realistic results. Color correction may be applied via internal camera software, or it can be applied in post-processing using photo editing software after the image has been downloaded onto a computer.
Photo editing software can be applied manually or automatically.

Not all images need color correction; some are fine as they are. However, many images can be improved with a little correction. The shortcoming of many automated image correction techniques used currently is that they apply the same "correction" to all images regardless of whether it is required or not.

A common method of improving the quality of an image is to use histogram contrast equalization. A
histogram provides a global description of the appearance of an image by charting the number of pixels at each tone level. Contrast equalization involves increasing the dynamic range of an image by maximizing the spread between adjacent tone levels. While equalization extracts information in regions where tone levels are tightly compressed, equalization can also cause hue shifts and over saturation.

Other commonly used methods include the "gray world" approach which assumes the average color of an image should be some predefined value of "gray" e.g. half the value of the maximum intensity for each color component. Also the white patch approach is similar to the gray world method but assumes that the maximum value of each channel should correspond to full white. There are many other methods of color correction each with its own merits and each work to some extent on a specific type of image.

These methods are typically applied as global techniques for the correction of digital color data in any type of image. However, each color correction algorithm or process only provides satisfactory results for a limited range of images. For example, a gray world approach may not work for a winter image containing a lot of snow. In such an image the average intensity of the image will be legitimately quite high, so manipulating the image to set average intensity to a lower "gray"
value would give the image an artificial unsatisfactory dull appearance.

The ImageChecker is a content based process for color reproduction assessment.
Image content is analyzed to first determine the quality of the color in the image, and then to determine whether color correction is necessary. If necessary, the color correction can be applied automatically or manually, or the image can be passed over for printing, discarded or deleted. The image content is analyzed to detect the presence of objects or characteristics including but not limited to:

- sky - people - skin - eyes - teeth - foliage - grass - an indoor or outdoor environment - etc.

Based on the image content, the quality of the image color reproduction is determined, followed by a decision indicating whether or not color correction is necessary. If it is determined that color correction is necessary, the most appropriate image correction is recommended based on the image content and the extent to which color correction is necessary.

The lmageChecker uses the color of reference objects detected in the image to make a decision about how to adjust the color. Reference objects like skin, teeth, whites of eyes, and foliage will be detected and their color analysed. From the target objects and their ideal colors (or ideal color range), the difference between detected objects and their ideal colors will be calculated.
The ideal colors may be distinct colors or they may consist of a range. They will be based on established reference color guidelines or guidelines will be developed. If the absolute value of the difference between the colors of the objects in an image and their ideal colors (or ideal color ranges) exceeds a threshold, then the quality of the image color reproduction is deemed to be inferior and color correction is recommended.
The image may then be corrected automatically or manually, or the image can be passed over for printing, discarded or deleted.

WHITE BALANCE
When a camera has been calibrated to correctly capture white, the camera is then considered to be "white balanced". Once the camera is calibrated for white, other colors should be captured accurately.
The white balance is a setting that compensates for the differences in color temperature of the ambient lighting. The light spectrum is scientifically described in terms of color temperature, and is measured in degrees Kelvin ( K). Photographers use three standard light color temprratures. The first, 55000 K, is called "daylight" for natural outdoors lighting, while the other two are incandescent (artificial light) color temperature standards: 3200 K for tungsten studio lamps and 34000 K for photo lamps and photofloods. Fluorescent tube manufacturers produce tubes with various color temperatures, the most common of which are: Warm White (3000-3500 N, Cool White (4100-4200 19 and Daylight (6000-7000 N.

The lower the color temperature the redder the light, and the higher the color temperature the bluer the light. In both analog and digital electronic cameras that use CCD and CMOS
sensors to capture the image, the white balance must be adjusted to ensure that all colors in the scene will be represented faithfully. It can be adjusted automafically by the camera, by selec6ng presets (tungsten, fluorescent, etc.) or by aiming the lens at a totally white surface (the white card) and selecting "lock white balance."
Under the proposed project, images will be analyzed to detect objects that should be white, e.g. teeth, eyes, clouds, and determining how white these objects actually appear in the image. If "white" objects are significantly outside a range of expected shades of white, then the image will be flagged as having poor white balance.

SIMILAR IMAGE DETECTION

Most personal image collections contain sequences of very similar images.
These are either taken automatically with a bracketing function on the camera or manually by taldng a number of pictures of the same subject in the hope that one will be satisfactory. Bracketing takes images at a variety of different settings in a burst of 2, 3 or 5 frames.

When selecting images for a photo book, to upload to a web site, or to print, it is desirable to avoid including repeat images and very similar images. Exact repeat images are easily detected. Similar image detection is more difficult. Similar images exhibiting exposure differences, angular differences, shifting, and flash level differences need to be detected within an image database. It is anticipated that different techniques will have to be implemented for the detection of each type of difference. When similar images are detected the software user will be warned and given the chance to select one or more of the similar images. Where possible the best quality image will be suggested.

A number of commercial and shareware products address the duplicate image problem using data transmission methods. Cyclic Redundancy Check (CRC) and Message-Digest (MD) hash functions, which do not look explicitly at file content, are useful in finding duplicates of any electronic file type. For detection of very similar images, however, details are required on image content.

Rudimentary image similarity approaches using global color histograms are ineffective in near-duplicate detection. The ImageChecker uses image content information including but not limited to exposure, flash level, orientation, and positioning, to make the similarity decision more robust.
JPEG ARTIFACTS

There are two types of image compression: lossless and lossy. With lossless compression only redundant information is removed; as a result the original and decompressed images are identical.
Lossy image compression takes advantage of the limitations of the human visual system and removes information that people do not easily see. This way, greater compression is achieved, but the original and the decompressed files are no longer the same. Sometimes the differences between the original and compressed images are visible and those differences are called artifacts.

JPEG compression is most commonly used as a lossy compression. JPEG (or JPG) stands for Joint Photographic Experts Group, and was named for the organization that developed the compression format. There are various levels of JPEG compression. High compression produces lower quality images and smaller files while lower compression produces higher quality images and larger file sizes.

One very common and noticeable JPEG artifact is blocking. In over-compressed JPEG images 8x8 pixel blocks are seen all over the image. Another common problem is "ringing"
around sharp edges in the image. Ringing produces abnormally bright pixels or dark pixels in the region of an edge.

A number of companies sell software to reduce the appearance of JPEG
blockiness. No companies sell software (to our knowledge) to automatically detect the presence of JPEG
blockiness or ringing in images. The ImageChecker automatically detects the presence of JPEG blockiness and ringing. This involves comparing the edge intensities at the boundaries between the 8x8 pixel blocks to the edge intensities inside the pixel blocks. Ringing is detected by finding occurrences of high contrast "sparkles"
near edges. In a cluttered image ringing will be more difficult to detect.

Claims