EP2143069A1 - Method for red eye detection in a digital image - Google Patents

Method for red eye detection in a digital image

Info

Publication number
EP2143069A1
EP2143069A1 EP07723937A EP07723937A EP2143069A1 EP 2143069 A1 EP2143069 A1 EP 2143069A1 EP 07723937 A EP07723937 A EP 07723937A EP 07723937 A EP07723937 A EP 07723937A EP 2143069 A1 EP2143069 A1 EP 2143069A1
Authority
EP
European Patent Office
Prior art keywords
regions
redness
face
skin
red
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Withdrawn
Application number
EP07723937A
Other languages
German (de)
French (fr)
Inventor
Claudio Cusano
Francesca Gasparini
Raimondo Schettini
Paolo Gallina
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Universita degli Studi di Milano Bicocca
Original Assignee
Universita degli Studi di Milano
Telecom Italia SpA
Universita degli Studi di Milano Bicocca
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Universita degli Studi di Milano, Telecom Italia SpA, Universita degli Studi di Milano Bicocca filed Critical Universita degli Studi di Milano
Publication of EP2143069A1 publication Critical patent/EP2143069A1/en
Withdrawn legal-status Critical Current

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00Image analysis
    • G06T7/0002Inspection of images, e.g. flaw detection
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00Image enhancement or restoration
    • G06T5/90Dynamic range modification of images or parts thereof
    • G06T5/94Dynamic range modification of images or parts thereof based on local image properties, e.g. for local contrast enhancement
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00Image analysis
    • G06T7/90Determination of colour characteristics
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/18Eye characteristics, e.g. of the iris
    • G06V40/193Preprocessing; Feature extraction
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/20Special algorithmic details
    • G06T2207/20004Adaptive image processing
    • G06T2207/20012Locally adaptive
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/30Subject of image; Context of image processing
    • G06T2207/30196Human being; Person
    • G06T2207/30201Face
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/30Subject of image; Context of image processing
    • G06T2207/30216Redeye defect

Definitions

  • the present invention relates to a method for red eye detection in a digital image, in particular a method in which the red eye detection is fully automatic and does not need user intervention.
  • the method of the invention maximises the detection rate, minimizing at the same time the "false positive" rate.
  • the red eye effect is a well known problem in photography. It is often seen in amateur shots taken with a built-in flash, but the problem is also well known to professional photographers.
  • Red eye is the red reflection of the blood vessels in the retina caused when a strong and sudden light strikes the eye, in particular when the pupil is expanded in dim light to allow more light to enter the eye.
  • red-eye removal solutions Most of them are semi-automatic or manual solutions. The user has to either click on the red-eye or draw a box containing it before the removal algorithm can find the red-eye pixels and correct them.
  • a method for removing a red eye from an image which includes 1) calculating a weighted red value for each pixel in the image from green, red and blue colour values and a luminance value of each pixel in the image, 2) selecting a plurality of pixels in the image having weighted red values greater than a threshold as red eye pixels, and 3) correcting some of the red eye pixels to remove the red eye from the image.
  • a definition of weighted red value is given and it represents the likelihood that a pixel is a purple-red pixel forming part of a red eye, is disclosed.
  • the software determines if each red eye region so identified is proximate to a facial region. If the facial region has a colour that is within the range of threshold skin colour values, then it is probably a pupil with red eye that is proximate to a facial region.
  • US patent application n. 2004/0213476 is relative to a system and method of detecting and correcting red-eye in a digital image.
  • measures of pixel redness in the digital image are computed.
  • a preliminary set of candidate red eye pixel areas is identified based on the computer pixel redness measures.
  • Each candidate red eye pixel area having a computed redness contrast relative to at least one respective neighbouring pixel area less then a prescribed redness contrast threshold is filtered from the preliminary set.
  • each candidate red eye pixel area located in an area of the digital image having a computed greyscale contrast relative to at least one respective neighbouring pixel area less than a prescribed greyscale contrast threshold is filtered from the preliminary set.
  • at least one candidate red eye pixel area in the image is detected.
  • Pixels in each candidate red eye pixel area are classified based on the redness and skin tone coloration. For each candidate red eye pixel area, an initial pixel mask identifying all pixels in the candidate red eye pixel area classified as red and non-skin tone is generated. A final pixel mask is generated based on each initial pixel mask. Red eye in the digital image is corrected by darkening and desaturating original colour values identified as red in the final pixel mask, when original colour values of pixels identified as red in the final pixel mask are desaturated by respective amounts that vary with pixel location in the final pixel mask.
  • the defective colour areas are recorded with a colour spectrum deviating form the actual colour spectrum of said area without colour defects, wherein basic areas in the image are identified on the basis of features which are common for these recorded defective areas, said basic areas supporting an increased likelihood to include defective areas, and the processing is then reduced to the basic areas to identifies borderlines and/or centres of the defective areas, and afterwards, it is identified whether the localized basic area or areas deemed to be defective are defective or not, and finally, if a location basic area has been identified to be defective, a correction mask is created to correct the visual appearance of the defective area.
  • Redeye is searched for only within these regions, seeking areas with high redness satisfying some geometric constraints.
  • a novel redeye removal algorithm is then applied automatically to the red eyes identified, and opportunely smoothed to avoid unnatural transitions between the corrected and original parts.
  • Experimental results on a set of over 450 images are reported.
  • a multi-resolution neural network approach is then exploited to create an analogous probability map for candidate faces. These two distributions are then combined to identify the most probable facial regions in the image. Redeye is then searched for within these regions, seeking areas with high redness and applying geometric constraints to limit the number of false hits. The redeye removal algorithm is then applied automatically to the red eyes identified. Candidate areas are opportunely smoothed to avoid unnatural transitions between the corrected and the original parts of the eyes. Experimental results of application of this procedure on a set of over 300 images are presented. Summary of the invention
  • the present invention is relative to a method to detect red eye(s) in a digital image, in particular a fully automatic detection method which involves no user's activity or interaction.
  • One of the main goals of the method of the invention is to obtain a high red-eye detection rate, so that most of the red eyes in the image are detected, and a very low "false positive" rate, i.e. only "real" red eyes are recognized and not other image details.
  • the method of the invention includes a face detection step. It is indeed desirable to determine whether or not there is a face in the image: once a face region is detected, the task of detecting red-eyes within the face region becomes simpler than processing the whole image.
  • an automatic face detector e.g., a computer-implemented algorithm, is preferably used.
  • any face detector that fulfills the requirement can be used in the invention.
  • the face detector operates in accordance to the method described in P. Viola and M.J.
  • the output of the face detection step comprises a first set of face regions, which are substantially delimited portions of the original input image, and for each identified face region in the input image the face detection module generates a bounding box enclosing the respective face.
  • the method of the invention comprises an additional step of skin detection in which the whole input image is processed. In this step, potential skin regions present in the image are analyzed in search for other potential face candidates that have not been detected by the face detector.
  • a skin detection algorithm based on the colour information of the pixels is applied to the input image.
  • the preferred colour space used to select pixel regions as skin regions is the luminance-chrominance space.
  • skin pixels are defined as those pixels having a chrominance within a given interval. More preferably a suitable algorithm used in this step of the invention can be found in F. Gasparini, R. Schettini "Skin segmentation using multiple thresholding" Proc. Internet imaging VII, Vol. SPIE 6061 (S. Santini, R. Schettini, T. Gevers eds.), pp. 60610F-1 , 60610F-8, 2006.
  • the output of this skin detection algorithm is a set of candidate skin regions.
  • a masking step is performed on the input image. Large “red” and “dark” regions (the meaning of “dark” and “red” will become clearer in the following detailed description) are removed form the input image to easier the task of skin detection (step detailed below).
  • the output of the skin detector algorithm comprises a relatively large set of candidate skin regions. Indeed, among the detected skin regions not only faces are included (which are the target regions because they contain eyes), but any other body part having a skin colour (i.e. arms, legs, necks, etc), background elements, clothes, etc. It is therefore preferred, that the skin detection process comprises a further step having the aim of limiting the number of skin regions to be further processed.
  • the skin regions number reduction is preferably performed using a first set of geometrical constraints.
  • a face has a typical geometrical shape: it is substantially elliptic and therefore its
  • roundness value can be measured, it has a possible range of ratios between its width and length, and preferably includes several other purely geometric characteristics (i.e., the presence of "holes” such eyes and mouth, etc) that can be used for identifying the skin regions which are probably also faces.
  • the face regions detected by the face detector are subtracted from the skin regions set. Those regions already identified by the face detector are considered good face candidates and therefore there is no need of processing them further.
  • a second aspect of the invention relates to a method of red eye detection.
  • the face regions obtained by applying a face detector and preferably a skin detector being limited using geometrical constraints are analyzed to identify red eye artifacts.
  • red eye artifacts have a dominant purple-brown component.
  • red eyes appear as bright spots both in a redness and in a purple-brown map.
  • Applicants have noticed that an evaluation of the magnitude of the purple-brown component within the image increases the probability of detecting red eye artifacts while decreasing the occurrence of false positive.
  • the analysis of the purple-brown component allows a good discrimination between hair and red eye artifacts within a face region.
  • R edness(x,y) LJ 2 x * » 0r ) - fo x,y) + B fr y j X .
  • PurpleBrown(;c, y) Max[ ⁇ , B(x, y) - G(x, y)] .
  • Two monochromatic maps are generated by this step of the method of the invention: a redness map and a purple-brown map, in which the original pixel colour data of the pixels of the face and skin regions under consideration is replaced by their redness value or the purple-brown value, respectively.
  • Many other red regions i.e. regions that appear "bright” in the redness map
  • the local maxima of the redness are first detected, in order to calculate the following quantities only on those regions.
  • a second set of geometric constraints is preferably created. Based on geometric considerations on the typical geometric shape of a red-eye, on the fact that eyes are located substantially at the centre of the face and not at its boundary, etc, the number of candidate red-eye regions is reduced.
  • a colour constraint is imposed on the candidate red-eye regions, preferably only on those regions satisfying the geometric constraints' second set.
  • the colour constraint requires that only those candidate red-eye regions whose pixels satisfy the following equations:
  • an additional step of the invention is performed, to detect the second eye.
  • the candidate red-eye regions already processed in the previous step to detect the first red eye are thus re-processed, loosening the geometric and colour conditions.
  • the red eye region selected as second red-eye is the one having the highest R ⁇ PB values.
  • the red eyes so detected are also corrected using a suitable algorithm. Any known algorithm can be used in this step of the method of the invention.
  • the algorithm to correct the red-eye region comprises a smoothing masking step (the resulting smoothed mask is referred in the following to as MASKsmooth) and a pixel color changing step of the red-eye region pixels.
  • the smoothing step is for example described in F. Gasparini, R. Schettini, "Automatic redeye removal for smart enhancement of photos of unknown origin", Proc. of the 8th International Conference on Visual Information Systems, Amsterdam, July 2005, Springer, Lecture Notes in Computer Sciences, Vol. 3736, 2005, pp. 226-233 and in F. Chazli, F. Gasparini, R. Schettini, "A modular procedure for automatic redeye correction in digital photos", Proc. Color Imaging IX: Processing, Hardcopy, and Applications IX, Proceedings of SPIE Vol. 5293 (R. Eschbach, G.G. Marcu eds.), pp.139- 147, 2004.
  • each pixel of coordinates (x,y) belonging to the red eye region(s) is then corrected.
  • a weighted average, based on a weighting factor w, of the B and G components is calculated.
  • the weighting factor w depends on the redness intensity of the pixel under consideration.
  • the original RGB colors of the pixel are then replaced with
  • R n ⁇ v (x,y) R(x, y) - (l - MASKsmooth) + MASKsmooth ⁇ (B(x, y) + G(x, y)) - — , W
  • B new (x, y) B(x, y) - (l - MASKsmooth) + MASKsmooth ⁇ (B(x, y) + G(x, y)) ⁇ — and
  • G new (x, y) G(x, y) (l - MASKsmooth) + MASKsmooth ⁇ (B(x, y) + G(x, y)) - — .
  • - fig. 1 is a schematic flow diagram of the method of the invention
  • - figs. 2 and 3 are exemplary input images and their corresponding processed results according to a step of face detection of the method of the invention
  • - figs. 4 and 5 are processed images according to an additional step of the invention of the input images of figs. 2 and 3, respectively;
  • - figs. 6 and 7 are processed images according to an additional step of the invention of the input images of figs. 4 and 5, and of the processed images of figs. 2 and 3, respectively;
  • - figs. 8 and 9 are processed images according to an additional step of the invention of the input images of figs. 6 and 7, respectively;
  • - fig. 10 represents the processed image of fig. 3 (top image), the same image filtered using a filter of the method of the invention (middle image), and a corresponding redness map (bottom image) according to an additional step of the method of the invention;
  • fig. 10a represents the redness mask of the region selected in fig. 9;
  • - fig. 11 represents the processed image of fig. 2 (top image), a redness map of the same image (middle image) and a purple-brown map (bottom image) of the same;
  • - figs. 12-13-14 represent the local redness maxima of the redness maps of figs. 10 -11 -
  • - figs. 15 and 16 represent the red eyes detected in the bounding box obtained in the image of fig. 3 and in the geometric skin-face region obtained for the same image, respectively;
  • - fig. 16a shows the additional red-eye detection in the geometric skin-face region of fig. 3; - figs. 17 and 18 shows the original and corrected images of fig. 2 and 3.
  • a flow chart representing the steps of the method of the invention according to a preferred embodiment of the same is depicted.
  • the method is applicable to any digital input image 100.
  • the digital colour input image 100 is processed according to the method of the invention which generates a digital colour output image 200 wherein the occurrences of red eyes present in the input image 100 have been detected and optionally corrected.
  • the input image 100 can be obtained for example using a digital camera, a mobile phone or the like (not shown). It can also be downloaded from the Internet, received by e-mail or stored in a removable memory storage device, such as a CD-Rom or diskette (also not shown).
  • the steps of the method of the invention of fig. 1 are implemented as one or more respective software modules that are for example executable on a computer (workstation).
  • a computer on which modules of the method may execute includes a processing unit, a system memory and a system bus that couples the processing unit to the various components of the computer.
  • the processing unit may have one or more processor.
  • the software modules of the method are incorporated in a printer and, once initiated, operate automatically.
  • the printer includes sufficient memory and processing resources to execute the programming instruction corresponding to the method of the invention.
  • the printer may for example include a "red eye correction button" that can be pressed in order to activate the method of the invention.
  • the modules may be incorporated in a photographic digital camera that takes and stores the image 100.
  • the input image 100 comprises a plurality of pixels that can be considered, at least logically, to be organized in an array or matrix having a plurality or rows and columns. Each pixel has an X- coordinate, which identifies the particular column within which the pixel is located, and a Y- coordinate, which identifies the particular row within which the pixel is located.
  • each pixel contains digital data which may be in the well known Red, Greed; Blue (RGB) colour space.
  • RGB Red, Greed
  • RGB Blue
  • the input image 100 can be in different formats other that RGB, such as, for example, Cyan, Magenta, Yellow, Black (CMYK); Hue, Saturation, Value (HSV), among others.
  • a length and a height of the image 100 are also defined, being the number of pixels along the X and Y direction, respectively. The longer between the length and the height is the maximum dimension (Max Dim) of the image 100.
  • the output image 200 has preferably the same number of pixels as the input image 100 organized in the same array. Those pixels identified, as detailed in the following, as corresponding to occurrences of red eyes are detected and preferably corrected, while all the other pixels remain unchanged.
  • the image size is re-scaled to a "standard size", so that the maximum dimension (Max Dim) of the rescaled image has always the same value (i.e., it is equal for example to 240 pixels, however other pixel numbers can be used).
  • a minimum and maximum face area is also preferably defined. As an example, it is supposed, from geometric consideration, that the maximum (minimum) possible area occupied by a face within the rescaled input image 100 is equal to:
  • the minimum and maximum area of a face are preferably to be modified, i.e. the two above written values are selected for faces having a height of at least 1/10 of the total height of the input image, which are the "most probable" face dimensions in pictures.
  • the subsequent step of the method of the invention is a face detection step.
  • the whole input image 100 (possibly rescaled in step 1) is processed by a suitable algorithm so that human faces are searched and located through the image 100.
  • a bounding box BB is generated in step 2a surrounding or encompassing each identified face.
  • the bounding box BB is preferably rectangular and each pixel within the bounding box is identified with its x and y coordinates. According to an embodiment of the invention, this step 2 of the method operates in accordance with the algorithm described in the Viola - Jones paper already identified above.
  • the algorithm used in the preferred embodiment of the invention is a face detector based on the combination of multiple "rectangle features", computed as the difference of the sum of pixel values in different rectangular areas of the input image.
  • the rectangle features are computed very quickly using an alternative image representation called the "Integral Image”.
  • AdaBoost® learning algorithm Using the AdaBoost® learning algorithm, a small number of critical visual features is selected from a very large set of potential features. The learning algorithm is executed several times to produce multiple classifiers which are combined in a "cascade". The cascade structure allows background regions of the image to be quickly discarded while spending more computation on promising face-like regions. Every square sub-window of the input image is processed by the trained cascade, and those sub-windows which are accepted by all the classifiers are classified as faces. The probability of a region to be a face depends on how many times the region has been considered a face in the sub-sampled images.
  • a binary mask 102 is thus created in step 2a, the dimensions of which are preferably equal to those of the rescaled input image 100. In mask 102, all pixels belonging to a region classified as a face region (the bounding box BB) have the value of "1" (which corresponds to white pixels), while all pixels outside the face regions have the value of "0" (which corresponds to black pixels).
  • n face bounding boxes BB are thus located included in mask 102.
  • fig. 2 and 3 The results of this method step 2 are illustrated in fig. 2 and 3.
  • the rescaled input image 100 at the far left is processed by the face detection algorithm and different face regions are located, which are represented as rectangles in the middle image 101.
  • the final binary mask 102 is represented at the far right: some of the rectangles previously selected as probable face regions have been disregarded as being too small compared to the average size face region.
  • Each rectangle represents the output bounding box BB containing a face. Both faces in the input image 100 are detected by the face detection algorithm and an additional bounding box is also represented which does not correspond to any "real" face in the input image 100.
  • the step 2 of face detection is applied to a different input image 100'.
  • the partially rotated man face has not been found by the algorithm, only the frontal child face is recognized and a single bounding box BB is created.
  • the resulting masked image 102' represents the original image 100 masked with the binary mask obtained using the face detection method: the regions corresponding to the "0" of the mask are deleted from the original image and only the region corresponding to the "1" of the mask are kept and shown.
  • the method of the invention preferably further comprises a skin detection step 3.
  • the whole input image 100, or preferably the rescaled image as detailed above is processed to detect regions that may contain "skin": this step in particular discriminates between skin pixels and non-skin pixels. Skin is preferably detected using a colour analysis of the image pixels, as better outlined below.
  • the skin detection step 3 is substantially an additional face detection step with the scope of detecting possible faces that have been "missed” by the face detector. According to the skin detection step, the detection of faces is based on the fact that a face contains "skin" and thus skin clusters are searched.
  • a first red and dark regions removal step 3a is performed on the input image 100.
  • a redness value for each pixel of the input image 100 (or the rescaled image) having coordinates (x,y) is calculated using the following equation:
  • RednessQc RednessQc,, (1) where R, G , B are red, green and blue component pixel values in the RGB colour space of the input image 100. Clearly, a similar redness calculation can be performed in different colour spaces depending on the input image colour format.
  • a redness mask is created in which all pixels of the original image whose computed redness is greater than a first redness threshold are set equal to "1", whereas the remaining pixels are set equal to "0".
  • the redness threshold is equal to 1.2.
  • the redness and darkness masks are combined in an AND operation so that the resulting mask is a binary matrix having values "1" corresponding to the pixels that have value "1" both in the redness and in darkness mask. All the other pixels (i.e.
  • pixel being "0" in both the redness and darkness masks or pixel being “0” either in the redness or in the darkness mask) are set equal to "0".
  • the resulting mask is then applied to the rescaled original input image 100 and the pixels corresponding to the "1" values are removed from the image 100.
  • a new masked image is therefore created, which is formed by applying the complementary mask of the AND combination of the redness and darkness masks. These remaining regions of the original rescaled input image 100 are called in the following darkness and redness masked image.
  • redness mask "small" red areas, i.e. smaller than 5/1000 of the rescaled input image 100 area, are removed because they can contain potential red eyes artifacts and thus need to be considered in the red eye detection phase. Therefore, in the masked image obtained after the application of the redness and darkness mask on the rescaled input image 100, these "small” red areas, i.e. areas the redness of which is above the redness threshold, are still present.
  • the skin detection step 3 takes place.
  • the skin detection algorithm which is preferably a recall oriented algorithm, based on the hypothesis that skin pixels exhibit similar colour coordinates in a properly chosen colour space, processes the rescaled input image 100, preferably masked using the redness and darkness masks.
  • a "skin cluster boundary" is defined, expressly defining the boundaries of the skin cluster in a certain colour space.
  • the rescaled redness and darkness masked input image 100 is transformed by the skin detection algorithm from the RGB colour space (or any other colour space in which the input image 100 is stored) to the YCbCr space, where Y is the luma component and Cb and Cr are the blue and red chroma components for each pixel.
  • the conversion matrix from one colour space to another is known in the art.
  • the rescaled and masked input image 100 is thus segmented in a plurality of m candidate skin regions SR, each region being a cluster of pixels satisfying eq. (3).
  • the areas of the candidate skin regions SR are evaluated: in case of too small overall skin regions, i.e., if the sum of the areas of all skin regions SR found by the skin detection algorithm is smaller than two times the minimum face area (see step 1 of the method of the invention where the minimum face area is calculated), a colour balancing method is applied to the input image 100, in a colour balancing step 5, such as Gray World.
  • This algorithm is based on the fact that the colors in a sufficiently complex scene average to gray.
  • the algorithm estimates the average colors of the image and shifts the average color so it becomes gray.
  • a different average colour may depend on the type of light used to illuminate the scenes or on defects of the camera used to take the input image 100.
  • a different colours balancing algorithm can be used, such as a white patch algorithm, as the one described in K. Barnard, V. Cardei, B. Funt, "A Comparison of Computational Color Constancy Algorithms-Part I: Methodology and Experiments with Synthesized Data” , IEEE Transactions on Image Processing 11 (9) (2002) 972-983.
  • For each pixel of the input image 100 preferably rescaled and masked using the redness and darkness masks, whose computed Cb and Cr values satisfy eq.
  • a binary skin mask 104 is then the output of the skin detection module 3.
  • the result of the skin detection step applied to the input image 100 of fig. 2 is shown.
  • the white regions represent the candidate skin regions SR.
  • the skin detection step 3 (and the redness and darkness masking 3a) is performed on input image 100' of fig. 3, as shown in fig. 5.
  • the result is displayed as masked image 104', which is obtained applying the skin binary mask to the original image 100'.
  • First the skin binary mask for input image 100' is calculated and then the regions of the binary mask set equal to 0 are removed from image 100', while the regions in which the corresponding mask is set equal to 1 are maintained. As visible, a single skin region SR is detected.
  • two binary masks are available: a mask 102 from the face detector step 2 including n bounding boxes BB, one for each candidate face, and a mask 104 from the skin detection step 3 containing m candidate skin regions SR.
  • the two masks are then preferably "subtracted" in step 6 of the method of the invention, i.e. regions identified as faces by the face detection algorithm (i.e. the bounding boxes BB) are set equal to 0 in the binary mask 104 obtained by the skin detection algorithm.
  • a new skin-face binary mask 105 is thus obtained, in which pixels are set equal to 1 if they belong to the regions of skin NOT identified as face regions by the face detector algorithm. In the skin-face mask 105 therefore all regions that are skin but not faces (in the sense of candidate face regions detected by the face detector) are included.
  • the corresponding masked image i.e.
  • the rescaled input image 100 masked with the skin-face mask 105 is thus supposed to include background elements having "skin colour", faces missed by the face detector algorithm and possible other body parts (such as arms, legs, etc.). These "skin-face” regions are referred to as SFR.
  • figs. 6 and 7 The result of the application of the subtraction step 6 is visualized in figs. 6 and 7.
  • fig. 6 the masks 102 and 104 of figs. 2 and 4, respectively, and the resulting skin-face mask 105 are shown
  • fig. 7 the masked image 105', which is the original (rescaled and masked) image 100 masked using the skin-face mask, is depicted showing the remaining skin regions SFR.
  • a geometric analysis step 7 is then preferably performed in the method of the invention on the skin-face mask 105.
  • the skin-face mask 105 is analyzed and at least one of the following geometric constrains is verified: all skin-face regions SFR having an area smaller than the minimum face area (defined in step 1 of the method of the invention) are ruled out. - Discontinuity (D) calculation. "Holes" within the skin-face region SFR are searched.
  • a second binary skin-face mask 106 is thus calculated.
  • This mask 106 contains as pixels having value set equal to 1 the skin-face regions SFR of mask 105 whose pixels satisfy the following equation
  • geometric threshold (5) wherein according to a preferred embodiment the geometric threshold has a value of 1.6. All the other skin-face regions SFR not satisfying eq. (5), are set equal to 0.
  • the geometric analysis step 7 thus reduces the number of skin-face regions which are candidates to contain faces.
  • the effect of the application of the geometric analysis step 7 is shown. In the original input image 100, in which two persons are shown, both faces have been detected by the face detector of step 2 (two correctly detected bounding boxes BB).
  • the image masked with the skin-face mask 106 is now rescaled again to return to its original size, i.e. to the original input image size, in order to process the image further at its full resolution.
  • the bounding boxes BB obtained via the face detector algorithm of step 2a are then analyzed in a subsequent step 20 of the invention. This step may be performed immediately after the step of face detection 2.
  • Each bounding box BB is processed using the skin detector of step 3. If at least 50% of the total area delimited by the bounding box is recognized as a skin region(s), the pixels within the box BB are filtered using the redness filter of equation (1) (i.e. only pixels having a redness less than 1.2 are maintained).
  • Each masked bounding box BB is then transformed to the YCbCr space and only the pixels satisfying eq. (3) are kept, thus generating a new filtered bounding box FBB for each bounding box BB of mask 102.
  • non- convex regions are closed using morphological operations.
  • the skin detector of step 3 reveals one or more skin regions within the bounding box BB whose area(s) is less than 50% of the total area of the bounding box BB, only the redness filter is applied, but not eq. (3), obtaining for each bounding box BB of mask 102 having "small" skin portions a masked bounding box MBB.
  • Masked bounding boxes MBB and filtered bounding boxes FBB form a mask 109.
  • the redness mask (see eq. (1)) is again applied to the skin-face regions which have been rescaled back to their full resolution.
  • this (sub)step corresponds to the box "definition of the single skin-face region" in the analysis step 7. Due to the rescaling, additional "extremely red” pixels may have been reintroduced and may create disturbance in the following step of red-eye detection.
  • step 8 of the method of the invention in which the red eye artifacts are detected, only the face regions of the input image identified by the face detector and additionally processed as explained in step 20 above (which are the masked bounding boxes and the filtered bounding boxes), and the geometric skin-face regions obtained masking the rescaled image 100 with mask 106 obtained at the end of the geometric analysis step 7 are further analyzed. Both those regions are the candidate face regions CFR in which the algorithm for red eye detection looks for red eyes.
  • the masks 109 and 106 are used to segment the original input image 100: only the regions of the original image 100 having pixels set to 1 in masks 109 or mask 106 are further analyzed in the following steps of the method of the invention. In other words, only the "1" regions of masks 106 and 109 are kept of the original image 100 and are the "candidate face regions" CFR, all the other regions (which are "0" both in mask 106 and in mask 109) are removed. Each candidate face region is processed as outlined below. At the end of the processing, all the following is repeated for the next candidate face region till all of them have been processed.
  • PurpleBrownO, y) Max[ ⁇ , B ⁇ x, y) - G ⁇ x, y)] (6) where B and G are the blue and green components in the RGB color space.
  • the redness values calculation creates for each candidate face region CFR a monochrome image 110, in gray scale, wherein the highly intense red regions appear as bright spots.
  • the purple-brown values calculation generates for each candidate face region CFR a second monochrome image 111 wherein bright regions appear, which are the regions in which the purple-brown component is dominant. Applicants have noted that the purple-brown component is an important component of red eyes' artifacts.
  • the bounding box BB originated applying the face detector to original image 100' of fig. 3 is shown (top image).
  • the redness filter and then the skin detector are applied, generating the filtered bounding box FBB (middle image).
  • the results of the redness calculation, i.e., the redness values as defined in eq. (1) calculated for each pixel of the filtered bounding box FBB and then plotted, are represented as image 110' (bottom image).
  • one of the bounding boxes BB obtained applying the face detector step to the image 100 is depicted (top image).
  • the respective corresponding redness map 110 and purple-brown map 111 are also shown (middle and bottom images, respectively).
  • the redness map is shown for the skin region GSFR (see masked image 106' of fig. 9) detected by the skin detector in the image 100' of fig. 3.
  • Figure 10 shows that red eyes are identified as bright spots both in the redness and purple- brown maps.
  • the purple-brown and redness maps are preferably smoothed using, for example, a bi- dimensional Gaussian filter, in order to remove small discontinuities and highlights local redness maxima.
  • the Gaussian filter is calculated from the following equation (7):
  • redness and purple-brown maps 110, 111 obtained using eq. (1) and (6) for each candidate face region CFR.
  • the local maxima of the redness are also calculated. This calculation selects a given number of "red regions" RLM which might be red eyes to be optionally corrected.
  • the imextendedmax function identifies all regional maxima that are greater than a specified threshold. It computes the extended-maxima transform, which is the regional maxima of the H-maxima transform. H is a nonnegative scalar.
  • a regional maximum is a connected set of pixels of constant intensity from which it is impossible to reach a point with higher intensity without first descending; that is, a connected component of pixels with the same intensity value, t, surrounded by pixels that all have a value less than t.
  • the local maxima with external boundary pixel having a value lower than 50%, 40%, 30%, 20%, 10% of the maximum value t are then calculated and summed.
  • PrMax is the redness smoothed using the Gaussian filter and then normalized.
  • each redness local maximum region RLM is processed according to the following sub-steps.
  • Too large local maxima regions are also discarded. Too large means for example larger than 0,01 of the total area of the candidate face region.
  • red eye region has the following characteristics:
  • the upper and lower limits depend, among other, on the colour space of the input image. If several regions satisfy equations (11) and (12), the two having higher PR are considered to be red eyes RE. If no region satisfies the above mentioned equations, no redeye RE is detected.
  • a single region satisfying eq. (11) and (12) is found (the number of red eyes found is calculated in step 8a of the method of the invention), then the region is considered to be a red eye RE and a subsequent step 10 of the method of the invention is performed to locate the second eye of the face.
  • the red-eye detection step is shown.
  • both red eyes RE of the bounding box of image 100' are immediately detected using the geometric and colour constraints above described.
  • a single red eye RE is detected and thus the following step of the invention is performed.
  • the second red eye is not immediately recognized due, most probably, to the presence of glasses.
  • the second eye locator step 10 it is assumed that frequently in an input image both eyes of a person are shown. Therefore, after the localization of the first red eye region RE, all the local redness maxima RLM considered in the sub-steps above outlined, with the exception of all regions RLM too close or too far from the first red eye region RE already detected, are again processed.
  • the distance between the first detected red eye region RE and the candidate second red eye region is preferably comprised between 3 and 10 times the major axis of an ellipsis encircling the first detected red eye region RE. Additionally, also redness local maxima regions having an area 25% smaller than, or larger than 2 times, the area of the first red eye region RE already located are discarded.
  • a new geometric and colour analysis of the redness local maxima regions RLM (already calculated using the redness map 110) is performed.
  • the constraints for the detection of the second red eye are less strict than for the first one.
  • the new geometric constraints to be satisfied by a local maxima to be considered the second red eye are the following: O > 17% e ⁇ 0.85 (13)
  • PB fifth lower purple-brown limit
  • ⁇ & R fifth lower redness limit
  • the third and fourth lower redness and purple-brown limits are smaller or equal to the first and second redness and purple-brown limits, respectively.
  • the above values are equal to:
  • PB 0 and R > 1.4.
  • the region having the highest PB R value is considered to be second red eye region RE2.
  • the detection of the second eye RE2 is shown in fig. 16a. Before the local maxima regions processing, the region already identified as the first red eye RE is removed from the image. The second red eye RE2 is thus immediately identified, as shown in the mask at the right. An optional additional step 11 of the method of the invention is thus performed in order to correct the red eyes RE, RE2 artefacts detected.
  • the algorithm used in this step is the colour correction of the red-eye(s). If a pixel has been detected as belonging to a red eye region RE, RE2 it is replaced with a substantially monochrome pixel, as detailed below. First of all, a smoothing mask is created having the size of the red-eye region to be corrected. This mask avoids unnatural transitions between corrected and original parts of the eye.
  • the weighting factor w is calculated as follows:
  • Equation (15) above the minimum value is taken between the lred value and 1.
  • the value of lred is normalized with respect to its maximum value reached in the red region under consideration.
  • G new (JC, y) G(x, y) - (l - MASKsmooth) + MASKsmooth ⁇ (B(x, y) + G(x, y)) — .
  • Each pixel of the red eye region is thus replaced with a pixel having the above calculated colour. It is however avoided to apply the mentioned correction to the possible white pixels (the flesh "glint”) and black (eyelashes) present in the red eye region.
  • a corrected digital image 200 is thus created.
  • Figs. 17 and 18 show the red eye correction on image 100 and 100' obtaining image 200 and 200' respectively. From the modified picture, it is visible the "natural" appearance of the corrected eyes.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Ophthalmology & Optometry (AREA)
  • Human Computer Interaction (AREA)
  • Multimedia (AREA)
  • Quality & Reliability (AREA)
  • Image Analysis (AREA)
  • Image Processing (AREA)

Abstract

The present invention relates to a method for red-eye detection in a digital input image (100, 100'), comprising: a face detection step to identify one or more face regions (SR, BB) within the input image (100, 100'), a redness and purple-brown mapping step, wherein for each pixel of said face regions (SR, BB) having (x, y) coordinates in the input image (100, 100') its redness and purple-brown values are calculated as follows (formula (I)), where f and f are functions, α-(β+γ)=0; α, μ, v > 0; β, γ ≥ 0, and a candidate red-eye regions (LRM) identification step, said candidate red eye regions (LRM) being portions of said face regions (SR, BB) having a computed redness average (R) and a computed purple-brown average (PB ) which satisfy: PB > first lower purple-brown limit and R > first lower redness limit, and lower PR limit < PR = R PB < upper PR limit. A further aspect of the invention relates to a method identifying at least one face in a digital image.

Description

Method for red eye detection in a digital image Technical field
The present invention relates to a method for red eye detection in a digital image, in particular a method in which the red eye detection is fully automatic and does not need user intervention.
Additionally, the method of the invention maximises the detection rate, minimizing at the same time the "false positive" rate.
Technological background
The red eye effect is a well known problem in photography. It is often seen in amateur shots taken with a built-in flash, but the problem is also well known to professional photographers.
Red eye is the red reflection of the blood vessels in the retina caused when a strong and sudden light strikes the eye, in particular when the pupil is expanded in dim light to allow more light to enter the eye.
Fixing red-eye artefacts digitally became an important skill with the advent of digital technologies which permit to acquire digitalized images either directly with a digital camera or converting traditional photos by scanners. Also the widespread use of small devices with built-in flash, including cell phones and handheld computers, produces a large amount of digital photographs that are potentially red-eye affected.
Currently, many image processing software applications in the market offer red-eye removal solutions. Most of them are semi-automatic or manual solutions. The user has to either click on the red-eye or draw a box containing it before the removal algorithm can find the red-eye pixels and correct them.
Other software applications require little or no user intervention and the software identifies the red eye automatically.
In the US patent application n. 2005/0232481 in the name of Donghui Wu, a method for removing a red eye from an image, which includes 1) calculating a weighted red value for each pixel in the image from green, red and blue colour values and a luminance value of each pixel in the image, 2) selecting a plurality of pixels in the image having weighted red values greater than a threshold as red eye pixels, and 3) correcting some of the red eye pixels to remove the red eye from the image. A definition of weighted red value is given and it represents the likelihood that a pixel is a purple-red pixel forming part of a red eye, is disclosed. Additionally, the software determines if each red eye region so identified is proximate to a facial region. If the facial region has a colour that is within the range of threshold skin colour values, then it is probably a pupil with red eye that is proximate to a facial region.
US patent application n. 2004/0213476 is relative to a system and method of detecting and correcting red-eye in a digital image. In one aspect, measures of pixel redness in the digital image are computed. A preliminary set of candidate red eye pixel areas is identified based on the computer pixel redness measures. Each candidate red eye pixel area having a computed redness contrast relative to at least one respective neighbouring pixel area less then a prescribed redness contrast threshold is filtered from the preliminary set. In addition, each candidate red eye pixel area located in an area of the digital image having a computed greyscale contrast relative to at least one respective neighbouring pixel area less than a prescribed greyscale contrast threshold is filtered from the preliminary set. In another aspect, at least one candidate red eye pixel area in the image is detected. Pixels in each candidate red eye pixel area are classified based on the redness and skin tone coloration. For each candidate red eye pixel area, an initial pixel mask identifying all pixels in the candidate red eye pixel area classified as red and non-skin tone is generated. A final pixel mask is generated based on each initial pixel mask. Red eye in the digital image is corrected by darkening and desaturating original colour values identified as red in the final pixel mask, when original colour values of pixels identified as red in the final pixel mask are desaturated by respective amounts that vary with pixel location in the final pixel mask.
In the US patent n. 6885766 in the name of Held A. et al., a method for automatically correcting colour defective areas in an image is described. In the mentioned method, the defective colour areas are recorded with a colour spectrum deviating form the actual colour spectrum of said area without colour defects, wherein basic areas in the image are identified on the basis of features which are common for these recorded defective areas, said basic areas supporting an increased likelihood to include defective areas, and the processing is then reduced to the basic areas to identifies borderlines and/or centres of the defective areas, and afterwards, it is identified whether the localized basic area or areas deemed to be defective are defective or not, and finally, if a location basic area has been identified to be defective, a correction mask is created to correct the visual appearance of the defective area. In "Automatic redeye removal for smart enhancement of photos of unknown origin" written by F. Gasparini and R. Schettini, published in Proc. of the 8th International Conference on Visual Information Systems, Amsterdam, July 2005, Springer, Lecture Notes In Computer Sciences, Vol. 3736, 2005, pages 226-233, a modular procedure for automatic correction of redeye artefact in images of unknown origin, maintaining the natural appearance of the eyes is outlined. First, a smart colour balancing procedure is applied. This phase not only facilitates the subsequent step of processing, but also improves the overall appearance of the output image. Combining the results of a colour-based face detector and of a face detector based on a multi- resolution neural network the most likely facial region are identified. Redeye is searched for only within these regions, seeking areas with high redness satisfying some geometric constraints. A novel redeye removal algorithm is then applied automatically to the red eyes identified, and opportunely smoothed to avoid unnatural transitions between the corrected and original parts. Experimental results on a set of over 450 images are reported.
In "A modular procedure for automatic redeye correction in digital photos", written by R. Schettini, F. Gasparini and F. Chazli, published in Proc. Color Imaging IX: Processing, Hardcopy, and Applications IX, Proceedings of SPIE Vol. 5293 (R. Eschbach, G. G. Marcueds.), pages 139-147, 2004, an algorithm for the automatic removal of "redeye" from digital photos is disclosed. First an adaptive colour cast removal algorithm is applied to correct the colour photo. This phase not only facilitates the subsequent step processing, but also improves the overall appearance of the output image. A skin detector, based mainly on the chromatic distribution of the image, creates a probability map of skin-like region. A multi-resolution neural network approach is then exploited to create an analogous probability map for candidate faces. These two distributions are then combined to identify the most probable facial regions in the image. Redeye is then searched for within these regions, seeking areas with high redness and applying geometric constraints to limit the number of false hits. The redeye removal algorithm is then applied automatically to the red eyes identified. Candidate areas are opportunely smoothed to avoid unnatural transitions between the corrected and the original parts of the eyes. Experimental results of application of this procedure on a set of over 300 images are presented. Summary of the invention
The present invention is relative to a method to detect red eye(s) in a digital image, in particular a fully automatic detection method which involves no user's activity or interaction. One of the main goals of the method of the invention is to obtain a high red-eye detection rate, so that most of the red eyes in the image are detected, and a very low "false positive" rate, i.e. only "real" red eyes are recognized and not other image details.
The method of the invention includes a face detection step. It is indeed desirable to determine whether or not there is a face in the image: once a face region is detected, the task of detecting red-eyes within the face region becomes simpler than processing the whole image. For this purpose, an automatic face detector, e.g., a computer-implemented algorithm, is preferably used. For the actual detection, any face detector that fulfills the requirement can be used in the invention. In a preferred embodiment of the invention, the face detector operates in accordance to the method described in P. Viola and M.J. Jones "Robust Real-Time Face Detection", International Journal of Computer Vision 57(2), pages 137-154 (2004), downloaded from the Internet on March 1 , 2007 at http://www.vision.caltech.edu/html-files/EE148-2005- Spring/pprs/viola04ijcv.pdf. Those skilled in the art will recognize that the method of the invention may be used with other systems designed to recognize human faces in images. In the selection of a suitable face detection what is of importance is that the detection of faces occurs fully automatically. It is furthermore advantageous that the detection rate is reasonably high and the false negative rate, i.e., faces are detected even though there is no face present, is reasonably low.
According to the above described algorithm, the output of the face detection step comprises a first set of face regions, which are substantially delimited portions of the original input image, and for each identified face region in the input image the face detection module generates a bounding box enclosing the respective face.
Any face detector can be used in the method of the invention, as long as it classifies image portions in face and non-face regions. As examples, face detector algorithms can be based on neural networks, learning algorithms, etc. It is known that face detection in a digital image is a challenging task because the overall appearance of faces ranges widely in scale, rotation, orientation and pose, as well as in facial expression and lighting conditions. Any algorithm used to detect faces in images has therefore difficulties to recognize all possible faces, with particular reference to rotated faces, and thus it is probable that some faces are missed in this processing step. According to a first preferred aspect, the method of the invention comprises an additional step of skin detection in which the whole input image is processed. In this step, potential skin regions present in the image are analyzed in search for other potential face candidates that have not been detected by the face detector. According to a preferred embodiment, a skin detection algorithm based on the colour information of the pixels is applied to the input image. The preferred colour space used to select pixel regions as skin regions is the luminance-chrominance space. In this space, "skin pixels" are defined as those pixels having a chrominance within a given interval. More preferably a suitable algorithm used in this step of the invention can be found in F. Gasparini, R. Schettini "Skin segmentation using multiple thresholding" Proc. Internet imaging VII, Vol. SPIE 6061 (S. Santini, R. Schettini, T. Gevers eds.), pp. 60610F-1 , 60610F-8, 2006.
The output of this skin detection algorithm is a set of candidate skin regions. Preferably, prior to the skin detector processing detailed above, a masking step is performed on the input image. Large "red" and "dark" regions (the meaning of "dark" and "red" will become clearer in the following detailed description) are removed form the input image to easier the task of skin detection (step detailed below).
Generally, the output of the skin detector algorithm comprises a relatively large set of candidate skin regions. Indeed, among the detected skin regions not only faces are included (which are the target regions because they contain eyes), but any other body part having a skin colour (i.e. arms, legs, necks, etc), background elements, clothes, etc. It is therefore preferred, that the skin detection process comprises a further step having the aim of limiting the number of skin regions to be further processed.
The skin regions number reduction is preferably performed using a first set of geometrical constraints. A face has a typical geometrical shape: it is substantially elliptic and therefore its
"roundness value" can be measured, it has a possible range of ratios between its width and length, and preferably includes several other purely geometric characteristics (i.e., the presence of "holes" such eyes and mouth, etc) that can be used for identifying the skin regions which are probably also faces. Preferably, before the above described geometric selection, the face regions detected by the face detector are subtracted from the skin regions set. Those regions already identified by the face detector are considered good face candidates and therefore there is no need of processing them further.
A second aspect of the invention relates to a method of red eye detection. According to this second aspect of the invention, the face regions obtained by applying a face detector and preferably a skin detector being limited using geometrical constraints, are analyzed to identify red eye artifacts.
Applicants have observed that red eye artifacts have a dominant purple-brown component.
Therefore, red eyes appear as bright spots both in a redness and in a purple-brown map. Applicants have noticed that an evaluation of the magnitude of the purple-brown component within the image increases the probability of detecting red eye artifacts while decreasing the occurrence of false positive. In particular, the analysis of the purple-brown component allows a good discrimination between hair and red eye artifacts within a face region.
For this purpose, for each face region obtained from a face detector and preferably from a skin detector, a corresponding redness and purple-brown map is evaluated. In particular, redness and purple-brown values of each pixel having (x,y) coordinates of the face region are defined and calculated as follows: where f is a generic function and a-(β+y)=0; α>0, β,γ >0; a, β, y, are not limited to integer values, and δ is a normalization factor that can have any desired value depending on the preferred normalization; PurpleBrown(x,y) = f (μ - B(x,y) - v - G(x,y)), where f is a generic function; μ and v are both positive values, preferably both < 2, more preferably of about 1.
In the equations above, it has been assumed that the input image is in the RGB colour space, where R (Red), G (Green) and B (Blue) are the space generators. However, any input image format space can be used in the method of the invention, corresponding purple-brown and redness equations are calculated from the above ones using known transformation matrixes. In a preferred embodiment of the invention, the redness and purple-brown take the following form: in the redness equation preferably α= γ=δ=λ e β=0, so that
Redness(x,y) = LJ 2 x *»0r) - fox,y) + Bfryj X .
I I R(χ.y) J) while in the purple-brown equation μ = v = 1 so that it results
PurpleBrown(;c, y) = Max[θ, B(x, y) - G(x, y)] .
In an additional embodiment of the invention, the purple-brown map is calculated using the same equation as above, but without the max function, i.e., PurpleBrown(x,;/) = [/?(*, y) - G{χ, y)]. After this first calculation, all values of the mapping which result negative are set equal to zero.
Two monochromatic maps are generated by this step of the method of the invention: a redness map and a purple-brown map, in which the original pixel colour data of the pixels of the face and skin regions under consideration is replaced by their redness value or the purple-brown value, respectively. Many other red regions (i.e. regions that appear "bright" in the redness map) may be included in a input image, not only red eyes. Therefore, to detected red eyes among all the regions having a high average redness value and/or a high average purple-brown value, additional constraints are imposed. Only the regions fulfilling the following constraints are to be considered "real" red- eyes.
Preferably, the local maxima of the redness are first detected, in order to calculate the following quantities only on those regions.
A second set of geometric constraints is preferably created. Based on geometric considerations on the typical geometric shape of a red-eye, on the fact that eyes are located substantially at the centre of the face and not at its boundary, etc, the number of candidate red-eye regions is reduced.
A colour constraint is imposed on the candidate red-eye regions, preferably only on those regions satisfying the geometric constraints' second set. In detail, the colour constraint requires that only those candidate red-eye regions whose pixels satisfy the following equations:
[ PB > first lower purple-brown limit <S R > first lower redness limit, or
PB > second lower purple-brown limit & R > second lower redness limit ], and lower PR limit < PR = R - PB < upper PR limit, where R is the average redness and PB is the average purple-brown of the candidate red-eye region under consideration, are considered to be red eyes. Preferably, the above limits are equal to
[ PB > 10 <& R > 0.3 , or
PB > 0.02 & R > 0.5 , ] and OA < PR = R P~B < 50, which have been found from experiments. If no region satisfies the above equations, no red eye is detected and the next candidate face or skin region is processed, creating new redness and purple-brown masks. Otherwise, if more than two regions satisfy the above equations, preferably the detected red eyes are the two region having the highest and second highest R PB values.
In case a single candidate red eye region satisfies the above equation, preferably an additional step of the invention is performed, to detect the second eye. Indeed, it is preferred for a good aesthetic appearance that no red eye is corrected than a single red eye among two is corrected. The candidate red-eye regions already processed in the previous step to detect the first red eye are thus re-processed, loosening the geometric and colour conditions. Again, the red eye region selected as second red-eye is the one having the highest R PB values. Preferably, the red eyes so detected are also corrected using a suitable algorithm. Any known algorithm can be used in this step of the method of the invention.
In a preferred embodiment, the algorithm to correct the red-eye region comprises a smoothing masking step (the resulting smoothed mask is referred in the following to as MASKsmooth) and a pixel color changing step of the red-eye region pixels. The smoothing step is for example described in F. Gasparini, R. Schettini, "Automatic redeye removal for smart enhancement of photos of unknown origin", Proc. of the 8th International Conference on Visual Information Systems, Amsterdam, July 2005, Springer, Lecture Notes in Computer Sciences, Vol. 3736, 2005, pp. 226-233 and in F. Chazli, F. Gasparini, R. Schettini, "A modular procedure for automatic redeye correction in digital photos", Proc. Color Imaging IX: Processing, Hardcopy, and Applications IX, Proceedings of SPIE Vol. 5293 (R. Eschbach, G.G. Marcu eds.), pp.139- 147, 2004.
The color of each pixel of coordinates (x,y) belonging to the red eye region(s) is then corrected. For each pixel a weighted average, based on a weighting factor w, of the B and G components is calculated. The weighting factor w depends on the redness intensity of the pixel under consideration. The original RGB colors of the pixel are then replaced with
Rnαv (x,y) = R(x, y) - (l - MASKsmooth) + MASKsmooth (B(x, y) + G(x, y)) - — , W
Bnew (x, y) = B(x, y) - (l - MASKsmooth) + MASKsmooth (B(x, y) + G(x, y)) — and
Gnew (x, y) = G(x, y) (l - MASKsmooth) + MASKsmooth (B(x, y) + G(x, y)) - — .
Using this procedure, the possible white pixels present within the red eye regions are left unaltered and the corrected red eye region(s) has a "natural" appearance. Brief description of the drawings
Further features and advantages of a method for red eye detection in a digital image according to the present invention will become more clear from the following detailed description thereof, given with reference to the accompanying drawings, where:
- fig. 1 is a schematic flow diagram of the method of the invention; - figs. 2 and 3 are exemplary input images and their corresponding processed results according to a step of face detection of the method of the invention;
- figs. 4 and 5 are processed images according to an additional step of the invention of the input images of figs. 2 and 3, respectively;
- figs. 6 and 7 are processed images according to an additional step of the invention of the input images of figs. 4 and 5, and of the processed images of figs. 2 and 3, respectively;
- figs. 8 and 9 are processed images according to an additional step of the invention of the input images of figs. 6 and 7, respectively;
- fig. 10 represents the processed image of fig. 3 (top image), the same image filtered using a filter of the method of the invention (middle image), and a corresponding redness map (bottom image) according to an additional step of the method of the invention; fig. 10a represents the redness mask of the region selected in fig. 9;
- fig. 11 represents the processed image of fig. 2 (top image), a redness map of the same image (middle image) and a purple-brown map (bottom image) of the same; - figs. 12-13-14 represent the local redness maxima of the redness maps of figs. 10 -11 -
10a respectively; - figs. 15 and 16 represent the red eyes detected in the bounding box obtained in the image of fig. 3 and in the geometric skin-face region obtained for the same image, respectively;
- fig. 16a shows the additional red-eye detection in the geometric skin-face region of fig. 3; - figs. 17 and 18 shows the original and corrected images of fig. 2 and 3.
Preferred embodiments of the invention
With initial reference to fig. 1 , a flow chart representing the steps of the method of the invention according to a preferred embodiment of the same is depicted. The method is applicable to any digital input image 100. In detail, the digital colour input image 100 is processed according to the method of the invention which generates a digital colour output image 200 wherein the occurrences of red eyes present in the input image 100 have been detected and optionally corrected. The input image 100 can be obtained for example using a digital camera, a mobile phone or the like (not shown). It can also be downloaded from the Internet, received by e-mail or stored in a removable memory storage device, such as a CD-Rom or diskette (also not shown).
In one embodiment, the steps of the method of the invention of fig. 1 are implemented as one or more respective software modules that are for example executable on a computer (workstation). In general, a computer on which modules of the method may execute includes a processing unit, a system memory and a system bus that couples the processing unit to the various components of the computer. The processing unit may have one or more processor.
In a preferred embodiment, the software modules of the method are incorporated in a printer and, once initiated, operate automatically. The printer includes sufficient memory and processing resources to execute the programming instruction corresponding to the method of the invention. The printer may for example include a "red eye correction button" that can be pressed in order to activate the method of the invention. Alternatively, the modules may be incorporated in a photographic digital camera that takes and stores the image 100. The input image 100 comprises a plurality of pixels that can be considered, at least logically, to be organized in an array or matrix having a plurality or rows and columns. Each pixel has an X- coordinate, which identifies the particular column within which the pixel is located, and a Y- coordinate, which identifies the particular row within which the pixel is located. In addition, each pixel contains digital data which may be in the well known Red, Greed; Blue (RGB) colour space. It should be understood that the input image 100 can be in different formats other that RGB, such as, for example, Cyan, Magenta, Yellow, Black (CMYK); Hue, Saturation, Value (HSV), among others.
A length and a height of the image 100 are also defined, being the number of pixels along the X and Y direction, respectively. The longer between the length and the height is the maximum dimension (Max Dim) of the image 100. As better described below, the output image 200 has preferably the same number of pixels as the input image 100 organized in the same array. Those pixels identified, as detailed in the following, as corresponding to occurrences of red eyes are detected and preferably corrected, while all the other pixels remain unchanged. As a possible initial step 1 of the method of the invention, the image size is re-scaled to a "standard size", so that the maximum dimension (Max Dim) of the rescaled image has always the same value (i.e., it is equal for example to 240 pixels, however other pixel numbers can be used).
A minimum and maximum face area is also preferably defined. As an example, it is supposed, from geometric consideration, that the maximum (minimum) possible area occupied by a face within the rescaled input image 100 is equal to:
Minimum area of a face= (maximum dimension of the image/10)2 X (3/8) Maximum area of a face= (area of the input image)/2.
For very different face sizes in an image, however, the minimum and maximum area of a face are preferably to be modified, i.e. the two above written values are selected for faces having a height of at least 1/10 of the total height of the input image, which are the "most probable" face dimensions in pictures.
The subsequent step of the method of the invention, indicated as 2 in the flow chart of fig. 1 , is a face detection step. In the face detection step, the whole input image 100 (possibly rescaled in step 1) is processed by a suitable algorithm so that human faces are searched and located through the image 100. A bounding box BB is generated in step 2a surrounding or encompassing each identified face.
The bounding box BB is preferably rectangular and each pixel within the bounding box is identified with its x and y coordinates. According to an embodiment of the invention, this step 2 of the method operates in accordance with the algorithm described in the Viola - Jones paper already identified above.
More in detail, the algorithm used in the preferred embodiment of the invention is a face detector based on the combination of multiple "rectangle features", computed as the difference of the sum of pixel values in different rectangular areas of the input image. The rectangle features are computed very quickly using an alternative image representation called the "Integral Image".
Using the AdaBoost® learning algorithm, a small number of critical visual features is selected from a very large set of potential features. The learning algorithm is executed several times to produce multiple classifiers which are combined in a "cascade". The cascade structure allows background regions of the image to be quickly discarded while spending more computation on promising face-like regions. Every square sub-window of the input image is processed by the trained cascade, and those sub-windows which are accepted by all the classifiers are classified as faces. The probability of a region to be a face depends on how many times the region has been considered a face in the sub-sampled images. The most probable region among those classified as probable faces is used to determine the "average" face size: all regions which have been considered as faces by the algorithm, but which are too small or to big compared to the averaged face size (= area of the most probable face region), are thus disregarded. "Too small" or "too large" means for example larger or smaller than 8 and %, respectively, of the average face size. Face regions having a probability of less than 30% to be faces are also preferably discarded. A binary mask 102 is thus created in step 2a, the dimensions of which are preferably equal to those of the rescaled input image 100. In mask 102, all pixels belonging to a region classified as a face region (the bounding box BB) have the value of "1" (which corresponds to white pixels), while all pixels outside the face regions have the value of "0" (which corresponds to black pixels).
From the above described algorithm, n face bounding boxes BB are thus located included in mask 102.
The results of this method step 2 are illustrated in fig. 2 and 3. In fig. 2, the rescaled input image 100 at the far left is processed by the face detection algorithm and different face regions are located, which are represented as rectangles in the middle image 101. The final binary mask 102 is represented at the far right: some of the rectangles previously selected as probable face regions have been disregarded as being too small compared to the average size face region. Each rectangle represents the output bounding box BB containing a face. Both faces in the input image 100 are detected by the face detection algorithm and an additional bounding box is also represented which does not correspond to any "real" face in the input image 100.
In fig. 3, the step 2 of face detection is applied to a different input image 100'. As shown, the partially rotated man face has not been found by the algorithm, only the frontal child face is recognized and a single bounding box BB is created. The resulting masked image 102' represents the original image 100 masked with the binary mask obtained using the face detection method: the regions corresponding to the "0" of the mask are deleted from the original image and only the region corresponding to the "1" of the mask are kept and shown. The method of the invention preferably further comprises a skin detection step 3. In this step 3, the whole input image 100, or preferably the rescaled image as detailed above, is processed to detect regions that may contain "skin": this step in particular discriminates between skin pixels and non-skin pixels. Skin is preferably detected using a colour analysis of the image pixels, as better outlined below.
The skin detection step 3 is substantially an additional face detection step with the scope of detecting possible faces that have been "missed" by the face detector. According to the skin detection step, the detection of faces is based on the fact that a face contains "skin" and thus skin clusters are searched.
Before the skin detection phase, preferably a first red and dark regions removal step 3a is performed on the input image 100. In this optional step 3a, a redness value for each pixel of the input image 100 (or the rescaled image) having coordinates (x,y) is calculated using the following equation:
RednessQc,,) (1) where R, G , B are red, green and blue component pixel values in the RGB colour space of the input image 100. Clearly, a similar redness calculation can be performed in different colour spaces depending on the input image colour format.
Additionally, also a darkness value for each pixel of the input image 100 is calculated:
d ,ark . ness ( .x,y ,)= _ (2)
From the rescaled input image 100, a redness mask is created in which all pixels of the original image whose computed redness is greater than a first redness threshold are set equal to "1", whereas the remaining pixels are set equal to "0". Preferably the redness threshold is equal to 1.2. Additionally, all pixels of the rescaled original image 100 having a darkness smaller than a darkness threshold, which is preferably equal to 60, are set equal to 1 , while all the other pixels are set equal to 0, creating a darkness mask. The redness and darkness masks are combined in an AND operation so that the resulting mask is a binary matrix having values "1" corresponding to the pixels that have value "1" both in the redness and in darkness mask. All the other pixels (i.e. pixel being "0" in both the redness and darkness masks or pixel being "0" either in the redness or in the darkness mask) are set equal to "0". The resulting mask is then applied to the rescaled original input image 100 and the pixels corresponding to the "1" values are removed from the image 100. A new masked image is therefore created, which is formed by applying the complementary mask of the AND combination of the redness and darkness masks. These remaining regions of the original rescaled input image 100 are called in the following darkness and redness masked image. Clearly, the same result would have been obtained by masking the original image with a mask resulting from the AND operation of a redness mask formed by "1" values corresponding to pixels having redness smaller than 1.2 and a darkness mask formed by "1" values corresponding to pixels having darkness larger than 60.
From the above computed redness mask, "small" red areas, i.e. smaller than 5/1000 of the rescaled input image 100 area, are removed because they can contain potential red eyes artifacts and thus need to be considered in the red eye detection phase. Therefore, in the masked image obtained after the application of the redness and darkness mask on the rescaled input image 100, these "small" red areas, i.e. areas the redness of which is above the redness threshold, are still present.
After this optional redness and darkness masking step 3a, the skin detection step 3 takes place. The skin detection algorithm, which is preferably a recall oriented algorithm, based on the hypothesis that skin pixels exhibit similar colour coordinates in a properly chosen colour space, processes the rescaled input image 100, preferably masked using the redness and darkness masks. A "skin cluster boundary" is defined, expressly defining the boundaries of the skin cluster in a certain colour space. In the preferred embodiment of skin detection step 3, the rescaled redness and darkness masked input image 100 is transformed by the skin detection algorithm from the RGB colour space (or any other colour space in which the input image 100 is stored) to the YCbCr space, where Y is the luma component and Cb and Cr are the blue and red chroma components for each pixel. The conversion matrix from one colour space to another is known in the art. After the above mentioned transformation, only those pixels whose Cb and Cr coordinates satisfy the following equations are considered to be skin pixels: lower Cb limit < Cb < upper Cb limit (3) lower Cr limit < Cr < upper Cr limit, where, preferably, lower Cb limit = 74, upper Cb limit = 124, lower Cr limit = 134, and upper Cr limit = 185.
These limiting values (74-124 and 134-185) for the Cb and Cr coordinates, so that only pixels within the intervals of eq. (3) are considered to be skin pixels, are obtained according to the method described in F. Gasparini, R. Schettini "Skin segmentation using multiple thresholding" Proc. Internet imaging VII, Vol. SPIE 6061 (S. Santini, R. Schettini, T. Gevers eds.), pp. 60610F-1 , 6061 OF-8, 2006.
The rescaled and masked input image 100 is thus segmented in a plurality of m candidate skin regions SR, each region being a cluster of pixels satisfying eq. (3). In step 4 of the method of the invention, the areas of the candidate skin regions SR are evaluated: in case of too small overall skin regions, i.e., if the sum of the areas of all skin regions SR found by the skin detection algorithm is smaller than two times the minimum face area (see step 1 of the method of the invention where the minimum face area is calculated), a colour balancing method is applied to the input image 100, in a colour balancing step 5, such as Gray World. This algorithm is based on the fact that the colors in a sufficiently complex scene average to gray. Therefore, the algorithm estimates the average colors of the image and shifts the average color so it becomes gray. A different average colour may depend on the type of light used to illuminate the scenes or on defects of the camera used to take the input image 100. Alternatively, a different colours balancing algorithm can be used, such as a white patch algorithm, as the one described in K. Barnard, V. Cardei, B. Funt, "A Comparison of Computational Color Constancy Algorithms-Part I: Methodology and Experiments with Synthesized Data" , IEEE Transactions on Image Processing 11 (9) (2002) 972-983. For each pixel of the input image 100, preferably rescaled and masked using the redness and darkness masks, whose computed Cb and Cr values satisfy eq. (3), the corresponding mask value is asserted, e.g., set to "1". For those pixels whose computed Cb and Cr coordinates do not satisfy (3), the corresponding mask value is de-asserted, e.g., set to "0". A binary skin mask 104 is then the output of the skin detection module 3. In fig. 4, the result of the skin detection step applied to the input image 100 of fig. 2 is shown. First the input image 100 is rescaled (the image shown in fig. 4 is indeed the rescaled image) and masked using the redness and darkness masking: the pixels regions set equal to 1 both in the redness and in the darkness mask are first removed (the result of this masking step is non shown in fig. 4) according to step 3a of the method of the invention and then the result of the skin detection algorithm 3 is shown as binary skin mask 104. The white regions represent the candidate skin regions SR.
Analogously, the skin detection step 3 (and the redness and darkness masking 3a) is performed on input image 100' of fig. 3, as shown in fig. 5. The result is displayed as masked image 104', which is obtained applying the skin binary mask to the original image 100'. First the skin binary mask for input image 100' is calculated and then the regions of the binary mask set equal to 0 are removed from image 100', while the regions in which the corresponding mask is set equal to 1 are maintained. As visible, a single skin region SR is detected. At this stage, therefore, two binary masks are available: a mask 102 from the face detector step 2 including n bounding boxes BB, one for each candidate face, and a mask 104 from the skin detection step 3 containing m candidate skin regions SR. The two masks are then preferably "subtracted" in step 6 of the method of the invention, i.e. regions identified as faces by the face detection algorithm (i.e. the bounding boxes BB) are set equal to 0 in the binary mask 104 obtained by the skin detection algorithm. A new skin-face binary mask 105 is thus obtained, in which pixels are set equal to 1 if they belong to the regions of skin NOT identified as face regions by the face detector algorithm. In the skin-face mask 105 therefore all regions that are skin but not faces (in the sense of candidate face regions detected by the face detector) are included. The corresponding masked image, i.e. the rescaled input image 100 masked with the skin-face mask 105, is thus supposed to include background elements having "skin colour", faces missed by the face detector algorithm and possible other body parts (such as arms, legs, etc.). These "skin-face" regions are referred to as SFR.
The result of the application of the subtraction step 6 is visualized in figs. 6 and 7. In fig. 6 the masks 102 and 104 of figs. 2 and 4, respectively, and the resulting skin-face mask 105 are shown, while in fig. 7 the masked image 105', which is the original (rescaled and masked) image 100 masked using the skin-face mask, is depicted showing the remaining skin regions SFR.
To limit the number of false hits, i.e. to detect the "skin-face" regions, SFR, having low probability of being "real" faces, a geometric analysis step 7 is then preferably performed in the method of the invention on the skin-face mask 105. In this step 7, the skin-face mask 105 is analyzed and at least one of the following geometric constrains is verified: all skin-face regions SFR having an area smaller than the minimum face area (defined in step 1 of the method of the invention) are ruled out. - Discontinuity (D) calculation. "Holes" within the skin-face region SFR are searched.
Indeed, eyes, mouth, etc represent holes in the skin-face regions of mask 105 and, therefore, if a discontinuity within a skin-face region is present, this region has a higher probability to be a "good" face region. Skin-face regions SFR including one or more holes have a discontinuity value set equal to 1 , otherwise the regions have a D=O.5. - Solidity (S). The ratio between the area of the processed skin-face region SFR and the area of the smallest convex region included in the skin-face region under issue defines solidity. A low solidity means that the region presents many loops and bends (such as in a hand), while a high solidity represents a "roundish" object. The ideal solidity of a face is equal to 1. - Ratio (R). The ratio between the minor (MinAxis) and major axis (MaxAxis) of an equivalent ellipsis surrounding the skin-face region SFR analyzed is calculated. This is an index of the ellipticity of the skin-face region.
Eccentricity (ε). This value indicates the length and narrowness of the skin-face region under issue. It is obtained using the following formulae: R = MinAxis_ (4)
MaxAxis
ε = Jl -R2 . • After having calculated in step 7 of the method of the invention at least one or preferably all of the geometric quantities above defined for each skin-face region SFR of mask 105, or optionally other geometric values which may be of interest, a second binary skin-face mask 106 is thus calculated. This mask 106 contains as pixels having value set equal to 1 the skin-face regions SFR of mask 105 whose pixels satisfy the following equation
(D + S + E + R) > geometric threshold (5) wherein according to a preferred embodiment the geometric threshold has a value of 1.6. All the other skin-face regions SFR not satisfying eq. (5), are set equal to 0. The geometric analysis step 7 thus reduces the number of skin-face regions which are candidates to contain faces. The SFR that remain after the geometrical constraints imposed on step 7 of the invention, i.e., skin- face region satisfying eq. (5), are called geometric skin-face regions GSFR forming mask 106. In fig. 8, the effect of the application of the geometric analysis step 7 is shown. In the original input image 100, in which two persons are shown, both faces have been detected by the face detector of step 2 (two correctly detected bounding boxes BB). Additional potential "faces" regions SFR have then been found by the skin detector algorithm (see mask 105 on the left and fig. 6). From the application of the geometric constraints of eq. (5), the skin-face regions SFR, which are potentially faces, are reduced to one. The single geometric skin-face region GSFR is depicted in mask 106. In fig. 9, masked image 106' is obtained applying the geometrical constraints of eq. (5) to the masked image 105' of fig. 7. Only the geometric skin-face region GSFR corresponding to the undetected man face is now present as a possible face region.
As an additional preferred step of the invention, the image masked with the skin-face mask 106 is now rescaled again to return to its original size, i.e. to the original input image size, in order to process the image further at its full resolution. The bounding boxes BB obtained via the face detector algorithm of step 2a are then analyzed in a subsequent step 20 of the invention. This step may be performed immediately after the step of face detection 2. Each bounding box BB is processed using the skin detector of step 3. If at least 50% of the total area delimited by the bounding box is recognized as a skin region(s), the pixels within the box BB are filtered using the redness filter of equation (1) (i.e. only pixels having a redness less than 1.2 are maintained). Each masked bounding box BB is then transformed to the YCbCr space and only the pixels satisfying eq. (3) are kept, thus generating a new filtered bounding box FBB for each bounding box BB of mask 102. Preferably, non- convex regions are closed using morphological operations.
Otherwise, if the skin detector of step 3 reveals one or more skin regions within the bounding box BB whose area(s) is less than 50% of the total area of the bounding box BB, only the redness filter is applied, but not eq. (3), obtaining for each bounding box BB of mask 102 having "small" skin portions a masked bounding box MBB. Masked bounding boxes MBB and filtered bounding boxes FBB form a mask 109.
Preferably, the redness mask (see eq. (1)) is again applied to the skin-face regions which have been rescaled back to their full resolution. In fig. 1 , this (sub)step corresponds to the box "definition of the single skin-face region" in the analysis step 7. Due to the rescaling, additional "extremely red" pixels may have been reintroduced and may create disturbance in the following step of red-eye detection.
In the next step 8 of the method of the invention, in which the red eye artifacts are detected, only the face regions of the input image identified by the face detector and additionally processed as explained in step 20 above (which are the masked bounding boxes and the filtered bounding boxes), and the geometric skin-face regions obtained masking the rescaled image 100 with mask 106 obtained at the end of the geometric analysis step 7 are further analyzed. Both those regions are the candidate face regions CFR in which the algorithm for red eye detection looks for red eyes.
More in detail, to perform the red-eye detection, the masks 109 and 106 are used to segment the original input image 100: only the regions of the original image 100 having pixels set to 1 in masks 109 or mask 106 are further analyzed in the following steps of the method of the invention. In other words, only the "1" regions of masks 106 and 109 are kept of the original image 100 and are the "candidate face regions" CFR, all the other regions (which are "0" both in mask 106 and in mask 109) are removed. Each candidate face region is processed as outlined below. At the end of the processing, all the following is repeated for the next candidate face region till all of them have been processed. The following steps of the method of the invention are thus detailed only for a single candidate face region CFR, being the processing of the remaining substantially identical. The redness and purple-brown values of all the pixels contained in a selected candidate face region CFR are calculated. In particular, for each pixel of said region, its redness value is calculated as indicated in eq. (1), while its purple-brown value is defined as follow:
PurpleBrownO, y) = Max[θ, B{x, y) - G{x, y)] (6) where B and G are the blue and green components in the RGB color space. The redness values calculation creates for each candidate face region CFR a monochrome image 110, in gray scale, wherein the highly intense red regions appear as bright spots. The purple-brown values calculation generates for each candidate face region CFR a second monochrome image 111 wherein bright regions appear, which are the regions in which the purple-brown component is dominant. Applicants have noted that the purple-brown component is an important component of red eyes' artifacts.
In fig. 10, the bounding box BB originated applying the face detector to original image 100' of fig. 3 is shown (top image). To the bounding box BB, the redness filter and then the skin detector are applied, generating the filtered bounding box FBB (middle image). The results of the redness calculation, i.e., the redness values as defined in eq. (1) calculated for each pixel of the filtered bounding box FBB and then plotted, are represented as image 110' (bottom image). In fig. 11 , one of the bounding boxes BB obtained applying the face detector step to the image 100 is depicted (top image). The respective corresponding redness map 110 and purple-brown map 111 are also shown (middle and bottom images, respectively). In fig. 10a the redness map is shown for the skin region GSFR (see masked image 106' of fig. 9) detected by the skin detector in the image 100' of fig. 3.
Figure 10 shows that red eyes are identified as bright spots both in the redness and purple- brown maps. The purple-brown and redness maps are preferably smoothed using, for example, a bi- dimensional Gaussian filter, in order to remove small discontinuities and highlights local redness maxima. The Gaussian filter is calculated from the following equation (7):
F(x,y) = e ^ (7) where σ is not a constant, but depends on the ratio between the total number of pixels in the considered candidate face region and the number of pixels classified as skin. This ratio is called
"RatioSkin" and σ is given by the following equation σ = 2.5* RatioSkin.
It can bee seen that in a background face, the sigma is small, while in a forefront face the sigma is rather large. Indeed, in forefront faces, eyes are relatively large and thus it is not suitable to analyze small redness variations. On the other hand, in small faces also small redness variations may correspond to red eyes.
Given the redness and purple-brown maps 110, 111 obtained using eq. (1) and (6) for each candidate face region CFR, the local maxima of the redness are also calculated. This calculation selects a given number of "red regions" RLM which might be red eyes to be optionally corrected.
In greater detail, the local redness maxima are calculated using a Matlab® function called BW = imextendedmax(l,H). The imextendedmax function identifies all regional maxima that are greater than a specified threshold. It computes the extended-maxima transform, which is the regional maxima of the H-maxima transform. H is a nonnegative scalar.
A regional maximum is a connected set of pixels of constant intensity from which it is impossible to reach a point with higher intensity without first descending; that is, a connected component of pixels with the same intensity value, t, surrounded by pixels that all have a value less than t.
The local maxima with external boundary pixel having a value lower than 50%, 40%, 30%, 20%, 10% of the maximum value t are then calculated and summed. As maximum region RML, a region including at least three maxima is selected, for example: RML=(imextendedmax(PrMax,0.5,4)+imextendedmax(PrMax,0.4,4)+imextendedmax(PrMax,0.3 ,4)+imextendedmax(PrMax,0.2,4)+imextendedmax(PrMax,0.1 ,4)) > 3, which means that the expression above written should result a local maximum region at least three times to be considered a RLM. PrMax is the redness smoothed using the Gaussian filter and then normalized.
In figs. 12, 13 and 14, the local maxima RLM of images 110 and 110' (with the reference number 110', both images of fig. 10 and of fig. 10a are indicated) are respectively shown. In order to select from all the local maxima regions RLM those having a high probability of being red eyes, a geometrical and colour evaluation of each regions RLM is thus performed. Each redness local maximum region RLM is processed according to the following sub-steps.
1. All the regions of local maxima RLM which are too close to the boundary of the candidate face region CFR are discarded. Preferably, after the removal of the redness local maxima regions at the boundary of the candidate face region, new redness local maxima are calculated. This step is performed in order to avoid the undetection of some red eye(s) (which may be much smaller local maxima compared to the initial ones) due to the extreme redness of some contour regions.
2. Too large local maxima regions are also discarded. Too large means for example larger than 0,01 of the total area of the candidate face region.
3. If after sub-steps 1 and 2, the redness local maxima region(s) total area is still more than 35% wider than the candidate face region area, the processing of this face region is ended and the next candidate face region is examined. If this occurs, it means that this extremely wide red region(s) probably represents skin, such as sunburnt skin or a pull.
4. The following geometrical quantities are calculated for each candidate face region (CFR): eccentricity (ε); ratio (R) between the major and minor axis of the equivalent ellipsis surrounding the candidate face region; ratio (F) between the height and length of a rectangle bounding the candidate face region. From all these quantities, only candidate face regions satisfying the following constraints are kept in the following steps:
R = MnAxiS > 0.6, MaxAxis
F = ^^ > 0.6. height
- position of the local maximum region RLM with respect to the candidate face region. In a ideal face the position of the eye (xeye.yeye) should not be at the boundary of the candidate face region having a given length and height, that is only regions : xc < 0.93 * Length (9)
yc < 0.97 * height
where Xe and yc are the centers of the local maximum region.
- occupation index (O). This value is used to disregard redness maxima regions having holes inside. Accepted local maxima regions are those whose ratio between the redness local maximum region area and the area of an ellipsis with axis MinAxis and MaxAxis is preferably
O = ^^ > 0.2. (10) π • MinAxis • MaxAxis
- Solidity (S). Redness local maxima regions the redness of which is comprisedbetween 0.07% and 0.95% are accepted. 5. If after sub-step 4, there are still more than eight local maxima of the redness, then the analysis of the candidate face region CFR is terminated and the next candidate face region is analyzed. Otherwise, the method of the invention proceeds with the following sub-step 6.
6. Colour analysis. Of each redness local maximum region accepted in sub-step 4, the following quantities are calculated: - white pixel number. Frequently, within a red eye, one or more white pixel(s) is(are) present, due to the flash of the camera which has taken the input image. The presence of white pixel(s) within a local maximum of redness increases the probability of having detected a red eye artifact.
- For each redness local maximum region, the average redness (R ) and the average purple-brown (PB ) within the region are calculated. From experimental results, Applicants have noted that a red eye region has the following characteristics:
[ PB > first lower purple-brown limit & R > first lower redness limit or PB > second lower purple-brown limit & R > second lower redness limit] and lower PR limit < PR = R - PB < upper PR limit.
Preferably, according to experimental results, the equations above are written as
/" PB > 10 & R > 0.3 or (11) PB > 0.02 & R > 0.57 and 0Λ < PR = R - P~B < 50. (12)
The upper and lower limits depend, among other, on the colour space of the input image. If several regions satisfy equations (11) and (12), the two having higher PR are considered to be red eyes RE. If no region satisfies the above mentioned equations, no redeye RE is detected.
If a single region satisfying eq. (11) and (12) is found (the number of red eyes found is calculated in step 8a of the method of the invention), then the region is considered to be a red eye RE and a subsequent step 10 of the method of the invention is performed to locate the second eye of the face. In figs. 15 and 16 the red-eye detection step is shown. In fig. 15 both red eyes RE of the bounding box of image 100' are immediately detected using the geometric and colour constraints above described. In fig. 16, in which only the skin-face region of image 100' is shown, a single red eye RE is detected and thus the following step of the invention is performed. The second red eye is not immediately recognized due, most probably, to the presence of glasses.
In the second eye locator step 10, it is assumed that frequently in an input image both eyes of a person are shown. Therefore, after the localization of the first red eye region RE, all the local redness maxima RLM considered in the sub-steps above outlined, with the exception of all regions RLM too close or too far from the first red eye region RE already detected, are again processed. The distance between the first detected red eye region RE and the candidate second red eye region is preferably comprised between 3 and 10 times the major axis of an ellipsis encircling the first detected red eye region RE. Additionally, also redness local maxima regions having an area 25% smaller than, or larger than 2 times, the area of the first red eye region RE already located are discarded. A new geometric and colour analysis of the redness local maxima regions RLM (already calculated using the redness map 110) is performed. The constraints for the detection of the second red eye are less strict than for the first one. The new geometric constraints to be satisfied by a local maxima to be considered the second red eye are the following: O > 17% e < 0.85 (13)
R > 0.5F> 0.57, and, regarding the colours: PB > third lower purple-brown limit & R > third lower redness limit or
PB > fourth lower purple-brown limit & R > fourth lower redness limit, or
PB > fifth lower purple-brown limit <& R > fifth lower redness limit, where in particular the third and fourth lower redness and purple-brown limits are smaller or equal to the first and second redness and purple-brown limits, respectively. Preferably, the above values are equal to:
PB > 8 and R > 0.3 or
PB > 0.01 and R > 0.4 (14) or
PB > 0 and R > 1.4. Among the regions satisfying both the geometric and the colours constraints of equations (13) and (14), the region having the highest PB R value is considered to be second red eye region RE2.
The detection of the second eye RE2 is shown in fig. 16a. Before the local maxima regions processing, the region already identified as the first red eye RE is removed from the image. The second red eye RE2 is thus immediately identified, as shown in the mask at the right. An optional additional step 11 of the method of the invention is thus performed in order to correct the red eyes RE, RE2 artefacts detected. The algorithm used in this step is the colour correction of the red-eye(s). If a pixel has been detected as belonging to a red eye region RE, RE2 it is replaced with a substantially monochrome pixel, as detailed below. First of all, a smoothing mask is created having the size of the red-eye region to be corrected. This mask avoids unnatural transitions between corrected and original parts of the eye.
The smoothing mask has a size equal to the size of the skin or face region under consideration in which the red eye regions RE, RE2 are found. It is obtained smoothing through a Gaussian filter the RE mask. Then, for each pixel having coordinates (x,y) within the red eye region RE, RE2 to be corrected, the average of B and G is calculated: Average (x,y)= (G(x,y)+ B(x,y))/2 This average value is then multiplied by a weighting factor w which depends on the redness intensity of the pixel itself. The higher is the redness intensity, the lower is the resulting value. A low value results in a dark correction. The weighting factor w is calculated as follows:
In equation (15) above, the minimum value is taken between the lred value and 1. The value of lred is normalized with respect to its maximum value reached in the red region under consideration.
Preferably coeff=1.3 and lred is the redness intensity of the face or skin region considered. . The original RGB colour components are then replaced by w Rnew (jc, y) = R(χ, y) - (l - MASKsmooth) + MASKsmooth ■ (B(x, y) + G(x, y)) - — ,
w Bnew (x > y) = β(x > ^) - O - MASKsmooth) + MASKsmooth (B(x, y) + G(x, y)) - — ,
Gnew (JC, y) = G(x, y) - (l - MASKsmooth) + MASKsmooth (B(x, y) + G(x, y)) — .
Each pixel of the red eye region is thus replaced with a pixel having the above calculated colour. It is however avoided to apply the mentioned correction to the possible white pixels (the flesh "glint") and black (eyelashes) present in the red eye region.
A corrected digital image 200 is thus created.
Using this correction, the white pixels possibly present within the red eye are left unmodified.
Figs. 17 and 18 show the red eye correction on image 100 and 100' obtaining image 200 and 200' respectively. From the modified picture, it is visible the "natural" appearance of the corrected eyes.

Claims

Claims
1. A method for red-eye detection in a digital input image (100,100'), comprising:
- a face detection step to identify one or more face regions (SR, BB) within the input image (100,100'), - a redness and purple-brown mapping step, wherein for each pixel of said face regions
(SR1BB) having (x,y) coordinates in the input image (100,100') its redness and purple- brown values are calculated as follows
Re dness(X,y)
Purple - brown(x, y) = f\μ - B(x, y) - v - G(x, y)), where f and f are functions, α-(β+γ)=0; α,μ,v>0; β.γ≥O, - a candidate red-eye regions (LRM) identification step, said candidate red eye regions
(LRM) being portions of said face regions (SR.BB) having a computed redness average (R ) and a computed purple-brown average (PB ) which satisfy:
PB > first lower purple-brown limit and R > first lower redness limit, and lower PR limit < PR = R - PB < upper PR limit.
2. The method according to claim 1 , wherein the step of identifying candidate red-eye regions comprises the step of considering additionally as candidate red eye regions (LRM) regions portions of said face regions (SR1BB) having
PB > second lower purple-brown limit & R > second lower redness limit and lower PR limit < PR = R PB < upper PR limit, wherein the second lower purple-brown limit is lower than the first lower purple-brown limit and the second lower redness limit is higher than the first lower redness limit.
3. The method according to claim 1 or 2, comprising the step of
- reducing the number of candidate red-eye regions (LRM) using a first set of one or more geometrical constraints.
4. The method according to any of the preceding claims, wherein said candidate red-eye regions (LRM) are local maxima regions of said redness mapping.
5. The method according to any of the preceding claims, wherein said face detection step comprises a sub-step of creating a binary mask (102, 104) in which all pixels belonging to a region classified as said face region (BB1SR) have the value of 1 , while all pixels outside the face regions have the value of 0.
6. The method according to any of the preceding claims, comprising a step of masking said input image (100,100') using a redness mask to remove from said input image (100,100') regions larger than expected red-eye regions, before said redness and purple-brown mapping step.
7. The method according to claim 6, wherein regions having a redness higher than a redness threshold are removed from the input image (100,100').
8. The method according to any of the preceding claims, comprising a step of masking said input image (100,100') using a darkness mask, before said redness and purple- brown mapping step.
9. The method according to claim 8, wherein regions having a darkness lower than a darkness threshold and a redness higher than a redness threshold are removed from the input image (100,100').
10. The method according to any of the preceding claims, wherein said face detection step comprises the step of a face detection outputting one or more bounding boxes (BB) and of a skin detection outputting one or more skin regions (SR).
11. The method according to claim 10, wherein said skin detection step comprises the step of mapping said input image into YCbCr color space and identifying as skin pixels those pixels comprised within a Cr interval and a Cb interval.
12. The method according to any of the preceding claims, comprising the step of correcting original color values of pixels belonging to a region identified as a red eye region
(RE, RE2) by smoothing and desaturating their original colour values.
13. The method according to claim 12, wherein said desaturation step comprises the step of replacing the original RGB colors values of said pixels belonging to said red-eye region (RE, RE2) with new RGB color values depending on the original redness intensity values of the pixels of said red-eye region (RE, RE2).
14. The method according to any of the preceding claims, the method further comprising before said candidate red-eye regions (LRM) identification step, the step of - reducing the number of face regions (SR1BB) using a second set of one or more geometrical constraints.
15. The method according to any of the preceding claims, the method further comprising, before said face detection step, a step of rescaling said input image size to a standard size, so that the maximum dimension (Max Dim), which is the longest between height and length, of said rescaled image has a fixed value.
16. The method according to any of the preceding claims, wherein said redness mapping step is calculated by using the following redness values:
17. The method according to any of the preceding claims, wherein said purple-brown mapping step is computed using the following purple-brown values:
PurpleBrown(x, y) = Max[θ, B(x, y) - G(x, y)] .
18. The method according to any of the preceding claims comprising
- A counting step to count the number of candidate red-eye regions (LRM) identified in said identification step, and - candidate second red-eye region (RE2) identification step, said candidate second red eye region (RE) being portions of said face regions (SR1BB) having a computed redness average (R ) and a computed purple-brown average (PB ) which satisfy:
PB > third lower purple-brown limit & R > third lower redness limit or PB > fourth lower purple-brown limit & R > fourth lower redness limit, or
PB > fifth lower purple-brown limit <& R >fifth lower redness limit.
19. A method of identifying at least one face in a digital image (100,100') comprising the steps of: applying a face detector on the digital image (100,100') to obtain one or more face regions (BB); - applying a skin detector on the digital image (100,100') to obtain a first set of one or more candidate skin regions (SR), wherein said skin regions (SR) are defined as those pixels of said input image (100,100') having their colour components in a given colour space within given intervals;
- reducing the number of said skin regions (SR) using a second set of geometric constraints obtaining a second set of skin regions (GSFR), and
- considering as faces said face regions and said second set of skin regions.
20. The method according to claim 19, wherein said skin detection step comprises the step of mapping said input image (100,100') into YCbCr colour space and identifying as skin pixels those pixels comprised within a Cr interval and a Cb interval.
21. The method according to claim 19 or 20,further including the step of
- selecting a set of candidate red-eye regions (LRM) computing redness measures on pixels belonging to said face regions (BB) and to said skin regions (SR), so that said red-eye regions have a redness above a given threshold.
22. The method according to any of claims 19-21, comprising, before said reduction step, a step of removal from said candidate skin regions (SR) of said face regions (BB).
23. The method according to claims 19-22 comprising the steps of:
- calculating the sum of the areas of all skin regions (SR) found in said skin detection step,
- comparing said sum with a fixed value, - applying a colour balancing method to said input image (100, 100') in case said sum is smaller than said fixed value.
24. Printer apt to print digital input images (100,100') including one or more software programs operating according to the method of claims 1-18 and/or according to the method of claims 19-23.
EP07723937A 2007-04-03 2007-04-03 Method for red eye detection in a digital image Withdrawn EP2143069A1 (en)

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/EP2007/002996 WO2008119368A1 (en) 2007-04-03 2007-04-03 Method for red eye detection in a digital image

Publications (1)

Publication Number Publication Date
EP2143069A1 true EP2143069A1 (en) 2010-01-13

Family

ID=38828753

Family Applications (1)

Application Number Title Priority Date Filing Date
EP07723937A Withdrawn EP2143069A1 (en) 2007-04-03 2007-04-03 Method for red eye detection in a digital image

Country Status (2)

Country Link
EP (1) EP2143069A1 (en)
WO (1) WO2008119368A1 (en)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8170332B2 (en) 2009-10-07 2012-05-01 Seiko Epson Corporation Automatic red-eye object classification in digital images using a boosting-based framework
CN105247570A (en) * 2013-05-28 2016-01-13 马维尔国际贸易有限公司 Systems and methods for red-eye correction

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6016354A (en) * 1997-10-23 2000-01-18 Hewlett-Packard Company Apparatus and a method for reducing red-eye in a digital image
US7116820B2 (en) * 2003-04-28 2006-10-03 Hewlett-Packard Development Company, Lp. Detecting and correcting red-eye in a digital image
US7333653B2 (en) * 2003-08-29 2008-02-19 Hewlett-Packard Development Company, L.P. Detecting and correcting redeye in an image
JP4757559B2 (en) * 2004-08-11 2011-08-24 富士フイルム株式会社 Apparatus and method for detecting components of a subject
US8374403B2 (en) * 2005-05-16 2013-02-12 Cisco Technology, Inc. Methods and apparatus for efficient, automated red eye detection

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See references of WO2008119368A1 *

Also Published As

Publication number Publication date
WO2008119368A1 (en) 2008-10-09

Similar Documents

Publication Publication Date Title
US7953251B1 (en) Method and apparatus for detection and correction of flash-induced eye defects within digital images using preview or other reference images
US8254674B2 (en) Analyzing partial face regions for red-eye detection in acquired digital images
JP5547730B2 (en) Automatic facial and skin beautification using face detection
EP1807805B1 (en) Method and apparatus for red-eye detection in an acquired digital image
US8520089B2 (en) Eye beautification
US8331666B2 (en) Automatic red eye artifact reduction for images
US20060280363A1 (en) Image processing apparatus and method, computer program, and storage medium
US20170221186A1 (en) Device for and method of enhancing quality of an image
JP2007504719A (en) System and method for detecting and correcting red eyes in an image suitable for embedded applications
JP2002183729A (en) Blond-hair-pixel removing method in image skin-color detection
WO2008119368A1 (en) Method for red eye detection in a digital image
Schettini et al. A modular procedure for automatic red eye correction in digital photos
CN117522802A (en) Retina image preprocessing method and device, electronic equipment and storage medium
Safonov et al. Automatic Red Eye Correction
Kong et al. Adaptive image backlight compensation for mobile phone users

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

17P Request for examination filed

Effective date: 20091103

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HU IE IS IT LI LT LU LV MC MT NL PL PT RO SE SI SK TR

RAP1 Party data changed (applicant data changed or rights of an application transferred)

Owner name: UNIVERSITA DEGLI STUDI DI MILANO - BICOCCA

DAX Request for extension of the european patent (deleted)
17Q First examination report despatched

Effective date: 20130621

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE APPLICATION IS DEEMED TO BE WITHDRAWN

18D Application deemed to be withdrawn

Effective date: 20131101