AU759361B2 - Using eigenvalues and eigenvectors to determine an optimal resampling method for a transformed image - Google Patents

Description

S&F Ref: 541440

AUSTRALIA

PATENTS ACT 1990 COMPLETE SPECIFICATION FOR A STANDARD PATENT

ORIGINAL

Name and Address of Applicant: Actual Inventor(s): Address for Service: Canon Kabushiki Kaisha 30-2, Shimomaruko 3-chome, Ohta-ku Tokyo 146 Japan David Geoffrey Slack-Smith Spruson Ferguson St Martins Tower,Level 31 Market Street Sydney NSW 2000 Using Eigenvalues and Eigenvectors to Determine an Optimal Resampling Method for a Transformed Image Invention Title: ASSOCIATED PROVISIONAL APPLICATION DETAILS [33] Country [31] Applic. No(s) AU PQ5974 [32] Application Date 02 Mar 2000 The following statement is a full description of this invention, including the best method of performing it known to me/us:- IP Australia Documents received on: 2 MAR 2001 Batch No: 5815c 1 USING EIGENVALUES AND EIGENVECTORS TO DETERMINE AN OPTIMAL RESAMPLING METHOD FOR A TRANSFORMED IMAGE Technical Field of the Invention The present invention relates generally to the field of image processing and in particular, to a method and apparatus for efficient image resampling using eigenvalues and eigenvectors. The invention also relates to a computer program product including a computer readable medium having recorded thereon a computer program for efficient image resampling using eigenvalues and eigenvectors.

Background Art In the field of computer graphics, it is common to geometrically transform an image from one space to another. Monochrome or colour images are usually represented by a 2-dimensional array of pixels. Monochrome images require a single colour component per pixel to select a grey scale value and each pixel of a colour image is generally represented by a point in a 3-dimensional colour space, RGB (red, green, blue) being the system generally used by Cathode Ray Tube displays CRTs). One conventional method for rendering a transformed image [hereinafter called the "destination image"], is to use point-sampling. Point sampling approximates the colour of a pixel in a destination image using the value of the pixel closest to that pixel's centre's inverse-transformed location in the original image [hereinafter called the "source image"].

20 Other commonly used methods for rendering a transformed image include supersampling, bilinear interpolation, and/or a combination of supersampling and bilinear interpolation known as resampling.

Supersampling works by choosing a set of sample points centred around a destination pixel centre, inverse-transforming each sample point, and setting the destination pixel's colour to a weighted-sum of all of the inverse-transformed sample 541440 2 points' colour values. A commonly chosen set of points is a square grid of points within a current pixel, being equally spaced with equal weights.

Bilinear interpolation works by inverse-transforming the centre of each destination pixel and interpolating bilinearly between the four nearest source pixel centres by a weighted-sum where the weights depend upon the inverse-transformed point's distances from the surrounding pixel centres).

Resampling is a method that chooses a set of sample points centred around a destination pixel centre, inverse-transforms each sample point and for each sample point bilinearly interpolates between its nearest 4 source pixel centres to get a value for the sample point. Resampling then sets the destination pixel's colour to equal a weighted sum of the sample points' interpolated colour values.

Unfortunately, the above techniques have a number of disadvantages. Firstly, all of the methods, including point sampling, can cause visual artefacts in a resulting rendered image. In this connection, supersampling and bilinear interpolation set a 15 destination pixel to the weighted-sum of the source pixels in a neighbourhood around the oooo destination pixel's centre's inverse-transformed (or back-mapped) position. In practice, to obtain a rendered image relatively free of visual artefacts, substantially all of the source *'**.pixels covered by a back-mapped destination pixel should contribute to the value calculated for the destination pixel. As an example, Fig. 6 shows an image 1000 transformed by a matrix (not shown) using bilinear interpolation and Fig. 7 shows the •.image 1000 transformed by a matrix (not shown) using supersampling. In Figs. 6 and 7 source pixels 1007) are represented by phantom lines with a centre point represented by a dot, destination pixels 1005 and back mapped destination pixels 1003 are represented "by un-broken lines with a centre point. As seen in Figs. 6 and 7, when the image 1000 is expanded in one direction and compressed in another direction to produce a transformed image 1001, the back-mapped destination pixel 1003 is expanded and compressed in the 541440 3 opposite directions. The destination pixel 1005 and back-mapped destination pixel 1003 of Fig. 7 also include a 2 x 2 sampling grid with each block of the sampling grid having a centre point. As seen in Fig. 6, using bilinear interpolation with a weighted average of four pixels to calculate the value to be assigned to the destination pixel 1005, means that only the source pixels 1007, 1009, 1011 and 1013 contribute to the destination pixel 1005.

Therefore, there are source pixels at the outer extremes of the back-mapped destination pixel 1003 that will not contribute to the value calculated for the destination pixel 1005, resulting in visual artefacts in the rendered image 1001. Similarly, as seen in Fig. 7, when supersampling is used to calculate the value to be assigned to the destination pixel 1005, there are source pixels at one outer extreme of the back-mapped destination pixel 1003 that will not contribute to the value calculated for the destination pixel 1005. .Further, as seen in Fig. 7, when using supersampling a larger area of the back-mapped destination pixel 1003 has no source pixel contributing to the output destination pixel 1005, which will result in visual artefacts in the rendered image.

Secondly, supersampling and bilinear interpolation generally require relatively oooo more work processing and memory access) than point-sampling does. In particular, a S"weighted sum calculation generally requires component-wise multiplication and addition.

For colour images, supersampling and bilinear interpolation can require tripling (or quadrupling, with transparency) the number of additions and multiplications required to calculate a weighted-sum. Thirdly, point-sampling results in visual artefacts for almost any transformation applied to an image. Fourthly, supersampling is inappropriate for image enlargement. Finally, bilinear interpolation is inappropriate for image reduction.

In addition, point sampling requires one source pixel to be read in order to .o generate a destination pixel. Supersampling requires a source pixel to be read for every sample point (of which there are usually at least 4) and a weighted-sum to be calculated per-component per-pixel. Bilinear interpolation requires that 4 pixels be read and bilinear 541440 4 interpolation between colour values performed on a per-component per-pixel basis.

Resampling requires that 4 pixels be read and bilinear interpolation be performed for every sample point followed by a weighted-sum per-component per-pixel. Therefore, additional pixel reads are required as the complexity of the sampling method increases.

There are other known higher-order, interpolation methods that can be used to render a transformed image. However, the higher-order methods can require even more work than the above methods. For example, bicubic interpolation would need to read 16 source pixels per destination pixel.

In the case of an affine transformation, or any transformation that can be locally approximated by an affine transformation, one known method of filtering an image is to determine the correct sampling method to use based on properties of the source matrix.

In accordance with this affine transformation method, the absolute value of the determinant of the source matrix can be compared to one. If the absolute value is less than one, supersampling can be used on the reduction, and bilinear interpolation can be 15 used otherwise. However, the affine transformation method still does not cope with all .oo o S• "possible cases. For example, transformations that expand a source image in one direction and compress the source image in another direction will not be dealt with in an optimal manner.

Disclosure of the Invention It is an object of the present invention to substantially overcome, or at least ameliorate, one or more disadvantages of existing arrangements.

According to one aspect of the present invention there is provided a method of :filtering a first image to generate a second image using one of a plurality of sampling processes, said first image and second image being related by an affine transformation, said affine transformation comprising a linear transformation component, said method 541440 characterised in that said one sampling process is selected depending upon at least one eigenvalue of the linear transformation component.

According to another aspect of the present invention there is provided a method of filtering image data, said method including the steps of: accessing a first set of discrete sample values of said image data; and calculating a second set of discrete sample values using one of a plurality of filtering processes selected depending on the eigenvalues of a linear transformation component relating said image data.

According to still another aspect of the present invention there is provided a method of filtering a first image to generate a second image using one of a plurality of sampling methods, wherein said first image and second image are related by an affine transformation comprising a linear transformation component, said method comprising the steps of: if the eigenvalues of said linear transformation component are real, selecting said sampling process depending on the magnitudes of said -eigenvalues of said linear transformation component; and filtering said first image using said selected sampling process to :generate said second image; otherwise, representing the linear transformation component as a matrix and decomposing said matrix into a lower triangular matrix and a rotation matrix; selecting said sampling process depending on the magnitudes of the eigenvalues of said triangular matrix; and filtering said first image using said selected sampling process to generate said second image.

541440 6- According to still another aspect of the present invention there is provided a method of filtering a first image to generate a second image, said first image and second image being related by a non-affine transformation, said transformation being approximated by a plurality of affine transformations over different regions of the image, each affine transformation comprising a linear transformation component, said method characterised in that one of a plurality of sampling processes is selected for each region depending on at least one eigenvalue of said region's corresponding said linear transformation component.

According to still another aspect of the present invention there is provided an apparatus for filtering a first image to generate a second image using one of a plurality of sampling processes, said first image and second image being related by an affine transformation, said affine transformation comprising a linear transformation component, said apparatus being characterised in that said apparatus comprises a means for selecting said one sampling process depending upon at least one eigenvalue of the linear transformation component.

According to still another aspect of the present invention there is provided an S: apparatus for filtering image data, said apparatus comprising: access mean for accessing a first set of discrete sample values of said image data; and calculation means for calculating a second set of discrete sample values using of a plurality of filtering processes selected depending on the eigenvalues of a linear transformation component relating said image data.

According to still another aspect of the present invention there is provided an apparatus for filtering a first image to generate a second image using one of a plurality of sampling methods, wherein said first image and second image are related by an affine 541440 7 transformation comprising a linear transformation component, said apparatus comprising: means for selecting said sampling process depending on the magnitudes of the eigenvalues of said linear transformation component, and filtering said first image using said selected sampling process to generate said second image if the eigenvalues of said linear transformation component are real, otherwise representing the linear transformation component as a matrix and decomposing said matrix into a lower triangular matrix and a rotation matrix; selecting said sampling process depending on the magnitudes of the eigenvalues of said triangular matrix; and filtering said first image using said selected sampling process to generate said second image.

According to still another aspect of the present invention there is provided an apparatus for filtering a first image to generate a second image, said first image and second image being related by a non-affine transformation, said transformation being approximated by a plurality of affine transformations over different regions of the image, "each affine transformation comprising a linear transformation component, said apparatus being characterised in that said apparatus comprises selection means for selecting one of a plurality of sampling processes for each region, depending on at least one eigenvalue of 20 said region's corresponding linear transformation component.

According to still another aspect of the present invention there is provided a computer readable medium, having a computer program recorded thereon, wherein the program comprises code to perform a method of filtering a first image to generate a second image using one of a plurality of sampling processes, said first image and second image being related by an affine transformation, said affine transformation comprising a linear transformation component, said method being characterised in that said one 541440 8 sampling process is selected depending upon at least one eigenvalue of the linear transformation component.

According to still another aspect of the present invention there is provided a computer readable medium, having a computer program recorded thereon, wherein the program is configured to perform a method of filtering image data, said program comprising: code for accessing a first set of discrete sample values of said image data; and code for calculating a second set of discrete sample values using one of a plurality of filtering processes selected depending on the eigenvalues of a linear transformation component relating said image data.

According to still another aspect of the present invention there is provided a computer readable medium, having a computer program recorded thereon, wherein the program is configured to perform a method of filtering a first image to generate a second image using one of a plurality of sampling methods, wherein said first image and second image are related by an affine transformation comprising a linear transformation 4 component, said program comprising: if the eigenvalues of said linear transformation component are real, code for selecting said sampling process depending on the magnitudes of said eigenvalues of said linear transformation component; and oooo9* S. 20 code for filtering said first image using said selected sampling process 9*99 to generate said second image; .9 99• 9 otherwise, code for representing the linear transformation component as a matrix 9.

9* 9 and decomposing said matrix into a lower triangular matrix and a rotation matrix; 541440 9 code for selecting said sampling process depending on the magnitudes of the eigenvalues of said triangular matrix; and code for filtering said first image using said selected sampling process to generate said second image.

Brief Description of the Drawings A number of preferred embodiments of the present invention will now be described with reference to the drawings, in which: Figs. 1 and show a flow diagram of the method of of filtering an image in accordance with the preferred embodiment; Fig. 2 shows a flow diagram of the method of supersampling in accordance with S the preferred embodiment; Fig. 3 shows a flow diagram of the method of resampling in accordance with the preferred embodiment; and Fig. 4 shows an image reduced and rotated using a prior art supersampling method utilising a 3 x 3 sampling grid; Fig. 5 shows an image reduced and rotated using a prior art supersampling :o of method utilising a 2 x 2 sampling grid; Fig. 6 shows an image transformed by a matrix using a prior art bilinear S interpolation sampling method; Fig. 7 shows an image transformed by a matrix using a prior art supersampling method utilising a 2 x 2 sampling grid; Fig. 8 shows an image transformed using resampling in accordance with the preferred embodiment; Fig. 9 is a schematic block diagram of a general purpose computer upon which the preferred embodiment of the present invention can be practiced.

Detailed Description including Best Mode 541440 Where reference is made in any one or more of the accompanying drawings to steps and/or features, which have the same reference numerals, those steps and/or features have for the purposes of this description the same function(s) or operation(s), unless the contrary intention appears.

Some portions of the detailed description which follow are explicitly or implicitly presented in terms of algorithms and symbolic representations of operations on data within a computer memory computer code). These algorithmic descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. An algorithm is here, and generally, conceived to be a self-consistent sequence of steps leading to a desired result. The steps are those requiring physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated. It has proven convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or "the like.

.It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities. Unless specifically stated otherwise e as apparent from the following discussions, it is appreciated that throughout the present 20 description, discussions utilising terms such as "processing", "computing", "generating", 0 "creating", "operating", "communicating", "rendering", "providing", and "linking" or the e e like, refer to the action and processes of a computer system, or similar electronic device, *o that manipulates and transforms data represented as physical (electronic) quantities within :the computer system's registers and memories into other data similarly represented as physical quantities within the computer system memories or registers or other such information storage, transmission or display devices.

541440 11 The preferred embodiment is a method of filtering an image that takes advantage of knowledge of the local properties of the transformation that is being applied to the image in order to produce the best approximation of the image in an efficient manner. In accordance with the preferred embodiment, the affine transformation's matrix is used to determine the best sampling method to use on a transformed image. The preferred method uses the eigenvalues and eigenvectors of the linear transformation's matrix to determine the best sampling method to use on the transformed image; or decomposes the matrix into a triangular matrix and a rotation matrix and uses the triangular matrix's eigenvalues and the corresponding eigenvectors, post-multiplied by the rotation matrix, to determine the best sampling method to use on the transformed image.

An affine transformation can be decomposed into a linear transformation composed with a translation. The linear transformation can be represented by a matrix, component or operator. A linear transformation can be decomposed into a scale, a horizontal shear, and a rotation. If the sample points being transformed are represented by row vectors, then the shear matrix (and the matrix representing scale followed by shear) is a lower triangular matrix a matrix with only zero entries above the main diagonal). Further, every matrix A has certain eigenvalues ?i and corresponding eigenvectors vi such that viA Xivi. The eigenvalues of a matrix can be found as the solutions of the equation det(A XiI) 0 known as the characteristic equation. In the *20 case of a 2 x 2 matrix, the characteristic equation will be a quadratic.

Fig. 1 is a flowchart showing a method of filtering an image, in accordance with the preferred embodiment. The process begins at step 101, where given an affine transformation to be applied to the image, it is determined whether or not the affine transformation represents a translation by an integral number of pixels. If the affine transformation represents a translation by an integral number of pixels then point 541440 12 sampling is used to render the transformed image, at step 103. In carrying out the rendering process, point-sampling can be used to copy the source buffer or alternatively, the destination buffer's pixel data can be made a reference to the source buffer's pixel data, rather than a copy. If the affine transformation does not represent a translation by an integral number of pixels, at step 101, the process proceeds to step 105, where the discriminant of the affine transformation matrix's characteristic equation is calculated. At the next step 107, if the discriminant is less than zero the eigenvalues of the transformation matrix are complex conjugates) the process proceeds to step 109. At step 109, the transformation matrix is decomposed into the product of a lower triangular matrix representing a scale followed by a shear and an orthogonal matrix representing a rotation, such that transforming any sample point with the transformation matrix is equivalent to transforming it with the triangular matrix and then rotating it with the rotation matrix. The eigenvalues for the lower triangular matrix are also determined at step 109 using the entries along the triangular matrix's main diagonal. The eigenvalues for the lower triangular matrix are then used to classify the transformation matrix as a pure reduction, a pure non-reduction, or a mixed reduction.

o oo At the next step 111, if both eigenvalues have magnitudes less than 1 the transformation matrix is classified as a pure reduction), then the process proceeds to step 113, where supersampling is used to render the transformed image. The smaller of the ooooi 20 two calculated eigenvalues is used in the supersampling process and is indicative of the necessary size of the sampling grid to be used in the supersampling process. The supersampling process in accordance with the preferred embodiment will be explained in more detail later in this document. If the magnitudes of the eigenvalues are not less than o one at step 111, the process proceeds to step 115, where if the eigenvalues have magnitudes greater than or equal to one the transformation matrix is classified as a pure non-reduction) the process proceeds to step 117. At step 117, bilinear interpolation 541440 13 is used to render the transformed image. If the magnitudes of the eigenvalues are not greater than or equal to one at step 115 the transformation matrix is classified as a mixed reduction), the process proceeds to step 119, where the eigenvector corresponding to the smaller of the two eigenvalues is calculated. At the next step 121, the eigenvector is transformed, using the rotation matrix determined at step 109, to give a vector indicative of the direction along which sample points could best be chosen. Points along the line passing through the centre of the pixel in the direction of the eigenvector could be chosen or sample points could be chosen from a lattice based on their proximity to this line. The eigenvector is also normalised at step 121. At the next step 123, resampling is used to render the transformed image. The affine transformation, the smaller eigenvalue and the normalised rotated eigenvector are used in the resampling process in accordance with the preferred embodiment. The magnitude of the smaller eigenvalue is indicative of the necessary number of sample points needed to be used in the resampling process. The resampling process in accordance with the preferred embodiment will be described in more detail later in this document.

If, at step 107, the discriminant is not less than zero, the process proceeds to step 125, where if the discriminant is equal to zero both the transformation matrix's eigenvalues are the same), the process proceeds to step 127. At step 127, the absolute value of the transformation matrix's determinant is calculated and can be used to distinguish between reduction and non-reduction. Reductions will be supersampled, nonreductions will be bilinearly interpolated. The process continues to step 129, where if the absolute value of the transformation matrix is less than one the transformation matrix is classified as a reduction), supersampling is used to render the transformed image, at step 113. Otherwise, the process proceeds to step 117 the transformation matrix is classified as a non-reduction), where bilinear interpolation is used to render the transformed image.

541440 14 If the discriminant is positive, then the transformation matrix has a pair of distinct real eigenvalues. The eigenvalues are calculated and used to classify the transformation as a pure reduction, a pure non-reduction, or a mixed reduction. Therefore, at step 125, if the discriminant is not equal to zero, the process proceeds to step 135, where the eigenvalues of the transformation matrix are calculated. At the next step 137, if both eigenvalues have magnitudes of less than one the transformation matrix is classified as a pure reduction) then the process proceeds to step 113, where supersampling is used to render the transformed image. Otherwise, the process proceeds to step 141, where if both eigenvalues have magnitudes greater than or equal to one the transformation matrix is classified as a pure non-reduction), then the process proceeds to step 117. At step 117, bilinear interpolation is used to render the transformed image. If at step 141, the magnitudes of the eigenvalues are not greater than or equal to one the transformation matrix is classified as a mixed reduction), the process proceeds to step 145, where the eigenvector corresponding to the smaller of the two eigenvalues is calculated and normalised. At the next step 123, resampling is used to render the transformed image. Again, the affine transformation, the smaller eigenvalue and the normalised eigenvector are used in the resampling process in accordance with the preferred embodiment. The method uses the smaller eigenvalue to determine the number of samples, and the corresponding normalised eigenvector to determine the positions of 20 the samples. Fig. 8 shows an image 800 transformed using resampling, in accordance with the preferred embodiment, using two sample points 803, 805 for a destination pixel 809. As with Figs. 4 to 7, the source pixels 811) which contribute to the destination pixel 809 are shown in phantom lines with an illustrated centre point. The resulting transformed image 807 is compressed in the direction of the eigenvector corresponding to the smaller eigenvalue, and expanded in the direction of the other eigenvector. As seen in Fig. 8, many more source pixels, 811 to 818, contribute to the calculation of the 541440 destination pixel 809 than with the bilinear interpolation and supersampling methods seen in Figs. 6 and 7, respectively. The process concludes at step 149.

Fig. 2 is a flowchart showing the method of supersampling, in accordance with the preferred embodiment. The process begins at step 201, where if the smallest eigenvalue calculated at steps 109 or 135, is less than the Sample Level Threshold, the process proceeds to step 203. The Sample Level Threshold, in accordance with the preferred embodiment, may be approximately equal to V 2 when choosing between a 2 x 2 sampling grid and a 3 x 3 sampling grid. At step 203, the transformed image is super sampled using a 3 x 3 supersampling grid 401, as seen in Fig. 4. Again, source pixels are shown in Fig. 4 as phantom lines with each source pixel having an illustrated centre point.

The sampling grid is shown superimposed on the destination pixel 403. All subpixels of the supersampling grid are of equal area 1/9 of the pixel size). If the smallest eigenvalue is not less than the Sample Level Threshold at step 201, the process proceeds to step 205, where the transformed image is super sampled using a 2 x 2 supersampling grid 501, as seen in Fig. 5. Again, all subpixels of the supersampling grid are of equal i area 1/4 of the pixel size). As seen in Figs. 4 and 5, more source pixels (shown in .phantom lines) covered by a back-mapped destination pixel will often contribute to the value of the output destination pixel when the 3 x 3 sampling grid 401 is used than with the 2 x 2 sampling grid 501. The supersampling process concludes at step 207. Although, *oooo 20 the preferred embodiment selects between a 2 x 2 and 3 x 3 sampling grid, it will be appreciated by those skilled in the art that the preferred method could be extended to o°* select between more n x n grids. In this instance, other values can be chosen for the sample level threshold producing different results.

3 is a flowchart showing the method of resampling, in accordance with the preferred embodiment. The process begins at step 301, where if the smallest eigenvalue calculated at steps 109 or 135 is less than the sample level threshold, the process proceeds 541440 16 to step 303. At step 303, the back-mapped sample points si of the input sample points are calculated for every destination pixel. The back-mapped sample points si are calculated by solving the equation Asi di, where the sample points are d- 1 x v/3, do x, dl x v/3, where x represents the pixel centre. As previously described, the resampling process uses the smaller eigenvalue and the corresponding normalised (possibly rotated) eigenvector v. The destination pixel is set to the weighted average of the sample point's values with weights 1/4, wo= 1/2 and w /4 at step 307.

If the smallest eigenvalue is not less than the Sample Level Threshold at step 301, the process proceeds to step 305, where the back-mapped sample points si of the input sample points are calculated for every destination pixel, where the sample points are do= x v/4, dl x v/4, where x represents the pixel centre. The destination pixel is set to the average of the sample points with weights wo= 1/2 and wl= V2 at step 308. The process concludes at step 309.

An algorithm suitable for implementing the preferred method, as seen in Fig. 1, is presented below as pseudocode: if affine transformation is translation by integral number of pixels :then Point-sample.

else Calculate discriminant of affine transformation matrix's characteristic equation.

if discriminant is less than 0 then Decompose matrix into the product of a lower triangular matrix and a rotation matrix.

Take eigenvalues from diagonal of lower triangular matrix if both eigenvalues have magnitudes less than 1 then 541440 17 call supersample, passing the smaller eigenvalue.

else if both eigenvalues have magnitudes greater than or equal tol then Bilinearly interpolate else Calculate smaller eigenvalue's corresponding eigenvector.

Transform this eigenvector by the rotation matrix.

Normalise eigenvector.

call resample, passing the affine transformation, the smaller eigenvalue and the normalised rotated eigenvector.

end if else if discriminant equals 0 then if absolute value of determinant of matrix is less than 1 then call supersample, passing the square root of the magnitude of the S* determinant.

else Bilinearly interpolate.

20 end if else Calculate eigenvalues of matrix.

if both eigenvalues have magnitudes less than 1 then call supersample, passing the smaller eigenvalue.

else if both eigenvalues have magnitudes greater than or equal to 1 541440 18 then Bilinearly interpolate else Calculate smaller eigenvalue's corresponding eigenvector.

Normalise eigenvector.

call resample, passing the affine transformation, the smaller eigenvalue and the normalised eigenvector.

end if end if end if An algorithm suitable for implementing the preferred supersampling method, as seen in Fig. 2, is presented below as pseudo computer code: procedure supersample(destination buffer, source buffer, A, X) begin if X< SAMPLE LEVELTHRESHOLD then 0 Supersample with a supersampling grid (all subpixels of equal area (1/9 of the pixel)).

else 20 Supersample with a supersampling grid (all subpixels of equal area (1/4 of the pixel)).

if end An algorithm suitable for implementing the preferred resampling method, as seen in Fig. 3, is presented below as pseudocode: 541440 19 procedure resample(destination buffer, source buffer, A, k, v) begin if k< SAMPLELEVELTHRESHOLD then for every destination pixel do Let the weights be 1/4, wo= 1/2, wl= 1/4..

Set the sample points to d-i x v/3, do x d.

1 x v/3 where x is the pixel centre.

Inverse transform the sample points by solving Asi di.

10 Set the destination pixel's value to the weighted average of the samples.

end for else for every destination pixel do Set the sample points to di x v/4, where x is the pixel centre.

15 Inverse transform the sample points by solving Asi di.

Set the destination pixel's value to the average of the samples.

end for end if end In accordance with a second embodiment of the present invention, the method of filtering an image can be extended to deal with general (non-affine) mappings, by calculating the Jacobian matrix of the mapping per-pixel and then calculating the translational part of the nearest affine map. The translational part of the nearest affine map is calculated by calculating the difference of the destination pixel centre's position 541440 and the inverse-mapped point transformed by the Jacobian matrix. The nearest affine mapping is then defined by the Jacobian matrix composed with a translation by the difference of the destination pixel centre's position and the inverse-mapped point transformed by the Jacobian matrix. In accordance with the second embodiment, the Jacobian matrix does not have to be recalculated per-pixel, but can be calculated more frequently in areas of high rate of change of Jacobian.

The method of Fig. 1 is preferably practiced using a conventional generalpurpose computer system 900, such as that shown in Fig. 9 wherein the processes of Fig. 1 can be implemented as software, such as an application program executing within the computer system 900. In particular, the steps of the method of filtering an image are effected by instructions in the software that are carried out by the computer. The software may be divided into two separate parts; one part for carrying out the preferred method; and another part to manage the user interface between the latter and the user. The software may be stored in a computer readable medium, including the storage devices described below, for example. The software is loaded into the computer from the computer readable medium, and then executed by the computer. A computer readable medium having such software or computer program recorded on it is a computer program e.

9*o* product. The use of the computer program product in the computer preferably effects an advantageous apparatus for filtering an image in accordance with the embodiments of the oo.j invention.

The computer system 900 comprises a computer module 901, input devices such o ooo-as a keyboard 902 and mouse 903, output devices including a printer 915 and a display o oo device 914. A Modulator-Demodulator (Modem) transceiver device 916 is used by the oo o computer module 901 for communicating to and from a communications network 920, for example connectable via a telephone line 921 or other functional medium. The 541440 21 modem 916 can be used to obtain access to the Internet, and other network systems, such as a Local Area Network (LAN) or a Wide Area Network (WAN).

The computer module 901 typically includes at least one processor unit 905, a memory unit 906, for example formed from semiconductor random access memory (RAM) and read only memory (ROM), input/output interfaces including a video interface 907, and an 1/O interface 913 for the keyboard902 and mouse 903 and optionally a joystick (not illustrated), and an interface 908 for the modem 916. A storage device 909 is provided and typically includes a hard disk drive 910 and a floppy disk drive 911. A magnetic tape drive (not illustrated) may also be used. A CD-ROM drive 912 is typically provided as a non-volatile source of data. The components 905 to 913 of the computer module 901, typically communicate via an interconnected bus 904 and in a manner which results in a conventional mode of operation of the computer system 900 known to those in the relevant art. Examples of computers on which the embodiments can be practised include IBM-PC's and compatibles, Sun Sparcstations or alike computer systems evolved therefrom.

0 Safe Typically, the application program of the preferred embodiment is resident on 0 SO the hard disk drive 910 and read and controlled in its execution by the processor 905.

Intermediate storage of the program and any data fetched from the network 920 may be accomplished using the semiconductor memory 906, possibly in concert with the hard 559505 OO So I.°o 20 disk drive 910. In some instances, the application program may be supplied to the user fo* encoded on a CD-ROM or floppy disk and read via the corresponding drive 912 or 911, 05 S or alternatively may be read by the user from the network 920 via the modem device 916.

Still further, the software can also be loaded into the computer system 900 from other r, a'* computer readable medium including magnetic tape, a ROM or integrated circuit, a magneto-optical disk, a radio or infra-red transmission channel between the computer module 901 and another device, a computer readable card such as a PCMCIA card, and 541440 22 the Internet and Intranets including email transmissions and information recorded on websites and the like. The foregoing is merely exemplary of relevant computer readable mediums. Other computer readable mediums may be practiced without departing from the scope and spirit of the invention.

The method of filtering an image may alternatively be implemented in dedicated hardware such as one or more integrated circuits performing the functions or sub functions of Fig. 1. Such dedicated hardware may include graphic processors, digital signal processors, or one or more microprocessors and associated memories.

The aforementioned preferred method comprises a particular control flow. There are many other variants of the preferred methods which use different control flows without departing from the spirit or scope of the invention. Furthermore one or more of the steps of the preferred methods may be performed in parallel rather than sequentially.

The foregoing describes only some embodiments of the present invention, and modifications and/or changes can be made thereto without departing from the scope and spirit of the invention, the embodiments being illustrative and not restrictive. For example, a linear transformation component or operator need not be represented in a matrix representation as described above, to be utilised in accordance with the preferred embodiment.

In the context of this specification, the word "comprising" means "including ooo.

S" 20 principally but not necessarily solely" or "having" or "including" and not "consisting only of'. Variations of the word comprising, such as "comprise" and "comprises" have cd m corresponding meanings.

541440

Claims (87)

1. A method of filtering a first image to generate a second image using one of a plurality of sampling processes, said first image and second image being related by an affine transformation, said affine transformation comprising a linear transformation component, said method characterised in that said one sampling process is selected depending upon one or more eigenvalues of the linear transformation component.

2. The method according to claim 1, wherein said eigenvalues are used to select from supersampling, bilinear interpolation, and resampling as the said one sampling process.

3. The method according to claim 1, wherein said sampling process comprises supersampling if the magnitudes of all eigenvalues of said linear transformation component are less than a predetermined number.

4. The method according to claim 1, wherein said sampling process comprises bilinear interpolation if the magnitudes of all eigenvalues of said linear transformation component are greater than or equal to a predetermined number.

The method according to claim 1, wherein said sampling process comprises resampling if the magnitude of only one of the eigenvalues of said linear transformation component is less than a predetermined number. 7 25

6. The method according to any one of claims 3 to 5, wherein said predetermined S\ number is equal to one. 541440 24

7. The method according to claim 1, wherein said associated linear transformation component is represented by a matrix and decomposed into a lower triangular matrix and a rotation matrix if said eigenvalues are complex.

8. The method according to claim 7, wherein said sampling process is selected depending on the eigenvalues of said triangular matrix.

9. The method according to claim 8, wherein said sampling process comprises supersampling if the magnitudes of the eigenvalues of said triangular matrix are less than a predetermined number.

The method according to claim 8, wherein said sampling process comprises bilinear interpolation if the magnitudes of the eigenvalues of said triangular matrix are greater than or equal to a predetermined number.

11. The method according to claim 8, wherein said sampling process comprises resampling if the magnitude of only one of the eigenvalues of said triangular matrix is less than a predetermined number.

12. The method according to any one of claims 9 to 11, wherein said predetermined number is equal to one.

13. The method according to any one of claims 1 to 6, wherein the structure of a 25 sampling grid used to filter said image, is determined depending on the eigenvalues of said linear transformation component. said linear transformation component. 541440

14. The method according to any one of claims 7 to 12, wherein the structure of a sampling grid used to filter said image, is determined depending on the eigenvalues of said triangular matrix.

The method according to any one of claims 1 to 6 or 13, wherein the structure of a sampling grid used to filter said image, is determined depending on the eigenvalues and corresponding eigenvectors of said linear transformation component.

16. The method according to any one of claims 7 to 12 or 14, wherein the structure of a sampling grid used to filter said image, is determined depending on the eigenvalues and corresponding eigenvectors of said triangular matrix.

17. The method according to any one of claims 7 to 12, 14 or 16, wherein said eigenvectors are transformed using said rotation matrix.

18. The method according to claims 15, 16 or 17, wherein points selected for said grid are on a line passing through a pixel centre in a direction of one of said eigenvectors or, where relevant, rotated eigenvectors.

19. The method according to claims 15, 16 or 17, wherein points selected for said grid are in close proximity to a line passing through a pixel centre in a direction of one of said eigenvectors or, where relevant, rotated eigenvectors. 25

20. A method of filtering image data, said method including the steps of: accessing a first set of discrete sample values of said image data; and 541440 26 calculating a second set of discrete sample values using one of a plurality of filtering processes selected depending on the eigenvalues of a linear transformation component relating said image data.

21. The method according to claim 20, wherein said eigenvalues are used to select from supersampling, bilinear interpolation, and resampling as the said one sampling process.

22. The method according to claim 20, wherein said filtering process comprises supersampling if the magnitudes of all eigenvalues of said linear transformation component are less than a predetermined number.

23. The method according to claim 20, wherein said filtering process comprises bilinear interpolation if the magnitudes of all eigenvalues of said linear transformation component are greater than or equal to a predetermined number.

24. The method according to claim 20, wherein said filtering process comprises S resampling if the magnitude of only one of the eigenvalues of said linear transformation component is less than a predetermined number.

The method according to any one of claims 22 to 24, wherein said predetermined number is equal to one.

26. The method according to claim 20, wherein said linear transformation component is represented by a matrix and is decomposed into a lower triangular matrix \and a rotation matrix if said eigenvalues are complex. 541440 27

27. The method according to claim 26, wherein said filtering process is selected depending on the eigenvalues of said triangular matrix.

28. The method according to claim 27, wherein said filtering process comprises supersampling if the magnitudes of the eigenvalues of said triangular matrix are less than a predetermined number.

29. The method according to claim 27, wherein said filtering process comprises bilinear interpolation if the magnitudes of the eigenvalues of said triangular matrix are greater than or equal to a predetermined number.

The method according to claim 27, wherein said filtering process comprises resampling if the magnitude of only one of the eigenvalues of said triangular matrix is less than a predetermined number.

31. The method according to any one of claims 28 to 30, wherein said predetermined number is equal to one.

32. The method according to any one of claims 20 to 25, wherein the structure of a sampling grid used to filter said image, is determined depending on the eigenvalues of said linear transformation component.

33. The method according to any one of claims 26 to 31, wherein the structure of a sampling grid used to filter said image, is determined depending on the eigenvalues of said triangular matrix. 541440 28

34. The method according to any one of claims 20 to 25 or 32, wherein the structure of a sampling grid used to filter said image, is determined depending on the eigenvalues and corresponding eigenvectors of said linear transformation component.

The method according to any one of claims 26 to 31 or 33, wherein the structure of a sampling grid used to filter said image, is determined depending on the eigenvalues and corresponding eigenvectors of said triangular matrix.

36. The method according to claims 26 to 31, 33 or 35, wherein said eigenvectors are transformed using said rotation matrix.

37. The method according to claims 34, 35 or 36, wherein points selected for said grid are on a line passing through a pixel centre in a direction of one of said eigenvectors or, where relevant, rotated eigenvectors.

38. The method according to claims 34, 35 or 36, wherein points selected for said grid are in close proximity to a line passing through a pixel centre in a direction of one of said eigenvectors or, where relevant, rotated eigenvectors.

39. A method of filtering a first image to generate a second image using one of a plurality of sampling methods, wherein said first image and second image are related by *e an affine transformation comprising a linear transformation component, said method comprising the steps of: 25 if the eigenvalues of said linear transformation component are real, 541440 29 selecting said sampling process depending on the magnitudes of said eigenvalues of said linear transformation component; and filtering said first image using said selected sampling process to generate said second image; otherwise, representing the linear transformation component as a matrix and decomposing said matrix into a lower triangular matrix and a rotation matrix; selecting said sampling process depending on the magnitudes of the eigenvalues of said triangular matrix; and filtering said first image using said selected sampling process to generate said second image.

The method according to claim 39, wherein said eigenvalues are used to select from supersampling, bilinear interpolation, and resampling as the said one sampling process.

41. The method according to claim 39, wherein said sampling process comprises supersampling if the magnitudes of the eigenvalues of said linear transformation component or said triangular matrix are less than a predetermined number. S

42. The method according to claim 39, wherein said sampling process comprises bilinear interpolation if the magnitudes of eigenvalues of said linear transformation S component or said triangular matrix are greater than or equal to a predetermined number. o• 541440

43. The method according to claim 39, wherein said sampling process comprises resampling if the magnitude of only one of the eigenvalues of said linear transformation component or said triangular matrix are less than a predetermined number.

44. The method according to any one of claims 41 to 43, wherein said predetermined number is equal to one.

The method according to any one of claims 39 to 44, wherein the structure of a sampling grid used to filter said first image, is determined depending on the eigenvalues of said linear transformation component.

46. The method according to any one of claims 39 to 44, wherein the structure of a sampling grid used to filter said first image, is determined depending on the eigenvalues and corresponding eigenvectors of said linear transformation component or said triangular 15 matrix.

47. The method according to claims 45 or 46, wherein points selected for said grid are on a line passing through a pixel centre in a direction of one of said eigenvectors. 20o

48. The method according to claims 45 or 46, wherein points selected for said grid 090;:- are in close proximity to a line passing through a pixel centre in a direction of one of said eigenvectors.

49. A method of filtering a first image to generate a second image, said first image and second image being related by a non-affine transformation, said transformation being approximated by a plurality of affine transformations over different regions of the image, 541440 31 each affine transformation comprising a linear transformation component, said method characterised in that one of a plurality of sampling processes is selected for each region depending on at least one eigenvalue of said region's corresponding said linear transformation component.

An apparatus for filtering a first image to generate a second image using one of a plurality of sampling processes, said first image and second image being related by an affine transformation, said affine transformation comprising a linear transformation component, said apparatus being characterised in that said apparatus comprises a means for selecting said one sampling process depending upon at least one eigenvalue of the linear transformation component.

51. The apparatus according to claim 50, wherein said eigenvalues are used to select from supersampling, bilinear interpolation, and resampling as the said one sampling 15 process.

52. The apparatus according to claim 50, wherein said sampling process comprises supersampling if the magnitudes of all eigenvalues of said linear transformation .o*oo component are less than a predetermined number.

53. The apparatus according to claim 50, wherein said sampling process comprises bilinear interpolation if the magnitudes of all eigenvalues of said linear transformation S"component are greater than or equal to a predetermined number. 541440 32

54. The apparatus according to claim 50, wherein said sampling process comprises resampling if the magnitude of only one of the eigenvalues of said linear transformation component is less than a predetermined number.

55. The apparatus according to any one of claims 52 to 54, wherein said predetermined number is equal to one.

56. The apparatus according to claim 50, wherein said associated linear transformation component is represented by a matrix and decomposed into a lower triangular matrix and a rotation matrix if said eigenvalues are complex.

57. The apparatus according to claim 56, wherein said sampling process is selected depending on the eigenvalues of said triangular matrix. oooo

58. The apparatus according to claim 57, wherein said sampling process comprises S""supersampling if the magnitudes of the eigenvalues of said triangular matrix are less than a predetermined number. o•.

59. The apparatus according to claim 57, wherein said sampling process comprises bilinear interpolation if the magnitudes of the eigenvalues of said triangular matrix are S greater than or equal to a predetermined number. .i

60. The apparatus according to claim 57, wherein said sampling process comprises resampling if the magnitude of only one of the eigenvalues of said triangular matrix is less than a predetermined number. 541440 33

61. The apparatus according to any one of claims 58 to 60, wherein said predetermined number is equal to one.

62. The apparatus according to any one of claims 50 to 55, wherein the structure of a sampling grid used to filter said image, is determined depending on the eigenvalues of said linear transformation component.

63. The apparatus according to any one of claims 56 to 61, wherein the structure of a sampling grid used to filter said image, is determined depending on the eigenvalues of said triangular matrix.

64. The apparatus according to any one of claims 50 to 55 or 62, wherein the structure of a sampling grid used to filter said image, is determined depending on the eigenvalues and corresponding eigenvectors of said linear transformation component.

65. The apparatus according to any one of claims 56 to 61 or 63, wherein the structure of a sampling grid used to filter said image, is determined depending on the eigenvalues and corresponding eigenvectors of said triangular matrix.

66. The apparatus according to any one of claims 56 to 61, 63 or 65, wherein said eigenvectors are transformed using said rotation matrix.

67. The apparatus according to claims 64, 65 or 66, wherein points selected for said grid are on a line passing through a pixel centre in a direction of one of said eigenvectors or, where relevant, rotated eigenvectors. 541440 -34

68. The apparatus according to claims 64, 65 or 66, wherein points selected for said grid are in close proximity to a line passing through a pixel centre in a direction of one of said eigenvectors or, where relevant, rotated eigenvectors.

69. An apparatus for filtering image data, said apparatus comprising: access mean for accessing a first set of discrete sample values of said image data; and calculation means for calculating a second set of discrete sample values using one of a plurality of filtering processes selected depending on the eigenvalues of a linear transformation component relating said image data.

The apparatus according to claim 69, wherein said eigenvalues are used to select from supersampling, bilinear interpolation, and resampling as the said one sampling process.

71. The apparatus according to claim 69, wherein said filtering process comprises supersampling if the magnitudes of all eigenvalues of said linear transformation component are less than a predetermined number.

72. The apparatus according to claim 69, wherein said filtering process comprises bilinear interpolation if the magnitudes of all eigenvalues of said linear transformation component are greater than or equal to a predetermined number.

73. The apparatus according to claim 69, wherein said filtering process comprises resampling if the magnitude of only one of the eigenvalues of said linear transformation component is less than a predetermined number. 541440

74. The apparatus according to any one of claims 71 to 73, wherein said predetermined number is equal to one.

75. An apparatus for filtering a first image to generate a second image using one of a plurality of sampling methods, wherein said first image and second image are related by an affine transformation comprising a linear transformation component, said apparatus comprising: means for selecting said sampling process depending on the magnitudes of the eigenvalues of said linear transformation component, and filtering said first image using said selected sampling process to generate said second image if the eigenvalues of said linear transformation component are real, otherwise representing the linear transformation component as a matrix and decomposing said matrix into a lower triangular matrix and a rotation matrix; 15 selecting said sampling process depending on the magnitudes of the eigenvalues of said triangular matrix; and filtering said first image using said selected sampling process to generate said second image. o 20

76. The apparatus according to claim 75, wherein said eigenvalues are used to select from supersampling, bilinear interpolation, and resampling as the said one sampling process.

77. The apparatus according to claim 75, wherein said sampling process comprises supersampling if the magnitudes of the eigenvalues of said linear transformation component or said triangular matrix are less than a predetermined number. 541440 36

78. The apparatus according to claim 75, wherein said sampling process comprises bilinear interpolation if the magnitudes of eigenvalues of said linear transformation component or said triangular matrix are greater than or equal to a predetermined number.

79. The apparatus according to claim 75, wherein said sampling process comprises resampling if the magnitude of only one of the eigenvalues of said linear transformation component or said triangular matrix are less than a predetermined number.

80. The apparatus according to any one of claims 77 to 79, wherein said predetermined number is equal to one.

81. An apparatus for filtering a first image to generate a second image, said first image and second image being related by a non-affine transformation, said transformation oooo being approximated by a plurality of affine transformations over different regions of the image, each affine transformation comprising a linear transformation component, said apparatus being characterised in that said apparatus comprises selection means for selecting one of a plurality of sampling processes for each region, depending on at least one eigenvalue of said region's corresponding linear transformation component.

82. A computer readable medium, having a computer program recorded thereon, wherein the program comprises code to perform a method of filtering a first image to generate a second image using one of a plurality of sampling processes, said first image and second image being related by an affine transformation, said affine transformation comprising a linear transformation component, said method being characterised in that 541440 37 said one sampling process is selected depending upon at least one eigenvalue of the linear transformation component.

83. A computer readable medium, having a computer program recorded thereon, wherein the program is configured to perform a method of filtering image data, said program comprising: code for accessing a first set of discrete sample values of said image data; and code for calculating a second set of discrete sample values using one of a plurality of filtering processes selected depending on the eigenvalues of a linear transformation component relating said image data.

84. A computer readable medium, having a computer program recorded thereon, wherein the program is configured to perform a method of filtering a first image to generate a second image using one of a plurality of sampling methods, wherein said first 15 image and second image are related by an affine transformation comprising a linear transformation component, said program comprising: if the eigenvalues of said linear transformation component are real, code for selecting said sampling process depending on the magnitudes oooo• *~of said eigenvalues of said linear transformation component; and code for filtering said first image using said selected sampling process Goes to generate said second image; :otherwise, code for representing the linear transformation component as a matrix and decomposing said matrix into a lower triangular matrix and a rotation matrix; 541440 38 code for selecting said sampling process depending on the magnitudes of the eigenvalues of said triangular matrix; and code for filtering said first image using said selected sampling process to generate said second image.

A method of filtering a first image to generate a second image using one of a plurality of sampling processes, substantially as herein described with reference to the accompanying drawings.

86. An apparatus for filtering a first image to generate a second image using one of a plurality of sampling processes, substantially as herein described with reference to the accompanying drawings.

87. A computer readable medium, having a computer program recorded thereon, 15 wherein the program is configured to perform a method of filtering a first image to generate a second image using one of a plurality of sampling methods, wherein said first ,image and second image are related by an affine transformation comprising a linear transformation component, said method being substantially as herein described with reference to the accompanying drawings. DATED this Eighth Day of February, 2001 Canon Kabushiki Kaisha Patent Attorneys for the Applicant SPRUSON FERGUSON 541440