GB2501936A

GB2501936A - Micro-lens array with micro-lens subsets displaced from a regular lattice pattern

Info

Publication number: GB2501936A
Application number: GB1208317.6A
Authority: GB
Inventors: Benoit Vandame
Original assignee: Canon Inc
Current assignee: Canon Inc
Priority date: 2012-05-11
Filing date: 2012-05-11
Publication date: 2013-11-13
Anticipated expiration: 2032-05-11
Also published as: GB201216108D0; GB201208317D0; GB2501950A; WO2013167758A1; GB2501936B; GB2501950B

Abstract

A micro-lens array for an imaging device has micro-lenses located on the micro-lens array relatively to a regular lattice. The micro-lens array has micro-lens subsets, each sub-set having a two dimensional array of micro lenses, wherein micro-lenses of each sub-set are displaced relative to the regular lattice according to a common pattern, the common pattern defining different displacements for each micro-lens of the sub-sets. Each micro-lens of the subset may have a focal distance f. Micro-lenses of each sub-set may be displaced relative to the regular lattice according to a displacement pattern, which defines the displacement of each micro-lens as integer multiples (k, l) of unit vectors, said unit vectors having a magnitude which is a function of f /N.

Description

MICRO LENS ARRAY AND IMAGING APPARATUS

The invention relates to light-field camera and to a micro-lens array for light-field camera.

Light-Field cameras record 4D (four dimensional) light-field data which can be transformed into various reconstructed images like re-focused images with freely selected focal distance that is the depth of the image plane which is in focus. A re-focused image is built by projecting the various 4D light-field pixels into a 2D (two dimensional) image. Unfortunately the resolution of a re-focused image varies with the focal distance.

For example, the publication US 201 0/0265381A1, "Imaging Device" proposes an imaging device with a micro-lens array where the micro-lenses are displaced from an equidistant arrangement to a non-linear arrangement according to the height of the image on the imaging element. The pitch between the micro-lenses changes from the centre to periphery of the micro-lens array. The displacement provides an optical correction in order to compensate the image displacement implied by the geometric distortion of the main lens. The publication provides however no solution for improving the resolution of reconstructed images when the focal distance changes.

Particular aspects of light-field cameras will first be exposed.

Light-Field cameras design.

We consider light-field cameras which record a 4D light-field on a single sensor like a 2D regular array of pixels. Such light-Field cameras can be for instance: 1) a plenoptic camera comprising a main lens, an array of lenses and a sensor 12; or 2) a multi-camera array comprising an array of lenses and a sensor, but without main lens.

The array of lenses is often a micro-device, which is commonly named a micro-lens array.

Figure 1 illustrates a plenoptic camera 1 with three major elements: the main lens 10, the micro-lens array 11 and the sensor 12. Figure 2 illustrates a multi-camera array 2 with two major elements: the micro-lens array 11 and the single sensor 12.

Optionally spacer or spacer material may be located between the micro-lens array around each lens and the sensor to prevent light from one lens overlapping with the light of other lenses at the sensor side.

It is worth noting that the multi-camera array can be considered as a particular case of plenoptic cameras where the main lens has an infinite focal length. Indeed, a lens with an infinite focal length has no impact on the rays of light. The present invention is applicable to plenoptic cameras as well as multi camera arrays.

4D Light-Field data

Figure 4 illustrates the image which is recorded at the sensor. The sensor of a light-field camera records an image of the scene which is made of a collection of 2D micro-images, also called small images, arranged within a 2D image. Each small image is produced by a lens from the array of lenses. Each small image is represented by a circle, the shape of that small image being function of the shape of the micro-lens. A pixel of the sensor is located by its coordinates (x,y). p is the distance in pixels between two centres of contiguous micro lens images. The micro-lenses are chosen such as p is larger than a pixel width. A micro-lens image is referenced by its coordinates (i,j). Some pixels might not receive any light from any micro-lens; those pixels are discarded. Indeed, the space between the micro-lenses can be masked to prevent photons falling outside of a lens (if the micro-lenses are square or another close packed shape, no masking is needed). However most of the pixels receive the light from one micro-lens. The pixels are associated with four coordinates (x,y) and (i,f). The centre of the micro-lens image (i,f) on the sensor is labelled (x1,y,,). Figure 4 illustrates the first micro-lens image (0,0) centred on (x00,y10). The pixels of the sensor 12 are arranged in a regular rectangular lattice.

The micro-lenses are arranged in a regular rectangular lattice. The pixels lattice and the micro-lenses lattice are relatively rotated by 0. The coordinate can be written in function of the 4 parameters: p. 0 and (x00,y00): = pcosOi-psin9j+x00 (1) = psinOi+pcosOj+y00 Figure 4 also illustrates how an object, represented by the black squares 3, in the scene is simultaneously visible on numerous micro-lens images. The distance w between two consecutive imaging points of the same object 3 on the sensor is known as the disparity. The disparity depends on the physical distance between the camera and the object. w converges to p as the object becomes closer to the camera.

Depending on the light-field camera design, w is either larger or smaller than p (if d is respectively larger or smaller than f, (see Figure 3 and next sub-section about the geometrical property of the light-field camera). Figure 4 illustrates a case where w is smaller than p. An important characteristic is the number r of consecutive lenses through which an object is imaged. r is in units of /. It is estimated by considering the cumulated disparity on consecutive lenses: )w -pr < p. One obtains the following characteristic for the number of replications, r: p (2) LpH Where LaJ denotes the ceiling value of a. This equation is an estimation which assumes that the micro-lens images are squared with no left-over space (i.e. Close packed or abutting). r is given for one dimension, an object is therefore visible in r2 micro-lens images considering the 2D grid of micro-lens. Without the ceiling function, r would be a non-integer value, r is in fact an average approximation. In practice, an object can be seen r or r + 1 times depending on rounding effect.

Geometrical property of the light-field camera

The previous section introduced w the disparity of a given observed object, and p the distance between two consecutive micro-lens images. Both distances are defined in pixel units. They are converted into physical distances (meters) W and P by multiplying respectively w and p by the pixels size 6 of the sensor: W = 8w and P=8p.

The distances W and Pcan be computed knowing the characteristics of the plenoptic camera. Figure 3 gives a schematic view of the plenoptic camera with the following elements: * The main lens 10 is an ideal thin lens with a focal distance F. * The micro lens array 11 is made of micro-lenses having a focal distance f.

The pitch of the micro-lenses is 0. The micro-lens array is located at the fix distance D from the main lens. The micro-lenses might have any shape like circular or squared. The diameter of the shape is smaller or equal to. One can consider the particular case where the micro-lenses are pinholes. In this case the following equation remains valid with / = d.

* The sensor 12 is made of a squared lattice of pixels having each a physical size of 3. 3 is in unit of meter per pixel. The sensor is located at the fix distance d from the micro-lens array.

* The object (not visible in the figure 3) is located at the distance z of the main lens. This object is focused by the main lens at a distance z' from the main lens. The disparity of the object between two consecutive lens is equal to W. The distance between 2 micro-lens image centres is P. Following the mathematics of thin lenses we have:

I I I (3)

z z' F From the Thales law we can derive that: D-z'D-z-d (4) 0 W Mixing the 2 previous equations the following equation is easily demonstrated: (5) This equation gives the relation between the physical object located at distance z from the main lens and the disparity W of the corresponding views of that object.

This relation is build using geometrical considerations and does not assume that the object is in focus at the sensor side. The focal length f of the micro-lenses and other properties such as the lens apertures allow determining if the micro-lens images observed on the sensor are in focus. In practice, one typically tunes the distance D and a once for all using the relation: 1 1 1 +-=-(6) D-z' d f The micro-lens images observed on the sensor of an object located at distance z from the main lens appears in focus so long as the circle of confusion is smaller than the pixel size 6. In practice the range [zm,zJ of distances z which allows observing in focus micro-images is large and can be optimized depending on the focal length f, the apertures of the main lens and the micro-lenses, the distances D and d: for instance one can tune the micro-lens camera to have a range of z from 1 meter to infinity [1,ccj.

Also from the Thales law one derives P: D+d D (7) P=çbe The ratio e defines the enlargement between the micro-lens pitch and the micro-lens images pitch projected at the sensor side.

Variation of the disparity The light-field camera being designed, the values D, ci, f and F are tuned and fixed. The disparity IV varies with z, the object distance. One can note particular values of W: * is the disparity for an object at distance z1 such that the micro-lens images are exactly in focus, it corresponds to equation (6) . Mixing equations (4) and (6) one obtains: (8) iiç1, is the disparity for an object located at distance z =aF from the main lens. According to equation (5) one obtains: W= 1-(9)

D-P a-i

The variation of disparity is an important property of the light-field camera. The ratio flJ. /F1fQ is a good indicator of the variation of disparity. Indeed the micro-lens images of objects located at ZJh(v. are sharp and the light field camera is designed to observed objects around z, which are also in focus. The ratio is computed with equations (8) and (9): 1+ (10) d D -a_F a-i The ratio is very close to one. In practice the variations of disparity is typically within few percent around W. The present inventor has further brought to light the following aspects.

Image refocusing method A major interest of the light-field cameras is the ability to compute 2D images where the focal distance is freely adjustable. To compute a 2D image out of the 4D light-field, the small images observed on the sensor are zoomed, shifted and summed. A given pixel (x,y) of the sensor associated with the micro-lens (i,j) is projected into a 2D image according to the following equation: J x = s.(g(x-x:j)+x,) (11) = s.(g(y-y.7)+y:j) Where (X, Y) is the coordinate of the projected pixel on the 2D refocused image. The coordinate (X,Y) is not necessarily integer. The pixel value at location (x,y) is projected into the 2D refocused image using common image interpolation technique.

Parameter s controls the size of the 2D refocused image, and g controls the plane which is in focus (the plane perpendicular to the optical axis, for which the 2D image is in focus) as well as the zoom performed on the small images. The output image is s times the sensor image size. In this formulation the size of the re-focused image is independent from the parameter g, and the small images are zoomed by sg.

The previous equation can be reformulated due to the regularity of position of the centres of the micro-lens images.

= sgx+sp(1-g)(cos8 /-sinS j)+s(1-g)x00 12 Y = sgy+sp(1-g)(sinO i+cosO j)+s(1-g)v00 The parameter g can be expressed as function of p and w. It is computed by simple geometry. It corresponds to the zoom that must be performed on the micro-lens images, using their centres as reference, such that the various zoomed views of a same objects get superposed. One deduces the following relation: p (13) p-w This relation is used to select the distance z of the objects in focus in the projected image. The value of g can be negative depending on the light-field camera design. A negative value means that the micro-lens images need to be inverted before being summed. One notices that r = ]gj.

Including this last relation into equation (12) one rewrites the projection equation: X = p (14) Y = sgy -sgw(sin 01 + cosO j) + p The last formulation has the great advantage to simplify the computation of the projected coordinate by splitting the pixel coordinates (x,y) and the lens coordinates

(i,j) of the 4D light-field.

Sampling property of the refocused image The different pixels of the light-field image are projected into the re-focused image according to the above described method and define a set of projected coordinates (X,Y) into the grid of the refocused image. It has been recognized by the present inventors that the distribution of the set of projected coordinates is an important property which can be used to characterise the resolution of the refocused image, and in particular, the regularity or homogeneity of the distribution. As will be explained later the present invention addresses this homogeneity.

It is not trivial to characterise the homogeneity of the projected 4D light-field pixels into the 2D re-focused image. To study this property one considers a simple projection equation assuming that the rotation angle 0 is zero, and the coordinate of the first micro-lens centre (x00,y00) is equal to (0,0). This assumption does not impact the proposed study. One obtains the following simplified projection equation with ii = sg x = ux-uwi = (x-n) p-w (15) sp1 Y = uy-uw/ = _____ p-w This set of equation shows a simple relation between the 4 dimensions x,y,i,j and the projected coordinates (X, Y). The value u = sg is a constant independent of w if s = kig where Ic is any constant. In this condition, the size of the re-focused image is function of w and is equal to kig times the size of the original image.

Figure 5 illustrates the 1 D projected coordinate X for a particular settings. s = 0.5, g=7.677, w=151.05, u=3.83 and p=l'73.67. The x-axis shows the projected coordinatesX, the y-axis indicates the micro-lens coordinates t of the projected pixels. One notices that 8 micro-lenses contribute to the observed projected coordinatesX, which in this case is equal to r+1. The distribution of the projected points X is not homogeneous since the values Ii and H representing respectively the minimum and the maximum sampling steps between 2 consecutive projected coordinates xare substantially different from each other. In this example, the projected coordinates are nearly superposed, clustered in groups of eight.

Figure 6 illustrates the same view with the same settings except that w=151.25. The distribution of the projected points X is homogeneous, the projected points being distributed with equal spacing along the axis X. One notices this ideal case where Ii = H = u/w}. Where a} denotes the fractional part of a. w} plays a major role in the maximum sampling steps between the projected coordinates.

Several cases of h and H occur depending on {w} 1. With {w} = 0: h = 0 and H = u. On average, r projected coordinates X overlap. The distance between 2 non-overlapped consecutive X is constant and equal to H. The projected coordinates define a perfect sampling with a constant sampling step equal to H = u.

2. With {w} = ;i/ N where ii and N are positive integers such as 0< n <N «= r. In this case the number of overlapped projected coordinates X is equal, on average, to rgcd(n,N)/N where gcd(n,N)refers to the greatest common divisor between n and N. The projected coordinates define a perfect sampling with a constant sampling step equal to H =ugcd(n,N)/N. The sampling step is smaller if N is a prime number. Indeed, if A! is not a prime number, the number of overlapped coordinates increase as well as the sampling step. The perfect sampling of the projected coordinates X with the smallest sampling step is obtained for w} = n/r and gcd(;i,r) = 1.

3. With {w} = /2/ N where ii and N are positive integers such as 0< ii < A and N> r. In this case there are no overlapped projected coordinatesX. But the sampling defined by the projected coordinates is not perfect: Ii =ugcd(n,N)/N and H=u-h. The projected pixels are clustered, in other word some projected pixels are sampled with a small sampling step equal to Ii, where other pixels are sampled with a larger sampling step H: the projected coordinates appear clustered.

H is a good indicator to estimate the resolution of the re-focused image. Figure 7 illustrates the normalized sampling step as a function of {w} for a conventional light field camera characterized by p=l73.67 and w53 150. This function is built from the 3 cases described above: points surrounded by the black circles depict the first case (h=0 and {w}=0); points surrounded by the empty circles depict the second case (all the possible regular grids) other points lying on the dark line segments correspond to the third case with all possible n values and any N> r (all possible irregular grids). The best possible resolution is given by h = H/u =1/r =1/7.

Problem of light-field cameras

The projection of the 4D light-field pixels defines a set of projected points having a distribution which depends on the selected focal distance (i.e. The object plane which it is desired to be in focus). As explained above, the resolution of the re-focused image highly depends on the distribution of the projected coordinates(X, Y). The resolution can be estimated by the maximum sampling step H. Unfortunately, H depends on {w} and varies from values of u to u/r. Variations of Hmake the resolution of the re-focused image vary. The present inventors have recognized that the distribution characterizes the resolution of the projected image. It is an object or at least one aspect of the invention to mitigate sampling variation in reconstructed images and therefore to obtain a more constant resolution of the reconstructed images for any selected focal distance.

The present invention discloses a micro-lens array, suitable for use in a light-field camera for example, where the centres of the micro-lenses are slightly displaced versus a regular lattice. The small displacements are preferably defined by characteristic parameters of the light-field camera and in certain embodiments can advantageously be arranged in order to provide an optimal distribution of the projected image.

In accordance with one aspect of the present invention there is provided a micro-lens array for an imaging device comprising micro-lenses located on the micro-lens array relatively to a regular lattice, wherein the micro-lens array comprises a plurality of micro-lens sub-sets, each sub-set comprising a two dimensional array of (Q) micro lenses, and wherein micro-lenses of each sub-set are displaced relative to the regular lattice according to a common pattern, the common pattern defining different displacements for each micro-lens of the subsets.

The common pattern defines a displacement model whereby each micro-lens of a sub-set is located according to the common pattern but differently located with respect to the regular lattice. The pattern, in certain embodiments, defines a number of possible displacements and hence positions, for each micro-lenses. Although it is simpler for the actual displacements of micro-lenses in different sub sets to be the same, embodiments of the invention allow different subsets of the plurality to have different displacements, while still adhering to the same, common pattern or model of displacements. However, the pattern or model is such that even with a certain degree of flexibility provided for each lens displacement, the relative displacements between micro-lenses of a subset adhere to a controlled relationship, and such relationship is observed similarly across subsets of the plurality.

Thanks to these characteristics, the resolution of an image reconstructed from the micro-images is improved over conventional light-field cameras. In particular a good diversity of sampling is obtained while variations of resolution are avoided. Thus a more regular resolution is obtained for any focalization distance. Typically, all subsets of the micro-lens array share a common pattern. However, the plurality of subsets need not encompass the whole micro-lens array. It could be envisaged for example that a first common pattern could apply to a first plurality of subsets, and a second common pattern could apply to a second plurality.

Advantageously, the common pattern defines each displacement as a function of the position (1,1) of each micro-lens within the sub-set. The common pattern may further define each displacement as a function of the number (Q) of micro-lenses in each sub-set. The obtained dispositions of the micro-lenses provide for an advantageous distribution for the projected image and reduce the super-position or the clustering of pixels in a reconstructed image, which permits to obtain a more constant resolution for any focalization distance.

According to one embodiment, the common pattern defines displacements in integer multiples of unit displacement vectors. The vectors are preferably orthogonal vectors.

These features allow reducing the super-position or the clustering of pixels while reconstructing an image from the micro-images. The magnitude (r) of the unit displacement vectors is advantageously a function of the focal distance of micro-lenses.

In another embodiment, the magnitude of the unit displacement vectors is a function of the number (Q) of micro-lenses in each sub-set. The multiple of the unit vectors for each micro-lens may further be a function of the position of the micro-lens within the sub-set. The resulting dispositions of the micro-lenses provide for an advantageous distribution for the projected image and reduce the super-position or the clustering of pixels in a reconstructed image.

In a particular embodiment, the sub-sets comprise a square array of NxN = Q micro-lenses. Furthermore, the common pattern defines a plurality of possible displacements for each micro-lens, each of said plurality being equivalent in modulo N. In a further embodiment, the displacement of at least one micro-lens in each sub-set is zero. This allows providing a common reference in each micro-lens sub-set so that the computation of a reconstructed image is simplified. Furthermore, it allows the relative position between the micro-lenses to be precisely obtained. This feature also simplifies the fabrication of the micro-lens array.

Advantageously, the common pattern and the associated displacements are independent of the location of the sub-set within the micro-lens array.

The micro-lens array as set out above may be embodied in an imaging device including a photo-sensor having an array of pixels, each micro-lens projecting an image of a scene on an associated region of the photo-sensor forming a micro-image.

In accordance with another aspect of the present invention there is provided a micro-lens array for an imaging device comprising micro-lenses located on the micro-lens array relatively to a regular lattice, wherein the micro-lens array comprises a plurality of micro-lens sub-sets, each sub-set comprising an array of NxN micro lenses, each micro-lens of the subset having a focal distance f, wherein micro-lenses of each sub-set are displaced relative to the regular lattice according to a displacement pattern, said displacement pattern defining the displacement of each micro-lens as an integer multiple of unit vectors, said unit vectors having a magnitude r wherein v is a function off/N.

These features allow reducing the super-position or the clustering of pixels when reconstructing an image from the micro-images. Advantageously, the magnitude has a fixed value representing characteristics of the imaging device.

According to a further aspect, the present invention also relates to an imaging device comprising the disclosed micro-lens array.

In embodiments of the invention, the displacement (if any) of each micro-lens is of the order of 1/1000 of the micro-lens pitch. Embodiments may therefore have displacements in the range of approximately 0-5pm, or approximately.0-lOpm for

example.

Embodiments of the invention will now be described, by way of example only, and with reference to the following drawings in which: Figure 1 is a schematic view of a light-field camera; Figure 2 is a schematic view of a particular light-field camera; Figure 3 is a detailed view of a light-field camera made of perfect lenses; Figure 4 is a schematic view of the 4D light-field data recorded by the 2D image

sensor of a light-field camera;

Figure 5 is an illustration of the coordinates of the projected 4D Light-Field pixels into the 20 projected image in a normal case; Figure 6 is an illustration of the coordinates of the projected 4D Light-Field pixels into the 20 projected image in a case associated with a particular disparity; Figure 7 illustrates a normalized maximum sampling step of the projected! reconstructed image from 4D light-field data for a conventional light-field camera; Figure 8 is a schematic view of a micro-lens array according to an embodiment of the invention with displaced micro-lenses for a super-resolution factor N=2; Figure 9 is a schematic view of the a micro-lens array according to an embodiment of the invention with displaced micro-lenses for a super-resolution factor N=3; Figure 10 illustrates a normalized maximum sampling step of the projected! reconstructed from 4D light-field data obtained with the micro-lens array of figure 8 with displaced micro-lenses for a super-resolution factor of N=2; Figure 11 illustrates a normalized maximum sampling step of the projected! reconstructed from 4D light4ield data obtained with the micro-lens array of figure 8 with displaced micro-lenses for a super-resolution factor of N=3.

Micro-lens array with displaced micro-lenses The pixels of the 4D light-field image are projected into a re-focus image. As described above, the maximum sampling step of the projected coordinates depends on {w} the fractional part of the disparity. The variations of sampling step are due to the superposition or clustering of the projected coordinates for certain values of {w} as illustrated in figure 5.

To decrease the superposition or the clustering of the projected coordinates (X,Y), the micro-lens images are shifted as compared to a regular array, so as to reduce or prevent overlapping or clustering of projected pixels. In other words, in embodiments of the invention the centre of a given micro-lens (i,j) is shifted by the given shift (A,(i,j),A1(i,j)) so that the modified projected coordinates (X',Y') of this new light-

field camera would become:

Jx' = ux-uw.Q+A(i,j))= X-uwA,(i,j) = uy -uw (J + A7(i,j))= y -uwA,(i,j) (A,(i,j),A1(i,j)) are shifts given in unit of distance between the micro-lens centres, or the micro-lens image centres. The motivation of moving the micro-lenses is to have a perfect and constant sampling of the projected coordinates (X',Y') for any w=LwJ+n/At where N is a selected positive integer smaller or equal to r, and n is any integer such as nc[O,N[.ln conventional light-field, the sampling step is a function of n for a given N. If the sampling step is made independent of ii, then N acts as a super-resolution factor. Equation (16) becomes: = x' = iv(x-LvJi)-ni -A(i,j)(NLwj+n) (17) = Y' = N(y-Lwj')-nj-A1(i,j)(NLwJ+n) (X", Y") are normalized projected coordinates such that (X", Y") are integers for a perfect sampling of the projected coordinates (X',Y'). For a perfect sampling A(i,j)(NLwJ+n) and A1(i,j)(NLWJ+n) must also be integers respectively equal to k(i,j) and 1(4]). These constraints give us the following values for (A:(i,j),A,(i,j)): A! \_ _________ 4 1 \ _________ Ls;j,J)-(18) ivLmvj+n ivLwj+n The displacement of the micro-lens images depends on w. In other words, for a given micro-lens displacement (A(i,j),A1(i,j)), the shift of the corresponding micro-lens image depends on the disparity w. The previous equations can be approximated by taking into consideration the two considerations: 1) w>> N; and 2) the variations of w are small. w can be considered constant and equal to w,.

Indeed, it has been shown (Cf. equation (1 0)) that the ratio Jf which is equal to WaF / Wf,, is typically very close to 1. In this condition, equation (18) can be approximated by: AJJ, I k(i,j) A I I(i,j) Vtio(;u. N M,cu. JV (19) A1(i,j) 9 k(i,j) A1(i,j) 9 KJ) 0C7U The second line of the previous equation is given knowing that w.

where o is the physical size of a pixel. The approximation of (AJJ,j),A1(/,j)) does not depend on the disparity w. Thus, by using this approximation, it is possible to build a micro-lens array with an irregular grid of micro-lenses such as the projected coordinates (X', Y') do not overlap or cluster as it happens for the projected coordinates (X, Y) of conventional light-field imaging devices.

Condition for an optimal micro-lens displacement for optimal homogeneity A remaining question is how to define the 2 functions k(i,j) and 10,1) to have optimum micro-lens displacements such that the projected coordinate (X",Y") have a minimum clustering, and a perfect sampling when w = n / N. Equation (17) can be simplified considering equation (19) and w>> N x" = tx = N(x-Lw)-n/-kQ,j) (20) = Y' = N(v -LwJi)-i -1(1,]) To obtain a perfect sampling the set of projected coordinates (X", Y") defined by the various lens coordinates (i,j) must have all possible integer values whatever n, and also the number of contiguous lenses to obtain the perfect sampling must be minimum and equal to N (considering one dimension). This constraint can be reformulated by taking into consideration modular arithmetic modulo AT: N-Hi)-ni -k(i,j) -ni -k(J,j) (modN) 21 1 F' NC-Hi)-nj-i(i,j) -n/-10,1) (modN) k(i,j) and 1(1,]) are 2 periodic functions, one period is defined with (i,f)e[O,N[2, into [o,N[. One is searching k(i,j) and l(i,j) such that for any given ii E [o,N[, all the set of projected coordinates (x"modN,y"modN) defined by h-A' I h-A' I -, -.

(imodN,jmodN)e[O,NL is equal to b,5 where àab is the Dirac function located at (a,h) with (a,h) being integer numbers.

Theoretical solution To solve equation (21) the following linear solutions are considered: Jk(i,J) Ai+B/+K (modN) 22 t 1Q,j) Ci+Ej+L (modN) Equation (21) becomes: fx= -ni-Ai-Bj-K (modN) 23 Y" -nj-Ci-Ej-L (modiv) For the second member of the previous equation, one derives the value of Ci, which also appear in the first member after multiplying it by C: -Cni-CAi-CB/-CK (modN) Ci -Y -n-Ej-L (modN) Replacing the second member into the first member (24) becomes: Xu?e_J(n2 +n(A + E)+ AE-BC)-(Y'4-LXn + A)-CK (mod N) (25) The set of projected coordinates located at (x"modN,Y"modN) must cover all coordinates 5 for (/ mod N,j mod N) [o, NL whatever n c[o, N[. With equation (25) one deduces that X(modN) must have any possible values whatever Y"(modN). Thus the second order polynomial function,n(n)=n2+n(A+E)+AE_BC must verify: gcd(in(n) mod N, N) = 1 Vn E [o, N[ gcd(n2 +n(A +E)+ AE-BC(modN),N)= 1 Vn E [O,N[ The NxN micro-lenses defines a sub-set of micro-lenses. For a given sub-set the values A,B,C,E,K,L are freely selected according to the previous equation. The parameters A,B,C,E,K,L may take different values in the different sub-sets.

Parameters K,L define which if any micro-lens from a given subset is not displaced with respect to the regular lattice.

Experimental solution Many values A,B,C,E verify equation (26). The special case: A=O, B=T, C=1, E = 1, K = 0 and L = 0 is detailed in this section. The proposed solution has the following form: JkQ,J) 77 (mod N) (27) 1i(i,i) 1+1 (modN) T is a free parameter which has been experimentally determined for various values N. The experimentation consists in testing various values of T e [o,N[ such that the constraint gcd(tn(n)mod N, N) = I is respected for any [o, N[. The following table indicates the smallest value of T according to that constraint: 1 0 6 1 11 3 16 4 21 1 2 1 7 1 12 3 17 1 22 3 3 1 8 1 13 1 18 1 23 1 4 1 9 1 14 1 19 3 24 1 3 10 3 15 1 20 3 25 3 It follows that the periodic functions k(i,j) and 1(1,]) are fully characterized and thus the shifts (A,(i,j),A1(i,j)) of the micro-lens image versus the regular grid are also fully characterized. The shifts are given in unit of (/,]). To convert the shifts into physical unit at the micro-lens side, the shifts must be multiplied by. The physical shifts (A1(i,j),0/)) at the micro-lens side are computed easily by combining equation (8) and (19): Ai,j) L.±.k@,J) A1(i,j) i(f,J) (28) c/N c/N The physical shifts can be decomposed in the increment r = ZN which is multiplied by the integers values given by k(i,j) and 10,1) to obtain the physical shifts. The increment r is independent from the characteristics of the main lens.

Thus the main lens can be replaced by any optical system which delivers a focus images located perpendicularly to the main optical axis at location z (as illustrated in Figure 3). This invention applies to many light-field cameras such as: an array of cameras (as illustrated in Figure 2); a plenoptic camera (as illustrated in Figure 1); a plenoptic camera where the main lens is a zoom which delivers zoomed images in focus at location z. The design of the micro-lens array is therefore defined by: * The focal distance f of the micro-lenses.

* The average pitch 0 between consecutive lenses.

* The distance d between the micro-lens array and the sensor.

* The pixel size 6 of the sensor.

* The super-resolution factor N which is freely selected between [1,r].

* The micro-lens centres (p,,p) are located following the equation: = iOJAk(i,J) (29) = It should be recalled that the functions k(4j) and 1(J,j) are defined modulo N: thus the centers are valid as well as u, +at%,p, +a.th,4J whatever a being an integer. Consequently the displacements can be negative.

The micro-lens array is designed according to the previous settings. If the size of the micro-lenses is equal to the pitch, then the micro-lenses might have a very small overlaps due to displacement the micro-lenses versus the squared lattice. This issue is solved by designing micro-lenses such that the micro-lens size is smaller than _Ia(N_1X. The shape of the micro-lenses can be circular, rectangular or any shape without any modification of the previous equations. The number of micro-lens (i,i) to be designed in the micro-lens array is defined such that (J�e,Jpe) is equal to the physical size of the sensor. The micro-lens array being designed, it is located at distance d from the sensor. It is interesting to note that the above demonstration remains valid whatever the coordinates of the first lens are and whatever the angular position between the micro-lens array and the lattice of pixels is.

Micro-lens array design An imaging device including the above proposed arrangement as well as a micro-lens array will now be described. The following values are chosen for the different parameters: Symbols Values Comments F 70mm Main focal distance / 2mm Micro-lens focal distance ci 2.3mm Distance between the micro-lens array and the sensor 1mm Micro-lens pitch 0.004mm/pixel Physical size of pixel from the sensor z. 5000mm Object is located at 5 meters from focus the main lens 70.994mm Distance between the main lens of the focus plan of object z1,5 D 86.327mm Distance between the main lens and the micro-lens array such that images on sensor is in focus.

D-z' 15.33mm Distance between: the focus plan of the object z observed through the main lens, and the micro-lens array.

e 1.0266 Enlargement P 1. 0266mm Pitch in physical unit of the micro-lens images projected on the sensor P 256.S6pixel Pitch in pixel unit of the micro-lens images projected on the sensor 1.15mm Disparity in physical unit observed on the sensor of the object located at distance z from the main lens.

287.5pixel Disparity in pixel unit observed on the sensor of the object located at distance Zjo0, from the main lens.

r 8 Averaged number of replications for an object located at distance z from the main lens.

Figure 8 illustrates a case where the super-resolution factor is chosen to beiv = 2 which also corresponds to the size of the N by N sub-set. In this case the increment v = is equal to r = I.74,wn. Figure 8 shows the displacements of the micro-lenses versus a regular lattice. The regular lattice is defined by the equidistant dashed lines 0,1,2,3 extending in both directions / andj. The directions I and fare preferably perpendicular. In a conventional micro-lens array the centres of the micro-lenses are located at the intersections of the lines defining the regular lattice to form a regular grid of equidistant micro-lenses. According to an embodiment of the present invention the micro-lenses are arranged in the following way on the array 11 in accordance with formula 29. Blocks of N by N, N being the super-resolution factor having here the value 2, micro-lenses indicated by the bold dashed squares form a subsets of micro-lenses 200, 220, 240, 202, 222, 242.... The blocks or micro-lens sub-sets are replicated in I and f directions such that the micro-lens subsets are adjacently disposed in the two directions: subsets 200,220,240.. .and subsets 202,222,242... are adjacently disposed in direction I, while subsets 200,202... and subsets 220,222... are adjacently disposed in direction j. The micro-lens array is therefore formed of a plurality of micro-lens sub-sets disposed in a tiling form. In the present example the micro-lenses of each sub-set are all identical displaced in view of simplifying calculation of a reconstructed image. However, different arrangements can be given to the micro-lens sub-sets over the micro-lens array. This can be done for example by choosing different values for A,B,C,E in different micro-lens sub-sets.

The bold arrows indicate the displacement as a shift vector of the micro-lens centres.

The amount of displacement is given by a multiple of a fixed increment t-in accordance with formula 29 and the table at the end of this paragraph. The arrows displayed in the figure have been artificially zoomed for illustration purpose.

A plurality of micro-lenses are shifted with respect to the regular lattice: in the illustration micro-lenses ClO, C30, C12, C32... are shifted by r in the directionj.

Micro-lenses Cli, C31... are shifted by r in the direction I. Micro-lenses COl, C21... are shifted by r in the directions / and f. It follows that micro-lenses are set out of regular alignment in a particular way that reduces the superposition or the clustering of pixels in a reconstructed image. Preferably each micro-lens in each micro-lens sub-set is displaced by a different shift vector to increase the resolution of a reconstructed image. Each shift vector has a shift magnitude and a direction.

Optionally at least one micro-lens in each subset is not displaced with respect to the regular lattice: in the illustration the centre of micro-lenses COO, C20, C02, C22...

belonging respectively to subsets 200,220,202,222... are located on the intersection lines of the regular lattice, here at the top left corner of each subset (K = L = 0). In practice, the micro-lenses which are not displaced can be any of the lenses of a sub-set. The position of a micro-lens which is not displaced can vary from one sub-set to another. Preferably one micro-lens is not displaced in each of the sub-set and the other micro-lenses of the sub-set have relative displacements with respect to the un-displaced micro-lens. The displaced micro-lenses in each sub-set have displacements which are defined relative to an un-displaced micro-lens.

As described above, the displacements are determined such that the superposition or the clustering of pixels in a reconstructed image is decreased (see also figure 10).

The values k(,j), 1(1,]) and for the first sub-set of 2x2 micro-lenses illustrated in figure 8 are given in the following table: 0 0 0 0 0 0 1 0 0 1 1 0 1 1 1 1 1 1 0 0 The micro-lens array illustrated in Figures 8 and 9 is made unitarily of glass or synthetic glass. Possible processes for forming the micro-lenses on a glass plate includes lithography and/or etching and/or melting techniques.

Figure 9 is similar to figure 8 but illustrates a case with a super-resolution factor of N = 3. In this case the increment is r = 1.16/in? . The micro-lenses are arranged in a similar way as in figure 8 except that the subsets of micro-lenses are composed of 3 by 3 micro-lenses. The subsets 300,330,303,333 of micro-lenses are adjacently disposed in the two directions I and j. The displacement of the micro-lenses versus the regular lattice is similar to figure 8.

A plurality of micro-lenses are shifted with respect to the regular lattice: micro-lenses Gb... are shifted by r in the direction]. Micro-lenses C12... are shifted by 2v in the direction 1. Micro-lenses COl, C31... are shifted by r in the direction I and j.

Micro-lenses C20... are shifted by 2v in the direction j. Micro-lenses C12... are shifted by 2r in the direction I. Micro-lenses C02, C32... are shifted by 2r in the direction I and /. Micro-lenses Cli... are shifted by v in direction and by 2r in direction j. Micro-lenses C22... are shifted byt-in direction j and by 2r in direction Optionally at least one micro-lens in a sub-set is not displaced with respect to the regular lattice: in the illustration the center of micro-lenses COO, C30... belonging respectively to subsets 300,330... are located on the intersections of the regular lattice, preferably at the top left corner of each subset (K=L=0). In practice, the micro-lenses which are not displaced can be any of the lenses of a sub-set. The position of the micro-lens which is not displaced can vary from one sub-set to another. Preferably one micro-lens is not displaced in each of the sub-set and the other micro-lenses of the sub-set have relative displacements with respect to the un-displaced micro-lens. The displaced micro-lenses in each sub-set have displacements which are defined relative to the un-displaced micro-lens.

As described above, the displacements are determined such that the superposition or the clustering of pixels in a reconstructed image is reduced (see also figure 11).

The values k(I,j), 1(1,]) and for the first sub-set of 3x3 micro-lenses illustrated in figure 9 are given in the following table:

JH H H H

o o 0 0 0 0 1 0 0 1 r 2 0 0 2 20 2r o i 1 1 1 1 1 2 �+2r 2 1 1 0 2�+v o 2 2 2 2r 20+2r 1 2 2 0 0+2v 20 2 2 2 1 2�+2r 20+r It is important to note that the relative positions of the micro-lenses in each N by N size sub-set (N being the number of micro-lenses in each directions /,]) is defined modulo N. Thus all displacements modulo N will also be solutions. This means that if a given displacement (p1,ji1) is a solution, then the displacement (y;+aN,p;+bN) with a and b integers is also solution.

Resolution of the projected image The resolution of the projected image can be estimated by computing its maximum sampling step H as for the conventional light-field camera made of a lens array arranged following a square lattice as presented in figure 7.

The projected coordinates (X',Y'), obtained by the proposed micro-lens array with a super-resolution factor N, defines a set of points in the 2D projected/reconstructed.

The set of points according to the proposed micro-lens array is characterized by the maximum sampling step H'. The values of H' have a simple expression for projected coordinates obtained with a disparity having the form: {w}=n/(NilJ) with n and M being positive integers such as o«=n<1tc«=Lr!ivj. The maximum sampling step is equal to H'=ugcd(n,M)/(NM). The largest value of H' is u/N, one recalls that the largest value H obtained for the conventional square lattice micro-lens array is equal to u.

Figure 10 illustrates the normalized f-P/u values with the super-resolution factor N = 2 as a function of the fractional pad of the disparity [w}. The corresponding characteristic parameters of the light-field camera are the one given above. The dashed line recalls the normalized H/u values obtained with a conventional light-field camera equipped with a regular square lattice micro-lens array. Similarly, Figure 11 illustrates the normalized H/u values with the super-resolution factor N = 3.

On can observe in figures 9 and 10 that the resolution of the reconstructed image varies less than a conventional (dash lines) with regular square lattice. Therefore, a more regular resolution is obtained with the proposed micro-lens array. The regularity of the resolution increases with the value of the super-resolution factor N. The present invention also applies for light-field cameras made of an array of lenses and one sensor as illustrated in figure 2. The array of lenses is designed with the equation (29).

It is interesting to note that the maximum sampling step H' is independent from the rotation angle 0 between the pixel lattice and the micro-lens array. Therefore, the rotation angle e has no impact on the resolution of the re-focus image.

It will be understood that the present invention has been described above purely by way of example, and modification of detail can be made within the scope of the invention. Each feature disclosed in the description, and (where appropriate) the claims and drawings may be provided independently or in any appropriate combination.

Claims

CLAIMS1. A micro-lens array for an imaging device comprising micro-lenses located on the micro-lens array relatively to a regular lattice, wherein: the micro-lens array comprises a plurality of micro-lens subsets, each sub-set comprising a two dimensional array of (0) micro lenses, wherein micro-lenses of each sub-set are displaced relative to the regular lattice according to a common pattern, the common pattern defining different displacements for each micro-lens of the sub-sets.
2. The micro-lens array of claim 1 wherein said common pattern defines each displacement as a function of the position (i,j) of each micro-lens within the sub-set.
3. The micro-lens array of claims 1 or 2 wherein said common pattern defines each displacement as a function of the number (0) of micro-lenses in each sub-set.
4. The micro-lens array of any of the preceding claims wherein said common pattern defines displacements in integer multiples of unit displacement vectors.
5. The micro-lens array of claim 4 wherein the magnitude (v) of said unit displacement vectors is a function of focal distance of micro-lenses.
6. The micro-lens array of claims 4 or 5 wherein the magnitude of said unit displacement vectors is a function of the number (0) of micro-lenses in each sub-set.
7. The micro-lens array of any of claims 4 to 6 wherein the multiple of said unit vectors for each micro-lens is a function of the position (i,j)of the micro-lens within the sub-set.
8. The micro-lens array of any of the preceding claims wherein said sub-sets comprise a square array of A5KIV= Q micro-lenses.
9. The micro-lens array of claim 8 wherein said common pattern defines a plurality of possible displacements for each micro-lens, each of said plurality being equivalent in moduloN.
10. The micro-lens array of any of the preceding claims wherein the displacement of at least one micro-lens in each sub-set is zero.
11. The micro-lens array of any of the preceding claims wherein the common pattern and the displacements are independent of the location of the sub-set in the micro-lens array.
12. The micro-lens array according to Claim 4, wherein said integer multiples (k, I) [kQ,j) Ai+Bj+K (modN) are given by. . . I(s,j) Ct+Ej+L (modN) where NxNdefines the size of the sub-set in number of micro-lenses, and the values A,B,C,E being determined as a solution of the equation: gcd((n2 +n(A+E)+AE-BC)modN),N)= 1 Vn E [o,v[
13. An imaging device comprising a micro-lens array according to any of claims 1 to 12 and a photo-sensor having an array of pixels, each micro-lens projecting an image of a scene on an associated region of the photo-sensor forming a micro-image.
14. The imaging device of claim 13, wherein said common pattern defines displacements in integer multiples of unit displacement vectors and wherein the magnitude (r) of said unit displacement vectors is given by v = where f is the micro-lens focal distance, B is the physical size of a sensor pixel, d is the distance between the micro-lens array and the sensor and JJxN defines the size of the sub-set in number of micro-lenses.
15. A micro-lens array for an imaging device comprising micro-lenses located on the micro-lens array relatively to a regular lattice, wherein: the micro-lens array comprises a plurality of micro-lens sub-sets, each sub-set comprising an array of NxN micro lenses, each micro-lens of the subset having a focal distance f, wherein micro-lenses of each sub-set are displaced relative to the regular lattice according to a displacement pattern, said displacement pattern defining the displacement of each micro-lens as integer multiples (Ic, I) of unit vectors, said unit vectors having a magnitude r wherein the magnitude r is a function of f/N.
16. The micro-lens array of claim 15, wherein the displacement of each micro-lens is defined as a function of the position (i,j) of the micro-lens within the sub-set, and Ik(i,j) Ai+Bj+K (modN) wherein said integer multiples (k, I) are given by --CZ+Ej+L (modN) the values A,B,C,E being determined as a solution of the equation: gcd((n2 +n(A+ E)+ AE-BC)modN),N)= I Vn e [O,N[
17. An imaging device comprising a micro-lens array according to claim 15 or claim 16, and a photo-sensor having an array of pixels, arranged so that each micro-lens projects an image of a scene on an associated region of the photo-sensor forming a micro-image, said sensor and said micro-lens array having a separation d and pixels of said photo sensor having size 5 wherein =