WO2018002697A1 - Method for the enhancement of digital image resolution by applying a unique processing of partially overlapping low-resolution images - Google Patents
Method for the enhancement of digital image resolution by applying a unique processing of partially overlapping low-resolution images Download PDFInfo
- Publication number
- WO2018002697A1 WO2018002697A1 PCT/IB2016/055200 IB2016055200W WO2018002697A1 WO 2018002697 A1 WO2018002697 A1 WO 2018002697A1 IB 2016055200 W IB2016055200 W IB 2016055200W WO 2018002697 A1 WO2018002697 A1 WO 2018002697A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- resolution
- image
- point
- images
- points
- Prior art date
Links
- 238000000034 method Methods 0.000 title claims abstract description 36
- 238000012545 processing Methods 0.000 title claims description 19
- 238000004364 calculation method Methods 0.000 claims abstract description 11
- 238000012937 correction Methods 0.000 claims abstract description 6
- 230000015572 biosynthetic process Effects 0.000 claims abstract description 3
- 230000003287 optical effect Effects 0.000 claims abstract description 3
- 230000007246 mechanism Effects 0.000 claims description 3
- 230000001788 irregular Effects 0.000 claims description 2
- 238000003672 processing method Methods 0.000 claims 2
- 239000011159 matrix material Substances 0.000 description 4
- 230000008569 process Effects 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000009877 rendering Methods 0.000 description 2
- 230000009466 transformation Effects 0.000 description 2
- 238000013476 bayesian approach Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 239000003086 colorant Substances 0.000 description 1
- 230000000295 complement effect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 238000011835 investigation Methods 0.000 description 1
- 238000000844 transformation Methods 0.000 description 1
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/73—Deblurring; Sharpening
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10004—Still image; Photographic image
- G06T2207/10012—Stereo images
Definitions
- the invention relates to the field of electronic processing of images (photographs) and specifically - to the enhancement of the electronic image resolution by using the data of several overlapping low-resolution images.
- This invention provides the possibility to create automatically an image of enhanced resolution from several partially overlapping low-resolution images captured at different positions and angles of cameras.
- the camera positions and the spatial surface, which is used for the interrelation between the low-resolution images are reconstructed from the initial low-resolution images.
- the intensity of high-resolution pixels is calculated based on the projections of spatial surface points rendered by them in low-resolution images.
- the 3D surface of a captured object (the earth surface, buildings, crowns of trees, etc.) used for finding of image projections captured in photographs is restored from the initial low-resolution images.
- Reconstruction of higher resolution images is calculated using the overlapping low-resolution areas of photographs.
- Higher resolution photographs are obtained following all the processing steps. These photos correspond to the locking position and angle of the original photos (i.e. they are contain the same images) but has a higher image resolution so the image captured in them becomes more detailed. The image will be more detailed in those areas where the greater number of photographs overlapped.
- the first stage applying the method of photogrammetry a 3D surface model captured in photographs is found: positions and turning angle of cameras are restored, as well as 3D mesh of the object captured in photographs is calculated.
- the second phase the radial undistortion of the lens for all low-resolution images is performed. The size of the correction required is determined using the geometric distortion coefficients obtained during photogrammetry.
- the calculation of the intensities of higher resolution pixels is carried out.
- a 3D point located on a 3D mesh of the captured object is found.
- the projection point is found.
- a quadrangular area of the point environment, the angles of which are also projected to all low-resolution images, is marked around the 3D point. Only those low-resolution images, where the wanted point is well distinguished, are selected for further calculations. From the low-resolution points located in the projections of the 3D point environments an equation system is formed, by resolving which the wanted intensity of the increased resolution point is found.
- Fig. 1 Overlap of (a) the low-resolution, (b) the high-resolution, (c) the high- and low-resolution environment points of the wanted point environment.
- Fig. 2 Environment of the wanted point in different low-resolution images.
- resolution means a parameter of an image reproduced in electronic equipment and describing the amount of image forming points within a length/width/space unit.
- pixel or "point” refers to a square/cubic area of the image occupied by the point.
- pixel intensity or “intensity” is understood as the pixel brightness (for monochrome images) or as the brightness of the original colours of the pixel colour (red, green and blue) (for colour images).
- low-resolution image or “lower- resolution image” means a photograph of an object, where the captured object is clearly visible but fine details of the object are rendered abstractly.
- high-resolution image or “enhanced resolution image” means an improved photograph of the same object, in which fine details of the object are rendered in greater detail (eg., you can read the note, which was not possible to read in the original low-resolution images).
- rendering and processing of images can be carried out in electronic devices capable of working with images, such as computer, video camera, video player, video rendering device (e.g., TV, monitor) .
- electronic devices capable of working with images such as computer, video camera, video player, video rendering device (e.g., TV, monitor) .
- video rendering device e.g., TV, monitor
- image electronic devices capable of processing of digital information variable memory modules, at least one processor, input and output devices, etc.
- High-resolution images render the captured object in greater detail and can therefore be used solving more sophisticated tasks.
- capturing of the objects in high resolution is limited by various reasons: technical (optics of photographic device does not zoom the image to the correct level), legal (resolution of satellite and infrared photographs is limited by law due to their strategic importance), etc.
- image resolution enhancement techniques using the lower-resolution images. Part of the methods enhance the image resolution by using a single low-resolution image, other methods do the same by using two or more low-resolution images.
- the image resolution is increased by various means of interpolation
- low-resolution images are identified by the found angular points and there are attempts using algorithmic means to identify minor changes in camera position and possibilities of their compensation.
- algorithmic means to identify minor changes in camera position and possibilities of their compensation.
- Such methods may additionally use other image processing techniques, such as noise removal, sharpness enhancement, etc.
- Previously discussed resolution enhancement methods treat the captured image as a plane and is therefore required that the original low-resolution images would be captured from a single point (the unchanging position of the camera).
- Method presented in this description for the increase of the image resolution uses the original low-resolution images captured from different positions of the camera and at different angles of inclination in respect of the captured object.
- the formation of enhanced resolution images consists of three enhancement stages.
- the 3D mesh of the object is restored from the captured photographs.
- the 3D mesh is restored using the methodology of photogrammetry by identifying the position and turning angles of the camera in respect of the captured object.
- the correction of optical distortions is performed using the values of geometric distortions obtained during photogrammetry.
- the intensities of points forming the image of enhanced resolution are determined.
- the present invention is new because by the use of the said three image resolution enhancement stages the images of enhanced resolution are obtained, when original low- resolution images were captured at different positions and inclination angles of the camera.
- the descriptive method works better if the rays are drawn through each point of the surface and positions of the cameras seeing these points are at the angle greater than 5 degrees.
- Each of the original low- resolution images V t is changed to an image of enhanced resolution V/ .
- the enhancement of the resolution is performed by a freely selected value but it is recommended not greater than 1.5 times, because when increasing more than 1.5 times, the results obtained at the time of the resolution enhancement calculation can be unstable.
- Fig. 1 shows a possible enhancement of the resolution by the aforementioned 1.5 times: a) part shows the side of an image consisting of two low-resolution points; b) part shows the side of an image consisting of three points of enhanced resolution; c) shows the overlap of low-resolution and higher-resolution images.
- a) part shows the side of an image consisting of two low-resolution points; b) part shows the side of an image consisting of three points of enhanced resolution; c) shows the overlap of low-resolution and higher-resolution images.
- each pixel of enhanced resolution is being processed separately, i.e., the entire functional processing sequence is applied to each formed pixel of enhanced-resolution V/ .
- An auxiliary Z plane is drawn through the point X.
- the direction of the normal of the plane Z is opposite to the direction of the radius R t .
- a square environment S of the point Q l is created.
- the square environment S of the point Q l is made of high-resolution pixels W u l v e S .
- the number of pixels, i.e. the size of the environment, is a matter of free choice.
- Fig. 1, b) shows the square environment of the point, the side length of which is equal to three high -resolution points.
- a new set of images K is formed from the former set of images L.
- the image from the set L is transferred to a new set of images K, if two conditions are met:
- the intensity values G u k v are calculated in the projection points W v for all lower-resolution images V k e K . For the determination of the intensity bicubic interpolation is used.
- the processing mechanism is repeated for other remaining points of the image V/ .
- Fig. 2 shows the environment of the wanted point in different low-resolution images.
- each pixel is understood as the mean intensity of the four-square surface area occupied by this pixel.
- Lower-resolution pixels overlap imperfectly, so several high-resolution pixels fall into the foursquare of each pixel (some of which fall partially).
- the value of each lower-resolution point can be calculated using the weighted average formula:
- g tj the intensity of the low-resolution pixel
- m the value of the side of the point environment S
- g u ' v - the intensity of the enhanced resolution pixel
- w u ' v - the weight of each high-resolution pixel is calculated as the ratio of g u ' v overlapping area with g ⁇ j .
- Fig. 1 shows the environment of the point in low-resolution image (part a) and high-resolution image (part b) as well as the overlapping areas of high- and low-resolution points (part c).
- x the intensities of all wanted points of high-resolution image
- b the intensities of the original low-resolution pixels
- a - weighting matrix corresponding to the area of each high-resolution pixel overlapping with low resolution pixels.
- Fig. shows the example where the environment of high-resolution points consists of 9 unknown points and, therefore, the equation system must consist of at least 9 equations.
- the environment of low-resolution points consists of 4 points.
- the matrix A will be the same for all calculated points (or it may be extended if more than 3 images will be used for calculations), and the vector b will be filled with the values of the environment points of the wanted point. Below are presented restrictions for the resolution enhancement mechanism.
- Resolution enhancement in individual locations of a formed large high-resolution photograph will be different.
- the level of detail of the enhanced resolution image strongly depends on the amount of overlapping photographs and the total point density in the environment of the wanted point i.e., the surface of the object in the photographs taken from a closer distance has a higher resolution than in the photographs taken from a greater distance and, therefore, in an attempt to increase the resolution of the photographs taken from a closer distance by using the overlapping photographs taken from a greater distance, the result will be unstable. If the coordinates of the points of the overlapping photographs will highly correlate with each other the photographs might complement each other insufficiently because there will be no new information that could be used to enhance the resolution of the restored/formed photograph. However, due to the roughness of the investigated surface and the change in the position of the camera in other areas of the same photograph, correlation of the points will be lower and as a result the image of the enhanced resolution will be more detailed.
Landscapes
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Image Processing (AREA)
Abstract
The object of the present invention is a method for the composition of high- resolution image performing automated enhancement of image resolution from several partially overlapping low-resolution images captured at different positions and angles of photo cameras Formation of the enhanced resolution image includes the 3 main stages: in the first stage a 3D model of the surface captured in photographs is identified: positions and turning angles of cameras are restored, as well as the 3D surface of the object captured in photographs is calculated; in the second stage a correction of optical distortions of all lower-resolution images is performed, the required amount of correction is determined using the geometric distortion coefficients received in the first phase; in the third stage, the calculation of the intensities of enhanced resolution pixels is carried out - to this end, for each new enhanced resolution point a 3D point located on a 3D mesh of the captured object is found, when projecting the 3D point to all lower-resolution points, the projections of the 3D point are found, a four-square area of the point environment, the angles of which are also projected to all lower-resolution images, is marked around the 3D point, from the lower-resolution points located in the projections of the 3D point environments an equation system is formed, by resolving which the wanted intensity of the enhanced resolution point is found. This paper describes the principles for the enhancement of the monochrome image resolution, but it can also be used for colour images. In the case of colour images the colour of each enhanced resolution pixel is determined by adapting the described method to each original colour (red, green and blue) individually.
Description
METHOD FOR THE ENHANCEMENT OF DIGITAL IMAGE RESOLUTION BY APPLYING A UNIQUE PROCESSING OF PARTIALLY OVERLAPPING LOW- RESOLUTION IMAGES
FIELD OF THE INVENTION
The invention relates to the field of electronic processing of images (photographs) and specifically - to the enhancement of the electronic image resolution by using the data of several overlapping low-resolution images.
DESCRIPTION OF RELATED ART
This invention provides the possibility to create automatically an image of enhanced resolution from several partially overlapping low-resolution images captured at different positions and angles of cameras. During the process the camera positions and the spatial surface, which is used for the interrelation between the low-resolution images, are reconstructed from the initial low-resolution images. After geometric corrections of initial low-resolution images, the intensity of high-resolution pixels is calculated based on the projections of spatial surface points rendered by them in low-resolution images.
Below is considered the state of the art, i.e., information, which is publicly available at the moment.
There is known the patent US8582922B2 (published on 12 November 2013), in which the method for the image resolution enhancement using two or more low- resolution images is presented. Method for the resolution enhancement is based on the calculation of the high-resolution pixel values of the low-resolution pixels using a principle of weighted interpolation calculation. This calculation method makes no reference about the images that were captured at different camera rotation angles in respect of the object, three-dimensional surface is not applied to enhance the resolution, therefore, images where the object is rendered from one plane or planes, which are only at few-degree angles to the said object plane, can be used for such a method.
There is also known the patent application US20040170340A1 (published on 2 September 2004), which presents the method of image resolution enhancement, where the image of enhanced resolution is obtained from at least two lower-resolution images using a Bayesian approach. However, such resolution enhancement method can be used only when the orientation of the original low-resolution images and their turning at an object are very close to each other.
There is also known the document US20030012457 (published on 12 January 2003), where a flat higher resolution image is formed from multiple low-resolution images of panels. The basic principle of this method is that all information is collected into a single image and later unnecessary details are cropped according to a certain pattern.
In most cases, at the technical level documents most often associated with image resolution enhancement techniques that are oriented to computing and further processing of multiple low-resolution images applying different techniques are found. Also, the documents usually consider images restored in two-dimensional plane, which coincides with the image plane. Because the faster development of image processing and composition technologies as a science and application trend started only in the last decade (especially in the last years), so it can be said that at the moment image processing technologies are going through a breakthrough. This is due to the fact that the current computer hardware is able to process quickly large amounts of information without oversimplification of the original images, to say, in pursuance of better results it allowed to work with images in a complex way by applying various sophisticated methodologies.
The vast majority of image resolution enhancement techniques in the current technical level are based on the reconstruction of small changes in the position of the camera (shift and turn). Search for positions is carried out in 2D space coinciding with the image plane. Once the optimum positions have been found the increased resolution has been calculated for the entire area of the restored image at the same time. However, in case of major changes in camera position and locking direction, homography-based methods do not work because the images can no longer overlap due to the prospective transformation of objects captured in them at a different angle.
The method described in the present invention performs entry of images during the photogrammetry process. With the use of the key-points and 3D model bundle adjustment the image interrelationships and camera positions in 3D space are found. Due to a different locking position of photographs the objects captured in each overlapping image are visible in different places of the image, i.e., it is not possible to carry out the increase of the resolution for the entire image at once, because the images can not overlap even after the necessary transformations. This problem is solved by calculating each pixel of increased resolution separately. Sequence of steps applied in the present invention restores all points of higher resolution by finding to each of them a respective point on the 3D surface, marking a local surface plane in it and projecting this small square area to all low- resolution images where this 3D surface location is locked. According to the obtained projections, in each selected image are found the intensities of the projection points, which are then used for the conclusion of the system of equations. Having solved the system of equations, the obtained central environmental intensity of the investigation point is saved in an image of increased resolution.
SUMMARY OF THE INVENTION
In the course of 3D image conversion, the 3D surface of a captured object (the earth surface, buildings, crowns of trees, etc.) used for finding of image projections captured in photographs is restored from the initial low-resolution images. Reconstruction of higher resolution images is calculated using the overlapping low-resolution areas of photographs. Higher resolution photographs are obtained following all the processing steps. These photos correspond to the locking position and angle of the original photos (i.e. they are contain the same images) but has a higher image resolution so the image captured in them becomes more detailed. The image will be more detailed in those areas where the greater number of photographs overlapped.
When forming a higher-resolution image three main stages are applied:
In the first stage, applying the method of photogrammetry a 3D surface model captured in photographs is found: positions and turning angle of cameras are restored, as well as 3D mesh of the object captured in photographs is calculated.
In the second phase, the radial undistortion of the lens for all low-resolution images is performed. The size of the correction required is determined using the geometric distortion coefficients obtained during photogrammetry.
In the third stage, the calculation of the intensities of higher resolution pixels is carried out. To this end, for each new higher-resolution point a 3D point located on a 3D mesh of the captured object is found. When projecting the 3D point to all low-resolution images, the projection point is found. A quadrangular area of the point environment, the angles of which are also projected to all low-resolution images, is marked around the 3D point. Only those low-resolution images, where the wanted point is well distinguished, are selected for further calculations. From the low-resolution points located in the projections of the 3D point environments an equation system is formed, by resolving which the wanted intensity of the increased resolution point is found.
BRIEF DESCRIPTION OF THE DRAWINGS
Fig. 1. Overlap of (a) the low-resolution, (b) the high-resolution, (c) the high- and low-resolution environment points of the wanted point environment.
Fig. 2. Environment of the wanted point in different low-resolution images.
DESCRIPTION OF THE PREFERRED EMBODIMENTS
For the purposes of this description the term "resolution" means a parameter of an image reproduced in electronic equipment and describing the amount of image forming points within a length/width/space unit.
For the purposes of this description the term "pixel" or "point" refers to a square/cubic area of the image occupied by the point.
For the purposes of this description the term "pixel intensity" or "intensity" is understood as the pixel brightness (for monochrome images) or as the brightness of the original colours of the pixel colour (red, green and blue) (for colour images).
For the purposes of this description the term "low-resolution image" or "lower- resolution image" means a photograph of an object, where the captured object is clearly visible but fine details of the object are rendered abstractly.
For the purposes of this description the term "high-resolution image" or "enhanced resolution image" means an improved photograph of the same object, in which fine details of the object are rendered in greater detail (eg., you can read the note, which was not possible to read in the original low-resolution images).
In case of the present invention, rendering and processing of images can be carried out in electronic devices capable of working with images, such as computer, video camera, video player, video rendering device (e.g., TV, monitor) . For the presented processing of the image electronic devices capable of processing of digital information (various memory modules, at least one processor, input and output devices, etc.) are needed.
High-resolution images render the captured object in greater detail and can therefore be used solving more sophisticated tasks. However, capturing of the objects in high resolution is limited by various reasons: technical (optics of photographic device does not zoom the image to the correct level), legal (resolution of satellite and infrared photographs is limited by law due to their strategic importance), etc. There are a lot of different image resolution enhancement techniques using the lower-resolution images. Part of the methods enhance the image resolution by using a single low-resolution image, other methods do the same by using two or more low-resolution images. In the first case, the image resolution is increased by various means of interpolation, and in the second case, low-resolution images are identified by the found angular points and there are attempts using algorithmic means to identify minor changes in camera position and possibilities of their compensation. Such methods may additionally use other image processing techniques, such as noise removal, sharpness enhancement, etc.
Previously discussed resolution enhancement methods treat the captured image as a plane and is therefore required that the original low-resolution images would be captured from a single point (the unchanging position of the camera). Method presented in this description for the increase of the image resolution uses the original low-resolution images captured from different positions of the camera and at different angles of inclination in
respect of the captured object. In the present invention the formation of enhanced resolution images consists of three enhancement stages.
In the first stage of the resolution enhancement the 3D mesh of the object is restored from the captured photographs. The 3D mesh is restored using the methodology of photogrammetry by identifying the position and turning angles of the camera in respect of the captured object.
In the second stage, the correction of optical distortions is performed using the values of geometric distortions obtained during photogrammetry.
In the third stage, the intensities of points forming the image of enhanced resolution are determined.
The present invention is new because by the use of the said three image resolution enhancement stages the images of enhanced resolution are obtained, when original low- resolution images were captured at different positions and inclination angles of the camera. The descriptive method works better if the rays are drawn through each point of the surface and positions of the cameras seeing these points are at the angle greater than 5 degrees.
The image resolution enhancement stages are described below in more detail:
The enhancement of the resolution is performed from the N low-resolution images forming a set of low-resolution images L = ty0,Vl, ... ,VN ). Each of the original low- resolution images Vt is changed to an image of enhanced resolution V/ . When changing the image resolution, the geometric dimensions of the image remain unchanged, while the number of image-forming points increases. The enhancement of the resolution is performed by a freely selected value but it is recommended not greater than 1.5 times, because when increasing more than 1.5 times, the results obtained at the time of the resolution enhancement calculation can be unstable.
Fig. 1 shows a possible enhancement of the resolution by the aforementioned 1.5 times: a) part shows the side of an image consisting of two low-resolution points; b) part shows the side of an image consisting of three points of enhanced resolution; c) shows the overlap of low-resolution and higher-resolution images.
At every stage of resolution enhancement each pixel of enhanced resolution is being processed separately, i.e., the entire functional processing sequence is applied to each formed pixel of enhanced-resolution V/ .
Below the separate phases (steps) of functional processing sequences are described in more detail:
1. Radius Rt crossing the point Ql of a higher-resolution image, which we desire to calculate, is drawn through the center Ct of the lower-resolution image Vt . At the intersection of the radius and the reconstructed three-dimensional surface the point X is marked.
2. An auxiliary Z plane is drawn through the point X. The direction of the normal of the plane Z is opposite to the direction of the radius Rt .
3. Around the point Ql , located in the calculated high-resolution image, a square environment S of the point Ql is created. The square environment S of the point Ql is made of high-resolution pixels Wu l v e S . The number of pixels, i.e. the size of the environment, is a matter of free choice. Fig. 1, b) shows the square environment of the point, the side length of which is equal to three high -resolution points.
4. Radiuses are drawn through the neighbouring points Wu l v e S of the point Ql , at the intersection of which with the auxiliary plane Z three-dimensional points Tuv are marked.
5. At this stage, superposition of a two-dimensional image with three- dimensional image takes place. The auxiliary plane Z is tilted/turned so that at the point X its normal Nz would coincide with the normal Nx of the three-dimensional surface. Points Tuv are turned together with the plane Z .
6. Using projection matrices calculated during at the time of photogrammetry, three-dimensional points Tuv are designed and their projections WM* in each lower-resolution image Vk e L are derived. Because of the tilt of the plane Z in respect of the three-dimensional surface, in each image the projections of points W^v will be rendered as irregular quadrangles, which are the best correspondent with the projection of the area of a three-
dimensional surface in each image.
7. A new set of images K is formed from the former set of images L. The image from the set L is transferred to a new set of images K, if two conditions are met:
a) All projections W v fall into the image Vk , i.e., VW «, v : 0 < u < wv vk ,0 < v < h v,k , here w v,k and h vk is the width and height of the image Vk , respectively.
b) The normal Nx of the point X forms an angle of less than 60° with a radius R, of the image V, , i.e., arccos <— . Where
IHIHJ 3
n - the normal Nx of the point X written in a vector form, and r - radius of the image Vk - Rk written in a vector form.
8. The intensity values Gu k v are calculated in the projection points W v for all lower-resolution images Vk e K . For the determination of the intensity bicubic interpolation is used.
9. According to the description given in chapter "Equation System Composition" and obtained intensity values Gu k v of low-resolution images a system of normal equations for the calculation of the intensities Gu'v of the enhanced resolution pixels is formed.
10. Having solved the equation system, the medium element xi i = M /2 corresponding to the intensity Gu'v of the central enhanced resolution point is selected from the vector x and entered into the point Q' in the enhanced resolution image V/ .
11. The processing mechanism is repeated for other remaining points of the image V/ .
Fig. 2 shows the environment of the wanted point in different low-resolution images.
Below is provided an explanation concerning (composition) of the aforementioned equation system:
Each pixel is understood as the mean intensity of the four-square surface area occupied by this pixel. Lower-resolution pixels overlap imperfectly, so
several high-resolution pixels fall into the foursquare of each pixel (some of which fall partially). In general, the value of each lower-resolution point can be calculated using the weighted average formula:
Here gtj - the intensity of the low-resolution pixel, m - the value of the side of the point environment S (Fig. 1. shows the example, the side of which is equal to 3), gu'v - the intensity of the enhanced resolution pixel, wu'v - the weight of each high-resolution pixel is calculated as the ratio of gu'v overlapping area with g{j .
Value of high resolution point Qx l y is calculated from the projection of this point environment S in each low-resolution image. Fig. 1 shows the environment of the point in low-resolution image (part a) and high-resolution image (part b) as well as the overlapping areas of high- and low-resolution points (part c).
Using the formula (1), the normal equation system is concluded, which is written using matrix form:
Ax = b
Here x - the intensities of all wanted points of high-resolution image, b - the intensities of the original low-resolution pixels, A - weighting matrix corresponding to the area of each high-resolution pixel overlapping with low resolution pixels.
Taking into account the amount of high-resolution points (the unknowns) falling into the environment of the point the required quantity of equations in the system is determined. The number of equations must be equal to or greater than the number of wanted points (the unknowns). Fig. shows the example where the environment of high-resolution points consists of 9 unknown points and, therefore, the equation system must consist of at least 9 equations. The environment of low-resolution points consists of 4 points. To achieve the required amount of equations, we have to use the data of at least
3 low resolution images, thus forming a system of 12 equations, each of which will have 9 unknowns.
On the basis of the example presented in Fig. 1 and the formula (1), as well as the available intensity values Gu k v of low-resolution pixels of the wanted point, a normal equation system is concluded for the calculation of the intensities Gu'v of the high-resolution pixels. Matrix A is filled with the ratios of the overlapping point areas. Each equation corresponds to the intensity estimate of a single low-resolution point made up of the estimates of unknown high-resolution points according to the formula (1). Vector b is filled with the intensity values Gu k v of low-resolution points. In the example shown the first 4 equations describe the environment points of the first low- resolution image, the next 4 - of the second, and the last 4 - of the third image:
Having solved the equation system, in the vector x we find the intensities Gu'v of the enhanced resolution points. x— [ j J , G12 , G13 , G21 , G22 , G23 , G31 , G32 , G33 ]
The matrix A will be the same for all calculated points (or it may be extended if more than 3 images will be used for calculations), and the vector b will be filled with the values of the environment points of the wanted point.
Below are presented restrictions for the resolution enhancement mechanism.
Resolution enhancement in individual locations of a formed large high-resolution photograph will be different. The level of detail of the enhanced resolution image strongly depends on the amount of overlapping photographs and the total point density in the environment of the wanted point i.e., the surface of the object in the photographs taken from a closer distance has a higher resolution than in the photographs taken from a greater distance and, therefore, in an attempt to increase the resolution of the photographs taken from a closer distance by using the overlapping photographs taken from a greater distance, the result will be unstable. If the coordinates of the points of the overlapping photographs will highly correlate with each other the photographs might complement each other insufficiently because there will be no new information that could be used to enhance the resolution of the restored/formed photograph. However, due to the roughness of the investigated surface and the change in the position of the camera in other areas of the same photograph, correlation of the points will be lower and as a result the image of the enhanced resolution will be more detailed.
In the areas of the photographs abstract objects (sky, shiny surface, rippled water, etc.) are captured or there are no sufficient amount of overlapping photographs, the resolution is increased from the original low-resolution pixels using a bicubical interpolation method.
Although this paper describes the principles for the enhancement of the monochrome image resolution, but this method can also be used for colour images. In the case of colour images the colour of each enhanced resolution pixel is determined by adapting the described method to each original colour (red, green and blue) individually.
The above description of the preferred embodiments is provided in order to illustrate and describe the present invention. This is not an exhaustive or limiting description, seeking to determine the exact form or embodiment. The above description should be considered more like an illustration, rather than a limitation. It is evident that numerous modifications and variations may be obvious to the specialists of that field. Embodiment is chosen and described so that the experts of this field would clarify in the best way the principles of this invention and the best practical application for various embodiments with various modifications suitable for a particular use or application of the embodiment. It is intended that the scope of the invention is defined in the claim appended thereto and its
equivalents, where all of the said terms have meaning within the widest range, unless indicated otherwise.
In the embodiment options described by the specialists of this field changes can be created without deviations from the scope of this invention as specified in the following claim.
Claims
1. Automatic composition and processing method of partially overlapping low- resolution images captured from different positions and angles of the photo camera with a view to form a high-resolution image therefrom,
comprising:
the collection of information from low-resolution images;
the processing mechanism; and
the formation of the enhanced resolution image;
characterised in that
this processing method includes the following three functional processing stages:
in the first resolution enhancement stage using the method of photogrammetry from low-resolution photographs the camera positions and turning angle in the space are found, and a 3D mesh of the captured object is reproduced;
in the second resolution enhancement stage a correction of optical distortions is performed using the data of geometric distortions obtained during the photogrammetry;
in the third stage the intensity values of the points making up the enhanced resolution image are calculated.
2. Method for automatic composition and processing of digital images according to claim 1 characterised in that the said 3 processing stages consist of the following functional steps:
a) radius Rt crossing the point Ql of a high-resolution image, which we desire to calculate, is drawn through the center Cl of the low-resolution image Vl , where at the intersection of the radius and the reconstructed three- dimensional surface the point X is marked;
b) an auxiliary plane Z is drawn through the point X, where the direction of the normal of the plane Z is opposite to the direction of the radius Rt ;
c) around the point Ql , located in the high-resolution image, a square environment S of the point Ql is created, where the square environment S of the point Ql is made of high-resolution pixels Wu l v e S , where the number of pixels, i.e. the size of the environment, is a matter of free choice;
d) radiuses are drawn through the neighbouring points Wu l v e S of the point
Ql , at the intersection of which with the auxiliary plane Z three-dimensional points Tuv are marked;
e) the superposition of a two-dimensional image with three-dimensional image takes place, where the auxiliary plane Z is tilted/turned so that at the point X its normal Nz would coincide with the normal Nx of the three- dimensional surface, and points Tuv are tilted/turned so that they would remain at the same locations of the plane Z ;
f) using the photogrammetry results, three-dimensional points Tuv are designed and their projections WM* in each lower-resolution image Vk e L are derived, where because of the tilt of the plane Z in respect of the three- dimensional surface, in each image the projections of points W v will be rendered as irregular quadrangles, which are the best correspondent with the projection of the area of a three-dimensional surface in each image;
g) a new set of images K is formed from the former set of images L, the image from the set L is transferred to a new set of images K, if two conditions are met:
aa) all projections W v fall into the image Vk , i.e., VW, : 0 < u≤ wv ,0 < v < h, , here w, and h h is the width and height of the image Vk , respectively;
bb) the normal Nx of the point X forms an angle of less than 60° with a radius Rk of the image Vk or , where n - the
normal Nx of the point X written in a vector form, and r - radius of the image Vk - Rk written in a vector form;
h) The intensity values Gu k v are calculated in the projection points Wu* for all
lower-resolution images Vk e K , where bicubic interpolation method is used for the determination of the intensities;
i) based on the obtained intensity values Gu k v of the lower-resolution images a system of normal equations for the calculation of the intensities Gu'v of the enhanced resolution pixels is formed;
j) having solved the equation system, the medium element xi i = M /2 corresponding to the intensity Gu'v of the central enhanced resolution point is selected from the vector x and entered into the point Q' within the enhanced resolution image.
3. Method for automatic composition and processing of digital images according to claim 2 characterised in that the said functional steps from a) to j) are repeated for the remaining points of the image V .
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
EP16779181.3A EP3479342A1 (en) | 2016-06-29 | 2016-08-31 | Method for the enhancement of digital image resolution by applying a unique processing of partially overlapping low-resolution images |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
LTLT2016-509 | 2016-06-29 | ||
LT2016509A LT6525B (en) | 2016-06-29 | 2016-06-29 | Method for the enhancement of digital image resolution by applying a unique processing of partially overlaping low resolution images |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2018002697A1 true WO2018002697A1 (en) | 2018-01-04 |
Family
ID=57124067
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/IB2016/055200 WO2018002697A1 (en) | 2016-06-29 | 2016-08-31 | Method for the enhancement of digital image resolution by applying a unique processing of partially overlapping low-resolution images |
Country Status (3)
Country | Link |
---|---|
EP (1) | EP3479342A1 (en) |
LT (1) | LT6525B (en) |
WO (1) | WO2018002697A1 (en) |
Cited By (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN109493419A (en) * | 2018-11-09 | 2019-03-19 | 武大吉奥信息技术有限公司 | A kind of method and device of oblique photograph data acquisition digital surface model |
CN109731171A (en) * | 2019-01-11 | 2019-05-10 | 王伟 | Medical space security protection system |
CN110176035A (en) * | 2019-05-08 | 2019-08-27 | 深圳市易尚展示股份有限公司 | Localization method, device, computer equipment and the storage medium of index point |
CN111729307A (en) * | 2020-07-30 | 2020-10-02 | 腾讯科技(深圳)有限公司 | Virtual scene display method, device, equipment and storage medium |
WO2022159093A1 (en) * | 2021-01-22 | 2022-07-28 | Uab Diabetis | System, method, and apparatus for temperature asymmetry measurement of body parts |
US11453891B2 (en) | 2017-05-10 | 2022-09-27 | The Regents Of The University Of California | Directed editing of cellular RNA via nuclear delivery of CRISPR/CAS9 |
US11667903B2 (en) | 2015-11-23 | 2023-06-06 | The Regents Of The University Of California | Tracking and manipulating cellular RNA via nuclear delivery of CRISPR/CAS9 |
Citations (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US8582922B2 (en) * | 2010-11-11 | 2013-11-12 | Mitsubishi Electric Corporation | Image processing device and method |
Family Cites Families (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7239428B2 (en) | 2001-06-11 | 2007-07-03 | Solectronics, Llc | Method of super image resolution |
JP3729404B2 (en) | 2002-06-27 | 2005-12-21 | 東海ゴム工業株式会社 | Anti-vibration bush |
-
2016
- 2016-06-29 LT LT2016509A patent/LT6525B/en unknown
- 2016-08-31 WO PCT/IB2016/055200 patent/WO2018002697A1/en unknown
- 2016-08-31 EP EP16779181.3A patent/EP3479342A1/en active Pending
Patent Citations (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US8582922B2 (en) * | 2010-11-11 | 2013-11-12 | Mitsubishi Electric Corporation | Image processing device and method |
Non-Patent Citations (4)
Title |
---|
DANIEL HERRERA CASTRO: "FROM IMAGES TO POINT CLOUDS PRACTICAL CONSIDERATIONS FOR THREE- DIMENSIONAL COMPUTER VISION", 1 January 2015 (2015-01-01), XP055362842, Retrieved from the Internet <URL:http://jultika.oulu.fi/files/isbn9789526208534.pdf> [retrieved on 20170407] * |
LIPSKI CHRISTIAN ET AL: "Correspondence and Depth-Image Based Rendering a Hybrid Approach for Free-Viewpoint Video", IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, IEEE SERVICE CENTER, PISCATAWAY, NJ, US, vol. 24, no. 6, 1 June 2014 (2014-06-01), pages 942 - 951, XP011550125, ISSN: 1051-8215, [retrieved on 20140602], DOI: 10.1109/TCSVT.2014.2302379 * |
YASUTAKA FURUKAWA ET AL: "Multi-View Stereo: A Tutorial3 Structure from Motion", 30 May 2015 (2015-05-30), XP055362693, Retrieved from the Internet <URL:http://www.cse.wustl.edu/~furukawa/papers/fnt_mvs.pdf> [retrieved on 20170406] * |
ZINGER S ET AL: "Free-viewpoint depth image based rendering", JOURNAL OF VISUAL COMMUNICATION AND IMAGE REPRESENTATION, ACADEMIC PRESS, INC, US, vol. 21, no. 5-6, 1 July 2010 (2010-07-01), pages 533 - 541, XP027067829 * |
Cited By (10)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11667903B2 (en) | 2015-11-23 | 2023-06-06 | The Regents Of The University Of California | Tracking and manipulating cellular RNA via nuclear delivery of CRISPR/CAS9 |
US11453891B2 (en) | 2017-05-10 | 2022-09-27 | The Regents Of The University Of California | Directed editing of cellular RNA via nuclear delivery of CRISPR/CAS9 |
CN109493419A (en) * | 2018-11-09 | 2019-03-19 | 武大吉奥信息技术有限公司 | A kind of method and device of oblique photograph data acquisition digital surface model |
CN109731171A (en) * | 2019-01-11 | 2019-05-10 | 王伟 | Medical space security protection system |
CN109731171B (en) * | 2019-01-11 | 2021-04-13 | 宋小梅 | Safety protection system for medical place |
CN110176035A (en) * | 2019-05-08 | 2019-08-27 | 深圳市易尚展示股份有限公司 | Localization method, device, computer equipment and the storage medium of index point |
CN110176035B (en) * | 2019-05-08 | 2021-09-28 | 深圳市易尚展示股份有限公司 | Method and device for positioning mark point, computer equipment and storage medium |
CN111729307A (en) * | 2020-07-30 | 2020-10-02 | 腾讯科技(深圳)有限公司 | Virtual scene display method, device, equipment and storage medium |
CN111729307B (en) * | 2020-07-30 | 2023-08-22 | 腾讯科技(深圳)有限公司 | Virtual scene display method, device, equipment and storage medium |
WO2022159093A1 (en) * | 2021-01-22 | 2022-07-28 | Uab Diabetis | System, method, and apparatus for temperature asymmetry measurement of body parts |
Also Published As
Publication number | Publication date |
---|---|
LT6525B (en) | 2018-05-10 |
EP3479342A1 (en) | 2019-05-08 |
LT2016509A (en) | 2018-01-10 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
WO2018002697A1 (en) | Method for the enhancement of digital image resolution by applying a unique processing of partially overlapping low-resolution images | |
US8855441B2 (en) | Method and apparatus for transforming a non-linear lens-distorted image | |
US9959653B2 (en) | Mosaic oblique images and methods of making and using same | |
US6252577B1 (en) | Efficient methodology for scaling and transferring images | |
US8749694B2 (en) | Methods and apparatus for rendering focused plenoptic camera data using super-resolved demosaicing | |
US8803918B2 (en) | Methods and apparatus for calibrating focused plenoptic camera data | |
US8860833B2 (en) | Blended rendering of focused plenoptic camera data | |
US8665341B2 (en) | Methods and apparatus for rendering output images with simulated artistic effects from focused plenoptic camera data | |
US8675997B2 (en) | Feature based image registration | |
US8217956B1 (en) | Method and apparatus for rendering spherical panoramas | |
US20150170405A1 (en) | High resolution free-view interpolation of planar structure | |
EP3640886B1 (en) | Homography rectification | |
WO2003047234A2 (en) | System and method for providing multi-sensor super-resolution | |
KR20150050347A (en) | Imaging device and computer performing super-resolution image processing and performing method thereof | |
JP4094303B2 (en) | 3D information acquisition apparatus and 3D information acquisition method | |
CN116402934A (en) | Automatic texture mapping method, terminal and storage medium for three-dimensional reconstruction | |
Hadhoud et al. | New trends in high resolution image processing | |
CN110503603A (en) | A method of it is up-sampled based on guidance and obtains light field weight focus image | |
JP4925112B2 (en) | Free viewpoint image generation method | |
JP7514125B2 (en) | Image processing device and program thereof | |
van der Walt et al. | A polygon-based interpolation operator for super-resolution imaging | |
Borkowski | Digital image processing in measurement of ice thickness on power transmission lines: a Rough Set approach | |
TWI738510B (en) | Image overlay method of semiconductor element | |
CN117710209A (en) | Image processing method, apparatus, electronic device, and computer-readable storage medium | |
Matviychuk et al. | Exploring the manifold of image patches |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 16779181 Country of ref document: EP Kind code of ref document: A1 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
ENP | Entry into the national phase |
Ref document number: 2016779181 Country of ref document: EP Effective date: 20190129 |