Image fusion method based on nonlinear weight
Technical Field
The invention relates to image processing, in particular to a method for fusing images based on nonlinear weight.
Background
The gradual image fusion algorithm is a linear pixel-level image fusion algorithm, generates a weight model by linear weight according to an overlapping area of images to be spliced, generates pixels of the overlapping area according to the linear weight model, has higher efficiency and is more suitable for real-time application compared with the optimal suture line image fusion algorithm.
At present, scholars at home and abroad have already carried out a lot of research and improvement on gradual evolutionary algorithms.
Wandan, Liuhui et al have proposed a trigonometric function weighted image stitching algorithm, this algorithm has proposed a trigonometric function weighted image fusion model, through employing this model, have reduced the image stitching trace effectively, has dealt with the problem that the boundary is sharp well, make the image transition part comparatively smooth, but the complicated picture overlap part of stitching still has comparatively obvious ghost image phenomenon. (Wandan, Liuhui, Like, etc.. an image stitching algorithm of trigonometric function weight [ J ] infrared technique, 2017,39 (1): 53-57.)
Wangshuai, Sunwei et al propose a weighted fusion image splicing algorithm based on brightness unification, which considers the problem that images with different brightness may generate excessively fast brightness in the image fusion stage, before image splicing and image fusion, the two images are subjected to brightness preprocessing, the images in the original RGB color space are converted into the images in the l alpha beta color space, brightness linear transformation is performed on l components of the images, and then the images are converted into the RGB images again, so that the unification of the brightness of the two images is realized. Although the curve weight image fusion model provided by the method improves the ghost situation to a certain extent, the ghost phenomenon visible to the naked eye still occurs. (King, Sunwei, Zingiberensis, etc. based on the unified weighted fusion image mosaic algorithm of brightness [ J ] Shandong science, 2014,27 (3): 44-50.)
Xiu Chunbo, Ma Yunfei proposes an improved gradual image fusion model with square weight, the model carries out square operation on traditional linear weight, the model effectively optimizes excessive color of image fusion, the excessive color of the middle area of a spliced image is more natural, but the defect of the model is obvious, the weight of the model is always greater than the linear weight, and the ghost phenomenon is more severe. When the method is used for splicing simple images, the effect is good, but when the method is used for complex images, double images are very obvious. (Xiu Chunbo, Ma)Yunfei.Image Stitching Based on Improved Gradual FusionAlgorithm[A].The 31stChinese Control and Decision Conference[C].New York:IEEE,2019:2962-2966.)
The research on the gradual fade-out algorithm has made a certain progress and has achieved a certain result, but the following problems still exist: (1) the existing method mainly aims at the problems of excessive color of the image and the processing of the splicing seam, although the visual effect is improved on the whole, the details are poor, and the middle area is possibly fuzzy; (2) the existing improved method generally has the phenomenon of double images with different degrees, and the visual effect is influenced to be poorer.
Disclosure of Invention
The purpose of the invention is as follows: the invention provides an image fusion method based on nonlinear weight, which improves the image fusion quality and reduces the image ghost generated after the splicing of complex images.
The technical scheme is as follows: a method for image fusion based on nonlinear weight comprises the following steps:
(1) inputting a first image and a second image, and extracting feature points of the first image and the second image through an SURF algorithm;
(2) performing feature matching on the feature points of the first image and the second image through a FLANN algorithm;
(3) warping the image through a homography algorithm, deforming the second image to enable the transformed images of the first image and the second image to be basically aligned, and mechanically splicing;
(4) and applying a nonlinear weight model to perform image fusion on the transformed images of the first image and the second image.
Further, in the step (2), the specific steps of feature matching are as follows:
a. taking the feature points of the first image as a training set and the feature points of the second image as a query set, and acquiring Euclidean distances between all feature points in the training set and feature points in the query set;
b. by comparing Euclidean distances, retaining the nearest point and the next nearest point of the Euclidean distance between each training set feature point and the query set feature point, and abandoning the rest matching;
c. if the nearest Euclidean distance and the second nearest Euclidean distance satisfy the following formula:
keeping the matching pair, otherwise discarding the matching pair; wherein, the ratio is a threshold value for judging the difference degree between the matching pair of the nearest Euclidean distance and the matching pair of the next nearest Euclidean distance; the ratio value is 0.4-0.6; the higher the ratio value is, the more the number of the matching pairs is, and the lower the matching precision is; the lower the ratio value is, the fewer the number of matching pairs is, and the higher the matching precision is.
Further, in step (3), the second image is deformed to fill a blank area without image information with pure black, that is, RGB components of the image are all 0.
Further, in the step (4), the first image and the second image are both divided into two parts according to the feature point matching condition, wherein the part with the matching feature points is an overlapping region, and the region without the matching feature points is a non-overlapping region.
Further, in step (4), the nonlinear weight model is: the first image and the second image are in the non-overlapping region C1And C2In the overlap region (C), each occupies 100% of the weight1∩C2) The resulting pixel follows the following equation:
C(x,y)=W1C1(x,y)+W2C2(x,y)
wherein, C1As a first image, C2Is a second image; w1Is the weight of the first image, W2Is the weight of the second image; c1(x, y) is the pixel value of the first image, C2(x, y) are pixel values of the second image; nonlinear weight W1And W2The values of (c) follow the following formula:
wherein D is1To be overlappedTotal width of area, D2And the distance between the current pixel and the left boundary of the overlapping area is taken as left, the left is the left boundary of the splicing overlapping area, and right is the right boundary of the splicing overlapping area.
Further, in the step (4), the image fusion is performed on the converted images of the first and second images by applying the nonlinear weight model, and when the overlap region is generated, the converted image of the second image has the weight of the solid black pixel which is always 0, and the solid black pixel does not participate in the generation of the overlap region.
Has the advantages that: compared with the prior art, the invention has the following remarkable effects: according to the invention, the influence of the first image on the overlapped image close to the left boundary part and the influence of the second image on the overlapped image close to the right boundary part are amplified by using the nonlinear weight model, so that the pixel change of the splicing region close to the boundary is smoother, the image ghost of the overlapped part can be effectively reduced, the visual interference factors such as ghost, crack and the like are effectively reduced, and the visual effect is remarkably improved.
Drawings
FIG. 1 is a schematic view of an image fusion process according to the present invention;
FIG. 2 is a weight diagram of a nonlinear weight image fusion model according to the present invention;
FIG. 3 is a first image to be fused;
FIG. 4 is a second image to be fused;
FIG. 5 is a diagram of the overall effect of the fused images;
FIG. 6 is a diagram illustrating conventional progressive image fusion model weights;
FIG. 7 is a diagram of the overall effect of the conventional gradual fade-out algorithm after image fusion;
FIG. 8 is a comparison graph of local magnification effect after the conventional gradual fade-out algorithm and the image of the present invention are fused.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings.
As shown in fig. 1, the method for image fusion based on nonlinear weight includes the following steps:
(1) Inputting a first image and a second image, and performing feature extraction on the first image and the second image through an SURF algorithm;
the feature extraction is the first step of image stitching application, the function of the feature extraction is to extract and describe key features of images, the invention uses SURF algorithm to extract features, and the specific implementation steps are as follows: a. constructing a Hessian matrix of the image; b. constructing a Gaussian pyramid; c. preliminarily positioning the feature points; d. calculating the vector direction of the feature points; e. a descriptor of the feature points is formed.
(2) Performing feature matching on the feature points of the first image and the second image through a FLANN algorithm;
the role of feature matching is to eliminate feature mismatches for most complex images. The specific implementation steps are as follows:
a. taking the feature points of the first image as a training set and the feature points of the second image as a query set, and acquiring Euclidean distances between all feature points in the training set and feature points in the query set;
b. by comparing Euclidean distances, retaining the nearest point and the next nearest point of the Euclidean distance between each training set feature point and the query set feature point, and abandoning the rest matching;
c. if the nearest Euclidean distance and the second nearest Euclidean distance satisfy the following formula:
the matching pair is retained, otherwise the matching pair is discarded. Wherein, the ratio is a threshold value for judging the difference degree between the matching pair of the nearest Euclidean distance and the matching pair of the next nearest Euclidean distance; the ratio value is 0.4-0.6; the higher the ratio value is, the more the number of the matching pairs is, and the lower the matching precision is; the lower the ratio value is, the fewer the number of matching pairs is, and the higher the matching precision is.
(3) Warping the image through a homography algorithm to enable the first image and the second image to be basically aligned, and mechanically splicing; the method comprises the following specific steps:
a. the homography transform is a 3 x 3 transform rectangle H that maps points on one image to corresponding points on the other image, and is expressed by the following formula:
b. for a set of corresponding points (x) before and after image warping1,y1) And (x)2,y2) The mapping relationship between the two is shown as the following formula:
and (3) applying the mapping formula to carry out image transformation, generating a transformed image, and carrying out mechanical splicing.
(4) Applying a nonlinear weight model to perform image fusion on the first image and the second image; the method comprises the following specific steps:
a. and according to the result of image feature matching, generating a first image of the non-spliced region by 100% weight of the first image pixels, and generating a second image of the non-spliced region by 100% weight of the second image pixels.
b. And fusing the pixels of the first image and the second image by using a nonlinear weight model to form a spliced area of the image, and sequentially synthesizing the spliced area of the image with the non-spliced area of the first image and the non-spliced area of the second image to generate a fused image.
As shown in FIG. 2, the first image and the second image are in the non-overlapping region C 1And C2In the overlap region (C), each occupies 100% of the weight1∩C2) The resulting pixel follows the following equation:
C(x,y)=W1C1(x,y)+W2C2(x,y)
wherein, C1As a first image, C2Is a second image; w1Is the weight of the first image, W2Is the weight of the second image, C1(x, y) is the pixel value of the first image, C2(x, y) is the pixel value of the second image, the nonlinear weight W1And W2The values of (c) follow the following formula:
wherein D is1For the total width of the overlap region, D2And the distance between the current pixel and the left boundary of the overlapping area is taken as left, the left is the left boundary of the splicing overlapping area, and right is the right boundary of the splicing overlapping area.
The images shown in fig. 3 and fig. 4 are spliced by using the invention, the effect image after image fusion is shown in fig. 5, a nonlinear weight image fusion model is applied to image fusion, the influence of the first image on the weight generated by the part of pixels close to the left border of the splice (for example, at the position of x ═ left in fig. 2) is amplified, the influence of the second image on the weight generated by the part of pixels close to the right border of the splice (for example, at the position of x ═ right in fig. 2) is amplified, and the overlapped area (C) is enabled to be formed1∩C2) The pixel change close to the boundary is smoother, the possibility of image ghosting generated by the overlapped part can be effectively reduced, so that visual interference factors such as ghosting and cracks are effectively reduced, and the visual impression of the finally generated fusion image is improved.
As shown in fig. 6, the image fusion is performed by applying a conventional gradual image fusion algorithm, which specifically includes: the first image and the second image are in the non-overlapping region C1And C2In the overlap region (C), each occupies 100% of the weight1∩C2) The resulting pixel follows the following equation:
C(x,y)=W1C1(x,y)+W2C2(x,y)
wherein, C1As a first image, C2Is a second image; w1Is the weight of the first image, W2Is the weight of the second brushed image; c1(x, y) is the pixel value of the first image, C2(x, y) are pixel values of the second image; weight W1And W2The values of (c) follow the following formula:
wherein D is1For the total width of the overlap region, D2Is as followsDistance of the front pixel from the left boundary of the overlap region.
The images shown in fig. 3 and 4 are spliced by using a traditional gradual image fusion algorithm, and an effect image after image fusion is shown in fig. 7.
As shown in fig. 8, 8(a) performs image fusion by using a conventional progressive fading-out algorithm, and ghost images of text portions after image fusion are particularly serious, which greatly affects detail observation of fused images; and 8(b) the method is used for image fusion, the detail part information of the fused image is well displayed, no obvious ghost image exists, the definition is obviously improved compared with that of the traditional algorithm, and the subjective effect is better.
The Quality score evaluation is performed by applying a Quality score evaluation method (Pavan C M, Rajiv S. objective and objective Quality evaluation Assessment of Stitched Images for visual Reality [ J ]. IEEETRANSACTION ON IMAGE PROCESSING,2019,28(11):5620-563) proposed by Pavan C M and Rajiv S, using the IMAGE Stitched by the conventional method in FIG. 7 and the IMAGE Stitched by the present invention in FIG. 5 as input Images, and the objective Quality evaluation score is obtained by comparing the original IMAGE with the Stitched IMAGE at the pixel level and introducing a difference model into a regression vector machine. In order to make the result more reliable, other original images and the stitched image are introduced for quality score comparison, 4 additional experiments are added, and the obtained quality scores are shown in table 1.
TABLE 1 Objective quality score comparison
Therefore, although the quality evaluation scores of all groups of experiments fluctuate, the quality evaluation scores obtained by using the algorithm of the invention are higher than those obtained by using the traditional gradual evolution algorithm, and the quality evaluation method has better performance in the aspect of objective quality evaluation.
By combining subjective observation and objective quality evaluation scores, the nonlinear weight image fusion method of the invention achieves better effect and higher score, so that the algorithm of the invention has higher algorithm quality.