CN111754408A - High-real-time image splicing method - Google Patents

High-real-time image splicing method Download PDF

Info

Publication number
CN111754408A
CN111754408A CN202010607671.XA CN202010607671A CN111754408A CN 111754408 A CN111754408 A CN 111754408A CN 202010607671 A CN202010607671 A CN 202010607671A CN 111754408 A CN111754408 A CN 111754408A
Authority
CN
China
Prior art keywords
image
feature point
point
feature
pixel
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202010607671.XA
Other languages
Chinese (zh)
Inventor
徐冬冬
寇旗旗
程德强
高蕊
赵凯
陈亮亮
唐守锋
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
China University of Mining and Technology CUMT
Original Assignee
China University of Mining and Technology CUMT
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by China University of Mining and Technology CUMT filed Critical China University of Mining and Technology CUMT
Priority to CN202010607671.XA priority Critical patent/CN111754408A/en
Publication of CN111754408A publication Critical patent/CN111754408A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4038—Image mosaicing, e.g. composing plane images from plane sub-images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G06F18/2413—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on distances to training or reference patterns
    • G06F18/24147—Distances to closest patterns, e.g. nearest neighbour classification
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/20—Image enhancement or restoration using local operators
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/70—Denoising; Smoothing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
    • G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/46—Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features
    • G06V10/462—Salient features, e.g. scale invariant feature transforms [SIFT]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10004—Still image; Photographic image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20024—Filtering details
    • G06T2207/20028—Bilateral filtering
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20212—Image combination
    • G06T2207/20221—Image fusion; Image merging

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Data Mining & Analysis (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Evolutionary Biology (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • General Engineering & Computer Science (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Multimedia (AREA)
  • Image Analysis (AREA)

Abstract

The invention relates to an image splicing method with high real-time performance, belongs to the technical field of image processing, and solves the problems of complex calculation, low precision and poor real-time performance of the conventional image splicing method. The method comprises the following steps: acquiring a first image and a second image to be spliced; extracting feature point sets of the first image and the second image; describing each feature point in the feature point set by using CLBP (common Link protocol) codes to obtain a descriptor set; based on the descriptor set, performing initial matching on the feature points by adopting a nearest neighbor ratio classifier to obtain a feature point pair set; rejecting mismatching characteristic point pairs in the characteristic point pair set, and obtaining a projection transformation matrix between the first image and the second image; and fusing based on the projection transformation matrix to obtain a spliced image. According to the method, the characteristic points which are uniformly distributed in the image are extracted, and the CLBP codes are adopted to describe the characteristic points, so that excessive characteristic points are prevented from participating in the matching process, the calculation complexity is reduced, and the calculation precision and the splicing instantaneity are improved.

Description

High-real-time image splicing method
Technical Field
The invention relates to the technical field of image processing, in particular to an image splicing method with high real-time performance.
Background
In recent years, with the rise of artificial intelligence, vision is one of the main captured information sources, and a wide-area image containing rich information becomes an urgent need, and image stitching has been widely researched and focused as a key technology thereof. The method is applied to a plurality of fields such as geological survey, biomedicine, civil entertainment and the like. Therefore, the method for splicing the images quickly and accurately has very important significance.
Image stitching is to combine a plurality of single images into an image after correlation processing. The fusion method mainly comprises a series of steps of input image preprocessing, feature point detection, feature point matching, geometric correction matrix acquisition and image fusion. At present, the detection of feature points of image splicing in the prior art is mainly based on an SIFT and SURF method, the SIFT algorithm can keep unchanged the scale and rotation of an image and has certain robustness to illumination, but due to the complexity of calculation, the defect of real-time property is brought. The SURF algorithm is superior to the SIFT algorithm in speed because an integral image is used in feature point detection. Both are to detect the feature points on the whole image, however, the images to be stitched are only partially overlapped, and needless time is consumed for detecting excessive feature points, and the proper reduction of the feature points does not affect the registration accuracy. In addition, the differentiability of the feature descriptor directly affects the subsequent steps of feature matching, image correction and the like. The SIFT describes the feature points by using the gradient direction histogram of the feature point neighborhood, and the calculation is complex, thereby bringing the expenditure on time.
The method mainly has the following defects that firstly, the detected feature points are regionally concentrated, the redundancy of the feature points increases the computational complexity, the precision is low, and the requirement on the real-time property of image splicing is difficult to meet in the conventional image splicing method based on SURF and SIFT; secondly, in the existing image splicing method, feature points are often described based on the main direction of the positioning feature points, so that the complexity of calculation is increased, and the time consumption is long.
Disclosure of Invention
In view of the above analysis, the present invention aims to provide an image stitching method with high real-time performance, so as to solve the problems of complicated calculation, low precision and poor real-time performance of the existing image stitching method.
The invention provides an image splicing method with high real-time performance, which comprises the following steps:
acquiring a first image and a second image to be spliced;
extracting feature point sets of the first image and the second image by using an adaptive non-maximum suppression-based SURF algorithm; wherein each feature point in the set of feature points is uniformly distributed on the corresponding image;
describing each feature point in the feature point set by using CLBP (common line code) coding to obtain descriptor sets of the first image and the second image;
based on the descriptor sets of the first image and the second image, performing initial feature point matching by adopting a nearest neighbor ratio classifier to obtain feature point pair sets of the first image and the second image;
rejecting mismatching characteristic point pairs in the characteristic point pair set by using a random sampling consistency method, and obtaining a projection transformation matrix between the first image and the second image;
and fusing the first image and the second image based on the projective transformation matrix to obtain a spliced image.
Further, the following procedures are executed to extract the feature point set of the first image or the second image:
sequentially processing the first image or the second image by a plurality of filters with different sizes to obtain a plurality of image layers with consistent image size and different scales in a discrete space and a filter response value of each pixel point of an image in each image layer;
the image layers are averagely divided into a plurality of groups of image piles according to the size, wherein the sizes of the images in the first image layer in the first group of image piles to the last image layer in the last group of image piles are sequentially increased, and the adjacent two groups of image piles contain the image layers of the images with the same size;
removing pixel points with the filtering response value smaller than a first preset threshold value in each image layer to obtain a preliminary characteristic point set;
and taking any preliminary feature point as a feature point to be operated, adaptively selecting a corresponding inhibition radius, performing non-maximum inhibition in the image layer where the feature point to be operated is located and the image layer adjacent to the feature point to be operated, removing pixel points with filter response values smaller than the filter response value of the feature point to be operated, traversing each preliminary feature point, and obtaining a feature point set of the first image or the second image.
Further, the filter size for obtaining the image of the ith image layer in the ith group of image piles is:
L＝3×[2o+1×(l+1)+1]，
wherein o is 1, 2, 3.. m, l is 1, 2, 3.. t,
the image dimensions of the l image layer in the o image pile are obtained as follows:
Figure BDA0002559731460000031
wherein L is0To obtain the filter size of the first image layer image in the 1 st group of image piles, S0Is the scale of the first image layer image within the 1 st set of image piles.
Further, before describing the feature points, the method further includes:
obtaining the offset of each feature point coordinate in the first image and the second image feature point set by using a Taylor series interpolation method;
and correcting the coordinates of the corresponding characteristic points in the discrete space by using the coordinate offset to obtain the coordinates of all the characteristic points in the first image and the second image in the continuous space, and determining the positions of the corresponding characteristic points based on the coordinates.
Further, the feature points are described to obtain corresponding feature point descriptors in the following manner:
constructing a circle neighborhood with a preset radius by taking any feature point in the feature point set as a center, and determining pixel points included in the circle neighborhood according to the position coordinates of any feature point;
describing each pixel by adopting CLBP coding to obtain a descriptor of the pixel: contrast binary pattern
Figure BDA0002559731460000041
Positive and negative binary pattern
Figure BDA0002559731460000042
And amplitude binary pattern
Figure BDA0002559731460000043
Descriptor contrast binary pattern based on multiple pixel points in circle neighborhood
Figure BDA0002559731460000044
Positive and negative binary pattern
Figure BDA0002559731460000045
And amplitude binary pattern
Figure BDA0002559731460000046
Obtaining a statistical histogram corresponding to the characteristic point by the corresponding coding mode;
converting the statistical histogram into a feature vector, and performing normalization processing on the feature vector to obtain a descriptor of a feature point;
and traversing each feature point to obtain a descriptor set corresponding to the feature point set.
Further, the descriptor of each pixel point is obtained by the following formula:
Figure BDA0002559731460000047
Figure BDA0002559731460000048
CLBP_CP,R＝s(gc-cI)，
Figure BDA0002559731460000049
Figure BDA00025597314600000410
Figure BDA00025597314600000411
wherein P is the number of pixel points in a sampling region with the pixel point to be described as the center, R is the radius of the sampling region, gcThe gray value of the pixel point to be described is obtained; gpThe gray value of the P-th pixel point in the sampling area is obtained; m ispThe absolute value of the difference of the gray values of the P-th pixel point and the pixel point to be described in the sampling region is obtained; c is all m in the circular neighborhood of any feature pointpThe mean value of (a); c. CIMean value of pixel values, U (LBP), of all pixel points in a circular neighborhood of any feature pointP,R) Is a positive and negative binary pattern
Figure BDA0002559731460000051
Number of coding cycle transitions, U (CLBP _ M)P,R) As amplitude binary patterns
Figure BDA0002559731460000052
The number of code cycle hops.
Further, the initial matching of the feature points is performed by using a nearest neighbor ratio classifier to obtain a feature point pair set, which is specifically as follows:
acquiring the Euclidean distance between any feature point in the feature point set of the first image and each feature point in the feature point set of the second image to be spliced, and determining the Euclidean distance between the nearest feature point and the next nearest feature point and any feature point in the feature point set of the first image;
if the ratio of the Euclidean distance between the nearest neighbor feature point and the second nearest neighbor feature point is smaller than a preset value, judging that any feature point in the first image is matched with the nearest neighbor feature point in the second image and is a feature point pair;
and traversing all the feature points in the first image feature point set to obtain a feature point set.
Further, rejecting mismatching feature point pairs in the feature point pair set by the following method:
step 1, randomly selecting four groups of characteristic point pairs from the preliminarily obtained characteristic point pair set, and obtaining a projection transformation matrix between a first image and a second image based on coordinate data of the characteristic point pairs;
step 2, calculating the projection error of each characteristic point pair in the characteristic point pair set and the projection transformation matrix; if the projection error is smaller than a second preset threshold value, the characteristic point pairs are counted into an inner point set;
step 3, comparing the number of the characteristic point pairs in the inner point set with a threshold value;
taking the number of the feature point pairs in the inner point set obtained by the first calculation as a threshold, repeating the steps S1 to S2, obtaining the number of the feature point pairs in the corresponding inner point set, comparing the number with the result (namely, the threshold) obtained by the first calculation, taking the current inner point set as a preferred inner point set if the number is larger than the threshold, taking the inner point set obtained at the previous time as the preferred inner point set if the number is smaller than the threshold, performing iteration for a preset number of times, namely, taking the number of the feature point pairs in the preferred inner point set as the threshold, repeating the steps S1 to S3, and taking the preferred inner point set obtained at the last iteration as an optimal inner point set, namely, eliminating the feature point pair set of the mismatching feature point pairs.
Further, the first image and the second image are fused to obtain a spliced image in the following way:
transforming the first image and the second image into the same plane by using the optimal projective transformation matrix;
in the same plane, acquiring the proportion of each pixel point in the overlapping area of the first image and the second image, and acquiring a fusion coefficient of the first image and the second image according to the proportion;
fusing the first image and the second image based on the fusion coefficient by the following formula to obtain a spliced image:
Figure BDA0002559731460000061
wherein I (x, y) is the obtained stitched image, I1Is a set of pixel points, I, in the first image2For the set of pixel points in the second image, (x, y) represents the coordinates of the pixel points within the obtained stitched image, (x1,y1) Is the coordinate of the pixel point of the part of the first image which is not overlapped with the second image, (x)2,y2) Is the coordinate of the pixel point of the second image which is not overlapped with the first image, (x'1,y'1) Is the coordinate of the pixel point of the part of the first image which is overlapped with the second image, (x'2,y'2) And coordinates of pixel points of the second image which are overlapped with the first image are obtained.
Further, the fusion coefficient of the first image and the second image is obtained by the following formula:
Figure BDA0002559731460000071
w2＝1-w1，xa<xi<xb，i＝1、2、3...n,
wherein x isiThe abscissa of the ith pixel point in the overlapping region of the first image and the second image, n is the number of the pixel points in the overlapping region, and xaIs the abscissa, x, of the left edge of the overlap regionbIs the abscissa of the right edge of the overlap region, w1The fusion coefficient w of the ith pixel point in the overlapping region relative to the first image2And the fusion coefficient of the ith pixel point in the overlapping area relative to the second image is obtained.
Compared with the prior art, the invention can realize at least one of the following beneficial effects:
1. the method extracts the feature point sets of the first image and the second image to be spliced based on the self-adaptive non-maximum suppression SURF algorithm, so that each feature point in the feature point sets is uniformly distributed on the corresponding image, the redundant feature points are filtered, the feature point matching precision is improved, the calculation complexity of a projection transformation matrix in the feature point matching process and obtained is reduced, the calculated amount is greatly reduced, and the real-time performance of image splicing is improved;
2. according to the invention, each feature point in the feature point set is described by using CLBP coding, and the obtained feature point descriptor has scale invariance, rotation invariance and illumination invariance, so that the step of determining the direction of the feature point by the traditional SURF is omitted, the calculated amount is greatly reduced, and the real-time property is further improved while the image splicing precision is not influenced.
In the invention, the technical schemes can be combined with each other to realize more preferable combination schemes. Additional features and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. The objectives and other advantages of the invention will be realized and attained by the structure particularly pointed out in the written description and drawings.
Drawings
The drawings are only for purposes of illustrating particular embodiments and are not to be construed as limiting the invention, wherein like reference numerals are used to designate like parts throughout.
Fig. 1 is a flowchart of an image stitching method with high real-time performance according to an embodiment of the present invention.
Detailed Description
The accompanying drawings, which are incorporated in and constitute a part of this application, illustrate preferred embodiments of the invention and together with the description, serve to explain the principles of the invention and not to limit the scope of the invention.
The invention discloses a high-real-time image splicing method. As shown in fig. 1, the method comprises the steps of:
step S1, acquiring a first and a second image to be stitched.
Step S2, extracting feature point sets of the first image and the second image by using an SURF algorithm based on adaptive non-maximum suppression; wherein, each feature point in the feature point set is uniformly distributed on the corresponding image.
And step S3, describing each feature point in the feature point set by using CLBP coding to obtain a descriptor set of the first image and the second image.
And step S4, based on the descriptor sets of the first image and the second image, performing initial feature point matching by using a nearest neighbor ratio classifier to obtain feature point pair sets of the first image and the second image.
And step S5, rejecting mismatching characteristic point pairs in the characteristic point pair set by using a random sampling consistency method, and obtaining a projection transformation matrix between the first image and the second image.
And step S6, fusing the first image and the second image based on the projective transformation matrix to obtain a spliced image.
In step S1, the image is subjected to denoising preprocessing to obtain a first image and a second image to be stitched. The method comprises the following specific steps:
if the image to be spliced is a color image, firstly converting the image into a gray image, and filtering and denoising the converted gray image. The invention adopts a bilateral filtering method, the bilateral filtering is a nonlinear filtering method, and the spatial position correlation and the gray level similarity of the neighborhood of the pixel point are considered at the same time, so that the purpose of denoising is achieved, and the image edge can be well stored. The image obtained after bilateral filtering and denoising is as follows:
Figure BDA0002559731460000091
Figure BDA0002559731460000092
jp is a pixel value obtained after filtering processing is carried out on a pixel point at a position p, Kp is a normalization term of a bilateral filter, omega represents a neighborhood of the pixel point with the coordinate p, and Ip、IqPixel values, σ, of pixel points at positions p, q, respectivelysIs the spatial location kernel parameter, σ, of a pixel pointrKernel parameters representing pixel values.
In step S2, the following procedure is performed to extract a feature point set of the first image or the second image.
And S2.1, sequentially processing the first image or the second image by a plurality of filters with different sizes to obtain a plurality of image layers with consistent image size and different scales in discrete space and a filter response value of each pixel point of the image in each image layer.
Specifically, a Hessian matrix H (x, y, s) at any pixel point (x, y) in the first image or the second image with the scale s is defined as:
Figure BDA0002559731460000093
wherein, (x, y) is the position coordinate of the pixel point, Dxx(x, y, s) is a second order differential of Gaussian
Figure BDA0002559731460000094
Convolution with the image at a pixel point (x, y), Dxy(x, y, s) is
Figure BDA0002559731460000095
Convolution with the image at a pixel point (x, y), Dyy(x, y, s) is
Figure BDA0002559731460000096
Convolution with the image at pixel point (x, y).
Wherein the content of the first and second substances,
Figure BDA0002559731460000101
the principle that the filter changes the image scale is the same as the Gaussian differential function, the size of the image is not changed, the filter template and the image are subjected to convolution operation, and a plurality of image layers with gradually increased scales and completely consistent sizes are constructed by changing the size of the filter template.
The filter response value of each pixel point in the image of each image layer is as follows:
det(Happrox)＝Dxx(x,y,s)Dyy(x,y,s)-(0.9Dxy(x,y,s))2，
wherein det (H)approx) Representing the filter response value of a pixel, s being the image scale, Dxx(x, y, s) is the filter response value of the filter template in the x-direction at (x, y, s), Dyy(x, y, s) is the filter response value of the template in the y direction at (x, y, s), Dxy(x, y, s) is the filter response value of the filter template at (x, y, s) in the xy direction.
And S2.2, averagely dividing the image layers into a plurality of groups of image piles according to the size of the image piles, wherein the image scales from the first image layer in the first group of image piles to the last image layer in the last group of image piles are sequentially increased, and the adjacent two groups of image piles contain image layers with images of the same scale. The image layers of the images with the same scale are arranged in the two adjacent groups of image piles, so that the obtained image piles can be ensured to contain the images with all scales, and meanwhile, the calculated amount can be reduced by adopting a method of sampling at intervals in the large-scale images.
The filter size used to obtain the image of the ith image layer in the ith group of image piles is:
L＝3×[2o+1×(l+1)+1]，
wherein o is 1, 2, 3.. m, l is 1, 2, 3.. t,
the image dimensions of the l image layer in the o image pile are obtained as follows:
Figure BDA0002559731460000102
wherein L is0To obtain the filter size of the first image layer image in the 1 st group of image piles, S0Is the scale of the first image layer image within the 1 st set of image piles.
Illustratively, the filter used to obtain the first image layer image in the 1 st group of image piles is a filter template with a size of 9 × 9 and a scale of 1.2, and this is used as the reference filter.
And S2.3, removing the pixel points of which the filter response values in each image layer are smaller than a first preset threshold value to obtain a preliminary feature point set. In particular, the first preset threshold value may be set empirically, preferably in the order of 3.
And S2.4, in order to ensure the scale invariance of the first image and the second image, taking any preliminary feature point as a feature point to be operated, adaptively selecting a corresponding inhibition radius, performing non-maximum inhibition in an image layer where the feature point to be operated is located and an image layer adjacent to the image layer, removing pixel points with a filtering response value smaller than that of the feature point to be operated, traversing each preliminary feature point, and obtaining a feature point set of the first image or the second image.
Specifically, the formula of the adaptively selected suppression radius r is as follows:
ri＝min||Xi-Xj||，
s.t.det(Happrox(Xi))≤0.9det(Happrox(Xj))，
wherein, XiAs a position vector (x) of the feature point to be operatedi,yi,si)，XjPosition vectors (x) of other characteristic points in the image layer of the characteristic point to be operatedj,yj,sj)，det(Happrox(X)) is a filter response value of the feature point.
And determining a characteristic point with a filtering response value in the image layer larger than 0.9 times of the filtering response value of the characteristic point to be operated by taking any characteristic point as the characteristic point to be operated, calculating the Euclidean distance between the characteristic point to be operated and the characteristic point to be operated, and taking the minimum Euclidean distance as the suppression radius of the characteristic point to be operated.
Adaptive non-maximum suppression is performed within the image pile groups, with only one adjacent layer for the first and last layers of each group, so that the two layers do not perform non-maximum suppression. By adaptive non-maximum suppression, more feature points can be suppressed in an image area with a large number of feature points, and a small number of feature points can be suppressed in an image area with sparse feature points, so that the feature points are uniformly distributed. Meanwhile, the calculation amount of subsequent feature point matching is reduced, and the solving precision of the projective transformation matrix is improved, so that the image splicing precision and the real-time performance are improved.
Preferably, before describing the feature points, the method further includes:
and obtaining the offset of each feature point coordinate in the first image and the second image feature point set by using a Taylor series interpolation method.
And correcting the coordinates of the corresponding characteristic points in the discrete space by using the coordinate offset to obtain the coordinates of all the characteristic points in the first image and the second image in the continuous space, and determining the positions of the corresponding characteristic points based on the coordinates.
Let b (X) be a response value of (X, y, s) at the feature point X, and the taylor series expansion is:
Figure BDA0002559731460000121
the corresponding offset is:
Figure BDA0002559731460000122
the new coordinates of the feature points fitted by the difference of taylor series
Figure BDA0002559731460000123
Is composed of
Figure BDA0002559731460000124
In step S3, the feature points are described to obtain corresponding feature point descriptors by:
and S3.1, constructing a circular neighborhood with a preset radius size by taking any feature point in the feature point set as a center, preferably, the preset radius size is 10S, and S is the scale of the image to which the any feature point belongs. And determining pixel points included in the circle neighborhood according to the position coordinates of the characteristic points, namely judging whether other pixel points are in the circle neighborhood or not according to the position coordinates of the characteristic points.
And S3.2, considering that the SURF algorithm is used for describing the feature points, the directions of the feature points are required to be distributed, and the calculation complexity and the calculation amount are increased. Therefore, in order to solve the above problems, the present invention describes each pixel by using CLBP coding to obtain a descriptor of the pixel: contrast binary pattern
Figure BDA0002559731460000131
Positive and negative binary pattern
Figure BDA0002559731460000132
And amplitude binary pattern
Figure BDA0002559731460000133
Specifically, the descriptor of each pixel point is obtained by the following formula:
Figure BDA0002559731460000134
Figure BDA0002559731460000135
CLBP_CP,R＝s(gc-cI)，
Figure BDA0002559731460000136
Figure BDA0002559731460000137
Figure BDA0002559731460000138
wherein P is the number of pixel points in a sampling region with the pixel point to be described as the center, R is the radius of the sampling region, gcThe gray value of the pixel point to be described is obtained; gpThe gray value of the P-th pixel point in the sampling area is obtained; m ispThe absolute value of the difference of the gray values of the P-th pixel point and the pixel point to be described in the sampling region is obtained; c is all m in the circular neighborhood of any feature pointpThe mean value of (a); c. CIMean value of pixel values, U (LBP), of all pixel points in a circular neighborhood of any feature pointP,R) Is a positive and negative binary pattern
Figure BDA0002559731460000139
Number of coding cycle transitions, U (CLBP _ M)P,R) As amplitude binary patterns
Figure BDA00025597314600001310
The number of code cycle hops.
Step S3.3, descriptor contrast binary pattern based on multiple pixel points in circle neighborhood
Figure BDA00025597314600001311
Positive and negative binary pattern
Figure BDA00025597314600001312
And amplitude binary pattern
Figure BDA00025597314600001313
And obtaining a statistical histogram corresponding to the feature point by the corresponding coding mode.
In particular, the method comprises the following steps of,
Figure BDA0002559731460000141
there are ten coding modes that can be used,
Figure BDA0002559731460000142
there are ten coding modes, CLBP _ CP,RThere are two modes.
When CLBP _ CP,RWhen the content is equal to 0, the content,
Figure BDA0002559731460000143
there are ten coding modes, CLBP _ CP,RWhen the number is equal to 1, the alloy is put into a container,
Figure BDA0002559731460000144
there are ten coding modes.
The statistical histogram of the feature points thus obtained (denoted as
Figure BDA0002559731460000145
) The abscissa of (a) is 30 coding modes, and the ordinate is the number of pixel points corresponding to the corresponding coding mode in the circle neighborhood.
Step S3.4, convert the statistical histogram into a feature vector V ═ x1,x2,x3.....x30]And normalizing the feature vectors of the feature points by the following formula to obtain descriptors of the feature points:
Figure BDA0002559731460000146
then P ═ P1,p2,...,p30And the normalized feature vector is used as the feature vector. So far, the obtained feature vector of the feature point has rotation invariance, illumination invariance and scale invariance.
And S3.5, traversing each feature point to obtain a descriptor set corresponding to the feature point set.
Step S4, preferably, based on the feature vectors of the feature points obtained in step S3, performing initial feature point matching on the feature points in the first image and the second image by using a nearest neighbor ratio classifier, and obtaining a feature point pair set, which is as follows:
the Euclidean distance between any feature point in the feature point set of the first image and each feature point in the feature point set of the second image to be spliced is obtained, and the Euclidean distance between the nearest feature point and the Euclidean distance between the nearest feature point and the feature point in the feature point set of the first image are determined, wherein the specific formula is as follows:
Figure BDA0002559731460000147
wherein, P1Feature vectors, P, corresponding to any feature point in the first image2,N1Is the feature vector of the feature point in the second image which is most adjacent to any feature point in the first image, P2,N2The NNDR is a characteristic vector of a characteristic point next adjacent to any characteristic point in the first image in the second image, and the NNDR is a ratio of Euclidean distances between the nearest adjacent characteristic point and the next adjacent characteristic point.
If the ratio of the euclidean distance between the nearest neighbor feature point and the second nearest neighbor feature point is smaller than a preset value, it is determined that any feature point in the first image matches with the nearest neighbor feature point in the second image, and the result is a feature point pair, and preferably, the preset value is 0.8.
And traversing all the feature points in the first image feature point set to obtain a feature point pair set.
In step S5, preferably, in consideration of the existence of random errors, the obtained feature point pair set still has a certain mismatch, and the random sampling consensus algorithm (RANSAC) of the present invention executes the following procedure to remove the mismatch feature point pairs in the feature point pair set.
S5.1, randomly selecting four groups of characteristic point pairs from the preliminarily obtained characteristic point pair set, and obtaining a projection transformation matrix between the first image and the second image based on the coordinate data of the characteristic point pairs, wherein the formula is as follows:
Figure BDA0002559731460000151
wherein (x)1,y1),(x2,y2) Respectively are the position coordinates of the characteristic points in the first image and the characteristic points in the matched second image, and H is a projection transformation matrix.
S5.2, calculating the projection error of each characteristic point pair in the characteristic point pair set and the projection transformation matrix; and if the projection error is smaller than a second preset threshold value, the characteristic point pairs are counted into an inner point set. The second preset threshold is preferably set according to the required accuracy requirement of the stitched image.
Specifically, the projection error is obtained by the following formula:
Figure BDA0002559731460000152
where e (h) is a projection error, and N is the number of feature point pairs included in the feature point pair set.
S5.3, comparing the number of the characteristic point pairs in the inner point set with a threshold value;
and (3) taking the number of the feature point pairs in the inner point set obtained by the first calculation as a threshold, repeating the steps from S5.1 to S5.2, obtaining the number of the feature point pairs in the corresponding inner point set, comparing the number with the result (namely, the threshold) obtained by the first calculation, taking the current inner point set as a preferred inner point set if the number is greater than the threshold, taking the inner point set obtained at the previous time as a preferred inner point set if the number is less than the threshold, and performing iteration, namely taking the number of the feature point pairs in the preferred inner point set as the threshold, and repeating the steps from S5.1 to S5.3, preferably, specifically setting the iteration number K according to the precision requirement for matching the feature points. If the number of the feature point pairs in the inner point set obtained by the K-th iteration is larger than the number of the feature point pairs in the preferred inner point set obtained by the K-1-th iteration, taking the inner point set obtained by the K-th iteration as an optimal inner point set, namely, rejecting the feature point set of the mismatching feature point pairs; and if the number of the characteristic point pairs in the preferred inner point set obtained by the K-1 iteration is less than the number of the characteristic point pairs in the preferred inner point set obtained by the K-1 iteration, taking the preferred inner point set obtained by the K-1 iteration as an optimal inner point set, and determining a projection transformation matrix corresponding to the optimal inner point set as an optimal projection transformation matrix.
Preferably, the first image and the second image are fused to obtain a stitched image in the following manner.
The first image and the second image are transformed into the same plane using the optimal projective transformation matrix obtained in step S5.
In the same plane, obtaining the proportion of each pixel point in the overlapping area of the first image and the second image, and obtaining the fusion coefficient of the first image and the second image according to the proportion, specifically as follows:
Figure BDA0002559731460000161
w2＝1-w1，xa<xi<xb，i＝1、2、3...n,
wherein I (x, y) is the obtained stitched image, I1Set of pixel points for the part of the first image that is not coincident with the second image, I2The pixel point set of the part of the second image which is not overlapped with the first image in the second image (x, y) represents the coordinates of the pixel points in the obtained spliced image (x1,y1) Is the coordinate of pixel point in the first image coincident with the second image, (x)2,y2) And coordinates of pixel points of the second image which are overlapped with the first image are obtained.
And fusing the first image and the second image based on the fusion coefficient by the following formula to obtain a spliced image:
Figure BDA0002559731460000171
wherein I (x, y) is the obtained stitched image, I1Is a set of pixel points, I, in the first image2For the set of pixel points in the second image, (x, y) represents the coordinates of the pixel points within the obtained stitched image, (x1,y1) Is the coordinate of the pixel point of the part of the first image which is not overlapped with the second image, (x)2,y2) Is the coordinate of the pixel point of the second image which is not overlapped with the first image, (x'1,y'1) Is the coordinate of the pixel point of the part of the first image which is overlapped with the second image, (x'2,y'2) And coordinates of pixel points of the second image which are overlapped with the first image are obtained.
Compared with the prior art, the image splicing method with high real-time performance provided by the invention has the advantages that firstly, the characteristic point sets of the first image and the second image to be spliced are extracted based on the self-adaptive non-maximum suppression SURF algorithm, so that each characteristic point in the characteristic point sets is uniformly distributed on the corresponding image, the redundant characteristic points are filtered, the matching precision of the characteristic points is improved, the calculation complexity of the characteristic point matching process and the projection transformation matrix is reduced, the calculation amount is greatly reduced, and the real-time performance of image splicing is improved; secondly, each feature point in the feature point set is described by using CLBP coding, and the obtained feature point descriptor has scale invariance, rotation invariance and illumination invariance, so that the step of determining the direction of the feature point by the traditional SURF is omitted, the calculated amount is greatly reduced, and the real-time performance of image splicing is further improved while the image splicing precision is not influenced.
Those skilled in the art will appreciate that all or part of the flow of the method implementing the above embodiments may be implemented by a computer program, which is stored in a computer readable storage medium, to instruct related hardware. The computer readable storage medium is a magnetic disk, an optical disk, a read-only memory or a random access memory.
The above description is only for the preferred embodiment of the present invention, but the scope of the present invention is not limited thereto, and any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope of the present invention are included in the scope of the present invention.

Claims (10)

1. The image splicing method with high real-time performance is characterized by comprising the following steps of:
acquiring a first image and a second image to be spliced;
extracting feature point sets of the first image and the second image by using an adaptive non-maximum suppression-based SURF algorithm; wherein each feature point in the set of feature points is uniformly distributed on the corresponding image;
describing each feature point in the feature point set by using CLBP (common line code) coding to obtain descriptor sets of the first image and the second image;
based on the descriptor sets of the first image and the second image, performing initial feature point matching by adopting a nearest neighbor ratio classifier to obtain feature point pair sets of the first image and the second image;
rejecting mismatching characteristic point pairs in the characteristic point pair set by using a random sampling consistency method, and obtaining a projection transformation matrix between the first image and the second image;
and fusing the first image and the second image based on the projective transformation matrix to obtain a spliced image.
2. The image stitching method according to claim 1, wherein the following process is performed to extract the feature point set of the first image or the second image:
sequentially processing the first image or the second image by a plurality of filters with different sizes to obtain a plurality of image layers with consistent image size and different scales in a discrete space and a filter response value of each pixel point of an image in each image layer;
the image layers are averagely divided into a plurality of groups of image piles according to the size, wherein the sizes of the images in the first image layer in the first group of image piles to the last image layer in the last group of image piles are sequentially increased, and the adjacent two groups of image piles contain the image layers of the images with the same size;
removing pixel points with the filtering response value smaller than a first preset threshold value in each image layer to obtain a preliminary characteristic point set;
and taking any preliminary feature point as a feature point to be operated, adaptively selecting a corresponding inhibition radius, performing non-maximum inhibition in the image layer where the feature point to be operated is located and the image layer adjacent to the feature point to be operated, removing pixel points with filter response values smaller than the filter response value of the feature point to be operated, traversing each preliminary feature point, and obtaining a feature point set of the first image or the second image.
3. The image stitching method according to claim 2, wherein the filter size for obtaining the image of the ith image layer in the ith group of image piles is:
L＝3×[2o+1×(l+1)+1]，
wherein o is 1, 2, 3.. m, l is 1, 2, 3.. t,
the image dimensions of the l image layer in the o image pile are obtained as follows:
Figure FDA0002559731450000021
wherein L is0To obtain the filter size of the first image layer image in the 1 st group of image piles, S0Is the scale of the first image layer image within the 1 st set of image piles.
4. The image stitching method according to claim 1, further comprising, before describing the feature points:
obtaining the offset of each feature point coordinate in the first image and the second image feature point set by using a Taylor series interpolation method;
and correcting the coordinates of the corresponding characteristic points in the discrete space by using the coordinate offset to obtain the coordinates of all the characteristic points in the first image and the second image in the continuous space, and determining the positions of the corresponding characteristic points based on the coordinates.
5. The image stitching method according to claim 4, characterized in that the feature points are described to obtain corresponding feature point descriptors by:
constructing a circle neighborhood with a preset radius by taking any feature point in the feature point set as a center, and determining pixel points included in the circle neighborhood according to the position coordinates of any feature point;
describing each pixel point in the circular neighborhood by adopting CLBP (compact circular bit back) coding to obtain a descriptor of the pixel point: contrast binary pattern
Figure FDA0002559731450000022
Positive and negative binary pattern
Figure FDA0002559731450000023
And amplitude binary pattern
Figure FDA0002559731450000031
Descriptor contrast binary pattern based on multiple pixel points in circle neighborhood
Figure FDA0002559731450000032
Positive and negative binary pattern
Figure FDA0002559731450000033
And amplitude binary pattern
Figure FDA0002559731450000034
Obtaining a statistical histogram corresponding to the characteristic point by the corresponding coding mode;
converting the statistical histogram into a feature vector, and performing normalization processing on the feature vector to obtain a descriptor of a feature point;
and traversing each feature point to obtain a descriptor set corresponding to the feature point set.
6. The image stitching method according to claim 5, wherein the descriptor of each pixel point is obtained by the following formula:
Figure FDA0002559731450000035
Figure FDA0002559731450000036
CLBP_CP,R＝s(gc-cI)，
Figure FDA0002559731450000037
Figure FDA0002559731450000038
Figure FDA0002559731450000039
wherein P is the number of pixel points in a sampling region with the pixel point to be described as the center, R is the radius of the sampling region, gcThe gray value of the pixel point to be described is obtained; gpThe gray value of the P-th pixel point in the sampling area is obtained; m ispThe absolute value of the difference of the gray values of the P-th pixel point and the pixel point to be described in the sampling region is obtained; c is all m in the circle neighborhood of any one feature pointpThe mean value of (a); c. CIMean value of pixel values, U (LBP), of all pixel points in the circular neighborhood of any feature pointP,R) Is a positive and negative binary pattern
Figure FDA0002559731450000041
Number of coding cycle transitions, U (CLBP _ M)P,R) As amplitude binary patterns
Figure FDA0002559731450000042
The number of code cycle hops.
7. The image stitching method according to claim 1, wherein the initial matching of the feature points is performed by using a nearest neighbor ratio classifier to obtain a feature point pair set, which is specifically as follows:
acquiring the Euclidean distance between any feature point in the feature point set of the first image and each feature point in the feature point set of the second image to be spliced, and determining the Euclidean distance between the nearest feature point and the next nearest feature point and any feature point in the feature point set of the first image;
if the ratio of the Euclidean distance between the nearest neighbor feature point and the second nearest neighbor feature point is smaller than a preset value, judging that any feature point in the first image is matched with the nearest neighbor feature point in the second image and is a feature point pair;
and traversing all the feature points in the first image feature point set to obtain a feature point set.
8. The image stitching method according to claim 4, wherein the mismatching feature point pairs in the feature point pair set are eliminated by:
step 1, randomly selecting four groups of characteristic point pairs from the preliminarily obtained characteristic point pair set, and obtaining a projection transformation matrix between a first image and a second image based on coordinate data of the characteristic point pairs;
step 2, calculating the projection error of each characteristic point pair in the characteristic point pair set and the projection transformation matrix; if the projection error is smaller than a second preset threshold value, the characteristic point pairs are counted into an inner point set;
step 3, comparing the number of the characteristic point pairs in the inner point set with a threshold value;
taking the number of the feature point pairs in the inner point set obtained by the first calculation as a threshold, repeating the steps S1 to S2, obtaining the number of the feature point pairs in the corresponding inner point set, comparing the number with the result (namely, the threshold) obtained by the first calculation, taking the current inner point set as a preferred inner point set if the number is larger than the threshold, taking the inner point set obtained at the previous time as the preferred inner point set if the number is smaller than the threshold, performing iteration for a preset number of times, namely, taking the number of the feature point pairs in the preferred inner point set as the threshold, repeating the steps S1 to S3, and taking the preferred inner point set obtained at the last iteration as an optimal inner point set, namely, eliminating the feature point pair set of the mismatching feature point pairs.
9. The image stitching method according to claim 8, wherein the first image and the second image are fused to obtain a stitched image by:
transforming the first image and the second image into the same plane by using the optimal projective transformation matrix;
in the same plane, acquiring the proportion of each pixel point in the overlapping area of the first image and the second image, and acquiring a fusion coefficient of the first image and the second image according to the proportion;
fusing the first image and the second image based on the fusion coefficient by the following formula to obtain a spliced image:
Figure FDA0002559731450000051
wherein I (x, y) is the obtained stitched image, I1Is a set of pixel points, I, in the first image2For the set of pixel points in the second image, (x, y) represents the coordinates of the pixel points within the obtained stitched image, (x1,y1) Is the coordinate of the pixel point of the part of the first image which is not overlapped with the second image, (x)2,y2) Is the coordinate of the pixel point of the second image which is not overlapped with the first image, (x'1,y'1) Is the coordinate of the pixel point of the part of the first image which is overlapped with the second image, (x'2,y'2) And coordinates of pixel points of the second image which are overlapped with the first image are obtained.
10. The image stitching method according to claim 9, wherein the fusion coefficient of the first image and the second image is obtained by the following formula:
Figure FDA0002559731450000052
w2＝1-w1，xa<xi<xb，i＝1、2、3...n,
wherein x isiThe abscissa of the ith pixel point in the overlapping region of the first image and the second image, n is the number of the pixel points in the overlapping region, and xaIs the abscissa, x, of the left edge of the overlap regionbIs the abscissa of the right edge of the overlap region, w1The fusion coefficient w of the ith pixel point in the overlapping region relative to the first image2And the fusion coefficient of the ith pixel point in the overlapping area relative to the second image is obtained.
CN202010607671.XA 2020-06-29 2020-06-29 High-real-time image splicing method Pending CN111754408A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202010607671.XA CN111754408A (en) 2020-06-29 2020-06-29 High-real-time image splicing method

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010607671.XA CN111754408A (en) 2020-06-29 2020-06-29 High-real-time image splicing method

Publications (1)

Publication Number Publication Date
CN111754408A true CN111754408A (en) 2020-10-09

Family

ID=72678098

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010607671.XA Pending CN111754408A (en) 2020-06-29 2020-06-29 High-real-time image splicing method

Country Status (1)

Country Link
CN (1) CN111754408A (en)

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130223683A1 (en) * 2012-02-24 2013-08-29 Canon Kabushiki Kaisha Method and Apparatus for Generating Image Description Vector, Image Detection Method and Apparatus
CN107610183A (en) * 2017-09-25 2018-01-19 天津大学 New striped projected phase height conversion mapping model and its scaling method
CN107918927A (en) * 2017-11-30 2018-04-17 武汉理工大学 A kind of matching strategy fusion and the fast image splicing method of low error
CN111080529A (en) * 2019-12-23 2020-04-28 大连理工大学 Unmanned aerial vehicle aerial image splicing method for enhancing robustness

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130223683A1 (en) * 2012-02-24 2013-08-29 Canon Kabushiki Kaisha Method and Apparatus for Generating Image Description Vector, Image Detection Method and Apparatus
CN107610183A (en) * 2017-09-25 2018-01-19 天津大学 New striped projected phase height conversion mapping model and its scaling method
CN107918927A (en) * 2017-11-30 2018-04-17 武汉理工大学 A kind of matching strategy fusion and the fast image splicing method of low error
CN111080529A (en) * 2019-12-23 2020-04-28 大连理工大学 Unmanned aerial vehicle aerial image splicing method for enhancing robustness

Non-Patent Citations (4)

* Cited by examiner, † Cited by third party
Title
ZHENHUA GUO ET AL.: ""A Completed Modeling of Local Binary Pattern Operator for Texture Classification"", 《IEEE TRANSACTIONS ON IMAGE PROCESSING》 *
储蓄: ""基于改进SURF算法图像匹配方法研究"", 《CNKI硕士电子期刊》 *
毋小省等: "基于改进的区域纹理谱描述符的图像检索", 《激光与红外》 *
许晓帆等: ""基于自适应非极大值抑制的 SIFT 改进算法"", 《电子设计工程》 *

Similar Documents

Publication Publication Date Title
CN109118473B (en) Angular point detection method based on neural network, storage medium and image processing system
CN111667506B (en) Motion estimation method based on ORB feature points
CN111445389A (en) Wide-view-angle rapid splicing method for high-resolution images
CN109858527B (en) Image fusion method
CN109559273B (en) Quick splicing method for vehicle bottom images
CN113592923B (en) Batch image registration method based on depth local feature matching
CN109712071A (en) Unmanned plane image mosaic and localization method based on track constraint
CN111199558A (en) Image matching method based on deep learning
Flenner et al. Resampling forgery detection using deep learning and a-contrario analysis
CN114359591A (en) Self-adaptive image matching algorithm with edge features fused
CN103955950A (en) Image tracking method utilizing key point feature matching
CN117557565B (en) Detection method and device for lithium battery pole piece
Zhao et al. Learning probabilistic coordinate fields for robust correspondences
CN106934395B (en) Rigid body target tracking method adopting combination of SURF (speeded Up robust features) and color features
Li et al. Efficient properties-based learning for mismatch removal
CN112101283A (en) Intelligent identification method and system for traffic signs
CN111754408A (en) High-real-time image splicing method
Yao et al. ORB Feature Matching Algorithm Based on Multi-Scale Feature Description Fusion and Feature Point Mapping Error Correction
CN113221914B (en) Image feature point matching and mismatching elimination method based on Jacobsad distance
CN115439669A (en) Feature point detection network based on deep learning and cross-resolution image matching method
Kang et al. An adaptive fusion panoramic image mosaic algorithm based on circular LBP feature and HSV color system
Luanyuan et al. MGNet: Learning Correspondences via Multiple Graphs
Shen et al. A detector-oblivious multi-arm network for keypoint matching
CN110059651B (en) Real-time tracking and registering method for camera
CN111754402A (en) Image splicing method based on improved SURF algorithm

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination