CN115661368B - Image matching method, device, server and storage medium - Google Patents
Image matching method, device, server and storage medium Download PDFInfo
- Publication number
- CN115661368B CN115661368B CN202211598037.XA CN202211598037A CN115661368B CN 115661368 B CN115661368 B CN 115661368B CN 202211598037 A CN202211598037 A CN 202211598037A CN 115661368 B CN115661368 B CN 115661368B
- Authority
- CN
- China
- Prior art keywords
- image
- matching
- matched
- acquiring
- trained
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 53
- 238000013528 artificial neural network Methods 0.000 claims abstract description 50
- 238000000605 extraction Methods 0.000 claims abstract description 50
- 238000012545 processing Methods 0.000 claims abstract description 26
- 238000004458 analytical method Methods 0.000 claims description 38
- 229920001651 Cyanoacrylate Polymers 0.000 claims description 16
- 239000004830 Super Glue Substances 0.000 claims description 16
- 230000008520 organization Effects 0.000 claims description 13
- 230000004044 response Effects 0.000 claims description 13
- 239000011159 matrix material Substances 0.000 claims description 11
- 238000012937 correction Methods 0.000 claims description 6
- 238000005457 optimization Methods 0.000 claims description 3
- 230000000694 effects Effects 0.000 abstract description 11
- 238000012549 training Methods 0.000 description 8
- 230000008569 process Effects 0.000 description 5
- 238000004364 calculation method Methods 0.000 description 4
- 238000010586 diagram Methods 0.000 description 4
- 230000006870 function Effects 0.000 description 2
- 230000004888 barrier function Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 239000003086 colorant Substances 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000002349 favourable effect Effects 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000011176 pooling Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/30—Computing systems specially adapted for manufacturing
Landscapes
- Image Analysis (AREA)
Abstract
The application provides an image matching method, an image matching device, a server and a storage medium. The method specifically comprises the following steps: acquiring an image set to be matched, and acquiring an image pre-pairing algorithm corresponding to an acquisition type; performing pre-matching processing on images in an image set to be matched to obtain a plurality of initial image matching pairs; acquiring a characteristic region block of each image in an image set to be matched, and acquiring a characteristic descriptor corresponding to the image by adopting a pre-trained neural network and a characteristic extraction network; and acquiring a matching result of the image set to be matched by adopting a pre-trained matching network so as to realize image three-dimensional reconstruction processing based on the matching result. By the method, the neural network participates in the extraction efficiency of the feature descriptors corresponding to the images in the image set to be matched, the matching accuracy is improved, and the efficiency and the effect of three-dimensional reconstruction according to the image set to be matched are improved.
Description
Technical Field
The present application relates to the field of image matching technologies, and in particular, to an image matching method, an image matching apparatus, a server, and a storage medium.
Background
When the image information is used for three-dimensional reconstruction of the target object, sufficient feature information needs to be extracted according to the image information, and image matching is completed according to the feature information, so that more complete three-dimensional information of the target object can be calculated, and the three-dimensional reconstruction is realized. Therefore, completing image matching is an important step for realizing three-dimensional reconstruction.
In the prior art, scale Invariant Feature Transform (SIFT) feature extraction algorithm is often adopted for image matching, the complexity is high, the real-time requirement is difficult to meet, random Sample Consensus (RANSAC) algorithm which is often used for image matching has no upper limit on iteration times, an optimal solution cannot be obtained frequently, image matching is difficult to realize, the time required by three-dimensional reconstruction is long, and the obtained effect is poor.
Therefore, rapidly and accurately completing image matching is still a technical problem to be solved urgently.
Disclosure of Invention
The application provides an image matching method, an image matching device, a server and a storage medium, which are used for solving the technical problems that in the prior art, the complexity of a feature extraction algorithm is high, an optimal solution is difficult to obtain by an image matching algorithm, so that images are difficult to match, and the three-dimensional reconstruction time is long and the effect is poor.
In a first aspect, the present application provides an image matching method, including:
acquiring an image set to be matched, and acquiring an image pre-matching algorithm corresponding to the acquisition type according to the acquisition type of the image set to be matched;
performing pre-matching processing on the images in the image set to be matched according to the image pre-matching algorithm to obtain a plurality of initial image matching pairs;
for each image in the image set to be matched, acquiring a characteristic region block of the image, and acquiring a characteristic descriptor corresponding to the image by adopting a pre-trained organization neural network, a pre-trained AffiniNet neural network and a HardNet characteristic extraction network;
and acquiring a matching result of the image set to be matched by adopting a pre-trained SuperGlue matching network according to the characteristic descriptors corresponding to the initial image matching pairs and each image in the image set to be matched so as to realize image three-dimensional reconstruction processing based on the matching result.
In a possible implementation manner, the acquiring, for each image in the image set to be matched, a feature region block of the image includes:
for each image in the image set to be matched, carrying out black plug Hessain matrix analysis on the image to obtain an analysis result;
obtaining an inner corner point corresponding to the image according to the analysis result;
and taking the inner corner point as a center, and acquiring a characteristic region block of the image according to a first preset pixel length.
In a possible implementation manner, the performing blackplug Hessain matrix analysis on the image to obtain an analysis result includes:
for the imageIs calculated for each pixel point (x, y) in (a), the image is ∑ based>The second partial derivatives in the x-direction, the y-direction and the second derivative in the xy-direction are taken into consideration in order to obtain a Hessain matrix->;
For each pixel point (x, y), respectively obtaining a Hessain response value corresponding to each pixel point (x, y);
Corresponding said each pixel pointAnd arranging according to the corresponding position relation of the pixel points on the image to obtain the analysis result.
In a possible implementation manner, obtaining an internal corner corresponding to the image according to the analysis result includes:
according to the analysis result, adopting a formula:
obtaining the area corresponding to the second preset pixel length extending to the vicinity by taking the corner point as the centerPoints less than zero are internal angle points;
wherein,the Hessain response value corresponding to the pixel point (x, y) in the image is obtained; />In question for the image>First derivative in x-direction; />Is the image->The first derivative in the y direction.
In a possible implementation manner, the obtaining a feature descriptor corresponding to the image by using a pre-trained origin neural network, a pre-trained affinity net neural network, and a HardNet feature extraction network includes:
performing direction and deformation correction processing on the characteristic region block by adopting the pre-trained optimization neural network and the pre-trained AffiniNet neural network to obtain a characteristic region image corresponding to the corrected characteristic region block;
and extracting a preset number of feature descriptors corresponding to the feature region image by adopting the HardNet feature extraction network according to the feature region image.
In one possible implementation, the method further includes:
for each image in the image set to be matched, acquiring a gray level image corresponding to the image, and acquiring a color gradient corresponding to the image by adopting a Harris algorithm according to the gray level image;
then, the obtaining a feature region block of the image according to a first preset pixel length with the inner corner as a center includes:
and taking the inner corner point as a center, and acquiring a characteristic region block of the image according to the first preset pixel length and the color gradient.
In a second aspect, the present application provides an image matching apparatus, comprising:
the image acquisition module is used for acquiring an image set to be matched and acquiring an image pre-matching algorithm corresponding to the acquisition type according to the acquisition type of the image set to be matched;
the pre-matching module is used for performing pre-matching processing on the images in the image set to be matched according to the image pre-matching algorithm to obtain a plurality of initial image matching pairs;
the characteristic extraction module is used for acquiring a characteristic region block of each image in the image set to be matched, and acquiring a characteristic descriptor corresponding to the image by adopting a pre-trained organization neural network, a pre-trained AffiniNet neural network and a HardNet characteristic extraction network;
and the image matching module is used for acquiring the matching result of the image set to be matched by adopting a pre-trained SuperGlue matching network according to the characteristic descriptors corresponding to the initial image matching pairs and each image in the image set to be matched so as to realize image three-dimensional reconstruction processing based on the matching result.
In addition, optionally, the feature extraction module is specifically configured to:
for each image in the image set to be matched, carrying out black plug Hessain matrix analysis on the image to obtain an analysis result;
obtaining an inner corner point corresponding to the image according to the analysis result;
and taking the inner corner as a center, and acquiring a characteristic region block of the image according to a first preset pixel length.
Optionally, the feature extraction module is specifically configured to:
for the imageIs calculated for each pixel point (x, y) in (a), the image is ∑ based>The second partial derivatives in the x-direction, the y-direction and the xy-direction are obtained to obtain the Hessain matrix->;
For each pixel point (x, y), respectively obtaining a Hessain response value corresponding to each pixel point (x, y);
Corresponding said each pixel pointAnd arranging according to the corresponding position relation of the pixel points on the image to obtain the analysis result.
Optionally, the feature extraction module is specifically configured to:
according to the analysis result, adopting a formula:
obtaining the area corresponding to the second preset pixel length extending to the vicinity by taking the corner point as the centerPoints less than zero are internal angle points;
wherein,the Hessain response value corresponding to the pixel point (x, y) in the image is obtained; />In question for the image>The first derivative in the x direction; />Is the image->The first derivative in the y direction.
Optionally, the image matching module is specifically configured to:
performing direction and deformation correction processing on the characteristic region block by adopting the pre-trained organization neural network and the pre-trained AffiniNet neural network to obtain a characteristic region image corresponding to the corrected characteristic region block;
and extracting a preset number of feature descriptors corresponding to the feature region image by adopting the HardNet feature extraction network according to the feature region image.
Optionally, the feature extraction module is further configured to:
for each image in the image set to be matched, acquiring a gray level image corresponding to the image, and acquiring a color gradient corresponding to the image by adopting a Harris algorithm according to the gray level image;
the feature extraction module is specifically configured to:
and taking the inner corner point as a center, and acquiring a characteristic region block of the image according to the first preset pixel length and the color gradient.
In a third aspect, the present application provides a server, comprising:
a processor and a memory;
the memory is used for storing executable instructions of the processor;
wherein the processor is configured to perform the method of the first aspect via execution of the executable instructions.
In a fourth aspect, the present application provides a computer-readable storage medium having stored thereon computer-executable instructions for implementing the method according to the first aspect when executed by a processor.
The application provides an image matching method, an image matching device, a server and a storage medium. The method specifically comprises the following steps: acquiring an image set to be matched, and acquiring an image pre-matching algorithm corresponding to an acquisition type according to the acquisition type of the image set to be matched; performing pre-matching pair processing on images in an image set to be matched according to an image pre-matching pair algorithm to obtain a plurality of initial image matching pairs; for each image in an image set to be matched, acquiring a characteristic region block of the image, and acquiring a characteristic descriptor corresponding to the image by adopting a pre-trained organization neural network, a pre-trained AffiniNet neural network and a HardNet characteristic extraction network; and acquiring a matching result of the image set to be matched by adopting a pre-trained SuperGlue matching network according to the characteristic descriptors corresponding to the plurality of initial image matching pairs and each image in the image set to be matched so as to realize image three-dimensional reconstruction processing based on the matching result. By the method, images in the image sets to be matched are pre-paired, the feature descriptors corresponding to the images are obtained by adopting a pre-trained Orientation neural network, a pre-trained Affininet neural network and a HardNet feature extraction network, finally, the pre-paired initial image pair and the feature descriptors corresponding to the initial image pair are input into a pre-trained SuperGlue matching network, and the matching result of the image sets to be matched is obtained.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings used in the embodiments or the description of the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present application, and those skilled in the art can obtain other drawings without inventive labor.
Fig. 1 is a schematic flowchart of a first embodiment of an image matching method provided in the present application;
fig. 2 is a schematic flowchart of a second embodiment of an image matching method provided in the present application;
FIG. 3 is a schematic structural diagram of an embodiment of an image matching apparatus provided in the present application;
fig. 4 is a schematic structural diagram of an embodiment of a server provided in the present application.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some embodiments of the present application, but not all embodiments. All other embodiments that can be made by one skilled in the art based on the embodiments in the present application in light of the present disclosure are within the scope of the present application.
The terms "first," "second," "third," "fourth," and the like in the description of the application and the above-described figures, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the application described herein are capable of operation in sequences other than those illustrated or described herein. Moreover, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
When the target object is three-dimensionally reconstructed through the image information of the target object, accurate three-dimensional information corresponding to the target object needs to be calculated according to the two-dimensional image information of the target object, and in order to obtain the three-dimensional information, pixel identification and alignment, that is, image matching, needs to be performed on the target object. In the prior art, a commonly used SIFT feature extraction algorithm in the image matching process is high in complexity, high time and resource cost are consumed in calculation, the real-time requirement is difficult to meet, a commonly used RANSAC algorithm in the matching process of the extracted feature descriptors estimates parameters of model fitting data from a point set containing inner points and outer points in an iteration mode, the iteration frequency of the algorithm has no upper limit, the calculation amount is large, an optimal solution cannot be obtained frequently, and the matching effect is unstable. The three-dimensional reconstruction performed by image matching based on the two algorithms is time-consuming and poor in effect.
Therefore, the technical idea of the present application is that: how to improve the image matching efficiency and accuracy so as to save time and improve the reconstruction effect when performing three-dimensional reconstruction according to image information.
The technical solution of the present application will be described in detail below with reference to specific examples. It should be noted that the following specific embodiments may be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.
Fig. 1 is a schematic flowchart of a first embodiment of an image matching method provided in the present application. Referring to fig. 1, the method includes:
s101, obtaining an image set to be matched, and obtaining an image pre-matching algorithm corresponding to the collection type according to the collection type of the image set to be matched.
In this embodiment, the image set to be matched is an image set captured for a target object to be three-dimensionally reconstructed, images in the image set may be captured by corresponding devices at different times and different positions and at different angles, and images with different dimensional characteristics of the target object may be reflected, so that each image in the image set to be matched has different degrees of deformation in directions and angles with respect to a front view of the target object in each direction. Optionally, the device may be a digital camera, a mobile phone, a tablet computer, or other devices with an image capturing function.
In this embodiment, the acquisition type of the image set to be matched includes a type of a device used for acquiring an image, an environment type when the image is acquired, the number of the acquired images, and the like, and the image pre-matching algorithm includes an exhaustion method, a bag-of-words method, a sequential sequence method, a Global Positioning System (GPS) assisted clustering method, and the like. If the number of the images in the image set to be matched does not exceed 200, acquiring an exhaustive method to pre-match the images of the image set to be matched; if the images in the image set to be matched are acquired by the same equipment and the different images are spaced at the same physical distance, acquiring a sequence method to pre-match the images of the image set to be matched with an algorithm; if the image set to be matched is in an outdoor environment during image acquisition and the acquisition equipment can mark a GPS of an acquisition position, acquiring a GPS assisted clustering method to pre-match an algorithm for the image of the image set to be matched; if the image set to be matched is in an outdoor environment during image acquisition but cannot acquire a GPS of an acquisition position, acquiring a bag-of-words method to pre-match an algorithm for the image of the image set to be matched.
Step S102, carrying out pre-matching pair processing on the images in the image set to be matched according to an image pre-matching pair algorithm, and obtaining a plurality of initial image matching pairs.
In this embodiment, according to a pre-pairing algorithm obtained according to the acquisition type of the image set to be matched, pre-pairing processing is performed on images in the image set to be matched, so as to obtain a plurality of pairwise matched initial image matching pairs corresponding to the image set to be matched.
In this embodiment, if the image pre-pairing algorithm of the image set to be matched is obtained by the exhaustive method, pre-pairing is performed on each image in the image set to be matched and the other images; if the sequence method is acquired to be the image pre-pairing algorithm of the image set to be matched, each image in the image set to be matched can be only pre-paired with the image with the closest physical interval during shooting; if the GPS assisted clustering method is acquired to perform the image pre-matching algorithm for the image set to be matched, the images in the image set to be matched can be grouped according to the GPS information and then pre-matched, and the two images with the most similar GPS information are always used as the most preferable and matched pair; and if the acquired bag-of-words method is the image pre-pairing algorithm of the image set to be matched, performing image pre-pairing based on the bag-of-words model.
Step S103, for each image in the image set to be matched, obtaining a characteristic region block of the image, and obtaining a characteristic descriptor corresponding to the image by adopting a pre-trained Orientation neural network, a pre-trained AffiniNet neural network and a HardNet characteristic extraction network.
In this embodiment, for each image in an image set to be matched, a feature region block of the image is obtained, and a feature descriptor corresponding to the image is obtained by using a pre-trained optimization neural network, a pre-trained AffineNet neural network, and a HardNet feature extraction network. The pre-trained neural network has strong information comprehensive capability, self-learning, self-organization and self-adaptability, can well coordinate various input information relations, has high timeliness when a plurality of images are processed in parallel, and has strong robustness.
And S104, acquiring a matching result of the image set to be matched by adopting a pre-trained SuperGlue matching network according to the characteristic descriptors corresponding to the plurality of initial image matching pairs and each image in the image set to be matched so as to realize image three-dimensional reconstruction processing based on the matching result.
In this embodiment, a plurality of initial image matching pairs and feature descriptors corresponding to images in each matching pair are input into a pre-trained SuperGlue matching network, if a ratio of a feature descriptor that can be matched by the input initial image matching pair to a feature descriptor corresponding to an image in the input matching pair exceeds a preset threshold, it is determined that two images in the initial image matching pair can be matched, and a matching result is output as information of a successfully matched initial image matching pair, so as to implement image three-dimensional reconstruction processing based on the matching result. Optionally, the preset threshold is labeled match _ threshold, and a specific parameter value thereof may be set to 0.2.
In this embodiment, the pre-trained SuperGlue matching network is an initial matching pair of an image set obtained based on history, a feature descriptor obtained by the pre-trained orientitation neural network, the pre-trained AffineNet neural network and the HardNet feature extraction network is obtained as an input of a training set, a matching result of the initial matching pair marked with the history image set is used as an output of the training set, and the original SuperGlue matching network is trained to obtain the pre-trained SuperGlue matching network. As the feature descriptors acquired by the HardNet feature extraction network in the input set of the training set are richer than those acquired by the original SuperPoint feature extraction network and have higher confidence, the pre-trained SuperGlue matching network has higher and more accurate matching precision than the original SuperGlue matching network.
In the embodiment, an image pre-matching algorithm corresponding to an acquisition type is acquired by acquiring an image set to be matched and according to the acquisition type of the image set to be matched; performing pre-matching processing on images in an image set to be matched according to an image pre-matching algorithm to obtain a plurality of initial image matching pairs; for each image in an image set to be matched, acquiring a characteristic region block of the image, and acquiring a characteristic descriptor corresponding to the image by adopting a pre-trained organization neural network, a pre-trained AffiniNet neural network and a HardNet characteristic extraction network; and acquiring a matching result of the image set to be matched by adopting a pre-trained SuperGlue matching network according to the characteristic descriptors corresponding to the plurality of initial image matching pairs and each image in the image set to be matched so as to realize image three-dimensional reconstruction processing based on the matching result. By the method provided by the embodiment, images in an image set to be matched are pre-paired, a pre-trained organization neural network, a pre-trained AffiniNet neural network and a HardNet feature extraction network are adopted to obtain feature descriptors corresponding to the images, finally, a pre-paired initial image pair and feature descriptors corresponding to the initial image pair are input into a pre-trained SuperGlue matching network to obtain a matching result of the image set to be matched, the participation of the neural network improves the extraction efficiency of the feature descriptors corresponding to each image in the image set to be matched, improves the matching accuracy, and improves the efficiency and effect of three-dimensional reconstruction according to the image set to be matched.
On the basis of the above embodiment, a specific implementation manner of obtaining the feature descriptor corresponding to the image by using the pre-trained organization neural network, the pre-trained AffineNet neural network and the HardNet feature extraction network in step S103 includes:
and performing direction and deformation correction processing on the characteristic region block by adopting a pre-trained organization neural network and a pre-trained AffiniNet neural network to obtain a characteristic region image corresponding to the corrected characteristic region block.
And extracting a preset number of feature descriptors corresponding to the feature region images by adopting a HardNet feature extraction network according to the feature region images.
In this embodiment, the pre-trained origin neural network and the pre-trained AffineNet neural network are obtained by training the origin neural network and the AffineNet neural network, with feature region blocks having different degrees of directions and deformations due to different acquisition directions and angles as inputs of a training set, and with corrected corresponding feature region images as outputs of the training set. The training set can be adjusted according to the acquisition type of the image set to be matched, illustratively, when an outdoor building is taken as a target object, because different lenses of the five-eye camera have a fixed angle relationship, the shooting angles of the data source can be enriched, and the unmanned aerial vehicle can prevent interference caused by barrier shielding during overhead shooting, so that the training set can use an image acquired by the unmanned aerial vehicle carrying the five-eye camera as a source image set of a characteristic region block; when indoor still is taken as a target object, an image acquisition environment with single background color needs to be established, and an image obtained by shooting with the same equipment is taken as a source image set of a characteristic region block so as to control variables as much as possible.
In this embodiment, the HardNet feature extraction network Padding all convolution layers with a round of 0 (except the last convolution layer) to ensure that the image size is unchanged, and no pooling layer is provided to ensure the feature descriptor performance. And extracting a preset number of feature descriptors corresponding to the feature region images by using a HardNet feature extraction network. Alternatively, the preset number may be 128 dimensions.
Fig. 2 is a schematic flowchart of a second embodiment of an image matching method provided in the present application. Referring to fig. 2, on the basis of the above-mentioned embodiment of fig. 1, in step S103, for each image in the image set to be matched, a specific implementation manner of obtaining the feature region block of the image is as follows:
step S201, for each image in the image set to be matched, carrying out black plug Hessain matrix analysis on the image, and obtaining an analysis result.
In one possible implementation manner, step S201 further includes:
for imagesIs calculated for each pixel point (x, y) in (a), the image is ∑ based>The second partial derivatives in the x-direction, the y-direction and the xy-direction are obtained to obtain the Hessain matrix->。
For each pixel point (x, y), respectively obtaining a Hessain response value corresponding to each pixel point (x, y)。
Corresponding each pixel pointAnd arranging according to the corresponding position relation of the pixel points on the image to obtain an analysis result.
In this embodiment, for each image in the set of images to be matched, the image is processedIs calculated for each pixel point (x, y) in (a), the image is ∑ based>The second partial derivatives in the x-direction, the y-direction and the xy-direction are obtained to obtain the Hessain matrix->And obtaining Hessain response value corresponding to each pixel point (x, y)The corresponding ^ or ^ of each pixel point>And arranging according to the corresponding position relation of the pixel points on the image to obtain a Hessain matrix analysis result with the same size as the analyzed image.
And S202, acquiring inner corner points corresponding to the images according to the analysis result.
In a specific implementation manner, step S202 further includes:
according to the analysis result, the formula is adopted:
Obtaining the area corresponding to the second preset pixel length extending to the vicinity by taking the corner point as the centerPoints less than zero are interior angles.
Wherein,the Hessain response value corresponding to the pixel point (x, y) in the image is obtained; />Is an image->The first derivative in the x direction; />Is an image->The first derivative in the y direction.
In the present embodiment, it is preferred that,is maximum and->The point of (A) corresponds to a point of increased Gaussian curvature, so willIs maximum and->The point of (a) is defined as the corner point. Each image in the set of images to be matched has a plurality of corner points. The corner points are important features of the image, and the corner points can effectively reduce the data volume of information while keeping the important features of the image and the figure, effectively improve the calculation speed, are favorable for reliable matching of the image and have important effects on understanding and analyzing the image and the figure.
In this embodiment, in order to make the obtained feature region block include more effective feature points, an area corresponding to a second preset pixel length extending to the vicinity with the corner point as the center is obtainedPoints less than zero are interior angle points. The inner corner point is a local maximum point of a region which extends near the corner point and corresponds to the second preset pixel length, contains more image information of the target object, is more stable than the corner point, and is easier to obtain an effective characteristic region block. Alternatively, the second preset pixel length may be 4 pixels.
And S203, taking the inner corner point as a center, and acquiring a characteristic region block of the image according to the first preset pixel length.
In this embodiment, the feature region block corresponding to one interior corner may be a plurality of feature region blocks obtained according to different first preset pixel lengths. Optionally, the first preset pixel length may be any pixel length in a range from 15 pixels to 25 pixels, and 6 different pixel lengths may be determined in the range to obtain 6 feature region blocks with different sizes.
In this embodiment, internal corner points including more image information are determined according to a result of analysis of a black matrix of each image in an image set to be matched, and one or more feature region blocks of the image are obtained according to the internal corner points, namely, seed points, as a center and according to a first preset pixel length. The characteristic region block acquired by taking the inner corner as the central point has effective characteristics of more target objects, the effectiveness of the subsequently acquired characteristic descriptors can be guaranteed, the image matching accuracy is improved, and the imaging effect of three-dimensional reconstruction is guaranteed.
On the basis of the embodiment of fig. 2, the method further includes:
and for each image in the image set to be matched, acquiring a gray image corresponding to the image, and acquiring a color gradient corresponding to the image by adopting a Harris algorithm according to the gray image.
Then, taking the inner corner point as the center, and obtaining the characteristic region block of the image according to the first preset pixel length, including:
and taking the inner corner point as a center, and acquiring a characteristic region block of the image according to the first preset pixel length and the color gradient.
In this embodiment, in an R (red) G (green) B (blue) type color image shot by a device having an image shooting function in an image set to be matched, in an RGB model, assuming that three channels have equal values, that is, R = G = B, color information of the pixel point may be represented by the same gray scale value, and the range of the gray scale value is 0 to 255. After the gray level image of each image in the image set to be matched is obtained, the calculation amount of the image can be reduced, and the Harris algorithm can be adopted conveniently to obtain the color gradient corresponding to the image. The color gradient may embody the image information represented by different colors in the image.
In this embodiment, the feature region block including the image color gradient information may be obtained according to the first preset pixel length and the color gradient with the inner corner point as the center. The first predetermined pixel length may be anywhere from 15 pixels to 25 pixels.
Fig. 3 is a schematic structural diagram of an embodiment of an image matching apparatus provided in the present application. Referring to fig. 3, the image matching apparatus 300 includes: an image acquisition module 301, a pre-matching module 302, a feature extraction module 303 and an image matching module 304. The image acquisition module 301 is configured to acquire an image set to be matched, and acquire an image pre-pairing algorithm corresponding to an acquisition type according to the acquisition type of the image set to be matched; the pre-pairing module 302 is configured to perform pre-pairing processing on images in an image set to be matched according to an image pre-pairing algorithm to obtain a plurality of initial image matching pairs; the feature extraction module 303 is configured to acquire a feature region block of each image in the image set to be matched, and acquire a feature descriptor corresponding to the image by using a pre-trained organization neural network, a pre-trained AffineNet neural network and a HardNet feature extraction network; and the image matching module 304 is configured to obtain a matching result of the image set to be matched by using a pre-trained SuperGlue matching network according to the plurality of initial image matching pairs and the feature descriptor corresponding to each image in the image set to be matched, so as to implement image three-dimensional reconstruction processing based on the matching result.
In addition, optionally, the feature extraction module 303 is specifically configured to: for each image in the image set to be matched, carrying out black plug Hessain matrix analysis on the image to obtain an analysis result; obtaining an inner angle point corresponding to the image according to the analysis result; and taking the inner corner point as a center, and acquiring a characteristic region block of the image according to the first preset pixel length.
Optionally, the feature extraction module 303 is specifically configured to: for imagesIs calculated for each pixel point (x, y) in (a), the image is ∑ based>Obtaining a Hessain matrix by using the second-order partial derivatives in the x direction and the y direction and the second-order derivative in the xy direction(ii) a For each pixel point (x, y), respectively obtaining a Hessain response value corresponding to each pixel point (x, y)(ii) a Corresponding each pixel point is judged>And arranging according to the corresponding position relation of the pixel points on the image to obtain an analysis result.
Optionally, the feature extraction module 303 is specifically configured to: according to the analysis result, the formula is adopted:acquire an image pick>The Gaussian curvature corresponding to each pixel point (x, y) is->(ii) a Based on the Gaussian curvature>The enlarged points are taken as angular points of the image; acquiring the position in a region corresponding to a second preset pixel length extending to the vicinity by taking the corner point as the center>Points less than zero are internal angle points; wherein +>The Hessain response value corresponding to the pixel point (x, y) in the image is obtained; />Is an image->The first derivative in the x direction; />Is an image->The first derivative in the y direction.
Optionally, the image matching module 304 is specifically configured to: adopting a pre-trained Orientation neural network and a pre-trained Affininet neural network to perform direction and deformation correction processing on the characteristic region block so as to obtain a characteristic region image corresponding to the corrected characteristic region block; and extracting a preset number of feature descriptors corresponding to the feature region images by adopting a HardNet feature extraction network according to the feature region images.
Optionally, the feature extraction module 303 is further configured to: acquiring a gray level image corresponding to each image in an image set to be matched, and acquiring a color gradient corresponding to the image by adopting a Harris algorithm according to the gray level image; the feature extraction module 303 is specifically configured to: and taking the inner corner point as a center, and acquiring a characteristic region block of the image according to the first preset pixel length and the color gradient.
The image matching device provided in the embodiment of the present application can implement the technical solution of any one of the above method embodiments, and the implementation principle and the beneficial effect thereof are similar, and are not described herein again.
Fig. 4 is a schematic structural diagram of an embodiment of a server provided in the present application. Referring to fig. 4, the server 400 includes: a processor 401 and a memory 402.
The memory 402 is used to store executable instructions for the processor 401.
Wherein the processor 401 is configured to execute the technical solution in any of the foregoing method embodiments via executing the executable instructions.
The server provided by the present application is used for executing the technical solution in any of the foregoing method embodiments, and the implementation principle and the technical effect are similar, which are not described herein again.
The present application further provides a computer-readable storage medium, where computer-executable instructions are stored in the computer-readable storage medium, and when the computer-executable instructions are executed by a processor, the computer-executable instructions are used to implement the technical solution in any one of the foregoing method embodiments, which are similar in implementation principle and technical effect and are not described herein again.
Those of ordinary skill in the art will understand that: all or a portion of the steps of implementing the above-described method embodiments may be performed by hardware associated with program instructions. The program may be stored in a computer-readable storage medium. When executed, the program performs steps comprising the method embodiments described above; and the aforementioned storage medium includes: various media that can store program codes, such as ROM, RAM, magnetic or optical disks.
Finally, it should be noted that: the above embodiments are only used for illustrating the technical solutions of the present application, and not for limiting the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some or all of the technical features may be equivalently replaced; and the modifications or the substitutions do not make the essence of the corresponding technical solutions depart from the scope of the technical solutions of the embodiments of the present application.
Claims (12)
1. An image matching method, comprising:
acquiring an image set to be matched, and acquiring an image pre-matching algorithm corresponding to the acquisition type according to the acquisition type of the image set to be matched;
performing pre-matching processing on the images in the image set to be matched according to the image pre-matching algorithm to obtain a plurality of initial image matching pairs;
for each image in the image set to be matched, carrying out black plug Hessain matrix analysis on the image to obtain an analysis result;
acquiring an internal angle point corresponding to the image according to the analysis result;
taking the inner corner point as a center, acquiring a characteristic region block of the image according to a first preset pixel length, and acquiring a characteristic descriptor corresponding to the image by adopting a pre-trained organization neural network, a pre-trained AffiniNet neural network and a HardNet characteristic extraction network;
and acquiring a matching result of the image set to be matched by adopting a pre-trained SuperGlue matching network according to the characteristic descriptors corresponding to the initial image matching pairs and each image in the image set to be matched so as to realize image three-dimensional reconstruction processing based on the matching result.
2. The method according to claim 1, wherein performing a Hessain matrix analysis on the image to obtain an analysis result comprises:
for the imageIs calculated for each pixel point (x, y) in (a), the image is ∑ based>The second partial derivatives in the x-direction, the y-direction and the xy-direction are obtained to obtain the Hessain matrix->;
For each pixel point (x, y), respectively obtaining a Hessain response value corresponding to each pixel point (x, y);
3. The method of claim 2, wherein obtaining the internal corner corresponding to the image according to the analysis result comprises:
according to the analysis result, adopting a formula:
obtaining the area corresponding to the second preset pixel length extending to the vicinity by taking the corner point as the centerPoints less than zero are internal angle points;
4. The method as claimed in any one of claims 1 to 3, wherein the obtaining of the feature descriptor corresponding to the image by using the pre-trained organization neural network, the pre-trained AffiniNet neural network and the HardNet feature extraction network comprises:
performing direction and deformation correction processing on the characteristic region block by adopting the pre-trained optimization neural network and the pre-trained AffiniNet neural network to obtain a characteristic region image corresponding to the corrected characteristic region block;
and extracting a preset number of feature descriptors corresponding to the feature region image by adopting the HardNet feature extraction network according to the feature region image.
5. The method of claim 1, further comprising:
for each image in the image set to be matched, acquiring a gray image corresponding to the image, and acquiring a color gradient corresponding to the image by adopting a Harris algorithm according to the gray image;
then, the obtaining a feature region block of the image according to a first preset pixel length with the inner corner as a center includes:
and taking the inner corner point as a center, and obtaining a characteristic region block of the image according to the first preset pixel length and the color gradient.
6. An image matching apparatus, characterized by comprising:
the image acquisition module is used for acquiring an image set to be matched and acquiring an image pre-pairing algorithm corresponding to the acquisition type according to the acquisition type of the image set to be matched;
the pre-matching module is used for performing pre-matching processing on the images in the image set to be matched according to the image pre-matching algorithm to obtain a plurality of initial image matching pairs;
the characteristic extraction module is used for acquiring a characteristic region block of each image in the image set to be matched, and acquiring a characteristic descriptor corresponding to the image by adopting a pre-trained organization neural network, a pre-trained AffiniNet neural network and a HardNet characteristic extraction network;
the image matching module is used for acquiring a matching result of the image set to be matched by adopting a pre-trained SuperGlue matching network according to the plurality of initial image matching pairs and the feature descriptor corresponding to each image in the image set to be matched so as to realize image three-dimensional reconstruction processing based on the matching result;
the feature extraction module is specifically configured to:
for each image in the image set to be matched, carrying out black plug Hessain matrix analysis on the image to obtain an analysis result;
obtaining an inner corner point corresponding to the image according to the analysis result;
and taking the inner corner point as a center, and acquiring a characteristic region block of the image according to a first preset pixel length.
7. The apparatus of claim 6, wherein the feature extraction module is specifically configured to:
for the imageIs calculated for each pixel point (x, y) in (a), the image is ∑ based>The second partial derivatives in the x-direction, the y-direction and the xy-direction are obtained to obtain the Hessain matrix->;
For each pixel point (x, y), respectively obtaining a Hessain response value corresponding to each pixel point (x, y);
8. The apparatus of claim 7, wherein the feature extraction module is specifically configured to:
according to the analysis result, adopting a formula:
obtaining the area corresponding to the second preset pixel length extending to the vicinity by taking the corner point as the centerPoints less than zero are internal angle points;
9. The apparatus according to any one of claims 6 to 8, wherein the image matching module is specifically configured to:
performing direction and deformation correction processing on the characteristic region block by adopting the pre-trained organization neural network and the pre-trained AffiniNet neural network to obtain a characteristic region image corresponding to the corrected characteristic region block;
and extracting a preset number of feature descriptors corresponding to the feature region image by adopting the HardNet feature extraction network according to the feature region image.
10. The apparatus of claim 6, wherein the feature extraction module is further configured to:
for each image in the image set to be matched, acquiring a gray image corresponding to the image, and acquiring a color gradient corresponding to the image by adopting a Harris algorithm according to the gray image;
the feature extraction module is specifically configured to:
and taking the inner corner point as a center, and acquiring a characteristic region block of the image according to the first preset pixel length and the color gradient.
11. A server, comprising:
a processor and a memory;
the memory is used for storing executable instructions of the processor;
wherein the processor is configured to perform the method of any of claims 1-5 via execution of the executable instructions.
12. A computer-readable storage medium having computer-executable instructions stored therein, which when executed by a processor, are configured to implement the method of any one of claims 1 to 5.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202211598037.XA CN115661368B (en) | 2022-12-14 | 2022-12-14 | Image matching method, device, server and storage medium |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202211598037.XA CN115661368B (en) | 2022-12-14 | 2022-12-14 | Image matching method, device, server and storage medium |
Publications (2)
Publication Number | Publication Date |
---|---|
CN115661368A CN115661368A (en) | 2023-01-31 |
CN115661368B true CN115661368B (en) | 2023-04-11 |
Family
ID=85022936
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202211598037.XA Active CN115661368B (en) | 2022-12-14 | 2022-12-14 | Image matching method, device, server and storage medium |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN115661368B (en) |
Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN110009722A (en) * | 2019-04-16 | 2019-07-12 | 成都四方伟业软件股份有限公司 | Three-dimensional rebuilding method and device |
WO2020206903A1 (en) * | 2019-04-08 | 2020-10-15 | 平安科技(深圳)有限公司 | Image matching method and device, and computer readable storage medium |
Family Cites Families (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20100309225A1 (en) * | 2009-06-03 | 2010-12-09 | Gray Douglas R | Image matching for mobile augmented reality |
CN101984463A (en) * | 2010-11-02 | 2011-03-09 | 中兴通讯股份有限公司 | Method and device for synthesizing panoramic image |
CN110781911B (en) * | 2019-08-15 | 2022-08-19 | 腾讯科技(深圳)有限公司 | Image matching method, device, equipment and storage medium |
-
2022
- 2022-12-14 CN CN202211598037.XA patent/CN115661368B/en active Active
Patent Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2020206903A1 (en) * | 2019-04-08 | 2020-10-15 | 平安科技(深圳)有限公司 | Image matching method and device, and computer readable storage medium |
CN110009722A (en) * | 2019-04-16 | 2019-07-12 | 成都四方伟业软件股份有限公司 | Three-dimensional rebuilding method and device |
Also Published As
Publication number | Publication date |
---|---|
CN115661368A (en) | 2023-01-31 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
WO2020228446A1 (en) | Model training method and apparatus, and terminal and storage medium | |
US9530073B2 (en) | Efficient descriptor extraction over multiple levels of an image scale space | |
CN110232326B (en) | Three-dimensional object recognition method, device and storage medium | |
CN108345821B (en) | Face tracking method and device | |
JP2016513843A (en) | Reduction of object detection time by using feature spatial localization | |
CN103426190B (en) | The method and system of image reconstruction | |
WO2022179549A1 (en) | Calibration method and apparatus, computer device, and storage medium | |
CN112435223B (en) | Target detection method, device and storage medium | |
WO2020014913A1 (en) | Method for measuring volume of object, related device, and computer readable storage medium | |
CN112861666A (en) | Chicken flock counting method based on deep learning and application | |
CN114332183A (en) | Image registration method and device, computer equipment and storage medium | |
CN116664892A (en) | Multi-temporal remote sensing image registration method based on cross attention and deformable convolution | |
CN117876608B (en) | Three-dimensional image reconstruction method, three-dimensional image reconstruction device, computer equipment and storage medium | |
Badki et al. | Robust radiometric calibration for dynamic scenes in the wild | |
CN112102379B (en) | Unmanned aerial vehicle multispectral image registration method | |
CN115661368B (en) | Image matching method, device, server and storage medium | |
CN111914890B (en) | Image block matching method between images, image registration method and product | |
US9098746B2 (en) | Building texture extracting apparatus and method thereof | |
CN109493279B (en) | Large-scale unmanned aerial vehicle image parallel splicing method | |
CN110751163B (en) | Target positioning method and device, computer readable storage medium and electronic equipment | |
CN106651950B (en) | Single-camera pose estimation method based on quadratic curve perspective projection invariance | |
CN114913246A (en) | Camera calibration method and device, electronic equipment and storage medium | |
CN110020624B (en) | Image recognition method, terminal device and storage medium | |
CN114140429A (en) | Real-time parking space detection method and device for vehicle end | |
Otoya et al. | Real-time non-invasive leaf area measurement method using depth images |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant | ||
GR01 | Patent grant |