CN112801141A - Heterogeneous image matching method based on template matching and twin neural network optimization - Google Patents
Heterogeneous image matching method based on template matching and twin neural network optimization Download PDFInfo
- Publication number
- CN112801141A CN112801141A CN202110022206.4A CN202110022206A CN112801141A CN 112801141 A CN112801141 A CN 112801141A CN 202110022206 A CN202110022206 A CN 202110022206A CN 112801141 A CN112801141 A CN 112801141A
- Authority
- CN
- China
- Prior art keywords
- image
- matching
- point
- neural network
- pixel
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 61
- 238000013528 artificial neural network Methods 0.000 title claims abstract description 46
- 238000005457 optimization Methods 0.000 title claims abstract description 18
- 238000012545 processing Methods 0.000 claims abstract description 24
- 238000007781 pre-processing Methods 0.000 claims abstract description 5
- 230000003287 optical effect Effects 0.000 claims description 20
- 238000004364 calculation method Methods 0.000 claims description 13
- 238000001914 filtration Methods 0.000 claims description 12
- 238000001514 detection method Methods 0.000 claims description 7
- 238000012549 training Methods 0.000 claims description 7
- 230000001629 suppression Effects 0.000 claims description 6
- 238000009432 framing Methods 0.000 claims description 3
- 238000009499 grossing Methods 0.000 claims description 3
- 238000013507 mapping Methods 0.000 claims description 3
- 230000000694 effects Effects 0.000 abstract description 8
- 238000006243 chemical reaction Methods 0.000 abstract description 6
- 238000000605 extraction Methods 0.000 abstract description 4
- 238000005259 measurement Methods 0.000 description 5
- 238000003708 edge detection Methods 0.000 description 4
- 238000003384 imaging method Methods 0.000 description 3
- 238000004458 analytical method Methods 0.000 description 2
- 238000005034 decoration Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 238000002474 experimental method Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012360 testing method Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000001427 coherent effect Effects 0.000 description 1
- 238000010835 comparative analysis Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000005764 inhibitory process Effects 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
- G06V10/75—Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries
- G06V10/751—Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4007—Scaling of whole images or parts thereof, e.g. expanding or contracting based on interpolation, e.g. bilinear interpolation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/20—Image enhancement or restoration using local operators
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/70—Denoising; Smoothing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/13—Edge detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10032—Satellite or aerial image; Remote sensing
- G06T2207/10044—Radar image
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Computing Systems (AREA)
- Health & Medical Sciences (AREA)
- Software Systems (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- General Engineering & Computer Science (AREA)
- Molecular Biology (AREA)
- Biomedical Technology (AREA)
- Data Mining & Analysis (AREA)
- Mathematical Physics (AREA)
- Computational Linguistics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Biophysics (AREA)
- Databases & Information Systems (AREA)
- Medical Informatics (AREA)
- Multimedia (AREA)
- Image Analysis (AREA)
- Image Processing (AREA)
Abstract
A heterogeneous image matching method based on template matching and twin neural network optimization belongs to the field of computer image processing and solves the problems that the existing heterogeneous image matching is difficult and the contour edge is thick and the positioning accuracy is low when Sobel operators carry out feature extraction. The method comprises the following steps: preprocessing an SAR image; image half-pixel processing; converting sketch images based on Sobel operators; matching the templates; the twin neural networks are fine matched. The method is suitable for image processing operation of matching two different source images with different sizes in the same area scene based on template matching. In the template matching process, the sketch image conversion based on the Sobel operator is carried out on the heterogeneous image, the matching effect is enhanced, and the improved twin neural network is applied for fine matching subsequently, so that the heterogeneous image matching effect is better, and the matching precision is higher.
Description
Technical Field
The invention belongs to the technical field of computer image processing, and particularly relates to a heterogeneous image matching method based on template matching and twin neural network optimization.
Background
Image matching is a very important task in the field of computer vision and image processing, and is mainly used for matching two or more images acquired at different times, different sensors, different viewing angles and different shooting conditions. The image matching is the basis of processing and application of various images, and the matching effect directly influences the processing and application work of subsequent images.
The heterogeneous image matching is a process of performing similarity and consistency analysis on corresponding relations of image content, characteristics, structures, relations, textures, gray levels and the like through images acquired by different source sensors to seek the same image target. Heterogeneous image matching methods can be roughly divided into two categories: region-based matching methods and feature-based matching methods. Region-based matching methods directly utilize the gray information of an image or some transformation of the gray information for similarity measurement. The method divides the sets according to the gray level, and a certain corresponding relation often exists between the sets, and the corresponding relation is a process without gray mapping constraint and characteristic extraction when solving the matching of the heterogeneous images. The method takes a template as a unit, and calculates the similarity degree of a current window and the template at each position of an image according to a certain similarity measurement criterion, so the method focuses on the design of the similarity measurement criterion.
The gray scale difference of the same pixel points among the heterogeneous images is large, but because the images in the same area are collected, the edge information of the target area can be clearly distinguished after the images are subjected to edge detection. The adopted edge detection mainly comprises a Canny operator, a Prewitt operator, a Sobel operator and the like, wherein the Sobel operator is a discrete difference operator and is used for calculating an approximate value of the gray scale of the image brightness function. Using this operator at any point in the image will produce the corresponding gray scale vector or its normal vector. The Sobel operator has two, one for detecting horizontal edges and the other for detecting vertical edges. Compared with the Prewitt operator, the Sobel operator weights the influence of the position of the pixel, so that the edge blurring degree can be reduced, and the effect is better. However, the Sobel operator detects the edge according to the phenomenon that the gray weighting difference of upper, lower, left and right adjacent points of the pixel point reaches an extreme value at the edge, has a smoothing effect on noise, provides more accurate edge direction information, and has insufficient edge positioning accuracy; moreover, the Sobel operator detects the extracted edge of the edge image to be coarse, and the image is not an expected result, so that the error range in matching is large.
Disclosure of Invention
In view of the difficulties in heterogeneous image matching and the problems of thick outline edge and low positioning precision in feature extraction by a Sobel operator, the invention provides a heterogeneous image matching method based on template matching and twin neural network optimization. The method is mainly suitable for image processing operation of matching two different source images with different sizes in the same area scene based on template matching. In the template matching process, the sketch image conversion based on the Sobel operator is carried out on the heterogeneous image, the matching effect is enhanced, and the improved twin neural network is applied for fine matching subsequently, so that the heterogeneous image matching effect is better, and the matching precision is higher.
The technical scheme adopted by the invention for solving the technical problem is as follows:
the invention discloses a heterogeneous image matching method based on template matching and twin neural network optimization, which comprises the following steps:
firstly, preprocessing an SAR image;
step two, image half-pixel processing;
step three, converting the sketch image based on a Sobel operator;
step four, template matching;
and step five, precisely matching the twin neural networks.
Further, the specific operation steps of the first step are as follows:
and carrying out median filtering processing on the SAR image to be matched so as to remove light spots and speckle noise of the SAR image to obtain a de-noised image.
Further, the formula of the median filtering is as follows:
y(i)＝Med[x(i-N),...,x(i),...,x(i+N)]
in the formula, an odd-length L-long window is defined, where L is 2N +1, N is a positive integer, and at a certain time, the signal sample values in the window are x (i-N), …, x (i), …, and x (i + N), where x (i) is the signal sample value located at the center of the window, y (i) represents the output value of median filtering, and Med represents the median calculation.
Further, the specific operation steps of the second step are as follows:
and respectively carrying out half-pixel processing on the optical image and the preprocessed SAR image by adopting a bilinear interpolation method, and converting the size of the image into four times of the original image.
Further, the bilinear interpolation method has the following formula:
in the formula, f (R)1) Represents R1Pixel value of dot, f (R)2) Represents R2Pixel value of dot, f (A)1) To representA1Pixel value of dot, f (A)2) Is represented by A2Pixel value of dot, f (A)3) Is represented by A3Pixel value of dot, f (A)4) Is represented by A4Pixel value of a point, f (P) represents pixel value of unknown point P, x1Represents a known point A1、A3Abscissa of (a), x2Represents a known point A2、A4Abscissa of (a), y1Represents a known point A1、A2Ordinate of (a), y2Represents a known point A3、A4M denotes the abscissa of the unknown point P, n denotes the ordinate of the unknown point P, R1Is represented by the formula A1Coordinate points, R, co-ordinate with the ordinate and with the abscissa of the unknown point P2Is represented by the formula A3Coordinate points, A, having the same ordinate and the same abscissa as the unknown point P1Coordinate points representing pixel points at the upper left corner of the image, A2Coordinate points representing pixel points in the upper right corner of the image, A3Coordinate points, A, representing pixel points at the lower left corner of the image4Coordinate points, A, representing pixels in the lower right corner of the image1、A2、A3、A4Are all known;
known as A1And A4The coordinates of the unknown point P in the area are obtained.
Further, the specific operation steps of the third step are as follows:
s3.1, utilizing Sobel operator to extract edge characteristics
Utilizing a Sobel edge filter to extract edge features of an optical image and an SAR image: the convolution kernels of the Sobel operator are a horizontal detection convolution kernel and a vertical detection convolution kernel, and the convolution kernels are respectively applied to the input image to generate independent measurement values of gradient components in each direction, wherein A represents an original image, and G represents a gradient component in each directionxAnd GyThe gray values of the image detected by the transverse and longitudinal edges are respectively represented, and then the calculation formula is as follows:
g is to bexAnd GyCombining together to find the magnitude of the absolute value of the gradient G and the direction of the gradient G at each point; comparing the absolute value of the gradient G at a certain point with the middle value of the gray value of the whole image, and if the gradient G at the point is greater than the middle value of the gray value of the whole image, considering the point as an edge point;
s3.2 non-maximum suppression
And (3) processing the image obtained in the S3.1 by adopting non-maximum value suppression, and smoothing the image after edge thinning to obtain a corresponding sketch image.
Further, the specific operation steps of the step four are as follows:
and carrying out template matching according to the obtained sketch image, matching the region of the SAR sketch image in the optical sketch image, framing the SAR sketch image by using a rectangular frame, outputting an upper left-corner pixel point of the rectangular frame in the optical sketch image, and outputting nine adjacent pixel points taking the upper left-corner pixel point as a center.
Furthermore, the template matching adopts a normalized correlation coefficient matching algorithm, and the calculation formula is as follows:
in the formula, R (x, y) represents a matching result, T 'represents a template image, I' represents an image to be matched, x represents an abscissa of the upper left corner of the window dividing block in the image to be matched, y represents an ordinate of the upper left corner of the window dividing block in the image to be matched, x 'represents an abscissa of each pixel point in the window dividing block, and y' represents an ordinate of each pixel point in the window dividing block.
Further, the specific operation steps of the step five are as follows:
and inputting the registered optical image and SAR image in the training set as the training set into two input ends of a twin neural network for training, outputting the trained model, calling the model to perform image matching on nine pixel points which are output by template matching and take the upper left corner pixel point as the center, outputting the image with the highest degree of acquaintance as the final SAR image in the area of the optical image, and outputting the pixel point coordinate corresponding to the image with the highest degree of acquaintance as the final result.
Furthermore, the twin neural network comprises two input ends, two neural networks and a Loss module; respectively inputting the two input images into two neural networks, and respectively mapping the input images to new spaces by the two neural networks to form the representation of the input images in the new spaces; and finally, evaluating the similarity of the two input images through Loss calculation.
The invention has the beneficial effects that:
the invention starts from edge detection and processes the heterogeneous images so as to facilitate subsequent matching work. According to the method, the medium value filtering processing is carried out on the SAR image to be matched, so that light spots and speckle noise of the SAR image can be removed, and a denoised image is obtained; respectively carrying out half-pixel processing on the optical image and the SAR image after median filtering processing by adopting a bilinear interpolation method, and then carrying out sketch image conversion based on a Sobel operator, wherein in the step, the image after edge thinning is obtained by further processing by using non-maximum value inhibition after the Sobel operator so as to solve the problem that the edge image extracted by the conventional Sobel operator is thicker; performing template matching on the processed sketch image to obtain an approximate matching point coordinate; and respectively matching nine pixel points with the obtained coordinate point as the center by adopting a twin neural network, selecting an optimal point for outputting, and obtaining a final matching coordinate so as to solve the problem of insufficient edge positioning precision. According to the method, edge detection and template matching are carried out on different source images, the problem of difficulty in matching caused by different heterogeneous image imaging principles in the existing matching method is solved, meanwhile, the error of a matching result is reduced by adopting template matching, and matching comparison is carried out in the neighborhood of the matching result by adopting a twin neural network subsequently, so that the matching accuracy is further improved.
Drawings
FIG. 1 is a flow chart of the heterogeneous image matching method based on template matching and twin neural network optimization according to the present invention.
FIG. 2 is a schematic structural diagram of a twin neural network.
FIG. 3 is a comparison of the two methods in a harbor scenario. a represents the matching result of the SIFT matching method of the existing original brightness image, b represents the SAR image to be matched, and c represents the matching result of the invention.
Fig. 4 is a comparison diagram of two methods in an urban scene. a represents the matching result of the SIFT matching method of the existing original brightness image, b represents the SAR image to be matched, and c represents the matching result of the invention.
Fig. 5 is a comparison graph of two methods in a river scene. a represents the matching result of the SIFT matching method of the existing original brightness image, b represents the SAR image to be matched, and c represents the matching result of the invention.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings.
The core of the invention is that: for the heterogeneous images to be matched, in the process of template matching, the sketch image conversion based on the Sobel operator is adopted, so that the matching difficulty and complexity are reduced, the sketch image more suitable for template matching is formed, and then the precise matching is further performed through the twin neural network, so that the matching accuracy is improved.
As shown in FIG. 1, the heterogeneous image matching method based on template matching and twin neural network optimization of the present invention mainly comprises the following steps:
step one, SAR image preprocessing
When the sketch image is subjected to image matching, more accurate matching can be better performed according to the contour information in the image. Because the image aimed by the invention is the heterogeneous image matching of the optical image and the SAR image, and the imaging principle of the SAR image causes the SAR image to have coherent speckle noise, the SAR image is preprocessed firstly. The Sobel operator is a discrete difference operator, and the Sobel operator is used for realizing the detection of the image edge by adding the gray value weight difference of four adjacent pixel points of the upper, the lower, the left and the right of each pixel in the image and achieving an extreme value at the edge. The SAR image preprocessing method based on Sobel operator includes the following steps:
carrying out median filtering processing on the SAR image, wherein the formula of the median filtering is as follows:
y(i)＝Med[x(i-N),...,x(i),...,x(i+N)]
in the formula, an odd-length L-long window is defined, where L is 2N +1, N is a positive integer, and at a certain time, the signal sample values in the window are x (i-N), …, x (i), …, and x (i + N), where x (i) is the signal sample value located at the center of the window, y (i) represents the output value of median filtering, and Med represents the median calculation.
And carrying out median filtering on the SAR image to be matched to remove light spots and speckle noise of the SAR image so as to obtain a de-noised image.
Step two, image half-pixel processing
Respectively carrying out half-pixel processing on the optical image and the SAR image after median filtering processing by adopting a bilinear interpolation method, and converting the size of the image into four times of the original image, wherein the formula of the bilinear interpolation method is as follows:
in the formula, f (R)1) Represents R1Pixel value of dot, f (R)2) Represents R2Pixel value of dot, f (A)1) Is represented by A1Pixel value of dot, f (A)2) Is represented by A2Pixel value of dot, f (A)3) Is represented by A3Pixel value of dot, f (A)4) Is represented by A4Pixel value of a point, f (P) represents pixel value of unknown point P, x1Representing known pointsA1、A3Abscissa of (a), x2Represents a known point A2、A4Abscissa of (a), y1Represents a known point A1、A2Ordinate of (a), y2Represents a known point A3、A4M denotes the abscissa of the unknown point P, n denotes the ordinate of the unknown point P, R1Is represented by the formula A1Coordinate points, R, co-ordinate with the ordinate and with the abscissa of the unknown point P2Is represented by the formula A3Coordinate points, A, having the same ordinate and the same abscissa as the unknown point P1Coordinate points representing pixel points at the upper left corner of the image, A2Coordinate points representing pixel points in the upper right corner of the image, A3Coordinate points, A, representing pixel points at the lower left corner of the image4Coordinate points, A, representing pixels in the lower right corner of the image1、A2、A3、A4Are all known.
Known as A1And A4The coordinates of the unknown point P in the area are obtained.
Step three, sketch image conversion based on Sobel operator
S3.1, utilizing Sobel operator to extract edge characteristics
Compared with other operators such as canny, the Sobel operator has a good characteristic effect and is accurate in edge positioning when the edge contour of the SAR image is actually extracted, and the SAR image with gradually changed gray scale and more noise can be better processed, so that the Sobel edge filter is selected to extract the edge characteristics of the optical image and the SAR image.
The convolution kernels employed by the Sobel operator can be divided into horizontal detection convolution kernels and vertical detection convolution kernels, and the convolution kernels can be applied to the input image separately to generate separate measurements of the gradient components in each direction, G if the original image is denoted by axAnd GyThe gray values of the image detected by the transverse and longitudinal edges are respectively represented, and then the calculation formula is as follows:
g is to bexAnd GyCombined together to find the magnitude of the absolute value of the gradient G at each point and the direction of the gradient G. And comparing the absolute value of the gradient G at a certain point with the middle value of the gray value of the whole image, and if the absolute value of the gradient G at the certain point is greater than the middle value of the gray value of the whole image, considering the point as an edge point.
S3.2 non-maximum suppression
Because the Sobel operator has the problem of thick outline edge when extracting edge features, the invention adopts non-maximum suppression to process the image obtained in S3.1, and the image is smoothed after edge thinning to obtain a sketch image.
Step four, template matching
And carrying out template matching according to the obtained sketch image, matching the region of the SAR sketch image in the optical sketch image, framing the SAR sketch image by using a rectangular frame, and outputting the upper left corner pixel point of the rectangular frame in the optical sketch image as a matching result and a precision index.
The applied Template Matching algorithm (Template Matching) is one of the methods for finding a specific target in an image, the principle of this method is very simple, traverse every possible position in the image, compare the similarity with the Template everywhere, output the area with the highest similarity as the target area, through comparison, the Matching precision of the normalized correlation coefficient Matching algorithm (CV _ T M _ CCOEFF _ normal) in Template Matching is higher, its calculation formula is as follows:
in the formula, R (x, y) represents a matching result, T 'represents a template image, I' represents an image to be matched, x represents an abscissa of the upper left corner of the window dividing block in the image to be matched, y represents an ordinate of the upper left corner of the window dividing block in the image to be matched, x 'represents an abscissa of each pixel point in the window dividing block, and y' represents an ordinate of each pixel point in the window dividing block.
After template matching, nine adjacent pixel points with the upper left corner pixel point as the center are output
Step five, fine matching of twin neural networks
Firstly, inputting a corresponding registered optical image and SAR image which are used as a training set into two input ends of a twin neural network for training, outputting a trained model, calling the model to perform image matching on nine pixel points which are output by template matching and take the upper left corner pixel point as the center, outputting an image with the highest recognition degree as a final SAR image in an optical image area, and outputting the pixel point coordinate corresponding to the image with the highest similarity degree as a final result.
The structure of the twin neural network is shown in fig. 2. The purpose of a twin neural network is to measure how similar two inputs are. The twin neural Network has two inputs (Input1 and Input2) and the two Input images are respectively Input into two neural networks (Network1 and Network2) which respectively map the Input images into new spaces to form representations of the Input images in the new spaces. Through the calculation of Loss, the similarity of two input images is evaluated.
And carrying out picture matching on nine pixel points taking the upper left corner pixel point as the center by using the improved twin neural network, and selecting the optimal pixel point coordinate as a result to output.
Detailed description of the invention
The feasibility of the method provided by the invention is verified in the following specific tests. Compared with the existing template matching method, the method disclosed by the invention is used for carrying out comparative analysis on three aspects of feature point extraction, correct matching rate and matching speed.
1. Working conditions
In the experiment, an intel core i9-9900k CPU @3.60ghz 16 processor is adopted, a PC (personal computer) of Windows10 is operated, the display card is 2 GeForce RTX 1080Ti, and the programming language is Python language.
2. Analysis of experimental content and results
As shown in fig. 3, it can be seen by comparing the original SAR image that the image results matched by the present invention are more similar, and then the upper left pixel point of the two matching results and the upper left pixel point of the actual SAR image are further output to perform a test and output an accuracy rate with a pixel point error smaller than 5, as shown in table 1, thereby proving that the accuracy rate of the method of the present invention is greatly improved.
Wherein, the calculation formula of the correct matching rate is as follows:
in the formula, P is the correct matching rate, Q is the correct matching number with a matching error of 5 pixels or less, and N is the total matching number.
In different cases, the comparison results of the present invention with the existing template matching method are shown in fig. 3 to 5. The results of the experiments were compared in three scenes, namely, port (fig. 3), city (fig. 4) and river (fig. 5), and the results of the correct matching rates are shown in table 1.
As can be seen from fig. 3 to 5 and table 1, the correct matching rate of the present invention is higher than that of the existing template matching method under different conditions.
TABLE 1 comparison of correct matching ratio (%)
| Traditional template matching method | The invention | |
| Port port | 61.33 | 91.58 |
| City | 65.47 | 88.72 |
| River flow | 52.19 | 79.65 |
The three experimental results show that the sketch image conversion based on the Sobel operator solves the problem that common characteristics of images on characteristics such as gray scale, brightness, color and the like are difficult to obtain and difficult to match due to different imaging principles of related heterogeneous images, forms a sketch image with image information, reduces the matching difficulty, and solves the problem of low matching accuracy due to the fact that a twin neural network is added for fine matching. Meanwhile, the adopted twin neural network structure is simpler, so that the matching rate is higher during application.
The foregoing is only a preferred embodiment of the present invention, and it should be noted that, for those skilled in the art, various modifications and decorations can be made without departing from the principle of the present invention, and these modifications and decorations should also be regarded as the protection scope of the present invention.
Claims (10)
1. The heterogeneous image matching method based on template matching and twin neural network optimization is characterized by comprising the following steps of:
firstly, preprocessing an SAR image;
step two, image half-pixel processing;
step three, converting the sketch image based on a Sobel operator;
step four, template matching;
and step five, precisely matching the twin neural networks.
2. The heterogeneous image matching method based on template matching and twin neural network optimization according to claim 1, wherein the specific operation steps of step one are as follows:
and carrying out median filtering processing on the SAR image to be matched so as to remove light spots and speckle noise of the SAR image to obtain a de-noised image.
3. The heterogeneous image matching method based on template matching and twin neural network optimization according to claim 2, wherein the formula of the median filter is as follows:
y(i)＝Med[x(i-N),...,x(i),...,x(i+N)]
in the formula, an odd-length L-long window is defined, where L is 2N +1, N is a positive integer, and at a certain time, the signal sample values in the window are x (i-N), …, x (i), …, and x (i + N), where x (i) is the signal sample value located at the center of the window, y (i) represents the output value of median filtering, and Med represents the median calculation.
4. The heterogeneous image matching method based on template matching and twin neural network optimization according to claim 1, wherein the specific operation steps of step two are as follows:
and respectively carrying out half-pixel processing on the optical image and the preprocessed SAR image by adopting a bilinear interpolation method, and converting the size of the image into four times of the original image.
5. The heterogeneous image matching method based on template matching and twin neural network optimization according to claim 4, wherein the bilinear interpolation method has the following formula:
in the formula, f (R)1) Represents R1Pixel value of dot, f (R)2) Represents R2Pixel value of dot, f (A)1) Is represented by A1Pixel value of dot, f (A)2) Is represented by A2Pixel value of dot, f (A)3) Is represented by A3Pixel value of dot, f (A)4) Is represented by A4Pixel value of a point, f (P) represents pixel value of unknown point P, x1Represents a known point A1、A3Abscissa of (a), x2Represents a known point A2、A4Abscissa of (a), y1Represents a known point A1、A2Ordinate of (a), y2Represents a known point A3、A4M denotes the abscissa of the unknown point P, n denotes the ordinate of the unknown point P, R1Is represented by the formula A1Coordinate points, R, co-ordinate with the ordinate and with the abscissa of the unknown point P2Is represented by the formula A3Coordinate points, A, having the same ordinate and the same abscissa as the unknown point P1Coordinate points representing pixel points at the upper left corner of the image, A2Coordinate points representing pixel points in the upper right corner of the image, A3Coordinate points, A, representing pixel points at the lower left corner of the image4Coordinate points, A, representing pixels in the lower right corner of the image1、A2、A3、A4Are all known;
known as A1And A4The coordinates of the unknown point P in the area are obtained.
6. The heterogeneous image matching method based on template matching and twin neural network optimization according to claim 1, wherein the specific operation steps of step three are as follows:
s3.1, utilizing Sobel operator to extract edge characteristics
Utilizing a Sobel edge filter to extract edge features of an optical image and an SAR image: the convolution kernels of the Sobel operator are a horizontal detection convolution kernel and a vertical detection convolution kernelThe convolution kernels are applied separately to the input image to generate separate measures of the gradient components in each direction, denoted the original image by A, GxAnd GyThe gray values of the image detected by the transverse and longitudinal edges are respectively represented, and then the calculation formula is as follows:
g is to bexAnd GyCombining together to find the magnitude of the absolute value of the gradient G and the direction of the gradient G at each point; comparing the absolute value of the gradient G at a certain point with the middle value of the gray value of the whole image, and if the gradient G at the point is greater than the middle value of the gray value of the whole image, considering the point as an edge point;
s3.2 non-maximum suppression
And (3) processing the image obtained in the S3.1 by adopting non-maximum value suppression, and smoothing the image after edge thinning to obtain a corresponding sketch image.
7. The heterogeneous image matching method based on template matching and twin neural network optimization according to claim 1, wherein the specific operation steps of step four are as follows:
and carrying out template matching according to the obtained sketch image, matching the region of the SAR sketch image in the optical sketch image, framing the SAR sketch image by using a rectangular frame, outputting an upper left-corner pixel point of the rectangular frame in the optical sketch image, and outputting nine adjacent pixel points taking the upper left-corner pixel point as a center.
8. The heterogeneous image matching method based on template matching and twin neural network optimization according to claim 7, wherein the template matching adopts a normalized correlation coefficient matching algorithm, and the calculation formula is as follows:
in the formula, R (x, y) represents a matching result, T 'represents a template image, I' represents an image to be matched, x represents an abscissa of the upper left corner of the window dividing block in the image to be matched, y represents an ordinate of the upper left corner of the window dividing block in the image to be matched, x 'represents an abscissa of each pixel point in the window dividing block, and y' represents an ordinate of each pixel point in the window dividing block.
9. The heterogeneous image matching method based on template matching and twin neural network optimization according to claim 8, wherein the specific operation steps of step five are as follows:
inputting the registered optical image and SAR image in the training set into two input ends of a twin neural network for training, outputting the trained model, calling the model to perform image matching on nine pixel points which are output by template matching and take the upper left corner pixel point as the center, outputting the image with the highest recognition degree as the final SAR image in the area of the optical image, and outputting the pixel point coordinate corresponding to the image with the highest recognition degree as the final result.
10. The heterogeneous image matching method based on template matching and twin neural network optimization of claim 9, wherein the twin neural network comprises two inputs, two neural networks, a Loss module; respectively inputting the two input images into two neural networks, and respectively mapping the input images to new spaces by the two neural networks to form the representation of the input images in the new spaces; and finally, evaluating the similarity of the two input images through Loss calculation.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110022206.4A CN112801141B (en) | 2021-01-08 | 2021-01-08 | Heterogeneous image matching method based on template matching and twin neural network optimization |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110022206.4A CN112801141B (en) | 2021-01-08 | 2021-01-08 | Heterogeneous image matching method based on template matching and twin neural network optimization |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112801141A true CN112801141A (en) | 2021-05-14 |
| CN112801141B CN112801141B (en) | 2022-12-06 |
Family
ID=75809145
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110022206.4A Active CN112801141B (en) | 2021-01-08 | 2021-01-08 | Heterogeneous image matching method based on template matching and twin neural network optimization |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112801141B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115862377A (en) * | 2023-02-08 | 2023-03-28 | 山东科技大学 | Intelligent parking reservation method and system |
| CN117037159A (en) * | 2023-10-09 | 2023-11-10 | 网思科技股份有限公司 | Oil painting authenticity identification method, device and storage medium based on convolutional neural network |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20110142297A1 (en) * | 2009-12-16 | 2011-06-16 | Eye Controls, Llc | Camera Angle Compensation in Iris Identification |
| CN102654902A (en) * | 2012-01-16 | 2012-09-05 | 江南大学 | Contour vector feature-based embedded real-time image matching method |
| CN107680095A (en) * | 2017-10-25 | 2018-02-09 | 哈尔滨理工大学 | The electric line foreign matter detection of unmanned plane image based on template matches and optical flow method |
| CN108960258A (en) * | 2018-07-06 | 2018-12-07 | 江苏迪伦智能科技有限公司 | A kind of template matching method based on self study depth characteristic |
| CN109409292A (en) * | 2018-10-26 | 2019-03-01 | 西安电子科技大学 | The heterologous image matching method extracted based on fining characteristic optimization |
| CN110674719A (en) * | 2019-09-18 | 2020-01-10 | 北京市商汤科技开发有限公司 | Target object matching method and device, electronic equipment and storage medium |
| CN111783548A (en) * | 2020-06-04 | 2020-10-16 | 河海大学 | SAR image and visible light image matching method based on improved feature extraction and game theory hypergraph |
-
2021
- 2021-01-08 CN CN202110022206.4A patent/CN112801141B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20110142297A1 (en) * | 2009-12-16 | 2011-06-16 | Eye Controls, Llc | Camera Angle Compensation in Iris Identification |
| CN102654902A (en) * | 2012-01-16 | 2012-09-05 | 江南大学 | Contour vector feature-based embedded real-time image matching method |
| CN107680095A (en) * | 2017-10-25 | 2018-02-09 | 哈尔滨理工大学 | The electric line foreign matter detection of unmanned plane image based on template matches and optical flow method |
| CN108960258A (en) * | 2018-07-06 | 2018-12-07 | 江苏迪伦智能科技有限公司 | A kind of template matching method based on self study depth characteristic |
| CN109409292A (en) * | 2018-10-26 | 2019-03-01 | 西安电子科技大学 | The heterologous image matching method extracted based on fining characteristic optimization |
| CN110674719A (en) * | 2019-09-18 | 2020-01-10 | 北京市商汤科技开发有限公司 | Target object matching method and device, electronic equipment and storage medium |
| CN111783548A (en) * | 2020-06-04 | 2020-10-16 | 河海大学 | SAR image and visible light image matching method based on improved feature extraction and game theory hypergraph |
Non-Patent Citations (6)
| Title |
|---|
| BRANDON懂你: "图像处理之双线性插值法", 《HTTPS://BLOG.CSDN.NET/QQ_37577735/ARTICLE/DETAILS/80041586》 * |
| JUNG ROK KIM等: "Sobel Edge-based Image Template Matching in FPGA", 《2020 IEEE INTERNATIONAL CONFERENCE ON CONSUMER ELECTRONICS - ASIA (ICCE-ASIA)》 * |
| 刘国成著: "《人群异常行为数字图像处理与分析》", 31 January 2020 * |
| 宋伟东等: "《遥感影像几何纠正与三维重建》", 30 April 2011 * |
| 李佳芮等: "融合边缘信息和扩展邻域的异源图像匹配", 《计算机工程与应用》 * |
| 谢志华等: "可见光图像与合成孔径雷达图像的快速配准", 《激光与光电子学进展》 * |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115862377A (en) * | 2023-02-08 | 2023-03-28 | 山东科技大学 | Intelligent parking reservation method and system |
| CN117037159A (en) * | 2023-10-09 | 2023-11-10 | 网思科技股份有限公司 | Oil painting authenticity identification method, device and storage medium based on convolutional neural network |
| CN117037159B (en) * | 2023-10-09 | 2024-03-19 | 网思科技股份有限公司 | Oil painting authenticity identification method, device and storage medium based on convolutional neural network |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112801141B (en) | 2022-12-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109410207B (en) | NCC (non-return control) feature-based unmanned aerial vehicle line inspection image transmission line detection method | |
| CN112819772B (en) | High-precision rapid pattern detection and recognition method | |
| CN110866924B (en) | Line structured light center line extraction method and storage medium | |
| CN108596197B (en) | Seal matching method and device | |
| CN107301661A (en) | High-resolution remote sensing image method for registering based on edge point feature | |
| CN112801141B (en) | Heterogeneous image matching method based on template matching and twin neural network optimization | |
| CN109034017A (en) | Head pose estimation method and machine readable storage medium | |
| CN110070574B (en) | Binocular vision stereo matching method based on improved PSMAT net | |
| CN110348263A (en) | A kind of two-dimensional random code image recognition and extracting method based on image recognition | |
| CN116758077B (en) | Online detection method and system for surface flatness of surfboard | |
| CN108470338B (en) | A kind of water level monitoring method | |
| CN111783583B (en) | SAR image speckle suppression method based on non-local mean algorithm | |
| CN104200434B (en) | Non-local mean image denoising method based on noise variance estimation | |
| CN111369605A (en) | Infrared and visible light image registration method and system based on edge features | |
| CN111354047B (en) | Computer vision-based camera module positioning method and system | |
| CN112017130B (en) | Image restoration method based on self-adaptive anisotropic total variation regularization | |
| CN107944497A (en) | Image block method for measuring similarity based on principal component analysis | |
| CN116503462A (en) | Method and system for quickly extracting circle center of circular spot | |
| CN111027637A (en) | Character detection method and computer readable storage medium | |
| CN109671084B (en) | Method for measuring shape of workpiece | |
| CN113642397B (en) | Object length measurement method based on mobile phone video | |
| CN114332079A (en) | Plastic lunch box crack detection method, device and medium based on image processing | |
| CN108647605B (en) | Human eye gaze point extraction method combining global color and local structural features | |
| CN117315670A (en) | Water meter reading area detection method based on computer vision | |
| CN111951178A (en) | Image processing method and device for remarkably improving image quality and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |