CN112085092B - Graph matching method and device based on space-time continuity constraint - Google Patents
Graph matching method and device based on space-time continuity constraint Download PDFInfo
- Publication number
- CN112085092B CN112085092B CN202010935742.9A CN202010935742A CN112085092B CN 112085092 B CN112085092 B CN 112085092B CN 202010935742 A CN202010935742 A CN 202010935742A CN 112085092 B CN112085092 B CN 112085092B
- Authority
- CN
- China
- Prior art keywords
- matrix
- target
- objective function
- homography transformation
- homography
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
- G06V10/75—Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries
- G06V10/751—Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
- G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/46—Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features
- G06V10/462—Salient features, e.g. scale invariant feature transforms [SIFT]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Landscapes
- Engineering & Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Theoretical Computer Science (AREA)
- Multimedia (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Computing Systems (AREA)
- Databases & Information Systems (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- Software Systems (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a graph matching method and device based on space-time continuity constraint, the method comprises the steps of calculating an affinity matrix, adopting a first-order similarity matrix, calculating the matching degree of edges, namely a second-order similarity matrix, embedding the motion information of a target into an optimization objective function of a graph matching problem, so that two optimization variables exist in the optimization objective function: homography transformation matrixHAnd an allocation matrixXCorrespondingly, an alternative optimization algorithm is provided, one optimization variable is fixed each time, the other optimization variable is solved, and the iteration is circulated until the objective function converges or reaches the preset maximum iteration number, in addition, the verification based on multi-graph matching as the last step is also provided in the technical scheme, the method is particularly suitable for judging whether the condition of losing the target occurs in the planar target tracking process, an execution basis can be provided for the follow-up tracking behavior, and the speed and accuracy of image matching are improved by the technology.
Description
Technical Field
The invention belongs to the field of image processing, relates to a graph matching method oriented to an augmented reality technology, and particularly relates to a graph matching method and device based on space-time continuity constraint.
Background
Planar target tracking algorithms are one of the core software components in augmented reality technology. In the planar target tracking algorithm, the planar target is characterized as a graph structure, so that better characterization stability can be obtained, however, the difficulty of the characterization method is how to realize the matching problem of the graph of the previous frame and the graph of the current frame of the target.
A common graph matching method is to convert the graph matching problem into a secondary distribution problem. However, the quadratic assignment problem belongs to the NP-Hard problem, i.e. the problem that all non-deterministic polynomial problems can be reduced within the polynomial time complexity, and cannot be solved within the polynomial time. To this end, most methods obtain an approximate solution to the problem by making further relaxation of the quadratic assignment problem. The method based on tree search converts the graph matching problem into a tree search problem, searches two graphs respectively based on a heuristic search strategy from an initial node, and gradually expands isomorphic nodes and edges until the isomorphic nodes cannot be further expanded. In order to obtain a better matching result, the above graph matching methods generally need to be repeatedly performed from different initial states or nodes for multiple times, so that the problem of low matching speed of the methods is common.
Disclosure of Invention
Aiming at the problems, the invention provides a graph matching method and device based on space-time continuity constraint by utilizing the characteristic that a video sequence has space-time continuity, and provides a good initial state for the optimization process of graph matching by embedding the motion information of a target into a graph matching optimization objective function, thereby greatly improving the speed and accuracy of graph matching.
The technical scheme of the invention is as follows: the method for matching the graph based on the space-time continuity constraint comprises the following steps:
(1) Calculating an affinity matrix: constructing a first-order similarity matrix according to the matching degree of the vertexes, constructing a second-order similarity matrix according to the matching degree of the vertexes and the edges, and splicing the first-order similarity matrix and the second-order similarity matrix according to diagonal lines to obtain an affinity matrix;
(2) Constructing an objective function, embedding motion information of an objective into a graph matching problem to obtain the objective function, and enabling the objective function to have two optimization variables: the method comprises the steps of estimating an initial value of a homography transformation matrix of a target in a current frame according to a change trend of the homography transformation matrix of the target in a plurality of previous frames, and obtaining an allocation matrix of a current target frame image according to the initial value of the homography transformation matrix of the target in the current frame;
(3) Designing an optimization algorithm: utilizing an alternating optimization algorithm to optimize a variable homography transformation matrix and an allocation matrix of the objective function, fixing one of the optimization variables each time, solving the other optimization variable, and performing loop iteration until the objective function optimization variable converges or reaches a preset maximum iteration number;
(4) Multiple-graph matching verification: by using the loop consistency verification method, the reliability of the current matching result is determined by checking whether the initial graph of the target, the graph of the previous frame and the graph of the current frame meet the loop consistency constraint.
The invention further adopts the technical scheme that: when constructing the first-order similarity matrix in the step (1), constructing the first-order similarity matrix according to the SURF local feature descriptors of the vertexes, and when constructing the second-order similarity matrix, constructing the second-order similarity matrix according to the SURF local feature descriptors of the vertexes connected in pairs and the included angles of the edges of the two pairs of vertexes.
The invention further adopts the technical scheme that: when the initial value of the homography transformation matrix is estimated in the step (2), the initial value of the homography transformation matrix of the target in the current frame is estimated according to the change trend of the homography transformation matrix of the target in a plurality of previous frames by adopting linear interpolation, and then the distribution matrix of the current target frame image is calculated according to the initial value of the homography transformation matrix of the target in the current frame by adopting a weighted random walk algorithm.
The invention further adopts the technical scheme that: the specific process of the alternative optimization algorithm in the step (3) is that after the distribution matrix of the current target frame image is obtained according to the initial value of the homography transformation matrix in the current frame, the homography transformation matrix is re-estimated according to the distribution matrix, and iteration is repeated in sequence until the preset maximum iteration number is reached or the convergence condition of the objective function optimization variable is met, wherein the convergence condition is that the change amplitude of the distribution matrix or the homography transformation matrix in two continuous iterations is smaller than a threshold value.
The invention further adopts the technical scheme that: the method for re-estimating the homography transformation matrix after the distribution matrix is obtained is that the matching relation of the vertexes is obtained according to the distribution matrix, and the homography transformation matrix is re-estimated according to the random sampling consistency.
The technical scheme of the invention is as follows: the device comprises a calculation affinity matrix module, a construction objective function module, a design optimization algorithm module and a multi-graph matching verification module, wherein the calculation affinity matrix module constructs a first-order similarity matrix according to the matching degree of vertexes for an image of a previous frame of a target and an image of a current frame, constructs a second-order similarity matrix according to the matching degree of vertexes and edges, and splices the first-order similarity matrix and the second-order similarity matrix according to diagonal lines to obtain an affinity matrix; the objective function constructing module is used for embedding the motion information of the target into the graph matching problem to obtain an objective function, so that two optimization variables exist in the objective function: the method comprises the steps of estimating an initial value of a homography transformation matrix of a target in a current frame according to a change trend of the homography transformation matrix of the target in a plurality of previous frames, and obtaining an allocation matrix of a current target frame image according to the initial value of the homography transformation matrix of the target in the current frame; the design optimization algorithm module utilizes an alternating optimization algorithm to optimize a variable homography transformation matrix and an allocation matrix of an objective function, one of the optimization variables is fixed each time, the other optimization variable is solved, and the iteration is circulated until the objective function optimization variable converges or reaches the preset maximum iteration number; the multi-image matching verification module utilizes a cyclic consistency verification method to determine the reliability of a current matching result by checking whether the initial image of a target, the image of a previous frame and the image of a current frame meet cyclic consistency constraint.
The invention further adopts the technical scheme that: and when constructing a second-order similarity matrix, the calculated affinity matrix module constructs the second-order similarity matrix according to the SURF local feature descriptors of the vertexes connected in pairs and the included angles of the edges of the two pairs of vertexes.
The invention further adopts the technical scheme that: when the initial value of the homography transformation matrix is estimated, the constructing objective function module estimates the initial value of the homography transformation matrix of the target in the current frame according to the change trend of the homography transformation matrix of the target in a plurality of previous frames by adopting linear interpolation, and then adopts a weighted random walk algorithm to calculate the distribution matrix of the current target frame image according to the initial value of the homography transformation matrix of the target in the current frame.
The invention further adopts the technical scheme that: the specific process of the alternating optimization algorithm in the design optimization algorithm module is as follows: after the distribution matrix of the current target frame image is calculated according to the initial value of the homography transformation matrix in the current frame, the homography transformation matrix is re-estimated according to the distribution matrix, and iteration is repeated in sequence until the preset maximum iteration number is reached or the convergence condition of the objective function optimization variable is met, wherein the convergence condition is that the change amplitude of the distribution matrix or the homography transformation matrix in two continuous iterations is smaller than a threshold value.
The invention further adopts the technical scheme that: the method for re-estimating the homography transformation matrix after obtaining the distribution matrix in the design optimization algorithm module is to obtain the matching relation of the vertexes according to the distribution matrix and re-estimate the homography transformation matrix according to the random sampling consistency.
The graph matching method and device based on space-time continuity constraint provided by the invention have the beneficial effects that: by embedding the motion information of the target into the graph matching objective function, a good initial state is provided for the optimization process of graph matching, when the affinity matrix is calculated, not only is a first-order similarity matrix adopted, but also the matching degree of edges, namely a second-order similarity matrix, is calculated.
Drawings
FIG. 1 is a flow chart of an embodiment of the matching method of the present invention;
fig. 2 is a schematic diagram of a module structure of the present invention.
Detailed Description
In order to further describe the technical scheme of the invention in detail, the embodiment is implemented on the premise of the technical scheme of the invention, and detailed implementation modes and specific steps are given.
Fig. 1 is a schematic flow chart of an embodiment of the matching method of the present invention, and specifically includes the following implementation steps:
(1) Calculating an affinity matrix: and constructing a first-order similarity matrix according to the matching degree of the vertexes, constructing a second-order similarity matrix according to the matching degree of the vertexes and the edges, and splicing the first-order similarity matrix and the second-order similarity matrix according to diagonal lines to obtain an affinity matrix.
The specific implementation process is as follows: in general, when calculating the affinity matrix, only the matching of the vertices, namely a first-order similarity matrix, is considered, and the continuity of the object motion in the video is considered, namely the included angle between the edge of the graph of the previous frame object and the edge of the graph of the current frame object is not too large, so that in the embodiment, when calculating the affinity matrix, not only the first-order similarity matrix, but also the matching of the edges, namely a second-order similarity matrix, is adopted, and the first-order similarity matrix is constructed according to the SURF local feature descriptors of the vertices, and the specific process is as follows: previous frame map G 1 ＝(V 1 ,E 1 ) V of it 1 Vertices representing a previous frame of the graph, E 1 Graph G representing edges between nodes, current frame 2 ＝(V 2 ,E 2 )，G 1 ,G 2 The number of the nodes is n respectively 1 ,n 2 . Then a first order similarity matrix
When G 1 SURF local feature descriptor and G for vertex i 2 SURF local feature descriptor matching for middle vertex j, first order similarity matrix X ij 1 and vice versa. Calculating a second-order similarity matrix according to SURF local feature descriptors of vertexes connected in pairs and included angles of edges of the vertexes>
SURF local feature descriptors for pairs of connected vertices are identical and the angles of the edges of two pairs of vertices are similar, A ij 1 and vice versa. The first-order similarity matrix and the second-order similarity are combinedAnd splicing the matrixes according to diagonal lines, and filling 0 in the vacant positions to obtain an affinity matrix K.
(2) Constructing an objective function, embedding motion information of an objective into a graph matching problem to obtain the objective function, and enabling the objective function to have two optimization variables: the method comprises the steps of estimating an initial value of a homography transformation matrix of a target in a current frame according to a change trend of the homography transformation matrix of the target in a plurality of previous frames, and obtaining an allocation matrix of a current target frame image according to the initial value of the homography transformation matrix of the target in the current frame;
the specific implementation process is as follows: the optimization objective of the conventional graph matching problem is to solve a distribution matrix, which is used for determining the correspondence between the vertices in two graphs, and in this embodiment, the motion information of the objective, that is, the vertices of the graph of the planar objective in different frames, are associated through a homography matrix, and are embedded into an optimization objective function of a secondary distribution problem, where the optimization objective function is expressed as:
argmax(x T Kx)-min(p 1 -Hp 2 )
wherein the method comprises the steps of
K represents an affinity matrix, p 1 Representing the position of the vertex in the previous frame, p 2 Representing the position of the matching vertex in the subsequent frame, H represents the homography matrix. The constructed optimization objective function has two optimization variables, namely an allocation matrix X and a homography matrix H, wherein the allocation matrix is used for determining whether the topological structures of the two graphs keep consistent, and the homography matrix is used for determining whether the vertexes of the two graphs accord with the same geometric transformation relation.
Preferred embodiments of the invention are: when the initial value of the homography transformation matrix is estimated, the initial value of the homography transformation matrix of the target in the current frame is estimated according to the change trend of the homography transformation matrix of the target in a plurality of previous frames by adopting linear interpolation, and then the distribution matrix of the current target frame image is calculated according to the initial value of the homography transformation matrix of the target in the current frame by adopting a weighted random walk algorithm, and the specific implementation process is as follows: firstly, estimating an initial value of a homography matrix H, and adopting linear interpolation according to the change trend of a homography transformation matrix of a target in a plurality of previous frames, wherein the method specifically comprises the following steps: the former video frame sequences are used as independent variables, the homography transformation matrix is used as the dependent variables for modeling, the initial value of the homography transformation matrix H of the target in the current frame is estimated, and the distribution matrix X of the current picture is calculated by adopting a weighted random walk algorithm, specifically: firstly, selecting the optimal initial node of random walk according to the estimated homography transformation matrix H, then randomly accessing a certain adjacent node from the initial node, establishing an allocation matrix X, allocating higher weight for the nodes with more access times, and repeating the access until the allocation matrix X is stable.
(3) Designing an optimization algorithm: and (3) utilizing an alternating optimization algorithm to optimize a variable homography transformation matrix and an allocation matrix of the objective function, fixing one of the optimization variables each time, solving the other optimization variable, and performing loop iteration until the objective function optimization variable converges or reaches a preset maximum iteration number.
Preferred embodiments of the invention are: after the distribution matrix of the current target frame image is obtained according to the initial value of the homography transformation matrix in the current frame, the homography transformation matrix is re-estimated according to the distribution matrix, and iteration is repeated in sequence until the preset maximum iteration number is reached or the convergence condition of the objective function optimization variable is met, wherein the convergence condition is that the change amplitude of the distribution matrix or the homography transformation matrix in two continuous iterations is smaller than a threshold value.
Preferred embodiments of the invention are: the method for re-estimating the homography transformation matrix after the distribution matrix is obtained is that the homography transformation matrix is re-estimated by adopting a random sampling consistency algorithm according to the matching relation of the vertexes obtained by the distribution matrix.
The specific implementation mode is as follows: obtaining a matching relation of the vertexes according to the obtained distribution matrix X, and re-estimating a homography transformation matrix H by adopting a random sampling consistency algorithm according to the matching relation of the vertexes, wherein the homography transformation matrix H is specifically as follows: randomly extracting 4 non-collinear matching points from the matching point set, calculating a transformation matrix H, calculating the difference between the corresponding relation of the residual points in the matching point set and the transformation matrix, adding the internal point set M if the error is smaller than a threshold (set as 5 in the embodiment), and iteratively searching for the transformation matrix which enables the internal point set M to be maximum, wherein the transformation matrix H calculated on the maximum internal point set M is used as a final result.
Because there are two optimization variables with different physical meanings in the objective function to be optimized, namely, the distribution matrix X and the homography matrix H, so that the objective function has non-convexity and cannot obtain a closed solution, for this reason, the alternative optimization algorithm adopted in this embodiment fixes one of the optimization variables at a time, solves the other optimization variable, and repeatedly solves the homography transformation matrix H and the distribution matrix X until the preset maximum iteration number is reached or the convergence condition is satisfied, the convergence condition is that the variation amplitude of the distribution matrix X or the homography transformation matrix H in two consecutive iterations is smaller than the threshold, and the variation amplitude threshold in this embodiment is 5%.
(4) Multiple-graph matching verification: by using the loop consistency verification method, the reliability of the current matching result is determined by checking whether the initial graph of the target, the graph of the previous frame and the graph of the current frame meet the loop consistency constraint.
The specific implementation mode is as follows: planar target tracking is a continuous process, and not only is the images of the previous frame and the current frame matched, but also whether the two images are consistent with the initial image of the target is required to be confirmed, so that whether the current tracking result is reliable is confirmed, multiple image matching is carried out on the initial image of the target, the images of the previous frame and the images of the current frame, whether the three images are completely matched is judged based on the constraint of cyclic consistency, and the successful matching standard is as follows: the affinity matrices calculated for the three graphs agree between each other. If the initial image of the target and the image of the previous frame are used, the image of the current frame can be successfully matched, and the initial image of the target can be successfully matched according to the image of the current frame and the image of the previous frame, the three images are considered to meet the cycle consistency constraint, if the cycle consistency constraint is not met, the tracking failure is indicated, and the flag variable F is set as False; otherwise, the tracking is successful, and the flag variable F is set to True.
As shown in fig. 2, the specific embodiments of the present invention are: the device for matching the images based on space-time continuity constraint is constructed and comprises a calculation affinity matrix module 1, a construction objective function module 2, a design optimization algorithm module 3 and a multi-image matching verification module 4, wherein the calculation affinity matrix module 1 constructs a first-order similarity matrix according to the matching degree of vertexes and constructs a second-order similarity matrix according to the matching degree of vertexes and edges, and the first-order similarity matrix and the second-order similarity matrix are spliced according to diagonal lines to obtain an affinity matrix; the objective function constructing module 2 embeds motion information of an object into a graph matching problem to obtain an objective function, so that two optimization variables exist in the objective function: the method comprises the steps of estimating an initial value of a homography transformation matrix of a target in a current frame according to a change trend of the homography transformation matrix of the target in a plurality of previous frames, and obtaining an allocation matrix of a current target frame image according to the initial value of the homography transformation matrix of the target in the current frame; the design optimization algorithm module 3 utilizes an alternating optimization algorithm to optimize a variable homography transformation matrix and an allocation matrix of an objective function, one of the optimization variables is fixed each time, the other optimization variable is solved, and the iteration is circulated until the objective function optimization variable converges or reaches the preset maximum iteration number; the multi-graph matching verification module 4 determines the reliability of the current matching result by checking whether the initial graph of the target, the graph of the previous frame and the graph of the current frame satisfy the cyclic consistency constraint by using the cyclic consistency verification method.
Preferred embodiments of the invention are: when constructing a first-order similarity matrix, the affinity matrix calculation module 1 constructs a first-order similarity matrix according to the SURF local feature descriptors of the vertexes, and when constructing a second-order similarity matrix, constructs a second-order similarity matrix according to the SURF local feature descriptors of the vertexes connected in pairs and the included angles of the edges of the two pairs of vertexes.
Preferred embodiments of the invention are: when the initial value of the homography transformation matrix is estimated, the objective function constructing module 2 adopts linear interpolation to estimate the initial value of the homography transformation matrix of the target in the current frame according to the change trend of the homography transformation matrix of the target in a plurality of previous frames, and then adopts a weighted random walk algorithm to calculate the distribution matrix of the current target frame image according to the initial value of the homography transformation matrix of the target in the current frame.
Preferred embodiments of the invention are: the specific process of the alternating optimization algorithm in the design optimization algorithm module 3 is as follows: after the distribution matrix of the current target frame image is calculated according to the initial value of the homography transformation matrix in the current frame, the homography transformation matrix is re-estimated according to the distribution matrix, and iteration is repeated in sequence until the preset maximum iteration number is reached or the convergence condition of the objective function optimization variable is met, wherein the convergence condition is that the change amplitude of the distribution matrix or the homography transformation matrix in two continuous iterations is smaller than a threshold value.
Preferred embodiments of the invention are: the method for re-estimating the homography transformation matrix after the distribution matrix is obtained in the design optimization algorithm module 3 is that the homography transformation matrix is re-estimated by adopting a random sampling consistency algorithm according to the matching relation of vertexes obtained by the distribution matrix.
In this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, apparatus.
The foregoing is a further detailed description of the invention in connection with the preferred embodiments, and it is not intended that the invention be limited to the specific embodiments described. It will be apparent to those skilled in the art that several simple deductions or substitutions may be made without departing from the spirit of the invention, and these should be considered to be within the scope of the invention.
Claims (10)
1. A graph matching method based on space-time continuity constraint, comprising the steps of:
(1) Calculating an affinity matrix: constructing a first-order similarity matrix according to the matching degree of the vertexes, constructing a second-order similarity matrix according to the matching degree of the vertexes and the edges, and splicing the first-order similarity matrix and the second-order similarity matrix according to diagonal lines to obtain an affinity matrix;
(2) Constructing an objective function: embedding the motion information of the target into the graph matching problem to obtain an objective function, so that two optimization variables exist in the objective function: the method comprises the steps of estimating an initial value of a homography transformation matrix of a target in a current frame according to a change trend of the homography transformation matrix of the target in a plurality of previous frames, and obtaining an allocation matrix of a current target frame image according to the initial value of the homography transformation matrix of the target in the current frame;
(3) Designing an optimization algorithm: utilizing an alternating optimization algorithm to optimize a variable homography transformation matrix and an allocation matrix of the objective function, fixing one of the optimization variables each time, solving the other optimization variable, and performing loop iteration until the objective function optimization variable converges or reaches a preset maximum iteration number;
(4) Multiple-graph matching verification: determining the reliability of the current matching result by checking whether the initial diagram of the target, the diagram of the previous frame and the diagram of the current frame meet the cyclic consistency constraint or not by using a cyclic consistency verification method;
wherein, step (2) specifically includes: the motion information of the target, namely the vertex of the graph of the planar target in different frames is associated through a homography matrix and is embedded into an objective function of the secondary distribution problem, and the objective function is expressed as:
argmax(x T Kx)-min(p 1 -Hp 2 )
wherein the method comprises the steps of
n 1 、n 2 The number of nodes respectively representing the previous frame and the next frame, K represents an affinity matrix, and p 1 Representing the position of the vertex in the previous frame, p 2 Representing the position of the matched vertex in the following frame, H represents a homography matrix, and the constructed objective function has two optimization variables, namely an allocation matrix X and a homography matrix H, wherein the allocation matrix is used for determining whether the topological structure of two graphs keeps consistent or not, and the homography matrix is used forTo determine whether the vertices of the two graphs conform to the same geometric transformation relationship.
2. The graph matching method based on space-time continuity constraint of claim 1, wherein when the first-order similarity matrix is constructed in the step (1), the first-order similarity matrix is constructed according to the SURF local feature descriptors of the vertexes, and when the second-order similarity matrix is constructed, the second-order similarity matrix is constructed according to the SURF local feature descriptors of the vertexes connected in pairs and the included angles of the edges of the two pairs of vertexes.
3. The graph matching method based on space-time continuity constraint according to claim 1, wherein when the initial value of the homography transformation matrix is estimated in the step (2), the initial value of the homography transformation matrix of the target in the current frame is estimated by linear interpolation according to the change trend of the homography transformation matrix of the target in the previous frames, and then the distribution matrix of the current target frame image is calculated according to the initial value of the homography transformation matrix of the target in the current frame by a weighted random walk algorithm.
4. The graph matching method based on space-time continuity constraint according to claim 3, wherein the specific process of the alternating optimization algorithm in the step (3) is that after the distribution matrix of the current target frame image is obtained according to the initial value of the homography transformation matrix in the current frame, the homography transformation matrix is re-estimated according to the distribution matrix, and the iteration is repeated in sequence until the preset maximum iteration number is reached or the convergence condition of the objective function optimization variable is met, wherein the convergence condition is that the change amplitude of the distribution matrix or the homography transformation matrix in two continuous iterations is smaller than a threshold value.
5. The graph matching method based on space-time continuity constraint according to claim 4, wherein the method for re-estimating the homography transformation matrix after obtaining the distribution matrix is to obtain the matching relation of the vertexes according to the distribution matrix, and re-estimate the homography transformation matrix according to the random sampling consistency.
6. A graph matching apparatus based on space-time continuity constraints, the apparatus comprising: the system comprises an affinity matrix calculation module, an objective function construction module, a design optimization algorithm module and a multi-image matching verification module, wherein the affinity matrix calculation module constructs a first-order similarity matrix according to the matching degree of vertexes for an image of a previous frame of a target and an image of a current frame of the target, constructs a second-order similarity matrix according to the matching degree of vertexes and edges, and splices the first-order similarity matrix and the second-order similarity matrix according to diagonal lines to obtain an affinity matrix; the objective function constructing module is used for embedding the motion information of the target into the graph matching problem to obtain an objective function, so that two optimization variables exist in the objective function: the method comprises the steps of estimating an initial value of a homography transformation matrix of a target in a current frame according to a change trend of the homography transformation matrix of the target in a plurality of previous frames, and obtaining an allocation matrix of a current target frame image according to the initial value of the homography transformation matrix of the target in the current frame; the design optimization algorithm module utilizes an alternating optimization algorithm to optimize a variable homography transformation matrix and an allocation matrix of an objective function, one of the optimization variables is fixed each time, the other optimization variable is solved, and the iteration is circulated until the objective function optimization variable converges or reaches the preset maximum iteration number; the multi-image matching verification module is used for determining the reliability of a current matching result by checking whether an initial image of a target, an image of a previous frame and an image of a current frame meet a cyclic consistency constraint or not by using a cyclic consistency verification method; the construction objective function module specifically comprises: the motion information of the target, namely the vertex of the graph of the planar target in different frames is associated through a homography matrix and is embedded into an objective function of the secondary distribution problem, and the objective function is expressed as:
argmax(x T Kx)-min(p 1 -Hp 2 )
wherein the method comprises the steps of
n 1 、n 2 The number of nodes respectively representing the previous frame and the next frame, K represents an affinity matrix, and p 1 Representing the position of the vertex in the previous frame, p 2 And representing the positions of the matched vertexes in the subsequent frame, wherein H represents a homography matrix, and the constructed objective function has two optimization variables, namely an allocation matrix X and a homography matrix H, wherein the allocation matrix is used for determining whether the topological structures of the two graphs keep consistent, and the homography matrix is used for determining whether the vertexes of the two graphs accord with the same geometric transformation relation.
7. The graph matching device based on space-time continuity constraint of claim 6, wherein the computation affinity matrix module constructs a first-order similarity matrix according to the SURF local feature descriptors of the vertices when constructing the first-order similarity matrix, and constructs a second-order similarity matrix according to the SURF local feature descriptors of the vertices connected in pairs and the included angles between the edges of the two pairs of vertices when constructing the second-order similarity matrix.
8. The graph matching device based on space-time continuity constraint of claim 6, wherein the construction objective function module estimates an initial value of a homography transformation matrix of a target in a current frame according to a change trend of the homography transformation matrix of the target in a plurality of previous frames by linear interpolation when estimating the initial value of the homography transformation matrix, and then calculates an allocation matrix of the current target frame image according to the initial value of the homography transformation matrix of the target in the current frame by a weighted random walk algorithm.
9. The graph matching device based on space-time continuity constraint according to claim 8, wherein the specific process of the alternate optimization algorithm in the design optimization algorithm module is as follows: after the distribution matrix of the current target frame image is calculated according to the initial value of the homography transformation matrix in the current frame, the homography transformation matrix is re-estimated according to the distribution matrix, and iteration is repeated in sequence until the preset maximum iteration number is reached or the convergence condition of the objective function optimization variable is met, wherein the convergence condition is that the change amplitude of the distribution matrix or the homography transformation matrix in two continuous iterations is smaller than a threshold value.
10. The graph matching device based on space-time continuity constraint according to claim 9, wherein the method of re-estimating the homography transformation matrix after obtaining the distribution matrix in the design optimization algorithm module is to obtain the matching relation of the vertexes according to the distribution matrix, and re-estimate the homography transformation matrix according to the random sampling consistency.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010935742.9A CN112085092B (en) | 2020-09-08 | 2020-09-08 | Graph matching method and device based on space-time continuity constraint |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010935742.9A CN112085092B (en) | 2020-09-08 | 2020-09-08 | Graph matching method and device based on space-time continuity constraint |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112085092A CN112085092A (en) | 2020-12-15 |
| CN112085092B true CN112085092B (en) | 2023-06-20 |
Family
ID=73732119
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010935742.9A Active CN112085092B (en) | 2020-09-08 | 2020-09-08 | Graph matching method and device based on space-time continuity constraint |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112085092B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116310417B (en) * | 2023-03-10 | 2024-04-26 | 济南大学 | Approximate graph matching method and system based on shape context information |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110443261A (en) * | 2019-08-15 | 2019-11-12 | 南京邮电大学 | A kind of more figure matching process restored based on low-rank tensor |
| CN111242221A (en) * | 2020-01-14 | 2020-06-05 | 西交利物浦大学 | Image matching method, system and storage medium based on image matching |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9898564B2 (en) * | 2014-01-15 | 2018-02-20 | Sage Software, Inc. | SSTA with non-gaussian variation to second order for multi-phase sequential circuit with interconnect effect |
-
2020
- 2020-09-08 CN CN202010935742.9A patent/CN112085092B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110443261A (en) * | 2019-08-15 | 2019-11-12 | 南京邮电大学 | A kind of more figure matching process restored based on low-rank tensor |
| CN111242221A (en) * | 2020-01-14 | 2020-06-05 | 西交利物浦大学 | Image matching method, system and storage medium based on image matching |
Non-Patent Citations (1)
| Title |
|---|
| A robustlocalsparsetrackerwithglobal consistencyconstraint;Xinhua You 等;《Signal Processing》;第1-11页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112085092A (en) | 2020-12-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110427877B (en) | Human body three-dimensional posture estimation method based on structural information | |
| CN108345890B (en) | Image processing method, device and related equipment | |
| CN111666960B (en) | Image recognition method, device, electronic equipment and readable storage medium | |
| US8755630B2 (en) | Object pose recognition apparatus and object pose recognition method using the same | |
| CN111598796B (en) | Image processing method and device, electronic equipment and storage medium | |
| CN106952338B (en) | Three-dimensional reconstruction method and system based on deep learning and readable storage medium | |
| TWI700599B (en) | Method and device for embedding relationship network diagram, computer readable storage medium and computing equipment | |
| CN112347550A (en) | Coupling type indoor three-dimensional semantic graph building and modeling method | |
| CN112330699B (en) | Three-dimensional point cloud segmentation method based on overlapping region alignment | |
| CN112084849A (en) | Image recognition method and device | |
| KR20180035359A (en) | Three-Dimensional Space Modeling and Data Lightening Method using the Plane Information | |
| CN112085092B (en) | Graph matching method and device based on space-time continuity constraint | |
| CN115546601A (en) | Multi-target recognition model and construction method, device and application thereof | |
| CN112668608A (en) | Image identification method and device, electronic equipment and storage medium | |
| CN114519306A (en) | Decentralized terminal node network model training method and system | |
| CN114841309A (en) | Data processing method and device and electronic equipment | |
| KR20210058638A (en) | Apparatus and method for image processing | |
| CN116975347A (en) | Image generation model training method and related device | |
| KR20200133424A (en) | Method and system for learning self-organizing generative networks | |
| CN115439534A (en) | Image feature point matching method, device, medium, and program product | |
| CN115423927A (en) | ViT-based multi-view 3D reconstruction method and system | |
| US11393069B2 (en) | Image processing apparatus, image processing method, and computer readable recording medium | |
| CN115995085A (en) | Complex layout image-text recognition discipline knowledge graph embedded learning method | |
| CN113222016A (en) | Change detection method and device based on cross enhancement of high-level and low-level features | |
| CN114596342B (en) | Parallel matching method based on satellite images, storage medium and computer equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |