CN109872343B - Weak texture object posture tracking method, system and device - Google Patents
Weak texture object posture tracking method, system and device Download PDFInfo
- Publication number
- CN109872343B CN109872343B CN201910105602.6A CN201910105602A CN109872343B CN 109872343 B CN109872343 B CN 109872343B CN 201910105602 A CN201910105602 A CN 201910105602A CN 109872343 B CN109872343 B CN 109872343B
- Authority
- CN
- China
- Prior art keywords
- contour
- image
- point
- posture
- matching
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Landscapes
- Image Analysis (AREA)
Abstract
The invention provides a method, a system and a device for tracking the posture of a weak texture object, wherein the method comprises the following steps: predicting the attitude of the input current frame image to obtain a predicted object attitude: acquiring object contour shape data according to the predicted object posture; searching and matching contour shape characteristics on the current live-action image according to the contour shape data of the object; and carrying out attitude calculation according to the matching information. The method utilizes the 3D model of the weak texture object and the live-action image of the target object, does not need to rely on a large amount of prior data for training, and therefore can realize real-time posture tracking on common computing equipment.
Description
Technical Field
The embodiment of the invention relates to the field of computer vision, in particular to a method, a system and a device for tracking the posture of a weak texture object.
Background
Three-dimensional object posture tracking is a typical task in the field of computer vision, and by calculating 6-degree-of-freedom (R, t) posture data of a three-dimensional object in a scene in real time, the external world has a clear understanding of the position and orientation of the three-dimensional object in the scene, and the three-dimensional object posture tracking has a great number of applications in the fields of augmented reality, industrial assistance, robots and the like. The three-dimensional object can be divided into rich texture and weak texture according to the richness degree of the surface texture.
The rich texture has rich texture information, so that the rich texture can be extracted and described by utilizing a mature computer vision algorithm, then a matching pair with a 3D model is obtained by methods such as 2D image feature matching and the like, attitude estimation and tracking are carried out, and finally the 6-degree-of-freedom attitude of the object is obtained.
The weak texture object is also a very common object, which is as small as screws, wrenches and various parts in a production workshop and as large as almost all household appliances, automobiles, consumer electronics products and the like in life, and can be said to be full of daily life and social production of people. The gesture tracking with 6 degrees of freedom is carried out on the weak texture object, so that a plurality of very practical problems can be solved, for example, the function of carrying out virtual button marking on household appliances in daily life so as to realize a specification, for example, the function of carrying out virtual decoration on an automobile so as to realize multi-dimensional display, for example, the function of assisting workers in assembling parts in industry and assisting a robot in accurately grabbing objects and the like can be solved. These augmented reality applications or industrial production applications require real-time acquisition of 6-degree-of-freedom pose information of a weakly textured object so that auxiliary digital content or tags can be accurately placed on the real position of the object, thereby allowing a user to quickly obtain useful information in the same field of view, improving productivity and production efficiency. However, the surface of a weak texture object is often single in color, has no abundant texture, even has the characteristics of high light, specular reflection and the like, and is very lack of or even has no reliable visual features, so that the traditional computer vision algorithm cannot well solve the tracking problem of the 6-degree-of-freedom posture of the object.
Chinese patent application CN107679537A of prior art 1 discloses a texture-free spatial target pose estimation algorithm based on contour point ORB feature matching, which uses ORB features to perform feature extraction on the contour of a multi-viewpoint projection image of a texture-free object, and performs feature matching on ORB features in a target image to be detected with the contour, thereby obtaining 2D-3D matching information. And finally, solving the attitude parameters by combining the accuracy assistance of feature matching.
US patent US9892543B2 of prior art 2 discloses a system and method for estimating pose of a non-textured object, the estimation of pose being performed on a weakly textured object by a method of machine learning. Firstly, projection rendering is carried out on a 3D model of the weak texture object, different viewpoints, different illumination conditions and different scale distances are exhausted as far as possible, and a series of training data sets are obtained. And during training, the tree structure is adopted to index the blocks of the target object in the image. And performing matching retrieval on the input target image to be detected and the data set, and finally deducing the 6-degree-of-freedom posture of the target object in the current image.
However, in the process of implementing the invention, the inventor finds that the prior art has at least the following problems:
the chinese patent application CN107679537A of prior art 1 uses a conventional computer vision algorithm, which has the disadvantage that it is not possible to extract sufficient reliable visual features using ORB features for the contour of the projected image of most non-textured objects. In the process of carrying out ORB feature matching on an image to be detected, the feature uniqueness is poor and the feature matching outlier rate is extremely high due to the fact that object contour information in the image of the real environment is very easily interfered by noise, background, shielding and the like. Therefore, the pose solution with 6 degrees of freedom in a real environment is very easy to fail.
US patent US9892543B2 of prior art 2 uses a newer machine learning algorithm, which has the disadvantage that a training data set of a very large amount of data needs to be constructed for each weakly textured object to cover as common situations and poses as possible. The success rate and the precision of the method depend on the scale of the training data set, so that the method has high cost in the practical field, has no convenient expandability and cannot be quickly applied to another weak texture object. Another disadvantage of this method is that the matching and pose inference process is time-consuming and depends on a lot of computing resources, and thus does not have the capability of real-time pose tracking on general computing devices (such as smart phones, AR glasses, etc.).
In addition, weak texture objects in a real environment generally need to face the situation that occlusion, background interference, highlight material, specular reflection and the like seriously affect the performance of computer vision or machine learning algorithms, and the methods in the prior art have no good system robustness to the problems, and are prone to tracking failure or attitude error in the using process.
It should be noted that the above background description is only for the sake of clarity and complete description of the technical solutions of the present invention and for the understanding of those skilled in the art. Such solutions are not considered to be known to the person skilled in the art merely because they have been set forth in the background section of the invention.
Disclosure of Invention
In view of the foregoing problems, an object of embodiments of the present invention is to provide a method, a system, and a device for tracking a pose of a weak texture object, where a 3D model of the weak texture object and a live-action image of a target object are used, and training is performed without relying on a large amount of prior data, so that real-time pose tracking can be implemented on a common computing device.
In order to achieve the above object, an embodiment of the present invention provides a method for tracking a weak texture object pose, including: predicting the attitude of an input current frame image to obtain a predicted object attitude, which specifically comprises the following steps: inputting a current frame image; if the current frame is the initial frame, adjusting the default posture of the system as the predicted object posture rotation matrix R0＝RadjustTranslation vector t0＝tadjustThe default pose is represented by a default rotation matrix and translation vector, denoted as Rinit，tinitThe value is obtained by optimal calculation according to the following conditions:
wherein M represents a 3D model of the object; i represents an image; ori (m) represents the center of the 3D model of the object; ori (i) represents the center of the image; proj (; R, t) denotes projection according to attitude (R, t); area () represents a calculation Area;
adjusting the default posture according to a certain step length (Δ R, Δ t):
ΔR＝exp(δrx，δry，δrz)T，Δt＝(δtx，δty，δtz)T
wherein Δ R is a preset rotation step, and Δ t is a preset translation step; δ tx, δ ty, δ tz are step lengths of unit translation; δ rx, δ ry, δ rz is the step size of the unit rotation vector; exp () is an exponential mapping function under the SO (3) group, and converts the rotation amount into a rotation matrix;
the calculation criterion of the adjusted posture according to the SE (3) group is as follows:
Radjust＝ΔR*Rinit，tadjust＝ΔR*tinit+Δt
the predicted object pose (R)0，t0) Is R0＝Radjust，t0＝tadjust(ii) a Or, if it is currentlyIf the frame is not the initial frame, then the pose of the previous frame is used as the predicted object pose rotation matrix R0＝RprevTranslation vector t0＝tprev(ii) a Acquiring object contour shape data according to the predicted object posture, specifically comprising: performing projection rendering on the 3D model of the object according to the predicted object posture, and extracting a projection outline; sampling the outline information, and carrying out statistics on the local color information of the outline to obtain object outline shape data; searching and matching contour shape characteristics on the current live-action image according to the contour shape data of the object; and carrying out attitude calculation according to the matching information.
The embodiment of the invention also provides a system for tracking the posture of the weak texture object, which comprises the following components: the image acquisition unit is used for acquiring a real-scene image of the target object; the human-computer interaction unit is used for displaying the acquired real scene image of the target object on the display unit; a data storage unit for storing a 3D model of the object; a rendering unit, configured to perform projection rendering of the 3D model of the object according to the predicted object pose, specifically configured to: performing projection rendering on a 3D model of an object according to the predicted object posture, rendering a binary foreground image, and extracting a 2D contour of the foreground image; the communication synchronization unit is used for carrying out data synchronization on the rendered plane image and the image acquired by the image acquisition unit; the computing unit is used for predicting the posture of the input current frame image to obtain a predicted object posture, acquiring object contour shape data according to the predicted object posture, searching and matching contour shape characteristics on the current live-action image according to the object contour shape data, and performing posture calculation according to matching information; the computing unit is specifically configured to: judging whether the input current frame image is an initial frame; if the current frame is the initial frame, adjusting the default posture of the system as the predicted object posture rotation matrix R0＝RadjustTranslation vector t0＝tadjustThe default pose is represented by a default rotation matrix and translation vector, denoted as Rinit，tinitThe value is obtained by optimal calculation according to the following conditions:
wherein M represents a 3D model of the object; i represents an image; ori (m) represents the center of the 3D model of the object; ori (i) represents the center of the image; proj (; R, t) denotes projection according to attitude (R, t); area () represents a calculation Area;
adjusting the default posture according to a certain step length (Δ R, Δ t):
ΔR＝exp(δrx，δry，δrz)T，Δt＝(δtx，δty，δtz)T
wherein Δ R is a preset rotation step, and Δ t is a preset translation step; δ tx, δ ty, δ tz are step lengths of unit translation; δ rx, δ ry, δ rz is the step size of the unit rotation vector; exp () is an exponential mapping function under the SO (3) group, and converts the rotation amount into a rotation matrix;
the calculation criterion of the adjusted posture according to the SE (3) group is as follows:
Radjust＝ΔR*Rinit，tadjust＝ΔR*tinit+Δt
the predicted object pose (R)0，t0) Is R0＝Radjust，t0＝tadjust(ii) a If the current frame is not the initial frame, the pose of the previous frame is used as the predicted object pose rotation matrix R0＝RprevTranslation vector t0＝tprev。
The embodiment of the present invention further provides a weak texture object posture tracking device, including a memory and a processor, wherein: the memory is used for storing codes and documents; the processor is configured to execute the code and documents stored in the memory to implement the aforementioned method steps.
As can be seen from the above, the embodiment of the invention combines the projection contour of the 3D model of the weak texture object and the color information on the live-action image to perform statistical description of the local shape features, and the contour extraction is not performed on the live-action image, so that the interference of noise in the real-action image on the edge contour information can be dealt with; secondly, the embodiment of the invention carries out discretized conditional interval sampling when carrying out statistics on the local characteristics around the object outline, reserves the local area of a strong structure, and fades the local area of a weak structure and a flat structure, thereby well coping with the situations of background interference, shielding and the like; thirdly, the whole system only depends on the outer contour of the object and does not need to depend on the inner surface of the object, so that the embodiment of the invention can reduce the influence of the interference to the minimum under the complex conditions of high light material, specular reflection and the like, and has high system robustness; in addition, the input of the system in the embodiment of the invention only needs the 3D model of the weak texture object and the live-action image of the target object, and does not need to rely on a large amount of prior data for training, thereby having very high expansibility and adaptability. Compared with the traditional operators such as ORB, SIFT and the like, the used local feature statistical method is low in calculation complexity, and can achieve real-time posture tracking on common computing equipment (such as smart phones, AR glasses and the like).
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art are briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on the drawings without creative efforts.
FIG. 1 is a schematic flow chart of a method for tracking a posture of a weak texture object according to an embodiment of the present invention;
fig. 2 is a schematic flowchart of a process of predicting a pose of an input current frame image according to an embodiment of the present invention;
FIG. 3 is a schematic flow chart of acquiring object contour shape data according to a predicted object posture according to an embodiment of the present invention;
fig. 4 is a schematic flowchart of a process of searching and matching contour shape features on a current live-action image according to an embodiment of the present invention;
FIG. 5 is a schematic flow chart illustrating an attitude calculation according to matching information according to an embodiment of the present invention
FIG. 6 is a schematic diagram of an architecture of a weak texture object pose tracking system according to an embodiment of the present invention;
fig. 7 is a schematic structural diagram of another weak texture object pose tracking apparatus according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the technical solutions of the embodiments of the present invention will be described clearly and completely with reference to the accompanying drawings of the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all embodiments of the present invention. All other embodiments, which can be obtained by a person skilled in the art without any inventive step based on the embodiments of the present invention, are within the scope of the present invention.
The embodiment of the invention provides a weak texture object posture tracking method which mainly comprises object posture prediction, object contour shape data acquisition, real image contour shape searching and matching and posture calculation. As shown in fig. 1, the method specifically comprises the following steps:
The detailed mode of the step can be seen in fig. 2.
103, judging whether the current frame is an initial frame, if so, entering a step 104; if not, step 105 is entered.
In this embodiment, if the system never enters the tracking process, for example, a camera is just turned on, or the tracking state of the previous frame of the system is failed, the current frame is considered as the initial frame, otherwise, the current frame is a non-initial frame.
And 104, adjusting the default posture of the system to be used as the predicted posture of the object.
In this embodiment modeFor the initial frame, the system default pose is adjusted as the predicted object pose, i.e., (R)0，t0) Is (R)adjust，tadjust) Wherein R is0＝Radjust，t0＝tadjustR denotes a rotation matrix and t denotes a translation vector.
The default posture adjustment method is as follows:
the default pose is represented by a default rotation matrix and translation vector, denoted as Rinit，tinitThe value is obtained by optimal calculation according to the following conditions:
wherein M represents a 3D model of the object; i represents an image; ori (m) represents the center of the 3D model of the object; ori (i) represents the center of the image; proj (; R, t) denotes projection according to attitude (R, t); area () represents a calculation region Area.
Default posture Rinit，tinitAfter the model M is projected, the center of the model M is just positioned at the center of the image, and the area occupied by the projection area in the picture is the largest.
Adjusting the default posture according to a certain step length (Δ R, Δ t):
ΔR＝exp(δrx，δry，δrz)T，Δt＝(δtx，δty，δtz)T
wherein Δ R is a preset rotation step, and Δ t is a preset translation step; δ rx, δ ry, δ rz is the step size of the unit rotation vector; δ tx, δ ty, δ tz are step lengths of unit translation; exp () is an exponential mapping function under the SO (3) group, and converts the rotation amount into a rotation matrix, wherein SO (3) is a 3-dimensional special orthogonal group, any 3-dimensional rotation matrix belongs to SO (3), and the operation follows the SO (3) algorithm.
The calculation criterion of the adjusted posture according to the SE (3) group is as follows:
Radjust＝ΔR*Rinit，tadjust＝ΔR*tinit+Δt
wherein SE (3) is a 3-dimensional special Euclidean group and represents rigid body transformation, namely rotation and translation.
The predicted object pose (R)0，t0) Is (R)adjust，tadjust) Wherein R is0＝Radjust，t0＝tadjust。
In this embodiment, for a non-initial frame, the pose (R) of the previous frame may be used directlyprev，tprev) As predicted object attitude (R)0，t0) Wherein R is0＝Rprev，t0＝tprev。
And 2, acquiring object contour shape data according to the predicted object posture.
The detailed mode of the step can be seen in fig. 3.
And 202, performing projection rendering on the 3D model of the object according to the predicted object posture.
In the present embodiment, the object attitude R is predicted based on the predicted object attitude0＝Radjust，t0＝tadjustOr R0＝Rprev，t0＝tprevAnd performing projection rendering of the 3D model of the object.
In this embodiment, in order to reduce the complexity of the calculation, only the binarized foreground map Fg may be rendered, and then the 2D contour of the foreground map Fg may be extracted. Preferably, only the outline is extracted, and an outline extraction method, such as a method of statistical Structural Analysis of partitioned Binary Images by BorderFollowing, may be used, although other methods may be used, and are not limited.
It should be noted that, in the present embodiment, since the contour extraction is performed on the projected binary image, not on the live-action target image, the contour extraction is not interfered by the noise of the live-action image, and the obtained contour can be ensured to be accurate.
In this embodiment, after obtaining the projected 2D contour, the contour needs to be information sampled.
In the actual operation process, any sampling brings about a certain degree of shape information loss. The currently commonly used simple sampling method is to perform sampling at uniform intervals along the contour line to obtain a series of 2D contour sampling points. However, the shape characteristics of the object in the actual field of view are neglected in the uniform interval sampling mode, and the structural characteristics cannot be well preserved. For example, when sampling is sparse, shape loss is large; when the sampling is dense, the calculation amount is multiplied.
It should be noted that in this embodiment, an uneven sampling manner is adopted, that is, an adaptive sampling step length is set according to the local structural features of the contour, strong structural features are retained, flat regions and weak structural regions are thinned, the original shape is retained to the maximum extent, and the sampling density is not increased.
The adaptive sampling step calculation method is as follows:
λi＝λ0*Weigh(θ)，θ＝Angle(Pi-1)
wherein λ isiIs a self-adaptive sampling step length; lambda [ alpha ]0A base step size; theta represents more than one sampling point Pi-1Angle (P) formed at both sides of the contour line as the centeri-1) (ii) a Weigh (θ) represents a weighting factor for this angle.
The angle θ closer to 180 ° indicates that the region is less structural and flatter, whereas the region is more structural and sharper. Different weighting coefficients are set according to different included angles theta, a smaller coefficient is set in a region with strong structure, and a larger coefficient is set in a region with weak structure.
And step 205, counting the local color information of the contour to obtain object contour shape data.
In the present embodiment, the contour point obtained by sampling the contour information based on the adaptive sampling step is { P }i(u，v；x，y，z)|i＝[1，N]N contour points in total. Wherein the contour point PiBoth the 3D point position (x, y, z) in the 3D model of the object and the 2D position (u, v) on the projection image are included.
After N contour points are obtained, statistics of contour local color information is needed to obtain finally needed object contour shape data. In the present embodiment, the statistics are performed on the live view image of the target object in the previous frame.
First, each contour point P is calculatediNormal direction vector n ofi：
ni＝(vi-vi+1，ui+1-ui)
Wherein u isi，viIs a contour point PiThe pixel coordinates of (a); u. ofi+1，vi+1Is a contour point Pi+1The pixel coordinates of (a); the normal calculation formula is to express niAnd Pi、Pi+1The line between the points is vertical.
Then, pixels on the live view image are counted along the positive direction and the negative direction of the normal vector, and a color histogram is calculated as a local shape description of the contour point and is recorded as an internal shape and an external shape, respectively.
Finally, the internal shapes and the external shapes of all the contour points are respectively superposed to form the contour shape data Appearance of the objectobj：
Appearanceobj＝{∑iIntApperancei，∑iExtAppearancei}
Wherein, AppeanceiRepresenting a contour point PiIntApperatence ofiRepresenting the internal shape, ExtAppearanceiRepresenting the outer shape, Histo () is a statistical color histogram function, α is a step factor.
If L pixels are counted along the positive direction and the negative direction of the normal vector, the shape feature of the point is described by 2L +1 pixels. Therefore, the total number of points to be examined is N × 2L +1, and compared with the conventional method in which local features such as ORB and SIFT are required to extract features from the whole image, the amount of computation is reduced geometrically, and thus the method is very efficient in performing statistics on the contour local color information in the present embodiment.
And 3, searching and matching the contour shape characteristics on the current live-action image according to the contour shape data of the object.
The detailed mode of the step can be seen in fig. 4.
In the present embodiment, the object attitude R is predicted based on the predicted object attitude0＝Radjust，t0＝tadjustOr R0＝Rprev，t0＝tprevSampling point of the profile { Pi(u，v；x，y，z)|i＝[1，N]Projecting on the current live-action image.
In the present embodiment, each contour sampling point P is calculated according to the method of the aforementioned step 205iNormal direction vector n ofiThe gradient of each pixel is calculated on the current live-action image in the positive and negative directions of the normal.
In the present embodiment, a pixel gradient threshold value is set in advance. If the pixel gradient exceeds a threshold value, the pixel is considered as a candidate point; and if the pixel gradient does not exceed the threshold, returning to continue to search along the normal for candidate points that satisfy the gradient threshold. Marking the candidate point as { Ci|i＝[1，m]It can be set that m candidate points are found in total.
And 305, taking the pixel as a candidate point, counting local color information on two sides of the candidate point, comparing the local color information with the contour shape data of the object, and finding out a matching point according to the similarity.
In the present embodiment, each candidate point C is usediCounting color histograms on two sides of the normal line by taking the color histogram as a center to obtain local shape data application of the candidate pointcThen, the calculated object contour shape data is compared with the calculated object contour shape dataobjA similarity measure is performed. The similarity measure is a measure for comprehensively evaluating the degree of closeness between two things. The closer two things are, the larger their similarity measure is, and the further apart the two things are, the smaller their similarity measure is. The methods for measuring similarity are various and generally selected according to practical problems. For example, the distance L1 or L2 or other means may be used in this embodiment, and is not particularly limited herein.
In the present embodiment, the most similar point in the candidate points is found by the similarity measure, a similarity threshold may be set, and when the similarity measure exceeds the threshold, the point may be regarded as a matching point.
And step 306, inserting the found matching points into the 2D-3D matching pairs.
In this embodiment, the found matching points are inserted into a 2D-3D matching pair. The 2D coordinates in the 2D-3D matching pair come from the matching points, and the 3D coordinates come from the coordinates of the corresponding contour sampling points on the 3D model of the object.
And 4, carrying out attitude calculation according to the matching information.
The detailed manner of this step can be seen in fig. 5.
And step 401, solving the PnP by using a RANSAC mechanism according to the 2D-3D matching pair.
In the present embodiment, the PnP (passive-n-point) solution algorithm is to obtain the position and the posture of the object by calculating the projection relationship of a plurality of 2D-3D matching pairs. RANSAC (RANdom SAmple Consensus) can iteratively estimate parameters of an optimal mathematical model from a set of observed datasets comprising inliers (data that build the model) and outliers (data that cannot adapt to the model).
In the present embodiment, the number of inliers is determined from the reprojection error after the PnP is solved by the RANSAC mechanism, and it is determined whether or not the number of inliers exceeds a preset threshold, for example, 8.
As shown in fig. 6, the embodiment of the present invention further provides a system for tracking the posture of a weak texture object, where the input of the system is the live-action image of the target object collected by the image collection unit. The target object live-action image is displayed in a display unit of the equipment through a man-machine interaction unit, and the object 3D model is stored in a data storage unit. When the gesture tracking is carried out, the rendering unit is responsible for carrying out projection rendering on the 3D model of the object, the rendered plane image is subjected to data synchronization with the real image of the target object acquired by the image acquisition unit through the communication synchronization unit, gesture tracking calculation with 6 degrees of freedom is carried out in the calculation unit together with the real image of the target object, and finally gesture data of the target object are obtained.
The computing unit is specifically configured to perform attitude prediction on an input current frame image to obtain a predicted object attitude, obtain object contour shape data according to the predicted object attitude, perform search and matching of contour shape features on a current live-action image according to the object contour shape data, and perform attitude calculation according to matching information.
It should be noted that the specific technical details of the above-mentioned weak texture object posture tracking system are similar to the weak texture object posture tracking method, and in particular, the novel non-uniform adaptive step size contour point sampling algorithm and the efficient shape feature description and matching algorithm of the object contour involved in the processes of object posture prediction, object contour shape data acquisition, live-action image contour shape search and matching, and 6-degree-of-freedom posture calculation by the computing unit are similar to the aforementioned weak texture object posture tracking method, and therefore detailed description is omitted.
As shown in fig. 7, an embodiment of the present invention further provides a weak texture object pose tracking apparatus, including a memory and a processor, where:
a memory 701 for storing codes and documents;
a processor 702 for executing the code and documents stored in the memory for implementing the method for weak texture three-dimensional object pose estimation as described above.
The specific technical details of the above-mentioned weak texture object posture tracking device are similar to those of the above-mentioned weak texture object posture tracking method, and therefore detailed descriptions thereof are omitted.
As can be seen from the above, the embodiment of the invention combines the projection contour of the 3D model of the weak texture object and the color information on the live-action image to perform statistical description of the local shape features, and the contour extraction is not performed on the live-action image, so that the interference of noise in the real-action image on the edge contour information can be dealt with; secondly, the embodiment of the invention carries out discretized conditional interval sampling when carrying out statistics on the local characteristics around the object outline, reserves the local area of a strong structure, and fades the local area of a weak structure and a flat structure, thereby well coping with the situations of background interference, shielding and the like; thirdly, the whole system only depends on the outer contour of the object and does not need to depend on the inner surface of the object, so that the embodiment of the invention can reduce the influence of the interference to the minimum under the complex conditions of high light material, specular reflection and the like, and has high system robustness; in addition, the input of the system in the embodiment of the invention only needs the 3D model of the weak texture object and the live-action image of the target object, and does not need to rely on a large amount of prior data for training, thereby having very high expansibility and adaptability. Compared with the traditional operators such as ORB, SIFT and the like, the used local feature statistical method is low in calculation complexity, and can achieve real-time posture tracking on common computing equipment (such as smart phones, AR glasses and the like).
Those skilled in the art will understand that all or part of the steps in the method according to the above embodiments may be implemented by a program instructing related hardware to complete, where the program is stored in a storage medium and includes several instructions to enable a device (which may be a single chip, a chip, etc.) or a processor (processor) to execute all or part of the steps in the method according to the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
The embodiments in the present specification are described in a progressive manner, and the same and similar parts among the embodiments can be referred to each other, and each embodiment focuses on the differences from the other embodiments.
Finally, it should be noted that: the foregoing description of various embodiments of the invention is provided to those skilled in the art for the purpose of illustration. It is not intended to be exhaustive or to limit the invention to a single disclosed embodiment. Various alternatives and modifications of the invention, as described above, will be apparent to those skilled in the art. Thus, while some alternative embodiments have been discussed in detail, other embodiments will be apparent or relatively easy to derive by those of ordinary skill in the art. The present invention is intended to embrace all such alternatives, modifications, and variances which have been discussed herein, and other embodiments which fall within the spirit and scope of the above application.
Claims (10)
1. A weak texture object posture tracking method is characterized by comprising the following steps:
predicting the attitude of an input current frame image to obtain a predicted object attitude, which specifically comprises the following steps: inputting a current frame image; if the current frame is the initial frame, adjusting the default posture of the system as the predicted object posture rotation matrix R0＝RadjustTranslation vector t0＝tadjust(ii) a Or, if the current frame is not the initial frame, using the pose of the previous frame as the predicted object pose rotation matrix R0＝RprevTranslation vector t0＝tprev；
Acquiring object contour shape data according to the predicted object posture, specifically comprising: performing projection rendering on the 3D model of the object according to the predicted object posture, and extracting a projection outline; sampling the outline information, and carrying out statistics on the local color information of the outline to obtain object outline shape data;
searching and matching contour shape characteristics on the current live-action image according to the contour shape data of the object;
carrying out attitude calculation according to the matching information;
the default pose is represented by a default rotation matrix and translation vector, denoted as Rinit，tinitThe value is obtained by optimal calculation according to the following conditions:
wherein M represents a 3D model of the object; i represents an image; ori (m) represents the center of the 3D model of the object; ori (i) represents the center of the image; proj (; R, t) denotes projection according to attitude (R, t); area () represents a calculation Area;
adjusting the default posture according to a certain step length (Δ R, Δ t):
ΔR＝exp(δrx，δry，δrz)T，Δt＝(δtx，δty，δtz)T
wherein Δ R is a preset rotation step, and Δ t is a preset translation step; δ tx, δ ty, δ tz are step lengths of unit translation; δ rx, δ ry, δ rz is the step size of the unit rotation vector; exp () is an exponential mapping function under the SO (3) group, and converts the rotation amount into a rotation matrix;
the calculation criterion of the adjusted posture according to the SE (3) group is as follows:
Radjust＝ΔR*Rinit，tadjust＝ΔR*tinit+Δt
the predicted object pose (R)0，t0) Is R0＝Radjust，t0＝tadjust。
2. The weak texture object pose tracking method according to claim 1, wherein the extracting of the projection contour specifically comprises:
rendering a binarized foreground image, and extracting a 2D contour of the foreground image.
3. The weak texture object pose tracking method according to claim 1, wherein the sampling of the contour information specifically comprises:
setting a self-adaptive sampling step length, and sampling the profile information by adopting a non-uniform sampling mode;
the self-adaptive sampling step calculation mode is as follows:
λi＝λ0*Weigh(θ)，θ＝Angle(Pi-1)；
wherein λ is0A base step size; theta represents more than one sampling point Pi-1Angle (P) formed at both sides of the contour line as the centeri-1) (ii) a Weigh (θ) represents the weighting factor for this angle;
the contour point obtained by sampling the contour information according to the self-adaptive sampling step length is Pi(u，v；x，y，z)|i＝[1，N]In which the contour point PiBoth the 3D point position (x, y, z) in the 3D model of the object and the 2D position (u, v) on the projection image are included.
4. The method for tracking the posture of the weakly-textured object according to claim 3, wherein the step of performing statistics on the local color information of the contour to obtain the contour shape data of the object specifically comprises the following steps:
calculating each contour point PiNormal direction vector n ofi：ni＝(vi-vi+1，ui+1-ui)；
Wherein u isi，viIs a contour point PiThe pixel coordinates of (a); u. ofi+1，vi+1Is a contour point Pi+1The pixel coordinates of (a); n isiAnd Pi、Pi+1The connecting line between the points is vertical;
counting pixels on the live-action image along the positive direction and the negative direction of the normal vector, calculating a color histogram of the pixels, taking the color histogram as the local shape description of the contour point, and respectively recording the local shape description as an internal shape and an external shape;
respectively superposing the internal shapes and the external shapes of all the contour points to form the contour shape data Appearance of the objectobj：
Appearanceobj＝{∑iIntApperancei，∑iExtAppearancei}
Wherein Histo () is a statistical color histogram function; appeanceiRepresenting a contour point PiIntApperatence ofiRepresenting the internal shape, ExtAppearanceiRepresenting the outer shape, α is the step factor.
5. The method for tracking the pose of a weak texture object according to claim 4, wherein the searching and matching of the contour shape features on the current live-action image according to the contour shape data of the object specifically comprises:
sampling point P of the profileiProjecting on the current live-action image, and calculating the gradient of each pixel on the current live-action image along the positive direction and the negative direction of the normal of the contour sampling point;
presetting a pixel gradient threshold, if the pixel gradient exceeds the threshold, the pixel is a candidate point, and recording the candidate point as { C }i|i＝[1，m]}；
Counting local color information on two sides of the candidate point, and comparing the local color information with the object contour shape data;
finding out the most similar point in the candidate points as a matching point through similarity measurement, or presetting a similarity threshold value and finding out the candidate points with the similarity larger than the threshold value as the matching points;
and inserting the matching points into a 2D-3D matching pair, wherein the 2D coordinates in the 2D-3D matching pair are from the matching points, and the 3D coordinates are from the coordinates of the corresponding contour sampling points on the 3D model of the object.
6. The weak texture object posture tracking method according to claim 5, wherein the performing posture calculation according to the matching information specifically comprises:
solving PnP by using a RANSAC mechanism according to the 2D-3D matching pair, and determining the number of interior points according to the reprojection error;
and presetting an interior point number threshold, if the number of the interior points exceeds the threshold, successfully tracking the current frame, and outputting the posture with 6 degrees of freedom.
7. A weak texture object pose tracking system, comprising:
the image acquisition unit is used for acquiring a real-scene image of the target object;
the human-computer interaction unit is used for displaying the acquired real scene image of the target object on the display unit;
a data storage unit for storing a 3D model of the object;
a rendering unit, configured to perform projection rendering of the 3D model of the object according to the predicted object pose, specifically configured to: performing projection rendering on a 3D model of an object according to the predicted object posture, rendering a binary foreground image, and extracting a 2D contour of the foreground image;
the communication synchronization unit is used for carrying out data synchronization on the rendered plane image and the acquired real image of the target object;
a computing unit for predicting the input current frame image to obtain the predicted object gesture, obtaining the object outline shape data according to the predicted object gesture, searching and matching the outline shape characteristics on the current real scene image according to the object outline shape data, and performing gesture according to the matching informationState resolving; the computing unit is specifically configured to: judging whether the input current frame image is an initial frame; if the current frame is the initial frame, adjusting the default posture of the system as the predicted object posture rotation matrix R0＝RadjustTranslation vector t0＝tadjustThe method specifically comprises the following steps:
the default pose is represented by a default rotation matrix and translation vector, denoted as Rinit，tinitThe value is obtained by optimal calculation according to the following conditions:
wherein M represents a 3D model of the object; i represents an image; ori (m) represents the center of the 3D model of the object; ori (i) represents the center of the image; proj (; R, t) denotes projection according to attitude (R, t); area () represents a calculation Area;
adjusting the default posture according to a certain step length delta R, delta t:
ΔR＝exp(δrx，δry，δrz)T，Δt＝(δtx，δty，δtz)T
wherein, δ tx, δ ty, δ tz are the step length of unit translation; δ rx, δ ry, δ rz is the step size of the unit rotation vector; exp () is an exponential mapping function under the SO (3) group, and converts the rotation amount into a rotation matrix;
the calculation criterion of the adjusted posture according to the SE (3) group is as follows:
Radjust＝ΔR*Rinit，tadjust＝ΔR*tinit+Δt
the predicted attitude of the object is R0＝Radjust，t0＝tadjust(ii) a If the current frame is not the initial frame, the pose of the previous frame is used as the predicted object pose rotation matrix R0＝RprevTranslation vector t0＝tprev。
8. The weak texture object pose tracking system of claim 7, wherein the computing unit is further specifically configured to:
setting a self-adaptive sampling step length, and sampling the profile information by adopting a non-uniform sampling mode; the self-adaptive sampling step calculation mode is as follows:
λi＝λ0*Weigh(θ)，θ＝Angle(Pi-1)；
wherein λ is0A base step size; theta represents more than one sampling point Pi-1An included angle is formed on two sides of the contour line as a center; weigh (θ) represents the weighting factor for this angle;
the contour point obtained by sampling the contour information according to the self-adaptive sampling step length is Pi(u，v；x，y，z)|i＝[1，N]In which the contour point PiBoth the 3D point position (x, y, z) in the 3D model of the object and the 2D position (u, v) on the projection image;
calculating each contour point PiNormal direction vector n ofi：ni＝(vi-vi+1，ui+1-ui)；
Wherein u isi，viIs the pixel coordinate of the contour point Pi; u. ofi+1，vi+1Is a contour point Pi+1The pixel coordinates of (a); n isiAnd Pi、Pi+1The connecting line between the points is vertical;
counting pixels on the live-action image along the positive direction and the negative direction of the normal vector, calculating a color histogram of the pixels, taking the color histogram as the local shape description of the contour point, and respectively recording the local shape description as an internal shape and an external shape;
respectively superposing the internal shapes and the external shapes of all the contour points to form the contour shape data Appearance of the objectobj：
Appearanceobj＝{∑iIntApperancei，∑iExtAppearancei}
Where Histo () is a statistical color histogramA function; IntApperatenceiRepresenting the internal shape, ExtAppearanceiRepresenting the outer shape.
9. The weak texture object pose tracking system of claim 8, wherein the computing unit is further specifically configured to:
sampling point P of the profileiProjecting on the current live-action image, and calculating the gradient of each pixel on the current live-action image along the positive direction and the negative direction of the normal of the contour sampling point; presetting a pixel gradient threshold, if the pixel gradient exceeds the threshold, the pixel is a candidate point, and recording the candidate point as { C }i|i＝[1，m]}; counting local color information on two sides of the candidate point, and comparing the local color information with the object contour shape data; finding out the most similar point in the candidate points as a matching point through similarity measurement, or presetting a similarity threshold value and finding out the candidate points with the similarity larger than the threshold value as the matching points; inserting the matching points into a 2D-3D matching pair, wherein 2D coordinates in the 2D-3D matching pair come from the matching points, and 3D coordinates come from coordinates of corresponding contour sampling points on the 3D model of the object;
solving PnP by using a RANSAC mechanism according to the 2D-3D matching pair, and determining the number of interior points according to the reprojection error; and presetting an interior point number threshold, if the number of the interior points exceeds the threshold, successfully tracking the current frame, and outputting the posture with 6 degrees of freedom.
10. A weak texture object pose tracking apparatus, the apparatus comprising a memory and a processor, wherein:
the memory is used for storing codes and documents;
the processor for executing the code and documents stored in the memory for implementing the method steps of any of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910105602.6A CN109872343B (en) | 2019-02-01 | 2019-02-01 | Weak texture object posture tracking method, system and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910105602.6A CN109872343B (en) | 2019-02-01 | 2019-02-01 | Weak texture object posture tracking method, system and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109872343A CN109872343A (en) | 2019-06-11 |
| CN109872343B true CN109872343B (en) | 2020-03-17 |
Family
ID=66918593
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910105602.6A Active CN109872343B (en) | 2019-02-01 | 2019-02-01 | Weak texture object posture tracking method, system and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109872343B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111783986A (en) * | 2020-07-02 | 2020-10-16 | 清华大学 | Network training method and device and posture prediction method and device |
| CN113298870B (en) * | 2021-05-07 | 2023-03-28 | 中国科学院深圳先进技术研究院 | Object posture tracking method and device, terminal equipment and storage medium |
| CN114067041B (en) * | 2022-01-14 | 2022-06-14 | 深圳大学 | Material generation method and device of three-dimensional model, computer equipment and storage medium |
Family Cites Families (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN100483462C (en) * | 2002-10-18 | 2009-04-29 | 清华大学 | Establishing method of human face 3D model by fusing multiple-visual angle and multiple-thread 2D information |
| CN101719286B (en) * | 2009-12-09 | 2012-05-23 | 北京大学 | Multiple viewpoints three-dimensional scene reconstructing method fusing single viewpoint scenario analysis and system thereof |
| CN102385754B (en) * | 2010-08-30 | 2016-12-07 | 三星电子株式会社 | A kind of object tracking methods and equipment |
| CN102184551A (en) * | 2011-05-10 | 2011-09-14 | 东北大学 | Automatic target tracking method and system by combining multi-characteristic matching and particle filtering |
| US10157468B2 (en) * | 2014-05-07 | 2018-12-18 | Nec Corporation | Object detection device, object detection method, and object detection system |
| CN105096377B (en) * | 2014-05-14 | 2019-03-19 | 华为技术有限公司 | A kind of image processing method and device |
| CN106447725B (en) * | 2016-06-29 | 2018-02-09 | 北京航空航天大学 | Spatial target posture method of estimation based on the matching of profile point composite character |
| CN107679537B (en) * | 2017-05-09 | 2019-11-19 | 北京航空航天大学 | A kind of texture-free spatial target posture algorithm for estimating based on profile point ORB characteristic matching |
| CN108509848B (en) * | 2018-02-13 | 2019-03-05 | 视辰信息科技(上海)有限公司 | The real-time detection method and system of three-dimension object |
| CN109035327B (en) * | 2018-06-25 | 2021-10-29 | 北京大学 | Panoramic camera attitude estimation method based on deep learning |
-
2019
- 2019-02-01 CN CN201910105602.6A patent/CN109872343B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN109872343A (en) | 2019-06-11 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107045631B (en) | Method, device and equipment for detecting human face characteristic points | |
| Aldoma et al. | Multimodal cue integration through hypotheses verification for rgb-d object recognition and 6dof pose estimation | |
| CN109472820B (en) | Monocular RGB-D camera real-time face reconstruction method and device | |
| US9417700B2 (en) | Gesture recognition systems and related methods | |
| CN108229416B (en) | Robot SLAM method based on semantic segmentation technology | |
| CN109872343B (en) | Weak texture object posture tracking method, system and device | |
| CN110751097B (en) | Semi-supervised three-dimensional point cloud gesture key point detection method | |
| CN111144349B (en) | Indoor visual relocation method and system | |
| Taylor et al. | Fusion of multimodal visual cues for model-based object tracking | |
| CN109272577B (en) | Kinect-based visual SLAM method | |
| CN111709980A (en) | Multi-scale image registration method and device based on deep learning | |
| CN113393503B (en) | Classification-driven shape prior deformation category-level object 6D pose estimation method | |
| Zhang et al. | Hand-held monocular SLAM based on line segments | |
| CN114332214A (en) | Object attitude estimation method and device, electronic equipment and storage medium | |
| US20230019499A1 (en) | Image processing system and method | |
| CN107808391B (en) | Video dynamic target extraction method based on feature selection and smooth representation clustering | |
| JP6955081B2 (en) | Electronic devices, systems and methods for determining object orientation | |
| CN117351078A (en) | Target size and 6D gesture estimation method based on shape priori | |
| EP2237227A1 (en) | Video sequence processing method and system | |
| CN106406507B (en) | Image processing method and electronic device | |
| US20230020713A1 (en) | Image processing system and method | |
| CN116416305B (en) | Multi-instance pose estimation method based on optimized sampling five-dimensional point pair characteristics | |
| CN110647925A (en) | Rigid object identification method and device based on improved LINE-MOD template matching | |
| Nakano | Stereo vision based single-shot 6d object pose estimation for bin-picking by a robot manipulator | |
| Guo et al. | A hybrid framework based on warped hierarchical tree for pose estimation of texture-less objects |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |