CN111340873A - Method for measuring and calculating object minimum outer envelope size of multi-view image - Google Patents

Method for measuring and calculating object minimum outer envelope size of multi-view image Download PDF

Info

Publication number
CN111340873A
CN111340873A CN202010128916.0A CN202010128916A CN111340873A CN 111340873 A CN111340873 A CN 111340873A CN 202010128916 A CN202010128916 A CN 202010128916A CN 111340873 A CN111340873 A CN 111340873A
Authority
CN
China
Prior art keywords
feature points
confidence
points
point cloud
feature
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202010128916.0A
Other languages
Chinese (zh)
Other versions
CN111340873B (en
Inventor
何力
朱蕾
陈炜楠
管贻生
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Guangdong University of Technology
Original Assignee
Guangdong University of Technology
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Guangdong University of Technology filed Critical Guangdong University of Technology
Priority to CN202010128916.0A priority Critical patent/CN111340873B/en
Publication of CN111340873A publication Critical patent/CN111340873A/en
Application granted granted Critical
Publication of CN111340873B publication Critical patent/CN111340873B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/60—Analysis of geometric attributes
    • G06T7/62—Analysis of geometric attributes of area, perimeter, diameter or volume
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01B—MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
    • G01B11/00—Measuring arrangements characterised by the use of optical techniques
    • G01B11/002—Measuring arrangements characterised by the use of optical techniques for measuring two or more coordinates
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01B—MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
    • G01B11/00—Measuring arrangements characterised by the use of optical techniques
    • G01B11/24—Measuring arrangements characterised by the use of optical techniques for measuring contours or curvatures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10028—Range image; Depth image; 3D point clouds
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
    • Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
    • Y02P90/30—Computing systems specially adapted for manufacturing

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Data Mining & Analysis (AREA)
  • General Engineering & Computer Science (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Evolutionary Biology (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Geometry (AREA)
  • Image Analysis (AREA)

Abstract

The application discloses a method for measuring and calculating the minimum object outer envelope size of a multi-view image, which comprises the following steps: respectively shooting images of an object to be measured from two different angles by using terminal equipment, and calculating spatial coordinates of pixel points in the shot images in a world coordinate system so as to obtain a three-dimensional point cloud model of the images; separating and extracting the point cloud of the object from the three-dimensional point cloud model of the image to complete the point cloud extraction of the object; and for the point cloud of the object, generating a vertex of the surface of the object by using a directed envelope box, and obtaining a characteristic vector through principal component analysis, so as to obtain the minimum outer envelope of the object and obtain the size of the object. According to the method, the actual application requirement of the logistics industry on package information acquisition is better met by acquiring the multi-view images of the object and extracting the three-dimensional point cloud model to estimate the size; the method realizes process automation, does not need manual operation except shooting, and has wide application scenes and convenient use.

Description

Method for measuring and calculating object minimum outer envelope size of multi-view image
Technical Field
The application relates to the field of logistics and image processing, in particular to a method for measuring and calculating the minimum outer envelope size of an object of a multi-view image.
Background
The wide application of regional economic integration and internet technology enables the logistics industry to be revolutionarily developed. Refined distribution and intelligent operation are basic requirements of modern logistics, and the weight and volume of packages are the most basic elements in the logistics. At present, the acquisition of parcel weight information is very convenient, and the acquisition of volume information has the problems of complex operation, low effect and the like, thereby limiting the development of logistics intellectualization to a certain extent.
Along with the popularization of mobile phones and the improvement of image processing technology, logistics practitioners measure the length, width and height of packages by using mobile phone cameras, sorting and delivering of the packages can be simplified, and logistics efficiency is improved. However, the processing of parcel image information and the like and the calculation and measurement of the minimum outer envelope of parcels with irregular shapes and the like (such as tables and chairs) are two major difficulties.
Disclosure of Invention
The application aims to provide a method for measuring and calculating the minimum outer envelope size of an object in a multi-view image, which is used for acquiring the outer envelope of the object by using the multi-view image shot by a terminal device to obtain the size information of the object.
In order to realize the task, the following technical scheme is adopted in the application:
an object minimum outer envelope dimension measuring and calculating method of a multi-view image comprises the following steps:
respectively shooting images of an object to be measured from two different angles by using terminal equipment, and calculating spatial coordinates of pixel points in the shot images in a world coordinate system so as to obtain a three-dimensional point cloud model of the images;
separating and extracting the point cloud of the object from the three-dimensional point cloud model of the image to complete the point cloud extraction of the object;
and for the point cloud of the object, generating a vertex of the surface of the object by using a directed envelope box, and obtaining a characteristic vector through principal component analysis, so as to obtain the minimum outer envelope of the object and obtain the size of the object.
Further, the calculating a spatial coordinate of a pixel point in the shot image in a world coordinate system to obtain a three-dimensional point cloud model of the image includes:
if the origins of the camera coordinate system and the world coordinate system coincide with each other, the same object has the same depth under the two coordinate systems, and therefore, the pixel point p (u, v) in the image shot by the camera has the spatial coordinate relation (X) with the pixel point p (u, v) in the world coordinate systemw,Yw,Zw) Comprises the following steps:
Figure BDA0002395265430000021
wherein f is the focal length of the camera, the pixel coordinate (u) of the image center0,v0) And the physical size of each pixel of the light sensing device in the camera is dxAnd dy；
The point (x, y) corresponding to the pixel point p (u, v) in the image in the camera coordinate system satisfies the relationship:
Figure BDA0002395265430000022
let the coordinate of a point P on the object in the world coordinate system be (X)w，Yw，Zw) According to two times of shooting at different angles, X can be calculated and obtained due to the intersection point of the spatial positions of the corresponding pixels in the two shot imageswAnd YwAnd P is in a ray O from the optical centerCp is above;
note OCAnd OC' is the center of the lens when the object is photographed twice from different angles, respectively, then P is the ray OCp and OCPoint of intersection of 'p', the point of intersectionI.e. seeking points that satisfy the geometric constraints on the orders so that the reprojection error is minimal:
C(p，p′)＝d(P，p)2+d(P，p′)2
where C (P, P ') is the reprojection error value, P represents a point on the object, P and P' are the same point in the image taken twice by the camera, respectively, d ()2Solving a function for the Euclidean distance;
by solving the above equation, Z is obtainedw(ii) a According to the same method, the spatial coordinates of all pixel points in the shot image are calculated, and the coordinate points form a three-dimensional point cloud model of the image, so that the three-dimensional point cloud expression of the image is completed.
Further, the separating and extracting the point cloud of the object from the three-dimensional point cloud model of the image to complete the point cloud extraction of the object includes:
extracting the characteristics of the collected point cloud data, judging and classifying the characteristic expressions of the extracted characteristic points, and dividing the characteristic points into two types of high confidence degree and low confidence degree;
for the high-confidence characteristic points, taking the category corresponding to the maximum value in the characteristic vectors of the characteristic points as the category of the characteristic points; for the feature points with low confidence coefficient, establishing a similarity expression matrix between the feature points by using position information between the feature points;
taking the similarity expression matrix as the association between the feature points with low confidence coefficient, and classifying the feature points with low confidence coefficient into the category of the feature points with high confidence coefficient with the maximum association;
and summarizing and combining the feature points divided into the same category to realize semantic segmentation of the point cloud.
Further, the discriminating and classifying the feature expression of the extracted feature points to classify the feature points into two categories, namely high confidence level and low confidence level, includes:
and (3) distinguishing the confidence coefficient of feature expression by taking n as a threshold value for the proportion of the maximum value to the second maximum value in the feature vectors of the feature points: and taking the feature points with the ratio of the maximum value to the second largest value in the feature vectors larger than n as high-confidence feature points, and taking the feature points smaller than n as low-confidence feature points.
Further, for the feature points with low confidence coefficient, establishing a similarity expression matrix between the feature points by using the position information between the feature points, including:
constructing a feature correlation matrix M between feature points by using feature expression vectors of the feature points with low confidencefv(ii) a Normalizing the coordinates of the feature points, and constructing a distance correlation matrix M between the feature points by using the distances of the normalized coordinates between the feature pointsdm(ii) a The similarity expression matrix M between feature points is | Mfv-Mdm|。
Further, the characteristic correlation matrix MfvIn (2), feature expression vector V of any two feature pointsi、VjThe similarity of (A) is as follows:
Figure BDA0002395265430000031
cov (V)i，Vj) Is a Vi、VjCovariance of D (V)i)、D(Vj) Are each Vi、VjThe variance of (c).
Further, the distance correlation matrix MdmThe P, Q distance between any two feature points is:
Figure BDA0002395265430000032
wherein (x)P，yP，zP)、(xQ，yQ，zQ) The normalized coordinates of P, Q representing the feature points.
Further, the classifying the low-confidence feature points into the category where the high-confidence feature points with the maximum association degree are located by using the similarity expression matrix as the association between the feature points includes:
and constructing a network graph by using graph theory, and classifying the low-confidence characteristic points into the category of the high-confidence characteristic points with the maximum relevance by combining the similarity expression matrix.
Further, the constructing a network graph by using graph theory, and classifying the low-confidence feature points into the category of the high-confidence feature points with the maximum association degree by combining the similarity expression matrix includes:
establishing a network graph, taking all feature points as vertexes in the network graph, taking the association between adjacent feature points as edges of the graph, and defining the weight of each edge according to the similarity between the feature points, wherein the weight of each edge is defined by the similarity expression matrix; and then, calculating the probability of each low-confidence characteristic point reaching each high-confidence characteristic point by utilizing the weight of the edge, taking the class of the high-confidence characteristic point with the maximum probability as the class of the low-confidence characteristic point, and dividing the low-confidence characteristic points into the class.
The application provides a terminal device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method for measuring and calculating the minimum outer envelope size of an object in a multi-view image according to the first aspect when executing the computer program.
The present application provides a computer readable storage medium storing a computer program, which when executed by a processor implements the steps of the method for object minimum outer envelope dimension estimation for multi-view images of the aforementioned first aspect.
The application has the following technical characteristics:
1. the method for acquiring the multi-view images of the object and extracting the three-dimensional point cloud model to estimate the size by using the terminal equipment, such as a mobile phone and the like, is provided, and the actual application requirement of the logistics industry on acquisition of the parcel information is better met; the method realizes process automation, does not need manual operation except shooting, and has wide application scenes and convenient use.
2. The method simplifies the processes of sorting, classifying and delivering the packages, improves the logistics efficiency and is beneficial to promoting the development of modern logistics intelligent operation.
Drawings
FIG. 1 is a schematic flow diagram of the process of the present application;
FIG. 2 is a schematic diagram of a pinhole model of a camera;
FIG. 3 is a schematic diagram of calculating object space coordinates from different perspectives;
FIGS. 4 (a), (b), (c) are schematic diagrams of point clouds illustrating an environment and an object from three different angles, respectively;
fig. 5 (a), (b), and (c) are point cloud schematic diagrams showing chairs separated from the three-dimensional point cloud environment at three different angles, respectively.
Detailed Description
In view of the problems of complicated operation, low efficiency and the like in the logistics industry of acquiring volume information of parcels (particularly parcels with irregular shapes), the method for measuring and calculating the minimum object outer envelope size of the multi-view image is provided. The method uses point cloud to express packages shot in multiple visual angles; performing semantic segmentation on a multi-angle image acquired by a mobile phone camera by utilizing a deep convolutional neural network, taking planes for placing logistics packages such as a desktop and a floor as segmentation targets, and separating the packages from the placing planes where the packages are placed; and according to the package point cloud obtained by segmentation, obtaining the length, width and height of the minimum external cuboid.
As shown in fig. 1, the present application provides a method for measuring and calculating the minimum object outer envelope size of a multi-view image, comprising the following steps:
and S1, respectively shooting images of the object to be measured from two different angles by using the terminal equipment, and calculating the space coordinates of pixel points in the shot images in a world coordinate system, thereby obtaining a three-dimensional point cloud model of the images.
In the embodiment of the application, the terminal device adopts a mobile phone, and the images of the object are respectively acquired from the left side surface and the right side surface of the object through a rear camera of the mobile phone. In the pinhole model structure of the camera, the camera's internal parameters are known quantities related to the camera's own properties, including the focal length f, the pixel coordinates (u) of the image center0,v0) And the physical size of each pixel of the light sensing device in the camera is dxAnd dy。
If the origin points of the camera coordinate system and the world coordinate system are coincidentThe single and same object in the two coordinate systems has the same depth, so that the pixel point p (u, v) in the image shot by the camera and the space coordinate relation (X) of the pixel point in the world coordinate systemw,Yw,Zw) Comprises the following steps:
Figure BDA0002395265430000051
wherein Z iscRepresenting the coordinates of the object in the Z-axis in the camera coordinate system.
The point (x, y) corresponding to the pixel point p (u, v) in the image in the camera coordinate system satisfies the relationship:
Figure BDA0002395265430000052
as shown in FIG. 2, OCXCYCZCAs camera coordinate system, OwXwYwZwIs a world coordinate system. The coordinate of a point P on the object in the world coordinate system is (X)w，Yw，Zw) According to the two times of shooting at different angles, X can be calculated as the intersection point of the spatial positions of the corresponding pixels in the two shot images is shown in FIG. 3wAnd YwAnd P is in a ray O from the optical centerCp is above.
Since in the imaging model shown in FIG. 2, the depth of the P point (i.e., Z)w) Is a ray which passes through the midpoint p of the picture plane starting from the center of the camera lens, for ZwAs shown in fig. 3:
OCand OC' is the center of the lens when the object is photographed twice from different angles, respectively, then P is the ray OCp and OCThe intersection of 'p', i.e., the point that seeks to satisfy the geometric constraint on the stage, minimizes the reprojection error:
C(p，p′)＝d(P，p)2+d(P，p′)2
wherein C (P, P ') is a reprojection error value, P represents a point on the object, P and P'The same point in the image, d ()2Solving the function for the euclidean distance, i.e.:
Figure BDA0002395265430000061
therefore, after two shots at different angles, the formula C (P, P') ═ d (P, P)2+d(P，p′)2Containing only an unknown quantity of ZwThus solving for Zw。
According to the same method, the spatial coordinates of all pixel points in the shot image are calculated, and the coordinate points form a three-dimensional point cloud model of the image, so that the three-dimensional point cloud expression of the image is completed.
And S2, separating and extracting the point cloud of the object from the three-dimensional point cloud model of the image by using the deep neural network, and finishing the point cloud extraction of the object.
The method and the device combine the characteristic description of the point cloud and the distance between the point pair to construct the similarity expression of the characteristic points. The characteristic description is the Pearson similarity of the characteristic points in the pre-training point cloud classification; the distance between the point pairs is represented by the euclidean distance. Introducing a random walk algorithm in the two-dimensional image segmentation field into the three-dimensional point cloud processing, and classifying a two-dimensional similarity matrix, so as to realize the efficient segmentation of the three-dimensional point cloud, wherein the method specifically comprises the following steps:
and S2.1, extracting the characteristics of the three-dimensional point cloud model, judging and classifying the characteristic expression of the extracted characteristic points, and classifying the characteristic points into two types of high confidence degree and low confidence degree.
Specifically, for the ratio of the maximum value to the second largest value in the feature vectors of the feature points, n is used as a threshold, and the confidence coefficient of feature expression is distinguished: and taking the feature points with the ratio of the maximum value to the second largest value in the feature vectors larger than n as high-confidence feature points, and taking the feature points smaller than n as low-confidence feature points. n is a constant that can be adjusted.
S2.2, for the high-confidence characteristic points, taking the category corresponding to the maximum value in the characteristic vectors of the characteristic points as the category of the characteristic points; for the feature points with low confidence coefficient, establishing a similarity expression matrix between the feature points by using the position information between the feature points, wherein the similarity expression matrix comprises the following steps:
constructing a feature correlation matrix M between feature points by using feature expression vectors of the feature points with low confidencefv(ii) a Normalizing the coordinates of the feature points, and constructing a distance correlation matrix M between the feature points by using the distances of the normalized coordinates between the feature pointsdm(ii) a The similarity expression matrix M between feature points is | Mfv-Mdm|。
The characteristic correlation matrix MfvIn (2), feature expression vector V of any two feature pointsi、VjThe similarity of (A) is as follows:
Figure BDA0002395265430000071
cov (V)i，Vj) Is a Vi、VjCovariance of D (V)i)、D(Vj) Are each Vi、VjThe variance of (c).
The distance correlation matrix MdmThe P, Q distance between any two feature points is:
Figure BDA0002395265430000072
wherein (x)P，yP，ZP)、(xQ，yQ，zQ) The normalized coordinates of P, Q representing the feature points.
S2.3, using the similarity expression matrix as the association between the feature points with low confidence coefficient, and classifying the feature points with low confidence coefficient into the category of the feature points with high confidence coefficient with the maximum association degree, including:
constructing a network graph by using graph theory, and classifying the low-confidence characteristic points into the category of the high-confidence characteristic points with the maximum association degree by combining the similarity expression matrix, wherein the method specifically comprises the following steps:
establishing a network graph, taking all feature points as vertexes in the network graph, taking the association between adjacent feature points as edges of the graph, and defining the weight of each edge according to the similarity between the feature points, wherein the weight of each edge is defined by the similarity expression matrix; and then, calculating the probability of each low-confidence characteristic point reaching each high-confidence characteristic point by utilizing the weight of the edge, taking the class of the high-confidence characteristic point with the maximum probability as the class of the low-confidence characteristic point, and dividing the low-confidence characteristic points into the class.
And S2.4, summarizing and combining the feature points divided into the same category to realize semantic segmentation of the point cloud and extract the point cloud of the object.
As shown in fig. 4, for the chair to be subjected to outer envelope extraction, the diagrams (a) to (c) in fig. 4 show the environment of the chair from three different angles, and the diagrams (a) to (c) in fig. 5 are point cloud schematic diagrams showing the chair separated from the chair in the three-dimensional point cloud environment from three different angles.
S3, for the point cloud of the object, generating the vertex of the object surface by using an oriented envelope box (OBB), and obtaining a characteristic vector, namely the main axis of the OBB, through Principal Component Analysis (PCA); the principal component analysis is to calculate a covariance matrix formed by covariance between any two variables, judge the correlation size between the variables, extract a group of linear uncorrelated variable sets as principal components, and obtain the eigenvector of the covariance matrix, namely the direction of the OBB envelope box, so as to obtain the minimum outer envelope of the object and obtain the length, width and height dimensions of the object, thereby facilitating stacking and storage of goods in practical application.
The above embodiments are only used to illustrate the technical solutions of the present application, and not to limit the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; such modifications and substitutions do not substantially depart from the spirit and scope of the embodiments of the present application and are intended to be included within the scope of the present application.

Claims (10)

1. A method for measuring and calculating the minimum object outer envelope size of a multi-view image is characterized by comprising the following steps:
respectively shooting images of an object to be measured from two different angles by using terminal equipment, and calculating spatial coordinates of pixel points in the shot images in a world coordinate system so as to obtain a three-dimensional point cloud model of the images;
separating and extracting the point cloud of the object from the three-dimensional point cloud model of the image to complete the point cloud extraction of the object;
and for the point cloud of the object, generating a vertex of the surface of the object by using a directed envelope box, and obtaining a characteristic vector through principal component analysis, so as to obtain the minimum outer envelope of the object and obtain the size of the object.
2. The method for measuring and calculating the minimum outer envelope size of an object in a multi-view image according to claim 1, wherein the step of calculating the space coordinates of pixel points in the captured image in a world coordinate system to obtain a three-dimensional point cloud model of the image comprises:
if the origins of the camera coordinate system and the world coordinate system coincide with each other, the same object has the same depth under the two coordinate systems, and therefore, the pixel point p (u, v) in the image shot by the camera has the spatial coordinate relation (X) with the pixel point p (u, v) in the world coordinate systemw，Yw，Zw) Comprises the following steps:
Figure FDA0002395265420000011
wherein f is the focal length of the camera, the pixel coordinate (u) of the image center0，v0) And the physical size of each pixel of the light sensing device in the camera is dxAnd dy；
The point (x, y) corresponding to the pixel point p (u, v) in the image in the camera coordinate system satisfies the relationship:
Figure FDA0002395265420000012
let the coordinate of a point P on the object in the world coordinate system be (X)w，Yw，Zw) According to two times of shooting at different angles, X can be calculated and obtained due to the intersection point of the spatial positions of the corresponding pixels in the two shot imageswAnd YwAnd P is in a ray O from the optical centerCp is above;
note OCAnd OC' is the center of the lens when the object is photographed twice from different angles, respectively, then P is the ray OCp and OCThe intersection of 'p', i.e., the point that seeks to satisfy the geometric constraint on the stage, minimizes the reprojection error:
C(p，p′)＝d(P，p)2+d(P，p′)2
where C (P, P ') is the reprojection error value, P represents a point on the object, P and P' are the same point in the image taken twice by the camera, respectively, d ()2Solving a function for the Euclidean distance;
by solving the above equation, Z is obtainedw(ii) a According to the same method, the spatial coordinates of all pixel points in the shot image are calculated, and the coordinate points form a three-dimensional point cloud model of the image, so that the three-dimensional point cloud expression of the image is completed.
3. The method for measuring and calculating the minimum outer envelope dimension of the object in the multi-view image according to claim 1, wherein the step of separating and extracting the point cloud of the object from the three-dimensional point cloud model of the image comprises the steps of:
extracting the characteristics of the collected point cloud data, judging and classifying the characteristic expressions of the extracted characteristic points, and dividing the characteristic points into two types of high confidence degree and low confidence degree;
for the high-confidence characteristic points, taking the category corresponding to the maximum value in the characteristic vectors of the characteristic points as the category of the characteristic points; for the feature points with low confidence coefficient, establishing a similarity expression matrix between the feature points by using position information between the feature points;
taking the similarity expression matrix as the association between the feature points with low confidence coefficient, and classifying the feature points with low confidence coefficient into the category of the feature points with high confidence coefficient with the maximum association;
and summarizing and combining the feature points divided into the same category to realize semantic segmentation of the point cloud.
4. The method for measuring and calculating the minimum object envelope dimension of a multi-view image according to claim 3, wherein the step of performing discriminant classification on the feature expressions of the extracted feature points to classify the feature points into two categories, namely high confidence and low confidence, comprises the steps of:
and (3) distinguishing the confidence coefficient of feature expression by taking n as a threshold value for the proportion of the maximum value to the second maximum value in the feature vectors of the feature points: and taking the feature points with the ratio of the maximum value to the second largest value in the feature vectors larger than n as high-confidence feature points, and taking the feature points smaller than n as low-confidence feature points.
5. The method for measuring and calculating the minimum outer envelope size of an object in a multi-view image according to claim 3, wherein for the feature points with low confidence, establishing a similarity expression matrix between the feature points by using the position information between the feature points comprises:
constructing a feature correlation matrix M between feature points by using feature expression vectors of the feature points with low confidencefv(ii) a Normalizing the coordinates of the feature points, and constructing a distance correlation matrix M between the feature points by using the distances of the normalized coordinates between the feature pointsdm(ii) a The similarity expression matrix M between feature points is | Mfv-Mdm|。
6. The method as claimed in claim 5, wherein the feature correlation matrix M is a matrix of minimum object envelop dimensionsfvIn (2), feature expression vector V of any two feature pointsi、VjThe similarity of (A) is as follows:
Figure FDA0002395265420000031
cov (V)i，Vj) Is a Vi、VjCovariance of D (V)i)、D(Vj) Is divided intoIs other than Vi、VjThe variance of (a);
the distance correlation matrix MdmThe P, Q distance between any two feature points is:
Figure FDA0002395265420000032
wherein (x)P，yP，zP)、(xQ，yQ，zQ) The normalized coordinates of P, Q representing the feature points.
7. The method for measuring and calculating the minimum object outer envelope size of the multi-view image according to claim 3, wherein the step of classifying the feature points with low confidence degree into the category of the feature points with high confidence degree with the maximum relevance degree by using the similarity expression matrix as the relevance among the feature points comprises the steps of:
and constructing a network graph by using graph theory, and classifying the low-confidence characteristic points into the category of the high-confidence characteristic points with the maximum relevance by combining the similarity expression matrix.
8. The method for measuring and calculating the minimum object outer envelope size of the multi-view image according to claim 7, wherein the step of constructing the network graph by using graph theory and combining the similarity expression matrix to classify the low-confidence feature points into the category of the high-confidence feature points with the maximum association degree comprises the following steps:
establishing a network graph, taking all feature points as vertexes in the network graph, taking the association between adjacent feature points as edges of the graph, and defining the weight of each edge according to the similarity between the feature points, wherein the weight of each edge is defined by the similarity expression matrix; and then, calculating the probability of each low-confidence characteristic point reaching each high-confidence characteristic point by utilizing the weight of the edge, taking the class of the high-confidence characteristic point with the maximum probability as the class of the low-confidence characteristic point, and dividing the low-confidence characteristic points into the class.
9. A terminal device comprising a memory, a processor and a computer program stored in said memory and executable on said processor, characterized in that the steps of the method according to any of claims 1-8 are implemented when the computer program is executed by the processor.
10. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out the steps of the method according to any one of claims 1 to 8.
CN202010128916.0A 2020-02-28 2020-02-28 Object minimum outer envelope size measuring and calculating method for multi-view image Active CN111340873B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202010128916.0A CN111340873B (en) 2020-02-28 2020-02-28 Object minimum outer envelope size measuring and calculating method for multi-view image

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010128916.0A CN111340873B (en) 2020-02-28 2020-02-28 Object minimum outer envelope size measuring and calculating method for multi-view image

Publications (2)

Publication Number Publication Date
CN111340873A true CN111340873A (en) 2020-06-26
CN111340873B CN111340873B (en) 2023-05-23

Family

ID=71187181

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010128916.0A Active CN111340873B (en) 2020-02-28 2020-02-28 Object minimum outer envelope size measuring and calculating method for multi-view image

Country Status (1)

Country Link
CN (1) CN111340873B (en)

Cited By (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112001412A (en) * 2020-07-10 2020-11-27 浙江大华技术股份有限公司 Multi-view vehicle association method and related device
CN112308912A (en) * 2020-11-03 2021-02-02 长安大学 System, device and method for acquiring road surface disease homologous multi-characteristic image
CN112738393A (en) * 2020-12-25 2021-04-30 珠海西山居移动游戏科技有限公司 Focusing method and device
CN113034695A (en) * 2021-04-16 2021-06-25 广东工业大学 Wasserstein distance-based object envelope multi-view reconstruction and optimization method
CN113160414A (en) * 2021-01-25 2021-07-23 北京豆牛网络科技有限公司 Automatic identification method and device for remaining amount of goods, electronic equipment and computer readable medium
CN113888618A (en) * 2021-09-30 2022-01-04 北京工业大学 Object single piece separation method and system
CN114998665A (en) * 2022-08-04 2022-09-02 创新奇智(广州)科技有限公司 Image category identification method and device, electronic equipment and storage medium
CN115410135A (en) * 2022-11-01 2022-11-29 中国民航大学 Autonomous-type-carried aviation luggage feature perception reconstruction method and system and application thereof

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104751146A (en) * 2015-04-13 2015-07-01 中国科学技术大学 Indoor human body detection method based on 3D (three-dimensional) point cloud image
CN107025685A (en) * 2017-04-11 2017-08-08 南京林业大学 Airborne building summit cloud modeling method under topology ambiguity
CN109255813A (en) * 2018-09-06 2019-01-22 大连理工大学 A kind of hand-held object pose real-time detection method towards man-machine collaboration
CN109785379A (en) * 2018-12-17 2019-05-21 中国科学院长春光学精密机械与物理研究所 The measurement method and measuring system of a kind of symmetric objects size and weight
CN110148169A (en) * 2019-03-19 2019-08-20 长安大学 A kind of vehicle target 3 D information obtaining method based on PTZ holder camera
CN110402399A (en) * 2017-01-03 2019-11-01 创新科技有限公司 Laser radar system and method for the object that detects and classify
CN110543850A (en) * 2019-08-30 2019-12-06 上海商汤临港智能科技有限公司 Target detection method and device and neural network training method and device

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104751146A (en) * 2015-04-13 2015-07-01 中国科学技术大学 Indoor human body detection method based on 3D (three-dimensional) point cloud image
CN110402399A (en) * 2017-01-03 2019-11-01 创新科技有限公司 Laser radar system and method for the object that detects and classify
CN107025685A (en) * 2017-04-11 2017-08-08 南京林业大学 Airborne building summit cloud modeling method under topology ambiguity
CN109255813A (en) * 2018-09-06 2019-01-22 大连理工大学 A kind of hand-held object pose real-time detection method towards man-machine collaboration
CN109785379A (en) * 2018-12-17 2019-05-21 中国科学院长春光学精密机械与物理研究所 The measurement method and measuring system of a kind of symmetric objects size and weight
CN110148169A (en) * 2019-03-19 2019-08-20 长安大学 A kind of vehicle target 3 D information obtaining method based on PTZ holder camera
CN110543850A (en) * 2019-08-30 2019-12-06 上海商汤临港智能科技有限公司 Target detection method and device and neural network training method and device

Cited By (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112001412A (en) * 2020-07-10 2020-11-27 浙江大华技术股份有限公司 Multi-view vehicle association method and related device
CN112308912A (en) * 2020-11-03 2021-02-02 长安大学 System, device and method for acquiring road surface disease homologous multi-characteristic image
CN112308912B (en) * 2020-11-03 2023-09-15 长安大学 System, device and method for obtaining homologous multi-feature image of pavement disease
CN112738393A (en) * 2020-12-25 2021-04-30 珠海西山居移动游戏科技有限公司 Focusing method and device
CN112738393B (en) * 2020-12-25 2022-08-09 珠海西山居移动游戏科技有限公司 Focusing method and device
CN113160414A (en) * 2021-01-25 2021-07-23 北京豆牛网络科技有限公司 Automatic identification method and device for remaining amount of goods, electronic equipment and computer readable medium
CN113160414B (en) * 2021-01-25 2024-06-07 北京豆牛网络科技有限公司 Automatic goods allowance recognition method, device, electronic equipment and computer readable medium
CN113034695A (en) * 2021-04-16 2021-06-25 广东工业大学 Wasserstein distance-based object envelope multi-view reconstruction and optimization method
CN113888618A (en) * 2021-09-30 2022-01-04 北京工业大学 Object single piece separation method and system
CN113888618B (en) * 2021-09-30 2024-05-10 北京工业大学 Object single-piece separation method and system
CN114998665A (en) * 2022-08-04 2022-09-02 创新奇智(广州)科技有限公司 Image category identification method and device, electronic equipment and storage medium
CN115410135A (en) * 2022-11-01 2022-11-29 中国民航大学 Autonomous-type-carried aviation luggage feature perception reconstruction method and system and application thereof

Also Published As

Publication number Publication date
CN111340873B (en) 2023-05-23

Similar Documents

Publication Publication Date Title
CN111340873B (en) Object minimum outer envelope size measuring and calculating method for multi-view image
US10528616B2 (en) Systems and methods for automatically generating metadata for media documents
CN109829398B (en) Target detection method in video based on three-dimensional convolution network
CN109903331B (en) Convolutional neural network target detection method based on RGB-D camera
US8175412B2 (en) Method and apparatus for matching portions of input images
WO2022170844A1 (en) Video annotation method, apparatus and device, and computer readable storage medium
US10558844B2 (en) Lightweight 3D vision camera with intelligent segmentation engine for machine vision and auto identification
CN112927353B (en) Three-dimensional scene reconstruction method, storage medium and terminal based on two-dimensional target detection and model alignment
CN106570480B (en) A kind of human action classification method based on gesture recognition
JP2014081347A (en) Method for recognition and pose determination of 3d object in 3d scene
CN108229416A (en) Robot SLAM methods based on semantic segmentation technology
CN109272577B (en) Kinect-based visual SLAM method
AU2011301774A1 (en) A method for enhancing depth maps
CN107358189B (en) Object detection method in indoor environment based on multi-view target extraction
WO2023024766A1 (en) Object size identification method, readable storage medium and object size identification system
CN112329662B (en) Multi-view saliency estimation method based on unsupervised learning
CN113501167A (en) Binocular vision-based small traditional Chinese medicine package positioning method
CN116958434A (en) Multi-view three-dimensional reconstruction method, measurement method and system
JP2014199559A (en) Viewpoint estimation device and sorter learning method therefor
CN106778763B (en) Image representation method based on attribute graph
Proenca et al. SHREC’15 Track: Retrieval of Oobjects captured with kinect one camera
Zhang et al. Kinect-based universal range sensor for laboratory experiments
Asif et al. Model-free segmentation and grasp selection of unknown stacked objects
Le et al. Geometry-Based 3D Object Fitting and Localizing in Grasping Aid for Visually Impaired
Zhang et al. Deep Learning Based Cross-View Human Detection System

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant
CB03 Change of inventor or designer information
CB03 Change of inventor or designer information

Inventor after: Guan Yisheng

Inventor after: Zhu Lei

Inventor after: Chen Weinan

Inventor after: He Li

Inventor before: He Li

Inventor before: Zhu Lei

Inventor before: Chen Weinan

Inventor before: Guan Yisheng