CN116255912A - Method and system for measuring static volume of package - Google Patents
Method and system for measuring static volume of package Download PDFInfo
- Publication number
- CN116255912A CN116255912A CN202211710256.2A CN202211710256A CN116255912A CN 116255912 A CN116255912 A CN 116255912A CN 202211710256 A CN202211710256 A CN 202211710256A CN 116255912 A CN116255912 A CN 116255912A
- Authority
- CN
- China
- Prior art keywords
- package
- projection
- logistics
- training
- semantic segmentation
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000003068 static effect Effects 0.000 title claims abstract description 24
- 238000000034 method Methods 0.000 title claims description 33
- 230000011218 segmentation Effects 0.000 claims abstract description 121
- 238000005259 measurement Methods 0.000 claims abstract description 33
- 238000001514 detection method Methods 0.000 claims abstract description 23
- 238000000691 measurement method Methods 0.000 claims abstract description 8
- 238000012549 training Methods 0.000 claims description 95
- 238000001914 filtration Methods 0.000 claims description 20
- 238000004891 communication Methods 0.000 claims description 13
- 238000011156 evaluation Methods 0.000 claims description 10
- 230000008859 change Effects 0.000 claims description 3
- 238000004519 manufacturing process Methods 0.000 claims description 3
- 238000012216 screening Methods 0.000 claims description 3
- 230000008569 process Effects 0.000 description 9
- 239000011159 matrix material Substances 0.000 description 4
- 230000006870 function Effects 0.000 description 3
- 238000002372 labelling Methods 0.000 description 3
- 238000012545 processing Methods 0.000 description 3
- 238000004422 calculation algorithm Methods 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- PCTMTFRHKVHKIS-BMFZQQSSSA-N (1s,3r,4e,6e,8e,10e,12e,14e,16e,18s,19r,20r,21s,25r,27r,30r,31r,33s,35r,37s,38r)-3-[(2r,3s,4s,5s,6r)-4-amino-3,5-dihydroxy-6-methyloxan-2-yl]oxy-19,25,27,30,31,33,35,37-octahydroxy-18,20,21-trimethyl-23-oxo-22,39-dioxabicyclo[33.3.1]nonatriaconta-4,6,8,10 Chemical compound C1C=C2C[C@@H](OS(O)(=O)=O)CC[C@]2(C)[C@@H]2[C@@H]1[C@@H]1CC[C@H]([C@H](C)CCCC(C)C)[C@@]1(C)CC2.O[C@H]1[C@@H](N)[C@H](O)[C@@H](C)O[C@H]1O[C@H]1/C=C/C=C/C=C/C=C/C=C/C=C/C=C/[C@H](C)[C@@H](O)[C@@H](C)[C@H](C)OC(=O)C[C@H](O)C[C@H](O)CC[C@@H](O)[C@H](O)C[C@H](O)C[C@](O)(C[C@H](O)[C@H]2C(O)=O)O[C@H]2C1 PCTMTFRHKVHKIS-BMFZQQSSSA-N 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000005094 computer simulation Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000004880 explosion Methods 0.000 description 1
- 238000009499 grossing Methods 0.000 description 1
- 230000001788 irregular Effects 0.000 description 1
- 230000035945 sensitivity Effects 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01B—MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
- G01B11/00—Measuring arrangements characterised by the use of optical techniques
- G01B11/02—Measuring arrangements characterised by the use of optical techniques for measuring length, width or thickness
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01B—MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
- G01B11/00—Measuring arrangements characterised by the use of optical techniques
- G01B11/02—Measuring arrangements characterised by the use of optical techniques for measuring length, width or thickness
- G01B11/03—Measuring arrangements characterised by the use of optical techniques for measuring length, width or thickness by measuring coordinates of points
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/40—Image enhancement or restoration using histogram techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/70—Denoising; Smoothing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/50—Depth or shape recovery
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/60—Analysis of geometric attributes
- G06T7/62—Analysis of geometric attributes of area, perimeter, diameter or volume
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/25—Determination of region of interest [ROI] or a volume of interest [VOI]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
- G06V10/267—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion by performing operations on regions, e.g. growing, shrinking or watersheds
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/70—Labelling scene content, e.g. deriving syntactic or semantic representations
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/30—Computing systems specially adapted for manufacturing
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Multimedia (AREA)
- Computational Linguistics (AREA)
- Geometry (AREA)
- Image Analysis (AREA)
Abstract
The invention relates to a static volume measurement method and a static volume measurement system for packages. It comprises the following steps: carrying out semantic segmentation on the RGB image of the logistics package by utilizing a constructed logistics package semantic segmentation model to obtain a package detection basic segmentation mask of the logistics package; converting the depth data of the package detection basic segmentation mask into 3D point cloud coordinates, and projecting each point to a calibration belt surface to obtain a projection surface segmentation mask after projection; and dividing the mask on the projection surface, determining the wrapping height of the logistics package based on the projection distance from the 3D point cloud coordinates to the calibration belt surface, and determining the wrapping length and the wrapping width of the logistics package according to the minimum circumscribed rectangle of the mask divided on the projection surface. The invention can rapidly and effectively realize the measurement of the volume of the package and improve the precision and reliability of the measurement of the volume of the package.
Description
Technical Field
The invention relates to a volume measurement method and a volume measurement system, in particular to a static volume measurement method and a static volume measurement system for packages.
Background
In recent years, with the rapid development of logistics technology, in the express sorting business, the requirement on parcel volume measurement technology is also continuously improved. In thousands of packages, sorting packages of different volumes is an important technical link, namely, the accuracy of the package volume measurement results directly influences the proceeding of the subsequent express sorting process.
When larger errors occur in the measurement of the volume of the package, the accuracy of express sorting can be greatly reduced, and therefore the efficiency of express package transportation is reduced. In addition, in logistics, the volume of packages transported sometimes needs to be counted, and more accurate measurement results are needed.
At present, in the express industry, the accuracy of package volume measurement is not high enough, and high-precision volume measurement also needs to rely on manual measurement to measure the size of the package, but manual measurement is difficult to express and corresponds to massive packages to be measured. Therefore, how to make a rapid and accurate volume measurement of a package is a challenge.
Disclosure of Invention
The invention aims to overcome the defects in the prior art and provide a static volume measurement method and a static volume measurement system for a package, which can rapidly and effectively measure the size of the package and improve the accuracy and reliability of measurement of the size of the package.
According to the technical scheme provided by the invention, the static volume measurement method of the package comprises the following steps:
providing a logistics package camera for acquiring a depth map and RGB images, wherein the logistics package camera is calibrated based on the acquired depth so as to determine a calibration belt surface for logistics package conveying after calibration;
acquiring a depth map and RGB (red, green and blue) images of a logistics parcel based on a logistics parcel camera, and performing semantic segmentation on the RGB images of the logistics parcel by using a constructed logistics parcel semantic segmentation model so as to extract a parcel detection basic segmentation mask of the logistics parcel;
converting the depth data of the package detection basic segmentation mask into 3D point cloud coordinates, and projecting each point to a calibration belt surface to obtain a projection surface segmentation mask after projection;
dividing a mask on a projection surface, determining the wrapping height of the logistics package based on the projection distance from the 3D point cloud coordinates to a calibration belt surface, and determining the wrapping length and the wrapping width of the logistics package according to the minimum circumscribed rectangle of the mask divided on the projection surface;
the static volume of the flow wrap is determined based on the wrap height, the wrap length, and the wrap width determined above.
The constructed logistics package semantic segmentation model comprises the following steps:
constructing a logistics package semantic segmentation basic model, wherein the logistics package semantic segmentation basic model is based on a network frame of deep Labv3+, and a network backbone part of the network frame adopts an Xreception model;
training the constructed logistics package semantic segmentation basic model, wherein the training of the logistics package semantic segmentation basic model comprises a pre-training stage and a semantic segmentation training stage which are sequentially carried out;
training the logistics package semantic segmentation basic model by adopting an ImageNet-1k data set in a pre-training stage so as to obtain pre-training weight parameters of the logistics package semantic segmentation basic model after the pre-training stage is finished;
in the semantic segmentation training stage, acquiring a package RGB training image for manufacturing a training sample by using a logistics package camera, marking the package RGB training image and selecting an ROI (region of interest) to generate a semantic segmentation training data set;
based on the generated semantic segmentation training data set, carrying out semantic segmentation training on the logistics package semantic segmentation basic model after the pre-training stage so as to generate a logistics package semantic segmentation model by the logistics package semantic segmentation basic model after reaching a target training state.
In the pre-training stage, initializing the weight by adopting a Kaiming method, and pre-training by adopting a mode of fixed iteration times;
the learning rate attenuation strategy is sectional attenuation, and the weight with the highest average accuracy is selected as a pre-training weight parameter.
In the semantic segmentation training stage, the learning rate attenuation strategy adopts multi-step long attenuation, and the iteration number of the network training is not less than 3000;
the evaluation index is evaluated by using the average cross ratio and the prediction accuracy, the evaluation index is calculated every 50 generations, and the change curve of each evaluation index data is recorded.
When the projection screen segmentation mask is obtained based on the package detection basic segmentation mask, the method comprises the following steps:
converting depth data corresponding to the package detection basic segmentation mask into corresponding 3D point cloud coordinates, and projecting each point cloud to a calibration belt surface to determine a projection distance and the 3D point cloud coordinates of a projection point after projection;
and converting the 3D point cloud coordinates of the projection points into 2D coordinates on the calibration belt surface based on the depth camera internal parameters of the logistics package camera, and obtaining the projection surface segmentation mask based on the converted 2D coordinates on the calibration belt surface.
When the projection distance is determined by projecting the point cloud to the calibration belt surface, the method comprises the following steps:
configuring a projection distance threshold, and screening the projection distance by utilizing the configured projection distance threshold to delete the projection distance less than the configured projection distance threshold;
and filtering all the projection points by using a point cloud radius filter to delete outliers around the current projection point.
And after converting the 3D point cloud coordinates of the projection points into 2D coordinates of the calibration belt surface, deleting the projection points positioned outside the depth map range.
Configuring a height filtering threshold value to filter the height of the projection point based on the configured height filtering threshold value;
after the heights of the projection points are filtered, carrying out histogram statistics on the heights of all the remaining projection points, accumulating the number of each height in the histogram from small to large according to the height value, and configuring the corresponding height as the package height of the logistics package when the accumulated height number is larger than a preset total proportion threshold value of the total number of all the heights.
Dividing a mask for a projection plane, and determining a maximum area communication area of the mask divided by the projection screen and contour points of the maximum area communication area;
determining the vertex of the minimum bounding rectangle of the maximum area communication area based on the contour point of the maximum area communication area, converting the vertex of the minimum bounding rectangle into 3D point cloud coordinates, and determining the Euclidean distance between adjacent vertexes, wherein,
among the determined Euclidean distances, the configuration with the small distance is the width of the logistics package, and the configuration with the large distance is the length of the logistics package.
A static volumetric measurement system for a package, comprising a volumetric measurement controller, wherein,
and measuring the volume of the logistics package by adopting the method for the depth map and the RGB image of any logistics package.
The invention has the advantages that: acquiring a depth map and RGB (red, green and blue) images of a logistics parcel based on a logistics parcel camera, and performing semantic segmentation on the RGB images of the logistics parcel by using a constructed logistics parcel semantic segmentation model so as to extract a parcel detection basic segmentation mask of the logistics parcel; converting the depth data of the package detection basic segmentation mask into 3D point cloud coordinates, and projecting each point to a calibration belt surface to obtain a projection surface segmentation mask after projection; dividing a mask on a projection surface, determining the wrapping height of the logistics package based on the projection distance from the 3D point cloud coordinates to a calibration belt surface, and determining the wrapping length and the wrapping width of the logistics package according to the minimum circumscribed rectangle of the mask divided on the projection surface; the measurement of the package size can be realized rapidly and effectively, and the accuracy and the reliability of the package size measurement are improved.
Drawings
FIG. 1 is a flow chart of an embodiment of the present invention.
FIG. 2 is a schematic diagram of an embodiment of the 3D point cloud projection of the present invention.
Detailed Description
The invention will be further described with reference to the following specific drawings and examples.
In order to quickly and effectively realize measurement of the volume of the package and improve the accuracy and reliability of measurement of the volume of the package, according to one embodiment of the invention, the volume measurement method comprises the following steps:
providing a logistics package camera for acquiring a depth map and RGB images, wherein the logistics package camera is calibrated based on the acquired depth so as to determine a calibration belt surface for logistics package conveying after calibration;
acquiring a depth map and RGB (red, green and blue) images of a logistics parcel based on a logistics parcel camera, and performing semantic segmentation on the RGB images of the logistics parcel by using a constructed logistics parcel semantic segmentation model so as to extract a parcel detection basic segmentation mask of the logistics parcel;
converting the depth data of the package detection basic segmentation mask into 3D point cloud coordinates, and projecting each point to a calibration belt surface to obtain a projection surface segmentation mask after projection;
dividing a mask on a projection surface, determining the wrapping height of the logistics package based on the projection distance from the 3D point cloud coordinates to a calibration belt surface, and determining the wrapping length and the wrapping width of the logistics package according to the minimum circumscribed rectangle of the mask divided on the projection surface;
the static volume of the flow wrap is determined based on the wrap height, the wrap length, and the wrap width determined above.
Generally, conveying of logistics packages is achieved through a belt, in the process of conveying the logistics packages through the belt, images of conveying the logistics packages are obtained through the belt by means of a logistics package camera, wherein the images obtained through the logistics package camera comprise a depth map and RGB images, the logistics package camera can adopt the existing commonly used industrial camera mode, and the fact that the depth map and the RGB images of the logistics packages can be obtained is the right.
When a depth map and an RGB image are acquired by using a logistics package camera, the logistics package camera needs to be calibrated, and after calibration, a belt surface in the RGB image can be determined. For logistics package camera calibration, the conventional form can be adopted, and a calibration embodiment is given below.
When the method is used for calibrating, for a region of interest, depth data (the depth data specifically refers to the depth value of each pixel in the depth map) in the depth map of the region of interest are obtained, the depth data are converted into 3D point cloud coordinates by using a depth camera reference matrix of a logistics package camera, the depth camera reference matrix of the logistics package camera is a characteristic parameter of the logistics package camera, and the mode and the process for converting the depth data of the region of interest into the 3D point cloud coordinates based on the depth camera reference matrix can be consistent with the prior art, and specifically, the depth data can be converted into the 3D point cloud coordinates.
And carrying out plane fitting on the converted 3D point cloud coordinates by utilizing a random consistency algorithm (RANSAC, random Sample Consensus), wherein when carrying out plane fitting by utilizing the random consistency algorithm, three points are randomly selected in the 3D point cloud coordinates at first, so that a plane equation can be calculated by utilizing the selected three points by utilizing the technical means commonly used in the technical field. After the plane equation is calculated, the distances from all the remaining points to the plane are counted, and a distance fitting threshold is set to divide the other 3D point clouds into in-plane points and out-of-plane points.
The distance fitting threshold can be generally selected according to the distribution condition of the point cloud of the belt surface, if the fluctuation of the point cloud of the belt surface is small, the distance threshold is correspondingly set to be a small point, for example, when the distance fitting threshold is set to 1500, namely, the point with the distance to the plane being smaller than 1500 is regarded as an in-plane sample point, otherwise, the point is regarded as an out-of-model point.
In specific implementation, the calibration process of the belt surface fitting can be completed by using a computer model function in a PCL library, and finally, the optimal plane parameters meeting the requirements are selected, so that the calibrated belt surface is obtained.
After the logistics package camera is calibrated, when the logistics package is normally conveyed by the belt, a depth map and RGB images of the region of interest can be realized, and the region of interest is generally the region utilized in the calibration. After the RGB image of the logistic package is obtained, the constructed logistic package semantic segmentation model is utilized to carry out semantic segmentation on the RGB image of the logistic package, and when the logistic package exists in the RGB image, a package detection basic segmentation mask of the logistic package can be extracted and obtained.
When the logistics package is subjected to semantic segmentation, the conventional common target detection mode can be adopted, and generally, the position of the logistics package and the area of the position of the logistics package can be obtained when the logistics package is subjected to semantic segmentation. One embodiment of semantic segmentation target detection for deep labv3+ based network frameworks is given below. The logistics package semantic segmentation model for the network framework based on deep Labv3+ comprises the following steps:
the constructed logistics package semantic segmentation model comprises the following steps:
constructing a logistics package semantic segmentation basic model, wherein the logistics package semantic segmentation basic model is based on a network frame of deep Labv3+, and a network backbone part of the network frame adopts an Xreception model;
training the constructed logistics package semantic segmentation basic model, wherein the training of the logistics package semantic segmentation basic model comprises a pre-training stage and a semantic segmentation training stage which are sequentially carried out;
training the logistics package semantic segmentation basic model by adopting an ImageNet-1k data set in a pre-training stage so as to obtain pre-training weight parameters of the logistics package semantic segmentation basic model after the pre-training stage is finished;
in the semantic segmentation training stage, acquiring a package RGB training image for manufacturing a training sample by using a logistics package camera, marking the package RGB training image and selecting an ROI (region of interest) to generate a semantic segmentation training data set;
based on the generated semantic segmentation training data set, carrying out semantic segmentation training on the logistics package semantic segmentation basic model after the pre-training stage so as to generate a logistics package semantic segmentation model by the logistics package semantic segmentation basic model after reaching a target training state.
In the implementation, the logistics package semantic segmentation model based on the deep Labv3+ network frame and the Xnaption model is adopted for the network trunk part, and due to the fact that cavity convolution is introduced, the loss of information can be reduced while the receptive field is increased. In order to obtain a final logistics package semantic segmentation model, a pre-training stage and a semantic segmentation training stage are needed to be sequentially carried out on a logistics package semantic segmentation basic model.
Corresponding training conditions are required to be configured in the pre-training stage and the semantic segmentation training stage, and the training conditions configured in the pre-training stage and the semantic segmentation training stage are exemplified below. Specifically, in the pre-training stage, using an ImageNet-1k data set for training to obtain, in order to improve convergence speed and prevent gradient explosion, initializing weights by using a Kaimag method, training a logistics package semantic segmentation basic model by using 400000 times of fixed iteration times, and selecting the weight with the highest average accuracy as a pre-training weight parameter; the initial learning rate is set to 0.001 and the learning rate decay strategy is a segmented decay.
Specifically, the ImageNet-1k dataset is an existing common public dataset, after the training conditions of the pre-training stage are configured, the pre-training of the logistics package semantic segmentation basic model can be realized, and the specific training method and process based on the configured training conditions can be consistent with the existing one, and are not repeated here.
In the semantic segmentation training stage, the weight of the logistic package semantic segmentation basic model uses a pre-training weight parameter; the training initial learning rate is set to be 0.05, and the learning rate attenuation strategy adopts multi-step long attenuation. The iteration number of training is not less than 3000, the evaluation index is evaluated by using the average cross ratio and the prediction accuracy, the evaluation index is calculated every 50 generations, and the change curve of each evaluation index data is recorded. And adjusting training parameters according to the results of the evaluation indexes to perform multiple training, and selecting the weight with the optimal accuracy as the final weight parameter of the logistic package semantic segmentation basic model, thereby obtaining the logistic package semantic segmentation model.
In the specific implementation, the learning rate attenuation strategy and the multi-step long attenuation strategy adopting the sectional attenuation are consistent with the prior art, generally, the configured training conditions are all common in the technical field, the training mode and the training process according to the configured training conditions can be consistent with the prior art, and the training can be realized to obtain the logistics package semantic segmentation model.
The training data set comprises a plurality of training samples for the semantic segmentation training stage, and the training samples in the training data set can be manufactured by the following modes: sequentially placing packages with different sizes on a belt surface, collecting a package depth map and RGB images through a logistics package camera arranged above the belt, and respectively placing the collected depth map and RGB images under different folders; then, carrying out package labeling on the RGB images of the acquired package through a data set labeling tool Labelme, and setting gray values of a package area and a background area to 255 and 0 respectively in a coding mode so as to obtain a package standard segmentation mask (namely the actual position of the logistic package in each RGB image); all the data sets after labeling are randomly divided into training sets and verification sets according to the required proportion.
The method comprises the steps that an interested area is manually selected for an RGB image of a package, wherein the interested area is mainly an area on a belt surface where the package possibly appears, and interference of other areas on package processing can be avoided by pre-selecting the interested area; and solving the maximum circumscribed rectangle of the region of interest by utilizing findContours and minArearact functions in Opencv, and acquiring RGB images of the corresponding region according to the solved maximum circumscribed rectangle to serve as input images of the semantic segmentation model. Specifically, the region of interest is consistent with the region of interest selected during the calibration, the region of interest refers to a region where the package appears on the belt surface, when the belt is used for conveying the logistics package, the package of the region of interest is the package region when the logistics package appears in the region of interest, and other regions are regarded as background regions.
When the method is used, after the RGB image of the region of interest is obtained by using the logistics parcel camera, semantic segmentation is carried out on the RGB image by using the logistics parcel semantic segmentation model, so that a parcel detection basic segmentation mask of logistics parcel can be obtained. The package detection basic segmentation model of the logistics package is the position of the logistics package on the belt surface and the area of the position on the belt.
In one embodiment of the present invention, when obtaining a projection screen division mask based on a package detection basic division mask, the method includes:
converting depth data corresponding to the package detection basic segmentation mask into corresponding 3D point cloud coordinates, and projecting each point cloud to a calibration belt surface to determine a projection distance and the 3D point cloud coordinates of a projection point after projection;
and converting the 3D point cloud coordinates of the projection points into 2D coordinates on the calibration belt surface based on the depth camera internal parameters of the logistics package camera, and obtaining the projection surface segmentation mask based on the converted 2D coordinates on the calibration belt surface.
Specifically, when the point cloud is projected to the calibration belt surface to determine the projection distance, the method comprises the following steps:
configuring a projection distance threshold, and screening the projection distance by utilizing the configured projection distance threshold to delete the projection distance less than the configured projection distance threshold;
and filtering all the projection points by using a point cloud radius filter to delete outliers around the current projection point.
In implementation, as shown in fig. 2, the distance from the 3D point cloud coordinate to the calibrated belt surface is calculated, the distance is determined, and the projection distance threshold is selected according to the height range of the measured package, for example, 1.2 times the maximum height of the package to be measured is selected as the projection distance threshold. And deleting the point cloud coordinates smaller than the projection distance threshold value, and reserving points with the projection distance larger than the projection distance threshold value.
In fig. 2, ABCD refers to the top side of the logistic package and a 'B' C 'D' refers to the bottom side of the logistic package. h refers to the height difference between the top surface of the package and the bottom surface of the logistics package, namely the height of the logistics package. The belt plane in fig. 2 is the calibration belt plane, and since the logistic package is placed on the belt plane, the bottom surface a 'B' C 'D' of the logistic package coincides with the calibration belt plane. In making a static volume measurement of a package, it is necessary to determine the height h, which can be determined by the following description.
For 2D coordinates, which are generally coordinates under a pixel coordinate system, the pixel coordinate system can be constructed in a conventional common mode, and because the 2D coordinates and the 3D point cloud coordinates have a one-to-one correspondence, the two coordinates can be directly converted based on the depth camera internal parameters of the logistics package camera, and the specific conversion mode and the process can be consistent with the conventional mode.
Filtering all the projection points by using a point cloud radius filter, setting a radius threshold and a number of adjacent points threshold for each point, wherein the radius threshold can be set as the maximum adjacent point distance from each point to be the center, and the adjacent points exceeding the distance can not be regarded as the projection points. The threshold number of neighboring points is configured to be the minimum number of neighboring points per point, and a center point less than this number will be considered an outlier.
After the radius threshold value and the number threshold value of the adjacent points are set, counting the number of the adjacent points of the projection points in the radius threshold range by taking each point as a central point, and judging the central point with the number of the adjacent points smaller than the number threshold value of the adjacent points as an outlier and deleting the outlier.
After the initial deletion of outliers, the radius threshold and the number of neighboring points threshold are then adjusted, followed by a further radius filtering. The outliers around the projection points can be obviously reduced through twice radius filtering;
in particular, in order to reduce the sensitivity of the second-time radius filtering, the radius threshold of the second-time point cloud radius filter and the number threshold of the adjacent points are smaller than the radius threshold of the first-time point cloud radius filter and the number threshold of the adjacent points.
Further, after converting the 3D point cloud coordinates of the projection points into 2D coordinates of the calibration belt surface, deleting the projection points positioned outside the depth map range.
Specifically, the 2D coordinates of the projection plane are obtained by converting 3D point cloud coordinates of the projection points, and meanwhile, whether the horizontal coordinate value and the vertical coordinate value of the 2D coordinates are within the dimension range of the depth map is determined, that is, whether the 2D coordinate value has a negative value or is larger than the maximum value of the length and the width of the depth map is determined, the projection points within the dimension range of the depth map are reserved only, and the pixel positions of the projection points in the depth map are obtained according to the inverse operation from the depth values to the calculation formula of the projection points, so that the segmentation mask for wrapping the depth map is further obtained.
And setting 0.8 times of the minimum package area as a package area threshold, filtering out a connected domain of which the area is smaller than the package area threshold in the depth map segmentation mask, and obtaining corresponding coordinates of the connected domain in the projection plane according to the position of the connected domain which accords with the area threshold, thereby obtaining the package projection plane segmentation mask.
In practice, the type of package to be transported is generally stable for a belt transporting the logistic package, i.e., the minimum package area can be determined.
In one embodiment of the invention, a height filtering threshold is configured to filter the height of the projection point based on the configured height filtering threshold;
after the heights of the projection points are filtered, carrying out histogram statistics on the heights of all the remaining projection points, accumulating the number of each height in the histogram from small to large according to the height value, and configuring the corresponding height as the package height of the logistics package when the accumulated height number is larger than a preset total proportion threshold value of the total number of all the heights.
Specifically, the configuration height filtering threshold includes a minimum parcel height threshold and a maximum parcel height threshold, where the minimum parcel height threshold may be generally 0.5 times the minimum height of the parcel to be tested, and the maximum parcel height threshold may be 1.2 times the maximum height of the parcel to be tested.
And filtering out projection points of which the heights of all positions in the projection surface segmentation mask are smaller than a minimum parcel height threshold value and larger than a maximum parcel height threshold value.
And counting the left projection points in the height histogram, accumulating the number of each height in the histogram from small to large according to the height value, wherein the height when the accumulated height number is greater than 90% of the total height is used as the package height, namely, at the moment, the preset total ratio threshold is 90%, and of course, the preset total ratio threshold can be other conditions, and can be specifically selected according to actual needs so as to meet the requirement of taking the package height as a criterion.
From the above description, it will be appreciated that the type of logistic parcel is determined, and therefore the height range of the logistic parcel is generally known, and therefore, the minimum parcel height threshold and the maximum parcel height threshold can be determined.
In one embodiment of the invention, a mask is segmented on a projection plane, and a maximum area communication area of the mask and contour points of the maximum area communication area are determined;
determining the vertex of the minimum bounding rectangle of the maximum area communication area based on the contour point of the maximum area communication area, converting the vertex of the minimum bounding rectangle into 3D point cloud coordinates, and determining the Euclidean distance between adjacent vertexes, wherein,
among the determined Euclidean distances, the configuration with the small distance is the width of the logistics package, and the configuration with the large distance is the length of the logistics package.
Specifically, a mask is divided for a projection plane, after a communication area with the largest area is obtained, a findContours function in Opencv is used for obtaining contour points of the largest communication area, and then downsampling and Gaussian filtering are respectively used for processing the contour points, so that the effects of further reducing interference of protruding points around the contour and smoothing the contour can be achieved; therefore, the mask for dividing the projection surface can be used for determining the communication area with the largest area by adopting the technical means commonly used in the technical field.
In the specific implementation, through the Gaussian filtering and the filtering of the point cloud radius filter, the damage treatment of the wrapping edge can be realized, and the shape of the wrapped after the treatment is approximately regarded as a cuboid.
The projection segmentation mask profile obtains the minimum circumscribed rectangular vertex, converts each vertex into a 3D point cloud coordinate according to the depth camera internal reference matrix, and obtains the Euclidean distance between the adjacent vertices by using a two-dimensional form, wherein the smaller distance is the width of the logistics package, and the larger distance is the length of the logistics package.
In the specific implementation, the determination of the minimum circumscribed rectangular vertex of the projection surface segmentation mask outline can be realized by adopting the conventional common technical means. The specific manner and process of determining the Euclidean distance between adjacent vertices using the two-dimensional form may be consistent with existing techniques, particularly in terms of being able to determine the desired Euclidean distance.
In conclusion, the static size measurement of the logistics package can be realized, namely the package height, the package length and the package width of the logistics package can be obtained, and the volume measurement of the package can be realized according to the size of the logistics package.
During the implementation, most of the measured logistics packages are regular cuboid packages, and the volume of the logistics packages can be determined according to the volume calculation mode of the cuboid only by determining the package height, the package length and the package width, so that the size of the space occupied by the logistics packages can be determined. Furthermore, the volume measurement here may not be considered for irregular shaped objects.
In summary, a static volumetric measurement system of packages is available, and in one embodiment of the invention, includes a volumetric measurement controller, wherein,
and measuring the volume of the logistics package by adopting the method for the depth map and the RGB image of any logistics package.
Specifically, the volume measurement controller may use the conventional computer equipment, that is, the computer equipment is used to perform the above calibration and construct the semantic segmentation model of the logistic package, and the specific manner and process for measuring the volume of the logistic package may refer to the above description and will not be repeated here.
From the above description, it can be seen that the volumetric controller can be divided into the following parts according to the operation of the volumetric controller:
the plane calibration module is used for demarcating a region of interest of the plane of the belt and demarcating to obtain a calibrated belt surface;
the semantic segmentation module is used for constructing a logistics package semantic segmentation model and carrying out semantic segmentation on the logistics package to be measured so as to obtain a package detection basic segmentation mask;
the 3D point cloud data processing module is used for converting the depth map corresponding to the basic segmentation mask into 3D point cloud coordinates through the package detection; the projection distance and the projection point coordinates are obtained by projecting the point cloud coordinates to the belt surface; and filtering the projection point coordinates through a radius filter to obtain projection point coordinates with outliers removed.
The package static volume measuring module is used for obtaining the package height through the projection distance; obtaining the minimum circumscribed rectangle of the mask contour by projection segmentation and obtaining the length and width of the package; and calculating the static volume of the package by using a volume formula according to the length, width and height.
The present invention is not limited to the above embodiments, but is capable of numerous modifications, equivalents and modifications within the scope of the present invention as described in the foregoing detailed description can be made by those skilled in the art without departing from the scope of the present invention.
Claims (10)
1. A method of static volumetric measurement of a package, the volumetric measurement comprising:
providing a logistics package camera for acquiring a depth map and RGB images, wherein the logistics package camera is calibrated based on the acquired depth so as to determine a calibration belt surface for logistics package conveying after calibration;
acquiring a depth map and RGB (red, green and blue) images of a logistics parcel based on a logistics parcel camera, and performing semantic segmentation on the RGB images of the logistics parcel by using a constructed logistics parcel semantic segmentation model so as to extract a parcel detection basic segmentation mask of the logistics parcel;
converting the depth data of the package detection basic segmentation mask into 3D point cloud coordinates, and projecting each point to a calibration belt surface to obtain a projection surface segmentation mask after projection;
dividing a mask on a projection surface, determining the wrapping height of the logistics package based on the projection distance from the 3D point cloud coordinates to a calibration belt surface, and determining the wrapping length and the wrapping width of the logistics package according to the minimum circumscribed rectangle of the mask divided on the projection surface;
the static volume of the flow wrap is determined based on the wrap height, the wrap length, and the wrap width determined above.
2. The static volumetric measurement method of parcels according to claim 1, wherein the semantic segmentation model of the constructed logistic parcel comprises:
constructing a logistics package semantic segmentation basic model, wherein the logistics package semantic segmentation basic model is based on a network frame of deep Labv3+, and a network backbone part of the network frame adopts an Xreception model;
training the constructed logistics package semantic segmentation basic model, wherein the training of the logistics package semantic segmentation basic model comprises a pre-training stage and a semantic segmentation training stage which are sequentially carried out;
training the logistics package semantic segmentation basic model by adopting an ImageNet-1k data set in a pre-training stage so as to obtain pre-training weight parameters of the logistics package semantic segmentation basic model after the pre-training stage is finished;
in the semantic segmentation training stage, acquiring a package RGB training image for manufacturing a training sample by using a logistics package camera, marking the package RGB training image and selecting an ROI (region of interest) to generate a semantic segmentation training data set;
based on the generated semantic segmentation training data set, carrying out semantic segmentation training on the logistics package semantic segmentation basic model after the pre-training stage so as to generate a logistics package semantic segmentation model by the logistics package semantic segmentation basic model after reaching a target training state.
3. The method for measuring static volume of package according to claim 2, wherein in the pre-training stage, the weights are initialized by using a Kaiming method, and the pre-training is performed by using a fixed iteration number;
the learning rate attenuation strategy is sectional attenuation, and the weight with the highest average accuracy is selected as a pre-training weight parameter.
4. The method for measuring the static volume of the package according to claim 2, wherein in the semantic segmentation training stage, a learning rate attenuation strategy adopts multi-step long attenuation, and the iteration number of network training is not less than 3000;
the evaluation index is evaluated by using the average cross ratio and the prediction accuracy, the evaluation index is calculated every 50 generations, and the change curve of each evaluation index data is recorded.
5. The method of claim 1 to 4, wherein the step of obtaining a projection screen division mask based on the package detection basic division mask comprises:
converting depth data corresponding to the package detection basic segmentation mask into corresponding 3D point cloud coordinates, and projecting each point cloud to a calibration belt surface to determine a projection distance and the 3D point cloud coordinates of a projection point after projection;
and converting the 3D point cloud coordinates of the projection points into 2D coordinates on the calibration belt surface based on the depth camera internal parameters of the logistics package camera, and obtaining the projection surface segmentation mask based on the converted 2D coordinates on the calibration belt surface.
6. The method of claim 5, wherein projecting the point cloud onto the calibration belt surface to determine the projection distance comprises:
configuring a projection distance threshold, and screening the projection distance by utilizing the configured projection distance threshold to delete the projection distance less than the configured projection distance threshold;
and filtering all the projection points by using a point cloud radius filter to delete outliers around the current projection point.
7. The method of claim 5, wherein the projection points are deleted after converting the 3D point cloud coordinates of the projection points into 2D coordinates of the calibration belt surface.
8. The method of claim 1 to 4, wherein a height filtering threshold is configured to filter the height of the projection point based on the configured height filtering threshold;
after the heights of the projection points are filtered, carrying out histogram statistics on the heights of all the remaining projection points, accumulating the number of each height in the histogram from small to large according to the height value, and configuring the corresponding height as the package height of the logistics package when the accumulated height number is larger than a preset total proportion threshold value of the total number of all the heights.
9. The wrapped static volumetric measurement method according to any of claims 1-4, wherein a mask is segmented for a projection plane, and a maximum area connected region of the mask is determined for the mask segmentation, and contour points of the maximum area connected region are determined;
determining the vertex of the minimum bounding rectangle of the maximum area communication area based on the contour point of the maximum area communication area, converting the vertex of the minimum bounding rectangle into 3D point cloud coordinates, and determining the Euclidean distance between adjacent vertexes, wherein,
among the determined Euclidean distances, the configuration with the small distance is the width of the logistics package, and the configuration with the large distance is the length of the logistics package.
10. A static volumetric measurement system for a package, comprising a volumetric measurement controller, wherein,
the depth map and RGB image of any one of the logistic packages, the logistic package volume is measured by the method of any one of the preceding claims 1 to 9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211710256.2A CN116255912A (en) | 2022-12-29 | 2022-12-29 | Method and system for measuring static volume of package |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211710256.2A CN116255912A (en) | 2022-12-29 | 2022-12-29 | Method and system for measuring static volume of package |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116255912A true CN116255912A (en) | 2023-06-13 |
Family
ID=86683481
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211710256.2A Pending CN116255912A (en) | 2022-12-29 | 2022-12-29 | Method and system for measuring static volume of package |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116255912A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117612061A (en) * | 2023-11-09 | 2024-02-27 | 中科微至科技股份有限公司 | Visual detection method for package stacking state for stacking separation |
| CN117765065A (en) * | 2023-11-28 | 2024-03-26 | 中科微至科技股份有限公司 | Target detection-based single-piece separated package rapid positioning method |
| CN117830389A (en) * | 2023-11-28 | 2024-04-05 | 中科微至科技股份有限公司 | RGBD camera-based real-time position detection method for single-piece separation package |
-
2022
- 2022-12-29 CN CN202211710256.2A patent/CN116255912A/en active Pending
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117612061A (en) * | 2023-11-09 | 2024-02-27 | 中科微至科技股份有限公司 | Visual detection method for package stacking state for stacking separation |
| CN117765065A (en) * | 2023-11-28 | 2024-03-26 | 中科微至科技股份有限公司 | Target detection-based single-piece separated package rapid positioning method |
| CN117830389A (en) * | 2023-11-28 | 2024-04-05 | 中科微至科技股份有限公司 | RGBD camera-based real-time position detection method for single-piece separation package |
| CN117765065B (en) * | 2023-11-28 | 2024-06-04 | 中科微至科技股份有限公司 | Target detection-based single-piece separated package rapid positioning method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN116255912A (en) | Method and system for measuring static volume of package | |
| CN111027547B (en) | Automatic detection method for multi-scale polymorphic target in two-dimensional image | |
| CN115082467B (en) | Building material welding surface defect detection method based on computer vision | |
| US10115035B2 (en) | Vision system and analytical method for planar surface segmentation | |
| US8462992B2 (en) | Method of change detection for building models | |
| US8254723B2 (en) | System and method for extracting boundary elements of an object | |
| CN115100221B (en) | Glass defect segmentation method | |
| CN110286124A (en) | Refractory brick measuring system based on machine vision | |
| CN112254635B (en) | Volume measurement method, device and system | |
| CN112330593A (en) | Building surface crack detection method based on deep learning network | |
| CN112651968A (en) | Wood board deformation and pit detection method based on depth information | |
| CN113362385A (en) | Cargo volume measuring method and device based on depth image | |
| CN110399820B (en) | Visual recognition analysis method for roadside scene of highway | |
| CN115546202B (en) | Tray detection and positioning method for unmanned forklift | |
| CN101114337A (en) | Ground buildings recognition positioning method | |
| CN114396875A (en) | Rectangular parcel volume measurement method based on vertical shooting of depth camera | |
| CN115170570A (en) | Fabric fuzzing and pilling detection method based on gray level run-length matrix | |
| CN111008642A (en) | High-resolution remote sensing image classification method and system based on convolutional neural network | |
| CN111814739A (en) | Method, device and equipment for detecting express package volume and storage medium | |
| CN115046653A (en) | Cross-regional stored grain temperature variation detection method based on temperature field cloud picture | |
| CN107256421B (en) | Method for rapidly counting rice and wheat seeds | |
| CN117635615B (en) | Defect detection method and system for realizing punching die based on deep learning | |
| CN116309817A (en) | Tray detection and positioning method based on RGB-D camera | |
| CN114140485A (en) | Method and system for generating cutting track of main root of panax notoginseng | |
| CN114202548A (en) | Forklift pallet positioning method and device, storage medium and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |