CN116883998B - Article labeling method and device based on millimeter wave image and electronic equipment - Google Patents
Article labeling method and device based on millimeter wave image and electronic equipment Download PDFInfo
- Publication number
- CN116883998B CN116883998B CN202310737145.9A CN202310737145A CN116883998B CN 116883998 B CN116883998 B CN 116883998B CN 202310737145 A CN202310737145 A CN 202310737145A CN 116883998 B CN116883998 B CN 116883998B
- Authority
- CN
- China
- Prior art keywords
- pixel
- value
- color
- dimensional
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000002372 labelling Methods 0.000 title claims abstract description 47
- 238000010606 normalization Methods 0.000 claims abstract description 103
- 238000013507 mapping Methods 0.000 claims abstract description 90
- 239000011159 matrix material Substances 0.000 claims abstract description 35
- 238000000034 method Methods 0.000 claims abstract description 22
- 238000005070 sampling Methods 0.000 claims abstract description 18
- 238000012545 processing Methods 0.000 claims abstract description 11
- 238000001514 detection method Methods 0.000 claims description 11
- 238000004891 communication Methods 0.000 claims description 10
- 239000003086 colorant Substances 0.000 abstract description 6
- 230000000877 morphologic effect Effects 0.000 abstract description 4
- 238000005516 engineering process Methods 0.000 description 9
- 230000008569 process Effects 0.000 description 7
- 238000007689 inspection Methods 0.000 description 6
- 238000004364 calculation method Methods 0.000 description 3
- 238000003384 imaging method Methods 0.000 description 3
- 238000004590 computer program Methods 0.000 description 2
- 238000013480 data collection Methods 0.000 description 2
- 238000013135 deep learning Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 230000018109 developmental process Effects 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000006467 substitution reaction Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 238000012512 characterization method Methods 0.000 description 1
- 238000004040 coloring Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000009434 installation Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000007781 pre-processing Methods 0.000 description 1
- 230000011218 segmentation Effects 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000012549 training Methods 0.000 description 1
- 230000007723 transport mechanism Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/70—Labelling scene content, e.g. deriving syntactic or semantic representations
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/56—Extraction of image or video features relating to colour
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/7715—Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/30—Computing systems specially adapted for manufacturing
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Computing Systems (AREA)
- Databases & Information Systems (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Image Processing (AREA)
Abstract
The invention provides an article labeling method, device and electronic equipment based on millimeter wave images, wherein the method comprises the following steps: acquiring pixel three-dimensional data of three-dimensional pixel points of a three-dimensional millimeter wave sample image; carrying out logarithmic mapping and normalization processing on the pixel three-dimensional data, and then carrying out maximum projection in the depth direction to obtain two-dimensional projection data, a pixel normalization gray value and a pixel normalization index value; sampling the color matrix in the depth direction to obtain a depth color index table; determining a pixel color value by combining the pixel normalized gray value based on the depth color value matched by the pixel normalized index value in the depth color index table; and obtaining an article labeling result based on the pixel color value and the color two-dimensional millimeter wave image obtained by the two-dimensional projection data. According to the embodiment of the invention, the image is subjected to color mapping through the depth information of the pixels, the three-dimensional millimeter wave image can be compressed into a two-dimensional color image, the morphological characteristics and the boundaries of the object are highlighted by utilizing the colors of the image, and the labeling efficiency is improved.
Description
Technical Field
The invention relates to the technical field of millimeter wave image detection, in particular to an article labeling method, an article labeling device and electronic equipment based on millimeter wave images.
Background
Along with the development of science and technology, the application of human body security inspection equipment based on millimeter wave imaging technology in the public security field is in the vigorous development period. The application of the deep learning technology improves the recognition efficiency of the human body security inspection products to a great extent. However, the deep learning technology requires massive image samples to perform model training, and millimeter wave images generated by millimeter wave human body security inspection equipment are poor in resolution and easy to be interfered by surrounding environments, and meanwhile, clothes and carried articles worn by inspected personnel have diversity and complexity, so that people with abundant experience are required to label the image samples, the consumed labor cost is high, and the efficiency is low.
In order to improve the efficiency of image labeling, related technologies process three-dimensional millimeter wave image data into two-dimensional plane images, extract object images through a target detection technology, and identify object categories through an image identification technology. However, the two-dimensional plane image in the related art is usually a gray level image, and because of the diversity and complexity of the content of the millimeter wave image, three-dimensional pixel points of human body features and object features of the gray level image are easy to fuse with each other, so that the object boundary is blurred, and the labeling efficiency and accuracy are affected.
Disclosure of Invention
The present invention aims to solve at least one of the technical problems existing in the prior art. Therefore, the invention provides an article marking method, device and electronic equipment based on millimeter wave images, which can realize coloring of three-dimensional pixel points of human bodies and articles, highlight morphological characteristics and boundaries of the articles and improve marking efficiency.
In a first aspect, an embodiment of the present invention provides an article labeling method based on millimeter wave images, including:
acquiring a three-dimensional millimeter wave sample image, and determining pixel three-dimensional data of each three-dimensional pixel point of the three-dimensional millimeter wave sample image;
carrying out logarithmic mapping and normalization processing on the pixel three-dimensional data to obtain pixel normalized data;
performing maximum projection in the depth direction according to the pixel normalization data to obtain two-dimensional projection data of the three-dimensional pixel point, a pixel normalization gray value and a pixel normalization index value, wherein the pixel normalization gray value is a gray value of a position corresponding to the maximum depth value, and the pixel normalization index value is the maximum depth value;
up-sampling the preset color matrix in the depth direction to obtain a depth color index table;
matching a depth color value in the depth color index table according to the pixel normalization index value, and determining a pixel color value according to the pixel normalization gray value and the depth color value;
after traversing the three-dimensional millimeter wave sample image to obtain the pixel color value of each three-dimensional pixel point, generating a color two-dimensional millimeter wave image based on the pixel color value and the two-dimensional projection data;
and performing target detection based on the color two-dimensional millimeter wave image to obtain an article labeling result.
According to some embodiments of the invention, the pixel three-dimensional data includes pixel height data, pixel width data and pixel depth data, and the performing logarithmic mapping and normalization processing on the pixel three-dimensional data to obtain pixel normalized data includes:
taking absolute values of the pixel three-dimensional data, and carrying out logarithmic mapping to obtain three-dimensional mapping data of the three-dimensional pixel points, wherein the three-dimensional mapping data comprises height mapping data, width mapping data and depth mapping data;
determining a mapping maximum value and a mapping minimum value of the three-dimensional pixel points according to the three-dimensional mapping data, wherein the mapping maximum value consists of a maximum value of height mapping data, a maximum value of width mapping data and a maximum value of depth mapping data in the three-dimensional mapping data, and the mapping minimum value consists of a minimum value of height mapping data, a minimum value of width mapping data and a minimum value of depth mapping data in the three-dimensional mapping data;
and determining the pixel normalization data of each three-dimensional pixel point according to the mapping maximum value, the mapping minimum value and the three-dimensional mapping data, wherein the pixel normalization data comprises pixel height normalization data, pixel width normalization data and pixel depth normalization data.
According to some embodiments of the present invention, the performing maximum projection in a depth direction according to the pixel normalization data to obtain two-dimensional projection data, a pixel normalization gray value, and a pixel normalization index value of the three-dimensional pixel point includes:
determining the maximum value of the pixel depth normalization data in the pixel depth normalization data as the pixel normalization index value;
projecting the pixel height normalized data and the pixel width normalized data to a position indicated by the maximum value of the pixel depth normalized data to obtain the two-dimensional projection data;
and determining a gray value corresponding to the position indicated by the maximum value of the pixel depth normalization data as the pixel normalization gray value.
According to some embodiments of the present invention, the performing depth upsampling on a preset color matrix to obtain a depth color index table includes:
acquiring a color sequence consisting of a plurality of color elements;
constructing the color matrix according to the color sequence, wherein the number of rows of the color matrix is 3, and the number of columns of the color matrix is the number of the color elements;
and upsampling the color matrix in the depth direction based on a bilinear interpolation method, and determining the obtained sampling matrix as the depth color index table, wherein the number of lines of the sampling matrix is 3, and the number of columns of the sampling matrix is the maximum depth of the three-dimensional millimeter wave image data.
According to some embodiments of the invention, the matching the depth color value in the depth color index table according to the pixel normalization index value includes:
determining a target column of the depth color index table according to the pixel normalization index value;
acquiring a first target row value, a second target row value and a third target row value corresponding to the target column;
and determining the first target line value, the second target line value and the third target line value as the depth color value, wherein the first target line value is used for indicating an R color value, the second target line value is used for indicating a G color value and the third target line value is used for indicating a B color value.
According to some embodiments of the invention, the determining a pixel color value from the pixel normalized gray value and the depth color value comprises:
determining a product of the pixel normalized gray value and the first target line value as a target R color value;
determining a product of the pixel normalized gray value and the second target line value as a target G color value;
determining the product of the pixel normalized gray value and the third target line value as a target B color value;
and obtaining the pixel color value according to the target R color value, the target G color value and the target R color value.
According to some embodiments of the present invention, the object detection based on the color two-dimensional millimeter wave image, to obtain an object labeling result, includes:
when the difference value between the pixel color values of adjacent two-dimensional pixel points in the color two-dimensional millimeter wave image is larger than a preset color threshold value, determining the corresponding two-dimensional pixel points as boundary pixel points;
and determining an object boundary based on the plurality of boundary pixel points, carrying out image recognition on the content in the object boundary, and determining the recognized result as the object labeling result.
In a second aspect, an embodiment of the present invention provides an article marking device based on millimeter wave images, including at least one control processor and a memory for communication connection with the at least one control processor; the memory stores instructions executable by the at least one control processor to enable the at least one control processor to perform the millimeter wave image based item labeling method of the first aspect described above.
In a third aspect, an embodiment of the present invention provides an electronic device, including an article marking apparatus based on millimeter wave images as described in the second aspect.
In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium storing computer-executable instructions for performing the method for labeling items based on millimeter wave images according to the first aspect.
The article labeling method based on the millimeter wave image at least has the following beneficial effects: acquiring a three-dimensional millimeter wave sample image, and determining pixel three-dimensional data of each three-dimensional pixel point of the three-dimensional millimeter wave sample image; carrying out logarithmic mapping and normalization processing on the pixel three-dimensional data to obtain pixel normalized data; performing maximum projection in the depth direction according to the pixel normalization data to obtain two-dimensional projection data of the three-dimensional pixel point, a pixel normalization gray value and a pixel normalization index value, wherein the pixel normalization gray value is a gray value of a position corresponding to the maximum depth value, and the pixel normalization index value is the maximum depth value; up-sampling the preset color matrix in the depth direction to obtain a depth color index table; matching a depth color value in the depth color index table according to the pixel normalization index value, and determining a pixel color value according to the pixel normalization gray value and the depth color value; after traversing the three-dimensional millimeter wave sample image to obtain the pixel color value of each three-dimensional pixel point, generating a color two-dimensional millimeter wave image based on the pixel color value and the two-dimensional projection data; and performing target detection based on the color two-dimensional millimeter wave image to obtain an article labeling result. According to the technical scheme provided by the embodiment of the invention, the image is subjected to color mapping through the depth information of each pixel, the three-dimensional millimeter wave image can be compressed into a two-dimensional color image, the morphological characteristics and the boundaries of the object are highlighted by utilizing the color of the image, and the labeling efficiency is improved.
Drawings
FIG. 1 is a flow chart of a method for labeling items based on millimeter wave images provided by one embodiment of the invention;
FIG. 2 is a comparison of item labels for a color two-dimensional map and a gray two-dimensional map provided by one embodiment of the present invention;
FIG. 3 is a flow chart of a log mapping and normalization process provided by another embodiment of the present invention;
FIG. 4 is a flow chart of maximum projection provided by another embodiment of the present invention;
FIG. 5 is a flow chart of a depth color index table determination according to another embodiment of the present invention;
FIG. 6 is a flow chart for determining depth color values provided by another embodiment of the present invention;
FIG. 7 is a flow chart for determining pixel color values provided by another embodiment of the present invention;
FIG. 8 is a flow chart of labeling items provided in another embodiment of the invention;
fig. 9 is a block diagram of an article marking device based on millimeter wave images according to another embodiment of the present invention.
Detailed Description
Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein like or similar reference numerals refer to like or similar elements or elements having like or similar functions throughout. The embodiments described below by referring to the drawings are illustrative only and are not to be construed as limiting the invention.
In the description of the present invention, it should be understood that references to orientation descriptions such as upper, lower, front, rear, left, right, etc. are based on the orientation or positional relationship shown in the drawings, are merely for convenience of description of the present invention and to simplify the description, and do not indicate or imply that the apparatus or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, and thus should not be construed as limiting the present invention.
In the description of the present invention, a number means one or more, a number means two or more, and greater than, less than, exceeding, etc. are understood to not include the present number, and above, below, within, etc. are understood to include the present number. The description of the first and second is for the purpose of distinguishing between technical features only and should not be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated or implicitly indicating the precedence of the technical features indicated.
In the description of the present invention, unless explicitly defined otherwise, terms such as arrangement, installation, connection, etc. should be construed broadly and the specific meaning of the terms in the present invention can be reasonably determined by a person skilled in the art in combination with the specific contents of the technical scheme.
The embodiment of the invention provides an article labeling method and device based on millimeter wave images and electronic equipment, wherein the article labeling method based on the millimeter wave images comprises the following steps: acquiring a three-dimensional millimeter wave sample image, and determining pixel three-dimensional data of each three-dimensional pixel point of the three-dimensional millimeter wave sample image; carrying out logarithmic mapping and normalization processing on the pixel three-dimensional data to obtain pixel normalized data; performing maximum projection in the depth direction according to the pixel normalization data to obtain two-dimensional projection data of the three-dimensional pixel point, a pixel normalization gray value and a pixel normalization index value, wherein the pixel normalization gray value is a gray value of a position corresponding to the maximum depth value, and the pixel normalization index value is the maximum depth value; up-sampling the preset color matrix in the depth direction to obtain a depth color index table; matching a depth color value in the depth color index table according to the pixel normalization index value, and determining a pixel color value according to the pixel normalization gray value and the depth color value; after traversing the three-dimensional millimeter wave sample image to obtain the pixel color value of each three-dimensional pixel point, generating a color two-dimensional millimeter wave image based on the pixel color value and the two-dimensional projection data; and performing target detection based on the color two-dimensional millimeter wave image to obtain an article labeling result. According to the technical scheme provided by the embodiment of the invention, the image is subjected to color mapping through the depth information of each pixel, the three-dimensional millimeter wave image can be compressed into a two-dimensional color image, the morphological characteristics and the boundaries of the object are highlighted by utilizing the color of the image, and the labeling efficiency is improved.
The control method of the embodiment of the present invention is further described below based on the drawings.
Referring to fig. 1, fig. 1 is a flowchart of an article labeling method based on millimeter wave images according to an embodiment of the present invention, where the article labeling method based on millimeter wave images includes, but is not limited to, the following steps:
s11, acquiring a three-dimensional millimeter wave sample image, and determining pixel three-dimensional data of each three-dimensional pixel point of the three-dimensional millimeter wave sample image;
s12, carrying out logarithmic mapping and normalization processing on the pixel three-dimensional data to obtain pixel normalized data;
s13, carrying out maximum projection in the depth direction according to the pixel normalization data to obtain two-dimensional projection data of a three-dimensional pixel point, a pixel normalization gray value and a pixel normalization index value, wherein the pixel normalization gray value is a gray value of a position corresponding to the maximum depth value, and the pixel normalization index value is the maximum depth value;
s14, up-sampling the preset color matrix in the depth direction to obtain a depth color index table;
s15, matching a depth color value in a depth color index table according to the pixel normalization index value, and determining a pixel color value according to the pixel normalization gray value and the depth color value;
s16, traversing the three-dimensional millimeter wave sample image to obtain a pixel color value of each three-dimensional pixel point, and generating a color two-dimensional millimeter wave image based on the pixel color value and the two-dimensional projection data;
and S17, performing target detection based on the color two-dimensional millimeter wave image to obtain an article labeling result.
It should be noted that, the three-dimensional millimeter wave sample image may be collected by the millimeter wave human body security inspection device, or may also directly collect the image data of the three-dimensional millimeter wave sample image by the millimeter wave human body security inspection device, that is, the data of each three-dimensional pixel point, and the specific data collection mode may be adjusted according to the different collection devices, and the data collection mode is not limited in this embodiment.
It should be noted that, the pixel three-dimensional data in this embodiment may include height data, width data, and depth data, where the expression of the pixel three-dimensional data is: data=d (H, W, C), 0.ltoreq.h.ltoreq.h, 0.ltoreq.w.ltoreq.w, 0.ltoreq.c.ltoreq.c, where H is a height maximum value of the pixel three-dimensional Data, H is a height Data of the pixel three-dimensional Data, W is a width maximum value of the pixel three-dimensional Data, W is a width Data of the pixel three-dimensional Data, C is a depth (or channel) maximum value of the pixel three-dimensional Data, and C is a depth Data (or channel number) of the pixel three-dimensional Data.
It should be noted that, the logarithmic mapping is a process of mapping a real number to a positive real number, and because the three-dimensional data is based on a space coordinate system, mapping a numerical value to a positive real number can reduce the calculation complexity of the subsequent depth projection and improve the calculation efficiency. Similarly, the color is mapped through the depth information, so that only the depth difference among different pixel points is required to be represented, and the computational complexity in the process of computing the depth projection can be effectively reduced through normalization.
In the millimeter wave imaging security inspection field, three-dimensional depth information of human bodies, clothes or articles is different due to the difference of sizes in the same plane, so in order to compress a three-dimensional image into a colorful two-dimensional image, the embodiment establishes a mapping relation between depth and color, and determines a color value of each pixel by using the three-dimensional depth of each pixel, so that a colorful two-dimensional image is obtained. In order to achieve the above-mentioned effects, in this embodiment, a depth color index table is obtained by upsampling a preset color matrix in a depth direction, so that each color value of the depth color index table and depth information establish a mapping relationship, so that a corresponding depth color value is conveniently indexed according to the normalized depth value of a three-dimensional pixel, that is, the color values matched by pixels with different depths are different, and the resolvability of the color two-dimensional millimeter wave image is improved.
It should be noted that, according to the above description, after the data preprocessing is completed to obtain the pixel normalized data, three-dimensional coordinates of each three-dimensional pixel point after normalization are actually obtained, the depth direction, that is, the Z-axis direction, and the maximum value projection is performed based on the Z-axis direction to obtain the two-dimensional projection data. In the two-dimensional projection data, the height data and the width data still have a certain value range, namely, H is more than or equal to 0 and less than or equal to H, W is more than or equal to 0 and less than or equal to W, but the value of c in the depth direction is fixed, so that the height data and the width data are projected into a plane. Because the marking of millimeter wave imaging does not need to accurately obtain the actual outline of the article, and only needs to determine the type of the article according to the general outline of the article, the embodiment can embody the maximum depth of each three-dimensional pixel point by carrying out maximum projection in the depth direction (Z axis), so that the colors of the three-dimensional pixel points with the same depth are approximate, the colors of the three-dimensional pixel points with different depths have larger difference, the article distinguishing is more facilitated, and the influence of the over similar colors on the marking is avoided.
It should be noted that, according to the description of the embodiment, the depth color index table can index different color values for different depths, and after the depth color value is indexed by the pixel normalization index value, the embodiment multiplies the normalized gray value by the pixel normalization gray value according to the depth color value, so as to obtain the pixel color value of the whole pixel point, and under the condition of having the pixel color value and the two-dimensional pixel data projected to a certain plane, a person skilled in the art is familiar with how to construct a color two-dimensional millimeter wave image, and the specific image construction process is not limited.
It should be noted that, the above operations are all repeated for one of the three-dimensional pixel points of the three-dimensional millimeter wave sample image in order to obtain the whole color two-dimensional millimeter wave image, the steps are repeatedly executed for each three-dimensional pixel point of the three-dimensional millimeter wave sample image in a traversing manner, and after the pixel color value corresponding to each three-dimensional pixel point is determined, the color two-dimensional millimeter wave image is generated.
It should be noted that, compared with the gray level map in the related art, boundary confusion is easy to occur, in this embodiment, object detection and labeling are performed based on the color two-dimensional millimeter wave image, and object boundaries can be rapidly determined through color distinction, for example, as shown in fig. 2, the left side of fig. 2 is the object labeling of this embodiment based on the color two-dimensional millimeter wave image, and the objects in the box and the rest of the image can have obvious differences in color level, so that object recognition can be rapidly performed through both the object detection algorithm and the manual labeling, and the object labeling result is effectively improved.
In addition, in an embodiment, the pixel three-dimensional data includes pixel height data, pixel width data, and pixel depth data, and referring to fig. 3, step S12 further includes, but is not limited to, the following steps:
s31, taking absolute values of three-dimensional data of pixels, and carrying out logarithmic mapping to obtain three-dimensional mapping data of three-dimensional pixel points, wherein the three-dimensional mapping data comprises height mapping data, width mapping data and depth mapping data;
s32, determining a mapping maximum value and a mapping minimum value of the three-dimensional pixel point according to the three-dimensional mapping data, wherein the mapping maximum value consists of a maximum value of height mapping data, a maximum value of width mapping data and a maximum value of depth mapping data in the three-dimensional mapping data, and the mapping minimum value consists of a minimum value of height mapping data, a minimum value of width mapping data and a minimum value of depth mapping data in the three-dimensional mapping data;
and S33, determining pixel normalization data of each three-dimensional pixel point according to the mapping maximum value, the mapping minimum value and the three-dimensional mapping data, wherein the pixel normalization data comprises pixel height normalization data, pixel width normalization data and pixel depth normalization data.
It should be noted that the description is based on the embodiment shown in fig. 1The pixel three-dimensional data of the three-dimensional pixel point respectively have three dimensions of height, width and depth, and the embodiment performs logarithmic mapping after taking absolute values for the data of each dimension, so that the three-dimensional mapping data can better reflect the characteristics of the three-dimensional mapping data, and the three-dimensional mapping data can be obtained by calculating according to the following formula: d, d 1 (h,w,c)＝|d(h,w,c)|,d 2 (h,w,c)＝10×log 10 (d 1 (h, w, c)), where d 1 (h, w, c) is the absolute value of the three-dimensional data of the pixel, d 2 (h, w, c) is three-dimensional mapping data.
It should be noted that, in order to implement the normalization process, the data boundary needs to be determined, so the mapping maximum value and the mapping minimum value of the three-dimensional pixel point in this embodiment, that is, the maximum value and the minimum value in three dimensions of height, width and depth, may be specifically obtained by the following formula: d, d max ＝max{d 2 (h,w,c)，0≤h≤H,0≤w≤W,0≤c≤C}，d min ＝min{d 2 (h,w,c)，0≤h≤H,0≤w≤W,0≤c≤C}，Wherein d max To map maximum value d min To map the minimum value, d 3 (h, w, c) is pixel normalized data.
In the above calculation process, each dimension may be calculated, so as to obtain pixel height normalization data, pixel width normalization data, and pixel depth normalization data corresponding to the pixel normalization data.
In addition, in an embodiment, referring to fig. 4, step S13 further includes, but is not limited to, the following steps:
s41, determining the maximum value of the pixel depth normalization data in the pixel depth normalization data as a pixel normalization index value;
s42, projecting the pixel height normalized data and the pixel width normalized data to a position indicated by the maximum value of the pixel depth normalized data to obtain two-dimensional projection data;
s43, determining a gray value corresponding to the position indicated by the maximum value of the pixel depth normalization data as a pixel normalization gray value.
It should be noted that, according to the description of the embodiment shown in fig. 1, the pixel normalization index value is the maximum depth value, and therefore, after the pixel normalization data is calculated according to the manner of the embodiment shown in fig. 3, the maximum value of the pixel depth normalization data can be determined as the pixel normalization index value through simple judgment of the numerical value.
It should be noted that, according to the description of the embodiment shown in fig. 1, the maximum projection is to determine the maximum depth value in the three-dimensional coordinates, then project the remaining two coordinates to the Z-axis plane corresponding to the maximum depth value, for example, for the three-dimensional pixel point d (h, w, c), after determining the maximum depth value as 10 according to the above steps, the two-dimensional projection data obtained after projection is d 4 (h,w,10)，
It should be noted that, in order to improve the distinguishing effect, the pixel normalized gray value may be the maximum gray value in the two-dimensional projection data, that is, the following expression is satisfied: g=max { d 3 (h, e, C), 0.ltoreq.c.ltoreq.C }, and similarly, the pixel normalization index value satisfies the following expression: id=argmax { d 3 (h, w, C), 0.ltoreq.c.ltoreq.C }, where G is the pixel normalized gray value and ID is the pixel normalized index value.
In addition, in an embodiment, referring to fig. 5, step S14 further includes, but is not limited to, the following steps:
s51, acquiring a color sequence composed of a plurality of color elements;
s52, constructing a color matrix according to the color sequence, wherein the number of rows of the color matrix is 3, and the number of columns of the color matrix is the number of color elements;
and S53, up-sampling the color matrix in the depth direction based on a bilinear interpolation method, and determining the obtained sampling matrix as a depth color index table, wherein the number of lines of the sampling matrix is 3, and the number of columns of the sampling matrix is the maximum depth of the three-dimensional millimeter wave image data.
It should be noted that, the color sequence of this embodiment may use a common rainbow color, and obtaining 6 colours therefrom as colour elements, e.g. red, orange, yellow,The green, cyan and blue form a color sequence, and because the image colors mainly determine RGB color values, the constructed color matrix can be a matrix of 3 rows, each row corresponds to R color values, G color values and B color values respectively, and the color matrix expression is as follows: rgb= [ red, orange, yellow, green, blue] 3×6 Wherein the number of columns of the color matrix is 6 and the number of rows is 3.
It should be noted that, in this embodiment, the up-sampling is performed by using the bilinear difference, which can perform linear interpolation in the height and width directions, so as to improve the accuracy of up-sampling, and make the color characterization of the depth color index table in the height and width dimensions more accurate. After upsampling, the resulting depth color index table is rgb= [ sample (RGB)] 3×C 。
In addition, in an embodiment, referring to fig. 6, step S15 further includes, but is not limited to, the following steps:
s61, determining a target column of a depth color index table according to the pixel normalization index value;
s62, acquiring a first target row value, a second target row value and a third target row value corresponding to the target column;
s63, determining a first target line value, a second target line value and a third target line value as depth color values, wherein the first target line value is used for indicating an R color value, the second target line value is used for indicating a G color value, and the third target line value is used for indicating a B color value.
It should be noted that, according to the description of the above embodiment, each element in the depth color index table is a color value mapped to a depth, the pixel normalization index value is a maximum depth value, in this embodiment, a target column is determined from the depth index table by using the pixel normalization index value, 3 row values corresponding to the column are color values which are represented by depths and correspond to three-dimensional pixel points, that is, the depth color values of this embodiment, the depths of the pixel points are different, the matched depth color values are different, for example, the pixel normalization index value is 5, and the element in the 5 th column is selected from the depth color index table as the depth color value.
It should be noted that, according to the description of the above embodiment, the depth color index table is a matrix of 3 rows, so that the first target row value, the second target row value and the third target row value can be obtained, and the embodiment uses the first target row value to indicate the R color value, the second target row value to indicate the G color value, and the third target row value to indicate the B color value, and of course, the color value corresponding to each row can also be adjusted, which is not limited in this embodiment.
In addition, in an embodiment, referring to fig. 7, step S15 further includes, but is not limited to, the following steps:
s71, determining the product of the pixel normalized gray value and the first target line value as a target R color value;
s72, determining the product of the pixel normalized gray value and the second target line value as a target G color value;
s73, determining the product of the pixel normalized gray value and the third target line value as a target B color value;
s74, pixel color values are obtained according to the target R color value, the target G color value and the target R color value.
It should be noted that, according to the description of the above embodiment, since the gray value of the present embodiment is normalized, after determining the depth color value, the pixel color value can be obtained by multiplying the gray value, specifically, the pixel color value can be obtained by the following formula: pixel= [ g×rgb (ID) [0], g×rgb (ID) [1], g×rgb (ID) [2] ], wherein RGB (ID) is a depth color value, 0 is a first target line value, 1 is a second target line value, 2 is a third target line value, and Pixel is a Pixel color value.
In addition, in an embodiment, referring to fig. 8, step S17 further includes, but is not limited to, the following steps:
s81, when the difference value between pixel color values of adjacent two-dimensional pixel points in the color two-dimensional millimeter wave image is larger than a preset color threshold value, determining the corresponding two-dimensional pixel points as boundary pixel points;
s82, determining the object boundary based on the plurality of boundary pixel points, carrying out image recognition on the content in the object boundary, and determining the recognized result as an object labeling result.
After the two-dimensional millimeter wave image of color is obtained, the boundary pixel point can be determined by the difference of pixel color values of the adjacent two-dimensional pixel points, for example, the two articles are mapped with different depths due to different depths, so that the two articles are different in color, the difference of the color values of the boundary pixel point in the image is larger, and therefore, the embodiment can carry out boundary judgment through the preset color threshold value, and the article segmentation is realized.
After the division of the articles, the types of the articles may be determined through image recognition in the related art, so as to label the articles, which will not be described in detail in this embodiment.
As shown in fig. 9, fig. 9 is a block diagram of an article marking device based on millimeter wave images according to an embodiment of the present invention. The invention also provides an article labeling device based on the millimeter wave image, which comprises:
the processor 901 may be implemented by a general purpose central processing unit (Central Processing Unit, CPU), a microprocessor, an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, etc. for executing related programs to implement the technical solutions provided by the embodiments of the present application;
the Memory 902 may be implemented in the form of a Read Only Memory (ROM), a static storage device, a dynamic storage device, or a random access Memory (Random Access Memory, RAM). Memory 902 may store an operating system and other application programs, and when implementing the technical solutions provided in the embodiments of the present disclosure through software or firmware, relevant program codes are stored in memory 902, and the processor 901 is used to invoke and execute the article labeling method based on millimeter wave images in the embodiments of the present disclosure;
an input/output interface 903 for inputting and outputting information;
the communication interface 904 is configured to implement communication interaction between the device and other devices, and may implement communication in a wired manner (e.g. USB, network cable, etc.), or may implement communication in a wireless manner (e.g. mobile network, WIFI, bluetooth, etc.);
a bus 905 that transfers information between the various components of the device (e.g., the processor 901, the memory 902, the input/output interface 903, and the communication interface 904);
wherein the processor 901, the memory 902, the input/output interface 903 and the communication interface 904 are communicatively coupled to each other within the device via a bus 905.
The embodiment of the application also provides electronic equipment, which comprises the article marking device based on the millimeter wave image.
The embodiment of the application also provides a storage medium, which is a computer readable storage medium, and the storage medium stores a computer program, and the computer program realizes the article marking method based on the millimeter wave image when being executed by a processor.
The memory, as a non-transitory computer readable storage medium, may be used to store non-transitory software programs as well as non-transitory computer executable programs. In addition, the memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid state storage device. In some embodiments, the memory optionally includes memory remotely located relative to the processor, the remote memory being connectable to the processor through a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof. The apparatus embodiments described above are merely illustrative, in which the elements illustrated as separate components may or may not be physically separate, implemented to reside in one place, or may be distributed over a plurality of network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
Those of ordinary skill in the art will appreciate that all or some of the steps, systems, and methods disclosed above may be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application specific integrated circuit. Such software may be distributed on computer readable media, which may include computer storage media (or non-transitory media) and communication media (or transitory media). The term computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data, as known to those skilled in the art. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital Versatile Disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by a computer. Furthermore, as is well known to those of ordinary skill in the art, communication media typically include computer readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave or other transport mechanism, and may include any information delivery media.
While the preferred embodiment of the present invention has been described in detail, the present invention is not limited to the above embodiments, and those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit and scope of the present invention, and these equivalent modifications or substitutions are included in the scope of the present invention as defined in the appended claims.
Claims (10)
1. An article labeling method based on millimeter wave images is characterized by comprising the following steps:
acquiring a three-dimensional millimeter wave sample image, and determining pixel three-dimensional data of each three-dimensional pixel point of the three-dimensional millimeter wave sample image;
carrying out logarithmic mapping and normalization processing on the pixel three-dimensional data to obtain pixel normalized data;
performing maximum projection in the depth direction according to the pixel normalization data to obtain two-dimensional projection data of the three-dimensional pixel point, a pixel normalization gray value and a pixel normalization index value, wherein the pixel normalization gray value is a gray value of a position corresponding to the maximum depth value, and the pixel normalization index value is the maximum depth value;
up-sampling the preset color matrix in the depth direction to obtain a depth color index table;
matching a depth color value in the depth color index table according to the pixel normalization index value, and determining a pixel color value according to the pixel normalization gray value and the depth color value;
after traversing the three-dimensional millimeter wave sample image to obtain the pixel color value of each three-dimensional pixel point, generating a color two-dimensional millimeter wave image based on the pixel color value and the two-dimensional projection data;
and performing target detection based on the color two-dimensional millimeter wave image to obtain an article labeling result.
2. The method for labeling objects based on millimeter wave images according to claim 1, wherein the pixel three-dimensional data includes pixel height data, pixel width data and pixel depth data, the performing logarithmic mapping and normalization processing on the pixel three-dimensional data to obtain pixel normalized data comprises:
taking absolute values of the pixel three-dimensional data, and carrying out logarithmic mapping to obtain three-dimensional mapping data of the three-dimensional pixel points, wherein the three-dimensional mapping data comprises height mapping data, width mapping data and depth mapping data;
determining a mapping maximum value and a mapping minimum value of the three-dimensional pixel points according to the three-dimensional mapping data, wherein the mapping maximum value consists of a maximum value of height mapping data, a maximum value of width mapping data and a maximum value of depth mapping data in the three-dimensional mapping data, and the mapping minimum value consists of a minimum value of height mapping data, a minimum value of width mapping data and a minimum value of depth mapping data in the three-dimensional mapping data;
and determining the pixel normalization data of each three-dimensional pixel point according to the mapping maximum value, the mapping minimum value and the three-dimensional mapping data, wherein the pixel normalization data comprises pixel height normalization data, pixel width normalization data and pixel depth normalization data.
3. The method for labeling an article based on a millimeter wave image according to claim 2, wherein the performing maximum projection in a depth direction according to the pixel normalization data to obtain two-dimensional projection data, a pixel normalization gray value, and a pixel normalization index value of the three-dimensional pixel point comprises:
determining the maximum value of the pixel depth normalization data in the pixel depth normalization data as the pixel normalization index value;
projecting the pixel height normalized data and the pixel width normalized data to a position indicated by the maximum value of the pixel depth normalized data to obtain the two-dimensional projection data;
and determining a gray value corresponding to the position indicated by the maximum value of the pixel depth normalization data as the pixel normalization gray value.
4. The method for labeling objects based on millimeter wave images according to claim 1, wherein the step of upsampling the preset color matrix in the depth direction to obtain a depth color index table comprises the steps of:
acquiring a color sequence consisting of a plurality of color elements;
constructing the color matrix according to the color sequence, wherein the number of rows of the color matrix is 3, and the number of columns of the color matrix is the number of the color elements;
and upsampling the color matrix in the depth direction based on a bilinear interpolation method, and determining the obtained sampling matrix as the depth color index table, wherein the number of lines of the sampling matrix is 3, and the number of columns of the sampling matrix is the maximum depth of the three-dimensional millimeter wave image data.
5. The millimeter wave image based item labeling method of claim 4, wherein the matching the depth color value in the depth color index table according to the pixel normalized index value comprises:
determining a target column of the depth color index table according to the pixel normalization index value;
acquiring a first target row value, a second target row value and a third target row value corresponding to the target column;
and determining the first target line value, the second target line value and the third target line value as the depth color value, wherein the first target line value is used for indicating an R color value, the second target line value is used for indicating a G color value and the third target line value is used for indicating a B color value.
6. The millimeter wave image based item labeling method of claim 5, wherein the determining a pixel color value from the pixel normalized gray value and the depth color value comprises:
determining a product of the pixel normalized gray value and the first target line value as a target R color value;
determining a product of the pixel normalized gray value and the second target line value as a target G color value;
determining the product of the pixel normalized gray value and the third target line value as a target B color value;
and obtaining the pixel color value according to the target R color value, the target G color value and the target R color value.
7. The method for labeling objects based on millimeter wave images according to claim 1, wherein the object detection based on the color two-dimensional millimeter wave images to obtain object labeling results comprises:
when the difference value between the pixel color values of adjacent two-dimensional pixel points in the color two-dimensional millimeter wave image is larger than a preset color threshold value, determining the corresponding two-dimensional pixel points as boundary pixel points;
and determining an object boundary based on the plurality of boundary pixel points, carrying out image recognition on the content in the object boundary, and determining the recognized result as the object labeling result.
8. An article marking device based on millimeter wave images, which is characterized by comprising at least one control processor and a memory for being in communication connection with the at least one control processor; the memory stores instructions executable by the at least one control processor to enable the at least one control processor to perform the millimeter wave image-based item labeling method of any of claims 1-7.
9. An electronic device comprising the millimeter wave image-based item labeling apparatus of claim 8.
10. A computer-readable storage medium storing computer-executable instructions for causing a computer to perform the millimeter wave image-based item labeling method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310737145.9A CN116883998B (en) | 2023-06-20 | 2023-06-20 | Article labeling method and device based on millimeter wave image and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310737145.9A CN116883998B (en) | 2023-06-20 | 2023-06-20 | Article labeling method and device based on millimeter wave image and electronic equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116883998A CN116883998A (en) | 2023-10-13 |
| CN116883998B true CN116883998B (en) | 2024-04-05 |
Family
ID=88267169
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310737145.9A Active CN116883998B (en) | 2023-06-20 | 2023-06-20 | Article labeling method and device based on millimeter wave image and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116883998B (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2019184709A1 (en) * | 2018-03-29 | 2019-10-03 | 上海智瞳通科技有限公司 | Data processing method and device based on multi-sensor fusion, and multi-sensor fusion method |
| CN112634244A (en) * | 2020-12-28 | 2021-04-09 | 博微太赫兹信息科技有限公司 | Three-dimensional complex image processing method and system for target detection |
| CN114140517A (en) * | 2021-11-19 | 2022-03-04 | 深圳市优必选科技股份有限公司 | Object pose identification method and device, visual processing equipment and readable storage medium |
| CN114707013A (en) * | 2022-04-08 | 2022-07-05 | Oppo广东移动通信有限公司 | Image color matching method and device, terminal and readable storage medium |
| CN115174774A (en) * | 2022-06-29 | 2022-10-11 | 上海飞机制造有限公司 | Depth image compression method, device, equipment and storage medium |
| CN115797665A (en) * | 2023-02-02 | 2023-03-14 | 深圳佑驾创新科技有限公司 | Image feature-based image and single-frame millimeter wave radar target matching method |
-
2023
- 2023-06-20 CN CN202310737145.9A patent/CN116883998B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2019184709A1 (en) * | 2018-03-29 | 2019-10-03 | 上海智瞳通科技有限公司 | Data processing method and device based on multi-sensor fusion, and multi-sensor fusion method |
| CN112634244A (en) * | 2020-12-28 | 2021-04-09 | 博微太赫兹信息科技有限公司 | Three-dimensional complex image processing method and system for target detection |
| CN114140517A (en) * | 2021-11-19 | 2022-03-04 | 深圳市优必选科技股份有限公司 | Object pose identification method and device, visual processing equipment and readable storage medium |
| CN114707013A (en) * | 2022-04-08 | 2022-07-05 | Oppo广东移动通信有限公司 | Image color matching method and device, terminal and readable storage medium |
| CN115174774A (en) * | 2022-06-29 | 2022-10-11 | 上海飞机制造有限公司 | Depth image compression method, device, equipment and storage medium |
| CN115797665A (en) * | 2023-02-02 | 2023-03-14 | 深圳佑驾创新科技有限公司 | Image feature-based image and single-frame millimeter wave radar target matching method |
Non-Patent Citations (2)
| Title |
|---|
| 利用深度学习进行毫米波图像违禁物体识别;张健;王卫民;唐洋;;计算机与数字工程;20200720(第07期);219-224 * |
| 多参数亮度值重映射的颜色传递方法;仲红玉;尹丽菊;高明亮;邹国锋;申晋;王炫;;红外与激光工程;20180525(第05期);159-166 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116883998A (en) | 2023-10-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105740856B (en) | A kind of pointer instrument registration read method based on machine vision | |
| US9064178B2 (en) | Edge detection apparatus, program and method for edge detection | |
| CN104966089B (en) | A kind of method and device of image in 2 D code edge detection | |
| CN102739951A (en) | Image processing apparatus, image processing method, and program | |
| CN103186904A (en) | Method and device for extracting picture contours | |
| CN109871829B (en) | Detection model training method and device based on deep learning | |
| CN107220664B (en) | Oil bottle boxing and counting method based on structured random forest | |
| CN110189347B (en) | Method and terminal for measuring volume of object | |
| CN102739911A (en) | Image processing apparatus, image processing method, and computer readable medium | |
| CN113592839B (en) | Distribution network line typical defect diagnosis method and system based on improved fast RCNN | |
| CN110598795A (en) | Image difference detection method and device, storage medium and terminal | |
| Rahimzadeganasl et al. | Automatic building detection based on CIE LUV color space using very high resolution pleiades images | |
| CN113343976A (en) | Anti-highlight interference engineering measurement mark extraction method based on color-edge fusion feature growth | |
| CN109658523A (en) | The method for realizing each function operation instruction of vehicle using the application of AR augmented reality | |
| CN109492573A (en) | A kind of pointer read method and device | |
| CN116883998B (en) | Article labeling method and device based on millimeter wave image and electronic equipment | |
| CN110751690B (en) | Visual positioning method for milling machine tool bit | |
| CN112396638A (en) | Image processing method, terminal and computer readable storage medium | |
| Han | Tree height measurement based on image processing with 3-points correction | |
| CN116125489A (en) | Indoor object three-dimensional detection method, computer equipment and storage medium | |
| CN115294035A (en) | Bright point positioning method, bright point positioning device, electronic equipment and storage medium | |
| CN115170552A (en) | Image quality evaluation method, device, equipment and storage medium | |
| CN112036268B (en) | Component identification method and related device | |
| CN114581536A (en) | Image color difference detection method based on feature perception and multi-channel learning | |
| CN112966594A (en) | Environment sensing method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |