CN113240790A - Steel rail defect image generation method based on 3D model and point cloud processing - Google Patents
Steel rail defect image generation method based on 3D model and point cloud processing Download PDFInfo
- Publication number
- CN113240790A CN113240790A CN202110401797.6A CN202110401797A CN113240790A CN 113240790 A CN113240790 A CN 113240790A CN 202110401797 A CN202110401797 A CN 202110401797A CN 113240790 A CN113240790 A CN 113240790A
- Authority
- CN
- China
- Prior art keywords
- steel rail
- defect
- model
- information
- point cloud
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 230000007547 defect Effects 0.000 title claims abstract description 187
- 229910000831 Steel Inorganic materials 0.000 title claims abstract description 169
- 239000010959 steel Substances 0.000 title claims abstract description 169
- 238000000034 method Methods 0.000 title claims abstract description 66
- 238000012545 processing Methods 0.000 title claims abstract description 29
- 230000002950 deficient Effects 0.000 claims abstract description 48
- 238000004088 simulation Methods 0.000 claims abstract description 27
- 238000013507 mapping Methods 0.000 claims abstract description 16
- 238000003672 processing method Methods 0.000 claims abstract description 12
- 238000012549 training Methods 0.000 claims abstract description 9
- 238000009499 grossing Methods 0.000 claims abstract description 6
- 238000013528 artificial neural network Methods 0.000 claims abstract description 4
- 230000003321 amplification Effects 0.000 claims description 25
- 238000003199 nucleic acid amplification method Methods 0.000 claims description 25
- 230000003416 augmentation Effects 0.000 claims description 9
- 238000009826 distribution Methods 0.000 claims description 7
- 230000008569 process Effects 0.000 claims description 6
- 238000006243 chemical reaction Methods 0.000 claims description 4
- 230000007704 transition Effects 0.000 abstract description 8
- 238000002372 labelling Methods 0.000 abstract description 5
- 238000001514 detection method Methods 0.000 abstract description 3
- 238000010586 diagram Methods 0.000 description 13
- 238000004364 calculation method Methods 0.000 description 6
- 230000006870 function Effects 0.000 description 6
- 230000000694 effects Effects 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 4
- 238000011160 research Methods 0.000 description 4
- 230000008859 change Effects 0.000 description 3
- 238000012423 maintenance Methods 0.000 description 3
- 230000002441 reversible effect Effects 0.000 description 3
- 239000000243 solution Substances 0.000 description 3
- 230000003042 antagnostic effect Effects 0.000 description 2
- 238000013135 deep learning Methods 0.000 description 2
- 201000010099 disease Diseases 0.000 description 2
- 208000037265 diseases, disorders, signs and symptoms Diseases 0.000 description 2
- 238000000605 extraction Methods 0.000 description 2
- 238000009877 rendering Methods 0.000 description 2
- 238000013526 transfer learning Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000005266 casting Methods 0.000 description 1
- 230000001609 comparable effect Effects 0.000 description 1
- 238000009795 derivation Methods 0.000 description 1
- 238000002347 injection Methods 0.000 description 1
- 239000007924 injection Substances 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000003062 neural network model Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 238000003860 storage Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- General Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- Computing Systems (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Biomedical Technology (AREA)
- Molecular Biology (AREA)
- Biophysics (AREA)
- General Engineering & Computer Science (AREA)
- Artificial Intelligence (AREA)
- Mathematical Physics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Health & Medical Sciences (AREA)
- Computer Graphics (AREA)
- Geometry (AREA)
- Investigating Materials By The Use Of Optical Means Adapted For Particular Applications (AREA)
Abstract
The invention provides a steel rail defect image generation method based on a 3D model and point cloud processing, which comprises the steps of firstly, constructing a normal steel rail 3D model and a defect steel rail initial 3D model; then, amplifying the number of initial 3D models of the defective steel rail and the types of the defects by a point cloud processing method; then, smoothing the transition difference between the defect and the background based on the curvature and the inverse curvature; and finally, realizing automatic labeling by a label mapping method, and simultaneously increasing the diversity and complexity of the background of the steel rail simulation data by a texture replacement method. The method provided by the invention can generate unlimited high-quality marked steel rail defect simulation data. The data can be used for later neural network training or auxiliary training, and the influence of insufficient data samples on a steel rail defect detection task is effectively improved.
Description
Technical Field
The invention relates to the technical field of data set generation, in particular to a steel rail defect image generation method based on a 3D model and point cloud processing.
Background
China is a big railway country, and the total length of the railway is over 13.1 kilometers. As the length of railways increases, maintenance of the state of the railways becomes increasingly important. In the maintenance of railway conditions, the automated location and identification of rail surface defects is one of the important factors.
As an important technical means for detecting the service state of the rail transit infrastructure, the steel rail surface defect detection technology based on the deep learning technology trains a neural network model through a large amount of defect image data, so that various disease defects on the surface of a steel rail are identified and positioned. In an actual scene, the number of the defective steel rails is rare due to good maintenance, so that a batch of simulation data generated according to the characteristics of the defective steel rails is a good scheme for independent research or research together with real data.
The research on simulation data generation technology has been more tried in other fields, but the research on generating the simulation data of the surface defects of the steel rail directly is less. In 2014, south China university, a generation method and a generation system of a dendritic shrinkage porosity defect simulation image of a casting are provided, a defect image is subjected to binarization conversion and histogram matching, a new false data set is generated by combining a method of superposing a real background image and diffusing a defect edge region so as to increase a test sample and a learning sample, but the method is only used for generating a defect on a two-dimensional image by an image processing method, and the quality of the generated defect is not ideal. Zhejiang university proposed a method for recognizing surface image defects of injection molded products based on transfer learning in 2019, and generated a batch of false sample sets by using a method for generating an antagonistic network, and Zhejiang industry university proposed a method for generating drain pipe defect detection training data based on PGGAN transfer learning in 2019, which generated virtual data by using a technology for generating an antagonistic network. The generation methods based on generation of the countermeasure network all have common defects:
(1) a large amount of real data is needed to participate in the training process of the generator, and if the real data is less, a simulation data set with high quality cannot be obtained;
(2) the tag information of the data cannot be synchronously acquired when the data is generated, and a large amount of manpower is consumed for marking in the later period;
(3) the types of defects generated are random and the generated result cannot be controlled.
In addition to the above common defects, since the size information of the rail defect in the rail data set is relatively fine compared to the scene where the rail defect is located, that is, the occupation ratio in the whole image is small, it is difficult to generate the virtual data by using the generation countermeasure network to retain the defect information, and thus the generated rail has a relatively real overall effect but lacks critical defect information.
Disclosure of Invention
The embodiment of the invention provides a steel rail defect image generation method based on a 3D model and point cloud processing, which is used for solving the problems in the prior art.
In order to achieve the purpose, the invention adopts the following technical scheme.
A steel rail defect image generation method based on 3D model and point cloud processing comprises the following steps:
constructing a normal steel rail 3D model, and adding initial defect information in data of the normal steel rail 3D model to obtain an initial defective steel rail 3D model;
performing amplification processing on the initial defect information by a point cloud processing method to obtain a plurality of extension defect information and further obtain one or more defect steel rail extension 3D models;
smoothing the defective steel rail extension 3D model by an operation method based on curvature and inverse curvature to obtain a defective steel rail target 3D model;
converting the smoothed defective steel rail target 3D model to obtain a steel rail defect simulation data set;
and mapping the extended defect position information of the defective steel rail target 3D model into a steel rail defect simulation data set by a label mapping method to obtain target label information, wherein the target label information is used for neural network training.
Preferably, the step of performing amplification processing on the initial defect information by using a point cloud processing method to obtain a plurality of extended defect information, and the step of further obtaining a plurality of extended defect 3D models of the defective steel rail comprises:
carrying out format conversion on the point cloud libraries of the normal steel rail 3D model and the defect steel rail initial 3D model to respectively obtain a normal steel rail point cloud model and a defect steel rail point cloud model;
extracting point cloud information of a defective steel rail point cloud model, and performing amplification treatment to obtain a plurality of extension defect information;
and obtaining a plurality of defect steel rail extension 3D models based on the plurality of extension defect information.
Preferably, extracting point cloud information of the defect steel rail point cloud model comprises:
passing through type
DefectPoint＝{(x，y，z)|(x，y，z)∈RailPoint，z≠f(x)} (1)
Acquiring information of points which do not meet the curve function of the surface of the normal steel rail; wherein f (x) is a surface curve function of the fitted normal steel rail;
obtaining information Defect Point of all point cloud parts which do not satisfy the surface distribution relation of the steel rail based on the surface curve function of the normal steel rail, and calculating the coordinate X of the minimum square frame which can enclose all point cloud parts which do not satisfy the surface distribution relation of the steel railL、YL、XRAnd YR(ii) a Wherein, XLAnd YLAre respectively as
XL＝min(x|(x，y，z)∈DefectPoint)，YL＝min(y|(x，y，z)∈DefectPoint)，XRAnd YRAre each XR＝max(x|(x，y，z)∈DefectPoint)，YR＝max(y|(x，y，z)∈DefectPoint)；
Based on XL、YL、XRAnd YRExtracting all point cloud information Defect BoxPoint { (X, y, z) | (X, y, z) ∈ RailPoint, and X { (X, y, z) } in a minimum square frame capable of enclosing all point cloud parts which do not meet the distribution relation of the surface of the steel railL≤x≤XR，YL≤y≤YR}(2)。
Preferably, the plurality of extended defect information includes volume augmentation defect information, shape augmentation defect information, number augmentation defect information, and location augmentation defect information;
the volume amplification defect information is obtained by a formula ScalPoint ═ { alpha (x, y, z) | (x, y, z) ∈ Defect BoxPoint, and alpha ∈ (0,1) } (3), wherein a represents a volume scaling factor;
shape-amplified defect information pass-through
ShapePoint＝{(α1x，α2y，α3z)|(x，y，z)∈DefectBoxPoint，α1，α2，α3∈(0，1]Obtaining (4);
in the formula, alpha1、α2And alpha3Scale factors respectively representing three dimensions of coordinate axes x, y and z;
the quantity amplification defect information is obtained by calculating the range of the point cloud area of the defect steel rail point cloud model in the square frame, which can be deviated;
position amplification defect information passing mode
FinalPoint { (x + offsetX, y + offsetY, z) | (x, y, z) ∈ detectbutoxypoint ═ ScaleBoxPoint vent { (5).
Preferably, the smoothing the extended 3D model of the defective steel rail by the operation method based on the curvature and the inverse curvature to obtain the target 3D model of the defective steel rail includes:
stretching extension defect information to a rail surface coordinate system in a normal rail 3D model by using a formula CurvePoint { (x, y, z + f (x)) | (x, y, z) ∈ FinaPoint } (6);
by passing
AntiCurvePoint
＝{(x，y，z-f(x))|(x，y，z)∈DefectBoxPoint∪ScalePoint∪ShapePoint}
FinalPoint＝{(x+offsetX，y+offsetY，z)|(x，y，z)∈AntiCurvePoint}
CurvePoint { (x, y, z + f (x)) | (x, y, z) ∈ FinalPoint } (7) remove/add curvature information of the square frame.
Preferably, mapping extended defect position information of the defective steel rail target 3D model into a steel rail defect simulation dataset by a label mapping method, and obtaining target label information includes:
passing through type
the texture of the rail defect simulation dataset is replaced by the formula resultPicture ═ TexturePicture × R + Original Picture × (1-R), R ∈ (0,1) (9).
According to the technical scheme provided by the embodiment of the invention, the steel rail defect image generation method based on the 3D model and the point cloud processing is designed, and a huge amount of amplified steel rail defect models with the same comparable effect as a manually-kneaded steel rail defect model are automatically generated according to a small amount of preliminary manually-kneaded defective steel rail simulation models; meanwhile, in the process of expanding the defects by using the point cloud generating processing method, the curvature of the steel rail is fitted, and then the smooth transition between the defects and the 3D model background information of the steel rail is designed by curvature and inverse curvature operation, so that the expanded 3D model of the defective steel rail is more real. After a two-dimensional steel rail defect simulation data set is obtained by rendering a 3D model of a defective steel rail, a texture set is constructed by using a real data set of a normal steel rail, and textures of simulation data are replaced by randomly selecting the textures from the texture set, so that the background of the generated steel rail defect simulation data set becomes more real. The label information is extracted while the defect 3D model is generated, and is mapped to the two-dimensional data set through a mapping method, so that the automation work of the label information is realized, and the generation efficiency is improved.
Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
FIG. 1 is a processing flow chart of a steel rail defect image generation method based on a 3D model and point cloud processing according to the present invention;
FIG. 2 is a schematic diagram of a 3D model of a normal steel rail in a steel rail defect image generation method based on a 3D model and point cloud processing provided by the invention;
FIG. 3 is a schematic diagram of an initial 3D model of a defective steel rail in a steel rail defect image generation method based on a 3D model and point cloud processing provided by the invention;
FIG. 4 is a schematic process diagram of a point cloud processing method in the steel rail defect image generation method based on the 3D model and the point cloud processing provided by the invention;
FIG. 5 is a schematic diagram of a steel rail surface curvature fitting operation in a steel rail defect image generation method based on a 3D model and point cloud processing provided by the invention;
FIG. 6 is an effect diagram after curvature and inverse curvature operations in the steel rail defect image generation method based on the 3D model and point cloud processing provided by the invention;
FIG. 7 is a schematic diagram of curvature stretching operation in the steel rail defect image generation method based on 3D model and point cloud processing provided by the invention;
FIG. 8 is a schematic diagram of a reverse curvature stretching operation in the steel rail defect image generation method based on a 3D model and point cloud processing provided by the invention;
FIG. 9 is a schematic diagram illustrating the principle of automatic extraction of tag information in a steel rail defect image generation method based on a 3D model and point cloud processing according to the present invention;
fig. 10 is a schematic diagram illustrating an automatic extraction effect of tag information in a steel rail defect image generation method based on a 3D model and point cloud processing according to the present invention;
fig. 11 is a schematic diagram of a texture replacement result in the steel rail defect image generation method based on the 3D model and the point cloud processing provided by the present invention.
Detailed Description
Reference will now be made in detail to embodiments of the present invention, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to the same or similar elements or elements having the same or similar function throughout. The embodiments described below with reference to the accompanying drawings are illustrative only for the purpose of explaining the present invention, and are not to be construed as limiting the present invention.
As used herein, the singular forms "a", "an", "the" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and/or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof. It will be understood that when an element is referred to as being "connected" or "coupled" to another element, it can be directly connected or coupled to the other element or intervening elements may also be present. Further, "connected" or "coupled" as used herein may include wirelessly connected or coupled. As used herein, the term "and/or" includes any and all combinations of one or more of the associated listed items.
It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the prior art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.
For the convenience of understanding the embodiments of the present invention, the following description will be further explained by taking several specific embodiments as examples in conjunction with the drawings, and the embodiments are not to be construed as limiting the embodiments of the present invention.
The invention provides a generation method of a steel rail defect image data set, which is used for realizing the following purposes:
1. generating expectable steel rail defect simulation data under the condition that real defective steel rail data do not exist;
2. the marking information is automatically extracted to avoid the complexity of manual marking;
3. amplifying defects by a point cloud processing method to increase the speed of generating a defect data set based on a 3d model method;
4. and generating more real steel rail defect simulation data through texture replacement and curvature fitting.
Referring to fig. 1, the method for generating a rail defect image based on 3D model and point cloud processing provided by the invention comprises the following steps:
s1, constructing a normal steel rail 3D model, and adding initial defect information into data of the normal steel rail 3D model to obtain an initial defective steel rail 3D model; in this step, a modeling tool in the prior art, such as solidworks or other 3D model construction software, can be used for constructing a 3D model of the normal steel rail, and a corresponding 3D model of the normal steel rail is constructed according to the size and proportion of the actual steel rail (i.e. the real steel rail is modeled in a virtual environment); then adding initial defect information on the normal steel rail 3D model by using wrap or other software according to the disease characteristics (defect information) to be generated, thereby obtaining a small batch of defective virtual defect steel rail initial 3D models;
s2, performing amplification processing on the initial defect information through a point cloud processing method to obtain a plurality of extended defect information and further obtain one or more defect steel rail extended 3D models; the method comprises the following steps of extracting defect information in an initial 3D model of the defective steel rail, and changing the form, size and position of the defect information, so that a large number of various virtual 3D models of the steel rail with the defects can be amplified;
s3, smoothing the defect steel rail extension 3D model through an operation method based on curvature and inverse curvature to enable defect information to appear more naturally on the steel rail 3D model, and obtaining a defect steel rail target 3D model;
s4, after generating a large number of diversified defective steel rail target 3D models with smoothly fused defect characteristics, converting the defective steel rail target 3D models to obtain a steel rail defect simulation data set; the method specifically comprises the steps of developing an acquisition tool for acquiring the information of the top surfaces of the steel rails in batches through 3dmax, and converting a defective steel rail target 3D model set into a two-dimensional steel rail defect simulation data set by using the acquisition tool;
s5, mapping the extended defect position information of the defective steel rail target 3D model to the steel rail defect simulation data set through a label mapping method, so that target label information for neural network training can be acquired without manual labeling.
In the preferred embodiment of the present invention, step S1 is specifically to first construct a 3D model of a normal rail 500mm long according to the size and proportion of the real rail by using 3D modeling software, as shown in fig. 2. And then adding defect information to the surface of the steel rail by using model fine tuning software or common 3D modeling software to obtain an initial 3D model of the defective steel rail, as shown in FIG. 3.
After the initial 3D model of the defective steel rail is obtained, the number and diversity of the 3D models of the defective steel rail are expanded by utilizing a designed point cloud expansion method. As shown in fig. 4, format conversion is performed on a point cloud library of a normal steel rail 3D model and a defective steel rail initial 3D model, for example, by a python point cloud library, and a normal steel rail point cloud model and a defective steel rail point cloud model are obtained respectively. And for the defective steel rail point cloud model RailPoint, extracting a defect point cloud part Defect Point through the characteristic that the defect does not accord with the surface distribution of the normal steel rail, and performing amplification treatment to obtain a plurality of extension defect information. The method can be specifically operated through the following sub-steps:
as shown in the cross-section of the rail in figure 5,
the curve of the upper surface of the steel rail can be mainly divided into 5 sections of circular arcs which are respectively called S1, S2, S3, S4 and S5 from left to right, and the circle centers of the circular arcs are marked as O1,O2，O3，O4，O5. Wherein the curvature radius of S1 and S5 is R3Radius of curvature R of S2, S42. Radius of curvature of S3 is R1. Other parameters in fig. 1: a. the1Is the x-axis length of the first arc, A2Is the x-axis length of the second arc, A3The x-axis length of the third arc. The surface curve function of the steel rail can be obtained through mathematical derivation:
whereby the expression of the defect point cloud part is
Deffectpoint { (x, y, z) | (x, y, z) ∈ rallpoint, z ≠ f (x) } (1); wherein (x, y, z) is used to represent the coordinate of the midpoint of the three-dimensional point cloud, where x, y, z represents the value of each individual dimension in the coordinate, and specifically z represents the depth of the steel rail, the x-axis is the transverse axis of the steel rail, and y is the longitudinal axis of the steel rail, and since the cross-sections of the steel rail are the same, the depth is only related to the x-axis; firstly, points belonging to RailPoint are taken out through an equation (1), points of which the x coordinate and the z coordinate do not accord with the function relationship of z ═ f (x) are removed, and the coordinates of the rest points are assigned to Defect Point;
calculating the coordinate X of the smallest square box capable of enclosing the defect point cloud partsL、YL、XRAnd YR(ii) a Wherein XLAnd YLAre respectively as
XL＝min(x|(x，y，z)∈DefectPoint)，YL＝min(y|(x，y，z)∈DefectPoint)，XRAnd YRIs XR＝max(x|(x，y，z)∈DefectPoint)，YR＝max(y|(x，y，z)∈DefectPoint)；
Is specifically processed by the above formulaProcedure XLFor example, the following steps are carried out: firstly, taking out the x coordinate of a point belonging to the Defect Point; selecting the smallest value in the X coordinates to be assigned to XL(ii) a The other same principles are adopted;
based on XL、YL、XRAnd YRAll point cloud information Defect BoxPoint { (X, y, z) | (X, y, z) ∈ RailPoint, and X { (X, y, z) } in the minimum square frame of the defect steel rail point cloud part can be extractedL≤x≤XR，YL≤y≤YR}(2)。
For the extracted point cloud information in the square frame, only changing the x coordinate, the y coordinate and the z coordinate or simultaneously changing only two of 3 coordinates can change the shape of the defect, and changing the values of three coordinates in the same proportion can change the size of the defect. Diversity amplification can be achieved by the above-described method to obtain extended defect information.
Further, the extended defect information includes volume amplification defect information, shape amplification defect information, number amplification defect information, and position amplification defect information.
The volume amplification defect information is obtained by a formula ScalPoint ═ { alpha (x, y, z) | (x, y, z) ∈ Defect BoxPoint, and alpha ∈ (0,1] } (3), wherein a represents a volume scaling factor, and the smaller the value is, the smaller the defect point cloud after amplification is;
shape-amplified defect information pass-through
ShapePoint＝{(α1x，α2y，α3z)|(x，y，z)∈DefectBoxPoint，α1，α2，α3∈(0，1]Obtaining (4);
in the formula, alpha1、α2And alpha3Scaling factors respectively representing three dimensions of x, y and z of a coordinate axis, wherein when the scaling factors of the dimensions are different, the scaling factors of the dimensions can lead the stretching degrees of the dimensions to be different, so that the original defect point cloud is deformed, and the purpose of shape amplification is achieved;
and simultaneously calculating the offset ranges (offset X and offset Y) of the point clouds in the square frame on the surface of the steel rail, and randomly offsetting the point clouds on any position on the steel rail to realize quantitative amplification.
The position amplification defect information is obtained by a formula FinalPoint { (x + offset X, y + offset Y, z) | (x, y, z) ∈ DetectbOxPoint [. U.ScaleBoxPoint [. U.ShapePoint } (5).
And digging out point cloud information of corresponding positions on the normal steel rail point cloud model, filling the point cloud information FinaPoint with amplified forms and quantities contained in the square frame into the point cloud information, obtaining a defective steel rail 3D point cloud file, and converting the defective steel rail 3D point cloud file into a 3D model file, thereby realizing the scheme of point cloud amplification.
When the point cloud information is filled into a normal steel rail 3D point cloud file, the problem that transition between a square frame containing a defect point cloud and a background point cloud is unnatural can be faced. Such as left 1 in fig. 6. In view of this problem, the corresponding operation of curvature and inverse curvature is designed in the preferred embodiment provided by the present invention to solve this problem. The effect after the curvature and the reverse curvature operation are shown in fig. 6 as left 2 and left 3, respectively. Curvature and reverse curvature operations are some of the optimization operations that are performed in consideration of the cause of the occurrence of the unnatural transition of the edge. The transition is not natural because the reference frame of the square frame is not a horizontal plane as the reference frame when the smallest square frame containing the defect information is extracted and filled in. When the defect is filled into the normal steel rail 3D point cloud, the edge of the normal steel rail 3D point cloud is not horizontal, so if the defect is directly filled, the edge is raised or sunken as shown in the left 1 of FIG. 7. Therefore, curvature operation is adopted to stretch the defect point cloud information FinalPoint to a coordinate system (f (x)) of the steel rail surface in the normal steel rail 3D model, and the calculation formula is as follows { (x, y, z + f (x)) | (x, y, z) ∈ FinalPoint }.
As shown in fig. 7, left two. The transition of the square frame containing the defect point cloud to the background point cloud becomes more natural through the curvature operation, but this problem is not completely solved. Therefore, the operation of inverse curvature is proposed in this embodiment, mainly considering that the coordinate system of the square frame of the extracted defect point cloud is not flat, and it also has the curvature of the steel rail, as shown in fig. 8. Therefore, the curvature information of the square frame is first eliminated by the inverse curvature operation, and then the curvature information of the part to be filled is added by the curvature operation during filling, the formula is as follows
AntiCurvePoint
＝{(x，y，z-f(x))|(x，y，z)∈DefectBoxPoint∪ScalePoint∪ShapePoint}
FinalPoint＝{(x+offsetX，y+offsetY，z)|(x，y，z)∈AntiCurvePoint}
CurvePoint＝{(x，y，z+f(x))|(x，y，z)∈FinalPoint} (7)。
Through the above operations, a 3D model of the target defective steel rail is generated as shown in fig. 6 at the left 3.
In the preferred embodiment of the present invention, the operation of the annotation information mapping method of step S5 is as shown in fig. 9, which maps the generated defect location information onto the final two-dimensional image, and the format of the annotation information is a target-level annotation in the YOLO format (the annotation information in the YOLO format uses a quadruple (X, Y, W, H) to label the location of an object in the picture.
First, the coordinates of the upper left corner (x1, y1) and the coordinates of the lower right corner (x2, y2) of the square box, which are coordinates relative to the modeled origin at the lower center of the rail plane, may be obtained during point cloud processing. When mapping to the annotation information in the YOLO format, it is necessary to know the relative value of the center point coordinate of the square frame on the picture, and the relative values of the width and height, and the coordinate to be obtained at this time is the coordinate of the origin of the image. The map information is obtained by calculation as shown in fig. 9. Therefore, automatic labeling of the labeling information is realized, and the effect of automatic labeling is shown in fig. 10.
In one specific embodiment, as shown in fig. 9, first, the size of the final image is fixed (only including the rail surface area) in the point cloud stage, and is Wrail*HrailThe information of the box is also already obtained in the point cloud processing stage, i.e. (xl, yl), (x2, y 2).
By calculation formula
(8) And obtaining the position information and the volume information of a square frame surrounding the defect in the target 3D model of the defective steel rail.
Wherein, (x1+ x2)/2+ WrailAnd/2 is the abscissa of the center point of the square frame relative to the upper left corner of the rail surface (the position corresponding to the image origin), and divided by WrailI.e. relative coordinates, and the calculation process of y is the same as that of y. The relative position of the center point of the final square frame is (X, Y) in the upper diagram.
Meanwhile, the relative size of the frame, i.e., the size of the rail surface, is the same as the information at the point cloud stage, and thus is (W, H) in the calculation formula (8).
After the two-dimensional data set is obtained, a texture set is constructed by using a two-dimensional image of a normal steel rail, and then the texture of the virtual data set is replaced in a pixel superposition mode. Calculation formula
Result scene is texture × R + OriginalPicture × (1-R), R in R ∈ (0,1) (9) is the intensity parameter for texture replacement, texture scene is a picture randomly extracted from the texture set (scaled to the same size as the virtual picture in order to enable pixel-level overlay), and OriginalPicture is the virtual picture. The results after replacement are shown in FIG. 11.
In summary, the steel rail defect image generation method based on the 3D model and the point cloud processing provided by the invention includes the following steps:
and manufacturing a normal steel rail 3D model by utilizing solidworks or other 3D model construction software, and finely adjusting the normal steel rail 3D model by wrap or other software to obtain a batch of defective steel rail 3D models with defects.
And expanding the number and diversity of the defective steel rail 3D models by a point cloud processing method. For diversity we varied the size, shape of the defect. For the number, we randomly offset the positions of the defect information after the form change and the original defect information on the rail. Thereby obtaining an infinite and sufficient 3D model of the defect steel rail. And meanwhile, the curvature of the steel rail 3D model is fitted, and the transition of the defects and the background information of the steel rail 3D model is smoothed through the curvature and the inverse curvature operation.
The virtual rail surface map of the defective steel rail 3D model generated by us is collected through 3dmax or other software in a batch rendering mode. Meanwhile, in order to make the background information of the virtual rail surface image more complex and diversified, a texture set is constructed through the texture of the normal image, and one texture information of the rail surface image is randomly selected from the texture set to replace the texture information of the rail surface image.
And mapping the defect positions acquired in the point cloud processing process to a final rail surface diagram by a designed marking information mapping method, and forming a format of a YOLO marking.
The method provided by the invention has the following beneficial effects:
the automatic and high-speed generation of the rail simulation data set can be realized without the participation of any real data. The generated data can be used for network training or auxiliary training based on deep learning at a later stage;
the generated virtual data are provided with corresponding label information, manual marking is not needed, the investment of human resources is greatly reduced, and meanwhile, the point cloud-based augmentation mode is high in speed and can generate a large amount of data;
the defects amplified by the point cloud processing method appear and transition on the steel rail more naturally by adopting curvature and inverse curvature operation, so that the similarity between the artificially kneaded model and the point cloud expanded model reaches a higher level;
texture information is extracted through real normal steel rail data to replace the texture of the generated data, and the background diversity and authenticity of the generated defect data are guaranteed.
Those of ordinary skill in the art will understand that: the figures are merely schematic representations of one embodiment, and the blocks or flow diagrams in the figures are not necessarily required to practice the present invention.
From the above description of the embodiments, it is clear to those skilled in the art that the present invention can be implemented by software plus necessary general hardware platform. Based on such understanding, the technical solutions of the present invention may be embodied in the form of a software product, which may be stored in a storage medium, such as ROM/RAM, magnetic disk, optical disk, etc., and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the method according to the embodiments or some parts of the embodiments.
The embodiments in the present specification are described in a progressive manner, and the same and similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for apparatus or system embodiments, since they are substantially similar to method embodiments, they are described in relative terms, as long as they are described in partial descriptions of method embodiments. The above-described embodiments of the apparatus and system are merely illustrative, and the units described as separate parts may or may not be physically separate, and the parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.
The above description is only for the preferred embodiment of the present invention, but the scope of the present invention is not limited thereto, and any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope of the present invention are included in the scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims (6)
1. A steel rail defect image generation method based on 3D model and point cloud processing is characterized by comprising the following steps:
constructing a normal steel rail 3D model, and adding initial defect information in data of the normal steel rail 3D model to obtain an initial defective steel rail 3D model;
performing amplification processing on the initial defect information by a point cloud processing method to obtain a plurality of extension defect information and further obtain one or more defect steel rail extension 3D models;
smoothing the defective steel rail extension 3D model by an operation method based on curvature and inverse curvature to obtain a defective steel rail target 3D model;
converting the smoothed defective steel rail target 3D model to obtain a steel rail defect simulation data set;
and mapping the extended defect position information of the defective steel rail target 3D model to the steel rail defect simulation data set by a label mapping method to obtain target label information, wherein the target label information is used for neural network training.
2. The method of claim 1, wherein the step of performing an amplification process on the initial defect information by a point cloud processing method to obtain a plurality of extended defect information, and the step of further obtaining a plurality of extended 3D models of defective rails comprises:
carrying out format conversion on the point cloud libraries of the normal steel rail 3D model and the defect steel rail initial 3D model to respectively obtain a normal steel rail point cloud model and a defect steel rail point cloud model;
extracting point cloud information of the defective steel rail point cloud model, and performing amplification treatment to obtain a plurality of extension defect information;
and obtaining the plurality of defect steel rail extension 3D models based on the plurality of extension defect information.
3. The method of claim 2, wherein the extracting point cloud information of the defect steel rail point cloud model comprises:
passing through type
DefectPoint＝{(x,y,z)|(x,y,z)∈RailPoint,z≠f(x)} (1)
Acquiring information of points which do not meet the curve function of the surface of the normal steel rail; wherein f (x) is a surface curve function of the fitted normal steel rail;
obtaining information Defect Point of all point cloud parts which do not satisfy the surface distribution relation of the steel rail based on the surface curve function of the normal steel rail, and calculating the coordinate X of the minimum square frame which can enclose all point cloud parts which do not satisfy the surface distribution relation of the steel railL、YL、XRAnd YR(ii) a Wherein, XLAnd YLAre each XL＝min(x|(x,y,z)∈DefectPoint),YL＝min(y|(x,y,z)∈DefectPoint)，XRAnd YRAre each XR＝max(x|(x,y,z)∈DefectPoint),YR＝max(y|(x,y,z)∈DefectPoint)；
Based on XL、YL、XRAnd YRExtracting all point cloud information Defect BoxPoint { (X, y, z) | (X, y, z) ∈ RailPoint, and X { (X, y, z) } in the minimum square frame capable of enclosing all point cloud parts which do not meet the distribution relation of the surface of the steel railL≤x≤XR,YL≤y≤YR} (2)。
4. The method of claim 3, wherein the plurality of extended defect information comprises volume augmentation defect information, shape augmentation defect information, number augmentation defect information, and location augmentation defect information;
the volume amplification defect information is obtained by a formula ScalPoint ═ { alpha (x, y, z) | (x, y, z) ∈ Defect BoxPoint, and alpha ∈ (0,1) } (3), wherein alpha represents a volume scaling factor;
the shape-amplified defect information is passed through
ShapePoint＝{(α1x,α2y,α3z)|(x,y,z)∈DefectBoxPoint,α1,α2,α3∈(0,1]Obtaining (4); in the formula, alpha1、α2And alpha3Scale factors respectively representing three dimensions of coordinate axes x, y and z;
the quantity amplification defect information is obtained by calculating the range of the point cloud area of the defect steel rail point cloud model in the square frame, wherein the range of the point cloud area can be deviated;
the position amplification defect information is through
FinalPoint { (x + offsetX, y + offsetY, z) | (x, y, z) ∈ detectbutoxypoint ═ ScaleBoxPoint vent { (5).
5. The method according to claim 4, wherein the smoothing the extended 3D model of the defective steel rail by the operation method based on the curvature and the inverse curvature to obtain the target 3D model of the defective steel rail comprises:
stretching the extension defect information to a rail surface coordinate system in the normal rail 3D model by using a formula CurvePoint { (x, y, z + f (x)) | (x, y, z) ∈ FinaPoint } (6);
by passing
AntiCurvePoint
＝{(x,y,z-f(x))|(x,y,z)∈DefectBoxPoint∪ScalePoint∪ShapePoint}
FinalPoint＝{(x+offsetX,y+offsetY,z)|(x,y,z)∈AntiCurvePoint}
CurvePoint { (x, y, z + f (x)) | (x, y, z) ∈ FinalPoint } (7) remove/add curvature information of the square frame.
6. The method of claim 5, wherein the mapping extended defect location information of the target 3D model of the defective rail into the rail defect simulation dataset by a label mapping method, and obtaining target label information comprises:
passing through type
replacing the texture of the rail defect simulation dataset by the formula resultPicture ═ TexturePicture × R + Original Picture × (1-R), R ∈ (0,1) (9).
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110401797.6A CN113240790B (en) | 2021-04-14 | 2021-04-14 | Rail defect image generation method based on 3D model and point cloud processing |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110401797.6A CN113240790B (en) | 2021-04-14 | 2021-04-14 | Rail defect image generation method based on 3D model and point cloud processing |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113240790A true CN113240790A (en) | 2021-08-10 |
| CN113240790B CN113240790B (en) | 2024-06-14 |
Family
ID=77128270
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110401797.6A Active CN113240790B (en) | 2021-04-14 | 2021-04-14 | Rail defect image generation method based on 3D model and point cloud processing |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113240790B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114529689A (en) * | 2022-04-24 | 2022-05-24 | 广州易道智慧信息科技有限公司 | Ceramic cup defect sample amplification method and system based on antagonistic neural network |
| GB2613879A (en) * | 2021-12-17 | 2023-06-21 | Zeta Motion Ltd | Automated inspection system |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107995225A (en) * | 2017-12-26 | 2018-05-04 | 国网河南省电力公司信息通信公司 | A kind of security even analysis method towards complex network |
| CN111982910A (en) * | 2020-07-06 | 2020-11-24 | 华南理工大学 | Weak supervision machine vision detection method and system based on artificial defect simulation |
| CN112070712A (en) * | 2020-06-05 | 2020-12-11 | 宁波大学 | Printing defect detection method based on self-encoder network |
| US20210004946A1 (en) * | 2019-07-02 | 2021-01-07 | MakinaRocks Co., Ltd. | Systems and methods for detecting flaws on panels using images of the panels |
-
2021
- 2021-04-14 CN CN202110401797.6A patent/CN113240790B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107995225A (en) * | 2017-12-26 | 2018-05-04 | 国网河南省电力公司信息通信公司 | A kind of security even analysis method towards complex network |
| US20210004946A1 (en) * | 2019-07-02 | 2021-01-07 | MakinaRocks Co., Ltd. | Systems and methods for detecting flaws on panels using images of the panels |
| CN112070712A (en) * | 2020-06-05 | 2020-12-11 | 宁波大学 | Printing defect detection method based on self-encoder network |
| CN111982910A (en) * | 2020-07-06 | 2020-11-24 | 华南理工大学 | Weak supervision machine vision detection method and system based on artificial defect simulation |
Non-Patent Citations (1)
| Title |
|---|
| 周洲: "《轮芯缺陷检测及缺陷样本生成技术》", 《全国优秀硕士论文》 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| GB2613879A (en) * | 2021-12-17 | 2023-06-21 | Zeta Motion Ltd | Automated inspection system |
| CN114529689A (en) * | 2022-04-24 | 2022-05-24 | 广州易道智慧信息科技有限公司 | Ceramic cup defect sample amplification method and system based on antagonistic neural network |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113240790B (en) | 2024-06-14 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113450307B (en) | Product edge defect detection method | |
| CN112862811B (en) | Material microscopic image defect identification method, equipment and device based on deep learning | |
| WO2020165557A1 (en) | 3d face reconstruction system and method | |
| CN107767453B (en) | Building LIDAR point cloud reconstruction optimization method based on rule constraint | |
| CN104134234A (en) | Full-automatic three-dimensional scene construction method based on single image | |
| CN110263794B (en) | Training method of target recognition model based on data enhancement | |
| CN105678683A (en) | Two-dimensional storage method of three-dimensional model | |
| CN109840483B (en) | Landslide crack detection and identification method and device | |
| CN113240790A (en) | Steel rail defect image generation method based on 3D model and point cloud processing | |
| CN110110646A (en) | A kind of images of gestures extraction method of key frame based on deep learning | |
| CN105095581B (en) | A kind of casting shrinking hole defect image generation method | |
| CN110223376B (en) | Three-dimensional particle reconstruction method based on single accumulated particle material image | |
| CN115937461B (en) | Multi-source fusion model construction and texture generation method, device, medium and equipment | |
| CN111598803B (en) | Point cloud filtering method based on variable-resolution voxel grid and sparse convolution | |
| CN108197555B (en) | Real-time face fusion method based on face tracking | |
| CN115937626A (en) | Automatic generation method of semi-virtual data set based on instance segmentation | |
| CN109598782A (en) | Building historical relic restorative procedure, storage medium based on dimensional Modeling Technology | |
| CN114359269A (en) | Virtual food box defect generation method and system based on neural network | |
| CN117496085A (en) | Parameterized three-dimensional model generation method and system based on hand-drawn sketch | |
| CN111462084B (en) | Image vectorization printing bleeding point prediction system and method based on random forest | |
| CN115620287A (en) | Indoor point cloud scene semantic segmentation method based on patch context characteristics | |
| CN112002019B (en) | Method for simulating character shadow based on MR mixed reality | |
| CN115273080A (en) | Lightweight visual semantic odometer method for dynamic scene | |
| CN114119874A (en) | Single image reconstruction high-definition 3D face texture method based on GAN | |
| CN110348458A (en) | A kind of image texture classification method based on local direction extreme value mode |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |