CN115222731A - Train fastener abnormity detection method based on two-dimensional image-point cloud mapping - Google Patents

Train fastener abnormity detection method based on two-dimensional image-point cloud mapping Download PDF

Info

Publication number
CN115222731A
CN115222731A CN202211091685.6A CN202211091685A CN115222731A CN 115222731 A CN115222731 A CN 115222731A CN 202211091685 A CN202211091685 A CN 202211091685A CN 115222731 A CN115222731 A CN 115222731A
Authority
CN
China
Prior art keywords
fastener
point cloud
train
dimensional image
dimensional
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202211091685.6A
Other languages
Chinese (zh)
Other versions
CN115222731B (en
Inventor
秦娜
万字朋
黄德青
倪思杰
周重合
蔡重阳
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Southwest Jiaotong University
Original Assignee
Southwest Jiaotong University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Southwest Jiaotong University filed Critical Southwest Jiaotong University
Priority to CN202211091685.6A priority Critical patent/CN115222731B/en
Publication of CN115222731A publication Critical patent/CN115222731A/en
Application granted granted Critical
Publication of CN115222731B publication Critical patent/CN115222731B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0004—Industrial image inspection
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
    • G01N21/84—Systems specially adapted for particular applications
    • G01N21/88—Investigating the presence of flaws or contamination
    • G01N21/8851—Scan or image signal processing specially adapted therefor, e.g. for scan signal adjustment, for detecting different kinds of defects, for compensating for structures, markings, edges
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
    • G06T17/05—Geographic models
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/11—Region-based segmentation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • G06T7/75—Determining position or orientation of objects or cameras using feature-based methods involving models
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
    • G01N21/84—Systems specially adapted for particular applications
    • G01N21/88—Investigating the presence of flaws or contamination
    • G01N21/8851—Scan or image signal processing specially adapted therefor, e.g. for scan signal adjustment, for detecting different kinds of defects, for compensating for structures, markings, edges
    • G01N2021/8887—Scan or image signal processing specially adapted therefor, e.g. for scan signal adjustment, for detecting different kinds of defects, for compensating for structures, markings, edges based on image processing techniques
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10028—Range image; Depth image; 3D point clouds
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • General Engineering & Computer Science (AREA)
  • Computing Systems (AREA)
  • Mathematical Physics (AREA)
  • Artificial Intelligence (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Geometry (AREA)
  • Molecular Biology (AREA)
  • Computer Graphics (AREA)
  • Quality & Reliability (AREA)
  • Remote Sensing (AREA)
  • Signal Processing (AREA)
  • Chemical & Material Sciences (AREA)
  • Analytical Chemistry (AREA)
  • Biochemistry (AREA)
  • Immunology (AREA)
  • Pathology (AREA)
  • Image Analysis (AREA)
  • Train Traffic Observation, Control, And Security (AREA)

Abstract

The invention discloses a train fastener abnormity detection method based on two-dimensional image-point cloud mapping, which comprises the following steps: reading a two-dimensional image and three-dimensional point cloud image data of the fastener, which are acquired by acquisition equipment, from a server according to a shooting sequence; positioning a fastener in a two-dimensional image based on an improved YOLOv4 target detection algorithm; dividing the fastener based on a region growing algorithm, and calculating the geometric center and the rotation angle of the fastener to obtain a region to be detected; fitting planes in the point cloud based on a RANSAC algorithm, and respectively calculating the average distance between the pixel points in the two to-be-detected regions and the fitting planes; calculating the opening and closing angle of the geometric space between the fastener and the fitting plane; and carrying out train fastener abnormity detection through the size relation between the opening and closing angle and the threshold value. The difference between the detection result and the true value of the method provided by the invention is not more than 0.2mm, and the detection accuracy of the fastener, the reliable and stable operation of the train and the rapid detection can be ensured.

Description

Train fastener abnormity detection method based on two-dimensional image-point cloud mapping
Technical Field
The invention relates to the technical field of train fastener abnormity detection, in particular to a train fastener abnormity detection method based on two-dimensional image-point cloud mapping.
Background
The subway is an important transportation tool for people to go out, plays a very important role in relieving urban traffic pressure, promoting urban economic development and the like, and simultaneously, whether a train can run safely and stably influences the personal safety of manpower, material resources and even passengers. With the promulgation of the new generation artificial intelligence development planning, artificial intelligence is raised as a national strategy, the application of the artificial intelligence technology in the urban rail transit field is increasingly wide, and the urban rail transit intelligence not only carries out strategic deployment responding to the country, but also is a trend following the development of the era. In recent years, with the development and popularization of related technologies such as 5G technology and computer hardware, the growing data and the low-cost and high-efficiency computing power provide data and computing power support for the development of artificial intelligence technology, so that the technology barrier is broken through continuously. Traditional manual work mode of patrolling and examining, not only consuming time and power, patrolling and examining the in-process maintainer moreover and probably accompanying potential safety threat, combine together urban rail transit with artificial intelligence, carry out state monitoring to the train through machine vision's mode, can overcome the drawback that the manual work was patrolled and examined to a certain extent.
At present, machine vision is widely applied to the field of rail transit, from the monitoring of human faces and body temperatures of stations entering and exiting to the monitoring of foreign matter invasion in vehicles and beside rails, and a widely applied machine vision technology provides safe and stable riding experience for people and protects driving and navigation for safe traveling of passengers. However, machine vision has a larger promotion space in the aspect of train abnormal state detection, in recent years, 3D point cloud data is widely used to supplement distance information for a two-dimensional image, effectively reduce and eliminate interference of dust, oil stains and the like on detection, and meanwhile, the two-dimensional image is combined with point cloud data to further expand the information content of the data, so that a new idea and judgment logic are provided for solving the problem that a common two-dimensional image cannot be detected. The train fastening pieces are numerous and distributed on the surfaces of various electrical boxes such as a PA box, a storage battery box, a BOP box and the like of the train, are used for fixing the electrical boxes and protecting important parts, equipment and the like in the boxes from being damaged, the equipment in the electric box is key electric equipment of the train, and the safety of the train is directly influenced by ensuring that the electric box is in a safe state, so that the state detection of the electric box is very important to the safe and stable operation of the train.
To sum up, the following key points need to be solved in the mode of combining the two-dimensional image with the point cloud data to realize the train fastener state detection in the practical application scene: 1. the algorithm model has strong robustness, and the core key point of fault detection based on computer vision is the detection reliability, so that the fault detection identification is required to have higher accuracy. 2. The algorithm model needs to have strong universality, and the algorithm can be suitable for fasteners in different vehicle numbers, different positions, different devices and different illumination environments. 3. The algorithm model is fast in detection, and due to the fact that the time for a train to stop at a station is limited, in order to not affect normal operation of the train, train faults need to be overhauled within a set time, and therefore fault detection is required to have the characteristic of being real-time and fast, namely, the detection time is as short as possible.
Disclosure of Invention
In order to solve the problems in the prior art, the invention provides a train fastener abnormity detection method based on two-dimensional image-point cloud mapping, which solves the problems in the background technology.
In order to achieve the purpose, the invention provides the following technical scheme: a train fastener abnormity detection method based on two-dimensional image to point cloud mapping comprises the following steps:
s1, reading a two-dimensional image and three-dimensional point cloud image data of a fastener, which are acquired by acquisition equipment, from a server according to a shooting sequence;
s2, positioning a fastener in a two-dimensional image based on an improved YOLOv4 target detection algorithm;
s3, segmenting the fastener based on a region growing algorithm, and calculating the geometric center and the rotation angle of the fastener to obtain a region to be detected;
s4, fitting a plane in the point cloud based on a RANSAC algorithm, and respectively calculating the average distance between the pixel points in the two to-be-detected regions and the fitting plane;
s5, calculating the opening and closing angle of the geometric space between the fastener and the fitting plane;
and S6, detecting the abnormality of the train fastener according to the size relation between the opening and closing angle and the threshold value.
Preferably, in step S2, the method specifically includes:
s21, loading a trained and improved YOLOv4 target detection model, and performing target detection on two-dimensional images of a train PA box fastener, a PH box fastener, a storage battery box fastener and a BOP box fastener according to the fastener types;
s22, intercepting the rectangular frame image of the target detection according to the target detection result and recording the coordinates of four vertexes of the rectangular frame
Figure 187307DEST_PATH_IMAGE002
、
Figure 905864DEST_PATH_IMAGE004
、
Figure 484613DEST_PATH_IMAGE006
、
Figure 997634DEST_PATH_IMAGE008
。
Preferably, the improved YOLOv4 target detection model is specifically that in a trunk feature extraction network, CSPdarknet53 is replaced by MobilenetV2, so that the positioning speed is increased.
Preferably, in step S3, the method specifically includes:
s31, selecting the center of the target detection rectangular frame as a seed point for region growing, and segmenting a fastener part from the captured image in the step S2;
s32, calculating the geometric center and the rotation angle of the segmented fastener image;
s33, fitting a straight line penetrating through the fastener based on the geometric center and the rotation angle, and selecting two end points in the directions of the upper end and the lower end of the fastener;
and S34, expanding the selected two end points to the periphery, and acquiring a region containing 4 x 4 pixel points as a region to be detected.
Preferably, in step S4, the method specifically includes:
s41, calculating a global optimal plane in the point cloud image based on a RANSAC algorithm, wherein the calculation result comprises four parameters of A, B, C and D, and the fitting plane is expressed as:
Figure 412566DEST_PATH_IMAGE010
；
s42, mapping the area to be detected back to three-dimensional point cloud data, and screening available point cloud data;
s43, calculating an upper detection area and a lower detection area
Figure 731552DEST_PATH_IMAGE012
Average distance to the fitting plane
Figure 40173DEST_PATH_IMAGE014
、
Figure 848729DEST_PATH_IMAGE016
According to the formula of calculating the distance from the point to the surface:
Figure 610012DEST_PATH_IMAGE018
wherein the content of the first and second substances,
Figure 309853DEST_PATH_IMAGE020
is the three-dimensional left information of the pixel points in the point cloud space,
Figure 269718DEST_PATH_IMAGE022
is a coordinate of the X-axis,
Figure 124542DEST_PATH_IMAGE024
is a coordinate of a Y axis and is a coordinate of a Y axis,
Figure 763334DEST_PATH_IMAGE026
is the Z-axis coordinate.
Preferably, the calculation formula of the geometric space opening and closing angle between the fastener and the fitting plane is as follows:
Figure 627384DEST_PATH_IMAGE028
wherein the content of the first and second substances,
Figure 441757DEST_PATH_IMAGE029
、
Figure 874006DEST_PATH_IMAGE031
the average distances from the upper detection area and the lower detection area to the fitting plane are respectively,
Figure 344302DEST_PATH_IMAGE033
、
Figure DEST_PATH_IMAGE035
the coordinates of the upper point and the lower point of the Y axis of the fastener are shown.
Preferably, in step S6, if the opening/closing angle is smaller than the threshold value
Figure DEST_PATH_IMAGE037
Judging that the state of the fastener is normal, otherwise, judging that the fastener is in an abnormal state:
Figure DEST_PATH_IMAGE039
preferably, the threshold is specifically a maximum included angle between the fastener cover plate and the fitting plane in a normal locking state as a threshold
Figure DEST_PATH_IMAGE041
。
The beneficial effects of the invention are:
1) According to the invention, the robot trolley controls the mechanical arm, image data are collected at fixed points according to the three-dimensional map and uploaded to the server, and the switching state of the train fastener is judged in a mode of combining two-dimensional data with three-dimensional point cloud data, so that a large number of negative samples are not needed, and the problem of rare negative samples in an actual operation scene is solved.
2) According to the detection method for mapping the pixel points back to the three-dimensional point cloud data based on the two-dimensional image positioning, the segmentation and the preprocessing of the fastener parts are all performed on the two-dimensional image, the traversing of the whole three-dimensional point cloud data is avoided, the image processing and identifying time is greatly reduced, the calculation amount of an algorithm is reduced, the burden of a server is lightened, and the detection speed is high.
3) According to the invention, the target detection network is improved according to actual requirements, the algorithm efficiency is further improved on the premise of not influencing the positioning accuracy of the algorithm, meanwhile, the target detection algorithm based on deep learning has strong robustness, the influence caused by factors such as shooting angle change, field illumination condition change and the like caused by trolley positioning deviation can be effectively avoided, and the part to be detected can be accurately positioned.
4) The method is based on the region growing algorithm, selects proper seed points, segments the lock catch image without needing a large amount of time, skillfully acquires the geometric center and the rotation angle information of the fastener, fits a straight line and selects the region to be detected, and ensures that the region to be selected can embody the state information of the whole fastener.
5) The invention provides a plane fitting method based on an RANCAC algorithm, selects a plurality of areas to be detected, reflects the on-off state of a fastener in a mode of judging the distance between a point and a plane, accords with the judgment logic of the on-off state of the fastener in actual life, has strong algorithm robustness, is suitable for all rotary opening and closing parts of a subway train, and can be tried to be popularized to other scenes with similar fasteners, such as trucks, high-speed trains and the like.
Drawings
FIG. 1 is a schematic flow chart of a method for detecting train fastener anomalies according to the present invention;
FIG. 2 is a schematic diagram of the calculation of the spatial opening and closing angle of the fastener with reference to the surface of the box (fitting plane);
FIG. 3 is a schematic diagram of the improved YOLOv4 target detection model of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Referring to fig. 1-3, the present invention provides a technical solution: a train fastener abnormity detection method based on two-dimensional image to point cloud mapping is characterized in that a robot trolley carries a 3D camera through a mechanical arm, the trolley stops and controls the mechanical arm to collect image data at a proper angle after running to a specified position, and the following fasteners take lock catches as an example, as shown in figure 1, the method specifically comprises the following steps:
step 1: reading fastener image data collected by the trolley from the server according to the actual shooting sequence
The robot trolley uses laser to measure distance, a three-dimensional map is built, navigation is carried out to a place where a fastener exists according to map information, a shooting angle is adjusted through a control mechanical arm, after a preset acquisition position is reached, two-dimensional image and three-dimensional point cloud image data are acquired, the acquired two-dimensional image and three-dimensional image are calibrated, pixel points in the three-dimensional point cloud correspond to pixel points of the two-dimensional image one by one, and the images are sequentially uploaded to a server to be detected in real time.
And 2, step: positioning fastener in two-dimensional image based on improved YOLOv4 target detection algorithm
The improved YOLOv4 target detection algorithm is adopted in the positioning algorithm, the YOLOv4 target detection algorithm based on deep learning has strong robustness, even if the shooting angle is deviated due to slight positioning deviation of the trolley or abnormal conditions such as partial oil stain, rainwater and dust exist on a fastener, the target can be accurately positioned, and compared with the traditional method, the accuracy and the stability are greatly improved. The main improvement point in the improved model of the YOLOv4 target detection algorithm is to replace CSPdark net53 of a trunk feature extraction network with MobilenetV2, and the purpose of doing so is to adopt a lightweight network to accelerate the positioning speed, as shown in FIG. 3. Before network training, firstly, a data enhancement mode is utilized, the data set is expanded by adding noise, translation, rotation and the like to an original training data set, and then the original training data set is sent to a network for training. After the model training is finished, the trained model is loaded, the lock catch can be positioned on the two-dimensional image in a rectangular frame mode, and coordinates of four top points of the rectangular frame relative to the upper left corner (origin point) of the original image are recorded
Figure DEST_PATH_IMAGE043
。
And step 3: the method comprises the steps of partitioning a fastener based on a region growing algorithm, calculating the geometric center and the rotation angle of the fastener, fitting a straight line penetrating through the upper end and the lower end of the fastener, and selecting two end points in the directions of the upper end and the lower end of the fastener in a self-adaptive mode.
The basic idea of the region growing segmentation algorithm is to group pixels with similar properties together to form a region, and the growing is stopped until no pixel point meeting the similarity requirement is included. In the invention, the fastener is generally a metal part, the color of the fastener is obviously different from the surface of the box body, and by utilizing the characteristic, the center of the target detection rectangular frame is firstly calculated through the coordinates of the four vertexes of the rectangular frame recorded in the previous step, the coordinates are also coordinates relative to the upper left corner (origin) of the original image, and the center coordinates are as follows:
Figure DEST_PATH_IMAGE045
Figure DEST_PATH_IMAGE047
and selecting the point as a seed point for region growth, and segmenting the fasteners in the image based on a region growth algorithm. After the segmentation is finished, the minimum circumscribed rectangle of the segmented area is obtained, the center of the rectangle is the geometric center of the fastener, and the included angle formed by the minimum circumscribed rectangle and the x axis of the original image is the rotation angle
Figure DEST_PATH_IMAGE049
. On the basis, the geometric center is determined
Figure DEST_PATH_IMAGE051
And a rotation angle
Figure 323628DEST_PATH_IMAGE049
Fitting a straight line through the fastener:
Figure 195769DEST_PATH_IMAGE053
and taking the geometric center as a starting point, and selecting end points at two ends in a self-adaptive manner along the directions of the two ends to the upper end and the lower end of the fastener respectively. At this time, since the pixel points in the three-dimensional point cloud necessarily have the corresponding pixel points in the two-dimensional image, but the pixel points in the two-dimensional image may be invalid points in the three-dimensional point cloud, in order to avoid interference of the invalid point cloud on detection, the two selected end points are expanded around as a center, and two square areas are obtained to serve as the areas to be detected.
And 4, step 4: and fitting a plane in the point cloud based on a RANSAC algorithm, mapping the selected to-be-detected region back to three-dimensional point cloud data, and respectively calculating the average distance between the two to-be-detected regions and the fitting plane.
The fastening piece is generally positioned on the surface of an electric box with a flat surface, the surface of the box body occupies most of the whole image in a shot image, points in the plane are assumed to be inner points and points out of the plane are assumed to be outer points based on the RANSAC algorithm, the global optimal plane in the point cloud image is calculated in an iterative mode, namely the fitting plane of the surface of the box body, the iterative calculation result contains four parameters of A, B, C and D, and the fitted plane can be expressed as follows:
Figure 48187DEST_PATH_IMAGE055
and then mapping the coordinates of the pixel points of the to-be-detected region back to the three-dimensional point cloud data, traversing all the pixel points in the region, screening available point cloud data, and avoiding interference of useless data on a detection result. Calculating the distance between each available pixel point in the region and the fitting plane, and calculating the distance formula according to the distance between the point and the plane to obtain the region to be detected
Figure 802517DEST_PATH_IMAGE057
The average distance d to the fitting plane can be expressed as:
Figure 8370DEST_PATH_IMAGE059
wherein, the first and the second end of the pipe are connected with each other,
Figure DEST_PATH_IMAGE060
is the three-dimensional left information of the pixel points in the point cloud space,
Figure 640078DEST_PATH_IMAGE062
is a coordinate of the X-axis,
Figure 273184DEST_PATH_IMAGE063
is a coordinate of the Y axis and is a coordinate of the Y axis,
Figure 842706DEST_PATH_IMAGE065
and the coordinate is Z-axis coordinate, n is the total number of effective pixel points in the area to be detected, and i is a specific pixel point.
And 5: and detecting the abnormality of the train fastener.
Generally, when the fastener is in an unfastened state, the cover plate of the fastener is in a tilted state, and the opening and closing angle of the cover plate from the surface of the box body is far larger than that of the fastener in a normal fastening state (in a normal state, the surface of the fastener is almost parallel to the surface of the box body, and the opening and closing angle is small), so that the opening and closing angle is used as a judgment basis for the abnormal state of the fastener. As shown in fig. 2, the opening and closing angle of the geometric space between the surface of the fastener and the surface of the box body is further calculated, and the calculation formula of the angle is as follows:
Figure DEST_PATH_IMAGE067
wherein the content of the first and second substances,
Figure DEST_PATH_IMAGE068
、
Figure DEST_PATH_IMAGE069
the average distances from the upper detection area and the lower detection area to the fitting plane are respectively,
Figure DEST_PATH_IMAGE071
、
Figure DEST_PATH_IMAGE073
the coordinates of the upper point and the lower point of the Y axis of the fastener are obtained.
And setting a maximum included angle between the cover plate of the fastener and a fitting plane (the surface of the box body) in a normal locking state as a threshold value, and if the opening and closing angle of the fastener to be detected is smaller than the threshold value of the included angle, judging that the locking state is normal, otherwise, judging that the fastener is in an abnormal state.
Figure DEST_PATH_IMAGE075
If the fastener on-off state is detected to be abnormal, alarm information and positioning information are sent to prompt workers to overhaul in time, and more serious consequences are prevented.
The invention provides a complete solution for monitoring the state of the train fastener, the failure of the train in a real running environment is a small-probability event, fewer negative samples can be obtained, the whole set of algorithm designed by the invention does not need a large number of negative samples, the opening and closing state of the fastener is judged in a distance and angle detection mode by combining two-dimensional information and three-dimensional point cloud, and the judgment logic of the actual detection of a maintainer is better met.
The method carries out positioning through an improved YOLOv4 target detection algorithm based on deep learning, replaces a trunk characteristic extraction network with a lighter-weight MobilenetV2 to reduce the parameter quantity of the network, greatly improves the positioning speed on the premise of ensuring the positioning accuracy, tests under the CPU hardware environment, and only needs 0.15s for the positioning speed, meanwhile, the method based on deep learning has strong robustness after being trained by a large amount of data, and can still accurately position the target under the condition that the scene illumination condition changes and the shooting angle deviation is caused by the trolley positioning error.
The method is designed to calculate the distance between a point and a plane and further calculate a spatial included angle to represent the opening and closing state of the fastener, a to-be-detected area capable of covering the whole fastener is selected from a two-dimensional image through positioning, segmentation and fitting of a straight line, three-dimensional point cloud data acquired by a high-precision three-dimensional camera is combined, pixel points are mapped back to the point cloud data, the global optimal plane is fitted in the point cloud based on the RANSAC algorithm by ingeniously utilizing the characteristic that the surface of an electric box is flat, the average distance between the to-be-detected area and the fitting plane is calculated, and through tests, the difference between a detection result and a true value is not more than 0.2mm, so that the accuracy of fastener detection and the reliable and stable operation of a train can be guaranteed.
Although the present invention has been described in detail with reference to the foregoing embodiments, it will be apparent to those skilled in the art that various changes in the embodiments and/or modifications of the invention can be made, and equivalents and modifications of some features of the invention can be made without departing from the spirit and scope of the invention.

Claims (8)

1. A train fastener abnormity detection method based on two-dimensional image to point cloud mapping is characterized by comprising the following steps:
s1, reading a two-dimensional image and three-dimensional point cloud image data of a fastener, which are acquired by acquisition equipment, from a server according to a shooting sequence;
s2, positioning a fastener in a two-dimensional image based on an improved YOLOv4 target detection algorithm;
s3, segmenting the fastener based on a region growing algorithm, and calculating the geometric center and the rotation angle of the fastener to obtain a region to be detected;
s4, fitting a plane in the point cloud based on a RANSAC algorithm, and respectively calculating the average distance between the pixel points in the two to-be-detected regions and the fitting plane;
s5, calculating a geometric space opening and closing angle of the fastener and the fitting plane;
and S6, detecting the abnormity of the train fastener according to the size relation between the opening and closing angle and the threshold value.
2. The train fastener anomaly detection method based on two-dimensional image-to-point cloud mapping according to claim 1, characterized in that: in step S2, the method specifically includes:
s21, loading a trained and improved YOLOv4 target detection model, and performing target detection on two-dimensional images of a train PA box fastener, a PH box fastener, a storage battery box fastener and a BOP box fastener according to the fastener types;
s22, intercepting a rectangular frame image of the target detection according to the target detection result and recording coordinates of four vertexes of the rectangular frame
Figure 511619DEST_PATH_IMAGE002
、
Figure 796818DEST_PATH_IMAGE004
、
Figure 789045DEST_PATH_IMAGE006
、
Figure 687731DEST_PATH_IMAGE008
。
3. The train fastener anomaly detection method based on two-dimensional image-to-point cloud mapping according to claim 2, characterized in that: the improved YOLOv4 target detection model is specifically characterized in that in a trunk feature extraction network, CSPdark net53 is replaced by MobilenetV2, and the positioning speed is accelerated.
4. The train fastener anomaly detection method based on two-dimensional image-to-point cloud mapping according to claim 1, characterized in that: in step S3, the method specifically includes:
s31, selecting the center of the target detection rectangular frame as a seed point for region growth, and segmenting fastener parts in the intercepted image in the step S2;
s32, calculating the geometric center and the rotation angle of the segmented fastener image;
s33, fitting a straight line penetrating through the fastener based on the geometric center and the rotation angle, and selecting two end points in the directions of the upper end and the lower end of the fastener;
and S34, expanding the selected two end points to the periphery, and acquiring a region containing 4 x 4 pixel points as a region to be detected.
5. The train fastener anomaly detection method based on two-dimensional image-to-point cloud mapping according to claim 1, characterized in that: in step S4, the method specifically includes:
s41, calculating a global optimal plane in the point cloud image based on a RANSAC algorithm, wherein the calculation result comprises four parameters of A, B, C and D, and the fitting plane is expressed as:
Figure 867040DEST_PATH_IMAGE010
；
s42, mapping the area to be detected back to three-dimensional point cloud data, and screening available point cloud data;
s43, calculating an upper detection area and a lower detection area
Figure 125852DEST_PATH_IMAGE012
Average distance to fitting plane
Figure 238164DEST_PATH_IMAGE014
、
Figure 42172DEST_PATH_IMAGE016
According to the distance calculation formula from the point to the surface:
Figure 239935DEST_PATH_IMAGE018
wherein, the first and the second end of the pipe are connected with each other,
Figure 302438DEST_PATH_IMAGE020
is the three-dimensional left information of the pixel points in the point cloud space,
Figure 534836DEST_PATH_IMAGE022
is a coordinate of an X-axis and is a coordinate of a Z-axis,
Figure 509745DEST_PATH_IMAGE024
is a coordinate of a Y axis and is a coordinate of a Y axis,
Figure 194805DEST_PATH_IMAGE026
is the Z-axis coordinate.
6. The train fastener anomaly detection method based on two-dimensional image to point cloud mapping according to claim 1, characterized in that: the calculation formula of the geometric space opening and closing angle of the fastener and the fitting plane is as follows:
Figure 546152DEST_PATH_IMAGE028
wherein the content of the first and second substances,
Figure 882324DEST_PATH_IMAGE030
、
Figure 293714DEST_PATH_IMAGE032
the average distances from the upper detection area and the lower detection area to the fitting plane are respectively,
Figure 200490DEST_PATH_IMAGE034
、
Figure 355528DEST_PATH_IMAGE036
the coordinates of the upper point and the lower point of the Y axis of the fastener are obtained.
7. The train fastener anomaly detection method based on two-dimensional image-to-point cloud mapping according to claim 1, characterized in that: in the step S6, if the opening and closing angle is smaller than the threshold value
Figure 296939DEST_PATH_IMAGE038
Judging that the state of the fastener is normal, otherwise, judging that the fastener is in an abnormal state:
Figure 128497DEST_PATH_IMAGE040
。
8. the train fastener anomaly detection method based on two-dimensional image-to-point cloud mapping of claim 7, wherein: the threshold is specifically that the maximum included angle between the fastener cover plate and the fitting plane in the normal locking state is used as the threshold
Figure 522570DEST_PATH_IMAGE042
。
CN202211091685.6A 2022-09-07 2022-09-07 Train fastener abnormity detection method based on two-dimensional image-point cloud mapping Active CN115222731B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202211091685.6A CN115222731B (en) 2022-09-07 2022-09-07 Train fastener abnormity detection method based on two-dimensional image-point cloud mapping

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202211091685.6A CN115222731B (en) 2022-09-07 2022-09-07 Train fastener abnormity detection method based on two-dimensional image-point cloud mapping

Publications (2)

Publication Number Publication Date
CN115222731A true CN115222731A (en) 2022-10-21
CN115222731B CN115222731B (en) 2022-12-02

Family

ID=83617886

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202211091685.6A Active CN115222731B (en) 2022-09-07 2022-09-07 Train fastener abnormity detection method based on two-dimensional image-point cloud mapping

Country Status (1)

Country Link
CN (1) CN115222731B (en)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116386016A (en) * 2023-05-22 2023-07-04 杭州睿影科技有限公司 Foreign matter treatment method and device, electronic equipment and storage medium
CN117456285A (en) * 2023-12-21 2024-01-26 宁波微科光电股份有限公司 Metro shielding door foreign matter detection method based on TOF camera and deep learning model

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111122602A (en) * 2020-01-03 2020-05-08 重庆大学 Three-dimensional camera-based straddle type monorail finger-shaped plate abnormity detection system and method
US20200175311A1 (en) * 2018-11-29 2020-06-04 Element Ai Inc. System and method for detecting and tracking objects
CN112365461A (en) * 2020-11-06 2021-02-12 北京格灵深瞳信息技术有限公司 Fastener loosening identification method, system, terminal and storage medium
CN213163963U (en) * 2020-08-12 2021-05-11 国琳五金电子(苏州)有限公司 Assembly line device of fastener real-time quality detection system
CN112991347A (en) * 2021-05-20 2021-06-18 西南交通大学 Three-dimensional-based train bolt looseness detection method
CN113362467A (en) * 2021-06-08 2021-09-07 武汉理工大学 Point cloud preprocessing and ShuffleNet-based mobile terminal three-dimensional pose estimation method
CN114037703A (en) * 2022-01-10 2022-02-11 西南交通大学 Subway valve state detection method based on two-dimensional positioning and three-dimensional attitude calculation

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20200175311A1 (en) * 2018-11-29 2020-06-04 Element Ai Inc. System and method for detecting and tracking objects
CN111122602A (en) * 2020-01-03 2020-05-08 重庆大学 Three-dimensional camera-based straddle type monorail finger-shaped plate abnormity detection system and method
CN213163963U (en) * 2020-08-12 2021-05-11 国琳五金电子(苏州)有限公司 Assembly line device of fastener real-time quality detection system
CN112365461A (en) * 2020-11-06 2021-02-12 北京格灵深瞳信息技术有限公司 Fastener loosening identification method, system, terminal and storage medium
CN112991347A (en) * 2021-05-20 2021-06-18 西南交通大学 Three-dimensional-based train bolt looseness detection method
CN113362467A (en) * 2021-06-08 2021-09-07 武汉理工大学 Point cloud preprocessing and ShuffleNet-based mobile terminal three-dimensional pose estimation method
CN114037703A (en) * 2022-01-10 2022-02-11 西南交通大学 Subway valve state detection method based on two-dimensional positioning and three-dimensional attitude calculation

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
HANXIAO JIANG等: "OPD: Single-view 3D Openable Part Detection", 《HTTPS://ARXIV.ORG/PDF/2203.16421.PDF》 *
李子沁: "基于深度学习的接触网紧固件缺陷检测算法研究", 《中国优秀硕士学位论文全文数据库 (工程科技Ⅱ辑)》 *
钟志峰等: "基于改进YOLOv4的轻量化目标检测算法", 《计算机应用》 *

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116386016A (en) * 2023-05-22 2023-07-04 杭州睿影科技有限公司 Foreign matter treatment method and device, electronic equipment and storage medium
CN116386016B (en) * 2023-05-22 2023-10-10 杭州睿影科技有限公司 Foreign matter treatment method and device, electronic equipment and storage medium
CN117456285A (en) * 2023-12-21 2024-01-26 宁波微科光电股份有限公司 Metro shielding door foreign matter detection method based on TOF camera and deep learning model

Also Published As

Publication number Publication date
CN115222731B (en) 2022-12-02

Similar Documents

Publication Publication Date Title
CN115222731B (en) Train fastener abnormity detection method based on two-dimensional image-point cloud mapping
Dhiman et al. Pothole detection using computer vision and learning
Banić et al. Intelligent machine vision based railway infrastructure inspection and monitoring using UAV
CN106290388B (en) A kind of insulator breakdown automatic testing method
CN111337941A (en) Dynamic obstacle tracking method based on sparse laser radar data
CN110458161B (en) Mobile robot doorplate positioning method combined with deep learning
CN111126381A (en) Insulator inclined positioning and identifying method based on R-DFPN algorithm
CN110298842A (en) A kind of rail clip image position method based on super-pixel node sequencing
CN114549549B (en) Dynamic target modeling tracking method based on instance segmentation in dynamic environment
CN112197773A (en) Visual and laser positioning mapping method based on plane information
CN116152068A (en) Splicing method for solar panel images
Di Stefano et al. Automatic 2D-3D vision based assessment of the attitude of a train pantograph
CN116977328B (en) Image quality evaluation method in active vision of vehicle bottom robot
CN116524382A (en) Bridge swivel closure accuracy inspection method system and equipment
CN115909216A (en) Cargo ship hatch detection method and system based on laser radar and monocular camera
CN115077563A (en) Vehicle positioning accuracy evaluation method and device and electronic equipment
Li et al. SLAM of mobile robot based on the joint optimization of LiDAR and camera
CN116620360B (en) Rail car positioning system and method
Huang et al. FRVO-Mono: Feature-Based Railway Visual Odometry With Monocular Camera
Jiang et al. A straight-line-based vanishing point detection method for railway environmental images
CN116661500A (en) Rotor unmanned aerial vehicle line patrol control method and device based on vision and storage medium
Qian et al. An Improved Pose Estimation Method in Dynamic Scene Based on Hierarchical Masking and Point Features Classification
Zhang et al. Weld Seam Detection Method with Rotational Region Proposal Network
Li A high-efficiency method of pantograph collector strip wearing inspection based on stereo vision
CN113256736B (en) Multi-camera visual SLAM method based on observability optimization

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant