CN115222731B - Train fastener abnormity detection method based on two-dimensional image-point cloud mapping - Google Patents

Train fastener abnormity detection method based on two-dimensional image-point cloud mapping Download PDF

Info

Publication number
CN115222731B
CN115222731B CN202211091685.6A CN202211091685A CN115222731B CN 115222731 B CN115222731 B CN 115222731B CN 202211091685 A CN202211091685 A CN 202211091685A CN 115222731 B CN115222731 B CN 115222731B
Authority
CN
China
Prior art keywords
fastener
point cloud
train
detection
dimensional image
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202211091685.6A
Other languages
Chinese (zh)
Other versions
CN115222731A (en
Inventor
秦娜
万字朋
黄德青
倪思杰
周重合
蔡重阳
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Southwest Jiaotong University
Original Assignee
Southwest Jiaotong University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Southwest Jiaotong University filed Critical Southwest Jiaotong University
Priority to CN202211091685.6A priority Critical patent/CN115222731B/en
Publication of CN115222731A publication Critical patent/CN115222731A/en
Application granted granted Critical
Publication of CN115222731B publication Critical patent/CN115222731B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0004—Industrial image inspection
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
    • G01N21/84—Systems specially adapted for particular applications
    • G01N21/88—Investigating the presence of flaws or contamination
    • G01N21/8851—Scan or image signal processing specially adapted therefor, e.g. for scan signal adjustment, for detecting different kinds of defects, for compensating for structures, markings, edges
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
    • G06T17/05—Geographic models
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/11—Region-based segmentation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • G06T7/75—Determining position or orientation of objects or cameras using feature-based methods involving models
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
    • G01N21/84—Systems specially adapted for particular applications
    • G01N21/88—Investigating the presence of flaws or contamination
    • G01N21/8851—Scan or image signal processing specially adapted therefor, e.g. for scan signal adjustment, for detecting different kinds of defects, for compensating for structures, markings, edges
    • G01N2021/8887—Scan or image signal processing specially adapted therefor, e.g. for scan signal adjustment, for detecting different kinds of defects, for compensating for structures, markings, edges based on image processing techniques
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10028—Range image; Depth image; 3D point clouds
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • General Engineering & Computer Science (AREA)
  • Computing Systems (AREA)
  • Mathematical Physics (AREA)
  • Artificial Intelligence (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Geometry (AREA)
  • Molecular Biology (AREA)
  • Computer Graphics (AREA)
  • Quality & Reliability (AREA)
  • Remote Sensing (AREA)
  • Signal Processing (AREA)
  • Chemical & Material Sciences (AREA)
  • Analytical Chemistry (AREA)
  • Biochemistry (AREA)
  • Immunology (AREA)
  • Pathology (AREA)
  • Image Analysis (AREA)
  • Train Traffic Observation, Control, And Security (AREA)

Abstract

The invention discloses a train fastener abnormity detection method based on two-dimensional image-point cloud mapping, which comprises the following steps: reading a two-dimensional image and three-dimensional point cloud image data of the fastener, which are acquired by acquisition equipment, from a server according to a shooting sequence; positioning a fastener in a two-dimensional image based on an improved YOLOv4 target detection algorithm; dividing the fastener based on a region growing algorithm, and calculating the geometric center and the rotation angle of the fastener to obtain a region to be detected; fitting a plane in the point cloud based on a RANSAC algorithm, and respectively calculating the average distance between the pixel points in the two to-be-detected regions and the fitting plane; calculating the opening and closing angle of the geometric space between the fastener and the fitting plane; and carrying out train fastener abnormity detection through the size relation between the opening and closing angle and the threshold value. The difference between the detection result and the true value of the method provided by the invention is not more than 0.2mm, and the detection accuracy of the fastener, the reliable and stable operation of the train and the quick detection can be ensured.

Description

Train fastener abnormity detection method based on two-dimensional image-point cloud mapping
Technical Field
The invention relates to the technical field of train fastener abnormity detection, in particular to a train fastener abnormity detection method based on two-dimensional image-point cloud mapping.
Background
The subway is an important transportation tool for people to go out, plays a very important role in relieving urban traffic pressure, promoting urban economic development and the like, and simultaneously, whether a train can run safely and stably influences the personal safety of manpower, material resources and even passengers. With the promulgation of the new generation artificial intelligence development planning, artificial intelligence is raised as a national strategy, the application of the artificial intelligence technology in the urban rail transit field is increasingly wide, and the urban rail transit intelligence not only carries out the strategic deployment in responding to the country, but also is the trend of complying with the development of the era. In recent years, with the development and popularization of related technologies such as 5G technology and computer hardware, the growing data and the low-cost and high-efficiency computing power provide data and computing power support for the development of artificial intelligence technology, so that the technology barrier is broken through continuously. Traditional manual inspection mode, not only consuming time and wasting power, moreover inspection in-process maintainer probably accompanies latent safety threat, combines together urban rail transit and artificial intelligence, carries out state monitoring to the train through machine vision's mode, can overcome the drawback that the manual work was patrolled and examined to a certain extent.
At present, machine vision is widely applied to the field of rail transit, from the monitoring of human faces and body temperatures of stations entering and exiting to the monitoring of foreign matter invasion in vehicles and beside rails, and a widely applied machine vision technology provides safe and stable riding experience for people and protects driving and navigation for safe traveling of passengers. However, machine vision has a larger promotion space in the aspect of train abnormal state detection, in recent years, 3D point cloud data is widely applied, distance information is supplemented for a two-dimensional image, interference of dust, oil stains and the like on detection is effectively reduced, meanwhile, the information content of the data is further expanded by combining the two-dimensional image with the point cloud data, and a new idea and judgment logic are provided for solving the problem that a common two-dimensional image cannot be detected. The train fastening pieces are numerous and distributed on the surfaces of various electrical boxes such as a PA box, a storage battery box, a BOP box and the like of the train, are used for fixing the electrical boxes and protecting important parts, equipment and the like in the boxes from being damaged, the equipment in the electric box is the key electric equipment of the train, and the safety of the train is directly influenced by ensuring that the electric box is in a safe state, so that the condition detection of the electric box is very important to the safe and stable operation of the train.
To sum up, the following key points need to be solved in the mode of combining the two-dimensional image with the point cloud data to realize the train fastener state detection in the practical application scene: 1. the algorithm model has strong robustness, and the core key point of fault detection based on computer vision is the detection reliability, so that the fault detection identification is required to have higher accuracy. 2. The algorithm model needs to have strong universality, and the algorithm can be suitable for fasteners in different vehicle numbers, different positions, different devices and different illumination environments. 3. The algorithm model is fast in detection, and due to the fact that the time for a train to stop at a station is limited, in order to not affect normal operation of the train, train faults need to be overhauled within a set time, and therefore fault detection is required to have the characteristic of being real-time and fast, namely, the detection time is as short as possible.
Disclosure of Invention
In order to solve the problems in the prior art, the invention provides a train fastener abnormity detection method based on two-dimensional image-point cloud mapping, which solves the problems in the background technology.
In order to achieve the purpose, the invention provides the following technical scheme: a train fastener abnormity detection method based on two-dimensional image to point cloud mapping comprises the following steps:
s1, reading a two-dimensional image and three-dimensional point cloud image data of a fastener, which are acquired by acquisition equipment, from a server according to a shooting sequence;
s2, positioning a fastener in a two-dimensional image based on an improved YOLOv4 target detection algorithm;
s3, segmenting the fastener based on a region growing algorithm, and calculating the geometric center and the rotation angle of the fastener to obtain a region to be detected;
s4, fitting a plane in the point cloud based on a RANSAC algorithm, and respectively calculating the average distance between the pixel points in the two to-be-detected regions and the fitting plane;
s5, calculating a geometric space opening and closing angle of the fastener and the fitting plane;
and S6, detecting the abnormality of the train fastener according to the size relation between the opening and closing angle and the threshold value.
Preferably, in step S2, the method specifically includes:
s21, loading a trained improved YOLOv4 target detection model, and performing target detection on two-dimensional images of a train PA box fastener, a PH box fastener, a storage battery box fastener and a BOP box fastener according to the fastener types;
s22, intercepting a rectangular frame image of the target detection according to the target detection result and recording coordinates of four vertexes of the rectangular frame
Figure 187307DEST_PATH_IMAGE002
、
Figure 905864DEST_PATH_IMAGE004
、
Figure 484613DEST_PATH_IMAGE006
、
Figure 997634DEST_PATH_IMAGE008
。
Preferably, the improved YOLOv4 target detection model is specifically that in a trunk feature extraction network, CSPdarknet53 is replaced by MobilenetV2, so that the positioning speed is increased.
Preferably, in step S3, the method specifically includes:
s31, selecting the center of the target detection rectangular frame as a seed point for region growth, and segmenting fastener parts in the intercepted image in the step S2;
s32, calculating the geometric center and the rotation angle of the segmented fastener image;
s33, fitting a straight line penetrating through the fastener based on the geometric center and the rotation angle, and selecting two end points in the directions of the upper end and the lower end of the fastener;
and S34, expanding the area to the periphery by taking the selected two end points as centers, and acquiring an area containing 4*4 pixel points as an area to be detected.
Preferably, in step S4, the method specifically includes:
s41, calculating a global optimal plane in the point cloud image based on the RANSAC algorithm, wherein the calculation result comprises four parameters of A, B, C, D, and the fitting plane is expressed as:
Figure 412566DEST_PATH_IMAGE010
；
s42, mapping the area to be detected back to three-dimensional point cloud data, and screening available point cloud data;
s43, calculating an upper detection area and a lower detection area
Figure 731552DEST_PATH_IMAGE012
Average distance to the fitting plane
Figure 40173DEST_PATH_IMAGE014
、
Figure 848729DEST_PATH_IMAGE016
According to the distance calculation formula from the point to the surface:
Figure 610012DEST_PATH_IMAGE018
wherein the content of the first and second substances,
Figure 309853DEST_PATH_IMAGE020
is the three-dimensional left information of the pixel points in the point cloud space,
Figure 269718DEST_PATH_IMAGE022
is a coordinate of the X-axis,
Figure 124542DEST_PATH_IMAGE024
is a coordinate of the Y axis and is a coordinate of the Y axis,
Figure 763334DEST_PATH_IMAGE026
is the Z-axis coordinate.
Preferably, the calculation formula of the geometric space opening and closing angle between the fastener and the fitting plane is as follows:
Figure 627384DEST_PATH_IMAGE028
wherein the content of the first and second substances,
Figure 441757DEST_PATH_IMAGE029
、
Figure 874006DEST_PATH_IMAGE031
the average distances from the upper and lower detection areas to the fitting plane are respectively,
Figure 344302DEST_PATH_IMAGE033
、
Figure DEST_PATH_IMAGE035
the coordinates of the upper point and the lower point of the Y axis of the fastener are shown.
Preferably, in the step S6, if the opening/closing angle is smaller than the threshold value
Figure DEST_PATH_IMAGE037
Judging that the state of the fastener is normal, otherwise, judging that the fastener is in an abnormal state:
Figure DEST_PATH_IMAGE039
preferably, the threshold is specifically a maximum included angle between the fastener cover plate and the fitting plane in a normal locking state as a threshold
Figure DEST_PATH_IMAGE041
。
The invention has the beneficial effects that:
1) According to the invention, the robot trolley controls the mechanical arm, image data are collected at fixed points according to the three-dimensional map and uploaded to the server, and the switching state of the train fastener is judged in a mode of combining two-dimensional data with three-dimensional point cloud data, so that a large number of negative samples are not needed, and the problem of rare negative samples in an actual operation scene is solved.
2) According to the detection method for mapping the pixel points back to the three-dimensional point cloud data based on the two-dimensional image positioning, the segmentation and the preprocessing of the fastener parts are all performed on the two-dimensional image, the traversing of the whole three-dimensional point cloud data is avoided, the image processing and identifying time is greatly reduced, the calculation amount of an algorithm is reduced, the burden of a server is lightened, and the detection speed is high.
3) According to the invention, the target detection network is improved according to actual requirements, the algorithm efficiency is further improved on the premise of not influencing the positioning accuracy of the algorithm, meanwhile, the target detection algorithm based on deep learning has strong robustness, the influence caused by factors such as shooting angle change, field illumination condition change and the like caused by trolley positioning deviation can be effectively avoided, and the part to be detected can be accurately positioned.
4) The method is based on the region growing algorithm, selects proper seed points, segments the lock catch image without needing a large amount of time, skillfully acquires the information of the geometric center and the rotation angle of the fastener, fits a straight line and selects the region to be detected, and ensures that the selected region can reflect the state information of the whole fastener.
5) The invention provides a plane fitting method based on an RANCAC algorithm, selects a plurality of areas to be detected, reflects the on-off state of a fastener in a mode of judging the distance between a point and a plane, accords with the judgment logic of the on-off state of the fastener in actual life, has strong algorithm robustness, is suitable for all rotary opening and closing parts of a subway train, and can be tried to be popularized to other scenes with similar fasteners, such as trucks, high-speed trains and the like.
Drawings
FIG. 1 is a schematic flow chart of a method for detecting train fastener anomalies according to the present invention;
FIG. 2 is a schematic diagram of the calculation of the spatial opening and closing angle of the fastener with reference to the surface of the box (fitting plane);
FIG. 3 is a schematic diagram of the improved YOLOv4 target detection model of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Referring to fig. 1-3, the present invention provides a technical solution: a train fastener abnormity detection method based on two-dimensional image to point cloud mapping is characterized in that a robot trolley carries a 3D camera through a mechanical arm, the trolley stops and controls the mechanical arm to collect image data at a proper angle after running to a specified position, and the following fasteners take lock catches as an example, as shown in figure 1, the method specifically comprises the following steps:
step 1: reading fastener image data collected by the trolley from the server according to the actual shooting sequence
The robot trolley utilizes laser ranging to construct a three-dimensional map, navigates to a place where a fastener exists according to map information, adjusts a shooting angle through a control mechanical arm, starts to collect two-dimensional images and three-dimensional point cloud image data after reaching a preset collecting position, at the moment, the collected two-dimensional images and three-dimensional images are calibrated, pixel points in the three-dimensional point cloud correspond to pixel points of the two-dimensional images one by one, and the images are sequentially uploaded to a server for real-time detection.
Step 2: positioning fastener in two-dimensional image based on improved YOLOv4 target detection algorithm
The improved YOLOv4 target detection algorithm is adopted in the positioning algorithm, the YOLOv4 target detection algorithm based on deep learning has strong robustness, and even if the shooting angle is deviated due to slight deviation of trolley positioning or abnormal conditions such as partial oil stain, rainwater, dust and the like exist on a fastener, the target can be accurately positioned, and compared with the traditional method, the positioning algorithm has the advantages of higher accuracy and stabilityAnd (5) greatly lifting. The main improvement point in the improved model of the Yolov4 target detection algorithm is to replace the trunk feature extraction network from CSPdarknet53 to MobileneetV 2, and the purpose of doing so is to adopt a lightweight network to accelerate the positioning speed, as shown in FIG. 3. Before network training, firstly, a data enhancement mode is utilized, the data set is expanded by adding noise, translation, rotation and the like to an original training data set, and then the original training data set is sent to a network for training. After the model training is finished, the trained model is loaded, the lock catch can be positioned on the two-dimensional image in a rectangular frame mode, and four vertex coordinates of the rectangular frame relative to the upper left corner (original point) of the original image are recorded
Figure DEST_PATH_IMAGE043
。
And 3, step 3: the method comprises the steps of partitioning a fastener based on a region growing algorithm, calculating the geometric center and the rotation angle of the fastener, fitting a straight line penetrating through the upper end and the lower end of the fastener, and selecting two end points in the directions of the upper end and the lower end of the fastener in a self-adaptive mode.
The basic idea of the region growing segmentation algorithm is to group pixels with similar properties together to form a region, and the growing is stopped until no pixel point meeting the similarity requirement is included. According to the invention, the fastener is generally a metal part, the color of the fastener is obviously different from the surface of the box body, and by utilizing the characteristic, the center of the target detection rectangular frame is firstly calculated through coordinates of four vertexes of the rectangular frame recorded in the previous step, the coordinates are the coordinates relative to the upper left corner (origin) of the original image, and the coordinates of the center are as follows:
Figure DEST_PATH_IMAGE045
Figure DEST_PATH_IMAGE047
and selecting the point as a seed point for region growth, and segmenting the fasteners in the image based on a region growth algorithm. After the segmentation is finished, the minimum circumscribed rectangle and moment of the segmented area are solvedThe center of the figure is the geometric center of the fastener, and the included angle formed by the minimum external rectangle and the x axis of the original image is the rotation angle
Figure DEST_PATH_IMAGE049
. Based on the above, the geometric center is determined
Figure DEST_PATH_IMAGE051
And a rotation angle
Figure 323628DEST_PATH_IMAGE049
Fitting a straight line through the fastener:
Figure 195769DEST_PATH_IMAGE053
and taking the geometric center as a starting point, and selecting end points at two ends in a self-adaptive manner along the directions of the two ends to the upper end and the lower end of the fastener respectively. At this time, since the pixel points in the three-dimensional point cloud necessarily have the corresponding pixel points in the two-dimensional image, but the pixel points in the two-dimensional image may be invalid points in the three-dimensional point cloud, in order to avoid interference of the invalid point cloud on detection, the two selected end points are expanded around as a center, and two square areas are obtained to serve as the areas to be detected.
And 4, step 4: and fitting a plane in the point cloud based on a RANSAC algorithm, mapping the selected to-be-detected region back to three-dimensional point cloud data, and respectively calculating the average distance between the two to-be-detected regions and the fitting plane.
The fastener is generally positioned on the surface of an electric box with a flat surface, the surface of the box body occupies most of the whole image in a shot image, points in a plane are assumed to be inner points and points out of the plane are assumed to be outer points based on an RANSAC algorithm, a global optimal plane in a point cloud image is calculated in an iterative mode, namely a fitting plane of the surface of the box body, the iterative calculation result contains four parameters A, B, C, D, and the fitted plane can be expressed as:
Figure 48187DEST_PATH_IMAGE055
and then mapping the coordinates of the pixel points in the region to be detected back to the three-dimensional point cloud data, traversing all the pixel points in the region, screening available point cloud data, and avoiding interference of useless data on a detection result. Calculating the distance between each available pixel point in the region and the fitting plane, and calculating the region to be detected according to a point-to-plane distance calculation formula
Figure 802517DEST_PATH_IMAGE057
The average distance d to the fitted plane can be expressed as:
Figure 8370DEST_PATH_IMAGE059
wherein the content of the first and second substances,
Figure DEST_PATH_IMAGE060
is the three-dimensional left information of the pixel points in the point cloud space,
Figure 640078DEST_PATH_IMAGE062
is a coordinate of the X-axis,
Figure 273184DEST_PATH_IMAGE063
is a coordinate of the Y axis and is a coordinate of the Y axis,
Figure 842706DEST_PATH_IMAGE065
is the Z-axis coordinate, n is the total number of effective pixel points in the area to be detected, and i is a specific pixel point.
And 5: and detecting the abnormity of the train fastener.
Generally, when the fastener is in an unfastened state, the cover plate of the fastener is in a tilted state, and the opening and closing angle of the cover plate from the surface of the box body is far larger than that of the fastener in a normal locking state (in a normal state, the surface of the fastener is almost parallel to the surface of the box body, and the opening and closing angle is small), so that the opening and closing angle is used as a judgment basis for the abnormal state of the fastener. As shown in fig. 2, the opening and closing angle of the geometric space between the surface of the fastener and the surface of the box body is further calculated, and the calculation formula of the angle is as follows:
Figure DEST_PATH_IMAGE067
wherein, the first and the second end of the pipe are connected with each other,
Figure DEST_PATH_IMAGE068
、
Figure DEST_PATH_IMAGE069
the average distances from the upper and lower detection areas to the fitting plane are respectively,
Figure DEST_PATH_IMAGE071
、
Figure DEST_PATH_IMAGE073
the coordinates of the upper point and the lower point of the Y axis of the fastener are shown.
And setting the maximum included angle between the cover plate of the fastener and the fitting plane (the surface of the box body) in the normal locking state as a threshold value, and judging that the locking state is normal if the opening and closing angle of the fastener to be detected is smaller than the threshold value of the included angle, otherwise, judging that the fastener is in an abnormal state.
Figure DEST_PATH_IMAGE075
If the fastener on-off state is detected to be abnormal, alarm information and positioning information are sent to prompt workers to overhaul in time, and more serious consequences are prevented.
The invention provides a complete solution for monitoring the state of the train fastener, the failure of the train in a real running environment is a small-probability event, fewer negative samples can be obtained, a whole set of algorithm designed by the invention does not need a large number of negative samples, the opening and closing state of the fastener is judged in a distance and angle detection mode by combining two-dimensional information and three-dimensional point cloud, and the judgment logic of the actual detection of a maintainer is more consistent.
The method carries out positioning through an improved YOLOv4 target detection algorithm based on deep learning, replaces a trunk characteristic extraction network with a lighter-weight MobilenetV2 to reduce the parameter quantity of the network, greatly improves the positioning speed on the premise of ensuring the positioning accuracy, tests under the CPU hardware environment, and only needs 0.15s for the positioning speed, meanwhile, the method based on deep learning has strong robustness after being trained by a large amount of data, and can still accurately position the target under the condition that the scene illumination condition changes and the shooting angle deviation is caused by the trolley positioning error.
The method is designed to calculate the distance between a point and a surface, further calculate a space included angle to represent the opening and closing state of the fastener, select a to-be-detected area capable of covering the whole fastener from a two-dimensional image through positioning, dividing and fitting straight lines, combine three-dimensional point cloud data acquired by a high-precision three-dimensional camera, map pixel points back to the point cloud data, skillfully utilize the surface smoothness characteristic of an electric box body, fit a global optimal plane in the point cloud based on an RANSAC algorithm, calculate the average distance between the to-be-detected area and the fitting plane, and test results show that the difference between a detection result and a true value is not more than 0.2mm, so that the accuracy of fastener detection and the reliable and stable operation of a train can be ensured.
Although the present invention has been described in detail with reference to the foregoing embodiments, it will be apparent to those skilled in the art that various changes in the embodiments and/or modifications of the invention can be made, and equivalents and modifications of some features of the invention can be made without departing from the spirit and scope of the invention.

Claims (4)

1. A train fastener abnormity detection method based on two-dimensional image to point cloud mapping is characterized by comprising the following steps:
s1, reading a two-dimensional image and three-dimensional point cloud image data of a fastener, which are acquired by acquisition equipment, from a server according to a shooting sequence;
s2, positioning a fastener in a two-dimensional image based on an improved YOLOv4 target detection algorithm;
s3, segmenting the fastener based on a region growing algorithm, and calculating the geometric center and the rotation angle of the fastener to obtain a region to be detected;
s4, fitting a plane in the point cloud based on a RANSAC algorithm, and respectively calculating the average distance between the pixel points in the two to-be-detected regions and the fitting plane;
s5, calculating a geometric space opening and closing angle of the fastener and the fitting plane;
s6, detecting the abnormity of the train fastener according to the size relation between the opening and closing angle and the threshold value;
in step S2, the method specifically includes:
s21, loading a trained improved YOLOv4 target detection model, and performing target detection on two-dimensional images of a train PA box fastener, a PH box fastener, a storage battery box fastener and a BOP box fastener according to the fastener types;
s22, intercepting a rectangular frame image of the target detection according to the target detection result and recording four vertex coordinates X of the rectangular frame min 、Y min 、X max 、X max ；
The improved YOLOv4 target detection model is specifically characterized in that in a trunk feature extraction network, CSPdark net53 is replaced by MobilenetV2, and the positioning speed is accelerated;
in step S4, the method specifically includes:
s41, calculating a global optimal plane in the point cloud image based on the RANSAC algorithm, wherein the calculation result comprises four parameters of A, B, C, D, and the fitting plane is expressed as:
Ax+By+Cz+D＝0；
s42, mapping the area to be detected back to three-dimensional point cloud data, and screening available point cloud data;
s43, calculating an upper detection area X and a lower detection area X i ,Y i ,Z i Mean distance d to the fitting plane 1 、d 2 According to the distance calculation formula from the point to the surface:
Figure FDA0003918995010000021
wherein, X i ,Y i ,Z i Is the three-dimensional left information, X, of a pixel point in a point cloud space i As a coordinate of the X axis, Y i, As a Y-axis coordinate, Z i Is a Z-axis coordinate;
the geometric space opening and closing angle of the fastener and the fitting plane is calculated according to the following formula:
Figure FDA0003918995010000022
wherein, d 1 、d 2 The average distances from the upper and lower detection areas to the fitting plane, Y min 、Y max The coordinates of the upper point and the lower point of the Y axis of the fastener are shown.
2. The train fastener anomaly detection method based on two-dimensional image-to-point cloud mapping according to claim 1, characterized in that: in step S3, the method specifically includes:
s31, selecting the center of the target detection rectangular frame as a seed point for region growth, and segmenting fastener parts in the intercepted image in the step S2;
s32, calculating the geometric center and the rotation angle of the segmented fastener image;
s33, fitting a straight line penetrating through the fastener based on the geometric center and the rotation angle, and selecting two end points in the directions of the upper end and the lower end of the fastener;
and S34, expanding the area to the periphery by taking the selected two end points as the center, and acquiring the area containing 4*4 pixel points as the area to be detected.
3. The train fastener anomaly detection method based on two-dimensional image-to-point cloud mapping according to claim 1, characterized in that: in the step S6, if the opening/closing angle is smaller than the threshold, it is determined that the fastener state is normal, otherwise, the fastener is in an abnormal state:
Figure FDA0003918995010000031
4. the train fastener anomaly detection method based on two-dimensional image-to-point cloud mapping according to claim 3, characterized in that: the threshold is specifically that the maximum included angle between the fastener cover plate and the fitting plane in the normal locking state is used as a threshold.
CN202211091685.6A 2022-09-07 2022-09-07 Train fastener abnormity detection method based on two-dimensional image-point cloud mapping Active CN115222731B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202211091685.6A CN115222731B (en) 2022-09-07 2022-09-07 Train fastener abnormity detection method based on two-dimensional image-point cloud mapping

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202211091685.6A CN115222731B (en) 2022-09-07 2022-09-07 Train fastener abnormity detection method based on two-dimensional image-point cloud mapping

Publications (2)

Publication Number Publication Date
CN115222731A CN115222731A (en) 2022-10-21
CN115222731B true CN115222731B (en) 2022-12-02

Family

ID=83617886

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202211091685.6A Active CN115222731B (en) 2022-09-07 2022-09-07 Train fastener abnormity detection method based on two-dimensional image-point cloud mapping

Country Status (1)

Country Link
CN (1) CN115222731B (en)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116386016B (en) * 2023-05-22 2023-10-10 杭州睿影科技有限公司 Foreign matter treatment method and device, electronic equipment and storage medium
CN117456285A (en) * 2023-12-21 2024-01-26 宁波微科光电股份有限公司 Metro shielding door foreign matter detection method based on TOF camera and deep learning model

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111122602A (en) * 2020-01-03 2020-05-08 重庆大学 Three-dimensional camera-based straddle type monorail finger-shaped plate abnormity detection system and method
CN112365461A (en) * 2020-11-06 2021-02-12 北京格灵深瞳信息技术有限公司 Fastener loosening identification method, system, terminal and storage medium
CN213163963U (en) * 2020-08-12 2021-05-11 国琳五金电子(苏州)有限公司 Assembly line device of fastener real-time quality detection system
CN112991347A (en) * 2021-05-20 2021-06-18 西南交通大学 Three-dimensional-based train bolt looseness detection method
CN113362467A (en) * 2021-06-08 2021-09-07 武汉理工大学 Point cloud preprocessing and ShuffleNet-based mobile terminal three-dimensional pose estimation method

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11030476B2 (en) * 2018-11-29 2021-06-08 Element Ai Inc. System and method for detecting and tracking objects
CN114037703B (en) * 2022-01-10 2022-04-15 西南交通大学 Subway valve state detection method based on two-dimensional positioning and three-dimensional attitude calculation

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111122602A (en) * 2020-01-03 2020-05-08 重庆大学 Three-dimensional camera-based straddle type monorail finger-shaped plate abnormity detection system and method
CN213163963U (en) * 2020-08-12 2021-05-11 国琳五金电子(苏州)有限公司 Assembly line device of fastener real-time quality detection system
CN112365461A (en) * 2020-11-06 2021-02-12 北京格灵深瞳信息技术有限公司 Fastener loosening identification method, system, terminal and storage medium
CN112991347A (en) * 2021-05-20 2021-06-18 西南交通大学 Three-dimensional-based train bolt looseness detection method
CN113362467A (en) * 2021-06-08 2021-09-07 武汉理工大学 Point cloud preprocessing and ShuffleNet-based mobile terminal three-dimensional pose estimation method

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
OPD: Single-view 3D Openable Part Detection;Hanxiao Jiang等;《https://arxiv.org/pdf/2203.16421.pdf》;20220331;1-22 *
基于改进YOLOv4的轻量化目标检测算法;钟志峰等;《计算机应用》;20220710;第42卷(第7期);2201-2209 *
基于深度学习的接触网紧固件缺陷检测算法研究;李子沁;《中国优秀硕士学位论文全文数据库 (工程科技Ⅱ辑)》;20220115(第01期);C033-84 *

Also Published As

Publication number Publication date
CN115222731A (en) 2022-10-21

Similar Documents

Publication Publication Date Title
CN115222731B (en) Train fastener abnormity detection method based on two-dimensional image-point cloud mapping
Dhiman et al. Pothole detection using computer vision and learning
Banić et al. Intelligent machine vision based railway infrastructure inspection and monitoring using UAV
CN109615611B (en) Inspection image-based insulator self-explosion defect detection method
CN110133440A (en) Electric power unmanned plane and method for inspecting based on Tower Model matching and vision guided navigation
CN105243381B (en) Failure automatic identification detection system and method based on 3D information
CN106290388A (en) A kind of insulator breakdown automatic testing method
CN114743119B (en) High-speed rail contact net hanger nut defect detection method based on unmanned aerial vehicle
CN111126381A (en) Insulator inclined positioning and identifying method based on R-DFPN algorithm
CN112197773B (en) Visual and laser positioning mapping method based on plane information
CN114004977A (en) Aerial photography data target positioning method and system based on deep learning
CN111046950A (en) Image processing method and device, storage medium and electronic device
CN110298842A (en) A kind of rail clip image position method based on super-pixel node sequencing
Wang et al. Autonomous landing of multi-rotors UAV with monocular gimbaled camera on moving vehicle
CN115063579A (en) Train positioning pin looseness detection method based on two-dimensional image and three-dimensional point cloud projection
Di Stefano et al. Automatic 2D-3D vision based assessment of the attitude of a train pantograph
Shajahan et al. Automated inspection of monopole tower using drones and computer vision
CN116977328B (en) Image quality evaluation method in active vision of vehicle bottom robot
CN113885504A (en) Autonomous inspection method and system for train inspection robot and storage medium
NL2035822A (en) Image reconstruction method for detecting defects of solar panels on warehouse ceiling
CN116524382A (en) Bridge swivel closure accuracy inspection method system and equipment
CN115909216A (en) Cargo ship hatch detection method and system based on laser radar and monocular camera
CN115077563A (en) Vehicle positioning accuracy evaluation method and device and electronic equipment
CN113963107A (en) Large target three-dimensional reconstruction method and system based on binocular vision
Li et al. SLAM of mobile robot based on the joint optimization of LiDAR and camera

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant