CN112949483B

CN112949483B - Non-contact rail stretching displacement real-time measurement method based on fast R-CNN

Info

Publication number: CN112949483B
Application number: CN202110226890.8A
Authority: CN
Inventors: 厉小润; 程嘉昊; 王森荣; 黎金辉; 王晶; 林超; 李秋义; 鄢祖建; 杨艳丽; ***
Original assignee: Zhejiang University ZJU; China Railway Siyuan Survey and Design Group Co Ltd
Current assignee: Zhejiang University ZJU; China Railway Siyuan Survey and Design Group Co Ltd
Priority date: 2021-03-01
Filing date: 2021-03-01
Publication date: 2022-05-03
Anticipated expiration: 2041-03-01
Also published as: CN112949483A

Abstract

The invention discloses a non-contact rail stretching displacement real-time measurement method based on Faster R-CNN, and belongs to the field of high-speed rail structure monitoring. The method comprises the following steps that a sign board which comprises at least two circles and is obviously different from the background in color is designed to be adhered to the side surface of the tail of the front section of rail and the side surface of the same side of the rear section of rail; calibrating the image to serve as a training sample of a deep learning model for signboard detection, and training the deep learning model; and detecting a signboard area in the image by using a depth learning model for the real-time image obtained at each detection point, filtering the invalid frame image according to the detection result, detecting the circle center in the signboard area of the rest valid frame images, roughly calculating the rail stretching displacement, and then performing smooth filtering on the roughly calculated result to realize the real-time measurement of the rail stretching displacement.

Description

Non-contact rail stretching displacement real-time measurement method based on fast R-CNN

Technical Field

The invention belongs to the field of high-speed rail structure monitoring, and particularly relates to a non-contact rail stretching displacement real-time measuring method based on fast R-CNN.

Background

In order to meet the requirements of various complex regional environments, the long and large continuous beam bridge cannot be avoided on the high-speed railway. In the use process of the steel rail telescopic regulator and the rail lifting device at the beam joint on the related line, the defects of inclined pulling crack of a sleeper, larger deformation of a scissors fork of the rail lifting device, even blockage and the like occur, and larger maintenance workload and economic loss are caused. Because the maintenance workload of the steel rail expansion adjuster is large, and the steel rail expansion adjuster is one of three weak links of a high-speed rail track structure, the monitoring requirement of a high-speed railway engineering department on the rail structure of the steel rail expansion adjuster area is very urgent.

The track displacement change monitoring is a key link of monitoring the track structure of a steel rail telescopic regulator area, and the current track displacement change monitoring mainly adopts a vibrating string type sensor and an optical fiber grating sensing mode from the aspects of precision and implementation. Currently, the main disadvantages of the existing monitoring methods include:

(1) generally, the method can only be used for monitoring on a working point, has a limited monitoring range in space and time, and cannot realize large-scale real-time measurement;

(2) all are contact sensing mode, and the sensor that sets up on track structure is great potential safety hazard to the operation of high-speed motor car.

With the increase of bandwidth of railway networks in geometric orders of magnitude, high-speed railways are gradually installing cameras along the lines. The monitoring research of the high-speed rail structure by taking a camera along the high-speed rail as the front end and taking non-contact sensing such as image recognition and the like as means has very important practical significance and application prospect.

However, due to the complex field environment of the high-speed rail, the imaging result is affected by various factors such as illumination, wind and rain, train vibration and the like, and the perspective distortion of the camera, the rail extension displacement measured based on the simple image recognition method cannot meet the requirement of the rail structure monitoring precision.

Disclosure of Invention

Aiming at the structural characteristics that the high-speed railway track is respectively in a longitudinal belt shape and is distributed in a vertical layered manner, the invention provides a non-contact type rail stretching displacement real-time measurement method based on fast R-CNN, wherein a sign board which comprises at least two circles and has obvious color difference with the background is designed and is adhered to the side surface of the tail part of the previous section of rail and the side surface of the same side of the next section of rail; calibrating the image to be used as a training sample of a deep learning model for signboard detection, and training the deep learning model; and detecting a signboard area in the image by using a deep learning model for the real-time image obtained at each detection point, filtering the invalid frame image according to the detection result, detecting the circle center in the signboard area of the rest valid frame images, roughly calculating the rail stretching displacement, and then performing smooth filtering on the roughly calculated result to realize the real-time measurement of the rail stretching displacement.

In order to achieve the purpose, the invention adopts the technical scheme that:

a non-contact rail stretching displacement real-time measurement method based on fast R-CNN comprises the following steps:

step1, aiming at any detection point, arranging a camera at a fixed position on one side of a pair of rail joints to be detected, so that the pair of rail joints are positioned in a target monitoring range, respectively sticking a signboard at least comprising two circles on the side surface of the tail part of a front section of rail and the side surface of the same side of a rear section of rail, wherein all circle centers on the signboard are on the same straight line, and the connection line of the circle centers is parallel to the extending direction of a track;

step2, collecting sample images under different working conditions in a target detection range, marking original pixel coordinates of four corner points of each marking plate in the sample images, and converting the original pixel coordinates of the corner points into original pixel coordinates of the top points of the detection frame; the original pixel coordinates of two vertexes on the diagonal line of the rectangular detection frame are used as a signboard detection label, and a sample image with the label is used as a training sample set;

step3, establishing a deep learning model, and training the deep learning model by using the training sample set obtained in the step 2;

and 4, step 4: acquiring a rail video at each detection point in real time through a camera, extracting m frames at equal intervals from the video corresponding to the current moment to serve as an image to be detected at the current moment, detecting a signboard area in the image by using a trained deep learning model, and filtering an invalid frame image according to a detection result; detecting the circle center in the detected signboard area, roughly calculating the rail expansion displacement corresponding to each effective frame image, and taking the average value as an initial detection result;

and 5: sequencing and smoothly filtering the initial detection results of all effective frame images, and outputting the filtered average value as a final result;

step 6: and (5) repeating the steps 4 to 5, and executing the detection at the next moment to realize the real-time measurement of the rail stretching displacement.

Compared with the prior art, the invention has the advantages that: the invention designs a sign board which comprises at least two circles and has obvious color difference with the background, and the sign board is adhered to the side surface of the tail part of the previous section of rail and the side surface of the same side of the next section of rail; calibrating the image to serve as a training sample of a deep learning model for signboard detection, and training the deep learning model; and detecting a signboard area in the image by using a deep learning model for the real-time image obtained at each detection point, filtering the invalid frame image according to the detection result, detecting the circle center in the signboard area of the rest valid frame image, roughly calculating the rail extension displacement, and then carrying out smooth filtering on the roughly calculated result to realize the real-time measurement of the rail extension displacement.

The method is based on a non-contact measurement mode, does not affect the normal work of the rail, is high in safety, can be applied to large-scale real-time measurement of the rail extension displacement of the steel rail, and can be popularized to scenes of distance detection by using signboards, such as stock rail-stock rail, stock rail-switch rail extension displacement and the like.

Drawings

FIG. 1 is a flow chart of a non-contact rail stretching displacement real-time measurement method based on Faster R-CNN according to the present invention;

FIG. 2 is a signboard drawing view in the present embodiment;

fig. 3 is a structural diagram of a deep learning model employed in the present embodiment;

FIG. 4 is a test chart including 2 signboards in the present embodiment;

FIG. 5 is a diagram illustrating the result of image preprocessing before ellipse fitting in the present embodiment;

fig. 6 is a diagram showing the detection results of 2 signboards and 6 circle centers and the detection results of the distance in the present embodiment. .

Detailed Description

In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is described in detail below with reference to specific examples. Specific embodiments are described below to simplify the present disclosure. It is to be understood that the invention is not limited to the embodiments described and that various modifications thereof are possible without departing from the basic concept, and that such equivalents are intended to fall within the scope of the claims appended hereto.

The technical solution in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. All other embodiments, which can be obtained by a person skilled in the art without making any creative effort based on the embodiments in the present invention, belong to the protection scope of the present invention.

As shown in FIG. 1, the non-contact rail stretching displacement real-time measurement method based on Faster R-CNN provided by the invention mainly comprises the following steps:

and 4, step 4: acquiring a rail video at each detection point in real time through a camera, extracting m frames from the video corresponding to the current moment at medium intervals to serve as an image to be detected at the current moment, detecting a signboard area in the image by using a trained deep learning model, and filtering an invalid frame image according to a detection result; detecting the circle center in the detected signboard area, roughly calculating the rail expansion displacement corresponding to each effective frame image, and taking the average value as an initial detection result;

The following describes a specific implementation manner, and the working condition parameters in this embodiment are: the resolution of the camera is 400 ten thousand pixels, the imaging height is 2.5m, and the imaging is carried out by visible light in the daytime.

Designing a marking plate.

As shown in fig. 2, a black matrix signboard containing 3 white circles is designed, and centers of the 3 circles are aligned. The method comprises the following steps of attaching signboards to the tail ends of a pair of rails, wherein the telescopic displacement distance is to be measured, attaching the signboards to the side faces of the rails, and enabling the circle center connecting line of the signboards to be parallel to the rails during attaching.

And (II) calibrating the deep learning model training sample.

Acquiring sample images under different working conditions within a target detection range, wherein fig. 4 is a sample image acquired in the embodiment, each image includes 2 marker boards, each marker board includes 3 circles, and for each sample image:

(2.1) manually marking four corner points of the signboard in the sample picture, and then converting coordinates of the four corner points into vertex coordinates of a rectangular detection box of the signboard. Recording original pixel coordinates (x (i), y (i)) of four corner points of a certain signboard in the sample picture, wherein i is 1,2,3 and 4;

(2.2) converting the original pixel coordinates of the four corner points into the original pixel coordinates of the vertex of the detection frame, wherein the conversion formula is as follows:

in the formula, P_lefttopTo detect the original pixel coordinates of the top left vertex of the frame, P_rightdownThe original pixel coordinates of the right lower vertex of the detection frame are x (i) and y (i), which are the original pixel coordinates of the ith angular point respectively;

the original pixel coordinates of the top left corner vertex and the bottom right corner vertex of the rectangular detection frame calibrated in the way are used as the detection labels of the signboards during the training of the deep learning model.

And (III) training a deep learning model.

And taking the sample marked in the step two as a training sample, and training the marker detection model based on deep learning. Wherein the deep learning model adopts a Faster R-CNN model pre-trained on a COCO train2017 data set. And (3) inputting the model during training into the sample calibrated in the step (2) and subjected to data amplification, and outputting the model into the top left corner and the bottom right corner vertex coordinates of the rectangular detection frame of the signboard in the input picture and the confidence coefficient of the coordinates of the signboard detection frame.

The feature extraction backbone networks of the model are ResNet50 and FPN, and the structure is shown in FIG. 3. And (3) performing data augmentation on the sample obtained by the calibration in the step (2), and then taking one part of the sample as a training set and one part of the sample as a verification set, wherein the data augmentation mode of the sample comprises random horizontal overturning, random cutting and random change of hue, contrast and brightness. The optimizer used in model training is Adam, and the learning rate adjustment strategy is a Cosine mode. And testing indexes on the verification set when the model trains 1 epoch on the training set, and finishing the training when the indexes of the verification set do not rise any more.

And (IV) roughly calculating the track expansion and contraction displacement.

4.1) acquiring a rail video at each detection point in real time through a camera, and extracting m frames from the video corresponding to the current moment at medium intervals to serve as an image to be detected at the current moment;

4.2) detecting the signboard region in the image by using the trained deep learning model, wherein the detection result is shown in FIG. 6, and invalid frame images are filtered according to the detection result, and the invalid frame images meet the following conditions: the number of the detected signboard areas by using the trained deep learning model is less than 2, and the confidence coefficient is greater than 90.

4.3) detecting the circle center in the detected signboard area, comprising:

4.3.1) carrying out median filtering on the image to reduce image noise; in this embodiment, the convolution kernel of size 5 is used to perform median filtering on the image;

4.3.2) converting the image after median filtering into a gray scale map, wherein the pixel gray scale value calculation formula is as follows:

Gray(x,y)＝min(R,G,B)

in the formula, R, G, B are three channels of the color space, Gray (x, y) is the Gray value at the pixel point (x, y) of the original image;

4.3.3) binarizing the gray level image:

the binarization of the gray level image is realized by setting a certain threshold, setting the pixel point with the gray level lower than the threshold as 0, and setting the point with the gray level higher than the threshold as 255, so that the image has only two gray levels, and the binarization image is obtained by the following formula:

in the formula, Binary (x, y) is the gray value of the Binary image at the pixel point (x, y), and T is a set threshold;

in the binarization process of the gray level image, a proper threshold value is selected as a relatively critical part, the weather of a shooting environment and the gray level value of a shot picture can be influenced day and night, and the gray level threshold values which need to be set are different, so that the dynamic average local threshold value is selected to carry out binarization on the image in the patent, and the binarization effect can be good.

4.3.4) removing black and white connected regions with area smaller than the threshold value in the binary image, and performing opening and closing operation with the core size of 5 × 5 to make the edges of the connected regions smoother; in this embodiment, the small black and white connected regions with an area smaller than 400 in the binarized image are removed, and the opening and closing operation with a core size of 5 × 5 is performed to make the edges of the connected regions smoother.

4.3.5) extracting the contour of the processed binary image, and screening the contour, wherein the screening process comprises the following steps:

STEP1, removing the outline with the number of vertexes less than or equal to 25;

STEP2. removing the outline of the surrounding area which is a black area; the effect obtained by this example is shown in fig. 5.

STEP3, carrying out ellipse fitting on each contour based on a least square method, and removing the contour of which the ratio of the ellipse area obtained by fitting to the actual area of the contour is more than or equal to 1.1 or less than or equal to 0.9;

STEP4, removing the outline of which the ratio of the ellipse perimeter obtained by fitting to the actual perimeter of the outline is more than or equal to 1.15 or less than or equal to 0.85;

STEP5, performing convex hull treatment on each contour, wherein the ratio of the area after convex hull removal to the actual area of the contour is more than or equal to 1.1 or less than or equal to 0.9;

STEP6. fitting circle center:

if the number of the remaining outlines in each signboard area is less than 3, the detection fails, and the image is deleted; otherwise, randomly selecting 3 contours from the rest contours, performing principal component analysis on coordinates of the circle centers of the ellipses obtained by fitting the contours, selecting 3 contours with the largest variance ratio as 3 circle center areas in the detected signboards in each combination in a trial mode, and using the circle center coordinates obtained after ellipse fitting of the 3 contours as fitting results.

The method in which the least squares method is used for ellipse fitting is as follows:

the equation for an arbitrary ellipse on a plane can be expressed as:

x²+Axy+By²+Cx+Dy+E＝0

assuming Pi (xi, yi) (i is 1,2,3 … … N, N ≧ 5) is N points on the edge of the ellipse, the objective function fitted according to the least squares principle is:

when the objective function F is 0 with respect to the bias of A, B, C, D, E, which takes the minimum value, a coefficient A, B, C, D, E may be calculated.

According to A, B, C, D, E, the center (x0, y0), the major semiaxis a, the minor semiaxis b and the rotation angle theta of the ellipse parameters are obtained through calculation, and the calculation formula is as follows:

of the resulting ellipse parameters, (x)₀,y₀) Namely the detected center coordinates.

4.4) roughly calculating the rail expansion displacement corresponding to each effective frame image according to the fitted circle center coordinates, so that the distance between the two sections of rails can be represented by the average distance between the centers of 3 pairs of circles on the signboards stuck on the two sections of rails. The distance between the centers of the left circle and the right circle of the signboard is d₀The specific calculation steps are as follows:

translating the centers of 3 circles detected by the left signboard in the y-axis direction to enable the average y-coordinate values of the centers of the circles detected by the left and right signboards to be equal;

calculating the average value d of the pixel distance between 3 pairs of circle centers after translation₁；

Calculating the average value d of the pixel distance between the centers of the leftmost circle and the rightmost circle of the left signboard and the right signboard₂；

The distance d between the two rails can be calculated by:

in the formula (d)₀The distance between the centers of the left and right boundary circles of the signboard is shown.

The expansion displacement distance of the rail can be obtained by subtracting the initial rail distance from the current rail distance, and the average value of the expansion displacement quantity of the rail corresponding to each effective frame image is used as an initial detection result.

And (V) precisely calculating the rail expansion displacement.

Sequencing and smoothly filtering the initial detection results of all effective frame images, and outputting the filtered average value as a final result; the smoothing filtering specifically includes: and for the sorted initial detection results, removing the maximum value of the front p% and the minimum value of the last p% and then calculating an average value, wherein the average value is used as a smooth filtering result, and p is a threshold value. In this example, p is 10%.

In the subsequent real-time detection process, the fifth to sixth steps are repeatedly executed, and the real-time measurement of the rail extension displacement can be realized.

Compared with the prior art, the method provided by the embodiment of the invention realizes a non-contact type rail stretching displacement real-time measurement method based on the Faster R-CNN, and designs the marker plate which comprises at least two circles and has obvious color difference with the background, and the marker plate is adhered to the side surface of the tail part of the previous section of rail and the side surface of the same side of the next section of rail; calibrating the image to serve as a training sample of a deep learning model for signboard detection, and training the deep learning model; and detecting a signboard area in the image by using a deep learning model for the real-time image obtained at each detection point, filtering the invalid frame image according to the detection result, detecting the circle center in the signboard area of the rest valid frame images, roughly calculating the rail stretching displacement, and then performing smooth filtering on the roughly calculated result to realize the real-time measurement of the rail stretching displacement.

The objects, technical solutions and advantages of the present invention will be more clearly described by the accompanying drawings shown in the embodiments of the present invention. It should be noted that the specific embodiments described herein are merely illustrative of the invention and do not limit the invention. All equivalents, modifications, and the like which come within the spirit and scope of the principles and concepts of the invention are desired to be protected.

Claims

1. A non-contact rail stretching displacement real-time measurement method based on fast R-CNN is characterized by comprising the following steps:

2. The non-contact rail expansion and contraction displacement real-time measurement method based on Faster R-CNN as claimed in claim 1, wherein the color inside the three circular areas on the signboard is significantly different from the color outside the circular areas.

3. The method for measuring non-contact rail extension displacement in real time based on Faster R-CNN as claimed in claim 1, wherein the deep learning model is fast R-CNN pre-trained on COCO train2017 data set.

4. The method for measuring the non-contact rail stretching displacement in real time based on the Faster R-CNN as claimed in claim 1, further comprising the step of data augmentation of the sample image in the training sample set between the step2 and the step3, wherein the step comprises random horizontal inversion, random cropping, and random change of hue, contrast and brightness.

5. The method for measuring the non-contact rail expansion and contraction displacement based on Faster R-CNN as claimed in claim 1, wherein the step S2 comprises:

2.1) acquiring sample images in different working conditions in a target detection range, and marking original pixel coordinates (x (i), y (i)) of four corner points of each marking plate in the sample images, wherein i is 1,2,3 and 4;

2.2) converting the original pixel coordinates of the four corner points into the original pixel coordinates of the vertex of the detection frame, wherein the conversion formula is as follows:

2.3) using the original pixel coordinates of two vertexes on the diagonal line of the rectangular detection frame as the signboard detection label.

6. The non-contact rail stretching displacement real-time measurement method based on Faster R-CNN as claimed in claim 1, wherein the invalid frame image in step4 satisfies the following conditions: the number of the detected signboard areas by using the trained deep learning model is less than 2, and the confidence coefficient is greater than 90.

7. The non-contact rail stretching displacement real-time measurement method based on fast R-CNN as claimed in claim 1, wherein the circle center is detected in the detected signboard area in step4, specifically:

4.1) carrying out median filtering on the image to reduce image noise;

4.2) converting the image after median filtering into a gray scale map, wherein the pixel gray scale value calculation formula is as follows:

Gray(x，y)＝min(R，G，B)

4.3) binarizing the gray level image:

respectively setting screening intervals for R, G, B channels, judging each pixel on the image one by one, setting the pixel of 3 components in the interval range as 255, and otherwise, setting the pixel as 0 to obtain a binary image, wherein the formula is as follows:

4.4) removing black and white connected regions with the area smaller than the threshold value in the binary image, and performing opening and closing operation with the core size of 5 x 5 to enable the edges of the connected regions to be smoother;

4.5) extracting the contour of the processed binary image, and screening the contour, wherein the screening process comprises the following steps:

a) removing the outline with the number of the vertex points less than or equal to 25;

b) removing the outline of the surrounding area which is a black area;

c) carrying out ellipse fitting on each contour based on a least square method, and removing the contour of which the ratio of the ellipse area obtained by fitting to the actual area of the contour is more than or equal to 1.1 or less than or equal to 0.9;

d) removing the outline of which the ratio of the ellipse perimeter obtained by fitting to the actual perimeter of the outline is more than or equal to 1.15 or less than or equal to 0.85;

e) carrying out convex hull treatment on each contour, wherein the ratio of the area after convex hull removal to the actual area of the contour is more than or equal to 1.1 or less than or equal to 0.9;

4.6) fitting circle center:

8. The non-contact rail stretching displacement real-time measurement method based on fast R-CNN as claimed in claim 7, wherein the rail stretching displacement corresponding to each effective frame image is roughly calculated according to the fitted circle center coordinates, specifically:

The distance d between the two rails can be calculated by:

in the formula, d₀The distance between the centers of the left and right boundary circles of the signboard is shown.

9. The non-contact rail stretching displacement real-time measurement method based on fast R-CNN as claimed in claim 1, wherein the smoothing filtering in step5 is specifically: and for the sorted initial detection results, removing the maximum value of the front p% and the minimum value of the last p% and then calculating an average value, wherein the average value is used as a smooth filtering result, and p is a threshold value.