CN114565596A - Steel surface crack detection and prediction method based on deep learning and video understanding - Google Patents

Steel surface crack detection and prediction method based on deep learning and video understanding Download PDF

Info

Publication number
CN114565596A
CN114565596A CN202210212511.4A CN202210212511A CN114565596A CN 114565596 A CN114565596 A CN 114565596A CN 202210212511 A CN202210212511 A CN 202210212511A CN 114565596 A CN114565596 A CN 114565596A
Authority
CN
China
Prior art keywords
crack
video
steel surface
characteristic points
points
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202210212511.4A
Other languages
Chinese (zh)
Inventor
高铁山
余倩倩
刘金杉
卢昱杰
张伟平
顾祥林
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Tongji University
Original Assignee
Tongji University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Tongji University filed Critical Tongji University
Priority to CN202210212511.4A priority Critical patent/CN114565596A/en
Publication of CN114565596A publication Critical patent/CN114565596A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/044—Recurrent networks, e.g. Hopfield networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10016—Video; Image sequence
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
    • Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
    • Y02P90/30—Computing systems specially adapted for manufacturing

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • General Health & Medical Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Biomedical Technology (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Biophysics (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Quality & Reliability (AREA)
  • Multimedia (AREA)
  • Image Analysis (AREA)

Abstract

The invention provides a steel surface crack detection and prediction method based on deep learning and video understanding, which comprises the following steps of: step 1, acquiring and storing a video of crack opening and closing of a crack on the surface of steel under the action of fatigue load, and editing the stored video; step 2, training the SuperPoint network through videos, tracking and recording motion information of characteristic points of the steel surface in the videos; step 3, screening out feature points around the crack by a threshold-based adaptive feature point screening method; step 4, fitting the form of the crack by taking the detected characteristic points as basic information to obtain basic parameter information of the crack and digitalizing the basic parameter information; and 5, taking historical development information of the cracks, basic parameter information of the cracks and basic information of the steel surface as input, taking the development condition of the cracks after N cycles as output to train an LSTM neural network, and predicting the crack development rate of the steel surface in real time through the LSTM neural network after the training is finished.

Description

Steel surface crack detection and prediction method based on deep learning and video understanding
Technical Field
The invention belongs to a steel plate apparent crack identification and detection and development prediction method, and particularly relates to a steel surface crack detection and prediction method based on deep learning and video understanding.
Background
In recent years, the overall economic level of China is continuously improved, the comprehensive national force is gradually enhanced, the infrastructure construction of China as an important fulcrum for pulling economy shows a vigorous trend, the construction scale of steel structure bridges is continuously large, and the number and the quantity of large-span bridges are continuously expanded. However, the environment of the bridge in the construction, maintenance and operation stages is very complicated, so that quality problems sometimes occur in the using process. Among them, cracks are one of the most common defects of structural members.
With the progress of the technology, the traditional method for detecting and evaluating the bridge cracks through manual work is gradually replaced by a new method due to the defects of high subjectivity of results, higher requirements on professional experience of detection personnel, lower precision and efficiency and the like. With the rapid development of computer technology, the deep learning method gradually replaces the traditional image processing technology to become the mainstream method of image classification, target detection and instance segmentation, is widely applied to the detection and monitoring work in the civil engineering industry, and has achieved a lot of achievements. The bridge crack detection method based on deep learning is also applied to the building industry, and has the advantages of being more accurate and convenient than manual work. In addition, the characteristic point tracking technology based on deep learning is developed and advanced, but there are many blanks in the research on the aspect of detection and prediction of the apparent cracks of the steel bridge based on the characteristic point tracking.
A journal article 'research on metal surface microwave nondestructive inspection method' of the air force radar academy 2001 provides a nondestructive inspection method for detecting cracks on a metal surface by using a coaxial probe in a microwave frequency band. However, the system needs an additional microwave signal generator, the equipment is complex, and the coaxial probe has a small size, which brings certain difficulty to the detection work.
The invention discloses a metal surface defect image identification nondestructive testing device and method (application publication number CN107782733A) disclosed in 2017 of seventh-ninth research institute of China Ship re-engineering group company, and provides the metal surface defect image identification nondestructive testing device. The method has the advantages that the equipment which is arranged on the mobile unit is used for detecting a certain area, the detection range is large, the detection method is simple and quick, the detection time and the labor cost are reduced, and the influence of the subjectivity of detection personnel on the detection result is eliminated. However, the algorithm depends on parameters set manually, and the adaptability is poor.
A journal article 'infrared thermal imaging nondestructive testing of fatigue cracks of steel bridges', published in 2016 of Wuhan science and technology university, utilizes the constitutive relation between the temperature change of the steel surface and the stress thereof, and adopts an infrared thermal imaging method to test the distortion temperature of the hole edge of the component so as to determine the cracks of the component. The infrared detection has the advantages of non-contact, high efficiency, simple operation, no coupling and the like. But the detection effect is easily affected by the shape, position, etc. of the defect.
The research on the crack detection and evaluation of the steel box girder based on the metal magnetic memory method is experimentally researched in the paper of the university of Chongqing traffic 2017 Master' Steel box girder crack detection test research based on metal magnetic memory, and a new magnetic memory detection method for the crack of the steel box girder is provided. The method overcomes the defects of the traditional nondestructive detection and can diagnose the stress concentration in the ferromagnetic metal component. However, the size of the probe has a large impact on the accuracy of the results.
Journal article "microstrip antenna sensor for metal crack detection and measurement" 2020 of north and middle university proposes a metal crack detection and characterization sensor based on microstrip antenna radiation principle. The sensor is made of high-temperature-resistant alumina ceramic and metal silver paste through a screen printing process and can be applied to a high-temperature and high-pressure severe environment. And has the detection capability of submillimeter cracks. However, this approach still requires manual field installation of the sensor, and once installed, the sensor is difficult to relocate to other structures.
A journal article of 2021-year-old university of northeast forestry, namely bridge crack detection research based on a convolutional neural network, provides a bridge crack detection technology based on an unmanned aerial vehicle on the basis of the convolutional neural network. The method is superior to the traditional image processing technology in effect, and can be used for crack identification of roads, house buildings and the like. However, since the method only uses the static features of the object, the detection accuracy can be limited to the pixel level only.
A journal article 2021 of Zhejiang university, namely 'feature pyramid network-based oversized image crack identification and detection method' provides an automatic crack detection method based on a feature pyramid deep learning network. The method has no strict requirement on the specific resolution of the crack image, and provides a potential solution for future crack detection. However, this method still only utilizes the static characteristics of the fracture.
The university of electronic technology, the thesis of acoustic sensing array detection technology for stress crack events inside metal parts, in 2021, provides a method for detecting crack events in real time by monitoring acoustic signals during processing with an acoustic emission monitoring technology, which can detect 20 μm cracks. However, the method has high requirements on the acoustic emission device and is not friendly to the application scene of the actual engineering. Furthermore, this method does not allow quantitative estimation of crack width from the defect signal. Many studies have been made on image shape determination methods based on computer vision, but in the field of building engineering, research on the identification and detection of water leakage mainly focuses on determining the severity of water leakage by calculating the area of the water leakage, and performing temporal sequence prediction on temporary unmanned hunting based on various neural network structures.
In summary, the non-destructive testing method for the steel surface crack mostly remains to be applied to the material characteristics such as sound, light, electricity, magnetism and the like, and the dynamic information of the crack opening and closing is rarely used. However, the prediction of crack development is mostly limited to the recognition of physical phenomena, and the flexibility is poor. As the field of deep learning and video understanding develops, more algorithms are proposed for processing of video and prediction for the future. These new algorithms can be applied in the field of steel surface crack detection and prediction.
Disclosure of Invention
The present invention is made to solve the above problems, and an object of the present invention is to provide a steel surface crack detection and prediction method based on deep learning and video understanding.
The invention provides a steel surface crack detection and prediction method based on deep learning and video understanding, which is characterized by comprising the following steps of: step 1, acquiring and storing a video of crack opening and closing of a crack on the surface of steel under the action of fatigue load, and editing the stored video;
step 2, training a SuperPoint network through videos, tracking motion information of characteristic points of the steel surface in the videos and recording the motion information;
step 3, screening out feature points around the crack according to the displacement difference between adjacent feature points by a threshold-based self-adaptive feature point screening method;
step 4, fitting the form of the crack by taking the detected characteristic points as basic information to obtain basic parameter information of the crack and digitalizing the basic parameter information;
and 5, taking historical development information of the cracks, basic parameter information of the cracks and basic information of the steel surface as input, taking the development condition of the cracks after N cycles as output to train an LSTM neural network, and predicting the crack development rate of the steel surface in real time through the LSTM neural network after the training is finished.
In the steel surface crack detection and prediction method based on deep learning and video understanding provided by the invention, the method can also have the following characteristics: the step 1 comprises the following substeps:
step 1-1, shooting a video by using a camera, and selecting camera equipment and adjusting related parameters to optimize imaging according to the distance between a shot object and the camera, the illumination of the environment where the shot object is located and the shooting angle when shooting the video;
step 1-2, calibrating a camera, acquiring internal parameters of the camera, keeping the internal parameters of the camera unchanged, erecting the camera beside a steel surface containing cracks to shoot a video of the steel surface under the action of a working load, and measuring a shooting distance and a shooting inclination angle;
and 1-3, editing the video, wherein the length of each video segment is ten seconds, and each video segment comprises 5 vibration cycles.
In the steel surface crack detection and prediction method based on deep learning and video understanding provided by the invention, the method can also have the following characteristics: in the step 1-2, calibrating the camera comprises the following substeps:
step 1-2-1, printing a checkerboard, and pasting the checkerboard on a plane as a calibration object;
step 1-2-2, shooting photos in different directions for the calibration object by adjusting the direction of the calibration object or the camera;
step 1-2-3, extracting angular points of the checkerboard from the picture;
step 1-2-4, estimating five internal parameters and six external parameters of a camera under the ideal distortion-free condition;
1-2-5, estimating a distortion coefficient under the actual radial distortion by using a least square method;
step 1-2-6, optimizing internal parameters, external parameters and distortion coefficients by a maximum likelihood method, improving estimation precision,
wherein, adopt opencv to carry out when carrying out the calibration to the camera, the concrete process is as follows:
circularly reading the photos;
detecting the corner points of the checkerboard through a findChessboardCorrers function;
sub-pixel refinement is carried out on the corner points through a find4QuadCornerSubpix function;
corner points are displayed by drawChessboss detectors;
calibrating according to the ideal coordinates and the actual image coordinates through a calibretacarama function to obtain internal parameters, external parameters and distortion coefficients;
the backprojection error is calculated by the projectPoints function.
In the steel surface crack detection and prediction method based on deep learning and video understanding provided by the invention, the method can also have the following characteristics: the step 2 comprises the following substeps:
step 2-1, performing preliminary learning on the SuperPoint network on a virtual training set through the MagicPoint;
step 2-2, performing homography transformation on the screenshot of the video, and further training a SuperPoint network;
step 2-3, searching feature points of the steel surface in the video through a VGG neural network, and calibrating the feature points on the steel surface in the video by taking the searched feature points as a basis;
and 2-4, the SuperPoint network tracks the movement of all the characteristic points and records the position information and the movement information of the characteristic points according to the characteristic points obtained by training and tracking.
In the steel surface crack detection and prediction method based on deep learning and video understanding provided by the invention, the method can also have the following characteristics: the method comprises the following steps of screening out characteristic points around the crack by a threshold-based self-adaptive classifier in step 3:
step 3-1, calculating the average value of the relative displacement between cracks on the whole steel surface according to the distance between the characteristic points and the relative displacement;
3-2, judging that a crack exists between two adjacent characteristic points when the comparative displacement between the two adjacent characteristic points is more than 2 times of the average value, and screening the two points from all the characteristic points;
step 3-3, repeating the steps 3-1 to 3-2, screening out all characteristic points around the crack,
wherein, the comparison displacement d' is the ratio of the relative displacement d between two adjacent characteristic points and the displacement s between two adjacent characteristic points, and the formula is as follows:
Figure BDA0003532290060000071
in the steel surface crack detection and prediction method based on deep learning and video understanding provided by the invention, the method can also have the following characteristics: step 4 comprises the following substeps:
step 4-1, connecting the middle points of two adjacent characteristic points to fit a curve of the crack, and properly prolonging the curve;
4-2, setting the displacement difference between two adjacent characteristic points as the opening degree of the crack;
and 4-3, taking the opening degree between two adjacent characteristic points as input, taking the final extension length of the crack as output, training a BP (back propagation) neural network, calculating to obtain the real length of the crack and the position of the tip of the fitted crack through the BP neural network, and obtaining the length of the crack, the opening size of the crack under load and the inclination angle of the crack at a specific position.
Action and Effect of the invention
According to the steel surface crack detection and prediction method based on deep learning and video understanding, the determination of the position of the crack on the steel surface is completed for the first time through the characteristic point tracking, the digital expression of the basic parameter information of the crack is completed through the proposed self-adaptive threshold algorithm, and the prediction of the future development of the crack is completed through the existing information. According to the method, the existence of the crack is detected and the crack development trend is predicted according to the existing video information of the crack opening and closing of the steel surface by establishing the characteristic point tracking and crack growth prediction model based on deep learning, and some defects of the existing crack evaluation and prediction method can be made up.
Drawings
FIG. 1 is a schematic diagram of a hardware device in an embodiment of the invention;
FIG. 2 is a flow chart of a steel surface crack detection and prediction method based on deep learning and video understanding in an embodiment of the invention;
FIG. 3 is a schematic diagram of feature point tracking and screening in an embodiment of the invention;
FIG. 4 is a schematic diagram of crack length and dip calculations by comparing displacement differences of feature points in an embodiment of the present invention;
FIG. 5 is a schematic diagram of the embodiment of the present invention, in which the fracture morphology and the fracture basic parameter information are fitted according to the screened feature point positions and the displacement information;
fig. 6 is a schematic input and output diagram of the LSTM neural network during training in the embodiment of the present invention.
Detailed Description
In order to make the technical means and functions of the present invention easy to understand, the present invention is specifically described below with reference to the embodiments and the accompanying drawings.
< example one >
Fig. 1 is a schematic diagram of a hardware device in a first embodiment of the present invention.
As shown in fig. 1, the hardware device in this embodiment includes a CCD industrial camera 1, a microscope lens 2, a light supplement lamp 3, a transverse limit beam 4, a Z-direction slide rail 5, a Y-direction slide rail 6, an X-direction slide rail 7, an X-direction slide groove 8, a steel plate 9 to be measured, a central circular hole 10, a linear cutting seam 11, and a pre-split seam 12.
In the embodiment, a steel plate with the size of 1170x90x10mm is selected as a steel plate sample to be detected;
pre-perforating the geometric center of the steel plate, wherein the diameter of the perforated round hole is 5 mm;
pre-opening an initial crack in the direction perpendicular to the length direction of the steel plate and in the collinear direction of the diameter of the pre-opening round hole;
the initial loss of the initial crack was 30%, i.e. the initial gap length was 16 mm. Before the fatigue test is formally carried out, the steel plate containing the initial damage is subjected to fatigue pre-cracking, so that a fatigue crack of 1mm is generated at the end part of a notch of the steel plate.
Drawing calibration paper, placing the calibration paper at the pre-splitting cut in parallel, placing a high-definition optical high-power microscope right above the pre-splitting cut, aligning and connecting a microscope power supply;
after the observation distance is adjusted, calibrating the microscope to be used for measuring the size of the follow-up crack;
and (3) measuring the cracks by using a calibrated microscope, and respectively measuring the sizes of the cracks under the action of no load and 135KN static load for subsequent reference.
Fig. 2 is a flowchart of a steel surface crack detection and prediction method based on deep learning and video understanding in the first embodiment of the present invention.
As shown in fig. 2, the method for detecting and predicting the steel surface crack based on deep learning and video understanding of the embodiment includes the following steps:
step 1, fatigue loading is carried out on a steel plate, the frequency is 0.5Hz, the loading amplitude is 13.5KN-135KN, video shooting is carried out by using a calibrated camera at a distance of 1000mm from a test piece, the shot video is clipped, and the length of each video section after clipping is 10 seconds and comprises 5 vibration periods.
In the step 1, opencv is adopted for camera calibration, and the specific process is as follows:
circularly reading the photos;
detecting the corner points of the checkerboard through a findChessboardCorrers function;
sub-pixel refinement is carried out on the corner points through a find4QuadCornerSubpix function;
corner points are displayed by drawChessboss detectors;
calibrating according to the ideal coordinates and the actual image coordinates through a calibretacarama function to obtain internal parameters, external parameters and distortion coefficients;
the backprojection error is calculated by the projectPoints function.
Fig. 3 is a schematic diagram of feature point tracking and screening according to an embodiment of the present invention.
As shown in fig. 3, the motion information of the feature points of the steel surface in the video is tracked through a SuperPoint network, and the feature points around the crack are screened out through a threshold-based adaptive feature point screening method, which specifically includes the following steps:
and 2, training the SuperPoint network through the video, tracking the motion information of the characteristic points of the steel surface in the video and recording the motion information.
In this embodiment, the structural characteristic parameters and the loss functions of the SuperPoint network are as follows:
the SuperPoint network is an end-to-end feature point and descriptor extraction network, the network structure is an encoder-decoder structure similar to a semantic segmentation network, a complete picture is input, and deep features of the picture are extracted through a shared encoder;
and then respectively outputting the feature points and the descriptors through two decoders of the feature points and the descriptors, wherein the network structure is shown in table 1, which is different from the method of firstly detecting the feature points and then calculating the descriptors by a completely manual design algorithm, and the feature points and the descriptors are generated in parallel.
Table 1SuperPoint network architecture
Figure BDA0003532290060000111
As shown in table 1, each row in table 1 is a convolution channel, the first number is an input channel, the middle two numbers are convolution kernel sizes, the last number is a convolution kernel number, and "+ pooling" means performing a maximum pooling operation after convolution;
the structure of the shared encoder is similar to that of a VGG network convolution structure, the first six layers are subjected to 3 × 3 convolution each time and then subjected to 2 × 2 maximum pooling, and the shared encoder is subjected to operations such as convolution kernel pooling, and then subjected to picture dimensionality reduction, deep features are extracted, and subsequent calculated amount is reduced;
the feature map output from the feature point decoder is up-sampled 8 times to output the feature map having the same size as the original image via 1/8 in which the feature point decoder and the descriptor decoder output the feature map having the size of the original image.
The loss function used is consistent with that of SuperPoint. The loss function is composed of a characteristic point loss and a descriptor loss, and is represented by the following formula:
L(X,X′,D,D′,Y,Y′,S)＝Lp(X,Y)+Lp(X′,Y′)+λLd(D,D′,S) (2)
in the formula (2), X and D are feature point feature maps and descriptor feature maps of the original image output after input to the network, Y is a label value of the feature point of the original image, X ', D ', and Y ' corresponding to the input pictures are pictures of the original image after homography transformation, and the rest meanings are the same as X, D, and Y, and S is described by formula (5). Lp and Ld represent the characteristic point loss and the descriptor loss, respectively, and the hyperparameter λ is used to balance the characteristic point detection loss and the descriptor loss. The specific formula of Lp is as follows:
Figure BDA0003532290060000121
in the formula (3), Hc，WcRespectively representing the height and width of the feature point feature map. x is the number ofhw，yhwRespectively, the values of X and Y at (h, w). The specific formula of lp is as follows:
Figure BDA0003532290060000122
in the formula (4), xhwkIs denoted by xhwThe value at the k channel. lpSo that xhwAs large as possible on the lane corresponding to the tag value y.
LdThe specific formula is as follows:
Figure BDA0003532290060000131
in the formula (5), dhw，dh′w'represents the values of D, D' at (h, w), (h ', w'), respectively. Because the encoder performs eight-fold down-sampling, the point in the output descriptor feature map corresponds to the picture unit of the pixel point of 8 × 8 in the input picture. Shwh′w′Is used for judging dhwCorresponding to whether the center position of the input picture unit is d 'after being subjected to homography conversion which is consistent with the original image'h′w′Corresponding to the central position of the input picture unit in the neighborhood, Shwh′w′Is used to judge dhw，d′h′w′Whether the corresponding position is the same or not in the original drawingAnd (4) approaching. Shwh′w′The symbol "1" indicates that the corresponding positions in the original image are close to each other, and corresponds to the forward direction, and vice versa. Shwh′w′And ldThe specific formula is as follows:
Figure BDA0003532290060000132
in the formula (6), phw，ph′w' respectively represent dhw，d′h′w′Location center Hp of corresponding input picture unithwIs to phwThe homography conversion is performed in the same manner as the original image.
ld(d；d'；s)＝ld·s·max(0，mp-dTd')+(1-s)·max(0，dTd'-mn) (7)
In the formula (7), the hyperparameter λ d is used for balancing the positive corresponding loss and the negative corresponding loss in the descriptor, and the hyperparameter mpFor the forward correspondence threshold, mnNegative corresponds to a threshold.
Step 3, screening out the characteristic points around the crack according to the displacement difference between the adjacent characteristic points by a threshold-based self-adaptive characteristic point screening method, which comprises the following specific steps:
step 3-1, calculating the average value of the relative displacement between the cracks on the whole structural surface according to the distance between the characteristic points and the relative displacement;
step 3-2, when the comparative displacement between two adjacent characteristic points is more than 2 times of the average value, determining that cracks exist between the two points, and screening the two points from all the points;
step 3-3, repeating the steps 3-1 to 3-2, screening out all characteristic points around the crack,
the comparative displacement d' between adjacent feature points is the ratio between the relative displacement d between two points and the displacement s between two points.
Figure BDA0003532290060000141
FIG. 4 is a schematic diagram of crack length and dip calculation by comparing the displacement difference of feature points according to a first embodiment of the present invention; FIG. 5 is a schematic diagram of the first embodiment of the present invention, in which the fracture morphology and the fracture basic parameter information are fitted according to the screened feature point position and displacement information.
As shown in fig. 4 and 5, in step 4, the detected feature points are used as basic information, the morphology of the crack is fitted, the basic parameter information of the crack is obtained, and the basic parameter information is digitized, which specifically includes:
step 4-1, connecting the midpoints of two adjacent characteristic points to fit a crack curve, and properly prolonging the curve;
4-2, setting the displacement difference between two adjacent characteristic points as the opening degree of the crack;
and 4-3, taking the opening degree between two adjacent characteristic points as input, taking the final extension length of the crack as output, training a BP neural network, calculating through the BP neural network to obtain the real length of the crack and the position of the tip of the fitting crack, and obtaining the length of the crack, the size of an opening of the crack under load and the inclination angle of the crack at a specific position.
Fig. 6 is a schematic input/output diagram of an LSTM neural network during training according to an embodiment of the present invention.
As shown in fig. 6, in step 5, historical development information of the crack, basic parameter information of the crack and basic information of the steel surface are used as input, the development condition of the crack after N cycles is used as output to train an LSTM neural network, and after the training is completed, the crack development rate of the steel surface is predicted in real time through the LSTM neural network.
< example two >
In the embodiment, an orthotropic steel bridge deck is selected as a sample to be detected;
the selected orthotropic steel bridge deck is composed of a top plate, 3U-shaped ribs and two transverse clapboards, wherein the thickness of the top plate is 12mm, the thickness of the transverse clapboards is 10mm, and the thickness of the U-shaped ribs is 6 mm;
the orthotropic steel bridge deck is made of ISO 4950-2E355DD grade steel;
a 45mm long crack is arranged on the selected orthotropic steel bridge deck U-shaped rib;
the steel surface crack detection and prediction method based on deep learning and video understanding comprises the following steps:
step 1, carrying out fatigue loading on the steel bridge deck, wherein the frequency is 0.5Hz, the loading amplitude is 100KN-300KN, simultaneously, carrying out video shooting by using a calibrated camera at a distance of 1000mm from the test piece, and carrying out clipping processing on the shot video, wherein the length of each section of the video after clipping is 10 seconds and comprises 5 vibration periods.
In the step 1, opencv is adopted for camera calibration, and the specific process is as follows:
circularly reading the photos;
detecting the corner points of the checkerboard through a findChessboardCorrers function;
sub-pixel refinement is carried out on the corner points through a find4QuadCornerSubpix function;
corner points are displayed by drawChessboss detectors;
calibrating according to the ideal coordinates and the actual image coordinates through a calibretacarama function to obtain internal parameters, external parameters and distortion coefficients;
the backprojection error is calculated by the projectPoints function.
And 2, training the SuperPoint network through the video, tracking the motion information of the characteristic points of the steel surface in the video and recording the motion information.
Step 3, screening out the characteristic points around the crack according to the displacement difference between the adjacent characteristic points by a threshold-based self-adaptive characteristic point screening method, which comprises the following specific steps:
step 3-1, calculating the average value of the relative displacement between the cracks on the whole structural surface according to the distance between the characteristic points and the relative displacement;
3-2, when the comparative displacement between two adjacent characteristic points is more than 2 times of the average value, determining that cracks exist between the two points, and screening the two points from all the points;
step 3-3, repeating the steps 3-1 to 3-2, screening out all characteristic points around the crack,
the comparative displacement d' between adjacent feature points is the ratio between the relative displacement d between two points and the displacement s between two points.
Figure BDA0003532290060000161
And 4, fitting the form of the crack by taking the detected characteristic points as basic information to obtain basic parameter information of the crack and digitizing the basic parameter information, wherein the specific steps are as follows:
step 4-1, connecting the middle points of two adjacent characteristic points to fit a curve of the crack, and properly prolonging the curve;
4-2, setting the displacement difference between two adjacent characteristic points as the opening degree of the crack;
and 4-3, taking the opening degree between two adjacent characteristic points as input, taking the final extension length of the crack as output, training a BP (back propagation) neural network, calculating to obtain the real length of the crack and the position of the tip of the fitted crack through the BP neural network, and obtaining the length of the crack, the opening size of the crack under load and the inclination angle of the crack at a specific position.
And 5, taking historical development information of the cracks, basic parameter information of the cracks and basic information of the steel surface as input, taking the development condition of the cracks after N cycles as output to train an LSTM neural network, and predicting the crack development rate of the steel surface in real time through the LSTM neural network after the training is finished.
Effects and effects of the embodiments
According to the steel surface crack detection and prediction method based on deep learning and video understanding, the determination of the position of the crack on the steel surface through characteristic point tracking is completed for the first time, the digital expression of basic parameter information of the crack is completed through the proposed adaptive threshold algorithm, and the prediction of the future development of the crack is completed through the existing information. In the embodiment, the existence of the crack is detected and the crack development trend is predicted according to the existing video information of the opening and closing of the crack on the steel surface by establishing the characteristic point tracking and crack growth prediction model based on deep learning, and some defects of the existing crack evaluation and prediction method can be overcome.
The above embodiments are preferred examples of the present invention, and are not intended to limit the scope of the present invention.

Claims (6)

1. A steel surface crack detection and prediction method based on deep learning and video understanding is characterized by comprising the following steps:
step 1, acquiring and storing a video of crack opening and closing of a crack on the surface of steel under the action of fatigue load, and editing the stored video;
step 2, training a SuperPoint network through the video, tracking and recording motion information of the characteristic points of the steel surface in the video;
step 3, screening out the characteristic points around the crack according to the displacement difference between the adjacent characteristic points by a threshold-based self-adaptive characteristic point screening method;
step 4, fitting the form of the crack by taking the detected characteristic points as basic information to obtain basic parameter information of the crack and digitalizing the basic parameter information;
and 5, taking the historical development information of the crack, the basic parameter information of the crack and the basic information of the steel surface as input, taking the development condition of the crack after N cycles as output to train an LSTM neural network, and predicting the crack development rate of the steel surface in real time through the LSTM neural network after the training is finished.
2. The steel surface crack detection and prediction method based on deep learning and video understanding of claim 1, characterized in that:
wherein, step 1 includes the following substeps:
step 1-1, shooting the video by using a camera, and selecting camera equipment and adjusting related parameters to optimize imaging according to the distance between a shot object and the camera, the illumination of the environment where the shot object is located and the shooting angle when shooting the video;
step 1-2, calibrating the camera, acquiring the internal parameters of the camera, keeping the internal parameters of the camera unchanged, erecting the camera beside the steel surface containing the crack to shoot a video of the steel surface under the action of a working load, and measuring a shooting distance and a shooting inclination angle;
and 1-3, clipping the video to enable the length of each video to be ten seconds, wherein each video comprises 5 vibration cycles.
3. The steel surface crack detection and prediction method based on deep learning and video understanding according to claim 2, characterized in that:
in step 1-2, calibrating the camera includes the following substeps:
step 1-2-1, printing a checkerboard, and pasting the checkerboard on a plane as a calibration object;
step 1-2-2, shooting photos in different directions for the calibration object by adjusting the direction of the calibration object or the camera;
step 1-2-3, extracting angular points of the checkerboard from the picture;
step 1-2-4, estimating five internal parameters and six external parameters of the camera under the condition of ideal distortion-free;
1-2-5, estimating a distortion coefficient under the actual radial distortion by using a least square method;
step 1-2-6, optimizing the internal parameters, the external parameters and the distortion coefficients by a maximum likelihood method, improving estimation precision,
the calibration of the camera is carried out by adopting opencv, and the specific process is as follows:
reading the photos in a circulating way;
detecting the corner points of the checkerboard by a findChessboardCorrers function;
sub-pixel refinement is carried out on the corner through a find4QuadCornerSubpix function;
displaying the corner points by drawChessboss detectors;
calibrating according to the ideal coordinates and the actual image coordinates through a calibretacarama function to obtain the internal parameters, the external parameters and the distortion coefficients;
the backprojection error is calculated by the projectPoints function.
4. The steel surface crack detection and prediction method based on deep learning and video understanding according to claim 1, characterized in that:
wherein, step 2 includes the following substeps:
step 2-1, performing preliminary learning on the SuperPoint network on a virtual training set through MagicPoint;
2-2, carrying out homography transformation on the screenshot of the video, and further training the SuperPoint network;
step 2-3, searching the characteristic points of the steel surface in the video through a VGG neural network, and calibrating the characteristic points on the steel surface in the video by taking the searched characteristic points as a basis;
and 2-4, the SuperPoint network tracks the movement of all the characteristic points and records the position information and the movement information of the characteristic points according to the characteristic points obtained by training and tracking.
5. The steel surface crack detection and prediction method based on deep learning and video understanding according to claim 1, characterized in that:
the method comprises the following steps of (1) screening out the feature points around the crack by a threshold-based adaptive classifier in step 3, wherein the method comprises the following steps:
step 3-1, calculating an average value of the relative displacement between the cracks on the whole steel surface according to the distance between the characteristic points and the relative displacement;
step 3-2, when the comparative displacement between two adjacent characteristic points is more than 2 times of the average value, judging that the crack exists between the two points, and screening the two points from all the characteristic points;
step 3-3, repeating the steps 3-1 to 3-2, screening out all the characteristic points around the crack,
wherein the comparative displacement d' is a ratio of a relative displacement d between two adjacent feature points to a displacement s between two adjacent feature points, and the formula is as follows:
Figure FDA0003532290050000041
6. the steel surface crack detection and prediction method based on deep learning and video understanding according to claim 1, characterized in that:
wherein, step 4 comprises the following substeps:
4-1, fitting a curve of the crack at the midpoint of two adjacent characteristic points of a connecting line, and properly prolonging the curve;
4-2, setting the displacement difference between two adjacent characteristic points as the opening degree of the crack;
and 4-3, taking the opening degree between two adjacent feature points as input, taking the final extension length of the crack as output, training a BP (back propagation) neural network, calculating the true length of the crack and the position of the tip of the fitted crack through the BP neural network, and obtaining the length of the crack, the size of an opening of the crack under load and the inclination angle of the crack at a specific position.
CN202210212511.4A 2022-03-04 2022-03-04 Steel surface crack detection and prediction method based on deep learning and video understanding Pending CN114565596A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210212511.4A CN114565596A (en) 2022-03-04 2022-03-04 Steel surface crack detection and prediction method based on deep learning and video understanding

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210212511.4A CN114565596A (en) 2022-03-04 2022-03-04 Steel surface crack detection and prediction method based on deep learning and video understanding

Publications (1)

Publication Number Publication Date
CN114565596A true CN114565596A (en) 2022-05-31

Family

ID=81717263

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210212511.4A Pending CN114565596A (en) 2022-03-04 2022-03-04 Steel surface crack detection and prediction method based on deep learning and video understanding

Country Status (1)

Country Link
CN (1) CN114565596A (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116484921A (en) * 2023-06-21 2023-07-25 中国石油大学(华东) Crack size accurate quantification method for multi-physical-quantity feature fusion convolutional neural network

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116484921A (en) * 2023-06-21 2023-07-25 中国石油大学(华东) Crack size accurate quantification method for multi-physical-quantity feature fusion convolutional neural network
CN116484921B (en) * 2023-06-21 2023-08-18 中国石油大学(华东) Crack size accurate quantification method for multi-physical-quantity feature fusion convolutional neural network

Similar Documents

Publication Publication Date Title
Park et al. Concrete crack detection and quantification using deep learning and structured light
CN113516660B (en) Visual positioning and defect detection method and device suitable for train
US11714024B2 (en) Vision-based fatigue crack detection using feature tracking
CN109886939B (en) Bridge crack detection method based on tensor voting
JP2017049974A (en) Discriminator generator, quality determine method, and program
CN112950633B (en) Aluminum alloy weld joint surface defect detection method based on line structured light
Shahrokhinasab et al. Performance of image-based crack detection systems in concrete structures
CN110619328A (en) Intelligent ship water gauge reading identification method based on image processing and deep learning
Hou et al. Investigation on performance of RGB point cloud and thermal information data fusion for 3D building thermal map modeling using aerial images under different experimental conditions
CN109584206B (en) Method for synthesizing training sample of neural network in part surface flaw detection
CN112540089A (en) Application method of digital imaging system in concrete bridge crack detection and analysis
Wu et al. Non-contact measurement method of beam vibration with laser stripe tracking based on tilt photography
Zhou et al. UAV vision detection method for crane surface cracks based on Faster R-CNN and image segmentation
Hua et al. 3D reconstruction of concrete defects using optical laser triangulation and modified spacetime analysis
CN114565596A (en) Steel surface crack detection and prediction method based on deep learning and video understanding
Jiang et al. Bridge coating inspection based on two-stage automatic method and collision-tolerant unmanned aerial system
Oytun et al. Effect of Terrestrial Laser Scanning (TLS) parameters on the accuracy of crack measurement in building materials
Colombani et al. A study of multi-target image-based displacement measurement approach for field testing of bridges
CN115761487A (en) Method for quickly identifying vibration characteristics of small and medium-span bridges based on machine vision
Mojidra et al. Vision-based fatigue crack detection using global motion compensation and video feature tracking
CN113446932A (en) Non-contact crack measuring method and system
CN116503359A (en) Road surface crack monitoring method based on road side camera
CN116958837A (en) Municipal facilities fault detection system based on unmanned aerial vehicle
Chen et al. Modal frequency identification of stay cables with ambient vibration measurements based on nontarget image processing techniques
CN111583241B (en) Mobile detection method and device for regularity of ultra-large area honeycomb products

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination