CN117994753B - Vision-based device and method for detecting abnormality of entrance track of car dumper - Google Patents
Vision-based device and method for detecting abnormality of entrance track of car dumper Download PDFInfo
- Publication number
- CN117994753B CN117994753B CN202410398164.8A CN202410398164A CN117994753B CN 117994753 B CN117994753 B CN 117994753B CN 202410398164 A CN202410398164 A CN 202410398164A CN 117994753 B CN117994753 B CN 117994753B
- Authority
- CN
- China
- Prior art keywords
- feature extraction
- track
- image
- image set
- identification
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 36
- 230000005856 abnormality Effects 0.000 title claims abstract description 29
- 238000000605 extraction Methods 0.000 claims abstract description 131
- 238000013507 mapping Methods 0.000 claims abstract description 28
- 238000001514 detection method Methods 0.000 claims abstract description 18
- 230000002159 abnormal effect Effects 0.000 claims abstract description 17
- 238000004458 analytical method Methods 0.000 claims abstract description 16
- 238000012216 screening Methods 0.000 claims abstract description 11
- 238000007781 pre-processing Methods 0.000 claims abstract description 10
- 238000012549 training Methods 0.000 claims description 28
- 230000000007 visual effect Effects 0.000 claims description 13
- 238000013527 convolutional neural network Methods 0.000 claims description 10
- 238000012423 maintenance Methods 0.000 claims description 10
- 230000009545 invasion Effects 0.000 claims description 8
- 230000000737 periodic effect Effects 0.000 claims description 4
- 238000011176 pooling Methods 0.000 claims description 4
- 238000012795 verification Methods 0.000 claims description 3
- 238000012545 processing Methods 0.000 description 7
- 238000000926 separation method Methods 0.000 description 6
- 239000003550 marker Substances 0.000 description 3
- 230000007613 environmental effect Effects 0.000 description 2
- 239000000284 extract Substances 0.000 description 2
- 238000005299 abrasion Methods 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000004140 cleaning Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 230000007774 longterm Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
Landscapes
- Image Analysis (AREA)
Abstract
The application discloses a device and a method for detecting abnormality of an entrance track of a car dumper based on vision, wherein the device comprises the following steps: and an image acquisition module: the image acquisition device is used for acquiring a target image set to be detected; and a mapping identification module: the method comprises the steps of preprocessing a target image set to be detected, screening out key frame images, extracting features of the key frame images to obtain preprocessed image features, and forming a mark image set by the marks endowed by the preprocessed image features and the corresponding images; and the feature extraction module is used for: the method comprises the steps of inputting a marked image set into a feature extraction model, identifying and identifying the marked image set, and carrying out joint analysis to obtain a feature extraction result; seam state determination module: the method comprises the steps of determining a track joint state according to a feature extraction result; and the detection judging module is used for: the method is used for judging whether the entrance track of the car dumper is abnormal according to the track joint state and combining the traffic state signal, and solves the problem that detection accuracy is poor due to the influence of foreign matters on image acquisition equipment in abnormal detection work of the car dumper.
Description
Technical Field
The application relates to the technical field of data processing, in particular to a device and a method for detecting abnormality of an entrance track of a car dumper based on vision.
Background
The dumper is large mechanical equipment widely applied to ports, power plants, mines and other places and used for dumping and carrying cargos on transportation means such as trains, automobiles and the like. The dumper inlet rail is an important component of the whole system, and the running state of the dumper inlet rail directly affects the safety and efficiency of the system. However, due to long-term use, abrasion, environmental factors, etc., abnormality such as deformation, breakage, foreign matter, etc., may occur in the rail, and if such abnormality cannot be found and handled in time, serious safety accidents may be caused.
The problem that the detection accuracy is poor due to the fact that foreign matters affect the image acquisition device in the existing abnormality detection work of the car dumper in the prior art is solved, and the car dumper can be better maintained finally.
Disclosure of Invention
The application provides a device and a method for detecting abnormality of an entrance track of a car dumper based on vision, which are used for solving the technical problems in the background technology.
The technical scheme adopted for solving the technical problems is as follows: a vision-based tipper entrance track anomaly detection device, comprising:
And an image acquisition module: the image acquisition device is used for acquiring multi-angle images of the entrance track of the car dumper to obtain a target image set to be detected;
And a mapping identification module: the method comprises the steps of preprocessing a target image set to be detected, screening out key frame images, extracting features of the key frame images to obtain preprocessed image features, carrying out mapping identification based on space positions on the preprocessed image features, endowing each feature with an identification, and forming a mark image set by the endowed identification and the corresponding image;
And the feature extraction module is used for: the feature extraction model is used for inputting the marked image set into a feature extraction model, the feature extraction model is generated by acquiring sample image data and performing convolutional neural network training, the feature extraction model comprises a plurality of parallel feature extraction branches, each branch corresponds to a specific image feature extraction method, and the feature extraction model carries out identification and identification on the marked image set and carries out joint analysis to obtain a feature extraction result;
seam state determination module: the method comprises the steps of determining a track joint state according to a feature extraction result;
and the detection judging module is used for: and the device is used for judging whether the entrance track of the dumper is abnormal according to the track joint state and the traffic state signal.
The application also provides a method for detecting the abnormality of the entrance track of the car dumper based on vision, which comprises the following steps:
S1: performing multi-angle image acquisition on an entrance track of the car dumper by adopting an image acquisition device to obtain a target image set to be detected;
S2: preprocessing a target image set to be detected, screening out a key frame image, extracting features of the key frame image to obtain preprocessed image features, carrying out mapping identification based on space positions on the preprocessed image features, endowing each feature with an identification, and forming a mark image set by the endowed identification and the corresponding image;
S3: inputting the marked image set into a feature extraction model, performing convolutional neural network training generation by the feature extraction model through obtaining sample image data, wherein the feature extraction model comprises a plurality of parallel feature extraction branches, each branch corresponds to a specific image feature extraction method, and the feature extraction model performs identification recognition on the marked image set and performs joint analysis to obtain a feature extraction result;
s4: determining the track joint state according to the feature extraction result;
s5: and judging whether the entrance track of the tippler is abnormal according to the track joint state and the traffic state signal.
According to the vision-based method for detecting the abnormality of the entrance track of the car dumper, the image acquisition device is used for acquiring the multi-angle image of the entrance track on site, determining a target image set to be detected, preprocessing the target image set to be detected, determining a mark image set based on the mapping identification of the space position, calling sample image data, performing convolutional neural network training, generating a feature extraction model, inputting the mark image set into the feature extraction model, performing identification and joint analysis, determining a feature extraction result, determining the track joint state, finally, combining with a traffic state signal based on the track joint state, performing abnormal operation control triggering warning, and realizing more accurate detection of the entrance track of the car dumper, so that the accident hidden danger caused by the abnormality of the entrance track of the car dumper can be better maintained, and the accident hidden danger caused by the abnormality of the entrance track of the car dumper is reduced.
Preferably, S1: the step of acquiring the multi-angle image of the entrance track of the car dumper by adopting the image acquisition device to obtain the target image set to be detected comprises the following steps: the image acquisition device is adopted to conduct acquisition angle self-adjustment in a mapping angle interval, a plurality of groups of acquisition images corresponding to the image acquisition devices one by one are acquired, the image acquisition device is matched with the acquisition angle marks on the plurality of groups of acquisition images to obtain a target image set to be detected, the image acquisition device is arranged according to the size and shape of an entrance track of the car dumper, and the arrangement position needs to be considered during arrangement to acquire a main body target of the car dumper. After the layout of the image acquisition equipment is completed, each image acquisition device can be adjusted according to the acquisition angle range preset by the image acquisition device.
Preferably, the screening method of the key frame image in S2 includes: and identifying the target image set to be detected mapped by each image acquisition device, traversing the target image set to be detected, and screening out images capable of representing the track state or change, wherein the images are key frame images.
Preferably, the mapping identification based on the spatial location in S2 includes: the key frame image is expressed in a space position, the object is transferred from the two-dimensional image to the three-dimensional space for expression, corresponding position relation determination is carried out according to the expression in the three-dimensional space, and each feature is endowed with an identifier, wherein the identifier can be the coordinates of the feature in the image or other forms of identifiers.
Preferably, the training method of the feature extraction model in S3 includes:
a1: collecting sample image data, and using the sample image data to supervise and train a first feature extraction branch, wherein the first feature extraction branch comprises a multi-stage convolution layer, a multi-stage pooling layer and a full connection layer;
A2: inputting the sample image data into a first feature extraction branch for precision prediction, and taking the sample image data as a training sample if the prediction precision is lower than a preset precision threshold;
A3: training the second feature extraction branch by using the training sample;
a4: repeating iterative training verification to obtain an Nth feature extraction branch;
A5: and (3) paralleling the first feature extraction branch and the second feature extraction branch until the Nth feature extraction branch to obtain a feature extraction model.
By continuously constructing more optimized feature extraction branches, the processing capacity of the model on complex image features can be effectively enhanced, and the prediction accuracy is higher.
Preferably, the step S3 of identifying the marker image set by the feature extraction model and performing joint analysis to obtain a feature extraction result includes:
b1: each feature extraction branch is synchronously processed, a plurality of groups of convolution features are determined, and the plurality of groups of convolution features are in one-to-one correspondence with the feature extraction branches;
B2: classifying the plurality of groups of convolution features based on the mapping identification of the spatial position in the S2, and determining a plurality of types of convolution features;
b3: and (5) checking and selecting the highest frequency term in the convolution characteristics of a plurality of classes as a characteristic extraction result.
And determining the feature extraction result according to the joint analysis, so that the obtained feature extraction result has more representative features, and the features can reflect the overall information of the image.
Preferably, the track joint state includes a track connection state and a track separation state; the traffic state signal is a traffic signal sent when the track joint state is a track connection state; the traffic state signal and the track joint state have consistency, namely the traffic state signal is required to have the track joint state, and vice versa, if one of the traffic state signal and the track joint state does not appear, the other traffic state signal does not appear, so that when the track joint state and the traffic state signal are consistent, the track is judged to be in the connection state, and the entrance track of the tippler is judged to be normal.
Preferably, a preset operation and maintenance period is set, a feature extraction result of a periodic node is read, track invasion foreign matters are determined, visual influence judgment is carried out on the track invasion foreign matters, if the visual influence degree is out of limit, the inlet track of the dumper is judged to be abnormal, the visual influence degree is out of limit, the visual influence judgment indicates that the foreign matters have obvious influence on track operation and safety, and the track invasion foreign matters must be found and processed in time at the moment, so that the safety and the efficiency of track operation are improved.
The application has the following substantial effects:
1. According to the vision-based method for detecting the abnormality of the entrance track of the car dumper, the image acquisition device is used for acquiring the multi-angle image of the entrance track on site, determining a target image set to be detected, preprocessing the target image set to be detected, determining a mark image set based on the mapping identification of the space position, calling sample image data, performing convolutional neural network training, generating a feature extraction model, inputting the mark image set into the feature extraction model, performing identification and joint analysis, determining a feature extraction result, determining the track joint state, and finally, based on the track joint state, combining with a traffic state signal, performing abnormal operation control triggering warning, so that the entrance track of the car dumper is detected more accurately, further better maintenance can be obtained, and accident potential caused by the abnormality of the entrance track of the car dumper is reduced;
2. the vision-based method for detecting the abnormality of the entrance track of the car dumper can acquire images at multiple angles and can enable the acquired images to correspond to the acquisition positions one by one, and when the entrance track of the car dumper is abnormal, the abnormal positions can be accurately detected, so that the maintenance is convenient;
3. The vision-based method for detecting the abnormality of the entrance track of the dumper processes and screens the acquired images, extracts the characteristics of the images, detects the entrance track of the dumper by combining the traffic state signals, has more accurate detection results, and is convenient for maintaining the entrance track of the dumper.
Drawings
Fig. 1 is a block diagram of a vision-based tippler entrance track abnormality detection device in accordance with a first embodiment of the present application;
Fig. 2 is a schematic flow chart of a detection step of a method for detecting abnormality of an entrance track of a tipper based on vision in a second embodiment of the present application;
fig. 3 is a flowchart of a training step of the feature extraction model in the second embodiment of the present application.
Detailed Description
The technical scheme of the application is further specifically described by the following specific examples.
Example 1
As shown in fig. 1, a vision-based device for detecting abnormality of an entrance track of a car dumper includes:
And an image acquisition module: the image acquisition device is used for acquiring multi-angle images of the entrance track of the car dumper to obtain an image set of a target to be detected, is a general image acquisition device, comprises a CCD sensor image acquisition device, and is distributed according to the size and shape of the entrance track of the car dumper, and the main body target of the car dumper can be acquired by taking the distribution position into consideration when the image acquisition device is distributed. The image acquisition devices are distributed around the track according to the position of the on-site entrance track. After the layout of the image acquisition devices is completed, each image acquisition device can be automatically adjusted according to the preset acquisition angle range of the image acquisition device, if necessary, the acquisition angle of each image acquisition device can be manually adjusted, the angle interval of each image acquisition device based on the on-site entrance track is determined, and then the image acquisition device is controlled to acquire images to obtain a target image set to be detected;
And a mapping identification module: the method comprises the steps of preprocessing a target image set to be detected, screening out key frame images, extracting features of the key frame images to obtain preprocessed image features, carrying out mapping identification based on space positions on the preprocessed image features, endowing each feature with an identification, and forming a mark image set by the endowed identification and the corresponding image;
And the feature extraction module is used for: the feature extraction model is used for inputting the marked image set into a feature extraction model, the feature extraction model is generated by acquiring sample image data and performing convolutional neural network training, the feature extraction model comprises a plurality of parallel feature extraction branches, each branch corresponds to a specific image feature extraction method, and the feature extraction model carries out identification and identification on the marked image set and carries out joint analysis to obtain a feature extraction result;
seam state determination module: the method comprises the steps of determining a track joint state according to a feature extraction result;
and the detection judging module is used for: and the device is used for judging whether the entrance track of the dumper is abnormal according to the track joint state and the traffic state signal.
The device for detecting the abnormality of the entrance track of the dumper based on vision acquires multi-angle images of the entrance track on site through the image acquisition device, determines a target image set to be detected, pre-processes the target image set to be detected, and determines a mark image set based on a mapping mark of a space position, invokes sample image data, performs convolutional neural network training, generates a feature extraction model, inputs the mark image set into the feature extraction model, performs mark recognition and joint analysis, determines a feature extraction result, determines a track joint state, and finally performs triggering warning of abnormal operation control by combining with a traffic state signal based on the track joint state, thereby realizing more accurate detection of the entrance track of the dumper, further better maintaining and reducing accident hidden dangers caused by the abnormality of the entrance track of the dumper.
Example two
As shown in fig. 2, a vision-based method for detecting abnormality of an entrance track of a car dumper includes:
S1: the method comprises the steps of adopting an image acquisition device to acquire multi-angle images of an entrance track of the car dumper to obtain a target image set to be detected: and carrying out self-adjustment on the acquisition angles based on the mapping angle interval by adopting an image acquisition device, determining a plurality of groups of acquisition images, wherein the plurality of groups of acquisition images correspond to the image acquisition device one by one, and carrying out attribution and acquisition angle marking of the image acquisition device on the plurality of groups of acquisition images to generate the target image set to be detected.
The image acquisition device with adjustable angle is controlled to be distributed at the position of the entrance track of the car dumper, the acquisition angle is self-adjusted in the mapping angle interval, a plurality of groups of acquisition images corresponding to the image acquisition devices one by one are acquired, the plurality of groups of acquisition images are subjected to matching of the image acquisition devices and acquisition angle marking, and the matching is that the image acquisition device identifiers for acquiring the plurality of groups of acquisition images are used for generating a target image set to be detected. By arranging the image acquisition device with the adjustable acquisition angle at the position of the entrance track of the car dumper, images of the on-site entrance track can be effectively captured from a plurality of angles, so that more comprehensive and accurate information is provided. Meanwhile, the requirements of different track positions and different environmental conditions can be met through distributed arrangement and adjustable angle ranges.
S2: preprocessing a target image set to be detected, screening out a key frame image, extracting features of the key frame image to obtain preprocessed image features, carrying out mapping identification based on space positions on the preprocessed image features, endowing each feature with an identification, and forming a mark image set by the endowed identification and the corresponding image: the mapping identification based on the space position refers to representing the target image set to be detected in the space position, transferring the object from the two-dimensional image to the three-dimensional space for representation, and determining the corresponding position relation according to the representation in the three-dimensional space. Therefore, multi-angle joint judgment can be realized, and when effective features cannot be identified in a local area due to foreign matter invasion at a certain angle, joint judgment can be carried out through other angles. Before the mapping identification based on the space position is carried out, the target image set to be detected is required to be preprocessed, required image data is screened, unnecessary image data is removed, the subsequent mapping identification based on the space position is convenient, and the overall efficiency is improved. And identifying the preprocessed target image set to be detected, traversing the target image set to be detected, and selecting images which can better represent the track state or change, wherein the images are called key frame images. And evaluating the key frame image by using an image definition evaluation algorithm to obtain a definition value. Setting a definition threshold, if the definition of a certain key frame image is lower than the threshold, considering that the image does not meet the requirement, performing image enhancement processing, wherein the image enhancement algorithm comprises but is not limited to a denoising algorithm, contrast and brightness adjustment, binarization and the like, selecting and using according to actual conditions, and sorting the processed image to generate a preprocessed image set. And extracting the characteristics of the color, the shape, the texture and the like of the preprocessed image set to obtain a preprocessed image characteristic set. And (3) determining the spatial position of the preprocessed image feature set, determining the spatial position of each feature through coordinate transformation according to the position of the preprocessed image feature in the image, mapping and identifying, and mapping each preprocessed image feature with the spatial position of each preprocessed image feature in the image, so that an identification is given to each feature. The identifier can be the coordinates of the features in the image, or can be other types of identifiers, and the identifiers and the corresponding images form a new set, namely a marked image set. Wherein each element in the set of marker images is an image with spatial location information. By preprocessing the key frame image, a basis can be provided for the subsequent mapping identification based on the spatial position to determine the marked image set, and the efficiency of the mapping identification based on the spatial position is improved.
S3: inputting the marked image set into a feature extraction model, performing convolutional neural network training generation by the feature extraction model through obtaining sample image data, wherein the feature extraction model comprises a plurality of parallel feature extraction branches, each branch corresponds to a specific image feature extraction method, and the feature extraction model performs identification recognition on the marked image set and performs joint analysis to obtain feature extraction results: inputting the marked image set into a feature extraction model, performing synchronous processing based on each feature extraction branch, and determining a plurality of groups of convolution features, wherein the plurality of groups of convolution features are in one-to-one correspondence with the feature extraction branches; classifying the plurality of groups of convolution features based on the identification space position to determine a plurality of types of convolution features; and correcting and selecting the highest frequency item in various convolution characteristics aiming at the multi-type convolution characteristics, and taking the highest frequency item as the characteristic extraction result.
The identification and recognition joint analysis means that a plurality of branch convolution features exist in the same space, analysis and comparison are carried out on the plurality of branch convolution features, and the optimal branch convolution features are selected. The corresponding branch convolution features under the same spatial position are extracted, and the extracted branch convolution features are independently used as one type of branch convolution features. And extracting the convolution characteristic category of the branch with the highest frequency as the convolution characteristic of the space position at the characteristic extraction result of each branch for the image. The set of marker images is input into a trained feature extraction model. The feature extraction model comprises a plurality of feature extraction branches, and each branch corresponds to a specific image feature extraction method. Through the synchronization processing of each feature extraction branch, several groups of convolution features are extracted from the input image set. Convolutional features are an important type of feature in convolutional neural networks that are commonly used for image recognition and classification tasks. The convolution features are in one-to-one correspondence with the previous feature extraction branches, each of which outputs a set of convolution features. Since each feature extraction branch may extract different convolution features for the same image or image region, these features need to be classified. The convolved features are separated into categories based on the spatial location identity of each feature. For the convolution features in each category, analyzing the occurrence frequency of the convolution features, namely which features occur most frequently, selecting the convolution feature with the highest frequency in each category as the representative feature of the category, and combining the representative features in all the categories to form a final feature extraction result. And determining the feature extraction result according to the joint analysis, so that the obtained feature extraction result has more representative features, and the features can reflect the overall information of the image.
As shown in fig. 3, during training of the feature extraction model, a large amount of sample image data is acquired through big data, the large amount of sample image data is screened, sample image data which does not accord with the training is identified and removed, sample image data which accords with the training is identified and reserved, corresponding sample image data is obtained, and first a first feature extraction branch is trained by using the sample image data. The first feature extraction branch consists of a plurality of convolution layers, a plurality of pooling layers and a full connection layer, wherein the plurality of convolution layers are used for extracting local features of the image; the multi-layer pooling layer is used for reducing the dimension of the feature and simultaneously retaining important information; the full connection layer is used to map the extracted features to predetermined categories. After training the first feature extraction branch, sample image data is input into the branch for verification. If the prediction accuracy of the model is lower than a preset accuracy threshold, the sample image data are used as training samples, and the training samples are unqualified sample data. Training a second feature extraction branch by using a training sample, wherein the second feature extraction branch is similar to the first branch in structure and is an optimized version of the first structure branch so as to better process complex image features misjudged by the first branch. Repeating the steps to obtain more optimized third feature extraction branches until the Nth feature extraction branch, wherein each branch is more optimized than the previous branch, and connecting all trained branches in parallel to obtain a feature extraction model. By continuously constructing more optimized feature extraction branches, the processing capacity of the model on complex image features can be effectively enhanced, and the prediction accuracy is higher.
S4: determining the track joint state according to the feature extraction result: the track joint state comprises a track connection state and a track separation state, and the track joint state refers to a gap which appears at a connecting place between two sections of tracks and is the condition of the gap. When a large gap appears between the rails at two ends to cause the blockage of goods in the transportation process, the rail state is indicated to be a rail separation state, and if the gap between the rails does not influence the goods transportation, the rail joint state is indicated to be a rail connection state.
S5: judging whether the entrance track of the dumper is abnormal or not according to the track joint state and the traffic state signal: the traffic state signal refers to a transmission signal sent when the track joint state is the track connection state, and the traffic state signal has consistency with the track joint state, namely the traffic state signal is required to appear in the track joint state, and vice versa, if one of the traffic state signal and the track joint state does not appear, the other traffic state signal does not appear. The track separation state and the forbidden signal are identical, and the forbidden signal is generated in the track separation state. Based on the track joint state, the traffic state signal is combined to trigger and warn abnormal operation control. The system state can be better monitored and adjusted through consistency judgment so as to ensure normal operation, and when the traffic state signal is consistent with the track joint state and the track separation is judged, the inlet track is judged to be abnormal.
In order to improve the safety and efficiency of the track operation, a preset operation and maintenance period is set, the feature extraction result of the periodic node is read, and the track intrusion foreign matters are determined; performing visual influence judgment on the track invasion foreign matters, and generating a foreign matter treatment instruction if the visual influence degree is out of limit; and transmitting the foreign matter treatment instruction to a personnel mobile terminal, and carrying out operation and maintenance management on the invasion foreign matters.
When the foreign matter appears in the track, if the foreign matter is smaller, the judgment of the seam state of the track is not affected, the influence of the smaller foreign matter can be eliminated through the image acquisition devices arranged at other positions, and if the foreign matter is larger to interfere with the image acquisition devices, the image acquisition devices cannot acquire effective information, and further processing is needed.
The predetermined operation and maintenance period is a fixed maintenance period, and the equipment is maintained every time a certain time passes. A preset operation and maintenance period is set first, and the feature extraction result of the periodic node is read in each operation and maintenance period. And analyzing and judging according to the result of the feature extraction to determine whether the foreign matter invaded by the track exists. If it is determined that there is a foreign object that is intruded into the track, a visual impact determination is made. The visual influence relates to analyzing the size, shape, color, position and other characteristics of the foreign matters to evaluate the influence degree of the foreign matters on the track operation and the safety, and judging that the track at the entrance of the dumper is abnormal if the visual influence judgment shows that the foreign matters have obvious influence on the track operation and the safety, namely the visual influence degree exceeds the limit. And then cleaning, removing, repairing and the like are carried out on the track so as to ensure the normal operation and safety of the track. By combining feature extraction and visual influence judgment, the foreign matters invaded by the track can be timely found and processed, so that the safety and the efficiency of the track operation are improved.
The above-described embodiment is only a preferred embodiment of the present application, and is not limited in any way, and other variations and modifications may be made without departing from the technical aspects set forth in the claims.
Claims (7)
1. Vision-based tippler entrance track anomaly detection device, which is characterized by comprising:
And an image acquisition module: the image acquisition device is used for acquiring multi-angle images of the entrance track of the car dumper to obtain a target image set to be detected;
And a mapping identification module: the method is used for preprocessing a target image set to be detected, screening out key frame images, extracting features of the key frame images to obtain preprocessed image features, carrying out mapping identification based on spatial positions on the preprocessed image features, endowing each feature with an identification, forming a mark image set by the endowed identification and the corresponding image, and the mapping identification based on the spatial positions comprises the following steps: representing the key frame image in a space position, transferring an object from a two-dimensional image to a three-dimensional space for representation, determining a corresponding position relation according to the representation in the three-dimensional space, and endowing each feature with a mark;
And the feature extraction module is used for: the method is used for inputting the marked image set into a feature extraction model, the feature extraction model is generated by acquiring sample image data and carrying out convolutional neural network training, the feature extraction model comprises a plurality of parallel feature extraction branches, each branch corresponds to a specific image feature extraction method, the feature extraction model carries out identification and identification on the marked image set and carries out joint analysis to obtain a feature extraction result,
The feature extraction model carries out identification and identification on the marked image set and carries out joint analysis, and the feature extraction result comprises the following steps:
b1: each feature extraction branch is synchronously processed, a plurality of groups of convolution features are determined, and the plurality of groups of convolution features are in one-to-one correspondence with the feature extraction branches;
B2: classifying the plurality of groups of convolution features based on the mapping identification of the spatial position, and determining a plurality of types of convolution features;
b3: checking and selecting the highest frequency item in a plurality of classes of convolution characteristics as a characteristic extraction result;
seam state determination module: the method comprises the steps of determining a track joint state according to a feature extraction result;
and the detection judging module is used for: and the device is used for judging whether the entrance track of the dumper is abnormal according to the track joint state and the traffic state signal.
2. The vision-based method for detecting the abnormality of the entrance track of the car dumper is used for realizing the detection of the abnormality of the entrance track of the car dumper by the vision-based device for detecting the abnormality of the entrance track of the car dumper as claimed in claim 1, and is characterized by comprising the following steps:
S1: performing multi-angle image acquisition on an entrance track of the car dumper by adopting an image acquisition device to obtain a target image set to be detected;
S2: preprocessing a target image set to be detected, screening out a key frame image, extracting features of the key frame image to obtain preprocessed image features, carrying out mapping identification based on space positions on the preprocessed image features, endowing each feature with an identification, and forming a mark image set by the endowed identification and the corresponding image;
S3: inputting the marked image set into a feature extraction model, performing convolutional neural network training generation by the feature extraction model through obtaining sample image data, wherein the feature extraction model comprises a plurality of parallel feature extraction branches, each branch corresponds to a specific image feature extraction method, and the feature extraction model performs identification recognition on the marked image set and performs joint analysis to obtain a feature extraction result;
s4: determining the track joint state according to the feature extraction result;
s5: and judging whether the entrance track of the tippler is abnormal according to the track joint state and the traffic state signal.
3. The vision-based method for detecting abnormality of entrance track of car dumper according to claim 2, wherein S1: the step of acquiring the multi-angle image of the entrance track of the car dumper by adopting the image acquisition device to obtain the target image set to be detected comprises the following steps: and carrying out acquisition angle self-adjustment in the mapping angle interval by adopting the image acquisition device, acquiring a plurality of groups of acquisition images corresponding to the image acquisition devices one by one, and carrying out matching of the image acquisition devices and acquisition angle marking on the plurality of groups of acquisition images to obtain a target image set to be detected.
4. The vision-based method for detecting abnormality of entrance track of car dumper according to claim 2, wherein the screening method of the key frame image in S2 comprises: and identifying the target image set to be detected mapped by each image acquisition device, traversing the target image set to be detected, and screening out images capable of representing the track state or change, wherein the images are key frame images.
5. The vision-based tipper entrance track anomaly detection method of claim 4, wherein the feature extraction model training method in S3 comprises:
a1: collecting sample image data, and using the sample image data to supervise and train a first feature extraction branch, wherein the first feature extraction branch comprises a multi-stage convolution layer, a multi-stage pooling layer and a full connection layer;
A2: inputting the sample image data into a first feature extraction branch for precision prediction, and taking the sample image data as a training sample if the prediction precision is lower than a preset precision threshold;
A3: training the second feature extraction branch by using the training sample;
a4: repeating iterative training verification to obtain an Nth feature extraction branch;
A5: and (3) paralleling the first feature extraction branch and the second feature extraction branch until the Nth feature extraction branch to obtain a feature extraction model.
6. The vision-based tipper entrance track anomaly detection method of claim 2, wherein the track joint state includes a track connected state and a track separated state; the traffic state signal is a traffic signal sent when the track joint state is a track connection state; and when the seam state of the track is consistent with the traffic state signal, judging that the track is in a connection state, and judging that the entrance track of the dumper is normal.
7. The method for detecting the abnormality of the entrance track of the dumper based on vision according to claim 2, wherein a predetermined operation and maintenance period is set, the feature extraction result of the periodic node is read, the track invasion foreign matter is determined, the visual influence judgment is carried out on the track invasion foreign matter, and if the visual influence degree is out beyond the limit, the abnormality of the entrance track of the dumper is judged.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410398164.8A CN117994753B (en) | 2024-04-03 | 2024-04-03 | Vision-based device and method for detecting abnormality of entrance track of car dumper |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410398164.8A CN117994753B (en) | 2024-04-03 | 2024-04-03 | Vision-based device and method for detecting abnormality of entrance track of car dumper |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117994753A CN117994753A (en) | 2024-05-07 |
| CN117994753B true CN117994753B (en) | 2024-06-07 |
Family
ID=90895668
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410398164.8A Active CN117994753B (en) | 2024-04-03 | 2024-04-03 | Vision-based device and method for detecting abnormality of entrance track of car dumper |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117994753B (en) |
Citations (22)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2019232831A1 (en) * | 2018-06-06 | 2019-12-12 | 平安科技（深圳）有限公司 | Method and device for recognizing foreign object debris at airport, computer apparatus, and storage medium |
| CN111340754A (en) * | 2020-01-18 | 2020-06-26 | 中国人民解放军国防科技大学 | Method for detecting and classifying surface defects based on aircraft skin |
| WO2020154096A1 (en) * | 2019-01-22 | 2020-07-30 | Fyusion, Inc. | Damage detection from multi-view visual data |
| CN111626203A (en) * | 2020-05-27 | 2020-09-04 | 北京伟杰东博信息科技有限公司 | Railway foreign matter identification method and system based on machine learning |
| CN112200483A (en) * | 2020-10-22 | 2021-01-08 | 卡斯柯信号有限公司 | Automatic inspection system and automatic inspection method for subway trackside equipment |
| CN112257572A (en) * | 2020-10-20 | 2021-01-22 | 神思电子技术股份有限公司 | Behavior identification method based on self-attention mechanism |
| CN113361354A (en) * | 2021-05-28 | 2021-09-07 | 国能朔黄铁路发展有限责任公司 | Track component inspection method and device, computer equipment and storage medium |
| CN113705412A (en) * | 2021-08-23 | 2021-11-26 | 中国铁道科学研究院集团有限公司 | Deep learning-based multi-source data fusion track state detection method and device |
| CN114187441A (en) * | 2021-12-08 | 2022-03-15 | 深圳市商汤科技有限公司 | Contact net part defect detection method and device, electronic equipment and storage medium |
| WO2022105179A1 (en) * | 2020-11-23 | 2022-05-27 | 平安科技（深圳）有限公司 | Biological feature image recognition method and apparatus, and electronic device and readable storage medium |
| CN114972299A (en) * | 2022-06-16 | 2022-08-30 | 沈阳工业大学 | Railway track defect detection method based on deep migration learning |
| CN115096996A (en) * | 2022-05-31 | 2022-09-23 | 广西大学 | Rail transit train welding quality detection method based on improved Mask R-CNN |
| CN115187522A (en) * | 2022-06-22 | 2022-10-14 | 南昌航空大学 | Skin seam measuring method and device based on data fusion |
| CN115601558A (en) * | 2022-10-24 | 2023-01-13 | 吉林大学（Cn） | Single turnout state detection system and detection method and semi-automatic data labeling method |
| CN115675566A (en) * | 2022-11-11 | 2023-02-03 | 洛阳双瑞橡塑科技有限公司 | Track monitoring system and monitoring method |
| CN116160179A (en) * | 2023-01-03 | 2023-05-26 | 浙江圣洁钛业科技有限公司 | Automatic welding clamping platform of location |
| CN116206265A (en) * | 2023-05-05 | 2023-06-02 | 昆明轨道交通四号线土建项目建设管理有限公司 | Protection alarm device and method for rail transit operation maintenance |
| CN116612457A (en) * | 2023-05-29 | 2023-08-18 | 北京工商大学 | Track bed foreign matter detection method based on deep unsupervised learning |
| CN117058097A (en) * | 2023-08-14 | 2023-11-14 | 安徽环球基因科技有限公司 | Identification method and system for antibody rapid detection reagent |
| CN117079138A (en) * | 2023-09-09 | 2023-11-17 | 西北工业大学 | Building damage assessment method based on multi-view information intelligent extraction and fusion |
| CN117218107A (en) * | 2023-10-07 | 2023-12-12 | 南京铁道职业技术学院 | Rail defect detection method and system |
| CN117392610A (en) * | 2023-11-08 | 2024-01-12 | 大连升泰生物科技有限公司 | Intelligent monitoring method and system for abnormal state of farmed fish |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20170032250A1 (en) * | 2015-07-29 | 2017-02-02 | Ching-Ping Chang | Machine Status And User Behavior Analysis System |
| US20180096595A1 (en) * | 2016-10-04 | 2018-04-05 | Street Simplified, LLC | Traffic Control Systems and Methods |
| US20230286556A1 (en) * | 2022-03-08 | 2023-09-14 | University Of South Carolina | Autonomous drone for railroad track inspection |
-
2024
- 2024-04-03 CN CN202410398164.8A patent/CN117994753B/en active Active
Patent Citations (22)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2019232831A1 (en) * | 2018-06-06 | 2019-12-12 | 平安科技（深圳）有限公司 | Method and device for recognizing foreign object debris at airport, computer apparatus, and storage medium |
| WO2020154096A1 (en) * | 2019-01-22 | 2020-07-30 | Fyusion, Inc. | Damage detection from multi-view visual data |
| CN111340754A (en) * | 2020-01-18 | 2020-06-26 | 中国人民解放军国防科技大学 | Method for detecting and classifying surface defects based on aircraft skin |
| CN111626203A (en) * | 2020-05-27 | 2020-09-04 | 北京伟杰东博信息科技有限公司 | Railway foreign matter identification method and system based on machine learning |
| CN112257572A (en) * | 2020-10-20 | 2021-01-22 | 神思电子技术股份有限公司 | Behavior identification method based on self-attention mechanism |
| CN112200483A (en) * | 2020-10-22 | 2021-01-08 | 卡斯柯信号有限公司 | Automatic inspection system and automatic inspection method for subway trackside equipment |
| WO2022105179A1 (en) * | 2020-11-23 | 2022-05-27 | 平安科技（深圳）有限公司 | Biological feature image recognition method and apparatus, and electronic device and readable storage medium |
| CN113361354A (en) * | 2021-05-28 | 2021-09-07 | 国能朔黄铁路发展有限责任公司 | Track component inspection method and device, computer equipment and storage medium |
| CN113705412A (en) * | 2021-08-23 | 2021-11-26 | 中国铁道科学研究院集团有限公司 | Deep learning-based multi-source data fusion track state detection method and device |
| CN114187441A (en) * | 2021-12-08 | 2022-03-15 | 深圳市商汤科技有限公司 | Contact net part defect detection method and device, electronic equipment and storage medium |
| CN115096996A (en) * | 2022-05-31 | 2022-09-23 | 广西大学 | Rail transit train welding quality detection method based on improved Mask R-CNN |
| CN114972299A (en) * | 2022-06-16 | 2022-08-30 | 沈阳工业大学 | Railway track defect detection method based on deep migration learning |
| CN115187522A (en) * | 2022-06-22 | 2022-10-14 | 南昌航空大学 | Skin seam measuring method and device based on data fusion |
| CN115601558A (en) * | 2022-10-24 | 2023-01-13 | 吉林大学（Cn） | Single turnout state detection system and detection method and semi-automatic data labeling method |
| CN115675566A (en) * | 2022-11-11 | 2023-02-03 | 洛阳双瑞橡塑科技有限公司 | Track monitoring system and monitoring method |
| CN116160179A (en) * | 2023-01-03 | 2023-05-26 | 浙江圣洁钛业科技有限公司 | Automatic welding clamping platform of location |
| CN116206265A (en) * | 2023-05-05 | 2023-06-02 | 昆明轨道交通四号线土建项目建设管理有限公司 | Protection alarm device and method for rail transit operation maintenance |
| CN116612457A (en) * | 2023-05-29 | 2023-08-18 | 北京工商大学 | Track bed foreign matter detection method based on deep unsupervised learning |
| CN117058097A (en) * | 2023-08-14 | 2023-11-14 | 安徽环球基因科技有限公司 | Identification method and system for antibody rapid detection reagent |
| CN117079138A (en) * | 2023-09-09 | 2023-11-17 | 西北工业大学 | Building damage assessment method based on multi-view information intelligent extraction and fusion |
| CN117218107A (en) * | 2023-10-07 | 2023-12-12 | 南京铁道职业技术学院 | Rail defect detection method and system |
| CN117392610A (en) * | 2023-11-08 | 2024-01-12 | 大连升泰生物科技有限公司 | Intelligent monitoring method and system for abnormal state of farmed fish |
Non-Patent Citations (5)
| Title |
|---|
| Research on surface defect detection system and method of train bearing cylindrical roller based on surface scanning;Yudong Bao 等;《Journal of Mechanical Science and Technology 》;20230902;4507–4519 * |
| Rongdong Yu 等.Leveraging Graph Databases for Automated OPC UA Information Model Construction.《ICNCC '23: Proceedings of the 2023 12th International Conference on Networks, Communication and Computing》.2024, 294–299. * |
| 基于Faster R-CNN模型X-射线图像的焊接缺陷检测（英文）;郭文明;刘凯;渠慧帆;;北京邮电大学学报;20191231(第06期);24-32 * |
| 基于YOLO改进算法的轨道扣件状态检测研究;王兵水;郑树彬;李立明;钟倩文;;智能计算机与应用;20201231(第01期);145-151 * |
| 基于图像特征的轨道表面缺陷识别研究;郝永旺;魏文军;曹玉新;姜永涛;;电子世界;20180723(第14期);7-9 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN117994753A (en) | 2024-05-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110261436B (en) | Rail fault detection method and system based on infrared thermal imaging and computer vision | |
| Akagic et al. | Pavement crack detection using Otsu thresholding for image segmentation | |
| CN111079747B (en) | Railway wagon bogie side frame fracture fault image identification method | |
| CN110254468B (en) | Intelligent online detection device and detection method for track surface defects | |
| CN111797890A (en) | Method and system for detecting defects of power transmission line equipment | |
| EP0567059A1 (en) | Object recognition system and abnormality detection system using image processing | |
| CN113436157B (en) | Vehicle-mounted image identification method for pantograph fault | |
| CN109766746A (en) | A kind of track foreign matter detecting method of unmanned plane video | |
| Yusof et al. | Deep convolution neural network for crack detection on asphalt pavement | |
| CN107798293A (en) | A kind of crack on road detection means | |
| KR20190024447A (en) | Real-time line defect detection system | |
| CN111080650A (en) | Method for detecting looseness and loss faults of small part bearing blocking key nut of railway wagon | |
| Zhao et al. | Image-based comprehensive maintenance and inspection method for bridges using deep learning | |
| CN111783700B (en) | Automatic recognition and early warning method and system for pavement foreign matters | |
| CN110910363A (en) | Insufficient solder joint detection method, system and medium based on machine vision and deep learning | |
| CN111091551A (en) | Method for detecting loss fault of brake beam strut opening pin of railway wagon | |
| CN113903180B (en) | Method and system for detecting vehicle overspeed on expressway | |
| CN115527364A (en) | Traffic accident tracing method and system based on radar vision data fusion | |
| CN111080599A (en) | Fault identification method for hook lifting rod of railway wagon | |
| CN117994753B (en) | Vision-based device and method for detecting abnormality of entrance track of car dumper | |
| Hashmi et al. | Computer-vision based visual inspection and crack detection of railroad tracks | |
| CN111553500B (en) | Railway traffic contact net inspection method based on attention mechanism full convolution network | |
| CN116110006B (en) | Scenic spot tourist abnormal behavior identification method for intelligent tourism system | |
| Song et al. | Modeling and optimization of semantic segmentation for track bed foreign object based on attention mechanism | |
| CN111325708A (en) | Power transmission line detection method and server |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant |