CN111832415A

CN111832415A - Intelligent truck safety protection system for container hoisting operation

Info

Publication number: CN111832415A
Application number: CN202010541326.0A
Authority: CN
Inventors: 张韶越; 何志成; 尚继辉; 张宾; 陈小虎; 时金利; 张骏
Original assignee: Beiyu Technology Shanghai Co ltd; Aerospace Intelligent Manufacturing Shanghai Technology Co ltd
Current assignee: Beiyu Technology Shanghai Co ltd; Aerospace Intelligent Manufacturing Shanghai Technology Co ltd
Priority date: 2020-06-15
Filing date: 2020-06-15
Publication date: 2020-10-27
Anticipated expiration: 2040-06-15
Also published as: CN111832415B

Abstract

The invention relates to a truck safety intelligent protection system for container hoisting operation. The system mainly comprises a hardware system consisting of field industrial control equipment, a network switch, a signal trigger, a digital camera, a laser radar and other parts; the software system is composed of algorithms such as video analysis, signal processing, communication control and the like. In the process of hoisting the goods carried by the truck, a hardware system collects the goods in real time, a software system analyzes the goods in real time, and three accidents are mainly prevented. Firstly, whether the truck is wrongly hoisted or not, secondly, whether the truck head is hit or not and thirdly, whether the truck is dragged or not and the hanger is hung or not. If the lifting exceeds a certain preset height, or the head of the truck enters a lifting operation area, or the load moves transversely during the unloading process of the lifting appliance, the system sends an alarm to the crane to prevent the damage of the truck and the injury and death of drivers. Compared with the prior art, the artificial intelligence algorithm based on the deep neural network is adopted, various loaded goods, truck trays and truck heads in videos can be intelligently identified, and therefore the functions of preventing the truck from being lifted, preventing the truck heads from being smashed and preventing the trucks from being dragged are achieved by one set of system. The system can intelligently distinguish the load condition of the truck and the types of the pallets, is slightly influenced by the environment, has low misjudgment rate, and can be applied to the application of safety production such as unmanned monitoring in the lifting process of the tire crane container in a port.

Description

Intelligent truck safety protection system for container hoisting operation

Technical Field

The invention belongs to the technical field of machine vision based on an artificial intelligence algorithm, and relates to a video automatic tracking and monitoring technology for a specific interested object.

Background

The truck safety protection system (hereinafter referred to as an ATL system) is mainly used for avoiding accidents of lifting a truck, crashing a head of the truck or dragging a lifting appliance when a wharf tire crane or a track crane unloads the truck. Taking the example where the typical truck load is a container: the handling of containers is a special type of work that, when lifting containers, can cause two serious accidents, because the locking feet of the truck are not fully opened: one is that the spreader lifts the container and truck together or at one end, and the truck is actuated without substantial separation from the load. Both of these accidents can result in damage to the container, spreader and truck, and more seriously, in casualty accidents of the truck driver. Another accident that may occur during the actual hoisting process is that the parking position is incorrect when the truck is unloaded, so that the head of the truck enters the original container hoisting area. At the moment, if a crane driver is left out of observation or cannot observe due to limited conditions, the lifting appliance is still lowered according to the original operation process, and then an accident that the heavy lifting appliance smashes and hits the head of the truck can occur.

The traditional truck safety protection system only focuses on anti-hoisting protection, and has two technical routes of using laser radar alone and using a monitoring camera alone. The former uses laser scanning distance measurement principle, can only obtain real-time data on one scanning line, and then compares the real-time data with a pre-stored template to judge whether a truck is lifted along with a load. However, in practice, the conditions of trucks and loads are very strange, and after the conditions are combined with each other, hundreds of possible conditions exist, data obtained by a single laser scanning line is very non-intuitive, and huge time and management cost are required for developers to pay, and in the case of a crane failure, the data are manually debugged to make all possible template data. In addition, the laser radar is greatly interfered by outdoor complex meteorological conditions such as rain, snow, haze and the like, the lifting system of the laser radar is independently used, the misjudgment rate is high, and the function of preventing the truck head from being smashed cannot be realized.

Compared with the technology of singly using the laser radar, the technology of singly using the monitoring camera has the greatest advantages that real-time video can be obtained, and the technology is very intuitive. Moreover, a single image capture is equivalent to an increase from one dimension of one line to two dimensions of one plane, so that the data volume is greatly increased. However, an increase in the total data amount brings about a simultaneous increase in effective data and interference data. Although it is clear for the monitoring personnel to quickly determine whether there is a lifting accident, the traditional automatic identification algorithm also presents a great challenge. The prior typical digital image anti-lifting recognition method is to artificially divide a plurality of sub-blocks covering a truck tray in an image and then track the image of each frame in the sub-blocks. And calculating the vector value of the vertical upward movement of the current frame relative to the previous frame in the sub-block to judge whether the truck bracket is hoisted. The biggest hidden danger of the method is that sub-blocks of tracking calculation are artificially divided. In particular, variations in truck model, truck load and truck park position can cause the position of the pallet in the video to vary greatly, with the result that the sub-block may not always track exactly the part of the truck tray, may track either truck load causing false positives or track background causing false negatives. Similarly, because the subblock area is very small, the conventional system cannot realize the functions of preventing the vehicle head from being smashed and dragging.

SUMMARY OF THE PATENT FOR INVENTION

The invention aims to overcome the defects in the prior art and provide an automatic monitoring system which adopts an artificial intelligence algorithm to perform real-time video analysis and is used for preventing a truck from being lifted, smashed and dragged. The system has a self-learning function, high recognition rate, wide adaptive environment, quick installation and deployment and high cost performance.

The system utilizes the laser-assisted visual measurement detection principle, integrates artificial intelligence and automatic control technology, can realize real-time detection of the positions of the lifting appliance, the load and the truck, formulates a series of multiple safety control strategies through a software algorithm, can avoid the accidents that the truck is lifted, smashed and pulled due to human errors, and avoids danger while ensuring the field operation efficiency. Typically, when the tire crane works in a port, if accident potential is detected, the system can realize the operation control of the lifting mechanism and ensure the safety of the field bridge operation.

The functions of the invention can be realized by the following technical scheme:

a truck safety intelligent protection system for container hoisting operation mainly comprises hardware devices such as field industrial control equipment, a network switch, a signal trigger, a digital camera, a laser radar and the like. The acquisition subsystem is an algorithm processing system consisting of an image processing software module, a communication module and a trigger control logic. After the system is powered on, the images and the laser radar acquire data in real time, mode recognition is carried out, and according to the crane PLC signal state and the self mode recognition result of the system, the monitoring state is defaulted to enter and the data are transmitted according to the requirement. When the system detects that the truck is hung incorrectly, if the hanging height exceeds a first threshold value, a first alarm is sent to the PLC; if the hoist altitude continues to rise, the alarm is again sent to the PLC and continues until human intervention is involved.

The hardware core of the system is a detection component, and the hardware core of the system comprises two types, namely a digital camera and a laser radar. Both are mounted on the side face, and the anti-lifting alarm is carried out on the truck on one lane. The digital cameras are respectively arranged at two ends of the tire crane, and the installation positions of the digital cameras can ensure that the truck tray and the truck load can enter the camera view field at different truck unloading positions; and meanwhile, when the truck is in a normal unloading position, the head of the truck cannot occupy the range of the camera visual field exceeding 2/3. The laser radar is arranged at the lower middle part of the tyre crane and is 200 mm-1000 mm away from the ground. The scanning plane of the laser radar has an included angle within +/-15 degrees with the horizontal plane, so that a plurality of tires on the side surface of the truck can be transversely swept when the truck is in a normal operation position. The two components are complementary comprehensively, and the separation condition of the boxcar can be accurately identified by one hundred percent.

The system software consists of the following parts: the system comprises a collection service module, a separation identification module, a report data module, a PLC communication module and a fault alarm module. The acquisition service module is an algorithm processing system consisting of an image processing software module, a communication module and a trigger control logic. In the process of loading and unloading the load, the real-time acquisition of images and the real-time transmission of laser radar data are realized; the separation identification module is mainly used for realizing an ATL function, namely monitoring in real time to avoid accidents in the box unloading process; the report data module is mainly used for recording historical data including partial original images and laser radar readings, so that on one hand, the artificial intelligence recognition system can be trained regularly to improve the recognition accuracy, and on the other hand, the reports are kept for later use; and the last PLC communication and fault alarm module is used for realizing data interaction between the system and the crane PLC so as to alarm and indicate abnormal states.

The core of the system software is the algorithm writing of the separation identification module. The method is divided into a laser radar data pattern recognition algorithm and a camera image pattern recognition algorithm. The laser radar data pattern recognition algorithm has the function of automatically recognizing one of the typical light bar patterns so as to determine the current state of the truck and the container and distinguish whether the truck is hoisted by the following container or not. When the truck container is normally separated, the waveform is basically unchanged; once the hub is lifted, the waveform data at the partial tire can change significantly. The waveform of a certain tire on the total waveform data disappears according to the lifted position. And after the system monitors the change, comparing the camera data to find that the situation that the container truck is lifted is really happened, and alarming.

The digital camera image pattern recognition algorithm monitors two areas before and after each other, and each field area is about 2.5m by 1.9 m. The detection capability of three types of interested targets of a truck head, a truck bracket and a truck load is formed through an artificial intelligence algorithm. If the truck head is detected in the image and the proportion of the field of view occupied by the truck head exceeds a preset value (generally 2/3 field area), an anti-smashing alarm is started.

If a truck load and truck tray are detected, then further detecting whether the lower edge of the load is separating from the upper edge of the tray, whether the load has moved laterally, and whether the upper edge of the tray has risen, to determine whether the separation is normal, whether the truck is towing the load, or whether the truck is being hoisted.

In order to reduce misjudgments, the invention also determines the system state and part of system parameters by detecting whether loads exist in the visual field and whether the truck tray has a baffle or not and integrating a plurality of indexes transmitted to the system by the crane PLC. For example, if the truck is unloaded, or the truck tray is an inner tray with baffles and no lock heads, or the lifting appliance is in an unlocking state, and lifting or dragging prevention accidents cannot happen in the states, the system is in a monitoring state only, and false alarm cannot happen.

The invention uses artificial intelligence technology to judge whether the truck is lifted or smash-proof. In particular to an image segmentation algorithm realized by adopting a deep neural network. Neural networks are mathematical models that apply a structure similar to brain neurosynaptic connections for information processing. If the number of layers of the neural network is very large, it can be called a deep neural network. The deep neural network segmentation algorithm used by the invention is composed of a plurality of convolution layers, a pooling layer and a full-connection layer. After an input image enters a network, after a plurality of times of convolution and pooling operations, the input image is changed into a row of feature vector data, and after the feature vectors are subjected to full-connection layer operation, the selection probability of each label class is finally calculated for each pixel. The label categories are divided into four categories, namely truck nose, truck tray, truck load and background. And (4) taking the label with the maximum selection probability, assigning the label to the current pixel, and traversing all the pixels to obtain the segmentation result of the interested object on the picture. This process of inputting an image into a deep neural network with existing parameters and then outputting a classification label image is referred to as forward propagation.

In order to obtain the parameters of each layer of the deep neural network with the best effect, optimal fitting needs to be carried out according to training data. The process of optimizing the fitting adopts a gradient descent method, and the aim of minimizing the loss function is achieved through gradual iteration. And in the process of calculating the gradient once, calculation is started from the output end of the deep neural network, and the gradient of each layer of parameters is obtained by gradually reversing from the output layer to the input layer by utilizing a chain method. This process is called back propagation. The training data for back propagation is obtained from raw data acquired by a manually marked digital camera and a lidar. The more training data, the better the parameters of the network are optimized, and the higher the final recognition rate of the whole algorithm is.

The mass of training data for back propagation is stored on an off-site cloud server, which is called a training end. All labeling and training processes are performed at the training end. A high-performance industrial personal computer used in the system field is called a deployment end, and only the structure and the optimized parameters of the deep neural network model are stored. The deployment end realizes image segmentation through forward propagation on one hand, and stores new effective operation data on the other hand, and periodically and remotely transmits the data back to the training end. And the training end adds the operation data obtained after manual cleaning, sorting and marking into the original training data to form a truck lifting-prevention video big database. The training end carries on training irregularly, optimizes parameters of all layers of the deep neural network, and updates the parameters to the deployment end, so that the recognition rate of various vehicle heads, trays and loads is improved. Therefore, the software composition of the system has certain machine learning capacity, and the intelligent performance is higher and higher along with the accumulation of big data until the accuracy is completely comparable with the accuracy of manual judgment.

Drawings

FIG. 1 is a diagram of the hardware connections of the present invention;

FIG. 2 is a diagram of a deep neural network used in the software system of the present invention;

FIG. 3 is a flow chart of software function determination according to the present invention;

FIG. 4 is a schematic diagram showing the installation positions of main hardware of the system in embodiment 1;

fig. 5 shows the result of the intelligent video analysis performed by the software system of the present invention on the actual picture taken by the digital camera module of the system in embodiment 1. 5a is a front camera picture and 5b is a rear camera picture.

The notation in the figure is:

the system comprises 1-a mathematical camera, 2-a laser radar, 3-a high-performance industrial personal computer, 4-an intelligent video analysis module, 5-a tire crane, 6-a truck (comprising a vehicle head and a bracket) and 7-a truck load (a container is arranged in the place).

Detailed Description

The invention is described in detail below with reference to the figures and specific embodiments. The present embodiment is implemented on the premise of the technical solution of the present invention, and a detailed implementation manner and a specific operation process are given, but the scope of the present invention is not limited to the following embodiments.

Example 1:

the technical scheme of the invention is applied to automatic monitoring of the operation of the tire crane of the actual container wharf, and is used for preventing accidents that a container truck with a bracket lock head (called as an outer container truck) for unloading goods from the beginning breaks down at the head of the container truck and is mistakenly lifted due to the locking of the lock head.

As shown in fig. 4, a digital camera 1 is respectively installed at the front and the rear of the tire crane 5 at a height of about 1.4m from the ground; a laser radar 2 is arranged at the height of about 50cm from the ground at the lower part of the middle part, and the scanning plane of the laser radar forms an angle of about 15 degrees with the horizontal plane; the high-performance industrial personal computer 3 is arranged in an electric room of the tire crane 5. The industrial personal computer 3 is connected with the digital camera 1 and the laser radar 2 through optical fibers to receive data and is connected with a tire crane PLC in an electric room through a network cable.

The working process of the invention is that after the collection truck 6 enters the fields of view of the front and rear digital cameras 1, the system software can automatically identify and segment the region of the object of interest in the field of view, as shown in fig. 5a and 5 b. Fig. 5a shows the segmentation result of the front camera, and it can be seen that the system software automatically identifies three types of objects, i.e., a head (tracker), a container (box), and a truck tray (outer), and their areas. The car head in this example does not occupy more area than the left side 1/3 of the picture, so the anti-pound alarm of the system is not triggered. Fig. 5b shows the segmentation result of the rear camera, and the system automatically recognizes two types of objects and their areas, i.e., the container (box) and the truck tray (outer). When the tyre crane sling begins to unload the container, the sling firstly contacts the upper edge of the container and then locks the lifting lug on the container, at the moment, the PLC firstly sends a sling locking signal to the system, and the process lasts for a plurality of seconds. The system can enter an alarm preparation state according to the PLC signal and the self intelligent analysis result, and continuously updates the height of the real-time container truck bracket and the horizontal position of the real-time container to the reference value in the system until a box locking signal sent by the PLC disappears. And then the tyre crane enters the lifting process, the locking and box-closing signal disappears, the lifting appliance keeps the locking state of the lifting lug lock head on the container, the lifting appliance does not contact the upper edge of the container, and the PLC correspondingly sends a locking and box-closing signal to the system. During the period of locking out the non-box signal, the system continuously does two processes. Firstly, in the current video, the bracket height that intelligent analysis arrived compares with the benchmark value of saving before to realize the function that the truck prevented lifting. And secondly, comparing the horizontal position of the current container with a previously stored reference value, thereby realizing the function of being responsible for preventing dragging. If the height of the current container truck bracket exceeds a reference value of 20cm or the transverse displacement of the container exceeds the reference value of 20cm, an alarm signal is sent to the PLC; if the height exceeds the reference value by 25cm or the displacement exceeds 25cm, the lifting process of the lifting appliance is cut off in addition to the alarm signal until manual intervention.

Through innovation and improvement of the invention, the application example can automatically adapt to various truck head models and bracket models, and the misjudgment rate is continuously reduced along with accumulation of a system database until the accuracy rate of manual monitoring judgment is completely comparable to that of manual monitoring judgment. Therefore, the implementation example can completely replace manual monitoring and guarantee the safety of the tire crane operation process.

The embodiments described above are intended to facilitate the understanding and use of the invention by those skilled in the art. It will be readily apparent to those skilled in the art that various modifications to these embodiments may be made, and the generic principles described herein may be applied to other embodiments without the use of the inventive faculty. Therefore, the present invention is not limited to the above embodiments, and those skilled in the art should make improvements and modifications within the scope of the present invention based on the disclosure of the present invention.

Claims

1. The utility model provides a truck safety intelligence protection system for container hoist and mount operation which characterized in that, this system hardware constitution includes digital camera (1), laser radar (2) and high performance industrial computer (3). The digital camera (1) and the laser radar (2) are connected with the high-performance industrial personal computer (3) through optical fibers.

2. The utility model provides a truck safety intelligence protection system for container hoist and mount operation which characterized in that, this system software constitution includes intelligent video analysis module (4). The intelligent video analysis module (4) adopts a deep neural network algorithm, and can find three types of interested objects, namely a truck head, a truck tray and a truck load in an image through a pixel discrimination method.

3. The intelligent truck safety protection system for container lifting operation as claimed in claim 1, wherein the system hardware is characterized in that the digital cameras (1) are installed at two ends of the tyre crane. The installation position of the digital camera can ensure that the truck tray and the truck load can enter the camera view field at different truck unloading positions; and meanwhile, when the truck is in a normal unloading position, the head of the truck cannot occupy the range of the camera visual field exceeding 2/3.

4. The intelligent truck safety protection system for container hoisting operation according to claim 1, wherein the system hardware is characterized in that the laser radar (2) is installed at the middle lower part of the tyre crane and is 200 mm-1000 mm away from the ground. The scanning plane of the laser radar has an included angle within +/-15 degrees with the horizontal plane, so that a plurality of tires on the side surface of the truck can be transversely swept when the truck is in a normal operation position.

5. The intelligent truck safety protection system for container lifting operation as claimed in claim 2, wherein the system software is characterized in that the video analysis module (4) adopts a deep neural network to realize an image segmentation algorithm. The segmentation algorithm assigns a label to each pixel on the input image. The tag is selected from one of a truck head, a truck tray, a truck load, and a background. The specific selection process is as follows: and transmitting the pictures into a neural network as an input item, and outputting an output item as the selection probability of each pixel for each label class. And taking the label with the maximum selection probability and assigning the label to the current pixel. This process is called forward propagation.

6. The intelligent truck safety protection system for container lifting operation as claimed in claim 5, wherein the system software is characterized in that the video analysis module (4), the deep neural network in the image processing algorithm adopted, and the parameters of each layer of the network are optimally fitted according to the training data. The process of optimizing the fitting adopts a gradient descent method, and the aim of minimizing the loss function is achieved through gradual iteration. The process of calculating the gradient in a single pass is called back propagation. The training data for back propagation is obtained from raw data acquired by a manual marking digital camera (1) and a laser radar (2).

7. The intelligent truck safety protection system for container lifting operation as claimed in claim 6, wherein the video analysis module (4) adopts a deep neural network in an image processing algorithm, and a large amount of training data for back propagation is stored on an off-site cloud server, and the server is called a training end. All labeling and training processes are performed at the training end. And a high-performance industrial personal computer (3) used in the system field is called a deployment end and only stores the structure and optimized parameters of the deep neural network model. The deployment end realizes image segmentation through forward propagation on one hand, and stores new effective operation data on the other hand, and periodically and remotely transmits the data back to the training end. And the training end adds the operation data obtained after manual cleaning, sorting and marking into the original training data to form a truck lifting-prevention video big database. The training end carries on training irregularly, optimizes parameters of all layers of the deep neural network, and updates the parameters to the deployment end, so that the recognition rate of various vehicle heads, trays and loads is improved. Therefore, the software composition of the system has certain machine learning capacity, and the intelligent performance is higher and higher along with the accumulation of big data until the accuracy is completely comparable with the accuracy of manual judgment.