CN111723704A - Raspberry pie-based van body door opening monitoring method - Google Patents

Abstract

The invention discloses a raspberry group-based van body door opening monitoring method, which comprises the steps of acquiring the driving speed of a van based on a distance sensing unit, acquiring an image at the tail part of the van, extracting a door area through the image time difference characteristics or a deep learning network, carrying out edge extraction and morphological processing on the intercepted door image, extracting a straight line set by adopting Hough transform based on the acquired binary image, screening the upper edges of doors at two sides according to the extracted heuristic rule, and calculating the opening of the doors according to the upper edges. And evaluating the state of the truck based on the speed and the opening degree of the carriage door, giving an alarm and feeding back when a potential risk exists, and carrying out arresting management on the truck with the unclosed carriage door at the outlet. The method can quantitatively monitor the opening angle of the freight car door, screen the linear set through heuristic rules, accurately acquire the ridge line of the freight car door, and simultaneously identify the car type and the ROI area of the freight car door by establishing the deep learning network, thereby improving the robustness of the system and reducing the calculated amount.

Description

Raspberry pie-based van body door opening monitoring method

Technical Field

The invention belongs to the field of logistics transportation, and particularly relates to a raspberry pi-based van body door opening monitoring method.

Background

In modern economic society, the ability to quickly respond to the market and customers and to establish a supply chain has become one of the most important factors for inter-enterprise competition. Therefore, the cultivation and improvement of the logistics capacity of enterprises become an important driving force for the development of modern logistics. The logistics park has obvious scale advantages and aggregation effects in the aspects of economic scale, geographical distribution, construction operation modes and the like. At present, China basically forms the construction and development situation of logistics parks from south to north and from east to west in China. The logistics park has large occupied area, generally takes the land for storage, transportation, processing and the like as the main part, and also comprises certain facility land for information, consultation, maintenance, comprehensive service and the like matched with the logistics park. After the logistics park appears, the pressure of logistics on urban traffic is reduced, the scale benefit of logistics operation is improved, and the requirements of large-scale development trend of warehouse construction and development of goods intermodal transportation are met. As a world logistics large country, more than 2000 logistics parks exist in China, and the construction of logistics infrastructure achieves positive results.

Although the logistics park in China has achieved obvious effects, a lot of new technical requirements are generated in development. For example, the logistics park has insufficient service capacity and low informatization and networking levels. Means of data acquisition and interaction in each link of logistics transportation are not abundant enough, and monitoring measures are not enough. The monitoring and standard requirements of the truck on the access and traffic management of the logistics park are more and more prominent.

Compared with other common trucks, the van truck has the advantages of flexibility, high working efficiency, large transportation amount, safety, reliability and the like. Therefore, the van is widely used in logistics transportation. The van truck is all-weather, can finish transportation tasks on various roads in the city and in urban areas, is safer and tidier than common trucks, and can not wet goods when raining. The van must set up the back door, and the setting of door provides the condition for the loading and unloading goods, and the opening angle of door must reach certain design requirement, just can accomplish the safe loading and unloading of goods completely and give full play to van's service function. According to the specification of ZBT52006 general technical condition of van, the opening angle of the rear door of the van is 270 degrees.

If the compartment door of a van in logistics transportation is not closed when the van is driven at a high speed, the goods can fall off to cause economic loss; the car door swaying left and right can hurt the pedestrians, and traffic accidents are caused. There are many reports about such accidents, such as in Ningbo, a heavy van is not firm because of the door, and the door is opened to damage the height limiting rod of the toll station; in Wenzhou, a truck has a car door which is not closed, so that a walking old man is hit by the car door; in addition, in Shanxi, the rear compartment door of the vehicle is not closed, and the vehicle is thrown away in the driving process, so that the passing vehicles are threatened to avoid, and serious traffic accidents are caused by the difference.

At present, some technical schemes related to car door detection exist, such as the chinese patent application with application number 2018102987891, whether a car door is closed is judged through the on-off of a contact block, the chinese patent application with application number 2016200489718 acquires the opening and closing information of the car door through a contact type mechanical door lock, and the chinese patent application with application number 2017109280143 realizes the sensing of opening degree detection through a photoelectric emitting and receiving module so as to realize the monitoring of illegal opening of a private car. The above schemes are all used for detecting the state of the vehicle door from the inside of the vehicle, and external supervision cannot be realized.

The freight car brings great potential safety hazard for road transportation because the compartment door is not closed, therefore, in the management of the access of a logistics park, equipment and a system which can automatically identify whether the compartment door of the freight car is closed are urgently needed, and the automatic detection, reminding and control of the state of the compartment door of the compartment type freight car can timely prevent the occurrence of cargo loss and personnel injury accidents.

The raspberry pie has been proposed as a single board computer for many years, and although the raspberry pie is proposed for the field of education at first, due to the open source hardware characteristic, a large amount of support of the open source community is obtained, the related software resources are abundant, and the speed of developing new applications can be faster. Currently, a widely used raspberry pi 3B has a plurality of camera interfaces. Only the credit card-sized raspberry pie has strong processing capability and abundant interfaces, is widely used in various mobile or embedded solutions, and has high cost performance.

SUMMARY OF THE PATENT FOR INVENTION

In view of the above requirements, the invention aims to provide a raspberry-group-based van body door opening monitoring method and system, which are used for monitoring the state of a van door of a truck at positions such as an unloading opening, a road, an entrance and an exit in a cargo yard, limiting the travel of the truck in an unreasonable state and giving out a warning in time. And as the freight train is frequently required to carry out multi-point loading and unloading of cargos in the whole truck in the park, the truck door can be fixed through the chain lock without being completely closed when the truck door is allowed to move at low speed in the park for convenient operation. Therefore, corresponding to the operation characteristics of the logistics park, the invention also needs to effectively monitor the opening degree of the compartment door of the van, namely the opening degree.

The truck carriage opening monitoring method provided by the invention takes a raspberry group as a processing platform, obtains the opening of a carriage door by acquiring images of a truck from the outside and obtaining characteristics of the carriage door through image processing, judges the state of the truck by matching with a distance sensor to detect the speed of the truck and gives an alarm feedback when a potential risk exists, carries out arresting management on the truck with the carriage door not closed at an outlet and requires timely rectification and elimination of hidden danger. In order to extract the contour line of the carriage door from the image with rich and complex information, the characteristics of the carriage door area need to be intercepted firstly, the invention adopts two differential images to respectively obtain the upper boundary line and the lower boundary line of the carriage door area according to the characteristics of an application occasion, and extracts two long ridge lines of the carriage top according to the characteristics of a carriage, thereby obtaining the upper frame of the carriage door. And (3) aiming at the intercepted carriage door image, obtaining a binary edge image through image processing, screening the upper edges of the carriage doors on two sides according to a heuristic rule summarized from the image characteristics based on Hough transform, and calculating the opening degree of the carriage doors according to the upper edges. In order to obtain the exact door opening angle in the world coordinate system, the images are transformed into the world coordinate system by perspective transformation before or after the differential images are obtained.

In order to enable the door area of each type of truck to utilize the size of images as much as possible by adjusting the pitching of a camera, optimize the quality of filed door images and improve the robustness, accuracy and precision of detection processing, a yolo-v 3-tiny-based improved deep learning network model is established in a monitoring controller, the network is trained through a training sample of images fused by collected truck side front main body images and side images, and the angle for carrying out image collection on the tail of the truck is adjusted based on truck types identified by the network model during online application. Meanwhile, based on the network, the method for acquiring the door area can be replaced, and specifically, by adding the truck tail image marked with the door area in the training sample, the anchor frame of the door area in the truck tail image can be acquired based on the network during online application.

The technical scheme of the invention is that the invention provides a raspberry pi-based van body door opening monitoring method, which comprises the following steps:

s1, calculating the running speed V of the truck based on the trigger interval of the first detection module and the second detection module in the distance sensing unit;

s2, acquiring a truck number by scanning an ID card in the truck, and continuously acquiring a first tail image and a second tail image of the truck based on the triggering of a third detection module;

s3, carrying out differential operation on the first tail image and the second tail image to obtain a first differential image, and carrying out differential operation on the second tail image by taking the road surface image as a background to obtain a second differential image;

s4, based on the first and second difference images, marking the longitudinal position of the transverse continuous non-0 element length exceeding the set value, marking the longitudinal position of the longitudinal continuous non-0 element length exceeding the set value in a row mode, and intercepting the second tail image by taking the minimum common external rectangle of the row and column marks as a reference to obtain a freight car door image;

s5, converting the freight car door image into a first car door image under a world coordinate system based on a perspective inverse transformation matrix, carrying out graying processing on the first car door image to obtain a second car door image, carrying out binarization processing on the second car door image to obtain a third car door image, and carrying out edge extraction on the third car door image to obtain a fourth car door image;

s6, detecting a straight line in the fourth carriage door image based on Hough transform, screening by a heuristic rule of the azimuth relation between the straight line and a first row transverse line in the row mark to obtain an upper carriage door frame and an upper carriage door edge line, and calculating the opening degree of the carriage door according to the obtained result;

and S7, evaluating the running state of the truck according to the current speed and the opening degree of the carriage door of the truck and managing and controlling the truck and/or the gateway controller according to the evaluation result.

Optionally, the step S4 includes the following processing: searching a first transverse line with the length exceeding a set value and the row number being minimum in the first differential image, searching a second transverse line with the length exceeding the set value and the row number being maximum in the second differential image, and searching a first column line and a second column line with the length exceeding the set value and the column number being minimum and maximum respectively in the second differential image; taking the minimum common external rectangle of the first and second transverse lines and the first and second row lines as a reference, and taking the minimum common external rectangle as an ROI area of the truck tail image after expanding a certain range outwards;

the step S7 includes the following processing: and when the speed V is greater than a set value and the opening alpha of the car door is greater than the set value, an alarm signal is sent to a vehicle-mounted alarm and an acousto-optic unit through an output module, and/or a brake falling instruction is sent to a gateway controller.

Optionally, the step S1 includes the following processing: if the time interval between the times of the vehicle detected by the first and second detection modules is recorded as Δ t1 and the time interval between the times of the vehicle leaving the first and second detection modules is recorded as Δ t2, the calculation formula of the speed V is:

wherein L is the distance between the first and second detection modules.

Optionally, the inverse perspective transform matrix in step S5 is obtained by:

let the perspective inverse transform matrix be:

it converts the pixel point coordinates (u, v,1) in the image plane coordinate system to point coordinates (x, y,1) in the world coordinate system:

(x,y,1)＝(u,v,1)*T，

acquiring two coordinates of four pixel points by detecting imaging points of four known reference objects which are distributed in a rectangular shape on a road in an image, substituting the coordinates into the conversion formula to obtain eight linear equations, and obtaining eight linear equations from the linear equations and the equation a₁₃+a₂₃+a₃₃Solving to obtain the perspective inverse transformation matrix under the constraint of an equation set formed by 1;

and, the cargo door image is transformed into a first door image in a world coordinate system by:

optionally, the step S5 includes the following processing:

graying processing is carried out by adopting a mode of carrying out weighted average on R, G, B components, an Otsu threshold value method is adopted in binarization processing, the third compartment door image is obtained after multiple times of corrosion, expansion and corrosion after threshold value segmentation, and edge detection is carried out by adopting a Canny operator.

Optionally, the step S6 includes the following processing:

marking the intersection points of the first transverse line in the second differential image and the first and second alignment lines respectively by taking the first transverse line in the line mark as a first transverse line and the head and tail alignment lines in the line mark as a first alignment line and a second alignment line respectively, and correspondingly marking the left and right intersection points as PHL and PHR respectively in the first door image,

then, the detection and screening of the straight line are carried out:

first, for the fourth door image I₀After Hough transform is carried out on each foreground pixel point, a theta-r first plane matrix corresponding to the image is obtained through accumulation, wherein r represents the distance of a vertical line of a straight line from an original point, theta is the angle of the vertical line to a transverse axis,

then, cleaning the first plane matrix, and removing the value smaller than a set threshold value T_houghObtaining a second planar matrix

Each of which is not 0 element l_i,j(r_i,j,θ_i,j) Corresponding to a straight line; for image I₀Traversing and cleaning are carried out, only pixel points corresponding to the linear elements in the second planar matrix are left, and the linear elements l corresponding to the pixel points are identified_i,j(r_i,j,θ_i,j) Obtaining a cleaned image I₁(ii) a By applying to the image I₁Go through the traversal as the matrix

Each non-0 value of the linear elements in the list C_i,jTo record the image I corresponding to the straight line₁All the pixels in the chain table are arranged from small to large according to the abscissa of the pixel, and meanwhile, an array L is established and a first column element L of the array L is used_i,jTo record linked list C_i,jThe second column element is used for recording indexes i and j of the corresponding linked list; sorting the array L from big to small according to the first row element values, recording the sorting result as an array PL,

thirdly, straight lines and elements corresponding to the carriage door upper frame parallel to the first transverse line and the carriage door side edges connected with the carriage door upper frame are searched, the first N rows of elements of the PL are counted, the elements corresponding to the carriage door upper frame and the side edges are removed, and the corresponding linked lists are correspondingly removed in a linked list set,

and circularly performing the following processing on the updated array and the chain table, wherein k is from 1 to the line number of the array PL:

aiming at the kth row element of the array PL, obtaining indexes i and j according to the value of the second row element, and traversing the linked list C corresponding to the elements_i,jAnd obtaining the distance dL between the minimum abscissa pixel point PL and the first transverse line and the distance dR between the maximum abscissa pixel point PR and the first transverse line in the nodes, if min (dL, dR)<ds, then the linked list C corresponding to the element_i,jAs the carriage door upper frame candidate linked list;

and finally, calculating the opening degree of the carriage door for each candidate linked list:

if it corresponds to a straight line l_i,jTheta of_i,jSatisfies | theta in a predetermined neighborhood of pi/2_i,j-π/2|＜θ_sRespectively determining that the opening degrees of the left compartment door and the right compartment door are pi/2 according to the closer of the abscissa of the point PL and the abscissa of which point of the upper frame end point PHL and PHR of the compartment door,

otherwise, if theta_i,jIf the opening degree of the compartment door is more than pi/2, the calculation formula of the opening degree of the compartment door is defined as,

otherwise if theta_i,jIf the opening degree of the compartment door is less than pi/2, the calculation formula of the opening degree of the compartment door is defined as,

otherwise, defining the opening degree of the compartment door to be 0;

finally, when the opening degree of the carriage door is not 0, the angle theta is determined according to the first transverse line_UTo correct the degree of opening of the valve,

optionally, the step S6 includes the following processing:

pixel point (u, v) and line element l_i,j(r_i,j,θ_i,j) The corresponding judgment mode is as follows:

taking a branch point H (r) on the straight line_i,j·cosθ_i,j,r_i,j·sinθ_i,j) Denoted as H (u ', v'),

calculating J-r²- (u-u '+ v-v'), if | J | ≦ then determining that the pixel point corresponds to the straight line, otherwise not, wherein the pixel point is a preset small positive number;

to point M (x)_M,y_M) To a straight line l_i,j(r_i,j,θ_i,j) Is calculated using the following equation:

d＝|y_M·sinθ_i,j+x_M·cosθ_i,j-r_i,j|。

optionally, the step S6 includes the following processing:

before the linear screening, the image is rotated by a preset angle such as 20-50 degrees by using a second perspective transformation, the characteristic line of the carriage door is obtained through the linear screening, then the characteristic line is converted into a world coordinate system by using a second perspective inverse transformation, and then the opening degree of the carriage door is obtained.

Optionally, the step S6 includes the following processing:

first, for the fourth door image I₀Selecting a pixel point from the leftmost upper end as a seed point, and establishing a linked list C comprising the pixel point_kSearching the next point within the range of the threshold value ds in the field from left to right and from top to bottom, and recording the position (u, v) of the next point in the linked list until the searched point is empty; repeating the above processing until each foreground point belongs to a linked list; screening the linked list set, and removing the linked list with the length less than a set value;

second, for each chain table C after screening_kGenerating a binary image with the node positions (u, v) corresponding to the foreground points, and after carrying out Hough transform on the image, only keeping the element theta-r with the maximum value, and marking as l_k(r_k,θ_k)；

Third, for each straight line l_kAccording to its corresponding linked list C_kAnd obtaining the distance dL between the minimum abscissa pixel point CL and the first transverse line and the distance dR between the maximum abscissa pixel point CR and the first transverse line in the nodes, if min (dL, dR)<ds, then the linked list C corresponding to the element_kAs the carriage door upper frame candidate linked list;

and finally, calculating the opening degree of the compartment door for each candidate linked list.

In another embodiment of the present invention, there is also provided a raspberry-based van body door opening monitoring method, which includes the following steps:

p1, establishing an image recognition module: the image recognition module adopts a network model improved based on yolo-v3-tiny, takes 832 multiplied by 832 images as input, and inserts a convolution layer with a convolution kernel of 3 multiplied by 3 and a pooling layer of 2 multiplied by 2 in sequence before the 0 th layer of the yolo-v3-tiny network, and the number of the filters of the two layers is 8;

p2, obtaining training samples: acquiring images of the vehicle at a preset position of the channel, and forming a training data set by taking a truck tail image marked with a compartment door area as an image sample;

p3, training the network model off line: performing parameter configuration on network training, and performing offline training on the network model by using the acquired data set to obtain an image recognition model;

p4, on-line monitoring of the opening of a freight car:

SP1, calculating the running speed V of the truck based on the trigger intervals of the first detection module and the second detection module; acquiring a truck number by scanning an ID card in the truck;

SP2, acquiring a current tail image of the truck based on the triggering of a third detection module, obtaining a door area anchor frame after the image identification model processing, and intercepting the current tail image by taking the anchor frame as a reference to obtain a truck door image;

SP3, extracting the edges of the truck carriage door images, extracting a straight line set through Hough transform, searching two longest cargo carriage top ridge lines with an angle in the pi/2 neighborhood from the straight line set, and labeling the intersection points of the ridge lines and the upper edge of the anchor frame in the truck carriage door images;

SP4, converting the freight car door image into a first car door image under a world coordinate system based on a perspective inverse transformation matrix, carrying out graying processing on the first car door image to obtain a second car door image, carrying out binarization processing on the second car door image to obtain a third car door image, and carrying out edge extraction on the third car door image to obtain a fourth car door image;

SP5, detecting a straight line in the fourth carriage door image based on Hough transform, screening by a heuristic rule of the orientation relation of the straight line and the marked intersection point to obtain an upper carriage door frame and an upper carriage door edge line, and calculating the opening degree of the carriage door according to the obtained result;

and SP6, evaluating the running state of the truck according to the current speed and the opening degree of the carriage door of the truck and managing and controlling the truck and/or the gateway controller according to the evaluation result.

Preferably, the step P2 further includes the following steps:

two pictures collected by a side-view camera and a main camera positioned in front of the side of the vehicle are fused into a sample picture in an up-and-down arrangement mode, and the sample marked with the vehicle type is added into the training data set;

the step SP2 further includes the following steps:

the image preprocessing module fuses vehicle pictures acquired by the main camera and the side-view camera into pictures to be detected in a row arrangement mode and then inputs the pictures into the image recognition model to obtain vehicle type information of the vehicle; adjusting the pitch angle of a camera holder and/or the focal length of a camera according to the identified vehicle type, and acquiring a current tail image of the truck;

the step SP4 further includes the following steps:

and determining the perspective inverse transformation matrix according to the preset relation between the pitch angle and the perspective inverse transformation.

Preferably, the truck transportation information is interacted with a database of the server through a communication interface, and information such as a door image, a truck number and a door opening degree is stored in the database.

Preferably, the monitoring controller controls the lighting unit to supplement light based on the detection of the environment by the illuminance sensing unit.

Compared with the prior art, the scheme of the invention has the following advantages: according to the method, the tail images of the trucks in the driving process of the logistics park are collected, the extraction process and the opening degree calculation method of the truck carriage door features are developed based on the analysis and research of the application situation characteristics, the opening degree of the truck carriage door can be accurately obtained, and therefore the safety of the van transportation is improved. Meanwhile, the quantification of the opening degree also provides a basis for the efficiency optimization of multi-point loading and unloading in the goods yard. Compared with the traditional method of relying on vehicle-mounted sensing detection, the method can provide external supervision, ensure that the compartment door is in a closed state before the delivery of the logistics park, and improve the service level of the park. The processing mode of setting the ROI area for the freight car door and the corresponding judgment processing process of the pixel point and the straight line element after Hough transformation both reduce the calculated amount and improve the processing efficiency. The application of heuristic rules in the screening of the edges of the compartment door can accurately acquire the ridge lines of the compartment door. Based on the yolo-v3-tiny improved deep learning network, the van door area of the truck tail image of each vehicle type can be rapidly obtained, the image acquisition angle of the overhead camera is adjusted based on the vehicle type, and the robustness and the precision of the detection of the opening degree of the van door are improved; and the network is suitable for the calculation force characteristics of the raspberry group and the characteristics of the collected images, so that the device based on the movable embedded type raspberry group platform can effectively process the automatic management of truck transportation in a logistics park.

Drawings

FIG. 1 is a view of a raspberry-based door opening monitoring device and system for a van body;

FIG. 2 is a block diagram of a raspberry pi-based van body door opening monitoring controller;

FIG. 3 is a schematic view of a door area of a van;

FIG. 4 is a schematic view of a logistics park road; FIG. 5 is a fragmentary schematic view of a truck haul road;

FIG. 6 is a partial schematic view of a transport corridor in front of a barrier; FIG. 7 is a partial schematic view of a loading and unloading port;

FIG. 8 is a diagram of an image recognition module deep learning network architecture; FIG. 9 is a wagon tail image;

FIG. 10 is a truck tail image under a world coordinate system after the perspective is converted;

FIG. 11 is a schematic view of a region ROI of an image of a truck tail; FIG. 12 is a door grayed image;

FIG. 13 is a binary image of a door; FIG. 14 is a binarized image after morphological processing;

fig. 15 is a screened binary image of the door edge.

Wherein:

10000 van type freight car door opening monitoring system; 1000 van body door opening monitoring devices, 2000 servers, 3000 vehicle mounted alarm devices, 4000 gateway controllers;

the system comprises a 100 monitoring controller, a 200 user interface unit, a 300 acousto-optic unit, a 400 illumination sensing unit, a 500 illuminating unit, a 600 distance sensing unit, a 700 image acquisition unit, an 800 communication interface and a 900 scanning identification unit;

910 scanning for terminals; 710 switching array, 720 camera, 721 primary camera, 722 overhead camera, 723 side camera; 601 a first detection module, 602 a second detection module, 603 a third detection module;

the system comprises a 110 input module, a 120 main processing module, a 130 image preprocessing module, a 140 image identification module, a 150 opening degree calculation module, a 160 output module and a 170 storage module;

a van 10, a vehicle passage 11, a pedestrian passage 12 and a loading and unloading port (operation port) 13;

21 upright posts, 22 lifting and releasing rotating shafts and 23 brake levers; 31 road sign.

Detailed Description

Preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, but the present invention is not limited to only these embodiments. The invention is intended to cover alternatives, modifications, equivalents and alternatives which may be included within the spirit and scope of the invention.

In the following description of the preferred embodiments of the present invention, specific details are set forth in order to provide a thorough understanding of the present invention, and it will be apparent to those skilled in the art that the present invention may be practiced without these specific details.

The invention is described in more detail in the following paragraphs by way of example with reference to the accompanying drawings. It should be noted that the drawings are in simplified form and are not to precise scale, which is only used for convenience and clarity to assist in describing the embodiments of the present invention. The front and rear of the invention are relative, according to the advancing direction of the vehicle.

Example 1

At present, the logistics park in China develops rapidly, so that a plurality of technical requirements are brought forward, such as the management and service capacity, the informatization level and the like of the logistics park are all to be improved. In order to perfect data acquisition and automatic management capacity in each link of logistics transportation, aiming at the problem that the state of a van carriage door of a van truck in a logistics park is lack of monitoring at present, the system and the device for monitoring the opening degree of a van carriage based on raspberry serving are provided by the embodiment by combining the demand of logistics transportation service and the characteristics of a raspberry serving processing platform.

As shown in fig. 1, the method of the present invention employs a raspberry-based van transportation management control system 10000, which includes a monitoring controller 100, a user interface unit 200, an acousto-optic unit 300, a distance sensing unit 600, an image acquisition unit 700, a scanning identification unit 900, and a communication interface 800, where the communication interface 800 is further connected to a gateway controller 4000 and a server 2000.

The acousto-optic unit 300 performs information prompt on a truck driver and an operator through sound and/or light, for example, normal/abnormal states are represented by different colors of light and twinkling, and state prompt is performed through voice information; the user interface unit 200 includes an operation panel and a display screen for entering parameters, initiating operations, and performing information interaction; according to the detection of the distance sensing unit 600 on the vehicle, the image acquisition unit 700 performs time-sharing multi-angle image acquisition on the vehicle after being triggered, and the scanning identification unit 900 acquires information such as a truck number by scanning an ID card in the truck.

In logistics transportation, van trucks have the advantage of being all-weather, and are therefore heavily used. As shown in fig. 3, the van has a rear door, and the cargo door is provided to facilitate loading and unloading of cargo. However, the loading and unloading time periods of the trucks are all over the day, the labor intensity of the operation is high, particularly, workers are easy to get tired at night, and the drivers and the loaders are often separated from each other, so that the drivers and the loaders are not smooth to hand over after the loading and unloading operation is finished, and the situation that the inspection of whether the door of the truck compartment is closed is lack of main responsibility is easily caused, so that accidents of personnel injury and material loss caused by opening of the door of the truck compartment are frequently caused in news reports. Therefore, in the management of the logistics park, a method and a system for automatically monitoring the state of the doors of the access and moving trucks from the outside are needed. Moreover, for a logistics park, goods are often assembled on trucks, and the compartment door can be fixed through a chain lock when the truck is allowed to move at low speed in a field for convenient operation between a plurality of close-distance loading and unloading places. Therefore, the opening degree, namely the opening degree, of the compartment door of the van is monitored by combining the operation characteristics of the logistics park, and the abnormality is found in time and is alarmed.

Referring to fig. 1, in the monitoring system, a raspberry-based van body door opening monitoring device 1000 as a main body takes an embedded processor raspberry as a main control processing platform, and includes the monitoring controller 100, a user interface unit 200, an acousto-optic unit 300, a distance sensing unit 600, an image acquisition unit 700, a scanning identification unit 900 and a communication interface 800.

The image acquisition unit 700 in turn includes a switching array 710 and a plurality of cameras 720, with the monitoring controller effecting acquisition of image information of the vehicle from one or more of the plurality of cameras through the switching array 710. As shown in fig. 5 and 6, the camera 720 includes a main camera 721 for capturing an image of the vehicle in front of the tunnel side, an overhead camera 722 for capturing an image of the rear door of the vehicle, and a side-view camera 723 for capturing an image of the side of the vehicle. The time-sharing and multi-angle image acquisition is carried out on the truck through the cameras arranged at the loading and unloading port and the barrier gate area, the time-sharing acquisition of the image is triggered through the differential detection of the distance sensing unit 600, namely, the image in a specific direction is acquired when the truck is captured to move to a preset position.

Based on the acquired image information, the raspberry-based van body door opening monitoring controller extracts the image characteristics of the van body door. As shown in fig. 1 and 2, the monitoring controller 100 includes an input module 110, a main processing module 120, an image preprocessing module 130, an image recognition module 140, an opening calculation module 150, an output module 160, and a storage module 170. The input module 110 obtains setting parameters and user operation instructions, vehicle position signals collected by each vehicle detection module, and vehicle-mounted radio frequency card ID information through the user interface unit 200, the distance sensing unit 600, and the scanning identification unit 900. The storage module 170 is used for storing intermediate data, archived files and the like in the information processing process of each module; the output module 160 transmits the control information of the main processing module to the gateway controller 4000, the vehicle-mounted alarm 3000, and the like through the external communication interface 800, and the input and output modules also perform information interaction with the external server 2000 through the communication interface.

As shown in fig. 4, after a truck enters a yard in a logistics park, the truck passes through a vehicle passage 11 to pass through various warehouses, cargo handling points A, B, and the like, and in order to improve road safety, some parks divert the flow of people and vehicles and move pedestrians and non-motor vehicles through a pedestrian passage 12. Referring to fig. 4, 5, 6 and 7, in order to monitor the truck door, the present invention obtains an image of the truck tail through an overhead camera 722, and then extracts the features of interest through processing the obtained image, so as to obtain the opening degree of the truck door. In order to alarm the truck in time when the abnormal state is monitored, the number of the truck is obtained through the scanning terminal 910 in the scanning identification unit on the inverted L-shaped walking bar, and a warning signal is sent to the vehicle-mounted alarm 3000 on the truck through the wireless communication interface.

Specifically, as shown in fig. 1, 5 and 6, when the vehicle moves forward from the road shown in fig. 5 to reach the barrier area shown in fig. 6, the plurality of cameras in the image capturing unit are respectively disposed at different positions of the vehicle driving passage, and the distance sensing unit 600 includes a first detection module 601 and a second detection module 602 which are sequentially disposed at the side of the road in the vehicle moving direction, and a third detection module 603 which is located above the center of the road in front of the second detection module 602; the third detection module 603 is fixed to the inverted-L bar supported by the column 21 together with the overhead camera 722 and the scanning terminal 910.

The overhead camera in the image acquisition unit is triggered a preset distance behind the vehicle leaving the third detection module. Referring to fig. 5, preferably, before the transportation road or the barrier, the overhead camera 726 is triggered after the vehicle leaves the third detection module 603 for a preset time period, for example, 3 seconds after leaving, to capture the truck tail image. In the loading/unloading port shown in fig. 7, the third detection module may be fixed at a predetermined distance in front of the vehicle. Fig. 7 shows a refrigerated cargo handling area, and in the case of the square-partitioned type adopted for ordinary cargo handling, an overhead camera may be installed in front of the handling point in the manner shown in fig. 5 and 6.

The first detection module and the second detection module are positioned beside the road and are also used for detecting the running speed of the truck. And a main processing module in the monitoring controller calculates the running speed V of the vehicle by recording the time interval delta t1 when the vehicle is detected by the first detection module and the second detection module and the time interval delta t2 when the vehicle leaves the first detection module and the second detection module:

wherein L is the distance between the first and second detection modules.

For the multipoint loading and unloading shown in fig. 4, when the vehicle speed is detected to be greater than the set value and the opening degree of the car door is detected to be greater than the set value, the output module sends out an alarm signal to the vehicle-mounted alarm and the acousto-optic unit in a wireless communication mode through the communication interface.

In order to extract the characteristics of the carriage door from the image of the tail part of the truck, the invention intercepts the part of the carriage door area from the image and processes the part of the carriage door area in order to reduce the calculation amount. Specifically, the overhead camera is used for successively acquiring images twice, and a first tail image and a second tail image of the truck are respectively obtained, and optionally, both are represented by grayscale images. Carrying out differential operation on the first tail image and the second tail image to obtain a first differential image; and taking the road surface image collected by the overhead camera as a background image, and carrying out difference operation on the second tail image and the background image to obtain a second difference image.

Aiming at the first differential image, searching a first transverse line with the minimum line number and the number of non-0 elements exceeding a set value, such as 10 pixels, in the image according to a connection relation from top to bottom in the line direction; and searching a second transverse line with the number of non-0 elements exceeding a set value and the largest row number in the image from bottom to top in the row direction for the second differential image. Similarly, the number of non-0 elements in the second differential image, namely the first column line with the length exceeding the set value and the smallest column number is searched from left to right, and the second column line with the length exceeding the set value and the largest column number is searched from right to left. As shown in fig. 11, the minimum common circumscribed rectangle of the first and second horizontal lines and the first and second column lines is used as a reference, and the ROI is obtained by expanding a certain range outwards, for example, by 10-20 pixels in the horizontal and vertical directions, and then the ROI is taken as a wagon tail image from the second tail image.

Preferably, the setting values are selected proportionally according to the width of the image resolution, such as 1/40-1/20, which is preferably the transverse width or the longitudinal height. Preferably, the background image may be acquired periodically, and the last background image before the tail image acquisition is triggered is used as a reference background image.

Preferably, before the search is performed, non-0 elements in the first and second difference images are replaced by 1, and then the first and second difference images are average-filtered by a horizontal operator and a vertical operator, respectively, and the filtered values are rounded. The horizontal operator and the vertical operator are vectors with certain lengths and element sums of 1. Through the filtering processing, the burr signals of non-car door areas such as a car roof and the like caused by the change of the image acquisition parameters can be removed.

In order to obtain the position, the size characteristic and the opening degree of the carriage door in the world coordinate system, the image of the freight carriage door collected by the overhead camera is converted into an image in the world coordinate system by using a perspective inverse transformation matrix. Let the perspective inverse transform matrix be:

it converts the pixel point coordinates (u, v,1) in the image plane coordinate system to point coordinates (x, y,1) in the world coordinate system: (x, y,1) ═ u, v,1) × T.

When the shooting angle is kept unchanged during image acquisition of the overhead view camera, the matrix can be obtained through the standard reference object in an off-line state and preset, and the preset matrix parameters can be directly called during on-line application. Preferably, as shown in figure 5,road signs 31 comprising four reference points in a rectangular distribution may also be provided as reference objects on the road. Detecting the imaging points of the four reference points in the image to obtain two coordinates of the four pixel points, substituting the coordinates into the conversion formula to obtain eight linear equations, and obtaining the eight linear equations from the linear equations and the equation a₁₃+a₂₃+a₃₃And solving the constraint of nine linear equation sets formed by 1 to obtain the inverse perspective transformation matrix. Then, the cargo door image may be transformed into a first door image in a world coordinate system by:

preferably, the coordinates of the four reference points in the world coordinate system are expressed as relative values in a ratio of the length to the width of the rectangle. Preferably, the four reference points are marked by using different colors of reflected light different from the background road surface as rectangular blocks, and after the gray scale image is subjected to edge acquisition, the center of gravity of each searched rectangular block is used as the image coordinate of the corresponding reference point.

The freight car door image is transformed into a first car door image under a world coordinate system based on the inverse perspective transformation matrix, then the door opening degree characteristic is extracted from the image, and the validity verification can be carried out on the image through angle measurement under the world coordinate system. Preferably, the inverse perspective transformation and the graying processing may be performed before the ROI region is cut out, as shown in fig. 10.

With reference to fig. 12 and 13, a second car door image is obtained by performing a graying process on the first car door image, and a third car door image is obtained by performing a binarization process on the second car door image. The graying processing adopts a mode of carrying out weighted average on R, G, B components, and the binarization processing adopts an Otsu threshold method. As can be seen from fig. 13, the directly obtained binary image includes many foreground detail features that are irrelevant to the extraction of the car door features, such as door locks, lock catches, license plates, reflective strips, and the like. However, due to the apparent gradient change in illumination, the perspective images inside and outside the car door are distinct. Based on the observation and analysis of these features, the present invention obtains the binarized image of the door core area as the third door image by performing a plurality of times of erosion, expansion and mathematical morphology processing on the binarized image, as shown in fig. 14. In fig. 14, the black part is the area where the door is opened, and there are some small pieces of black interference information around the area, but the target area of the door is basically distinguished from the background, and the morphological processing greatly simplifies the calculation of the subsequent door feature extraction.

Referring to fig. 15, after edge detection is performed on the third door image, a fourth door image is obtained, then straight lines in the fourth door image are detected based on hough transform, and after the straight lines and the heuristic rule of the azimuth relation of the first horizontal line are screened, an upper door frame and an upper edge line of the door are obtained, and the opening degree of the door is calculated according to the obtained upper door frame and upper edge line. Preferably, the edge detection uses Canny operator, wherein in fig. 15, the image is processed in reverse color for easy observation.

And marking intersection points of the first transverse line and the first and second row lines in the second differential image according to the row and column characteristics, and correspondingly marking the left and right intersection points as the end points PHL and PHR of the upper frame of the car door in each car door image. Specifically, the detection and screening of the straight line of the door edge are carried out according to the following processes:

first, for the fourth door image I₀And after Hough transformation is carried out on each foreground pixel point, accumulating to obtain a theta-r first plane matrix corresponding to the image, wherein r represents the normal distance between a straight line and the origin, and theta is the angle of the vertical line relative to the horizontal axis. Specifically, a first plane matrix is established, wherein the value range of a row vector theta is 0-pi, r is obtained for each foreground point by using a transformation equation r ═ u · cos theta + vssin theta based on theta, and 1 is added to the matrix, namely theta row r column elements corresponding to the array according to the values of theta and r.

Then, cleaning the first plane matrix, and removing the value smaller than a set threshold value T_houghThe value of the element(s), i.e., the mark removal point(s), is 0, a second planar matrix is obtained

Each of whichNon-zero element l_i,j(r_i,j,θ_i,j) Corresponding to a straight line; for image I₀Traversing and cleaning are carried out, only pixel points corresponding to the linear elements in the second plane matrix are left, and the linear elements l corresponding to the pixel points can be identified by a symbiotic matrix_i,j(r_i,j,θ_i,j) Obtaining a cleaned image I₁(ii) a By applying to the image I₁Go through the traversal as the matrix

Each non-zero value of the linear elements in the list C_i,jTo record the image I corresponding to the straight line₁All the pixels in the chain table, and the nodes in the chain table are arranged from small to large according to the abscissa of the pixels; at the same time, an array L is established and its first column element L is used_i,jTo record linked list C_i,jThe index i and j of the corresponding linked list are recorded by a second column element such as a character string; and sorting the array L from large to small according to the element values in the first column, and recording the sorting result as an array PL.

Thirdly, searching straight lines and elements corresponding to the upper carriage door frame parallel to the first transverse line and the carriage door side edges connected with the upper carriage door frame, counting the first N rows of elements of the PL, removing the elements corresponding to the upper carriage door frame and the side edges, and correspondingly removing the corresponding linked lists in the linked lists in a centralized manner; and circularly performing the following processing on the updated array and the chain table set, wherein k is from 1 to the line number of the array PL:

aiming at the kth row element of the array PL, obtaining indexes i and j according to the value of the second row element, and traversing the linked list C corresponding to the elements_i,jAnd obtaining the distance dL between the minimum abscissa pixel point PL and the first transverse line and the distance dR between the maximum abscissa pixel point PR and the first transverse line in the nodes, if min (dL, dR)<ds, then the linked list C corresponding to the element_i,jAnd the candidate linked list of the carriage door upper frame is used.

And finally, calculating the opening degree of the compartment door for each candidate linked list:

if it corresponds to a straight line l_i,jTheta of_i,jWithin a set neighborhood of pi/2Satisfies | theta_i,j-π/2|＜θ_sRespectively determining that the opening degrees of the left carriage door and the right carriage door are pi/2 according to the fact that the abscissa of the point PL is closer to the abscissa of one of the upper frame end points PHL and PHR of the carriage door;

otherwise, if theta_i,jIf the opening degree of the compartment door is more than pi/2, the calculation formula of the opening degree of the compartment door is defined as,

otherwise if theta_i,jIf the opening degree of the compartment door is less than pi/2, the calculation formula of the opening degree of the compartment door is defined as,

otherwise, defining the opening degree of the compartment door to be 0;

finally, when the opening degree of the carriage door is not 0, the angle theta is determined according to the first transverse line_UTo correct the degree of opening of the valve,

the door opening angle α shown in fig. 15 is calculated to be 41.966 degrees. Under a world coordinate system, measurement and technology are carried out through a measuring tape, and the real opening degree of the compartment door is 42.27 degrees. The error of 0.304 degree exists between the calculation result and the actual value, the visible detection error is very small, the engineering application requirement is met, and the accurate detection of the opening of the freight car is realized.

Preferably, only the first horizontal line, that is, only the pixel points in yd pixels in the vertical direction below the upper frame of the car door, may be subjected to region extraction to obtain an image I₀. Preferably, when the chain table is used for recording the pixel points corresponding to the straight line, the chain table C is also matched_i,jPerforming split representation, traversing C from front to back_i,jIf the distance between one node and the adjacent node is larger than the distance threshold d2, the node is recorded by using a new linked list.

Preferably, wherein T is_houghCan be taken as relativeA smaller value such as 10 to 15, or about 1/20 for a binary image width; the threshold d2 may be larger, for example, 15-25, or 1/15 of the binary image width.

Because the upper part of each door is provided with two edges after the compartment door is opened, preferably, the value of N is 5 to 8. Optionally, in the third step, the side edges and the upper edges in the vertical direction of the car door are removed, and the linked list and the group PL are cleaned accordingly. Specifically, among the several straight lines of the greatest length: finding out a straight line with a direction angle theta of 0 or pi and a distance between a head pixel element and a tail pixel element of a corresponding linked list and a terminal point PHL and a PHR of the carriage door upper frame, which are respectively smaller than a threshold value ds, and judging the straight line as an upper edge; then finding out a straight line with the direction angle theta near the preset angle value and one of the elements of the head and tail pixels of the corresponding linked list and the end points PHL and PHR of the carriage door upper frame, wherein one of the distances between the element and the end points PHL and PHR of the carriage door upper frame is less than a threshold value ds, and judging the straight line as a side edge. Preferably, the preset angle value is preset offline according to the length of the upper frame of the compartment door and the height of the vehicle type. Or preferably, the side edges are determined by searching two approximately symmetrical left and right straight lines having the largest length after the edge extraction of the image, starting from both ends of the second horizontal line in the second difference image.

In the prior art, a certain pixel point (u · ctg θ + r/sin θ) is generally determined by hough transform equation r ═ u · cos θ + vssin θ or linear equation v ═ u · ctg θ + r/sin θ after conversion₀,v₀) Whether or not on the straight line. And repeating the trigonometric operation of the number of the pixels to be detected for a plurality of times for each straight line in the second plane matrix after the preliminary screening. For this reason, to improve the calculation efficiency, it is preferable that the pixel points (u, v) and the line elements l_i,j(r_i,j,θ_i,j) The corresponding judgment method is as follows:

firstly, a branch point H (r) is taken on the straight line_i,j·cosθ_i,j,r_i,j·sinθ_i,j) And is recorded as H (u ', v'), and then J-r is calculated for each pixel point to be judged²- (u.u '+ v.v'), if | J | ≦ then determining that the pixel point corresponds to the straight line, otherwise not, wherein the pixel point is a preset small positive number.

Preferably, the point M (x) is pointed to_MyM) to a straight line l_i,j(r_i,j,θ_i,j) Is calculated using the following equation:

d＝|y_M·sinθ_i,j+x_M·cosθ_i,j-r_i,j|。

as shown in fig. 6, at the outlet of the logistics park or the goods yard, when it is detected that the freight car to be discharged is not tightly closed, the monitoring controller sends a gate falling instruction to the gateway controller 4000 to prevent the freight car from being discharged; meanwhile, besides the signal of the vehicle-mounted alarm, the acousto-optic unit 300 is used for reminding workers of timely intervention and intervention, the truck is required to be rectified and changed, the compartment door is tightly closed, and accidents are prevented.

Specifically, as shown in fig. 6, after receiving a brake drop command, the barrier controller 4000 controls the lifting and releasing of the brake lever 23 mounted on the pillar 21 through the lifting and releasing shaft 22. The monitoring controller interacts the truck transportation information with a database in the server 2000 through a communication interface, after the verification of the van door and other states and information is passed, the gate rod is controlled to be erected through the aisle gate controller, a truck can leave, and the rod is dropped after the truck is detected to leave; when the state verification fails, such as the abnormal state of the car door, the information prompt and alarm are carried out through the acousto-optic unit and the display module in the user interface unit, and the brake lever is in a brake closing state.

The vehicle access information is recorded correctly while the vehicle is allowed to access. Preferably, the monitoring controller stores the door image, the truck main body image collected by the main camera, the truck number, the opening degree of the door, the access time and other information in the database. Preferably, the server 2000 is provided with a logistics transportation information database of the ERP system.

Preferably, the freight car door opening monitoring device further comprises an illumination sensing unit and a lighting unit, and the monitoring controller controls the lighting unit to supplement light based on the detection of the illumination sensing unit on the environment.

When the abnormal state of the freight car door is detected, the freight car needs to be alarmed in time, in order to send out an alarm signal to a target freight car, on one hand, the alarm signal can be identified by identifying a license plate, on the other hand, the freight car can be identified by a radio frequency card and the like, and the method is more effective particularly when the ID of the radio frequency card corresponds to the processing flow in other information systems such as ERP and the like. Moreover, the radio frequency card can be portable and mobile, and can perform read-write operation in other units or departments. For this purpose, a scanning identification unit is provided in a door opening monitoring device of a van body to perform ID identification of a portable card for an identification card such as a radio frequency card.

Preferably, the card can be written, for example, the information of the passing time, the place and the like of the card is recorded, and a basis is provided for activity recording and tracing on a logistics chain of the van.

Example 2:

in order to obtain the main body contour features of the transportation truck for processing such as vehicle type recognition, entering and exiting image archiving, and the like, as shown in fig. 1, fig. 2, fig. 5, and fig. 6, unlike embodiment 1, the present embodiment uses the main camera 721 to collect the vehicle body image, and extracts the vehicle type of the vehicle after being processed by the image recognition module in the monitoring controller. As shown in fig. 5 and 6, the overhead view camera base is provided with the angle-adjustable pan/tilt head, and according to the identified vehicle type, the main processing module of the monitoring controller can optimize image acquisition of the overhead view camera according to a preset angle based on the size of the vehicle type, so that the truck tail image can utilize the size of the image as much as possible.

The automatic acquisition of the vehicle type information needs to establish an identification model, and the invention adopts a deep learning network to identify the vehicle type based on the vehicle image shot at a specific angle. The traditional vehicle detection method is based on detection of preset target vehicle characteristics, in recent years, a target detection algorithm is greatly broken through, and with the development of artificial intelligence and deep learning technology, a method for manually extracting a characteristic making classifier is gradually replaced by using a convolutional neural network to perform an image understanding task. Due to the characteristic of fast speed of algorithms such as Yolo (abbreviation of You only look once) and SSD, the algorithms have more potential to be used on an embedded processing platform. Many specific deep learning algorithm implementations can only meet the detection accuracy and detection speed on the GPU at the same time. However, since the GPU is expensive and has high heat generation, it is difficult to load the GPU on a portable platform, and the embedded platform with low cost cannot achieve the effect of real-time detection due to the lack of a large-capacity GPU. Due to the end-to-end design of the Yolo model, the implementation process is simple, the characteristics of the picture are extracted only once, the speed is high, and the Yolo model becomes one of the classical target detection models.

Based on the force calculation characteristic of the raspberry group, the yolo-v3-tiny network is selected as a prototype model for deep learning of vehicle type recognition. The yolo-v3-tiny network is a lightweight model of the latest optimized version of the yolo network, and has the characteristics of strong generalization capability, relatively low computational complexity and high recognition processing efficiency.

In the yolo-v3-tiny network, each grid unit feature map predicts 3 candidate frames, and each candidate frame needs four coordinates and five basic parameters including a confidence coefficient, so the number of convolution kernels of an output layer is B x (M +5), where B is the number of candidate frames and M is the number of categories. Compared with yolo-v3, the predicted output branch number of the yolo-v3-tiny network is reduced from 3 to 2, namely, the feature diagram adopts two types of 13 x 13 and 26 x 26, and the calculation amount is reduced.

After deep testing and analysis, the situation that in the management of a goods yard in a logistics park, images of vehicles can be acquired through shooting in a specific area and a specific angle is found, and therefore structural optimization can be conducted on a deep learning network in a targeted mode. Specifically, referring to fig. 5, the main camera 721 located in front of the vehicle side is triggered when both the first detection module 601 and the second detection module 602 of the vehicle detect the vehicle. Marking pictures collected by a main camera to form a training data set, and performing off-line training on the network model by using the data set to obtain a vehicle type recognition model; when the vehicle type recognition model runs on line, the image recognition module processes the picture to be detected by the vehicle type recognition model to obtain the vehicle type information of the vehicle. And adjusting the pitch angle of the aerial view camera pan-tilt according to the identified vehicle type, and determining a perspective inverse transformation matrix according to the preset relation between the pitch angle and the perspective inverse transformation.

Referring to fig. 2, preferably, an image preprocessing module 130 is provided in the main control unit, and is configured to fuse vehicle images acquired by the main camera and the side cameras into a sample image in a column arrangement manner, and label the sample image to form a training data set; when the system runs on line, the image to be detected is generated through the fusion processing and then is input to the image identification module.

Preferably, the overhead camera adopts a camera capable of digital focusing control, and the focal length of image acquisition is adjusted according to the prior size of the identified vehicle type, so that the truck tail door area image occupies more than 2/3 of the image space. Preferably, the shooting angles of the main camera and the side camera can be adjusted, so that the car body occupies 40% -70% of the picture space when the target van is shot.

Preferably, the side view camera captures an image such that the target vehicle is in the lower half of the field of view, then intercepts the lower half of the picture, similarly sets the main view camera, and then places the intercepted main view camera picture and side view camera picture above and below the sample picture respectively to form a fused picture.

The yolo-v3-tiny network removed the output layer of the 52 x 52 profile compared to yolo-v3, also reducing detection of small size targets accordingly. In the invention, the target vehicle has remarkable geometric size characteristics through the training sample acquired by the image acquisition mode.

Referring to fig. 8, in order to fully utilize the advantages of the deep learning network, the invention combines the image features to modify the normalized size of the sample picture from 416 × 416 to 832 × 832; and the structure of the yolo-v3-tiny network is modified accordingly: a convolution layer with a convolution kernel of 3 multiplied by 3 and a pooling layer of 2 multiplied by 2 are sequentially inserted before the 0 th layer of the yolo-v3-tiny network, and the number of the filters of the two layers is N, wherein N is an even number between 4 and 16.

Preferably, N is 8. Since repeated tests show that the recognition effect is better when N is 8, the method can better link the extraction of 16 features in the subsequent layers.

Since the yolo output layer convolution kernel has the number of B × (M +5) and M is 9 in the test, the number of convolution kernels is 42, and the yolo output layer outputs 13 × 13 × 42 candidate frames and 26 × 26 × 42 candidate frames and their identified classes, respectively, as shown in fig. 6.

Through image acquisition in the process that various vehicles pass through an access and a transportation channel, images acquired based on a main camera and a side camera are fused into a sample picture, and the vehicles in the picture are subjected to frame selection and type marking, so that samples under various natural conditions such as different illumination and meteorological conditions are acquired, the samples are abundant, and then offline training is performed on the modified network. In order to simplify image acquisition, video shooting can be carried out on road traffic, then conversion from a video image to one frame of picture is realized through a video processing system, and the converted picture is screened and then fused and labeled to form a training sample.

And taking the collected images of the vehicles on the transportation channel as a data set, and dividing the images into a training set and a testing set, wherein various van vehicles are labeled according to types, and non-van vehicles in the sample are labeled as others. Compared with the detection results of the common yolo-v3-tiny network, the improved network has improved recognition performance on various vehicle types, and the accuracy of six common dry-chamber trucks of 3T, 5T, 8T, 10T, 25T and 30T and two refrigeration-chamber trucks of 3T and 10T is improved by 1-6 percentage points respectively. Meanwhile, the light intensity is found to have great influence on the recognition effect of the truck type in the experiment, and the recognition effect of the small truck at night is relatively good.

Because the deep learning network has strong generalization and knowledge self-learning, the invention extracts the position information of the compartment door area based on the single same yolo-v3-tiny network, and concretely, the compartment door image sample marked with the compartment door frame is added into the training data set and the filter number of the convolution layer before the yolo layer is correspondingly modified. During online operation, the acquired van door image is identified by the network model trained offline, the anchor frame of the van door area in the image is detected, and the area in the anchor frame is used as the ROI of the van tail image. By sharing the same network among a plurality of identification tasks, the model complexity is reduced, and the identification capability and the management efficiency are improved.

Example 3

The difference between the first embodiment and the second embodiment is that the present embodiment obtains the door edge straight line by a method of connected domain search. Specifically, the detection and screening of the straight line of the door edge are carried out according to the following processes:

first, for the fourth door image I₀Selecting a pixel point from the leftmost upper end as a seed point, and establishing a linked list C comprising the pixel point_kSearching the next point within the range of the threshold value ds in the field from left to right and from top to bottom, and recording the position (u, v) of the next point in the linked list until the searched point is empty; repeating the above processing until each foreground point belongs to a linked list; screening the linked list set, and removing the linked list with the length less than a set value;

second, for each chain table C after screening_kGenerating a binary image with the node positions (u, v) corresponding to the foreground points, and after carrying out Hough transform on the image, only keeping the element theta-r with the maximum value, and marking as l_k(r_k,θ_k)；

Third, for each straight line l_kAccording to its corresponding linked list C_kAnd obtaining the distance dL between the minimum abscissa pixel point CL and the first transverse line and the distance dR between the maximum abscissa pixel point CR and the first transverse line in the nodes, if min (dL, dR)<ds, then the linked list C corresponding to the element_kAs the carriage door upper frame candidate linked list;

and finally, calculating the opening degree of the compartment door for each candidate linked list:

if it corresponds to a straight line l_kTheta of_kSatisfies | theta in a predetermined neighborhood of pi/2_k-π/2|＜θ_sRespectively determining the opening degrees of the left compartment door and the right compartment door to be pi/2 according to the closer of the abscissa of the point CL and the abscissa of which point of the end points PHL and PHR of the upper frame of the compartment door,

otherwise, determining linked list C_kThe uppermost end point CH and the lowermost end point CL in the above-mentioned group define the calculation formula of the door opening degree,

if the conditions are not met, defining the opening degree of the compartment door to be 0;

finally, when the opening degree of the carriage door is not 0, the angle theta is determined according to the first transverse line_UTo correct the degree of opening of the valve,

preferably, the perspective inverse transformation is performed to obtain a first perspective inverse transformation, the first perspective inverse transformation is performed to obtain an image in a world coordinate system, the image is rotated by a preset angle, such as 20 to 50 degrees, through a second perspective transformation, the characteristic line of the car door is obtained through linear screening, the characteristic line of the car door is then converted into the world coordinate system through the second perspective inverse transformation, and finally the opening degree of the car door is obtained.

Through the processing of the second perspective transformation and the second perspective inverse transformation, the angle theta of the straight line at the edge of the truck can be avoided from 0 and pi/2 as much as possible, so that the interval 0 is avoided in the triangular calculation, and the calculation precision is improved.

Preferably, after the second difference image is subjected to edge extraction, the obtained edge image is subjected to hough transform to obtain a first alignment line and a second alignment line corresponding to a vertical edge straight line at the top of the carriage, the mean value of the direction angles theta of the two alignment lines is calculated to obtain the driving direction angle of the vehicle, and an alarm is given when the driving direction angle exceeds a preset value range in the neighborhood of pi/2, so that collision damage to the outside caused by yaw driving possibly caused when the vehicle is poor in visual condition or the attention of a driver is not concentrated is prevented.

Preferably, the shutter speed of the overhead camera is optimized and adjusted by the speed calculation, so that the quality of the acquired truck tail image is improved.

While the embodiments of the present invention have been described above, these embodiments are presented as examples and do not limit the scope of the invention. These embodiments may be implemented in other various ways, and various omissions, substitutions, combinations, and changes may be made without departing from the spirit of the invention. These embodiments and modifications are included in the scope and gist of the invention, and are also included in the invention described in the claims and the equivalent scope thereof.

Claims (9)

1. A raspberry pi-based van body door opening monitoring method comprises the following steps:

s1, calculating the running speed V of the truck based on the trigger interval of the first detection module and the second detection module in the distance sensing unit;

s2, acquiring a truck number by scanning an ID card in the truck, and continuously acquiring a first tail image and a second tail image of the truck based on the triggering of a third detection module;

s3, carrying out differential operation on the first tail image and the second tail image to obtain a first differential image, and carrying out differential operation on the second tail image by taking the road surface image as a background to obtain a second differential image;

s4, based on the first and second difference images, marking the longitudinal position of the transverse continuous non-0 element length exceeding the set value, marking the longitudinal position of the longitudinal continuous non-0 element length exceeding the set value in a row mode, and intercepting the second tail image by taking the minimum common external rectangle of the row and column marks as a reference to obtain a freight car door image;

s5, converting the freight car door image into a first car door image under a world coordinate system based on a perspective inverse transformation matrix, carrying out graying processing on the first car door image to obtain a second car door image, carrying out binarization processing on the second car door image to obtain a third car door image, and carrying out edge extraction on the third car door image to obtain a fourth car door image;

s6, detecting a straight line in the fourth carriage door image based on Hough transform, screening by a heuristic rule of the azimuth relation between the straight line and a first row transverse line in the row mark to obtain an upper carriage door frame and an upper carriage door edge line, and calculating the opening degree of the carriage door according to the obtained result;

and S7, evaluating the running state of the truck according to the current speed and the opening degree of the carriage door of the truck and managing and controlling the truck and/or the gateway controller according to the evaluation result.

2. The raspberry pi based van body door opening monitoring method according to claim 1,

the step S4 includes the following processing: searching a first transverse line with the length exceeding a set value and the row number being minimum in the first differential image, searching a second transverse line with the length exceeding the set value and the row number being maximum in the second differential image, and searching a first column line and a second column line with the length exceeding the set value and the column number being minimum and maximum respectively in the second differential image; taking the minimum common external rectangle of the first and second transverse lines and the first and second row lines as a reference, and taking the minimum common external rectangle as an ROI area of the truck tail image after expanding a certain range outwards;

the step S7 includes the following processing: and when the speed V is greater than a set value and the opening alpha of the car door is greater than the set value, an alarm signal is sent to a vehicle-mounted alarm and an acousto-optic unit through an output module, and/or a brake falling instruction is sent to a gateway controller.

3. The raspberry pi based van body door opening monitoring method according to claim 1,

the step S1 includes the following processing: if the time interval between the times of the vehicle detected by the first and second detection modules is recorded as Δ t1 and the time interval between the times of the vehicle leaving the first and second detection modules is recorded as Δ t2, the calculation formula of the speed V is:

wherein L is the distance between the first and second detection modules.

4. The raspberry pi based van compartment door opening monitoring method according to claim 1, wherein the inverse perspective transformation matrix in step S5 is obtained by:

let the perspective inverse transform matrix be:

it converts the pixel point coordinates (u, v,1) in the image plane coordinate system to point coordinates (x, y,1) in the world coordinate system:

(x,y,1)＝(u,v,1)*T，

acquiring two coordinates of four pixel points by detecting imaging points of four known reference objects which are distributed in a rectangular shape on a road in an image, substituting the coordinates into the conversion formula to obtain eight linear equations, and obtaining eight linear equations from the linear equations and the equation a₁₃+a₂₃+a₃₃Solving to obtain the perspective inverse transformation matrix under the constraint of an equation set formed by 1;

and, the cargo door image is transformed into a first door image in a world coordinate system by:

5. the raspberry pi based van body door opening monitoring method according to claim 1,

the step S5 includes the following processing:

graying by means of weighted averaging of R, G, B components, acquiring a third door image by an Otsu threshold value method in binarization processing, performing multiple corrosion, expansion and corrosion after threshold value segmentation, and performing edge detection by using a Canny operator;

the step S6 includes the following processing:

marking the intersection points of the first transverse line in the second differential image and the first and second alignment lines respectively by taking the first transverse line in the line mark as a first transverse line and the head and tail alignment lines in the line mark as a first alignment line and a second alignment line respectively, and correspondingly marking the left and right intersection points as PHL and PHR respectively in the first door image,

then, the detection and screening of the straight line are carried out:

first, for the fourth door image I₀After Hough transform is carried out on each foreground pixel point, a theta-r first plane matrix corresponding to the image is obtained through accumulation, wherein r represents the distance of a vertical line of a straight line from an original point, theta is the angle of the vertical line to a transverse axis,

then, cleaning the first plane matrix, and removing the value smaller than a set threshold value T_houghObtaining a second planar matrix

Each of which is not 0 element l_i,j(r_i,j,θ_i,j) Corresponding to a straight line; for image I₀Traversing and cleaning are carried out, only pixel points corresponding to the linear elements in the second planar matrix are left, and the linear elements l corresponding to the pixel points are identified_i,j(r_i,j,θ_i,j) Obtaining a cleaned image I₁(ii) a By applying to the image I₁Go through the traversal as the matrix

Each non-0 value of the linear elements in the list C_i,jTo record the image I corresponding to the straight line₁All the pixels in the chain table are arranged from small to large according to the abscissa of the pixel, and meanwhile, an array L is established and a first column element L of the array L is used_i,jTo record linked list C_i,jThe second column element is used for recording indexes i and j of the corresponding linked list; sorting the array L from big to small according to the first row element values, recording the sorting result as an array PL,

thirdly, straight lines and elements corresponding to the carriage door upper frame parallel to the first transverse line and the carriage door side edges connected with the carriage door upper frame are searched, the first N rows of elements of the PL are counted, the elements corresponding to the carriage door upper frame and the side edges are removed, and the corresponding linked lists are correspondingly removed in a linked list set,

and circularly performing the following processing on the updated array and the chain table, wherein k is from 1 to the line number of the array PL:

to is directed atObtaining indexes i and j of the kth row element of the array PL according to the value of the second row element of the array PL, and traversing the linked list C corresponding to the elements_i,jAnd obtaining the distance dL between the minimum abscissa pixel point PL and the first transverse line and the distance dR between the maximum abscissa pixel point PR and the first transverse line in the nodes, if min (dL, dR)<ds, then the linked list C corresponding to the element_i,jAs the carriage door upper frame candidate linked list;

and finally, calculating the opening degree of the carriage door for each candidate linked list:

if it corresponds to a straight line l_i,jTheta of_i,jSatisfies | theta in a predetermined neighborhood of pi/2_i,j-π/2|＜θ_sRespectively determining that the opening degrees of the left compartment door and the right compartment door are pi/2 according to the closer of the abscissa of the point PL and the abscissa of which point of the upper frame end point PHL and PHR of the compartment door,

otherwise, if theta_i,jIf the opening degree of the compartment door is more than pi/2, the calculation formula of the opening degree of the compartment door is defined as,

otherwise if theta_i,jIf the opening degree of the compartment door is less than pi/2, the calculation formula of the opening degree of the compartment door is defined as,

otherwise, defining the opening degree of the compartment door to be 0;

finally, when the opening degree of the carriage door is not 0, the angle theta is determined according to the first transverse line_UTo correct the degree of opening of the valve,

6. the raspberry pi based van body door opening monitoring method according to claim 5, wherein the step S6 includes the following steps:

pixel point (u, v) and straight lineElement l_i,j(r_i,j,θ_i,j) The corresponding judgment mode is as follows:

taking a branch point H (r) on the straight line_i,j·cosθ_i,j,r_i,j·sinθ_i,j) Denoted as H (u ', v'),

calculating J-r²- (u-u '+ v-v'), if | J | ≦ then determining that the pixel point corresponds to the straight line, otherwise not, wherein the pixel point is a preset small positive number;

to point M (x)_M,y_M) To a straight line l_i,j(r_i,j,θ_i,j) Is calculated using the following equation:

d＝|y_M·sinθ_i,j+x_M·cosθ_i,j-r_i,j|。

7. the raspberry pi based van body door opening monitoring method according to claim 1, wherein the step S6 includes the following steps:

before the linear screening, the image is rotated by a preset angle such as 20-50 degrees by using a second perspective transformation, the characteristic line of the carriage door is obtained through the linear screening, then the characteristic line is converted into a world coordinate system by using a second perspective inverse transformation, and then the opening degree of the carriage door is obtained.

8. The raspberry pi based van body door opening monitoring method according to claim 1,

the step S6 includes the following processing:

first, for the fourth door image I₀Selecting a pixel point from the leftmost upper end as a seed point, and establishing a linked list C comprising the pixel point_kSearching the next point within the range of the threshold value ds in the field from left to right and from top to bottom, and recording the position (u, v) of the next point in the linked list until the searched point is empty; repeating the above processing until each foreground point belongs to a linked list; screening the linked list set, and removing the linked list with the length less than a set value;

second, for each chain table C after screening_kGenerating a binary image with the node positions (u, v) corresponding to the foreground points, and after carrying out Hough transform on the image, only keeping the element theta-r with the maximum value, and marking as l_k(r_k,θ_k)；

Third, for each straight line l_kAccording to its corresponding linked list C_kAnd obtaining the distance dL between the minimum abscissa pixel point CL and the first transverse line and the distance dR between the maximum abscissa pixel point CR and the first transverse line in the nodes, if min (dL, dR)<ds, then the linked list C corresponding to the element_kAs the carriage door upper frame candidate linked list;

and finally, calculating the opening degree of the compartment door for each candidate linked list.

9. A raspberry pi-based van body door opening monitoring method comprises the following steps:

p1, establishing an image recognition module: the image recognition module adopts a network model improved based on yolo-v3-tiny, takes 832 multiplied by 832 images as input, and inserts a convolution layer with a convolution kernel of 3 multiplied by 3 and a pooling layer of 2 multiplied by 2 in sequence before the 0 th layer of the yolo-v3-tiny network, and the number of the filters of the two layers is 8;

p2, obtaining training samples: acquiring images of the vehicle at a preset position of the channel, and forming a training data set by taking a truck tail image marked with a compartment door area as an image sample;

p3, training the network model off line: performing parameter configuration on network training, and performing offline training on the network model by using the acquired data set to obtain an image recognition model;

p4, on-line monitoring of the opening of a freight car:

SP1, calculating the running speed V of the truck based on the trigger intervals of the first detection module and the second detection module; acquiring a truck number by scanning an ID card in the truck;

SP2, acquiring a current tail image of the truck based on the triggering of a third detection module, obtaining a door area anchor frame after the image identification model processing, and intercepting the current tail image by taking the anchor frame as a reference to obtain a truck door image;

SP3, extracting the edges of the truck carriage door images, extracting a straight line set through Hough transform, searching two longest cargo carriage top ridge lines with an angle in the pi/2 neighborhood from the straight line set, and labeling the intersection points of the ridge lines and the upper edge of the anchor frame in the truck carriage door images;

SP4, converting the freight car door image into a first car door image under a world coordinate system based on a perspective inverse transformation matrix, carrying out graying processing on the first car door image to obtain a second car door image, carrying out binarization processing on the second car door image to obtain a third car door image, and carrying out edge extraction on the third car door image to obtain a fourth car door image;

SP5, detecting a straight line in the fourth carriage door image based on Hough transform, screening by a heuristic rule of the orientation relation of the straight line and the marked intersection point to obtain an upper carriage door frame and an upper carriage door edge line, and calculating the opening degree of the carriage door according to the obtained result;

and SP6, evaluating the running state of the truck according to the current speed and the opening degree of the carriage door of the truck and managing and controlling the truck and/or the gateway controller according to the evaluation result.

Images