CN117953470B - Expressway event identification method and device of panoramic stitching camera - Google Patents
Expressway event identification method and device of panoramic stitching camera Download PDFInfo
- Publication number
- CN117953470B CN117953470B CN202410346321.0A CN202410346321A CN117953470B CN 117953470 B CN117953470 B CN 117953470B CN 202410346321 A CN202410346321 A CN 202410346321A CN 117953470 B CN117953470 B CN 117953470B
- Authority
- CN
- China
- Prior art keywords
- expressway
- event
- training
- time
- monitoring
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 20
- 238000012544 monitoring process Methods 0.000 claims abstract description 78
- 239000013598 vector Substances 0.000 claims abstract description 76
- 238000012549 training Methods 0.000 claims abstract description 66
- 230000004927 fusion Effects 0.000 claims abstract description 24
- 238000004458 analytical method Methods 0.000 claims abstract description 9
- 238000001514 detection method Methods 0.000 claims description 21
- 230000002159 abnormal effect Effects 0.000 claims description 12
- 238000000605 extraction Methods 0.000 claims description 12
- 238000012545 processing Methods 0.000 claims description 12
- 230000006870 function Effects 0.000 claims description 8
- 230000000007 visual effect Effects 0.000 claims description 7
- 238000005266 casting Methods 0.000 claims description 6
- 239000000463 material Substances 0.000 claims description 6
- 230000005540 biological transmission Effects 0.000 claims description 5
- 238000010276 construction Methods 0.000 claims description 5
- 239000011159 matrix material Substances 0.000 claims description 3
- 230000002708 enhancing effect Effects 0.000 claims description 2
- 238000012216 screening Methods 0.000 claims description 2
- 206010039203 Road traffic accident Diseases 0.000 description 4
- 238000011835 investigation Methods 0.000 description 4
- 230000006399 behavior Effects 0.000 description 3
- 238000004590 computer program Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000003709 image segmentation Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 239000007787 solid Substances 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000002372 labelling Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Landscapes
- Traffic Control Systems (AREA)
- Image Processing (AREA)
- Closed-Circuit Television Systems (AREA)
Abstract
The invention provides a highway event identification method of a panoramic stitching camera, which comprises the following steps: s1, acquiring a highway monitoring video stream as a training sample, wherein the monitoring video stream is a video material of a panoramic 360-degree view formed by splicing a plurality of monitoring cameras; s2, constructing a fusion module comprising a target feature vector and a position vector and a space-time probability prediction module as a highway event recognition model; s3, training is carried out in the expressway event recognition model, each training sample is firstly input into a fusion module of a target feature vector and a position vector to realize fusion, then a space-time probability prediction module is adopted to predict and recognize the track of each type of target, semantic analysis of the expressway event is realized, and a trained expressway event recognition model is obtained; s4, acquiring a monitoring video stream to be detected, inputting the monitoring video stream to a training-completed expressway event recognition model for identifying expressway events, and further obtaining an expressway event recognition result.
Description
Technical Field
The invention belongs to the technical field of computer vision, and relates to a highway event identification method and device for a panoramic stitching camera.
Background
The existing expressway event detection system mostly depends on a traditional monitoring camera, and can be applied to aspects such as traffic flow monitoring, traffic illegal behavior detection, accident monitoring, road jam monitoring, tunnel safety monitoring and the like. However, the following problems exist: (1) limited field of view: conventional monitoring cameras generally only monitor a specific area in a fixed manner, and have a limited field of view. The omnibearing and panoramic monitoring view angle cannot be provided; (2) low resolution: the resolution of the traditional monitoring camera is relatively low, and the details of the image are not clear enough. This can affect the ability to identify and analyze small objects or distant objects; (3) monitoring blind areas: because the view angle of the camera is limited, a monitoring blind area exists. This means that events in the blind zone cannot be captured, which presents a certain risk to the safety monitoring.
Disclosure of Invention
In order to overcome the problems of limited visual field, low resolution, blind area and the like of the traditional monitoring camera, the invention provides a highway event identification method and device of a panoramic spliced camera, which can provide more comprehensive information and help traffic management departments to discover and process various events in time; the efficiency and the accuracy of event detection are greatly improved, and the pressure of manual monitoring is reduced.
The technical scheme adopted by the invention is as follows:
the highway event identification method of the panoramic spliced camera comprises the following specific steps:
step S1, obtaining at least 10 sections of monitoring video streams of each type of expressways including congestion and slow running, abnormal parking, abnormal running, pedestrian non-motor vehicles and road surface casting objects as training samples, wherein the monitoring video streams are video materials of panoramic 360-degree vision formed by splicing a plurality of monitoring cameras;
Step S2, constructing a fusion module comprising a target feature vector and a position vector and a space-time probability prediction module as a highway event recognition model;
Step S3, inputting the training samples in the step S1 into the expressway event recognition model in the step S2 for training, wherein each training sample is firstly input into a fusion module of a target feature vector and a position vector to realize fusion, and then a space-time probability prediction module is adopted to predict and recognize the track of various targets so as to realize semantic analysis of expressway events and obtain a trained expressway event recognition model;
and S4, acquiring a monitoring video stream to be detected, inputting the monitoring video stream into the expressway event recognition model trained in the step S3 for identifying expressway events, and further obtaining the recognition result of the expressway events.
Further, the training samples in step S1 need to be pre-trained, specifically including:
extracting vehicle targets corresponding to trucks, cars and buses in each video image frame through a feature extraction network;
the vehicle object is marked and the image of the marked vehicle object is pre-trained to identify the vehicle object from the video image frames.
Further, the feature extraction network in step S1 adopts YoloV to input training sample V T with a time length T into the feature extraction network to output the extracted features, and the width and height of the input video image frame are set to be w×h.
Further, the construction of the expressway event recognition model in step S2 is specifically as follows:
The feature images output by Conv2, conv3, conv4 and Conv5 in YoloV network are C2, C3, C4 and C5 respectively, the output feature images are generated by the subsequent stage of YoloV network to form a multi-scale feature vector theta= [ theta 1,θ2,θ3,θ4 ] of each camera, the image frames of 4 monitoring cameras are spliced, and theta i represents the feature vector of the No. 1-4 monitoring cameras;
screening and enhancing features of vehicles, non-motor vehicles and pedestrians in overlapping areas of adjacent monitoring cameras by adopting a cross-attention algorithm:
；
Wherein, In order to enhance the post feature vector, f is a cross-attention mechanism algorithm, which is specifically implemented as follows:
；
Wherein, For/>Is characterized by transpose; d k denotes a vector matrix of vehicles, non-motor vehicles and pedestrians related to highway events, in order to prevent/>Is too large;
Meanwhile, position sensing is carried out among different camera views, and marking is carried out by adopting a position embedding vector omega= [ omega 1,ω2,ω3,ω4 ], wherein omega i represents a position vector of a No. 1-4 monitoring camera;
Constructing a space-time probability prediction model:
Discrete time t= (m,) n., the spatiotemporal trajectory of the object within (i.e., from time m to time n, and m < n) is defined as:
；
wherein j ε I j,Ij represents a feature set that includes vehicles, non-motor vehicles, and pedestrians; Representing the characteristic space-time vector after fusion operation of the characteristic vector and the position vector acquired by the monitoring camera; /(I) Representing space-time vectors of various targets from time m to time n, wherein m is less than n; /(I)A space-time vector representing the occurrence of various targets at the moment m;
After the characteristic space-time vectors are obtained, classification is carried out according to the moving track probabilities of the vehicle, the non-motor vehicle and the pedestrian targets in different expressway events, so that the identification of expressway event categories is realized, and the track probability of q-type events is calculated as follows:
；
Wherein, y i(Pj) is a probability function, and the training stage carries out the training of the probability function according to the known expressway event sample; the trajectory probability representing the q-class event occurring in the time span N, y q(Pj) is the pre-policy probability, P j is the feature space-time vector, i is the number representing the monitoring camera,/> The probability of detecting a q-class event within a time span N;
Meanwhile, the detection stage can match the events in the weighted file expressway event class quilt domain obtained by training, and finally outputs the identified expressway event type, namely the expressway event identification result, which is expressed as:
；
Wherein, Detecting an event with the category q in the time t as a detection result;
the occurrence probability of the event with the category q is detected in the time t' for predicting the probability;
Is the weight obtained through training of the sequence length T.
Further, the monitoring video stream to be detected in step S4 is a video material based on a panoramic 360-degree field of view formed by splicing a plurality of monitoring cameras.
The highway event recognition device of the panoramic stitching camera is used for executing the highway event recognition method of the panoramic stitching camera, and comprises a processor, wherein the processor is internally provided with:
the system comprises a training sample acquisition module, a processing module and a processing module, wherein the training sample acquisition module is used for acquiring at least 10 sections of monitoring video streams of each type of expressways including congestion slow running, abnormal parking, abnormal running, pedestrian non-motor vehicles and road surface casting objects as training samples, and the monitoring video streams are video materials based on panoramic 360-degree visual fields formed by splicing a plurality of monitoring cameras;
The model building module is used for building a fusion module comprising a target feature vector and a position vector and a space-time probability prediction module as a highway event recognition model;
The model training module is used for inputting training samples into the expressway event recognition model for training, wherein each training sample is firstly input into the fusion module of the target feature vector and the position vector for realizing fusion, and then a space-time probability prediction module is used for predicting and recognizing the tracks of various targets to realize semantic analysis of expressway events, so that the expressway event recognition model after training is obtained;
The recognition result output module is used for inputting the monitoring video stream to be detected into the training-completed expressway event recognition model to recognize the expressway event, so as to obtain the recognition result of the expressway event.
Further, a feature extraction module is further arranged in the processor and used for pre-training the training text and extracting features of images of the marked vehicle targets.
Further, a memory for storing data or instructions is connected to the processor.
Further, the processor is connected with a transmission device for transmitting data with an external device.
Further, the processor is connected with an input/output device for inputting or outputting information.
The invention has the beneficial effects that: the panoramic stitching camera can provide a high-resolution and panoramic view angle by fusing the images of a plurality of cameras, and effectively monitor and record various traffic events and conditions occurring on the expressway. Firstly, the panoramic stitching camera can realize the omnibearing monitoring of the whole road section. By stitching the images of multiple cameras, a panoramic view can be obtained, covering a wider area. This is very helpful for detecting highway events such as traffic accidents, traffic violations, road jams, construction conditions, etc. Compared with the traditional camera which can only fixedly monitor a certain specific area, the panoramic spliced camera can provide more comprehensive information, and help traffic management departments to timely find and process various events. Second, panoramic stitching cameras may also provide high definition images and video. The spliced images of the cameras can provide higher resolution and more detailed details, so that the identification and analysis of traffic events are more accurate and reliable. This is of great importance for investigation of accident causes, evidence of illicit activity, evidence provision and investigation of deployment. In addition, the panoramic stitching camera can also be combined with image processing and intelligent algorithms to realize automatic event detection and identification. By analyzing and identifying the spliced images, events such as traffic accidents, vehicle illegal behaviors, road jams and the like can be automatically detected, and an alarm can be timely sent to related departments. This greatly improves the efficiency and accuracy of event detection, reducing the pressure of manual monitoring. In conclusion, the panoramic stitching camera has important application value in expressway event detection. The system can provide an omnibearing monitoring visual angle, high-definition images and videos and an automatic event detection and identification function, provides powerful support for traffic management departments, and improves the efficiency and accuracy of event processing.
Drawings
FIG. 1 is a schematic diagram of the construction logic of the highway event recognition model of the present invention;
Fig. 2 is a schematic diagram of a method for constructing a highway event recognition model according to the present invention.
Detailed Description
The invention will be further illustrated with reference to the following specific examples, without limiting the invention to these specific embodiments. It will be appreciated by those skilled in the art that the invention encompasses all alternatives, modifications and equivalents as may be included within the scope of the claims.
Example 1
Referring to fig. 1 and 2, the invention provides a highway event identification method of a panoramic stitching camera, which comprises the following specific steps:
S1, acquiring at least 10 sections of monitoring video streams of each type of expressways including congestion and slow running, abnormal parking, abnormal running, pedestrian non-motor vehicles and road surface casting objects as training samples, wherein the monitoring video streams are video materials of panoramic 360-degree vision formed by splicing a plurality of monitoring cameras;
The training samples need to be pre-trained, and specifically comprise:
extracting vehicle targets corresponding to trucks, cars and buses in each video image frame through a feature extraction network;
the vehicle object is marked and the image of the marked vehicle object is pre-trained so that the vehicle object can be identified from the video image frames.
The feature extraction network in this embodiment adopts YoloV, inputs training sample V T with time length T into the feature extraction network to output the extracted features, and sets the width and height of the input video image frame to be w×h (412×412), and the output of each stage used by the YoloV backbone network is as shown in the following table one:
Output of each stage of a table-image segmentation network
S2, constructing a fusion module comprising a target feature vector and a position vector and a space-time probability prediction module as a highway event recognition model;
the highway event recognition model is constructed specifically as follows:
The feature images output by Conv2, conv3, conv4 and Conv5 in YoloV network are C2, C3, C4 and C5 respectively, the feature images are combined into an overall feature vector theta= [ theta 1,θ2,θ3,θ4 ] by generating multi-scale features of each camera in the subsequent stage of YoloV network, and in order to better acquire panoramic images, the invention adopts the steps of splicing image pictures of 4 monitoring cameras, wherein theta i represents the feature vectors of No. 1-4 monitoring cameras;
because panoramic stitching is employed, adjacent cameras have overlapping areas; in order to effectively extract the characteristics of the overlapping area of the adjacent monitoring pictures, the cross attention algorithm is adopted to screen and strengthen the characteristics of vehicles, non-motor vehicles and pedestrians in the overlapping area of the adjacent monitoring cameras:
；
Wherein, In order to enhance the post feature vector, f is a cross-attention mechanism algorithm, which is specifically implemented as follows:
；
Wherein, For/>Is characterized by transpose; d k denotes a vector matrix of vehicles, non-motor vehicles and pedestrians related to highway events, in order to prevent/>Is too large;
Meanwhile, the panoramic camera based on splicing needs to perform position sensing among different camera views, the position sensing is performed among different camera views, and the position sensing is marked by adopting a position embedding vector omega= [ omega 1,ω2,ω3,ω4 ], wherein omega i represents the position vector of the No. 1-4 monitoring camera;
expressway event identification based on panoramic stitching cameras needs to be combined with feature vectors and position vectors extracted by each camera, comprehensive judgment is carried out according to changes of space-time trajectories of vehicles, non-motor vehicles and pedestrians, and a space-time probability prediction model is constructed:
Discrete time t= (m,) n., the spatiotemporal trajectory of the object within (i.e., from time m to time n, and m < n) is defined as:
；
wherein j ε I j,Ij represents a feature set that includes vehicles, non-motor vehicles, and pedestrians; Representing the characteristic space-time vector after fusion operation of the characteristic vector and the position vector acquired by the monitoring camera; /(I) Representing space-time vectors of various targets from time m to time n, wherein m is less than n; /(I)A space-time vector representing the occurrence of various targets at the moment m;
After the characteristic space-time vectors are obtained, classification is carried out according to the moving track probabilities of the vehicle, the non-motor vehicle and the pedestrian targets in different expressway events, so that the identification of expressway event categories is realized, and the track probability of q-type events is calculated as follows:
；
Wherein, y i(Pj) is a probability function, and the training stage carries out the training of the probability function according to the known expressway event sample; the trajectory probability representing the q-class event occurring in the time span N, y q(Pj) is the pre-policy probability, P j is the feature space-time vector, i is the number representing the monitoring camera,/> The probability of detecting a q-class event within a time span N;
Meanwhile, the detection stage matches the events in the weighted file expressway event class quilt area obtained by training, and finally outputs the identified expressway event type, namely the expressway event identification result, which is expressed as:
；
Wherein, Detecting an event with the category q in the time t as a detection result;
the occurrence probability of the event with the category q is detected in the time t' for predicting the probability;
Is the weight obtained through training of the sequence length T. The weight is obtained through training, the invention comprises two parts of training and recognition application, the weight is obtained through training, and then recognition is carried out through the weight.
S3, inputting the training samples in the step S1 into the expressway event recognition model in the step S2 for training, wherein each training sample is firstly input into a fusion module of a target feature vector and a position vector to realize fusion, and then a space-time probability prediction module is adopted to predict and recognize the track of various targets so as to realize semantic analysis of expressway events and obtain a trained expressway event recognition model;
S4, acquiring a monitoring video stream to be detected, inputting the monitoring video stream into the expressway event recognition model trained in the step S3 for identifying expressway events, and further obtaining the recognition result of the expressway events. The monitoring video stream to be detected is a video material based on a panoramic 360-degree visual field formed by splicing a plurality of monitoring cameras.
According to the invention, characteristic vector and position vector information in pictures are collected according to the spliced panoramic camera; then, track prediction and detection are carried out on objects including vehicles, non-motor vehicles, pedestrians and the like based on the space-time probability model, so that recognition and detection on expressway events are realized.
The panoramic stitching camera can provide a high-resolution and panoramic view angle by fusing the images of a plurality of cameras, and effectively monitor and record various traffic events and conditions occurring on the expressway. Firstly, the panoramic stitching camera can realize the omnibearing monitoring of the whole road section. By stitching the images of multiple cameras, a panoramic view can be obtained, covering a wider area. This is very helpful for detecting highway events such as traffic accidents, traffic violations, road jams, construction conditions, etc. Compared with the traditional camera which can only fixedly monitor a certain specific area, the panoramic spliced camera can provide more comprehensive information, and help traffic management departments to timely find and process various events. Second, panoramic stitching cameras may also provide high definition images and video. The spliced images of the cameras can provide higher resolution and more detailed details, so that the identification and analysis of traffic events are more accurate and reliable. This is of great importance for investigation of accident causes, evidence of illicit activity, evidence provision and investigation of deployment. In addition, the panoramic stitching camera can also be combined with image processing and intelligent algorithms to realize automatic event detection and identification. By analyzing and identifying the spliced images, events such as traffic accidents, vehicle illegal behaviors, road jams and the like can be automatically detected, and an alarm can be timely sent to related departments. This greatly improves the efficiency and accuracy of event detection, reducing the pressure of manual monitoring. In conclusion, the panoramic stitching camera has important application value in expressway event detection. The system can provide an omnibearing monitoring visual angle, high-definition images and videos and an automatic event detection and identification function, provides powerful support for traffic management departments, and improves the efficiency and accuracy of event processing.
Example two
The embodiment provides a highway event recognition device of a panoramic stitching camera, which is used for executing the highway event recognition method of the panoramic stitching camera, and comprises a processor, wherein the processor is internally provided with:
the system comprises a training sample acquisition module, a processing module and a processing module, wherein the training sample acquisition module is used for acquiring at least 10 sections of monitoring video streams of each type of expressways including congestion slow running, abnormal parking, abnormal running, pedestrian non-motor vehicles and road surface casting objects as training samples, and the monitoring video streams are video materials based on panoramic 360-degree visual fields formed by splicing a plurality of monitoring cameras;
the feature extraction module is used for pre-training the training text and extracting features of an image of a marked vehicle target;
The model building module is used for building a fusion module comprising a target feature vector and a position vector and a space-time probability prediction module as a highway event recognition model;
The model training module is used for inputting training samples into the expressway event recognition model for training, wherein each training sample is firstly input into the fusion module of the target feature vector and the position vector for realizing fusion, and then a space-time probability prediction module is used for predicting and recognizing the tracks of various targets to realize semantic analysis of expressway events, so that the expressway event recognition model after training is obtained;
The recognition result output module is used for inputting the monitoring video stream to be detected into the training-completed expressway event recognition model to recognize the expressway event, so as to obtain the recognition result of the expressway event.
The processor of this embodiment may include a Central Processing Unit (CPU), or an Application SPECIFIC INTEGRATED Circuit (ASIC), or may be configured as one or more integrated circuits that implement embodiments of the present application.
The processor in this embodiment is connected with a memory for storing data or instructions. The memory may include mass storage for data or instructions. By way of example, and not limitation, the memory may comprise a hard disk drive (HARD DISK DRIVE, abbreviated HDD), a floppy disk drive, a Solid state drive (Solid STATE DRIVE, abbreviated SSD), flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (Universal Serial Bus, abbreviated USB) drive, or a combination of two or more of these. The memory may include removable or non-removable (or fixed) media, where appropriate. The memory may be internal or external to the data processing apparatus, where appropriate. In a particular embodiment, the memory is a Non-Volatile (Non-Volatile) memory. In particular embodiments, the Memory includes Read-Only Memory (ROM) and random access Memory (Random Access Memory RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (Programmable Read-Only Memory, abbreviated PROM), an erasable PROM (Erasable Programmable Read-Only Memory, abbreviated EPROM), an electrically erasable PROM (ELECTRICALLY ERASABLE PROGRAMMABLE READ-Only Memory, abbreviated EEPROM), an electrically rewritable ROM (ELECTRICALLY ALTERABLE READ-Only Memory, abbreviated EAROM), or a FLASH Memory (FLASH), or a combination of two or more of these. The RAM may be a Static Random-Access Memory (SRAM) or a dynamic Random-Access Memory (Dynamic Random Access Memory DRAM), where the DRAM may be a fast page mode dynamic Random-Access Memory (Fast Page Mode Dynamic Random Access Memory, FPMDRAM), an extended data output dynamic Random-Access Memory (Extended Date Out Dynamic Random Access Memory, EDODRAM), a synchronous dynamic Random-Access Memory (Synchronous Dynamic Random-Access Memory, SDRAM), or the like, as appropriate.
The memory may be used to store or cache various data files that need to be processed and/or communicated, as well as possible computer program instructions for execution by the processor. The processor reads and executes the computer program instructions stored in the memory.
The processor in this embodiment is connected with a transmission device for transmitting data with an external device. The transmitting device may be used to receive or transmit data via a network. Specific examples of the network described above may include a wired or wireless network provided by a communication provider of the electronic device. In one example, the transmission device includes a network adapter (Network Interface Controller, simply referred to as a NIC) that can connect to other network devices through the base station to communicate with the internet. In one example, the transmission device may be a Radio Frequency (RF) module, which is used to communicate with the internet wirelessly.
The processor in this embodiment is connected with an input/output device for inputting or outputting information. The input-output device is used for inputting or outputting information. In this embodiment, the input information may be a surveillance video stream, and the output information may be a current traffic state of a highway toll station, and the like.
Example III
The expressway event identification method of the panoramic stitching camera provided by the embodiment is different from that of the first embodiment in that:
The method for identifying the expressway event is applied to identifying the expressway traffic jam event, and specifically comprises the following steps:
Congestion detection is combined with scene panorama stitching, firstly, images acquired by each camera are stitched into a panoramic picture, the width and the height of each video are set to be 412 x 412, then feature graphs C2, C3, C4 and C5 are output through YoloV networks Conv2, conv3, conv4 and Conv5, and the feature graphs are combined into a global feature vector through the multi-scale features of each camera generated in the subsequent stage of the YoloV network The θ vector is in (412/32) × (412/32) format, as shown in Table two below:
Output of each stage of the table two congestion image segmentation network
Features of the vehicle are then filtered and enhanced, and then position-embedded vectors are employedMarking is carried out.
In this embodiment, the panoramic camera based on stitching needs to perform position sensing between different camera views, that is, position embedding vectors are adoptedLabeling, wherein/>Representing the position vectors of the monitoring cameras No. 1-4.
Then judging whether the current congestion situation occurs or not according to the running track of the vehicle in each monitoring picture, namely, for the congestion eventWherein the time span is n=180 seconds, that is, the length of the sliding window is judged, and y is the detection result of the congestion event type, that is, the congestion event is detected within 180 seconds from the time t. /(I)To predict the probability, i.e., the occurrence probability of an event with a congestion category is detected at time t' (when the probability is greater than 70%, the occurrence of the event is determined).
The foregoing examples illustrate only a few embodiments of the application, which are described in greater detail and are not to be construed as limiting the scope of the application. It should be noted that it will be apparent to those skilled in the art that several variations and modifications can be made without departing from the spirit of the application, which are all within the scope of the application. Accordingly, the scope of the application should be assessed as that of the appended claims.
Claims (9)
1. The highway event identification method of the panoramic spliced camera is characterized by comprising the following specific steps of:
step S1, obtaining at least 10 sections of monitoring video streams of each type of expressways including congestion and slow running, abnormal parking, abnormal running, pedestrian non-motor vehicles and road surface casting objects as training samples, wherein the monitoring video streams are video materials of panoramic 360-degree vision formed by splicing a plurality of monitoring cameras;
step S2, constructing a fusion module comprising a target feature vector and a position vector and a space-time probability prediction module as a highway event recognition model; the construction of the expressway event recognition model is specifically as follows:
The feature images output by Conv2, conv3, conv4 and Conv5 in YoloV network are C2, C3, C4 and C5 respectively, the output feature images are generated by the subsequent stage of YoloV network to form a multi-scale feature vector theta= [ theta 1,θ2,θ3,θ4 ] of each camera, the image frames of 4 monitoring cameras are spliced, and theta i represents the feature vector of the No. 1-4 monitoring cameras;
screening and enhancing features of vehicles, non-motor vehicles and pedestrians in overlapping areas of adjacent monitoring cameras by adopting a cross-attention algorithm:
θ′i＝f(θi-1,θi,θi+1)
Wherein θ' i is the enhanced feature vector, and f is the cross-attention mechanism algorithm, which is specifically implemented as follows:
Wherein, Transpose the features of θ i+1; d k denotes a vector matrix of vehicles, non-maneuvers and pedestrians related to highway events in order to prevent/>Is too large;
Meanwhile, position sensing is carried out among different camera views, and marking is carried out by adopting a position embedding vector omega= [ omega 1,ω2,ω3,ω4 ], wherein omega i represents a position vector of a No. 1-4 monitoring camera;
Constructing a space-time probability prediction model:
The spatiotemporal trajectory of the object within the discrete time t= (m...degree., n) is defined as:
wherein j ε I j,Ij represents a feature set that includes vehicles, non-motor vehicles, and pedestrians; Representing the characteristic space-time vector after fusion operation of the characteristic vector and the position vector acquired by the monitoring camera; /(I) Representing space-time vectors of various targets from time m to time n, wherein m is less than n; /(I)A space-time vector representing the occurrence of various targets at the moment m;
After the characteristic space-time vectors are obtained, classification is carried out according to the moving track probabilities of the vehicle, non-maneuver and pedestrian targets in different expressway events, so that the identification of expressway event categories is realized, and the track probability of q-type events is calculated as follows:
Wherein, y i(Pj) is a probability function, and the training stage carries out the training of the probability function according to the known expressway event sample; the trajectory probability representing the q-class event occurring in the time span N, y q(Pj) is the pre-policy probability, P j is the feature space-time vector, i is the number representing the monitoring camera,/> The probability of detecting a q-class event within a time span N;
Meanwhile, the detection stage matches the events in the weighted file expressway event class passive domain obtained by training, and finally outputs the identified expressway event type, namely the expressway event identification result
Wherein C t,q is a detection result, namely an event with the category of q is detected in t time;
the occurrence probability of the event with the category q is detected in the time t' for predicting the probability;
The weight is obtained through training of the sequence length T;
Step S3, inputting the training samples in the step S1 into the expressway event recognition model in the step S2 for training, wherein each training sample is firstly input into a fusion module of a target feature vector and a position vector to realize fusion, and then a space-time probability prediction module is adopted to predict and recognize the track of various targets so as to realize semantic analysis of expressway events and obtain a trained expressway event recognition model;
and S4, acquiring a monitoring video stream to be detected, inputting the monitoring video stream into the expressway event recognition model trained in the step S3 for identifying expressway events, and further obtaining the recognition result of the expressway events.
2. The highway event recognition method of the panoramic stitching camera according to claim 1, wherein: the training samples in step S1 need to be pre-trained, specifically including:
extracting vehicle targets corresponding to trucks, cars and buses in each video image frame through a feature extraction network;
the vehicle object is marked and the image of the marked vehicle object is pre-trained to identify the vehicle object from the video image frames.
3. The highway event recognition method of the panoramic stitching camera according to claim 2, wherein: the feature extraction network in step S1 adopts YoloV to input training sample V T with time length T into the feature extraction network to output the extracted features, and the width and height of the input video image frame are set to be w×h.
4. The method for identifying highway events by using the panoramic stitching camera according to claim 1, wherein: the monitoring video stream to be detected in the step S4 is a video material based on a panoramic 360-degree view formed by splicing a plurality of monitoring cameras.
5. A highway event recognition device of a panoramic stitching camera for performing the highway event recognition method of a panoramic stitching camera as recited in any one of claims 1-4, the recognition device comprising a processor, wherein: the processor is internally provided with:
the system comprises a training sample acquisition module, a processing module and a processing module, wherein the training sample acquisition module is used for acquiring at least 10 sections of monitoring video streams of each type of expressways including congestion slow running, abnormal parking, abnormal running, pedestrian non-motor vehicles and road surface casting objects as training samples, and the monitoring video streams are video materials based on panoramic 360-degree visual fields formed by splicing a plurality of monitoring cameras;
The model building module is used for building a fusion module and a space-time probability prediction module which comprise target feature vectors and position vectors to serve as a highway event recognition model;
The model training module is used for inputting training samples into the expressway event recognition model for training, wherein each training sample is firstly input into the fusion module of the target feature vector and the position vector for realizing fusion, and then a space-time probability prediction module is used for predicting and recognizing the tracks of various targets to realize semantic analysis of expressway events, so that the expressway event recognition model after training is obtained;
The recognition result output module is used for inputting the monitoring video stream to be detected into the training-completed expressway event recognition model to recognize the expressway event, so as to obtain the recognition result of the expressway event.
6. The highway event recognition device of the panoramic stitching camera of claim 5, wherein: and the processor is also internally provided with a feature extraction module which is used for pre-training the training text and extracting the features of the image of the marked vehicle target.
7. The highway event recognition device of the panoramic stitching camera of claim 5, wherein: the processor is connected with a memory for storing data or instructions.
8. The highway event recognition device of the panoramic stitching camera of claim 5, wherein: the processor is connected with a transmission device for transmitting data with an external device.
9. The highway event recognition device of the panoramic stitching camera of claim 5, wherein: the processor is connected with an input/output device for inputting or outputting information.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410346321.0A CN117953470B (en) | 2024-03-26 | 2024-03-26 | Expressway event identification method and device of panoramic stitching camera |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410346321.0A CN117953470B (en) | 2024-03-26 | 2024-03-26 | Expressway event identification method and device of panoramic stitching camera |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117953470A CN117953470A (en) | 2024-04-30 |
| CN117953470B true CN117953470B (en) | 2024-06-18 |
Family
ID=90792401
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410346321.0A Active CN117953470B (en) | 2024-03-26 | 2024-03-26 | Expressway event identification method and device of panoramic stitching camera |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117953470B (en) |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116486472A (en) * | 2022-12-29 | 2023-07-25 | 超级视线科技有限公司 | Group behavior identification method and system for monitoring video |
| CN117649632A (en) * | 2024-01-29 | 2024-03-05 | 杭州感想科技有限公司 | Expressway event identification method and device based on multi-source traffic data |
Family Cites Families (37)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8970665B2 (en) * | 2011-05-25 | 2015-03-03 | Microsoft Corporation | Orientation-based generation of panoramic fields |
| US20120300020A1 (en) * | 2011-05-27 | 2012-11-29 | Qualcomm Incorporated | Real-time self-localization from panoramic images |
| CN104156965B (en) * | 2014-08-13 | 2017-10-03 | 徐州工程学院 | A kind of automatic quick joining method of Mine Monitoring image |
| KR101946019B1 (en) * | 2014-08-18 | 2019-04-22 | 삼성전자주식회사 | Video processing apparatus for generating paranomic video and method thereof |
| US9599706B2 (en) * | 2015-04-06 | 2017-03-21 | GM Global Technology Operations LLC | Fusion method for cross traffic application using radars and camera |
| US20170169572A1 (en) * | 2015-12-15 | 2017-06-15 | Le Holdings (Beijing) Co., Ltd. | Method and electronic device for panoramic video-based region identification |
| CN107249096B (en) * | 2016-06-14 | 2021-02-26 | 杭州海康威视数字技术股份有限公司 | Panoramic camera and shooting method thereof |
| CN107085842B (en) * | 2017-04-01 | 2020-04-10 | 上海讯陌通讯技术有限公司 | Self-learning multipath image fusion real-time correction method and system |
| CN107122765B (en) * | 2017-05-22 | 2021-05-14 | 成都通甲优博科技有限责任公司 | Panoramic monitoring method and system for expressway service area |
| US20190068941A1 (en) * | 2017-08-28 | 2019-02-28 | David Anthony O'Sullivan | Method and process for transmitting a live video stream, with a 360-degree view, from the perspective of a participant in an activity |
| CN108810616B (en) * | 2018-05-31 | 2019-06-14 | 广州虎牙信息科技有限公司 | Object localization method, image display method, device, equipment and storage medium |
| US10699376B1 (en) * | 2018-10-03 | 2020-06-30 | Ambarella International Lp | eMirror with 3-in-1 stitching by non-rectilinear warping of camera views |
| CN110515378A (en) * | 2019-08-09 | 2019-11-29 | 西安电子科技大学 | A kind of intelligent Target searching method applied to unmanned boat |
| CN111369439B (en) * | 2020-02-29 | 2023-05-23 | 华南理工大学 | Panoramic all-around image real-time splicing method for automatic parking space identification based on all-around |
| CN111931627A (en) * | 2020-08-05 | 2020-11-13 | 智慧互通科技有限公司 | Vehicle re-identification method and device based on multi-mode information fusion |
| WO2022067606A1 (en) * | 2020-09-30 | 2022-04-07 | 中国科学院深圳先进技术研究院 | Method and system for detecting abnormal behavior of pedestrian, and terminal and storage medium |
| CN112800913B (en) * | 2021-01-20 | 2022-06-28 | 同济大学 | Pavement damage data space-time analysis method based on multi-source feature fusion |
| CN112793564B (en) * | 2021-02-07 | 2022-07-22 | 杨东杰 | Autonomous parking auxiliary system based on panoramic aerial view and deep learning |
| CN113688761B (en) * | 2021-08-31 | 2024-02-20 | 安徽大学 | Pedestrian behavior category detection method based on image sequence |
| CN114241372A (en) * | 2021-12-09 | 2022-03-25 | 江苏和正特种装备有限公司 | Target identification method applied to sector-scan splicing |
| US20230260266A1 (en) * | 2022-02-15 | 2023-08-17 | Waymo Llc | Camera-radar data fusion for efficient object detection |
| CN114842332A (en) * | 2022-04-12 | 2022-08-02 | 苏州艾吉威机器人有限公司 | Stock position detection method and system |
| CN114898361A (en) * | 2022-05-11 | 2022-08-12 | 安徽大学 | Peach orchard fruit state identification and counting method and system |
| CN115223191A (en) * | 2022-05-18 | 2022-10-21 | 华南农业大学 | Method for identifying and tracking group health pig behaviors |
| WO2023241536A1 (en) * | 2022-06-13 | 2023-12-21 | Anlyvis Limited | A system and method for detecting traffic events |
| CN115147723B (en) * | 2022-07-11 | 2023-05-09 | 武汉理工大学 | Inland ship identification and ranging method, inland ship identification and ranging system, medium, equipment and terminal |
| CN115661188A (en) * | 2022-11-08 | 2023-01-31 | 南京莱斯电子设备有限公司 | Road panoramic target detection tracking method under edge computing platform |
| CN116341710A (en) * | 2023-02-03 | 2023-06-27 | 中国地质大学（武汉） | Traffic accident prediction method and device, electronic equipment and storage medium |
| CN116797926A (en) * | 2023-06-09 | 2023-09-22 | 中国人民解放军32398部队 | Robot multi-mode near-field environment sensing method and system |
| CN116778435A (en) * | 2023-07-07 | 2023-09-19 | 华南理工大学 | Vehicle axle weight distribution identification method based on computer vision technology |
| CN117173096A (en) * | 2023-07-21 | 2023-12-05 | 华南理工大学 | Door surface defect detection method based on improved YOLOv8 network |
| CN117274036A (en) * | 2023-08-22 | 2023-12-22 | 合肥辉羲智能科技有限公司 | Parking scene detection method based on multi-view and time sequence fusion |
| CN117115752A (en) * | 2023-09-25 | 2023-11-24 | 四川路桥建设集团交通工程有限公司 | Expressway video monitoring method and system |
| CN117612060A (en) * | 2023-11-08 | 2024-02-27 | 浙江中智达科技有限公司 | Video early warning system, method, equipment and medium based on artificial intelligent detection |
| CN117636131A (en) * | 2023-12-05 | 2024-03-01 | 国网甘肃省电力公司兰州供电公司 | Yolo-I model-based small target identification method and related device |
| CN117392615B (en) * | 2023-12-12 | 2024-03-15 | 南昌理工学院 | Anomaly identification method and system based on monitoring video |
| CN117557466B (en) * | 2024-01-11 | 2024-04-09 | 中国科学院空天信息创新研究院 | Optical remote sensing image target image enhancement method and device based on imaging conditions |
-
2024
- 2024-03-26 CN CN202410346321.0A patent/CN117953470B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116486472A (en) * | 2022-12-29 | 2023-07-25 | 超级视线科技有限公司 | Group behavior identification method and system for monitoring video |
| CN117649632A (en) * | 2024-01-29 | 2024-03-05 | 杭州感想科技有限公司 | Expressway event identification method and device based on multi-source traffic data |
Also Published As
| Publication number | Publication date |
|---|---|
| CN117953470A (en) | 2024-04-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Aboah | A vision-based system for traffic anomaly detection using deep learning and decision trees | |
| Bura et al. | An edge based smart parking solution using camera networks and deep learning | |
| CN110738857B (en) | Vehicle violation evidence obtaining method, device and equipment | |
| CN109448365B (en) | Comprehensive monitoring method for road traffic system in cross-scale space-based platform region | |
| CN111800507A (en) | Traffic monitoring method and traffic monitoring system | |
| KR102035592B1 (en) | A supporting system and method that assist partial inspections of suspicious objects in cctv video streams by using multi-level object recognition technology to reduce workload of human-eye based inspectors | |
| CN112163525B (en) | Event type prediction method and device, electronic equipment and storage medium | |
| CN110580808B (en) | Information processing method and device, electronic equipment and intelligent traffic system | |
| US20120212617A1 (en) | System and method for detecting traffic violations on restricted roadways | |
| KR102122850B1 (en) | Solution for analysis road and recognition vehicle license plate employing deep-learning | |
| CN117953470B (en) | Expressway event identification method and device of panoramic stitching camera | |
| US11288519B2 (en) | Object counting and classification for image processing | |
| CN116311166A (en) | Traffic obstacle recognition method and device and electronic equipment | |
| CN113470009B (en) | Illegal umbrella opening detection and identification method and device, electronic equipment and storage medium | |
| Ramadan et al. | Real-time Automated License Plate Recognition and Tracking of Runaway Vehicles | |
| CN111627224A (en) | Vehicle speed abnormality detection method, device, equipment and storage medium | |
| Yadav et al. | Post-Crash Detection and Traffic Analysis | |
| Kiac et al. | ADEROS: artificial intelligence-based detection system of critical events for road security | |
| KR20230082207A (en) | System for analyzing traffic data based on npu and method thereof | |
| CN114882709A (en) | Vehicle congestion detection method and device and computer storage medium | |
| CN114241373A (en) | End-to-end vehicle behavior detection method, system, equipment and storage medium | |
| CN113850112A (en) | Road condition identification method and system based on twin neural network | |
| CN113743316A (en) | Vehicle jamming behavior identification method, system and device based on target detection | |
| Phuong et al. | Estimating the traffic density from traffic cameras | |
| CN113435352B (en) | Civilized city scoring method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant |