CN112069983A - Low-illumination pedestrian detection method and system for multi-task feature fusion shared learning - Google Patents
Low-illumination pedestrian detection method and system for multi-task feature fusion shared learning Download PDFInfo
- Publication number
- CN112069983A CN112069983A CN202010917093.XA CN202010917093A CN112069983A CN 112069983 A CN112069983 A CN 112069983A CN 202010917093 A CN202010917093 A CN 202010917093A CN 112069983 A CN112069983 A CN 112069983A
- Authority
- CN
- China
- Prior art keywords
- illumination
- network
- low
- pedestrian
- pedestrian detection
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000005286 illumination Methods 0.000 title claims abstract description 174
- 238000001514 detection method Methods 0.000 title claims abstract description 168
- 230000004927 fusion Effects 0.000 title claims abstract description 94
- 238000012549 training Methods 0.000 claims abstract description 63
- 238000011144 upstream manufacturing Methods 0.000 claims abstract description 9
- 230000006870 function Effects 0.000 claims description 29
- 238000000926 separation method Methods 0.000 claims description 28
- 238000000034 method Methods 0.000 claims description 15
- 230000007246 mechanism Effects 0.000 claims description 11
- 238000013527 convolutional neural network Methods 0.000 claims description 7
- 238000000354 decomposition reaction Methods 0.000 claims description 6
- 230000000007 visual effect Effects 0.000 claims description 5
- 238000004590 computer program Methods 0.000 claims description 4
- 238000009499 grossing Methods 0.000 claims description 3
- 238000010606 normalization Methods 0.000 claims description 2
- 238000005070 sampling Methods 0.000 claims description 2
- 238000012360 testing method Methods 0.000 description 5
- 238000010586 diagram Methods 0.000 description 4
- 238000011156 evaluation Methods 0.000 description 4
- 230000004438 eyesight Effects 0.000 description 4
- 238000012544 monitoring process Methods 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 3
- 241000282414 Homo sapiens Species 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 2
- 230000006399 behavior Effects 0.000 description 2
- 238000013135 deep learning Methods 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 238000002474 experimental method Methods 0.000 description 2
- 230000006872 improvement Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012795 verification Methods 0.000 description 2
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 description 1
- 101000713575 Homo sapiens Tubulin beta-3 chain Proteins 0.000 description 1
- 238000012300 Sequence Analysis Methods 0.000 description 1
- 102100036790 Tubulin beta-3 chain Human genes 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000012512 characterization method Methods 0.000 description 1
- 230000004456 color vision Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000009432 framing Methods 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 238000002310 reflectometry Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 230000007704 transition Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/103—Static body considered as a whole, e.g. static pedestrian or occupant recognition
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
- G06F18/253—Fusion techniques of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/29—Graphical models, e.g. Bayesian networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/52—Surveillance or monitoring of activities, e.g. for recognising suspicious objects
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Physics & Mathematics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- General Engineering & Computer Science (AREA)
- Evolutionary Computation (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Biology (AREA)
- Bioinformatics & Computational Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Molecular Biology (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- General Health & Medical Sciences (AREA)
- Health & Medical Sciences (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Multimedia (AREA)
- Human Computer Interaction (AREA)
- Image Analysis (AREA)
- Image Processing (AREA)
Abstract
The invention discloses a low-illumination pedestrian detection method and system based on multitask feature fusion shared learning, which are used for acquiring normal and low-illumination pedestrian data sets; pre-training the image illumination enhancement network by using normal and low illumination pedestrian data sets; pre-training a pedestrian detection network by using a normal illumination pedestrian data set; designing a multi-task feature fusion module capable of fusing features between an upstream task and a downstream task, performing feature fusion and sharing on the two networks, and constructing a low-illumination pedestrian detection network for multi-task feature fusion shared learning; importing the two pre-training models into a low-illumination pedestrian detection network, and training by using normal and low-illumination data sets to obtain a low-illumination pedestrian detection model for multi-task feature fusion and shared learning; and detecting the detected image by using a low-illumination pedestrian detection model of multi-task feature fusion sharing learning to obtain the position of the pedestrian in the image. The invention can accurately and efficiently detect the position of the pedestrian in the low-illumination image.
Description
Technical Field
The invention belongs to the technical field of computer vision target detection, and particularly relates to a low-illumination pedestrian detection method and system based on multi-task feature fusion shared learning.
Background
With the rapid development of world economy, people frequently come and go between different regions, different cities and different countries, the public safety hidden danger caused by the frequent coming and going of people causes related safety departments to consume much energy. Currently, video surveillance devices, which are important components of urban security systems, are widely used and installed in public areas such as roads, streets, schools, and shopping malls. The devices are mainly used for recording and storing things happening at related places, so that people can conveniently complete the requirements of remote monitoring, emergency command and the like, and the public safety of the society is guaranteed. Pedestrians are the main bodies in video monitoring, and the research and analysis of the behaviors of the pedestrians by using an intelligent technology are important components of the intelligent monitoring technology. Pedestrian detection is one of the key technologies in this field. Pedestrian detection is an important problem in the field of computer vision, and has enjoyed great success in recent years under controlled conditions.
In each application scene in the field of computer vision, pedestrians are very important analysis targets, and the important prior condition that a machine can complete subsequent tasks or interact with human beings is that the pedestrian targets in the behavior environment of the machine can be correctly identified. Pedestrian detection can be directly applied to scenes such as indoor and outdoor mobile robots, automatic driving of automobiles, security monitoring and the like, so that the pedestrian detection attracts the attention of people in recent years. Pedestrian detection has a significant impact on many applications, such as driver assistance systems, robotic navigation, and intelligent video surveillance systems. The continuous development of deep learning makes a great contribution to the performance of pedestrian detection.
Generally, pedestrian detection is a particular area of object detection. From this point of view, the pedestrian detection algorithms in deep learning can be divided into two categories: anchor boxes based methods and keypoint based methods. Convolutional Neural Networks (CNN) were first introduced into target detection in RCNN, which allows detection without the need to manually design features. The keypoint-based target detection algorithm generates a target bounding box by detecting and grouping keypoints. This greatly simplifies the output of the network and eliminates the need to design anchor boxes.
Although the pedestrian detection algorithm described above still achieves satisfactory performance under normal lighting conditions. Most of them do not consider pedestrian detection in low light environments. In practical applications, normal lighting conditions are not always guaranteed. In contrast, low light environments are very common. The main reason for poor pedestrian detection performance in low-light environments is that color and texture information in the input signal obtained under low-light conditions is severely distorted. However, color and texture information plays a crucial role in pedestrian detection. In order to solve the problem of low-illumination pedestrian detection, infrared pedestrian detection is researched at present, but the infrared pedestrian detection needs to be carried out under an infrared image, the original low-illumination image detection cannot be directly used, and the detection cost can be greatly increased.
Disclosure of Invention
Aiming at the defects or improvement requirements of the prior art, the invention provides a low-illumination pedestrian detection method and system based on multi-task feature fusion shared learning, and solves the problem of poor detection effect of pedestrians in a low-illumination environment.
To achieve the above object, according to one aspect of the present invention, there is provided a low-light pedestrian detection method based on multitask feature fusion shared learning, including:
s1: acquiring a normal illumination pedestrian data set and a low illumination pedestrian data set;
s2: constructing an image illumination enhancement network, wherein the image illumination enhancement network comprises a decomposition network and an enhancement network, and the image illumination enhancement network is trained by utilizing the normal illumination pedestrian data set and the low illumination pedestrian data set to obtain an image illumination enhancement pre-training model;
s3: constructing a self-calibration separate attention module SCSAB, which combines a self-calibration convolution network and a separate attention network, for collecting information of each spatial position in an input image to extend a field of view of each convolution layer;
s4: constructing a self-calibration separated attention hourglass network, wherein a basic module of the self-calibration separated attention hourglass network consists of SCSAB;
s5: constructing a pedestrian detection network, wherein the pedestrian detection network takes the self-calibration separated attention hourglass network as a backbone network, and trains the pedestrian detection network by using the normal illumination pedestrian data set to obtain a pedestrian detection pre-training model;
s6: based on multi-task feature fusion shared learning, designing a multi-task feature fusion module capable of fusing features between an upstream task and a downstream task, performing feature fusion and sharing on the image illumination enhancement network and the pedestrian detection network, and constructing a low-illumination pedestrian detection network for multi-task feature fusion shared learning;
s7: and importing the image illumination enhancement pre-training model and the pedestrian detection pre-training model into a low-illumination pedestrian detection network for multi-task feature fusion shared learning, training the low-illumination pedestrian detection network for multi-task feature fusion shared learning by utilizing the normal illumination pedestrian data set and the low-illumination pedestrian data set to obtain a low-illumination pedestrian detection model for multi-task feature fusion shared learning, and detecting an image to be detected through the low-illumination pedestrian detection model for multi-task feature fusion shared learning to obtain the position of a pedestrian in the image to be detected.
In some alternative embodiments, step S6 includes:
adding the characteristics of the first convolution network of the image illumination enhancement network and the characteristics of the second last SCSAB of the self-calibration separation attention hourglass network, then taking an average number, then passing through a sigmoid function, feeding back to the two networks in the next iteration, adding the characteristics of the fourth last convolution network of the image illumination enhancement network and the characteristics of the first SCSAB of the self-calibration separation attention hourglass network, then taking the average number, then passing through the sigmoid function, and feeding back to the two networks in the next iteration, thereby constructing the low-illumination pedestrian detection network with multi-task feature fusion and shared learning.
In some optional embodiments, the image illumination enhancement network is constructed based on a RetinexNet convolutional neural network, wherein a loss function L of the image illumination enhancement networkenhComprises the following steps: l isenh=Lrecon+λirLir+λisLis,λirAnd λisIs a weight coefficient, Lrecon,LirAnd LisRepresenting reconstruction, reflectance and illumination smoothness loss functions, respectively.
In some alternative embodiments, the loss function L of the pedestrian detection networkcorComprises the following steps: l iscor=Ldet+Lpull+ηLpush+γLoffWherein, η and γ are each Lpull,LpushAnd LoffWeights of three loss functions, LdetFor corner loss, LpullFor grouping corners, LpushSeparation of the diagonal, LoffIs the offset loss.
In some of the alternative embodiments, the first and second,
Ldetfor corner loss, N is the number of objects in the image, α and β are the hyperparameters controlling the contribution of each corner, C, H and W represent the number of channels, height and width, respectively, input, paijIs the score at the (i, j) position of class a in the predicted image, yaijIs an original image without normalization;
Lofffor offset loss, okIs the offset of the label, xkAnd ykIs the x and y coordinates of the corner point k, n is the down-sampling factor,representing the difference of the predicted offset and the offset of the tag calculated using the smoothing loss,representing a predicted offset;
Lpullfor grouping corners, LpushThe corners are separated, m represents the number of objects,for the embedding of the upper left corner of object m,for the embedding of the lower right corner of object m, emIs composed ofAndaverage value of emAnd ejRepresenting the embedding of objects m and j, respectively.
In some optional embodiments, the method further comprises: and designing a multi-task learning mechanism with feature fusion and sharing, wherein the mechanism can fuse features between an upstream task and a downstream task and feed back the features to other networks during the next iteration.
In some optional embodiments, the feature fusion and sharing multitask learning mechanism is: assume that there are two tasks, task A and task B, respectively, task A being a network of convolutional layers CA1Is characterized by an outputConvolutional layer C in task B networkB1Is characterized by an outputCA2And CB2Are respectively CA1And CB1The next layer of the wound up layer of (a),is CA2The input characteristics of the convolutional layer(s),is CB2Input characteristics of the convolutional layer, FiThe shared features obtained for the ith end-to-end iteration are expressed as follows:when the value of i is 1, the value of i,is represented as follows:when the value of i is greater than 1,is represented as follows:Fi-1representing the shared characteristics obtained by the i-1 st end-to-end iteration.
In some optional embodiments, the overall training loss function of the multitask feature shared low-light pedestrian detection network is: l ═ Ldet+Lcor=Ldet+Lpull+ηLpush+γLoff+ζLenhL is the total loss, ζ is the illumination enhancement loss LenhThe weight of (c).
According to another aspect of the present invention, there is provided a low-light pedestrian detection system based on multitask feature fusion shared learning, including:
the data set module is used for acquiring a normal illumination pedestrian data set and a low illumination pedestrian data set;
the image illumination enhancement module is used for constructing an image illumination enhancement network, the image illumination enhancement network comprises a decomposition network and an enhancement network, and the normal illumination pedestrian data set and the low illumination pedestrian data set are used for training the image illumination enhancement network to obtain an image illumination enhancement pre-training model;
the pedestrian detection network is trained by using the normal illumination pedestrian data set to obtain a pedestrian detection pre-training model, wherein a basic module of the self-calibration separation attention hourglass network consists of a self-calibration separation attention module SCSAB, and the self-calibration separation attention module combines a self-calibration convolution network and a separation attention network and is used for collecting information of each spatial position in an input image to expand the visual field of each convolution layer;
the multitask feature fusion module is used for designing a multitask feature fusion module capable of fusing features between an upstream task and a downstream task based on multitask feature fusion shared learning, performing feature fusion and sharing on the image illumination enhancement network and the pedestrian detection network, and constructing a low-illumination pedestrian detection network of the multitask feature fusion shared learning;
the model training module is used for importing the image illumination enhancement pre-training model and the pedestrian detection pre-training model into the multitask feature fusion shared learning low-illumination pedestrian detection network, and training the multitask feature fusion shared learning low-illumination pedestrian detection network by utilizing the normal illumination pedestrian data set and the low illumination pedestrian data set to obtain a multitask feature fusion shared learning low-illumination pedestrian detection model;
and the image detection module is used for detecting the image to be detected by using the low-illumination pedestrian detection model of the multitask feature fusion shared learning to obtain the position of the pedestrian in the image to be detected.
In some optional embodiments, the image illumination enhancement network is constructed based on a RetinexNet convolutional neural network.
In some alternative embodiments, a self-calibration based discrete attention pedestrian detection network is constructed.
According to another aspect of the present invention, there is provided a computer readable storage medium, on which a computer program is stored, which when executed by a processor, implements the steps of any of the low-light pedestrian detection methods based on multitask feature fusion shared learning.
In general, compared with the prior art, the above technical solution contemplated by the present invention can achieve the following beneficial effects:
the invention provides a low-illumination pedestrian detection method and system based on multitask feature fusion shared learning, which can accurately and efficiently detect the position of a pedestrian in a low-illumination image; and the self-calibration separated attention module is creatively provided, and the module combines the self-calibration convolution and the separated attention network to effectively collect the information of each spatial position in the input image so as to expand the visual field of each convolution layer, thereby improving the detection performance.
Drawings
Fig. 1 is a schematic flowchart of a low-light pedestrian detection method based on multitask feature fusion shared learning according to an embodiment of the present invention;
FIG. 2 is a diagram of a low light pedestrian detection network architecture based on multi-tasking feature fusion shared learning in accordance with the present invention;
FIG. 3 is a self-calibrating discrete attention block diagram proposed by the present invention;
FIG. 4 is a schematic diagram of a low-light pedestrian detection system based on multi-tasking feature fusion shared learning of the present invention;
FIG. 5 is a workflow diagram of the feature fusion and sharing multitasking learning mechanism proposed by the present invention;
FIG. 6 is a graph comparing test results of the examples of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. In addition, the technical features involved in the embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.
The invention is mainly divided into four parts: the method comprises an image illumination enhancement pre-training model, a pedestrian detection pre-training model, a low illumination pedestrian detection model of multi-task feature fusion shared learning and a low illumination pedestrian detection model of multi-task feature fusion shared learning, wherein the pedestrian position in the image is deduced from the low illumination image.
The low-light pedestrian detection method based on the multitask feature fusion shared learning disclosed by the embodiment of the invention comprises the following steps as shown in figure 1:
s1: acquiring a normal illumination pedestrian data set and a low illumination pedestrian data set;
in the embodiment of the invention, the information can be directly obtained by photographing; CityPersons (a published large-scale pedestrian detection dataset) pedestrian detection training set and verification set can also be reduced in illumination, and in the embodiment of the present invention, an RGB-based spatial brightness adjustment algorithm in OpenCV (a BSD license (open source) -based distributed cross-platform computer vision library) is used, and the algorithm is adjusted based on the current RGB value size, that is, the larger the R, G, B value, the larger the adjustment, for example: the current pixel point is (100,200,50), the adjustment coefficient is 1.1, and then the adjustment is (110,220, 55). In the present embodiment, the adjustment coefficient used is 0.8. After reducing the brightness, the normal and low light CityPersons pedestrian detection training and verification sets were used as training and test sets.
S2: constructing an image illumination enhancement network, wherein the image illumination enhancement network comprises a decomposition network and an enhancement network, and training the image illumination enhancement network by utilizing a normal illumination pedestrian data set and a low illumination pedestrian data set to obtain an image illumination enhancement pre-training model;
the normal and low-illumination pedestrian detection training sets are used for independently training the image illumination enhancement network for 100 periods to obtain an image illumination enhancement pre-training model. The network structure of the image illumination enhancement network is shown in fig. 2, the image illumination enhancement network is constructed based on a RetinexNet convolutional neural network, and the network introduces a Retinex theory. The classical Retinex theory establishes a model for human color perception, and this theory assumes that the observed image can be decomposed into two parts, a reflection channel and an illumination channel. Let S represent the source image, it can be represented by S ═ R × I, where R represents the reflectance component and I represents the illumination component, and × represents the element-by-element multiplication. For the loss function, in order to ensure that the image after restoring the illumination can retain the object edge information and also retain the smooth transition of the illumination information, the following loss functions are used in the illumination enhancement network:
Lenh=Lrecon+λirLir+λisLis
wherein λ isirAnd λisCoefficients representing the balance of reflectivity and illuminance. Loss function Lrecon,LirAnd LisRepresenting reconstruction, reflectance and illumination smoothness functions, respectively.
S3: constructing a pedestrian detection network, wherein the pedestrian detection network takes two self-calibration separation attention-based hourglass networks as a backbone network, and trains the pedestrian detection network by using a normal illumination pedestrian data set to obtain a pedestrian detection pre-training model, wherein a basic module of the self-calibration separation attention hourglass network consists of a self-calibration separation attention module SCSAB (simple sequence analysis and clustering routine) which is combined with a self-calibration convolution network and a separation attention network and is used for collecting information of each spatial position in an input image to expand the visual field of each convolution layer;
in the embodiment of the invention, a pedestrian detection training set with normal illumination is used for independently training a pedestrian detection network for 100 periods to obtain a pedestrian detection pre-training model. The network structure of the pedestrian detection network is shown in fig. 2. For the pedestrian detection network, a target detection algorithm thought based on key points is referred, and the target detection algorithm based on the key points generates an object boundary box by detecting and grouping the key points, so that the output of the network is greatly simplified, the need of designing an anchor box is eliminated, and meanwhile, an attention mechanism is introduced into the network to further improve the detection performance. The Self-calibration discrete Attention Block module utilizes an autonomously designed hourglass network as a backbone network of a pedestrian detection network, the hourglass network consists of the Self-calibration discrete Attention Block (SCSAB), and as shown in fig. 3, the Self-calibration convolution and discrete Attention network module effectively collects information of each spatial position in an input image to expand the view field of each convolution layer, so that the detection performance is improved.
For the loss function of the pedestrian detection network, a focal loss function of α -2 and β -4 may be used, and let p beaijPredict the score at the (i, j) position of class a in the image and let yaijRaw images were not normalized.
Where N is the number of objects in the image and α and β are the hyper-parameters that control the contribution of each point. C, H and W represent the number of input channels, height and width, respectively.
When an input image passes through a convolutional layer, the output is typically smaller in size than the input image. Thus, location (x, y) in the image maps to location in the heat mapWhere n is the downsampling factor. Some accuracy may be lost when remapping the location from the heat map to the input image. To solve this problem, a position offset is predicted, which slightly adjusts the corner point positions and then remaps them to the input resolution.
Wherein o iskIs an offset, xkAnd ykAre the x and y coordinates of the corner point k. One set of offsets is predicted to be shared by the top left corner of all classes and another set of offsets is shared by the bottom right corner. For training, the offset loss is labeled LoffAnd applying the smoothed L1 loss as the offset loss:
wherein,representing the difference of the predicted offset and the offset of the tag calculated using the smoothing loss,indicating the predicted offset.
There may be multiple objects in the image and thus multiple upper left and lower right corners may be detected, so it is necessary to determine whether a pair of upper left and lower right corners are from the same bounding box. Order toFor the embedding of the upper left corner of object m,is the embedding of the lower right corner of object m. The network is trained to group corners using "pull" losses and to separate corners using "push" losses:
wherein e ismIs thatAndaverage value of emAnd ejRepresenting the embedding of objects m and j, respectively, ejIs thatAndis determined by the average value of (a) of (b),for the embedding of the upper left corner of object j,for the embedding of the lower right corner of object j, the total loss of training for pedestrian detection is as follows:
Lcor=Ldet+Lpull+ηLpush+γLoff,
wherein eta and gamma are each Lpull,LpushAnd LoffWeights of three loss functions, LcorIs the total loss of the pedestrian detection network.
S4: based on multi-task feature fusion and sharing learning, designing a multi-task feature fusion module capable of fusing features among different tasks, sharing the features of an image illumination enhancement network and a pedestrian detection network, adding the features of the first 3 x 3 convolution network of the enhancement network and the features of the second last SCSAB of a self-calibration separation attention hourglass network, then taking an average number, then passing through a sigmoid function, feeding back to the two networks in the next iteration, adding the features of the fourth last 3 x 3 convolution network of the enhancement network and the features of the first SCSAB of the self-calibration separation attention hourglass network, then taking the average number, then passing through the sigmoid function, and feeding back to the two networks in the next iteration to construct a low-illumination pedestrian detection network of multi-task feature fusion and sharing learning;
a multi-task learning mechanism with feature fusion and sharing is designed, features between an upstream task and a downstream task can be fused by the multi-task learning mechanism, and the features are fed back to other networks when next iteration is carried out. As shown in fig. 5, the multi-task learning mechanism for feature fusion and sharing is: assume that there are two tasks, task A and task B, respectively, task A being a network of convolutional layers CA1Is characterized by an outputConvolutional layer C in task B networkB1Is characterized by an outputCA2And CB2Are respectively CA1And CB1The next layer of the wound up layer of (a),is CA2The input characteristics of the convolutional layer(s),is CB2Input characteristics of the convolutional layer, FiThe shared features obtained for the ith end-to-end iteration are expressed as follows:when the value of i is 1, the value of i,is represented as follows:when the value of i is greater than 1,is represented as follows:Fi-1representing the shared characters obtained by the i-1 st end-to-end round iterationAnd (5) carrying out characterization.
The low-illumination pedestrian detection network for multi-task feature fusion and shared learning comprises: in order to further improve the performance of the two networks, a multitask feature fusion module is introduced on the basis, and the detailed structure of the multitask feature fusion module is shown in fig. 2. Wherein, in ithIn the iteration, in the multitask feature fusion module, the image illumination is respectively enhanced to the features in the sub-networksAndfeatures fused into pedestrian detection sub-networkAndand the two fused features are respectively marked asAndfeature(s)And featuresFrom the first 3 x 3 convolutional layer and the fourth last 3 x 3 convolutional layer in the enhancement network, respectively. Feature(s)From the characteristics of the second to last SCSAB in the self-calibrating split attention hourglass networkFrom the first to the secondOne SCSAB. Also, the size of these features is the same.Andis represented as follows:
then theAndafter the sigmoid function is entered, it is expressed asAndat i +1thIn the course of the iteration(s),are respectively connected withAndelement-by-element multiplication as input to the second convolutional layer of the enhancement network and the last SCSAB of the self-calibrating split attention hourglass network. The input is characterized by being represented asAndthe formula is as follows:
the same fusion and sharing method acts on the input of the last convolution in the enhancement network and the input of the second SCSAB in the self-calibrating split attention hourglass network. The input is characterized by being represented asAndthe formula is as follows:
finally, the training loss function L of the low-illumination pedestrian detection network for multi-task feature fusion shared learning is as follows:
L=Ldet+Lpull+ηLpush+γLoff+ζLenh,
wherein eta and gamma are each Lpull,LpushAnd LoffWeights of the three loss functions. ζ is the loss of light enhancement LenhThe weight of (c). The sum η is set to 0.1, γ is set to 1, and ζ is set to 0.05.
S5: importing the image illumination enhancement pre-training model and the pedestrian detection pre-training model into a low-illumination pedestrian detection network for multi-task feature fusion shared learning, and training the low-illumination pedestrian detection network for multi-task feature fusion shared learning by using a normal illumination pedestrian data set and a low-illumination pedestrian data set to obtain a low-illumination pedestrian detection model for multi-task feature fusion shared learning;
and training the low-illumination pedestrian detection network of the multitask feature fusion shared learning by the normal and low-illumination pedestrian detection training sets, simultaneously importing the image illumination enhancement pre-training model and the pedestrian detection pre-training model which are trained well before, and training for 100 periods to obtain the low-illumination pedestrian detection model of the multitask feature fusion shared learning.
S6: and detecting the image to be detected by using the low-illumination pedestrian detection model of the multi-task feature fusion shared learning to obtain the position of the pedestrian in the image. And inputting the low-illumination test set picture into a trained multi-task feature fusion and shared learning low-illumination pedestrian detection model for reasoning, and framing the position of the pedestrian in the image.
The invention also provides a low-light pedestrian detection system based on multitask feature fusion shared learning, which is used for realizing the low-light pedestrian detection method based on multitask feature fusion shared learning, and as shown in fig. 4, the low-light pedestrian detection system based on multitask feature fusion shared learning comprises:
a data set module 101, configured to obtain a normal-illumination pedestrian data set and a low-illumination pedestrian data set;
the image illumination enhancement module 102 is used for constructing an image illumination enhancement network, the image illumination enhancement network comprises a decomposition network and an enhancement network, and the image illumination enhancement network is trained by utilizing a normal illumination pedestrian data set and a low illumination pedestrian data set to obtain an image illumination enhancement pre-training model;
the pedestrian detection module 103 is used for constructing a pedestrian detection network, the pedestrian detection network takes two self-calibration separation attention-based hourglass networks as a backbone network, and the pedestrian detection network is trained by using a normal illumination pedestrian data set to obtain a pedestrian detection pre-training model, wherein a basic module of the self-calibration separation attention hourglass network consists of a self-calibration separation attention module SCSAB, and the self-calibration separation attention module combines a self-calibration convolution network and a separation attention network and is used for collecting information of each spatial position in an input image to expand the visual field of each convolution layer;
the multitask feature fusion module 104 is used for fusing features between upstream and downstream tasks, sharing features of the image illumination enhancement network and the pedestrian detection network, adding the features of the first 3 x 3 convolution network of the enhancement network and the features of the second last SCSAB of the self-calibration separation attention hourglass network, then taking an average number, then passing through a sigmoid function, feeding back to the two networks in the next iteration, adding the features of the fourth last 3 x 3 convolution network of the enhancement network and the features of the first SCSAB of the self-calibration separation attention hourglass network, then taking an average number, then passing through the sigmoid function, feeding back to the two networks in the next iteration, and constructing the multitask feature fusion shared learning low-illumination pedestrian detection network;
the model training module 105 is used for importing the image illumination enhancement pre-training model and the pedestrian detection pre-training model into a low-illumination pedestrian detection network for multi-task feature fusion shared learning, and training the low-illumination pedestrian detection network for multi-task feature fusion shared learning by using a normal illumination pedestrian data set and a low-illumination pedestrian data set to obtain a low-illumination pedestrian detection model for multi-task feature fusion shared learning;
and the image detection module 106 is configured to detect the image to be detected by using the low-illumination pedestrian detection model of the multitask feature fusion shared learning, so as to obtain the position of the pedestrian in the image.
Further, an image illumination enhancement network is constructed based on the RetinexNet convolutional neural network; a pedestrian detection network is constructed based on the self-calibration separation attention block provided by the invention.
The present invention also provides a computer storage medium having stored therein a computer program executable by a computer processor, the computer program executing the low-light pedestrian detection model method based on multitask feature fusion shared learning described above.
The invention is most suitable forThe test example is provided using normal and low light CityPersons data sets, including 2975 pictures each for normal and low light training sets and 500 pictures each for normal and low light test sets. The experiment was implemented in a pytoreh and trained using 3 RTX 2080Ti graphics, while applying Adam optimization algorithm. In the selection of experimental parameters, the set learning rate is 0.0001. The evaluation index complies with the evaluation standard of the california institute of technology: log mean Miss Rate (MR) per image-2),MR-2The lower the value of (a), the better the algorithm performance. By MR-2And the evaluation index is compared with other excellent pedestrian detection algorithms to prove the superiority of the invention. TABLE 1 by MR-2The evaluation indexes show the results of the comparison, and fig. 6 is a comparison graph of the results of the detection of the pedestrian position, in which (a) shows the input image, (b) shows the CSP detection result, (c) shows the ALFNet detection result, (d) shows the cenet detection result, (e) shows the CornerNet detection result, (f) shows the CornerNet-Saccade detection result, (g) shows the detection result of the present invention, and (h) shows the Benchmark detection result.
The invention selects some excellent pedestrian detection or target detection methods, such as: CSP, ALFNet, CenterNet, CornerNet, and CornerNet-Saccade. ALFNet is the most representative algorithm for pedestrian detection using anchor boxes, and CSP and CenterNet are the best algorithms for pedestrian detection and target detection based on a central point, respectively. Meanwhile, CornerNet and CornerNet-Saccade are representative algorithms of the corner-based object detection method. All algorithms in table 1 train the algorithms with normal and low-light training sets, so all of these pedestrian detection networks have the capability of processing low-light images, and the fairness of the experiment is ensured. In addition, in order to further explain the function of the multitask feature fusion module in the algorithm of the patent, the algorithm in table 2 cascades the RetinexNet illumination enhancement algorithm and the detection algorithm together in a cascading manner. As can be seen from the results in Table 2, the indexes of the cascade method are still not superior to the algorithm of the invention, which proves that the multitask feature fusion module plays an important role.
Table 1 comparison of the present invention with five excellent algorithms
Table 2 comparison of the results of the present invention with five excellent algorithms cascaded by RetinexNet
From the experimental results of the table, the algorithm has obvious advantages compared with the other five methods.
The parts not described in the specification are prior art or common general knowledge. The present embodiments are illustrative only and not intended to limit the scope of the present invention, and modifications and equivalents thereof by those skilled in the art are considered to fall within the scope of the present invention as set forth in the claims.
It should be noted that, according to the implementation requirement, each step/component described in the present application can be divided into more steps/components, and two or more steps/components or partial operations of the steps/components can be combined into new steps/components to achieve the purpose of the present invention.
It will be understood by those skilled in the art that the foregoing is only a preferred embodiment of the present invention, and is not intended to limit the invention, and that any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the scope of the present invention.
Claims (10)
1. A low-light pedestrian detection method based on multitask feature fusion shared learning is characterized by comprising the following steps:
s1: acquiring a normal illumination pedestrian data set and a low illumination pedestrian data set;
s2: constructing an image illumination enhancement network, wherein the image illumination enhancement network comprises a decomposition network and an enhancement network, and the image illumination enhancement network is trained by utilizing the normal illumination pedestrian data set and the low illumination pedestrian data set to obtain an image illumination enhancement pre-training model;
s3: constructing a self-calibration separate attention module SCSAB, which combines a self-calibration convolution network and a separate attention network, for collecting information of each spatial position in an input image to extend a field of view of each convolution layer;
s4: constructing a self-calibration separated attention hourglass network, wherein a basic module of the self-calibration separated attention hourglass network consists of SCSAB;
s5: constructing a pedestrian detection network, wherein the pedestrian detection network takes the self-calibration separated attention hourglass network as a backbone network, and trains the pedestrian detection network by using the normal illumination pedestrian data set to obtain a pedestrian detection pre-training model;
s6: based on multi-task feature fusion shared learning, designing a multi-task feature fusion module capable of fusing features between an upstream task and a downstream task, performing feature fusion and sharing on the image illumination enhancement network and the pedestrian detection network, and constructing a low-illumination pedestrian detection network for multi-task feature fusion shared learning;
s7: and importing the image illumination enhancement pre-training model and the pedestrian detection pre-training model into a low-illumination pedestrian detection network for multi-task feature fusion shared learning, training the low-illumination pedestrian detection network for multi-task feature fusion shared learning by utilizing the normal illumination pedestrian data set and the low-illumination pedestrian data set to obtain a low-illumination pedestrian detection model for multi-task feature fusion shared learning, and detecting an image to be detected through the low-illumination pedestrian detection model for multi-task feature fusion shared learning to obtain the position of a pedestrian in the image to be detected.
2. The method according to claim 1, wherein step S6 includes:
adding the characteristics of the first convolution network of the image illumination enhancement network and the characteristics of the second last SCSAB of the self-calibration separation attention hourglass network, then taking an average number, then passing through a sigmoid function, feeding back to the two networks in the next iteration, adding the characteristics of the fourth last convolution network of the image illumination enhancement network and the characteristics of the first SCSAB of the self-calibration separation attention hourglass network, then taking the average number, then passing through the sigmoid function, and feeding back to the two networks in the next iteration, thereby constructing the low-illumination pedestrian detection network with multi-task feature fusion and shared learning.
3. The method of claim 1, wherein the image illumination enhancement network is constructed based on a RetinexNet convolutional neural network, wherein a loss function L of the image illumination enhancement networkenhComprises the following steps: l isenh=Lrecon+λirLir+λisLis,λirAnd λisIs a weight coefficient, Lrecon,LirAnd LisRepresenting reconstruction, reflectance and illumination smoothness loss functions, respectively.
4. The method of claim 3, wherein the loss function L of the pedestrian detection networkcorComprises the following steps: l iscor=Ldet+Lpull+ηLpush+γLoffWherein, η and γ are each Lpull,LpushAnd LoffWeights of three loss functions, LdetFor corner loss, LpullFor grouping corners, LpushSeparation of the diagonal, LoffIs the offset loss.
5. The method of claim 4,
Ldetfor corner loss, N is the number of objects in the image, α and β are the hyperparameters controlling the contribution of each corner, C, H and W represent the number of channels input, respectively, highDegree and width, paijIs the score at the (i, j) position of class a in the predicted image, yaijIs an original image without normalization;
Lofffor offset loss, okIs an offset, xkAnd ykIs the x and y coordinates of the corner point k, n is the down-sampling factor,representing the difference of the predicted offset and the offset of the tag calculated using the smoothing loss,representing a predicted offset;
Lpullfor grouping corners, LpushThe corners are separated, m represents the number of objects,for the embedding of the upper left corner of object m,for embedding in the lower right corner of object m, emIs composed ofAndaverage value of emAnd ejRepresenting the embedding of objects m and j, respectively.
6. The method of claim 1, further comprising: and designing a multi-task learning mechanism with feature fusion and sharing, wherein the mechanism can fuse features between an upstream task and a downstream task and feed back the features to other networks during the next iteration.
7. The method of claim 6, wherein the feature fusion and sharing multitask learning mechanism is: assume that there are two tasks, task A and task B, respectively, task A being a network of convolutional layers CA1Is characterized by an outputConvolutional layer C in task B networkB1Is characterized by an outputCA2And CB2Are respectively CA1And CB1The next layer of the wound up layer of (a),is CA2The input characteristics of the convolutional layer(s),is CB2Input characteristics of the convolutional layer, FiThe shared features obtained for the ith end-to-end iteration are expressed as follows:when the value of i is 1, the value of i,is represented as follows:when the value of i is greater than 1,is represented as follows:Fi-1representing the shared characteristics obtained by the i-1 st end-to-end iteration.
8. The method of claim 5, wherein the overall training loss function of the multitask feature shared low-light pedestrian detection network is: l ═ Ldet+Lcor=Ldet+Lpull+ηLpush+γLoff+ζLenhL is the total loss, ζ is the illumination enhancement loss LenhThe weight of (c).
9. A low-light pedestrian detection system based on multitask feature fusion shared learning is characterized by comprising the following components in parts by weight:
the data set module is used for acquiring a normal illumination pedestrian data set and a low illumination pedestrian data set;
the image illumination enhancement module is used for constructing an image illumination enhancement network, the image illumination enhancement network comprises a decomposition network and an enhancement network, and the normal illumination pedestrian data set and the low illumination pedestrian data set are used for training the image illumination enhancement network to obtain an image illumination enhancement pre-training model;
the pedestrian detection network is trained by using the normal illumination pedestrian data set to obtain a pedestrian detection pre-training model, wherein a basic module of the self-calibration separation attention hourglass network consists of a self-calibration separation attention module SCSAB, and the self-calibration separation attention module combines a self-calibration convolution network and a separation attention network and is used for collecting information of each spatial position in an input image to expand the visual field of each convolution layer;
the multitask feature fusion module is used for designing a multitask feature fusion module capable of fusing features between an upstream task and a downstream task based on multitask feature fusion shared learning, performing feature fusion and sharing on the image illumination enhancement network and the pedestrian detection network, and constructing a low-illumination pedestrian detection network of the multitask feature fusion shared learning;
the model training module is used for importing the image illumination enhancement pre-training model and the pedestrian detection pre-training model into the multitask feature fusion shared learning low-illumination pedestrian detection network, and training the multitask feature fusion shared learning low-illumination pedestrian detection network by utilizing the normal illumination pedestrian data set and the low illumination pedestrian data set to obtain a multitask feature fusion shared learning low-illumination pedestrian detection model;
and the image detection module is used for detecting the image to be detected by using the low-illumination pedestrian detection model of the multitask feature fusion shared learning to obtain the position of the pedestrian in the image to be detected.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the low-light pedestrian detection method based on multitask feature fusion shared learning according to any one of claims 1 to 8.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202010917093.XA CN112069983B (en) | 2020-09-03 | 2020-09-03 | Low-light pedestrian detection method and system for multi-task feature fusion sharing learning |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202010917093.XA CN112069983B (en) | 2020-09-03 | 2020-09-03 | Low-light pedestrian detection method and system for multi-task feature fusion sharing learning |
Publications (2)
Publication Number | Publication Date |
---|---|
CN112069983A true CN112069983A (en) | 2020-12-11 |
CN112069983B CN112069983B (en) | 2024-03-26 |
Family
ID=73666641
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202010917093.XA Active CN112069983B (en) | 2020-09-03 | 2020-09-03 | Low-light pedestrian detection method and system for multi-task feature fusion sharing learning |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN112069983B (en) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN112862713A (en) * | 2021-02-02 | 2021-05-28 | 山东师范大学 | Attention mechanism-based low-light image enhancement method and system |
Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20140340555A1 (en) * | 2013-05-14 | 2014-11-20 | Canon Kabushiki Kaisha | Image sensing apparatus |
CN104317244A (en) * | 2014-09-28 | 2015-01-28 | 北京理工大学 | Reconfigurable manufacturing system part family construction method |
CN109711258A (en) * | 2018-11-27 | 2019-05-03 | 哈尔滨工业大学(深圳) | Lightweight face critical point detection method, system and storage medium based on convolutional network |
-
2020
- 2020-09-03 CN CN202010917093.XA patent/CN112069983B/en active Active
Patent Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20140340555A1 (en) * | 2013-05-14 | 2014-11-20 | Canon Kabushiki Kaisha | Image sensing apparatus |
CN104317244A (en) * | 2014-09-28 | 2015-01-28 | 北京理工大学 | Reconfigurable manufacturing system part family construction method |
CN109711258A (en) * | 2018-11-27 | 2019-05-03 | 哈尔滨工业大学(深圳) | Lightweight face critical point detection method, system and storage medium based on convolutional network |
Non-Patent Citations (1)
Title |
---|
陈双玉等: "基于双目立体视觉和SVM算法行人检测方法", 《华中科技大学学报(自然科学版)》, 16 October 2015 (2015-10-16), pages 141 * |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN112862713A (en) * | 2021-02-02 | 2021-05-28 | 山东师范大学 | Attention mechanism-based low-light image enhancement method and system |
CN112862713B (en) * | 2021-02-02 | 2022-08-09 | 山东师范大学 | Attention mechanism-based low-light image enhancement method and system |
Also Published As
Publication number | Publication date |
---|---|
CN112069983B (en) | 2024-03-26 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US10453197B1 (en) | Object counting and instance segmentation using neural network architectures with image-level supervision | |
CN110163187B (en) | F-RCNN-based remote traffic sign detection and identification method | |
Kumar et al. | A new vehicle tracking system with R-CNN and random forest classifier for disaster management platform to improve performance | |
CN111814595B (en) | Low-illumination pedestrian detection method and system based on multi-task learning | |
CN113420607A (en) | Multi-scale target detection and identification method for unmanned aerial vehicle | |
Huang et al. | Spatial-temproal based lane detection using deep learning | |
KR20180048407A (en) | Apparatus and method for detecting a lane | |
CN112084869A (en) | Compact quadrilateral representation-based building target detection method | |
CN113255443A (en) | Pyramid structure-based method for positioning time sequence actions of graph attention network | |
CN114943757A (en) | Unmanned aerial vehicle forest exploration system based on monocular depth of field prediction and depth reinforcement learning | |
Asgarian et al. | Fast drivable area detection for autonomous driving with deep learning | |
Ren et al. | A new multi-scale pedestrian detection algorithm in traffic environment | |
Li et al. | Detection of road objects based on camera sensors for autonomous driving in various traffic situations | |
CN112069983B (en) | Low-light pedestrian detection method and system for multi-task feature fusion sharing learning | |
Zhang et al. | An efficient deep neural network with color-weighted loss for fire detection | |
CN117576149A (en) | Single-target tracking method based on attention mechanism | |
He et al. | Real-time pedestrian warning system on highway using deep learning methods | |
Tan et al. | UAV image object recognition method based on small sample learning | |
Nataprawira et al. | Pedestrian Detection in Different Lighting Conditions Using Deep Neural Networks. | |
CN117636241A (en) | Low-light scene multi-mode pedestrian detection tracking method based on decision-level fusion | |
Dutta | Seeing Objects in Dark with Continual Contrastive Learning | |
CN117523491A (en) | Unmanned aerial vehicle-mounted fire automatic detection method and device | |
Malik et al. | High-Level Semantic Feature Detector: Pedestrian Detection Based On Improved Mask R-CNN Algorithm | |
Wang | Human Detection in aSequence of Thermal Images using Deep Learning | |
CN117423135A (en) | Pedestrian target detection method based on improved YOLOv8 lightweight network model |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant | ||
GR01 | Patent grant |