CN106897664A - A kind of pedestrian detection method based on distributed big data platform - Google Patents

Abstract

The present invention proposes a kind of pedestrian detection method based on distributed big data platform, the method can real-time processing online data, while magnanimity pedestrian detection result can also be saved in into distributed memory system HDFS.Employ the pedestrian detection model of SVM Adaboost, ensure that accuracy of detection, while distributed parallel computational model, greatly improve the calculating speed of service, solve training grader slow, the problems such as storage hardware is not enough, is highly suitable for current huge social public arena monitoring, the video realization treatment of intelligent transportation.

Description

A kind of pedestrian detection method based on distributed big data platform

Technical field

It is more particularly to a kind of to be based on spark Distributed Computing Platforms the present invention relates to machine learning algorithm parallelization field Pedestrian detection method.

Background technology

Traditional pedestrian detection method use video background modeling method, motion frame difference detection method, optical flow objective detection method, These methods are all first to extract target, and then the method by classifying extracts pedestrian.It is simpler that these methods are suitable for background Single scene, but in the case where background is more complicated, the target of extraction occurs adhesion phenomenon, in classification by its misidentification To be other targets, also have powerful connections and receive the influence of illumination, can be target by background flase drop.

Based on existing model, such as：Fast linear SVM models, Adaboost detection hog characteristic models, convolution god Through network, non-linear SVM models, DPM models, deep neural network model can be provided with fairly good precision, but problem It is exactly slow calculating speed, training pattern is slow.

AdaBoost is a kind of greedy iterative algorithm, and implementation is as follows：First, AdaBoost algorithms are by training set Each sample set assigns an identical weight W.Then, the algorithm is iterated computing.According to sample in each interative computation The classification error rate ε t, AdaBoost of this collection assigns a new weighted value Wt to each sample set in training set again. Wt is a function related to ε t, i.e. classification error rate ε t small sample set weighted value Wt is small, otherwise classification error rate ε t are big Sample set weighted value Wt it is big.AdaBoost algorithms can be successfully made to focus on classification using the method for assigning weights again wrong The big sample set of rate by mistake, finally makes the classification error rate of sample set reach the reasonable value of setting.AdaBoost algorithms are for same Training set train different graders (i.e. Weak Classifier), and then these weak classifier sets are got up, constitute one more Powerful grader (i.e. strong classifier).The algorithm is only required to error rates of weak classifiers and (slightly better than guesses at random less than 50% Survey), then it is reduced to designated value by way of the overall classification error rate of interative computation is by with index.

Spark is the real-time Computational frame of distribution that Apache is released, there is provided parallel programming model, user need to only call Related API can complete distributed treatment program, for the treatment of big data provides favourable condition, therefore, based on distribution The pedestrian detection of the real-time Computational frames of formula spark, can greatly provide training speed, image processing speed, substantially improve because It is precision caused by hardware device, speed issue.

SVM is the machine learning in a kind of VC dimensions theory and Structural risk minization basis based on Statistical Learning Theory System, is mainly used to process the classification problem of binary sample, according to limited sample information in the complexity of model and study energy Seek optimal compromise between power, to obtain best Generalization Ability.The method that algorithm is mapped using kernel function, by luv space Sample be mapped to high-dimensional feature space, in high-dimensional feature space, find a hyperplane, this hyperplane is correctly classified sample This, and make the interval between positive negative sample maximum.

During pedestrian detection, because the characteristic that possesses is more and the problems such as training sample imbalance, using single The problems such as AdaBoost algorithms of Weak Classifier easily trigger long training time and over-fitting.Herein for traditional AdaBoost The deficiency of algorithm, proposes to replace AdaBoost algorithms (SVM-AdaBoost) the enhancing grader of single Weak Classifier using SVM Classification capacity.

The content of the invention

A kind of pedestrian detection method based on distributed big data platform, it is characterised in that comprise the following steps：

In order to be able to preferably solve the problems, such as the training effectiveness under the conditions of mass data, the present invention proposes a kind of based on distribution The real-time Computational frames of formula Spark, improve the pedestrian detection parallelization computational methods of SVM-AdaBoost, using parallel meter of new generation Calculation technology, under conditions of training precision is not influenceed, improves the training speed of SVM, improves training effectiveness, realizes SVM- AdaBoost parallel computations on multiple nodes.

Image set from pedestrian detection is received by message collection mechanism Kafka, starts Spark clusters, created SparkContext。

Then pre-processing image data, is cut into the sum of multiple subsets, by picture stream by the picture training of positive and negative sample set The multiple elasticity distribution formula data set RDD of generation, independent calculating is carried out to each RDD.

Image is divided into small cell factory by hog feature extractions first, gradient is then calculated on each cell factory straight Fang Tu, and the result of calculating is normalized using a kind of pattern of block-by-block, finally each cell factory is returned corresponding special Levy description.

This method uses RBF as the kernel function of SVM：

Method is mainly comprised the following steps：

Step 1：Given training set sample S={ (x₁,y₁),...,(x_n,y_n), the parameter of cycle-index T, SVM kernel function σ={ σ₁,σ₂... }, C={ C₁,C₂,...}.

Step 2：Setting training set sample weights：W_i ¹=1/N, i=1 ..., N

Step 3：One SVM (RBFSVM) Weak Classifier based on Radial basis kernel function of training.

Step 4：Calculate training error rate h_t：If ξ_tMore than 50%, then return to step 3, Otherwise carry out step 5.

Step 5：The weight h of RBFSVM Weak Classifiers is reset according to error rate_t:

Step 6：Reset the weight of training sample set

I=1, K, N, C_tIt is normalized parameter.

Step 7：After T times circulates or reaches specified accuracy, the grader after output training

The scale of class where the initialization reuse sample of sample is marked, w is expressed as_i=1/C_n, the sample of so rare class This has weights higher, and the probability being selected in rule sampling is larger, cries easily be pumped in an iterative process, it is to avoid Grader ignores the phenomenon of rare class.To the 1-r of multiclass SVM, wheel changes commanders any type therein as positive class, and other are used as negative Class, if there is m class, carries out m following steps：

(1) w={ w=1/C_n| j=1,2 ..., n }, { weights of the N number of sample of initialization, C_nIt is sample in the affiliated class of sample Number

(2) k is made to represent the wheel number of iteration

(3) For i=1 to k do

(4) according to w_iBy being sampled (put back to) generation training set D to D_i

(5) in D_iUpper training base grader C_i

(6) C is used_iTo all sample classifications in former training set D

(7) basisCalculate weighted error

(8)ifε_i＞ 0.5then

(9) w={ w_j=1/C_n| j=1,2 ... N } { resetting the weights of N number of sample }

(10) 4 are returned

(11)End if

(12) basisAdjust the weights of each sample

(13)End for

Intuitively understand, first, when error rate is bigger, the weight of grader is smaller, this meets general explanation and divides Class device performance more high confidence level is also bigger, and proportion is also bigger in final voting.Secondly, in new samples distribution In renewal process, correct sample of classifying acts on smaller in the study of next grader, and the sample of misclassification is in next grader Effect in study is bigger, so can be that new classifier design is more concentrated on wrong before point of sample classification, makes entirety Classification performance improve.

Brief description of the drawings

Fig. 1 distribution pedestrian detection video detection flow charts.

Specific embodiment

To make the purpose of the present invention, technical scheme and advantage are of greater clarity, with reference to specific embodiment and join According to accompanying drawing, the present invention is described in more detail.

Spark platform clusters are built, using a server as Master, 4 servers are used as Slaver.Wherein Dependence between Master essential records data flow is simultaneously responsible for task scheduling and generates new RDD, and Slaver is main It is the storage of the calculating and data for realizing algorithm.Kafka receives video flowing, flat come cluster management spark by zookeeper Platform cluster, can reach stabilization, improve fault freedom.

It is the superiority of verification algorithm, we are using challenging MIT pedestrian storehouse and INRIA pedestrian storehouse Liang Ge storehouses Sample, wherein MIT pedestrian storehouse include 19240 pedestrian's positive sample coloured images, and without negative sample, INRIA pedestrian storehouse includes one Individual training storehouse and a test library, wherein training storehouse includes 4416 positive samples, test library includes 2126 positive samples, this examination Selection which part sample is tested as training and test sample, test sample is used for assessing false drop rate and the missing inspection of training sample Rate.

Pedestrian's video flowing is then initialized by Kafka message collections, starts SparkContext, SparkStreaming。

The picture that will be received is converted into spark streaming, and picture stream then is divided into many according to pedestrian detection frame Individual RDD carrys out burst treatment.

RDD elastic datas are concentrated carries out hog feature extractions to video flowing, the picture 64*128 pictures of the sample set in the present invention Element, is then 3781 dimension description after hog feature extractions.

Specific features extract as follows：

1) gray processing (regarding image as the 3-D view of (x, y, z) (gray scale))；

2) standardization (normalization) of color space is carried out to input picture using Gamma correction methods；Purpose is regulation figure The contrast of picture, reduces the influence caused by the shade and illumination variation of image local, while the interference of noise can be suppressed；

3) gradient (including size and Orientation) of each pixel of image is calculated；Primarily to capture profile information, while The interference that further weakened light shines.

4) small cells (such as 6*6 pixels/cell) is divided an image into；

5) histogram of gradients (numbers of different gradients) of each cell (cell factory) is counted, you can form each cell Descriptor；

6) block (block) will be constituted per several cell, the feature descriptor strings of all cell in a block Connection gets up just to obtain the HOG features descriptor of the block.

Learning sample collection, non-linear SVM model selections are trained with the pedestrian detection model based on SVM-Adaboost Be Radial basis kernel function.

Pattern function is：

It is translated into dual form

Wherein ＜ φ x_iφ (x) ＞ are inner product

Strengthen lifting svm classifier effect using Adaboost.Algorithm steps are as follows：

1) initialization D={ x are startedⁱ,y₁,.......,xⁿ,y_n},k_max,W_i(i)=1/n, i=1 ... .n

2) K ＜ -0

3) do K ＜-K+1

4) training is using according to W_kThe weak learner C of the D of (i) sampling_k

5)E_k＜-to using W_kThe C of the D measurements of (i)_kTraining error

6)

7)

8) Until k=k_max

9)Return C_kAnd a_kK=1 ... ..k_max(totality of Weighted Coefficients grader)

10)end

By judging the result yi of detection model, if greater than 0, then it is assumed that be pedestrian in the detection block, otherwise, then it is assumed that The detection block is background.

To detect the detection block for pedestrian by the join in spark by the data of detection block be put into one it is new In RDD, then merged to detection block, wherein merging closer detection block, deleted than larger and smaller detection Frame.The picture of the pedestrian detection frame after detection is finally saved in HDFS distributed file systems.

Specific embodiment of the invention is described above.It is to be appreciated that the invention is not limited in above-mentioned Particular implementation, those skilled in the art can within the scope of the claims make various deformations or amendments, this not shadow Sound substance of the invention.

Claims (6)

1. a kind of pedestrian detection method based on distributed big data platform, it is characterised in that comprise the following steps：

S1：Image set from pedestrian detection is received by message collection mechanism Kafka；

S2:Start Spark clusters, create SparkContext；Pretreatment image collection, then obtains the picture stream of positive negative sample, Picture stream is generated into multiple elasticity distribution formula data set RDD using pedestrian detection frame, independent calculating is carried out to each RDD；

S3：Data set in RDD is determined whether correspondence detection block is pedestrian by Linear SVM detection algorithm, if row People, the position of detection block is stored in new RDD；

S4：T weak SVM classifier of AdaBoost algorithm iterations circulation, strong point of the weak SVM classifier synthesis that each training is obtained Class device；

S5：All RDD for being detected as pedestrian are merged into a new RDD, will be the detection of same pedestrian in all detection blocks Frame merges；

S6：The position of the detection block after merging is drawn on picture, and is preserved to distributed file storage system HDFS.

2. pedestrian detection method according to claim 1, it is characterised in that：Using Kafka message collection mechanism, Hang Renjian Positive and negative sample set is surveyed by Kafka message acceptance mechanisms, then by the RDD receiving treatment of Sparkcontext.

3. pedestrian detection method according to claim 1, it is characterised in that：Described step S2 is specially：

(1)：The running environment of Spark Application is built, starts SparkContext；

(2)：SparkContext runs Executor resources to explorer application, and starts StandaloneExecutorbackend, Executor apply for Task to SparkContext；

(3)：Application program is distributed to Executor by SparkContext；

(4)：SparkContext is built into DAG figures, DAG figures is resolved into Stage, Taskset is sent into Task Scheduler, is finally sent to Executor and runs by Task Scheduler by Task；

(5)：Task runs on Executor, has run all resources of release.

4. pedestrian detection method according to claim 1, it is characterised in that：The step S2 also includes：HOG feature extractions Method, specially：

1) gray processing, regards image as a 3-D view of (x, y, z) gray scale；

2) standardization or normalization of color space are carried out to input picture using Gamma correction methods；Purpose is regulation image Contrast, reduces the influence caused by the shade and illumination variation of image local, while the interference of noise can be suppressed；

3) gradient of each pixel of image, including size and Orientation are calculated；Primarily to capture profile information, while further The interference that weakened light shines；

4) small cells is divided an image into；

5) histogram of gradients of each cell is counted, you can form the sub- descriptor of iamge description of each cell；

6) block will be constituted per several cell, the feature descriptor of all cell is together in series just in a block Obtain the HOG features descriptor of the block；

7) the HOG features descriptor of all block in image image is together in series and can be obtained by the HOG of target Feature descriptor, this is exactly the final characteristic vector for using that is available for classifying.

5. pedestrian detection method according to claim 4, it is characterised in that：The step S3 is specially：

(1) distributed elastic data set RDD objects are created；

(2) the intervention computing of DAG scheduling DAGScheduler modules, calculates the dependence between RDD, and the dependence between RDD is closed System is formed DAG；

(3) each Job is divided into multiple working stage Stage；A Main Basiss for dividing Stage are currently to calculate the factor Input whether be to determine, if it is by its point in same Stage, it is to avoid the message transmission between multiple Stage is opened Pin.

6. pedestrian detection method according to claim 4, it is characterised in that：The step S4 is specially：

(1) weight for initializing all training examples is 1/N, and N is input picture number；

(2) loop iteration m=1 ... M：

A) training Weak Classifier y_m(), minimizes it error function

Wherein, Xn sample sets, Wn initial weights, N sample numbers, ε_mIt is error；t_nRepresent true Real classification, m iterationses, the quantity of n sample sets；

B) the right of speech α of the Weak Classifier is calculated:

${\mathrm{\α}}_m=\ln \left\{\frac{1-{\mathrm{\ϵ}}_m}{{\mathrm{\ϵ}}_m}\right\}$

C weight) is updated:

Wherein, Z_mIt is standardizing factor, w_m+1, i is the m+1 times iteration Weight；t_iRepresent i-th of classification type；

Wherein Z_m:

Standardizing factor, make it is all w's and be 1

3) last grader is obtained：

$Y_M\left(x\right)=sign\left(\underset{m=1}{\overset{M}{\Σ}}{a}_m{y}_m\right(x\left)\right).$