CN103559237B - Semi-automatic image annotation sample generating method based on target tracking - Google Patents

Abstract

The invention discloses a semi-automatic image annotation sample generating method based on target tracking. The method comprises processes of target tracking and semi-automatic annotation. A serial of samples are generated through a target tracking mechanism, a template learning mechanism is designed for tracking and detecting target areas, detection on videos or images is performed by means of learned templates, manual annotation is utilized to help to perform determination, and therefore, annotation samples are generated. The semi-automatic image annotation sample generating method based on the target tracking has the advantages of being capable of obtaining a large amount of image annotation samples by means of less labor consumption.

Description

Semi-automatic image labeling sample generating method based on target following

Technical field

The present invention relates to a kind of semi-automatic image labeling sample generating method based on target following, belong to image procossing skill Art field.

Background technology

The target of image labeling is the corresponding relation set up between image-region and mark keyword.Image labeling passes through to build Mapping relations between vertical low-level visual feature and high-level semantic, can solve present in image retrieval to a certain extent " semantic gap " problem.Image labeling can be divided into manual mark and automatic marking two class.Carry out image using artificial mode Mark is to be also the most directly most effective way, but this is also the work taking time and effort very much.With Internet and The development of Digital image technology, view data magnanimity increases, and traditional artificial mask method every time can only be in piece image Object area is labeled, and is labeled more and more time-consuming effort using manually.Therefore increasing scholar's research is passed through Carry out automatic image annotation using machine learning method, be used statistical learning method to be also required to substantial amounts of mark sample conduct Training set, however, the collection of labeled data that presently, there are is relatively fewer.Therefore, the present invention proposes one kind and is based on target following Semi-automatic image labeling sample generating method, thus to obtain more image labeling samples by using less manpower consumption This.

Content of the invention

Goal of the invention：Object area in piece image can only be labeled every time for traditional artificial mask method Defect, the invention provides a kind of semi-automatic image labeling sample generating method based on target following, thus by less Human intervention obtain more marked image pattern.

Technical scheme：A kind of semi-automatic image labeling sample generating method based on target following, this method comprises two Process, its step is as follows：

Object tracking process：

(11) artificial mark area-of-interest in initial frame, as region to be tracked；

(12) according to tab area, generate initially positive negative sample；

(13) according to positive negative sample, generate original template；

(14) most like region is searched in the next frame according to template；

(15) according to tracking result more new template；

(16) return to step (14), iteration is followed the tracks of.

Annotation process：

(21) using study to template frame of video is detected；

(22) candidate region detecting is tracked；

(23) tracking sequence is classified or manually marked, retained positive sample sequence, removed error tracking sequence；

(24) positive sample sequence and its template are preserved；

(25) template being obtained using tracking, return to step (21) is iterated；

(26) manual confirmation is carried out to all image sequences producing of following the tracks of.

Beneficial effect：Compared with prior art, the semi-automatic image labeling sample based on target following that the present invention provides Generation method, carries out, from motion tracking, generating sample sequence by the area-of-interest in artificial selection video, and combines semi-automatic Label technology, is obtained by less manpower consumption and substantial amounts of has marked image pattern.

Brief description

Fig. 1 is the offset window schematic diagram in the embodiment of the present invention, and wherein, solid-line rectangle is positive sample, and dashed rectangle is Negative sample；

Fig. 2 is the sample extension convolution operator schematic diagram in the embodiment of the present invention；

Fig. 3 is the tracking sample sequence schematic diagram in the embodiment of the present invention, wherein, the big positive sample of front half section (bicycle), Second half section is negative sample；

Fig. 4 is the overall flow schematic diagram in the embodiment of the present invention.

Specific embodiment

With reference to specific embodiment, it is further elucidated with the present invention it should be understood that these embodiments are merely to illustrate the present invention Rather than restriction the scope of the present invention, after having read the present invention, the various equivalences to the present invention for the those skilled in the art The modification of form all falls within the application claims limited range.

Based on the semi-automatic image labeling sample generating method of target following, detailed process is as follows：

Object tracking process：

(1) give the video that resolution ratio is N × N, gradient is calculated to video initial frame, and manually marks out the rectangle of H × W Object area, the centre coordinate in rectangle object region is (m₀,n₀), as initial positive sample x₀, H and W is image to be tracked respectively The height of window and width.

(2) negative sample is selected according to initial positive sample, rectangle object region is entered with line displacement, generate negative sampleIts Middle Δ i ∈ [- H/2,0) ∪ (0, H/2], Δ j ∈ [- W/2,0) ∪ (0, W/2] represent the skew of abscissa and ordinate, such as Fig. 1 Shown.Define α_ΔiΔjFor negative sampleWeight, δ refers to Gaussian function standard deviation, to define used here as Gaussian function α_ΔiΔj：

(3) according to initial positive sample and negative sample, generate template w

The complexity directly calculating above-mentioned template is very high, by introducing h

The wherein schematic diagram of h is as shown in Figure 2.DefinitionIt is with (m₀,n₀) centered on 2H × 2W rectangular area, wherein comprise Region corresponding to all positive negative samples, by using h pairCarry out convolution, can very easily try to achieve templateBy using FFT and IFFT, computation complexity can be reduced, first with FFT by h andIt is transformed into frequency domain, Then dot product, finally recycles IFFT to be transformed into time domain and obtains template w.In the case of hardware supported, can also be added using GPU Speed.

(4) gradient is calculated to frame of video t, using template w_t-1Area the most close in all H × W area in search present frame Domain, defining distance function is

Directly calculation template and the complexity of all H × W rectangular areas distance in frame t are O (NNHW), in order to accelerate Calculating speed, can be split as follows：

WhereinIt is constant for present frame；The unit matrix of a H × W can be passed through to current Frame carries out convolution and obtains, and two-dimentional unit matrix is split, obtains one-dimensional vector, can carry out quick convolution；Convolution can be carried out by using template and present frame to obtain.Can be multiple by calculating by using FFT and IFFT Miscellaneous degree is greatly reduced, and first with FFT, w and present frame is transformed into frequency domain, then dot product, finally when being transformed into using IFFT Domain obtains the distance of each pixel in present frame.In order to improve efficiency further, can only calculate with (m_t-1,n_t-1) centered on Distance in H × W area.In the case of hardware supported, can also be accelerated using GPU further.

(5) pass through calculated distance and obtain positive negative sampleWhereinRepresent tracking result,Expression removesThe minimum target area of outer distance, utilizesBy equation below come more new template w_t

Wherein β ∈ (0,1) represents learning rate.

WillAdd positive sample data set, t=t+1, repeat step (4) (5) is iterated following the tracks of, until (m_t,n_t) surpass Go out image range, or artificial stopping is followed the tracks of.

Sample annotation process：

(1) using study to template w video is detected, obtain some candidate regions

(2) it is directed to each candidate region

A) generate trace template

B) distance of each region and template in frame t is calculated with reference to step (4)；

C) generate positive negative sample with reference to step (5)And more new template

D) at any time, execute following operation：

I. artificial judgment：Remove the tracking sample of mistake by artificial mark, retain correct sample；

Ii. automatic decision：Calculate sample sequenceWith the distance of template, remove apart from unstable sample This

If e) sample is retained, continues to follow the tracks of, t=t+1, repeat (b) (c) (d), untilBeyond image model Enclose, or artificial stopping is followed the tracks of；

(3) one group of template { w is obtained according to tracking result_i, i=1 ..., K ' }, K ' is the positive sample sequence number retaining Amount, obtains one group of positive sample sequence simultaneously

(4) from template list W=w ∪ { w_i, i=1 ..., K ' } and one template of middle random selection, return to step (), Repeat, until artificial stop；

(5) manual confirmation is carried out to the positive sample sequence getting, as shown in figure 3, removing false positive sample further.

Proved by test of many times, compared with prior art, its remarkable advantage is the method for the present invention：

(1) this method is directly using Pixel-level feature, rather than extracts the features such as histogram, it is to avoid the time of feature extraction disappears Consumption, can describe more judgement information in image-region simultaneously, such as shape, space structure etc..Gradient used herein As feature, but mechanism described herein such as can directly use gray scale using being applied to arbitrary pixel characteristic Or RGB triple channel color characteristic etc..

(2) for each positive sample, this method produces negative sample using skew, so that the calculating robustness of template is more By force, meanwhile, this method passes through to design convolution operator h, such as shown in step (3), using h come to extending positive negative sample, can be quick Generate a series of negative samples, its computational efficiency is higher.

(3) this method is decomposed using distance, such as shown in step (4), Euclidean distance higher for computation complexity is calculated and turns Turn to two convolutional calculation, such that it is able to be accelerated using FFT and IFFT, in the case of hardware supported, can also use GPU accelerates, and calculating speed and efficiency are higher.

(4) this method devises a kind of online template renewal mechanism, such as shown in step (5), is generated by every secondary tracking Positive negative sample, can directly update trace template, carry out weight without using all positive negative sample producing of following the tracks of before New training, its training is more easy and efficiency is higher.

(5) pass through once to mark, be tracked obtaining substantial amounts of image pattern, usual video is per second to comprise 30 frames, Therefore follow the tracks of and can produce within 30 seconds 900 samples, compared to existing technology, by using little manpower consumption, can obtain big The mark sample of amount.

For step (five), because image pattern is to be obtained by tracking, there is continuity, as shown in figure 3, therefore, only need Each sequence is divided into two sections, leading portion is positive sample, back segment is negative sample, and each sequence only needs once manually to mark, because This, taking of artificial mark is little.

Claims (4)

1. a kind of semi-automatic image labeling sample generating method based on target following it is characterised in that：Comprise two processes, its Step is as follows：

Object tracking process：

(11) artificial mark area-of-interest in initial frame, as region to be tracked；

(12) according to tab area, generate initially positive negative sample；

(13) according to positive negative sample, generate original template；

(14) most like region is searched in the next frame according to template；

(15) according to tracking result more new template；

(16) return to step (14), iteration is followed the tracks of；

Annotation process：

(21) using study to template frame of video is detected；

(22) candidate region detecting is tracked；

(23) tracking sequence is classified or manually marked, retained positive sample sequence, removed error tracking sequence；

(24) positive sample sequence and its template are preserved；

(25) template being obtained using tracking, return to step (21) is iterated；

(26) manual confirmation is carried out to all image sequences producing of following the tracks of；

In step (11), artificial mark area-of-interest in initial frame, as region to be tracked, given resolution ratio is N × N's Video, calculates gradient to video initial frame, and manually marks out the rectangle object region of H × W, and centre coordinate is (m₀,n₀), make For initial positive sample x₀；

In step (12), according to initial positive sample x₀Select negative sample, rectangle object region is entered with line displacement, generate negative sampleWherein Δ i ∈ [- H/2,0) ∪ (0, H/2], Δ j ∈ [- W/2,0) ∪ (0, W/2] represent the inclined of abscissa and ordinate Move；Define α_ΔiΔjFor negative sampleWeight, to define α used here as Gaussian function_ΔiΔj：

${\mathrm{\α}}_{\mathrm{\Δ}i\mathrm{\Δ}j}=g(\mathrm{\Δ}i,\mathrm{\Δ}j)\∝\exp (-(\frac{{\mathrm{\Δi}}^2}{2{\mathrm{\δ}}_i^2}+\frac{{\mathrm{\Δj}}^2}{2{\mathrm{\δ}}_j^2}))---\left(1\right)$

In step (13), according to initial positive sample and negative sample, generate trace template w

$w=\underset{\mathrm{\Δ}i=-H/2}{\overset{H/2}{\Σ}}\underset{\mathrm{\Δ}j=-W/2}{\overset{W/2}{\Σ}}{\mathrm{\α}}_{\mathrm{\Δ}i\mathrm{\Δ}j}(x-x_{\mathrm{\Δ}i\mathrm{\Δ}j})---\left(2\right)$

The complexity directly calculating above-mentioned template is very high, by introducing h

$h_{\mathrm{\Δ}i\mathrm{\Δ}j}=\left\{\begin{array}{cc}\underset{\mathrm{\Δ}i=-H/2}{\overset{H/2}{\Σ}}\underset{\mathrm{\Δ}j=-W/2}{\overset{W/2}{\Σ}}{\mathrm{\α}}_{\mathrm{\Δ}i\mathrm{\Δ}j}-{\mathrm{\α}}_{00}& if\mathrm{\Δ}i=0,\mathrm{\Δ}j=0\\ -{\mathrm{\α}}_{\mathrm{\Δ}i\mathrm{\Δ}j}& otherwise\end{array}\right.---\left(3\right)$

DefinitionIt is with (m₀,n₀) centered on 2H × 2W rectangular area, wherein contain the region corresponding to all positive negative samples, lead to Cross and utilize h pairCarry out convolution, try to achieve trace templateUsing FFT by h andIt is transformed into frequency domain, then dot product, Recycle IFFT to be transformed into time domain afterwards and obtain template w；In the case of hardware supported, accelerated using GPU.

2. the semi-automatic image labeling sample generating method based on target following as claimed in claim 1 it is characterised in that：

In step (14), gradient is calculated to frame of video t, using template w_t-1The most close in all H × W area in search present frame Region, define distance function be

$d(w,y)=\underset{i=1}{\overset{H}{\Σ}}\underset{j=1}{\overset{W}{\Σ}}{(w_{ij}-y_{ij})}^2---\left(4\right)$

Replace directly calculating Euclidean distance using the method splitting convolution：

$d(w,y)=\underset{i=1}{\overset{H}{\Σ}}\underset{j=1}{\overset{W}{\Σ}}(w_{ij}^2+y_{ij}^2-2w_{ij}{y}_{ij})=\underset{i=1}{\overset{H}{\Σ}}\underset{j=1}{\overset{W}{\Σ}}{w}_{ij}^2+\underset{i=1}{\overset{H}{\Σ}}\underset{j=1}{\overset{W}{\Σ}}{y}_{ij}^2-2\underset{i=1}{\overset{H}{\Σ}}\underset{j=1}{\overset{W}{\Σ}}{w}_{ij}{y}_{ij}---\left(5\right)$

WhereinIt is constant for present frame；Convolution is carried out to present frame by the unit matrix of a H × W Obtain, two-dimentional unit matrix is split, obtains one-dimensional vector, quick convolution can be carried out；By profit Carry out convolution with template and present frame to obtain；First with FFT, w and present frame are transformed into frequency domain, then dot product, finally in profit It is transformed into the distance that time domain obtains each pixel in present frame with IFFT；In order to improve efficiency further, only calculate with (m_t-1, n_t-1) centered on H × W area in distance；In the case of hardware supported, accelerated using GPU further.

3. the semi-automatic image labeling sample generating method based on target following as claimed in claim 2 it is characterised in that：Step Suddenly, in (15), positive negative sample is obtained by calculated distanceWhereinRepresent tracking result,Expression removesOutward The minimum target area of distance, utilizesBy equation below come more new template w_t

$w_t=(1-\mathrm{\β})w_{t-1}+\mathrm{\β}h\⊗x_t^{+}-\mathrm{\β}h\⊗x_t^{-}---\left(6\right)$

Wherein β ∈ (0,1) represents learning rate；

WillAdd positive sample data set, t=t+1, repeat step (14) (15) is iterated following the tracks of, until (m_t,n_t) beyond figure As scope, or artificial stopping is followed the tracks of.

4. the semi-automatic image labeling sample generating method based on target following as claimed in claim 3 it is characterised in that：Mark Note process,

(1) using study to template w video is detected, obtain some candidate regions

(2) it is directed to each candidate region

A) generate trace template

B) distance of each region and template in frame t is calculated with reference to step (14)；

C) generate positive negative sample with reference to step (15)And more new template

D) at any time, execute following operation：

I. artificial judgment：Remove the tracking sample of mistake by artificial mark, retain correct sample；

Ii. automatic decision：Calculate sample sequenceWith the distance of template, remove apart from unstable sample；

If e) sample is retained, continues to follow the tracks of, t=t+1, repeat (b) (c) (d), untilBeyond image range, or Artificial stopping is followed the tracks of；

(3) one group of template { w is obtained according to tracking result_i, i=1 ..., K'}, K' are the positive sample sequence quantity retaining, simultaneously Obtain one group of positive sample sequence

(4) from template list W=w ∪ { w_i, randomly choose a template, return to step () in i=1 ..., K'}, repeat to hold OK, until artificial stop；

(5) manual confirmation is carried out to the positive sample sequence getting, remove false positive sample further.