CN104156973A - Real-time three-dimensional video monitoring method based on stereo matching - Google Patents

Abstract

A real-time three-dimensional video monitoring method based on stereo matching comprises the following steps that firstly, a plurality of points in a scene image are utilized for carrying out self-calibration on two vidicons to obtain respective projection models; secondly, the two vidicons are respectively utilized for obtaining static scene images to obtain respective basic frames; thirdly, current frames are compared with the basic frames, and monitoring objects of the two current frames output by the two vidicons are respectively obtained; fourthly, center points of the monitoring objects of the two current frames are respectively calculated; fifthly, the center points of the monitoring objects of the two current frames are reversely projected through the projection models of the corresponding vidicons to obtain two three-dimensional straight lines; sixthly, the three-dimensional positions of the monitoring object are calculated according to the two three-dimensional straight lines; the third step, the fourth step, the fifth step and the sixth step are repeatedly carried out, and the scene can be monitored in real time. The method is good in real-time performance, can be applied to environments not facilitating monitoring and can solve the problem of shielding in the object monitoring.

Description

Real-time three-dimensional video frequency monitoring method based on Stereo matching

Technical field

The present invention relates to video frequency monitoring method, specifically a kind of real-time three-dimensional video frequency monitoring method based on Stereo matching.

Background technology

In recent years, video monitoring is all played the part of more and more important work in daily life and special occasions.Traditional video monitoring object is to detect the two-dimensional bodies in video, does not consider spatial relationship between object or the spatial relationship of object and specific region.In processing protected field (object) and surrounding environment, on the relative position relation of moving object, there are severely restricts like this, in the face of occlusion issue is also difficult to process.In addition, existing video monitoring system is carried out inspected object by the difference of frame in video or the difference of I frame and P frame, and the real-time of video monitoring is poor.

Summary of the invention

The object of this invention is to provide a kind of real-time three-dimensional video frequency monitoring method based on Stereo matching.

Concrete technical scheme of the present invention is:

A real-time three-dimensional video frequency monitoring method based on Stereo matching, it comprises the following steps:

Two camera self-calibrations are obtained to projection model separately at some o'clock in S1, use scene image;

S2, with described two video cameras, obtain static scene picture construction basic frame separately respectively;

S3, by present frame and basic frame are compared, obtain respectively the monitored object in two present frames of two video cameras output;

S4, calculate respectively the central point of two monitored object in present frame;

S5, by the central point of the monitored object in two present frames, the projection model back projection by corresponding video camera obtains two 3 d-lines respectively;

S6, by described two 3 d-lines, calculate the three-dimensional position of monitored object;

Repeating step S3-S6, realizes the real-time monitoring to scene.

In the above-mentioned real-time three-dimensional video frequency monitoring method based on Stereo matching, in order to overcome the impact on object detection such as the factor such as illumination, white noise, find out fast the monitored object in present frame, preferably, the basic frame of each video camera comprises some frame static scene images, and described some frame static scene images are taken respectively under different illumination conditions; Correspondingly, described step S3 comprises: by present frame respectively with described some frame static scene image ratios pair, from comparison result, select the image block of least residual, filter, then therefrom select maximum image block as the monitored object in present frame.

In the above-mentioned real-time three-dimensional video frequency monitoring method based on Stereo matching, preferably, described step S4 comprises: the monitored object in present frame is carried out to rim detection, obtain subject area; Calculate the mean value of the coordinate of all pixels in described subject area, as the coordinate of monitored object central point in present frame.

In the above-mentioned real-time three-dimensional video frequency monitoring method based on Stereo matching, preferably, this method for supervising also comprises the three-dimensional alarm range in customization scene, by the three-dimensional position of the monitored object calculating and this three-dimensional alarm range comparison, produces alerting signal while falling into wherein.

In the above-mentioned real-time three-dimensional video frequency monitoring method based on Stereo matching, the problem that cannot monitor while blocking in order to solve, this method for supervising further comprises movement locus prediction steps, this movement locus prediction steps comprises: when there is circumstance of occlusion, predict the central point of monitored object in present frame according to known point with simple motion model.

In the real-time three-dimensional video frequency monitoring method based on Stereo matching more of the present invention, described two video cameras are not to the distance of scene etc.

In the real-time three-dimensional video frequency monitoring method based on Stereo matching more of the present invention, the camera lens of described two video cameras does not at least collect the same characteristic features point of monitored object in some times.

In the real-time three-dimensional video frequency monitoring method based on Stereo matching more of the present invention, to the distance of scene not etc., and the camera lens of described two video cameras does not at least collect the same characteristic features point of monitored object to described two video cameras in some times.

The present invention has the following advantages:

1. real-time is good, and the video synthesis processing power at a plurality of visual angles is reached to 30 frames per second, meets the requirement of real-time of human eye observation.

2. can be applied to following unfavorable monitoring environment: a, two video cameras to the distance of scene not etc., one far away one near, for video camera far away, on image, small pixel error can cause site error larger in space; The camera lens of b, described two video cameras does not at least collect the same characteristic features point of monitored object in some times, for example two camera lens and object are almost at same straight line, because can not find identical unique point, so utilize the method for binocular unique point space coupling unworkable; C, only know the positional information of video camera, for outer parameters such as the intrinsic parameters such as focal length of camera and the anglecs of rotation, do not know.

3. can solve the occlusion issue in object monitoring.

Accompanying drawing explanation

Fig. 1 is the process flow diagram of the real-time three-dimensional video frequency monitoring method of an embodiment based on Stereo matching;

Fig. 2 is the mapping relations of spatial point and face.

Embodiment

Below in conjunction with drawings and Examples, the present invention is further described.These more detailed descriptions are intended to help to understand the present invention, and should not be used to limit the present invention.According to content disclosed by the invention, it will be understood by those skilled in the art that and can not need some or all these specific detail can implement the present invention.And in other cases, for fear of by innovation and creation desalination, do not describe well-known operating process in detail.

As shown in Figure 1, this real-time three-dimensional video frequency monitoring method based on Stereo matching comprises the following steps:

Two camera self-calibrations are obtained to projection model separately at some o'clock in step S1, use scene image.

Known point set: $\left|\stackrel{\→}{A_1{A}_2}\right|=$ $\left|\stackrel{\→}{A_3{A}_4}\right|=\left|\stackrel{\→}{A_5{A}_6}\right|=\·\·\·=a_1,$ $\left|\stackrel{\→}{A_1{A}_3}\right|=\left|\stackrel{\→}{A_2{A}_4}\right|=\·\·\·=a_2$ $\left|\stackrel{\→}{A_3{A}_5}\right|=\left|\stackrel{\→}{A_4{A}_6}\right|=\·\·\·=a_3.$ A ₁, a ₂, a ₃it is known point.

The coordinate of all points is known.In video camera imaging plane, parallel lines intersect at vanishing point.The world coordinates of vanishing point and photocentre can be used and some A _irelative coordinate represent, also mean the world coordinates O (o that has known photocentre simultaneously _x, o _y, o _z).In theory, a rectangle of knowing 4 point coordinate just can be obtained the coordinate of photocentre.

Also know the world coordinates P (p of camera lens simultaneously _x, p _y, p _z).So be exactly the normal vector of imaging plane, as shown in Figure 2.Thereby have:

$f=\left|\stackrel{\→}{\mathrm{PO}}\right|*\frac{\left|\stackrel{\→}{O^{\′}{A}^{\′}}\right|}{\left|\stackrel{\→}{\mathrm{OA}}\right|}---\left(1\right)$

Wherein, A, O ∈ xoy, A, O " ∈ parallglplane, ⊥ imaging plane, ⊥ parallel plane.From formula (1), can calculate focal distance f.If somewhat abundant some A _i, i ∈ 1,2 ... and coordinate, f _iby A _icalculate, so known imaging plane equation: ax+by+cz+d=0.

Next can obtain the expression formula of space vector of the coordinate axis of imaging plane, be write respectively as S _x(x, y, z) and S _y(x, y, z).According to space geometry relation, space any point M _ican be expressed as:

$M_i^{\′}=\stackrel{\→}{{\mathrm{PA}}_i}*\sqrt{\left|\stackrel{\→}{\mathrm{PO}}\right|}*f/(\stackrel{\→}{\mathrm{PO}}*\stackrel{\→}{{\mathrm{PM}}_i})+P---\left(2\right)$

M ' wherein _ia M _imapping at imaging plane.Therefore obtain mapping point S ' _x, S ' _y.So imaging plane any point A ' _ican be expressed as:

$\stackrel{\→}{O^{\′}{A}_i^{\′}}=x*\stackrel{\→}{O^{\′}{U}_x}+y*\stackrel{\→}{O^{\′}}{U}_y---\left(3\right)$

Wherein O ' is intersection point with imaging plane. with it is vector of unit length.

According to pixel and actual range relation, spatial point A ' on imaging plane _i(x _i, y _i, z _i) coordinate be:

$A_i^{\′}=x*\stackrel{\→}{O^{\′}{U}_x}+y*\stackrel{\→}{O^{\′}{U}_y}*+O^{\′},\left\{\begin{array}{c}x=(x_p-O_{\mathrm{px}}^{\′})*{\mathrm{DPI}}_x\\ y=(y_p-O_{\mathrm{py}}^{\′})*{\mathrm{DPI}}_y\end{array}\right.---\left(4\right)$

Wherein, DPI _i, i ∈ { x, the distance of a pixel on y} denotation coordination axle, O ' _pi, i ∈ { x, the pixel coordinate that y} is O '.Impact point A _obj(x _o, y _o, z _o) extended line on:

$\frac{x_o-p_x}{x_i-p_x}=\frac{y_o-p_y}{y_i-p_y}=\frac{z_o-p_z}{z_i-p_z}=t,t>1---\left(5\right)$

In certain embodiments, the self-calibration of video camera also can adopt document H.Wildenauer, A.Hanbury, " Robust camera self-calibration from monocular images of Manhattan worlds ", Computer Vision and Pattern Recognition (CVPR), IEEE Conference on, pp.2831-2838, the self-calibrating method in 2012..

Step S2, with described two video cameras, obtain static scene picture construction basic frame separately respectively.Here said static state refers to does not have moving object in scene.

Further, in order to overcome the impact on object detection such as the factor such as illumination, white noise, find out fast the monitored object in present frame.The basic frame of each video camera preferably includes some frame static scene images, and described some frame static scene images are taken respectively under different illumination conditions.

Step S3, by present frame and basic frame are compared, obtain respectively the monitored object in two present frames of two video cameras output.It is target detection.

When the basic frame of each video camera comprises some frame static scene images, this target detection step comprises: by present frame respectively with described some frame static scene image ratios pair, from comparison result, select the image block of least residual, filter, then therefrom select maximum image block as the monitored object in present frame.Filtration is wherein for filtering noise and undesired signal, because most of time, has some Noise and Interference signals, as illumination in the difference of present frame and basic frame.More particularly, can preset thresholding Thrd _noisefilter, be defined as picture plane difference Diff _ij, i ∈ [1, width], j ∈ [1, height]:

${\mathrm{Diff}}_{\mathrm{ij}}=\left\{\begin{array}{c}{\mathrm{Diff}}_{\mathrm{ij}}-{\mathrm{Thrd}}_{\mathrm{noise}},\mathrm{if}{\mathrm{Diff}}_{\mathrm{ij}}>{\mathrm{Thrd}}_{\mathrm{noise}}\\ 0,\mathrm{others}\end{array}\right.---\left(6\right)$

Difference compared to the difference with consecutive frame in video or I frame and P frame is distinguished object, by the difference of present frame and basic frame, comes inspected object speed faster, can meet the requirement of real-time.

Step S4, calculate respectively the central point of two monitored object in present frame.

The method of computing center's point comprises: the monitored object in present frame is carried out to a rim detection, for example, adopt canny operator to carry out rim detection, obtain subject area Area _obj; Then calculate described subject area Area _objthe mean value of the coordinate of interior all pixels, as the coordinate of monitored object central point in present frame, its central point is defined as:

P _center＝ΣP _ij/n _obj (7)

P wherein _ij∈ Area _obj, n _objfor meeting P _ij∈ Area _objthe quantity of point.

Step S5, by the central point of the monitored object in two present frames, the projection model back projection by corresponding video camera obtains two 3 d-lines respectively.; the projection model back projection that the central point of the monitored object in first video camera present frame is passed through to the first video camera obtains a 3 d-line; similarly, by the central point of the monitored object in second video camera present frame, the projection model back projection by the second video camera obtains another 3 d-line.

Step S6, by described two 3 d-lines, calculate the three-dimensional position of monitored object.

Two lines in space can be located a point.In above-mentioned steps S5, two video cameras provide respectively a space line equation, establish it and are with meet equation (5), and meet at a bit, this intersection point is the three-dimensional position of monitored object.But due to the existence of error, they also may be non-intersect.

Known and arbitrfary point A _l1with on arbitrfary point A _l2, on projector distance d be exactly with between distance:

$\stackrel{\→}{d}=\stackrel{\→}{A_{l1}{A}_{l2}}*\stackrel{\→}{D}/\left|\stackrel{\→}{D}\right|---\left(8\right)$

And with to meet at 1 A _obj.If excessive, can think Stereo matching mistake, so can define like this three-dimensional position of monitored object:

$A_{\mathrm{obj}}=\left\{\begin{array}{c}\mathrm{null},\left|\stackrel{\→}{d}\right|>d_{\mathrm{trh}}\\ (\stackrel{\→}{l1}-\stackrel{\→}{d}/2)\∩(\stackrel{\→}{l2}+\stackrel{\→}{d}/2),\mathrm{others}\end{array}\right.---\left(9\right)$

Error rate can be defined as:

Rate _err＝n _err/n _all (10)

N wherein _errfor working as A _objframe number during for null, n _allthe frame number all occurring at two video cameras for monitored object.

Repeating step S3-S6, can realize the real-time monitoring to scene.

Further, method for supervising of the present invention can also comprise the three-dimensional alarm range in customization scene, by the three-dimensional position of the monitored object calculating and this three-dimensional alarm range comparison, produces alerting signal while falling into wherein.This three-dimensional alarm range can freely be defined by user.

In addition, method for supervising of the present invention can also comprise movement locus prediction steps, this movement locus prediction steps comprises: when there is circumstance of occlusion (for example, when monitored object disappears and exists in present frame at another video camera in the present frame of a video camera), predict the central point of monitored object in present frame according to known point (central point of monitored object) with simple motion model (such as straight line, circumference, para-curve etc.).Its principle is that regular motion is regarded in the motion within a small time period as by object, meets rectilinear motion, parabolic motion or circular motion, and therefore according to known point, carrying out matching just can obtain its prediction locus.

Experiment one: to known 4 point measurement volume coordinates in scene, then with actual coordinate comparison, as shown in table 1 with said method.

The calculated value of volume coordinate of table 1 and the comparison of actual value (unit: centimetre)

By table 1, can be seen, method for supervising error of the present invention is 10 centimetres of left and right, and this satisfies the demands in most of scenes.

Experiment two: tested the video of four sections of object different motion, result is as shown in table 2.

Owing to there is no depth finding instrument, can not accurately obtain information a little the accuracy of verification method, we are with the Rate in formula (10) _erras a kind of tolerance, consider the impact of article diameters on error simultaneously.If in formula (8) be greater than 2 times of object, think and a little do not mate (mistake in corresponding table 2), between 2 times to 1 times, can accept reluctantly (qualifying in corresponding table 2); Between 0.5 times to 1 times, good matching result (in corresponding table 2 good) at last; Be less than 0.5 times, can regard as very good (in corresponding table 2 outstanding).

Table 2 recognition result (unit: %)

Sequence	1	2	3	4	Average
						Excellent rate	10.5	47.4	33.3	42.3	33.3
Rate of good	57.9	26.3	41.7	42.3	42.1
						Pass rate	26.3	21.1	16.7	11.5	18.9
Error rate	5.3	5.2	8.3	3.9	5.7

Table 2 shows the be positioned with higher accuracy rate of the inventive method to point, meets the requirement of supervisory system.

Claims (7)

1. the real-time three-dimensional video frequency monitoring method based on Stereo matching, is characterized in that, the method comprises the following steps:

Two camera self-calibrations are obtained to projection model separately at some o'clock in S1, use scene image;

S2, with described two video cameras, obtain static scene picture construction basic frame separately respectively;

S3, by present frame and basic frame are compared, obtain respectively the monitored object in two present frames of two video cameras output;

S4, calculate respectively the central point of two monitored object in present frame;

S5, by the central point of the monitored object in two present frames, the projection model back projection by corresponding video camera obtains two 3 d-lines respectively;

S6, by described two 3 d-lines, calculate the three-dimensional position of monitored object;

Repeating step S3-S6, realizes the real-time monitoring to scene.

2. the real-time three-dimensional video frequency monitoring method based on Stereo matching according to claim 1, is characterized in that, the basic frame of each video camera comprises some frame static scene images, and described some frame static scene images are taken respectively under different illumination conditions; Correspondingly, described step S3 comprises: by present frame respectively with described some frame static scene image ratios pair, from comparison result, select the image block of least residual, filter, then therefrom select maximum image block as the monitored object in present frame.

3. the real-time three-dimensional video frequency monitoring method based on Stereo matching according to claim 1, is characterized in that, described step S4 comprises: the monitored object in present frame is carried out to rim detection, obtain subject area; Calculate the mean value of the coordinate of all pixels in described subject area, as the coordinate of monitored object central point in present frame.

4. the real-time three-dimensional video frequency monitoring method based on Stereo matching according to claim 1, it is characterized in that, this method for supervising also comprises the three-dimensional alarm range in customization scene, by the three-dimensional position of the monitored object calculating and this three-dimensional alarm range comparison, while falling into wherein, produce alerting signal.

5. the real-time three-dimensional video frequency monitoring method based on Stereo matching according to claim 1, it is characterized in that, this method for supervising also comprises movement locus prediction steps, this movement locus prediction steps comprises: when there is circumstance of occlusion, predict the central point of monitored object in present frame according to known point with simple motion model.

6. the real-time three-dimensional video frequency monitoring method based on Stereo matching according to claim 1, is characterized in that, described two video cameras are not to the distance of scene etc.

7. the real-time three-dimensional video frequency monitoring method based on Stereo matching according to claim 1, is characterized in that, the camera lens of described two video cameras does not at least collect the same characteristic features point of monitored object in some times.