CN103517041A - Real-time full-view monitoring method and device based on multi-camera rotating scanning - Google Patents

Abstract

The invention provides a real-time full-view monitoring method and device based on multi-camera rotating scanning. 360-degree panorama environment full-view presenting is achieved, and ceaseless-renewal panoramic pictures are achieved. The method comprises the steps that first, initialization is carried out; second, each camera device collects one frame of picture, pixel intensity compensation is carried out, the sampling result of each device is stored in the corresponding sampling picture sequence of the device, and meanwhile information of the angle where a rotating cloud table is placed during current sampling is recorded; an initial picture of each picture sequence is displayed, according to the relative mounting positions of cameras, the position on a 360-degree panoramic picture of the initial frames in each group is determined; third, a picture sequence is selected, a previous frame A and a current frame B are extracted, feature points in an ROI zone are detected and described, feature points extracted in the frame A and the frame B are matched, a non-maximum value in feature detecting is adjusted according to the matching result to restrain a threshold value; fourth, a homography matrix is computed according to the matching result.

Description

Real time panoramic method for supervising and device based on polyphaser rotation sweep

Technical field

The invention belongs to image processing field and technical field of video monitoring, be specifically related to the rotation sweep overall view monitoring system of polyphaser.

Background technology

Existing panoramic imagery equipment is mainly divided three classes, and the one, rely on special imaging device to obtain image with great visual angle as fish eye lens; Another kind of is to rely on the picture splicing that a plurality of camera is taken the photograph that covers different area of visual field to form large scene image; The 3rd class is that the image sequence that relies on a rotating camera to take surrounding environment splices and merged panoramic imagery.But special imaging device image as captured in fish eye lens has serious distortion (CN102222337A); The method that Equations of The Second Kind polyphaser scene merges wants to cover 360 degree scopes needs a lot of imaging devices, and cost is high, system complex (CN102117008A; CN101866482B); The slewing of last class one camera is subject to the restriction real-time of rotary speed and joining method poor, and upgrading 360 degree scenes once needs chronic (CN101221351B).The overall view monitoring tasks of 360 degree, not only need the visual field of complete display, also will consider the factors such as real-time, and system cost.

Summary of the invention

The present invention is intended to overcome above defect, has proposed a kind of real time panoramic method for supervising and device based on polyphaser rotation sweep, and the panorama that completes 360 degree panorama environment presents, and realizes and constantly update panoramic picture.Providing on the basis of 360 degree panoramic pictures, avoided distortion, required image collecting device quantity is few, without specific (special) requirements such as wide-angle, flakes, and system is simple, has reached high real-time, and coherent consistent through many imaging devices imaging of calibration, the stability of a system is high.

A real time panoramic method for supervising for polyphaser rotation sweep, comprises the following steps:

Step 1, according to turntable rotary speed and viewing field of camera angle size, determine the sampling interval; According to turntable rotary speed, direction of rotation and angle of visual field size and definite area-of-interest; The parametric calibration of a plurality of image capture devices, at synchronization to each imaging device sampling initial frame, according to imaging device installation site and horizontal alignment side-play amount, be presented on the corresponding initial position of 360 degree panoramic pictures the residing angle information of rotary head while simultaneously recording current sampling;

Step 2, every picture pick-up device gather respectively a two field picture, and carry out pixel intensity compensation, and the sampled result of every equipment is kept at respectively in sampled images sequence separately, the residing angle information of rotary head while simultaneously recording current sampling; The initial pictures that shows each image sequence, according to the relative position of installing between camera, determines in every group that initial frame is in 360 degree panoramic picture Shang positions;

Step 3, choose an image sequence, extract previous frame A and present frame B, detect and describe the characteristic point in ROI region, then the characteristic point of extracting in A, B two frames is mated, the non-maximum of adjusting in feature detection according to matching result suppresses threshold value;

Step 4, according to matching result, calculate homography matrix, according to the parameter identification in homography matrix, whether meet splicing condition; Meet splicing condition, according to homography matrix, obtain B frame with respect to the side-play amount of A frame, adopt the method that is fade-in gradually to go out to eliminate splicing gap, according to side-play amount, B frame is spliced to A frame and shown on panoramic picture; If do not meet splicing condition, receive current angle information, on this basis, frame B is presented to the relevant position of panoramic picture.

Whether step 5, detection have next frame, if had, go to step one, otherwise finish.

Parametric calibration described in step 1 comprises: in the rotation of every imaging device through the frame of sampling in 0 of The Cloud Terrace, pass through images match, carry out the parametric calibration of whole imaging devices, comprise vertical shift calibration, horizontal-shift calibration, luminance gain calibration.

The non-maximum inhibition threshold value of adjusting in step 3 in feature detection adopts the feature point detecting method in surf (Speeded-Up Robust Features) algorithm to carry out the extraction of characteristic point; Adopt again ORB (Comparative Evaluation of Binary Features) characteristics algorithm to be described the characteristic point detecting; According to the characteristic point in A, B two frames, carry out characteristic matching afterwards, and matching result is screened; Revise the dependent thresholds in feature detection, characteristic point quantity is remained on and meet the least possible numerical value of coupling demand.

A real time panoramic supervising device for polyphaser rotation sweep, comprising:

Input module, by one or more image capture devices, be fixed on turntable, imaging device is taken different angles, and relative position is fixed, often through building after a while, once sample, the sampled images of each imaging device forms respectively an image sequence, requires the consecutive frame in each sequence need to have enough overlapping regions, and the buffer storage of up-to-date adjacent two frames is provided;

Calibration module, calibrates parameter between different imaging devices, comprises vertical shift calibration, horizontal-shift calibration, luminance gain calibration;

Concatenation module, in each image sequence, present frame all carries out characteristic matching with previous frame, calculates side-play amount, eliminates splicing seams simultaneously, determines present frame display position;

Splicing failure response module, while linear mosaic failure being detected in image processing module, the relative position of the image capture device that while utilizing present frame to gather, the The Cloud Terrace position of response and this image sequence are corresponding on turntable, estimates the display position of present frame in panoramic picture;

Display module, provides the buffer storage of image to be displayed and positional information, according to the information that image is processed or splicing failure handling module provides, in panoramic picture, with present frame, upgrades relevant position, completes the renewal of panoramic picture.

Beneficial effect of the present invention:

Real time panoramic method for supervising and device based on polyphaser rotation sweep disclosed by the invention, adopt the method for a plurality of image capture device rotary grasping image sequences, obtain the information of panorama environment, than passing one camera rotary splicing equipment, increased exponentially the renewal rate of panoramic picture, and required imaging device quantity is few, general camera can meet the demands, and has therefore kept the hardware system expense of low requirement; Analysis On Multi-scale Features point detecting method based on Hessian determinant and the ORB feature method of looking familiar make the present invention's characteristic matching stitching algorithm as compared with the past have higher yardstick consistency and illumination invariant, make Image Mosaics more stable and accurate; Meanwhile, utilize the angle information of The Cloud Terrace passback, situation that can the failure of aid in treatment Image Mosaics, improves the reliability of system.

Accompanying drawing explanation

Fig. 1 is two camera lens rotation table devices of the embodiment of the present invention;

Fig. 2 is the flow chart of embodiment of the present invention Algorithm in Creating Panoramic Image;

Fig. 3 is the embodiment of the present invention 360 degree panorama splicing schematic diagrames.

Specific implementation method

Below in conjunction with the accompanying drawing in the embodiment of the present invention, the technical scheme in the embodiment of the present invention is removed, intactly described, described embodiment is only the present invention's part embodiment, rather than whole embodiment.Below by the embodiment being described with reference to the drawings, be exemplary, only for explaining the present invention, and can not be interpreted as limitation of the present invention.Embodiment based in the present invention, the every other embodiment that those of ordinary skills obtain, belongs to the scope of protection of the invention.

As shown in Figure 1, adopt N picture pick-up device (getting N=3 in the present embodiment) in the present embodiment, on turntable, interval 360 °/N degree is fixed, and viewing field of camera angle is v _fain the present embodiment, get v _fa=20 °, sampled images resolution is L _width* L _height; Consecutive frame determined minimum degree is set as α _min(0 < α _min< 1), α wherein _mifor the empirical value of this method generally gets 0.3, reference frame (left figure) ROI is transverse axis L _width* α pixel is to L _widthpixel region, present frame (right figure) ROI is that transverse axis 0 pixel is to L _width* (1-α) pixel region; In addition, the turntable in the present embodiment turns clockwise along overlooking.The width of panorama viewing area is W, is highly H.

The present invention a kind of overall view monitoring method based on polyphaser rotary taking, as shown in Fig. 2 is, specifically comprise the following steps:

Step 1, initialization turntable, starts clockwise uniform rotation from 0 of turntable, and rotating speed is w.Sampling initial frame is also presented on the initial position of 360 degree panoramic pictures the residing angle information of rotary head while simultaneously recording current sampling.

Wherein, the sampling interval need to be determined according to turntable rotary speed and viewing field of camera angle size.According to viewing field of camera angle v _fa, consecutive frame degree of overlapping α, rotating speed w, can calculating sampling interval t:

$t=\frac{v_{\mathrm{fa}}(1-\mathrm{\&alpha;})}{w}<t_{\max }=\frac{v_{\mathrm{fa}}(1-{\mathrm{\&alpha;}}_{\min })}{w}$

Because all imaging devices are fixed on turntable, therefore all keep identical angular velocity of rotation w, the sampling interval is t.

Calibration comprises the following steps:

1. choosing the picture frame that camera 1 gathers is frame A, appoint that to get another camera sample frame be frame B, detect respectively the characteristic point in two frames, then the characteristic point of extracting in A, B two frames is mated, the non-maximum of adjusting in feature detection according to matching result suppresses threshold value; Concrete, described step comprises: adopt the feature point detecting method in surf (Speeded-Up Robust Features) algorithm to carry out the extraction of characteristic point; Adopt again ORB (Comparative Evaluation of Binary Features) characteristics algorithm to be described the characteristic point detecting; According to the characteristic point in A, B two frames, carry out characteristic matching afterwards, and matching result is screened.Detailed description is shown in step 3.

2. according to matching result, calculate homography matrix, according to the parameter identification in homography matrix, whether meet splicing condition.If do not meet splicing condition, make the corresponding camera of frame B again capture a two field picture 0 of The Cloud Terrace, repeating step 3-1, until meet splicing condition.

3. horizontal-shift calibration value x _i'=h ₁₃, vertical shift calibration value y _i'=h ₂₃, statistics A, B Liang Zheng overlapping region pixel intensity and ${\mathrm{\&epsiv;}}_i=\frac{\underset{(x,y)\&Element;\mathrm{\&Omega;},(x^{\&prime;},y^{\&prime;})\&Element;{\mathrm{\&Omega;}}^{\&prime;}}{\mathrm{\&Sigma;}}[I_A(x,y)-I_B(x^{\&prime;},y^{\&prime;})]}{n_{\mathrm{\&Omega;}}},$ Wherein (x, y) and (x ', y ') is respectively the pixel coordinate of mutually aiming in frame A and frame B, and Ω and Ω ' be the overlapping region of correspondence in frame A and frame B, and i is corresponding camera numbers, ε ₁=0.Repeat 1 to 3, until all cameras all pass through calibration.

Step 2, image sequence initialization.Every picture pick-up device gathers respectively a two field picture, and carries out pixel intensity compensation, for all pixels in present frame, I (x, y)=I (x, y)+ε _i, i is the imaging device numbering that this frame is corresponding.The sampled result of every equipment is kept at respectively in sampled images sequence separately, the residing angle information of rotary head while simultaneously recording current sampling.

According to the relative position of installing between camera, determine in every group that initial frame is in 360 degree panoramic picture Shang positions.The original position that sequence 1 image shows in panoramic picture is (X ₁, Y ₁), the correspondence position of sequence 2 images is (X ₂, Y ₂), the correspondence position of sequence 3 images is (X ₃, Y ₃), X wherein ₁=0,

$X_3=X_1+W\&times;\frac{2}{3}+{\mathrm{\&epsiv;}}_3,Y_1=0,Y_2=Y_3=.$

Step 3, chooses an image sequence, extracts previous frame A and present frame B, detects and describes the characteristic point in ROI region, then the characteristic point of extracting in A, B two frames is mated, and according to matching result, adjusts the dependent thresholds in feature detection; Concrete, described step 3 comprises: adopt the feature point detecting method in surf algorithm to carry out the extraction of characteristic point; Adopt again ORB characteristics algorithm to be described the characteristic point detecting; According to the characteristic point in A, B two frames, carry out characteristic matching afterwards, and matching result is screened; Revise the dependent thresholds in feature detection, characteristic point quantity is remained on and meet the least possible numerical value of coupling demand, with the saving-algorithm time.

Choosing of image characteristic point will consider that the brightness that environment brings changes and because supervisory control system moves the dimensional variation of bringing, will note the time overhead of algorithm simultaneously, guarantees can complete whole image works for the treatment of in a sampling interval t.The Analysis On Multi-scale Features point detecting method of the Hessian determinant of feature point detection based in surf algorithm, concrete: model metric space, the approximation of the Hessian matrix determinant of each pixel in computed image.Hessian matrix is:

$H(x,\mathrm{\&sigma;})=\left[\begin{array}{cc}{L}_{\mathrm{xx}}(x,\mathrm{\&sigma;})& L_{\mathrm{xy}}(x,\mathrm{\&sigma;})\\ L_{\mathrm{xy}}(x,\mathrm{\&sigma;})& L_{\mathrm{yy}}(x,\mathrm{\&sigma;})\end{array}\right]$

Wherein σ yardstick, is L _xxthat gaussian filtering second order is led

with the result of input picture convolution, L _xyand L _yycompute classes seemingly,

surf algorithm is conveniently used box Filtering Template with the convolution D of input picture for what calculate _xx, D _xy, D _yyreplace L _xx, L _xy, L _yy.Hessian determinant can be expressed as:

det(H _approx)＝D _xxD _yy-(0.9D _xy) ²

Set subsequently a threshold value θ _fcarry out non-maximum and suppress detected characteristics point, then in metric space interpolation, obtain stable characteristic point position and scale-value.θ in the present embodiment _finitial value be made as 400, and carry out self adaptation adjustment with splicing effect, in order to balance detection effect and time overhead.Concrete,

${\mathrm{\&theta;}}_f=\left\{\begin{array}{cc}{\mathrm{\&theta;}}_f+{\mathrm{\&Delta;\&theta;}}_f& M_{\mathrm{good}}>M_{\max }\\ {\mathrm{\&theta;}}_f-{\mathrm{\&Delta;\&theta;}}_f& M_{\mathrm{good}}\&le;M_{\min }\end{array}\right.$

Wherein, M _goodfor the characteristic point after screening is to number, M _maxand M _minbe respectively the adjustment boundary of setting, in the present embodiment, M _max=40, M _min=15.

Obtaining after characteristic point is that feature is described, for characteristic matching ready.Concrete, adopt the character description method in ORB algorithm herein, the feature coding by random point to formation two-value, then add the description that rBRIEF feature is carried out characteristic direction.

The description that obtains characteristic point is used BF (Brute Force) algorithm to carry out characteristic matching afterwards, and matching result is carried out to Preliminary screening, rejects the characteristic point pair that distance between two points is greater than maximum spacing 50%.The characteristic point retaining is to as the data of calculating homography matrix.

Step 4, according to matching result, calculates homography matrix, according to the parameter identification in homography matrix, whether meets splicing condition.

Homography matrix H is that source images coordinate is to the transformation matrix between target image coordinate.

$H=\left[\begin{array}{ccc}{h}_{11}& h_{12}& h_{13}\\ h_{21}& h_{22}& h_{23}\\ h_{31}& h_{32}& h_{33}\end{array}\right]$

If (x ₁, y ₁) and (x ₂, y ₂) be respectively pixel coordinate in pixel coordinate in original image and target image,

$\left(\begin{array}{c}{x}_2\\ y_2\end{array}\right)=\left(\begin{array}{c}\frac{h_{11}{x}_1+h_{12}{y}_1+h_{13}}{h_{31}{x}_1+h_{32}{y}_1+h_{33}}\\ \frac{h_{21}{x}_1+h_{22}{y}_1+h_{23}}{h_{31}{x}_1+h_{32}{y}_1+h_{33}}\end{array}\right)$

All cameras in native system are all fixed on turntable, in same level, therefore when normal splicing, do not have rotation transformation and projective transformation, and in homography matrix H, element should meet: h ₁₁, h ₂₂be approximately 1, h ₁₂, h ₂₁, h ₃₁, h ₃₂be approximately 0.Therefore ξ=h imposes a condition ₁₂+ h ₂₁+ h ₃₁+ h ₃₂, as ξ < ξ _mintime, think and splice successfully, otherwise be considered as splicing unsuccessfully ξ _mindetermine based on experience value, in the present embodiment ξ _min=0.1.

Step 5, meets splicing condition, according to homography matrix, obtains B frame with respect to the side-play amount of A frame, adopts the method that is fade-in gradually to go out to eliminate splicing gap, according to side-play amount, B frame is spliced to A frame and is shown on panoramic picture; If do not meet splicing condition, receive current angle information, on this basis, frame B is presented to the relevant position of panoramic picture.

If in step 4, splicing result is for to splice successfully, h in homography matrix ₁₃, h ₂₃be respectively horizontal offset x _offsetwith vertical offset y _offset.Because sample effect between two frames may there are differences, now there is obvious vestige at the splicing seams place of two width pictures, adopts here and is fade-in gradually to go out method processing overlapping region elimination seam.Concrete grammar is as follows:

Make I _a(x, y), I _b(x, y) is the image intensity that the pixel (x, y) in frame A and frame B is located, w _seamfor the width k of overlapping region is that current pixel is apart from the pixel count of overlapping region left margin.?

$I_B(x,y)=(1-\frac{k}{w_{\mathrm{seam}}})\&times;I_A(x,y)+\frac{k}{w_{\mathrm{seam}}}\&times;I_B(x,y).$

If splicing result is unsuccessfully in step 4, read the turntable angle information u of present frame, for sequence 1, the horizontal offset of present frame is

horizontal offset for sequence 2,3 present frames is

$W\&times;\frac{(\frac{360}{N}(n-1)+u)\mathrm{mod}360}{360},$

Wherein N be sequence always, n is current sequence number, 0≤n≤N, N=3 in the present embodiment, vertical offset equals the vertical offset of sequence previous frame separately.Final side-play amount should be added the offset calibration value of corresponding camera, x _offset=x _offset+ x _i', y _offset=y _offset+ y _i'.

As shown in Figure 3, finally according to horizontal offset and vertical offset, the present frame in each image sequence is presented to panoramic picture Zhong relevant position.

Step 6, detects whether there is next frame, if had, goes to step two.

The invention discloses a kind of real time panoramic supervising device based on polyphaser rotation sweep, comprise with lower module:

Input module, is fixed on turntable by one or more image capture devices, and imaging device is taken different angles, and relative position is fixed.Often through building after a while, once sample, the sampled images of each imaging device forms respectively an image sequence, requires the consecutive frame in each sequence need to have enough overlapping regions, and the buffer storage of up-to-date adjacent two frames is provided.As the image capturing system that has adopted in the present embodiment two camera lenses to form, as shown in Figure 1.

Calibration module, calibrates parameter between different imaging devices, comprises vertical shift calibration, horizontal-shift calibration, luminance gain calibration.

Concatenation module, in each image sequence, present frame all carries out characteristic matching with previous frame, calculates side-play amount, eliminates splicing seams simultaneously, determines present frame display position.

Splicing failure response module, while linear mosaic failure being detected in image processing module, the relative position of the image capture device that when this module utilizes present frame to gather, the The Cloud Terrace position of response and this image sequence are corresponding on turntable, estimates the display position of present frame in panoramic picture.

Display module, provides the buffer storage of image to be displayed and positional information, according to the information that image is processed or splicing failure handling module provides, in panoramic picture, with present frame, upgrades relevant position, completes the renewal of panoramic picture.

The explanation of above embodiment is just for helping to understand method of the present invention and core concept thereof; Meanwhile, for one of ordinary skill in the art, according to thought of the present invention, in concrete execution mode and range of application, all will change.In sum, this description should not be construed as limitation of the present invention.

Claims (4)

1. the real time panoramic method for supervising based on polyphaser rotation sweep, is characterized in that, comprises the following steps:

Step 1, according to turntable rotary speed and viewing field of camera angle size, determine the sampling interval; According to turntable rotary speed, direction of rotation and angle of visual field size and definite area-of-interest; The parametric calibration of a plurality of image capture devices, at synchronization to each imaging device sampling initial frame, according to imaging device installation site and horizontal alignment side-play amount, be presented on the corresponding initial position of 360 degree panoramic pictures the residing angle information of rotary head while simultaneously recording current sampling;

Step 2, every picture pick-up device gather respectively a two field picture, and carry out pixel intensity compensation, and the sampled result of every equipment is kept at respectively in sampled images sequence separately, the residing angle information of rotary head while simultaneously recording current sampling; The initial pictures that shows each image sequence, according to the relative position of installing between camera, determines in every group that initial frame is in 360 degree panoramic picture Shang positions;

Step 3, choose an image sequence, extract previous frame A and present frame B, detect and describe the characteristic point in ROI region, then the characteristic point of extracting in A, B two frames is mated, the non-maximum of adjusting in feature detection according to matching result suppresses threshold value;

Step 4, according to matching result, calculate homography matrix, according to the parameter identification in homography matrix, whether meet splicing condition; Meet splicing condition, according to homography matrix, obtain B frame with respect to the side-play amount of A frame, adopt the method that is fade-in gradually to go out to eliminate splicing gap, according to side-play amount, B frame is spliced to A frame and shown on panoramic picture; If do not meet splicing condition, receive current angle information, on this basis, frame B is presented to the relevant position of panoramic picture.

Whether step 5, detection have next frame, if had, go to step one, otherwise finish.

2. a kind of real time panoramic method for supervising based on polyphaser rotation sweep as claimed in claim 1, it is characterized in that, parametric calibration described in step 1 comprises: in the rotation of every imaging device through the frame of sampling in 0 of The Cloud Terrace, pass through images match, carry out the parametric calibration of whole imaging devices, comprise vertical shift calibration, horizontal-shift calibration, luminance gain calibration.

3. a kind of real time panoramic method for supervising based on polyphaser rotation sweep as claimed in claim 1 or 2, it is characterized in that, the non-maximum inhibition threshold value of adjusting in step 3 in feature detection adopts the feature point detecting method in surf (Speeded-Up Robust Features) algorithm to carry out the extraction of characteristic point; Adopt again ORB (Comparative Evaluation of Binary Features) characteristics algorithm to be described the characteristic point detecting; According to the characteristic point in A, B two frames, carry out characteristic matching afterwards, and matching result is screened; Revise the dependent thresholds in feature detection, characteristic point quantity is remained on and meet the least possible numerical value of coupling demand.

4. the real time panoramic supervising device based on polyphaser rotation sweep, is characterized in that, comprising:

Input module, by one or more image capture devices, be fixed on turntable, imaging device is taken different angles, and relative position is fixed, often through building after a while, once sample, the sampled images of each imaging device forms respectively an image sequence, requires the consecutive frame in each sequence need to have enough overlapping regions, and the buffer storage of up-to-date adjacent two frames is provided;

Calibration module, calibrates parameter between different imaging devices, comprises vertical shift calibration, horizontal-shift calibration, luminance gain calibration;

Concatenation module, in each image sequence, present frame all carries out characteristic matching with previous frame, calculates side-play amount, eliminates splicing seams simultaneously, determines present frame display position;

Splicing failure response module, while linear mosaic failure being detected in image processing module, the relative position of the image capture device that while utilizing present frame to gather, the The Cloud Terrace position of response and this image sequence are corresponding on turntable, estimates the display position of present frame in panoramic picture;

Display module, provides the buffer storage of image to be displayed and positional information, according to the information that image is processed or splicing failure handling module provides, in panoramic picture, with present frame, upgrades relevant position, completes the renewal of panoramic picture.

Images