CN104616304A - Self-adapting support weight stereo matching method based on field programmable gate array (FPGA) - Google Patents

Abstract

The invention provides a self-adapting support weight stereo matching method based on a field programmable gate array (FPGA). The method includes building a local stereo matching window for a left image and a right image in the FPGA, then calculating a gray level similarity function value and a Manhattan distance similarity function value according to the gray level similarity and the Manhattan distance similarity between local window inner matching points and local window center points to further acquire the weight price relation value of the local window matching points; finally calculating the price paradigmatic relation of each matching point, and utilizing the rule that the winner takes it all to acquire a parallax result of each pixel point. The method improves the integral matching method, can acquire the real-time dense parallax result and has high robustness.

Description

A kind of self-adaptation based on FPGA supports the solid matching method of weight

Technical field

The invention belongs to binocular stereo vision field, be specifically related to the solid matching method that a kind of self-adaptation based on FPGA supports weight.

Background technology

Binocular stereo vision is the physiological structure of direct modeling mankind binocular vision, it is the important technical of extraction of depth information under three-dimensional scenic, have in robot navigation, unmanned plane location and three-dimensional measurement and use widely, in stereoscopic vision, most critical and difficult point are the Stereo matching of binocular image, owing to needing the repetitive operation carrying out mass data in matching process, therefore on CPU, carry out Stereo matching to binocular image to need to spend a large amount of operation time, be difficult to the requirement meeting real-time.Such as, be the CPU of 1GHZ for dominant frequency, region Stereo Matching Algorithm is utilized to carry out dense disparity map calculating to the picture of two medium-sized sizes, need the time spending several seconds, the result of such low rate significantly limit the development of stereoscopic vision, more particularly needs the practice occasion obtaining real-time matching parallax result.Therefore, under the condition ensureing matching precision, how to improve the real-time of Stereo Matching Algorithm in recent years, become the focus of people's research.

Region Stereo Matching Algorithm comprises global area Stereo Matching Algorithm and regional area Stereo Matching Algorithm, global area Stereo Matching Algorithm matching accuracy is high, but calculation of complex, operand are large, GPU can be adopted realize in raising real-time, but in GPU processing procedure, power consumption is very large, the therefore application occasion of limiter reality.Corresponding regional area Stereo Matching Algorithm, the method is the parallax value being obtained each point to be matched by the similarity relationships of the local window on left images, calculated amount is little, and if a suitable Matching power flow relation can be found in matching process, also a good matching result can be obtained, and FPGA is as the programmable logic gate array of one, dirigibility is strong, and what utilize that the parallel pipelining process line technology of its inside can be real-time realizes regional area Stereo Matching Algorithm.

At present, FPGA is utilized to realize the real-time volume matching process (referring to document A Parallel reconfigurable architecture for real-time stereo vision) realized based on FPGA of the people such as SAD solid matching method (referring to document Hardware implementation of an SAD basedstereo vision algorithm), the Chen L proposition based on FPGA realization that real-time regional area solid matching method has the people such as Ambrosch K to propose.These class methods can be real-time acquire real-time dense parallax result, but the local window in matching process is fixed value, and do not distribute suitable weight information in the matching process to the match point in window, therefore at degree of depth point of discontinuity place, low texture place and scene repeat region place easily mate and make mistakes, matching accuracy rate is not high.

Summary of the invention

The deficiency that object of the present invention exists in matching accuracy rate and real-time for existing solid matching method, propose the solid matching method that a kind of self-adaptation based on FPGA supports weight, the method can solve degree of depth point of discontinuity place, low texture place and scene repeat region place and easily mate the problem of makeing mistakes, thus overall matching effect can be improved, simultaneously, the method utilizes the parallel pipelining process line technology of FPGA inside, the regional area Stereo matching walked abreast can be carried out in FPGA, obtain real-time dense parallax result, there is stronger robustness.

In order to solve the problems of the technologies described above, the invention provides the solid matching method that a kind of self-adaptation based on FPGA supports weight, left images is set up to the sectional perspective match window of m × n size in FPGA, the input data of video input mouth are pretreated left images, the clock of pixel is the system synchronization clock that each pixel of left images enters FPGA inside, with first in first out buffer, row cache is carried out to match point, with the d type flip flop of FPGA inside, row buffer memory is carried out to match point; Then according to match point in local window and the grey similarity between local window central point and manhatton distance similarity, ask for grey similarity functional value and the manhatton distance similarity function value of point to be matched, thus obtain the weight cost relation value w (p of local window match point, q), weight cost relation value w (p, q) such as formula shown in (1)

w(p,q)＝wd _k·wR _l(1)

In formula (1), wd _krepresent the respective value of the manhatton distance similarity function value of pixel p and q obtained by look-up table in FPGA inside, R _lrepresent that the gray-scale value of pixel p and q is in the grey similarity relation after rank conversion, wR _lrepresent the respective value of the grey similarity functional value of pixel p and q obtained by look-up table in FPGA inside; Finally according to the Matching power flow weight relationship value of described local matching window, calculate the cost paradigmatic relation of each match point, the criterion that then utilizes that the victor is a king, ask for the parallax result of each pixel; The cost paradigmatic relation of described each match point shown in (2),

$E(p,\stackrel{\‾}{p_d})=\frac{{\mathrm{\Σ}}_{q\∈N_P,\stackrel{\‾}{q_d}\∈N_{\stackrel{\‾}{\mathrm{Pd}}}}w(p,q)w(\stackrel{\‾}{p_d},\stackrel{\‾}{q_d})e_m(q,\stackrel{\‾}{q_d})}{{\mathrm{\Σ}}_{q\∈N_P,\stackrel{\‾}{q_d}\∈N_{\stackrel{\‾}{\mathrm{Pd}}}}w(p,q)w(\stackrel{\‾}{p_d},\stackrel{\‾}{q_d})}---\left(2\right)$

In formula (2), N _prepresent the local matching window of left image, represent the local matching window of right image, w (p, q) is the weight relationship value of match point q in left image, for the match point in right image weight relationship value, for the local window of left images, p point and point is corresponding, q point with point is corresponding, for the fiducial value of match point after rank conversion, expression formula is such as formula shown in (3):

$e_m(q,\stackrel{\‾}{q_d})=\left\{\begin{array}{cc}0& R_{\mathrm{pq}}=R_{\stackrel{\‾}{p_d}\stackrel{\‾}{q_d}}\\ 1& \mathrm{otherwise}\end{array}\right.---\left(3\right)$

In formula (3), R _pqrepresent the rank transformed value of q point, represent the rank transformed value of point.

Preferably, Gaussian function is selected to carry out pre-service to video image, filtering noise.

Compared with prior art, its remarkable advantage is in the present invention, and (1) the present invention carries out regional area Stereo matching to binocular image in hardware FPGA, thus can obtain real-time dense parallax result; (2) the present invention utilizes the grey similarity in local window between match point and window center point, manhatton distance similarity sets up adaptive weighting relation, thus can improve the accuracy rate in matching process; (3) the present invention is in the matching process of local window, substitutes original grey similarity functional relationships set occurrence by the value after rank conversion, thus can set up a fixing look-up table in FPGA inside, and the real-time being conducive to FPGA calculates; (4) the present invention is in cost polymerization process, match point is first carried out parallax value calculating again after rank conversion, thus elimination illumination and noise is on the impact of left images matching result.

Accompanying drawing explanation

Fig. 1 is the inventive method process flow diagram.

Fig. 2 is the present invention sets up m × n size in FPGA regional area Stereo matching window to left images.

Fig. 3 is the present invention carries out rank conversion in FPGA process schematic to pixel.

Fig. 4 is the similarity function look-up table that the present invention sets up in FPGA, and wherein (a) is manhatton distance similarity search table, and (b) is grey similarity look-up table.

Fig. 5 is that the present invention carries out cost aminated polyepichlorohydrin schematic diagram to corresponding match point in the local window of left images, and wherein (a) is the schematic diagram of left image window, and (b) is the schematic diagram of right image window.

Fig. 6 is that cost polymerization process of the present invention realizes schematic diagram in FPGA.

Embodiment

Composition graphs 1, the self-adaptation that the present invention is based on FPGA supports the solid matching method of weight, and step is as follows:

Step 1: utilize the left images that the left and right cameras acquisition polar curve demarcated is corrected, the left images collected is carried out filtering process and remove noise, pretreated left image is I _l(i, j) and right image are I _r(i, j);

The present invention selects Gaussian function to carry out filtering to image, thus the effective white Gaussian noise eliminated sensor and introduce.Filtering Template adopt (2k+1) × (2k+1) dimension (wherein k=1,2,3 ...) and discrete Gaussian convolution core, account form is as shown in formula (1):

$\left\{\begin{array}{c}{I}_l(i,j)=I_{r0}(i,j)*G(u,v)=\underset{u=-k}{\overset{k}{\mathrm{\Σ}}}\underset{v=-k}{\overset{k}{\mathrm{\Σ}}}{I}_{r0}(i+u,j+v)\·G(u,v)\\ I_r(i,j)=I_{r0}(i,j)*G(u,v)=\underset{u=-k}{\overset{k}{\mathrm{\Σ}}}\underset{v=-k}{\overset{k}{\mathrm{\Σ}}}{I}_{r0}(i+u,j+v)\·G(u,v)\\ G(u,v)=\frac{1}{{2\mathrm{\π\σ}}^2}{e}^{-\frac{u^2+v^2}{{2\mathrm{\σ}}^2}}\end{array}\right.---\left(1\right)$

In formula (1), (i, j) is the coordinate figure of image slices vegetarian refreshments, I _l0(i, j) is the original left image of input, I _r0(i, j) is the original right image of input, and (u, v) is discrete Gauss point coordinate, and G (u, v) is for discrete gaussian kernel function is at the normalized value at (u, v) place, and σ is Gaussian function scale-value.

Step 2: composition graphs 2, left images is set up to the sectional perspective match window of m × n size in FPGA, the input data of video input mouth are pretreated left images, the clock of pixel is the system synchronization clock that each pixel of left images enters FPGA inside, FIFO is first in first out buffer, row cache is carried out to match point, row buffer memory carries out buffer memory by the d type flip flop of FPGA inside, win is the local window set up, a parallel local window can be obtained thus, for the regional area Stereo matching of left images in FPGA.

Step 3: according to match point in local window and the grey similarity between local window central point and manhatton distance similarity, ask for grey similarity functional value f (the Δ c of point to be matched _pq) and manhatton distance similarity function value f (Δ d _pq), thus obtain weight cost relation value w (p, q) of local window match point, shown in (2):

w(p,q)＝f(Δc _pq)·f(Δd _pq) (2)

In formula (2), p and q is respectively the pixel in the central pixel point of images match window and window area, Δ c _pqrepresent the grey similarity relation of p and q point, Δ d _pqrepresent the manhatton distance similarity relationships of p and q point.Wherein Δ c _pq, Δ d _pqrelation such as formula shown in (3), f (Δ c _pq), f (Δ d _pq) function expression relation is such as formula shown in (4).

$\left\{\begin{array}{c}{\mathrm{\Δc}}_{\mathrm{pq}}=I_p-I_q\\ {\mathrm{\Δd}}_{\mathrm{pq}}=|x_p-x_q|+{|y}_p-y_q|\end{array}\right.---\left(3\right)$

$\left\{\begin{array}{c}f\left({\mathrm{\Δc}}_{\mathrm{pq}}\right)=\exp (-\frac{{\mathrm{\Δc}}_{\mathrm{pq}}}{{\mathrm{\τ}}_c})\\ f\left({\mathrm{\Δd}}_{\mathrm{pq}}\right)=\exp (-\frac{{\mathrm{\Δd}}_{\mathrm{pq}}}{{\mathrm{\τ}}_d})\end{array}\right.---\left(4\right)$

In formula (3), I _prepresent the gray-scale value of p point, I _qrepresent the gray-scale value of q point, (x _p, y _p) represent the ranks coordinate figure of p point, (x _q, y _q) representing the ranks coordinate figure of q point, the middle exp of formula (4) represents exponential function, τ _crepresent the weight proportion constant under color similarity function, τ _drepresent the weight proportion constant under manhatton distance similarity function.

Further, in the present invention, in order to make, grey similarity function is convenient to be calculated in FPGA, first carries out rank conversion, then utilize new grey similarity relation R to the gray-scale value of match point _pqreplace original Δ c _pq, the expression formula of rank conversion is as follows:

$R_{\mathrm{pq}}=\left\{\begin{array}{cc}-2& I_p-I_q<-{\mathrm{\&tau;}}_1\\ -1& -{\mathrm{\&tau;}}_1\&le;I_p-I_q\&le;-{\mathrm{\&tau;}}_2\\ 0& -{\mathrm{\&tau;}}_2<I_p-I_q\&le;{\mathrm{\&tau;}}_2\\ 1& {\mathrm{\&tau;}}_2<I_p-I_q\&le;{\mathrm{\&tau;}}_1\\ 2& I_p-I_q>{\mathrm{\&tau;}}_1\end{array}\right.---\left(5\right)$

τ in formula (5) ₁, τ ₂for the class condition of rank conversion, as fixed constant in computation process, I _prepresent the gray-scale value of p point, I _qrepresent the gray-scale value of q point.

Therefore according to the local window set up in formula (5) and Fig. 2, composition graphs 3, the gray-scale value of the subtracter of FPGA inside to match point in window and window center point is utilized to subtract each other and judge, thus determine that in FPGA inside in window, each match point rank converts grade, obtain because point each in local window is parallel, therefore the rank pixel after conversion is also occur so that the form of local window is parallel, thus does not affect the real-time of algorithm.

Composition graphs 4, according to new grey similarity functional value f (R _pq) and manhatton distance similarity function value f (Δ d _pq) in FPGA, set up look-up table, thus determine Matching power flow relation weighted value w (p, q) of each pixel, weight w (p, the q) expression formula of the local window match point therefore calculated in FPGA is such as formula shown in (6):

w(p,q)＝wd _k·wR _l(6)

In Fig. 4, Δ d _krepresent the manhatton distance of p and q point, wd _krepresent f (the Δ d obtained by look-up table in FPGA inside _pq) respective value, R _lrepresent the grey similarity relation of p and q point after rank conversion, wR _lrepresent the f (R obtained by look-up table in FPGA inside _pq) respective value.

Due to after above-mentioned steps conversion, for the regional area match window that size is certain, grey similarity functional value after its manhatton distance similarity function value and rank convert is all the discrete transform values in certain limit, therefore the foundation of look-up table can be mapped data by finite state machine in FPGA, the wd after wherein mapping _k, wR _ldata are all integer, and these data after normalization, can't affect the conversion of its match point weight, are so conducive to the calculating carrying out fixed-point integer in FPGA.

Step 4: composition graphs 5, according to the Matching power flow weight relationship value of local matching window, calculates the cost paradigmatic relation of each match point shown in (7), then utilize the criterion that the victor is a king (WTA), ask for the parallax result d of each pixel _q, shown in (9).

$E(p,\stackrel{\&OverBar;}{p_d})=\frac{{\mathrm{\&Sigma;}}_{q\&Element;N_P,\stackrel{\&OverBar;}{q_d}\&Element;N_{\stackrel{\&OverBar;}{\mathrm{Pd}}}}w(p,q)w(\stackrel{\&OverBar;}{p_d},\stackrel{\&OverBar;}{q_d})e_m(q,\stackrel{\&OverBar;}{q_d})}{{\mathrm{\&Sigma;}}_{q\&Element;N_P,\stackrel{\&OverBar;}{q_d}\&Element;N_{\stackrel{\&OverBar;}{\mathrm{Pd}}}}w(p,q)w(\stackrel{\&OverBar;}{p_d},\stackrel{\&OverBar;}{q_d})}---\left(7\right)$

In formula (7), N _prepresent the local matching window of left image, represent the local matching window of right image, w (p, q) is the weight relationship value of match point q in left image, for the match point in right image weight relationship value, for the local window of left images, p point and point is corresponding, q point with point is corresponding, for the fiducial value after match point rank conversion, expression formula is such as formula shown in (8):

$e_m(q,\stackrel{\&OverBar;}{q_d})=\left\{\begin{array}{cc}0& R_{\mathrm{pq}}=R_{\stackrel{\&OverBar;}{p_d}\stackrel{\&OverBar;}{q_d}}\\ 1& \mathrm{otherwise}\end{array}\right.---\left(8\right)$

In formula (8), R _pqrepresent the rank transformed value of q point, represent the rank transformed value of point.

$d_q=\arg \underset{d\&Element;S_d}{\min }E(p,\stackrel{\&OverBar;}{p_d})---\left(9\right)$

In formula (9), S _d={ d _min, d _min+ 1 ..., d _max, d _minfor minimum parallax value, d _maxfor maximum disparity value.

Composition graphs 6, in left images, partial weight window and local pixel window is set up according to FPGA, computing is carried out according to formula (7), inner at FPGA, in order to reduce the use of multiplier resources, the weight relationship value of left images is carried out in the process be multiplied, and utilizes the operation of displacement to replace the function (the n power form splitting into multiple 2 by data is sued for peace) of multiplier, the w in Fig. 6 _lrepresent the weight w (p, q) of left image, w _rrepresent the weight in right image the weight that w is the left images calculated with shifting function is amassed, e in Fig. 6 _mvalue determined by formula (8), its value can only be ' 1 ' or ' 0 ', therefore formula (7) cost polymerization process in, in FPGA, only need e _mlowest order and weighted value w in everybody carry out logical and (&) and operate, therefore according to formula (7), cost paradigmatic relation value can be obtained to after the normalization of weight relationship value

According to formula (9), what walk abreast to each matched pixel point in FPGA calculates (d _max-d _min+ 1) individual Matching power flow polymerizing value, and utilize the comparer of FPGA inside to calculate cost paradigmatic relation to be worth side-play amount corresponding to minimum pixel, to be the parallax value of this point.

Therefore, the present invention circulates above-mentioned steps 1 ~ step 6, can carry out the calculating of dense disparity map to binocular video stream.

In order to the present invention further by Simulation experiments validate beneficial effect of the present invention, emulation experiment is that the CYCLONE III EP3C120F780C8 fpga chip that provides in altera corp realizes, the size of the local matching window adopted is 11 × 11, the frame frequency of the binocular video stream collected is 60fps, image size resolution is 640 × 480, table 1 is the form of the hardware resource use that the inventive method utilizes quartus2 composing software to generate, table 2 is that the real-time performance of the inventive method and existing Stereo Matching Algorithm contrasts, frame frequency presses 60MHZ clock normalization.Experiment shows that the disparity range of the method search that the inventive method proposes than Chen is larger, and frame frequency is also higher, can obtain real-time, matching accuracy rate is a higher dense parallax result.

Table 1

Table 2

Claims (5)

1. the solid matching method of the support of the self-adaptation based on a FPGA weight, it is characterized in that, left images is set up to the sectional perspective match window of m × n size in FPGA, the input data of video input mouth are pretreated left images, the clock of pixel is the system synchronization clock that each pixel of left images enters FPGA inside, with first in first out buffer, row cache is carried out to match point, with the d type flip flop of FPGA inside, row buffer memory is carried out to match point.

2. as claimed in claim 1 based on the solid matching method of the self-adaptation support weight of FPGA, it is characterized in that, according to match point in local window and the grey similarity between local window central point and manhatton distance similarity, ask for grey similarity functional value and the manhatton distance similarity function value of point to be matched, thus obtain weight cost relation value w (p, q) of local window match point, weight cost relation value w (p, q) such as formula shown in (1)

w(p,q)＝wd _k·wR _l(1)

In formula (1), wd _krepresent the respective value of the manhatton distance similarity function value of pixel p and q obtained by look-up table in FPGA inside, R _lrepresent that the gray-scale value of pixel p and q is in the grey similarity relation after rank conversion, wR _lrepresent the respective value of the grey similarity functional value of pixel p and q obtained by look-up table in FPGA inside.

3. as claimed in claim 2 based on the solid matching method of the self-adaptation support weight of FPGA, it is characterized in that, according to the Matching power flow weight relationship value of described local matching window, calculate the cost paradigmatic relation of each match point, then utilize the victor is a king criterion, ask for the parallax result of each pixel.

4., as claimed in claim 3 based on the solid matching method of the self-adaptation support weight of FPGA, it is characterized in that,

The cost paradigmatic relation of described each match point shown in (2),

$E(p,\stackrel{\&OverBar;}{p_d})=\frac{{\mathrm{\&Sigma;}}_{q\&Element;N_P,\stackrel{\&OverBar;}{q_d}\&Element;N_{\stackrel{\&OverBar;}{P_d}}}w(p,q)w(\stackrel{\&OverBar;}{p_d},\stackrel{\&OverBar;}{q_d})e_m(q,\stackrel{\&OverBar;}{q_d})}{{\mathrm{\&Sigma;}}_{q\&Element;N_P,\stackrel{\&OverBar;}{q_d}\&Element;N_{\stackrel{\&OverBar;}{P_d}}}w(p,q)w(\stackrel{\&OverBar;}{p_d},\stackrel{\&OverBar;}{q_d})}---\left(2\right)$

In formula (2), N _prepresent the local matching window of left image, represent the local matching window of right image, w (p, q) is the weight relationship value of match point q in left image, for the match point in right image weight relationship value, for the local window of left images, p point and point is corresponding, q point with point is corresponding, for the fiducial value of match point after rank conversion, expression formula is such as formula shown in (3):

$e_m(q,\stackrel{\&OverBar;}{q_d})=\left\{\begin{array}{cc}0& R_{\mathrm{pq}}=R_{\stackrel{\&OverBar;}{p_d}\stackrel{\&OverBar;}{q_d}}\\ 1& \mathrm{otherwise}\end{array}\right.---\left(3\right)$

In formula (3), R _pqrepresent the rank transformed value of q point, represent the rank transformed value of point.

5., as claimed in claim 1 based on the solid matching method of the self-adaptation support weight of FPGA, it is characterized in that, select Gaussian function to carry out pre-service to video image, filtering noise.