CN100556153C - A kind of preprocess method of multi-view image - Google Patents

Abstract

The invention discloses a kind of preprocess method of multi-view image, at metric space target image and source images are carried out the extreme value detection by yardstick invariant features mapping algorithm, extract the yardstick invariant features transform characteristics vector of target image and source images key point, utilize the characteristic matching algorithm to obtain the key point pair set of source images and the accurate coupling of target image, and source images is carried out color correction by calculating the right property the taken advantage of sum of errors additive errors of key point, by key point between set up affine transformation relationship, correcting image behind the color correction is carried out geometric calibration, advantage is that the present invention is under the prerequisite that guarantees multi-view image color correction and geometric calibration accuracy, the robustness of color correction and the precision of mapping have been improved greatly, reduce the computation complexity of geometric calibration, improved the coding efficiency of multi-view image.

Description

A kind of preprocess method of multi-view image

Technical field

The present invention relates to a kind of processing method of multi-view image, especially relate to a kind of preprocess method of multi-view image.

Background technology

In real world, the vision content that the observer saw depends on the position of observer with respect to observed object, and the observer can freely select each different angle to remove to observe and analyze things.In traditional video system, real scene is selected decision with respect to the picture of a viewpoint by cameraman or director, the sequence of video images that the user can only watch video camera to be produced on single viewpoint passively, and can not freely select other viewpoint to observe real scene.The video sequence that these folk prescriptions make progress can only reflect a side of real-world scene.The free viewpoint video system can make the user freely select viewpoint to go to watch any side in the certain limit in the real-world scene, by the MPEG of International Standards Organization (Moving Picture Experts Group: be called the developing direction of video system of future generation Motion Picture Experts Group).

The multi-viewpoint video image technology is a core link in the free viewpoint video technology, and it can provide the video image information of the different angles of captured scene.Fig. 1 is the parallel camera system imaging of a many viewpoints schematic diagram, and wherein n+1 camera (or video camera) placed abreast to take multi-viewpoint video image.Utilize the information of a plurality of viewpoints in the multi-view point video signal can synthesize the image information of user-selected any viewpoint, reach the purpose of freely switching any visual point image.But since in gatherer process the baseline of each camera not at same trunnion axis, key elements such as scene illumination, camera CCD noise, shutter speed and exposure are inconsistent, can cause the color value difference of the image that different cameral gathers very big, and in vertical direction the skew of geometric position can take place, bring great difficulty for follow-up video coding, virtual viewpoint rendering and multi-view point video three-dimensional display.

Problem at above-mentioned existence, the method that a kind of typical multi-view image is handled has been proposed at present, as shown in Figure 2, the multi-view image that a plurality of cameras are collected, carry out pretreatment operation earlier, comprise color correction and geometric calibration, then pretreated image is encoded again, between the viewpoint of decoding, draw virtual visual point image at last.

It is the color correction means that adopt usually that zone coupling or histogram mate, and the zone coupling concerns in the most similar interregional color map of setting up, and with these mapping relations source images proofreaied and correct by target image and source images are carried out cluster segmentation; The histogram coupling has identical histogram distribution as long as satisfy source images with target image by calculating the accumulative histogram of target image and source images, just the histogram of target image can be mapped to source images.If but variations such as multi-view image rotates, yardstick convergent-divergent, view transformation, illumination conversion, or because block, the influence of factor such as noise, but zone coupling and histogram coupling just can not keep good matching, it is low to shine upon precision.

Geometric calibration need obtain the inner parameter of camera and the conversion that external parameter is realized coordinate system usually, thereby reaches the purpose of geometric position unanimity, but the method computation complexity that adopts this geometric calibration is than higher.

Summary of the invention

Technical problem to be solved by this invention provides and a kind ofly can guarantee effectively that the color of each visual point image is consistent with the geometric position, improves the multi-view image color correction and the geometric calibration method of the coding efficiency of multi-view image simultaneously.

The present invention solves the problems of the technologies described above the technical scheme that is adopted: a kind of preprocess method of multi-view image, and it may further comprise the steps:

(1) visual point image in the multi-view image that synchronization is taken by many view camera system is defined as target image, is designated as T, and other visual point image is defined as source images, is designated as S, the target image set of keypoints of from target image, extracting be designated as ^(T)P}, the source images set of keypoints of from source images, extracting be designated as ^(S)P}, the plane coordinates of objective definition image are x ' y ' coordinate system, and the plane coordinates of definition source images is the xy coordinate system;

(2) at metric space target image and source images are carried out the extreme value detection by yardstick invariant features mapping algorithm, extraction target image set of keypoints ^(T)P} and source images set of keypoints ^(S)The yardstick invariant features transform characteristics vector of each key point among the P};

(3) according to yardstick invariant features transform characteristics vector, obtain the source images key point by the characteristic matching algorithm ^(S)P (x ₁, y ₁) optimal candidate coupling crux point on target image ^(T)P (x ₁', y ₁'), and on source images, obtain the target image key point ^(T)P (x ₁', y ₁') optimal candidate coupling key point ^(S)P (x ₂, y ₂), then by two-way verification determine key point to ( ^(T)P (x ₁', y ₁', ^(S)P (x ₁, y ₁)) for the coupling key point right;

(4) target image and source image data are converted into the CIEXYZ color space from the RGB color space, be converted into the CIELAB color space from the CIEXYZ color space again, the 1st component of three components of CIELAB color space is that luminance component is designated as L, the 2nd component is that first color component is designated as a and the 3rd component is that second color component is designated as b;

(5) to the key point of target image and all couplings of source images to the key point pair set that constitutes ( ^(T)P (x ₁', y ₁'), ^(S)P (x ₁, y ₁)), by the absolute difference sum of L, a and each component of b is carried out minimization, calculate $(a_i,e_i)=\arg \underset{a_i,e_i}{\min }\underset{(x_1,y_1),({x_1}^{\&prime;},{{\mathrm{<figure-callout\; id="1"\; label="y"\; filenames="C20071016449800151.png"\; state="\{\{state\}\}">y</figure-callout>}}_1}^{\&prime;})\&Element;\mathrm{\&Omega;}}{\mathrm{\&Sigma;}}\mathrm{abs}({}^{\left(T\right)}I_i({{\mathrm{<figure-callout\; id="1"\; label="x"\; filenames="C20071016449800151.png"\; state="\{\{state\}\}">x</figure-callout>}}_1}^{\&prime;},{{\mathrm{<figure-callout\; id="1"\; label="y"\; filenames="C20071016449800151.png"\; state="\{\{state\}\}">y</figure-callout>}}_1}^{\&prime;})-(a_i\&CenterDot;{}^{\left(S\right)}I_i({\mathrm{<figure-callout\; id="1"\; label="x"\; filenames="C20071016449800151.png"\; state="\{\{state\}\}">x</figure-callout>}}_1,y_1)+e_i))$ Obtain the property the taken advantage of error a of L, a and each component of b _iWith additive errors e _i, wherein, Ω is the key point pair set, ^(S)I _i(x ₁, y ₁) be the color value of i component of source images, ^(T)I _i(x ₁', y ₁') be the color value of i component of target image, i=1,2,3;

(6) with the property the taken advantage of error a of L, a and each component of b _iWith additive errors e _iL, a and each component of b to each pixel of source images carry out color correction, ^(C)I _i(x ₁, y ₁)=a _i ^(S)I _i(x ₁, y ₁)+e _i, wherein, ^(C)I _i(x ₁, y ₁) be the color value of i the component of correcting image behind the color correction, i=1,2,3, and correcting image is transformed into the RGB color space;

(7) affine transformation of setting up from the pixel of correcting image to the pixel of target image is $X^{\&prime;}=\left(\begin{array}{c}{x_1}^{\&prime;}\\ {{\mathrm{<figure-callout\; id="1"\; label="y"\; filenames="C20071016449800151.png"\; state="\{\{state\}\}">y</figure-callout>}}_1}^{\&prime;}\end{array}\right)=\left(\begin{array}{cc}{a}_{11}& a_{12}\\ a_{21}& a_{22}\end{array}\right)\left(\begin{array}{c}{x}_1\\ {\mathrm{<figure-callout\; id="1"\; label="y"\; filenames="C20071016449800151.png"\; state="\{\{state\}\}">y</figure-callout>}}_1\end{array}\right)+\left(\begin{array}{c}{b}_1\\ b_2\end{array}\right)=\mathrm{AX}+B,$ The key point pair set ( ^(T)P (x ₁', y ₁'), ^(S)P (x ₁, y ₁)) in select the right position of the key point of three pairs of Euclidean distance minimums as initial value, calculate spin matrix A and translation vector B;

(8) with spin matrix A and translation vector B pixel (x to correcting image ₁, y ₁) pass through y ₃=a ₂₁x ₁+ a ₂₂y ₁+ b ₂On the y direction, carry out geometric calibration, obtain the pixel (x of image behind the geometric calibration ₁, y ₃).

The extraction of described yardstick invariant features transform characteristics vector comprises that the extraction of source images characteristic vector may further comprise the steps to the extraction of target image and source images characteristic vector:

A. pass through the conversion of yardstick invariant features at the pixel of metric space, by double gauss difference operator D (x to source images ₁, y ₁, σ)=(G (x ₁, y ₁, k σ)-G (x ₁, y ₁σ)) * I (x, y), calculate the response of the double gauss difference operator of pixel under the metric space different scale of source images, obtain the characteristic dimension geometric locus according to response again, on the characteristic dimension geometric locus, detect all Local Extremum, determine the scale size of the position and the present position of the preliminary key point of source images then according to Local Extremum, wherein, σ is the metric space factor, two-dimensional Gaussian function $G(x_1,{\mathrm{<figure-callout\; id="1"\; label="y"\; filenames="C20071016449800151.png"\; state="\{\{state\}\}">y</figure-callout>}}_1,\mathrm{\&sigma;})=\frac{1}{2\mathrm{\&pi;}{\mathrm{\&sigma;}}^2}{e}^{-({x_1}^2+{{\mathrm{<figure-callout\; id="1"\; label="y"\; filenames="C20071016449800151.png"\; state="\{\{state\}\}">y</figure-callout>}}_1}^2)/2{\mathrm{\&sigma;}}^2},$ K is the product factor, x ₁=(x ₁, y ₁, σ) ^TFor the extreme point position is designated as

B. by the three-dimensional quadratic function of match, determine the scale size of the position and the present position of the final key point of source images, and whether the principal curvatures of judging this key point is less than the principal curvatures threshold value of setting, if, determine that then this key point is unsettled edge response point, and remove this unsettled edge response point; Otherwise, with the extreme point position of this key point

Substitution metric space function $D\left({\hat{x}}_1\right)=D+\frac{1}{2}\frac{{\&PartialD;D}^T}{\&PartialD;x_1}{\hat{x}}_1$ Obtain the metric space functional value

Judge

Whether less than the metric space threshold value of setting, if determine that then this key point is the low contrast key point, and remove this low contrast key point;

C. according to the metric space L (x of source images key point ₁, y ₁, σ)=G (x ₁, y ₁, σ) * I (x ₁, y ₁), by the directioin parameter of each key point after definite unsettled edge response point of removal of the direction distribution characteristics of source images key point neighborhood territory pixel and the low contrast key point, the source images directioin parameter comprises the big or small m (x of gradient ₁, y ₁) and direction θ (x ₁, y ₁), $m(x_1,y_1)=\sqrt{{(L(x_1+1,y_1)-L(x_1-1,y_1))}^2+{(L(x_1,y_1+1)-L(x_1,y_1-1))}^2},$ θ(x ₁，y ₁)＝tan ^-1((L(x ₁，y ₁+1)-L(x ₁，y ₁-1))/(L(x ₁+1，y ₁)-L(x ₁-1，y ₁)))；

To the extraction of target image characteristic vector, in the extraction identical operations of x ' y ' plane coordinate system employing with the source images characteristic vector.

Described characteristic matching algorithm is: the key point pair set of target image and source images coupling ^(T)P (x ₁', y ₁'), ^(S)P (x ₁, y ₁)) in, for the source images key point ^(S)P (x ₁, y ₁), at first in maximum horizontal and vertical parallax hunting zone, the most similar key point on the ferret out image ^(T)P (x ₁', y ₁') with time similar key point ^(T)P (x ₂', y ₂'), calculating then with source images and target image key point is the interior Euclidean distance of N * N window at center, if Euclidean distance satisfies $\frac{\underset{N}{\mathrm{\&Sigma;}}E{({}^{\left(S\right)}P(x_1,y_1)-{\mathrm{\&mu;}}_1,{}^{\left(T\right)}P({x_1}^{\&prime;},{{\mathrm{<figure-callout\; id="1"\; label="y"\; filenames="C20071016449800151.png"\; state="\{\{state\}\}">y</figure-callout>}}_1}^{\&prime;})-{{\mathrm{\&mu;}}_1}^{\&prime;})}^2}{\underset{N}{\mathrm{\&Sigma;}}E{({}^{\left(S\right)}P(x_1,y_1)-{\mathrm{\&mu;}}_1,{}^{\left(T\right)}P({x_2}^{\&prime;},{y_2}^{\&prime;})-{{\mathrm{\&mu;}}_2}^{\&prime;})}^2}<{\mathrm{\&tau;}}^2,$ Wherein, μ ₁Be the source images key point ^(S)P (x ₁, y ₁) average in N * N window, μ ₁' be the target image key point ^(T)P (x ₁', y ₁') average in N * N window, μ ₂' be the target image key point ^(T)P (x ₂', y ₂') average in N * N window, N is the size of window, τ is a preset threshold, then thinks the most similar key point ^(T)P (x ₁', y ₁') be the source images key point ^(S)P (x ₁, y ₁) optimal candidate coupling on target image; For the target image key point ^(T)P (x ₁', y ₁') obtain optimal candidate coupling on source images by compute euclidian distances ^(S)P (x ₂, y ₂); Parallax from the source images to the target image is designated as d _{Sou → tar}, d _{Sou → tar}=(x ₁'-x ₁, y ₁'-y ₁), the parallax from the target image to the source images is designated as d _{Tar → sou}, d _{Tar → sou}=(x ₂-x ₁', y ₂-y ₁'), to d _{Sou → tar}And d _{Tar → sou}Carry out two-way verification, if | d _{Sou → tar}+ d _{Tar → sou}|＜2, then determine key point to ( ^(T)P (x ₁', y ₁'), ^(S)P (x ₁, y ₁)) for the coupling key point right.

Compared with prior art, the advantage of a kind of multi-view image color correction provided by the present invention and geometric calibration method is:

1) based on yardstick invariant features mapping algorithm, by extracting the yardstick invariant features transform characteristics vector that factors such as rotation, yardstick convergent-divergent, view transformation, illumination conversion are maintained the invariance, the coupling of feature between two width of cloth multi-view images that can realize differing greatly, compare with the existing operation of describing characteristics of image with zone or histogram, improved the robustness of color correction greatly;

2) according to yardstick invariant features transform characteristics vector, determine that by the characteristic matching algorithm key point of target image and all couplings of source images is right, improved the precision of mapping greatly;

3) utilize the property taken advantage of error a _iWith additive errors e _iL, a and each component of b to each pixel of source images carry out color correction, meet the image-forming principle of camera more, have improved the precision of multi-view image color correction;

4) affine transformation of the present invention's definition for the camera of rigid motion, does not need to obtain in advance the inner parameter and the external parameter of camera, greatly reduces the computation complexity of multi-view image geometric calibration like this;

5) multi-view image is carried out color correction and geometric calibration preliminary treatment after, improved the coding efficiency that multi-view image is encoded, and reduced the predicated error in estimation and the compensation process.

Description of drawings

Fig. 1 is the parallel camera system imaging of a many viewpoints schematic diagram;

Fig. 2 is a multi-view image handling process schematic diagram;

Fig. 3 is the coding structure schematic diagram of multi-view image;

Fig. 4 is the flow chart of the preprocess method of multi-view image of the present invention;

Fig. 5 a is the target image of " flamenco1 " many viewpoints test set;

Fig. 5 b is the source images of " flamenco1 " many viewpoints test set;

Fig. 5 c is the target image of " golf2 " many viewpoints test set;

Fig. 5 d is the source images of " golf2 " many viewpoints test set;

Fig. 6 a is the initial characteristic vector schematic diagram of target image of " flamenco1 " many viewpoints test set;

Fig. 6 b is the initial characteristic vector schematic diagram of source images of " flamenco1 " many viewpoints test set;

Fig. 6 c is the initial characteristic vector schematic diagram of target image of " golf2 " many viewpoints test set;

Fig. 6 d is the initial characteristic vector schematic diagram of source images of " golf2 " many viewpoints test set;

Fig. 7 a is the key point schematic diagram of target image after characteristic matching of " flamenco1 " many viewpoints test set;

Fig. 7 b is the key point schematic diagram of source images after characteristic matching of " flamenco1 " many viewpoints test set;

Fig. 7 c is the key point schematic diagram of target image after characteristic matching of " golf2 " many viewpoints test set;

Fig. 7 d is the key point schematic diagram of source images after characteristic matching of " golf2 " many viewpoints test set;

Fig. 8 a is the comparison schematic diagram of " flamenco1 " many viewpoints test set, and left side figure is a target image, and right figure is a correcting image;

Fig. 8 b is the comparison schematic diagram of " golf2 " many viewpoints test set, and left side figure is a target image, and right figure is a correcting image;

Fig. 9 a is the source images of " flamenco1 " many viewpoints test set color correction front and back and the aberration comparative result schematic diagram of target image;

Fig. 9 b is the source images of " golf2 " many viewpoints test set color correction front and back and the aberration comparative result schematic diagram of target image;

Figure 10 a is the comparison schematic diagram of " flamenco1 " many viewpoints test set, and left side figure is a target image, and right figure is the image behind geometric calibration;

Figure 10 b is the comparison schematic diagram of " golf2 " many viewpoints test set, and left side figure is a target image, and right figure is the image behind geometric calibration;

Figure 11 a be the source images of " flamenco1 " many viewpoints test set with source images Y component coding distortion performance curve ratio after color correction and geometric calibration preliminary treatment than schematic diagram;

Figure 11 b be the source images of " flamenco1 " many viewpoints test set with source images U component coding distortion performance curve ratio after color correction and geometric calibration preliminary treatment than schematic diagram;

Figure 11 c be the source images of " flamenco1 " many viewpoints test set with source images V component coding distortion performance curve ratio after color correction and geometric calibration preliminary treatment than schematic diagram;

Figure 12 a be the source images of " golf2 " many viewpoints test set with source images Y component coding distortion performance curve ratio after color correction and geometric calibration preliminary treatment than schematic diagram;

Figure 12 b be the source images of " golf2 " many viewpoints test set with source images U component coding distortion performance curve ratio after color correction and geometric calibration preliminary treatment than schematic diagram;

Figure 12 c be the source images of " golf2 " many viewpoints test set with source images V component coding distortion performance curve ratio after color correction and geometric calibration preliminary treatment than schematic diagram.

Embodiment

Embodiment describes in further detail the present invention below in conjunction with accompanying drawing.

At first describe the notion of yardstick invariant features of the present invention conversion below and ask for the problem that optimal candidate is mated key point by characteristic matching.

Yardstick invariant features conversion SIFT (Scale Invariant Feature Transform) algorithm at first carries out extreme value at metric space and detects, extract the SIFT characteristic vector that factors such as rotation, yardstick convergent-divergent, view transformation, illumination conversion are maintained the invariance then, the SIFT characteristic vector mainly comprises the position of key point, the scale size of key point present position and the directioin parameter of key point.

The extraction of the SIFT characteristic vector of one secondary multi-view image comprises that the extraction of source images characteristic vector may further comprise the steps to the extraction of target image and source images characteristic vector:

A. pass through the conversion of yardstick invariant features at the pixel of metric space, by double gauss difference operator D (x to source images ₁, y ₁, σ)=(G (x ₁, y ₁, k σ)-G (x ₁, y ₁σ)) * I (x, y), calculate the response of the double gauss difference operator of pixel under the metric space different scale of source images, obtain the characteristic dimension geometric locus according to response again, on the characteristic dimension geometric locus, detect all Local Extremum, determine the scale size of the position and the present position of the preliminary key point of source images then according to Local Extremum, wherein, σ is the metric space factor, two-dimensional Gaussian function $G(x_1,{\mathrm{<figure-callout\; id="1"\; label="y"\; filenames="C20071016449800151.png"\; state="\{\{state\}\}">y</figure-callout>}}_1,\mathrm{\&sigma;})=\frac{1}{2\mathrm{\&pi;}{\mathrm{\&sigma;}}^2}{e}^{-({{\mathrm{<figure-callout\; id="1"\; label="x"\; filenames="C20071016449800151.png"\; state="\{\{state\}\}">x</figure-callout>}}_1}^2+{{\mathrm{<figure-callout\; id="1"\; label="y"\; filenames="C20071016449800151.png"\; state="\{\{state\}\}">y</figure-callout>}}_1}^2)/2{\mathrm{\&sigma;}}^2},$ K is the product factor, x ₁=(x ₁, y ₁, σ) ^TFor the extreme point position is designated as

B. by the three-dimensional quadratic function of match, determine the scale size of the position and the present position of the final key point of source images, and whether the principal curvatures of judging this key point is less than the principal curvatures threshold value of setting, if, determine that then this key point is unsettled edge response point, and remove this unsettled edge response point; Otherwise, with the extreme point position of this key point Substitution metric space function $D\left({\hat{x}}_1\right)=D+\frac{1}{2}\frac{{\&PartialD;D}^T}{\&PartialD;x_1}{\hat{x}}_1$ Obtain the metric space functional value

Judge Whether less than the metric space threshold value of setting, if determine that then this key point is the low contrast key point, and remove this low contrast key point;

C. according to the metric space L (x of source images key point ₁, y ₁, σ)=G (x ₁, y ₁, σ) * I (x ₁, y ₁), by the directioin parameter of each key point after definite unsettled edge response point of removal of the direction distribution characteristics of source images key point neighborhood territory pixel and the low contrast key point, the source images directioin parameter comprises the big or small m (x of gradient ₁, y ₁) and direction θ (x ₁, y ₁), $m(x_1,y_1)=\sqrt{{(L(x_1+1,y_1)-L(x_1-1,y_1))}^2+{(L(x_1,{\mathrm{<figure-callout\; id="1"\; label="y"\; filenames="C20071016449800151.png"\; state="\{\{state\}\}">y</figure-callout>}}_1+1)-L(x_1,y_1-1))}^2},$ θ(x ₁，y ₁)＝tan ^-1((L(x ₁，y ₁+1)-L(x ₁，y ₁-1))/(L(x ₁+1，y ₁)-L(x ₁-1，y ₁)))。

To the extraction of target image characteristic vector, in the extraction identical operations of x ' y ' plane coordinate system employing with the source images characteristic vector.

The principal curvatures threshold value of She Dinging gets 10 in the present embodiment, and the metric space threshold value of setting gets 0.03.

Yet, do not have necessary relation between the SIFT characteristic vector of the target image of extraction and source images, in order to obtain the source images key point ^(S)P (x ₁, y ₁) optimal candidate coupling on target image, at first from source images to the target image maximum horizontal with in the vertical parallax hunting zone, the most similar key point on the ferret out image ^(T)P (x ₁', y ₁') with time similar key point ^(T)P (x ₂', y ₂'), calculating with source images and target image key point then is Euclidean distance in the N * N window at center, if to the most similar key point of the most similar key point of target image ^(T)P (x ₁', y ₁') with time similar key point ^(Y)P (x ₂', y ₂') Euclidean distance satisfy $\frac{\underset{N}{\mathrm{\&Sigma;}}E{({}^{\left(S\right)}P(x_1,y_1)-{\mathrm{\&mu;}}_1,{}^{\left(T\right)}P({x_1}^{\&prime;},{y_1}^{\&prime;})-{{\mathrm{\&mu;}}_1}^{\&prime;})}^2}{\underset{N}{\mathrm{\&Sigma;}}E{({}^{\left(S\right)}P(x_1,y_1)-{\mathrm{\&mu;}}_1,{}^{\left(T\right)}P({x_2}^{\&prime;},{y_2}^{\&prime;})-{{\mathrm{\&mu;}}_2}^{\&prime;})}^2}<{\mathrm{\&tau;}}^2,$ Wherein, μ ₁Be the source images key point ^(S)P (x ₁, y ₁) average in N * N window, μ ₁' be the target image key point ^(T)P (x ₁', y ₁') average in N * N window, μ ₂' be the target image key point ^(T)P (x ₂', y ₂') average in N * N window, N is the size of window, τ is a preset threshold, then thinks the most similar key point ^(T)P (x ₁', y ₁') be the source images key point ^(S)P (x ₁, y ₁) optimal candidate coupling key point on target image, the parallax d from the source images to the target image _{Sou → tar}=(x ₁-x ₁', y ₁'-y ₁); For the target image key point ^(T)P (x ₁', y ₁'), from target image to the source images maximum horizontal with in the vertical parallax hunting zone, can on source images, obtain optimal candidate coupling key point equally by compute euclidian distances ^(S)P (x ₂, y ₂), the parallax d from the target image to the source images _{Tar → sou}=(x ₂-x ₁', y ₂-y ₁').To parallax d _{Sou → tar}And d _{Tar → sou}Carry out two-way verification, if | d _{Sou → tar}+ d _{Tar → sou}|＜2, then determine ^(S)P (x ₁, y ₁) with ^(T)P (x ₁', y ₁') be that the key point of a pair of coupling is right.In the present embodiment, the quality that the key point of window size N and target image and source images coupling is right is relevant, excessive or too smallly all can not obtain more accurate coupling, our experiments show that when being the center with current key point, both sides respectively are 3 pixels, and promptly effect is best during N=7; Threshold tau is in theory between [0,1] between, quality and quantity that τ key point big or small and target image and source images coupling is right are relevant, when τ=1, all key points are all mated, and the quality of its coupling reduces, when τ=0, therefore all key points all are unmatched, should take all factors into consideration when choosing τ, our experiments show that τ=0.8 o'clock is that effect is best.

Ask at above-mentioned yardstick invariant features mapping algorithm with by characteristic matching on the basis of optimal candidate coupling key point, as follows in conjunction with the preprocess method step of Fig. 4 multi-view image of the present invention:

At first a visual point image in the multi-view image that the different cameral synchronization is taken by many view camera system is as target image, and other visual point image is as source images, the target image set of keypoints of extracting from target image be designated as ^(T)P}, the source images set of keypoints of extracting from source images be designated as ^(S)P}, the plane coordinates of objective definition image are x ' y ' coordinate system, and the plane coordinates of definition source images is the xy coordinate system.

At metric space target image and source images are carried out extreme value by SIFT and detect, extract the target image set of keypoints ^(T)P}={ ^(T)P ⁽¹⁾, ^(T)P ⁽²⁾..., ^(T)P ^(M)And the source images set of keypoints ^(S)P}={ ^(S)P ⁽¹⁾, ^(S)P ⁽²⁾..., ^(S)P ^(N)In the SIFT characteristic vector of each key point, comprise the position of key point, the scale size of key point present position and the directioin parameter of key point, wherein, M and N are respectively the key point number of target image and source images.

According to the SIFT characteristic vector of said extracted, the target image set of keypoints ^(T)P} and source images set of keypoints ^(S)Carry out characteristic matching between the P}, with the key point of all couplings to the key point pair set that constitutes be expressed as ( ^(T)P (x ₁', y ₁'), ^(S)P (x ₁, y ₁)).

With the data of target image and source images from the RGB color space conversion to the CIELAB color space, the 1st component of three components of CIELAB color space is that luminance component is designated as L, the 2nd component is that first color component is designated as a and the 3rd component is that second color component is designated as b.The CIELAB color space is to obtain on the basis of CIEXYZ color space, conversion from the RGB color space to the CIELAB color space, at first need from the RGB color notation conversion space to the CIEXYZ color space, and then from the CIEXYZ color notation conversion space to the CIELAB color space.As light source, the map table from the RGB color space to CIEXYZ is shown CIEXYZ with standard white light D65 $\left[\begin{array}{c}X\\ Y\\ Z\end{array}\right]=\left[\begin{array}{ccc}0.412453& 0.357580& 0.180423\\ 0.212671& 0.715160& 0.072169\\ 0.019334& 0.119193& 0.950227\end{array}\right]\left[\begin{array}{c}R\\ G\\ B\end{array}\right],$ Wherein [R, G, B] scope is in [0,1].The CIEXYZ color space is shown L=116f (Y/Y to the map table of CIELAB color space _n)-16, a=500[f (X/X _n)-f (Y/Y _n)], b=200[f (Y/Y _n)-f (Z/Z _n)],

Wherein,

X _n, Y _n, Z _nBe tristimulus values, X _n=95.047, Y _n=100.000, Z _n=108.883.The dynamic range of L is [0,100], and the dynamic range of a is [120,120], and the dynamic range of b is [120,120].

The definition property taken advantage of error a _iWith additive errors e _iDescribe the difference of color between image, the property taken advantage of error mainly causes by the spectral characteristic of vision system, and additive errors is caused by the drift of color value.To the key point pair set of target image and source images coupling ( ^(T)P (x ₁', y ₁'), ^(S)P (x ₁, y ₁)), by the absolute difference sum of L, a and each component of b is carried out minimization, calculate $(a_i,e_i)=\arg \underset{a_i,e_i}{\min }\underset{(x_1,y_1),({x_1}^{\&prime;},{y_1}^{\&prime;})\&Element;\mathrm{\&Omega;}}{\mathrm{\&Sigma;}}\mathrm{abs}({}^{\left(T\right)}I_i({x_1}^{\&prime;},{y_1}^{\&prime;})-(a_i\&CenterDot;{}^{\left(S\right)}I_i(x_1,y_1)+e_i))$ Obtain the property the taken advantage of error a of L, a and each component of b _iWith additive errors e _i, Ω is the key point pair set of target image and source images coupling here, ^(S)I _i(x ₁, y ₁) be the color value of i component of source images, ^(T)I _i(x ₁', y ₁') be the color value of i component of target image, i=1,2,3;

Obtaining the property the taken advantage of error a of L, a and each component of b _iWith additive errors e _iAfter, L, a and each component of b of each pixel of source images carried out color correction operation respectively, ^(C)I _i(x ₁, y ₁)=a _i ^(S)I _i(x ₁, y ₁)+e _i, wherein, ^(C)I _i(x ₁, y ₁) be the color value of i the component of correcting image behind the color correction, ^(S)I _i(x ₁, y ₁) be the color value of i component of source images, i=1,2,3, the correcting image with the CIELAB color space is transformed into the RGB color space then.Conversion from the CIELAB color space to the RGB color space at first needs from the CIELAB color notation conversion space to the CIEXYZ color space, and then from the CIEXYZ color notation conversion space to the RGB color space.Map table from the CIELAB color space to CIEXYZ is shown: at first define f _y=(L+16)/116, f _x=f _y+ a/500, f _z=f _yIf-b/200 is f _y＞δ, $Y=Y_n\&CenterDot;{f_y}^3,$ Otherwise Y=(f _y-16/116) 3 δ ²Y _nIf f _x＞δ, $X=X_n\&CenterDot;{f_x}^3,$ Otherwise X=(f _x-16/116) 3 δ ²X _nIf f _z＞δ, $Z=Z_n\&CenterDot;{f_z}^3,$ Otherwise Z=(f _z-16/116) 3 δ ²Z _n, δ=6/29 wherein, X _n, Y _n, Z _nBe tristimulus values, X _n=95.047, Y _n=100.000, Z _n=108.883.Map table from the CIEXYZ color space to the RGB color space is shown $\left[\begin{array}{c}R\\ G\\ B\end{array}\right]=\left[\begin{array}{ccc}3.240479& -1.537150& -0.498535\\ -0.969256& 1.875992& 0.041556\\ 0.055648& -0.204043& 1.057311\end{array}\right]\left[\begin{array}{c}X\\ Y\\ Z\end{array}\right].$

After correcting image was transformed into the RGB color space, the affine transformation of setting up from the pixel of correcting image to the pixel of target image was $X^{\&prime;}=\left(\begin{array}{c}{x_1}^{\&prime;}\\ {{\mathrm{<figure-callout\; id="1"\; label="y"\; filenames="C20071016449800151.png"\; state="\{\{state\}\}">y</figure-callout>}}_1}^{\&prime;}\end{array}\right)=\left(\begin{array}{cc}{a}_{11}& a_{12}\\ a_{21}& a_{22}\end{array}\right)\left(\begin{array}{c}{x}_1\\ {\mathrm{<figure-callout\; id="1"\; label="y"\; filenames="C20071016449800151.png"\; state="\{\{state\}\}">y</figure-callout>}}_1\end{array}\right)+\left(\begin{array}{c}{b}_1\\ b_2\end{array}\right)=\mathrm{AX}+B,$ The key point pair set ( ^(T)P (x ₁', y ₁'), ^(S)P (x ₁, y ₁)) in select the right position of the key point of three pairs of Euclidean distance minimums as initial value, calculate spin matrix A and translation vector B.

At last with spin matrix A and translation vector B pixel (x to correcting image ₁, y ₁) pass through y ₃=a ₂₁x ₁+ a ₂₂y ₁+ b ₂On the y direction, carry out geometric calibration, obtain the pixel (x of image behind the geometric calibration ₁, y ₃).

After multi-view image above-mentioned color correction of process and the geometric calibration pretreatment operation, again multi-view image is encoded, reduced the predicated error of disparity estimation and compensation, improved coding efficiency.

Below carry out multi-view image color correction and the pretreated subjectivity of geometric calibration and objective performance and coding efficiency with regard to the present invention and compare.

To " flamenco1 " that is provided by KDDI company, " glof2 " two groups of multi-view point video test sets adopt multi-view image preprocess method of the present invention.Fig. 5 a, Fig. 5 b are respectively the target image and the source images of " flamenco1 " many viewpoints test set, and Fig. 5 c, Fig. 5 d are respectively the target image and the source images of " glof2 " many viewpoints test set, and target image and source images size are 320 * 240.As can be seen from the figure, the color appearance of target image and source images is obviously inconsistent, and small offset is arranged in vertical direction, and it is carried out color correction and geometric calibration just seems very necessary.Target image and source images to " flamenco1 " and " glof2 ", adopt the SIFT algorithm to extract the initial characteristic vector of the initial characteristic vector of the target image of " flamenco1 " and " glof2 " and source images shown in Fig. 6 a, Fig. 6 b, Fig. 6 c and Fig. 6 d, owing to the influence of blocking, exposing, between some characteristic vector, there is not matching relationship between viewpoint; The key point that adopts the target image that extracts " flamenco1 " and " glof2 " behind the characteristic matching algorithm of the present invention and source images is shown in Fig. 7 a, Fig. 7 b, Fig. 7 c and Fig. 7 d, the matching relationship of target image key point and source images key point is very clear and definite, substantially eliminated the key point of mistake coupling, the scale size of the size Expressing key point present position of piece among Fig. 7 a, Fig. 7 b, Fig. 7 c and Fig. 7 d.

Adopt the present invention that the source images of " flamenco1 " and " glof2 " is carried out correcting image behind the color correction shown in the right figure of the right figure of Fig. 8 a and Fig. 8 b, from the subjective effect of image as can be seen, compare with the left figure of target image Fig. 8 b of the left figure of target image Fig. 8 a of " flamenco1 " and " glof2 ", its color appearance of image and target image behind employing this method color correction are very approaching, the meadow of the floor of " flamenco1 ", " glof2 " particularly, effect is fairly obvious.

The source images after " flamenco1 " and " glof2 " employing the inventive method is proofreaied and correct and the aberration of target image, with compare without the source images of overcorrect and the aberration of target image, its comparative result schematic diagram is respectively shown in Fig. 9 a and Fig. 9 b, the difference of representing color with the CIEDE2000 aberration, calculate each right aberration of key point respectively to coupling, from Fig. 9 a and Fig. 9 b as can be seen, adopt color calibration method of the present invention, the aberration that key point is right decreases drastically, and illustrates that the similitude that adopts the present invention to proofread and correct post-equalization image and target image is stronger.

Adopt image behind the geometric calibration of the present invention shown in the right figure of the right figure of Figure 10 a and 10b, to " flamenco1 ", the correcting image of " glof2 " carries out geometric calibration, with the baseline (Epipolar Line) of the target image shown in left figure of Figure 10 a and the left figure of 10b be complete level, illustrate that geometric calibration method of the present invention is effective.

Image after color correction and the geometric calibration pretreatment operation is encoded, the coding structure that is adopted as shown in Figure 3, the coding structure that is adopted is only predicted between viewpoint, predict is I-P-P-P, be that first viewpoint adopts I frame coding, other viewpoints are all obtained by previous view prediction, and each is all adopted identical predict constantly.To through pretreated source images with compare without the coding efficiency of pretreated source images, Figure 11 a, 11b and 11c have provided the comparative result of Y component, U component and the V component coding distortion performance curve of " flamenco1 " many viewpoints test set respectively, Figure 12 a, 12b and 12c have provided the comparative result of Y component, U component and the V component coding distortion performance curve of " glof2 " many viewpoints test set respectively, and the coded data form is YUV (4:2:0).For " flamenco1 " many viewpoints test set, the distortion performance of image after the preliminary treatment, the Y component has improved 0.3dB under same code rate, and the U component has improved 0.3dB under same code rate, the V component differs and is not very big, and the decline of 0.1～0.3dB is arranged at high code check end; For " glof2 " many viewpoints test set, the Y component has 0～0.1dB to descend under same code rate, and the U component has improved 0.6～0.7dB under same code rate, and the V component has improved 0.4～0.5dB under same code rate.From overall coding result as can be seen, adopt color correction of the present invention and geometric calibration preliminary treatment after, can improve the coding efficiency of multi-view image greatly, illustrate that color correction of the present invention and geometric calibration preliminary treatment are effective.

Claims (3)

1, a kind of preprocess method of multi-view image is characterized in that it may further comprise the steps:

(1) visual point image in the multi-view image that synchronization is taken by many view camera system is defined as target image, is designated as T, and other visual point image is defined as source images, is designated as S, the target image set of keypoints of from target image, extracting be designated as ^(T)P}, the source images set of keypoints of from source images, extracting be designated as ^(S)P}, the plane coordinates of objective definition image are x ' y ' coordinate system, and the plane coordinates of definition source images is the xy coordinate system;

(2) at metric space target image and source images are carried out the extreme value detection by yardstick invariant features mapping algorithm, extraction target image set of keypoints ^(T)P} and source images set of keypoints ^(S)The yardstick invariant features transform characteristics vector of each key point among the P};

(3) according to yardstick invariant features transform characteristics vector, obtain the source images key point by the characteristic matching algorithm ^(S)P (x ₁, y ₁) optimal candidate coupling crux point on target image ^(T)P (x ₁', y ₁'), and on source images, obtain the target image key point ^(T)P (x ₁', y ₁') optimal candidate coupling key point ^(S)P (x ₂, y ₂), then by two-way verification determine key point to ( ^(T)P (x ₁', y ₁'), ^(S)P (x ₁, y ₁)) for the coupling key point right;

(4) target image and source image data are converted into the CIEXYZ color space from the RGB color space, be converted into the CIELAB color space from the CIEXYZ color space again, the 1st component of three components of CIELAB color space is that luminance component is designated as L, the 2nd component is that first color component is designated as a and the 3rd component is that second color component is designated as b;

(5) to the key point of target image and all couplings of source images to the key point pair set that constitutes ( ^(T)P (x ₁', y ₁'), ^(S)P (x ₁, y ₁)), by the absolute difference sum of L, a and each component of b is carried out minimization, calculate $(a_i,e_i)=\arg \underset{a_i,e_i}{\min }\underset{(x_1,y_1),({x_1}^{\&prime;},{y_1}^{\&prime;})\&Element;\mathrm{\&Omega;}}{\mathrm{\&Sigma;}}\mathrm{abs}({\mathrm{}}^{\left(T\right)}{I}_i({x_1}^{\&prime;},{y_1}^{\&prime;})-(a_i\&CenterDot;{\mathrm{}}^{\left(S\right)}{I}_i(x_1,y_1)+e_i))$ Obtain the property the taken advantage of error a of L, a and each component of b _iWith additive errors e _i, wherein, Ω is the key point pair set, ^(S)I _i(x ₁, y ₁) be the color value of i component of source images, ^(T)I _i(x ₁', y ₁') be the color value of i component of target image, i=1,2,3;

(6) with the property the taken advantage of error a of L, a and each component of b _iWith additive errors e _iL, a and each component of b to each pixel of source images carry out color correction, ^(C)I _i(x ₁, y ₁)=a _i ^(S)I _i(x ₁, y ₁)+e _i, wherein, ^(C)I _i(x ₁, y ₁) be the color value of i the component of correcting image behind the color correction, i=1,2,3, and correcting image is transformed into the RGB color space;

(7) affine transformation of setting up from the pixel of correcting image to the pixel of target image is $X^{\&prime;}=\left(\begin{array}{c}{x_1}^{\&prime;}\\ {y_1}^{\&prime;}\end{array}\right)=\left(\begin{array}{cc}{a}_{11}& a_{12}\\ a_{21}& a_{22}\end{array}\right)\left(\begin{array}{c}{b}_1\\ b_2\end{array}\right)=\mathrm{AX}+B,$ The key point pair set ( ^(T)P (x ₁', y ₁'), ^(S)P (x ₁, y ₁)) in select the right position of the key point of three pairs of Euclidean distance minimums as initial value, calculate spin matrix A and translation vector B;

(8) with spin matrix A and translation vector B pixel (x to correcting image ₁, y ₁) pass through y ₃=a ₂₁x ₁+ a ₂₂y ₁+ b ₂On the y direction, carry out geometric calibration, obtain the pixel (x of image behind the geometric calibration ₁, y ₃).

2, the preprocess method of a kind of multi-view image as claimed in claim 1, it is characterized in that the extraction of described yardstick invariant features transform characteristics vector, comprise that the extraction of source images characteristic vector may further comprise the steps to the extraction of target image and source images characteristic vector:

A. pass through the conversion of yardstick invariant features at the pixel of metric space, by double gauss difference operator D (x to source images ₁, y ₁, σ)=(G (x ₁, y ₁, k σ)-G (x ₁, y ₁σ)) * I (x, y), calculate the response of the double gauss difference operator of pixel under the metric space different scale of source images, obtain the characteristic dimension geometric locus according to response again, on the characteristic dimension geometric locus, detect all Local Extremum, determine the scale size of the position and the present position of the preliminary key point of source images then according to Local Extremum, wherein, σ is the metric space factor, two-dimensional Gaussian function $G(x_1,y_1,\mathrm{\&sigma;})=\frac{1}{2\mathrm{\&pi;}{\mathrm{\&sigma;}}^2}{e}^{-({x_1}^2+{y_1}^2)/{2\mathrm{\&sigma;}}^2},$ K is the product factor, x ₁=(x ₁, y ₁, σ) ^TFor the extreme point position is designated as

B. by the three-dimensional quadratic function of match, determine the scale size of the position and the present position of the final key point of source images, and whether the principal curvatures of judging this key point is less than the principal curvatures threshold value of setting, if, determine that then this key point is unsettled edge response point, and remove this unsettled edge response point; Otherwise, with the extreme point position of this key point Substitution metric space function $D\left({\hat{x}}_1\right)=D+\frac{1}{2}\frac{{\&PartialD;D}^T}{{\&PartialD;x}_1}{\hat{x}}_1$ Obtain the metric space functional value

Judge Whether less than the metric space threshold value of setting, if determine that then this key point is the low contrast key point, and remove this low contrast key point;

C. according to the metric space L (x of source images key point ₁, y ₁, σ)=G (x ₁, y ₁, σ) * I (x ₁, y ₁), by the directioin parameter of each key point after definite unsettled edge response point of removal of the direction distribution characteristics of source images key point neighborhood territory pixel and the low contrast key point, the source images directioin parameter comprises the big or small m (x of gradient ₁, y ₁) and direction θ (x ₁, y ₁),

$m(x_1,y_1)=\sqrt{{(L(x_1+1,y_1)-L(x_1-1,y_1))}^2+{(L(x_1,y_1+1)-L(x_1,y_1-1))}^2},$

θ(x ₁，y ₁)＝tan ^-1((L(x ₁，y ₁-1)-L(x ₁，y ₁-1))/(L(x ₁+1，y ₁)-L(x ₁-1，y ₁)))；

To the extraction of target image characteristic vector, in the extraction identical operations of x ' y ' plane coordinate system employing with the source images characteristic vector.

3, the preprocess method of a kind of multi-view image as claimed in claim 1 is characterized in that described characteristic matching algorithm is: the key point pair set of target image and source images coupling ( ^(T)P (x ₁', y ₁'), ^(S)P (x ₁, y ₁)) in, for the source images key point ^(S)P (x ₁, y ₁), at first in maximum horizontal and vertical parallax hunting zone, the most similar key point on the ferret out image ^(T)P (x ₁', y ₁') with time similar key point ^(T)P (x ₂', y ₂'), calculating then with source images and target image key point is the interior Euclidean distance of N * N window at center, if Euclidean distance satisfies $\frac{\underset{N}{\mathrm{\&Sigma;}}E{({\mathrm{}}^{\left(S\right)}P(x_1,y_1)-{\mathrm{\&mu;}}_1,{\mathrm{}}^{\left(T\right)}P({x_1}^{\&prime;},{y_1}^{\&prime;})-{{\mathrm{\&mu;}}_1}^{\&prime;})}^2}{\underset{N}{\mathrm{\&Sigma;}}E{({\mathrm{}}^{\left(S\right)}P(x_1,y_1)-{\mathrm{\&mu;}}_1,{\mathrm{}}^{\left(T\right)}P({x_2}^{\&prime;},{y_2}^{\&prime;})-{{\mathrm{\&mu;}}_2}^{\&prime;})}^2}<{\mathrm{\&tau;}}^2,$ Wherein, μ ₁Be the source images key point ^(S)P (x ₁, y ₁) average in N * N window, μ ₁' be the target image key point ^(T)P (x ₁', y ₁') average in N * N window, μ ₂' be the target image key point ^(T)P (x ₂', y ₂') average in N * N window, N is the size of window, τ is a preset threshold, then thinks the most similar key point ^(T)P (x ₁', y ₁') be the source images key point ^(S)P (x ₁, y ₁) optimal candidate coupling on target image; For the target image key point ^(T)P (x ₁', y ₁') obtain optimal candidate coupling on source images by compute euclidian distances ^(S)P (x ₂, y ₂); Parallax from the source images to the target image is designated as d _{Sou → tar}, d _{Sou → tar}=(x ₁'-x ₁, y ₁'-y ₁), the parallax from the target image to the source images is designated as d _{Tar → sou}, d _{Tar → sou}=(x ₂-x ₁', y ₂-y ₁'), to d _{Sou → tar}And d _{Tar → sou}Carry out two-way verification, if | d _{Sou → tar}+ d _{Tar → sou}|＜2, then determine key point to ( ^(T)P (x ₁', y ₁'), ^(S)P (x ₁, y ₁)) for the coupling key point right.

Images