CN104700381A - Infrared and visible light image fusion method based on salient objects - Google Patents

Abstract

The invention provides an infrared and visible light image fusion method based on salient objects. The infrared and visible light image fusion method based on the salient objects includes following steps: building nonlinear scale space representation for an infrared image and a visible light image which respectively comprise a plurality of objects in a given scene; using a visual attention computational model to compute visual attention salient maps of the infrared image and the visible light image based on the nonlinear scale space representation of the images; using a return inhibition mechanism to select salient object areas from the infrared image and the visible light image based on the visual attention salient maps of the infrared image and the visible light image, and computing all salient object areas in the whole scene; performing rectification operation on the infrared image and the visible light image, using a pixel level fusion algorithm to perform fusion treatment on the salient object areas, and using a feature level fusion algorithm to perform fusion treatment on non-salient object areas; generating a fusion image of the infrared image and the visible light image of the whole scene by synthesizing results.

Description

A kind of based on the infrared of well-marked target and visible light image fusion method

Technical field

The present invention relates to a kind of multi-source image integration technology field, particularly a kind of based on the infrared of well-marked target and visible images multi-level Fusion method.

Background technology

Image co-registration (Image Fusion) refers to and multiresolution or multimedium view data is produced the Comprehensive Analysis Technique of new image by spatial registration and image information complementation.Compared with single-sensor image, fused images can utilize the information of each source images to greatest extent, improve resolution and sharpness, increase the sensitivity of image object perception, perceived distance and precision, antijamming capability etc., thus reduce imperfection and the uncertainty of target apperception, improve the accuracy rate to target four and scene interpretation ability.The general flow process of image co-registration as shown in the figure.First, some pretreatment operation are carried out, as filtering noise etc. to several source images; Secondly, time-space relation is carried out to image, is mapped in same space-time coordinates by them, to guarantee the consistance of their space-time positions; Again, corresponding method is adopted to process to the image after registration; Finally, carry out fusion treatment according to certain fusion rule and obtain fused images.

Photoelectricity and infrared detection sensor are the most common two kinds of load of the tactical reconnaissance such as early warning plane and unmanned plane platform.Visual light imaging, wave band 0.4 ~ 1.0 μm, is characterized in that resolution is high, good concealment, can obtain abundant contrast, CF information, but affect by ambient light illumination, can not work at night, without camouflage recognition capability.Infrared imaging is in wave band 8 ~ 14 μm (long infrared band) and 3 ~ 5 μm (middle-infrared band), be characterized in having the ability necessarily penetrating cigarette, mist, snow etc., good concealment, imaging resolution is higher, detection range is generally between a few km to tens kms, but climate affects, and when operating distance is far away, image is stable not.When each spot temperature of target itself changes greatly or thermal background emission characteristic is more weak, the detailed information of the target that infrared image comprises or background is less, and visible images then can comprise abundant detailed information; But in dark or have in cigarette, cloud, mist bad border, visible images is second-rate, and the target in infrared image is but still clear and legible.Visible images and infrared image are carried out fusion treatment, the impact of neighbourhood noise on target detection can be overcome, the shape of comprehensive master goal and minutia, and do not affect by ambient lighting, round-the-clockly can use, system performance is studied and judged to the detection of raising reconnaissance platforms and Image Intelligence analysis significant.

The key tactical that US Department of Defense formulates at different times in the works, has the task of quite a few to relate to multi-source image fusion aspect." LANTIAN " gondola playing good operational performance in the Gulf War is exactly a kind of image fusion system that the multiple sensors information superposition such as FLIR (Forward-Looking Infrared), laser ranging, visible light camera can be shown.American TI Company obtains in nineteen ninety-five from U.S.'s night vision and electronic sensor management board (NVESD) by infrared with the contract of three generations's LLL image fusion integrated design to Advanced Helicopter (AHP) sensing system, and signal processing function is completed by TMS32OC30DSP.2003, U.S. Department of Defense's proposition " transverse direction in Military transforming merges (Transformational aboutHorizontal Fusion) " white paper, focus on laterally merging especially data fusion, object is the horizontal interoperability particularly data fusion ability of strengthening system.2006, general office of US Department of Defense (OSD) issued the Annual Technical bulleted list that will carry out verifying within the coming years, to promote the fast development in these fields.

Domestic work of merging at multi-source image concentrates on algorithm research, comprises technique study multi-source images such as utilizing DT-CWT conversion, multi-scale wavelet sampling, region segmentation and merges.Existing systems technology is mostly confined to the simple superposition of the different phase of Same Scene or the image pixel of its different color channels or accepts or rejects computing.

Summary of the invention

Goal of the invention: technical matters to be solved by this invention is the deficiency for existing in existing infrared image and visible light image fusion method, provides a kind of based on the infrared of well-marked target and visible light image fusion method.

In order to solve the problems of the technologies described above, the invention discloses a kind of based on the infrared of well-marked target and visible light image fusion method, comprising the following steps:

Step 1, comprises infrared image and the visible images of some (more than one) target to given Same Scene, set up infrared and multiscale space that is visible images respectively and represent;

Step 2, represents on basis at image non-linear metric space, utilizes visual attention computation model to calculate infrared and vision attention that is visible images and significantly schemes.

Step 3, significantly schemes on basis at infrared and visible images vision attention, utilizes the machine-processed well-marked target region selected respectively in infrared and visible images of inhibition of return, and calculates all well-marked target regions in whole scene.

Step 4, carries out registration operation to infrared image and visible images, adopts Pixel-level blending algorithm to carry out fusion treatment to well-marked target region, to non-significant target area, adopts feature-based fusion algorithm to carry out fusion treatment.

Step 5, comprehensive well-marked target area merges result and non-significant area merges result, generate infrared image and the visual image fusion image of whole scene.

Step 1 of the present invention comprises and utilizes the infrared and multiscale space that is visible images of the method establishment of Nonlinear Scale Space Theory to represent; The multiscale space of infrared image and visible images represents and is respectively:

L:I (x ₁, y ₁) × t ₁→ I (x ₁, y ₁; t ₁), wherein, wherein L is metric space conversion, x ₁represent the horizontal ordinate in infrared image I, y ₁represent the ordinate in infrared image I, t ₁represent the scale factor of infrared image multiscale space.L:V (x ₂, y ₂) × t ₂→ V (x ₂, y ₂; t ₂), wherein, wherein L is metric space conversion, x ₂represent the horizontal ordinate in visible images V, y ₂represent the ordinate in visible images V, t ₂represent the scale factor of visible images multiscale space.

For infrared image: I (x ₁, y ₁; t ₁)=I (x ₁, y ₁; 0) * g (x ₁, y ₁; t ₁), infrared image scaling function $g_1(x_1,y_1;t_1)=\frac{1}{2\mathrm{\π}{t}_1}\exp \{-\frac{{x_1}^2+{y_1}^2}{2t_{1_1}}\};$

For visible images: V (x ₂, y ₂; t ₂)=V (x ₂, y ₂; 0) * g (x ₂, y ₂; t ₂), visible images scaling function $g_2(x_2,y_2;t_2)=\frac{1}{2\mathrm{\π}{t}_2}\exp \{-\frac{{x_2}^2+{y_2}^2}{2t_2}\};$

In step 2 of the present invention, set up the visual attention computation model of infrared image and visible images, the vision attention calculating infrared image and visible images is significantly schemed, comprise: calculate the brightness of infrared image and visible images, color and direction Low Level Vision characteristic pattern, and each characteristic remarkable picture of correspondence.

For infrared image:

The remarkable figure computing formula of brightness:

c represents high scale factor, and s represents low scale factor, and I (c) represents the infrared image of high metric space, and I (s) represents the infrared image of low metric space.

The remarkable figure computing formula of color:

Red green/green red biconjugate of infrared image founds image wherein IR (c) the high yardstick red channel image that is infrared image, the low yardstick red channel image that IR (s) is infrared image, the high yardstick green channel images that IG (c) is infrared image, the low yardstick green channel images that IG (s) is infrared image;

Indigo plant Huang/champac the biconjugate of infrared image founds image wherein, the high yardstick blue channel image that IB (c) is infrared image, the low yardstick blue channel image that IB (s) is infrared image, the high yardstick yellow channels image that IY (c) is infrared image, the low yardstick yellow channels image that IY (s) is infrared image.

The remarkable figure computing formula in direction:

The direction of infrared image is significantly schemed Wherein, the high dimension characteristic pattern that IO (c, θ) is infrared image, the low dimension characteristic pattern that IO (s, θ) is infrared image, θ value is $(0,\frac{\mathrm{\π}}{4},\frac{\mathrm{\π}}{2},\frac{3\mathrm{\π}}{4}).$

The vision attention calculating whole infrared image by normalization operator N () is significantly schemed:

IS＝N(I)+N(IRG)+N(IBY)+N(IO)。

For visible images:

The remarkable figure computing formula of brightness:

c represents high scale factor, and s represents low scale factor, and V (c) represents the visible images of high metric space, and V (s) represents the visible images of low metric space.

The remarkable figure computing formula of color:

Red green/green red biconjugate of visible images founds image wherein VR (c) the high yardstick red channel image that is visible images, the low yardstick red channel image that VR (s) is visible images, the high yardstick green channel images that VG (c) is visible images, the low yardstick green channel images that VG (s) is visible images;

Indigo plant Huang/champac the biconjugate of visible images founds image wherein, the high yardstick blue channel image that VB (c) is visible images, the low yardstick blue channel image that VB (s) is visible images, the high yardstick yellow channels image that VY (c) is visible images, the low yardstick yellow channels image that VY (s) is visible images.

The remarkable figure computing formula in direction:

The direction of visible images is significantly schemed wherein, the high dimension characteristic pattern that VO (c, θ) is visible images, the low dimension characteristic pattern that VO (s, θ) is visible images, θ value is $(0,\frac{\mathrm{\π}}{4},\frac{\mathrm{\π}}{2},\frac{3\mathrm{\π}}{4}).$

The vision attention calculating whole visible images by normalization operator N () is significantly schemed:

VS＝N(V)+N(VRG)+N(VBY)+N(VO)。

In step 3 of the present invention, significantly scheme on basis at infrared and visible images vision attention, choose the maximum point of gray-scale value in visual saliency map for blinkpunkt first, after selecting first well-marked target region, the gray-scale value of blinkpunkt is first set to zero, again select the point that in the remarkable figure of visual task, gray-scale value is maximum to be second time blinkpunkt, select the well-marked target region in infrared image and visible images successively, and remember M respectively _i, M _v, M _irepresent significant target area number in infrared image, M _vrepresent significant target area in visible images, the well-marked target region M in whole scene _sumfor: M _sum=M _i+ M _v-(M _i∩ M _v).

In step 4 of the present invention, for M _sumindividual salient region, adopts the linear weighted function blending algorithm that Pixel-level merges to carry out fusion treatment to well-marked target region, obtains the fused images F of marking area _saliency(i, j)=w _i(i, j) I (i, j)+w _v(i, j) V (i, j)+C, for M _iVindividual in infrared image and visible images all significant region, w is set _i(i, j)=w _v(i, j)=0.5; For M _i-IVindividual only significant region in infrared image, arranges w _i(i, j)=0.8, w _v(i, j)=0.2; For M _v-IVindividual only significant region in visible images, arranges w _i(i, j)=0.2, w _v(i, j)=0.8; To non-significant target area, adopt the fusion carrying out non-significant target area based on principal component analytical method, obtain fused images F _non-saliency(i, j);

In step 5 of the present invention, calculate the infrared image of whole scene and the fusion results of visible images, F (i, j)=F _saliency(i, j)+F _non-saliency(i, j).

Beneficial effect: the invention has the advantages that: the multi-level Fusion model setting up infrared image and visible images, visual attention computation model is utilized to select well-marked target in infrared image and visible images respectively, Pixel-level fusion is carried out to well-marked target region, ensure the fusion mass of infrared image and visible images, carry out feature-based fusion to other non-limiting background area, the data volume of process reduces reason greatly.The method that the present invention proposes, while guaranteeing well-marked target syncretizing effect, improves the fusion efficiencies to a large amount of non-significant target area, has taken into account fusion mass and efficiency.

Accompanying drawing explanation

To do the present invention below in conjunction with the drawings and specific embodiments and further illustrate, above-mentioned and/or otherwise advantage of the present invention will become apparent.

Fig. 1 is based on the infrared of well-marked target and visible light image fusion method process flow diagram

Fig. 2 is the infrared image of a certain scene

Fig. 3 is be the visible images of Same Scene with Fig. 2

Fig. 4 is the fused images obtained based on the infrared of well-marked target and visible light image fusion method

Embodiment

Composition graphs 1, the present invention is based on the infrared of well-marked target and visible images multi-level Fusion method, specifically comprises the following steps:

Step 1, utilizes the multiscale space of Nonlinear Scale Space Theory method establishment infrared image and visible images to represent.The infrared image being provided with certain scene is I (x ₁, y ₁), (x ₁, y ₁∈ R ²) x ₁represent the horizontal ordinate in infrared image I, y ₁represent the ordinate in infrared image I; Visible images is V (x ₂, y ₂), (x ₂, y ₂∈ R ²), x ₂represent the horizontal ordinate in visible images V, y ₂represent the ordinate in visible images V.Scale factor is designated as t ∈ R ₊, the metric space of infrared image and visible images represents and is respectively L:I (x ₁, y ₁) × t ₁→ I (x ₁, y ₁; t ₁), t ₁represent the scale factor of infrared image multiscale space.L:V (x ₂, y ₂) × t ₂→ V (x ₂, y ₂; t ₂), t ₂represent the scale factor of visible images multiscale space.

I(x ₁,y ₁；0)＝I(x ₁,y ₁)L _t+s(I)＝L _t(L _s(I)) (1)

V(x ₂,y ₂；0)＝V(x ₂,y ₂)L _t+s(V)＝L _t(L _s(V)) (2)

Wherein L is metric space conversion, and the initial value of yardstick t is the yardstick in original image, is generally 0.The diffusion equation that L meets is:

${\∂}_{t}L=-\mathrm{div}\stackrel{\→}{J}---\left(3\right)$

Wherein, represent the equilibrium response of diffusion, D is that a positive definite symmetric matrices represents diffusion tensor.If with parallel, then represent that diffusion is isotropic, otherwise be anisotropy.If D is a normal number, then corresponding metric space is linear-scale space; If D is a scalar function relevant to picture structure, then it is non-linear isotropy metric space; If D is the vector function relevant to picture structure, then it is Anisotropic Nonlinear metric space.

The linear-scale space represented with gaussian kernel function is a kind of Gaussian convolution smoothing process not producing new construction.

For infrared image, can obtain:

I (x ₁, y ₁; t ₁)=I (x ₁, y ₁; 0) * g (x ₁, y ₁; t ₁) infrared image scaling function

$g_1(x_1,y_1;t_1)=\frac{1}{2\mathrm{\π}{t}_1}\exp \{-\frac{{x_1}^2+{y_1}^2}{2t_{1_1}}\}---\left(4\right)$

For visible images, can obtain:

V (x ₂, y ₂; t ₂)=V (x ₂, y ₂; 0) * g (x ₂, y ₂; t ₂) visible images scaling function

$g_2(x_2,y_2;t_2)=\frac{1}{2\mathrm{\π}{t}_2}\exp \{-\frac{{x_2}^2+{y_2}^2}{2t_2}\}---\left(5\right)$

Gaussian function during linearity metric space represents in smoothed image while noise, also level and smooth important features, thus be difficult to extract these features on thick yardstick; Meanwhile, the use of isotropic linear diffusion equation causes the correspondence at edge between different scale to be difficult to determine, thus make some thickness yardsticks represent between correspondence calculate be difficult to.For making the details such as object edge in image not by smoothly, diffusion equation should the regional area of adaptive process image, thus keeps not wishing by fuzzy information, and namely method of diffusion should be non-linear.Represent with the Nonlinear Scale Space Theory of non-linear isotropic diffusion establishing equation image, can guarantee that the region contour in each scalogram picture is enhanced, and the correspondence be conducive between different scale calculates, its diffusion equation is such as formula shown in (6):

${\∂}_{t}L=-\mathrm{div}(g\left(\right||\▿L|\left|\right)\▿L)---\left(6\right)$

$g\left(\right||\▿L|\left|\right)=\frac{1}{1+{\left|\right|\▿L\left|\right|}^2/{\mathrm{\λ}}^2}$ Or $g\left(\right||\▿L|\left|\right)=\exp (-{\left|\right|\▿L\left|\right|}^2/{\mathrm{\λ}}^2)---\left(7\right)$

Wherein, λ >0 represents edge threshold.

Step 2, represents on basis at image non-linear metric space, utilizes visual attention computation model to calculate infrared and vision attention that is visible images and significantly schemes.

Represent on basis in step 1 pair infrared image and visible images metric space, extract color, brightness and direction respectively as the low-level feature guiding vision attention, calculate all kinds of characteristic remarkable picture by the central authorities-periphery difference of receptive field, obtain vision attention finally by normalization operator and significantly scheme.

(1) Low Level Vision characteristic pattern

For infrared image:

Brightness figure: note Ir (t ₁), Ig (t ₁) and Ib (t ₁) be respectively redness, green and blue channel in original infrared image, wherein t ₁represent scale factor, luminance graph is:

I(t ₁)＝(Ir(t ₁)+Ig(t ₁)+Ib(t ₁))/3 (8)

Color characteristic figure: with I (t ₁) to r (t ₁), g (t ₁) and b (t ₁) passage is normalized, can obtain sensu lato red, green, blue and Huang Si channel value, such as formula (9), (10), (11), shown in (12).

IR(t ₁)＝Ir(t ₁)-(Ig(t ₁)+Ib(t ₁))/2 (9)

IG(t ₁)＝Ig(t ₁)-(Ir(t ₁)+Ib(t ₁))/2 (10)

IB(t ₁)＝Ib(t ₁)-(Ir(t ₁)+Ig(t ₁))/2 (11)

IY(t ₁)＝Ir(t ₁)+Ig(t ₁)-2*(|Ir(t ₁)-Ig(t ₁)|+Ib(t ₁)) (12)

Wherein, negative value is set to 0.

Direction character figure: utilize Gabor filter to calculate the direction character figure of infrared image, shown in (13):

$\mathrm{IG}(x_1,y_1)=\frac{1}{2\mathrm{\π\σ\β}}\exp \{-\mathrm{\π}[\frac{{(x_1-x_0)}^2}{{\mathrm{\σ}}^2}+\frac{{(y_1-y_0)}^2}{{\mathrm{\β}}^2}\left]\right\}\exp \{i[{\mathrm{\ξ}}_0{x}_1+v_0{y}_1]\}---\left(13\right)$

Wherein (x ₀, y ₀) be receptive field centre coordinate in spatial domain, x ₀for horizontal ordinate, y ₀for ordinate; (ξ ₀, υ ₀) be the optimal spatial frequency of wave filter on frequency domain, ξ ₀for real part, υ ₀for imaginary part.α and β is the standard deviation of Gaussian function on x and y-axis direction respectively.The Gabor filter that the present invention gets four direction exports as directional vision characteristic pattern: n is the direction that constant 4, i represents different, and value is 0,1,2,3.

For visible images:

Brightness figure: note Vr (t ₂), Vg (t ₂) and Vb (t ₂) be respectively redness, green and blue channel in primary visible light image, wherein t ₂represent scale factor, then luminance graph is:

V(t ₂)＝(Vr(t ₂)+Vg(t ₂)+Vb(t ₂))/3 (14)

Color characteristic figure: with V (t ₂) to Vr (t ₂), Vg (t ₂) and Vb (t ₂) passage is normalized, can obtain sensu lato red, green, blue and Huang Si channel value, such as formula (15), (16), (17), shown in (18).

VR(t ₂)＝Vr(t ₂)-(Vg(t ₂)+Vb(t ₂))/2 (15)

VG(t ₂)＝Vg(t ₂)-(Vr(t ₂)+Vb(t ₂))/2 (16)

VB(t ₂)＝Vb(t ₂)-(Vr(t ₂)+Vg(t ₂))/2 (17)

VY(t ₂)＝Vr(t ₂)+Vg(t ₂)-2*(|Vr(t ₂)-Vg(t ₂)|+Vb(t ₂)) (18)

Wherein, negative value is set to 0.

Direction character figure: utilize Gabor filter to calculate the direction character figure of visible images, shown in (19):

$\mathrm{VG}(x_2,y_2)=\frac{1}{2\mathrm{\π\σ\β}}\exp \{-\mathrm{\π}[\frac{{(x_2-x_0)}^2}{{\mathrm{\σ}}^2}+\frac{{(y_2-y_0)}^2}{{\mathrm{\β}}^2}\left]\right\}\exp \{i[{\mathrm{\ξ}}_0{x}_2+v_0{y}_2]\}---\left(19\right)$

Wherein (x ₀, y ₀) be receptive field centre coordinate in spatial domain, x ₀for horizontal ordinate, y ₀for ordinate; (ξ ₀, υ ₀) be the optimal spatial frequency of wave filter on frequency domain, ξ ₀for real part, υ ₀for imaginary part.α and β is the standard deviation of Gaussian function on x and y-axis direction respectively.The Gabor filter that the present invention gets four direction exports as directional vision characteristic pattern: n is the direction that constant 4, i represents different, and value is 0,1,2,3.

(2) characteristic remarkable picture

For infrared image:

By brightness I, four color components IR, IG, IB, IY and the four direction characteristic pattern method of metric space represent, center image corresponds to the yardstick under high resolving power, periphery image corresponds to the yardstick under low resolution, realize receptive field and integrate the poor calculative strategy in wild central authorities-periphery, obtain each characteristic remarkable picture.

Brightness is significantly schemed:

The remarkable figure of brightness is produced by luminance contrast, represents that brightness is significantly schemed with I (c, s):

Wherein, c is the scale factor that infrared image metric space represents middle high-resolution, and s is low resolution scale factor, for central authorities-periphery difference calculates son, the respective pixel deducted in low-resolution image by the pixel in high-definition picture is realized.

Color is significantly schemed:

Red green/green red biconjugate of infrared image founds image:

Wherein IR (c) the high yardstick red channel image that is infrared image, the low yardstick red channel image that IR (s) is infrared image, the high yardstick green channel images that IG (c) is infrared image, the low yardstick green channel images that IG (s) is infrared image.

Indigo plant Huang/champac the biconjugate of infrared image founds image:

Wherein, the high yardstick blue channel image that IB (c) is infrared image, the low yardstick blue channel image that IB (s) is infrared image, the high yardstick yellow channels image that IY (c) is infrared image, the low yardstick yellow channels image that IY (s) is infrared image.

Direction is significantly schemed:

Utilize Gabor pyramid IO (σ, θ), can obtain image local directional information, wherein σ is scale factor, θ ∈ 0 °, and 45 °, 90 °, 135 ° }.By the calculating of local direction contrast, by the calculating of local direction contrast, direction significantly can be schemed IO (c, s, θ) and be encoded into one group:

For visible images:

By brightness V, four color components VR, VG, VB, VY and the four direction characteristic pattern method of metric space represent, center image corresponds to the yardstick under high resolving power, periphery image corresponds to the yardstick under low resolution, realize receptive field and integrate the poor calculative strategy in wild central authorities-periphery, obtain each characteristic remarkable picture.

Brightness is significantly schemed:

The remarkable figure of brightness is produced by luminance contrast, represents that brightness is significantly schemed with V (c, s):

Wherein, c is the scale factor that visible images metric space represents middle high-resolution, and s is low resolution scale factor, for central authorities-periphery difference calculates son, the respective pixel deducted in low-resolution image by the pixel in high-definition picture is realized.

Color is significantly schemed:

Red green/green red biconjugate of visible images founds image:

Wherein VR (c) the high yardstick red channel image that is visible images, the low yardstick red channel image that VR (s) is visible images, the high yardstick green channel images that VG (c) is visible images, the low yardstick green channel images that VG (s) is visible images.

Indigo plant Huang/champac the biconjugate of visible images founds image:

Wherein, the high yardstick blue channel image that VB (c) is visible images, the low yardstick blue channel image that VB (s) is visible images, the high yardstick yellow channels image that VY (c) is visible images, the low yardstick yellow channels image that VY (s) is visible images.

Direction is significantly schemed:

Utilize Gabor pyramid VO (σ, θ), can obtain image local directional information, wherein σ is scale factor, θ ∈ 0 °, and 45 °, 90 °, 135 ° }.By the calculating of local direction contrast, by the calculating of local direction contrast, direction significantly can be schemed VO (c, s, θ) and be encoded into one group:

(3) vision attention is significantly schemed

The low-level feature figure in brightness, color and direction has different dynamic ranges and extraction mechanism, and because all characteristic remarkable pictures are combined, the object that only conspicuousness is stronger in minority figure, may be covered by noise or conspicuousness is less in large spirogram object.Therefore, the present invention utilizes normalization operator N (), to strengthen the less characteristic remarkable picture in remarkable peak, and weakens the characteristic remarkable picture that there are a large amount of significantly peaks.Significantly scheme each category feature, the operation of this operator comprises: 1) this characteristic pattern of normalization to [0 ..., 1] and in scope, to eliminate the amplitude difference depending on feature; 2) the great average in all local except the overall situation is maximum is calculated ) use take advantage of this characteristic remarkable picture.

For infrared image:

Respectively the remarkable figure of brightness, color and direction character is normalized by normalization operator N (), obtains with be shown below.

$\stackrel{\‾}{I}=\underset{c}{\⊕}\underset{s}{\⊕}N\left(I(c,s)\right)---\left(24\right)$

$\stackrel{\‾}{\mathrm{IRG}}=\underset{c}{\⊕}\underset{s}{\⊕}N\left(\mathrm{IRG}(c,s)\right)---\left(25\right)$

$\stackrel{\‾}{\mathrm{IBY}}=\underset{c}{\⊕}\underset{s}{\⊕}N\left(\mathrm{IBY}(c,s)\right)---\left(26\right)$

$\stackrel{\‾}{\mathrm{IO}}=\underset{\mathrm{\θ}}{\mathrm{\Σ}}N\left(\underset{c}{\⊕}\underset{s}{\⊕}N\left(\mathrm{IO}(c,s,\mathrm{\θ})\right)\right)---\left(27\right)$

In formula, symbol ⊕ represents that pointwise is sued for peace.

The remarkable figure of vision attention has formula (22) to calculate:

$\mathrm{IS}=N\left(\stackrel{\‾}{I}\right)+N\left(\stackrel{\‾}{\mathrm{IRG}}\right)+N\left(\stackrel{\‾}{\mathrm{IBY}}\right)+N\left(\stackrel{\‾}{\mathrm{IO}}\right)---\left(22\right)$

For visible images:

Respectively the remarkable figure of brightness, color and direction character is normalized by normalization operator N (), obtains with be shown below.

$\stackrel{\‾}{V}=\underset{c}{\⊕}\underset{s}{\⊕}N\left(V(c,s)\right)---\left(24\right)$

$\stackrel{\‾}{\mathrm{VRG}}=\underset{c}{\⊕}\underset{s}{\⊕}N\left(\mathrm{VRG}(c,s)\right)---\left(25\right)$

$\stackrel{\‾}{\mathrm{VBY}}=\underset{c}{\⊕}\underset{s}{\⊕}N\left(\mathrm{VBY}(c,s)\right)---\left(26\right)$

$\stackrel{\‾}{\mathrm{VO}}=\underset{\mathrm{\θ}}{\mathrm{\Σ}}N\left(\underset{c}{\⊕}\underset{s}{\⊕}N\left(\mathrm{VO}(c,s,\mathrm{\θ})\right)\right)---\left(27\right)$

In formula, symbol ⊕ represents that pointwise is sued for peace.

The remarkable figure of vision attention has formula (22) to calculate:

$\mathrm{VS}=N\left(\stackrel{\‾}{V}\right)+N\left(\stackrel{\‾}{\mathrm{VRG}}\right)+N\left(\stackrel{\‾}{\mathrm{VBY}}\right)+N\left(\stackrel{\‾}{\mathrm{VO}}\right)---\left(22\right)$

Step 3, significantly schemes on basis at infrared and visible images vision attention, utilizes the machine-processed well-marked target region selected respectively in infrared and visible images of inhibition of return, and calculates all well-marked target regions in whole scene.

After calculating visual saliency map by step 2, choose the maximum point of gray-scale value in visual saliency map for blinkpunkt first, centered by this point, go out first well-marked target region by the rectangular selection of original image size 1/16.After first well-marked target regional choice goes out, the gray-scale value of blinkpunkt is first set to zero, again select the point that in the remarkable figure of visual task, gray-scale value is maximum be second time blinkpunkt, the rectangular selection of employing original image size 1/16 goes out second well-marked target region.Iterative cycles, selects the well-marked target region in original image one by one.Well-marked target region in the infrared image selected and visible images is respectively M _i, M _v.

Well-marked target region in whole scene is M _sum=M _i+ M _v-(M _i∩ M _v).

Step 4, carries out registration operation to infrared image and visible images, adopts Pixel-level blending algorithm to carry out fusion treatment to well-marked target region, to non-significant target area, adopts feature-based fusion algorithm to carry out fusion treatment.

Adopt key point matching method, registration operation is carried out to infrared image I and visible images V.

For salient region, the well-marked target calculated in whole scene by step 3 is M _sum, wherein, in infrared image and visible images, all significant areal is M _iV=M _i∩ M _v, only in infrared image, significant areal is M _i-IV=M _i-M _iV, only in visible images, significant areal is M _v-IV=M _v-M _iV, adopt the linear weighted function blending algorithm that Pixel-level merges to carry out fusion treatment to well-marked target region, obtain the fused images F of marking area _saliency(i, j), represents by formula (23):

F _saliency(i,j)＝w _I(i,j)·I(i,j)+w _V(i,j)·V(i,j)+C (23)

Wherein, I (i, j) and V (i, j) represents the pixel position in source images, and F (i, j) represents the pixel position in fused images.W _i(i, j) and w _v(i, j) is weights, and w _i(i, j)+w _v(i, j)=1.For M _iVindividual in infrared image and visible images all significant region, w is set _i(i, j)=w _v(i, j)=0.5; For M _i-IVindividual only significant region in infrared image, arranges w _i(i, j)=0.8, w _v(i, j)=0.2; For M _v-IVindividual only significant region in visible images, arranges w _i(i, j)=0.2, w _v(i, j)=0.8.

To non-significant target area, feature level Image Fusion is adopted to carry out fusion treatment to the non-significant target area in infrared image and visible images.The fusion carrying out non-significant target area based on principal component analysis (PCA) (Principal Component Analysis, PCA) method is adopted in this patent.

Infrared image and visible images are converted by PCA shift theory respectively, obtain each major component, the first principal component image of infrared image and visible images is carried out Histogram Matching, then the first Main classification full-colour image is replaced, obtain fused images F by PCA inverse transformation _non-saliency(i, j).

Step 5, comprehensive well-marked target area merges result and non-significant area merges result, generate infrared image and the visual image fusion image of whole scene.

The fused images F in well-marked target region in scene is calculated in step 4 _saliency(i, j) and non-significant target area image F _non-saliencyon the basis of (i, j), calculate the infrared image of whole scene and the fusion results of visible images, shown in (24).

F(i,j)＝F _saliency(i,j)+F _non-saliency(i,j) (24)

Embodiment

By an object lesson, implementation procedure of the present invention is described.

Fig. 2 is the infrared image of a certain scene, and Fig. 3 is the visible images of a certain scene.

According to described in step 1, the Nonlinear Scale Space Theory first setting up infrared image and visible images represents, edge threshold λ is taken as 0.5.

According to described in step 2, calculate the brightness of infrared image and visible images, color and directional vision characteristic pattern respectively, and brightness, color and direction are significantly schemed, the vision attention calculating infrared image and visible images is significantly schemed.

According to described in step 3, the well-marked target region calculated in infrared image is 5, and the well-marked target region in visible images is 4, and wherein in infrared image and visible images, all significant target is 3, and the well-marked target region in whole scene is 6.

According to described in step 4, a well-marked target region, 6 in scene is merged according to Pixel-level fusion method, other non-significant target area is merged according to feature level fusing method.

According to described in step 5, the fusion results of the fusion results in well-marked target region and non-significant target area is carried out comprehensively, obtains infrared image and the visual image fusion result of whole scene, as shown in Figure 4.

Effect of the present invention can be summarised as:

1) the multi-level Fusion model of image co-registration is set up, Pixel-level fusion method and feature level fusing method are adopted respectively to the well-marked target in scene and non-significant target, Pixel-level fusion is carried out to well-marked target, namely essence merges, feature-based fusion is carried out to other background area, namely slightly merges, while guaranteeing well-marked target syncretizing effect, improve the fusion efficiencies to a large amount of non-significant target area, meet with the visual cognition process height of the mankind.

2) utilize the multiscale space of Nonlinear Scale Space Theory method establishment image to represent, in smoothed image while noise, guarantee that in each scalogram picture, well-marked target profile is enhanced, and improves the fusion mass to well-marked target in scene.

3) visual attention computation model of infrared image and visible images is set up, from brightness, color and aspect, direction three metric objective conspicuousness, and the well-marked target calculated respectively by normalization operator and inhibition of return mechanism in infrared image and visible images, select the well-marked target in infrared image and visible images simultaneously, and carry out fusion treatment.

4) method that the present invention proposes not only can be used for the fusion treatment of infrared image and visible images, field can also be merged by further genralrlization to other multi-source image, as multispectral image and visual image fusion, SAR image and visible images, SAR image and infrared image etc., application prospect is extensive.

The invention provides a kind of based on the infrared of well-marked target and visible light image fusion method; the method and access of this technical scheme of specific implementation is a lot; the above is only the preferred embodiment of the present invention; should be understood that; for those skilled in the art; under the premise without departing from the principles of the invention, can also make some improvements and modifications, these improvements and modifications also should be considered as protection scope of the present invention.The all available prior art of each ingredient not clear and definite in the present embodiment is realized.

Claims (6)

1. based on the infrared of well-marked target and a visible light image fusion method, it is characterized in that, comprise the following steps:

Step 1, given Same Scene is comprised to infrared image and the visible images of some targets, the multiscale space setting up infrared image and visible images respectively represents;

Step 2, represents on basis at image non-linear metric space, sets up the visual attention computation model of infrared image and visible images, and the vision attention calculating infrared image and visible images is significantly schemed;

Step 3, significantly schemes on basis at infrared image and visible images vision attention, utilizes the machine-processed well-marked target region selected respectively in infrared image and visible images of inhibition of return, and calculates all well-marked target regions in whole scene;

Step 4, carries out registration operation to infrared image and visible images, adopts Pixel-level blending algorithm to carry out fusion treatment to well-marked target region, to non-significant target area, adopts feature-based fusion algorithm to carry out fusion treatment;

Step 5, comprehensive well-marked target area merges result and non-significant area merges result, generate infrared image and the visual image fusion image of whole scene.

2. according to claim 1ly a kind ofly to it is characterized in that based on the infrared of well-marked target and visible light image fusion method, in step 1, the multiscale space of infrared image and visible images represents and is respectively:

L:I (x ₁, y ₁) × t ₁→ I (x ₁, y ₁; t ₁), wherein, wherein L is metric space conversion, x ₁represent the horizontal ordinate in infrared image I, y ₁represent the ordinate in infrared image I, t ₁represent the scale factor of infrared image multiscale space;

L:V (x ₂, y ₂) × t ₂→ V (x ₂, y ₂; t ₂), wherein, wherein L is metric space conversion, x ₂represent the horizontal ordinate in visible images V, y ₂represent the ordinate in visible images V, t ₂represent the scale factor of visible images multiscale space;

For infrared image I (x ₁, y ₁; t ₁):

I(x ₁,y ₁；t ₁)＝I(x ₁,y ₁；0)*g(x ₁,y ₁；t ₁)，

Infrared image scaling function g ₁(x ₁, y ₁; t ₁) be:

$g_1(x_1,y_1;t_1)=\frac{1}{{2\mathrm{\πt}}_1}\exp \{-\frac{{x_1}^2+{y_1}^2}{{2t}_{1_1}}\},$

For visible images V (x ₂, y ₂; t ₂):

V(x ₂,y ₂；t ₂)＝V(x ₂,y ₂；0)*g(x ₂,y ₂；t ₂)，

Visible images scaling function g ₂(x ₂, y ₂; t ₂) be:

$g_2(x_2,y_2;t_2)=\frac{1}{{2\mathrm{\πt}}_2}\exp \{-\frac{{x_2}^2+{y_2}^2}{{2t}_2}\}.$

3. according to claim 2 a kind of based on the infrared of well-marked target and visible light image fusion method, it is characterized in that, in step 2, set up the visual attention computation model of infrared image and visible images, the vision attention calculating infrared image and visible images is significantly schemed, comprise: calculate the brightness of infrared image and visible images, color and direction Low Level Vision characteristic pattern, and each characteristic remarkable picture of correspondence;

For infrared image:

I (c, s) computing formula is significantly schemed in brightness:

c represents high scale factor, and s represents low scale factor, and I (c) represents the infrared image of high metric space, and I (s) represents the infrared image of low metric space;

Color remarkable figure computing method are as follows:

Red green-green red biconjugate of infrared image founds image IRG (c, s) computing formula and is:

wherein IR (c) the high yardstick red channel image that is infrared image, the low yardstick red channel image that IR (s) is infrared image, the high yardstick green channel images that IG (c) is infrared image, the low yardstick green channel images that IG (s) is infrared image;

Indigo plant Huang-champac the biconjugate of infrared image founds image IBY (c, s) computing formula and is:

wherein, the high yardstick blue channel image that IB (c) is infrared image, the low yardstick blue channel image that IB (s) is infrared image, the high yardstick yellow channels image that IY (c) is infrared image, the low yardstick yellow channels image that IY (s) is infrared image;

Direction remarkable figure computing method are as follows:

The direction of infrared image is significantly schemed IO (c, s, θ) computing formula and is:

wherein, the high dimension characteristic pattern that IO (c, θ) is infrared image, the low dimension characteristic pattern that IO (s, θ) is infrared image, θ value is

The vision attention calculating whole infrared image by normalization operator N () significantly schemes IS:

IS＝N(I)+N(IRG)+N(IBY)+N(IO)；

For visible images:

V (c, s) computing formula is significantly schemed in brightness:

c represents high scale factor, and s represents low scale factor, and V (c) represents the visible images of high metric space, and V (s) represents the visible images of low metric space;

Color remarkable figure computing method are as follows:

Red green-green red biconjugate of visible images founds image VRG (c, s) computing formula and is:

Wherein VR (c) the high yardstick red channel image that is visible images, the low yardstick red channel image that VR (s) is visible images, the high yardstick green channel images that VG (c) is visible images, the low yardstick green channel images that VG (s) is visible images;

Indigo plant Huang-champac the biconjugate of visible images founds image VBY (c, s) computing formula and is:

wherein, the high yardstick blue channel image that VB (c) is visible images, the low yardstick blue channel image that VB (s) is visible images, the high yardstick yellow channels image that VY (c) is visible images, the low yardstick yellow channels image that VY (s) is visible images;

The remarkable figure computing formula in direction:

The direction of visible images is significantly schemed VO (c, s, θ) computing formula and is:

Wherein, the high dimension characteristic pattern that VO (c, θ) is visible images, the low dimension characteristic pattern that VO (s, θ) is visible images, θ value is

The vision attention calculating whole visible images by normalization operator N () is significantly schemed:

VS＝N(V)+N(VRG)+N(VBY)+N(VO)。

4. according to claim 3 a kind of based on the infrared of well-marked target and visible light image fusion method, it is characterized in that, in step 3, significantly scheme on basis at infrared and visible images vision attention, choose the maximum point of gray-scale value in visual saliency map for blinkpunkt first, after selecting first well-marked target region, the gray-scale value of blinkpunkt is first set to zero, the point that in the remarkable figure of visual task, gray-scale value is maximum is again selected to be second time blinkpunkt, select the well-marked target region in infrared image and visible images successively, and remember M respectively _i, M _v, M _irepresent significant target area number in infrared image, M _vrepresent significant target area in visible images, the well-marked target region M in whole scene _sumfor:

$M_{\mathrm{sum}}=M_I+M_V-(M_I\∩M_V).$

5. according to claim 4ly a kind ofly to it is characterized in that based on the infrared of well-marked target and visible light image fusion method, in step 4, for M _sumindividual salient region, adopt the linear weighted function blending algorithm that Pixel-level merges to carry out fusion treatment to well-marked target region, the fused images obtaining marking area is:

F _saliency(i,j)＝w _I(i,j)·I(i,j)+w _V(i,j)·V(i,j)+C，

In infrared image and visible images, all significant region is M _iV, w is set _i(i, j)=w _v(i, j)=0.5; Only in infrared image, significant region is M _i-IV, w is set _i(i, j)=0.8, w _v(i, j)=0.2; Only in visible images, significant region is M _v-IV, w is set _i(i, j)=0.2, w _v(i, j)=0.8;

To non-significant target area, adopt the fusion carrying out non-significant target area based on principal component analytical method, obtain fused images F _non-saliency(i, j).

6. according to claim 5ly a kind ofly to it is characterized in that based on the infrared of well-marked target and visible light image fusion method, in step 5, calculate the infrared image of whole scene and the fusion results F (i, j) of visible images:

F(i,j)＝F _saliency(i,j)+F _non-saliency(i,j)。