CN116416175A - Image fusion method based on self-adaptive edge-preserving smooth pyramid - Google Patents
Image fusion method based on self-adaptive edge-preserving smooth pyramid Download PDFInfo
- Publication number
- CN116416175A CN116416175A CN202310423665.2A CN202310423665A CN116416175A CN 116416175 A CN116416175 A CN 116416175A CN 202310423665 A CN202310423665 A CN 202310423665A CN 116416175 A CN116416175 A CN 116416175A
- Authority
- CN
- China
- Prior art keywords
- pyramid
- image
- weight
- exposure
- image sequence
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000007500 overflow downdraw method Methods 0.000 title claims abstract description 17
- 238000001914 filtration Methods 0.000 claims abstract description 26
- 230000004927 fusion Effects 0.000 claims abstract description 22
- 238000012545 processing Methods 0.000 claims abstract description 15
- 230000003044 adaptive effect Effects 0.000 claims description 24
- 238000009499 grossing Methods 0.000 claims description 24
- 238000000034 method Methods 0.000 claims description 24
- 238000010606 normalization Methods 0.000 claims description 11
- 230000006978 adaptation Effects 0.000 claims description 8
- PCHJSUWPFVWCPO-UHFFFAOYSA-N gold Chemical compound [Au] PCHJSUWPFVWCPO-UHFFFAOYSA-N 0.000 claims 1
- 239000010931 gold Substances 0.000 claims 1
- 229910052737 gold Inorganic materials 0.000 claims 1
- 230000009466 transformation Effects 0.000 claims 1
- 230000007704 transition Effects 0.000 abstract description 3
- 238000013507 mapping Methods 0.000 description 7
- 238000005259 measurement Methods 0.000 description 7
- 230000000694 effects Effects 0.000 description 6
- 230000000007 visual effect Effects 0.000 description 5
- 230000006870 function Effects 0.000 description 4
- 230000014759 maintenance of location Effects 0.000 description 4
- 230000003247 decreasing effect Effects 0.000 description 3
- 125000001475 halogen functional group Chemical group 0.000 description 3
- 238000003384 imaging method Methods 0.000 description 3
- 230000002159 abnormal effect Effects 0.000 description 2
- 238000013459 approach Methods 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 2
- 238000010276 construction Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000002035 prolonged effect Effects 0.000 description 2
- 229920006395 saturated elastomer Polymers 0.000 description 2
- 238000000354 decomposition reaction Methods 0.000 description 1
- 238000002059 diagnostic imaging Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 238000007499 fusion processing Methods 0.000 description 1
- 238000005286 illumination Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 238000007670 refining Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/90—Dynamic range modification of images or parts thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20212—Image combination
- G06T2207/20221—Image fusion; Image merging
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Landscapes
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Image Processing (AREA)
Abstract
The invention discloses an image fusion method based on a self-adaptive edge-preserving smooth pyramid, and relates to the technical field of image processing. Comprising the following steps: acquiring an image sequence; obtaining an initial weight map of the image sequence according to the improved exposure weight, the local contrast weight and the saturation weight; taking a Gaussian pyramid of a brightness component of an image sequence as a guide pyramid, and carrying out weighted guide filtering processing on an initial weight graph to obtain a weight graph pyramid; constructing a self-adaptive edge retaining smooth pyramid of the weight map by utilizing a detail layer of the Laplacian pyramid of the image sequence and a coefficient of the weighted guide filtering; and reconstructing the self-adaptive edge-retaining smooth pyramid and the Laplacian pyramid to obtain a fusion image. The invention has the following advantages: considering the situation that the whole input picture sequence is darker or lighter due to the fact that the information and the light-shade transition in the space field are more natural, the details of the darkest and brightest areas are better reserved, and meanwhile, the generation of halation artifacts is effectively restrained.
Description
Technical Field
The invention relates to the technical field of image processing, in particular to an image fusion method based on a self-adaptive edge-preserving smooth pyramid.
Background
The luminance dynamic range of most natural scenes is typically much larger than the capture range of a single shot of a common digital camera. For example, the dynamic range of natural scenes can reach 8 orders of magnitude from weak starlight to bright sunlight, however, the image sensor used in a conventional digital camera can capture only two orders of magnitude of dynamic range. The contradiction between the High Dynamic Range (HDR) of the real world and the Low Dynamic Range (LDR) of the digital camera causes that a single image shot by the camera is difficult to retain all detail content, the image contrast is easy to be low, the detail information is seriously lost, and the problems of overexposure or underexposure occur in local areas and the like. This challenge can be addressed by HDR imaging, which can recover information from multiple LDR images with different exposures in the same HDR scene. Currently, HDR technology is applied in the fields of medical imaging, video surveillance, satellite remote sensing, and the like.
One approach to acquiring HDR images is to use dedicated HDR devices or develop special sensors that can capture a wide dynamic range. However, these specialized devices are not common and are too expensive for the average consumer. Furthermore, current display devices do not support the display of HDR images. Therefore, this method has not been widely used. Unlike hardware-based solutions, software solutions are cheaper, efficient, and easy to implement. In recent years, HDR imaging techniques have been able to generate HDR images using a common camera by capturing multiple exposure images. Using HDR techniques, a sequence of LDR images containing details in different luminance ranges is combined into one image that can more accurately preserve most of the information of the real scene and has better visual effects. Current HDR imaging techniques mainly involve two approaches, tone mapping-based and multi-exposure image fusion (MEF) -based.
The tone mapping based method mainly comprises two steps: HDR image construction and tone mapping. It is first necessary to construct an HDR image from an LDR image and then let the HDR image be displayed on a common LDR display device using tone mapping. This method requires specialized equipment and calculates multiple exposure parameters, and is therefore time consuming and limiting.
Unlike tone mapping-based methods, MEF methods avoid the construction of intermediate HDR images, but directly extract comprehensive image information from multiple low dynamic range images, without being limited by illumination and camera parameters, and synthesize it into a fused image with good contrast, vivid color, and more informative and perceptually attractive properties. Furthermore, the obtained high quality fused image can be displayed directly on the LDR device without any additional processing. The method is simple and effective, but tends to have low operation speed, and has the problems of fuzzy details, local color distortion, halation artifact and the like. The existing MEF method is not suitable for the condition that the whole brightness of an input image sequence is dark or bright, and the details of the brightest or darkest area cannot be well reserved in a fusion image.
Therefore, how to solve the problems of detail blurring, halation artifacts and improved adaptability to input images existing in the existing solutions is a urgent need for those skilled in the art.
Disclosure of Invention
In view of the above, the present invention provides an image fusion method based on an adaptive edge-preserving smooth pyramid, which combines an improved exposure weight algorithm with a fusion algorithm of an adaptive edge-preserving smooth pyramid, so as to effectively inhibit the generation of halation artifacts while better preserving the details of the darkest or brightest region.
In order to achieve the above purpose, the present invention adopts the following technical scheme: an image fusion method based on an adaptive edge preserving smooth pyramid, comprising:
acquiring an image sequence;
calculating an improved exposure weight according to the overall brightness of the image sequence and the relative brightness between adjacent images;
extracting local contrast information from a gray image of a source image by using a dense scale invariant feature descriptor, and calculating local contrast weight according to the maximum value of the same pixel position in all the images;
calculating a saturation weight using R, G and the standard deviation of each pixel in the B channel;
obtaining an initial weight map of the image sequence according to the improved exposure weight, the local contrast weight and the saturation weight;
taking a Gaussian pyramid of a brightness component of an image sequence as a guide pyramid, and carrying out weighted guide filtering processing on the initial weight graph to obtain a weight graph pyramid;
constructing a self-adaptive edge retaining smooth pyramid of the weight map by utilizing a detail layer of the Laplacian pyramid of the image sequence and a coefficient of the weighted guide filtering;
and reconstructing the self-adaptive edge-retaining smooth pyramid and the Laplacian pyramid to obtain a fusion image.
Preferably, calculating the improved exposure weight specifically includes:
carrying out graying treatment on the image sequence to obtain a graying image;
normalizing the graying image;
acquiring a pixel value of the normalized gray image;
calculating normalized average brightness of the image sequence according to the pixel value;
and adjusting the exposure weight of the dark part or the bright part in a self-adaptive mode according to the normalized average brightness of the image sequence.
The invention is to adapt to the situation that the whole input image sequence is darker or brighter, and the weight of the dark or bright part is self-adaptive according to the whole average brightness of the input image sequence.
Preferably, the method for adaptively adjusting the exposure weight of the dark part or the bright part according to the normalized average brightness of the image sequence specifically comprises the following steps:
when the normalized average luminance is smaller than the good exposure pixel value m, the exposure weight is:
when the normalized average luminance is greater than the good exposure pixel value m, the exposure weight is:
when the normalized average luminance is equal to the good exposure pixel value m, the exposure weight is:
wherein m represents a good exposure pixel value, and the highest weight is given when the normalized pixel value of the image sequence is m, and m is [0,1 ]]，I n (i, j) represents the pixel value at the (i, j) position after normalization of the nth image,
the pixel value of the N-th image at the (i, j) position after the gray level map normalization is represented, the value range of N is 1,2, …, N, N represents the number of input image sequences, alpha represents the global brightness adaptive factor, and lambda represents the relative brightness adaptive factor.
According to the good exposure pixel value m; when the average brightness of all image sequences is less than m, this means that the input image is entirely dark, and the weight of the dark portion is appropriately increased, and the weight of the bright portion is appropriately decreased, so that the information of the dark portion is better preserved. When the average brightness of all image sequences is greater than m, this means that the input image is overall bright, and the weight of the bright portion is appropriately increased, and the weight of the dark portion is appropriately decreased, so that the information of the bright portion is better preserved.
Preferably, the global luminance adaptation factor α reflects the offset between the global exposure and the good exposure pixel value m of the image sequence, which is defined as follows:
wherein h and w represent the length and width of the image, respectively;
in addition, when the difference in brightness between one image in the input image sequence and its neighboring exposure image is large, more good exposure pixels are generally contained, and therefore, the present invention proposes a relative brightness adaptation factor λ.
The relative brightness adaptation factor λ is used to adjust the normalized value near the range where the good exposure m pixel value weights 1, which is defined as follows:
wherein mean (I n ) The average luminance of the nth input image is represented, μ represents a fixed parameter, and μ has a value of 0.25.
The invention comprehensively considers the integral brightness of the image sequence and the relative brightness between the adjacent images, and can adaptively adjust the weight of the dark part or the bright part of the image sequence according to the input image by utilizing the self-adaptive factor. The adaptation to the overall darkness or brightness of the input image is improved to obtain a better appearance and more image details.
Preferably, the laplacian pyramid decomposes the image sequence into a base layer and a detail layer, wherein the base layer is used for capturing low-frequency information, and the detail layer is used for capturing high-frequency information; the laplacian pyramid has L layers, L being given by:
where h and w represent the length and width of the image, respectively,
representing return of less than or equal to log 2 An integer of min (h, w).
Preferably, a gaussian pyramid of a luminance component of an image sequence is used as a guide pyramid, and the initial weight map is subjected to weighted guide filtering processing to obtain a weight map pyramid, which specifically comprises:
using G { Y } as a guide pyramid by using a Gaussian pyramid of the luminance component of the image sequence n } (l) A representation;
decomposing the initial weight map into L-layer Gaussian pyramids with G { W } n } (l) A representation;
g { Y } is filtered based on weighted guidance n } (l) Is transferred to G { W ] n } (l) The initial weight map is subjected to preliminary smoothing treatment, and the specific algorithm is as follows:
wherein L is more than or equal to 0 and less than or equal to L, L represents the layer number of the Gaussian pyramid,
and->
Respectively representing the coefficients of the weighted guided filtering, +.>
And a weight map pyramid which is subjected to the primary smoothing treatment by the weighted guided filtering is represented.
In order to reduce the calculation cost, the invention only calculates
And->
And +.>
And->
Two-layer coefficients, other layers, if the value of l is less than 4,/>
And->
By->
And->
Is generated layer by upsampling interpolation of (a) if the value of l is greater than 4, the remaining layers are generated by +.>
And->
Is achieved and fixes the radius ζ and regularization parameter λ to 2 and 1/1024, respectively.
Preferably, constructing an adaptive edge preserving smooth pyramid of a weight map using detail layers of a laplacian pyramid of an image sequence and coefficients of weighted guided filtering specifically includes:
introducing the weighted guide filtering coefficient into the detail layer refinement weight map pyramid of the Laplacian pyramid, and constructing a self-adaptive edge-preserving smooth pyramid of the weight map, wherein the specific algorithm is as follows:
wherein,,
representing the use of a Gaussian smoothing filter pair +.>
Smoothing to ensure consistency and attenuate halation; l 1 {L{I n } (l) The high-frequency information of Laplacian pyramid, namely the edge between the abnormal exposure area and the normal exposure area, can be used for correcting the non-Gaussian smoothingProper weighting ensures that the weighting of proper exposed areas is not affected by smoothing; />
Coefficients representing weighted guided filtering, in the above formula,/->
Is also |L 1 {L{I n } (l) Coefficient of |, which controls the magnitude of the high frequency signal, +.>
Gradient retention capacity is determined at the border +.>
The gradient retention effect is good if the gradient is larger; in the flat area->
Smaller, the smoothing effect is good.
Preferably, the detailed weight graph pyramid and the Laplacian pyramid are reconstructed, and a specific algorithm for obtaining the fusion image is as follows:
wherein L { F (i, j) } is (l) Laplacian pyramid image representing layer I, L { I } n (i,j)} (l) Representing the pixel value at the first layer position (i, j) of the n-th input image laplacian pyramid, AES { W n (i,j)} (l) The adaptive edges representing the weight map remain smooth pyramids, and then fusion results are obtained by inverse laplace transform.
Preferably, local contrast measurements on the image sequence are used to preserve important detail information of the source image, such as texture and edges. These texture and edge information are contained in the gradient changes. The invention uses dense scale invariant feature (scale invariant feature Transform, SIFT) descriptors to extract local contrast information from a gray scale image of a source image, and the specific algorithm is as follows:
wherein C is n (i, j) represents a simple local contrast measurement index,
representation l 1 Norms (F/F)>
Representing an operator used to compute a non-normalized dense SIFT map of the input image. To better utilize memory, in each cell, a descriptor is generated using an 8bin direction histogram and a 2 x 2 array of cells. As previously described, for activity level measurement of the relevant pixel, a total number of a significant number of elements in the non-standardized descriptor is used. Since each element is not negative in the SIFT descriptor, at each pixel, the gray-scale map +.>
Mapping l of (2) 1 Norm as C n (i, j). Then, a weight distribution strategy that winners eat is adopted, namely, the local contrast weight is calculated by the maximum value of the same pixel position in all images:
wherein the method comprises the steps of
Local contrast weight values representing the position of the nth sheet (i, j).
Preferably, calculating the saturation weight specifically includes: as the exposure time of the photograph is prolonged, the resulting color becomes unsaturated and the visual effect is poor. Saturated color is theoryImagine and make the image look vivid. The standard deviation of each pixel in the R, G and B channels of each pixel is used as a measured value S, and the standard deviation of each pixel in the R, G and B channels is calculated to obtain the color saturation weight
Preferably, an initial weight map of the image sequence is obtained according to the improved exposure weight, the local contrast weight and the saturation weight, and normalization processing is performed on the initial weight map:
wherein,,
indicating improved exposure weight, +.>
Representing local contrast weight, +.>
Representing color saturation weights; />
Representing initial weights derived using improved exposure weights, local contrast weights, and color saturation weights; w (W) n (i, j) represents an initial weight map after normalization processing.
Compared with the prior art, the invention provides an image fusion method based on a self-adaptive edge-preserving smooth pyramid, which comprises the following steps: acquiring an image sequence; calculating an improved exposure weight according to the overall brightness of the image sequence and the relative brightness between adjacent images; extracting local contrast information from a gray image of a source image by using a dense scale invariant feature descriptor, and calculating local contrast weight according to the maximum value of the same pixel position in all the images; calculating a saturation weight using R, G and the standard deviation of each pixel in the B channel; obtaining an initial weight map of the image sequence according to the improved exposure weight, the local contrast weight and the saturation weight; taking a Gaussian pyramid of a brightness component of an image sequence as a guide pyramid, and carrying out weighted guide filtering processing on the initial weight graph to obtain a weight graph pyramid; constructing a self-adaptive edge retaining smooth pyramid of the weight map by utilizing a detail layer of the Laplacian pyramid of the image sequence and a coefficient of the weighted guide filtering; and reconstructing the self-adaptive edge-retaining smooth pyramid and the Laplacian pyramid to obtain a fusion image.
Compared with the existing scheme, the scheme provided by the invention has the following advantages: consider the situation that the whole of the input picture sequence is darker or lighter because of the information in the space field and the more natural light-dark transition. According to the invention, the fusion weight of each pixel point is calculated by improving the exposure evaluation weight function and combining the local contrast weight and the color saturation weight, the self-adaptive edge is provided to maintain a smooth pyramid to refine the weight graph, and finally, a high-quality fusion image is generated, the details of the darkest and brightest areas are better kept, and meanwhile, the generation of halation artifacts can be effectively inhibited, so that the method has a better visual effect.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are required to be used in the embodiments or the description of the prior art will be briefly described below, and it is obvious that the drawings in the following description are only embodiments of the present invention, and that other drawings can be obtained according to the provided drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flow chart of an image fusion method based on an adaptive edge preserving smooth pyramid;
FIG. 2 is a graph of an improved exposure weight function provided by an embodiment of the present invention;
FIG. 3 is a sequence of input multi-exposure images provided by an embodiment of the present invention;
FIG. 4 is a partial enlarged view of the final fused image result provided by the embodiment of the present invention;
fig. 5 is a partial enlarged view of the result obtained by using 4 image fusion algorithms in the prior art.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
The embodiment of the invention discloses an image fusion method based on improved exposure weight and a smooth pyramid, which is used for acquiring an image sequence; as shown in fig. 1, the image sequence is an LDR image sequence;
calculating an improved exposure weight according to the overall brightness of the image sequence and the relative brightness between adjacent images;
extracting local contrast information from a gray image of a source image by using a dense scale invariant feature descriptor, and calculating local contrast weight according to the maximum value of the same pixel position in all the images;
calculating a saturation weight using R, G and the standard deviation of each pixel in the B channel;
obtaining an initial weight map of the image sequence according to the improved exposure weight, the local contrast weight and the saturation weight;
taking a Gaussian pyramid of a brightness component of an image sequence as a guide pyramid, and carrying out weighted guide filtering processing on the initial weight graph to obtain a weight graph pyramid;
constructing a self-adaptive edge retaining smooth pyramid of the weight map by utilizing a detail layer of the Laplacian pyramid of the image sequence and a coefficient of the weighted guide filtering;
and reconstructing the self-adaptive edge-retaining smooth pyramid and the Laplacian pyramid to obtain a fusion image.
Further refining the weight map pyramid by using an adaptive edge preserving smooth pyramid, and introducing a detail layer of the Laplacian pyramid of the image sequence by using a coefficient of weighted guided filtering to refine the weight map pyramid, so as to construct the adaptive edge preserving smooth pyramid of the weight map;
specifically, the embodiment of the invention provides an improved exposure weight algorithm, which comprehensively considers the integral brightness of an image sequence and the relative brightness between adjacent images; the self-adaptive factor is utilized to self-adaptively adjust the weight of the dark part or the bright part of the image sequence according to the input image, so that the adaptability of the algorithm to the whole darkness or the brightness of the input image is improved, and better appearance and more image details are obtained.
In the embodiment of the invention, m=0.5, namely, the pixel with the normalized value of about 0.5 is endowed with the highest weight, the pixel far away from 0.5 is endowed with the lower weight, and the exposure weight is as follows:
wherein the method comprises the steps of
The pixel value of the (i, j) position after the normalization of the gray level diagram representing the nth image, the value range of N is 1,2, …, and N are the number of input image sequences; the above equation is an exposure weight when the overall brightness and the relative brightness are not considered. In order to better adapt to the situation that the whole of the input image is darker or the whole of the input image is brighter, the embodiment firstly adjusts the weight of the dark or bright part in a self-adaptive way according to the whole average brightness of the input image sequence, and calculates the average brightness of all the input image sequences. If the average brightness is less than 0.5, this means that the input image is entirely dark. At this time, the weight of the dark part is properly increased, and the weight of the bright part is properly reduced, so that the information of the dark part is better kept. If averageThe luminance is greater than 0.5, which means that the input image is overall bright, and the weight of the bright portion is appropriately increased and the weight of the dark portion is appropriately decreased. The present embodiment uses a global luminance adaptation factor α, which can reflect the offset between the overall exposure of the input image and 0.5, defined as follows:
where h, w denote the length and width of the image, respectively. In addition, when the luminance difference between one image in the input image sequence and its neighboring exposure image is large, more good exposure pixels are generally contained, and therefore, a relative luminance adaptive factor λ is proposed that adjusts the normalized value to be close to the range where the 0.5 pixel weight is 1.
Wherein mean (I n ) Represents the average luminance of the nth input image, and μ represents a fixed parameter. .
In summary, when the normalized average brightness is less than 0.5, the final exposure weight is:
when the normalized average brightness is greater than 0.5, the final exposure weight is
As shown in fig. 2, the dashed line indicates that the average luminance of all the input image sequences is less than 0.5, and the solid line indicates that the average luminance of all the input images is greater than 0.5. When the normalized average luminance is equal to 0.5, the exposure weight takes the original form.
Specifically, the embodiment of the invention provides a local contrast weight algorithm, and local contrast measurement is performed on an input image sequence to keep important detail information of a source image, such as texture and edge, and the like, wherein the texture and edge information is contained in gradient changes. The embodiment uses dense scale invariant feature (scale invariant feature Transform, SIFT) descriptors to extract local contrast information from a gray scale image of a source image, and the specific algorithm is as follows:
wherein C is n (i, j) is a simple local contrast measurement index,
representation l 1 Norms (F/F)>
Representing an operator used to compute a non-normalized dense SIFT map of the input image. To better utilize memory, in each cell, a descriptor is generated using an 8bin direction histogram and a 2 x 2 array of cells. As previously described, for activity level measurement of the relevant pixel, a total number of a significant number of elements in the non-standardized descriptor is used. Since each element is not negative in the SIFT descriptor, at each pixel, the gray-scale map +.>
Mapping l of (2) 1 Norm as C n (i, j). Then, a weight distribution strategy that winners eat is adopted, namely, the local contrast weight is calculated by the maximum value of the same pixel position in all images:
wherein the method comprises the steps of
Local contrast weight values representing the position of the nth sheet (i, j).
Specifically, the embodiment of the invention provides a method for calculating saturation weight, and as the exposure time of a photo is prolonged, the generated color becomes unsaturated, the visual effect is poor, the saturated color is ideal, and the image looks more vivid. R, G for each pixel in the image sequence and the standard deviation in the B channel are taken as measured values S, and R, G and the standard deviation of each pixel in the B channel are calculated to obtain color saturation weights
The specific algorithm is as follows:
wherein,,
pixel value, gamma, at (i, j) position after normalization of R, G, B three channels representing nth image n (i,j) R,G,B Representing the average of the three channels R, G, B of the nth input image at (i, j), the definition is as follows:
wherein,,
pixels representing a function of the three channels R, G, B of the image at (i, j); k represents three color channels of R, G, and B.
Specifically, the three metrics described above (improved exposure weight, local contrast weight, and color saturation weight) represent the contribution of each pixel of the input to the final result. Constructing an initial weight graph from the three calculated measurement indexes, and carrying out normalization processing on the initial weight graph:
wherein,,
indicating improved exposure weight, +.>
Representing local contrast weight, +.>
Representing color saturation weights; />
Representing initial weights derived using improved exposure weights, local contrast weights, and color saturation weights; w (W) n (i, j) represents an initial weight map after normalization processing.
The initial weight map from the above equation is noisy, discontinuous, and if it is used directly for fusion operations, unsatisfactory problems such as seams and noticeable halation artifacts are likely to occur. Therefore, it is important to smooth the denoising before fusion processing using these weight maps. To solve this problem, a multi-resolution method may be used, but this method does not preserve the details of the brightest or darkest areas well and risks creating halation artifacts. Edge preserving filters have been widely used for enhancement detail and multi-exposure image fusion. The embodiment of the invention adopts an edge-preserving smoothing technology to improve the Gaussian pyramid of the weight graph. The weight map and the LDR image sequence are respectively decomposed into a Gaussian pyramid of an L layer and a Laplacian pyramid of the L layer, wherein the Laplacian pyramid decomposition decomposes an input image into a base layer and a detail layer, the highest layer is the base layer, low-frequency information (global color information of the image) is captured, the other layers are detail layers, and high-frequency information (image edges and detail information) is captured. L is given by:
wherein the method comprises the steps of
Representing return of less than or equal to log 2 min (h, w) is the nearest integer, min (h, w) represents a minimum function. From the above equation, the number of pyramid layers is 2 less than that of the conventional pyramid, because appropriate reduction of the number of pyramid layers can reduce the details lost in the process of decomposing and reconstructing the pyramid, but the problems of halation artifacts are caused. By carefully looking at the weight map pyramid, it was found that improper smoothing around the edges is the main cause of halation artifacts. For example, over-exposed areas along the boundaries of normally exposed areas tend to get higher weights by improper smoothing, which leads to halo artifacts around the edges.
Unlike using Gaussian pyramid G { W n } (l) To fuse the differently exposed images, embodiments of the present invention propose a novel adaptive edge preserving smooth pyramid to fuse them. Embodiments of the present invention are based on weighted guided filtering (WGIF). First, decomposing the weight map into Gaussian pyramids, G { W ] n } (l) Is a weight graph pyramid to be smoothed, and a pyramid of luminance components of an input LDR image sequence is used as a guide pyramid (G { Y n } (l) A gaussian pyramid that is the luminance component of the input image). The proposed pyramid is based on an observation that WGIF can convert G { Y } n } (l) Is transferred to G (W) n } (l) . Let the coefficients of WGIF be expressed as
And->
In order to reduce the calculation cost, the embodiment of the invention only calculates +.>
And->
And
And->
If the value of l is smaller than 4, the other layers are +.>
And->
By->
And->
Is generated layer by upsampling interpolation of (a) if the value of l is greater than 4, the remaining layers are generated by +.>
And->
Is achieved and fixes the radius ζ and regularization parameter λ to 2 and 1/1024, respectively. The WGIF-based pyramid is then given as follows:
smoothed by WGIF
Details cannot be well preserved and there are problems with halation artifacts due to improper smoothing around the high-level edges. The halo artifact is eliminated by smoothing the improper weights to a wider area so that it is invisible, using a gaussian smoothing filter for each layer to attenuate the halo artifact, but smoothing reduces the weights of the properly exposed areas, resulting in a loss of detail in the bright areas. To solve this problem, embodiments of the present invention introduce a Laplacian pyramid detail layer of the input image to refine the weight map, using coefficients of weighted guided filtering +.>
To construct an adaptive edge preserving smooth pyramid of the weight graph. The specific algorithm is as follows:
Finally, the invention utilizes pyramid fusion to obtain a final fusion image, and the specific algorithm is as follows:
wherein L { F (i, j) } is (l) Laplacian pyramid image representing layer I, L { I } n (i,j)} (l) Representing the pixel value at the first layer position (i, j) of the n-th input image Laplacian pyramid, AES { W n (i,j)} (l) The adaptive edges representing the weight map remain pyramids.
FIG. 3 is a sequence of multiple exposure images input according to an embodiment of the present invention, and FIG. 4 is a final fused image result and a partial enlarged view according to an embodiment of the present invention; FIG. 5 is a partial enlarged view of the results obtained using 4 image fusion algorithms in the prior art; as can be seen from the partial enlarged views of fig. 4 and 5, the brightest portion of the cloud in fig. 4 is more clearly defined, and the halation is effectively suppressed at the edge of the hot air balloon. The method provided by the invention effectively inhibits the generation of halation artifact while better maintaining the details of the brightest or darkest area, and has better visual effect. The invention builds a self-adaptive edge-keeping smooth pyramid of the weight map by the thought of integral brightness adaptation of the input image sequence and the introduction of the detail layer refinement weight map, and effectively inhibits the generation of halation artifacts while better keeping the details of the darkest/brightest areas.
Compared with the existing scheme, the scheme provided by the invention has the following advantages: the invention considers the situation that the information in the space field and the light-shade transition are more natural and the whole input picture sequence is dark or bright. The method is characterized in that an improved exposure weight algorithm is innovatively provided, then the fusion weight of each pixel point is calculated by combining the local contrast weight and the color saturation weight, the weight map is thinned by innovatively providing a self-adaptive edge-preserving smooth pyramid, and finally, a high-quality fusion image is generated, so that the details of darkest and brightest areas can be better kept, and the generation of halation artifacts can be effectively restrained.
In the present specification, each embodiment is described in a progressive manner, and each embodiment is mainly described in a different point from other embodiments, and identical and similar parts between the embodiments are all enough to refer to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant points refer to the description of the method section.
The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (9)
1. An image fusion method based on an adaptive edge preserving smooth pyramid, comprising:
acquiring an image sequence;
calculating an improved exposure weight according to the overall brightness of the image sequence and the relative brightness between adjacent images;
extracting local contrast information from a gray image of a source image by using a dense scale invariant feature descriptor, and calculating local contrast weight according to the maximum value of the same pixel position in all the images;
calculating a saturation weight using R, G and the standard deviation of each pixel in the B channel;
obtaining an initial weight map of the image sequence according to the improved exposure weight, the local contrast weight and the saturation weight;
taking a Gaussian pyramid of a brightness component of an image sequence as a guide pyramid, and carrying out weighted guide filtering processing on the initial weight graph to obtain a weight graph pyramid;
constructing a self-adaptive edge retaining smooth pyramid of the weight map by utilizing a detail layer of the Laplacian pyramid of the image sequence and a coefficient of the weighted guide filtering;
and reconstructing the self-adaptive edge-retaining smooth pyramid and the Laplacian pyramid to obtain a fusion image.
2. The image fusion method based on the adaptive edge preserving smooth pyramid according to claim 1, wherein calculating the improved exposure weight specifically comprises:
carrying out graying treatment on the image sequence to obtain a graying image;
normalizing the graying image;
acquiring a pixel value of the normalized gray image;
calculating normalized average brightness of the image sequence according to the pixel value;
and adjusting the exposure weight of the dark part or the bright part in a self-adaptive mode according to the normalized average brightness of the image sequence.
3. The image fusion method based on the adaptive edge preserving smooth pyramid according to claim 2, wherein the method adaptively adjusts the exposure weight of the dark or bright part according to the normalized average brightness of the image sequence, specifically comprises:
when the normalized average luminance is smaller than the good exposure pixel value m, the exposure weight is:
when the normalized average luminance is greater than the good exposure pixel value m, the exposure weight is:
when the normalized average luminance is equal to the good exposure pixel value m, the exposure weight is:
wherein m represents a good exposure pixel value, and when the normalized pixel value of the image sequence is close to m, the highest weight is given, and m is [0,1 ]]，I n (i, j) represents the pixel value at the (i, j) position after normalization of the nth image,
the pixel value of the N-th image at the (i, j) position after the gray level map normalization is represented, the value range of N is 1,2, …, N, N represents the number of input image sequences, alpha represents the global brightness adaptive factor, and lambda represents the relative brightness adaptive factor.
4. A method of image fusion based on an adaptive edge preserving smooth pyramid as claimed in claim 3, characterized in that the global luminance adaptation factor α reflects the offset between the global exposure of the image sequence and the good exposure pixel value m, defined as follows:
wherein h and w represent the length and width of the image, respectively;
the relative brightness adaptation factor λ is used to adjust the normalized value near the range where the good exposure m pixel value weights 1, which is defined as follows:
wherein mean (I n ) Represents the average luminance of the nth input image, and μ represents a fixed parameter.
5. An image fusion method based on an adaptive edge preserving smooth pyramid as claimed in claim 1, wherein the laplacian pyramid decomposes the image sequence into a base layer for capturing low frequency information and a detail layer for capturing high frequency information; the laplacian pyramid has L layers, L being given by:
where h and w represent the length and width of the image, respectively,
representing return of less than or equal to log 2 An integer of min (h, w).
6. The image fusion method based on the adaptive edge preserving smooth pyramid according to claim 5, wherein a gaussian pyramid of a luminance component of an image sequence is used as a guide pyramid, and the initial weight map is subjected to weighted guide filtering processing to obtain a weight map pyramid, which specifically comprises:
using G { Y } as a guide pyramid by using a Gaussian pyramid of the luminance component of the image sequence n } (l) A representation;
decomposing the initial weight map into L-layer Gaussian pyramids with G { W } n } (l) A representation;
g { Y } is filtered based on weighted guidance n } (l) Is transferred to G { W ] n } (l) The initial weight map is subjected to preliminary smoothing treatment, and the specific algorithm is as follows:
wherein L is more than or equal to 0 and less than or equal to L, L represents the layer number of the Gaussian pyramid,
and->
Respectively representing the coefficients of the weighted guided filtering, +.>
And a weight map pyramid which is subjected to the primary smoothing treatment by the weighted guided filtering is represented.
7. The image fusion method based on improved exposure weights and smoothing pyramids as claimed in claim 6, wherein constructing the adaptive edge preserving smoothing pyramids of the weight map using the detail layers of the laplacian pyramids of the image sequence and the coefficients of the weighted guided filtering, specifically comprises:
introducing the weighted guide filtering coefficient into the detail layer refinement weight map pyramid of the Laplacian pyramid, and constructing a self-adaptive edge-preserving smooth pyramid of the weight map, wherein the specific algorithm is as follows:
wherein,,
representing the use of a Gaussian smoothing filter pair +.>
Smoothing, |L 1 {L{I n } (l) 'representing Laplacian' gold wordHigh frequency information of tower->
Representing the coefficients of the weighted guided filtering.
8. The image fusion method based on improved exposure weight and smooth pyramid according to claim 1, wherein the specific algorithm for reconstructing the adaptive edge-preserving smooth pyramid and the laplacian pyramid to obtain the fused image is as follows:
wherein L { F (i, j) } is (l) Laplacian pyramid image representing layer I, L { I } n (i,j)} (l) Representing the pixel value at the first layer position (i, j) of the n-th input image laplacian pyramid, AES { W n (i,j)} (l) The self-adaptive edge of the weight graph keeps a pyramid, and then a fusion result is obtained through inverse Laplacian transformation.
9. The image fusion method based on improved exposure weights and a smooth pyramid according to claim 1, wherein the initial weight map is normalized before being subjected to weighted guided filtering:
wherein,,
representation ofImproving exposure weight->
Representing local contrast weight, +.>
Representing color saturation weights; />
Representing initial weights derived using improved exposure weights, local contrast weights, and color saturation weights; w (W) n (i, j) represents an initial weight map after normalization processing.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310423665.2A CN116416175A (en) | 2023-04-20 | 2023-04-20 | Image fusion method based on self-adaptive edge-preserving smooth pyramid |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310423665.2A CN116416175A (en) | 2023-04-20 | 2023-04-20 | Image fusion method based on self-adaptive edge-preserving smooth pyramid |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116416175A true CN116416175A (en) | 2023-07-11 |
Family
ID=87059431
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310423665.2A Pending CN116416175A (en) | 2023-04-20 | 2023-04-20 | Image fusion method based on self-adaptive edge-preserving smooth pyramid |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116416175A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117939307A (en) * | 2024-03-19 | 2024-04-26 | 四川辰宇微视科技有限公司 | Self-adaptive brightness adjusting method suitable for fusion camera |
-
2023
- 2023-04-20 CN CN202310423665.2A patent/CN116416175A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117939307A (en) * | 2024-03-19 | 2024-04-26 | 四川辰宇微视科技有限公司 | Self-adaptive brightness adjusting method suitable for fusion camera |
| CN117939307B (en) * | 2024-03-19 | 2024-06-04 | 四川辰宇微视科技有限公司 | Self-adaptive brightness adjusting method suitable for fusion camera |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107845128B (en) | Multi-exposure high-dynamic image reconstruction method with multi-scale detail fusion | |
| CN112734650B (en) | Virtual multi-exposure fusion based uneven illumination image enhancement method | |
| CN111986120A (en) | Low-illumination image enhancement optimization method based on frame accumulation and multi-scale Retinex | |
| CN109767413B (en) | HDR method and device for resisting motion artifacts and portable terminal | |
| CN112767289A (en) | Image fusion method, device, medium and electronic equipment | |
| CN110189281B (en) | Multi-exposure infrared image fusion method | |
| WO2016139260A9 (en) | Method and system for real-time noise removal and image enhancement of high-dynamic range images | |
| JPWO2009072537A1 (en) | Image processing apparatus and method, program, and recording medium | |
| Lee et al. | Noise reduction and adaptive contrast enhancement for local tone mapping | |
| CN110852982A (en) | Self-adaptive exposure adjustment multi-scale entropy fusion underwater image enhancement method | |
| CN110675351B (en) | Marine image processing method based on global brightness adaptive equalization | |
| US8873880B2 (en) | Detection and/or enhancement of contrast differences in digital image data | |
| KR20120114899A (en) | Image processing method and image processing apparatus | |
| CN114862698B (en) | Channel-guided real overexposure image correction method and device | |
| CN111242860A (en) | Super night scene image generation method and device, electronic equipment and storage medium | |
| CN113379609B (en) | Image processing method, storage medium and terminal equipment | |
| CN116416175A (en) | Image fusion method based on self-adaptive edge-preserving smooth pyramid | |
| US9466007B2 (en) | Method and device for image processing | |
| Muniraj et al. | Underwater image enhancement by modified color correction and adaptive Look-Up-Table with edge-preserving filter | |
| CN110246086B (en) | Image processing method for color image enhancement and image processing system thereof | |
| CN111325680B (en) | Image brightening method capable of inhibiting local overexposure | |
| CN117830134A (en) | Infrared image enhancement method and system based on mixed filtering decomposition and image fusion | |
| CN116579953A (en) | Self-supervision water surface image enhancement method and related equipment | |
| CN114638764B (en) | Multi-exposure image fusion method and system based on artificial intelligence | |
| JP4664938B2 (en) | Image processing apparatus, image processing method, and program |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |