CN116309126B - Five-band multispectral image reconstruction method based on autoregressive model - Google Patents
Five-band multispectral image reconstruction method based on autoregressive model Download PDFInfo
- Publication number
- CN116309126B CN116309126B CN202310121777.2A CN202310121777A CN116309126B CN 116309126 B CN116309126 B CN 116309126B CN 202310121777 A CN202310121777 A CN 202310121777A CN 116309126 B CN116309126 B CN 116309126B
- Authority
- CN
- China
- Prior art keywords
- pixel
- band
- image
- original image
- unknown
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 36
- 238000001914 filtration Methods 0.000 claims abstract description 16
- 230000003287 optical effect Effects 0.000 claims description 25
- 239000011159 matrix material Substances 0.000 claims description 12
- 230000003595 spectral effect Effects 0.000 claims description 11
- 238000000701 chemical imaging Methods 0.000 claims description 4
- 230000000694 effects Effects 0.000 claims description 3
- 230000009466 transformation Effects 0.000 claims description 3
- 239000007787 solid Substances 0.000 description 7
- 238000001228 spectrum Methods 0.000 description 7
- 238000004422 calculation algorithm Methods 0.000 description 4
- 238000005070 sampling Methods 0.000 description 4
- 230000003044 adaptive effect Effects 0.000 description 3
- 238000010586 diagram Methods 0.000 description 3
- 238000011156 evaluation Methods 0.000 description 2
- 238000003384 imaging method Methods 0.000 description 2
- 238000010606 normalization Methods 0.000 description 2
- 239000012788 optical film Substances 0.000 description 2
- 230000000737 periodic effect Effects 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000002059 diagnostic imaging Methods 0.000 description 1
- 239000010432 diamond Substances 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000007747 plating Methods 0.000 description 1
- 238000004088 simulation Methods 0.000 description 1
- 239000000126 substance Substances 0.000 description 1
- 230000007704 transition Effects 0.000 description 1
- 238000002834 transmittance Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/70—Denoising; Smoothing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/10—Image enhancement or restoration using non-spatial domain filtering
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/20—Image enhancement or restoration using local operators
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/73—Deblurring; Sharpening
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20172—Image enhancement details
- G06T2207/20192—Edge enhancement; Edge preservation
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02A—TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE
- Y02A40/00—Adaptation technologies in agriculture, forestry, livestock or agroalimentary production
- Y02A40/10—Adaptation technologies in agriculture, forestry, livestock or agroalimentary production in agriculture
Landscapes
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Image Processing (AREA)
Abstract
The invention discloses a five-band multispectral image reconstruction method based on an autoregressive model, belongs to the technical field of multispectral image processing, and aims to solve the problems that the reconstruction precision of a guide image is insufficient, so that noise and edge artifacts exist in the reconstructed images of other bands in the prior art; acquiring a B-band original image, a C-band original image, a G-band original image, a Y-band original image and an R-band original image; modeling the correlation among the pixel points in the G-band original image in the horizontal-vertical direction by using an autoregressive model, introducing an error feedback model, and obtaining an optimal estimated value of an unknown pixel point in the G-band original image by minimizing the estimated error sum of all the pixel points in a local window to finish the reconstruction of the G-band original image; and taking the reconstructed G wave band image as a guide image, and respectively reconstructing the original images of the four wave bands B, C, Y, R by using guide filtering.
Description
Technical Field
The invention relates to a five-band multispectral image reconstruction method based on an autoregressive model, and belongs to the technical field of multispectral image processing.
Background
Multispectral images have more complete spatial and spectral information than RGB images, and are widely used in the fields of medical imaging, food detection, face recognition and the like. The multispectral image is obtained by multispectral filter array (Multispectral Filter Array, MSFA) imaging, and the technology can obtain the spectral information of all substances through one exposure by placing the filter array with mosaic mode at the front end of the image sensor. The image directly obtained from the filter array is referred to as the original image, and the process of estimating the remaining band loss information at each sampling point is referred to as the multispectral image reconstruction process. Compared with a color filter array (Color Filter Array, CFA), MSFA has lower spatial correlation between pixels in the same band due to the increase of the number of bands, a great amount of detail information is lost in the reconstruction process, and the conventional CFA image reconstruction method cannot be directly applied to the MSFA image reconstruction problem due to the great difference of spatial and spectral characteristics between multispectral images and RGB images, so the multispectral image reconstruction problem is a difficulty in recent years.
The interpolation algorithm has the advantages of low complexity, high calculation speed and the like, and is very suitable for solving the problem of multi-spectrum image reconstruction. The article "Multispectral demosaicking using guided filter" published in 2012 by Monno Y, tanaka M, okutomi M on the SPIE journal, digital Photography. The technical gist of the article is that the adaptive Gaussian upsampling is used to generate an effective guide image from the most densely sampled spectral components in MSFA, and the rest of wave bands are interpolated in combination with the guide filtering to complete the reconstruction of the multispectral image, which makes full use of the spatial correlation and spectral correlation between different pixels. The problem with this technique is that the accuracy of the guide image obtained based on the adaptive gaussian upsampling method is still insufficient, and there is a significant artifact near the edge, which severely distorts the reconstructed image in the remaining bands.
Disclosure of Invention
The invention provides a five-band multispectral image reconstruction method based on an autoregressive model, which aims to solve the problems that the reconstruction precision of a guide image is insufficient and noise and edge artifacts exist in the reconstructed images of other bands in the prior art.
The technical scheme for solving the technical problems is as follows:
the five-band multispectral image reconstruction method based on the autoregressive model comprises the following steps:
step one, designing a five-band snapshot type multispectral imaging system;
step two, acquiring a B-band original image, a C-band original image, a G-band original image, a Y-band original image and an R-band original image;
thirdly, selecting a local neighborhood window T with the size of 7 multiplied by 7 by taking each unknown pixel point in the G-band original image as a center, and renumbering pixels in the neighborhood;
modeling the correlation among the pixel points in the window T in the horizontal-vertical direction by using an autoregressive model;
fifthly, calculating model parameters of the unknown pixels by using a Gaussian gradient function between the intensities of the known pixel points;
step six, calculating a preliminary estimated value of the unknown pixel, and calculating model parameters of the known pixel by using the preliminary estimated value;
step seven, minimizing the total estimated error of all pixel points in the window T to obtain an unknown pixel point y in the G-band original image j Optimal estimate of (2)Completing the reconstruction of the G band original image;
and step eight, taking the reconstructed G wave band image as a guide image, and respectively reconstructing original images of four wave bands of B, C, Y, R by using guide filtering.
The system designed in the first step comprises an optical lens, an optical filter array, a photoelectric detector and a computer, wherein incident light enters the optical lens, is converged on the optical filter array through the optical lens, and the optical filter array carries out spectrum splitting on the broadband incident light to divide the broadband incident light into five wave bands of narrow wave band light respectively: each pixel in the filter array only passes through a single narrow-band optical signal, and emergent light split by the filter array is received by a photoelectric detector, the photoelectric detector converts the optical signal into an electric signal and transmits the electric signal to a computer, and the computer converts the electric signal into an image signal.
And step two, the electric signal output by the photoelectric detector is transmitted to a computer and converted into an image signal by the computer, wherein the image contains all information of five wave bands, and an original image of the corresponding wave band, namely a single-wave-band original image of B, C, G, Y, R wave bands, is obtained by downsampling the wave band pixels according to the corresponding spatial position of each wave band pixel in the optical filter array.
Selecting a local neighborhood window T with the size of 7 multiplied by 7 by taking each unknown pixel point in the G-band original image as the center, renumbering pixels in the neighborhood, and representing an unknown pixel set as { y } j |j=1, 2,..25 }, the known set of pixels is denoted as { x } i |i＝1,2,...,24}。
Modeling the correlation between the pixel points in the window T in the horizontal-vertical direction by using an autoregressive model, wherein the model comprises a model for estimating an unknown pixel by using a known pixel by using the correlation and a feedback model for calculating the error between the estimated unknown pixel and the known pixel, and the two models are expressed as follows:
wherein x is j,t For unknown pixel y j A known pixel adjacent in the t-th direction in the four neighbors, y i,t Is the known pixel x i Unknown pixels adjacent in the t-th direction in the four neighbors, b j,t 、b i,t Respectively represent x j,t ，y i,t Corresponding autoregressive model parameters, t=1, 2,3,4, corresponding to the four directions up, right, down, left, γ j And gamma i Respectively are provided withIs the estimated error of the two models.
Step five, calculating model parameters of the known pixels in the four adjacent areas of the unknown pixels by using a Gaussian gradient function between the intensities of the known pixels, wherein the unknown pixels are y j Is known in the t-th direction of pixel x j,t Model parameter b of (2) j,t The expression is as follows:
in sigma j Is the pixel y to be estimated j The standard deviation between known pixels in the four neighbors, t=1, 2,3,4, respectively represents the up, right, down, left directions.
Step six, calculating a preliminary estimated value of the unknown pixel, and calculating model parameters of the unknown pixel in the four adjacent areas of the known pixel by using the preliminary estimated value; calculating an unknown pixel y using (1) j Preliminary estimated value of (2)The expression is as follows:
wherein x is j,t Representing unknown pixel y j A known pixel in the t-th direction in the four neighborhoods, b j,t For its model parameters, t=1, 2,3,4, respectively representing the up, right, down, left four directions;
according to the known pixel x i And the preliminary estimated value of the unknown pixels in the four neighborhoodsSimilarity between known pixels x i Model parameters b in the t-th direction i,t The expression is as follows:
in sigma i Is the known pixel point x i Standard deviation of pixels in eight neighborhoods, t=1, 2,3,4, respectively representing four directions of up, right, down and left; after all weight parameters of each pixel point are calculated, normalization is carried out on the weight parameters, and the sum of the weights is guaranteed to be 1.
Step seven, minimizing the total estimated error of all pixel points in the window T to obtain an unknown pixel point y in the G-band original image j Optimal estimate of (2)Completing the reconstruction of the G band original image;
optimal estimation valueThe expression is as follows:
the above is rewritable in matrix form:
y＝argmin{||Py-Qx|| 2 } (7)
in the formula (7), y and x are respectively an unknown pixel point and a known pixel point set in the window T, I 13×13 For a 13 th order identity matrix, the remaining sub-components are defined as follows:
Q 2 ＝(I 4×4 ,0 4×20 ) (11)
in the formula (11), I 4×4 Is an identity matrix with the size of 4 multiplied by 4, 0 4×20 Is a zero matrix of size 4 x 20; the solution of equation (7) can be expressed as:
y＝(P T P) -1 P T Qx (12)
outputting a preliminary estimate of the center pixel in (12) and within window TAnd taking the y value with the smallest difference as an unknown pixel estimated value, traversing the window T through the whole image, and completing the reconstruction of the G-band original image.
Step eight, taking the reconstructed G wave band image as a guide image, and respectively reconstructing original images of four wave bands B, C, Y, R by using guide filtering; for two-dimensional images, the key assumption of guided filtering is that the guided image P and the estimated image q satisfy a linear relationship within one two-dimensional local window, as follows:
wherein omega is k Is a local neighborhood window with a pixel k as a center, i is any pixel point in the window, and P i And q i Pixel values at pixel i for the pilot image P and the estimated image q, respectively; (a) k ,b k ) Is a set of assumed constant linear coefficients within a window, and by minimizing the difference between the estimated image q and the original image I, the linear coefficients (a k ,b k ) The method comprises the steps of carrying out a first treatment on the surface of the This problem is equivalent to the least squares problem expressed as:
m in the formula i Is a binary mask, the value at the sampled point in the original image I is 1, the rest position value is 0, epsilon is regularization parameter for adjusting the effect of the guided filtering, I i Pixel values at pixel I for original image I; solving the above can obtain a linear coefficient (a k ,b k ) Expressed as:
in the formulas (15) and (16),is the variance of the guide image P, +.>Hadamard products representing the original image I, the original image P and both respectively are accumulated in a local window omega k An average value of the inner; for each pixel point k, it is contained in a different window ω i In using different windows omega i Linear coefficient (a) k ,b k ) Calculate its mean +.>The expression is as follows:
where ω is the number of pixels in the partial window, equation (13) becomes:
and interpolating the other B, C, Y, R four-band original images by using the linear transformation to finish the reconstruction of the other band original images.
Compared with the similar guided filtering multispectral image reconstruction method based on the adaptive Gaussian upsampling, the method has the advantages that the G-band original image reconstructed by using the autoregressive model is higher in precision, the Peak Signal-to-Noise Ratio (PSNR) and the structural similarity (Structural Similarity, SSIM) of the reconstructed image are respectively improved by 3.39%,5.17% and 1.55% and 2.07%, the running time of the algorithm is respectively reduced by 0.09S and 0.07S, the method has higher instantaneity, the phenomena of edge artifacts, noise and the like in the reconstructed image are reduced, the high-frequency information such as textures, edges and the like in the multispectral reconstructed image is more completely reserved, and the subjective visual effect is closer to a real image.
Drawings
FIG. 1 is a diagram of a five-band snapshot multispectral imaging system of the present invention;
FIG. 2 is a schematic diagram of a filter array according to the present invention;
FIG. 3 is a diagram of pixel arrangements within a neighborhood T according to the present invention;
FIG. 4 auto-regressive model (a) estimates an unknown pixel model (b) error feedback model in the horizontal-vertical direction;
FIG. 5 model parameter estimation (a) known pixel model parameters (b) unknown pixel model parameters
Figure 6balloon scene reconstructed image contrast (a) real image (b) real image partial graph (c) GF method reconstructed image partial graph (d) method reconstructed image partial graph
Fig. 7CD scene reconstructed image contrast (a) real image (b) real image partial image (c) GF method reconstructed image partial image (d) method reconstructed image partial image
Detailed Description
The invention is described in further detail below with reference to the accompanying drawings.
The multispectral image reconstruction method based on weighted guided filtering comprises the following steps:
step one, a five-band snapshot type multispectral imaging system is designed. As shown in FIG. 1, the imaging system consists of an optical lens 1, an optical filter array 2, a photoelectric detector 3 and a computer 4. The working process of the system is as follows: the parallel incident light reaches the optical lens 1, is converged by the optical lens 1 and reaches the optical filter array 2, the optical filter array 2 splits the converged light, five narrow-band lights obtained after the splitting are received by the photodetector 3 and converted into electric signals, and the electric signals are transmitted to the computer 4 and converted into image signals. The photo detector 3 is a CMOS image sensor or a CCD image sensor, and the designed five-band multispectral array is based on an optical film plating principle, as shown in fig. 2, the optical film at each pixel has the same or different spectral transmittance, specifically, the optical filter array 2 samples the spectral information of B, C, G, Y, R five bands altogether, B is a blue band, the spectral range is 420nm-470nm, and the center wavelength is 440nm; c is light blue wave band, the spectrum range is 470nm-490nm, and the center wavelength is 480nm; g is a green wave band, the spectrum range is 490nm-570nm, and the center wavelength is 510nm; y is a yellow wave band, the spectrum range is 570nm-620nm, and the center wavelength is 580nm; r is red wave band, the spectrum range is 620nm-760nm, and the center wavelength is 650nm. The filter array is composed of a plurality of 4 multiplied by 4 periodic subarrays with the same arrangement mode, and B, C, G, Y, R five wave band sampling points are uniformly distributed in each periodic subarray, wherein the space probability ratio of the G wave band sampling points is 1/2, and the space probability ratio of the other four wave band sampling points is the same and is 1/8.
Step two, obtaining a B-band original image, a C-band original image, a G-band original image, a Y-band original image and an R-band original image. In the original image containing all spectrum information obtained in the step one, according to the corresponding spatial position of each band pixel in the optical filter array, downsampling the band pixels to obtain a single-band original image of B, C, G, Y, R four bands respectively. In the above single-band original image, there are a large number of non-sampled points that do not contain the spectral information of the band in addition to the sampled points of the band, and the present invention estimates the spectral information of the band in the non-sampled points and reconstructs the band original image using the following steps.
And thirdly, selecting a local neighborhood window T with the size of 7 multiplied by 7 by taking each unknown pixel point in the G-band original image as a center, and renumbering pixels in the neighborhood. As shown in fig. 3, useThe open dots represent unknown pixels { y } j I j=1, 2,..25 }, the solid dots represent the known pixels { x } i I=1, 2,..24 }, with the center unknown pixel noted y 1 Pixel x is known 1 At y 1 On the left, the remaining unknown pixels and known pixels are sequentially arranged at intervals in the window T in a clockwise direction.
And step four, modeling the correlation between the pixel points in the window T in the horizontal-vertical direction by using an autoregressive model. FIG. 4 (a) shows a model of estimating an unknown pixel from known pixels using correlation, in which the center open dots represent the unknown pixel y j The solid dots within the four neighbours of the hollow dot represent a known pixel x j,t T=1, 2,3,4 corresponds to the upper, right, lower, left four directions, respectively; FIG. 4 (b) shows a feedback model, i.e. calculating the error between the estimated unknown pixel and the known pixel, in which the center solid dot is the known pixel x i The open dots within the four neighbours of the solid dot represent the estimated unknown pixel y i,t T=1, 2,3,4 corresponds to the upper, right, lower, left four directions, respectively. The two models are expressed as:
in the formula { b } j,t ,b i,t T=1, 2,3,4} represents x j,t ，y i,t Corresponding autoregressive model parameters, gamma j And gamma i Respectively, for which errors are estimated.
And fifthly, calculating model parameters of the unknown pixels by using a Gaussian gradient function between the intensities of the known pixel points. As shown in FIG. 5 (a), the center open dot represents an unknown pixel y j The solid dots within the four neighbours of the hollow dot represent a known pixel x j,t T=1, 2,3,4 corresponds to the up, right, down, left four directions, respectively, unknown pixel y j Is the model in the t-th directionParameter b j,t The expression is as follows:
in sigma j Is the pixel y to be estimated j Standard deviation between known pixels in the four neighbors.
Step six, calculating the preliminary estimated value of the unknown pixel, and calculating the model parameters of the known pixel by using the preliminary estimated value. Calculating an unknown pixel y using (3) j Preliminary estimated value of (2)The expression is as follows:
as shown in FIG. 5 (b), in the figure, the center solid dot represents a known pixel x i Diamonds in the four neighborhoods of solid circles represent preliminary estimates of unknown pixelst=1, 2,3,4 corresponds to the up, right, down, left directions, respectively, according to the known pixel x i And the preliminary estimated value +.>Similarity between known pixels x i Model parameters b in the t-th direction i,t The expression is as follows:
in sigma i Is the known pixel point x i Standard deviation of pixels in eight neighborhoods. After all weight parameters of each pixel point are calculated, normalization is carried out on the weight parameters, and the sum of the weights is guaranteed to be 1.
Step (a)Seventhly, minimizing the sum of estimation errors of all pixel points in the window T to obtain an unknown pixel point y in the G-band original image j Optimal estimate of (2)And (5) completing the reconstruction of the G-band original image. />The expression is as follows:
the above is rewritable in matrix form:
y＝argmin{||Py-Qx|| 2 } (26)
in the formula (7), y and x are respectively an unknown pixel point and a known pixel point set in the window T, I 13×13 For a 13 th order identity matrix, the remaining sub-components are defined as follows:
Q 2 ＝(I 4×4 ,0 4×20 ) (30)
in the formula (11), I 4×4 Is an identity matrix with the size of 4 multiplied by 4, 0 4×20 Is a zero matrix of size 4 x 20. The solution of equation (7) can be expressed as:
y＝(P T P) -1 P T Qx (31)
outputting a preliminary estimate of the center pixel in (12) and within window TAnd taking the y value with the smallest difference as an unknown pixel estimated value, traversing the window T through the whole image, and completing the reconstruction of the G-band original image.
And step eight, taking the reconstructed G wave band image as a guide image, and respectively reconstructing original images of four wave bands of B, C, Y, R by using guide filtering. For two-dimensional images, the key assumption of guided filtering is that the guided image P and the estimated image q satisfy a linear relationship within one two-dimensional local window, as follows:
wherein omega is k Is a local neighborhood window with a pixel k as a center, i is any pixel point in the window, and P i And q i The pixel values at pixel i are the pilot image P and the estimated image q, respectively. (a) k ,b k ) Is a set of assumed constant linear coefficients within a window, and by minimizing the difference between the estimated image q and the original image I, the linear coefficients (a k ,b k ). This problem is equivalent to the least squares problem expressed as:
m in the formula i Is a binary mask, the value of the sampled point in the original image I is 1, the rest position value is 0, epsilon is regularization parameter, and is 0.001 for adjusting the effect of guiding filtering, I i Is the pixel value of the original image I at pixel I. Solving the above can obtain a linear coefficient (a k ,b k ) Expressed as:
in the formulas (15) and (16),is the variance of the guide image P, +.>Hadamard products representing the original image I, the original image P and both respectively are accumulated in a local window omega k An average value in the above. For each pixel point k, it is contained in a different window ω i In using different windows omega i Linear coefficient (a) k ,b k ) Calculate its mean +.>The expression is as follows:
where ω is the number of pixels in the partial window, equation (13) becomes:
and interpolating the other B, C, Y, R four-band original images by using the linear transformation to finish the reconstruction of the other band original images.
Examples:
the invention is improved on the basis of a guided filtering-based multispectral image reconstruction (Multispectral demosaicking using guided filter, GF) method, and in order to more intuitively illustrate the advantages of the invention, matlab2021a software is used for carrying out simulation analysis on a CAVE data set and a TokyoTech data set on two methods, and algorithms are respectively compared from the two aspects of objective evaluation indexes and subjective visual effects of reconstructed images. The objective evaluation indexes comprise peak signal-to-noise ratio PSNR, structural similarity SSIM and algorithm running time, as shown in tables 1 and 2, the PSNR and SSIM of the reconstructed image are respectively improved by 3.39%,5.17% and 1.55%,2.07% on two data sets compared with the GF method, and the running time is reduced by 0.09s and 0.07s compared with the GF method, so that the estimated unknown pixel value is closer to the real pixel value, the reconstructed image is more similar to the original image, and the real-time performance is better; fig. 6 and 7 are reconstructed images of a balloon scene and a CD scene, respectively, both of which include (a) a real image, (b) a real image partial image, (c) a GF method reconstructed image partial image, and (d) the present method reconstructed image partial image. As can be seen from the figure, in the reconstructed image of the invention, only a small amount of artifacts exist near the edge, the transition between different areas of the image is smoother, and the noise points are fewer, which indicates that the method can improve the reconstruction accuracy of the image, has better adaptability to edge information, and has better comprehensive performance than the GF method.
TABLE 1
TABLE 2
Claims (5)
1. The five-band multispectral image reconstruction method based on the autoregressive model is characterized by comprising the following steps of:
step one, designing a five-band snapshot type multispectral imaging system;
step two, acquiring a B-band original image, a C-band original image, a G-band original image, a Y-band original image and an R-band original image;
thirdly, selecting a local neighborhood window T with the size of 7 multiplied by 7 by taking each unknown pixel point in the G-band original image as a center, and renumbering pixels in the neighborhood;
selecting a local neighborhood window T with the size of 7 multiplied by 7 by taking each unknown pixel point in the G-band original image as a center, renumbering pixels in the neighborhood, and representing an unknown pixel set as { y } j |j=1, 2,..25 }, the known set of pixels is denoted as { x } i |i＝1,2,...,24}；
Modeling the correlation among the pixel points in the window T in the horizontal-vertical direction by using an autoregressive model;
modeling correlations between pixel points within a window T in a horizontal-vertical direction using an autoregressive model, including a model that estimates an unknown pixel from a known pixel using the correlations and a feedback model that calculates an error between the estimated unknown pixel and the known pixel, the two models being expressed as:
wherein x is j,t For unknown pixel y j A known pixel adjacent in the t-th direction in the four neighbors, y i,t Is the known pixel x i Unknown pixels adjacent in the t-th direction in the four neighbors, b j,t 、b i,t Respectively represent x j,t ，y i,t Corresponding autoregressive model parameters, t=1, 2,3,4, corresponding to the four directions up, right, down, left, γ j And gamma i The estimation errors of the two models are respectively;
fifthly, calculating model parameters of the unknown pixels by using a Gaussian gradient function between the intensities of the known pixel points;
calculating model parameters of known pixels in the four neighborhoods of the unknown pixels by using Gaussian gradient functions among the intensities of the known pixel points, wherein the unknown pixels are y j Is known in the t-th direction of pixel x j,t Model parameter b of (2) j,t The expression is as follows:
in sigma j Is the pixel y to be estimated j The standard deviation among known pixel points in the four adjacent areas, wherein t=1, 2,3 and 4 respectively represent the upper, right, lower and left directions;
step six, calculating a preliminary estimated value of the unknown pixel, and calculating model parameters of the known pixel by using the preliminary estimated value;
calculating a preliminary estimated value of the unknown pixel, and calculating model parameters of the unknown pixel in the four adjacent areas of the known pixel by using the preliminary estimated value; calculating an unknown pixel y using (1) j Preliminary estimated value of (2)The expression is as follows:
wherein x is j,t Representing unknown pixel y j A known pixel in the t-th direction in the four neighborhoods, b j,t For its model parameters, t=1, 2,3,4, respectively representing the up, right, down, left four directions;
according to the known pixel x i And the preliminary estimated value of the unknown pixels in the four neighborhoodsSimilarity between known pixels x i Model parameters b in the t-th direction i,t The expression is as follows:
in sigma i Is the known pixel point x i Standard deviation of pixels in eight neighborhoods, t=1, 2,3,4, respectively representing four directions of up, right, down and left; after all weight parameters of each pixel point are calculated, normalizing the weight parameters to ensure that the sum of the weights is 1;
step seven, minimizing the total estimated error of all pixel points in the window T to obtain an unknown pixel point y in the G-band original image j Optimal estimate of (2)Completing the reconstruction of the G band original image;
and step eight, taking the reconstructed G wave band image as a guide image, and respectively reconstructing original images of four wave bands of B, C, Y, R by using guide filtering.
2. The method of claim 1, wherein the system designed in the first step includes four parts, i.e., an optical lens, an optical filter array, a photodetector and a computer, the incident light enters the optical lens, and is converged on the optical filter array through the optical lens, the optical filter array performs spectral splitting on the broadband incident light, and the narrow-band light divided into five bands is respectively: each pixel in the filter array only passes through a single narrow-band optical signal, and emergent light split by the filter array is received by a photoelectric detector, the photoelectric detector converts the optical signal into an electric signal and transmits the electric signal to a computer, and the computer converts the electric signal into an image signal.
3. The method for reconstructing a five-band multispectral image based on an autoregressive model according to claim 2, wherein the electric signal output by the photoelectric detector in the second step is transmitted to a computer and is converted into an image signal by the computer, the image contains all information of five bands, and according to the corresponding spatial position of each band pixel in the filter array, the original image of the corresponding band, namely, the single-band original image of B, C, G, Y, R five bands is obtained by downsampling the band pixel.
4. The method for reconstructing a five-band multispectral image based on an autoregressive model according to claim 1, wherein the seventh step is to minimize the sum of estimation errors of all pixel points in the window T to obtain an unknown pixel point y in the G-band original image j Optimal estimate of (2)Completing the reconstruction of the G band original image;
optimal estimation valueThe expression is as follows:
the above rewrites into a matrix form:
y＝arg min{||Py-Qx|| 2 } (7)
in the formula (7), y and x are respectively an unknown pixel point and a known pixel point set in the window T, I 13×13 For a 13 th order identity matrix, the remaining sub-components are defined as follows:
P 1 ＝{p 1 (i,j)}(i＝1,2,3,4,j＝1,2,...,13),
Q 1 ＝{q 1 (j,i)}(i＝1,2,...,24,j＝1,2,...,13),
Q 2 ＝(I 4×4 ,0 4×20 ) (11)
in the formula (11), I 4×4 Is an identity matrix with the size of 4 multiplied by 4, 0 4×20 Is a zero matrix of size 4 x 20; the solution of equation (7) is expressed as:
y＝(P T P) -1 P T Qx (12)
outputting a preliminary estimate of the center pixel in (12) and within window TAnd taking the y value with the smallest difference as an unknown pixel estimated value, traversing the window T through the whole image, and completing the reconstruction of the G-band original image.
5. The method for reconstructing a five-band multispectral image based on an autoregressive model according to claim 4, wherein the step eight is to reconstruct the original images of four bands B, C, Y, R by using guide filtering with the reconstructed G-band image as a guide image; for two-dimensional images, the key assumption of guided filtering is that the guided image P and the estimated image q satisfy a linear relationship within one two-dimensional local window, as follows:
wherein omega is k Is a local neighborhood window with a pixel k as a center, i is any pixel point in the window, and P i And q i Pixel values at pixel i for the pilot image P and the estimated image q, respectively; (a) k ,b k ) Is a set of assumed constant linear coefficients within a window, and by minimizing the difference between the estimated image q and the original image I, the linear coefficients (a k ,b k )，Calculated by using a least square method, expressed as:
m in the formula i Is a binary mask, the value at the sampled point in the original image I is 1, the rest position value is 0, epsilon is regularization parameter for adjusting the effect of the guided filtering, I i Pixel values at pixel I for original image I; solving the above to obtain a linear coefficient (a k ,b k ) Expressed as:
in the formulas (15) and (16),is the variance of the guide image P, +.>Hadamard products representing the original image I, the original image P and both respectively are accumulated in a local window omega k An average value of the inner; for each pixel point k, it is contained in a different window ω i In using different windows omega i Linear coefficient (a) k ,b k ) Calculate its mean +.>The expression is as follows:
where ω is the number of pixels in the partial window, equation (13) becomes:
and interpolating the other B, C, Y, R four-band original images by using the linear transformation to finish the reconstruction of the other band original images.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310121777.2A CN116309126B (en) | 2023-02-16 | 2023-02-16 | Five-band multispectral image reconstruction method based on autoregressive model |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310121777.2A CN116309126B (en) | 2023-02-16 | 2023-02-16 | Five-band multispectral image reconstruction method based on autoregressive model |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116309126A CN116309126A (en) | 2023-06-23 |
| CN116309126B true CN116309126B (en) | 2024-01-23 |
Family
ID=86831555
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310121777.2A Active CN116309126B (en) | 2023-02-16 | 2023-02-16 | Five-band multispectral image reconstruction method based on autoregressive model |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116309126B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117237474B (en) * | 2023-11-16 | 2024-02-09 | 长春理工大学 | Depth guidance-based on-chip integrated multispectral imaging reconstruction method |
| CN117288325B (en) * | 2023-11-27 | 2024-02-13 | 长春理工大学 | High-light-efficiency snapshot type multispectral imaging method and system |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20100084458A (en) * | 2009-01-16 | 2010-07-26 | 삼성전자주식회사 | Image interpolation method and apparatus using pattern characteristics of color filter array |
| CN110579279A (en) * | 2019-09-19 | 2019-12-17 | 西安理工大学 | design method of nine-spectral-band multispectral imaging system of single sensor |
-
2023
- 2023-02-16 CN CN202310121777.2A patent/CN116309126B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20100084458A (en) * | 2009-01-16 | 2010-07-26 | 삼성전자주식회사 | Image interpolation method and apparatus using pattern characteristics of color filter array |
| CN110579279A (en) * | 2019-09-19 | 2019-12-17 | 西安理工大学 | design method of nine-spectral-band multispectral imaging system of single sensor |
Non-Patent Citations (3)
| Title |
|---|
| Image Interpolation by Adaptive 2-D Autoregressive Modeling and Soft-Decision Estimation;Xiangjun Zhang 等;《IEEE》;第17卷(第17期);第887-896页 * |
| Multispectral demosaicking using guided filter;Yusuke Monno 等;《Proceedings of SPIE》;第1-8页 * |
| 基于隐式分段自回归模型的图像插值算法;任杰等;软件学报;第1248-1259页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116309126A (en) | 2023-06-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN116309126B (en) | Five-band multispectral image reconstruction method based on autoregressive model | |
| Menon et al. | Color image demosaicking: An overview | |
| US20220301114A1 (en) | Noise Reconstruction For Image Denoising | |
| EP2426639B1 (en) | Method for demosaicing a raw digital image, corresponding computer program and imaging or graphic circuit | |
| Khan et al. | Underwater image enhancement by wavelet based fusion | |
| Hu et al. | Convolutional sparse coding for RGB+ NIR imaging | |
| CN110211044B (en) | Multispectral imaging method and device based on demosaicing algorithm and principal component analysis | |
| CN116188305B (en) | Multispectral image reconstruction method based on weighted guided filtering | |
| Tao et al. | Hyperspectral image recovery based on fusion of coded aperture snapshot spectral imaging and RGB images by guided filtering | |
| Habtegebrial et al. | Deep convolutional networks for snapshot hypercpectral demosaicking | |
| CN110443865B (en) | Multispectral imaging method and device based on RGB camera and depth neural network | |
| JP7358817B2 (en) | Image processing device, imaging device, image processing method, and program | |
| US10237519B2 (en) | Imaging apparatus, imaging system, image generation apparatus, and color filter | |
| CN116152120B (en) | Low-light image enhancement method and device integrating high-low frequency characteristic information | |
| US7269295B2 (en) | Digital image processing methods, digital image devices, and articles of manufacture | |
| US7430334B2 (en) | Digital imaging systems, articles of manufacture, and digital image processing methods | |
| CN116029930A (en) | Multispectral image demosaicing method based on convolutional neural network | |
| US20240020796A1 (en) | Noise reconstruction for image denoising | |
| Pan et al. | Deep panchromatic image guided residual interpolation for multispectral image demosaicking | |
| Li et al. | Universal Demosaicking for Interpolation-Friendly RGBW Color Filter Arrays | |
| US20220247889A1 (en) | Raw to rgb image transformation | |
| Hagen et al. | Fourier domain filtering analysis for color polarization camera demosaicking | |
| CN108051087A (en) | A kind of eight passage multispectral camera design methods for fast imaging | |
| Zhang et al. | A generic multispectral demosaicking method based on inter-channel spectrum correlation | |
| JP2013187711A (en) | Image processing apparatus, imaging apparatus, and image processing method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |