CN116263947A

CN116263947A - Image fusion method, computer program product, storage medium and electronic device

Info

Publication number: CN116263947A
Application number: CN202210451856.5A
Authority: CN
Inventors: 唐金伟
Original assignee: Beijing Jigan Technology Co ltd
Current assignee: Beijing Jigan Technology Co ltd
Priority date: 2022-04-26
Filing date: 2022-04-26
Publication date: 2023-06-16

Abstract

The application provides an image fusion method, a computer program product, a storage medium and an electronic device. The image fusion method comprises the following steps: acquiring a gray image and a color image to be fused, wherein the color image comprises a channel image of a brightness channel and a channel image of a chromaticity channel; m frames of gray sub-images are decomposed from the gray images, and M frames of brightness sub-images are decomposed from channel images of the brightness channels, wherein the M frames of gray sub-images and the M frames of brightness sub-images respectively represent image information of the gray images and the channel images of the brightness channels in corresponding M frequency bands; fusing the M-frame gray sub-images and the M-frame brightness sub-images to obtain M-frame fused sub-images; superposing the M frames of fusion sub-images to obtain a brightness fusion image; and combining the brightness fusion image and the channel image of the chromaticity channel to obtain a color fusion image. The method performs image fusion through the sub-frequency bands, so that the image fusion process is finer, and the quality of the color fusion image is improved.

Description

Image fusion method, computer program product, storage medium and electronic device

Technical Field

The present invention relates to the field of image processing technology, and in particular, to an image fusion method, a computer program product, a storage medium, and an electronic device.

Background

In a large number of electronic image application fields, the desire of people for high-quality images is increasing, and the photographing performance of the traditional single-camera image acquisition device cannot meet the requirements of people on the image quality, so that the two-camera image acquisition device appears in recent years.

The double-camera image acquisition equipment comprises a black-and-white camera and a color camera, is mainly applied to a dark light environment (such as night and indoor), and is used for respectively acquiring a gray level image and a color image through the black-and-white camera and the color camera, and then fusing the gray level image and the color image by using a black-and-white color image fusion technology to obtain a fused image. The black-and-white color image fusion technology utilizes the characteristic that the detail information of the gray level image is more prominent than the detail information of the color image in a dark environment, and adopts the brightness information of the gray level image and the color information of the color image to be fused in the fused image, so that the image quality of the fused dark color image is higher than that of the dark color image obtained by the single-camera image acquisition equipment.

However, the existing black-and-white color image fusion technology is rough, and the quality improvement of the obtained fusion image relative to the color image before fusion is very limited.

Disclosure of Invention

An object of an embodiment of the present application is to provide an image fusion method, a computer program product, a storage medium and an electronic device, so as to improve the above technical problems.

In order to achieve the above purpose, the present application provides the following technical solutions:

in a first aspect, an embodiment of the present application provides an image fusion method, including: acquiring a gray level image and a color image to be fused; wherein the color image includes a channel image of a luminance channel and a channel image of a chrominance channel; decomposing M frames of gray sub-images from the gray images, and decomposing M frames of brightness sub-images from the channel images of the brightness channels; wherein M is an integer greater than 1, the M-frame gray sub-images represent image information of the gray image in corresponding M frequency bands, and the M-frame luminance sub-images represent image information of a channel image of the luminance channel in the M frequency bands; fusing the M-frame gray sub-images and the M-frame brightness sub-images to obtain M-frame fused sub-images; each frame of gray sub-image is used for fusing with one frame of brightness sub-image of the corresponding frequency band; superposing the M frame fusion sub-images to obtain a brightness fusion image; and combining the brightness fusion image and the channel image of the chromaticity channel to obtain a color fusion image.

According to the method, the color images are fused with the corresponding gray level images in the frequency division manner, and the sub-images in different frequency bands have different characteristics (for example, the sub-images in the high frequency band represent image details, the sub-images in the low frequency band represent the overall brightness and contrast of the images), so that the fusion process of the images is finer in the frequency division manner, and the quality of the color fusion images is improved.

In an implementation manner of the first aspect, the fusing the M-frame gray sub-image and the M-frame luminance sub-image to obtain an M-frame fused sub-image includes: calculating M frame fusion masks corresponding to the M frequency bands according to the gray sub-images and the brightness sub-images; wherein, the pixel value in the fusion mask characterizes: fusion weights of pixel values at the same positions in the corresponding luminance sub-image and the corresponding gray sub-image; the M-frame fusion mask is utilized to carry out weighted fusion on the M-frame gray sub-image and the M-frame brightness sub-image, so as to obtain the M-frame fusion sub-image; each frame of fusion mask is used for fusing a corresponding frame of gray sub-image and a corresponding frame of brightness sub-image.

The existing black-and-white color image fusion technology is basically only suitable for a dark light environment, and the characteristic that the detail information of the gray level image is more prominent than the detail information of the color image in the dark light environment is utilized, so that the brightness information of the gray level image is adopted in the fusion image and the color information of the color image is fused, and the image quality of the dark light color image obtained after fusion is higher than that of the dark light color image obtained by the single-camera image acquisition equipment. However, for a highlight environment, the detail information of the gray-scale image is not absolutely more prominent than the detail information of the color image, and thus the existing black-and-white color image fusion technique is not suitable for the highlight environment.

In the implementation manner, the fusion mask is calculated according to the gray sub-image and the brightness sub-image, namely, the pixel value (representing the fusion weight) in the fusion mask is determined by the content adaptation of the gray sub-image and the brightness sub-image, so that the fusion is carried out according to the fusion mask and the information in the gray sub-image is not necessarily prone to be adopted more, and therefore, the color fusion image with higher quality can be obtained in the highlight environment in the implementation manner, and the technical blank of the prior art is made up. Of course, for the dark light environment, the image fusion quality can be effectively improved compared with the existing method due to the adoption of the frequency division and the section fusion.

In an implementation manner of the first aspect, m=3, and the M frequency bands are a high frequency band, a medium frequency band, and a low frequency band, respectively.

The implementation manner gives a simple and effective way of dividing the frequency bands, the number of the frequency bands is not too large, and meanwhile, the decomposed sub-images can well express the characteristics of the images on the frequency domain: the details of the image are represented by the parts located in the high-frequency band and the middle-frequency band (the high-frequency band is small details and the middle-frequency band is larger details), so that the definition of the image is determined; the portion located in the low frequency band represents the overall brightness change of the image, and cannot determine the sharpness of the image. Therefore, different fusion strategies can be adopted to the sub-images according to the characteristics of the three frequency bands.

In an implementation manner of the first aspect, the calculating the M-frame fusion mask corresponding to the M frequency bands according to the gray sub-image and the luminance sub-image includes: calculating a fusion mask of the middle frequency band according to the gray sub-image of the middle frequency band and the brightness sub-image of the middle frequency band; and calculating a fusion mask of the high frequency band according to the gray level sub-image of the high frequency band and the brightness sub-image of the high frequency band; and calculating the fusion mask of the low frequency band according to the fusion mask of the medium frequency band and the fusion mask of the high frequency band.

The implementation mode provides the calculation mode of the fusion masks of different frequency bands, and the sub-images of the high frequency band and the middle frequency band represent the details of the images, so that the definition of the images is determined, and the fusion masks of the high frequency band and the middle frequency band can be directly calculated according to the pixel value in the sub-images; the sub-image of the low frequency band basically does not contain image details, and the definition of the image cannot be determined, so that the fusion mask of the low frequency band directly calculated according to the sub-image of the low frequency band is not accurate enough.

In one implementation manner of the first aspect, the fusion mask is a binarized image, if a pixel value in the fusion mask takes a first value, a pixel value in a corresponding gray sub-image is adopted as a pixel value in a corresponding fusion sub-image at the same position, and if a pixel value in the fusion mask takes a second value, a pixel value in a corresponding brightness sub-image is adopted as a pixel value in a corresponding fusion sub-image at the same position; the calculating the fusion mask of the low frequency band according to the fusion mask of the medium frequency band and the fusion mask of the high frequency band comprises the following steps: and if the pixel values of the fusion mask of the middle frequency band and the fusion mask of the high frequency band at the same position are both the first numerical value, setting the pixel value of the fusion mask of the low frequency band at the position as the first numerical value, otherwise, setting the pixel value of the fusion mask of the low frequency band at the position as the second numerical value.

The implementation mode provides a calculation mode of the fusion mask of the low frequency band when the fusion mask is a binary image.

In an implementation manner of the first aspect, the calculating the fusion mask of the mid-band according to the gray sub-image of the mid-band and the luminance sub-image of the mid-band includes: taking absolute values of pixel values in the gray sub-images of the middle frequency band to obtain a first mask calculation image; taking absolute values of pixel values in the brightness sub-images of the middle frequency band to obtain a second mask calculated image; and calculating the fusion mask of the middle frequency band according to the magnitude relation between the pixel values of the first mask calculation image and the second mask calculation image at the same position.

In the above implementation, for the sub-image of the mid-band, the pixel value size characterizes the detail information (i.e. the sharpness) in the original image, so the fused mask of the mid-band can be directly calculated by the size relation between the pixel values at the same position in the mask calculation image. Meanwhile, when the gray level image and the color image are decomposed, the pixel value in the sub-image generated by decomposition may appear as a negative number, so that the pixel value comparison is required to be performed after the absolute value of the pixel value in the sub-image is taken, and the error in judging the relationship of the pixel value and the magnitude due to the negative number is avoided.

In an implementation manner of the first aspect, the calculating the fusion mask of the mid-band according to the gray sub-image of the mid-band and the luminance sub-image of the mid-band includes: taking absolute values of pixel values in the gray sub-images of the middle frequency band to obtain a first mask calculation image; taking absolute values of pixel values in the brightness sub-images of the middle frequency band to obtain a second mask calculated image; filtering the first mask calculation image and the second mask calculation image by using a non-edge-protection smoothing filter to obtain a third mask calculation image and a fourth mask calculation image; and calculating the fusion mask of the middle frequency band according to the magnitude relation between the pixel values of the third mask calculation image and the fourth mask calculation image at the same position.

The inventor has found that black and white edges are likely to occur near some edges of color images, and that gray scale images do not. In the fusion result of the prior art, the black and white edges are reserved, and the image quality is seriously affected. In the above implementation manner, the edge in the image can be widened by the non-edge-preserving smoothing filter, and the widened edge covers the region where the black-white edge is located, so that when the image fusion is performed in these regions, the image information of the gray level image which does not contain the black-white edge can be adopted rather than the image information of the color image, and the black-white edge in the color fusion image is eliminated or at least weakened.

In one implementation manner of the first aspect, the fusion mask is a binarized image, if a pixel value in the fusion mask takes a first value, a pixel value in a corresponding gray sub-image is adopted as a pixel value in a corresponding fusion sub-image at the same position, and if a pixel value in the fusion mask takes a second value, a pixel value in a corresponding brightness sub-image is adopted as a pixel value in a corresponding fusion sub-image at the same position; the calculating the fused mask of the mid-band according to the magnitude relation between the pixel values of the third mask calculation image and the fourth mask calculation image at the same position comprises the following steps: and aiming at any pixel position in the fused mask of the middle frequency band, if the pixel value of the third mask calculated image at the position is larger than the product of the pixel value of the fourth mask calculated image at the position and the adjustment threshold value, setting the pixel value of the fused mask of the middle frequency band at the position as a first numerical value, otherwise, setting the pixel value of the fused mask of the middle frequency band at the position as a second numerical value.

In the implementation manner, considering that the imaging quality of the gray level image and the imaging quality of the color image are different in different environments, the direct pixel value comparison result may not be accurate, so that the difference in the image acquisition environment can be reflected by setting the adjustment threshold value, the pixel value of the brightness sub-image is adjusted according to the adjustment threshold value, and then the adjustment result is compared with the pixel value of the gray level sub-image to calculate the fusion mask, so that the calculation accuracy of the fusion mask is higher. In the alternative, the adjustment threshold may also be set for the gray sub-image.

In an implementation manner of the first aspect, after the acquiring the gray scale image and the color image to be fused and before the decomposing the M-frame gray scale sub-image from the gray scale image, the method further includes: adjusting the brightness of the gray scale image to be consistent with the color image; the calculating the M-frame fusion mask corresponding to the M frequency bands according to the gray sub-image and the brightness sub-image includes: calculating a fusion mask of the middle frequency band according to the gray level sub-image of the low frequency band and the brightness sub-image of the low frequency band; and calculating a fusion mask of the middle frequency band according to the gray sub-image of the middle frequency band and the brightness sub-image of the middle frequency band; and calculating a fusion mask of the high frequency band according to the gray level sub-image of the high frequency band and the brightness sub-image of the high frequency band.

In the above implementation manner, the color image is used as a fusion reference, that is, the gray image is fused with the color image, so that the brightness of the gray image is adjusted to be the same level as that of the color image, and the fusion masks of the high, medium and low frequency bands are calculated by respectively using the gray sub-images of the high, medium and low frequency bands and the brightness sub-images, so that the reduction of the calculation accuracy (especially the calculation influence on the fusion masks of the low frequency band) caused by the inconsistency of the brightness of the gray image and the brightness of the color image can be improved.

In an implementation manner of the first aspect, the decomposing the M-frame gray sub-image from the gray image includes: filtering the gray level image by using a first low-pass filter to obtain a gray level sub-image of a low frequency band; the cut-off frequency of the first low-pass filter is a boundary between a low frequency band and a medium frequency band; filtering the gray level image by using a second low-pass filter to obtain a temporary gray level image, and calculating a gray level sub-image of a high frequency band according to the gray level image and the temporary gray level image; the cut-off frequency of the second low-pass filter is a boundary between a middle frequency band and a high frequency band; and calculating the gray sub-image of the middle frequency band according to the gray image, the gray sub-image of the low frequency band and the gray sub-image of the high frequency band.

In the implementation manner, the gray sub-image of the low frequency band is obtained by filtering the gray image through the first low-pass filter, then the temporary gray image is obtained by filtering the gray image through the second low-pass filter, the temporary gray image is equivalent to the gray sub-image containing the middle and low frequency bands, the gray sub-image of the high frequency band can be calculated according to the gray image and the temporary gray image, and finally the gray sub-image of the middle frequency band can be calculated by utilizing the gray image, the gray sub-image of the low frequency band and the gray sub-image of the high frequency band.

In an implementation manner of the first aspect, the decomposing the M-frame gray sub-image from the gray image includes: calculating a shielding area mask according to the gray level image and the channel image of the brightness channel; the pixel value in the mask of the shielding area characterizes the probability that the position of the pixel value belongs to the shielding area, and the shielding area refers to an area which only exists in the color image but does not exist in the gray image in the shooting scene; the gray level image and the channel image of the brightness channel are weighted and fused by using the shielding area mask to obtain a gray level shielding fusion image; and decomposing the M frames of gray sub-images from the gray shielding fusion image.

In the case where the grayscale image and the color image are captured by the black-and-white camera and the color camera of the same electronic device, respectively, there is parallax between the grayscale image and the color image due to the different positions of the black-and-white camera and the color camera, that is, a part of the region exists only in the color image but does not exist in the grayscale image (or is blocked in the grayscale image), and another part of the region exists only in the grayscale image but does not exist in the color image (or is blocked in the color image).

If the color image is used as a fusion reference, an area which exists only in the color image but does not exist in the gray image in a shooting scene is defined as an occlusion area, the image information of the gray image in the occlusion area is supplemented by the image information of the color image in the occlusion area, and then the gray image (namely, the gray occlusion fusion image) after the color image and the supplementary information is fused, so that the scene area which is included by the color image and the gray image is consistent, and the subsequent image fusion effect can be improved.

In the above implementation, the pixel values in the mask of the occlusion region describe both the position of the occlusion region and can be used as a fusion weight for supplementing the image information of the gray image in the occlusion region.

In a second aspect, an embodiment of the present application provides an image fusion apparatus, including: the image acquisition component is used for acquiring a gray image and a color image to be fused; wherein the color image includes a channel image of a luminance channel and a channel image of a chrominance channel; an image decomposition component for decomposing M frames of gray sub-images from the gray images and M frames of brightness sub-images from the channel images of the brightness channels; wherein M is an integer greater than 1, the M-frame gray sub-images represent image information of the gray image in corresponding M frequency bands, and the M-frame luminance sub-images represent image information of a channel image of the luminance channel in the M frequency bands; the image fusion component is used for fusing the M-frame gray sub-images and the M-frame brightness sub-images to obtain M-frame fusion sub-images; each frame of gray sub-image is used for fusing with one frame of brightness sub-image of the corresponding frequency band; the image superposition component is used for superposing the M frame fusion sub-images to obtain a brightness fusion image; and the channel splicing assembly is also used for combining the brightness fusion image and the channel image of the chromaticity channel to obtain a color fusion image.

In a third aspect, embodiments of the present application provide a computer program product comprising computer program instructions which, when read and executed by a processor, perform the method provided by the first aspect or any one of the possible implementations of the first aspect.

In a fourth aspect, embodiments of the present application provide a computer readable storage medium having stored thereon computer program instructions which, when read and executed by a processor, perform the method provided by the first aspect or any one of the possible implementations of the first aspect.

In a fifth aspect, embodiments of the present application provide an electronic device, including: a memory and a processor, the memory having stored therein computer program instructions which, when read and executed by the processor, perform the method of the first aspect or any one of the possible implementations of the first aspect.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the embodiments of the present application will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present application and should not be considered as limiting the scope, and other related drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

Fig. 1 shows steps of an image fusion method provided in an embodiment of the present application;

fig. 2 shows sub-steps that may be included in step S120 of fig. 1;

FIG. 3 shows a schematic diagram of an exploded process of a gray scale image;

FIG. 4 shows a color image, a gray scale image, and a color fusion image contrast map;

fig. 5 shows functional components included in the image fusion apparatus provided in the embodiment of the present application;

fig. 6 shows a structure of an electronic device provided in an embodiment of the present application.

Detailed Description

In recent years, technology research such as computer vision, deep learning, machine learning, image processing, image recognition and the like based on artificial intelligence has been advanced significantly. Artificial intelligence (Artificial Intelligence, AI for short) is an emerging scientific technology for studying and developing theories, methods, techniques and application systems for simulating and extending human intelligence. The artificial intelligence discipline is a comprehensive discipline and relates to various technical categories such as chips, big data, cloud computing, internet of things, distributed storage, deep learning, machine learning, neural networks and the like. Computer vision is an important branch of artificial intelligence, and particularly, machine recognition is used in the world, and computer vision technologies generally include technologies such as face recognition, living body detection, fingerprint recognition and anti-counterfeiting verification, biological feature recognition, face detection, pedestrian detection, object detection, pedestrian recognition, image processing, image recognition, image semantic understanding, image retrieval, word recognition, video processing, video content recognition, behavior recognition, three-dimensional reconstruction, virtual reality, augmented reality, synchronous positioning and map construction, computational photography, robot navigation and positioning, and the like. With research and progress of artificial intelligence technology, the technology expands application in various fields, such as security protection, city management, traffic management, building management, park management, face passing, face attendance, logistics management, warehouse management, robots, intelligent marketing, computed photography, mobile phone images, cloud services, intelligent home, wearing equipment, unmanned driving, automatic driving, intelligent medical treatment, face payment, face unlocking, fingerprint unlocking, personnel verification, intelligent screen, intelligent television, camera, mobile internet, network living broadcast, beauty, make-up, medical beauty, intelligent temperature measurement and the like. The image fusion method in the embodiment of the application also utilizes the technologies of image processing and the like.

The black-and-white color image fusion technology is widely applied to the double-camera image acquisition equipment, but the inventor researches for a long time to find that the existing black-and-white color image fusion technology can improve the quality of images in a dark environment, but the technology still has various defects, such as:

firstly, in the prior art, a color image and a gray level image are generally fused directly, and the quality of the obtained fused image is improved very limited relative to the color image before fusion;

secondly, the prior art mainly utilizes the characteristic that the quality of a gray level image is higher than that of a color image in a dark light environment to design a fusion strategy, and the quality of each area in the gray level image is not necessarily higher than that of the color image in a highlight environment, so that the original fusion strategy cannot be directly applied to the highlight environment;

third, in the fusion result of the prior art, there is a case where there is a black-and-white edge at the image edge, and a specific example is referred to fig. 4.

The image fusion method provided by the embodiment of the application improves the technical defects by carrying out technical means such as frequency division fusion, calculation fusion mask and the like. It should be appreciated that in addition to the new solutions proposed by the embodiments of the present application, the above-mentioned defects existing in the prior art are also conclusions drawn by the inventors after practice and careful study, and thus should also be considered as contributions to the solution of the present application by the inventors during the inventive process.

The following description of the embodiments of the present application will be made clearly and completely with reference to the drawings in the embodiments of the present application, and it is apparent that the described embodiments are only some embodiments of the present application, not all embodiments. The components of the embodiments of the present application, which are generally described and illustrated in the figures herein, may be arranged and designed in a wide variety of different configurations. Thus, the following detailed description of the embodiments of the present application, as provided in the accompanying drawings, is not intended to limit the scope of the application, as claimed, but is merely representative of selected embodiments of the application. All other embodiments, which can be made by those skilled in the art based on the embodiments of the present application without making any inventive effort, are intended to be within the scope of the present application.

It should be noted that: like reference numerals and letters denote like items in the following figures, and thus once an item is defined in one figure, no further definition or explanation thereof is necessary in the following figures. Meanwhile, in the description of the present application, the terms "first", "second", and the like are used only to distinguish the description, and are not to be construed as indicating or implying relative importance.

The terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.

Fig. 1 illustrates steps of an image fusion method provided in an embodiment of the present application, which may be performed by, but is not limited to, the electronic device illustrated in fig. 6, and with respect to the structure of the electronic device, reference may be made to the following explanation with respect to fig. 6. Referring to fig. 1, the image fusion method includes:

step S110: and acquiring a gray level image and a color image to be fused.

The images to be fused are images to be fused, the images to be fused can comprise a frame of gray level image and a frame of color image, the aim of fusion is to calculate a new frame of color image according to the images to be fused, and the image quality of the color image is higher than that of the color image before fusion. Wherein the color image includes a channel image of a luminance channel and a channel image of a chrominance channel. For example, the directly acquired image is a YUV image, which is itself a "color image" required in step S110, wherein the Y channel is a luminance channel and the UV channel is a chrominance channel. For another example, the directly acquired image is an RGB image, and since the RGB image has no luminance channel, it is not a "color image" required in step S110, and the RGB image needs to be converted into a YUV image to obtain the "color image" required in step S110. The gray image has only one channel, which can be considered as having only a luminance channel, so the gray image should be fused with the channel image of the luminance channel of the color image at the time of fusion.

The present application is not limited as to how to obtain an image to be fused that satisfies the above requirements: for example, it may be that the image actually collected is received as the image to be fused from a camera of an electronic device (for example, a mobile phone, a wearable device, a robot, etc., which is not necessarily the same device as the electronic device performing step S110), the camera of the electronic device includes a black-and-white camera (which may be a natural light camera or an infrared camera) and a color camera, and the gray-scale image and the color image are acquired by using the black-and-white camera and the color camera, respectively; for another example, the color image and the gradation image stored in advance may be read directly from the memory of the electronic apparatus, or the like. It should be further appreciated that color images and grayscale images are content-specific to the same scene, otherwise it is not significant to fuse them. Note that, although it will be appreciated from the following that the partial implementation of the image fusion method in the embodiments of the present application has good performance in a highlight environment, the present application is not limited to the lighting conditions for acquiring the grayscale image and the color image: for example, it may be a color image and a grayscale image taken in a dark light environment, or a color image and a grayscale image taken in a highlight environment, or the like.

Step S120: m-frame gray sub-images are decomposed from the gray scale image, and M-frame luminance sub-images are decomposed from the channel images of the luminance channels.

Wherein M is an integer greater than 1, the M-frame gray sub-images represent image information of gray images in corresponding M frequency bands, and the M-frame brightness sub-images represent image information of channel images of brightness channels in the M frequency bands. The value of M is not limited in this application: for example, if the value of M is 3, the M frequency bands may be a high frequency band, a medium frequency band, and a low frequency band; for another example, if the value of M is 2, the M frequency bands may be a high frequency band and a low frequency band. The value of M can be determined by a person skilled in the art according to the actual situation.

It should be understood that the image in the spatial domain may be transformed into the frequency domain, and the "decomposition" in step S120 is to split the complete image signal (e.g., the gray scale image or the channel image of the luminance channel) into M sub-signals (e.g., the gray scale sub-image or the luminance sub-image) in the frequency domain. The frequency ranges of the M frequency bands can have overlapping portions or no overlapping portions, but after the frequency bands are combined together, the frequency ranges at least can cover the distribution range of the complete image signal in the frequency domain. It should be noted that, decomposing the complete image signal does not mean that the original image signal does not exist after the decomposition is completed, and the original image signal may not exist or may continue to be retained.

For simplicity, the case where the frequency ranges of the M frequency bands have no overlapping portion, and the M frequency bands are combined together to cover the full frequency band (the full frequency band is covered by the distribution range of the complete image signal in the frequency domain necessarily), will be taken as an example, to describe how to decompose the complete image signal into sub-signals in the M frequency bands in the frequency domain.

In one implementation, M-1 frequency points separating adjacent frequency bands may be determined first, e.g., m=3, and two different frequency points may be determined first in the frequency domain: a and b, and a > b. Then, the full band can be further divided into M bands according to the M-1 frequency points: for example, 3 frequency bands can be divided according to a and b, the frequency is higher than a, the frequency is middle between a and b, and the frequency is lower than b (the two points of a and b belong to which frequency band is not greatly). Finally, a filter can be designed according to the frequency band division condition, the complete image signal is filtered, and the sub-signals in the corresponding frequency band are decomposed: for example, designing a low pass filter with a cut-off frequency b may result in a sub-signal in the low frequency band, etc. Of course, the step of determining the frequency points and the frequency bands may be only what needs to be considered when designing the filter, and the operation actually performed on the image is only filtering by using the filter.

It should be understood that the specific values of the frequency points in the above examples can be determined by those skilled in the art according to the actual situation, and this application is not limited thereto. The high frequency band, the medium frequency band and the low frequency band do not necessarily have to correspond to a fixed frequency range, but may be a relative concept (meaning that the frequency range of the high frequency band is larger than the frequency range of the medium frequency band, which is larger than the frequency range of the low frequency band).

Next, how the gray-scale image is decomposed by the filter will be described with reference to steps S121 to S123 in fig. 2:

step S121: and filtering the gray level image by using a first low-pass filter to obtain a gray level sub-image of a low frequency band.

The cut-off frequency of the first low-pass filter is the boundary between the low-frequency band and the middle-frequency band, namely corresponds to the frequency point b in the example above. The first low-pass filter allows the passage of image signals having frequencies lower than the boundary between the low-frequency band and the intermediate-frequency band in the gray-scale image (referred to as mono), while the passage of image signals having frequencies higher than the boundary between the low-frequency band and the intermediate-frequency band is not allowed, thereby obtaining a gray-scale sub-image (referred to as mono_low) of the low-frequency band.

Step S122: and filtering the gray level image by using a second low-pass filter to obtain a temporary gray level image, and calculating a gray level sub-image of the high frequency band according to the gray level image and the temporary gray level image.

The cut-off frequency of the second low-pass filter is the boundary between the middle frequency band and the high frequency band, namely, the cut-off frequency corresponds to the frequency point a, a > b in the example. The second low-pass filter allows the passage of image signals having frequencies lower than the boundary between the middle and high frequency bands in the gray-scale image, while the passage of image signals having frequencies higher than the boundary between the middle and high frequency bands is impossible, resulting in a temporary image (noted mono temp).

Since the second low-pass filter retains the image signal lower than the boundary between the intermediate frequency band and the high frequency band, it is known that the temporary gray image retained after filtering by the second low-pass filter contains the image information of the gray sub-image of the intermediate frequency band and the image information of the gray sub-image of the low frequency band, and the gray image contains the image information of the gray sub-image of the high frequency band, the image information of the gray sub-image of the intermediate frequency band and the image information of the gray sub-image of the low frequency band, so that the gray image and the temporary gray image can be subtracted to obtain the gray sub-image of the high frequency band (referred to as mono_high), for example, the subtraction can refer to the subtraction of the pixel values of the corresponding positions in the image.

Step S123: and calculating the gray sub-image of the middle frequency band according to the gray image, the gray sub-image of the low frequency band and the gray sub-image of the high frequency band.

The mid-band gray sub-image (denoted mono_mid) may be obtained by subtracting the high-band gray sub-image and the low-band gray sub-image from the gray image, e.g., mono_mid=mono-mono_high-mono_low.

Referring to fig. 3, fig. 3 is a schematic diagram of the above-mentioned decomposition process of the gray-scale image, wherein the horizontal axis shows the frequency (gradually decreasing toward the right), and the horizontal axis also shows two frequency points a and b (the boundary between the high frequency band and the middle frequency band, and the boundary between the middle frequency band and the low frequency band), and the frequency response curves of the first low-pass filter (the cut-off frequency is b), the second low-pass filter (the cut-off frequency is a), and the signal curve of the gray-scale image mono. The resolution of the gray scale image by the low pass filter can be seen from fig. 3: filtering by a first low-pass filter to obtain a gray sub-image mono_low of a low frequency band; a temporary gray image (mono_low+mono_mid) obtained by filtering of the second low-pass filter; then, a gray sub-image (mono_high=mono-mono_temp) of a high frequency band and a gray sub-image (mono_mid=mono-mono_high-mono_low) of a medium frequency band can be calculated according to the gray image, the temporary gray image and the gray sub-image of a low frequency band, and the image decomposition mode is simple and efficient.

In an alternative, the mid-band gray sub-image may also be calculated based on the temporary gray image and the low-band gray sub-image based on steps S121-S122, without performing step S123, e.g., mono_mid=mono_temp-mono_low.

In addition, the selection of the first low-pass filter and the second low-pass filter in the above steps S121 to S123 is not limited in this application, and for example, a gaussian filter, a butterworth filter (butterworth), a box filter (box filter), or the like may be used.

Taking a gaussian filter as an example, the cut-off frequency of the gaussian filter can be adjusted by adjusting the sigma value of the gaussian filter, and the larger the sigma, the lower the cut-off frequency. For example, sigma=3 may be set for the first low-pass filter in step S121 described above, and sigma=1 may be set for the second low-pass filter in step S12. Note that, when designing a filter, those skilled in the art may not need to know exactly what the cut-off frequency corresponding to the gaussian filter with sigma=3 is, and it is only necessary to roughly know that the filter can filter out a low-frequency signal.

Steps S121-S123 use a low pass filter to decompose the gray scale image, and in other alternatives, use a high pass filter to decompose the image, such as:

Step A1: and filtering the gray level image by using a first high-pass filter to obtain a sub-gray level image of the high frequency band.

The cut-off frequency of the first high-pass filter is the boundary between the middle frequency band and the high frequency band. The first high-pass filter allows the passage of image signals having frequencies higher than the boundary between the intermediate frequency band and the high frequency band in the gray scale image, while the passage of image signals having frequencies lower than the boundary between the intermediate frequency band and the high frequency band is not allowed, thereby obtaining a gray scale sub-image of the high frequency band.

Step A2: and filtering the gray level image by using a second high-pass filter to obtain a temporary gray level image, and calculating a gray level sub-image of the low frequency band according to the gray level image and the temporary gray level image.

The cut-off frequency of the second high-pass filter is the boundary between the low frequency band and the middle frequency band. The second high-pass filter allows the image signal having a frequency higher than the boundary between the low-frequency band and the intermediate-frequency band in the gray-scale image to pass, while the image signal having a frequency lower than the boundary between the low-frequency band and the intermediate-frequency band cannot pass, thereby obtaining a temporary gray-scale image (still noted as mono_temp for simplicity, but noted that the meaning is different from that of the temporary gray-scale image in step S122).

Then, the grayscale image and the first temporary grayscale image are subtracted to obtain a low-band sub-image, for example, mono_low=mono-mono_temp.

Step A3: and calculating the gray sub-image of the middle frequency band according to the gray image, the gray sub-image of the low frequency band and the gray sub-image of the high frequency band.

Similar to step S123 described above, the gray sub-image of the mid-band can be represented by the formula: mono_mid=mono-mono_high-mono_low.

It is obvious that the resolution of the gray-scale image may be performed by using a combination of the low-pass filter and the high-pass filter at the same time, or the resolution of the gray-scale image may be performed by using another filter (e.g., a band-pass filter), which is not illustrated one by one.

It should be understood that the manner of decomposing the M-frame luminance sub-image from the channel image of the luminance channel is similar to the manner of decomposing the M-frame gray sub-image from the gray image described above, and will not be described herein.

The inventor researches that the sub-images (which may be gray sub-images or luminance sub-images) in different frequency bands have different characteristics, so that different fusion strategies can be adopted for the sub-images in different frequency bands generated by decomposition (see step S130 in detail), and thus the fusion effect may be better than that of the original images (which may be gray images or channel images of luminance channels) directly. For example, for the case of m=3, the high-band sub-image mainly represents tiny details (e.g., edges, textures) in the original image, the middle-band sub-image mainly represents slightly larger details in the original image, and the low-band sub-image mainly represents a place with gentle variation in the original image, which determines the overall brightness and contrast of the image. In addition, since the division of 3 frequency bands does not cause excessive frequency bands to greatly increase the complexity of calculation, the scheme of the application is mainly exemplified by the case of m=3 in the description.

Step S130: and fusing the M-frame gray level sub-images and the M-frame brightness sub-images to obtain M-frame fused sub-images.

Each frame of gray sub-image is used for fusing with one frame of brightness sub-image of a corresponding frequency band, namely, the image fusion of frequency division is carried out on M frequency bands.

In an alternative solution, step S130 may calculate an M-frame fusion mask, and use the M-frame fusion mask to fuse the M-frame gray sub-image and the M-frame luminance sub-image, which specifically includes the following steps:

step B1: and calculating M frame fusion masks corresponding to the M frequency bands according to the gray level sub-images and the brightness sub-images.

Step B2: and carrying out weighted fusion on the M-frame gray sub-images and the M-frame brightness sub-images by using an M-frame fusion mask to obtain M-frame fusion sub-images.

The fusion mask is a weighted image, i.e., each pixel value in the fusion mask characterizes a weight. Each frame of fusion mask corresponds to a frequency band and is used for fusing a frame of gray sub-image and a frame of brightness sub-image corresponding to the frequency band.

Specifically, the dimensions of the fusion mask, the corresponding gray sub-image, and the corresponding luminance sub-image are the same, and each pixel value in the fusion mask characterizes: and the fusion weights of the pixel values at the same positions in the corresponding brightness sub-image and gray sub-image. Accordingly, the corresponding luminance sub-image and gray sub-image can be subjected to pixel-by-pixel weighted fusion (weighted summation) using the fusion mask.

Optionally, if each pixel value in the fusion mask is a single value, only the fusion weight corresponding to one of the luminance sub-image and the gray sub-image can be directly represented, but the fusion weight corresponding to the other one can be calculated according to the pixel value, so that the fusion weights of the luminance sub-image and the gray sub-image can be regarded as the content represented by the pixel values in the fusion mask.

For example, if a pixel value m (a value range [0,1 ]) at a certain pixel position in the fusion mask directly represents a fusion weight corresponding to the gray sub-image at the pixel position, the fusion weight corresponding to the luminance sub-image at the pixel position may be 1-m. At this time, the fusion process can also be formulated as (m=3):

fusion_high＝colour_y_high*(1-mask_high)+mono_high*mask_high

fusion_mid＝colour_y_mid*(1-mask_mid)+mono_mid*mask_mid

fusion_low＝colour_y_low*(1-mask_low)+mono_low*mask_low

wherein fusion_high is a fusion sub-image of a high-frequency band, color_y_high is a brightness sub-image of the high-frequency band, mask_high is a fusion mask of the high-frequency band (wherein pixel values are fusion weights corresponding to mono_high); fusion_mid is a fusion sub-image of the middle frequency band, color_y_mid is a brightness sub-image of the middle frequency band, mask_mid is a fusion mask of the middle frequency band (wherein the pixel value is fusion weight corresponding to mono_mid); fusion_low is a fusion sub-image of the low frequency band, color_y_low is a brightness sub-image of the low frequency band, mask_low is a fusion mask of the low frequency band (wherein the pixel value is fusion weight corresponding to mono_low). Note that all of the above formulas are performed by pixel operation, meaning that the weighted summation is performed according to the above formulas for the pixels at the same position in the image involved in the calculation.

The pixel values in the fusion mask may take on consecutive values, such as the values in [0,1 ]. For example, if the pixel value at a certain pixel position in the fusion mask directly represents the fusion weight of the gray sub-image at the pixel position, the probability that the definition of the gray sub-image at the pixel position is higher than that of the brightness sub-image can be used as the pixel value, which means that if the gray sub-image at the pixel position is clearer, the pixel value of the fusion sub-image at the pixel position should adopt more information in the gray sub-image, otherwise, the pixel value of the fusion sub-image at the pixel position should adopt more information in the brightness sub-image.

Alternatively, the pixel values in the fusion mask may take discrete values, for example, the fusion mask may be a binary image, where the pixel values in the binary image are only two values, and the first value (e.g., 1) or the second value (e.g., 0) represents different meanings in weighted fusion. For example, taking 1 indicates that the pixel value of the corresponding fusion sub-image at the same position as the fusion mask adopts the pixel value in the corresponding gray sub-image (corresponding to the above formula, the fusion weight corresponding to the luminance sub-image is 0, so that the luminance sub-image does not affect the weighting result, the weighting result is completely from the gray sub-image), taking 0 indicates that the pixel value of the corresponding fusion sub-image at the same position adopts the pixel value in the corresponding luminance sub-image (corresponding to the above formula, the fusion weight corresponding to the gray sub-image is 0, so that the gray sub-image does not affect the weighting result, and the weighting result is completely from the luminance sub-image).

In particular, since the discrete values 0 and 1 also belong to [0,1], 0 and 1 can also be regarded as the probability that the sharpness of the gray sub-image is higher than that of the luminance sub-image.

As to how to calculate M frame fusion masks corresponding to M frequency bands from the gray sub-image and the luminance sub-image, there are at least two cases: firstly, calculating an M-frame fusion mask by using all gray sub-images and brightness sub-images; second, an M-frame fusion mask is calculated using a portion of the gray sub-image and the luminance sub-image. For the specific calculation modes of the above two cases, see the description below.

It should be appreciated that there are also ways to fuse the M-frame gray sub-image and the M-frame luminance sub-image without using a fusion mask, as will be described in detail below.

Step S140: and superposing the M frames of fusion sub-images to obtain a brightness fusion image.

Because the M-frame gray sub-images and the M-frame brightness sub-images are generated by decomposition, each frame of gray sub-image and each frame of brightness sub-image only contain part of image information of the gray image and the channel image of the brightness channel, and correspondingly, each frame of fused sub-image after fusion only contains part of image information of the brightness fused image. For example, the grayscale sub-image of the low band and the luminance sub-image of the low band include only the grayscale image and the image information of the low band of the channel image of the luminance channel, respectively, and the fusion sub-image of the low band also includes only the image information of the low band of the luminance fusion image.

Therefore, the M-frame fusion sub-images need to be overlapped to obtain a brightness fusion image containing complete image information, where "overlapping" can be understood as an operation of merging image information of different frequency bands, and in an implementation manner, the pixel values of corresponding positions of the M-frame fusion sub-images may be added to obtain the pixel value of the brightness fusion image at the same position, and may be expressed as:

fusion＝fusion_high+fusion_mid+fusion_low

wherein, fusion is a luminance fusion image, fusion_high is a fusion sub-image of a high frequency band, fusion_mid is a fusion sub-image of a medium frequency band, and fusion_low is a fusion sub-image of a low frequency band. Note that all of the above formulas are performed as pixel-by-pixel operations.

The inventor researches find that some ways of decomposing the gray level image and the channel image of the brightness channel may cause that pixel values exceeding a preset pixel value range (for example, [0,255 ]) exist in the obtained gray level sub-image and/or brightness sub-image, so that the finally obtained brightness fusion image also has out-of-range pixel values, which does not meet the requirements of some image formats.

For example, m=3, in the above step S123, the gray sub-image of the mid-band may be obtained by calculating the formula mono_mid=mono_high-mono_low, and in the subtraction process, the pixel value in mono_mid may be negative. Further, the fused sub-image of the mid-band is calculated by the formula fusion_mid=color_y_mid+mask_mid+mono_mid (1-mask_mid), and the pixel value in fusion_mid may also be negative because the pixel value in mono_mid may be negative. However, according to the above superposition formula, the luminance fusion image fusion=fusion_high+fusion_mid+fusion_low, and since the pixel value in the fusion_mid may be negative, the pixel value in the fusion may be negative, that is, the pixel value range beyond [0,255] may be obtained.

In one implementation, the pixel value range of the luminance fusion map may be controlled within a preset pixel value range by the following method.

For example, m=3, the preset pixel value range is [0,255], and the above superposition formula can be appropriately modified into the following form:

fusion＝Min(Max(fusion_high+fusion_mid+fusion_low，0)，255)

wherein, the Max function is used for controlling the pixel value of the brightness fusion image to be more than 0 (comprising 0), and the Min function is used for controlling the pixel value of the brightness fusion image to be less than 255 (comprising 255). Note that pixel-by-pixel operations are performed in the above formulas.

Step S150: and combining the brightness fusion image and the channel image of the chromaticity channel to obtain a color fusion image.

Because the gray level image and the channel image of the brightness channel are all single-channel images, the brightness fusion image generated after the two images are fused is also a single-channel image and only contains the brightness information of the image, so that the color fusion image simultaneously containing the brightness information and the color information can be obtained by combining the brightness fusion image and the channel image of the chromaticity channel. "bonding" herein is understood to mean channel stitching.

The image fusion method in fig. 1 is briefly summarized below: the method provides a method for fusing gray level images and color images in different frequency bands, and the sub-images in different frequency bands have different characteristics, so that the fusion mode of the frequency bands ensures that the image fusion process is finer, and the quality of the color fusion image is improved.

In one implementation mode of the method, the gray level image and the color image can be fused in a frequency division manner by utilizing fusion masks of different frequency bands, wherein the fusion masks are calculated according to the gray level sub-image and the brightness sub-image, namely, the pixel values (representing fusion weights) in the fusion masks are determined by the content adaptation of the gray level sub-image and the brightness sub-image, so that the fusion according to the fusion masks does not necessarily tend to adopt information in the gray level sub-image more, and the color fusion image with higher quality can be obtained in a high-brightness environment by the implementation mode, and the blank of the prior art is made up. Of course, for the dark light environment, the image fusion quality can be effectively improved compared with the existing method due to the adoption of the frequency division and the section fusion.

Next, based on the above embodiments, how to calculate an M-frame fusion mask when fusion is performed on an M-frame gray sub-image and an M-frame luminance sub-image using the fusion mask will be described in detail.

Taking the case of m=3 as an example, in one implementation, the fusion mask for the three high, medium, and low bands may be calculated by performing step D1 and step D2.

Step D1: calculating a fusion mask of the middle frequency band according to the gray sub-image of the middle frequency band and the brightness sub-image of the middle frequency band; and calculating a fusion mask of the high frequency band according to the gray level sub-image of the high frequency band and the brightness sub-image of the high frequency band.

First, how to calculate the fusion mask of the mid-band:

step E1: taking absolute values of pixel values in the gray sub-images of the middle frequency band to obtain a first mask calculation image; and taking the absolute value of the pixel value in the luminance sub-image of the middle frequency band to obtain a second mask calculated image.

According to the foregoing, when the image to be fused is decomposed, because a subtraction calculation mode is adopted, the pixel values in the gray sub-image and the brightness sub-image of the middle frequency band may have negative numbers, so in order to facilitate the subsequent comparison of the pixel values, the absolute values of the pixel values in the gray sub-image and the brightness sub-image of the middle frequency band need to be taken before the fusion mask of the middle frequency band is calculated, so as to ensure that the pixel values in the image are converted into non-negative numbers. In addition, the pixel values may be converted into non-negative numbers in other manners, for example, the pixel values in the gray sub-image of the middle band and the luminance sub-image of the middle band may be squared to obtain the first mask calculation image and the second mask calculation image, respectively.

Step E2: and calculating a fused mask of the middle frequency band according to the magnitude relation between the pixel values of the first mask calculation image and the second mask calculation image at the same position.

For example, if the fused mask of the middle band is a binarized image, that is, the pixel value in the fused mask of the middle band takes 1 (one possible value of the first value), the pixel value in the gray sub-image of the middle band is adopted by the pixel value in the fused sub-image of the middle band at the same position, and if the pixel value in the fused mask of the middle band takes 0 (one possible value of the second value), the pixel value in the fused sub-image of the middle band at the same position is adopted by the pixel value in the luminance sub-image of the middle band. At this time, the fused mask of the mid-band can be calculated according to the following rule:

for a pixel value at a certain pixel position in the first mask calculation image, if the pixel value is larger than the pixel value at the same pixel position in the second mask calculation image, the pixel value of the fusion mask of the middle frequency band at the pixel position is 1; otherwise, if the pixel value is not greater than the pixel value at the same pixel position in the second mask calculation image, the pixel value of the fused mask of the middle frequency band at the pixel position is taken to be 0. This rule can be expressed in pseudo-code as follows:

wherein, |mono_mid| is the first mask calculated image, |color_y_mid| is the second mask calculated image, and the symbol || represents taking absolute value. Note that the above pseudo code is performed pixel by pixel, meaning that the above pseudo code is performed one by one for the pixels at the same position in the image that participates in the comparison.

The principle of calculation can be that the gray sub-image and the luminance sub-image of the middle frequency band respectively represent detail information in the gray image and the channel image of the luminance channel, the first mask calculation image and the second mask calculation image are only the result that the gray sub-image and the luminance sub-image of the middle frequency band take absolute values, and therefore the detail information in the channel image of the gray image and the luminance channel respectively represent detail information, the detail information reflects the definition of the channel image of the gray image and the luminance channel, and the greater the pixel value, the more obvious the image detail and the higher the image definition.

For example, at a certain pixel position, if the pixel value in the first mask calculated image is larger, the definition of the gray scale image at the pixel position is higher than that of the channel image of the brightness channel, so the pixel value of the fusion mask of the middle frequency band at the pixel position should be set to be 1, which means that the pixel value in the gray scale sub-image of the middle frequency band is selected during fusion; if the pixel value in the second mask calculated image is larger, the definition of the channel image of the brightness channel at the pixel position is higher than that of the gray level image, so the pixel value of the fusion mask of the middle frequency band at the pixel position is set to be 0, and the pixel value in the brightness sub-image of the middle frequency band is selected during fusion.

In an alternative scheme of steps E1-E2, considering that the imaging quality of the gray level image and the color image in different environments is different, the direct pixel value comparison result may not be accurate, so that the difference in the image acquisition environment can be reflected by setting an adjustment threshold, the pixel value of the luminance sub-image is adjusted according to the adjustment threshold, and then the adjustment result is compared with the pixel value of the gray level sub-image to calculate the fusion mask (the pixel value of the gray level sub-image can be adjusted, and the method is similar), so that the calculation accuracy of the fusion mask is higher.

For example, for a first mask to calculate a pixel value of an image at a certain pixel position, if the pixel value is greater than the product of the pixel value of a second mask to calculate the image at the position and an adjustment threshold, the pixel value of a fused mask of a middle frequency band at the pixel position takes 1; otherwise, if the pixel value is not greater than the product of the pixel value of the second mask calculation image at the position and the adjustment threshold value, the pixel value of the fusion mask of the middle frequency band at the pixel position is taken as 0. The adjustment threshold may be a constant, for example, 0.5, 0.6, etc., and the adjustment threshold may be a preset value or a temporary calculated value, where the above rule is expressed by a pseudo code as follows:

Here, thr_mid is the adjustment threshold corresponding to the intermediate frequency band, and in particular, since 1 may also be the value of the adjustment threshold, the above scheme of the fused mask of the intermediate frequency band in the calculation without setting the adjustment threshold may be regarded as the case where thr_mid=1 in the above scheme of the fused mask of the intermediate frequency band in the calculation with setting the adjustment threshold. Note that the above pseudo code is executed pixel by pixel.

If the fused mask of the mid-band is not a binarized image, the fused mask of the mid-band may also be calculated in a similar manner to the above, for example, in step E2, the difference between the pixel values of the first mask calculation image and the second mask calculation image at the same position may be calculated, and the difference may be mapped to a value in [0,1] according to a preset mapping rule (but not only 0 and 1), as the pixel value of the fused mask of the mid-band at this position. For example, if the difference between the pixel values of the first mask calculation image and the second mask calculation image at a certain position is 200, the pixel value of the fusion mask of the prescribed middle band at the certain position is taken to be 0.9, if the difference between the pixel values of the first mask calculation image and the second mask calculation image at a certain position is 0, the pixel value of the fusion mask of the prescribed middle band at the certain position is taken to be 0.5, and so on.

In addition, if some image decomposition modes ensure that the pixel values in the gray sub-image and the brightness sub-image of the middle frequency band cannot generate negative numbers, the pixel values in the gray sub-image and the brightness sub-image of the middle frequency band can be directly compared, so that the pixel values in the fusion mask of the middle frequency band can be determined, and the first mask calculation image and the second mask calculation image are not required to be calculated.

The calculation method of the high-frequency band fusion mask may refer to the calculation method of the intermediate-frequency band fusion mask, for example, refer to the above steps E1-E2 or the alternative scheme of steps E1-E2 (adjustment threshold is set), and if the calculation method in the alternative scheme is adopted, it should be noted that the value of the adjustment threshold corresponding to the high-frequency band and the value of the adjustment threshold corresponding to the intermediate-frequency band may be the same or different.

Step D2: and calculating the fusion mask of the low frequency band according to the fusion mask of the medium frequency band and the fusion mask of the high frequency band.

For example, if the fused mask of the middle band is a binarized image, and the definition refers to the example in step D1, the fused mask of the low band may be calculated according to the following rule: if the pixel values of the fusion mask of the middle frequency band and the fusion mask of the high frequency band at the same position are both 1, the pixel value of the fusion mask of the low frequency band at the position is set to 1 (the fusion sub-image of the low frequency band adopts the pixel value of the gray sub-image of the low frequency band at the pixel position), otherwise, the pixel value of the fusion mask of the low frequency band at the position is set to 0 (the fusion sub-image of the low frequency band adopts the pixel value of the brightness sub-image of the low frequency band at the pixel position).

As described above, the sub-images (or the images further calculated based on the sub-images, such as the first mask calculation image) located in the high-band and the medium-band, and the pixel value size reflects the image sharpness, so that the pixel value (who clearly takes) in the fused mask of the high-band and the medium-band can be determined directly according to the size relationship between the pixel values; the sub-image (or the image further calculated based on the sub-image) located in the low frequency band has the pixel value only representing the overall brightness or contrast of the image, and basically does not determine the definition of the image, so that the fusion mask for calculating the low frequency band directly through the pixel value comparison is not accurate enough. Therefore, in step D2, the fusion mask of the low frequency band is calculated from the fusion mask of the middle frequency band and the fusion mask of the high frequency band, instead of directly calculating from the sub-image of the low frequency band.

In addition, in the above example, the color image is used as the fusion reference, that is, the grayscale image is fused to the color image, so that only when the image sharpness of the grayscale image is higher in both the high-frequency and middle-frequency portions than in the color image (the pixel values of the fusion mask in the middle-frequency and the fusion mask in the high-frequency are all 1), the information of the grayscale image is used in the low-frequency portion (the pixel value of the fusion mask in the low-frequency is 1), otherwise, the information of the color image is used (the pixel value of the fusion mask in the low-frequency is 0), that is, the information in the color image is more preferred.

Of course, the rule may also be adjustable, for example, the following rule may also be set: as long as one pixel value of the fusion mask of the middle frequency band and the fusion mask of the high frequency band at the same position takes 1, the pixel value of the fusion mask of the low frequency band at the position is set to 1, otherwise, the pixel value of the fusion mask of the low frequency band at the position is set to 0.

In addition, if the gray level image is used as a fusion reference, that is, the color image is fused to the gray level image, a corresponding set of rules may be set, which will not be described again.

In an alternative scheme, in order to facilitate statistics of the values of the pixel values of the fusion mask of the middle frequency band and the fusion mask of the high frequency band at the same position, a confidence coefficient flag may be set for each frequency band, where the confidence coefficient flag takes a boolean value, and if the pixel value of the gray sub-image at a certain pixel position is greater than the pixel value of the luminance sub-image at the pixel position, the confidence coefficient flag is set to true, otherwise, the confidence coefficient flag is set to false. If the confidence coefficient marks of the high frequency band and the medium frequency band are both true, setting the pixel value of the fusion mask of the low frequency band at the position to be 1, otherwise setting the pixel value of the fusion mask of the low frequency band at the position to be 0. Taking the calculation mode including the adjustment threshold as an example, the above rule can be expressed by pseudo code as follows:

The method comprises the steps of obtaining an image obtained by taking an absolute value of a pixel value in a gray level sub-image of a high frequency band as an I mono_high I, obtaining an image obtained by taking an absolute value of a pixel value in a brightness sub-image of the high frequency band as an I color_y_high I, obtaining a flag_mid as a confidence coefficient mark of a medium frequency band, obtaining a flag_high as a confidence coefficient mark of the high frequency band, obtaining thr_mid as an adjustment threshold value corresponding to the medium frequency band, obtaining thr_high as an adjustment threshold value corresponding to the high frequency band, and performing AND operation by & & representation, wherein the confidence coefficient mark is not required to be set for the low frequency band. Note that the above pseudo code is executed pixel by pixel.

Further, the inventors found that there are color images that have black and white edges before fusion due to an algorithm of an image processor (Image Signal Processor, abbreviated as ISP) and the like, and that the prior art cannot well process the black and white edges when fusion of black and white color images is performed, so that the black and white edges still exist in a fusion result, and image quality is seriously affected. In the following, mainly white edges are taken as examples, and possible solutions are described, black edges being similar.

Referring to fig. 4, wherein the top is the color image, there is a white border at the black vertical bar indicated by the arrow in the upper left corner of the color image; the middle is a gray image, the color image is compared, and white edges do not appear in the gray image at the same position.

If the gray-scale image and the color image are fused in the manner described in the steps D1-D2, for convenience of explanation, it is not easy to assume that the gray-scale image is clearer than the color image (the condition is basically satisfied in the dark environment), that is, the pixel value (taking absolute value) in the middle-high frequency sub-image of the gray-scale image is generally larger than that in the brightness sub-image, but for the color fusion image, the image information of the gray-scale image is used in a large probability only in a narrow range at the edge of the black vertical bar (because the edge belongs to the middle-high frequency, the gray-scale image is clearer than the color image), and the image information of the color image is used in the remaining large probability (because the remaining part belongs to the low frequency, the difference between the definition of the gray-scale image and the color image is not obvious), which leads to the fact that the color fusion image still has white edges in the peripheral part of the black vertical bar, because the white edges have a certain width in the color image, and are not only distributed in a narrow range at the edge of the black vertical bar.

Therefore, taking the middle frequency case as an example, in an alternative, after the first mask calculation image and the second mask calculation image are obtained in the step D1, the non-edge-preserving smoothing filter may be further used to smooth the first mask calculation image and the second mask calculation image, so as to improve the problem that white edges occur in the color fusion image, and as a result, referring to the lowest image in fig. 4, it can be seen that the color fusion image has no obvious white edges at the black vertical bars. The specific practice of this scheme is as follows:

Step D3 (performed following step D1, step D2 is not performed): and respectively filtering the first mask calculated image and the second mask calculated image by using the non-edge-protection smoothing filter to obtain a third mask calculated image and a fourth mask calculated image.

Step D4: and calculating a fused mask of the middle frequency band according to the magnitude relation between the pixel values of the third mask calculation image and the fourth mask calculation image at the same position.

The non-edge-preserving smoothing filtering is a type of smoothing filtering algorithm, and unlike edge-preserving smoothing filtering algorithms (e.g., guided filtering, bilateral filtering), the filtering algorithm does not preserve the edges of the image during filtering, i.e., performs indiscriminate smoothing processing on the image content, e.g., box filtering and gaussian filtering belong to the non-edge-preserving smoothing filtering.

In the step D4, the method for calculating the fused mask of the frequency band according to the third mask calculation image and the fourth mask calculation image is similar to the method for calculating the fused mask of the frequency band according to the first mask calculation image and the second mask calculation image in the previous step D2, only the first mask calculation image is replaced by the third mask calculation image, and the second mask calculation image is replaced by the fourth mask calculation image, so that the description is omitted.

The principle of white edge elimination in steps D3-D4 is briefly analyzed as follows:

the non-edge-preserving smoothing filter can blur edges in the image or can widen the distribution range of the edges in the image, and after the edges are widened, the edges still belong to middle-high frequency bands in the image although frequency components of the edges are reduced. In step D3, the edge portions of the black bars in the third mask calculated image and the fourth mask calculated image are widened, the widened edges cover the area where the white edges are located, and in the widened edge areas of the portions, the pixel values in the third mask calculated image are still larger (the gray scale image is clearer than the color image and is shown as the pixel values at the intermediate frequency are larger) as a whole, so that the area where the white edges are located still has a high probability of adopting the image information of the gray scale image rather than the image information of the color image, and the white edges in the color fusion image are eliminated or at least weakened.

It should be understood that the fusion mask of the high-frequency band may also be calculated in a manner similar to steps D1, D3, and D4 to eliminate or weaken the black-white edges in the color image, which is not described herein.

The above-described method of calculating the fusion mask does not use all of the M-frame gray scale sub-images and the M-frame luminance sub-images: for example, the gray sub-image of the low band and the luminance sub-image of the low band are not used, the reason for which has been analyzed above. However, there are schemes for calculating the M-frame fusion mask using all M-frame gray level sub-images and M-frame luminance sub-images, for example, the fusion mask of the low frequency band may be calculated by the same method as that for calculating the fusion mask of the medium and high frequency bands instead of the method of step D2.

It should be noted, however, that since the gray sub-image and the luminance sub-image of the low frequency band contain substantially no detail information of the image, the pixel value size thereof mainly characterizes the overall luminance and contrast of the image. Therefore, after the step S110 is performed, and before the step S120 is performed, the brightness of the gray-scale image and the color image needs to be adjusted to be consistent, so as to eliminate or weaken the influence of the brightness on the pixel value, so that the result of comparing the pixel values in the low frequency band is more significant, and the result of calculating the fusion mask in the low frequency band is more accurate.

The brightness uniformity adjustment here includes at least two cases: in the first case, when the color image is used as the fusion reference, the brightness of the color image is used as the reference for brightness adjustment, and the brightness of the gradation image is adjusted so that the brightness of the gradation image matches the brightness of the color image. In the second case, when the gradation image is used as the fusion reference, the brightness of the gradation image is used as the reference for brightness adjustment, and the brightness of the color image is adjusted so that the brightness of the color image matches the brightness of the gradation image. As to how to adjust the brightness, the present application is not limited, and for example, a histogram matching algorithm may be used to make a consistency adjustment for brightness, and so on.

In particular, since only the pixel values in the luminance channel represent luminance in the color image, the luminance adjustment of the color image may be performed only on the luminance channel.

It should be noted that the term "uniform" does not mean that the luminance of two frames of images is identical, but means that the luminance of two frames of images is substantially at the same level, and may be identical or may have a certain difference.

Some methods for calculating the fusion mask are described above, in which the fusion mask is calculated to fuse the gray sub-image and the luminance sub-image, but in some alternative schemes, the fusion mask may not be calculated, and fusion may be directly performed according to the magnitude relation between the pixel values of the gray sub-image and the luminance sub-image, so as to obtain an M-frame fusion sub-image.

For example, taking the case of mid-band as an example, the fusion may be performed according to the following rule: if the pixel value in the gray sub-image of the middle frequency band is larger than the pixel value in the brightness sub-image of the middle frequency band at a certain pixel position, the fused sub-image of the middle frequency band adopts the pixel value of the gray sub-image of the middle frequency band at the pixel position, otherwise, the pixel value of the brightness sub-image of the middle frequency band is adopted, so that the aim of selecting the gray sub-image of the middle frequency band and the clearer sub-image in the brightness sub-image of the middle frequency band as the fusion result is fulfilled.

Further, if the pixel value in the gray sub-image of the middle frequency band is larger than the pixel value in the luminance sub-image of the middle frequency band, the pixel value at the position in one frame of binarized image is set to be 1, otherwise, the pixel value at the position in the binarized image is set to be 0, after the pixel value in the binarized image is set, the pixel values are regarded as fusion weights and are used for fusing the gray sub-image of the middle frequency band and the luminance sub-image of the middle frequency band, and compared with the previous contents, the binarized image is the binarized middle frequency band fusion mask. Therefore, the above example of not calculating the fusion mask can achieve the same effect as the case of calculating the fusion mask and the fusion mask being a binarized image.

In the following, on the basis of the above embodiment, it is described how to decompose M-frame gray sub-images from gray images in consideration of occlusion:

because the positions of the black-and-white camera and the color camera on the electronic device are different, parallax exists between the gray-scale image and the color image acquired by the two cameras, namely, a part of the region exists in the color image only and does not exist in the gray-scale image (or is blocked in the gray-scale image), and another part of the region exists in the gray-scale image only and does not exist in the color image (or is blocked in the color image). As to how to calculate the occlusion region is not limited in this application, for example, it can be calculated by LR-Check algorithm.

If the color image is used as a fusion reference, an area which exists only in the color image but does not exist in the gray image in a shooting scene can be defined as an occlusion area, and the image information of the gray image in the occlusion area is supplemented by the image information of the color image in the occlusion area so as to ensure that scene areas included in the color image and the gray image are consistent, thereby improving the subsequent image fusion effect.

To achieve the above object, in one implementation, the decomposing process of the gray image in step S120 may further include the sub-steps of:

step C1: and calculating an occlusion region mask according to the gray level image and the channel image of the brightness channel.

Step C2: and carrying out weighted fusion on the gray level image and the channel image of the brightness channel by using the shielding region mask to obtain a gray level shielding fusion image.

Step C3: and decomposing M frames of gray sub-images from the gray shielding fusion image.

The occlusion region mask is an image describing the position of the occlusion region, or calculating the occlusion region mask may also be considered as calculating the position of the occlusion region. The size of the occlusion region mask and the size of the gray scale image and the color image are the same. Optionally, each pixel value in the occlusion region mask characterizes: the probability (the numerical value in [0,1 ]) that the position of the pixel value belongs to the occlusion region is taken. In particular, if the occlusion region mask is a binary image, for example, the pixel values thereof only take two values of 0 and 1, taking 0 to indicate that the current pixel does not belong to the occlusion region and taking 1 to indicate that the current pixel belongs to the occlusion region, the set formed by the pixels with the values of 1 is an accurate description of the position of the occlusion region.

Further, in order to supplement the image information of the gray image in the occlusion region with the image information of the color image in the occlusion region, the occlusion region mask may also be regarded as a weighted image, i.e. each pixel value in the occlusion region mask is further characterized: the channel image of the luminance channel corresponds to the fusion weight at that pixel location. Therefore, the channel image of the brightness channel and the gray image can be weighted and fused (weighted and summed) by using the shielding region mask, so that the effect of supplementing the image information of the gray image in the shielding region by using the channel image of the brightness channel is achieved.

The principle of the method can be understood that if the pixel value in the shielding region mask is calculated reasonably, the pixel value (fusion weight) in the shielding region mask is larger (approaching or equal to 1) in the real shielding region, and the pixel value (fusion weight) in the shielding region mask is smaller (approaching or equal to 0) outside the shielding region, so that in the gray shielding fusion image obtained after fusion, the image information in the shielding region is derived or mainly derived from the channel image of the brightness channel, and the image information outside the shielding region still retains or basically retains the original image information in the gray image, namely the image information which cannot be acquired by the gray image in the shielding region is effectively supplemented, thereby ensuring that the channel image of the brightness channel is consistent with the scene area included by the gray image, and further improving the subsequent image fusion effect.

When the weighting fusion is carried out, the fusion weight corresponding to the gray level image is directly characterized by the pixel value in the mask of the non-shielded area, but can be calculated according to the pixel value in the mask of the shielded area. For example, if a pixel value n (a value range [0,1 ]) at a certain pixel position in the mask of the occlusion region represents a fusion weight corresponding to a channel image of a luminance channel at the pixel position, the fusion weight corresponding to a gray image at the pixel position may be 1-n. At this time, the fusion process can also be formulated as: mono '=color_y + (1-n) mono, where color_y is a channel image of a luminance channel, mono is a gray image, and mono' is a gray occlusion fusion image. Note that all of the above formulas are performed as pixel-by-pixel operations.

As an alternative, each pixel value in the occlusion region mask can also be characterized: the probability that the position of the pixel value does not belong to the shielding region can be referred to the previous description for how the shielding region mask in the scheme is used, and the description is omitted.

The above describes some calculation methods of the mask of the shielding region, which are used for realizing the supplementation of the image information of the gray level image in the shielding region by the image information of the color image in the shielding region through the fusion of the channel image of the brightness channel and the gray level image, but in some alternative schemes, the mask of the shielding region is not calculated, and the two are fused directly according to whether the pixel position belongs to the shielding region or not, so as to obtain the gray level shielding fusion image.

For example, the fusion may be performed according to the following rules: if the pixel position is judged to belong to the shielding area, the gray shielding fusion image adopts the pixel value of the channel image of the brightness channel at the pixel position, otherwise, the pixel value of the gray image is adopted.

Further, if a certain pixel position belongs to an occlusion region, setting a pixel value at the position in a frame of binary image to be 1, otherwise, setting a pixel value at the position in the binary image to be 0, and after the pixel value in the binary image is set, regarding the pixel values as fusion weights for fusing a gray sub-image and a channel image of a brightness channel, wherein the binary image is a binary occlusion region fusion mask as can be seen easily when compared with the previous. Therefore, the above example of not calculating the occlusion region fusion mask can achieve the same effect as the case of calculating the occlusion region fusion mask and the occlusion region fusion mask being a binary image.

For the specific implementation of step C3, reference may be made to the foregoing, for example, steps S121-S123 or steps A1-A3, and will not be described here again.

In another implementation manner, if the gray image is used as the fusion reference, a region that exists only in the gray image but does not exist in the color image in the photographed scene is defined as an occlusion region, and the image information of the gray image in the occlusion region is supplemented to the channel image of the luminance channel, so that the channel image of the luminance channel and the scene region included in the gray image are consistent.

At this time, each pixel value in the occlusion region mask characterizes: the probability that the position of the pixel value belongs to the occlusion region (the definition of the occlusion region is different from the previous one), or the corresponding fusion weight of the gray image at the pixel position. And (3) carrying out weighted fusion on the gray level image and the channel image of the brightness channel by using the shielding region mask to obtain a brightness shielding fusion image, and then decomposing M frames of brightness sub-images from the brightness shielding fusion image to serve as a fusion basis of the subsequent step S130 (fusion of the M frames of gray level sub-images and the M frames of brightness sub-images).

Fig. 5 shows functional components included in the image fusion apparatus provided in the embodiment of the present application. Referring to fig. 5, the image fusion apparatus 200 includes:

an image acquisition component 210 for acquiring a gray image and a color image to be fused; wherein the color image includes a channel image of a luminance channel and a channel image of a chrominance channel;

an image decomposition component 220, configured to decompose M frames of gray sub-images from the gray scale images, and M frames of luminance sub-images from the channel images of the luminance channels; wherein M is an integer greater than 1, the M-frame gray sub-images represent image information of the gray image in corresponding M frequency bands, and the M-frame luminance sub-images represent image information of a channel image of the luminance channel in the M frequency bands;

The image fusion component 230 is configured to fuse the M-frame gray sub-images and the M-frame luminance sub-images to obtain M-frame fusion sub-images; each frame of gray sub-image is used for fusing with one frame of brightness sub-image of the corresponding frequency band;

an image superimposing module 240, configured to superimpose the M-frame fusion sub-images to obtain a brightness fusion image;

the channel stitching component 250 is further configured to combine the luminance fusion image and the channel image of the chrominance channel to obtain a color fusion image.

In one implementation of the image fusion apparatus 200, the image fusion component 230 calculates M frame fusion masks corresponding to the M frequency bands according to the gray sub-images and the luminance sub-images; wherein, the pixel value in the fusion mask characterizes: fusion weights of pixel values at the same positions in the corresponding luminance sub-image and the corresponding gray sub-image; the M-frame fusion mask is utilized to carry out weighted fusion on the M-frame gray sub-image and the M-frame brightness sub-image, so as to obtain the M-frame fusion sub-image; each frame of fusion mask is used for fusing a corresponding frame of gray sub-image and a corresponding frame of brightness sub-image.

In one implementation of the image fusion apparatus 200, m=3, where the M frequency bands are a high frequency band, a medium frequency band, and a low frequency band, respectively.

In one implementation of the image fusion apparatus 200, the image fusion component 230 calculates a fusion mask for a mid-band based on a gray sub-image for the mid-band and a luminance sub-image for the mid-band; and calculating a fusion mask of the high frequency band according to the gray level sub-image of the high frequency band and the brightness sub-image of the high frequency band; and calculating the fusion mask of the low frequency band according to the fusion mask of the medium frequency band and the fusion mask of the high frequency band.

In one implementation manner of the image fusion device 200, the fusion mask is a binarized image, if the pixel value in the fusion mask takes a first value, the pixel value in the corresponding fusion sub-image at the same position takes the pixel value in the corresponding gray sub-image, and if the pixel value in the fusion mask takes a second value, the pixel value in the corresponding fusion sub-image at the same position takes the pixel value in the corresponding brightness sub-image; the image fusion component 230 is further configured to set the pixel value of the fusion mask of the low frequency band at the position to a first value if the pixel values of the fusion mask of the medium frequency band and the fusion mask of the high frequency band at the same position are both taken as the first value, otherwise set the pixel value of the fusion mask of the low frequency band at the position to a second value.

In one implementation of the image fusion apparatus 200, the image fusion component 230 takes an absolute value of a pixel value in the gray sub-image of the mid-band to obtain a first mask calculation image; taking absolute values of pixel values in the brightness sub-images of the middle frequency band to obtain a second mask calculated image; and calculating the fusion mask of the middle frequency band according to the magnitude relation between the pixel values of the first mask calculation image and the second mask calculation image at the same position.

In one implementation of the image fusion apparatus 200, the image fusion component 230 takes an absolute value of a pixel value in the gray sub-image of the mid-band to obtain a first mask calculation image; taking absolute values of pixel values in the brightness sub-images of the middle frequency band to obtain a second mask calculated image; filtering the first mask calculation image and the second mask calculation image by using a non-edge-protection smoothing filter to obtain a third mask calculation image and a fourth mask calculation image; and calculating the fusion mask of the middle frequency band according to the magnitude relation between the pixel values of the third mask calculation image and the fourth mask calculation image at the same position.

In one implementation manner of the image fusion device 200, the fusion mask is a binarized image, if the pixel value in the fusion mask takes a first value, the pixel value in the corresponding fusion sub-image at the same position takes the pixel value in the corresponding gray sub-image, and if the pixel value in the fusion mask takes a second value, the pixel value in the corresponding fusion sub-image at the same position takes the pixel value in the corresponding brightness sub-image; the image fusion component 230 is further configured to, for any one pixel position in the fused mask of the middle band, set the pixel value of the fused mask of the middle band at the position to a first value if the pixel value of the third mask calculated image at the position is greater than the product of the pixel value of the fourth mask calculated image at the position and the adjustment threshold, otherwise set the pixel value of the fused mask of the middle band at the position to a second value.

In one implementation of the image fusion apparatus 200, the image fusion apparatus 200 further includes: a brightness adjustment component for adjusting brightness of the gray-scale image to be consistent with the color image after the gray-scale image and the color image to be fused are acquired and before the M-frame gray-scale sub-image is decomposed from the gray-scale image; the image fusion component 230 calculates a fusion mask of the middle frequency band according to the gray sub-image of the low frequency band and the brightness sub-image of the low frequency band; and calculating a fusion mask of the middle frequency band according to the gray sub-image of the middle frequency band and the brightness sub-image of the middle frequency band; and calculating a fusion mask of the high frequency band according to the gray level sub-image of the high frequency band and the brightness sub-image of the high frequency band.

In one implementation of the image fusion apparatus 200, the image decomposition component 220 filters the gray scale image with a first low pass filter to obtain a gray scale sub-image in a low frequency band; the cut-off frequency of the first low-pass filter is a boundary between a low frequency band and a medium frequency band; filtering the gray level image by using a second low-pass filter to obtain a temporary gray level image, and calculating a gray level sub-image of a high frequency band according to the gray level image and the temporary gray level image; the cut-off frequency of the second low-pass filter is a boundary between a middle frequency band and a high frequency band; and calculating the gray sub-image of the middle frequency band according to the gray image, the gray sub-image of the low frequency band and the gray sub-image of the high frequency band.

In one implementation of the image fusion apparatus 200, the image decomposition component 220 calculates an occlusion region mask from the grayscale image and the channel image of the luminance channel; the pixel value in the mask of the shielding area characterizes the probability that the position of the pixel value belongs to the shielding area, and the shielding area refers to an area which only exists in the color image but does not exist in the gray image in the shooting scene; the gray level image and the channel image of the brightness channel are weighted and fused by using the shielding area mask to obtain a gray level shielding fusion image; and decomposing the M frames of gray sub-images from the gray shielding fusion image.

The image fusion apparatus 200 according to the embodiment of the present application has been described in the foregoing method embodiments, and for brevity, reference may be made to the corresponding contents of the method embodiments where the apparatus embodiment is not mentioned.

Fig. 6 shows a structure of an electronic device 300 provided in an embodiment of the present application. Referring to fig. 6, the electronic device 300 includes: processor 310, memory 320, and communication interface 330, which are interconnected and communicate with each other by a communication bus 340 and/or other forms of connection mechanisms (not shown).

The processor 310 includes one or more (only one shown), which may be an integrated circuit chip, with signal processing capabilities. The processor 310 may be a general-purpose processor, including a central processing unit (Central Processing Unit, CPU), a micro control unit (Micro Controller Unit, MCU), a network processor (Network Processor, NP), or other conventional processor; but may also be a special purpose processor including a graphics processor (Graphics Processing Unit, GPU), a Neural network processor (Neural-network Processing Unit, NPU for short), a digital signal processor (Digital Signal Processor, DSP for short), an application specific integrated circuit (Application Specific Integrated Circuits, ASIC for short), a field programmable gate array (Field Programmable Gate Array, FPGA for short) or other programmable logic device, discrete gate or transistor logic device, discrete hardware components. Also, when the processor 310 is plural, some of them may be general-purpose processors, and another may be special-purpose processors.

The Memory 320 includes one or more (Only one shown in the drawings), which may be, but is not limited to, a random access Memory (Random Access Memory, RAM), a Read Only Memory (ROM), a programmable Read Only Memory (Programmable Read-Only Memory, PROM), an erasable programmable Read Only Memory (Erasable Programmable Read-Only Memory, EPROM), an electrically erasable programmable Read Only Memory (Electric Erasable Programmable Read-Only Memory, EEPROM), and the like.

The processor 310, as well as other possible components, may access, read, and/or write data from, the memory 320. In particular, one or more computer program instructions may be stored in memory 320 that may be read and executed by processor 310 to implement the image fusion methods provided by embodiments of the present application.

The communication interface 330 includes one or more (only one shown) that may be used to communicate directly or indirectly with other devices for data interaction. Communication interface 330 may include an interface for wired and/or wireless communication.

It is to be understood that the configuration shown in fig. 6 is illustrative only, and that electronic device 300 may also include more or fewer components than shown in fig. 6, or have a different configuration than shown in fig. 6. For example, the electronic device 300 may further include a camera (a color camera and a black-and-white camera) for capturing an image or video, and the captured image or the frame in the video may be used as the image to be fused in step S110; for another example, if the electronic device 300 does not need to communicate with other devices, the communication interface 330 may not be provided.

The components shown in fig. 6 may be implemented in hardware, software, or a combination thereof. The electronic device 300 may be a physical device such as a cell phone, wearable device, video camera, PC, notebook, tablet, server, robot, etc., or may be a virtual device such as a virtual machine, container, etc. The electronic device 300 is not limited to a single device, and may be a combination of a plurality of devices or a cluster of a large number of devices.

The embodiment of the application also provides a computer readable storage medium, and the computer readable storage medium stores computer program instructions which, when read and executed by a processor, execute the image fusion method provided by the embodiment of the application. For example, a computer-readable storage medium may be implemented as memory 320 in electronic device 300 in FIG. 6.

The present embodiments also provide a computer program product comprising computer program instructions which, when read and executed by a processor, perform the image fusion method provided by the embodiments of the present application.

The foregoing is merely exemplary embodiments of the present application and is not intended to limit the scope of the present application, and various modifications and variations may be suggested to one skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. An image fusion method, comprising:

acquiring a gray level image and a color image to be fused; wherein the color image includes a channel image of a luminance channel and a channel image of a chrominance channel;

decomposing M frames of gray sub-images from the gray images, and decomposing M frames of brightness sub-images from the channel images of the brightness channels; wherein M is an integer greater than 1, the M-frame gray sub-images represent image information of the gray image in corresponding M frequency bands, and the M-frame luminance sub-images represent image information of a channel image of the luminance channel in the M frequency bands;

fusing the M-frame gray sub-images and the M-frame brightness sub-images to obtain M-frame fused sub-images; each frame of gray sub-image is used for fusing with one frame of brightness sub-image of the corresponding frequency band;

superposing the M frame fusion sub-images to obtain a brightness fusion image;

and combining the brightness fusion image and the channel image of the chromaticity channel to obtain a color fusion image.

2. The method of image fusion according to claim 1, wherein the fusing the M-frame gray sub-image and the M-frame luminance sub-image to obtain an M-frame fused sub-image includes:

Calculating M frame fusion masks corresponding to the M frequency bands according to the gray sub-images and the brightness sub-images; wherein, the pixel value in the fusion mask characterizes: fusion weights of pixel values at the same positions in the corresponding luminance sub-image and the corresponding gray sub-image;

the M-frame fusion mask is utilized to carry out weighted fusion on the M-frame gray sub-image and the M-frame brightness sub-image, so as to obtain the M-frame fusion sub-image; each frame of fusion mask is used for fusing a corresponding frame of gray sub-image and a corresponding frame of brightness sub-image.

3. The image fusion method according to claim 2, wherein m=3, and the M frequency bands are a high frequency band, a medium frequency band, and a low frequency band, respectively.

4. The image fusion method according to claim 3, wherein calculating M frame fusion masks corresponding to the M frequency bands from the gray sub-image and the luminance sub-image comprises:

calculating a fusion mask of the middle frequency band according to the gray sub-image of the middle frequency band and the brightness sub-image of the middle frequency band; and calculating a fusion mask of the high frequency band according to the gray level sub-image of the high frequency band and the brightness sub-image of the high frequency band;

And calculating the fusion mask of the low frequency band according to the fusion mask of the medium frequency band and the fusion mask of the high frequency band.

5. The method of claim 4, wherein the fusion mask is a binary image, if the pixel value in the fusion mask takes a first value, the pixel value in the corresponding fusion sub-image at the same position takes the pixel value in the corresponding gray sub-image, and if the pixel value in the fusion mask takes a second value, the pixel value in the corresponding fusion sub-image at the same position takes the pixel value in the corresponding brightness sub-image;

the calculating the fusion mask of the low frequency band according to the fusion mask of the medium frequency band and the fusion mask of the high frequency band comprises the following steps:

and if the pixel values of the fusion mask of the middle frequency band and the fusion mask of the high frequency band at the same position are both the first numerical value, setting the pixel value of the fusion mask of the low frequency band at the position as the first numerical value, otherwise, setting the pixel value of the fusion mask of the low frequency band at the position as the second numerical value.

6. The image fusion method according to claim 4 or 5, wherein the calculating the fusion mask of the mid-band according to the gray sub-image of the mid-band and the luminance sub-image of the mid-band comprises:

Taking absolute values of pixel values in the gray sub-images of the middle frequency band to obtain a first mask calculation image; taking absolute values of pixel values in the brightness sub-images of the middle frequency band to obtain a second mask calculated image;

and calculating the fusion mask of the middle frequency band according to the magnitude relation between the pixel values of the first mask calculation image and the second mask calculation image at the same position.

7. The image fusion method according to claim 4 or 5, wherein the calculating the fusion mask of the mid-band according to the gray sub-image of the mid-band and the luminance sub-image of the mid-band comprises:

filtering the first mask calculation image and the second mask calculation image by using a non-edge-protection smoothing filter to obtain a third mask calculation image and a fourth mask calculation image;

and calculating the fusion mask of the middle frequency band according to the magnitude relation between the pixel values of the third mask calculation image and the fourth mask calculation image at the same position.

8. The method of claim 7, wherein the fusion mask is a binary image, if the pixel value in the fusion mask takes a first value, the pixel value in the corresponding fusion sub-image at the same position takes the pixel value in the corresponding gray sub-image, and if the pixel value in the fusion mask takes a second value, the pixel value in the corresponding fusion sub-image at the same position takes the pixel value in the corresponding brightness sub-image;

the calculating the fused mask of the mid-band according to the magnitude relation between the pixel values of the third mask calculation image and the fourth mask calculation image at the same position comprises the following steps:

and aiming at any pixel position in the fused mask of the middle frequency band, if the pixel value of the third mask calculated image at the position is larger than the product of the pixel value of the fourth mask calculated image at the position and the adjustment threshold value, setting the pixel value of the fused mask of the middle frequency band at the position as a first numerical value, otherwise, setting the pixel value of the fused mask of the middle frequency band at the position as a second numerical value.

9. The image fusion method of claim 3, wherein after said acquiring the gray-scale image and the color image to be fused and before said decomposing the M-frame gray-scale sub-image from the gray-scale image, the method further comprises:

Adjusting the brightness of the gray scale image to be consistent with the color image;

the calculating the M-frame fusion mask corresponding to the M frequency bands according to the gray sub-image and the brightness sub-image includes:

calculating a fusion mask of the middle frequency band according to the gray level sub-image of the low frequency band and the brightness sub-image of the low frequency band; and calculating a fusion mask of the middle frequency band according to the gray sub-image of the middle frequency band and the brightness sub-image of the middle frequency band; and calculating a fusion mask of the high frequency band according to the gray level sub-image of the high frequency band and the brightness sub-image of the high frequency band.

10. The method of any one of claims 3-9, wherein said decomposing M-frame gray sub-images from said gray images comprises:

filtering the gray level image by using a first low-pass filter to obtain a gray level sub-image of a low frequency band; the cut-off frequency of the first low-pass filter is a boundary between a low frequency band and a medium frequency band;

filtering the gray level image by using a second low-pass filter to obtain a temporary gray level image, and calculating a gray level sub-image of a high frequency band according to the gray level image and the temporary gray level image; the cut-off frequency of the second low-pass filter is a boundary between a middle frequency band and a high frequency band;

And calculating the gray sub-image of the middle frequency band according to the gray image, the gray sub-image of the low frequency band and the gray sub-image of the high frequency band.

11. The method of any one of claims 1-10, wherein said decomposing M-frame gray sub-images from said gray scale images comprises:

calculating a shielding area mask according to the gray level image and the channel image of the brightness channel; the pixel value in the mask of the shielding area characterizes the probability that the position of the pixel value belongs to the shielding area, and the shielding area refers to an area which only exists in the color image but does not exist in the gray image in the shooting scene;

the gray level image and the channel image of the brightness channel are weighted and fused by using the shielding area mask to obtain a gray level shielding fusion image;

and decomposing the M frames of gray sub-images from the gray shielding fusion image.

12. A computer program product comprising computer program instructions which, when read and executed by a processor, perform the method of any of claims 1-11.

13. A computer readable storage medium, having stored thereon computer program instructions which, when read and executed by a processor, perform the method of any of claims 1-11.

14. An electronic device, comprising: a memory and a processor, the memory having stored therein computer program instructions which, when read and executed by the processor, perform the method of any of claims 1-11.