Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the embodiments of the present invention will be described in detail below with reference to the accompanying drawings. However, it will be appreciated by those of ordinary skill in the art that numerous technical details are set forth in order to provide a better understanding of the present application in various embodiments of the present invention. However, the technical solution claimed in the present application can be implemented without these technical details and various changes and modifications based on the following embodiments.
A first embodiment of the present invention relates to an image processing method including the steps of: calculating the scores of all image quality indexes of the images; and processing the image according to the scores of all the image quality indexes of the image and a predefined strategy space. In this embodiment, since the policy space stores a plurality of image processing policies in advance, the electronic device may select a processing policy suitable for the image from the predefined policy space based on the score of each image quality index of the image, and process the image, so that the image captured in different scenes can be optimized by selecting a suitable manner.
The following describes details of the image processing method according to the present embodiment. The following disclosure provides implementation details for the purpose of facilitating understanding, and is not necessary to practice the present solution.
The image processing method in the present embodiment is applied to an electronic device. The electronic device may be a camera, a terminal, a server, a cloud server, etc. with processing capability. The image may be an infrared image or other image. As shown in fig. 1, the image processing method specifically includes step 101 and step 102.
Step 101: and acquiring the scores of all image quality indexes of the images.
Specifically, the image quality index may be contrast, sharpness, brightness, signal-to-noise ratio, dynamic range, noise, texture, sharpness, exposure range, black-and-white balance, or the like, and the image quality index used in this embodiment may be any combination of the above indices.
It should be noted that, as will be understood by those skilled in the art, in practical applications, the score of each image quality index may be determined based on the calculated value of each image quality index of the image. For example, if the value of the image quality index is proportional to the image quality, the value of each image quality index of the image is set as the score of the corresponding image quality index, and if the value of the image quality index is inversely proportional to the image quality, the reciprocal of the value of each image quality index of the image is set as the score of the corresponding image quality index. For another example, the value of the image quality index is normalized, and the score of the image quality index is determined according to the value after the normalization. The present embodiment does not limit the manner of calculating the score of the image quality index.
Step 102: and if the scores of the image quality indexes of the image do not meet the preset score requirement, selecting a processing strategy from a predefined strategy space based on the scores of the image quality indexes of the image, and processing the image to obtain a final image.
Specifically, a plurality of image processing strategies are stored in the strategy space in advance. The predefined policy space refers to the totality of processing policies of the images that can be taken that are set in advance.
It should be noted that, as will be understood by those skilled in the art, the processing strategy in the strategy space may be set according to the selected image quality index. For example, the image quality indicator for the evaluation image includes a signal-to-noise ratio, and the policy space may include various processing policies for improving the signal-to-noise ratio. The present embodiment does not limit the number of processing strategies in the strategy space.
In one example, the electronic device determines whether the score of each image quality index of the image meets a preset score requirement; if yes, the image is not processed; if not, go to step 102.
It should be noted that, as will be understood by those skilled in the art, in practical applications, the score requirement may be set according to the quality requirement of the developer for the final image, for example, the preset score requirement includes: the sum of the scores of the image quality indexes of the image is greater than a threshold value; or, for each image quality index, the score of the image quality index is greater than the preset score corresponding to the image quality index. The threshold may be set according to the number of image quality indicators, and the like.
It is worth mentioning that the images meeting the preset requirements are not processed, so that the waste of computing resources caused by processing high-quality images can be avoided.
In one example, the policy space includes multiple processing policies, and the electronic device sequentially selects the processing policies in the policy space and processes the image respectively until all the processing policies in the policy space are traversed. The electronic device selects an image with the highest total score of the scores of the image quality indexes from the processed images as a final image.
In one example, selecting a processing strategy from a predefined strategy space based on the scores of the image quality indexes of the image, and processing the image to obtain a final image, includes: determining at least one policy combination based on the scores of the image quality indicators of the images; the strategy combination at least comprises at least one processing strategy in the strategy space; processing the images respectively by using each processing strategy in the strategy combination; calculating the sum of the scores of all image quality indexes of the processed image; and screening the processed image with the highest sum as a final image. In particular, each processing strategy may correspond to one or more image processing algorithms, e.g., an image denoising algorithm, an image sharpening algorithm, etc.
It is worth mentioning that the images are respectively processed by adopting a plurality of strategy combinations, and the processed image with the optimal processing result is selected as the final image, so that the quality of the final image is higher.
The following exemplifies a method of determining at least one policy combination based on the scores of the respective image quality indicators of the images.
The method comprises the following steps: judging whether the score of the image quality index of the image is larger than the preset score of the image quality index or not aiming at each image quality index; and if not, determining the image quality index as the image quality index to be optimized. And respectively selecting at least one processing strategy from the processing strategies corresponding to the image quality indexes to be optimized in the strategy space, and combining to obtain a strategy combination. Specifically, for each image quality index, one or more processing strategies for improving the image quality index score are stored in the strategy space. And when the score of a certain image quality index of the current image is not more than the preset score of the image quality index, taking the image quality index as the image quality index to be optimized. When the strategy combination is constructed, at least one processing strategy is respectively selected from the processing strategies corresponding to the image quality indexes to be optimized.
Optionally, the execution order of the processing strategies corresponding to the various image quality indexes is limited in the strategy space. For example, if a sharpening processing strategy and a denoising processing strategy exist in the strategy combination, the denoising processing strategy is executed first, and then the sharpening processing strategy is executed. The user presets the execution sequence of each processing strategy according to experience, so that the number of strategy combinations can be effectively reduced, and resource waste caused by executing unnecessary strategy combinations is avoided.
For example, there are 3 image quality indexes to be optimized a1, a2, and A3, and the execution sequence of the processing strategies corresponding to each image quality index to be optimized is: a1 → a2 → A3, 2 treatment strategies corresponding to a1, 3 treatment strategies corresponding to a2 and 2 treatment strategies corresponding to A3, at least 2 × 3 × 2 — 12 strategy combinations can be constructed. The electronic device uses the 12 strategy combinations to perform optimization processing on the image in turn.
The method 2 comprises the following steps: judging whether the score of the image quality index of the image is larger than the preset score of the image quality index or not aiming at each image quality index; and if not, determining the image quality index as the image quality index to be optimized. Selecting N candidate strategy combinations from M pre-stored candidate strategy combinations as strategy combinations to be executed according to the image quality index to be optimized; m is not less than N, and M and N are positive integers.
For example, there are 3 image quality indexes for setting in the electronic apparatus for evaluating image quality, B1, B2, and B3, respectively. Correspondingly, 6 types of candidate strategy combinations are stored in the strategy space, the first type of candidate strategy combination is used for improving the score of B1, the second type of candidate strategy combination is used for improving the scores of B1 and B2, the third type of candidate strategy combination is used for improving the scores of B1, B2 and B3, the fourth type of candidate strategy combination is used for improving the score of B2, the fifth type of candidate strategy combination is used for improving the scores of B2 and B3, and the sixth type of candidate strategy combination is used for improving the score of B3. If the electronic equipment determines that only the score of B1 is not greater than the preset score of B1, the first-class candidate strategy combination is used as a strategy combination to be executed; if the score of only B2 is determined not to be larger than the preset score of B2, the candidate strategy combination of the fourth class is taken as the strategy combination to be executed; if the score of only B3 is determined not to be larger than the preset score of B3, the candidate strategy combination of the sixth class is taken as the strategy combination to be executed; if the score of B1 is not larger than the preset score of B1 and the score of B2 is not larger than the preset score of B2, the candidate strategy combination of the second type is used as the strategy combination to be executed; if the score of B1 is determined to be not greater than the preset score of B1, the score of B2 is determined to be not greater than the preset score of B2, and the score of B3 is determined to be not greater than the preset score of B3, the candidate strategy combination of the third class is taken as the strategy combination to be executed; and if the score of B2 is determined to be not more than the preset score of B2 and the score of B3 is determined to be not more than the preset score of B3, the fifth type of candidate strategy combination is taken as the strategy combination to be executed.
It is worth mentioning that the strategy combination of the optimized image is obtained based on the score of the image quality index, so that the processing strategy for processing the image is more targeted, and the quality of the image is more effectively improved.
It should be noted that, as will be understood by those skilled in the art, in practical applications, other manners may be used to limit the processing strategy for processing the image, and the present embodiment is not limited thereto.
In one example, the processing of the image by using each processing policy in the policy combination comprises: determining the value of a preset parameter of a processing strategy corresponding to the image quality index to be optimized in the strategy combination according to the score of the image quality index to be optimized of the image; and executing each processing strategy in the strategy combination on the image in sequence. Specifically, the electronic device may store the score of the image quality index and the constraint relationship of each parameter in the processing policy corresponding to the image quality index. After the processing strategy to be executed is determined, the score of the image quality index corresponding to the processing strategy is brought into the constraint relation of each parameter in the processing strategy so as to obtain the value of each parameter of the processing strategy to be executed.
It is worth mentioning that, based on the score of the image quality index to be optimized of the image, the value of the preset parameter of the processing strategy corresponding to the image quality index to be optimized in the strategy combination is determined, instead of randomly selecting a numerical value to determine the processing strength of the processing strategy, and the optimal value is determined through multiple attempts, so that the optimization speed of the image can be improved.
It should be noted that, in practical applications, values of preset parameters of the processing policy may also be set according to a preset rule, for example, starting from a preset value, and gradually increasing or gradually decreasing according to a preset difference value until a maximum value or a minimum value is traversed. The present embodiment does not limit the setting manner of the value of the parameter.
The above description is only for illustrative purposes and does not limit the technical aspects of the present invention.
In the image processing method provided in this embodiment, since the policy space stores the processing policies of the plurality of images in advance, the electronic device may select a processing policy suitable for the image from the predefined policy space based on the score of each image quality index of the image, and process the image, so that the image shot in different scenes can be optimized by selecting a suitable mode.
A second embodiment of the present invention relates to an image processing method. The present embodiment is further detailed in the first embodiment, and a method of acquiring the score of each image quality index of an image will be described as an example.
Specifically, as shown in fig. 2, the present embodiment includes steps 201 to 203, where step 203 is substantially the same as step 102 in the first embodiment, and is not repeated here, and differences are mainly introduced below.
Step 201: an overexposed area of the image is determined.
Specifically, the electronic device may determine the over-exposed regions of the image by calculating the brightness of the regions of the image.
Step 202: and calculating the scores of all image quality indexes of the image according to other areas except the over-exposure area in the image.
Specifically, in the process of calculating each image quality index of an image, the score calculation of each image quality index is performed excluding an overexposed area in the image.
It is worth mentioning that if a certain part of the image is in an overexposed state, most information of the area will be lost, and the calculation of the signal-to-noise ratio, the dynamic range and the like will lose great effect. And removing the scores of all the image quality indexes of the calculated image of the over-exposure area, so that the scores of all the image quality indexes of the calculated image have reference value, and further the processing efficiency of the image is improved.
Step 203: and if the scores of the image quality indexes of the image do not meet the preset score requirement, selecting a processing strategy from a predefined strategy space based on the scores of the image quality indexes of the image, and processing the image to obtain a final image.
The following takes an image as an infrared image, and image quality indexes including a signal-to-noise ratio and a dynamic range as an example, and the image processing method mentioned in this embodiment is exemplified.
At present, infrared imaging technologies are mature and mainly divided into active infrared night vision technologies and passive infrared night vision technologies. Among them, active infrared night vision technology is the most common. The active infrared night vision technology is a night vision technology that performs observation by actively irradiating and reflecting infrared light of an infrared source with a target, and is often used in a surveillance-type camera. The active infrared night vision technology has the advantages that the night vision imaging is carried out by emitting infrared rays by the active infrared night vision technology without the help of external environment light, the night vision range is wide, and the influence of the environment is small. However, due to the inherent characteristics of the infrared sensor, the infrared image generally has the problems of poor target-to-background contrast, edge blurring and the like, and the image signal-to-noise ratio and the contrast may be low, which need to be solved by an image processing technology. The image processing methods are also different for infrared images of different scenes. For example, in an indoor environment, a long-distance image is blurred and noisy due to problems such as ambient light, and a spatial filtering method is mainly used. In outdoor environment, strong sunlight may make the image brightness and contrast lower, and more methods of tone mapping may be used. Common filtering methods include Gaussian Filter (GF), Bilateral Filter (BF), guided filter (guided filter), median filter (median filter), and mean filter (mean filter). The slightly more complicated filtering methods include local mean filtering, intelligent fuzzy filtering (smart blur), surface fuzzy filtering (surface blur), mean shift filtering (mean shift filter), Bi-Exponential Edge-Preserving smoothing (seeps) filtering, etc. These filtering methods are often used to remove noise from the infrared map. Some filtering methods also have the capability of removing noise while maintaining edge characteristics. Tone mapping (tone mapping) methods are also commonly used in image post-processing, which primarily map image colors to adjust Dynamic Range (DR) and contrast. Common global tone mapping methods are histogram equalization, Gamma (Gamma) mapping, log correction, and piecewise gray scale transformation. The local tone mapping method includes a block median histogram, adaptive histogram equalization, and the like. In addition, a sharpening method is also commonly used in image post-processing, and the sharpening is to highlight high-frequency information, such as edges and contours, on an image so as to make the image clearer. The sharpening operation is the reverse operation of image smoothing, and commonly used methods include a differential method (such as a robert gradient operator method and a laplacian operator method), a high-pass filtering method (such as an Unsharp Masking (USM) method in PS), and a template matching method. In addition, the defogging algorithm is a hot spot studied in current image processing, and increases the visibility of an image by eliminating the fogging in the image. The more common and effective defogging algorithm is the dark channel defogging method of nakeming. Although there are many image processing methods for a single scene, there is no mature image processing method that can be uniformly applied for infrared images of multiple scenes. In addition, although there is a method for adjusting parameters in an image adaptive manner in the unilateral technique, the degree of adaptation is not high enough. Without clear and definite indexes, the image quality cannot be improved. Therefore, there is a need for an adaptive image processing method capable of guiding multiple scenes according to image indexes. Aiming at infrared images in various scenes, the image problem is difficult to uniformly solve by the existing image post-processing technology. In order to improve the quality and the objective feeling of an infrared image and enhance the self-adaption degree of processing a multi-scene infrared image, so that the infrared image can be automatically processed offline, the embodiment provides an image processing method suitable for multiple scenes indoors and outdoors. As shown in fig. 3, one implementation of the image processing method in this embodiment includes steps 301 to 311 described below.
Step 301: and acquiring an infrared image I, and carrying out overexposure prior processing on the infrared image to obtain an infrared image J.
Specifically, brightness is a relatively important concept in an image, and is expressed in various aspects of the image. In this embodiment, the brightness is used to determine the exposure of the image as prior information for improving the signal-to-noise ratio and the dynamic range. If a certain area of the infrared image is in an overexposed state, most information of the area is lost. If this region is not removed, the processing signal-to-noise ratio and dynamic range will be largely lost. Therefore, a flag (mask) may be created to ignore this region information. If one pixel (I, j) in the infrared image I belongs to an overexposed area, mask (I, j) is 0, and if the pixel (I, j) belongs to a normal exposure area, mask (I, j) is 1. Where i is the abscissa of the pixel and j is the ordinate of the pixel. The infrared image J is the image of the infrared image I with the mask. The electronic equipment distinguishes whether each pixel belongs to the over-exposure area through the mask.
Step 302: the score of the infrared image J is calculated.
Specifically, the image quality index includes a signal-to-noise ratio and a dynamic range. And calculating the values of the signal-to-noise ratio and the dynamic range of other areas except the over-exposure area in the infrared image J, and then scoring to obtain the image quality of the image. For example. And calculating the fraction of the infrared image J by using the formula a.
Formula a: score ═ μ SNR + (1- μ) × DR;
wherein score represents the fraction of infrared image J; mu represents the weight coefficient of the signal-to-noise ratio for weighing the proportion of the signal-to-noise ratio and the dynamic range to the total mass, and can be set by a user according to the requirement; SNR represents the signal-to-noise ratio; DR denotes dynamic range.
After the score of the infrared image J is calculated, a subsequent operation is performed to try different strategy combinations of SIGNAL-to-NOISE RATIO (SNR) improvement and Dynamic Range (DR) to find the strategy combination with the highest score. Namely:
where D denotes a policy combination corresponding to a final image, D1 denotes a first processing policy, D2 denotes a second processing policy, D1 denotes a policy space for increasing an SNR value, D2 denotes a policy space for increasing a DR value, and D1 and D2 constitute a total policy space. D1E D1 represents that D1 is the processing strategy selected from D1, D2E D2 represents that D2 is the processing strategy selected from D2, max represents a maximum function, and score represents the fraction of the infrared image J processed by the processing strategies D1 and D2.
It should be noted that the formula b is only an example, and in practical applications, a plurality of processing strategies may also be selected from one strategy space, and the present embodiment is not limited.
The following describes the snr and its processing strategy, respectively. The signal-to-noise ratio of an image is the ratio of the mean of the gray values to the standard deviation of the background. I.e. the signal-to-noise ratio can be calculated by formula c.
Formula c: SNR is musig/σsig;
Where SNR represents the signal-to-noise ratio, μ, of an imagesigRepresenting mean of grey values, σsigThe background standard deviation is indicated.
The higher the signal-to-noise ratio, the more information the image represents and the less noise. In order to improve the signal-to-noise ratio, it is necessary to reduce the unwanted noise and enhance the information of the image, mainly the edge features and contour texture of the image. In order to reduce noise and maintain the edge characteristics of the image, a denoising processing strategy, such as an edge-preserving filtering processing strategy, such as bilateral filtering, may be set in the strategy space. In order to enhance the edge characteristics of the image and compensate for the high frequency loss caused by filtering, the strategy space may set a sharpening processing strategy, such as Unsharp Masking (USM) sharpening.
Optionally, the processing order of each processing strategy is limited in the strategy space, for example, a denoising processing strategy is executed first, and then a sharpening processing strategy is executed.
The dynamic range and its handling strategy are exemplified below. The dynamic range refers to the range of light intensity from the brightest area to the darkest area, independent of noise. High-Dynamic Range (HDR) images can provide more Dynamic Range and image details and provide more sufficient image contrast than ordinary images. To improve the dynamic range, a defogging processing strategy and a tone mapping processing strategy can be used.
The foggy image dark channel has a large area of grayness, and the local uniform fog can be removed by using a local darkest point by using a dark channel defogging method.
There are many tone mapping processing strategies, and Gamma (Gamma) correction and block median histogram are the ones that perform well in this system. The principle of Gamma correction is simple, as expressed by formula d:
formula d: j. the design is a squareout=Jin γ;
Wherein, JoutRepresenting the intensity value of the normalized output image, JinRepresenting the luminance values of the normalized input image. Gamma ray>1, enhancing the contrast of a high gray level area of the image; gamma ray<1, the contrast of the low-gray area of the image is enhanced. The method of Gamma correction can be adapted locally and also to segmentation.
The block median histogram is changed from a histogram equalization method, the input image is firstly subjected to average blocking and is cut into a plurality of sub-images, then histogram statistics is respectively carried out on each sub-image, and the contrast of the image can be obviously enhanced.
Step 303: and judging whether the score of the infrared image J meets the score requirement. If yes, go to step 304, and if no, go to step 305.
Step 304: the infrared image J is taken as the final infrared image K. Step 311 is then performed.
Step 305: the score of the infrared image J is taken as the score to be compared.
Step 306: and selecting a strategy combination from the strategy space, and processing the infrared image J according to the strategy combination.
For example, if the selected policy combination includes a certain defogging processing policy, a certain denoising processing policy, a certain sharpening processing policy, and a certain tone mapping policy, the electronic device applies the certain defogging processing policy in the policy space to the infrared image J to obtain the infrared image J _ 1; carrying out a certain denoising processing strategy in a strategy space on the infrared image J _1 to obtain an infrared image J _ 2; carrying out a certain sharpening processing strategy in a strategy space on the infrared image J _2 to obtain an infrared image J _ 3; and (3) carrying out a certain tone mapping strategy in the strategy space on the infrared image J _3 to obtain an infrared image J _4, namely an infrared image L.
Step 307: the score of the infrared image L is calculated.
Step 308: and judging whether the score of the infrared image L is higher than the score to be compared. If yes, go to step 309, and if not, go to step 310.
Step 309: and taking the infrared image L as a final infrared image K, and taking the score of the infrared image L as a score to be compared. Step 310 is then performed.
Step 310: and judging whether all the strategy combinations in the strategy space are traversed. If yes, go to step 311, and if not, go to step 306.
Step 311: and outputting the final infrared image K.
The above description is only for illustrative purposes and does not limit the technical aspects of the present invention.
In the image processing method provided in this embodiment, since the policy space stores the processing policies of the plurality of images in advance, the electronic device may select a processing policy suitable for the image from the predefined policy space based on the score of each image quality index of the image, and process the image, so that the image shot in different scenes can be optimized by selecting a suitable mode. In addition, the scores of all the image quality indexes of the image calculated in the over-exposure area are removed, so that the calculated scores of all the image quality indexes of the image have reference value, and the processing efficiency of the image is improved.
The steps of the above methods are divided for clarity, and the implementation may be combined into one step or split some steps, and the steps are divided into multiple steps, so long as the same logical relationship is included, which are all within the protection scope of the present patent; it is within the scope of the patent to add insignificant modifications to the algorithms or processes or to introduce insignificant design changes to the core design without changing the algorithms or processes.
A third embodiment of the present invention relates to an electronic apparatus, as shown in fig. 4, including: at least one processor 401; and a memory 402 communicatively coupled to the at least one processor 401; the memory 402 stores instructions executable by the at least one processor 401, and the instructions are executed by the at least one processor 401, so that the at least one processor 401 can execute the image processing method according to the above embodiment.
The electronic device includes: one or more processors 401 and a memory 402, one processor 401 being exemplified in fig. 4. The processor 401 and the memory 402 may be connected by a bus or other means, and fig. 4 illustrates the connection by a bus as an example. The memory 402 is a non-volatile computer-readable storage medium for storing non-volatile software programs, non-volatile computer-executable programs, and modules, such as algorithms corresponding to the processing strategies in the strategy space in the embodiment of the present application, stored in the memory 402. The processor 401 executes various functional applications of the apparatus and data processing, i.e., implements the above-described image processing method, by executing nonvolatile software programs, instructions, and modules stored in the memory 402.
The memory 402 may include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store a list of options, etc. Further, the memory 402 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid state storage device. In some embodiments, memory 402 may optionally include memory located remotely from processor 401, which may be connected to an external device via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
One or more modules are stored in the memory 402 and, when executed by the one or more processors 401, perform the image processing method of any of the method embodiments described above.
The product can execute the method provided by the embodiment of the application, has corresponding functional modules and beneficial effects of the execution method, and can refer to the method provided by the embodiment of the application without detailed technical details in the embodiment.
A fourth embodiment of the present invention relates to a computer-readable storage medium storing a computer program. The computer program realizes the above-described method embodiments when executed by a processor.
That is, as can be understood by those skilled in the art, all or part of the steps in the method of the above embodiments may be implemented by a program instructing related hardware, where the program is stored in a storage medium and includes several instructions to enable a device (which may be a single chip, a chip, or the like) or a processor (processor) to execute all or part of the steps in the method of the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
It will be understood by those of ordinary skill in the art that the foregoing embodiments are specific embodiments for practicing the invention, and that various changes in form and details may be made therein without departing from the spirit and scope of the invention in practice.