CN116471489A - Image preprocessing method and device - Google Patents

Abstract

The application discloses an image preprocessing method and device, and belongs to the field of computers. Comprising the following steps: acquiring a target RAW domain image; extracting a RAW domain block from the target RAW domain image based on a coordinate mapping relationship, wherein the coordinate mapping relationship is determined based on a coordinate transformation relationship for preprocessing the target RAW domain image and an image blocking rule after preprocessing; preprocessing the RAW domain image block to obtain a target image block.

Description

Image preprocessing method and device

Technical Field

The application belongs to the field of computers, and particularly relates to an image preprocessing method and device.

Background

In the shooting process, a user can select a night scene mode to shoot, so that the image quality of shooting a night scene is improved.

In the night view mode, a series of complex image preprocessing is generally required to be performed on the exposed multi-frame original RAW domain image until the multi-frame image obtained by preprocessing can meet the input data format requirement of the neural network. Because the RAW is in an uncompressed lossless format, the RAW domain image is directly preprocessed, a very large memory space is required to be consumed, and redundant memory access operation exists, so that the operation performance is adversely affected, the power consumption value is increased, and the preprocessing efficiency is lowered.

Disclosure of Invention

The embodiment of the application aims to provide an image preprocessing method and device, which can solve the problems of high power consumption and low efficiency of RAW domain image preprocessing.

In a first aspect, an embodiment of the present application provides an image preprocessing method, including:

acquiring a target RAW domain image;

extracting a RAW domain block from the target RAW domain image based on a coordinate mapping relationship, wherein the coordinate mapping relationship is determined based on a coordinate transformation relationship for preprocessing the target RAW domain image and an image blocking rule after preprocessing;

preprocessing the RAW domain image block to obtain a target image block.

In a second aspect, an embodiment of the present application provides an image preprocessing apparatus, including:

an acquisition unit configured to acquire a target RAW domain image;

the extraction unit is used for extracting a RAW domain block from the target RAW domain image based on a coordinate mapping relation, wherein the coordinate mapping relation is determined based on a coordinate transformation relation for preprocessing the target RAW domain image and an image blocking rule after preprocessing;

the preprocessing unit is used for preprocessing the RAW domain image block to obtain a target image block.

In a third aspect, embodiments of the present application provide an electronic device comprising a processor and a memory storing a program or instructions executable on the processor, which when executed by the processor, implement the steps of the method as described in the first aspect.

In a fourth aspect, embodiments of the present application provide a readable storage medium having stored thereon a program or instructions which when executed by a processor implement the steps of the method according to the first aspect.

In a fifth aspect, embodiments of the present application provide a chip, where the chip includes a processor and a communication interface, where the communication interface is coupled to the processor, and where the processor is configured to execute a program or instructions to implement a method according to the first aspect.

In a sixth aspect, embodiments of the present application provide a computer program product stored in a storage medium, the program product being executable by at least one processor to implement the method according to the first aspect.

In the embodiment of the application, the RAW domain image block corresponding to the target image block is extracted from the target RAW domain image based on the coordinate mapping relation, and the RAW domain image block is preprocessed, so that compared with preprocessing of the whole image size, the memory space required by preprocessing is greatly reduced, a large amount of redundant memory application and operation in the preprocessing process are reduced, the preprocessing efficiency is improved, and the overall performance of an image processing algorithm is improved.

Drawings

Fig. 1 is one of flow diagrams of an image preprocessing method provided in an embodiment of the present application;

fig. 2 is a flowchart illustrating a method for extracting a RAW domain tile according to an embodiment of the present disclosure;

fig. 3 is a schematic diagram of color type arrangement in a target RAW domain image according to an embodiment of the present application;

fig. 4 is a schematic diagram of storing pixel values in a target RAW domain image according to an embodiment of the present application;

FIG. 5 is a schematic flow chart of an image processing method according to an embodiment of the present disclosure;

FIG. 6 is a second flowchart of an image processing method according to an embodiment of the present disclosure;

fig. 7 is a schematic structural diagram of an image preprocessing device according to an embodiment of the present application;

fig. 8 is a schematic structural diagram of an electronic device according to an embodiment of the present application;

fig. 9 is a schematic hardware structure of an electronic device according to an embodiment of the present application.

Detailed Description

Technical solutions in the embodiments of the present application will be clearly described below with reference to the drawings in the embodiments of the present application, and it is apparent that the described embodiments are some embodiments of the present application, but not all embodiments. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments in the present application are within the scope of the protection of the present application.

The terms first, second and the like in the description and in the claims, are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the terms so used are interchangeable under appropriate circumstances such that the embodiments of the application are capable of operation in sequences other than those illustrated or otherwise described herein, and that the objects identified by "first," "second," etc. are generally of a type and do not limit the number of objects, for example, the first object may be one or more. Furthermore, in the description and claims, "and/or" means at least one of the connected objects, and the character "/", generally means that the associated object is an "or" relationship.

The preprocessing method provided by the embodiment of the application is described in detail below through specific embodiments and application scenes thereof with reference to the accompanying drawings.

The related art generally requires preprocessing of an original RAW domain image before image processing based on a neural network. In order to meet the power consumption and memory requirements, it is necessary to block the preprocessed image in whole image, and use the block obtained by the whole image as the input data of the neural network for image processing. In general, a tile input as a neural network may be denoted as a target tile.

In the preprocessing process, a series of complex transformations are required for the original RAW domain image, for example, a plurality of preprocessing links such as BPC (bad point correction, dead point correction), unpack (decompression of the RAW domain image), LSC (Lens Shading Correction ), warp (affine transformation of image), VST (variance-stabilizing transformation, variance smooth transformation) and the like may be involved.

In view of the above problems, embodiments of the present application provide an image preprocessing method. Fig. 1 is one of flow diagrams of an image preprocessing method according to an embodiment of the present application, as shown in fig. 1, where the method includes:

step 110, acquiring a target RAW domain image.

Specifically, the RAW domain image is a digitized RAW image data in RAW format, and the RAW format is an unprocessed, uncompressed image format, and in this embodiment, the target RAW domain image is an image that needs to be preprocessed to perform subsequent image processing. In addition, in order to adapt to the input requirement of the neural network applied by the subsequent image processing, the target RAW domain image needs to be preprocessed and segmented, and then can be input into the neural network in the form of a target block.

Here, the target RAW domain image may be a single frame image, or may be a multi-frame image obtained by continuous acquisition, or may be a multi-frame image obtained by separate acquisition, which is not particularly limited in the embodiment of the present application. It is understood that the step 110 and the subsequent steps 120 and 130 can be used for single-frame image preprocessing and multi-frame image preprocessing.

Step 120, extracting a RAW domain tile from the target RAW domain image based on a coordinate mapping relationship, wherein the coordinate mapping relationship is determined based on a coordinate transformation relationship for preprocessing the target RAW domain image and an image blocking rule after preprocessing.

In order to avoid the memory and power consumption effects caused by preprocessing a target RAW domain image with the original size, the method and the device divide the target RAW domain image into blocks before preprocessing the RAW image, so that the target RAW domain image can be preprocessed in the form of RAW domain blocks with smaller size, the memory space required by preprocessing is reduced, and the computing efficiency of preprocessing is improved.

In consideration of the conventional scheme, the target RAW domain image with the original image size needs to be preprocessed and can obtain a target image block after preprocessing by an image block dividing method, the embodiment of the application can find out a link needing to perform coordinate transformation from preprocessed links, further determine the coordinate transformation relation in the preprocessing process, and further determine the coordinate transformation relation from the image to the target image block based on the preprocessed image block dividing rule, so that the coordinate mapping relation from the target RAW domain image to the target image block is obtained. The image blocking rule is to block the preprocessed image in whole image, so that the target block obtained by blocking can meet the rule of the input requirement of the neural network for image processing. The image blocking rules may depend on the image size of the target RAW domain image, as well as the input requirements of the neural network.

It will be appreciated that the coordinate mapping relationship herein reflects the mapping relationship between the coordinates in the target RAW domain image and the coordinates in the target tile. In addition, on the premise that the preprocessing link is unchanged and the image blocking rule is unchanged, the coordinate mapping relationship is also fixed, that is, the coordinate mapping relationship can be calculated and stored in advance so as to be conveniently and directly called when the step 120 is executed. For example, the manner of extracting the RAW domain image blocks from the pre-calculated coordinate mapping relationship can be hardened into the chip IP (Intellectual Property), so that the instantaneity of the preprocessing algorithm is further improved, and the application in the fields of high-definition real-time preview, high-quality rapid imaging and the like is facilitated.

And (3) reversely pushing based on the coordinate mapping relation, so that which coordinate in the target RAW domain image corresponds to each coordinate in the target block can be determined. Therefore, through the inverse coordinate mapping relation, which coordinates in the target image block correspond to which coordinates in the target RAW domain image after the inverse coordinate mapping, namely which coordinates in the target RAW domain image belong to the same target block after the coordinate mapping, and based on the inverse coordinate mapping relation, the image segmentation is performed on the target RAW domain image, so that the RAW domain image block can be obtained.

The RAW domain tiles in the single-frame target RAW domain image obtained herein are in one-to-one correspondence with the target tiles, and the RAW domain tiles can be understood as data in the target RAW domain image corresponding to the target tiles, and the RAW domain tiles are consistent with the size of the target tiles. It will be appreciated that, since the target tile is typically obtained by image blocking the preprocessed target RAW domain image, the size of the target tile must be smaller than the size of the target RAW domain image, and the size of the RAW domain tile must also be smaller than the size of the target RAW domain image.

And step 130, preprocessing the RAW domain block to obtain a target block.

Specifically, after obtaining the RAW domain tiles of smaller size, the RAW domain tiles may be preprocessed to obtain the target tiles. Compared with directly processing the target RAW domain image with the original size, the method has the advantages that memory resources occupied by preprocessing the RAW domain image block are smaller, and the preprocessing efficiency is higher. In addition, the preprocessing process for the RAW domain blocks with smaller sizes can be parallel, namely, the preprocessing can be performed on a plurality of RAW domain blocks in parallel, so that the preprocessing efficiency is further improved.

In addition, since the RAW domain tiles corresponding to the target tiles are extracted from the target RAW domain image before preprocessing, the target tiles are obtained after preprocessing the RAW domain tiles, and image blocking is not required after preprocessing.

In the embodiment of the application, the RAW domain image block corresponding to the target image block is extracted from the target RAW domain image based on the coordinate mapping relation, and the RAW domain image block is preprocessed, so that compared with preprocessing of the whole image size, the memory space required by preprocessing is greatly reduced, a large amount of redundant memory application and operation in the preprocessing process are reduced, the preprocessing efficiency is improved, and the overall performance of an image processing algorithm is improved.

In an alternative embodiment, fig. 2 is a flowchart of a method for extracting a RAW domain tile according to an embodiment of the present application, as shown in fig. 2, step 120 includes:

step 121, determining pixel coordinates of each pixel of the RAW domain tile in the target RAW domain image based on the coordinate mapping relationship.

Specifically, the coordinates in one target tile after the inverse coordinate mapping can be determined by performing the inverse extrapolation based on the coordinate mapping relationship, which coordinates in the target RAW domain image correspond to the coordinates of each pixel in the RAW domain tile corresponding to the target tile, specifically, which coordinates of which pixels in the target RAW domain image correspond to the coordinates of each pixel in the RAW domain tile corresponding to the target tile. Here, the pixel coordinates of each pixel of the RAW domain tile in the target RAW domain image, i.e., the pixel coordinates that need to be extracted from the target RAW domain image to construct the RAW domain tile.

For example, the pixel coordinates on a target tile are (x _src1 ,y _src1 ) The pixel coordinates of the pixel coordinates in the target RAW domain image can be obtained by inverse extrapolation based on the coordinate mapping relation as (x) _src0 ,y _src0 ) That is, the pixel coordinate of one pixel in the RAW domain image of the target tile corresponding to the target tile is (x) _src0 ,y _src0 )。

Step 122, determining a color type and a pixel value of each pixel in the RAW domain tile based on the pixel coordinates of each pixel of the RAW domain tile in the target RAW domain image, so as to construct the RAW domain tile.

Specifically, after obtaining the pixel coordinates of each pixel of the RAW domain tile in the target RAW domain image, the color type of each pixel may be determined based on the pixel coordinates in the target RAW domain image. For example, in a target RAW domain image arranged in GBRG, the color types of the respective pixels may include G (Green), B (Blue), R (Red). The specific color type of each pixel may be determined based on the color type corresponding to the pixel coordinates of each pixel in the target RAW domain image.

In addition, the pixel value of each pixel may be determined based on the pixel coordinates in the target RAW domain image. The pixel values are stored in different manners under different RAW formats. For example, the RAW format may be RAW8, RAW10, RAW12, or the like, wherein in the RAW10 format, each pixel value is stored with 10 bits, i.e., 5 bytes of storage are required for pixel values of every 4 pixels. Moreover, the storage rules of pixels under different RAW formats are different. The pixel coordinate is used to derive the storage position of the pixel value in combination with the pixel storage rule of the RAW format corresponding to the target RAW domain image, and then the pixel value corresponding to the pixel coordinate is read from the target RAW domain image.

It will be appreciated that the construction of the RAW domain tile may be achieved after the color type and pixel values of each pixel in the RAW domain tile are obtained.

In the embodiment of the application, the RAW domain image block corresponding to the target image block is extracted from the target RAW domain image based on the coordinate mapping relation, so that conditions are provided for preprocessing the RAW domain image block, redundant memory application and operation in the preprocessing process are reduced, and the preprocessing efficiency is improved.

In an alternative embodiment, fig. 3 is a schematic diagram of color type arrangement in a target RAW domain image provided in the embodiment of the present application, as shown in fig. 3, in step 122, determining, based on pixel coordinates of each pixel of the RAW domain tile in the target RAW domain image, a color type of each pixel in the target RAW domain image includes:

and determining the color type corresponding to each pixel based on the parity of the pixel coordinates of each pixel.

Specifically, fig. 3 shows color types corresponding to each pixel coordinate in the target RAW domain image arranged in GBRG mode. As shown in fig. 3, four pixel coordinates of two rows and two columns form a set of color spaces, wherein odd rows and odd columns represent green, odd rows and even columns represent blue, even rows and odd columns represent red, and even rows and even columns represent green. Based on this rule, after obtaining the pixel coordinates of each pixel, the color type corresponding to each pixel can be determined based on the parity of the pixel coordinates. It will be appreciated that parity herein is used to characterize coordinates as either odd or even.

It is understood that the pixel coordinates of a two-dimensional image may include an abscissa and an ordinate. In the pixel coordinates, the abscissa represents the column number, the ordinate represents the line number, the ordinate is the odd number, the abscissa is the odd number corresponding to the odd number line and the odd number column, and the ordinate represents green; the ordinate is odd, the abscissa is even and corresponds to odd lines and even columns, and blue is represented; the ordinate is even, the abscissa is odd corresponding to even row and odd column, and red is represented; the ordinate is even, and the abscissa is even for even rows and columns, representing green.

For example, for a target RAW domain image in the GBRG arrangement, mipi (Mobile Industry Processor Interface ) format of pack10, the pixel coordinates (x _src0 ,y _src0 ) The corresponding color type can be confirmed by a boolean type variable:

bool colorR＝(y _src0 ％2！＝0)&&(x _src0 ％2＝＝0)；

bool colorG＝((y _src0 ％2＝＝0)&&(x _src0 ％2＝＝0))||((y _src0 ％2！＝0)&&

(x _src0 ％2！＝0))；

bool colorB＝(y _src0 ％2＝＝0)&&(x _src0 ％2！＝0)；

here, since the pixel coordinates (x _src0 ,y _src0 ) The color type at that point can only be one of R, G, B, so from the above formula it can be determined if the boolean variable is true to determine the color type to which the pixel coordinates correspond. For example, bolcolor=1, i.e., the boolean variable of red R is true, i.e., the color type corresponding to the pixel coordinate is red.

In the embodiment of the application, the color type corresponding to each pixel is determined based on the parity of the pixel coordinates of each pixel, so that a RAW domain block is constructed, conditions are provided for preprocessing based on the RAW domain block, redundant memory application and operation in the preprocessing process are reduced, and the preprocessing efficiency is improved.

In an alternative embodiment, in step 122, determining the pixel value of each pixel in the RAW domain tile based on the pixel coordinates of each pixel in the RAW domain tile in the target RAW domain image includes:

and determining a storage position of each pixel in the target RAW domain image based on the pixel coordinates of each pixel, and extracting the pixel value of each pixel from the storage position.

Specifically, in the target RAW domain image, the extraction of the pixel values needs to be implemented in combination with pixel coordinates and pixel storage rules. The pixel storage rule herein is used to indicate the storage location of the pixel value corresponding to each pixel coordinate in the target RAW domain image. Thus, the pixel value extraction can be realized based on the pixel storage rule and by combining the pixel coordinates, the storage position of the pixel value at the pixel coordinates in the target RAW domain image is determined.

For example, fig. 4 is a schematic diagram of pixel value storage in a target RAW domain image provided in an embodiment of the present application, where, as shown in fig. 4, in the target RAW domain image in a RAW10 format, each pixel value is stored with 10 bits, 5 bytes are applied to each pixel value of 4 pixels, where 8 bits higher than each 4 pixel values (1 st pixel, 2nd pixel, 3rd pixel, 4th pixel) are stored in the first 4 bytes (1 st byte, 2nd byte, 3rd byte, 4th byte) of 5 bytes, and the remaining 2 bits of each 4 pixel values are stored in the 5th byte (5 th byte) of 5 bytes. Thus, the pixel value can be extracted by locating the storage space of the pixel value corresponding to the pixel coordinate, for example, which bits of what byte to store in, based on the pixel coordinate. Let every fourth pixel be a set of units with respect to fig. 4, then for the nth pixel in each row of the target RAW domain image, n-to-4 is left as n%4, indicating which pixel is in the set of four pixel units; n is rounded to 4 and is n/4, indicating which four-pixel set is in a row. It is clear that the upper 8 bits of the nth pixel of a certain row are at the (n% 4+5×n/4) th byte position, and the lower 2 bits are at the ((5*n/4) +4) byte position.

In the embodiment of the application, the pixel values of the pixels are extracted based on the pixel coordinates of the pixels to construct the RAW domain block, so that conditions are provided for preprocessing based on the RAW domain block, redundant memory application and operation in the preprocessing process are reduced, and the preprocessing efficiency is improved.

In an optional embodiment, in step 122, the determining a storage location of the pixels in the target RAW domain image based on the pixel coordinates of the pixels, and extracting the pixel values of the pixels from the storage location includes:

determining a column index position and a storage offset in the storage position based on the abscissa in the pixel coordinates;

a row index location in the storage location is determined based on the ordinate in the pixel coordinates.

Specifically, the abscissa in the pixel coordinates represents the column number of the pixel in the target RAW domain image. In the case where it is known that the pixel values of the target RAW domain image are stored in units of cell sets, the number of pixel values included in a group of cell sets, for example, the number of pixel values included in a group of cell sets in fig. 4, may be 4. Rounding the number of pixel values contained in the abscissa pair unit set to obtain the unit set of which pixel value is stored in one row, and taking the remainder of the number of pixel values contained in the abscissa pair unit set to obtain the pixel value of which pixel value is in the unit set, and combining the two pixel values to determine the column index position in the storage position of the pixel; in addition, the offset amount stored in the lower order of the pixel values, that is, the storage offset amount, can be determined based on the number of pixel values in the unit set.

The ordinate in the pixel coordinates represents the number of rows in the target RAW domain image where the pixel is located. The row index position of the pixel value can be determined based on the correspondence of the number of rows in the storage position.

After defining the row and column index locations of the pixel values, and storing the offset, the storage locations may be located based on the three and the pixel values may be extracted from the storage locations.

For example, in the storage space shown in fig. 4, for the pixel coordinates (x _src0 ,y _src0 ) The storage location can be calculated by the following formulas 1 to 3:

Sy＝x _src0 *stride (2)

where stride represents the number of bytes spanned by a row of pixels in the target RAW domain image, sx represents the column index position, sy represents the row index position, and Sd represents the storage offset of the x_src0 th pixel 2 bits lower in a row. That is, in the memory space of the target RAW domain image, the pixel coordinates (x _src0 ,y _src0 ) The storage position of the upper eight bits of the pixel value of (1) is mipi_data [ Sx+Sy)]The lower 2 bits are at the position mipi_data [ Sy+Sd ]]While the lower 2 bits of storage location are related to the element of the four-pixel set that the pixel is in, so the lower 2 bits of binary value are shifted to the right by a factor ((2 x (x_src0% 4))&0x 3) to perform right shift operation.

To sum up, the pixel value pit (x_src0, y_src0) is composed of two parts of high eight bits and low 2 bits, and is stored with an unsigned short integer type (bit width 16 bits), and the calculation process of formula 4 is as follows: piex (x) _src0 ,y _src0 )＝(mipi_data[Sy+Sx]<<2|mipi_data[Sy+Sd])>>((2*(x_src0％4))&0x3)(4)

Wherein, piex (x _src0 ,y _src0 ) In order to be a value of a pixel,>>representing a shift-by-bit right operation,<<representing a bitwise left shift operation, | represents a bitwise OR operation.

In an alternative embodiment, the coordinate mapping relationship applied in step 120 may be determined based on the following steps:

determining target links with coordinate transformation from all links in the preprocessing, and determining a first mapping relation based on the coordinate transformation relation of all the target links;

determining a second mapping relation based on the image blocking rule;

and determining the coordinate mapping relation based on the first mapping relation and the second mapping relation.

Specifically, considering the conventional preprocessing mode, preprocessing is first performed, image blocking is performed on the preprocessed image, and coordinate transformation of pixels exists both during preprocessing and during image blocking. Therefore, the mapping relation under the respective coordinate transformation can be determined for each link under pretreatment and each link of image blocking.

For each link of preprocessing, a link with coordinate transformation can be determined from the links and marked as a target link. For example, in the preprocessing, there is a coordinate transformation of the data image filling minor padding, so the data image filling can be recorded as a target link. It will be appreciated that, as the preprocessing includes differences in links, the number of target links determined therefrom may be one or more, and the order of execution of the target links may be different. After the target link is obtained, the coordinate transformation relation of the target link can be determined based on the coordinate transformation rule of the target link, and the coordinate transformation relation of each target link is multiplied to be used as the mapping relation of the coordinate transformation under the preprocessing link, namely the first mapping relation.

For the image blocking link, a mapping relation of coordinate transformation under the image blocking, namely a second mapping relation, can be determined based on an image blocking rule.

The first and second mappings may then be multiplied as a coordinate mapping between the target RAW domain image and the target tile.

For example, assume that the coordinates of the target RAW domain image before the data mirror filling are (x _src0 ,y _src0 ) The coordinates of the target RAW domain image after the data mirror filling are (x _src1 ,y _src1 ) Can be obtained by combining (x _src0 ,y _src0 ) Substituting the coordinate transformation relation filled with the data mirror image to obtain (x _src1 ,y _src1 ) Then correspondingly, the code can be transmitted through (x _src1 ,y _src1 ) The coordinate transformation relation is inversely deduced to obtain (x) _src0 ,y _src0 )。

Here, the coordinate transformation relation of the back-push data mirror filling can be expressed as the following formulas 5, 6:

in the packing _left 、padding _right 、padding _width The left boundary, the right boundary and the width of the filling area are respectively; packing _up 、padding _bottom 、padding _height The upper boundary, the lower boundary and the height of the filling area are respectively; src (Src) _width And rc _height The image width and the image height, respectively.

In the embodiment of the application, the coordinate mapping relation is determined by combining two stages of preprocessing and image blocking, so that the coordinate mapping relation can be directly called in the image preprocessing process to extract the RAW domain image block, and the image preprocessing efficiency is improved.

In an alternative embodiment, step 130 is followed by:

image processing is performed based on the target tile.

Specifically, the target tile obtained in step 130 may be a tile directly input as a neural network to which image processing is applied. Thus, after the target image block is obtained, the target image can be input into the neural network, and the target image block is subjected to image processing through the neural network so as to obtain an image processing result output by the neural network. The image processing herein may specifically be image classification, image segmentation, image enhancement, and the like, which is not specifically limited in the embodiments of the present application.

It can be understood that the image processing result herein may be obtained by stitching the processing result of each target tile corresponding to the target RAW domain image.

In an alternative embodiment, the image processing based on the target tile includes:

and carrying out multi-frame fusion based on the target image blocks corresponding to the multi-frame target RAW domain images.

Specifically, the preprocessing method can be applied to a multi-frame fused image processing scheme. In the multi-frame fused image processing scheme, the target RAW domain image acquired in step 110 includes a plurality of frame target RAW domain images acquired continuously, and a target block obtained by preprocessing the multi-frame target RAW domain image is used as an input of a neural network for multi-frame fusion, so that an image obtained by fusing the multi-frame target RAW domain images can be obtained. For example, the multi-frame target RAW domain image continuously shot in the night scene mode can be preprocessed to obtain the target image block, and the target image block is input into the night scene multi-frame fused neural network to obtain the night scene image with better quality.

For example, fig. 5 is a schematic flow chart of an image processing method according to an embodiment of the present application, and as shown in fig. 5, the method includes:

the image processing method is specific to the acquired t frame target RAW domain image, namely M1 to Mt in fig. 5. The pixel coordinates in the target RAW domain image can be noted as (x _src0 ,y _src0 ). In addition, in the process of exposing and capturing frames of t frames of target RAW domain images, optical flow configuration information can be combined, and how to extract RAW domain image blocks from RAW and images is calculated in advance in frame gaps based on a coordinate mapping relation, so that after capturing frames, target image blocks can be obtained through direct block preprocessing, and the calculation efficiency is further improved.

Step (1), extracting a RAW domain block from the target RAW domain image based on a coordinate mapping relation:

assuming that the number of predicted target tiles is m, i.e., corresponds to B1 through Bm in FIG. 5, then for the jth target tile Bj, one may be based onAnd extracting the RAW domain block Bj corresponding to the target block Bj from the target RAW domain image of each frame. In (1) the->And (3) representing the reverse mapping of the coordinate mapping relation, wherein n is the number of links requiring coordinate transformation in preprocessing and image partitioning, and Fi is the reverse mapping of the coordinate transformation relation Fi of the ith coordinate transformation link. It can be seen that B1 to Bm in fig. 5 represent RAW domain tiles corresponding to the target tiles B1 to Bm, and there is a RAW domain tile corresponding to each target tile under each frame of the target RAW domain image.

And (2) preprocessing the RAW domain blocks B1 to Bm respectively to obtain target blocks B1 to Bm. In fig. 5, the preprocessing stage may be Vst conversion.

And (3) respectively inputting the target image blocks B1 to Bm into a neural network for image processing, and obtaining prediction output blocks D1 to Dm respectively corresponding to the target image blocks output by the neural network. In fig. 5, forward represents forward reasoning, i.e., inputting the target image into the neural network for forward reasoning to achieve image processing.

And (4) assembling the prediction output blocks D1 to Dm to the corresponding memory positions in the whole graph so as to realize the splicing of the image blocks and obtain the complete processed image.

It will be appreciated that in actual implementation of the image processing method described above, the only thing consumed for each step is the memory required to process each tile. That is, after processing for a single tile is completed, the memory occupied by processing the tile may be released. After all the image blocks are processed, a complete image can be obtained, and the memory peak value consumed by the method only needs the memory of one complete image, and the block memory size is added when a plurality of image blocks are processed in parallel in the processing process. Compared with the prior art, the method for preprocessing the target RAW domain image with the original image size has the advantage that occupied memory is greatly reduced.

Based on any of the above embodiments, fig. 6 is a second flowchart of the image processing method provided in the embodiment of the present application, as shown in fig. 6, the image processing method may be divided into two parts, corresponding to a left frame and a right frame in fig. 6, where the left frame is used to implement preprocessing for a target RAW domain image to obtain a target tile, and the right frame is used to implement image processing for the target tile.

In the image processing method, first, preprocessing needs to be performed for a target RAW domain image:

in fig. 6, t frames of target RAW domain images, i.e., M1 to Mt, may be included. In the process of preprocessing the t frame target RAW domain image to obtain the target block, n preprocessing links, namely F1 to Fn, are needed, so that the forward mapping process from the target RAW domain image to the target block can be abstracted into productsBy->Can realize the image coordinates (x _src ,y _src ) To the target tile coordinates (x _dst ,y _dst ) Is a transition of (2). Accordingly, the reverse mapping process may be expressed asBy->Can be realized from the target block coordinates (x _dst ,y _dst ) To target RAW domain image coordinates (x _src ,y _src ) Is a transition of (2).

It will be appreciated that byThe corresponding RAW domain tiles in the t-frame target RAW domain image, i.e., b1 to bm in fig. 6, may be obtained from the located target tiles.

The target image block corresponding to the RAW image block and meeting the front-end input requirement of the Bayer neural network can be obtained through the preprocessing links from F1 to Fn in FIG. 6.

After the target tile is obtained, image processing may be performed for the target tile:

and inputting the target image blocks into a Bayer neural network for forward reasoning, so as to obtain the prediction output blocks corresponding to the target image blocks, and assembling the prediction output blocks to the corresponding memory positions in the whole image to realize image block splicing and obtain the complete processed image.

According to the image preprocessing method provided by the embodiment of the application, the execution subject can be an image preprocessing device. In the embodiment of the present application, an image preprocessing device is described by taking an example of an image preprocessing method performed by the image preprocessing device.

Fig. 7 is a schematic structural diagram of an image preprocessing device according to an embodiment of the present application, as shown in fig. 7, the device includes:

an acquisition unit 710 for acquiring a target RAW domain image;

an extracting unit 720, configured to extract a RAW domain tile from the target RAW domain image based on a coordinate mapping relationship, where the coordinate mapping relationship is determined based on a coordinate transformation relationship that performs preprocessing on the target RAW domain image and an image blocking rule after the preprocessing;

The preprocessing unit 730 is configured to preprocess the RAW domain tile to obtain a target tile.

Optionally, the extracting unit includes:

a positioning subunit, configured to determine, based on the coordinate mapping relationship, pixel coordinates of each pixel of the RAW domain tile in the target RAW domain image;

a construction subunit, configured to determine, based on pixel coordinates of each pixel of the RAW domain tile in the target RAW domain image, a color type and a pixel value of each pixel in the RAW domain tile, so as to construct the RAW domain tile.

Optionally, the construction subunit is configured to:

and determining the color type corresponding to each pixel based on the parity of the pixel coordinates of each pixel.

Optionally, the construction subunit is configured to:

and determining a storage position of each pixel in the target RAW domain image based on the pixel coordinates of each pixel, and extracting the pixel value of each pixel from the storage position.

Optionally, the construction subunit is configured to:

determining a column index position and a storage offset in the storage position based on the abscissa in the pixel coordinates;

a row index location in the storage location is determined based on the ordinate in the pixel coordinates.

Optionally, the method further comprises a mapping determining unit for:

determining target links with coordinate transformation from all links in the preprocessing, and determining a first mapping relation based on the coordinate transformation relation of all the target links;

determining a second mapping relation based on the image blocking rule;

and determining the coordinate mapping relation based on the first mapping relation and the second mapping relation.

Optionally, the processing unit is further configured to:

image processing is performed based on the target tile.

Optionally, the processing unit is configured to:

and carrying out multi-frame fusion based on the target image blocks corresponding to the multi-frame target RAW domain images.

In the embodiment of the application, the RAW domain image block corresponding to the target image block is extracted from the target RAW domain image based on the coordinate mapping relation, and the RAW domain image block is preprocessed, so that compared with preprocessing of the whole image size, the memory space required by preprocessing is greatly reduced, a large amount of redundant memory application and operation in the preprocessing process are reduced, the preprocessing efficiency is improved, and the overall performance of an image processing algorithm is improved.

The image preprocessing device in the embodiment of the application may be an electronic device, or may be a component in the electronic device, for example, an integrated circuit or a chip. The electronic device may be a terminal, or may be other devices than a terminal. By way of example, the electronic device may be a mobile phone, tablet computer, notebook computer, palm computer, vehicle-mounted electronic device, mobile internet appliance (Mobile Internet Device, MID), augmented reality (augmented reality, AR)/Virtual Reality (VR) device, robot, wearable device, ultra-mobile personal computer, UMPC, netbook or personal digital assistant (personal digital assistant, PDA), etc., but may also be a server, network attached storage (Network Attached Storage, NAS), personal computer (personal computer, PC), television (TV), teller machine or self-service machine, etc., and the embodiments of the present application are not limited in particular.

The image preprocessing device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiments of the present application.

The image preprocessing device provided in the embodiment of the present application can implement each process implemented by the embodiments of the methods of fig. 1 to 6, and in order to avoid repetition, a detailed description is omitted here.

Optionally, fig. 8 is a schematic structural diagram of an electronic device provided in the embodiment of the present application, as shown in fig. 8, and further provides an electronic device 800, including a processor 801 and a memory 802, where a program or an instruction capable of running on the processor 801 is stored in the memory 802, and the program or the instruction implements each step of the foregoing display method embodiment when executed by the processor 801, and the steps can achieve the same technical effect, so that repetition is avoided and no further description is provided herein.

The electronic device in the embodiment of the application includes the mobile electronic device and the non-mobile electronic device described above.

Fig. 9 is a schematic hardware structure of an electronic device according to an embodiment of the present application.

The electronic device 900 includes, but is not limited to: radio frequency unit 901, network module 902, audio output unit 903, input unit 904, sensor 905, display unit 906, user input unit 907, interface unit 908, memory 909, and processor 910.

Those skilled in the art will appreciate that the electronic device 900 may also include a power source (e.g., a battery) for powering the various components, which may be logically connected to the processor 910 by a power management system to perform functions such as managing charge, discharge, and power consumption by the power management system. The electronic device structure shown in fig. 9 does not constitute a limitation of the electronic device, and the electronic device may include more or less components than shown, or may combine certain components, or may be arranged in different components, which are not described in detail herein.

The input unit 904, in this embodiment, a camera, is configured to acquire a target RAW domain image;

the processor 910 is configured to extract a RAW domain tile from the target RAW domain image based on a coordinate mapping relationship, and pre-process the RAW domain tile to obtain a target tile; the coordinate mapping relationship is determined based on a coordinate transformation relationship for preprocessing the target RAW domain image and an image blocking rule after preprocessing.

In the embodiment of the application, the RAW domain image block corresponding to the target image block is extracted from the target RAW domain image based on the coordinate mapping relation, and the RAW domain image block is preprocessed, so that compared with preprocessing of the whole image size, the memory space required by preprocessing is greatly reduced, a large amount of redundant memory application and operation in the preprocessing process are reduced, the preprocessing efficiency is improved, and the overall performance of an image processing algorithm is improved.

It should be appreciated that in embodiments of the present application, the input unit 904 may include a graphics processor (Graphics Processing Unit, GPU) 9041 and a microphone 9042, with the graphics processor 9041 processing image data of still pictures or video obtained by an image capture device (e.g., a camera) in a video capture mode or an image capture mode. The display unit 906 may include a display panel 9061, and the display panel 9061 may be configured in the form of a liquid crystal display, an organic light emitting diode, or the like. The user input unit 907 includes at least one of a touch panel 9071 and other input devices 9072. Touch panel 9071, also referred to as a touch screen. The touch panel 9071 may include two parts, a touch detection device and a touch controller. Other input devices 9072 may include, but are not limited to, a physical keyboard, function keys (e.g., volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, and so forth, which are not described in detail herein.

The memory 909 may be used to store software programs as well as various data. The memory 909 may mainly include a first storage area storing programs or instructions and a second storage area storing data, wherein the first storage area may store an operating system, application programs or instructions (such as a sound playing function, an image playing function, etc.) required for at least one function, and the like. Further, the memory 909 may include a volatile memory or a nonvolatile memory, or the memory 909 may include both volatile and nonvolatile memories. The nonvolatile Memory may be a Read-Only Memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an Electrically Erasable EPROM (EEPROM), or a flash Memory. The volatile memory may be random access memory (Random Access Memory, RAM), static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double Data Rate SDRAM (ddr SDRAM), enhanced SDRAM (Enhanced SDRAM), synchronous DRAM (SLDRAM), and Direct RAM (DRRAM). Memory 909 in embodiments of the present application includes, but is not limited to, these and any other suitable types of memory.

Processor 910 may include one or more processing units; optionally, the processor 910 integrates an application processor that primarily processes operations involving an operating system, user interface, application programs, etc., and a modem processor that primarily processes wireless communication signals, such as a baseband processor. It will be appreciated that the modem processor described above may not be integrated into the processor 910.

The embodiment of the present application further provides a readable storage medium, where a program or an instruction is stored, and when the program or the instruction is executed by a processor, the processes of the foregoing display method embodiment are implemented, and the same technical effects can be achieved, so that repetition is avoided, and no further description is given here.

Wherein the processor is a processor in the electronic device described in the above embodiment. The readable storage medium includes computer readable storage medium such as computer readable memory ROM, random access memory RAM, magnetic or optical disk, etc.

The embodiment of the application further provides a chip, the chip includes a processor and a communication interface, the communication interface is coupled with the processor, and the processor is used for running a program or an instruction, so as to implement each process of the embodiment of the display method, and achieve the same technical effect, so that repetition is avoided, and no redundant description is provided here.

It should be understood that the chips referred to in the embodiments of the present application may also be referred to as system-on-chip chips, chip systems, or system-on-chip chips, etc.

The embodiments of the present application provide a computer program product, which is stored in a storage medium, and the program product is executed by at least one processor to implement the respective processes of the embodiments of the display method, and achieve the same technical effects, and are not repeated herein.

It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element. Furthermore, it should be noted that the scope of the methods and apparatus in the embodiments of the present application is not limited to performing the functions in the order shown or discussed, but may also include performing the functions in a substantially simultaneous manner or in an opposite order depending on the functions involved, e.g., the described methods may be performed in an order different from that described, and various steps may also be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

From the above description of the embodiments, it will be clear to those skilled in the art that the above-described embodiment method may be implemented by means of software plus a necessary general hardware platform, but of course may also be implemented by means of hardware, but in many cases the former is a preferred embodiment. Based on such understanding, the technical solutions of the present application may be embodied essentially or in a part contributing to the prior art in the form of a computer software product stored in a storage medium (such as ROM/RAM, magnetic disk, optical disk), comprising several instructions for causing a terminal (which may be a mobile phone, a computer, a server, or a network device, etc.) to perform the methods described in the embodiments of the present application.

The embodiments of the present application have been described above with reference to the accompanying drawings, but the present application is not limited to the above-described embodiments, which are merely illustrative and not restrictive, and many forms may be made by those of ordinary skill in the art without departing from the spirit of the present application and the scope of the claims, which are also within the protection of the present application.

Claims (10)

1. An image preprocessing method, characterized by comprising:

acquiring a target RAW domain image;

extracting a RAW domain block from the target RAW domain image based on a coordinate mapping relationship, wherein the coordinate mapping relationship is determined based on a coordinate transformation relationship for preprocessing the target RAW domain image and an image blocking rule after preprocessing;

preprocessing the RAW domain image block to obtain a target image block.

2. The image preprocessing method according to claim 1, wherein the extracting a RAW domain tile from the target RAW domain image based on a coordinate mapping relationship comprises:

determining pixel coordinates of each pixel of the RAW domain block in the target RAW domain image based on the coordinate mapping relation;

and determining the color type and the pixel value of each pixel in the RAW domain block based on the pixel coordinates of each pixel of the RAW domain block in the target RAW domain image so as to construct the RAW domain block.

3. The image preprocessing method according to claim 2, wherein determining pixel values of pixels in the RAW domain tile based on pixel coordinates of the pixels in the target RAW domain image, comprises:

And determining a storage position of each pixel in the target RAW domain image based on the pixel coordinates of each pixel, and extracting the pixel value of each pixel from the storage position.

4. The image preprocessing method according to claim 3, wherein said determining a storage location of each pixel in said target RAW domain image based on a pixel coordinate of each pixel comprises:

determining a column index position and a storage offset in the storage position based on the abscissa in the pixel coordinates;

a row index location in the storage location is determined based on the ordinate in the pixel coordinates.

5. The image preprocessing method according to any one of claims 1 to 4, characterized in that the determining of the coordinate mapping relation includes:

determining target links with coordinate transformation from all links in the preprocessing, and determining a first mapping relation based on the coordinate transformation relation of all the target links;

determining a second mapping relation based on the image blocking rule;

and determining the coordinate mapping relation based on the first mapping relation and the second mapping relation.

6. An image preprocessing apparatus, characterized by comprising:

An acquisition unit configured to acquire a target RAW domain image;

the extraction unit is used for extracting a RAW domain block from the target RAW domain image based on a coordinate mapping relation, wherein the coordinate mapping relation is determined based on a coordinate transformation relation for preprocessing the target RAW domain image and an image blocking rule after preprocessing;

the preprocessing unit is used for preprocessing the RAW domain image block to obtain a target image block.

7. The image preprocessing apparatus according to claim 6, wherein the extracting unit includes:

a positioning subunit, configured to determine, based on the coordinate mapping relationship, pixel coordinates of each pixel of the RAW domain tile in the target RAW domain image;

a construction subunit, configured to determine, based on pixel coordinates of each pixel of the RAW domain tile in the target RAW domain image, a color type and a pixel value of each pixel in the RAW domain tile, so as to construct the RAW domain tile.

8. The image preprocessing apparatus of claim 7, wherein said construction subunit is configured to:

and determining a storage position of each pixel in the target RAW domain image based on the pixel coordinates of each pixel, and extracting the pixel value of each pixel from the storage position.

9. The image preprocessing device as set forth in claim 8, wherein said construction subunit is configured to:

determining a column index position and a storage offset in the storage position based on the abscissa in the pixel coordinates;

a row index location in the storage location is determined based on the ordinate in the pixel coordinates.

10. The image preprocessing apparatus according to any one of claims 6 to 9, further comprising a map determining unit configured to:

determining target links with coordinate transformation from all links in the preprocessing, and determining a first mapping relation based on the coordinate transformation relation of all the target links;

determining a second mapping relation based on the image blocking rule;

and determining the coordinate mapping relation based on the first mapping relation and the second mapping relation.