CN110677577A

CN110677577A - Image processing method and device

Info

Publication number: CN110677577A
Application number: CN201810718266.8A
Authority: CN
Inventors: 李晓东
Original assignee: Hangzhou Hai Kang Hui Ying Technology Co Ltd
Current assignee: Hangzhou Hai Kang Hui Ying Technology Co Ltd
Priority date: 2018-07-03
Filing date: 2018-07-03
Publication date: 2020-01-10

Abstract

The application discloses an image processing method and device, and belongs to the field of image processing. The method is applied to the FPGA and comprises the following steps: in the process that the FPGA receives image data, when a line of image data is received, a first image formed by the line of image data is segmented into a plurality of first image blocks; enabling the plurality of first image blocks to be subjected to ISP processing in parallel through a plurality of image signal processing ISP modules in the FGPA to obtain a plurality of second image blocks; and splicing the plurality of second image blocks according to the position identification of each first image block in the plurality of first image blocks in the first image to obtain a second image. In the process of receiving the image data, when one line of image data is received, a first image formed by the one line of image data is blocked, and ISP processing is performed on a plurality of obtained first image blocks in parallel, so that the processing clock frequency of an FPGA (field programmable gate array) is reduced, and the image with higher resolution can be processed in real time.

Description

Image processing method and device

Technical Field

The present disclosure relates to the field of image processing, and in particular, to an image processing method and apparatus.

Background

With the rapid development of electronic equipment, people can shoot anytime and anywhere by using the electronic equipment. In order to obtain images with different effects, after the electronic device captures an image, it is necessary to perform ISP (image signal Processing) Processing on the image on an FPGA (field programmable gate Array) chip of the electronic device. In order to reduce the shooting delay, the electronic device often requires ISP processing on the image in real time.

At present, electronic equipment performs ISP processing on an image on an FPGA chip by adopting a line-by-line pipeline method for the image. However, with the development of image acquisition and processing technology, images taken by electronic devices have higher resolution and image frame rate; for example, the resolution is 4K, and the image frame rate is 60 frames per second. However, for image processing with a resolution of 4K and an image frame rate of 60 frames per second, when ISP processing is performed on an image in a progressive pipeline manner, the internal processing clock frequency of an FPGA chip is required to reach more than 498MHz, and at present, no FPGA chip can stably operate at such a high processing clock frequency. Therefore, the method cannot process images with high resolution and high image frame rate.

Disclosure of Invention

The application provides an image processing method and device, which can reduce the processing clock frequency of an FPGA (field programmable gate array), so that the image with higher resolution can be processed in real time. The technical scheme is as follows:

in a first aspect, an embodiment of the present application provides an image processing method, where the method includes:

in the process that the FPGA receives image data, when a line of image data is received, a first image formed by the line of image data is segmented into a plurality of first image blocks;

enabling the plurality of first image blocks to be subjected to ISP processing in parallel through a plurality of image signal processing ISP modules in the FGPA to obtain a plurality of second image blocks;

and splicing the plurality of second image blocks according to the position identification of each first image block in the plurality of first image blocks in the first image to obtain a second image.

In one possible implementation manner, the dividing the first image composed of the line of image data into a plurality of first image blocks includes:

and keeping the height of a first image formed by the line of image data unchanged, and partitioning the first image according to the horizontal direction to obtain a plurality of first image blocks.

In another possible implementation manner, the blocking the first image according to the horizontal direction to obtain a plurality of first image blocks includes:

partitioning the first image in the horizontal direction to obtain a plurality of third image blocks;

for each third image block, determining an image block adjacent to the third image block according to the position identifier of the third image block in the first image; determining a specified number of rows of pixel points adjacent to the edge pixel point of the third image block in the adjacent image blocks as extended pixel points of the third image block; and combining the extended pixel point and the third image block into a first image block.

In another possible implementation manner, the stitching the plurality of second image blocks according to the position identifier of each of the plurality of first image blocks in the first image to obtain a second image includes:

determining the splicing sequence of each second image block according to the position identification of each first image block in the first image;

and splicing the plurality of second image blocks according to the splicing sequence of each second image block in the horizontal direction to obtain a second image.

In another possible implementation manner, the stitching the plurality of second image blocks according to the stitching order of each second image block to obtain a second image includes:

splicing the plurality of second image blocks according to the splicing sequence of each second image block in the horizontal direction to obtain a third image;

and determining an extension pixel point in the third image, and removing the extension pixel point from the third image to obtain the second image.

respectively determining an extended pixel point in each second image block, and removing the extended pixel point in each second image block;

and splicing each second image block without the extended pixel points according to the splicing sequence of each second image block in the horizontal direction to obtain the second image.

In another possible implementation manner, after the splicing the plurality of second image blocks according to the position identifier of each first image block in the image data to obtain a second image, the method further includes:

and converting the image data of one line of the second image into image data of a specified transmission format, and outputting the converted image data of one line through a high-speed interface.

In another possible implementation manner, after the dividing the first image composed of the line of image data into a plurality of first image blocks, the method further includes:

for each first image block, a vertical blanking area and a horizontal blanking area are determined in the first image block, and the vertical blanking area and the horizontal blanking area are removed from the first image block.

In a second aspect, an embodiment of the present application provides an image processing apparatus, where the apparatus is applied in a field programmable gate array FPGA, and the apparatus includes:

the segmentation module is used for segmenting a first image formed by a line of image data into a plurality of first image blocks when the FPGA receives a line of image data in the process of receiving the image data;

the processing module is used for enabling the plurality of first image blocks to be subjected to ISP processing in parallel through a plurality of image signal processing ISP modules in the FGPA to obtain a plurality of second image blocks;

and the splicing module is used for splicing the plurality of second image blocks according to the position identification of each first image block in the plurality of first image blocks in the first image to obtain a second image.

In a possible implementation manner, the segmentation module is further configured to keep a height of a first image formed by the line of image data unchanged, and block the first image according to a horizontal direction to obtain a plurality of first image blocks.

In another possible implementation manner, the segmentation module is further configured to perform blocking on the first image according to a horizontal direction to obtain a plurality of third image blocks; for each third image block, determining an image block adjacent to the third image block according to the position identifier of the third image block in the first image; determining a specified number of rows of pixel points adjacent to the edge pixel point of the third image block in the adjacent image blocks as extended pixel points of the third image block; and combining the extended pixel point and the third image block into a first image block.

In another possible implementation manner, the stitching module is further configured to determine a stitching order of each second image block according to the position identifier of each first image block in the first image; and splicing the plurality of second image blocks according to the splicing sequence of each second image block in the horizontal direction to obtain a second image.

In another possible implementation manner, the stitching module is further configured to stitch the plurality of second image blocks in the horizontal direction according to the stitching order of each second image block to obtain a third image; and determining an extension pixel point in the third image, and removing the extension pixel point from the third image to obtain the second image.

In another possible implementation manner, the stitching module is further configured to determine an extended pixel in each second image block, and remove the extended pixel in each second image block; and splicing each second image block without the extended pixel points according to the splicing sequence of each second image block in the horizontal direction to obtain the second image.

In another possible implementation manner, the apparatus further includes:

and the output module is used for converting the image data of one line of the second image into the image data of the appointed transmission format and outputting the converted image data of one line through a high-speed interface.

In another possible implementation manner, the apparatus further includes:

the removing module is used for determining a vertical blanking area and a horizontal blanking area in each first image block and removing the vertical blanking area and the horizontal blanking area from the first image blocks.

In a third aspect, an embodiment of the present application provides a field programmable gate array FPGA, where the FPGA includes

At least one processor; and

at least one memory;

the at least one memory stores one or more programs configured to be executed by the at least one processor, the one or more programs including instructions for performing the method as set forth in the first aspect or any possible implementation of the first aspect.

In a fourth aspect, embodiments of the present application provide a non-transitory computer-readable storage medium for storing a computer program, which is loaded by a processor to execute the instructions of the method according to the first aspect or any possible implementation manner of the first aspect.

In the embodiment of the application, in the process of receiving image data by the FPGA, when a line of image data is received, a first image formed by the line of image data is divided into a plurality of first image blocks, and the plurality of first image blocks are subjected to ISP processing in parallel through a plurality of ISP modules in the FPGA to obtain a plurality of second image blocks; and splicing the plurality of second image blocks according to the position identification of each second image block in the plurality of second image blocks in the first image to obtain a second image. The first image composed of a line of received image data is blocked, and ISP processing is performed on the obtained plurality of first image blocks in parallel, so that the processing clock frequency of the FPGA is reduced, and real-time processing on the image with higher resolution can be realized.

Drawings

FIG. 1 is an architecture diagram of an FPGA provided by an embodiment of the present application;

fig. 2 is a flowchart of a method of an image processing method according to an embodiment of the present application;

FIG. 3 is a flowchart of another method for processing an image according to an embodiment of the present disclosure;

fig. 4 is a schematic diagram of an image processing method provided in an embodiment of the present application;

fig. 5 is a schematic structural diagram of an image processing apparatus according to an embodiment of the present application;

fig. 6 is a schematic structural diagram of another image processing apparatus provided in an embodiment of the present application;

fig. 7 is a schematic structural diagram of another image processing apparatus according to an embodiment of the present application.

Detailed Description

To make the objects, technical solutions and advantages of the present application more clear, embodiments of the present application will be described in further detail below with reference to the accompanying drawings.

Referring to fig. 1, an architecture of an FPGA provided in an embodiment of the present application includes: the system comprises an image partitioning module, a plurality of ISP processing modules, an image splicing module and an output module. The output end of the image partitioning module is connected with the input end of each ISP processing module. The output end of each ISP processing module is respectively connected with the input end of the image splicing module.

The image partitioning module is used for acquiring received image data, dividing a first image formed by a line of image data into a plurality of first image blocks when a line of image data is received in the process of receiving the image data, and respectively transmitting the plurality of first image blocks to the plurality of ISP modules. Wherein a first image block is transmitted to an ISP module.

And the ISP processing modules are used for receiving the first image blocks, performing ISP processing on the first image blocks in parallel to obtain second image blocks, and transmitting the second image blocks to the image splicing module.

The image splicing module is used for receiving a plurality of second image blocks, and splicing the plurality of second image blocks according to the position identification of each first image block in the first image to obtain a second image; and transmitting the second image to an output module.

And the output module is used for receiving the second image, converting one line of image data of the second image into image data of a specified transmission format, and outputting the converted one line of image data through the high-speed interface. The designated transmission format can be any transmission format which can be output through a high-speed interface; for example, the specified transmission format may be an AXI4(advanced extensible Interface) -Stream bus format.

It should be noted that the first image may be an image of any resolution; for example, the first image is an image of 1K resolution, an image of 2K resolution, an image of 4K resolution, an image of 8K resolution, an image of 16K resolution, or an image of 32K resolution. In the embodiments of the present disclosure, the resolution of the first image is not particularly limited. In the embodiment of the present application, the image frame rate of the first image is not particularly limited; for example, the image frame rate of the first image to be processed may be 60 frame rates per second.

The embodiment of the application provides an image processing method which is applied to an FPGA. Referring to fig. 2, the method includes:

step 201: in the process that the FPGA receives image data, when one line of image data is received, a first image formed by the one line of image data is divided into a plurality of first image blocks.

Step 202: and carrying out ISP processing on the plurality of first image blocks in parallel through a plurality of image signal processing ISP modules in the FGPA to obtain a plurality of second image blocks.

Step 203: and splicing the plurality of second image blocks according to the position identification of each first image block in the plurality of first image blocks in the first image to obtain a second image.

In the embodiment of the application, in the process of receiving image data by the FPGA, when a line of image data is received, a first image formed by the line of image data is divided into a plurality of first image blocks, and the plurality of first image blocks are subjected to ISP processing in parallel through a plurality of ISP modules in the FPGA to obtain a plurality of second image blocks; and splicing the plurality of second image blocks according to the position identification of each second image block in the plurality of second image blocks in the first image to obtain a second image. The first image is partitioned, and ISP processing is performed on the plurality of first image blocks in parallel, so that the processing clock frequency of the FPGA is reduced, and the image with higher resolution can be processed in real time.

The embodiment of the application provides an image processing method which is applied to an FPGA. Referring to fig. 3, the method includes:

step 301: in the process of receiving image data by the FPGA, when receiving a line of image data, the FPGA divides a first image formed by the line of image data into a plurality of first image blocks.

The electronic equipment comprises an image acquisition module and an FPGA, wherein the image acquisition module is used for acquiring image data and transmitting the acquired image data to the FPGA, and the image data are transmitted in a line and a row. The FPGA receives image data acquired by the image acquisition module, and when one line of image data is received, the FPGA keeps the height of a first image formed by the one line of image data unchanged, and blocks the first image according to the horizontal direction to obtain a plurality of first image blocks.

In one possible implementation, the stitching of the plurality of first image blocks is equal to the first image. Correspondingly, the step of the FPGA blocking the first image according to the horizontal direction to obtain a plurality of first image blocks may be: the FPGA determines the number of image blocks of the first image, and blocks the first image according to the number of the image blocks in the horizontal direction to obtain a plurality of first image blocks. The sizes of the first image blocks may be equal or unequal.

The FPGA may determine the number of image blocks into which the first image is cut according to the size of the first image, and accordingly, the step of determining the number of image blocks into which the first image is cut by the FPGA may be: the FPGA determines the pixel number range in which the pixel number is located according to the pixel number included in the first image, and determines the image block number of the first image cut from the corresponding relation between the pixel number range and the image block number according to the pixel number range.

In addition, the FPGA can also default to the number of image blocks. Correspondingly, the step of determining, by the FPGA, the number of image blocks into which the first image is cut may be: the FPGA takes the default image block number as the image block number of the first image to be cut out. In addition, the user can also set the number of image blocks into which the first image is cut. Correspondingly, the step of determining, by the FPGA, the number of image blocks into which the first image is cut may be: the FPGA receives the number of inputs as the number of image blocks into which the first image is sliced. In addition, the FPGA can also determine the number of image blocks into which the first image is cut according to the number of ISP modules it includes. Correspondingly, the number of image blocks is not more than the number of ISP modules included in the FPGA

For example, the first image is an image with a resolution of 4K (3840 × 2160), the number N of image blocks into which the first image is cut is 4, and the FPGA performs equal-size blocking on the first image. Accordingly, each first image block has a width of 3840/4 ═ 960 and a height of 2160.

In another possible implementation, when ISP processing is performed on each pixel in the image block, image data of pixels before and/or after the pixel may be used. Therefore, when the FPGA is used for partitioning, each image block is expanded by a plurality of columns of pixel points leftwards and/or rightwards. Correspondingly, the step of the FPGA segmenting the first image composed of the line of image data into a plurality of first image blocks may be implemented by the following steps (1) and (2), including:

(1): and the FPGA keeps the height of the first image unchanged, and blocks the first image according to the horizontal direction to obtain a plurality of third image blocks.

And the FPGA determines the number of image blocks of the first image, keeps the height of the first image unchanged, and blocks the first image according to the number of the image blocks in the horizontal direction to obtain a plurality of third image blocks. The size of each third image block may be equal or unequal. And splicing the plurality of third image blocks to obtain a first image.

In a possible implementation manner, when the FPGA divides the first image into a plurality of third image blocks, the FPGA determines a position identifier of each third image block in the first image, and stores the position identifier of each third image block in the first image, so that the image blocks are subsequently spliced based on the position identifier of each third image block in the first image. The location identifier may be a location serial number of the image block in the first image. For example, the FPGA slices the first image into four third image blocks, namely, a third image block 1, a third image block 2, a third image block 3, and a third image block 4 from left to right. The position identifiers of the third image block 1, the third image block 2, the third image block 3 and the third image block 4 in the first image are 1, 2, 3 and 4, respectively.

(2): and for each third image block, the FPGA determines an extended pixel point of the third image block, and the extended pixel point and the third image block form a first image block.

This step can be realized by the following steps (2-1) and (2-3), including:

(2-1): and for each third image block, the FPGA determines the image block adjacent to the third image block according to the position identifier of the third image block in the first image.

And the FPGA determines an image block adjacent to the third image block from the plurality of third image blocks according to the position identification of the third image block in the first image. For example, when the third image block is the leftmost image block in the first image, only one adjacent image block of the third image block is the image block on the right side of the third image block. When the third image is the rightmost image block in the first image, only one image block adjacent to the third image block is the image block on the left side of the third image block. When the third image block is a middle image block except the leftmost image block or the rightmost image block in the first image, two adjacent image blocks of the third image block are respectively the left image block and the right image block of the third image block.

(2-2): and the FPGA determines a first specified number of rows of pixel points adjacent to the edge pixel point of the third image block in the adjacent image blocks as extended pixel points of the third image block.

The edge pixel points of the third image block are the leftmost column of pixel points and/or the rightmost column of pixel points in the third image block. For example, when the third image block is the leftmost image block, the edge pixel point of the third image block is the rightmost row pixel point in the third image block. When the third image block is the rightmost image block, the edge pixel point of the third image block is the leftmost pixel point in the third image block. When the third image block is a middle image block, edge pixel points of the third image block are leftmost pixel points and rightmost pixel points in the third image block.

In addition, the first designated number may be set and changed as needed, and in the embodiment of the present application, the first designated number is not specifically limited; for example, the first specified number may be 2 or 3, etc.

(2-3): and the FPGA forms the extended pixel point and the third image block into a first image block.

And for each third image block, the FPGA forms the third image block and the extended pixel points of the third image block into a first image block.

It should be noted that after the FPGA splits the first image into a plurality of first image blocks, the FPGA determines a position identifier of each first image block in the first image, and stores the position identifier of each first image block in the first image, so as to perform image stitching subsequently.

Another point to be described is that when the FPGA blocks the first image to obtain a plurality of first image blocks, each first image block is extended by several rows of pixel points to the left and/or the right. Therefore, the result of the first image block after edge extension in ISP processing is consistent with the whole image line-by-line processing result, and the processing accuracy is improved.

For example, referring to fig. 4, the FPGA receives a first image to be processed, which is a 4K resolution image, through the AXI14-Stream bus. The FPGA divides the first image into N first image blocks, namely a first image block 1, a first image block 2, … …, and a first image block N, according to the horizontal direction. In fig. 4, N is 4 as an example.

When the first image is an image with a resolution of 4K (3840 × 2160), the second designated number is 2, and the number N of the first image split into image blocks is 4, the width of the leftmost image block in the first image is: 3840/N +2 ═ 962, height 2160. The width of the middle first image block in the first image is 2+3840/N +2 ═ 964, and the height is 2160. The rightmost first image block in the first image has a width of 2+ 3840/N962 and a height of 2160.

Note that the first picture includes a field blanking region and a line blanking region. Therefore, before the FPGA slices the first image, the first image may be blanked. And then slicing the first image after the blanking processing. In addition, the FPGA may also perform blanking processing on each first image block when the first image is divided into a plurality of first image blocks, instead of performing blanking processing on the first image first. Step 302 is then performed.

Wherein the blanking process includes field blanking and line blanking. Correspondingly, the step of blanking the first image by the FPGA may be: the FPGA determines a vertical blanking area and a horizontal blanking area in the first image, and the vertical blanking area and the horizontal blanking area are removed from the first image.

For each first image block, the step of blanking the first image block by the FPGA may be: the FPGA determines a vertical blanking area and a horizontal blanking area in the first image block, and the vertical blanking area and the horizontal blanking area are removed from the first image block.

Step 302: and the FPGA carries out ISP processing on the plurality of first image blocks in parallel through a plurality of ISP modules in the FPGA to obtain a plurality of second image blocks.

The FPGA comprises a plurality of ISP modules, and the plurality of first image blocks are subjected to ISP processing in parallel through the plurality of ISP processing modules to obtain a plurality of second image blocks. Wherein an ISP module processes a first image block.

It should be noted that the ISP processing includes at least one of black level correction, digital gain, white balance, color interpolation, color correction, gamma correction, color domain conversion, sharpening processing, auto exposure, auto white balance, and auto focus statistics. Therefore, for each first image block, the step of ISP processing of the first image block by the FPGA may be: the FPGA carries out at least one of black level correction, digital gain, white balance, color interpolation, color correction, gamma correction, color domain conversion, sharpening processing, automatic exposure, automatic white balance and automatic focusing statistics on the first image block to obtain a second image block.

It should be noted that, after the FPGA processes each first image block to obtain each second image block, for each first image block and each second image block, the FPGA associates the first image block and the second image block, so as to perform image block splicing subsequently.

For example, the FPGA performs ISP processing on the N first image blocks in parallel through the ISP module to obtain N second image blocks, which are respectively the second image block 1, the second image block 2, … …, and the second image block N.

Step 303: and the FPGA determines the splicing sequence of each second image block according to the position identification of each first image block in the first image.

The FPGA determines the position relation of each first image block according to the position identification of each first image block in the first image, determines the position relation of each first image block as the position relation of each second image block, and determines the splicing sequence of each second image block according to the position relation of each second image block.

For example, the FPGA divides the first image into 4 first image blocks, which are the first image block 1, the first image block 2, the first image block 3, and the first image block 4, respectively. The first image block 1, the first image block 2, the first image block 3 and the first image block 4 are in a position relationship that the first image block 1 is adjacent to the first image block 2, the first image block 2 is adjacent to the first image block 3, and the first image block 3 is adjacent to the first image block 4. The second image blocks corresponding to the first image block 1, the first image block 2, the first image block 3, and the first image block 4 are the second image block 1, the second image block 2, the second image block 3, and the second image block 4, respectively. Thus, the position relationships of the second image block 1, the second image block 2, the second image block 3, and the second image block 4 are determined to be that the second image block 1 is adjacent to the second image block 2, the second image block 2 is adjacent to the second image block 3, and the second image block 3 is adjacent to the second image block 4. Therefore, the splicing sequence of the second image block 1, the second image block 2, the second image block 3 and the second image block 4 obtained by the FPGA is as follows: a second image block 1, a second image block 2, a second image block 3 and a second image block 4.

Step 304: and the FPGA splices the plurality of second image blocks in the horizontal direction according to the splicing sequence of each second image block to obtain a second image.

In a possible implementation manner, when the first image block does not include the extended pixel point, the FPGA splices the plurality of second image blocks according to a horizontal direction according to an order of the plurality of second image blocks to obtain a second image.

For example, the second image blocks corresponding to the first image block 1, the first image block 2, the first image block 3, and the first image block 4 are the second image block 1, the second image block 2, the second image block 3, and the second image block 4. The FPGA determines the sequence of the plurality of second image blocks to be a second image block 1, a second image block 2, a second image block 3 and a second image block 4 respectively, and the second image block 1, the second image block 2, the second image block 3 and the second image block 4 are spliced to obtain a second image.

In another possible implementation manner, when the first image block includes the extended pixel, the FPGA may be first spliced, and then the extended pixel is deleted. Accordingly, step 304 may be implemented by the following steps (1) and (3), including:

(1): and the FPGA splices the plurality of second image blocks in the horizontal direction according to the splicing sequence of each second image block to obtain a third image.

(2): and the FPGA determines an extension pixel point in the third image, and removes the extension pixel point from the third image to obtain a second image.

And the FPGA determines the specified column expansion pixel points from the third image according to the position identification of each second image block in the third image. For each second image block, when the second image block is the leftmost image block, the FPGA determines the rightmost appointed row of pixel points of the second image block as extended pixel points. And when the second image block is the rightmost image block, the FPGA determines the leftmost appointed row of pixel points of the second image block as extended pixel points. And when the second image block is a middle image block, the FPGA determines the leftmost appointed row of pixel points and the rightmost appointed row of pixel points of the second image block as extended pixel points.

In another possible implementation manner, when the first image block includes the extended pixel, the FPGA may delete the extended pixel and then perform splicing. Accordingly, step 304 may be implemented by the following steps (a) to (B), including:

(A) the method comprises the following steps And the FPGA determines an extended pixel point in each second image block respectively and removes the extended pixel point in each second image block.

And for each second image block, the FPGA determines the extended pixel points with the specified number from the second image block according to the position identification of the first image block corresponding to the second image block in the first image. For example, for each second image block, when the second image block is the leftmost image block, the FPGA determines the designated column of pixel points on the rightmost side of the second image block as extended pixel points. And when the second image block is the rightmost image block, the FPGA determines the leftmost appointed row of pixel points of the second image block as extended pixel points. And when the second image block is a middle image block, the FPGA determines the leftmost appointed row of pixel points and the rightmost appointed row of pixel points of the second image block as extended pixel points.

(B) The method comprises the following steps And the FPGA splices each second image block without the extended pixel points according to the splicing sequence of each second image block to obtain a second image.

For example, the FPGA splices the N second image blocks to obtain a second image. When the first image is an image of 4K resolution, the second image is also an image of 4K resolution.

Step 305: the FPGA outputs a line of image data of the second image.

In one possible implementation, the FPGA outputs the second image directly through the output interface as a line of image data. For example, one line of image data of the second image is output to be displayed or stored.

In another possible implementation, the FPGA may perform format conversion on a line of image data of the second image and then output the converted line of image data. Correspondingly, the steps can be as follows: and the FPGA converts the one-line image data of the second image into the image data of the appointed transmission format and outputs the converted one-line image data of the second image through the high-speed interface.

It should be noted that, the specified transmission format may be set and changed as needed, and in the embodiment of the present application, the specified transmission format is not specifically limited; for example, the specified transport format may be the AXI14-Stream bus format. For example, the FPGA outputs image data of the second image through the AXI14-Stream bus.

Another point to be described is that the method can be applied to a scene in which an FPGA performs imaging. Therefore, in the process of receiving the image data by the FPGA, each time the FPGA receives one line of image data, the image corresponding to the one line of image data is processed through the

steps

301 and 305.

Another point to be noted is that, since the first image is blocked, the processing clock frequency of the FPGA is reduced from the original CLK1 to CLK2 — CLK 1/N. Therefore, real-time processing of images with higher resolution (for example, 4K resolution, 8K resolution, 16K resolution and 64K resolution) can be stably realized on the FPGA. For example, the FPGA inputs a line of image data through AXI14-Stream bus format, the bus clock frequency is 148.5MHz, the bus bit width is 64bit, and the conversion to progressive processing clock frequency is:

therefore, the processing clock frequency of the FPGA is up to 594 MHz. However, none of the current FPGA chips can stably run a processing clock of 594 MHz. In the embodiment of the application, the FPGA blocks the first image composed of the line of image data, and then processes the plurality of first image blocks obtained by blocking in parallel, so that the processing clock frequency can be reduced by N times. For example, when N is 2, the processing clock frequency of the FPGA is reduced to 594MHz/2 to 297 MHz. And the larger the value of N is, the lower the processing clock frequency of the FPGA is. Moreover, the current FPGA can stably operate a processing clock of 297 MHz. Therefore, the image with a larger resolution ratio can be processed in real time in the current FPGA, and the processing efficiency is improved.

The embodiment of the application provides an image processing device, which is applied to an FPGA and used for executing the steps executed by the FPGA in the image processing method. Referring to fig. 5, the apparatus includes:

a segmentation module 501, configured to segment a first image formed by a line of image data into a plurality of first image blocks when the FPGA receives the line of image data;

a processing module 502, configured to perform ISP processing on the multiple first image blocks in parallel through multiple ISP modules for graphics signal processing in the FGPA to obtain multiple second image blocks;

the stitching module 503 is configured to stitch the plurality of second image blocks according to the position identifier of each of the plurality of first image blocks in the first image, so as to obtain a second image.

In a possible implementation manner, the dividing module 501 is further configured to keep a height of a first image formed by the line of image data unchanged, and divide the first image into a plurality of first image blocks according to a horizontal direction.

In another possible implementation manner, the segmentation module 501 is further configured to segment the first image according to a horizontal direction to obtain a plurality of third image blocks; for each third image block, determining an image block adjacent to the third image block according to the position identifier of the third image block in the first image; determining a specified number of rows of pixel points adjacent to the edge pixel point of the third image block in the adjacent image blocks as expansion pixel points of the third image block; and combining the extended pixel point and the third image block into a first image block.

In another possible implementation manner, the stitching module 503 is further configured to determine a stitching order of each second image block according to the position identifier of each first image block in the first image; and splicing the plurality of second image blocks according to the splicing sequence of each second image block in the horizontal direction to obtain a second image.

In another possible implementation manner, the stitching module 503 is further configured to stitch the plurality of second image blocks according to the stitching order of each second image block, and obtain a third image; and determining an extension pixel point in the third image, and removing the extension pixel point from the third image to obtain the second image.

In another possible implementation manner, the stitching module 503 is further configured to determine an extended pixel point in each second image block, and remove the extended pixel point in each second image block; and splicing each second image block without the extended pixel points according to the splicing sequence of each second image block in the horizontal direction to obtain the second image.

In another possible implementation, referring to fig. 6, the apparatus further includes:

an output module 504, configured to convert one line of image data of the second image into image data in a specified transmission format, and output the converted one line of image data through a high-speed interface.

In another possible implementation, referring to fig. 7, the apparatus further includes:

a removing module 505, configured to, for each first image block, determine a vertical blanking area and a horizontal blanking area in the first image block, and remove the vertical blanking area and the horizontal blanking area from the first image block.

In the embodiment of the application, in the process of receiving image data by the FPGA, when a line of image data is received, a first image formed by the line of image data is divided into a plurality of first image blocks, and the plurality of first image blocks are subjected to ISP processing in parallel through a plurality of ISP modules in the FPGA to obtain a plurality of second image blocks; and splicing the plurality of second image blocks according to the position identification of each second image block in the plurality of second image blocks in the first image to obtain a second image. The first image is partitioned, and ISP processing is performed on the plurality of first image blocks in parallel, so that the processing clock frequency of the FPGA is reduced, the image with higher resolution can be processed in real time, the processing time is shortened due to parallel processing, and the processing efficiency is improved.

It should be noted that: in the image processing apparatus provided in the above embodiment, only the division of the above functional modules is taken as an example for illustration during image processing, and in practical applications, the above function distribution may be completed by different functional modules according to needs, that is, the internal structure of the apparatus is divided into different functional modules to complete all or part of the above described functions. In addition, the image processing apparatus and the image processing method provided by the above embodiments belong to the same concept, and specific implementation processes thereof are described in the method embodiments in detail and are not described herein again.

The embodiment of the application provides a non-volatile computer-readable storage medium for storing a computer program, wherein the computer program is loaded through an FPGA (field programmable gate array) to execute the instructions of the image processing method.

It will be understood by those skilled in the art that all or part of the steps for implementing the above embodiments may be implemented by hardware, or may be implemented by a program instructing relevant hardware, where the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.

The above description is only exemplary of the present application and should not be taken as limiting the present application, as any modification, equivalent replacement, or improvement made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. An image processing method is applied to a Field Programmable Gate Array (FPGA), and comprises the following steps:

2. The method of claim 1, wherein the dividing the first image formed by the line of image data into a plurality of first image blocks comprises:

3. The method of claim 2, wherein the partitioning the first image in a horizontal direction to obtain a plurality of first image blocks comprises:

4. The method according to claim 1, wherein the stitching the plurality of second image blocks according to the position identifier of each of the plurality of first image blocks in the first image to obtain a second image comprises:

5. The method according to claim 4, wherein the stitching the plurality of second image blocks according to the stitching order of each second image block to obtain a second image comprises:

6. The method according to claim 4, wherein the stitching the plurality of second image blocks according to the stitching order of each second image block to obtain a second image comprises:

7. The method according to any one of claims 1 to 6, wherein after the splicing the plurality of second image blocks according to the position identifier of each first image block in the image data to obtain a second image, the method further comprises:

8. The method according to claim 1, wherein after said dividing the first image composed of said one line of image data into a plurality of first image blocks, said method further comprises:

9. An image processing apparatus, wherein the apparatus is applied in a Field Programmable Gate Array (FPGA), the apparatus comprising:

10. The apparatus of claim 9,

the segmentation module is further configured to keep a height of a first image formed by the line of image data unchanged, and perform block division on the first image according to a horizontal direction to obtain a plurality of first image blocks.

11. The apparatus of claim 10,

the segmentation module is further configured to segment the first image according to a horizontal direction to obtain a plurality of third image blocks; for each third image block, determining an image block adjacent to the third image block according to the position identifier of the third image block in the first image; determining a specified number of rows of pixel points adjacent to the edge pixel point of the third image block in the adjacent image blocks as extended pixel points of the third image block; and combining the extended pixel point and the third image block into a first image block.

12. The apparatus of claim 9,

the splicing module is further configured to determine a splicing sequence of each second image block according to the position identifier of each first image block in the first image; and splicing the plurality of second image blocks according to the splicing sequence of each second image block in the horizontal direction to obtain a second image.

13. The apparatus of claim 12,

the splicing module is further configured to splice the plurality of second image blocks in the horizontal direction according to the splicing sequence of each second image block to obtain a third image; and determining an extension pixel point in the third image, and removing the extension pixel point from the third image to obtain the second image.

14. The apparatus of claim 12,

the splicing module is further configured to determine an extended pixel in each second image block, and remove the extended pixel in each second image block; and splicing each second image block without the extended pixel points according to the splicing sequence of each second image block in the horizontal direction to obtain the second image.

15. The apparatus of any of claims 9-14, further comprising:

16. The apparatus of claim 9, further comprising: