CN108564551B - Fisheye image processing method and fisheye image processing device - Google Patents
Fisheye image processing method and fisheye image processing device Download PDFInfo
- Publication number
- CN108564551B CN108564551B CN201810381677.2A CN201810381677A CN108564551B CN 108564551 B CN108564551 B CN 108564551B CN 201810381677 A CN201810381677 A CN 201810381677A CN 108564551 B CN108564551 B CN 108564551B
- Authority
- CN
- China
- Prior art keywords
- image
- data
- image processing
- block
- processing circuit
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000012545 processing Methods 0.000 title claims abstract description 140
- 238000003672 processing method Methods 0.000 title claims abstract description 24
- 238000013507 mapping Methods 0.000 claims abstract description 110
- 238000004364 calculation method Methods 0.000 claims abstract description 39
- 238000000034 method Methods 0.000 claims abstract description 31
- 238000005070 sampling Methods 0.000 claims description 10
- 238000007906 compression Methods 0.000 claims description 3
- 230000006835 compression Effects 0.000 claims description 3
- 238000004590 computer program Methods 0.000 claims description 3
- 238000003702 image correction Methods 0.000 abstract description 4
- 238000012937 correction Methods 0.000 description 10
- 239000011159 matrix material Substances 0.000 description 10
- 238000003384 imaging method Methods 0.000 description 7
- 238000000638 solvent extraction Methods 0.000 description 6
- 238000010586 diagram Methods 0.000 description 5
- 230000006837 decompression Effects 0.000 description 4
- 230000000903 blocking effect Effects 0.000 description 3
- 238000013519 translation Methods 0.000 description 3
- 238000004458 analytical method Methods 0.000 description 2
- 230000002457 bidirectional effect Effects 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 2
- 238000004422 calculation algorithm Methods 0.000 description 2
- 238000013144 data compression Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000001914 filtration Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 238000013500 data storage Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000008030 elimination Effects 0.000 description 1
- 238000003379 elimination reaction Methods 0.000 description 1
- 238000012806 monitoring device Methods 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 238000005096 rolling process Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/80—Geometric correction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/04—Context-preserving transformations, e.g. by using an importance map
- G06T3/047—Fisheye or wide-angle transformations
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4007—Scaling of whole images or parts thereof, e.g. expanding or contracting based on interpolation, e.g. bilinear interpolation
Landscapes
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Image Processing (AREA)
Abstract
The invention discloses a fish-eye image processing method and a device, wherein the method comprises the steps of determining the mapping relation between each pixel point in an output image and a pixel point in an input image, forming a mapping relation corresponding table, and sending the mapping relation corresponding table to an image processing circuit; sending data of an input image to an image processing circuit; and after receiving the data of the input image, the image processing circuit performs interpolation calculation on the data of the input image according to the mapping relation of the mapping relation corresponding table to obtain the data of each pixel point of the output image. The device comprises a processor and an image processing circuit and is used for realizing the fisheye image processing method. The invention can improve the processing efficiency of the fisheye image and save the bandwidth of a hardware circuit occupied in the image correction process.
Description
Technical Field
The present invention relates to the field of image processing, and in particular, to a method for processing an image captured by a capturing device with a fisheye lens and an apparatus for implementing the method.
Background
With the development of photographing devices, people have widely used various wide-angle lenses for taking pictures, in which a fisheye lens is a wide-angle lens having an extremely wide angle of view. Photos taken by using the fisheye lens are generally called fisheye images, and due to the fact that the field angle of the fisheye images is large, a single fisheye camera can acquire the content of 180-degree field information, and the characteristic enables the fisheye lens to be increasingly applied to the fields of automatic driving, panoramic monitoring, virtual reality and the like.
However, since the fisheye image itself has large distortion, the fisheye image needs to be corrected by an appropriate mapping relationship and distortion correction algorithm, and an undistorted image conforming to the human visual characteristics is output. When the fisheye image is corrected by being limited by the uncertainty of the mapping relation and the requirement of a distortion correction algorithm on a data source, most of the current processing methods are applied to realize the correction of the image by using a software program, and along with the fact that the pixels of the image are higher and higher, the correction of the fisheye image by using the software program causes poor real-time performance of image output, and real-time live broadcast in a video mode cannot be realized.
If a hardware circuit, such as an image processing circuit, is used to correct a fisheye image in real time, most of the current methods use a method of caching multiple rows of original pixels or directly obtaining corresponding pixel points from the whole image to perform distortion correction calculation, but this method results in large area of the image processing circuit, and the problems of low read-write efficiency of the dynamic random access memory, such as large bandwidth requirement and high hardware requirement.
Specifically, most of the existing fisheye image processing methods employ caching of multiple lines of original pixels or direct acquisition of corresponding pixels from the whole image for distortion correction calculation, for a multiple-line original data caching scheme, enough lines of image data must be stored to ensure that correct data can be obtained for calculation in the distortion correction process, so that the area of a Static Random Access Memory (SRAM) is required to be very large to meet the requirement of data storage, and the data of the SRAM needs to be updated in a rolling manner or a manner of loading partial data repeatedly according to lines, so that the operation is complex and too much bandwidth of an invalid dynamic random access memory is introduced. In addition, the depth of the static memory SRAM used in the method is limited by the image specification, and the upgrading and updating of the chip specification are inconvenient. The method for directly obtaining the corresponding pixel points from the whole image to carry out distortion correction calculation has large and discontinuous address jump when obtaining data, low efficiency of the dynamic random access memory and excessive and ineffective bandwidth of the dynamic random access memory.
Disclosure of Invention
The invention mainly aims to provide a fisheye image processing method for correcting fisheye images through cooperation of software programs and hardware circuits.
Another object of the present invention is to provide a fisheye image processing apparatus that reduces hardware circuit implementation difficulty and reduces bandwidth.
In order to achieve the main purpose, the fisheye image processing method provided by the invention comprises the steps of determining the mapping relation between each pixel point in an output image and a pixel point in an input image, forming a mapping relation corresponding table, and sending the mapping relation corresponding table to an image processing circuit; sending data of an input image to an image processing circuit; and after receiving the data of the input image, the image processing circuit performs interpolation calculation on the data of the input image according to the mapping relation of the mapping relation corresponding table to obtain the data of each pixel point of the output image.
According to the scheme, the image processing method realizes the correspondence of the mapping relation through the software program, and the image processing circuit is used for carrying out interpolation calculation on the input image according to the mapping relation corresponding table and obtaining the output image.
The method comprises the following steps that an output image is divided into a plurality of image blocks, each image block is determined to correspond to a source data block in an input image, and an image processing circuit calculates data of pixel points of one image block each time; wherein sending the mapping relationship correspondence table to the image processing circuit comprises: sending data of a mapping relation corresponding table corresponding to the image block to be processed; wherein sending data of the input image to the image processing circuit comprises: and sending the data of the source data block corresponding to the image block to be processed in the input image to an image processing circuit.
Therefore, the output image is processed in a blocking mode, the image processing circuit can only receive a small amount of data each time, for example, only receive data of the mapping relation corresponding table of the image block to be processed and data of the source data block image each time, and therefore the data amount input to the image processing circuit each time can be greatly reduced, and therefore the occupied bandwidth from the DDR memory to the image processing circuit is reduced.
Before sending the mapping relation corresponding table to the image processing circuit, compressing the data of the mapping relation corresponding table, and sending the compressed data of the mapping relation corresponding table to the image processing circuit; and after receiving the data of the compressed mapping relation corresponding table, the image processing circuit decompresses the received data. Preferably, compressing the data of the mapping relationship correspondence table includes: and carrying out equidistant sampling compression on the data of the mapping relation corresponding table.
Therefore, the data of the mapping relation corresponding table is compressed, so that the data volume sent to the image processing circuit by the processor is small, and the data volume transmitted between the processor and the image processing circuit is effectively reduced. Moreover, the adoption of the technology of equidistant sampling compression can effectively reduce the difficulty of data compression and decompression,
further, the step of decompressing the data in the compressed mapping relationship mapping table by the image processing circuit includes: and performing row-column interpolation calculation on the coordinate values of the edge of the image block, and performing bilinear interpolation calculation on the coordinate values of the middle area of the image block.
Therefore, the accuracy of the decompressed data can be improved by processing the edge coordinate values and the coordinate values of the middle area in different modes.
Further, the sending the data of the source data block corresponding to the image block to be processed in the input image to the image processing circuit includes: before sending the source data block data of the current image block to be processed, judging whether an overlapping area exists between the source data block corresponding to the previous adjacent image block and the current source data block, and if the overlapping area exists, sending data outside the overlapping area to an image processing circuit.
Therefore, by judging whether the source data blocks corresponding to the two adjacent image blocks have the overlapping area or not and transmitting the data in the overlapping area after eliminating the data in the overlapping area, the repeated transmission of the data in the overlapping area can be avoided, and the bandwidth occupied by the image processing circuit is effectively reduced.
Further, the sending the data of the source data block corresponding to the image block to be processed in the input image to the image processing circuit includes: and judging whether the data volume of the current source data block is larger than a preset threshold, if so, dividing the current calculated image block into a plurality of image sub-blocks, determining the source data block corresponding to each image sub-block, and sending the data of the source data block corresponding to one image sub-block to the image processing circuit each time.
Therefore, by blocking the image block again, the data volume of the source data block loaded to the image processing circuit each time can be ensured not to exceed the threshold, so that the area of the buffer of the image processing circuit can be reduced, the area of the image processing circuit can be reduced, and the cost of the image processing circuit can be reduced.
Further, the sending the data of the mapping relationship correspondence table corresponding to the image block to be processed includes: data of a mapping relation correspondence table corresponding to two or more image blocks is transmitted to an image processing circuit at a time.
Therefore, the data of the mapping relation corresponding table corresponding to the plurality of image blocks are transmitted at one time, so that the frequency of transmitting the data to the image processing circuit by the processor can be reduced, and the correction efficiency of the fisheye image is improved.
In order to achieve the above another object, the present invention further provides a fisheye image processing apparatus comprising a processor and an image processing circuit, wherein the processor is configured to implement, when executing a computer program: determining the mapping relation between each pixel point in the output image and the pixel point in the input image, forming a mapping relation corresponding table, and sending the mapping relation corresponding table to the image processing circuit; sending data of an input image to an image processing circuit; wherein the image processing circuit is configured to: and after receiving the data of the input image transmitted by the processor, carrying out interpolation calculation on the data of the input image according to the mapping relation of the mapping relation corresponding table, and calculating the data of each pixel point of the output image.
In a preferred aspect, the processor is further configured to: dividing an output image into a plurality of image blocks, determining that each image block corresponds to a source data block in an input image, and calculating data of pixel points of one image block by an image processing circuit each time; when the mapping relation corresponding table is sent to the image processing circuit, sending data of the mapping relation corresponding table corresponding to the image block to be processed; when the data of the input image is sent to the image processing circuit, the data of the source data block corresponding to the image block to be processed in the input image is sent to the image processing circuit.
According to the scheme, the image processing device comprises the processor and the image processing circuit, the processor realizes the correspondence of the mapping relation through the software program, and the image processing circuit is used for carrying out interpolation calculation on the input image according to the mapping relation correspondence table and obtaining the output image.
Drawings
Fig. 1 is a flowchart of an embodiment of a fisheye image processing method according to the invention.
Fig. 2 is a block diagram of a fisheye image processing apparatus according to an embodiment of the invention.
Fig. 3 is a schematic diagram of a plurality of coordinate systems of a fisheye image corrected by an embodiment of the fisheye image processing method of the invention.
Fig. 4 is a flowchart of coordinate system calculation in the fisheye image processing process according to the embodiment of the fisheye image processing method of the invention.
Fig. 5 is a schematic diagram illustrating blocking of an output image according to an embodiment of the fisheye image processing method of the invention.
Fig. 6 is a schematic diagram of a mapping relationship correspondence table and an image block in an embodiment of the fisheye image processing method of the invention.
Fig. 7 is a flowchart of a parameter analysis unit in an embodiment of the apparatus for processing a fisheye image according to the invention.
Fig. 8 is a flowchart of source data block re-blocking in an embodiment of the apparatus for processing fisheye image according to the invention.
Fig. 9 is a flowchart of the operation of the data source block loading unit in the embodiment of the fisheye image processing apparatus of the invention.
Fig. 10 is a schematic diagram of source data blocks corresponding to two adjacent image blocks in an embodiment of the fisheye image processing method of the invention.
Fig. 11 is a flowchart of the operation of the bilinear interpolation unit in the embodiment of the fisheye image processing device of the invention.
Fig. 12 is a flowchart of the operation of the interpolation result output unit in the embodiment of the fisheye image processing apparatus of the invention.
The invention is further explained with reference to the drawings and the embodiments.
Detailed Description
The fisheye image processing method is applied to various shooting devices, such as cameras, monitoring devices and the like, and can also be applied to mobile devices with a shooting function, such as electronic devices, such as smart phones, tablet computers or automobile data recorders and the like. The fisheye image processing device of the invention can be a device which is applied to various shooting devices and is realized by a processor and an image processing circuit.
The invention mainly processes the fisheye image by combining a software program and a hardware circuit. Since the fisheye image is captured by a capturing device with a fisheye lens, and the captured image is a two-dimensional image with distortion, each pixel of the image has its own row and column coordinates, in this embodiment, the original image captured by the capturing device is referred to as an input image, i.e., an image input to the fisheye image processing apparatus. Accordingly, the corrected image is also a two-dimensional image, each pixel in the image has its own row coordinate and column coordinate, and the corrected image is referred to as an output image, i.e., an image output by the fisheye image processing apparatus.
Therefore, in this embodiment, it is first required to determine which row coordinates of each pixel point in the output image correspond to which pixels in the input image, and determine which column coordinates of each pixel point in the output image correspond to which pixels in the input image. After determining the pixel correspondence between the output image and the input image, a mapping correspondence table is formed, and then the data of the input image is corrected, such as interpolated, by the image processing circuit according to the data of the mapping correspondence table, thereby obtaining the output image.
As shown in fig. 1, the fisheye image processing method of this embodiment first executes step S1 to determine the mapping relationship between each pixel point in the output image and the pixel point between the input images, and then executes step S2 to form a mapping relationship correspondence table according to the correspondence determined in step S1, and sends the mapping relationship correspondence table to the image processing circuit, so in this embodiment, steps S1 to S3 are implemented by a processor, that is, steps implemented by a computer program executed by the processor, and steps S4 to S9 are implemented by an image processing circuit, that is, steps implemented by an integrated circuit.
Referring to fig. 2, the fisheye image processing apparatus includes a processor 10, an image processing circuit 20, and a DDR controller 12, where the processor 10 needs to establish a mapping relationship corresponding table and compress the mapping relationship corresponding table. The image processing circuit 20 includes a data source block loading unit 21, a parameter parsing unit 22, a bilinear interpolation unit 23, and an interpolation result output unit 24, and the image processing circuit 20 reads data from the DDR memory 11 through the DDR controller 12, such as data of an input image, and outputs data of an output image to the DDR memory 11 through the DDR controller 12.
The process of establishing the mapping relationship correspondence table between the output image and the input image is described below with reference to fig. 3 and 4. According to the imaging principle of the fisheye lens, the fisheye lens can be approximately regarded as a hemisphere model, such as the hemisphere model shown in the upper half of fig. 3. One point in the world coordinate system is projected to a camera coordinate system through a rotation matrix R and a translation matrix T, and since light rays are refracted through a plurality of groups of lenses of the fisheye lens to form an image plane, a projection model from the camera coordinate system to the image plane is a nonlinear transformation process, and pixel points in the image plane can be transformed to the image coordinate system according to an internal reference matrix A of the camera through calculation of a multi-order formula. According to an imaging model of the fisheye lens, the method mainly adopts a known OpenCV or OCamCalib calibration tool to calibrate parameters of the camera, and obtains parameters of a rotation matrix R, a translation matrix T, an internal reference matrix A, a distortion coefficient and the like of the camera.
In order to enable the fisheye image with serious radial distortion to be more suitable for human eyes to watch, the method adopts a scheme of equal-rectangular panoramic expansion to correct the input image. Fisheye lens imaging is approximately regarded as a hemispherical model, and when an image formed by the fisheye lens imaging is unfolded to a rectangular plane, the image can be mapped according to the longitude and latitude of a spherical surface. According to the mapping relation between the fisheye image of the unit sphere and the rectangular expansion surface, each point (xd, yd) on the output image can be projected to a 3D spherical surface according to the mapping relation S to obtain 3D coordinates (X, Y, Z), the 3D coordinates are converted to a camera coordinate system (X ', Y ', Z ') according to a rotation matrix R and a translation matrix T, the position (X, Y) projected on the camera imaging plane can be obtained according to a projection model F, finally, the projection point (xs, ys) of the fisheye image in the fisheye image is calculated through a camera internal reference matrix A, and the process of establishing the mapping relation between the output image and the pixel point of the input image in the fisheye image correction process is shown in FIG. 4. Chinese patent application CN201511019998.0 discloses a calculation process of a mapping relationship between pixels of an output image and pixels of an input image, which is not described herein again.
In this embodiment, after determining the parameters of the shooting device, the processor 10 may determine a mapping relationship between each pixel point of the output image and a pixel point of the input image according to an existing calibration tool, and form a mapping relationship correspondence table according to the mapping relationship, for example, the mapping relationship correspondence table records row coordinates of which pixel points in the input image correspond to row coordinates of which pixel points in the output image, and records column coordinates of which pixel points in the input image correspond to each pixel point column coordinate of the output image.
Since the data of the mapping table needs to be sent to the image processing circuit 20, in order to reduce the amount of data sent to the image processing circuit 20, the data of the mapping table needs to be compressed in the present embodiment. For example, data compression is performed by equidistant sampling of coordinates in the mapping relationship correspondence table, for example, data of row coordinates and column coordinates corresponding to pixels of sampling points are calculated by a method of calculating an average value, and preferably, distances between two adjacent sampling points are equal. The compressed data does not include data except for the sampling points, so that after the image processing circuit receives the data of the mapping relation corresponding table, decompression needs to be performed, that is, the data except for the sampling points needs to be completely supplemented in an interpolation mode, so that the data of the original mapping relation corresponding table is obtained. In order to ensure that the data obtained by decompression is not distorted seriously, the sampling distance is not too long, for example, a sampling point should be set at intervals of three or five pixel points.
As the number of pixels of the photographing device is increased, the number of pixels of the output image is increased, and if the image processing circuit 20 processes data of a whole output image at a time, the image processing circuit 20 needs to have a large area, and for this reason, the output image needs to be partitioned, that is, step S3 is executed. As shown in fig. 5, the output image is divided into a plurality of image blocks, such as block _00, block _01 … block _ mn, etc., and preferably, the size of each image block is equal, i.e., the length and height of each image block are equal.
After the output image is divided into blocks, the image processing circuit 20 can calculate only the data of one image block at a time, so the image processing circuit 20 has a small amount of calculation at a time and the circuit area can be reduced. Since the image processing circuit 20 only needs to process the data calculation of one image block each time, the data of the mapping relationship correspondence table sent to the image processing circuit 20 each time may also only include the data of the corresponding image block, that is, the mapping relationship correspondence table needs to be split, and the data of the corresponding mapping relationship correspondence table in each image block is obtained.
Of course, since the amount of data in the mapping table is generally small, in order to avoid transmitting data from the processor 10 to the image processing circuit 20 multiple times, data in the mapping table corresponding to a plurality of image blocks may be transmitted to the image processing circuit 20 at one time. For example, data of the mapping table of a plurality of adjacent image blocks is transmitted at a time. Therefore, after the analysis of the plurality of image blocks corresponding to the current group of mapping relationship corresponding tables is finished, the data of the next group of mapping relationship corresponding tables is obtained.
As shown in fig. 6, the image processing circuit 20 processes data of one image block at a time, and therefore, in this embodiment, a data table LUT corresponding to each image block is provided, and each data table LUT includes description information of the source data block ROI and data of a mapping relationship corresponding table corresponding to the image block. The description information of the source data block ROI includes source data block information and valid flag bits required for interpolating and reconstructing each output image data block during image correction. For example, after determining the boundary of an image block of an output image, the mapping relationship correspondence table may be applied to determine the boundary of the image block corresponding to a pixel point in the input image according to the boundary information of the image block, and an image block located within the boundary of the pixel point in the input image is a source data block ROI corresponding to the image block, so that each image block has its own corresponding source data block.
After the output image is divided into a plurality of image blocks, and the source data block corresponding to each image block and the data in the mapping table are determined, step S4 is executed to determine the currently processed image block, for example, which image block needs to be processed currently, and then step S5 is executed, the image processing circuit loads the data of the source data block corresponding to the currently processed image block, at this time, the parameter analyzing unit 22 in the image processing circuit 20 will receive the data of the source data block of the currently processed image block and the data in the mapping table.
When the parameter analyzing unit 22 loads the data of the source data block and the data of the mapping relationship table of the currently processed image block, the steps shown in fig. 7 are executed, first, step S11 is executed to load the data of the mapping relationship table and the data of the source data block, and then step S12 is executed to decompress the data of the mapping relationship table. Since the data in the mapping relationship correspondence table is compressed in step S2, step S12 requires a process of decompressing the compressed data. For example, in decompression, the edge coordinate values of each image block are calculated by using a line interpolation and a column interpolation method, and the coordinate values of the middle area in the image block are calculated by using a bilinear interpolation method, so that the data of the mapping relationship correspondence table which is originally calculated is obtained.
Then, step S13 is executed to obtain interpolation guide information, such as boundary information of the source data block for interpolation calculation, and determine the interpolation calculation mode according to the boundary of the source data block, such as calculation by weighted average, weighted filter, etc., or other calculation modes, and obtain parameters such as weighting coefficients.
After step S11 is executed, step S14 is also executed to parse the data of the source data block, for example, to obtain the source data block location information required to guide bilinear interpolation calculation, and then step S15 is executed to obtain the loading guide information of the source data block, for example, the obtained parameters of the source data block, and extract the data including the current source data block re-partitioning flag, the current source data block or the re-partitioning source data block location information, so as to guide the loading of the current source data block data source.
Although the size of each image block is determined, the size of the source data block corresponding to each image block is uncertain, and if the data of the source data block corresponding to the currently processed image block is too large, the amount of data of the source data block that needs to be received by the image processing circuit is also very large. Since the buffer space in the image processing circuit 20 is often limited, if the data size of the source data block is too large, the buffer in the image processing circuit 20 cannot store all the data of the source data block corresponding to one image block.
Therefore, in this embodiment, it is necessary to determine whether the data amount of the source data block is too large, and if the data amount of the source data block is too large, the currently processed image block needs to be partitioned again, for example, one image block is divided into a plurality of image sub-blocks, so that the area of the source data block corresponding to each image sub-block is greatly reduced, and the corresponding data amount is also reduced. Therefore, the loading guidance information of the source data block obtained in step S15 includes the current source data block re-partitioning flag, that is, if the data amount of the current source data block is greater than the preset threshold, the re-partitioning flag will be generated, and at this time, the image processing circuit 20 loads data of the source data block corresponding to one image sub-block instead of data of the source data block of one image block each time.
Therefore, before the image processing circuit 20 loads the source data block, the processor needs to determine how many sets of boundary information are output to the image processing circuit, where each set of boundary information corresponds to the boundary of the source data block that is sent to the image processing circuit at a time. Referring to fig. 8, first, step S21 is executed to determine whether the data amount of the source data block is greater than a threshold, which in this embodiment is a preset threshold that can be set according to the size of the buffer of the image processing circuit 20. If the data size of the source data block is greater than the threshold, which indicates that the source data block corresponding to the currently parsed image block cannot be read into the buffer of the image processing circuit 20 at one time, step S23 is executed to output boundary information of four sets of source data blocks to the image processing circuit 20, where the four sets of source data blocks correspond to four image sub-blocks. If the data size of the source data block is not greater than the threshold, step S22 is executed to output the boundary information of a set of source data blocks, that is, the boundary of the source data block corresponding to the currently parsed image block, to the image processing circuit 20.
Therefore, by re-blocking the image block, whether the current source data block is re-blocked can be adaptively determined according to the size of the source data block corresponding to the currently analyzed image block, and the overlarge source data block is loaded in a multi-time carrying manner, so that the buffer resources are saved. It should be noted that the partitioning of the image block is not limited to four image sub-blocks, but may be divided into two, three, or eight image sub-blocks, and the size of the source data block may be adjusted accordingly.
In this embodiment, the data source block loading unit 21 in the image processing circuit 20 is configured to implement data loading of the source data block, that is, execute step S5. Because the fisheye image has serious distortion, the probability that the source data blocks corresponding to two adjacent image blocks or image sub-blocks of the output image have overlapping regions is high. Therefore, before the data source block loading unit 21 obtains the current source data block parameter, it needs to analyze whether the source data block corresponding to the currently analyzed image block has an overlapping area with the source data block area corresponding to the previous image block, if so, it needs to eliminate the overlapping area redundant data of the source data block corresponding to the current image block, and then loads the non-overlapping area source data block data to the image processing circuit, for example, loads data through the dynamic random access memory DDR.
As shown in fig. 9, the data source block loading unit 21 first executes step S31 to determine whether there is an overlapping area between the source data block that needs to be currently received and the previous source data block. Referring to fig. 10, there are many situations where the source data blocks corresponding to two adjacent images have overlapping areas, and fig. 10 shows several common situations. In fig. 10, under various conditions, A, B two points are respectively vertices of an upper left corner and a lower right corner of the current source data block image, C, D two points are vertices of an upper left corner and a lower right corner of the previous source data block image, as can be seen from fig. 10, in the former five conditions, an overlapping region exists between the region of the previous source data block and the region of the current source data block, for example, E, F two points under various conditions in fig. 10 are respectively vertices of an upper left corner and a lower right corner of the overlapping region image.
If it is determined that there is no overlapping area between the current source data block and the source data block corresponding to the adjacent image block, such as the last case shown in fig. 10, step S32 is executed to load the data of the current source data block to the image processing circuit 20. If there is an overlapping area between the current source data block and the source data block corresponding to the adjacent image block, step S33 is executed to eliminate redundant data between two source data blocks, i.e. to eliminate data outside the overlapping area, and then to load the data outside the overlapping area to the image processing circuit 20.
It can be seen that the amount of data read from the DDR memory to the buffer of the image processing circuit 20 can be reduced by clearing the redundant data of the overlap region, and the inventors have found that by actually processing the fisheye image, the data loading of most of the overlap region can be reduced by eliminating the redundant data, and the bandwidth of 55% of the bandwidth used between the DDR memory and the image processing circuit 20 can be reduced.
After the source data block of the input image is loaded, step S6 is executed to perform interpolation calculation on the image data of the source data block, for example, the bilinear interpolation unit 23 of the image processing circuit 20 performs interpolation calculation on the image data of the current source data block, as the steps shown in fig. 11 are executed. First, step S41 is executed to extract interpolation position information and acquire an interpolation data source. Because the image data of the source data block contains a large number of pixels, the interpolation calculation is to perform corresponding calculation on the chromaticity, the brightness value and the like of the pixel point of the input image according to the corresponding relation between the pixel point of the output image and the pixel point of the input image on the mapping relation corresponding table, so as to obtain the chromaticity and the brightness value of the pixel point corresponding to the output image. Therefore, step S41 is to obtain the interpolation position information from the interpolation guide information of step S13, and obtain the interpolation data source, that is, obtain the data of the pixel at the corresponding position in the source data block.
Then, step S42 is executed to acquire interpolation parameters and perform bilinear interpolation calculation. For example, the present embodiment completes the bidirectional interpolation by means of weighted filtering, and therefore, parameters of the weighted filtering, such as parameters of a filter matrix, weighting coefficients, and the like, need to be obtained during the interpolation calculation, and the bilinear interpolation calculation is performed immediately after the interpolation parameters are obtained. Finally, step S43 is executed to buffer the interpolation result, for example, the interpolation result is output to the buffer of the interpolation result for the interpolation result output unit 24 to use.
Finally, the output of the interpolation result is completed by the interpolation result output unit 24, and as shown in fig. 12, the interpolation result output unit 24 first acquires the result of the interpolation calculation from the bilinear interpolation unit 23, i.e., performs step S51, and then performs step S52, acquires the format of the output image, and rearranges the interpolation result according to the format of the output image. For example, the format of the acquired output image may be an output mode for acquiring the current image, such as acquiring information of YUV, YVU, NUV, NVU, or rotation, and accordingly rearranging the interpolation buffer result to output the interpolation result to the DDR memory 11 in a mode of utilizing the current DDR bus efficiency to the maximum.
Then, step S53 is executed to form data of an output image from the rearranged interpolation result data and output the data of the output image to the DDR memory 11, for example, the image processing circuit 20 outputs the output image data to the DDR controller 12 and then to the DDR memory 11 by the DDR controller 12.
After the current image block is processed, executing step S7, determining whether the current processed image block is the last image block, if so, executing step S8, outputting the data of the current processed image block to the DDR memory 11, and then ending, otherwise, executing step S9, loading the data of the source data block corresponding to the next image block, and returning to step S6, performing interpolation calculation on the current image block until all image blocks are calculated.
Therefore, the correction processing of the fisheye image is mainly realized by the following three steps: 1. establishing an imaging model for the fisheye camera, and calibrating parameters of the fisheye camera according to the imaging model; 2. establishing a backward mapping relation between an output image and an input image according to an equal-rectangular panoramic expansion mode, and generating a coordinate mapping table and a block information table supporting hardware inquiry through a software program; 3. and starting a hardware processing unit to complete a series of image reconstruction operations such as bidirectional interpolation and the like according to the software configuration information, the coordinate mapping table and the block information table, and outputting a final image.
In the present invention, the above steps 1 and 2 are implemented by software program, that is, the processor 10 executes the software program, and the step 3 is implemented by hardware, for example, by using the image processing circuit 20. It can be understood that the image processing circuit 20 of the present invention is an integrated circuit for implementing an image interpolation calculation function for fisheye image correction, and the fisheye image is processed by combining a software program with a hardware circuit, and the interpolation calculation speed of the hardware circuit is much faster than that of the software program, so that the present invention can avoid the problem of slow image output caused by implementing interpolation calculation purely through the software program.
In addition, the invention divides the output image into blocks, and the input image is correspondingly divided into a plurality of source data blocks, thus avoiding the repeated request operation of DDR memory when the data of the input image is loaded according to the line in the past, and having smaller data cache resources because the line cache unit is not needed. In addition, the invention adopts the redundant data elimination technology of the source data blocks corresponding to the adjacent image blocks, and reduces the bandwidth occupied by the input image when being loaded by about 55 percent.
In addition, the invention can reduce the data amount sent to the image processing circuit by the processor and also can reduce the bandwidth occupied by data transmission by compressing and transmitting the data of the mapping relation corresponding table. Because the calculation process of the mapping relation corresponding table is complex, the mapping relation corresponding table is calculated through a software program and transmitted to the image processing circuit, the complexity of the image processing circuit in the image processing process is reduced, and the hardware implementation is more convenient.
Of course, the above-mentioned solution is only a preferred embodiment of the present invention, and the practical application may also have more variations, for example, the mapping relationship correspondence table may be calculated by using other calibration tools, or the image block may be partitioned by using a fixed image size for the sub-partitioning, and these variations do not affect the implementation of the present invention, and should also be included in the protection scope of the present invention.
Claims (8)
1. The fisheye image processing method comprises the following steps:
the processor determines the mapping relation between each pixel point in the output image and the pixel point in the input image to form a mapping relation corresponding table, and sends the mapping relation corresponding table to the image processing circuit;
the processor sends data of an input image to the image processing circuit, divides an output image into a plurality of image blocks, and determines that each image block corresponds to a source data block in the input image;
the method is characterized in that:
after receiving the data of the input image, the image processing circuit performs interpolation calculation on the data of the input image according to the mapping relation of the mapping relation corresponding table to obtain the data of each pixel point of the output image, the image processing circuit calculates the data of the pixel point of an image block each time, the image processing circuit sets a data table corresponding to each image block, and each data table comprises the description information of the source data block and the data of the mapping relation corresponding table corresponding to the image block;
wherein the image processing circuitry is independent of the processor;
sending the mapping relationship correspondence table to an image processing circuit includes: sending data of a mapping relation corresponding table corresponding to the image block to be processed;
transmitting data of the input image to the image processing circuit includes: and sending the data of the source data block corresponding to the image block to be processed in the input image to the image processing circuit.
2. The fisheye image processing method according to claim 1, characterized in that:
before sending the mapping relation corresponding table to the image processing circuit, compressing the data of the mapping relation corresponding table, and sending the compressed data of the mapping relation corresponding table to the image processing circuit;
and after receiving the data of the compressed mapping relation corresponding table, the image processing circuit decompresses the received data.
3. The fisheye image processing method according to claim 2, characterized in that:
compressing the data of the mapping relation correspondence table includes: and carrying out equidistant sampling compression on the data of the mapping relation corresponding table.
4. The fisheye image processing method according to claim 3, characterized in that:
the decompressing calculation of the compressed data of the mapping relation corresponding table by the image processing circuit comprises: and performing row-column interpolation calculation on the coordinate values of the edges of the image blocks, and performing bilinear interpolation calculation on the coordinate values of the middle area of the image blocks.
5. The fisheye image processing method according to any one of claims 1 to 4, characterized in that:
sending the data of the source data block corresponding to the image block to be processed in the input image to the image processing circuit includes: before sending the source data block data of the current image block to be processed, judging whether an overlapping area exists between the source data block corresponding to the previous adjacent image block and the current source data block, and if the overlapping area exists, sending data outside the overlapping area to the image processing circuit.
6. The fisheye image processing method according to any one of claims 1 to 4, characterized in that:
sending data of a source data block corresponding to an image block to be processed in an input image to the image processing circuit includes:
judging whether the data volume of the current source data block is larger than a preset threshold value, if so, dividing the current calculated image block into a plurality of image sub-blocks, determining the source data block corresponding to each image sub-block, and sending a part of data of the source data block corresponding to the image sub-block to the image processing circuit each time.
7. The fisheye image processing method according to any one of claims 1 to 4, characterized in that:
the sending of the data of the mapping relationship correspondence table corresponding to the image block to be processed includes: and transmitting the data of the mapping relation corresponding table corresponding to more than two image blocks to the image processing circuit at one time.
8. Fisheye image processing apparatus comprising a processor and an image processing circuit independent of each other, wherein the processor is adapted to realize, when executing a computer program:
determining the mapping relation between each pixel point in the output image and the pixel point in the input image to form a mapping relation corresponding table, and sending the mapping relation corresponding table to an image processing circuit;
sending data of an input image to the image processing circuit, dividing an output image into a plurality of image blocks, and determining that each image block corresponds to a source data block in the input image;
the method is characterized in that:
the image processing circuit is to: after receiving the data of the input image transmitted by the processor, performing interpolation calculation on the data of the input image according to the mapping relation of the mapping relation corresponding table, calculating the data of each pixel point of the output image, and calculating the data of the pixel point of an image block each time, wherein the image processing circuit sets a data table corresponding to each image block, and each data table comprises the description information of the source data block and the data of the mapping relation corresponding table corresponding to the image block;
wherein sending the mapping relationship correspondence table to an image processing circuit comprises: sending data of a mapping relation corresponding table corresponding to the image block to be processed;
transmitting data of the input image to the image processing circuit includes: and sending the data of the source data block corresponding to the image block to be processed in the input image to the image processing circuit.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810381677.2A CN108564551B (en) | 2018-04-25 | 2018-04-25 | Fisheye image processing method and fisheye image processing device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810381677.2A CN108564551B (en) | 2018-04-25 | 2018-04-25 | Fisheye image processing method and fisheye image processing device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108564551A CN108564551A (en) | 2018-09-21 |
| CN108564551B true CN108564551B (en) | 2021-01-15 |
Family
ID=63536757
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810381677.2A Active CN108564551B (en) | 2018-04-25 | 2018-04-25 | Fisheye image processing method and fisheye image processing device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108564551B (en) |
Families Citing this family (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109886866B (en) * | 2019-01-15 | 2023-06-13 | 珠海亿智电子科技有限公司 | Self-adaptive affine transformation processing method and device |
| CN109785265B (en) * | 2019-01-16 | 2022-11-11 | 西安全志科技有限公司 | Distortion correction image processing method and image processing apparatus |
| CN110264395B (en) * | 2019-05-20 | 2023-11-28 | 深圳市森国科科技股份有限公司 | Lens calibration method and related device of vehicle-mounted monocular panoramic system |
| CN110570367A (en) * | 2019-08-21 | 2019-12-13 | 苏州科达科技股份有限公司 | Fisheye image correction method, electronic device and storage medium |
| CN110738615B (en) * | 2019-10-11 | 2023-09-19 | 珠海市杰理科技股份有限公司 | Fisheye image correction method, device, system and storage medium |
| CN111429382B (en) * | 2020-04-10 | 2024-01-19 | 浙江大华技术股份有限公司 | Panoramic image correction method and device and computer storage medium |
| CN113538237A (en) * | 2021-07-09 | 2021-10-22 | 北京超星未来科技有限公司 | Image splicing system and method and electronic equipment |
| CN115187448A (en) * | 2022-06-15 | 2022-10-14 | 奥比中光科技集团股份有限公司 | Chip-based image affine transformation method and chip |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2010051147A2 (en) * | 2008-10-31 | 2010-05-06 | General Instrument Corporation | Method and apparatus for transforming a non-linear lens-distorted image |
| CN103456031A (en) * | 2013-07-08 | 2013-12-18 | 江南大学 | Novel method for area image interpolation |
| CN105744137A (en) * | 2016-04-22 | 2016-07-06 | 王俊懿 | System for real-time correction and splicing of image formed by fish-eye lens through using hardware circuit |
| CN106204441A (en) * | 2016-06-27 | 2016-12-07 | Tcl集团股份有限公司 | The method and device that a kind of image local amplifies |
| CN106651808A (en) * | 2016-12-29 | 2017-05-10 | 北京爱奇艺科技有限公司 | Fisheye image conversion method and device |
| WO2017124759A1 (en) * | 2016-01-18 | 2017-07-27 | 深圳岚锋创视网络科技有限公司 | Method and device for rectifying image photographed by fish-eye lens |
| CN107220930A (en) * | 2017-08-02 | 2017-09-29 | 珠海全志科技股份有限公司 | Fish eye images processing method, computer installation and computer-readable recording medium |
-
2018
- 2018-04-25 CN CN201810381677.2A patent/CN108564551B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2010051147A2 (en) * | 2008-10-31 | 2010-05-06 | General Instrument Corporation | Method and apparatus for transforming a non-linear lens-distorted image |
| CN103456031A (en) * | 2013-07-08 | 2013-12-18 | 江南大学 | Novel method for area image interpolation |
| WO2017124759A1 (en) * | 2016-01-18 | 2017-07-27 | 深圳岚锋创视网络科技有限公司 | Method and device for rectifying image photographed by fish-eye lens |
| CN105744137A (en) * | 2016-04-22 | 2016-07-06 | 王俊懿 | System for real-time correction and splicing of image formed by fish-eye lens through using hardware circuit |
| CN106204441A (en) * | 2016-06-27 | 2016-12-07 | Tcl集团股份有限公司 | The method and device that a kind of image local amplifies |
| CN106651808A (en) * | 2016-12-29 | 2017-05-10 | 北京爱奇艺科技有限公司 | Fisheye image conversion method and device |
| CN107220930A (en) * | 2017-08-02 | 2017-09-29 | 珠海全志科技股份有限公司 | Fish eye images processing method, computer installation and computer-readable recording medium |
Non-Patent Citations (2)
| Title |
|---|
| 基于SDRAM缓存的实时视频图像几何校正***;陈文艺，等.;《电视技术》;20160517;第40卷(第5期);第105-106页 * |
| 陈文艺，等..基于SDRAM缓存的实时视频图像几何校正***.《电视技术》.2016,第40卷(第5期),第105-109页. * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108564551A (en) | 2018-09-21 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108564551B (en) | Fisheye image processing method and fisheye image processing device | |
| US11012620B2 (en) | Panoramic image generation method and device | |
| CN109194923B (en) | Video image processing device, system and method based on local non-uniform resolution | |
| US9262807B2 (en) | Method and system for correcting a distorted input image | |
| US12020401B2 (en) | Data processing systems | |
| JP4960992B2 (en) | Image processing method and image processing apparatus for fisheye correction and perspective distortion reduction | |
| CN111107333B (en) | Brightness correction method, system, equipment and computer readable storage medium | |
| US20220301121A1 (en) | Method and apparatus for correcting face distortion, electronic device, and storage medium | |
| US11582402B2 (en) | Image processing device | |
| CN107220930B (en) | Fisheye image processing method, computer device and computer readable storage medium | |
| CN110958362A (en) | Image correction system and method based on block table look-up | |
| US20210118104A1 (en) | Image processing | |
| CN111951193B (en) | Horizontal distortion correction method and horizontal distortion correction device for image | |
| US9269130B2 (en) | Image correction method using approximately non-linear regression approach and related image correction circuit | |
| WO2023024421A1 (en) | Method and system for splicing multiple channels of images, and readable storage medium and unmanned vehicle | |
| CN111213179A (en) | Image data processing method, device and system | |
| US20180286012A1 (en) | Image processing | |
| WO2023280266A1 (en) | Fisheye image compression method, fisheye video stream compression method and panoramic video generation method | |
| CN110533577B (en) | Fisheye image correction method and device | |
| CN110738615B (en) | Fisheye image correction method, device, system and storage medium | |
| WO2015098721A1 (en) | Image processing device, image processing method, image processing program, and imaging device | |
| CN113824894A (en) | Exposure control method, device, equipment and storage medium | |
| CN113538237A (en) | Image splicing system and method and electronic equipment | |
| CN115623217B (en) | Image preprocessing method, device and system | |
| KR102655332B1 (en) | Device and method for image correction |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |