WO2022127841A1 - 一种图像重映射方法以及图像处理装置 - Google Patents

一种图像重映射方法以及图像处理装置 Download PDF

Info

Publication number
WO2022127841A1
WO2022127841A1 PCT/CN2021/138619 CN2021138619W WO2022127841A1 WO 2022127841 A1 WO2022127841 A1 WO 2022127841A1 CN 2021138619 W CN2021138619 W CN 2021138619W WO 2022127841 A1 WO2022127841 A1 WO 2022127841A1
Authority
WO
WIPO (PCT)
Prior art keywords
pixel
image
row
column
pixels
Prior art date
Application number
PCT/CN2021/138619
Other languages
English (en)
French (fr)
Inventor
宋碧薇
陈纾悦
卢庆聪
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Publication of WO2022127841A1 publication Critical patent/WO2022127841A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/80—Geometric correction
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/04—Context-preserving transformations, e.g. by using an importance map

Definitions

  • the embodiments of the present application relate to the technical field of image processing, and in particular, to an image remapping method and an image processing apparatus.
  • Distortion is a common aberration in imaging systems.
  • the image presented on the imaging plane will be a distorted image (ie, a distorted image) compared to the original image, rather than the aforementioned original image.
  • the pre-distortion method is often used to convert the original image into a pre-distorted image that can cancel the distortion.
  • the processor calculates the pixel value of each pixel in the pre-distorted image based on the original image
  • the processor needs to find the A' point corresponding to the A pixel in the pre-distorted image from the original image, and assign the A' point to the original image.
  • the pixel value of is assigned to the A pixel in the predistorted image.
  • the pre-distorted image has different degrees of distortion in both the row and column directions compared to the original image, the horizontal and vertical coordinates of the A' point are not necessarily integers. Therefore, the processor needs to use an interpolation algorithm according to the A' point.
  • a number of pixels near the 'point' computes the pixel value of point A'. This loops until the processor calculates the pixel value for each pixel in the predistorted image. Then, after the processor inputs the pre-distorted image to the display device of the projection imaging system, the user can observe the original image obtained by distorting the pre-distorted image through the imaging plane.
  • the display driver in the traditional technology requires the image to be input into the driver in row order, and the pixels in each row of the sequence in the pre-distortion image correspond to the pixels of the original image are generally out of order, so the process of calculating the pre-distortion image involves A large number of out-of-order accesses to the original image content will affect the processing efficiency of the processor. Therefore, there is an urgent need for a solution that can improve the efficiency of processing raw images.
  • Embodiments of the present application provide an image remapping method and an image processing apparatus, which are used to improve the efficiency of generating a predistorted image based on an original image.
  • the present application provides an image remapping method, which can be applied to an imaging scene based on a projection imaging system.
  • the image processing apparatus first performs row conversion processing and then performs column conversion processing. Specifically, the image processing apparatus acquires each row of pixels of the original image, and acquires a row transformation parameter corresponding to each row of pixels.
  • the row transformation parameter is used to indicate the correspondence between the abscissa of the pixel in the original image and the abscissa of the pixel in the first image.
  • the image processing device determines the pixel value of each pixel in the first image according to each line of pixels in the original image and the line transformation parameter corresponding to each line of pixels.
  • the image processing apparatus obtains an image with pre-distortion in the row direction and no pre-distortion in the column direction (ie, the aforementioned first image). Then, the image processing apparatus will acquire each column of pixels of the first image, and acquire the column transformation parameters corresponding to each column of pixels.
  • the column transformation parameter is used to indicate the correspondence between the vertical coordinates of the pixels in the first image and the vertical coordinates of the pixels in the target image. Then, the image processing device determines the pixel value of each pixel in the target image according to each column of pixels in the first image and the column transformation parameter corresponding to each column of pixels.
  • the image processing device obtains an image with pre-distortion in the row direction and pre-distortion in the column direction (ie, the target image), and the target image is used to cancel the imaging distortion of the imaging device to display the original image.
  • the image processing apparatus when the image processing apparatus acquires each row of pixels of the original image, it can be specifically understood that the image processing apparatus reads the pixel value of each pixel in each row of pixels and the coordinates of the pixel in the original image. Among them, the coordinates of the pixel in the original image include abscissa and ordinate.
  • the image processing apparatus acquires a row transformation parameter corresponding to this row of pixels. Then, the image processing apparatus can calculate a row of pixels in the first image according to the aforementioned row of pixels and the row transformation parameters corresponding to this row of pixels. By analogy, the image processing apparatus will sequentially calculate each row of pixels in the first image.
  • the process from the original image to the first image can be referred to as a line transformation process.
  • the image processing device acquires each column of pixels in the first image
  • the image processing device reads the pixel value of each pixel in each column of pixels and the coordinates of the pixel in the first image.
  • the coordinates of the pixel in the first image include abscissa and ordinate.
  • the image processing device correspondingly acquires the column transformation parameters corresponding to this column of pixels.
  • the image processing apparatus can calculate a row of pixels in the target image according to the aforementioned row of pixels and the column transformation parameters corresponding to this row of pixels.
  • the image processing device will sequentially calculate each column of pixels in the target image.
  • the process from the first image to the target image can be referred to as a column transformation process.
  • the image processing apparatus has performed pre-distortion processing on the original image in the row direction and column direction respectively (ie the row transformation and column transformation), and the target image is an image that can cancel the imaging distortion of the projection imaging system. .
  • the image processing apparatus determines the target image (ie, the pre-distorted image) based on the original image
  • the image remapping is decomposed into two unidirectional mappings. Specifically, the image processing device first performs row transformation on each row of pixels in the original image, and then performs column transformation on each column of pixels in the original image based on the row transformation result (ie, the first image) to obtain the target image.
  • the image processing device when calculating a row of pixels in the first image, the image processing device only needs to sequentially read a row of pixels in the original image, and does not need to read out of order across rows; similarly, when the image processing device calculates the target image When there is one column of pixels in the first image, only one column of pixels in the first image needs to be read sequentially, and it is not necessary to read out of sequence across columns. Therefore, the image processing apparatus can read in order when processing the original image, thereby improving the efficiency of processing the original image by the image processing apparatus.
  • the row transformation parameter includes the abscissa of the first pixel value point corresponding to the first pixel in the original image, and the first pixel is any pixel in the first image , the first pixel value point is used to determine the pixel value of the first pixel, and the ordinate of the first pixel value point in the original image is the same as the ordinate of the first pixel in the first image.
  • the row transformation parameter can be represented by the abscissa of the first pixel value point in the original image, and the first pixel value point is the point corresponding to the first pixel in the first image to be determined, It can be understood that after the first pixel value point is distorted in the row direction, it falls to the position corresponding to the aforementioned first pixel. Therefore, when calculating the pixel value of the first pixel, it is necessary to find the value point of the first pixel in the original image. The pixel value of the location.
  • the vertical coordinate of the value point of the first pixel in the original image is the same as the vertical coordinate of the first pixel in the first image.
  • the coordinates are the same. Therefore, when the aforementioned image processing apparatus calculates the first pixel of a row in the first image, it only needs to acquire a row of pixels in the original image row by row.
  • the column transformation parameter includes the ordinate of the value point of the second pixel corresponding to the second pixel in the first image, and the second pixel is any one of the target images.
  • the second pixel value point is used to determine the pixel value of the second pixel
  • the abscissa of the second pixel value point in the first image is the same as the abscissa of the second pixel in the target image .
  • the column transformation parameter can be represented by the ordinate of the second pixel value point in the first image, and the second pixel value point is the point corresponding to the second pixel in the target image to be obtained, It can be understood that after the second pixel value point is distorted in the column direction, it falls to the position corresponding to the aforementioned second pixel. Therefore, when calculating the pixel value of the second pixel, it is necessary to find the value point of the second pixel in the first image. The pixel value of the position in .
  • the abscissa of the value point of the second pixel in the first image is the same as the horizontal coordinate of the second pixel in the target image.
  • the coordinates are the same. Therefore, when the aforementioned image processing apparatus calculates a column of second pixels in a target image, it only needs to acquire a column of pixels in the first image column by column.
  • the image processor in the traditional technology only involves the mapping parameters from the original image to the target image in the image processing process, and needs to remap both the horizontal and the column in one mapping process. Therefore, when calculating a pixel in the target image, the image processor is required to be able to read at least four pixels in a rectangular distribution from the original image. Since the distortion degree in the row direction is different from the distortion degree in the column direction, the image processor needs to acquire pixels out of sequence in the original image when calculating the value of each pixel in the target image one by one.
  • the solution of the present application only needs to acquire pixels row by row from the original image, and acquire pixels column by column from the first image. Therefore, the solution proposed in this application can improve the image processing efficiency of the image processing apparatus.
  • the aforementioned row transformation parameter is stored in a row mapping relationship table, and the position of the row transformation parameter in the row mapping relationship table is used to indicate the coordinates of a first pixel, where the first pixel is the pixel corresponding to the aforementioned row transform parameter. It can also be understood that the position of the row transformation parameter in the row mapping relationship table is used to indicate the coordinates of the first pixel determined based on the row transformation parameter.
  • the column transformation parameter is stored in the column mapping relationship table, and the position of the column transformation parameter in the column mapping relationship table is used to indicate the coordinates of the second pixel, wherein the second pixel corresponds to the aforementioned column transformation parameter pixel. It can also be understood that the position of the column transformation parameter in the column mapping relationship table is used to represent the coordinates of the second pixel determined based on the column transformation parameter.
  • the image processing apparatus determines the pixel value of each pixel in the first image according to each line of pixels in the original image and the line transformation parameter corresponding to each line of pixels, including: according to the The row transformation parameter determines the value point of the first pixel corresponding to the first pixel; at least one third pixel is determined according to the abscissa of the value point of the first pixel, and the third pixel is located at the value point of the first pixel. The same row; the pixel value of the first pixel is determined according to the pixel value of the at least one third pixel.
  • the image processing device determines the pixel value of each pixel in the target image according to each column of pixels in the first image and the column transformation parameter corresponding to each column of pixels, including: determining the second pixel according to the column transformation parameter. The corresponding value point of the second pixel; at least one fourth pixel is determined according to the ordinate of the value point of the second pixel, and the fourth pixel and the value point of the second pixel are located in the same column; according to the at least one fourth pixel The pixel value of the pixel determines the pixel value of the second pixel.
  • the abscissa of the first pixel value point is an integer.
  • the determining at least one third pixel according to the abscissa of the value point of the first pixel includes: determining the value point of the first pixel as the third pixel.
  • the determining of the pixel value of the first pixel according to the pixel value of the at least one third pixel includes: assigning the pixel value of the third pixel to the first pixel to obtain the pixel value of the first pixel.
  • the abscissa of the value point of the first pixel is a non-integer.
  • the determining at least one third pixel according to the abscissa of the value point of the first pixel includes: determining at least one pixel adjacent to the value point of the first pixel as the third pixel, and the pixel adjacent to the value point of the first pixel is the third pixel. The pixel is located in the same row as the value point of the first pixel.
  • the determining the pixel value of the first pixel according to the pixel value of the at least one third pixel includes: using an interpolation algorithm to determine the pixel value of the at least one third pixel according to the abscissa of the value point of the first pixel and the abscissa of the at least one third pixel.
  • a first weighted average value is obtained from the pixel value of a third pixel; the first weighted average value is assigned to the first pixel to obtain the pixel value of the first pixel.
  • the aforementioned pixel adjacent to the value point of the first pixel can be understood as one pixel to the left and/or one pixel to the right of the value point of the first pixel; it can also be understood as the second pixel from the left and/or the right The second pixel; it can even be understood as two pixels on the left and/or two pixels on the right.
  • the aforementioned pixel adjacent to the value point of the first pixel can be understood as one pixel to the left and/or one pixel to the right of the value point of the first pixel; it can also be understood as the second pixel from the left and/or the right The second pixel; it can even be understood as two pixels on the left and/or two pixels on the right.
  • the ordinate of the second pixel value point is an integer.
  • the determining at least one fourth pixel according to the ordinate of the value point of the second pixel includes: determining the value point of the second pixel as the fourth pixel.
  • the determining the pixel value of the second pixel according to the pixel value of the at least one fourth pixel includes: assigning the pixel value of the fourth pixel to the second pixel to obtain the pixel value of the second pixel.
  • the ordinate of the second pixel value point is a non-integer.
  • the determining at least one fourth pixel according to the ordinate of the value point of the second pixel includes: determining at least one pixel adjacent to the value point of the second pixel as the fourth pixel, and the pixel adjacent to the value point of the second pixel is the fourth pixel.
  • the pixel is located in the same column as the second pixel value point.
  • the determining the pixel value of the second pixel according to the pixel value of the at least one fourth pixel includes: adopting an interpolation algorithm to determine the pixel value of the at least one fourth pixel according to the ordinate of the value point of the second pixel and the ordinate of the at least one fourth pixel.
  • a second weighted average value is obtained from the pixel value of a fourth pixel; the second weighted average value is assigned to the second pixel to obtain the pixel value of the second pixel.
  • the image processing apparatus acquiring the row transformation parameters corresponding to the pixels of each row includes: reading the row transformation parameters corresponding to the pixels of each row from a memory in order in the row direction, and the memory stores the row transformation parameters corresponding to the pixels of each row.
  • a row mapping relation table where the row mapping relation table includes the row transformation parameters corresponding to each row of pixels in the original image.
  • the image processing apparatus obtains the column transformation parameters corresponding to the pixels of each column, including: sequentially reading the column transformation parameters corresponding to the pixels of each column from the memory according to the column direction, and the memory stores a column mapping relationship table, the column mapping relationship table Including the column transformation parameters corresponding to each column of pixels in the original image.
  • mapping relationship tables are stored in the memory, including a row mapping relationship table from the original image to the first image (that is, only a row-transformed image), and a column mapping relationship from the first image to the target image surface.
  • the mapping relationship table from the original image to the target image is used. Therefore, each calculation of the pixel value of a pixel in the solution of the traditional technology needs to refer to the original image that is not exactly in the same row and not exactly in the same column. Interpolate multiple pixels.
  • the method further includes: image processing The device stores the first pixels of each row of the first image in memory.
  • the process for the image processing apparatus to acquire the pixels of each column of the first image may specifically be: the image processing apparatus sequentially reads the pixels of each column of the first image from the memory.
  • the aforementioned first image does not appear in the image processing process, but only the original image and the pre-distorted image (ie, the target image).
  • the image processing apparatus may send the target image to the imaging apparatus, and the imaging apparatus is configured to perform imaging based on the aforementioned target image, so that the aforementioned target image is imaged based on the aforementioned
  • the target image projects the aforementioned original image on the imaging plane.
  • the imaging device may be a projection imaging system.
  • the present application provides an image processing apparatus, which may be an image processing chip in a projection imaging system or a processor with an image processing function.
  • the image processing device includes: an acquisition module and a calculation module.
  • the acquiring module is used for acquiring each row of pixels of the original image, and acquiring the row transformation parameters corresponding to each row of pixels.
  • the row transformation parameter is used to indicate the correspondence between the abscissa of the pixel in the original image and the abscissa of the pixel in the first image, where the first image has pre-distortion in the row direction and no pre-distortion in the column direction image.
  • a calculation module configured to determine the pixel value of each pixel in the first image according to each row of pixels in the original image and the row transformation parameter corresponding to each row of pixels.
  • the obtaining module is further configured to obtain each column of pixels of the first image, and obtain column transformation parameters corresponding to each column of pixels.
  • the column transformation parameter is used to indicate the correspondence between the vertical coordinates of the pixels in the first image and the vertical coordinates of the pixels in the target image, where the target image is an image with pre-distortion in the row direction and pre-distortion in the column direction.
  • the calculation module is further configured to determine the pixel value of each pixel in the target image according to each column of pixels in the first image and the column transformation parameters corresponding to each column of pixels, and the target image is used to offset the imaging device. image distortion to reveal the original image.
  • the row transformation parameter includes the abscissa of the first pixel value point corresponding to the first pixel in the original image, and the first pixel is any pixel in the first image , the first pixel value point is used to determine the pixel value of the first pixel, and the ordinate of the first pixel value point in the original image is the same as the ordinate of the first pixel in the first image.
  • the column transformation parameter includes the ordinate of the second pixel value point corresponding to the second pixel in the first image, the second pixel is any pixel in the target image, and the second pixel value point is defined as For determining the pixel value of the second pixel, the abscissa of the value point of the second pixel in the first image is the same as the abscissa of the second pixel in the target image.
  • the row transformation parameter is stored in a row mapping relationship table, and the position of the row transformation parameter in the row mapping relationship table is used to represent the coordinates of the first pixel determined based on the row transformation parameter . It can also be understood that the position of the row transformation parameter in the row mapping relationship table is used to indicate the coordinates of the first pixel, where the first pixel is a pixel corresponding to the aforementioned row transformation parameter.
  • the column transformation parameter is stored in the column mapping relationship table, and the position of the column transformation parameter in the column mapping relationship table is used to indicate the coordinates of the second pixel, where the second pixel is a pixel corresponding to the aforementioned column transformation parameter.
  • the position of the column transformation parameter in the column mapping relationship table is used to represent the coordinates of the second pixel determined based on the column transformation parameter.
  • the calculation module is specifically configured to: determine the value point of the first pixel corresponding to the first pixel according to the row transformation parameter; and determine the value point of the first pixel according to the abscissa of the value point of the first pixel Determine at least one third pixel, the third pixel and the value point of the first pixel are located in the same row; determine the pixel value of the first pixel according to the pixel value of the at least one third pixel; determine the pixel value of the first pixel according to the column transformation parameter The value point of the second pixel corresponding to the second pixel; at least one fourth pixel is determined according to the ordinate of the value point of the second pixel, and the fourth pixel and the value point of the second pixel are located in the same column; The pixel value of a fourth pixel determines the pixel value of the second pixel.
  • the image processing apparatus further includes a storage module.
  • the storage module is used for storing the first pixels of each row of the first image into the memory.
  • the row mapping relationship table and the column mapping relationship table are stored in a memory
  • the row mapping relationship table includes the row transformation parameter corresponding to each row of pixels in the original image
  • the table includes column transformation parameters corresponding to each column of pixels in the original image.
  • the obtaining module is specifically used for: reading the row transformation parameters corresponding to the pixels of each row from the memory in the row direction sequence; and reading the column transformation parameters corresponding to the pixels of each column from the memory in the column direction sequence.
  • the image processing apparatus further includes a storage module.
  • the sending module is configured to send the target image to the imaging device, and the imaging device is configured to perform imaging based on the aforementioned target image, so that the aforementioned original image is projected on the imaging plane based on the aforementioned target image.
  • the imaging device may be a projection imaging system.
  • the present application provides an image remapping method, which can be applied to an imaging scene based on a projection imaging system.
  • the image processing apparatus first performs row conversion processing and then performs column conversion processing. Specifically, the image processing apparatus obtains each column of pixels of the original image, and obtains a column transformation parameter corresponding to each column of pixels.
  • the column transformation parameter is used to indicate the correspondence between the vertical coordinates of the pixels in the original image and the vertical coordinates of the pixels in the second image. Then, the image processing apparatus determines the pixel value of each pixel in the second image according to each column of pixels in the original image and the column transformation parameter corresponding to each column of pixels.
  • the image processing apparatus obtains an image with pre-distortion in the column direction and no pre-distortion in the row direction (ie, the aforementioned second image). Then, the image processing apparatus will acquire each row of pixels of the second image, and acquire row transformation parameters corresponding to each row of pixels.
  • the row transformation parameter is used to indicate the correspondence between the abscissa of the pixel in the second image and the abscissa of the pixel in the target image.
  • the image processing device determines the pixel value of each pixel in the target image according to each row of pixels in the second image and the row transformation parameter corresponding to each row of pixels. At this time, the image processing device obtains an image with pre-distortion in the row direction and pre-distortion in the column direction (ie, the target image), and the target image is used to cancel the imaging distortion of the imaging device to display the original image.
  • the image processing apparatus when the image processing apparatus acquires each column of pixels in the original image, it can be specifically understood that the image processing apparatus reads the pixel value of each pixel in each column of pixels and the coordinates of the pixel in the original image. Among them, the coordinates of the pixel in the original image include abscissa and ordinate.
  • the image processing apparatus acquires a column transformation parameter corresponding to this column of pixels. Then, the image processing apparatus can calculate a row of pixels in the second image according to the aforementioned row of pixels and the column transformation parameters corresponding to this row of pixels. By analogy, the image processing apparatus will sequentially calculate each column of pixels in the second image.
  • the process from the original image to the second image can be referred to as a column transformation process.
  • the image processing apparatus when the image processing apparatus acquires each row of pixels of the second image, it can be specifically understood that the image processing apparatus reads the pixel value of each pixel in each row of pixels and the coordinates of the pixel in the second image.
  • the coordinates of the pixel in the second image include abscissa and ordinate.
  • the image processing apparatus acquires a row transformation parameter corresponding to this row of pixels. Then, the image processing apparatus can calculate a row of pixels in the target image according to the aforementioned row of pixels and the row transformation parameters corresponding to this row of pixels.
  • the image processing device will sequentially calculate each row of pixels in the target image.
  • the process from the second image to the target image can be referred to as a line transformation process.
  • the image processing apparatus has performed pre-distortion processing on the original image in the row direction and column direction respectively (ie the row transformation and column transformation), and the target image is an image that can cancel the imaging distortion of the projection imaging system. .
  • the image processing apparatus determines the target image (ie, the pre-distorted image) based on the original image
  • the image remapping is decomposed into two unidirectional mappings. Specifically, the image processing device first performs column transformation on each column of pixels in the original image, and then performs row transformation on each row of pixels in the original image based on the column transformation result (ie, the second image) to obtain the target image.
  • the image processing device when calculating a column of pixels in the second image, the image processing device only needs to sequentially read a column of pixels in the original image, and does not need to read out of sequence across columns; similarly, when the image processing device calculates the target When a row of pixels in an image is used, only one row of pixels in the second image needs to be read sequentially, and it is not necessary to read out of sequence across rows. Therefore, the image processing apparatus can read in order when processing the original image, thereby improving the efficiency of processing the original image by the image processing apparatus.
  • the column transformation parameter includes the ordinate of the value point of the fifth pixel corresponding to the fifth pixel in the original image, and the fifth pixel is any pixel in the second image , the value point of the fifth pixel is used to determine the pixel value of the fifth pixel, and the abscissa of the value point of the fifth pixel in the original image is the same as the abscissa of the fifth pixel in the second image.
  • the column transformation parameter can be represented by the ordinate of the value point of the fifth pixel in the original image, and the value point of the fifth pixel is the point corresponding to the fifth pixel in the second image to be obtained, It can be understood that the value point of the fifth pixel falls to the position corresponding to the fifth pixel after the column direction is distorted. Therefore, when calculating the pixel value of the fifth pixel, it is necessary to find the value point of the fifth pixel in the original image. The pixel value of the location.
  • the abscissa of the value point of the fifth pixel in the original image is the same as the horizontal coordinate of the fifth pixel in the second image.
  • the coordinates are the same. Therefore, when the aforementioned image processing apparatus calculates a column of fifth pixels in the second image, only one column of pixels in the original image needs to be acquired column by column.
  • the row transformation parameter includes the abscissa of the value point of the sixth pixel corresponding to the sixth pixel in the second image, and the sixth pixel is any pixel in the target image , the sixth pixel value point is used to determine the pixel value of the sixth pixel, and the ordinate of the sixth pixel value point in the second image is the same as the ordinate of the sixth pixel in the target image.
  • the row transformation parameter can be represented by the abscissa of the value point of the sixth pixel in the second image, and the value point of the sixth pixel is the point corresponding to the sixth pixel in the target image to be determined, It can be understood that after the sixth pixel value point is distorted in the row direction, it falls to the position corresponding to the aforementioned sixth pixel. Therefore, when calculating the pixel value of the sixth pixel, it is necessary to find the sixth pixel value point in the second image. The pixel value of the position in .
  • the vertical coordinate of the value point of the sixth pixel in the second image is the same as the vertical coordinate of the sixth pixel in the target image.
  • the coordinates are the same. Therefore, when calculating the sixth pixel of a row in the target image, the aforementioned image processing apparatus only needs to acquire a row of pixels in the second image row by row.
  • the aforementioned column transformation parameter is stored in a column mapping relationship table, and the position of the column transformation parameter in the column mapping relationship table is used to represent the coordinate of the fifth pixel determined based on the column transformation parameter . It can also be understood that the position of the column transformation parameter in the column mapping relationship table is used to indicate the coordinates of the fifth pixel, where the fifth pixel is a pixel corresponding to the foregoing column transformation parameter.
  • the row transformation parameter is stored in the row mapping relationship table, and the position of the row transformation parameter in the row mapping relationship table is used to represent the coordinates of the sixth pixel determined based on the row transformation parameter. It can also be understood that the position of the row transformation parameter in the row mapping relationship table is used to indicate the coordinates of the sixth pixel, where the sixth pixel is a pixel corresponding to the aforementioned row transformation parameter.
  • the image processing apparatus determines the pixel value of each pixel in the second image according to each column of pixels in the original image and the column transformation parameter corresponding to each column of pixels, including: according to the The column transformation parameter determines the value point of the fifth pixel corresponding to the fifth pixel; at least one seventh pixel is determined according to the ordinate of the value point of the fifth pixel, and the seventh pixel is located at the value point of the fifth pixel.
  • the same column; the pixel value of the fifth pixel is determined according to the pixel value of the at least one seventh pixel.
  • the image processing device determines the pixel value of each pixel in the target image according to each row of pixels in the second image and the row transformation parameter corresponding to each row of pixels, including: determining the relationship with the sixth pixel according to the row transformation parameter.
  • the corresponding value point of the sixth pixel; at least one eighth pixel is determined according to the abscissa of the value point of the sixth pixel, and the eighth pixel and the value point of the sixth pixel are located in the same row; according to the at least one eighth pixel
  • the pixel value of the pixel determines the pixel value of the sixth pixel.
  • the ordinate of the value point of the fifth pixel is an integer.
  • the determining at least one seventh pixel according to the ordinate of the value point of the fifth pixel includes: determining the value point of the fifth pixel as the seventh pixel.
  • the determining the pixel value of the fifth pixel according to the pixel value of the at least one seventh pixel includes: assigning the pixel value of the seventh pixel to the fifth pixel to obtain the pixel value of the fifth pixel.
  • the ordinate of the value point of the fifth pixel is a non-integer.
  • the determining at least one seventh pixel according to the ordinate of the value point of the fifth pixel includes: determining at least one pixel adjacent to the value point of the fifth pixel as the seventh pixel, and the pixel adjacent to the value point of the fifth pixel is the seventh pixel.
  • the pixel is located in the same column as the value point of the fifth pixel.
  • Determining the pixel value of the fifth pixel according to the pixel value of the at least one seventh pixel includes: using an interpolation algorithm to determine the at least one seventh pixel according to the ordinate of the value point of the fifth pixel and the ordinate of the at least one seventh pixel.
  • a first weighted average value is obtained from the pixel value of a seventh pixel; the first weighted average value is assigned to the fifth pixel to obtain the pixel value of the fifth pixel.
  • the abscissa of the value point of the sixth pixel is an integer.
  • the determining at least one eighth pixel according to the abscissa of the value point of the sixth pixel includes: determining the value point of the sixth pixel as the eighth pixel.
  • the determining the pixel value of the sixth pixel according to the pixel value of the at least one eighth pixel includes: assigning the pixel value of the eighth pixel to the sixth pixel to obtain the pixel value of the sixth pixel.
  • the abscissa of the value point of the sixth pixel is a non-integer.
  • the determining at least one eighth pixel according to the abscissa of the value point of the sixth pixel includes: determining at least one pixel adjacent to the value point of the sixth pixel as the eighth pixel, and the pixel adjacent to the value point of the sixth pixel is the eighth pixel.
  • the pixel and the value point of the sixth pixel are located in the same horizontal direction.
  • the determining the pixel value of the sixth pixel according to the pixel value of the at least one eighth pixel includes: using an interpolation algorithm to determine the at least one eighth pixel based on the abscissa of the value point of the sixth pixel and the abscissa of the at least one eighth pixel.
  • a second weighted average value is obtained from the pixel value of an eighth pixel; the second weighted average value is assigned to the sixth pixel to obtain the pixel value of the sixth pixel.
  • acquiring the column transformation parameter corresponding to each column of pixels by the image processing apparatus includes: sequentially reading the column transformation parameters corresponding to each column of pixels from a memory in a column direction, and the memory stores a A column mapping relationship table, where the column mapping relationship table includes the column transformation parameters corresponding to each column of pixels in the original image.
  • the image processing device obtains the row transformation parameters corresponding to the pixels of each row, including: reading the row transformation parameters corresponding to the pixels of each row from a memory in a row direction sequence, and the memory stores a row mapping relationship table, and the row mapping relationship table Including the line transformation parameters corresponding to each line of pixels in the original image.
  • mapping relationship tables are stored in the memory, including a column mapping relationship table from the original image to the second image (that is, only a column-transformed image), and a row mapping relationship from the second image to the target image surface.
  • the mapping relationship table from the original image to the target image is used. Therefore, each calculation of the pixel value of a pixel in the solution of the traditional technology needs to refer to the original image that is not exactly in the same row and not exactly in the same column. Interpolate multiple pixels.
  • the method further includes: image processing The device stores the fifth pixel of each column of the second image in memory.
  • the process for the image processing apparatus to acquire the pixels of each row of the second image may specifically be as follows: the image processing apparatus sequentially reads the pixels of each row of the second image from the memory.
  • the aforementioned second image does not appear in the image processing process, only the original image and the pre-distorted image (ie, the target image).
  • the image processing apparatus may send the target image to the imaging apparatus, and the imaging apparatus is configured to perform imaging based on the aforementioned target image, so that the aforementioned target image is imaged based on the aforementioned
  • the target image projects the aforementioned original image on the imaging plane.
  • the imaging device may be a projection imaging system.
  • the present application provides an image processing device, which may be an image processing chip in a projection imaging system or a processor with an image processing function.
  • the image processing device includes: an acquisition module and a calculation module.
  • the acquiring module is used for acquiring each column of pixels of the original image, and acquiring the column transformation parameters corresponding to each column of pixels.
  • the column transformation parameter is used to indicate the correspondence between the vertical coordinates of the pixels in the original image and the vertical coordinates of the pixels in the second image.
  • the calculation module is configured to determine the pixel value of each pixel in the second image according to each column of pixels in the original image and the column transformation parameter corresponding to each column of pixels.
  • the acquiring module is further configured to acquire the pixels of each row of the second image, and acquire the row transformation parameters corresponding to the pixels of each row.
  • the row transformation parameter is used to indicate the correspondence between the abscissa of the pixel in the second image and the abscissa of the pixel in the target image.
  • the calculation module is further configured to determine the pixel value of each pixel in the target image according to each row of pixels in the second image and the row transformation parameter corresponding to each row of pixels.
  • the image processing device obtains an image with pre-distortion in the row direction and pre-distortion in the column direction (ie, the target image), and the target image is used to cancel the imaging distortion of the imaging device to display the original image.
  • the column transformation parameter includes an ordinate of a fifth pixel value point corresponding to the fifth pixel in the original image, and the fifth pixel value point is used to determine the second image
  • the pixel value of the fifth pixel in , the abscissa of the value point of the fifth pixel in the original image is the same as the abscissa of the fifth pixel in the second image.
  • the row transformation parameter includes an abscissa in the second image of a sixth pixel value point corresponding to the sixth pixel, and the sixth pixel value point is used to determine the target image
  • the pixel value of the sixth pixel in , the ordinate of the value point of the sixth pixel in the second image is the same as the ordinate of the sixth pixel in the target image.
  • the aforementioned column transformation parameter is stored in a column mapping relationship table, and the position of the column transformation parameter in the column mapping relationship table is used to represent the coordinate of the fifth pixel determined based on the column transformation parameter .
  • the row transformation parameter is stored in the row mapping relationship table, and the position of the row transformation parameter in the row mapping relationship table is used to represent the coordinates of the sixth pixel determined based on the row transformation parameter.
  • the calculation module is specifically configured to: determine the value point of the fifth pixel corresponding to the fifth pixel according to the column transformation parameter; according to the ordinate of the value point of the fifth pixel At least one seventh pixel is determined, and the seventh pixel and the value point of the fifth pixel are located in the same column; the pixel value of the fifth pixel is determined according to the pixel value of the at least one seventh pixel.
  • the calculation module is specifically configured to: determine the value point of the sixth pixel corresponding to the sixth pixel according to the row transformation parameter; determine at least one eighth pixel according to the abscissa of the value point of the sixth pixel, the eighth pixel The pixel is located in the same row as the value point of the sixth pixel; the pixel value of the sixth pixel is determined according to the pixel value of the at least one eighth pixel.
  • the ordinate of the value point of the fifth pixel is an integer.
  • the calculation module is specifically configured to: determine the value point of the fifth pixel as the seventh pixel; assign the pixel value of the seventh pixel to the fifth pixel to obtain the pixel value of the fifth pixel.
  • the ordinate of the value point of the fifth pixel is a non-integer.
  • the calculation module is specifically configured to: determine at least one pixel adjacent to the value point of the fifth pixel as the seventh pixel, and the pixel adjacent to the value point of the fifth pixel is located in the same column as the value point of the fifth pixel.
  • the calculation module is specifically used for: using an interpolation algorithm to obtain a first weighted average value of the pixel value of the at least one seventh pixel according to the ordinate of the value point of the fifth pixel and the ordinate of the at least one seventh pixel; The first weighted average value is assigned to the fifth pixel to obtain the pixel value of the fifth pixel.
  • the abscissa of the value point of the sixth pixel is an integer.
  • the calculation module is specifically configured to: determine the value point of the sixth pixel as the eighth pixel; assign the pixel value of the eighth pixel to the sixth pixel to obtain the pixel value of the sixth pixel.
  • the abscissa of the value point of the sixth pixel is a non-integer.
  • the calculation module is specifically used to: determine at least one pixel adjacent to the value point of the sixth pixel as the eighth pixel, and the pixel adjacent to the value point of the sixth pixel and the value point of the sixth pixel are located in the same horizontal; Using an interpolation algorithm, according to the abscissa of the value point of the sixth pixel and the abscissa of the at least one eighth pixel, a second weighted average is obtained for the pixel value of the at least one eighth pixel; the second weighted average is assigned For the sixth pixel, the pixel value of the sixth pixel is obtained.
  • the image processing apparatus further includes a storage module.
  • the storage module is used for storing the fifth pixel of each column of the second image into the memory.
  • acquiring the column transformation parameter corresponding to each column of pixels by the image processing apparatus includes: sequentially reading the column transformation parameters corresponding to each column of pixels from a memory in a column direction, and the memory stores a A column mapping relationship table, where the column mapping relationship table includes the column transformation parameters corresponding to each column of pixels in the original image.
  • the obtaining module is specifically used for: reading the row transformation parameters corresponding to the pixels of each row from the memory in the row direction sequence; and reading the column transformation parameters corresponding to the pixels of each column from the memory in the column direction sequence.
  • the image processing apparatus further includes a storage module.
  • the sending module is configured to send the target image to the imaging device, and the imaging device is configured to perform imaging based on the aforementioned target image, so that the aforementioned original image is projected on the imaging plane based on the aforementioned target image.
  • the imaging device may be a projection imaging system.
  • the present application provides a mapping relationship construction method, which is used to calculate the row mapping relationship table and the column mapping relationship table introduced in the foregoing implementation method.
  • the method can be executed in the aforementioned image processing apparatus; it can also be executed by other computing devices, and the calculated row mapping relationship table and column mapping relationship table are stored in the aforementioned image processing apparatus) (or an external device connected to the image processing apparatus). memory).
  • the computing device obtains the coordinates of each calibration point in the calibration image; obtains the coordinates of each distorted point in the distorted image, the distorted image is an image obtained by the imaging distortion of the imaging device, and the distortion in the distorted image
  • the points correspond one-to-one with the calibration points in the calibration image.
  • the computing device determines a row mapping relationship table and a column mapping relationship table respectively according to the coordinates of the distortion point and the coordinates of the calibration point.
  • the computing device stores the foregoing row mapping relationship table and the foregoing column mapping relationship table in the memory.
  • the computing device determines the row mapping relationship table and the column mapping relationship table respectively according to the coordinates of the distortion point and the calibration point, including: S1, the fixed ordinate is the ordinate of the distortion point , with the abscissa of the calibration point as the independent variable and the abscissa of the corresponding distortion point as the dependent variable to construct a row transformation model. S2. Calculate the abscissa of each pixel in the third image on the original image according to the row transformation model to obtain the row mapping relationship table, where the coordinate of each pixel in the third image is an integer. S3.
  • the abscissa is fixed as the abscissa of the calibration point, and the ordinate of the calibration point is used as the independent variable and the ordinate of the corresponding distortion point is used as the dependent variable to construct a column transformation model.
  • S4. Calculate the ordinate of each pixel in the fourth image on the third image according to the column transformation model, and obtain the column mapping relationship table.
  • the coordinates of each pixel in the fourth image are integers, and the fourth image is the predistorted image of the calibration image.
  • the row mapping relationship table and the column mapping relationship table are respectively determined according to the coordinates of the distortion point and the calibration point, including: T1, the fixed ordinate is the ordinate of the distortion point, and The ordinate of the calibration point is the independent variable and the column transformation model is constructed with the ordinate of the corresponding distortion point as the dependent variable. T2. Calculate the vertical coordinate of each pixel in the fifth image on the original image according to the column transformation model, and obtain the column mapping relationship table, where the coordinate of each pixel in the fifth image is an integer. T3.
  • the abscissa is fixed as the abscissa of the calibration point, and the abscissa of the calibration point is used as the independent variable and the abscissa of the corresponding distortion point is used as the dependent variable to construct a row transformation model.
  • T4 Calculate the abscissa of each pixel in the sixth image on the fifth image according to the row transformation model, and obtain a row mapping relationship table, where the coordinates of each pixel in the sixth image are integers, and the sixth image is A pre-distorted image of this calibration image.
  • determining the row mapping relationship table and the column mapping relationship table respectively according to the coordinates of the distortion point and the calibration point including: taking the abscissa and ordinate of the distortion point as the self- variable, and the abscissa and ordinate of the calibration point are dependent variables, and a distortion model is constructed; according to the distortion model, the abscissa and ordinate of each pixel in the seventh image on the calibration image are calculated to obtain the distortion mapping table. , the abscissa and ordinate of each pixel in the seventh image are integers; the coordinates in the distortion mapping table are split along the horizontal direction to obtain the row mapping table; the coordinates in the distortion mapping table are Split along the vertical direction to get the column mapping relationship table.
  • the present application provides a computing device for computing the row mapping relationship table and the column mapping relationship table introduced by the foregoing implementation method.
  • the computing device includes an acquisition module and a determination module.
  • the acquisition module is used for acquiring the coordinates of each calibration point in the calibration image, and acquiring the coordinates of each distortion point in the distorted image.
  • the distorted image is an image obtained by imaging distortion of the imaging device, and the distorted points in the distorted image correspond to the calibration points in the calibration image one-to-one.
  • the determining module is used to determine the row mapping relationship table and the column mapping relationship table respectively according to the coordinates of the distortion point and the coordinates of the calibration point.
  • the computing device stores the foregoing row mapping relationship table and the foregoing column mapping relationship table in the memory.
  • the determining module is specifically configured to: fix the ordinate as the ordinate of the distortion point, use the abscissa of the calibration point as the independent variable and the abscissa of the corresponding distortion point as the dependent variable to construct a row transformation Model.
  • the abscissa of each pixel in the third image on the original image is calculated to obtain the row mapping relationship table, and the coordinate of each pixel in the third image is an integer.
  • the fixed abscissa is the abscissa of the calibration point
  • the ordinate of the calibration point is used as the independent variable and the ordinate of the corresponding distortion point is used as the dependent variable to construct a column transformation model.
  • the determining module is specifically configured to: fix the ordinate as the ordinate of the distortion point, use the ordinate of the calibration point as the independent variable and the ordinate of the corresponding distortion point as the dependent variable to construct a column transformation Model.
  • the column transformation model the ordinate of each pixel in the fifth image on the original image is calculated to obtain the column mapping relationship table, and the coordinate of each pixel in the fifth image is an integer.
  • the fixed abscissa is the abscissa of the calibration point, and the abscissa of the calibration point is used as the independent variable and the abscissa of the corresponding distortion point is used as the dependent variable to construct a row transformation model.
  • the determining module is specifically configured to: take the abscissa and ordinate of the distortion point as independent variables, and the abscissa and ordinate of the calibration point as dependent variables, to construct a distortion model Calculate the abscissa and ordinate of each pixel on this calibration image in the seventh image according to the distortion model, obtain the distortion mapping table, and the abscissa and ordinate of each pixel in the seventh image are integers;
  • the coordinates in the distortion mapping relationship table are split in the horizontal direction to obtain the row mapping relationship table; the coordinates in the distortion mapping relationship table are split in the vertical direction to obtain the column mapping relationship table.
  • the present application further provides an image processing apparatus, the image processing apparatus includes a processor and an internal memory, the processor is coupled to the internal memory; the processor is configured to read line mapping parameters from the external memory The table and the column map the parameter table into the internal memory; the internal memory also stores a program, when the program instructions stored in the internal memory are executed by the processor, the image processing apparatus can realize any of the first or third aspects described above. The method described in one embodiment.
  • the present application further provides an imaging device, including an image processing device and a display device.
  • the image processing device is configured to generate a target image according to the original image according to the method described in any one of the embodiments of the first aspect or the third aspect, and transmit the target image to the display device, so that the imaging device Project the original image.
  • the imaging device is a projection imaging system.
  • the present application further provides a computer-readable storage medium, comprising a computer program, the computer program being executed by a processor to implement the method described in any one of the embodiments of the foregoing first aspect or the third aspect.
  • the present application also provides a computer program product comprising instructions, the computer program product comprising computer program code, when the computer program code is run on a computer, the computer is made to perform the first or third aspects described above The method described in any one of the embodiments.
  • the embodiments of the present application have the following advantages:
  • the image processing apparatus when determining the target image (ie, the pre-distorted image) based on the original image, decomposes the image remapping into two single-directional mappings. Specifically, the image processing device first performs row transformation on each row of pixels in the original image, and then performs column transformation on each column of pixels in the original image based on the row transformation result (ie, the first image) to obtain the target image; Perform column transformation on each column of pixels in the image, and then perform row transformation on each row of pixels based on the result of the column transformation to obtain the target image.
  • the image processing device calculates a row of pixels in the first image, it only needs to read a row of pixels in the original image, and does not need to read across rows; similarly, the image processing device calculates a column of pixels in the target image. , only one column of pixels in the first image needs to be read, and it is not necessary to read across columns. Therefore, the image processing apparatus can read in order when processing the original image, thereby improving the efficiency of processing the original image by the image processing apparatus.
  • 1A is a simple structural diagram of a projection imaging system
  • FIG. 1B is an example diagram of the original image in this application.
  • FIG. 1C is an example diagram of a distorted image in the application
  • Fig. 2 is a flow chart of the image remapping method in the application
  • Fig. 3A is an example diagram of the row conversion process in this application.
  • 3B is an exemplary diagram of a column transformation process in the application.
  • Fig. 3C is an example diagram of row transformation and column transformation process in this application.
  • Fig. 4 is a flow chart of the mapping relationship construction method in this application.
  • 5A is an example diagram of a calibration image in the application
  • 5B is a schematic diagram of an embodiment of a method for constructing a mapping relationship in the application
  • 6A is another flowchart of the image remapping method in the application.
  • Fig. 6B is another example diagram of row transformation and column transformation process in the application.
  • FIG. 7 is a schematic diagram of an embodiment of an image processing apparatus in the present application.
  • FIG. 8 is a schematic diagram of another embodiment of the image processing apparatus in this application.
  • Embodiments of the present application provide an image remapping method and an image processing apparatus, which are used to improve the efficiency of generating a predistorted image based on an original image.
  • the image remapping method of the present application is mainly used to cancel the distortion generated by the imaging device based on the projection imaging system.
  • the following will introduce the principle of the distortion generation and the principle of canceling the distortion in combination with the structure of the projection imaging system:
  • FIG. 1A it is a schematic diagram of a simple structure of the projection imaging system 10 .
  • the projection imaging system 10 includes an optical module 101 and a display device 102 .
  • the display device 102 is used to generate an image that needs to be projected and imaged.
  • the light emitted by the display device 102 reaches the imaging plane 11 after passing through the optical module 101 , and an image is presented on the imaging plane 11 .
  • the aforementioned display device 102 may be a liquid crystal display (LCD), a digital light processing (DLP), a liquid crystal on silicon (LCOS), or the like.
  • the aforementioned optical module 101 may include one or more lenses, free-form mirrors, and the like.
  • the image presented on the aforementioned imaging plane 11 is not the image displayed by the display device 102 (hereinafter referred to as the original image), but is based on the aforementioned A distorted image (hereinafter referred to as a distorted image) is generated from the image.
  • the original image needs to be processed into a pre-distorted image with the opposite distortion direction and the same degree of distortion as the distorted image, and then, the aforementioned pre-distorted image is generated on the display device 102, so that the light emitted by the display device 102 passes through the optical
  • the module 101 reaches the imaging plane 11
  • the original image obtained by the pre-distorted image after distortion is displayed on the imaging plane 11 .
  • image distortion due to optical aberrations can be canceled.
  • the display device 102 generates a normal image as shown in FIG. 1B
  • the image projected by the aforementioned projection imaging system 10 onto the imaging plane 11 may be any distorted image as shown in FIG. 1C .
  • the original image displayed by the display device 102 is the normal image shown in FIG. 1B and the distorted image presented by the imaging plane 11 is the pincushion distorted image shown in (a) of FIG. 1C .
  • the aforementioned original image ie, the normal image shown in FIG. 1B
  • the imaging plane 11 can present a normal image as shown in FIG. 1B
  • the normal image ie, the original image
  • the normal image is obtained by distorting the barrel-shaped distorted image (ie, the pre-distorted image).
  • the image remapping method proposed in the embodiments of the present application can not only be applied to correct the distortion caused by the imaging device based on the projection imaging system, but also can establish the row and column directions between the original image and the pre-distorted image, respectively.
  • the above-mentioned original image is processed into a row-transformed image (or a column-transformed image) and an image after row-transformation and column-transformation (or after column-transformation and row-transformation). image after) to obtain a pre-distorted image. Since there is no need to access data across rows or columns, processing efficiency of the processor can be improved.
  • the distortion forms that can be corrected by the image remapping method in this embodiment of the present application may be pincushion distortion (as shown in (a) in FIG. 1C ), barrel distortion (as shown in (b) in FIG. 1C ) )), keystone distortion (as shown in (c) in Fig. 1C), sector distortion (as shown in (d) in Fig. 1C), and parallelogram distortion (as shown in (e) in Fig. 1C)
  • the distortion of any one of the above distortion forms may also be a combination or superposition of any at least two distortion forms of the foregoing multiple distortion forms, or may be distortion without a specific shape, or the like. There is no specific limitation here.
  • the aforementioned imaging device based on the aforementioned projection imaging system may be a head-up display (HUD) (also known as a head-up display), or a head-mounted display (HMD) (also known as a head-up display). called a head-mounted display).
  • HUD head-up display
  • HMD head-mounted display
  • the aforementioned head-up display HUD can be a display in the cockpit system of an aviation aircraft, which is installed above the central instrument panel of the cockpit as a window for exchanging information between the pilot and the aircraft; the aforementioned head-up display HUD can also be applied to the Internet of Vehicles (vehicle to everything , V2X) field, as a vehicle display, to provide driving information for the driver.
  • V2X Internet of Vehicles
  • the aforementioned head-mounted display HMD is often used in augmented reality (AR) technology and virtual reality (VR) technology, for example, AR glasses (also known as AR displays) and VR glasses (also known as VR displays) ).
  • AR augmented reality
  • VR virtual reality
  • the imaging plane that receives the image projected by the foregoing imaging device is the retina of the user.
  • the imaging device may also be an ordinary projector.
  • the imaging plane that receives the image projected by the projector is an image receiving device such as a curtain.
  • the image processing device performing the aforementioned image remapping method may include an image processing chip located in the aforementioned head-up display HUD or head-mounted display HMD, for example, a field programmable gate array (FPGA), A chip with image processing functions such as digital signal processing (DSP), central processing unit (CPU) or graphics processing unit (GPU).
  • FPGA field programmable gate array
  • DSP digital signal processing
  • CPU central processing unit
  • GPU graphics processing unit
  • the aforementioned display device 102 may be integrated with the aforementioned image processing device.
  • the display device 102 and the image processing device are both integrated into the aforementioned head-up display HUD (or head-mounted display HMD), and each pre-image calculated by the image processing device The distorted image can be immediately input to the aforementioned display device 102 for display.
  • the aforementioned display device 102 can also be distributed with the aforementioned image processing device, for example, the image processing device is externally placed in a computer connected to the aforementioned head-up display HUD (or head-mounted display HMD), the computer can It is transmitted to the display device 102 in the aforementioned head-up display HUD (or head-mounted display HMD) in a wired or wireless manner for display.
  • the image remapping method proposed by the present application will be introduced below based on the aforementioned scenarios.
  • this method as shown in FIG. 2 , if the image processing apparatus first performs row transformation on the original image, and then performs row transformation on the row transformed image. transform, the original image will perform the following steps:
  • Step 201 Acquire each row of pixels of the original image.
  • the original image is an image that is expected to be presented on the imaging plane without distortion.
  • the solution proposed in this application needs to perform pre-distortion processing on the original image to obtain a target image (ie, a pre-distorted image).
  • the aforementioned target image is input into the projection imaging system for projection, so that the aforementioned original image is presented on the imaging plane.
  • the image processing apparatus may read the aforementioned original image from an external memory through a data transmission interface or other communication interface, or receive the aforementioned original image from other external devices.
  • the image processing apparatus can temporarily store the obtained original image in the internal memory of the image processing apparatus, and reads a row of pixels from the internal memory every time a row of pixels is calculated.
  • the image processing apparatus will sequentially read the pixels of each row of the original image, and will also process the pixels of each row in sequence in the subsequent processing process.
  • a memory for example, double data rate synchronous dynamic random access memory (DDR SDRAM) (DDR) (DDR) for short
  • DDR SDRAM double data rate synchronous dynamic random access memory
  • the image processing apparatus when the image processing apparatus acquires each row of pixels of the original image, it can be specifically understood that the image processing apparatus reads the pixel value of each pixel in each row of pixels and the coordinates of the pixel in the original image. Among them, the coordinates of the pixel in the original image include abscissa and ordinate.
  • the process of acquiring each row of pixels in the original image by the image processing apparatus may specifically be as follows: the image processing apparatus reads each row of pixels in the original image from the memory. A row of pixels, that is, to read the pixels of consecutive abscissas in a row.
  • the image processing apparatus may read in the order of the row numbers, for example, after reading the pixels of the first row, read the pixels of the second row, and then read the pixels of the third row; The ordinal numbers are read in reverse order, for example, after reading the pixels of the second row, read the pixels of the first row.
  • the image processing apparatus may start reading from the first pixel of a certain row, or may start reading from the Nth pixel of a certain row. There is no specific limitation here.
  • Step 202 Obtain the row transformation parameter corresponding to each row of pixels.
  • step 201 and step 202 there is no time sequence limitation between step 201 and step 202 . That is to say, the image processing apparatus may execute step 201 first and then step 202, or may first execute step 202 and then execute step 201, and may also execute the aforementioned step 201 and step 202 at the same time, which is not limited herein.
  • the description will be made only by taking as an example that the image processing apparatus can first obtain a row of pixels from the original image, and then obtain row transformation parameters corresponding to the pixels of the row.
  • the row transformation parameter is a parameter used to perform row transformation processing on a certain row of pixels in the original image.
  • the row transformation process refers to the moving position of the pixel in the row direction, which can also be understood as the abscissa of the pixel has changed but the ordinate has not changed.
  • the image obtained from the original image by only line transformation processing is referred to as the first image.
  • the row transformation parameter is used to indicate the correspondence between the abscissa of the pixel in the original image and the abscissa of the pixel in the first image. It should also be understood that a change in the position of a pixel (ie, a movement of the pixel) is essentially a migration of the pixel value.
  • the row transformation parameter can be used to determine which pixel in the original image the value of the pixel value of a certain pixel in the first image comes from.
  • any pixel in the first image is called a first pixel
  • the first pixel corresponds to a point in the original image as a value point of the first pixel. Therefore, the row transformation parameter includes the abscissa in the original image of the first pixel value point corresponding to the first pixel, and the first pixel value point is used to determine the pixel value of the first pixel in the first image,
  • the ordinate of the value point of the first pixel in the original image is the same as the ordinate of the first pixel in the first image.
  • the row transformation parameter may be stored in the memory in the form of a row mapping relationship table, where the row mapping relationship table includes a plurality of row transformation parameters.
  • the number of row transformation parameters in the row mapping relationship table is equal to the number of pixels in the aforementioned first image, and each row transformation parameter in the row mapping relationship table is the same as that in the first image. There is a one-to-one correspondence between each pixel according to the coordinates.
  • the position of the row transformation parameter in the row mapping relationship table is used to represent the coordinates of the first pixel in the first image determined based on the row transformation parameter. Exemplarily, if the first image is an image of 1080 ⁇ 1080, the row mapping table corresponding to the original image is also 1080 ⁇ 1080. If the row transformation parameter a1 is located in the second row and the third column in the row mapping relationship table, then the row transformation parameter a1 is used to determine the pixel value of the pixel whose coordinates are (3, 2) in the first image.
  • the row mapping relationship table may be compressed and stored in the memory, that is, the compressed row mapping relationship table is stored in the memory.
  • the number of row transformation parameters in the compressed row mapping relationship table is smaller than the number of pixels in the foregoing first image.
  • the aforementioned image processing device obtains the row transformation parameters corresponding to the pixels of each row, which can be understood as reading the compressed row mapping relationship table and restoring the compressed row mapping relationship table to a complete row mapping relationship table, and then , and then read the row-to-row transformation parameters corresponding to the first image from the aforementioned complete row-mapping relation table; it can also be understood as reading several row-transformation parameters in the compressed row-mapping relation table, and then Several row transformation parameters read in the row mapping relationship table are restored to a row of row transformation parameters.
  • the foregoing restoration process may use an interpolation algorithm, which is not specifically limited here.
  • the aforementioned pixel value is a value assigned by a computer when the image is digitized, and represents the average brightness information of a certain pixel in the image.
  • the aforementioned original image may be a single-channel image, a three-channel image or a four-channel image.
  • a single-channel image is also called a grayscale image, and each pixel is represented by a pixel value.
  • the pixel value of the single-channel image is represented by 8 bits, the value range of the pixel value of the single-channel image is: 0 (black) to 255 (white).
  • the three-channel image generally refers to an RGB image, and the RGB image can present a color or a black and white image.
  • each pixel value of the three-channel image is represented by 8 bits, each pixel value is represented by three channel values, that is, the pixel value is represented by the superposition of red (0-255), green (0-255) and blue (0-255) .
  • the four-channel image adds a brightness channel to the three-channel image to represent the degree of transparency. If the original image is a single-channel image, the first image and the target image calculated by the image processing device are also single-channel images; if the original image is a three-channel image, the first image and the target image calculated by the image processing device are also Three-channel image; there is no specific limitation here. If the original image is a three-channel image, the pixel value of one pixel is composed of three channel values, and the same calculation needs to be performed for each channel value during the calculation process.
  • the process for the image processing apparatus to obtain the row transformation parameters corresponding to each row of pixels may specifically be: The row transformation parameter of the abscissa.
  • the image processing apparatus may read in the sequence of the row numbers, for example, after reading the row transformation parameters of the first row, read the row transformation parameters of the second row, and then read the row transformation parameters of the third row. parameters; it can also be read in reverse order of the row ordinal. For example, after reading the row transformation parameters of the second row, read the row transformation parameters of the first row.
  • the image processing apparatus may start reading from the first line transformation parameter of a certain line, or may start reading from the Nth line transformation parameter of a certain line. There is no specific limitation here.
  • the image processing apparatus may read the row mapping relationship table by reading the original image. That is to say, the order in which the image processing apparatus reads the pixels in the original image is consistent with the order in which the image processing apparatus reads the row transformation parameters in the row mapping relationship table. Exemplarily, if the image processing apparatus starts to read pixels from the Nth row and M column in the original image, the image processing apparatus should also start to read the row transformation parameters from the Nth row and Mth column in the row transformation parameter table.
  • Step 203 Determine the pixel value of each pixel in the first image according to each row of pixels in the original image and the row transformation parameter corresponding to each row of pixels.
  • the first image is an image with pre-distortion in the row direction and no pre-distortion in the column direction, that is, the first image is the result of row transformation processing of the original image to be obtained.
  • the process of calculating the first image by the image processing apparatus can be understood as the process of determining the pixel value of each pixel in the first image by the image processing apparatus. Because the position (ie, the coordinates) of each pixel in the first image is known, that is, the abscissa and the ordinate of each pixel in the first image are continuous integers. Therefore, the row transformation parameters of the first image can be queried according to the abscissa and ordinate of the first pixel, that is, the row transformation parameters with the same abscissa and ordinate can be queried according to the abscissa and ordinate of the first pixel.
  • FIG. 3A is used as an example for description.
  • the original image 31 is a 4 ⁇ 4 image (ie, an image with four rows and four columns), and the row mapping relationship table corresponding to the original image 31 is also four rows and four columns.
  • each pixel has a certain pixel value, which is represented by s(i, j), where i represents the abscissa and j represents the ordinate.
  • s(1, 1) represents the pixel value of the pixel whose abscissa is 1 and whose ordinate is 1.
  • the pixel value of each pixel needs to be calculated and obtained by the image processing apparatus according to the original image 31 and the row mapping relationship table 32 .
  • the abscissa of the pixel 331 is 1 and the ordinate is 1, that is, the pixel 331 is located in the first row and the first column in the first image.
  • the row transformation parameter corresponding to the pixel 331 is the row transformation parameter 321 corresponding to the first row and the first column in the row mapping relationship table 32 .
  • the image processing apparatus will determine the first pixel value point corresponding to the first pixel according to the row transformation parameter, and determine at least one third pixel according to the abscissa of the first pixel value point. Then, the image processing apparatus determines the pixel value of the first pixel according to the pixel value of the at least one third pixel. Wherein, the third pixel and the value point of the first pixel are located in the same row.
  • the abscissa of the value point of the first pixel may be an integer or a non-integer.
  • the image processing apparatus may directly determine that the value point of the first pixel is the third pixel, and then determine the value of the first pixel as the third pixel.
  • the pixel value of the third pixel is assigned to the first pixel to obtain the pixel value of the first pixel.
  • the row transformation parameter corresponding to the first pixel is the row transformation parameter 322 .
  • the value of the row transformation parameter 322 is 4
  • the row transformation parameter 332 is located in the second row of the row mapping relationship table 32
  • the pixel value of the pixel 332 is determined by the pixel of the fourth pixel in the second row in the original image.
  • the value (that is, the pixel value whose abscissa is 4 and the ordinate is 2) is determined.
  • 4 is the abscissa of the first pixel value point, which is the fourth pixel in the second row in the original image (that is, the abscissa is 4 and the ordinate is 2); the The third pixel is directly determined by the value point of the first pixel, that is, the third pixel is the value point of the first pixel.
  • the image processing apparatus needs to determine one or more third pixels near the non-integer. Specifically, the image processing apparatus determines that at least one pixel adjacent to the first pixel value point is the third pixel, and the pixel adjacent to the first pixel value point and the first pixel value point are located in the same row.
  • the image processing device uses an interpolation algorithm to obtain a first weighted average value of the pixel value of the at least one third pixel according to the abscissa of the value point of the first pixel and the abscissa of the at least one third pixel, and the The first weighted average value is assigned to the first pixel to obtain the pixel value of the first pixel.
  • the row transformation parameter corresponding to the first pixel is the row transformation parameter 321 .
  • the value of the row transformation parameter 321 is 1.6
  • the row transformation parameter 331 is located in the first row of the row mapping relationship table 32. Therefore, the value point of the first pixel is the point whose coordinates are (1.6, 1) in the original image.
  • the image processing apparatus needs to determine that the value point adjacent to the first pixel and the pixel located in the same row as the value point of the first pixel is the third pixel.
  • the image processing device can determine the pixel 311
  • 1.6 is the abscissa of the first pixel value point
  • the first pixel value point is the point (1.6, 1)
  • the third pixel is pixel 311 and pixel 312 .
  • the image processing apparatus may determine that two pixels to the left of the value point of the first pixel and two pixels to the right of the value point of the first pixel are the third pixel values, which are not specifically limited here.
  • what is listed in the foregoing example is a common one-dimensional linear interpolation.
  • the image processing device can also use interpolation methods such as bilinear interpolation in the row direction, multiple nonlinear interpolation, etc., which is not limited here. .
  • the image processing apparatus calculates each pixel in the first image
  • the aforementioned first image can be obtained.
  • the image processing apparatus when calculating the pixel value of a certain pixel in the first image, the image processing apparatus only needs one or more third pixels located in the same row as the pixel in the original image, and does not need to be located in other rows of pixels.
  • the pixel value of one pixel in the first image needs to be calculated, at least four pixels located in two rows in the original image need to be used. Therefore, the solution of the present application is beneficial to improve the efficiency of the image processing apparatus for calculating the pre-distorted image.
  • the image processing device since the image processing device only needs one row of pixels instead of two rows of pixels during calculation, the image processing device does not need to read two rows of pixels or two rows of pixels when reading the pixels or row transformation parameters in the memory.
  • the row transformation parameter avoids out-of-order access to the memory, reduces the number of activations of the memory, solves the problem of random reading and writing in complex remapping, and reduces the difficulty of hardware implementation.
  • the target image is calculated in parallel, the original image pixels required by multiple target pixels in parallel are also continuous, and redundant original images are not read, which reduces the required internal memory size to achieve high parallel computing.
  • the image processing apparatus may further store the aforementioned first image in the memory. It should be understood that, each time the image processing apparatus calculates the pixel value of a row of pixels of the first image, the pixel value of the row of pixels may be written into the memory.
  • the image processing device may also store the calculated row of pixels in the buffer, and after the pixels of the multiple rows are calculated, the image processing device writes the pixels of the aforementioned multiple rows into the memory in sequence. There is no specific limitation here.
  • the image processing apparatus can directly write the pixel value of each row of pixels into the memory row by row, or can perform transposition processing of the pixel value of each row of pixels, and write each row of pixels into the memory by column.
  • the image processing apparatus can directly write the pixel value of each row of pixels into the memory row by row, or can perform transposition processing of the pixel value of each row of pixels, and write each row of pixels into the memory by column.
  • Step 204 Acquire each column of pixels of the first image.
  • the first image is an image obtained from the original image through only line transformation processing.
  • the relevant introduction in the foregoing step 202 please refer to the relevant introduction in the foregoing step 202, which will not be repeated here.
  • the image processing apparatus sequentially reads the pixels of each column of the aforementioned first image from the memory. If the pixel value of each row of pixels of the image processing device is written into the memory row by row, the image processing device will read the pixels of each column of the first image in the memory by column. If the image processing device transposes the pixel values of the pixels in each row and writes the pixels in each row into the memory in columns, the image processing device will read the pixels in each column of the first image in the memory row by row .
  • Step 205 Obtain the column transformation parameter corresponding to each column of pixels.
  • step 204 and step 205 there is no time sequence limitation between step 204 and step 205 . That is to say, the image processing apparatus may perform step 204 first and then step 205, or may perform step 205 first and then perform step 204, and may also perform the aforementioned step 204 and step 205 at the same time, which is not limited herein.
  • the description will be given by taking an example that the image processing apparatus may first obtain a column of pixels from the first image, and then obtain column transformation parameters corresponding to the pixels of the column.
  • the column transformation parameter is a parameter used to perform column transformation processing on a certain column of pixels in the first image.
  • the column transformation process refers to the movement of the position of the pixel in the column direction, and it can also be understood that the ordinate of the pixel has changed but the abscissa has not changed.
  • the image obtained from the first image through only the column transformation process is called the target image, that is, the pre-distorted image based on the original image.
  • the target image is input into the projection imaging system, and the original image obtained by distorting the pre-distorted image will be displayed on the imaging plane.
  • the column transformation parameter is used to indicate the correspondence between the vertical coordinates of the pixels in the first image and the vertical coordinates of the pixels in the target image.
  • the column transformation parameter can be used to determine from which pixel in the first image the pixel value of a certain pixel in the target image comes from.
  • any pixel in the target image is called a second pixel, and the second pixel corresponds to a point in the first image as a value point of the second pixel.
  • the column transformation parameter includes the ordinate of the second pixel value point corresponding to the second pixel in the first image, and the second pixel value point is used to determine the pixel value of the second pixel in the target image,
  • the abscissa of the value point of the second pixel in the first image is the same as the abscissa of the second pixel in the target image.
  • the column transformation parameter may be stored in the memory in the form of a column mapping relationship table, where the column mapping relationship table includes a plurality of column transformation parameters.
  • the number of column transformation parameters in the column mapping relationship table is equal to the number of pixels in the target image, and each column transformation parameter in the column mapping relationship table is the same as each pixel in the target image. There is a one-to-one correspondence between them according to the coordinates.
  • the position of the column transformation parameter in the column mapping relationship table is used to represent the coordinates of the second pixel in the target image determined based on the column transformation parameter. Exemplarily, if the target image is an image of 1080 ⁇ 1080, the column mapping table corresponding to the target image is also 1080 ⁇ 1080. If the column transformation parameter b1 is located in the second row and the third column in the column mapping relationship table, then the column transformation parameter b1 is used to determine the pixel value of the pixel whose coordinates are (3, 2) in the target image.
  • the column mapping relationship table may be compressed and stored in the memory, that is, the compressed column mapping relationship table is stored in the memory.
  • the number of column transformation parameters in the compressed column mapping relationship table is smaller than the number of pixels in the aforementioned target image.
  • the aforementioned image processing apparatus obtains the column transformation parameter corresponding to each column of pixels, which can be understood as reading the compressed column mapping relationship table and restoring the compressed column mapping relationship table to a complete column mapping relationship table, and then , and then read a column of column transformation parameters corresponding to the target image from the aforementioned complete column mapping relationship table; it can also be understood as reading several column transformation parameters in the compressed column mapping relationship table, and then, from the compressed Several column transformation parameters read in the column mapping relationship table are restored to one column of column transformation parameters.
  • the foregoing restoration process may use an interpolation algorithm, which is not specifically limited here.
  • the process of acquiring the column transformation parameters corresponding to each column of pixels by the image processing device may specifically be as follows: the image processing device reads the column transformation parameters corresponding to each column of pixels from the memory by column, that is, reads consecutive pixels in a certain column.
  • the column transformation parameter of the ordinate may be read in the order of the columns, for example, after reading the row transformation parameters of the first column, read the column transformation parameters of the second column, and then read the column transformation parameters of the third column.
  • the parameters can also be read in reverse order of the column ordinal. For example, after reading the column transformation parameters of the second column, read the column transformation parameters of the first column.
  • the image processing apparatus may start reading from the first column transformation parameter of a certain column, or may start reading from the Nth column transformation parameter of a certain column. There is no specific limitation here.
  • the image processing apparatus may read the column mapping relationship table in a manner of reading the first image. That is to say, the order in which the image processing apparatus reads the first pixels in the first image is consistent with the order in which the image processing apparatus reads the column transformation parameters in the column mapping relationship table. Exemplarily, if the image processing apparatus starts to read the first pixel from the Nth row and M column in the first image, the image processing apparatus should also start to read the column transform from the Nth row and Mth column in the column transformation parameter table. parameter.
  • Step 206 Determine the pixel value of each pixel in the target image according to each column of pixels in the first image and the column transformation parameter corresponding to each column of pixels.
  • the target image is a pre-distorted image used to cancel the imaging distortion of the imaging device.
  • the target image is input into the projection imaging system, and the original image obtained by distorting the pre-distorted image will be displayed on the imaging plane.
  • the process of calculating the target image by the image processing apparatus can be understood as the process of determining the pixel value of each pixel in the target image by the image processing apparatus. Because the position (ie, the coordinates) of each pixel in the target image is known, that is, the abscissa and the ordinate of each pixel in the target image are continuous integers. Therefore, the column transformation parameters of the target image can be queried according to the abscissa and ordinate of the second pixel, that is, the column transformation parameters with the same abscissa and ordinate can be queried according to the abscissa and ordinate of the second pixel.
  • FIG. 3B is taken as an example for introduction.
  • the first image 33 is a 4 ⁇ 4 image, and the column mapping relationship table corresponding to the first image 33 also has four rows and four columns.
  • the pixel value of each pixel has been calculated by the image processing means.
  • the pixel value of each pixel in the target image 35 needs to be calculated by the image processing device according to the first image 33 and the column mapping relationship table 34.
  • the first pixel 351 in the first row of the target image 35 as an example, the horizontal direction of the pixel 351 The coordinate is 1, and the ordinate is 1, that is, the pixel 351 is located in the first row and first column in the target image.
  • the column transformation parameter corresponding to the pixel 351 is the column transformation parameter 341 corresponding to the first row and the first column in the column mapping relationship table 34 .
  • the image processing apparatus will determine the second pixel value point corresponding to the target pixel according to the column transformation parameter, and determine at least one fourth pixel according to the ordinate of the second pixel value point. Then, the image processing apparatus determines the pixel value of the target pixel according to the pixel value of the at least one fourth pixel. Wherein, the fourth pixel and the value point of the second pixel are located in the same row.
  • the abscissa of the value point of the second pixel may be an integer or a non-integer.
  • the image processing apparatus may directly determine that the value point of the second pixel is the fourth pixel, and then determine the value of the second pixel as the fourth pixel.
  • the pixel value of the fourth pixel is assigned to the second pixel to obtain the pixel value of the second pixel.
  • the column transformation parameter corresponding to the second pixel is the column transformation parameter 342 .
  • the value of the column transformation parameter 342 is 3
  • the column transformation parameter 342 is located in the fourth column of the column mapping relationship table 34
  • the pixel value of the pixel 342 is determined by the first image 33 , the third pixel in the fourth column
  • the pixel value (that is, the pixel value whose abscissa is 4 and the ordinate is 3) is determined.
  • 3 is the vertical and horizontal coordinates of the second pixel value point, and the second pixel value point is the third pixel in the fourth column in the first image 33 (that is, the abscissa is 4 and the vertical axis is 3).
  • the fourth pixel is directly determined by the value point of the second pixel, that is, the fourth pixel is the value point of the second pixel.
  • the image processing apparatus needs to determine one or more fourth pixels near the non-integer. Specifically, the image processing device determines that at least one pixel adjacent to the second pixel value point is the fourth pixel, and the pixel adjacent to the second pixel value point and the second pixel value point are located in the same row.
  • the image processing device uses an interpolation algorithm to obtain a second weighted average value of the pixel value of the at least one fourth pixel according to the abscissa of the value point of the second pixel and the abscissa of the at least one fourth pixel, and the The second weighted average value is assigned to the second pixel to obtain the pixel value of the second pixel.
  • the second pixel is the pixel 351
  • the column transformation parameter corresponding to the second pixel is the column transformation parameter 341 .
  • the value of the column transformation parameter 341 is 1.5
  • the column transformation parameter 341 is located in the first column of the column mapping relationship table 34 . Therefore, the second pixel value point is the point whose coordinates are (1, 1.5) in the first image 33 .
  • the image processing apparatus needs to determine that the value point adjacent to the second pixel, and the pixel located in the same column as the value point of the second pixel is the fourth pixel.
  • the image processing apparatus may determine that two pixels above the value point of the second pixel and two pixels below the value point of the second pixel are the fourth pixel values, which are not specifically limited here.
  • what is listed in the foregoing example is a common one-dimensional linear interpolation.
  • the image processing device can also use interpolation methods such as bilinear interpolation in the row direction, multiple nonlinear interpolation, etc., which is not limited here. .
  • the image processing apparatus has calculated each pixel in the target image.
  • Step 207 Send the aforementioned target image to the imaging device.
  • the imaging device is configured to perform imaging based on the aforementioned target image, so that the aforementioned original image is projected on the imaging plane based on the aforementioned target image.
  • step 207 is an optional step.
  • the image processing apparatus when determining the target image (ie, the pre-distorted image) based on the original image, decomposes the image remapping into two unidirectional mappings. As shown in FIG. 3C , the image processing device first performs row transformation on each row of pixels in the original image (ie, step 1 in FIG. 3C ), and then performs column transformation on each column of pixels based on the row transformation result (ie, the first image). Transform (ie, step 2 in Figure 3C ) to obtain the target image.
  • the image processing device calculates a row of pixels in the first image, it only needs to read a row of pixels in the original image, and does not need to read across rows; similarly, the image processing device calculates a column of pixels in the target image. , only one column of pixels in the first image needs to be read, and it is not necessary to read across columns. Therefore, the image processing apparatus can read in order when processing the original image, thereby improving the efficiency of processing the original image by the image processing apparatus.
  • the computing device may determine the row mapping relationship table and the column mapping relationship table in the following manner:
  • Step 401a Obtain the coordinates of each calibration point in the calibration image.
  • the calibration image is an image used to measure the imaging distortion of an imaging device (eg, the aforementioned head-up display HUD, head-mounted display HMD, projector, etc.), and the calibration image includes a plurality of calibration points with certain coordinates. Generally, a plurality of calibration points in the calibration image are arranged in the calibration image according to a certain geometric shape.
  • an imaging device eg, the aforementioned head-up display HUD, head-mounted display HMD, projector, etc.
  • the calibration image includes a plurality of calibration points with certain coordinates.
  • a plurality of calibration points in the calibration image are arranged in the calibration image according to a certain geometric shape.
  • the calibration image may be an image with only calibration points as shown on the left side of FIG. 5A , or an image including a standard grid as shown on the right side of FIG. 5A , which is not specifically limited in this application. If the calibration image is an image containing a standard grid, the vertex corner of the grid (ie, the intersection between adjacent grids) is the calibration point.
  • Step 401b Obtain the coordinates of each distorted point in the distorted image.
  • the distorted image is an image obtained by imaging distortion of the imaging device, and the distorted points in the distorted image correspond to the calibration points in the calibration image one-to-one. That is to say, a certain calibration point in the calibration image becomes a certain distortion point in the distorted image after being distorted.
  • the distorted image corresponding to the calibration image 501 may be shown as the distorted image 502 .
  • the calibration points in the calibration image 501 correspond to the distorted points in the distorted image 502 one-to-one.
  • mapping from 501 to 503 is obtained, that is, the pre-distortion mapping required by the parameter table (eg, the row mapping relationship table or the column mapping relationship table).
  • Step 402 Determine a row mapping relationship table and a column mapping relationship table respectively according to the coordinates of the distortion point and the coordinates of the calibration point.
  • the steps for the computing device to determine the row mapping relationship table and the column mapping relationship table are as follows:
  • the abscissa is fixed as the abscissa of the calibration point, and the ordinate of the calibration point is used as the independent variable and the ordinate of the corresponding distortion point is used as the dependent variable to construct a column transformation model.
  • (x i , y i ) are used to represent the coordinates of the calibration point on the original image
  • (u i , v i ) are used to represent the coordinates of the distorted point on the distorted image.
  • the computing device first calculates the distortion model that only performs row transformation on the calibration points, that is, calculates the model F ⁇ 1 ( xi , y i ) under row transformation according to the mapping from ( xi , v i ) to ( ui , v i ).
  • the computing device calculates the model F ⁇ 2 (x, y) under the row transformation according to the mapping of (x i , y' i ) to (u' i , y' i ). Then, the computing device calculates a distortion model that only performs column transformation on the calibration points, that is, calculates a model F ⁇ 1 (x, y) under column transformation according to the mapping from ( xi , y i ) to ( xi , v i ). Then, the computing device uses the distortion model to calculate the coordinates of each point on the target image on the original image, and obtains a row mapping relationship table and a column mapping relationship table:
  • i represents the abscissa of the pixel in the target image
  • j represents the ordinate of the pixel in the target image
  • width represents the width of the target image, which can be understood as the maximum value of the abscissa of the pixel in the target image
  • height represents the height of the target image, It can be understood as the maximum value of the ordinate of the pixel in the target image.
  • the computing device may also perform column transformation on the mapping relationship table from the original image to the target image, so that the foregoing row mapping relationship table and column mapping relationship table can be approximately obtained.
  • the following formula can be used to split:
  • M(i,j,0) represents the abscissa of the pixel in the original image corresponding to the ith column and the jth row of the target image
  • M(i,j,1) represents the original image corresponding to the ith column and the jth row of the target image
  • the vertical coordinate of the pixel in the middle; jj arg min
  • the parameter in the i-th column and the j-th row of the transformation parameter table (including only row transformation parameters) is the column transformation parameter in the parameter pair (including both row transformation parameters and column transformation parameters) in the i-th column of the original parameter table.
  • the row transformation parameters of similar parameter pairs that is, fix the ordinate and take the abscissa.
  • mapping relationship table is compressed, that is, the complete point-to-point mapping relationship is not stored. In this case, it can still be decomposed through the solution of the present application.
  • the compressed row mapping table and column mapping table are compressed, that is, the complete point-to-point mapping relationship is not stored. In this case, it can still be decomposed through the solution of the present application.
  • the present application Compared with the traditional technology to directly calculate the distortion mapping from (x i , y i ) to (u i , v i ), the present application first calculates the mapping from ( xi , v i ) to (u i , v i ), and then Calculate the mapping from (x i , y i ) to ( xi , v i ), therefore, the present application can obtain a row mapping relationship table and a column mapping relationship table.
  • the steps for determining the row mapping relationship table and the column mapping relationship table are as follows:
  • the fixed ordinate is the ordinate of the distortion point, and the ordinate of the calibration point is used as the independent variable and the ordinate of the corresponding distortion point is used as the dependent variable to construct a column transformation model.
  • the abscissa is fixed as the abscissa of the calibration point, and the abscissa of the calibration point is used as the independent variable and the abscissa of the corresponding distortion point is used as the dependent variable to construct a row transformation model.
  • T4 Calculate the abscissa of each pixel in the sixth image on the fifth image according to the row transformation model, and obtain a row mapping relationship table, where the coordinates of each pixel in the sixth image are integers, and the sixth image is A pre-distorted image of this calibration image.
  • (x i , y i ) are used to represent the coordinates of the calibration point on the original image
  • (u i , v i ) are used to represent the coordinates of the distorted point on the distorted image.
  • the computing device first calculates a distortion model that only performs column transformation on the calibration points, that is, calculates a model F ⁇ 2 (x, y) under column transformation according to the mapping from (u i , y i ) to (u i , v i ).
  • the computing device computes the model F ⁇ 3 (x, y) under the column transformation according to the mapping of (x' i , y i ) to (x' i , v' i ). Then, the computing device calculates a distortion model that only performs row transformation on the calibration points, that is, calculates a model F ⁇ 3 (x, y) under row transformation according to the mapping from (x i , y i ) to (u i , y i ). Then, the computing device uses the distortion model to calculate the coordinates of each point on the target image on the original image, and obtains a row mapping relationship table and a column mapping relationship table:
  • M Y (i, j) F ⁇ 3 (i, j)
  • M X (i, j) F ⁇ 3 (i, j), 0 ⁇ i ⁇ width, 0 ⁇ i ⁇ height;
  • i represents the abscissa of the pixel in the target image
  • j represents the ordinate of the pixel in the target image
  • width represents the width of the target image, which can be understood as the maximum value of the abscissa of the pixel in the target image
  • height represents the height of the target image, It can be understood as the maximum value of the ordinate of the pixel in the target image.
  • the computing device may also transform the mapping relationship table from the original image to the target image, so that the foregoing row mapping relationship table and column mapping relationship table can be approximately obtained.
  • ii argmin
  • the parameter in column i and row j (including only the column transformation parameters) is the parameter pair in the jth row of the original parameter table (including both row transformation parameters and column transformation parameters) whose row transformation parameters are most similar to i.
  • the column coordinate parameter of that is, fix the abscissa and take the ordinate.
  • Step 403 Store the row mapping relationship table and the column mapping relationship table in the memory.
  • the computing device stores the foregoing row mapping relationship table and column mapping relationship table in the memory.
  • the memory may be memory in a projection imaging system.
  • the memory is an external memory relative to the aforementioned image processing device, and the internal memory in the aforementioned image processing device can obtain the aforementioned row mapping relationship table and column mapping relationship table from the internal memory storing the row mapping relationship table and the column mapping relationship table.
  • the image remapping method proposed by the present application will be introduced below based on the aforementioned scenario.
  • the image processing apparatus first performs column transformation on the original image, and then performs row transformation on the image processed by the column transformation. transform, the original image will perform the following steps:
  • Step 601 Acquire each column of pixels of the original image.
  • the image processing apparatus when the image processing apparatus acquires each column of pixels in the original image, it can be specifically understood that the image processing apparatus reads the pixel value of each pixel in each column of pixels and the coordinates of the pixel in the original image. Among them, the coordinates of the pixel in the original image include abscissa and ordinate.
  • Step 602 Obtain the column transformation parameter corresponding to each column of pixels.
  • step 601 and step 602 there is no time sequence limitation between step 601 and step 602 . That is to say, the image processing apparatus may first perform step 601 and then perform step 602, or may perform step 602 first and then perform step 601, and may also perform the aforementioned step 601 and step 602 at the same time, which is not specifically limited here.
  • the description will be given by taking an example that the image processing apparatus can first obtain a column of pixels from the original image, and then obtain column transformation parameters corresponding to the pixels of the column.
  • the column transformation parameter is used to indicate the correspondence between the vertical coordinates of the pixels in the original image and the vertical coordinates of the pixels in the second image (ie, the column transformation result).
  • the second image is an image with pre-distortion in the column direction and no pre-distortion in the row direction, and the ordinate of the second image is an integer.
  • the column transformation parameter includes the ordinate of the fifth pixel value point corresponding to the fifth pixel in the original image, and the fifth pixel value point is used to determine the fifth pixel in the second image.
  • the abscissa of the value point of the fifth pixel in the original image is the same as the abscissa of the fifth pixel in the second image.
  • the foregoing column transformation parameter is stored in a column mapping relationship table, and the position of the column transformation parameter in the column mapping relationship table is used to represent the coordinate of the fifth pixel determined based on the column transformation parameter.
  • the process of acquiring the column transformation parameter corresponding to each column of pixels by the image processing apparatus may specifically be that the image processing apparatus sequentially reads the column transformation parameter corresponding to each column of pixels from the memory in the column direction. Specifically, according to the bit width of the memory, the number of column transformation parameters read each time is different, and the number of times the memory needs to be activated to read a column of column transformation parameters is different.
  • the image processing apparatus may read the column mapping relationship table by reading the original image. For details, please refer to the example in step 201.
  • Step 603 Determine the pixel value of each pixel in the second image according to each column of pixels in the original image and the column transformation parameter corresponding to each column of pixels.
  • the second image is an image with pre-distortion in the column direction and no pre-distortion in the row direction, that is, the second image is the result of the column transformation processing of the original image to be obtained.
  • the process of calculating the second image by the image processing apparatus can be understood as the process of determining the pixel value of each pixel in the second image by the image processing apparatus. Because the position (ie, the coordinates) of each pixel in the second image is known, that is, the abscissa and the ordinate of each pixel in the second image are continuous integers. Therefore, the column transformation parameters of the second image can be queried according to the abscissa and ordinate of the fifth pixel, that is, the column transformation parameters with the same abscissa and ordinate can be queried according to the abscissa and ordinate of the fifth pixel.
  • the image processing device first determines the value point of the fifth pixel corresponding to the fifth pixel according to the column transformation parameter; then, determines at least one seventh pixel according to the ordinate of the value point of the fifth pixel, the The seventh pixel and the value point of the fifth pixel are located in the same column; then, the pixel value of the fifth pixel is determined according to the pixel value of the at least one seventh pixel.
  • the image processing apparatus directly determines that the value point of the fifth pixel is the seventh pixel, and then determines the seventh pixel.
  • the pixel value of the pixel is assigned to the fifth pixel to obtain the pixel value of the fifth pixel.
  • the image processing apparatus determines that at least one pixel adjacent to the value point of the fifth pixel is the seventh pixel, and the The pixel adjacent to the value point of the fifth pixel is located in the same column as the value point of the fifth pixel; then, using an interpolation algorithm according to the ordinate of the value point of the fifth pixel and the ordinate of the at least one seventh pixel A first weighted average is obtained for the pixel value of the at least one seventh pixel; the first weighted average is assigned to the fifth pixel to obtain the pixel value of the fifth pixel.
  • Step 604 Acquire each row of pixels of the second image.
  • the second image is an image with pre-distortion in the column direction and no pre-distortion in the row direction, that is, the second image is the result of the column transformation processing of the original image to be obtained.
  • the image processing apparatus sequentially reads the pixels of each row of the aforementioned second image from the memory. If the pixel value of each column of pixels of the image processing device is written into the memory by column, the image processing device will read the pixels of each row of the second image in the memory by row. If the image processing device transposes the pixel values of each column of pixels and writes the pixels of each column into the memory row by row, the image processing device will read the pixels of each row of the second image in the memory by column .
  • Step 605 Obtain the row transformation parameter corresponding to each row of pixels.
  • step 604 there is no time sequence limitation between step 604 and step 605 . That is to say, the image processing apparatus may perform step 604 first and then step 605, or may perform step 605 first and then perform step 604, and may also perform the aforementioned step 604 and step 605 at the same time, which is not limited herein.
  • the description will be made only by taking as an example that the image processing apparatus may first obtain a row of pixels from the second image, and then obtain row transformation parameters corresponding to the pixels of the row.
  • the row transformation parameter is used to indicate the correspondence between the vertical coordinates of the pixels in the second image and the vertical coordinates of the pixels in the target image.
  • the row transformation parameter includes the abscissa of the sixth pixel value point corresponding to the sixth pixel in the second image, and the sixth pixel value point is used to determine the sixth pixel in the target image.
  • the ordinate of the value point of the sixth pixel in the second image is the same as the ordinate of the sixth pixel in the target image.
  • the row transformation parameter is stored in a row mapping relationship table, and the position of the row transformation parameter in the row mapping relationship table is used to represent the coordinates of the sixth pixel determined based on the row transformation parameter.
  • Step 606 Determine the pixel value of each target image according to each row of pixels in the second image and the row transformation parameter corresponding to each row of pixels.
  • the target image is the pre-distorted image used to cancel the imaging distortion of the imaging device.
  • the image processing device determines the value point of the sixth pixel corresponding to the sixth pixel according to the row transformation parameter; determines at least one eighth pixel according to the abscissa of the value point of the sixth pixel, and the eighth pixel
  • the value point of the sixth pixel is located in the same row; the pixel value of the sixth pixel is determined according to the pixel value of the at least one eighth pixel.
  • the image processing apparatus determines that the value point of the sixth pixel is the eighth pixel, and then determines the value of the eighth pixel.
  • the pixel value is assigned to the sixth pixel to obtain the pixel value of the sixth pixel.
  • the image processing apparatus determines that at least one pixel adjacent to the value point of the sixth pixel is the eighth pixel, and the adjacent pixel is the eighth pixel.
  • the pixel of the sixth pixel value point and the sixth pixel value point are located in the same horizontal; then, using an interpolation algorithm according to the abscissa of the sixth pixel value point and the abscissa of the at least one eighth pixel.
  • a second weighted average value is obtained from the pixel value of at least one eighth pixel; the second weighted average value is assigned to the sixth pixel to obtain the pixel value of the sixth pixel.
  • Step 607 Send the aforementioned target image to the imaging device.
  • the imaging device is configured to perform imaging based on the aforementioned target image, so that the aforementioned original image is projected on the imaging plane based on the aforementioned target image.
  • step 607 is an optional step.
  • the image processing apparatus when determining the target image (ie, the pre-distorted image) based on the original image, decomposes the image remapping into two unidirectional mappings. As shown in FIG. 6B , the image processing apparatus first performs column transformation on each column of pixels in the original image (ie, step 1 in FIG. 6B ), and then performs row transformation on each row of pixels based on the column transformation result (ie, the second image). Transformation (ie, step 2 in FIG. 6B ), to obtain the target image.
  • the image processing device calculates one column of pixels in the second image, it only needs to read one column of pixels in the original image, and does not need to read across columns; similarly, when the image processing device calculates one row of the target image In the case of pixels, only one row of pixels in the second image needs to be read, and it is not necessary to read across rows. Therefore, the image processing apparatus can read in order when processing the original image, thereby improving the efficiency of processing the original image by the image processing apparatus.
  • FIG. 7 it is a schematic structural diagram of an image processing apparatus 70 according to an embodiment of the present application.
  • the image processing device 70 may be located in a head-up display HUD (also known as a head-up display), a head-mounted display HMD or a projector.
  • the image processing device may be an image processing chip located in the aforementioned head-up display HUD, head-mounted display HMD or projector.
  • the steps in the foregoing embodiments corresponding to FIG. 2 and FIG. 6A may be performed by the image processing apparatus 70 .
  • the image processing apparatus 70 includes at least one processor 701 and at least one memory 702 .
  • the aforementioned processor 701 and the aforementioned memory 702 are interconnected by wires. It should be understood that only one processor 701 and one memory 702 are shown in FIG. 7 .
  • the processor 701 may be a field programmable gate array (FPGA), a digital signal process (DSP), a central processing unit (CPU) or a graphics processor (graphics processing unit). processing unit, GPU) and other functional units or functional modules with image processing functions.
  • FPGA field programmable gate array
  • DSP digital signal process
  • CPU central processing unit
  • GPU graphics processor
  • the aforementioned processor 701 may be a single-core processor; may also be a multi-core processor, for example, the processor 701 may be composed of multiple FPGAs or multiple DSPs. At this time, the processor 701 can process multiple images at the same time. For example, if the processor 701 includes two FPGAs, FPGA1 and FPGA2, after performing row transformation processing on Image 1, FPGA1 sends the row transformation result to FPGA2, and FPGA2 performs column transformation processing on the row transformation result of Image 1.
  • the FPGA1 While the FPGA2 performs column transformation processing on the row transformation result of the aforementioned image 1, the FPGA1 can continue to perform row transformation processing on the next image (for example, image 2), and then sends the row transformation result of the image 2 to FPGA2, The row transformation result of the image 2 is subjected to column transformation processing by the FPGA 2 .
  • processor 701 may refer to one or more devices, circuits, and/or processing cores for processing data (eg, computer program instructions).
  • the processor 701 can be a separate semiconductor chip, or can be integrated with other circuits into a semiconductor chip, for example, can form a system-on-chip with other circuits (such as codec circuits, hardware acceleration circuits, or various bus and interface circuits). (system-on-a-chip, SoC), or can also be integrated in the ASIC as a built-in processor of an application specific integrated circuit (ASIC), which can be packaged separately Or it can be packaged with other circuits.
  • SoC system-on-a-chip
  • ASIC application specific integrated circuit
  • the aforementioned memory 702 may be a read-only memory (ROM), other types of static storage devices that can store static information and instructions, or a random access memory (RAM) , or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), which is not specifically limited here.
  • ROM read-only memory
  • RAM random access memory
  • EEPROM electrically erasable programmable read-only memory
  • the aforementioned memory 702 may be a double-rate synchronous dynamic random access memory (DDR SDRAM) (DDR for short).
  • DDR SDRAM double-rate synchronous dynamic random access memory
  • the memory 702 may exist independently, but is connected to the aforementioned processor 701 .
  • the memory 702 may also be integrated with the aforementioned processor 701 . For example, integrated within one or more chips.
  • the memory 702 can be understood as the internal memory of the image processing apparatus 70, and the memory 702 is used for storing the row mapping relationship table, the column mapping relationship table and the original image read from the external memory.
  • the memory 702 is also used for storing intermediate images generated in the process of calculating the target image. Exemplarily, the row transformation result generated based on the original image (eg, the aforementioned first image), or the column transformation result generated based on the original image (eg, the aforementioned second image).
  • the memory 702 is also used to store program codes for executing the technical solutions of the embodiments of the present application.
  • the aforementioned program codes can be controlled and executed by the processor 701 , and various types of computer program codes executed can also be regarded as drivers of the processor 701 .
  • the image processing apparatus 70 further includes a communication interface 703, and the communication interface 703 is used for communication with an external memory or other devices.
  • the image processing apparatus 70 can receive instructions or data from other devices through the communication interface 703 .
  • the image processing apparatus 70 may read the row mapping relationship table, the column mapping relationship table and the original image from the external memory through the aforementioned communication interface 703 .
  • the aforementioned processor 701 may acquire, from an external memory or an internal memory, each line of pixels of the original image, and a line transformation parameter corresponding to each line of pixels. Then, the processor 701 determines the pixel value of each pixel in the first image according to each row of pixels in the original image and the row transformation parameter corresponding to each row of pixels, to obtain a first image. Then, the processor 701 stores the aforementioned first image in the internal memory. Then, the processor obtains each column of pixels of the first image from the aforementioned internal memory, and obtains the column transformation parameters corresponding to each column of pixels from the external memory or the internal memory.
  • the processor 701 determines the pixel value of each pixel in the target image according to each column of pixels in the first image and the column transformation parameter corresponding to each column of pixels to obtain the target image.
  • the processor 701 stores the aforementioned target image in an external memory.
  • the processor 701 determines the first pixel value point corresponding to the first pixel according to the row transformation parameter; then, determines at least one third pixel according to the abscissa of the first pixel value point, the The third pixel and the value point of the first pixel are located in the same row; then, the pixel value of the first pixel is determined according to the pixel value of the at least one third pixel.
  • the processor 701 determines the second pixel value point corresponding to the second pixel according to the column transformation parameter; then, determines at least one fourth pixel according to the ordinate of the second pixel value point, the The fourth pixel and the value point of the second pixel are located in the same column; then, the pixel value of the second pixel is determined according to the pixel value of the at least one fourth pixel.
  • the aforementioned processor 701 may acquire, from an external memory or an internal memory, each column of pixels of the original image, and a column transformation parameter corresponding to each column of pixels. Then, the processor 701 determines the pixel value of each pixel in the second image according to each column of pixels in the original image and the column transformation parameter corresponding to each column of pixels to obtain a second image. Then, the processor 701 stores the aforementioned second image in the internal memory. Then, the processor obtains each line of pixels of the second image from the aforementioned internal memory, and obtains line transformation parameters corresponding to each line of pixels from the external memory or the internal memory.
  • the processor 701 determines the pixel value of each pixel in the target image according to each row of pixels in the second image and the row transformation parameter corresponding to each row of pixels to obtain the target image.
  • the processor 701 stores the aforementioned target image in an external memory.
  • the processor 701 determines the value point of the fifth pixel corresponding to the fifth pixel according to the column transformation parameter; at least one seventh pixel is determined according to the ordinate of the value point of the fifth pixel, and the seventh pixel is The pixel and the value point of the fifth pixel are located in the same column; the pixel value of the fifth pixel is determined according to the pixel value of the at least one seventh pixel.
  • the processor 701 determines the value point of the sixth pixel corresponding to the sixth pixel according to the row transformation parameter; at least one eighth pixel is determined according to the abscissa of the value point of the sixth pixel, the eighth pixel.
  • the pixel is located in the same row as the value point of the sixth pixel; the pixel value of the sixth pixel is determined according to the pixel value of the at least one eighth pixel.
  • the processor 701 when determining the target image (ie, the pre-distorted image) based on the original image, the processor 701 decomposes the image remapping into two unidirectional mappings. Specifically, the processor 701 first performs row transformation on each row of pixels in the original image, and then performs column transformation on each column of pixels in the original image based on the row transformation result (ie, the first image) to obtain the target image; Perform column transformation on each column of pixels, and then perform row transformation on each row of pixels based on the column transformation result to obtain the target image.
  • the processor 701 calculates a row of pixels in the first image, it only needs to read a row of pixels in the original image, and does not need to read across rows; similarly, when the processor 701 calculates a column of pixels in the target image , and only one column of pixels in the first image needs to be read, not across columns. Therefore, the image processing apparatus can read in order when processing the original image, thereby improving the efficiency of processing the original image by the image processing apparatus.
  • the processor 701 when the processor 701 reads the row mapping relationship table, the column mapping relationship table, the original image, the first image, and the second image from the memory 702, the processor 701 does not need to read out of order across rows, reducing the amount of storage space in the memory 702. The number of times the memory cell is activated increases the useful life of the memory 702 .
  • Embodiments of the present application also provide a computer-readable storage medium, where a row mapping relationship table, a column mapping relationship table, and a program for generating a target image are stored in the computer-readable storage medium.
  • a row mapping relationship table a row mapping relationship table
  • a column mapping relationship table a program for generating a target image
  • the computer can be made to execute the method described in the embodiment shown in FIG. 2 or FIG. 6A.
  • the aforementioned image processing device 70 shown in FIG. 7 is a chip, which is also called a digital processing chip.
  • the chip includes a processing unit and a communication unit.
  • the processing unit acquires the program instruction through the communication unit, and the program instruction is executed by the processing unit, so that the processing unit executes the method steps described in the corresponding embodiment of FIG. 2 or FIG. 6A .
  • the processing unit is a circuit or an interface that integrates the above-mentioned processor 701 or is used to realize the function of the above-mentioned processor 701
  • the communication unit is a circuit or an interface that integrates the above-mentioned communication interface 703 or is used to realize the function of the above-mentioned communication interface 703 .
  • the storage unit when a storage unit is integrated in the chip, the storage unit may be a storage device such as a memory.
  • the processing unit in the chip may call the program code from the storage unit to implement the method steps described in the foregoing embodiment corresponding to FIG. 2 or FIG. 6A .
  • the chip When the chip is not integrated with a storage unit, the chip can be connected to a storage device such as an external memory through the aforementioned communication unit, so that the program code is obtained from the aforementioned external storage to realize the above-mentioned corresponding embodiment of FIG. 2 or FIG. 6A .
  • an embodiment of the present application provides a schematic structural diagram of an image processing apparatus 80 .
  • the foregoing method embodiments corresponding to FIG. 2 and FIG. 6A may all be based on the structure of the image processing apparatus 80 shown in FIG. 8 .
  • the image processing apparatus 80 includes a plurality of functional modules, and each of the aforementioned functional modules may be integrated into one processing unit, or each module may exist physically alone, or two or more modules may be integrated into one unit.
  • the above-mentioned integrated units may be implemented in the form of hardware, or may be implemented in the form of software functional units.
  • the image processing device 80 may be located in a head-up display HUD (also referred to as a head-up display) or a head-mounted display HMD.
  • the image processing device may be an image processing chip located in the aforementioned head-up display HUD or head-mounted display HMD.
  • the steps in the foregoing embodiments corresponding to FIG. 2 and FIG. 6A may be performed by the image processing apparatus 80 .
  • the image processing apparatus 80 includes an acquisition module 801 and a calculation module 802 .
  • the acquiring module 801 is configured to acquire each row of pixels of the original image, and acquire row transformation parameters corresponding to each row of pixels.
  • the row transformation parameter is used to indicate the correspondence between the abscissa of the pixel in the original image and the abscissa of the pixel in the first image, where the first image has pre-distortion in the row direction and no pre-distortion in the column direction image.
  • the calculation module 802 is configured to determine the pixel value of each pixel in the first image according to each row of pixels in the original image and the row transformation parameter corresponding to each row of pixels.
  • the obtaining module 801 is further configured to obtain each column of pixels in the first image, and obtain a column transformation parameter corresponding to each column of pixels, where the column transformation parameter is used to indicate the vertical dimension of the pixels in the first image.
  • the correspondence between the coordinates and the vertical coordinates of the pixels in the target image which is an image with pre-distortion in the row direction and pre-distortion in the column direction.
  • the calculation module 802 is further configured to determine the pixel value of each pixel in the target image according to each column of pixels in the first image and the column transformation parameter corresponding to each column of pixels, and the target image is used for The imaging distortion of the imaging device is counteracted to reveal the original image.
  • the calculation module 802 is specifically configured to: determine the value point of the first pixel corresponding to the first pixel according to the row transformation parameter; The coordinates determine at least one third pixel, and the third pixel and the value point of the first pixel are located in the same row; the pixel value of the first pixel is determined according to the pixel value of the at least one third pixel.
  • the second pixel value point corresponding to the second pixel is determined according to the column transformation parameter; at least one fourth pixel is determined according to the ordinate of the second pixel value point, the The fourth pixel and the value point of the second pixel are located in the same column; the pixel value of the second pixel is determined according to the pixel value of the at least one fourth pixel.
  • the image processing apparatus 80 further includes a storage module 803;
  • the storage module is used for storing the first pixels of each row of the first image into a memory.
  • the acquiring 801 module is specifically used for:
  • the column transformation parameters corresponding to the pixels in each column are sequentially read from the memory in the column direction.
  • the image processing apparatus 80 further includes a sending module 804;
  • the sending module 804 is configured to send the target image to the imaging device, where the imaging device is configured to perform imaging based on the aforementioned target image, so that the aforementioned original image is projected on the imaging plane based on the aforementioned target image.
  • the imaging device may be a projection imaging system.
  • each step of the above-mentioned method can be completed by a hardware integrated logic circuit in a processor or an instruction in the form of software.
  • the steps of the methods disclosed in conjunction with the embodiments of the present application may be directly embodied as executed by a hardware processor, or executed by a combination of hardware and software modules in the processor.
  • the software modules may be located in random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, registers and other storage media mature in the art.
  • the storage medium is located in the memory, and the processor reads the information in the memory, and completes the steps of the above method in combination with its hardware. To avoid repetition, detailed description is omitted here. It should also be understood that the first, second, third, fourth and various numerical numbers involved in this document are only distinctions made for convenience of description, and are not used to limit the scope of the embodiments of the present application.
  • the size of the sequence numbers of the above-mentioned processes does not mean the sequence of execution, and the execution sequence of each process should be determined by its functions and internal logic, and should not be dealt with in the embodiments of the present application. implementation constitutes any limitation.

Landscapes

  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Image Processing (AREA)

Abstract

本申请实施例公开了一种图像重映射方法以及图像处理装置，可以在基于原始图像确定目标图像(即预畸变图像)时，将图像重映射分解为两次单方向的映射。具体地，该图像处理装置先对原始图像中的每一行像素进行行变换，再基于行变换结果(即第一图像)对其中的每一列像素进行列变换，得到目标图像。也就是说，该图像处理装置在计算第一图像中的一行像素时，仅需要顺序读取原始图像中的一行像素，而不需要跨行乱序读取；类似的，图像处理装置在计算目标图像的一列像素时，也仅需要顺序读取第一图像中的一列像素，而不需要跨列乱序读取。因此，图像处理装置在处理原始图像时可以有序读取，进而提高了图像处理装置处理原始图像的效率。

Description

一种图像重映射方法以及图像处理装置
本申请要求于2020年12月18日提交中国国家知识产权局、申请号202011510725.7、申请名称为“一种图像重映射方法以及图像处理装置”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请实施例涉及图像处理技术领域，尤其涉及一种图像重映射方法以及图像处理装置。
背景技术
畸变是成像***普遍存在的一种像差。对于投影式成像***，若原始图像在投影过程中产生畸变，将导致呈现在成像平面的图像是相比于原始图像产生了畸变的图像(即畸变图像)，而不是前述原始图像。
在传统技术中，常采用预畸变的方式在把原始图像转换为可以抵消畸变的预畸变图像。具体地，处理器在基于原始图像计算预畸变图像中每个像素的像素值时，该处理器需要从原始图像中查找与预畸变图像中A像素对应的A’点，并将该A’点的像素值赋值给预畸变图像中的A像素。但是，由于预畸变图像相比于原始图像在行列两个方向上均有不同程度的畸变，因此该A’点的横纵坐标不一定为整数，因此，该处理器需要采用插值算法根据该A’点附近的多个像素计算出A’点的像素值。以此循环，直到该处理器计算出预畸变图像中的每个像素的像素值。然后，该处理器将该预畸变图像输入至投影式成像***的显示装置之后，用户便可以通过成像平面观察到由前述预畸变图像经过畸变而获得的原始图像。
由于，传统技术中的显示驱动要求图像按行顺序输入到驱动中，而预畸变图中顺序的每一行像素对应到原始图像的像素一般是乱序的，因此计算预畸变图像的过程中涉及到大量的对原始图像内容的乱序访问，将影响处理器的处理效率。因此，目前急需一种能够提高处理原始图像效率的方案。
发明内容
本申请实施例提供了一种图像重映射方法以及图像处理装置，用于提高基于原始图像生成预畸变图像的效率。
第一方面，本申请提供了一种图像重映射方法，该图像重映射方法可以应用于基于投影式成像***的成像场景中。在该方法中，图像处理装置先进行行变换处理再进行列变换处理。具体地，该图像处理装置获取原始图像的每一行像素，并且，获取该每一行像素对应的行变换参数。其中，该行变换参数用于指示该原始图像中像素的横坐标与第一图像中像素的横坐标之间的对应关系。然后，该图像处理装置分别根据该原始图像中的每一行像 素以及该每一行像素对应的行变换参数确定该第一图像中每个像素的像素值。此时，该图像处理装置获得了行方向存在预畸变而列方向不存在预畸变的图像(即前述第一图像)。然后，图像处理装置将获取该第一图像的每一列像素，并且，获取该每一列像素对应的列变换参数。其中，该列变换参数用于指示该第一图像中像素的纵坐标与目标图像中像素的纵坐标之间的对应关系。然后，该图像处理装置分别根据该第一图像中的每一列像素以及该每一列像素对应的列变换参数确定该目标图像中每个像素的像素值。此时，该图像处理装置获得了行方向存在预畸变且列方向存在预畸变的图像(即目标图像)，该目标图像用于抵消成像装置的成像畸变以展示该原始图像。
应当理解的是，图像处理装置获取原始图像的每一行像素，可以具体理解为，该图像处理装置读取到每一行像素中每一个像素的像素值和该像素在原始图像中的坐标。其中，该像素在原始图像中的坐标包含横坐标和纵坐标。前述图像处理装置每从原始图像中获取一行像素，相应的，该图像处理装置便获取这一行像素对应的行变换参数。然后，该图像处理装置根据前述一行像素和这一行像素对应的行变换参数便可以计算出第一图像中的一行像素。依次类推，该图像处理装置将依次计算出第一图像中的每一行像素。从原始图像到第一图像的过程可以称为行变换过程。类似的，图像处理装置获取第一图像的每一列像素，可以具体理解为，该图像处理装置读取到每一列像素中每一个像素的像素值和该像素在第一图像中的坐标。其中，该像素在第一图像中的坐标包含横坐标和纵坐标。前述图像处理装置每从第一图像中获取一列像素，相应的，该图像处理装置便获取这一列像素对应的列变换参数。然后，该图像处理装置根据前述一列像素和这一列像素对应的列变换参数便可以计算出目标图像中的一列像素。依次类推，该图像处理装置将依次计算出目标图像中的每一列像素。从第一图像到目标图像的过程可以称为列变换过程。此时，该图像处理装置已经对前述原始图像在行方向和列方向上分别进行了预畸变处理(即前述行变换和列变换)，该目标图像是能够抵消投影式成像***的成像畸变的图像。
本申请中，图像处理装置在基于原始图像确定目标图像(即预畸变图像)时，将图像重映射分解为两次单方向的映射。具体地，该图像处理装置先对原始图像中的每一行像素进行行变换，再基于行变换结果(即第一图像)对其中的每一列像素进行列变换，得到目标图像。也就是说，该图像处理装置在计算第一图像中的一行像素时，仅需要顺序读取原始图像中的一行像素，而不需要跨行乱序读取；类似的，图像处理装置在计算目标图像的一列像素时，也仅需要顺序读取第一图像中的一列像素，而不需要跨列乱序读取。因此，图像处理装置在处理原始图像时可以有序读取，进而提高了图像处理装置处理原始图像的效率。
在一种可选的实施方式中，该行变换参数包括与第一像素对应的第一像素取值点在该原始图像中的横坐标，该第一像素为该第一图像中的任一像素，该第一像素取值点用于确定该第一像素的像素值，该第一像素取值点在该原始图像中的纵坐标与该第一像素在该第一图像中的纵坐标相同。
本实施方式中，提出行变换参数可以用第一像素取值点在原始图像中的横坐标表示，该第一像素取值点是与待求的第一图像中的第一像素对应的点，可以理解为，第一像素取 值点在行方向发生畸变之后便落到了前述第一像素对应的位置，因此，在计算第一像素的像素值是需要查找第一像素取值点在原始图像中的位置的像素值。由于，第一像素和第一像素取值点之间仅是行方向存在畸变，因此，第一像素取值点在该原始图像中的纵坐标与该第一像素在该第一图像中的纵坐标相同。因此，前述图像处理装置在计算第一图像中的一行第一像素时，仅需要逐行获取原始图像中的一行像素。
在一种可选的实施方式中，该列变换参数包括与第二像素对应的第二像素取值点在该第一图像中的纵坐标，该第二像素为该述目标图像中的任一像素，该第二像素取值点用于确定该第二像素的像素值，该第二像素取值点在该第一图像中的横坐标与该第二像素在该目标图像中的横坐标相同。
本实施方式中，提出列变换参数可以用第二像素取值点在第一图像中的纵坐标表示，该第二像素取值点是与待求的目标图像中的第二像素对应的点，可以理解为，第二像素取值点在列方向发生畸变之后便落到了前述第二像素对应的位置，因此，在计算第二像素的像素值是需要查找第二像素取值点在第一图像中的位置的像素值。由于，第二像素和第二像素取值点之间仅是列方向存在畸变，因此，第二像素取值点在该第一图像中的横坐标与该第二像素在该目标图像中的横坐标相同。因此，前述图像处理装置在计算目标图像中的一列第二像素时，仅需要逐列获取第一图像中的一列像素。
相比于传统技术，传统技术中的图像处理器在图像处理过程中仅涉及从原始图像到目标图像的映射参数，需要在一次映射处理中对横和列同时进行重映射。因此，在计算目标图像中的一个像素时，就要求图像处理器至少能够从原始图像中读取呈矩形分布的四个像素。又由于在行方向的畸变程度与在列方向的畸变程度是不同的，因此，图像处理器在逐个计算目标图像中的每个像素值时，需要在原始图像中乱序获取像素。而本申请的方案则仅需要从原始图像中逐行获取像素，从第一图像中逐列获取像素。因此，本申请所提出的方案能够提高图像处理装置的图像处理效率。
在一种可选的实施方式中，前述行变换参数存储于行映射关系表中，该行变换参数在该行映射关系表中的位置用于指示第一像素的坐标，其中，该第一像素是与前述行变换参数对应的像素。也可以理解为，该行变换参数在该行映射关系表中的位置用于指示基于该行变换参数确定的第一像素的坐标。
此外，该列变换参数存储于列映射关系表中，该列变换参数在该列映射关系表中的位置用于指示第二像素的坐标，其中，该第二像素是与前述列变换参数对应的像素。也可以理解为，该列变换参数在该列映射关系表中的位置用于表示基于该列变换参数确定的第二像素的坐标。
在一种可选的实施方式中，图像处理装置分别根据该原始图像中的每一行像素以及该每一行像素对应的行变换参数确定该第一图像中每个像素的像素值，包括：根据该行变换参数确定与该第一像素对应的该第一像素取值点；根据该第一像素取值点的横坐标确定至少一个第三像素，该第三像素与该第一像素取值点位于同一行；根据该至少一个第三像素的像素值确定该第一像素的像素值。图像处理装置该分别根据该第一图像中的每一列像素以及该每一列像素对应的列变换参数确定该目标图像中每个像素的像素值，包括：根据该 列变换参数确定与该第二像素对应的该第二像素取值点；根据该第二像素取值点的纵坐标确定至少一个第四像素，该第四像素与该第二像素取值点位于同一列；根据该至少一个第四像素的像素值确定该第二像素的像素值。
在一种可选的实施方式中，该第一像素取值点的横坐标为整数。该根据该第一像素取值点的横坐标确定至少一个第三像素，包括：确定该第一像素取值点为该第三像素。该根据该至少一个第三像素的像素值确定该第一像素的像素值，包括：将该第三像素的像素值赋值于该第一像素，得到该第一像素的像素值。
在一种可选的实施方式中，该第一像素取值点的横坐标为非整数。该根据该第一像素取值点的横坐标确定至少一个第三像素，包括：确定至少一个邻近该第一像素取值点的像素为该第三像素，该邻近该第一像素取值点的像素与该第一像素取值点位于同一行。该根据该至少一个第三像素的像素值确定该第一像素的像素值，包括：采用插值算法根据该第一像素取值点的横坐标以及该至少一个第三像素的横坐标，对该至少一个第三像素的像素值求第一加权平均值；将该第一加权平均值赋值于该第一像素，得到该第一像素的像素值。
其中，前述邻近该第一像素取值点的像素，可以理解为，是第一像素取值点左边一个像素和/或右边一个像素；也可以理解为，是左边第二个像素和/或右边第二个像素；甚至还可以理解为，是左边两个像素和/或右边两个像素。具体此处不做限定。
在一种可选的实施方式中，该第二像素取值点的纵坐标为整数。该根据该第二像素取值点的纵坐标确定至少一个第四像素，包括：确定该第二像素取值点为该第四像素。该根据该至少一个第四像素的像素值确定该第二像素的像素值，包括：将该第四像素的像素值赋值于该第二像素，得到该第二像素的像素值。
在一种可选的实施方式中，该第二像素取值点的纵坐标为非整数。该根据该第二像素取值点的纵坐标确定至少一个第四像素，包括：确定至少一个邻近该第二像素取值点的像素为该第四像素，该邻近该第二像素取值点的像素与该第二像素取值点位于同一列。该根据该至少一个第四像素的像素值确定该第二像素的像素值，包括：采用插值算法根据该第二像素取值点的纵坐标以及该至少一个第四像素的纵坐标，对该至少一个第四像素的像素值求第二加权平均值；将该第二加权平均值赋值于该第二像素，得到该第二像素的像素值。
在一种可选的实施方式中，图像处理装置获取该每一行像素对应的行变换参数，包括：从存储器中按照行方向顺序读取该每一行像素对应的行变换参数，该存储器中存储有行映射关系表，该行映射关系表包括该原始图像中每一行像素对应的该行变换参数。图像处理装置获取该每一列像素对应的列变换参数，包括：从存储器中按照列方向顺序读取该每一列像素对应的列变换参数，该存储器中存储有列映射关系表，该列映射关系表包括该原始图像中每一列像素对应的列变换参数。
本实施方式中，存储器中存储了两张映射关系表，包括从原始图像到第一图像(即仅行变换的图像)的行映射关系表，以及，从第一图像到目标图像的列映射关系表。而传统技术的方案中，使用的是从原始图像到目标图像的映射关系表，因此，传统技术的方案中每计算一个像素的像素值需要参考原始图像中的位于不完全同行且不完全同列的多个像素进行插值。
在一种可选的实施方式中，该分别根据该原始图像中的每一行像素以及该每一行像素对应的行变换参数进行行变换处理，得到该第一图像之后，该方法还包括：图像处理装置存储该第一图像的每一行的该第一像素至存储器中。该图像处理装置获取该第一图像的每一列像素的过程具体可以为：图像处理装置从该存储器中依次读取该第一图像的每一列像素。传统技术的方案中，图像处理过程不会出现前述第一图像，仅有原始图像和预畸变图像(即目标图像)。
在一种可选的实施方式中，该图像处理装置生成目标图像之后，该图像处理装置可以将该目标图像发送至该成像装置，该成像装置用于基于前述目标图像进行成像，以使得基于前述目标图像在成像平面投射出前述原始图像。其中，该成像装置可以是投影式成像***。
第二方面，本申请提供了一种图像处理装置，该图像处理装置可以是投影式成像***中的图像处理芯片或具有图像处理功能的处理器。该图像处理装置包括：获取模块和计算模块。其中，获取模块，用于获取原始图像的每一行像素，以及获取该每一行像素对应的行变换参数。其中，该行变换参数用于指示该原始图像中像素的横坐标与第一图像中像素的横坐标之间的对应关系，该第一图像为行方向存在预畸变而列方向不存在预畸变的图像。
计算模块，用于分别根据该原始图像中的每一行像素以及该每一行像素对应的行变换参数确定该第一图像中每个像素的像素值。此外，该获取模块，还用于获取该第一图像的每一列像素，以及获取该每一列像素对应的列变换参数。其中，该列变换参数用于指示该第一图像中像素的纵坐标与目标图像中像素的纵坐标之间的对应关系，该目标图像为行方向存在预畸变且列方向存在预畸变的图像。此外，该计算模块，还用于分别根据该第一图像中的每一列像素以及该每一列像素对应的列变换参数确定该目标图像中每个像素的像素值，该目标图像用于抵消成像装置的成像畸变以展示该原始图像。
在一种可选的实施方式中，该行变换参数包括与第一像素对应的第一像素取值点在该原始图像中的横坐标，该第一像素为该第一图像中的任一像素，该第一像素取值点用于确定该第一像素的像素值，该第一像素取值点在该原始图像中的纵坐标与该第一像素在该第一图像中的纵坐标相同。该列变换参数包括与第二像素对应的第二像素取值点在该第一图像中的纵坐标，该第二像素为该述目标图像中的任一像素，该第二像素取值点用于确定该第二像素的像素值，该第二像素取值点在该第一图像中的横坐标与该第二像素在该目标图像中的横坐标相同。
在一种可选的实施方式中，该行变换参数存储于行映射关系表中，该行变换参数在该行映射关系表中的位置用于表示基于该行变换参数确定的第一像素的坐标。也可以理解为，该行变换参数在该行映射关系表中的位置用于指示第一像素的坐标，其中，该第一像素是与前述行变换参数对应的像素。该列变换参数存储于列映射关系表中，该列变换参数在该列映射关系表中的位置用于指示第二像素的坐标，其中，该第二像素是与前述列变换参数对应的像素。该列变换参数在该列映射关系表中的位置用于表示基于该列变换参数确定的第二像素的坐标。
在一种可选的实施方式中，该计算模块，具体用于：根据该行变换参数确定与该第一 像素对应的该第一像素取值点；根据该第一像素取值点的横坐标确定至少一个第三像素，该第三像素与该第一像素取值点位于同一行；根据该至少一个第三像素的像素值确定该第一像素的像素值；根据该列变换参数确定与该第二像素对应的该第二像素取值点；根据该第二像素取值点的纵坐标确定至少一个第四像素，该第四像素与该第二像素取值点位于同一列；根据该至少一个第四像素的像素值确定该第二像素的像素值。
在一种可选的实施方式中，该图像处理装置还包括存储模块。该存储模块，用于存储该第一图像的每一行的第一像素至存储器中。
在一种可选的实施方式中，该行映射关系表和该列映射关系表存储于存储器中，该行映射关系表包括该原始图像中每一行像素对应的该行变换参数，该列映射关系表包括该原始图像中每一列像素对应的列变换参数。该获取模块，具体用于：从存储器中按照行方向顺序读取该每一行像素对应的行变换参数；从存储器中按照列方向顺序读取该每一列像素对应的列变换参数。
在一种可选的实施方式中，该图像处理装置还包括存储模块。该发送模块，用于将该目标图像发送至该成像装置，该成像装置用于基于前述目标图像进行成像，以使得基于前述目标图像在成像平面投射出前述原始图像。其中，该成像装置可以是投影式成像***。
第三方面，本申请提供了一种图像重映射方法，该图像重映射方法可以应用于基于投影式成像***的成像场景中。在该方法中，图像处理装置先进行行变换处理再进行列变换处理。具体地，该图像处理装置获取原始图像的每一列像素，并且，获取该每一列像素对应的列变换参数。其中，该列变换参数用于指示该原始图像中像素的纵坐标与第二图像中像素的纵坐标之间的对应关系。然后，该图像处理装置分别根据该原始图像中的每一列像素以及该每一列像素对应的列变换参数确定该第二图像中每个像素的像素值。此时，该图像处理装置获得了列方向存在预畸变而行方向不存在预畸变的图像(即前述第二图像)。然后，图像处理装置将获取该第二图像的每一行像素，并且，获取该每一行像素对应的行变换参数。其中，该行变换参数用于指示该第二图像中像素的横坐标与目标图像中像素的横坐标之间的对应关系。然后，该图像处理装置分别根据该第二图像中的每一行像素以及该每一行像素对应的行变换参数确定该目标图像中每个像素的像素值。此时，该图像处理装置获得了行方向存在预畸变且列方向存在预畸变的图像(即目标图像)，该目标图像用于抵消成像装置的成像畸变以展示该原始图像。
应当理解的是，图像处理装置获取原始图像的每一列像素，可以具体理解为，该图像处理装置读取到每一列像素中每一个像素的像素值和该像素在原始图像中的坐标。其中，该像素在原始图像中的坐标包含横坐标和纵坐标。前述图像处理装置每从原始图像中获取一列像素，相应的，该图像处理装置便获取这一列像素对应的列变换参数。然后，该图像处理装置根据前述一列像素和这一列像素对应的列变换参数便可以计算出第二图像中的一列像素。依次类推，该图像处理装置将依次计算出第二图像中的每一列像素。从原始图像到第二图像的过程可以称为列变换过程。类似的，图像处理装置获取第二图像的每一行像素，可以具体理解为，该图像处理装置读取到每一行像素中每一个像素的像素值和该像素在第二图像中的坐标。其中，该像素在第二图像中的坐标包含横坐标和纵坐标。前述图像 处理装置每从第二图像中获取一行像素，相应的，该图像处理装置便获取这一行像素对应的行变换参数。然后，该图像处理装置根据前述一行像素和这一行像素对应的行变换参数便可以计算出目标图像中的一行像素。依次类推，该图像处理装置将依次计算出目标图像中的每一行像素。从第二图像到目标图像的过程可以称为行变换过程。此时，该图像处理装置已经对前述原始图像在行方向和列方向上分别进行了预畸变处理(即前述行变换和列变换)，该目标图像是能够抵消投影式成像***的成像畸变的图像。
本申请中，图像处理装置在基于原始图像确定目标图像(即预畸变图像)时，将图像重映射分解为两次单方向的映射。具体地，该图像处理装置先对原始图像中的每一列像素进行列变换，再基于列变换结果(即第二图像)对其中的每一行像素进行行变换，得到目标图像。也就是说，该图像处理装置在计算第二图像中的一列像素时，仅需要顺序读取原始图像中的一列像素，而不需要跨列乱序读取；类似的，图像处理装置在计算目标图像的一行像素时，也仅需要顺序读取第二图像中的一行像素，而不需要跨行乱序读取。因此，图像处理装置在处理原始图像时可以有序读取，进而提高了图像处理装置处理原始图像的效率。
在一种可选的实施方式中，该列变换参数包括与第五像素对应的第五像素取值点在该原始图像中的纵坐标，该第五像素为该第二图像中的任一像素，该第五像素取值点用于确定该第五像素的像素值，该第五像素取值点在该原始图像中的横坐标与该第五像素在该第二图像中的横坐标相同。
本实施方式中，提出列变换参数可以用第五像素取值点在原始图像中的纵坐标表示，该第五像素取值点是与待求的第二图像中的第五像素对应的点，可以理解为，第五像素取值点在列方向发生畸变之后便落到了前述第五像素对应的位置，因此，在计算第五像素的像素值是需要查找第五像素取值点在原始图像中的位置的像素值。由于，第五像素和第五像素取值点之间仅是列方向存在畸变，因此，第五像素取值点在该原始图像中的横坐标与该第五像素在该第二图像中的横坐标相同。因此，前述图像处理装置在计算第二图像中的一列第五像素时，仅需要逐列获取原始图像中的一列像素。
在一种可选的实施方式中，该行变换参数包括与第六像素对应的第六像素取值点在该第二图像中的横坐标，该第六像素为该目标图像中的任一像素，该第六像素取值点用于确定该第六像素的像素值，该第六像素取值点在该第二图像中的纵坐标与该第六像素在该目标图像中的纵坐标相同。
本实施方式中，提出行变换参数可以用第六像素取值点在第二图像中的横坐标表示，该第六像素取值点是与待求的目标图像中的第六像素对应的点，可以理解为，第六像素取值点在行方向发生畸变之后便落到了前述第六像素对应的位置，因此，在计算第六像素的像素值是需要查找第六像素取值点在第二图像中的位置的像素值。由于，第六像素和第六像素取值点之间仅是行方向存在畸变，因此，第六像素取值点在该第二图像中的纵坐标与该第六像素在该目标图像中的纵坐标相同。因此，前述图像处理装置在计算目标图像中的一行第六像素时，仅需要逐行获取第二图像中的一行像素。
在一种可选的实施方式中，前述列变换参数存储于列映射关系表中，该列变换参数在 该列映射关系表中的位置用于表示基于该列变换参数确定的第五像素的坐标。也可以理解为，该列变换参数在该列映射关系表中的位置用于指示第五像素的坐标，其中，该第五像素是与前述列变换参数对应的像素。此外，该行变换参数存储于行映射关系表中，该行变换参数在该行映射关系表中的位置用于表示基于该行变换参数确定的第六像素的坐标。也可以理解为，该行变换参数在该行映射关系表中的位置用于指示第六像素的坐标，其中，该第六像素是与前述行变换参数对应的像素。
在一种可选的实施方式中，图像处理装置分别根据该原始图像中的每一列像素以及该每一列像素对应的列变换参数确定该第二图像中每个像素的像素值，包括：根据该列变换参数确定与该第五像素对应的该第五像素取值点；根据该第五像素取值点的纵坐标确定至少一个第七像素，该第七像素与该第五像素取值点位于同一列；根据该至少一个第七像素的像素值确定该第五像素的像素值。图像处理装置该分别根据该第二图像中的每一行像素以及该每一行像素对应的行变换参数确定该目标图像中每个像素的像素值，包括：根据该行变换参数确定与该第六像素对应的该第六像素取值点；根据该第六像素取值点的横坐标确定至少一个第八像素，该第八像素与该第六像素取值点位于同一行；根据该至少一个第八像素的像素值确定该第六像素的像素值。
在一种可选的实施方式中，该第五像素取值点的纵坐标为整数。该根据该第五像素取值点的纵坐标确定至少一个第七像素，包括：确定该第五像素取值点为该第七像素。该根据该至少一个第七像素的像素值确定该第五像素的像素值，包括：将该第七像素的像素值赋值于该第五像素，得到该第五像素的像素值。
在一种可选的实施方式中，该第五像素取值点的纵坐标为非整数。该根据该第五像素取值点的纵坐标确定至少一个第七像素，包括：确定至少一个邻近该第五像素取值点的像素为该第七像素，该邻近该第五像素取值点的像素与该第五像素取值点位于同一列。该根据该至少一个第七像素的像素值确定该第五像素的像素值，包括：采用插值算法根据该第五像素取值点的纵坐标以及该至少一个第七像素的纵坐标，对该至少一个第七像素的像素值求第一加权平均值；将该第一加权平均值赋值于该第五像素，得到该第五像素的像素值。
在一种可选的实施方式中，该第六像素取值点的横坐标为整数。该根据该第六像素取值点的横坐标确定至少一个第八像素，包括：确定该第六像素取值点为该第八像素。该根据该至少一个第八像素的像素值确定该第六像素的像素值，包括：将该第八像素的像素值赋值于该第六像素，得到该第六像素的像素值。
在一种可选的实施方式中，该第六像素取值点的横坐标为非整数。该根据该第六像素取值点的横坐标确定至少一个第八像素，包括：确定至少一个邻近该第六像素取值点的像素为该第八像素，该邻近该第六像素取值点的像素与该第六像素取值点位于同一横。该根据该至少一个第八像素的像素值确定该第六像素的像素值，包括：采用插值算法根据该第六像素取值点的横坐标以及该至少一个第八像素的横坐标，对该至少一个第八像素的像素值求第二加权平均值；将该第二加权平均值赋值于该第六像素，得到该第六像素的像素值。
在一种可选的实施方式中，图像处理装置获取该每一列像素对应的列变换参数，包括：从存储器中按照列方向顺序读取该每一列像素对应的列变换参数，该存储器中存储有列映 射关系表，该列映射关系表包括该原始图像中每一列像素对应的该列变换参数。图像处理装置获取该每一行像素对应的行变换参数，包括：从存储器中按照行方向顺序读取该每一行像素对应的行变换参数，该存储器中存储有行映射关系表，该行映射关系表包括该原始图像中每一行像素对应的行变换参数。
本实施方式中，存储器中存储了两张映射关系表，包括从原始图像到第二图像(即仅列变换的图像)的列映射关系表，以及，从第二图像到目标图像的行映射关系表。而传统技术的方案中，使用的是从原始图像到目标图像的映射关系表，因此，传统技术的方案中每计算一个像素的像素值需要参考原始图像中的位于不完全同行且不完全同列的多个像素进行插值。
在一种可选的实施方式中，该分别根据该原始图像中的每一列像素以及该每一列像素对应的列变换参数进行列变换处理，得到该第二图像之后，该方法还包括：图像处理装置存储该第二图像的每一列的该第五像素至存储器中。该图像处理装置获取该第二图像的每一行像素的过程具体可以为：图像处理装置从该存储器中依次读取该第二图像的每一行像素。传统技术的方案中，图像处理过程不会出现前述第二图像，仅有原始图像和预畸变图像(即目标图像)。
在一种可选的实施方式中，该图像处理装置生成目标图像之后，该图像处理装置可以将该目标图像发送至该成像装置，该成像装置用于基于前述目标图像进行成像，以使得基于前述目标图像在成像平面投射出前述原始图像。其中，该成像装置可以是投影式成像***。
第四方面，本申请提供了一种图像处理装置，该图像处理装置可以是投影式成像***中的图像处理芯片或具有图像处理功能的处理器。该图像处理装置包括：获取模块和计算模块。其中，获取模块，用于获取原始图像的每一列像素，并且，获取该每一列像素对应的列变换参数。其中，该列变换参数用于指示该原始图像中像素的纵坐标与第二图像中像素的纵坐标之间的对应关系。计算模块，用于分别根据该原始图像中的每一列像素以及该每一列像素对应的列变换参数确定该第二图像中每个像素的像素值。此外，获取模块，还用于获取该第二图像的每一行像素，并且，获取该每一行像素对应的行变换参数。其中，该行变换参数用于指示该第二图像中像素的横坐标与目标图像中像素的横坐标之间的对应关系。计算模块，还用于分别根据该第二图像中的每一行像素以及该每一行像素对应的行变换参数确定该目标图像中每个像素的像素值。此时，该图像处理装置获得了行方向存在预畸变且列方向存在预畸变的图像(即目标图像)，该目标图像用于抵消成像装置的成像畸变以展示该原始图像。
在一种可选的实施方式中，该列变换参数包括与第五像素对应的第五像素取值点在该原始图像中的纵坐标，该第五像素取值点用于确定该第二图像中的该第五像素的像素值，该第五像素取值点在该原始图像中的横坐标与该第五像素在该第二图像中的横坐标相同。
在一种可选的实施方式中，该行变换参数包括与第六像素对应的第六像素取值点在该第二图像中的横坐标，该第六像素取值点用于确定该目标图像中的该第六像素的像素值，该第六像素取值点在该第二图像中的纵坐标与该第六像素在该目标图像中的纵坐标相同。
在一种可选的实施方式中，前述列变换参数存储于列映射关系表中，该列变换参数在该列映射关系表中的位置用于表示基于该列变换参数确定的第五像素的坐标。该行变换参数存储于行映射关系表中，该行变换参数在该行映射关系表中的位置用于表示基于该行变换参数确定的第六像素的坐标。
在一种可选的实施方式中，该计算模块，具体用于：根据该列变换参数确定与该第五像素对应的该第五像素取值点；根据该第五像素取值点的纵坐标确定至少一个第七像素，该第七像素与该第五像素取值点位于同一列；根据该至少一个第七像素的像素值确定该第五像素的像素值。该计算模块，具体用于：根据该行变换参数确定与该第六像素对应的该第六像素取值点；根据该第六像素取值点的横坐标确定至少一个第八像素，该第八像素与该第六像素取值点位于同一行；根据该至少一个第八像素的像素值确定该第六像素的像素值。
在一种可选的实施方式中，该第五像素取值点的纵坐标为整数。该计算模块，具体用于：确定该第五像素取值点为该第七像素；将该第七像素的像素值赋值于该第五像素，得到该第五像素的像素值。
在一种可选的实施方式中，该第五像素取值点的纵坐标为非整数。该计算模块，具体用于：确定至少一个邻近该第五像素取值点的像素为该第七像素，该邻近该第五像素取值点的像素与该第五像素取值点位于同一列。该计算模块，具体用于：采用插值算法根据该第五像素取值点的纵坐标以及该至少一个第七像素的纵坐标，对该至少一个第七像素的像素值求第一加权平均值；将该第一加权平均值赋值于该第五像素，得到该第五像素的像素值。
在一种可选的实施方式中，该第六像素取值点的横坐标为整数。该计算模块，具体用于：确定该第六像素取值点为该第八像素；将该第八像素的像素值赋值于该第六像素，得到该第六像素的像素值。
在一种可选的实施方式中，该第六像素取值点的横坐标为非整数。该计算模块，具体用于：确定至少一个邻近该第六像素取值点的像素为该第八像素，该邻近该第六像素取值点的像素与该第六像素取值点位于同一横；采用插值算法根据该第六像素取值点的横坐标以及该至少一个第八像素的横坐标，对该至少一个第八像素的像素值求第二加权平均值；将该第二加权平均值赋值于该第六像素，得到该第六像素的像素值。
在一种可选的实施方式中，该图像处理装置还包括存储模块。该存储模块，用于存储该第二图像的每一列的第五像素至存储器中。
在一种可选的实施方式中，图像处理装置获取该每一列像素对应的列变换参数，包括：从存储器中按照列方向顺序读取该每一列像素对应的列变换参数，该存储器中存储有列映射关系表，该列映射关系表包括该原始图像中每一列像素对应的该列变换参数。该获取模块，具体用于：从存储器中按照行方向顺序读取该每一行像素对应的行变换参数；从存储器中按照列方向顺序读取该每一列像素对应的列变换参数。
在一种可选的实施方式中，该图像处理装置还包括存储模块。该发送模块，用于将该目标图像发送至该成像装置，该成像装置用于基于前述目标图像进行成像，以使得基于前 述目标图像在成像平面投射出前述原始图像。其中，该成像装置可以是投影式成像***。
第五方面，本申请提供了一种映射关系构建方法，该方法用于计算前述实施方法所介绍的行映射关系表和列映射关系表。该方法可以在前述图像处理装置中执行；也可以由其他的计算设备执行，并在计算出的行映射关系表和列映射关系表存储至前述图像处理装置)(或与图像处理装置连接的外部存储器)中。在该方法中，计算设备获取标定图像中每个标定点的坐标；获取畸变图像中每个畸变点的坐标，该畸变图像为经成像装置的成像畸变而获得的图像，该畸变图像中的畸变点与该标定图像中的标定点一一对应。然后，该计算设备根据该畸变点的坐标和该标定点的坐标，分别确定行映射关系表和列映射关系表。
可选的，该计算设备将前述行映射关系表和前述列映射关系表存储于存储器中。
在一种可选的实施方式中，该计算设备根据该畸变点的坐标和该标定点的坐标，分别确定行映射关系表和列映射关系表，包括：S1、固定纵坐标为畸变点纵坐标，以标定点的横坐标为自变量并以对应畸变点的横坐标为因变量构建行变换模型。S2、根据该行变换模型计算第三图像中每个像素在原始图像上的横坐标，得到该行映射关系表，该第三图像中每个像素的坐标为整数。S3、固定横坐标为标定点横坐标，以标定点的纵坐标为自变量并以对应畸变点的纵坐标为因变量构建列变换模型。S4、根据该列变换模型计算第四图像中每个像素在该第三图像上的纵坐标，得到该列映射关系表，该第四图像中每个像素的坐标均为整数，该第四图像为该标定图像的预畸变图像。
在一种可选的实施方式中，该根据该畸变点的坐标和该标定点的坐标，分别确定行映射关系表和列映射关系表，包括：T1、固定纵坐标为畸变点纵坐标，以标定点的纵坐标为自变量并以对应畸变点的纵坐标为因变量构建列变换模型。T2、根据该列变换模型计算第五图像中每个像素在原始图像上的纵坐标，得到该列映射关系表，该第五图像中每个像素的坐标为整数。T3、固定横坐标为标定点横坐标，以标定点的横坐标为自变量并以对应畸变点的横坐标为因变量构建行变换模型。T4、根据该行变换模型计算第六图像中每个像素在该第五图像上的横坐标，得到行映射关系表，该第六图像中每个像素的坐标均为整数，该第六图像为该标定图像的预畸变图像。
在一种可选的实施方式中，该根据该畸变点的坐标和该标定点的坐标，分别确定行映射关系表和列映射关系表，包括：以该畸变点的横坐标和纵坐标为自变量，且，该标定点的横坐标和纵坐标为因变量，构建畸变模型；根据该畸变模型计算第七图像中每个像素在该标定图像上的横坐标和纵坐标，得到畸变映射关系表，该第七图像中每个像素的横坐标和纵坐标均为整数；将该畸变映射关系表中的坐标沿水平方向拆分，得到该行映射关系表；将该畸变映射关系表中的坐标沿垂直方向拆分，得到该列映射关系表。
第六方面，本申请提供了一种计算设备，该计算设备用于计算前述实施方法所介绍的行映射关系表和列映射关系表。该计算设备包括：获取模块和确定模块。其中，获取模块，用于获取标定图像中每个标定点的坐标，以及获取畸变图像中每个畸变点的坐标。其中，该畸变图像为经成像装置的成像畸变而获得的图像，该畸变图像中的畸变点与该标定图像中的标定点一一对应。确定模块，用于根据该畸变点的坐标和该标定点的坐标，分别确定行映射关系表和列映射关系表。
可选的，该计算设备将前述行映射关系表和前述列映射关系表存储于存储器中。
在一种可选的实施方式中，该确定模块，具体用于：固定纵坐标为畸变点纵坐标，以标定点的横坐标为自变量并以对应畸变点的横坐标为因变量构建行变换模型。根据该行变换模型计算第三图像中每个像素在原始图像上的横坐标，得到该行映射关系表，该第三图像中每个像素的坐标为整数。固定横坐标为标定点横坐标，以标定点的纵坐标为自变量并以对应畸变点的纵坐标为因变量构建列变换模型。根据该列变换模型计算第四图像中每个像素在该第三图像上的纵坐标，得到该列映射关系表，该第四图像中每个像素的坐标均为整数，该第四图像为该标定图像的预畸变图像。
在一种可选的实施方式中，该确定模块，具体用于：固定纵坐标为畸变点纵坐标，以标定点的纵坐标为自变量并以对应畸变点的纵坐标为因变量构建列变换模型。根据该列变换模型计算第五图像中每个像素在原始图像上的纵坐标，得到该列映射关系表，该第五图像中每个像素的坐标为整数。固定横坐标为标定点横坐标，以标定点的横坐标为自变量并以对应畸变点的横坐标为因变量构建行变换模型。根据该行变换模型计算第六图像中每个像素在该第五图像上的横坐标，得到行映射关系表，该第六图像中每个像素的坐标均为整数，该第六图像为该标定图像的预畸变图像。
在一种可选的实施方式中，该确定模块，具体用于：以该畸变点的横坐标和纵坐标为自变量，且，该标定点的横坐标和纵坐标为因变量，构建畸变模型；根据该畸变模型计算第七图像中每个像素在该标定图像上的横坐标和纵坐标，得到畸变映射关系表，该第七图像中每个像素的横坐标和纵坐标均为整数；将该畸变映射关系表中的坐标沿水平方向拆分，得到该行映射关系表；将该畸变映射关系表中的坐标沿垂直方向拆分，得到该列映射关系表。
第七方面，本申请还提供了一种图像处理装置，该图像处理装置包括处理器和内部存储器，该处理器和该内部存储器耦合；该处理器，用于从外部存储器中读取行映射参数表和列映射参数表至该内部存储器中；该内部存储器还存储有程序，当该内部存储器存储的程序指令被该处理器执行时使得该图像处理装置实现如前述第一方面或第三方面任意一种实施方式所介绍的方法。
第八方面，本申请还提供了一种成像装置，包括图像处理装置和显示装置。其中，该图像处理装置用于按照如前述第一方面或第三方面任意一种实施方式所介绍的方法根据原始图像生成目标图像，并将该目标图像传输至该显示装置，以使得该成像装置投射出该原始图像。可选的，该成像装置为投影式成像***。
第九方面，本申请还提供了一种计算机可读存储介质，包括计算机程序，该计算机序被处理器执行以实现如前述第一方面或第三方面任意一种实施方式所介绍的方法。
第十方面，本申请还提供了一种包含指令的计算机程序产品，该计算机程序产品包括计算机程序代码，当该计算机程序代码在计算机上运行时，使得计算机执行如前述第一方面或第三方面任意一种实施方式所介绍的方法。
从以上技术方案可以看出，本申请实施例具有以下优点：
本申请实施例中，图像处理装置在基于原始图像确定目标图像(即预畸变图像)时， 将图像重映射分解为两次单方向的映射。具体地，该图像处理装置先对原始图像中的每一行像素进行行变换，再基于行变换结果(即第一图像)对其中的每一列像素进行列变换，得到目标图像；或，先对原始图像中的每一列像素进行列变换，再基于列变换结果对每一行像素进行行变换，得到该目标图像。也就是说，该图像处理装置在计算第一图像中的一行像素时，仅需要读取原始图像中的一行像素，而不需要跨行读取；类似的，图像处理装置在计算目标图像的一列像素时，也仅需要读取第一图像中的一列像素，而不需要跨列读取。因此，图像处理装置在处理原始图像时可以有序读取，进而提高了图像处理装置处理原始图像的效率。
附图说明
为了更清楚地说明本申请实施例的技术方案，下面将对实施例描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本申请的一些实施例。
图1A为投影式成像***的一个简易结构图；
图1B为本申请中原始图像的一个示例图；
图1C为本申请中畸变图像的一个示例图；
图2为本申请中图像重映射方法的一个流程图；
图3A为本申请中行变换过程的一个示例图；
图3B为本申请中列变换过程的一个示例图；
图3C为本申请中行变换和列变换过程的一个示例图；
图4为本申请中映射关系构建方法的一个流程图；
图5A为本申请中标定图像的示例图；
图5B为本申请中映射关系构建方法一个实施例示意图；
图6A为本申请中图像重映射方法的另一个流程图；
图6B为本申请中行变换和列变换过程的另一个示例图；
图7为本申请中图像处理装置的一个实施例示意图；
图8为本申请中图像处理装置的另一个实施例示意图。
具体实施方式
下面将结合本申请实施例中的附图，对本申请实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例仅仅是本申请一部分实施例，而不是全部的实施例。
本申请的说明书和权利要求书及上述附图中的术语“第一”、“第二”、“第三”、“第四”等(如果存在)是用于区别类似的对象，而不必用于描述特定的顺序或先后次序。应该理解这样使用的数据在适当情况下可以互换，以便这里描述的实施例能够以除了在这里图示或描述的内容以外的顺序实施。此外，术语“包括”和“具有”以及他们的任何变形，意图在于覆盖不排他的包含，例如，包含了一系列步骤或单元的过程、方法、***、产品或设备不必限于清楚地列出的那些步骤或单元，而是可包括没有清楚地列出的或对于这些过程、方法、产品或设备固有的其它步骤或单元。
本申请实施例提供了一种图像重映射方法以及图像处理装置，用于提高基于原始图像 生成预畸变图像的效率。
本申请的图像重映射方法主要应用于抵消基于投影式成像***的成像装置产生的畸变，下面将结合投影式成像***的结构对畸变产生的原理以及抵消畸变的原理进行介绍：
如图1A所示，为投影式成像***10的简易结构示意图。其中，该投影式成像***10包括光学模组101和显示装置102。其中，显示装置102用于产生需要投影成像的图像。该显示装置102发出的光线经过光学模组101之后到达成像平面11，在该成像平面11呈现现出图像。
在实际应用中，前述显示装置102可能为液晶显示器(liquid crystal display，LCD)、数字光处理(digital light processing，DLP)或者硅基液晶(liquid crystal on silicon，LCOS)等。前述光学模组101可能包括一个或多个透镜、自由曲面镜等。
但是，由于投影式成像***10在成像过程中难免会存在像差，因此，呈现在前述成像平面11上的图像不是显示装置102显示的图像(后文称为原始图像)，而是该基于前述图像而产生畸变的图像(后文称为畸变图像)。为了抵消前述畸变，需要将原始图像处理成与畸变图像具有相反畸变方向以及相同畸变程度的预畸变图像，然后，在显示装置102上产生前述预畸变图像，以使得显示装置102发出的光线经过光学模组101而到达成像平面11时，成像平面11上呈现出的是由预畸变图像经过畸变而获得的原始图像。由此，可以抵消因光学像差而造成的图像畸变。示例性的，若显示装置102生成的是如图1B所示的正常图像，那么，前述投影式成像***10投射到成像平面11的图像可能是如图1C所示的任意一种畸变图像。示例性的，若当显示装置102显示的原始图像为图1B所示的正常图像且成像平面11呈现的畸变图像是如图1C中的(a)所示的枕形畸变图像。此时，可以将前述原始图像(即图1B所示的正常图像)处理为如图1C中的(b)所示的桶形畸变图像，并由显示装置102生成前述桶形畸变图像。于是，成像平面11便可以呈现出如图1B所示的正常图像，该正常图像(即原始图像)是由桶形畸变图像(即预畸变图像)经过畸变而获得的。
对此，传统技术常采用二次线性插值的方式，根据原始图像中呈矩形排布的四个像素计算预畸变图像中的一个像素，以此循环求出预畸变图像中的每一个像素的像素值。但是，在此过程中，每计算预畸变图像中的一个像素便需要从原始图像读取不完全同行且不完全同列的至少四个像素。因此，需要执行大量的对原始图像的乱序访问，这将影响处理器的处理效率。
对此，本申请实施例提出的图像重映射方法不仅可以应用于校正基于投影式成像***的成像装置引起的畸变，还能够建立原始图像和预畸变图像之间分别在行和列两个方向上的映射关系，分两次单方向的映射处理，将前述原始图像处理为经行变换的图像(或经列变换的图像)以及经行变换和列变换之后的图像(或经列变换和行变换之后的图像)，得到预畸变图像。由于，不需要跨行或跨列访问数据，因此，能够提高处理器的处理效率。
应当理解的是，本申请实施例中图像重映射方法所能校正的畸变形式，可以是枕形畸变(如图1C中的(a)所示)、桶形畸变(如图1C中的(b)所示)、梯形畸变(如图1C中的(c)所示)、扇形畸变(如图1C中的(d)所示)以及平行四边形畸变(如图1C中的(e)所示)中任意一种形式的畸变，也可以是前述多种畸变形式中任意至少两种畸变形 式的组合或叠加，还可以是没有具体形状的畸变等。具体此处不做限定。
应当注意的是，前述基于前述投影式成像***的成像装置可以是平视显示器(head up display，HUD)(也被称为抬头显示器)，也可以是头盔显示器(head mounted display，HMD)(也被称为头戴式显示器)。其中，前述平视显示器HUD可以是航空飞机座舱***中的显示器，安装于座舱中央仪表板的上方，作为飞行员与飞机之间交换信息的窗口；前述平视显示器HUD也可以应用于车联网(vehicle to everything，V2X)领域，作为车载显示器，为驾驶员提供行车信息。前述头盔显示器HMD常应用于增强现实(augmented reality，AR)技术和虚拟现实(virtual reality，VR)技术中，例如，AR眼镜(也被称为AR显示器)和VR眼镜(也被称为VR显示器)。当成像装置采用前述任意一种实现方式时，接收前述成像装置投射的图像的成像平面为用户的视网膜。此外，该成像装置也可以是普通的投影仪，此时，接收该投影仪投射的图像的成像平面为幕布等图像接收装置。
还应当注意的是，执行前述图像重映射方法的图像处理装置可以包括位于前述平视显示器HUD或头盔显示器HMD中的图像处理芯片，例如，现场可编程逻辑门阵列(field programmable gate array，FPGA)、数字信号处理(digital signal process，DSP)、中央处理器(central processing unit，CPU)或图形处理器(graphics processing unit，GPU)等具有图像处理功能的芯片。
另外，前述显示装置102可以前述图像处理装置集成于一体，例如，显示装置102和图像处理装置均集成于前述平视显示器HUD(或头盔显示器HMD)中，该图像处理装置计算出的每一张预畸变图像可以立即输入前述显示装置102进行显示。前述显示装置102也可以与前述图像处理装置分布式布置，例如，图像处理装置外置于与前述平视显示器HUD(或头盔显示器HMD)连接的计算机中，该计算机能够将计算出的预畸变图像通过有线或无线的方式传输至前述平视显示器HUD(或头盔显示器HMD)中的显示装置102进行显示。
下面将基于前述场景对本申请提出的图像重映射方法进行介绍，在该方法中，如图2所示，若图像处理装置先对原始图像进行行变换，再对经行变换处理后的图像进行列变换，则该原始图像将执行如下步骤：
步骤201、获取原始图像的每一行像素。
其中，该原始图像为希望呈现在成像平面上的不存在畸变的图像。本申请提出的方案需要对该原始图像进行预畸变处理，以获得目标图像(即预畸变图像)。通过将前述目标图像输入至投影式成像***进行投影，以使得在成像平面上呈现出前述原始图像。
本实施例中，该图像处理装置可以通过数据传输接口或其他通信接口从外部存储器读取前述原始图像，或从其他外部设备接收前述原始图像。该图像处理装置可以将前述获取到的原始图像暂存至图像处理装置的内部存储器中，每计算一行像素便从该内部存储器中读取一行像素。具体此处不做限定。但是，无论采用前述任何一种方式获取原始图像，该图像处理装置会依次读取该原始图像的每一行像素，并且，在后续处理过程中也会依次地对每一行像素进行处理。
在一种可选的实施方式中，若图像处理装置从存储器(例如，双倍速率同步动态随机存储器(double data rate synchronous dynamic random access memory，DDR SDRAM)(简称DDR))获取原始图像中每一行像素，那么，该过程具体可以为图像处理装置从存储原始图像的起始地址按照行方向顺序读取每一行像素。具体地，依据存储器的位宽每次读取像素的个数不同，图像处理装置读取完一行像素所需执行的读操作的次数不同。例如，若位宽为256bit，一个三通道的像素为8×3＝24bit，那么一次可以读10个像素。若该原始图像为100×100的图像，那么，该图像处理装置仅需要10次读取操作便可以读取到该原始图像的一整行像素。
应当理解的是，图像处理装置获取原始图像的每一行像素，可以具体理解为，该图像处理装置读取到每一行像素中每一个像素的像素值和该像素在原始图像中的坐标。其中，该像素在原始图像中的坐标包含横坐标和纵坐标。
此外，若图像处理装置从存储器中获取原始图像的每一行像素，则图像处理装置获取原始图像中的每一行像素的过程具体可以为：图像处理装置从存储器中按行读取原始图像中的每一行像素，即读取某一行中连续的横坐标的像素。可选的，该图像处理装置可以按照行的序数顺序读取，例如，读完第一行的像素之后，再读取第二行的像素，然后再读第三行的像素；也可以按行的序数逆序读取，例如，读完第二行的像素之后，再读取第一行的像素。此外，图像处理装置可以从某一行的第一个像素开始读取，也可以从某一行的第N个像素开始读取。具体此处不做限定。
步骤202、获取该每一行像素对应的行变换参数。
本实施例中，步骤201和步骤202之间无时间先后顺序的限定。也就是说，图像处理装置可以先执行步骤201再执行步骤202，也可以先执行步骤202再执行步骤201，还可以同时执行前述步骤201和步骤202，具体此处不做限定。在后续实施例中，仅以图像处理装置可以先从原始图像中获取一行像素，再从获取该行像素对应的行变换参数为例进行介绍。
该行变换参数为用于对原始图像中的某一行像素进行行变换处理的参数。该行变换处理指像素在行方向上移动位置，也可以理解为，像素的横坐标发生了改变而纵坐标没有发生改变。为便于介绍，将由原始图像仅经过行变换处理而获得的图像称为第一图像。那么，该行变换参数用于指示该原始图像中像素的横坐标与第一图像中像素的横坐标之间的对应关系。还应当理解的是，像素的位置改变(即像素的移动)实质上是像素值的迁移。因此，行变换参数可以用于确定第一图像中的某个像素的像素值的取值来自于原始图像中的哪个像素。为便于介绍，称第一图像中任意一个像素为第一像素，该第一像素对应于在原始图像中的点为第一像素取值点。于是，该行变换参数包括与第一像素对应的第一像素取值点在该原始图像中的横坐标，该第一像素取值点用于确定第一图像中的第一像素的像素值，该第一像素取值点在该原始图像中的纵坐标与该第一像素在该第一图像中的纵坐标相同。
可选的，该行变换参数可以以行映射关系表的形式存储于存储器中，该行映射关系表包括多个行变换参数。
在一种可选的实施方式中，该行映射关系表中行变换参数的个数与前述第一图像中像 素的个数相等，并且，行映射关系表中的各个行变换参数与第一图像中各个像素之间按照坐标一一对应。该行变换参数在该行映射关系表中的位置用于表示基于该行变换参数确定的第一图像中的第一像素的坐标。示例性的，若第一图像为1080×1080的图像，则该原始图像对应的行映射表也为1080×1080。若行变换参数a1位于行映射关系表中的第二行第三列，那么，该行变换参数a1用于确定第一图像中的坐标为(3，2)的像素的像素值。
在另一种可选的实施方式中，可以将行映射关系表压缩存储于存储器中，即该存储器中存储的是压缩的行映射关系表。压缩的行映射关系表中行变换参数的个数小于前述第一图像中像素的个数。此时，前述图像处理装置获取该每一行像素对应的行变换参数，可以理解为是，读取压缩的行映射关系表并将压缩的该行映射关系表恢复为完整的行映射关系表，然后，再从前述完整的行映射关系表中读取与第一图像对应的一行行变换参数；也可以理解为，读取压缩的行映射关系表中的若干个行变换参数，然后，将从压缩的该行映射关系表中读取的若干个行变换参数恢复为一行行变换参数。前述恢复过程可以采用插值算法，具体此处不做限定。
应当注意的是，前述像素值是图像被数字化时由计算机赋予的值，代表了图像中某一像素的平均亮度信息。前述原始图像可以为单通道图像、三通道图像或四通道图像。其中，单通道图像也被称为灰度图像，每个像素由一个像素值表示。若单通道图像的像素值采用8位表示，则该单通道图像的像素值的取值范围为：0(黑)～255(白)。其中，三通道图像一般指的是RGB图像，该RGB图像可以呈现出色彩，也可以表示黑白图像。若三通道图像的像素值采用8位表示，则每个像素值由三个通道值表示，即像素值由红(0～255)、绿(0～255)和蓝(0～255)叠加表示。四通道图像则是在三通道图像的基础上增加了一个亮度通道以表示透明程度。若原始图像为单通道图像，该图像处理装置计算出的第一图像和目标图像也为单通道图像；若原始图像为三通道图像，该图像处理装置计算出的第一图像和目标图像也为三通道图像；具体此处不做限定。若该原始图像为三通道图像，则一个像素的像素值是由三个通道值组成的，在计算过程中需要对每个通道值进行相同的计算。
本实施例中，图像处理装置获取每一行像素对应的行变换参数的过程，具体可以为，图像处理装置从存储器中按行读取每一行像素对应的行变换参数，即读取某一行中连续的横坐标的行变换参数。可选的，该图像处理装置可以按照行的序数顺序读取，例如，读完第一行的行变换参数之后，再读取第二行的行变换参数，然后再读第三行的行变换参数；也可以按行的序数逆序读取，例如，读完第二行的行变换参数之后，再读取第一行的行变换参数。此外，图像处理装置可以从某一行的第一个行变换参数开始读取，也可以从某一行的第N个行变换参数开始读取。具体此处不做限定。
本实施例中，图像处理装置可以采用读取原始图像的方式读取该行映射关系表。也就是说，图像处理装置读取原始图像中的像素的顺序与图像处理装置读取行映射关系表中的行变换参数的顺序是一致的。示例性的，若图像处理装置从原始图像中第N行M列开始读取像素，则该图像处理装置也应当从行变换参数表中的第N行第M列开始读取行变换参数。
步骤203、分别根据该原始图像中的每一行像素以及该每一行像素对应的行变换参数确定第一图像中每个像素的像素值。
其中，该第一图像为行方向存在预畸变而列方向不存在预畸变的图像，也就是说，该第一图像为待求的由原始图像经过行变换处理的结果。
本实施例中，图像处理装置计算该第一图像的过程，可以理解为，该图像处理装置确定第一图像中每个像素的像素值的过程。由于，该第一图像中每个像素的位置(即坐标)是已知的，即第一图像中的各个像素的横坐标和纵坐标均为连续的整数。因此，根据该第一像素的横坐标和纵坐标即可查询到该第一图像的行变换参数，即根据第一像素的横坐标和纵坐标查询具有相同横坐标和纵坐标的行变换参数。
为便于理解，以图3A为例进行介绍。其中，原始图像31为4×4的图像(即四行四列的图像)，与该原始图像31对应的行映射关系表也为四行四列。在原始图像31中，每个像素都有着确定的像素值，采用s(i，j)表示，其中，i代表横坐标，j代表纵坐标。例如，s(1，1)代表横坐标为1纵坐标为1的像素的像素值。在第一图像33中，每个像素的像素值需要图像处理装置根据原始图像31和行映射关系表32计算获得。以第一图像33的第一行第一列的像素331为例，该像素331的横坐标为1，纵坐标为1，即该像素331位于第一图像中的第一行第一列。那么，与该像素331对应的行变换参数为行映射关系表32中的第一行第一列对应的行变换参数321。
然后，该图像处理装置将根据该行变换参数确定与该第一像素对应的第一像素取值点，并根据该第一像素取值点的横坐标确定至少一个第三像素。然后，该图像处理装置根据该至少一个第三像素的像素值确定该第一像素的像素值。其中，该第三像素与该第一像素取值点位于同一行。
其中，该第一像素取值点的横坐标可能为整数，也可能为非整数。
在一种可选的实施方式中，若该第一像素取值点的横坐标为整数，那么，该图像处理装置可以直接确定该第一像素取值点为该第三像素，然后，将该第三像素的像素值赋值于该第一像素，得到该第一像素的像素值。
依然以前述图3A为例，若第一像素为像素332，那么，与该第一像素对应的行变换参数为行变换参数322。此时，该行变换参数322的取值为4，该行变换参数332位于行映射关系表32的第二行，则该像素332的像素值由原始图像中第二行第四个像素的像素值(即横坐标为4纵坐标为2的像素值)确定。在这个示例中，4为第一像素取值点的横坐标，该第一像素取值点为原始图像中第二行第四个像素(即横坐标为4纵坐标为2的像素)；该第三像素直接由第一像素取值点确定，即该第三像素为该第一像素取值点。
在另一种可选的实施方式中，若该第一像素取值点的横坐标为非整数，那么，该图像处理装置需要在该非整数附近的确定一个或多个第三像素。具体地，该图像处理装置确定至少一个邻近该第一像素取值点的像素为该第三像素，该邻近该第一像素取值点的像素与该第一像素取值点位于同一行。然后，该图像处理装置采用插值算法根据该第一像素取值点的横坐标以及该至少一个第三像素的横坐标，对该至少一个第三像素的像素值求第一加权平均值，将该第一加权平均值赋值于该第一像素，得到该第一像素的像素值。
依然以前述图3A为例，若第一像素为像素331，那么，与该第一像素对应的行变换参数为行变换参数321。此时，该行变换参数321的取值为1.6，该行变换参数331位于行映 射关系表32的第一行。因此，该第一像素取值点为原始图像中坐标为(1.6，1)的点。此时，该图像处理装置需要确定邻近该第一像素取值点，且，与该第一像素取值点位于同一行的像素为第三像素。由原始图像31可知，点(1.6，1)位于像素311(横坐标为1纵坐标为1)与像素312(横坐标为2纵坐标为1)之间，则该图像处理装置可以确定像素311和像素312为第三像素。若像素311的像素值为100，像素312的像素值为120，采用插值算法，可以确定加权平均值＝112。在这个示例中，1.6为第一像素取值点的横坐标，该第一像素取值点为点(1.6，1)，第三像素为像素311和像素312。
在实际应用中，该图像处理装置可以确定第一像素取值点左边两个像素以及第一像素取值点右边两个像素为第三像素值，具体此处不做限定。另外，前述示例中列举的是常见的一维线性插值，在实际应用中，该图像处理装置还可以采用行方向上的双线性插值、多次非线性插值等插值方法，具体此处不做限定。
以此类推，该图像处理装置计算完第一图像中的每个像素之后，便可以获得前述第一图像。
本实施例中，该图像处理装置在计算第一图像中的某一个像素的像素值时仅需要原始图像中的与该像素位于同一行的一个或多个第三像素，而不需要位于其他行的像素。而传统技术中，若需要计算第一图像中的一个像素的像素值，需要用到原始图像中的位于两行的至少四个像素。因此，本申请的方案，有利于提高图像处理装置计算预畸变图像的效率。此外，由于，图像处理装置在计算时仅需要一行像素而不需要两行像素，因此，图像处理装置在读取存储器中的像素或行变换参数时，也不需要读取两行像素或两行行变换参数，避免了对存储器的乱序访问，减少了激活存储器的次数，解决了复杂重映射中的随机读写问题，降低了硬件实现难度。在并行计算目标图像时，并行的多个目标像素所需的原始图像素也是连续的，不要读取冗余的原始图像，降低了实现高并行计算的所需的内部存储器大小。
本实施例中，图像处理装置在执行了步骤203之后，执行步骤204之前，该图像处理装置还可以将前述第一图像存储于存储器中。应当理解的是，该图像处理装置可以每计算出第一图像的一行像素的像素值便将该行像素的像素值写入存储器中。该图像处理装置也可以将计算出的一行像素存入缓存中，待计算出多行像素之后，该图像处理装置将前述多行像素依次写入存储器中。具体此处不不做限定。
此外，该图像处理装置可以直接将每行像素的像素值按行写入存储器中，也可以将每行像素的像素值进行转置处理，将每行像素按列写入存储器中。具体此处不做限定。
步骤204、获取该第一图像的每一列像素。
其中，第一图像为由原始图像仅经过行变换处理而获得的图像。具体请参阅前述步骤202中的相关介绍，此处不再赘述。
具体地，该图像处理装置从存储器中依次读取前述第一图像的每一列像素。若该图像处理装置每行像素的像素值按行写入存储器中，则该图像处理装置将按列读取该存储器中的第一图像的每一列像素。若该图像处理装置将每行像素的像素值进行转置处理，并将每行像素按列写入存储器中，则该图像处理装置将按行读取该存储器中的第一图像的每一列 像素。
步骤205、获取该每一列像素对应的列变换参数。
本实施例中，步骤204和步骤205之间无时间先后顺序的限定。也就是说，图像处理装置可以先执行步骤204再执行步骤205，也可以先执行步骤205再执行步骤204，还可以同时执行前述步骤204和步骤205，具体此处不做限定。在后续实施例中，仅以图像处理装置可以先从第一图像中获取一列像素，再从获取该列像素对应的列变换参数为例进行介绍。
该列变换参数为用于对第一图像中的某一列像素进行列变换处理的参数。该列变换处理指像素在列方向上移动位置，也可以理解为，像素的纵坐标发生了改变而横坐标没有发生改变。由第一图像仅经过列变换处理而获得的图像称为目标图像，即基于原始图像的预畸变图像。将该目标图像输入至投影式成像***中，成像平面上将展示出由预畸变图像经过畸变而获得的原始图像。该列变换参数用于指示该第一图像中像素的纵坐标与目标图像中像素的纵坐标之间的对应关系。还应当理解的是，像素的位置改变(即像素的移动)实质上是像素值的迁移。因此，列变换参数可以用于确定目标图像中的某个像素的像素值的取值来自于第一图像中的哪个像素。为便于介绍，称目标图像中任意一个像素为第二像素，该第二像素对应于在第一图像中的点为第二像素取值点。于是，该列变换参数包括与第二像素对应的第二像素取值点在该第一图像中的纵坐标，该第二像素取值点用于确定目标图像中的第二像素的像素值，该第二像素取值点在该第一图像中的横坐标与该第二像素在该目标图像中的横坐标相同。
可选的，该列变换参数可以以列映射关系表的形式存储于存储器中，该列映射关系表包括多个列变换参数。
在一种可选的实施方式中，该列映射关系表中列变换参数的个数与目标图像中像素的个数相等，并且，列映射关系表中的各个列变换参数与目标图像中各个像素之间按照坐标一一对应。该列变换参数在该列映射关系表中的位置用于表示基于该列变换参数确定的目标图像中的第二像素的坐标。示例性的，若目标图像为1080×1080的图像，则该目标图像对应的列映射表也为1080×1080。若列变换参数b1位于列映射关系表中的第二行第三列，那么，该列变换参数b1用于确定目标图像中的坐标为(3，2)的像素的像素值。
在另一种可选的实施方式中，可以将列映射关系表压缩存储于存储器中，即该存储器中存储的是压缩的列映射关系表。压缩的列映射关系表中列变换参数的个数小于前述目标图像中像素的个数。此时，前述图像处理装置获取该每一列像素对应的列变换参数，可以理解为是，读取压缩的列映射关系表并将压缩的该列映射关系表恢复为完整的列映射关系表，然后，再从前述完整的列映射关系表中读取与目标图像对应的一列列变换参数；也可以理解为，读取压缩的列映射关系表中的若干个列变换参数，然后，将从压缩的该列映射关系表中读取的若干个列变换参数恢复为一列列变换参数。前述恢复过程可以采用插值算法，具体此处不做限定。
本实施例中，图像处理装置获取每一列像素对应的列变换参数的过程，具体可以为，图像处理装置从存储器中按列读取每一列像素对应的列变换参数，即读取某一列中连续的 纵坐标的列变换参数。可选的，该图像处理装置可以按照列的序数顺序读取，例如，读完第一列的行变换参数之后，再读取第二列的列变换参数，然后再读第三列的列变换参数；也可以按列的序数逆序读取，例如，读完第二列的列变换参数之后，再读取第一列的列变换参数。此外，图像处理装置可以从某一列的第一个列变换参数开始读取，也可以从某一列的第N个列变换参数开始读取。具体此处不做限定。
本实施例中，图像处理装置可以采用读取第一图像的方式读取该列映射关系表。也就是说，图像处理装置读取第一图像中的第一像素的顺序与图像处理装置读取列映射关系表中的列变换参数的顺序是一致的。示例性的，若图像处理装置从第一图像中第N行M列开始读取第一像素，则该图像处理装置也应当从列变换参数表中的第N行第M列开始读取列变换参数。
步骤206、分别根据该第一图像中的每一列像素以及该每一列像素对应的列变换参数确定目标图像中每个像素的像素值。
其中，该目标图像为用于抵消成像装置成像畸变的预畸变图像。将该目标图像输入至投影式成像***中，成像平面上将展示出由预畸变图像经过畸变而获得的原始图像。
本实施例中，图像处理装置计算该目标图像的过程，可以理解为，该图像处理装置确定目标图像中每个像素的像素值的过程。由于，该目标图像中每个像素的位置(即坐标)是已知的，即目标图像中的各个像素的横坐标和纵坐标均为连续的整数。因此，根据该第二像素的横坐标和纵坐标即可查询到该目标图像的列变换参数，即根据第二像素的横坐标和纵坐标查询具有相同横坐标和纵坐标的列变换参数。
为便于理解，以图3B为例进行介绍。其中，第一图像33为4×4的图像，与该第一图像33对应的列映射关系表也为四行四列。在第一图像33中，每个像素的像素值已由图像处理装置计算出。而目标图像35中的各个像素的像素值需要图像处理装置根据第一图像33和列映射关系表34计算获得以目标图像35的第一行第一个的像素351为例，该像素351的横坐标为1纵坐标为1，即该像素351位于目标图像中的第一行第一列。那么，与该像素351对应的列变换参数为列映射关系表34中的第一行第一列对应的列变换参数341。
然后，该图像处理装置将根据该列变换参数确定与该目标像素对应的第二像素取值点，并根据该第二像素取值点的纵坐标确定至少一个第四像素。然后，该图像处理装置根据该至少一个第四像素的像素值确定该目标像素的像素值。其中，该第四像素与该第二像素取值点位于同一行。
其中，该第二像素取值点的横坐标可能为整数，也可能为非整数。
在一种可选的实施方式中，若该第二像素取值点的横坐标为整数，那么，该图像处理装置可以直接确定该第二像素取值点为该第四像素，然后，将该第四像素的像素值赋值于该第二像素，得到该第二像素的像素值。
依然以前述图3B为例，若第二像素为像素352，那么，与该第二像素对应的列变换参数为列变换参数342。此时，该列变换参数342的取值为3，该列变换参数342位于列映射关系表34的第四列，则该像素342的像素值由第一图像33中第四列第三个像素的像素值(即横坐标为4纵坐标为3的像素值)确定。在这个示例中，3为第二像素取值点的纵横 坐标，该第二像素取值点为第一图像33中第四列第三个像素(即横坐标为4纵坐标为3的像素)；该第四像素直接由第二像素取值点确定，即该第四像素为该第二像素取值点。
在另一种可选的实施方式中，若该第二像素取值点的横坐标为非整数，那么，该图像处理装置需要在该非整数附近的确定一个或多个第四像素。具体地，该图像处理装置确定至少一个邻近该第二像素取值点的像素为该第四像素，该邻近该第二像素取值点的像素与该第二像素取值点位于同一行。然后，该图像处理装置采用插值算法根据该第二像素取值点的横坐标以及该至少一个第四像素的横坐标，对该至少一个第四像素的像素值求第二加权平均值，将该第二加权平均值赋值于该第二像素，得到该第二像素的像素值。
依然以前述图3B为例，若第二像素为像素351，那么，与该第二像素对应的列变换参数为列变换参数341。此时，该列变换参数341的取值为1.5，该列变换参数341位于列映射关系表34的第一列。因此，该第二像素取值点为第一图像33中坐标为(1，1.5)的点。此时，该图像处理装置需要确定邻近该第二像素取值点，且，与该第二像素取值点位于同一列的像素为第四像素。由第一图像33可知，点(1，1.5)位于像素331(横坐标为1纵坐标为1)与像素334(横坐标为1纵坐标为2)之间，则该图像处理装置可以确定像素331和像素334为第四像素。若像素331的像素值为110，像素334的像素值为120，采用插值算法，可以确定加权平均值＝115。在这个示例中，1.5为第二像素取值点的纵坐标，该第二像素取值点为点(1，1.5)，第四像素为像素331和像素334。
在实际应用中，该图像处理装置可以确定第二像素取值点上边两个像素以及第二像素取值点下边两个像素为第四像素值，具体此处不做限定。另外，前述示例中列举的是常见的一维线性插值，在实际应用中，该图像处理装置还可以采用行方向上的双线性插值、多次非线性插值等插值方法，具体此处不做限定。
以此类推，该图像处理装置计算完目标图像中的每个像素。
步骤207、将前述目标图像发送至成像装置。
其中，该成像装置用于基于前述目标图像进行成像，以使得基于前述目标图像在成像平面投射出前述原始图像。
本实施例中，步骤207为可选的步骤。
本实施例中，图像处理装置在基于原始图像确定目标图像(即预畸变图像)时，将图像重映射分解为两次单方向的映射。如图3C所示，该图像处理装置先对原始图像中的每一行像素进行行变换(即图3C中步骤①)，再基于行变换结果(即第一图像)对其中的每一列像素进行列变换(即图3C中步骤②)，得到目标图像。也就是说，该图像处理装置在计算第一图像中的一行像素时，仅需要读取原始图像中的一行像素，而不需要跨行读取；类似的，图像处理装置在计算目标图像的一列像素时，也仅需要读取第一图像中的一列像素，而不需要跨列读取。因此，图像处理装置在处理原始图像时可以有序读取，进而提高了图像处理装置处理原始图像的效率。
在采用本申请实施提出的图像重映射方法之前，需要先确定行映射关系表和列映射关系表。如图4所示，计算设备可以采用如下方式确定行映射关系表和列映射关系表：
步骤401a、获取标定图像中每个标定点的坐标。
其中，标定图像为用于测量成像装置(例如，前述平视显示器HUD、头盔显示器HMD以及投影仪等)成像畸变的图像，该标定图像中包含多个具有确定坐标的标定点。一般地，该标定图像中的多个标定点按照一定的几何形状排布于标定图像中。
示例性的，标定图像可以如图5A左侧所示的仅有标定点的图像，也可以是如图5A右侧所示的包含标准网格的图像，具体本申请不做限定。若该标定图像为包含标准网格的图像，则网格的顶角(即相邻网格之间的交叉点)便为标定点。
步骤401b、获取畸变图像中每个畸变点的坐标。
其中，该畸变图像为经成像装置的成像畸变而获得的图像，该畸变图像中的畸变点与该标定图像中的标定点一一对应。也就是说，标定图像中的某个标定点经过畸变之后成为了畸变图像中的某个畸变点。示例性的，如图5B所示，若标定图像为501，当该成像装置引起的畸变为扇形畸变时，该标定图像501对应的畸变图像可以如畸变图像502所示。该标定图像501中的标定点与畸变图像502中的畸变点一一对应。例如，坐标为(x，y)的a1点经过畸变之后，坐标变为(u，v)(即a2点)。计算预畸变模型即需要计算501到503的映射，但事实上只有501和502是可观测到的，503是未知的。由于502与501之间的映射和501与503之间的映射其实是等价映射，因此实际计算的时候，只需要需要计算502到501的映射(即畸变图像到标定图像的映射)，就可以得到501到503的映射，也就是参数表(例如，行映射关系表或列映射关系表)所需要的预畸变映射。
步骤402、根据该畸变点的坐标和该标定点的坐标，分别确定行映射关系表和列映射关系表。
若图像处理装置采用先行后列的方式确定目标图像时，计算设备确定行映射关系表和列映射关系表的步骤如下所示：
S1、固定纵坐标为畸变点纵坐标，以标定点的横坐标为自变量并以对应畸变点的横坐标为因变量构建行变换模型。
S2、根据该行变换模型计算第三图像中每个像素在原始图像上的横坐标，得到该行映射关系表，该第三图像中每个像素的坐标为整数。
S3、固定横坐标为标定点横坐标，以标定点的纵坐标为自变量并以对应畸变点的纵坐标为因变量构建列变换模型。
S4、根据该列变换模型计算第四图像中每个像素在该第三图像上的纵坐标，得到该列映射关系表，该第四图像中每个像素的坐标均为整数，该第四图像为该标定图像的预畸变图像。
示例性的，采用(x i,y i)表示标定点在原始图像上的坐标，采用(u i,v i)表示畸变点在畸变图像上的坐标。计算设备先计算仅对标定点进行行变换的畸变模型，即根据(x i,v i)到(u i,v i)的映射计算行变换下的模型F α1(x i,y i)。可选的，该计算设备对(x i,v i)内的区域进行重采样，在行变换图像有效范围内得到一组均匀的标定点(x i,y' i)，并根据前述模型F α1(x,y)计算标定点(x i,y' i)在原始图像中对应的坐标(u' i,y' i)＝(F α1(x i,y' i),y' i)。然后，计 算设备根据(x i,y' i)到(u' i,y' i)的映射计算行变换下的模型F α2(x,y)。然后，计算设备计算仅对标定点进行列变换的畸变模型，即根据(x i,y i)到(x i,v i)的映射计算列变换下的模型F β1(x,y)。然后，计算设备利用畸变模型计算目标图上每个点的在原图上的坐标，得到行映射关系表和列映射关系表：
M X(i,j)＝F α2(i,j)，M Y(i,j)＝F β1(i,j)，0≤i＜width，0≤i＜height；
其中，i表示目标图像中像素的横坐标；j表示目标图像中像素的纵坐标；width代表目标图像的宽度，可以理解为是目标图像中像素的横坐标的最大值；height代表目标图像高度，可以理解为是目标图像中像素的纵坐标的最大值。
在另一种可选的实施方式中，计算设备也可以通过对从原始图像到目标图像的映射关系表进行列变换，以使得能够近似获得前述行映射关系表和列映射关系表。具体地，可以采用如下公式进行拆分：
M X(i,j)＝M(i,jj,0)，M Y(i,j)＝M(i,j,1)，0≤i＜width，0≤i＜height；
其中，M(i,j,0)表示目标图像第i列第j行对应的原始图像中像素的横坐标；M(i,j,1)表示目标图像第i列第j行对应的原始图像中像素的纵坐标；jj＝arg min|M(i,k,1)-j|，其中，jj表示使得M(i,k,1)最接近j的k的取值，即分解后的行变换参数表的第i列第j行的参数(仅包括行变换参数)是原始参数表中第i列的参数对(既包括行变换参数也包括列变换参数)中的列变换参数与j最相近的参数对的行变换参数，即固定纵坐标并取横坐标。
应当注意的是，在一些实现图像重映射的方法中，会对映射关系表进行压缩，即并不储存完整的点到点的映射关系，在这种情况下依然可以通过本申请的方案得到分解的压缩的行映射关系表和列映射关系表。
相比于传统技术直接计算(x i,y i)到(u i,v i)的畸变映射，本申请是先计算(x i,v i)到(u i,v i)的映射，再计算(x i,y i)到(x i,v i)的映射，因此，本申请可以获得行映射关系表和列映射关系表。
若图像处理装置采用先列后行的方式确定目标图像时，确定行映射关系表和列映射关系表的步骤如下所示：
T1、固定纵坐标为畸变点纵坐标，以标定点的纵坐标为自变量并以对应畸变点的纵坐标为因变量构建列变换模型。
T2、根据该列变换模型计算第五图像中每个像素在原始图像上的纵坐标，得到该列映射关系表，该第五图像中每个像素的坐标为整数。
T3、固定横坐标为标定点横坐标，以标定点的横坐标为自变量并以对应畸变点的横坐标为因变量构建行变换模型。
T4、根据该行变换模型计算第六图像中每个像素在该第五图像上的横坐标，得到行映射关系表，该第六图像中每个像素的坐标均为整数，该第六图像为该标定图像的预畸变图像。
示例性的，采用(x i,y i)表示标定点在原始图像上的坐标，采用(u i,v i)表示畸变点在畸变图像上的坐标。计算设备先计算仅对标定点进行列变换的畸变模型，即根据(u i,y i)到 (u i,v i)的映射计算列变换下的模型F β2(x,y)。可选的，该计算设备对(u i,y i)内的区域进行重采样，在列变换图像有效范围内得到一组均匀的标定点(x' i,y i)，并根据前述模型F β2(x,y)计算标定点(x' i,y i)在原始图像中对应的坐标(x' i,v' i)＝(x' i,F β2(x' i,y' i))。然后，计算设备根据(x' i,y i)到(x' i,v' i)的映射计算列变换下的模型F β3(x,y)。然后，计算设备计算仅对标定点进行行变换的畸变模型，即根据(x i,y i)到(u i,y i)的映射计算行变换下的模型F α3(x,y)。然后，计算设备利用畸变模型计算目标图上每个点的在原图上的坐标，得到行映射关系表和列映射关系表：
M Y(i,j)＝F β3(i,j)，M X(i,j)＝F α3(i,j)，0≤i<width，0≤i<height；
其中，i表示目标图像中像素的横坐标；j表示目标图像中像素的纵坐标；width代表目标图像的宽度，可以理解为是目标图像中像素的横坐标的最大值；height代表目标图像高度，可以理解为是目标图像中像素的纵坐标的最大值。
在另一种可选的实施方式中，计算设备也可以通过对从原始图像到目标图像的映射关系表进行变换，以使得能够近似获得前述行映射关系表和列映射关系表。具体地，可以采用如下公式进行拆分：M X(i,j)＝M(ii,j,0)，M Y(i,j)＝M(i,j,1)，0≤i<width，0≤i<height；
其中，ii＝argmin|M(k,j,1)-i|，其中，ii表示使得M(k,j,1)最接近i的k的取值，即分解后的列变换参数表的第i列第j行的参数(仅包括列变换参数)是原始参数表中第j行的参数对(既包括行变换参数也包括列变换参数)中的行变换参数与i最相近的那个参数对的列坐标参数，即固定横坐标并取纵坐标。
步骤403、将该行映射关系表和该列映射关系表存储于存储器中。
本实施例中，待计算设备计算出前述行映射关系表和列映射关系表之后，该计算设备将前述行映射关系表和列映射关系表存储于存储器中。该存储器可以是投影式成像***中的存储器。该存储器相对于前述图像处理装置是外部存储器，前述图像处理装置中的内部存储器可以从存储有行映射关系表和列映射关系表的内部存储器中获取前述行映射关系表和列映射关系表。
下面将基于前述场景对本申请提出的图像重映射方法进行介绍，在该方法中，如图6A所示，若图像处理装置先对原始图像进行列变换，再对经列变换处理后的图像进行行变换，则该原始图像将执行如下步骤：
步骤601、获取原始图像的每一列像素。
其中，该原始图像的定义具体可以参阅前述步骤201中的介绍。
应当理解的是，图像处理装置获取原始图像的每一列像素，可以具体理解为，该图像处理装置读取到每一列像素中每一个像素的像素值和该像素在原始图像中的坐标。其中，该像素在原始图像中的坐标包含横坐标和纵坐标。
步骤602、获取该每一列像素对应的列变换参数。
本实施例中，步骤601和步骤602之间无时间先后顺序的限定。也就是说，图像处理 装置可以先执行步骤601再执行步骤602，也可以先执行步骤602再执行步骤601，还可以同时执行前述步骤601和步骤602，具体此处不做限定。在后续实施例中，仅以图像处理装置可以先从原始图像中获取一列像素，再从获取该列像素对应的列变换参数为例进行介绍。
其中，该列变换参数用于指示该原始图像中像素的纵坐标与第二图像(即列变换结果)中像素的纵坐标之间的对应关系。该第二图像为列方向存在预畸变而行方向不存在预畸变的图像，该第二图像的纵坐标为整数。
可选的，该列变换参数包括与第五像素对应的第五像素取值点在该原始图像中的纵坐标，该第五像素取值点用于确定该第二图像中的该第五像素的像素值，该第五像素取值点在该原始图像中的横坐标与该第五像素在该第二图像中的横坐标相同。
可选的，前述列变换参数存储于列映射关系表中，该列变换参数在该列映射关系表中的位置用于表示基于该列变换参数确定的第五像素的坐标。
本实施例中，图像处理装置获取每一列像素对应的列变换参数的过程，具体可以为，图像处理装置从存储器中按照列方向顺序读取每一列像素对应的列变换参数。具体地，依据存储器的位宽每次读取列变换参数的个数不同，读取一列列变换参数需要激活存储器的次数不同。本实施例中，图像处理装置可以采用读取原始图像的方式读取该列映射关系表。具体请参阅步骤201中的示例。
步骤603、分别根据该原始图像中的每一列像素以及该每一列像素对应的列变换参数确定第二图像中每个像素的像素值。
其中，该第二图像为列方向存在预畸变而行方向不存在预畸变的图像，也就是说，该第二图像为待求的由原始图像经过列变换处理的结果。
本实施例中，图像处理装置计算该第二图像的过程，可以理解为，该图像处理装置确定第二图像中每个像素的像素值的过程。由于，该第二图像中每个像素的位置(即坐标)是已知的，即第二图像中的各个像素的横坐标和纵坐标均为连续的整数。因此，根据该第五像素的横坐标和纵坐标即可查询到该第二图像的列变换参数，即根据第五像素的横坐标和纵坐标查询具有相同横坐标和纵坐标的列变换参数。
具体地，该图像处理装置先根据该列变换参数确定与该第五像素对应的该第五像素取值点；然后，根据该第五像素取值点的纵坐标确定至少一个第七像素，该第七像素与该第五像素取值点位于同一列；然后，根据该至少一个第七像素的像素值确定该第五像素的像素值。
在一种可选的实施方式中，若前述第五像素取值点的纵坐标为整数，则该图像处理装置直接确定该第五像素取值点为该第七像素，然后，将该第七像素的像素值赋值于该第五像素，得到该第五像素的像素值。
在一种可选的实施方式中，若前述第五像素取值点的纵坐标为非整数，则该图像处理装置确定至少一个邻近该第五像素取值点的像素为该第七像素，该邻近该第五像素取值点的像素与该第五像素取值点位于同一列；然后，采用插值算法根据该第五像素取值点的纵坐标以及该至少一个第七像素的纵坐标，对该至少一个第七像素的像素值求第一加权平均 值；将该第一加权平均值赋值于该第五像素，得到该第五像素的像素值。
具体地，与前述步骤203的内容类似，请参阅前述步骤203中的介绍。
步骤604、获取该第二图像的每一行像素。
其中，该第二图像为列方向存在预畸变而行方向不存在预畸变的图像，也就是说，该第二图像为待求的由原始图像经过列变换处理的结果。
具体地，该图像处理装置从存储器中依次读取前述第二图像的每一行像素。若该图像处理装置每列像素的像素值按列写入存储器中，则该图像处理装置将按行读取该存储器中的第二图像的每一行像素。若该图像处理装置将每列像素的像素值进行转置处理，并将每列像素按行写入存储器中，则该图像处理装置将按列读取该存储器中的第二图像的每一行像素。
步骤605、获取该每一行像素对应的行变换参数。
本实施例中，步骤604和步骤605之间无时间先后顺序的限定。也就是说，图像处理装置可以先执行步骤604再执行步骤605，也可以先执行步骤605再执行步骤604，还可以同时执行前述步骤604和步骤605，具体此处不做限定。在后续实施例中，仅以图像处理装置可以先从第二图像中获取一行像素，再从获取该行像素对应的行变换参数为例进行介绍。
其中，该行变换参数用于指示该第二图像中像素的纵坐标与目标图像中像素的纵坐标之间的对应关系。
可选的，该行变换参数包括与第六像素对应的第六像素取值点在该第二图像中的横坐标，该第六像素取值点用于确定该目标图像中的该第六像素的像素值，该第六像素取值点在该第二图像中的纵坐标与该第六像素在该目标图像中的纵坐标相同。
可选的，该行变换参数存储于行映射关系表中，该行变换参数在该行映射关系表中的位置用于表示基于该行变换参数确定的第六像素的坐标。
具体地，请参阅前述步骤205对应的介绍。
步骤606、分别根据该第二图像中的每一行像素以及该每一行像素对应的行变换参数确定每个目标图像的像素值。
其中，该目标图像为用于抵消成像装置成像畸变的该预畸变图像。
具体地，该图像处理装置根据该行变换参数确定与该第六像素对应的该第六像素取值点；根据该第六像素取值点的横坐标确定至少一个第八像素，该第八像素与该第六像素取值点位于同一行；根据该至少一个第八像素的像素值确定该第六像素的像素值。
在一种可选的实施方式中，若前述第六像素取值点的横坐标为整数，则图像处理装置确定该第六像素取值点为该第八像素，然后，将该第八像素的像素值赋值于该第六像素，得到该第六像素的像素值。
在一种可选的实施方式中，若前述第六像素取值点的横坐标为非整数，则图像处理装置确定至少一个邻近该第六像素取值点的像素为该第八像素，该邻近该第六像素取值点的像素与该第六像素取值点位于同一横；然后，采用插值算法根据该第六像素取值点的横坐标以及该至少一个第八像素的横坐标，对该至少一个第八像素的像素值求第二加权平均值； 将该第二加权平均值赋值于该第六像素，得到该第六像素的像素值。
具体地，请参阅前述步骤206对应的介绍。
步骤607、将前述目标图像发送至成像装置。
其中，该成像装置用于基于前述目标图像进行成像，以使得基于前述目标图像在成像平面投射出前述原始图像。
本实施例中，步骤607为可选的步骤。
本实施例中，图像处理装置在基于原始图像确定目标图像(即预畸变图像)时，将图像重映射分解为两次单方向的映射。如图6B所示，该图像处理装置先对原始图像中的每一列像素进行列变换(即图6B中步骤①)，再基于列变换结果(即第二图像)对其中的每一行像素进行行变换(即图6B中步骤②)，得到目标图像。也就是说，该图像处理装置在计算第二图像中的一列像素时，仅需要读取原始图像中的一列像素，而不需要跨列读取；类似的，图像处理装置在计算目标图像的一行像素时，也仅需要读取第二图像中的一行像素，而不需要跨行读取。因此，图像处理装置在处理原始图像时可以有序读取，进而提高了图像处理装置处理原始图像的效率。
如图7所示，为本申请实施例提供的一种图像处理装置70的结构示意图。该图像处理装置70可以位于平视显示器HUD(也被称为抬头显示器)、头盔显示器HMD或投影仪中。该图像处理装置可以是位于前述平视显示器HUD、头盔显示器HMD或投影仪中的图像处理芯片。前述图2和图6A对应的实施例中的步骤可以由该图像处理装置70执行。
该图像处理装置70包括至少一个处理器701和至少一个存储器702。前述处理器701和前述存储器702通过线路互联。应当理解的是，图7仅示出了一个处理器701和一个存储器702。
其中，该处理器701可以是现场可编程逻辑门阵列(field programmable gate array，FPGA)、数字信号处理(digital signal process，DSP)、中央处理器(central processing unit，CPU)或图形处理器(graphics processing unit，GPU)等具有图像处理功能的功能单元或功能模块。
另外，前述处理器701可以是一个单核处理器；也可以是一个多核处理器，例如，该处理器701可以是由多个FPGA或多个DSP组成。此时，该处理器701可以同时对多张图像进行处理。例如，若处理器701中包含FPGA1和FPGA2两个FPGA，则由FPGA1在对图像1进行行变换处理之后，将行变换结果发送至FPGA2，由FPGA2对图像1的行变换结果进行列变换处理。在该FPGA2对前述图像1的行变换结果进行列变换处理的同时，该FPGA1可以继续对下一个图像(例如，图像2)进行行变换处理，然后，将图像2的行变换结果发送至FPGA2，由FPGA2对图像2的行变换结果进行列变换处理。
此外，处理器701可以指一个或多个装置、电路和/或用于处理数据(例如计算机程序指令)的处理核。该处理器701可以是个单独的半导体芯片，也可以跟其他电路一起集成为一个半导体芯片，例如，可以跟其他电路(如编解码电路、硬件加速电路或各种总线和接口电路)构成一个片上***(system-on-a-chip，SoC)，或者也可以作为一个特殊应用集 成电路(application specific integrated circuit，ASIC)的内置处理器集成在所述ASIC当中，该集成了处理器的ASIC可以单独封装或者也可以跟其他电路封装在一起。
此外，前述存储器702可以是只读存储器(read-only memory，ROM)，也可以是可存储静态信息和指令的其他类型的静态存储设备，也可以是随机存取存储器(random access memory，RAM)，也可以是可存储信息和指令的其他类型的动态存储设备，还可以是电可擦可编程只读存储器(electrically erasable programmable read-only memory，EEPROM)，具体此处不做限定。示例性的，前述存储器702可以为双倍速率同步动态随机存储器DDR SDRAM)(简称DDR)。该存储器702可以是独立存在，但与前述处理器701相连。可选的，该存储器702也可以和前述处理器701集成于一体。例如，集成于一个或多个芯片之内。
此外，该存储器702可以理解为是图像处理装置70的内部存储器，该存储器702用于存储从外部存储器读取的行映射关系表、列映射关系表以及原始图像。该存储器702还用于存储计算目标图像过程中产生的中间图像。示例性的，基于原始图像生成的行变换结果(例如，前述第一图像)，或，基于原始图像生成的列变换结果(例如，前述第二图像)。该存储器702还用于存储执行本申请实施例的技术方案的程序代码。前述程序代码可以由处理器701来控制执行，被执行的各类计算机程序代码也可被视为是处理器701的驱动程序。
可选的，该图像处理装置70还包括通信接口703，该通信接口703用于外部存储器或其他设备进行通信。图像处理装置70可以通过该通信接口703接收来自其他设备的指令或数据。示例性的，该图像处理装置70可以通过前述通信接口703从外部存储器中读取行映射关系表、列映射关系表以及原始图像。
示例性的，前述处理器701可以从外部存储器或内部存储器中获取原始图像的每一行像素，以及该每一行像素对应的行变换参数。然后，该处理器701分别根据该原始图像中的每一行像素以及该每一行像素对应的行变换参数确定该第一图像中每个像素的像素值，得到第一图像。然后，该处理器701将前述第一图像存储于内部存储器中。然后，该处理器再从前述内部存储器中获取该第一图像的每一列像素，并且，从外部存储器或内部存储器中获取每一列像素对应的列变换参数。然后，处理器701分别根据该第一图像中的每一列像素以及该每一列像素对应的列变换参数确定该目标图像中每个像素的像素值，得到目标图像。可选的，该处理器701将前述目标图像存储至外部存储器中。
可选的，该处理器701根据该行变换参数确定与该第一像素对应的该第一像素取值点；然后，根据该第一像素取值点的横坐标确定至少一个第三像素，该第三像素与该第一像素取值点位于同一行；然后，根据该至少一个第三像素的像素值确定该第一像素的像素值。
可选的，该处理器701根据该列变换参数确定与该第二像素对应的该第二像素取值点；然后，根据该第二像素取值点的纵坐标确定至少一个第四像素，该第四像素与该第二像素取值点位于同一列；然后，根据该至少一个第四像素的像素值确定该第二像素的像素值。
示例性的，前述处理器701可以从外部存储器或内部存储器中获取原始图像的每一列像素，以及该每一列像素对应的列变换参数。然后，该处理器701分别根据该原始图像中 的每一列像素以及该每一列像素对应的列变换参数确定该第二图像中每个像素的像素值，得到第二图像。然后，该处理器701将前述第二图像存储于内部存储器中。然后，该处理器再从前述内部存储器中获取该第二图像的每一行像素，并且，从外部存储器或内部存储器中获取每一行像素对应的行变换参数。然后，处理器701分别根据该第二图像中的每一行像素以及该每一行像素对应的行变换参数确定该目标图像中每个像素的像素值，得到目标图像。可选的，该处理器701将前述目标图像存储至外部存储器中。
可选的，该处理器701根据该列变换参数确定与该第五像素对应的该第五像素取值点；根据该第五像素取值点的纵坐标确定至少一个第七像素，该第七像素与该第五像素取值点位于同一列；根据该至少一个第七像素的像素值确定该第五像素的像素值。
可选的，该处理器701根据该行变换参数确定与该第六像素对应的该第六像素取值点；根据该第六像素取值点的横坐标确定至少一个第八像素，该第八像素与该第六像素取值点位于同一行；根据该至少一个第八像素的像素值确定该第六像素的像素值。
本实施例中，处理器701在基于原始图像确定目标图像(即预畸变图像)时，将图像重映射分解为两次单方向的映射。具体地，处理器701先对原始图像中的每一行像素进行行变换，再基于行变换结果(即第一图像)对其中的每一列像素进行列变换，得到目标图像；或，先对原始图像中的每一列像素进行列变换，再基于列变换结果对每一行像素进行行变换，得到该目标图像。也就是说，处理器701在计算第一图像中的一行像素时，仅需要读取原始图像中的一行像素，而不需要跨行读取；类似的，处理器701在计算目标图像的一列像素时，也仅需要读取第一图像中的一列像素，而不需要跨列读取。因此，图像处理装置在处理原始图像时可以有序读取，进而提高了图像处理装置处理原始图像的效率。
另外，由于该处理器701从存储器702读取行映射关系表、列映射关系表、原始图像、第一图像以及第二图像时，该处理器701无需跨行乱序读取，减少了存储器702中存储单元被激活的次数，提高了存储器702的使用寿命。
其余可以参考图2和图6A对应实施例中图像重映射方法，此处不再赘述。
本申请实施例中还提供了一种计算机可读存储介质，该计算机可读存储介质中存储有行映射关系表、列映射关系表，以及用于生成目标图像的程序，当前述计算机程序在计算机上运行时，可以使得计算机执行如前述图2或图6A所示实施例介绍的方法。
在一种可能的实现方式中，前述图7所示的图像处理装置70为芯片，该芯片也被称为数字处理芯片。该芯片包括处理单元和通信单元。其中，处理单元通过通信单元获取程序指令，该程序指令被处理单元执行，以使得该处理单元执行前述图2或图6A对应实施例介绍的方法步骤。具体地，该处理单元为集成了上述处理器701或者用于实现上述处理器701的功能的电路，该通信单元为集成了上述通信接口703或者用于实现上述通信接口703的功能的电路或接口。
可选的，当该芯片中集成了存储单元时，该存储单元可以是存储器等存储装置。此时，该芯片中的处理单元可以从该存储单元中调用程序代码以实现前述图2或图6A对应实施例所介绍的方法步骤。当该芯片未集成存储单元时，该芯片可以通过前述通信单元与外置 的存储器等存储装置连接，以使得从前述外置的存储器中获取程序代码以实现前述图2或图6A对应实施例所介绍的方法步骤。
如图8所示，为本申请实施例提供了一种图像处理装置80的结构示意图。前述图2和图6A对应的方法实施例均可以基于图8所示的图像处理装置80的结构。
该图像处理装置80包括多个功能模块，前述各个功能模块可以集成在一个处理单元中，也可以是各个模块单独物理存在，也可以两个或两个以上模块集成在一个单元中。上述集成的单元既可以采用硬件的形式实现，也可以采用软件功能单元的形式实现。
具体地，该图像处理装置80可以位于平视显示器HUD(也被称为抬头显示器)或头盔显示器HMD中。该图像处理装置可以是位于前述平视显示器HUD或头盔显示器HMD中的图像处理芯片。前述图2和图6A对应的实施例中的步骤可以由该图像处理装置80执行。具体地，该图像处理装置80包括获取模块801和计算模块802。
示例性的，获取模块801，用于获取原始图像的每一行像素，以及，获取该每一行像素对应的行变换参数。其中，该行变换参数用于指示该原始图像中像素的横坐标与第一图像中像素的横坐标之间的对应关系，该第一图像为行方向存在预畸变而列方向不存在预畸变的图像。
示例性的，计算模块802，用于分别根据该原始图像中的每一行像素以及该每一行像素对应的行变换参数确定该第一图像中每个像素的像素值。
示例性的，该获取模块801，还用于获取该第一图像的每一列像素，以及，获取该每一列像素对应的列变换参数，该列变换参数用于指示该第一图像中像素的纵坐标与目标图像中像素的纵坐标之间的对应关系，该目标图像为行方向存在预畸变且列方向存在预畸变的图像。
示例性的，该计算模块802，还用于分别根据该第一图像中的每一列像素以及该每一列像素对应的列变换参数确定该目标图像中每个像素的像素值，该目标图像用于抵消成像装置的成像畸变以展示该原始图像。
在一种可选的实施方式中，该计算模块802，具体用于：根据该行变换参数确定与该第一像素对应的该第一像素取值点；根据该第一像素取值点的横坐标确定至少一个第三像素，该第三像素与该第一像素取值点位于同一行；根据该至少一个第三像素的像素值确定该第一像素的像素值。
在一种可选的实施方式中，根据该列变换参数确定与该第二像素对应的该第二像素取值点；根据该第二像素取值点的纵坐标确定至少一个第四像素，该第四像素与该第二像素取值点位于同一列；根据该至少一个第四像素的像素值确定该第二像素的像素值。
在一种可选的实施方式中，该图像处理装置80还包括存储模块803；
该存储模块，用于存储该第一图像的每一行的该第一像素至存储器中。
在一种可选的实施方式中，该获取801模块，具体用于：
从存储器中按照行方向顺序读取该每一行像素对应的行变换参数；
从存储器中按照列方向顺序读取该每一列像素对应的列变换参数。
在一种可选的实施方式中，该图像处理装置80还包括发送模块804；
该发送模块804，用于将该目标图像发送至该成像装置，该成像装置用于基于前述目标图像进行成像，以使得基于前述目标图像在成像平面投射出前述原始图像。其中，该成像装置可以是投影式成像***。
其余可以参考图2或图6A对应实施例中图像处理装置的方法，此处不再赘述。
在实现过程中，上述方法的各步骤可以通过处理器中的硬件的集成逻辑电路或者软件形式的指令完成。结合本申请实施例所公开的方法的步骤可以直接体现为硬件处理器执行完成，或者用处理器中的硬件及软件模块组合执行完成。软件模块可以位于随机存储器，闪存、只读存储器，可编程只读存储器或者电可擦写可编程存储器、寄存器等本领域成熟的存储介质中。该存储介质位于存储器，处理器读取存储器中的信息，结合其硬件完成上述方法的步骤。为避免重复，这里不再详细描述。还应理解，本文中涉及的第一、第二、第三、第四以及各种数字编号仅为描述方便进行的区分，并不用来限制本申请实施例的范围。
应理解，本文中术语“和/或”，仅仅是一种描述关联对象的关联关系，表示可以存在三种关系，例如，A和/或B，可以表示：单独存在A，同时存在A和B，单独存在B这三种情况。另外，本文中字符“/”，一般表示前后关联对象是一种“或”的关系。
应理解，在本申请的各种实施例中，上述各过程的序号的大小并不意味着执行顺序的先后，各过程的执行顺序应以其功能和内在逻辑确定，而不应对本申请实施例的实施过程构成任何限定。
所属领域的技术人员可以清楚地了解到，为描述的方便和简洁，上述描述的***，装置和单元的具体工作过程，可以参考前述方法实施例中的对应过程，在此不再赘述。
以上实施例仅用以说明本申请的技术方案，而非对其限制；尽管参照前述实施例对本申请进行了详细的说明，本领域的普通技术人员应当理解：其依然可以对前述各实施例所记载的技术方案进行修改，或者对其中部分技术特征进行等同替换；而这些修改或者替换，并不使相应技术方案的本质脱离本申请各实施例技术方案的精神和范围。

Claims (16)

  1. 一种图像重映射方法，其特征在于，包括：
    获取原始图像的每一行像素；
    获取所述每一行像素对应的行变换参数，所述行变换参数用于指示所述原始图像中像素的横坐标与第一图像中像素的横坐标之间的对应关系，所述第一图像为行方向存在预畸变而列方向不存在预畸变的图像；
    分别根据所述原始图像中的每一行像素以及所述每一行像素对应的行变换参数确定所述第一图像中每个像素的像素值；
    获取所述第一图像的每一列像素；
    获取所述每一列像素对应的列变换参数，所述列变换参数用于指示所述第一图像中像素的纵坐标与目标图像中像素的纵坐标之间的对应关系，所述目标图像为行方向存在预畸变且列方向存在预畸变的图像；
    分别根据所述第一图像中的每一列像素以及所述每一列像素对应的列变换参数确定所述目标图像中每个像素的像素值，所述目标图像用于抵消成像装置的成像畸变以展示所述原始图像。
  2. 根据权利要求1所述的方法，其特征在于，
    所述行变换参数包括与第一像素对应的第一像素取值点在所述原始图像中的横坐标，所述第一像素为所述第一图像中的任一像素，所述第一像素取值点用于确定所述第一像素的像素值，所述第一像素取值点在所述原始图像中的纵坐标与所述第一像素在所述第一图像中的纵坐标相同；
    所述列变换参数包括与第二像素对应的第二像素取值点在所述第一图像中的纵坐标，所述第二像素为所述目标图像中的任一像素，所述第二像素取值点用于确定所述第二像素的像素值，所述第二像素取值点在所述第一图像中的横坐标与所述第二像素在所述目标图像中的横坐标相同。
  3. 根据权利要求2所述的方法，其特征在于，
    所述行变换参数存储于行映射关系表中，所述行变换参数在所述行映射关系表中的位置用于指示所述第一像素的坐标；
    所述列变换参数存储于列映射关系表中，所述列变换参数在所述列映射关系表中的位置用于指示所述第二像素的坐标。
  4. 根据权利要求2或3所述的方法，其特征在于，
    所述分别根据所述原始图像中的每一行像素以及所述每一行像素对应的行变换参数确定所述第一图像中每个像素的像素值，包括：
    根据所述行变换参数确定与所述第一像素对应的所述第一像素取值点；
    根据所述第一像素取值点的横坐标确定至少一个第三像素，所述第三像素与所述第一像素取值点位于同一行；
    根据所述至少一个第三像素的像素值确定所述第一像素的像素值；
    所述分别根据所述第一图像中的每一列像素以及所述每一列像素对应的列变换参数确 定所述目标图像中每个像素的像素值，包括：
    根据所述列变换参数确定与所述第二像素对应的所述第二像素取值点；
    根据所述第二像素取值点的纵坐标确定至少一个第四像素，所述第四像素与所述第二像素取值点位于同一列；
    根据所述至少一个第四像素的像素值确定所述第二像素的像素值。
  5. 根据权利要求1至4中任意一项所述的方法，其特征在于，所述分别根据所述原始图像中的每一行像素以及所述每一行像素对应的行变换参数确定所述第一图像中每个像素的像素值之后，所述方法还包括：
    存储所述第一图像的每一行的所述第一像素。
  6. 根据权利要求1至5中任意一项所述的方法，其特征在于，所述方法还包括：
    将所述目标图像发送至所述成像装置，所述成像装置用于基于所述目标图像进行成像，以使得基于所述目标图像在成像平面投射出所述原始图像。
  7. 一种图像处理装置，其特征在于，包括：
    获取模块，用于获取原始图像的每一行像素；
    所述获取模块，还用于获取所述每一行像素对应的行变换参数，所述行变换参数用于指示所述原始图像中像素的横坐标与第一图像中像素的横坐标之间的对应关系，所述第一图像为行方向存在预畸变而列方向不存在预畸变的图像；
    计算模块，用于分别根据所述原始图像中的每一行像素以及所述每一行像素对应的行变换参数确定所述第一图像中每个像素的像素值；
    所述获取模块，还用于获取所述第一图像的每一列像素；
    所述获取模块，还用于获取所述每一列像素对应的列变换参数，所述列变换参数用于指示所述第一图像中像素的纵坐标与目标图像中像素的纵坐标之间的对应关系，所述目标图像为行方向存在预畸变且列方向存在预畸变的图像；
    所述计算模块，还用于分别根据所述第一图像中的每一列像素以及所述每一列像素对应的列变换参数确定所述目标图像中每个像素的像素值，所述目标图像用于抵消成像装置的成像畸变以展示所述原始图像。
  8. 根据权利要求7所述的图像处理装置，其特征在于，
    所述行变换参数包括与第一像素对应的第一像素取值点在所述原始图像中的横坐标，所述第一像素为所述第一图像中的任一像素，所述第一像素取值点用于确定所述第一像素的像素值，所述第一像素取值点在所述原始图像中的纵坐标与所述第一像素在所述第一图像中的纵坐标相同；
    所述列变换参数包括与第二像素对应的第二像素取值点在所述第一图像中的纵坐标，所述第二像素为所述目标图像中的任一像素，所述第二像素取值点用于确定所述第二像素的像素值，所述第二像素取值点在所述第一图像中的横坐标与所述第二像素在所述目标图像中的横坐标相同。
  9. 根据权利要求8所述的图像处理装置，其特征在于，
    所述行变换参数存储于行映射关系表中，所述行变换参数在所述行映射关系表中的位 置用于指示所述第一像素的坐标；
    所述列变换参数存储于列映射关系表中，所述列变换参数在所述列映射关系表中的位置用于指示所述第二像素的坐标。
  10. 根据权利要求8或9所述的图像处理装置，其特征在于，
    所述计算模块，具体用于：
    根据所述行变换参数确定与所述第一像素对应的所述第一像素取值点；
    根据所述第一像素取值点的横坐标确定至少一个第三像素，所述第三像素与所述第一像素取值点位于同一行；
    根据所述至少一个第三像素的像素值确定所述第一像素的像素值；
    根据所述列变换参数确定与所述第二像素对应的所述第二像素取值点；
    根据所述第二像素取值点的纵坐标确定至少一个第四像素，所述第四像素与所述第二像素取值点位于同一列；
    根据所述至少一个第四像素的像素值确定所述第二像素的像素值。
  11. 根据权利要求7至10中任意一项所述的图像处理装置，其特征在于，所述图像处理装置还包括存储模块；
    所述存储模块，用于存储所述第一图像的每一行的所述第一像素至存储器中。
  12. 根据权利要求7至11中任意一项所述的图像处理装置，其特征在于，所述行映射关系表和所述列映射关系表存储于存储器中，所述行映射关系表包括所述原始图像中每一行像素对应的所述行变换参数，所述列映射关系表包括所述原始图像中每一列像素对应的列变换参数；
    所述获取模块，具体用于：
    从存储器中按照行方向顺序读取所述每一行像素对应的行变换参数；
    从存储器中按照列方向顺序读取所述每一列像素对应的列变换参数。
  13. 根据权利要求7至12中任意一项所述的图像处理装置，其特征在于，所述图像处理装置还包括发送模块；
    所述发送模块，用于将所述目标图像发送至所述成像装置，所述成像装置用于基于所述目标图像进行成像，以使得基于所述目标图像在成像平面投射出所述原始图像。
  14. 一种图像处理装置，其特征在于，包括：
    处理器和内部存储器，所述处理器和所述内部存储器耦合；
    所述处理器，用于从外部存储器中读取行映射参数表和列映射参数表至所述内部存储器中；
    所述内部存储器还存储有程序，当所述内部存储器存储的程序指令被所述处理器执行时使得所述图像处理装置实现权利要求1至6中任意一项所述的方法。
  15. 一种成像装置，其特征在于，包括：
    图像处理装置和显示装置；
    所述图像处理装置用于按照如权利要求1至6中任意一项所述的方法根据原始图像生成目标图像，并将所述目标图像发送至所述显示装置，以使得所述成像装置投射出所述原 始图像。
  16. 一种计算机可读存储介质，包括计算机程序，所述计算机序被处理器执行以实现如权利要求1至6中任意一项所述的方法。
PCT/CN2021/138619 2020-12-18 2021-12-16 一种图像重映射方法以及图像处理装置 WO2022127841A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202011510725.7 2020-12-18
CN202011510725.7A CN114648449A (zh) 2020-12-18 2020-12-18 一种图像重映射方法以及图像处理装置

Publications (1)

Publication Number Publication Date
WO2022127841A1 true WO2022127841A1 (zh) 2022-06-23

Family

ID=81990857

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/138619 WO2022127841A1 (zh) 2020-12-18 2021-12-16 一种图像重映射方法以及图像处理装置

Country Status (2)

Country Link
CN (1) CN114648449A (zh)
WO (1) WO2022127841A1 (zh)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115113424A (zh) * 2022-06-24 2022-09-27 利亚德光电股份有限公司 数据存取方法、装置、非易失性存储介质及图像处理设备
CN116152121A (zh) * 2023-04-20 2023-05-23 合肥高维数据技术有限公司 基于畸变参数的曲面屏生成方法、矫正方法

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116433535B (zh) * 2023-06-12 2023-09-05 合肥埃科光电科技股份有限公司 一种二次曲线拟合的点坐标去畸变方法、***及存储介质

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104363986A (zh) * 2014-10-31 2015-02-18 华为技术有限公司 一种图像处理方法和设备
CN104504677A (zh) * 2014-11-17 2015-04-08 南京泓众电子科技有限公司 一种基于可变视角的图像生成方法
CN107424126A (zh) * 2017-05-26 2017-12-01 广州视源电子科技股份有限公司 图像校正方法、装置、设备、***及摄像设备和显示设备
EP3496042A1 (en) * 2017-12-05 2019-06-12 Shortbite Ltd System and method for generating training images
CN111839580A (zh) * 2020-07-22 2020-10-30 桂林市啄木鸟医疗器械有限公司 牙片影像生成方法、装置、电子设备和存储介质

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104363986A (zh) * 2014-10-31 2015-02-18 华为技术有限公司 一种图像处理方法和设备
CN104504677A (zh) * 2014-11-17 2015-04-08 南京泓众电子科技有限公司 一种基于可变视角的图像生成方法
CN107424126A (zh) * 2017-05-26 2017-12-01 广州视源电子科技股份有限公司 图像校正方法、装置、设备、***及摄像设备和显示设备
EP3496042A1 (en) * 2017-12-05 2019-06-12 Shortbite Ltd System and method for generating training images
CN111839580A (zh) * 2020-07-22 2020-10-30 桂林市啄木鸟医疗器械有限公司 牙片影像生成方法、装置、电子设备和存储介质

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115113424A (zh) * 2022-06-24 2022-09-27 利亚德光电股份有限公司 数据存取方法、装置、非易失性存储介质及图像处理设备
CN115113424B (zh) * 2022-06-24 2024-06-04 利亚德光电股份有限公司 数据存取方法、装置、非易失性存储介质及图像处理设备
CN116152121A (zh) * 2023-04-20 2023-05-23 合肥高维数据技术有限公司 基于畸变参数的曲面屏生成方法、矫正方法

Also Published As

Publication number Publication date
CN114648449A (zh) 2022-06-21

Similar Documents

Publication Publication Date Title
WO2022127841A1 (zh) 一种图像重映射方法以及图像处理装置
US11392197B2 (en) Image rendering method, device, system, storage medium, image display method and computer device
US10152814B2 (en) Image distortion compensation display device and image distortion compensation method using the same
US10602032B2 (en) Method of correcting image distortion of optical device in display device and display device
CN1884968B (zh) 车辆用图像生成设备和方法
US10578868B2 (en) Head-mounted display and video data processing method thereof
US11164496B2 (en) Interrupt-free multiple buffering methods and systems
US20160171644A1 (en) Processing unaligned block transfer operations
US20220122233A1 (en) Method and apparatus for presenting image for virtual reality device, device and non-transitory computer-readable storage medium
CN105354809A (zh) 一种基于输出图像像素位置索引的预畸变方法和装置
US20170294176A1 (en) Image processing apparatus, image processing method, and storage medium
JP2024502772A (ja) 合成イメージの生成
JP2008543071A (ja) ワーピング演算ユニットを有する集積回路
CN110035273B (zh) 一种畸变校正方法、装置及使用其的显示设备
JP6000613B2 (ja) 画像処理装置、画像処理方法
JP2011008316A (ja) 情報処理装置、画像表示装置及び情報処理方法
US9761160B2 (en) Image processing device, display apparatus, image processing method, and program
US10699374B2 (en) Lens contribution-based virtual reality display rendering
CN112905514B (zh) 一种基于dmd的光刻设备数据传输***及其方法
US20150022539A1 (en) Image processing device and image processing method
US11055903B2 (en) Anti-aliasing graphic processing method, system, storage medium and apparatus
JP2017016511A (ja) 歪み補正画像処理装置及びプログラム
US10540788B2 (en) Radial gradient module
JP2013195963A (ja) 画像処理装置、集積回路装置及び画像表示システム
WO2023178539A1 (zh) 图像处理方法及装置

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21905767

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21905767

Country of ref document: EP

Kind code of ref document: A1