CN111314688B - Disparity map hole filling method and device and electronic system - Google Patents
Disparity map hole filling method and device and electronic system Download PDFInfo
- Publication number
- CN111314688B CN111314688B CN202010184648.4A CN202010184648A CN111314688B CN 111314688 B CN111314688 B CN 111314688B CN 202010184648 A CN202010184648 A CN 202010184648A CN 111314688 B CN111314688 B CN 111314688B
- Authority
- CN
- China
- Prior art keywords
- area
- region
- target
- image
- hole
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
- H04N13/106—Processing image signals
- H04N13/122—Improving the 3D impression of stereoscopic images by modifying image signal contents, e.g. by filtering or adding monoscopic depth cues
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Image Processing (AREA)
Abstract
The invention provides a parallax image hole filling method, a device and an electronic system; wherein, the method comprises the following steps: determining the area of a target hole region and an image region containing the target hole region in a preset disparity map; obtaining an integral image corresponding to an image area containing a target cavity area; and filling the target cavity region according to the region area and the integral graph of the target cavity region. In the method, in the process of filling the target cavity region, the region area of the target cavity region is considered, so that the matching degree of the filling method of the target cavity region and the target cavity region is higher, the condition that the noise point of the parallax map is changed into parallax white spots can be avoided, the quality of the parallax map is favorably improved, and the effect of image processing on the binocular camera image based on the parallax map is favorably improved.
Description
Technical Field
The invention relates to the technical field of image processing, in particular to a parallax image hole filling method, a parallax image hole filling device and an electronic system.
Background
The disparity map is an important intermediate result when processing images acquired based on binocular cameras. Holes are usually present in the disparity map in the initial state, and the holes easily affect the processing effect of subsequent images. For example, in the double-shot large aperture blurring project, in order to make blurring effect uniform out-of-focus blurring and clear in-focus, it is necessary to reasonably fill the holes in the disparity map, otherwise, blurring effect is difficult to achieve the intended purpose.
In the related art, the hole filling of the disparity map may be implemented by using various algorithms, for example, a scan line filling algorithm, a mean filtering algorithm, a 16-direction interpolation algorithm, histogram voting based on a segmentation region, and the like. The filling effect of the mean filtering algorithm is superior to that of the scanning line filling algorithm, but the size of the filtering window is fixed, and when the size of a cavity to be filled is changed greatly, the parallax image noise is easily changed into parallax white spots by the filling mode with the fixed size, so that the quality of the parallax image is reduced.
Disclosure of Invention
In view of the above, the present invention provides a disparity map hole filling method, device and electronic system to improve the quality of a disparity map and further improve the effect of performing image processing on a binocular image based on the disparity map.
In a first aspect, an embodiment of the present invention provides a method for filling a disparity map hole, where the method includes: determining the area of a target hole region and an image region containing the target hole region in a preset disparity map; obtaining an integral image corresponding to an image area containing a target cavity area; and filling the target cavity region according to the region area and the integral graph of the target cavity region.
Further, the step of determining the image region including the target hole region includes: determining a minimum circumscribed area of a preset shape of the target hole area in the disparity map; and expanding the minimum external region based on preset expansion parameters to obtain an image region containing the target cavity region.
Further, the minimum circumscribed area of the preset shape comprises a minimum circumscribed rectangle; the extension parameters include a width extension parameter and a height extension parameter.
Further, the step of obtaining an integral map corresponding to an image region including the target cavity region includes: calculating an integral graph corresponding to an image area containing the target cavity area:
wherein, IntegralImage represents an integral graph; idx represents a target hole area; i (u, v) represents a pixel value at the pixel (u, v) in the disparity map; and | w | represents an image region including the target hole region.
Further, before the step of calculating the integral map corresponding to the image region including the target hole region, the method further includes: calculating the average value of pixel values in an image area containing a target hole area; determining a pixel value threshold based on the average; for each pixel in the image region containing the target hole region, if the pixel value of the pixel is greater than or equal to the pixel value threshold, the pixel value of the pixel is set as an average value.
Further, the step of filling the target cavity region based on the region area of the target cavity region and the integral map includes:
for each pixel in the target hole region, setting the pixel value of the pixel to be:
d*(u,v)＝sum_D/sum_area：
wherein sum _ D represents the sum of pixel values in an image region centered on pixel (u, v) in the disparity map; the region width of the image region centered on the pixel (u, v) is w1Zone height of h1(ii) a IntegralImage represents an integral graph; idx represents a target hole area; w is a1Indicating a width of an image region including the target hole region; h is1Indicating the height of an image area containing a target hole area; sum _ area represents a difference between the area of the image region including the target hole region and the area of the target hole region.
Further, after the step of filling the target cavity region, the method further includes: and performing noise reduction processing on the filled disparity map to obtain a disparity map subjected to noise reduction processing.
Further, before the step of determining the area of the target hole region and the image region including the target hole region in the preset disparity map, the method further includes: acquiring a hole area existing in each row of pixel lines of the disparity map in a line-by-line scanning mode; for each row of pixel lines, judging whether mutually continuous hollow areas exist between the pixel line and the pixel line adjacent to the pixel line, and if so, combining the mutually continuous hollow areas; continuously executing the step of acquiring the hole regions existing in each row of pixel lines of the disparity map in a progressive scanning mode until the area of each hole region in the disparity map is not changed any more; and determining each hole area with no change as each target hole area in the disparity map.
In a second aspect, an embodiment of the present invention provides a disparity map hole filling apparatus, including: the determining module is used for determining the area of the target hole region and an image region containing the target hole region in a preset disparity map; the integral image acquisition module is used for acquiring an integral image corresponding to an image area containing a target cavity area; and the region filling module is used for filling the target cavity region according to the region area and the integral map of the target cavity region.
In a third aspect, an embodiment of the present invention provides an electronic system, including: the device comprises an image acquisition device, a processing device and a storage device; the image acquisition equipment is used for acquiring preview video frames or image data; the storage means has stored thereon a computer program which, when run by a processing device, performs the above-described disparity map hole filling method.
In a fourth aspect, the present invention provides a computer-readable storage medium, on which a computer program is stored, where the computer program is executed by a processing device to perform the steps of the above disparity map hole filling method.
The embodiment of the invention has the following beneficial effects:
according to the parallax image hole filling method, device and electronic system, firstly, the area of a target hole region and an image region containing the target hole region are determined in a preset parallax image; then, obtaining an integral image corresponding to an image area containing a target cavity area; filling the target cavity region according to the region area and the integral graph of the target cavity region; in the method, in the process of filling the target cavity region, the region area of the target cavity region is considered, so that the matching degree of the filling method of the target cavity region and the target cavity region is higher, the condition that the noise point of the parallax map is changed into parallax white spots can be avoided, the quality of the parallax map is favorably improved, and the effect of image processing on the binocular camera image based on the parallax map is favorably improved.
Additional features and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. The objectives and other advantages of the invention will be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.
In order to make the aforementioned and other objects, features and advantages of the present invention comprehensible, preferred embodiments accompanied with figures are described in detail below.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the embodiments or the prior art descriptions will be briefly described below, it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings can be obtained by those skilled in the art without creative efforts.
Fig. 1 is a schematic structural diagram of an electronic system according to an embodiment of the present invention;
fig. 2 is a flowchart of a disparity map hole filling method according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of a hole region in a disparity map according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of a hole region in another disparity map according to an embodiment of the present invention;
FIG. 5 is a schematic diagram of a minimum bounding rectangle of a target void region provided in an embodiment of the present invention;
fig. 6 is a schematic diagram of an image region obtained after the minimum bounding rectangle provided by the embodiment of the present invention is expanded;
fig. 7 is a schematic structural diagram of a parallax map hole filling device according to an embodiment of the present invention.
Detailed Description
To make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
In consideration of the fact that in the related art, the size of a filtering window is relatively fixed in a mode of filling a parallax image hole region through a mean filtering algorithm, when the size of a hole to be filled is changed greatly, parallax image noise is easily changed into parallax white spots, and the quality of a parallax image is further reduced, the embodiment of the invention provides a parallax image hole filling method, a device and an electronic system; the technology can be applied to various devices such as a server, a computer, a camera, a mobile phone, a tablet personal computer, a vehicle central control device and the like, can be realized by adopting corresponding software and hardware, and the embodiment of the invention is described in detail below.
The first embodiment is as follows:
first, an example electronic system 100 for implementing the disparity map hole filling method, apparatus and electronic system according to the embodiments of the present invention is described with reference to fig. 1.
As shown in FIG. 1, an electronic system 100 includes one or more processing devices 102, one or more memory devices 104, an input device 106, an output device 108, and one or more image capture devices 110, which are interconnected via a bus system 112 and/or other type of connection mechanism (not shown). It should be noted that the components and structure of the electronic system 100 shown in fig. 1 are exemplary only, and not limiting, and that the electronic system may have other components and structures as desired.
The processing device 102 may be a gateway or an intelligent terminal, or a device including a Central Processing Unit (CPU) or other form of processing unit having data processing capability and/or instruction execution capability, and may process data of other components in the electronic system 100 and may control other components in the electronic system 100 to perform desired functions.
The storage 104 may include one or more computer program products that may include various forms of computer-readable storage media, such as volatile memory and/or non-volatile memory. The volatile memory may include, for example, Random Access Memory (RAM), cache memory (cache), and/or the like. The non-volatile memory may include, for example, Read Only Memory (ROM), hard disk, flash memory, etc. On which one or more computer program instructions may be stored that may be executed by processing device 102 to implement client functionality (implemented by the processing device) and/or other desired functionality in embodiments of the present invention described below. Various applications and various data, such as various data used and/or generated by the applications, may also be stored in the computer-readable storage medium.
The input device 106 may be a device used by a user to input instructions and may include one or more of a keyboard, a mouse, a microphone, a touch screen, and the like.
The output device 108 may output various information (e.g., images or sounds) to the outside (e.g., a user), and may include one or more of a display, a speaker, and the like.
The image capture device 110 may capture preview video frames or picture data and store the captured preview video frames or picture data in the storage 104 for use by other components.
For example, the devices in the example electronic system for implementing the disparity map hole filling method, apparatus and electronic system according to the embodiments of the present invention may be integrally disposed, or may be disposed in a decentralized manner, such as integrally disposing the processing device 102, the storage device 104, the input device 106 and the output device 108, and disposing the image capturing device 110 at a specific position where a target image can be captured. When the above-described devices in the electronic system are integrally provided, the electronic system may be implemented as an intelligent terminal such as a camera, a smart phone, a tablet computer, a vehicle-mounted terminal, and the like.
Example two:
the embodiment provides a disparity map hole filling method, which is executed by a processing device in the electronic system; the processing device may be any device or chip having data processing capabilities. The processing equipment can independently process the received information, can also be connected with the server, jointly analyzes and processes the information, and uploads a processing result to the cloud.
As shown in fig. 2, the method comprises the steps of:
step S202, determining the area of a target hole region and an image region containing the target hole region in a preset disparity map;
generally, after images are collected for the same scene by a binocular camera, two images with the same scene but different visual angles are obtained. The disparity maps of the two images can be obtained in a characteristic point matching mode; the pixel values of the pixels of each point in the disparity map represent the disparity values of the pixels matched with each other in the two images. In many cases, the closer the two images are to the scene area of the camera, the greater the corresponding parallax. A disparity map may include one or more hole regions, and a hole region may include one or more adjacent pixels in the disparity map. The reason why the hole region is generated in the disparity map is generally two, one reason is that a calculation error occurs in the process of calculating the disparity maps of the two images, and the other reason is that a blocked region exists in the images, the blocked region is a region which cannot be seen by the binocular camera, or a region which cannot be seen by the left camera or the right camera in the binocular camera, and at this time, the hole region appears in the disparity map when the disparity map is calculated.
The target hole region may be one hole region in the disparity map, or may be a plurality of hole regions. In general, if a disparity map includes a plurality of hole regions, there is no communication between the hole regions. In actual implementation, each hole region in the disparity map may be determined as a target hole region one by one, or a plurality of hole regions in the disparity map may be determined as target hole regions at the same time. In one implementation, in order to fill the cavity region in a manner matching the area of the cavity region according to the area of the cavity region, a plurality of cavity regions in the disparity map may be used as target cavity regions one by one, and then the filling process is performed.
The area of the target void region may be expressed by the number of pixels included in the target void region, or may be expressed by other area expression methods. The image area including the target void area may be understood as the image area including the target void area and also including a part of pixel points of a non-void area around the target void area, and the filling value of the pixel points in the target void area may be determined according to pixel values of the pixel points. In general, the shape of the target hole region is various, and the shape of the image region including the target hole region may be preset to various shapes, such as a circle, an ellipse, a rectangle, or other irregular figures; the area of the graphic region can be preset as a fixed value, but in most cases, the area of the graphic region can be determined based on the area of the target cavity region; that is, if the area of the target hole region is large, the area of the image region including the target hole region is also large; if the area of the target hole region is small, the area of the image region including the target hole region is also small.
Step S204, obtaining an integral image corresponding to an image area containing a target cavity area;
it is assumed that, for the integral map of image a, it can be calculated specifically by: the value of any point (x, y) in the integral map is equal to the sum of the values of the point at the upper left corner of the image A and all points in the rectangular area enclosed by the point (x, y).
Specifically, in step S204, in the integral map corresponding to the image area, the value of any one point (a, b) in the integral map is equal to the sum of the values of all points in the rectangular area surrounded by the points (a, b) and the point (a, b) at the upper left corner of the image area. Considering that the image area including the target cavity area belongs to a part of the disparity map, an integral map corresponding to the whole disparity map can be obtained through calculation, an integral map area corresponding to the image area is intercepted from the integral map, and the integral map area is used as an integral map corresponding to the image area.
In step S206, the target cavity region is filled based on the region area and the integral map of the target cavity region.
When filling a target cavity region, it is more important to determine a filling value of each pixel point in the target cavity region; in the process of determining the filling value of the target cavity region, the area of the target cavity region and the integral value of each point in the integral map can be used as reference factors. Since the areas of different void regions are usually different, the filling values used when filling the void regions are also different.
Firstly, determining the area of a target hole region and an image region containing the target hole region in a preset disparity map; then, obtaining an integral image corresponding to an image area containing a target cavity area; filling the target cavity region according to the region area and the integral graph of the target cavity region; in the method, in the process of filling the target cavity region, the region area of the target cavity region is considered, so that the matching degree of the filling method of the target cavity region and the target cavity region is higher, the condition that parallax image noise points are changed into parallax white spots can be avoided, the quality of a parallax image is favorably improved, and the effect of processing images of binocular camera images based on the parallax image is favorably improved.
Example three:
the embodiment provides another disparity map hole filling method. Before filling the disparity map hole, in this embodiment, how to acquire a hole area in the disparity map is described first, including the following steps 02-08:
step 02, acquiring a hole area existing in each row of pixel lines of the disparity map in a line-by-line scanning mode;
in the initial state of the disparity map, the pixel points in the void region may not have pixel values, or the difference between the pixel values and the pixel values in the non-void region may be particularly large. Based on this, the disparity map can be scanned pixel by pixel first, and if there is a pixel without a pixel value or the pixel value exceeds a preset reasonable range, the pixel value of the void region can be set to the same preset value, which can be 0, 255 or other values.
In the progressive scanning process, aiming at each row of pixel rows in the disparity map, all the pixel points with the pixel values of preset values in the row are marked as target pixel points, and then if adjacent target pixel points exist, the adjacent target pixel points form a hollow area. In addition, the hole areas scanned in each line can be numbered in sequence. It is understood that the hole region may be obtained by scanning column by column or by serpentine scanning, in addition to the progressive scanning.
Step 04, judging whether mutually continuous hollow areas exist between each pixel row and the adjacent pixel row of the pixel row aiming at each pixel row, and if so, combining the mutually continuous hollow areas;
step 06, continuing to execute the step of acquiring the hole regions existing in each row of pixel rows of the disparity map in a progressive scanning mode until the area of each hole region in the disparity map is not changed any more;
and step 08, determining each hole area with no change in area as each target hole area in the disparity map.
For example, if the hole region a in the first pixel row and the hole region b in the second pixel row are continuous with each other, it can be understood that at least one pixel in the hole region a is adjacent to at least one pixel in the hole region b. In practical implementation, step 04 may be executed according to the arrangement order of the pixel rows, for example, starting from the first row of pixel rows, determining whether there are mutually continuous hollow regions between the pixel rows and the second row of pixel rows; after combining the mutually continuous hole regions, the total amount of the hole regions corresponding to the first and second rows of pixel rows is usually not greater than the total amount of the hole regions corresponding to the first and second rows of pixel rows before combining.
After the void areas in the first row and the second row of pixel rows are combined, whether the void areas which are continuous with the void areas corresponding to the first row and the second row of pixel rows exist in the void area in the third row or not is checked, and if the void areas exist, the combining process of the void areas is carried out; and repeating the steps until all the adjacent hole areas in the disparity map are combined.
For easy understanding, fig. 3 is a schematic diagram of a hole region in a disparity map; wherein, the shadow area is the merged cavity area. For convenience of processing, each hole region may be numbered, and as shown in fig. 4, each hole region may be numbered according to the position of the hole region; the number may be set in the hole region, i.e., one hole region corresponds to one number; or in each pixel of the void region; that is, each pixel in the void region is provided with a number, and pixels in the same void region correspond to the same number.
After the hole regions in the disparity map are obtained in the above manner, since the hole regions are independent from each other and are not communicated with each other, each hole region can be used as a target hole region, and then the region area of the target hole region and the image region including the target hole region are determined.
As described in the above embodiment, the number of pixels included in the target hole region may be determined as the region area of the target hole region. In the process of determining an image region including a target hole region, a minimum circumscribed region of a preset shape of the target hole region may be determined in a disparity map; the preset shape may be a rectangle, a square, a circle, or the like. The minimum circumscribed area of the preset shape can be understood as the size of the area of the preset shape, which can just contain the target cavity area.
In order to fill the pixels in the target hole region, enough parallax values are involved in the determination process of the filling values, and after the minimum circumscribed region is determined, the minimum circumscribed region is expanded based on preset expansion parameters to obtain an image region containing the target hole region. The preset expansion parameters can be determined according to the shape of the minimum circumscribed area; for example, when the minimum bounding region is a rectangle, the preset expansion parameters may include an expansion parameter of the length of the rectangle and an expansion parameter of the width of the rectangle; when the minimum circumscribed area is a circle, the preset expansion parameter may include an expansion parameter of a radius.
In a specific implementation manner, the minimum circumscribed area of the preset shape may be a minimum circumscribed rectangle; FIG. 5 is a schematic diagram of a minimum bounding rectangle of a target void region; the target hole area is irregular in shape, and the size of the minimum circumscribed rectangle can just contain the target hole area. For the minimum bounding rectangle, the extension parameters include a width extension parameter and a height extension parameter. The expansion parameter can be an expansion multiple, and the width of the minimum circumscribed rectangle is multiplied by the width expansion multiple to obtain the width of the expanded rectangle; and multiplying the height of the minimum circumscribed rectangle by the height expansion multiple to obtain the height of the expanded rectangle. The extension parameter may also be a specific extension value, for example, the width of the minimum circumscribed rectangle is added to the width extension value to obtain the width of the extended rectangle; and adding the height expansion value to the height of the minimum circumscribed rectangle to obtain the height of the expanded rectangle.
FIG. 6 is a diagram illustrating an image region obtained after the minimum bounding rectangle is expanded; the image area includes a minimum bounding rectangle and an extended area surrounding the minimum bounding rectangle. In practical implementation, the width extension value and the height extension value may be the same, and may be Δ, for examplestrideRepresents an extension value; when the width of the minimum circumscribed rectangle is width, the width w of the expanded image region is width +2 Δstride(ii) a When the height of the minimum circumscribed rectangle is height, the height h of the expanded image area is height +2 deltastride。
After the image area containing the target void area is determined, an integral image corresponding to the image area containing the target void area can be obtained; the method can be specifically realized by the following formula:
wherein, IntegralImage represents an integral graph; idx represents a target hole area; i (u, v) represents a pixel value at the pixel (u, v) in the disparity map; and | w | represents an image region including the target hole region.
Specifically, idx may be the number of the target hole region. If the disparity map comprises a plurality of hole areas, each hole area can be numbered for convenience of processing; then, each cavity area is taken as a target cavity area one by one, and an integral graph corresponding to the image area of the target cavity area is calculated, wherein idx is the number of the target cavity area.
In addition, because an abnormal maximum parallax value or an error parallax value may exist in the image region of the parallax map, if a certain pixel of the hollow region is located in the vicinity of the abnormal value or the error value, the filled value of the pixel is also likely to be an abnormal value, and after the pixel in the hollow region is filled with the abnormal value, an abnormal white spot parallax region having a certain area is formed as a whole. Based on this, before the step of calculating the integral map corresponding to the image area including the target hole area, the following steps 12 to 16 may be further performed:
step 12, calculating an average value of pixel values in an image region including a target hole region;
step 14, determining a pixel value threshold value based on the average value;
and step 16, setting the pixel value of each pixel in the image area containing the target hole area as an average value if the pixel value of the pixel is greater than or equal to the pixel value threshold.
For example, the pixel value threshold may be several times as large as the average value, e.g., the pixel value threshold is equal to three times the average value; if the pixel value of which pixel in the image region is greater than or equal to the pixel value threshold, the pixel value of the pixel is set as an average value, abnormal maximum values and error values in the image region can be eliminated through the method, abnormal white spot parallax regions are avoided when the hole regions are filled, and therefore the effect of filling the holes in the parallax image is improved.
In another implementation manner, the above step 12-16 may also be implemented after the integrogram corresponding to the image region including the target void region is acquired, that is, after the image region is updated through the step 12-16, the disparity map is updated based on the updated image region.
After the area of the target cavity region and the integral map corresponding to the image region including the target cavity region are obtained, filling processing can be performed on the target cavity region. In specific implementation, for each pixel in the target hole region, the pixel value of the pixel is set as:
d*(u,v)＝sum_D/sum_area：
the sum _ D can be specifically calculated by the following formula:
sum _ D represents the sum of pixel values in an image region centered on the pixel (u, v) in the disparity map; the region width of the image region centered on the pixel (u, v) is w1Zone height of h1(ii) a IntegralImage represents an integral graph; idx represents a target hole area; w is a1Indicating a width of an image region including the target hole region; h is1Indicating the height of an image area containing a target hole area; sum _ area represents a difference between the area of the image region including the target hole region and the area of the target hole region.
As can be seen from the above formula, the pixel value of the pixel in the target void region is not only related to the integral image corresponding to the image region including the target void region, but also related to the region area of the target void region; the sum _ area may be understood as an area of a region other than the target hole region in the image region including the target hole region. The area of the region can also be understood as the number of pixels included in the region.
In addition, since parallax noise may be generated in the parallax map after the filling of the holes in the parallax map is completed, it is necessary to perform noise reduction processing on the parallax map after the filling processing to obtain a parallax map after the noise reduction processing. Specifically, a median filter algorithm may be used for denoising, and a radius parameter of the median filter algorithm may be 3 × 3, although the parameter may also be other values.
In the method for filling the parallax image holes, the holes of the parallax image are filled on the basis of mean value filtering, the area of the holes in the parallax image is calculated and used as a reference value for filling, and the method can solve the problem that when the parallax image holes are filled by the traditional mean value filtering algorithm, the algorithm cannot be adaptive to the sizes of the holes, so that the filling effect of the parallax image holes is improved.
In addition, the above-mentioned disparity map hole filling method calculates an integral map for the image area of each hole area, and compared with the conventional method for calculating a global integral map of a disparity map, the method can greatly reduce the memory occupation, and can complete the filling of all hole areas in the disparity map with less memory consumption.
In a word, the parallax map hole filling method can more accurately perform hole parallax filling by self-adapting the size of the hole area; by processing the extremely large abnormal value and the error parallax value in the parallax image, the abnormal white spot parallax possibly existing in the traditional mean value filtering algorithm can be reduced; by carrying out local integral image calculation on the image area of each hole area, the parallax filling calculation amount can be reduced, the parallax hole filling can be completed quickly, and the platform memory occupation caused by the whole image integral image calculation is reduced.
Example four:
corresponding to the above method embodiment, referring to fig. 7, a schematic structural diagram of a disparity map hole filling device is shown, where the device includes:
a determining module 70, configured to determine, in a preset disparity map, a region area of a target hole region and an image region including the target hole region;
an integral image obtaining module 71, configured to obtain an integral image corresponding to an image area including the target cavity area;
and the region filling module 72 is configured to perform filling processing on the target cavity region according to the region area of the target cavity region and the integral map.
Firstly, determining the area of a target hole region and an image region containing the target hole region in a preset disparity map; then, obtaining an integral image corresponding to an image area containing a target cavity area; filling the target cavity region according to the region area and the integral graph of the target cavity region; in the method, in the process of filling the target cavity region, the region area of the target cavity region is considered, so that the matching degree of the filling method of the target cavity region and the target cavity region is higher, the condition that the noise point of the parallax map is changed into parallax white spots can be avoided, the quality of the parallax map is favorably improved, and the effect of image processing on the binocular camera image based on the parallax map is favorably improved.
Further, the determining module is further configured to: determining a minimum circumscribed area of a preset shape of the target hole area in the disparity map; and expanding the minimum external region based on preset expansion parameters to obtain an image region containing the target cavity region.
Further, the minimum circumscribed area of the preset shape comprises a minimum circumscribed rectangle; the extension parameters include a width extension parameter and a height extension parameter.
Further, the integral map obtaining module is further configured to: calculating an integral graph corresponding to an image area containing the target cavity area:
wherein, IntegralImage represents an integral graph; idx represents a target hole area; i (u, v) represents a pixel value at the pixel (u, v) in the disparity map; and | w | represents an image region including the target hole region.
Further, the above apparatus further comprises: the average value calculation module is used for calculating the average value of pixel values in an image area containing the target cavity area; a threshold determination module to determine a pixel value threshold based on the average; and the pixel value setting module is used for setting the pixel value of each pixel in the image area containing the target hole area as an average value if the pixel value of the pixel is greater than or equal to the pixel value threshold.
Further, the region filling module is further configured to: for each pixel in the target hole region, setting the pixel value of the pixel to be: d*(u,v)＝sum_D/sum_area；
sum _ D represents the sum of pixel values in an image region centered on the pixel (u, v) in the disparity map; the region width of the image region centered on the pixel (u, v) is w1Zone height of h1(ii) a IntegralImage represents an integral graph; idx represents a target hole area; w is a1Indicating a width of an image region including the target hole region; h is1Indicating the height of an image area containing a target hole area; sum _ area represents a difference between the area of the image region including the target hole region and the area of the target hole region.
Further, the above apparatus further comprises: and the noise reduction module is used for carrying out noise reduction processing on the filled disparity map to obtain the disparity map subjected to the noise reduction processing.
Further, the above apparatus further comprises: the device comprises a cavity area acquisition module, a disparity map generation module and a disparity map generation module, wherein the cavity area acquisition module is used for acquiring a cavity area existing in each row of pixel lines of the disparity map in a line-by-line scanning mode; the judging module is used for judging whether mutually continuous hollow areas exist between each pixel row and the adjacent pixel row of the pixel row aiming at each pixel row, and if so, the mutually continuous hollow areas are combined; the execution module is used for continuously executing the step of acquiring the hole regions existing in each row of pixel rows of the disparity map in a progressive scanning mode until the area of each hole region in the disparity map is not changed any more; and determining each hole area with no change as each target hole area in the disparity map.
The present embodiment also provides an electronic system, including: the device comprises an image acquisition device, a processing device and a storage device; the image acquisition equipment is used for acquiring preview video frames or image data; the storage means has stored thereon a computer program which, when run by a processing device, performs the above-described disparity map hole filling method.
The present embodiment also provides a computer-readable storage medium, on which a computer program is stored, where the computer program is executed by a processing device to perform the steps of the above disparity map hole filling method.
The method, the apparatus, and the computer program product for filling a disparity map hole provided in the embodiments of the present invention include a computer-readable storage medium storing a program code, where instructions included in the program code may be used to execute the method described in the foregoing method embodiments, and specific implementation may refer to the method embodiments, and will not be described herein again.
It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the system and the apparatus described above may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
In addition, in the description of the embodiments of the present invention, unless otherwise explicitly specified or limited, the terms "mounted," "connected," and "connected" are to be construed broadly, e.g., as meaning either a fixed connection, a removable connection, or an integral connection; can be mechanically or electrically connected; they may be connected directly or indirectly through intervening media, or they may be interconnected between two elements. The specific meaning of the above terms in the present invention can be understood in specific cases for those skilled in the art.
The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicate orientations or positional relationships based on the orientations or positional relationships shown in the drawings, and are only for convenience of description and simplicity of description, but do not indicate or imply that the device or element being referred to must have a particular orientation, be constructed and operated in a particular orientation, and thus, should not be construed as limiting the present invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance.
Finally, it should be noted that: although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art will understand that the following embodiments are merely illustrative of the present invention, and not restrictive, and the scope of the present invention is not limited thereto: any person skilled in the art can modify or easily conceive the technical solutions described in the foregoing embodiments or equivalent substitutes for some technical features within the technical scope of the present disclosure; such modifications, changes or substitutions do not depart from the spirit and scope of the embodiments of the present invention, and they should be construed as being included therein. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims (10)
1. A disparity map hole filling method is characterized by comprising the following steps:
determining the area of a target hole region and an image region containing the target hole region in a preset disparity map; the image area of the target cavity area comprises the target cavity area and partial pixel points of a non-cavity area around the target cavity area;
acquiring an integral image corresponding to the image area containing the target cavity area;
filling the target cavity region according to the region area of the target cavity region and the integral map;
the step of determining an image region including the target hole region includes: determining a minimum circumscribed area of a preset shape of the target hole area in the disparity map; and expanding the minimum external connection area based on preset expansion parameters to obtain an image area containing the target hole area.
2. The method of claim 1, wherein the minimum circumscribed area of the preset shape comprises a minimum circumscribed rectangle; the extension parameters include a width extension parameter and a height extension parameter.
3. The method of claim 1, wherein the step of obtaining the integral map corresponding to the image region containing the target hole region comprises:
calculating an integral map corresponding to the image area containing the target cavity area:
wherein, IntegralImage represents an integral graph; idx represents the target hole area; i (u, v) represents a pixel value at a pixel (u, v) in the disparity map; and | w | | represents the image area containing the target cavity area.
4. The method of claim 3, wherein before the step of calculating the integral map corresponding to the image region containing the target hole region, the method further comprises:
calculating the average value of the pixel values in the image area containing the target hole area;
determining a pixel value threshold based on the average;
and for each pixel in the image area containing the target hole area, if the pixel value of the pixel is greater than or equal to the pixel value threshold value, setting the pixel value of the pixel as the average value.
5. The method according to claim 1, wherein the step of filling the target cavity region based on the region area of the target cavity region and the integral map comprises:
for each pixel in the target hole region, setting the pixel value of the pixel as:
wherein sum _ D represents a sum of pixel values in an image region centered on a pixel (u, v) in the disparity map; IntegralImage represents an integral graph; idx represents the target hole area; w is a1Representing the width of the image area containing the target hole area; h is1Representing the height of the image area containing the target hole area; sum _ area represents a difference between the area of the image region including the target hole region and the area of the target hole region.
6. The method of claim 1, wherein after the step of filling the target void region, the method further comprises: and performing noise reduction processing on the filled disparity map to obtain the disparity map subjected to noise reduction processing.
7. The method according to claim 1, wherein before the step of determining the area of the target hole region and the image region containing the target hole region in the preset disparity map, the method further comprises:
acquiring a hole area existing in each row of pixel lines of the disparity map in a line-by-line scanning mode;
for each row of pixel lines, judging whether mutually continuous hollow areas exist between the pixel line and the pixel line adjacent to the pixel line, and if so, combining the mutually continuous hollow areas;
continuously executing the step of acquiring the hole regions existing in each row of pixel lines of the disparity map in a progressive scanning mode until the area of each hole region in the disparity map is not changed any more;
and determining each hole area with no change as each target hole area in the disparity map.
8. A disparity map hole filling apparatus, comprising:
the determining module is used for determining the area of a target hole region and an image region containing the target hole region in a preset disparity map; the image area of the target cavity area comprises the target cavity area and partial pixel points of a non-cavity area around the target cavity area;
the integral image acquisition module is used for acquiring an integral image corresponding to the image area containing the target cavity area;
the region filling module is used for filling the target cavity region according to the region area of the target cavity region and the integrogram;
the determination module is further to: determining a minimum circumscribed area of a preset shape of the target hole area in the disparity map; and expanding the minimum external connection area based on preset expansion parameters to obtain an image area containing the target hole area.
9. An electronic system, characterized in that the electronic system comprises: the device comprises an image acquisition device, a processing device and a storage device;
the image acquisition equipment is used for acquiring preview video frames or image data;
the storage device has stored thereon a computer program which, when executed by the processing apparatus, performs the disparity map hole filling method according to any one of claims 1 to 7.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processing device, performs the steps of the disparity map hole filling method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010184648.4A CN111314688B (en) | 2020-03-16 | 2020-03-16 | Disparity map hole filling method and device and electronic system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010184648.4A CN111314688B (en) | 2020-03-16 | 2020-03-16 | Disparity map hole filling method and device and electronic system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111314688A CN111314688A (en) | 2020-06-19 |
| CN111314688B true CN111314688B (en) | 2022-05-17 |
Family
ID=71148443
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010184648.4A Active CN111314688B (en) | 2020-03-16 | 2020-03-16 | Disparity map hole filling method and device and electronic system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111314688B (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113724369A (en) * | 2021-08-01 | 2021-11-30 | 国网江苏省电力有限公司徐州供电分公司 | Scene-oriented three-dimensional reconstruction viewpoint planning method and system |
| CN113781543B (en) * | 2021-11-10 | 2022-03-18 | 北京中科慧眼科技有限公司 | Binocular camera-based height limiting device detection method and system and intelligent terminal |
| CN114782450B (en) * | 2022-06-23 | 2022-10-25 | 北京航空航天大学杭州创新研究院 | Hole filling equipment control method, device, equipment and computer readable medium |
| CN117576256B (en) * | 2023-11-16 | 2024-04-26 | 元橡科技（北京）有限公司 | Parallax filling method, system and storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102307312A (en) * | 2011-08-31 | 2012-01-04 | 四川虹微技术有限公司 | Method for performing hole filling on destination image generated by depth-image-based rendering (DIBR) technology |
| CN103051908A (en) * | 2012-12-26 | 2013-04-17 | 四川虹微技术有限公司 | Disparity map-based hole filling device |
| CN103384343A (en) * | 2013-07-02 | 2013-11-06 | 南京大学 | Image cavity filling method and device thereof |
| CN103458257A (en) * | 2012-05-31 | 2013-12-18 | 财团法人工业技术研究院 | Hole filling method for multi-view disparity maps |
| CN103731657A (en) * | 2014-01-26 | 2014-04-16 | 冠捷显示科技（厦门）有限公司 | Hole filling processing method of hole-containing image processed by DIBR (Depth Image Based Rendering) algorithm |
-
2020
- 2020-03-16 CN CN202010184648.4A patent/CN111314688B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102307312A (en) * | 2011-08-31 | 2012-01-04 | 四川虹微技术有限公司 | Method for performing hole filling on destination image generated by depth-image-based rendering (DIBR) technology |
| CN103458257A (en) * | 2012-05-31 | 2013-12-18 | 财团法人工业技术研究院 | Hole filling method for multi-view disparity maps |
| CN103051908A (en) * | 2012-12-26 | 2013-04-17 | 四川虹微技术有限公司 | Disparity map-based hole filling device |
| CN103384343A (en) * | 2013-07-02 | 2013-11-06 | 南京大学 | Image cavity filling method and device thereof |
| CN103731657A (en) * | 2014-01-26 | 2014-04-16 | 冠捷显示科技（厦门）有限公司 | Hole filling processing method of hole-containing image processed by DIBR (Depth Image Based Rendering) algorithm |
Non-Patent Citations (1)
| Title |
|---|
| 《立体匹配得到的视差图孔洞填充》;CSDN博客;《https://blog.csdn.net/opencv_fjc/article/details/96451195》;20190718;"视差图和深度图的空洞填充"和程序部分 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111314688A (en) | 2020-06-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111314688B (en) | Disparity map hole filling method and device and electronic system | |
| US8818077B2 (en) | Stereo image matching apparatus and method | |
| JP5075757B2 (en) | Image processing apparatus, image processing program, image processing method, and electronic apparatus | |
| CN103426147B (en) | Image processing apparatus, image pick-up device and image processing method | |
| CN110517304B (en) | Method and device for generating depth map, electronic equipment and storage medium | |
| US8385732B2 (en) | Image stabilization | |
| CN112581904B (en) | Moire compensation method for brightness gray scale image of OLED (organic light emitting diode) screen | |
| CN111445487A (en) | Image segmentation method and device, computer equipment and storage medium | |
| CN112883983A (en) | Feature extraction method and device and electronic system | |
| CN113888438A (en) | Image processing method, device and storage medium | |
| CN111681165A (en) | Image processing method, image processing device, computer equipment and computer readable storage medium | |
| CN112580434A (en) | Face false detection optimization method and system based on depth camera and face detection equipment | |
| JP4990876B2 (en) | Image processing device | |
| CN108182666A (en) | A kind of parallax correction method, apparatus and terminal | |
| US7697775B2 (en) | Image signal processing apparatus and method, and program and recording medium used therewith | |
| CN111445411B (en) | Image denoising method, image denoising device, computer equipment and storage medium | |
| CN111885371A (en) | Image occlusion detection method and device, electronic equipment and computer readable medium | |
| CN109871814B (en) | Age estimation method and device, electronic equipment and computer storage medium | |
| JP2005339535A (en) | Calculation of dissimilarity measure | |
| CN113284075B (en) | Image denoising method and device, electronic device and storage medium | |
| CN112150532A (en) | Image processing method and device, electronic equipment and computer readable medium | |
| CN112435168B (en) | Reference block scaling method and computer readable storage medium | |
| CN111754417B (en) | Noise reduction method for video image, video matting method, device and electronic system | |
| CN115035227A (en) | Point cloud data processing method and device and electronic equipment | |
| CN115205456A (en) | Three-dimensional model construction method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |