CN108615221B - Light field angle super-resolution method and device based on shearing two-dimensional polar line plan - Google Patents
Light field angle super-resolution method and device based on shearing two-dimensional polar line plan Download PDFInfo
- Publication number
- CN108615221B CN108615221B CN201810315188.7A CN201810315188A CN108615221B CN 108615221 B CN108615221 B CN 108615221B CN 201810315188 A CN201810315188 A CN 201810315188A CN 108615221 B CN108615221 B CN 108615221B
- Authority
- CN
- China
- Prior art keywords
- resolution
- dimensional
- angle
- shearing
- light field
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
- 238000010008 shearing Methods 0.000 title claims abstract description 81
- 238000000034 method Methods 0.000 title claims abstract description 37
- 238000013527 convolutional neural network Methods 0.000 claims abstract description 17
- 230000004927 fusion Effects 0.000 claims description 22
- 238000005070 sampling Methods 0.000 claims description 19
- 238000000605 extraction Methods 0.000 claims description 5
- 230000003287 optical effect Effects 0.000 claims description 2
- 238000010586 diagram Methods 0.000 abstract description 23
- 230000000694 effects Effects 0.000 abstract description 13
- 230000006870 function Effects 0.000 description 21
- 238000012549 training Methods 0.000 description 10
- 238000003384 imaging method Methods 0.000 description 6
- 239000000126 substance Substances 0.000 description 4
- 230000000007 visual effect Effects 0.000 description 4
- 238000011084 recovery Methods 0.000 description 3
- 230000007547 defect Effects 0.000 description 2
- 239000000463 material Substances 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 238000009877 rendering Methods 0.000 description 2
- 230000004075 alteration Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000003247 decreasing effect Effects 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000007500 overflow downdraw method Methods 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/58—Optics for apodization or superresolution; Optical synthetic aperture systems
Landscapes
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Optics & Photonics (AREA)
- Image Processing (AREA)
Abstract
The invention discloses a light field angle super-resolution method and a light field angle super-resolution device based on a shearing two-dimensional polar line plan, wherein the method comprises the following steps: extracting a plurality of two-dimensional epipolar plane graphs from the low angular resolution light field; acquiring spatial low-frequency information of each two-dimensional epipolar plane map of a plurality of two-dimensional epipolar plane maps; obtaining a plurality of low-frequency polar line plane diagrams after shearing operation; achieving the target angular resolution; reconstructing angle information of the up-sampled low-frequency polar line plane graph through a convolutional neural network; obtaining an polar line plan with high spatial angular resolution after angular super-resolution; obtaining a plurality of polar plane diagrams with high angular spatial resolution under different parallaxes; obtaining a polar plane diagram with high spatial angular resolution; the high angular resolution lightfield is output. The method can achieve an excellent angle super-resolution effect on a large parallax light field with a certain sparse angle resolution, has strong robustness, and also has strong robustness on noise in a parallax image corresponding to a polar line plan.
Description
Technical Field
The invention relates to the technical field of computer vision, in particular to a light field angle super-resolution method and a light field angle super-resolution device based on a shearing two-dimensional polar line plan.
Background
Light field imaging is one of the most widely used means of capturing three-dimensional information of an object scene. Different from the traditional imaging mode, the light field not only records the intensity of light rays at a certain position, but also records the distribution condition of the light rays from a certain angle range at the position, so that the two-dimensional imaging is changed into the four-dimensional imaging comprising two space dimensions and two angle dimensions. Early light field acquisition devices were primarily multi-camera array systems and light field gantry systems, requiring custom-made expensive hardware facilities. With the development of light field imaging technology, single-camera handheld light field acquisition devices have also appeared in succession. However, due to the limitations of the resolution of the imaging sensor and the multi-dimensional data characteristics of the light field, the hand-held light field camera often has a trade-off relationship of spatial angular resolution, that is, when a higher spatial resolution is required, the camera angular resolution is lower, and vice versa.
To solve this problem, scholars at home and abroad often obtain the light field with high spatial angular resolution by using the light field with high spatial low angular resolution to perform the light field angular super-resolution or the visual angle difference method. These methods can be divided into two parts according to whether depth information is used or not. Methods that use depth information typically require computing scene depth information using an input light field, and then rendering a new view using the existing view and depth information. The method can perform angle super-resolution on a light field with larger parallax, but the calculation of the depth information is easily influenced by noise, object shielding and a low texture area, and the rendering of the visual angle is very dependent on the accuracy of the depth information, so that the obvious defect is often generated. While another type of method is generally based on the principle of estimating a four-dimensional light field using finite sampling points for light field angular super-resolution. The method has higher accuracy on the super-resolution of the light field angle, but often depends on denser visual angle sampling, and has poorer recovery effect on high-frequency information.
Disclosure of Invention
The present invention is directed to solving, at least to some extent, one of the technical problems in the related art.
Therefore, one purpose of the invention is to provide a light field angle super-resolution method based on a shearing two-dimensional polar line plan, which can achieve an excellent angle super-resolution effect on a large parallax light field with a certain sparse angle resolution, has strong robustness, and also has strong robustness on noise in a parallax image corresponding to the polar line plan.
The invention further aims to provide a light field angle super-resolution device based on the shearing of the two-dimensional polar plane diagram.
In order to achieve the above object, an embodiment of the present invention provides a super-resolution method for an optical field angle based on a shearing two-dimensional epipolar plane graph, which includes the following steps: extracting a plurality of two-dimensional epipolar plane graphs from the low angular resolution light field; obtaining spatial low-frequency information of each two-dimensional epipolar plane map of the plurality of two-dimensional epipolar plane maps through a one-dimensional Gaussian kernel function; performing shearing operation on each two-dimensional epipolar plane graph through a preset parallax value to obtain a plurality of low-frequency epipolar plane graphs after the shearing operation; carrying out angle and dimension up-sampling on the low-frequency epipolar line plane graphs after the plurality of shearing operations by a double-tri-interpolation method so as to achieve a target angle resolution; reconstructing angle information of the up-sampled low-frequency polar line plane graph through a convolutional neural network; restoring the spatial high-frequency information of the plurality of two-dimensional polar line plane graphs through non-blind deblurring operation to obtain polar line plane graphs with high spatial angular resolution after angle super-resolution; performing reverse shearing operation on the polar line plane graph after high-frequency information is recovered through the preset parallax value to obtain a plurality of polar line plane graphs with high angular spatial resolution under different parallaxes; fusing the plurality of high-angle spatial resolution epipolar plane graphs under different parallaxes through the epipolar plane graph to the corresponding parallax graph to obtain a high-spatial angular resolution epipolar plane graph; the high angular resolution lightfield is output.
According to the light field angle super-resolution method based on the shearing two-dimensional polar plane diagram, the polar plane diagram is utilized to perform angle super-resolution on the light field with low angular resolution, and the space and angle information of the light field can be utilized simultaneously; shearing the epipolar plane graph through a series of parallax candidate values, so that the aliasing phenomenon caused by sparse angle sampling and large parallax can be reduced; the position relation of the polar line plane graph after the angle super-resolution can be restored by utilizing the reverse shearing operation related to the angle super-resolution multiplying power; by fusing a series of polar line plane graphs after super-resolution, the optimal effect can be achieved on each parallax value, so that a large parallax light field with certain sparse angular resolution can achieve an excellent angle super-resolution effect, the robustness is very high, and the robustness is also very high for noise in the parallax graph corresponding to the polar line plane graph.
In addition, the light field angle super-resolution method based on the shearing two-dimensional polar line plan according to the above embodiment of the present invention may also have the following additional technical features:
further, in an embodiment of the present invention, a clipping value used when the plurality of two-dimensional epipolar plane graphs are subjected to a clipping operation is a disparity candidate value, wherein the clipped low-frequency epipolar plane graph is also a two-dimensional epipolar plane graph.
Further, in one embodiment of the present invention, the preset disparity value is obtained according to a multiple of angular super-resolution.
Further, in an embodiment of the present invention, the plurality of high-angle spatial resolution epipolar plane graphs under different parallaxes are fused by a fusion formula, wherein the fusion formula is:
wherein, WiIn order to fuse the weights needed for the fusion,
the limiting plane map, i ∈ 1, 2.
Further, in an embodiment of the present invention, the high angular spatial resolution epipolar plane plot at different parallaxes is:
wherein D isκThe non-blind deblurring operation is carried out,
to use the parallax value diF is a convolutional neural network operation, ELκ is a low frequency epipolar plan.
In order to achieve the above object, another embodiment of the present invention provides a super-resolution device for light field angle based on a sheared two-dimensional epipolar plane graph, including: the extraction module is used for extracting a plurality of two-dimensional polar line plane graphs according to the low-angle resolution light field; an obtaining module, configured to obtain, through a one-dimensional gaussian kernel function, spatial low-frequency information of each two-dimensional epipolar plane map of the multiple two-dimensional epipolar plane maps; the shearing module is used for carrying out shearing operation on each two-dimensional epipolar plane graph through a preset parallax value so as to obtain a plurality of low-frequency epipolar plane graphs after the shearing operation; the sampling module is used for carrying out angle and dimension up-sampling on the low-frequency epipolar line plane graphs after the plurality of shearing operations through a double-tri-interpolation method so as to achieve a target angle resolution; the reconstruction module is used for reconstructing angle information of the up-sampled low-frequency epipolar plane graph through a convolutional neural network; the restoring module is used for restoring the spatial high-frequency information of the plurality of two-dimensional polar line plane graphs through non-blind deblurring operation so as to obtain polar line plane graphs with high spatial angular resolution after angle super-resolution; the inverse shearing module is used for performing inverse shearing operation on the polar line plane graph after the high-frequency information is recovered through the preset parallax value so as to obtain a plurality of polar line plane graphs with high angular spatial resolution under different parallaxes; the fusion module is used for fusing the high-angle spatial resolution polar line plane maps under different parallaxes through the polar line plane map and the corresponding parallax map so as to obtain a high-spatial angular resolution polar line plane map; and the output module is used for outputting the high-angle resolution light field.
According to the light field angle super-resolution device based on the shearing two-dimensional polar line plan, the polar line plan is used for carrying out angle super-resolution on the light field with low angle resolution, and the space and angle information of the light field can be simultaneously utilized; shearing the epipolar plane graph through a series of parallax candidate values, so that the aliasing phenomenon caused by sparse angle sampling and large parallax can be reduced; the position relation of the polar line plane graph after the angle super-resolution can be restored by utilizing the reverse shearing operation related to the angle super-resolution multiplying power; by fusing a series of polar line plane graphs after super-resolution, the optimal effect can be achieved on each parallax value, so that a large parallax light field with certain sparse angular resolution can achieve an excellent angle super-resolution effect, the robustness is very high, and the robustness is also very high for noise in the parallax graph corresponding to the polar line plane graph.
In addition, the light field angle super-resolution device based on the shearing two-dimensional polar plane diagram according to the above embodiment of the present invention may also have the following additional technical features:
further, in an embodiment of the present invention, a clipping value used when the plurality of two-dimensional epipolar plane graphs are subjected to a clipping operation is a disparity candidate value, wherein the clipped low-frequency epipolar plane graph is also a two-dimensional epipolar plane graph.
Further, in one embodiment of the present invention, the preset disparity value is obtained according to a multiple of angular super-resolution.
Further, in an embodiment of the present invention, the plurality of high-angle spatial resolution epipolar plane graphs under different parallaxes are fused by a fusion formula, wherein the fusion formula is:
wherein, WiIn order to fuse the weights needed for the fusion,
the limiting plane map, i ∈ 1, 2.
Further, in an embodiment of the present invention, the high angular spatial resolution epipolar plane plot at different parallaxes is:
wherein D isκThe non-blind deblurring operation is carried out,
to use the parallax value diF is a convolutional neural network operation, ELκ is a low frequency epipolar plan.
Additional aspects and advantages of the invention will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the invention.
Drawings
The foregoing and/or additional aspects and advantages of the present invention will become apparent and readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings of which:
FIG. 1 is a flow diagram of a light field angle super-resolution method based on shearing a two-dimensional epipolar plane view according to one embodiment of the present invention;
FIG. 2 is a schematic diagram of a convolutional neural network according to one embodiment of the present invention;
FIG. 3 is a schematic structural diagram of a light field angle super-resolution device based on a cut two-dimensional epipolar plane diagram according to an embodiment of the present invention.
Detailed Description
Reference will now be made in detail to embodiments of the present invention, examples of which are illustrated in the accompanying drawings, wherein like or similar reference numerals refer to the same or similar elements or elements having the same or similar function throughout. The embodiments described below with reference to the drawings are illustrative and intended to be illustrative of the invention and are not to be construed as limiting the invention.
The method and the device for super-resolution of light field angles based on a cut two-dimensional epipolar plane graph according to an embodiment of the present invention will be described below with reference to the accompanying drawings.
FIG. 1 is a flow chart of a light field angle super-resolution method based on a shearing two-dimensional epipolar plane graph according to an embodiment of the present invention.
As shown in FIG. 1, the light field angle super-resolution method based on the shearing two-dimensional epipolar plane diagram comprises the following steps:
in step S101, a plurality of two-dimensional epipolar plane maps are extracted from the low angular resolution light field.
It can be appreciated that embodiments of the present invention extract a two-dimensional epipolar plane graph E for a low angular resolution light fieldL。
In particular, by fixing one of the four-dimensional light fieldsA two-dimensional polar plane diagram is extracted from the spatial coordinates and the angular coordinates, wherein L (x, y, s, t) is represented for a four-dimensional light field, wherein x and y are two spatial dimensions of the light field, and s and t are two angular dimensions of the light field. Two types of polar line plane graphs can be extracted by fixing a space coordinate and an angle coordinate, and the other two-dimensional polar line plane graph E is extracted by fixing y and ty*,t*(x, s), and the other is a two-dimensional epipolar plane graph E extracted by fixing x and sx*,s*(y, t). The epipolar plan contains one-dimensional spatial information and one-dimensional angular information of the light field. Hereinafter, two-dimensional polar line plane diagrams are both referred to as EL。
In step S102, spatial low frequency information of each of a plurality of two-dimensional epipolar plane maps is acquired by a one-dimensional gaussian kernel function.
It will be appreciated that embodiments of the present invention utilize a one-dimensional Gaussian kernel function κ for each epipolar line plan ELExtracting its spatial low-frequency information EL*κ。
Specifically, compared with the spatial resolution, the input light field has a low angular resolution, that is, the angular sampling of the light field is under-sampled, the difference of the view angles performed by using the depth information often has a large defect at the edge of the object, and the use of the non-depth reconstruction method to perform the angular super-resolution on the light field generates an aliasing phenomenon, which generates a ghost image in the generated new view angle. The invention uses gaussians to extract spatial low-frequency information, and the effect of the invention is equivalent to that of performing anti-aliasing processing on an epipolar plane diagram. Extraction of only epipolar line plan E using one-dimensional gaussian kernel κ -pairsLThe gaussian kernel function is:
where c is a scale adjustment parameter of the kernel function, σ is a shape adjustment parameter of the kernel function, and x ∈ [ -4 σ,4 σ ]. The kernel function is a discrete function, i.e. x only takes integer values. The scaling parameter c is such that the sum of the values of the kernel function within the interval is equal to 1. The shape adjustment parameter σ is adjusted according to the maximum parallax of adjacent viewing angles in the input light field, with the larger the parallax, the larger σ. For example, when the parallax between adjacent viewing angles of the light field is 4, σ is taken to be 1.5, and the size of the kernel function is 13 pixels.
In step S103, a clipping operation is performed on each two-dimensional epipolar plane map by a preset disparity value to obtain a plurality of low-frequency epipolar plane maps after the clipping operation.
It will be appreciated that embodiments of the invention utilise a series of disparity values for the low frequency epipolar plane plot ELSubjecting the residue to a shearing operation to obtain
Wherein the content of the first and second substances,
to use the parallax value diI ∈ 1, 2.. and N, N is the number of parallax values.
Specifically, embodiments of the present invention utilize a series of disparity values diI ∈ 1, 2.. and N, N are parallax value quantity, and the low-frequency epipolar line plan view E is obtainedLSubjecting the residue to a shearing operation to obtain
Wherein the content of the first and second substances,
to use the parallax value diThe shearing operation of (1).
Further, in an embodiment of the present invention, the clipping value used when the clipping operation is performed on the plurality of two-dimensional epipolar plane graphs is a parallax candidate value, wherein the clipped low-frequency epipolar plane graph is also a two-dimensional epipolar plane graph.
It will be appreciated that embodiments of the present invention provide a low frequency epipolar plan E of the inputLThe clipping values used in the clipping operation are a series of parallax candidates, and the resulting clipped epipolar plane graph is also a series of two-dimensional epipolar plane graphs.
In step S104, the low-frequency epipolar plane maps after the plurality of shearing operations are subjected to angular and dimensional up-sampling by a double-tri-interpolation method to achieve a target angular resolution.
It can be understood that the plan view of the low-frequency epipolar lines after the shearing operation
The angular dimension is upsampled by a bicubic interpolation method to achieve a desired angular resolution, expressed as
The up-sampling magnification is generally 2 to 4, and those skilled in the art can set the up-sampling magnification according to specific situations, and the up-sampling magnification is not particularly limited herein.
In step S105, the angle information is reconstructed from the up-sampled low-frequency epipolar plane map by the convolutional neural network.
It will be appreciated that embodiments of the present invention utilize a convolutional neural network to upsample a low frequency epipolar line plan
Reconstruct the angle information, expressed as
Where f is the convolutional neural network operation.
Specifically, as shown in fig. 2, the convolutional neural network used is a residual network:
f(E′L)＝E′L+R(E′L)，
wherein the content of the first and second substances,
is polar line plan obtained by extracting angle low-frequency information through Gaussian kernel function kappa and performing shearing operation, R is residual error network, wherein the residual error network comprises three convolution layers, the first layer L is1Comprising 64 1X 9 cores, a second layer L2Comprising 32 64X 5 cores, a third layer L3Comprising 1 32 x 5 kernel, connected to a modified linear unit behind each layer.
In the disabilityIn the training of the difference network, order expected residual r is E '-E'LWhere E' is the desired high angular resolution limit plan for extracting low frequency information via a one-dimensional kernel function. The loss function of the training is:
where n is the number of training limit plans. When training the residual network, each training limit plane graph is divided into limit plane subgraphs with the size of 17 × 17 pixels, and the division step size is 14 pixels. Every 64 limit plane subgraphs are trained as a batch. In order to avoid the influence of overfitting on the training result, data enhancement is adopted to process the training data, wherein the data enhancement comprises overturning, spatial downsampling and adding Gaussian noise with the average value of 0.
The weights of each layer of kernel in the residual network are initialized with a gaussian distribution with a mean value of 0 and a variance of 0.001. The number of iterations of the residual network training is 8 x 105Next, the process is carried out. The initial learning rate was 0.01, every 2.5 × 105The sub-iteration is reduced to 1/10 before. When passing 5X 105After the training, the learning rate was decreased to 0.0001 after two times. The impulse of the residual network is 0.9.
Training the residual network and performing light field angle super-resolution by using the network are performed under a Y channel of a YCbCr color space of a limit plan, and the other two channels are not processed. And after the angle information of the limit plane graph under the Y channel is reconstructed by using the network, the limit plane graph is synthesized into a final output limit plane graph like the other two channels.
In step S106, the spatial high-frequency information of the plurality of two-dimensional epipolar plane maps is restored by non-blind deblurring operation to obtain an epipolar plane map with high spatial angular resolution after angular super-resolution.
It will be appreciated that the non-blind deblurring operation D is utilizedκRestoring the high-frequency spatial information of the polar line plan to obtain the polar line plan with high spatial angular resolution after angular super-resolution
Wherein the non-blind deblurring operation DκThe kernel function used is a one-dimensional gaussian kernel function κ.
Since the kernel function κ for extracting the low-frequency information of the epipolar plane diagram is manually set, and the kernel function structure is not destroyed in steps S104 to S105, the embodiment of the present invention can well recover the high-frequency information removed by the gaussian kernel function κ using the non-blind deblurring operation.
In step S107, the polar plane map after the high frequency information is restored is subjected to a reverse shearing operation by a preset parallax value, so as to obtain a plurality of high-angle spatial resolution polar plane maps under different parallaxes.
In one embodiment of the invention, the preset disparity value is obtained from a multiple of the angular super-resolution.
It will be appreciated that the disparity values used in the inverse shearing operation vary according to a multiple of the angular super-resolution, i.e. diAnd/alpha, wherein alpha is the angular super-resolution magnification.
Further, in one embodiment of the present invention, the high angular spatial resolution epipolar plane plots at different parallaxes are:
wherein D isκThe non-blind deblurring operation is carried out,
to use the parallax value diF is a convolutional neural network operation, ELκ is a low frequency epipolar plan.
It will be appreciated that embodiments of the present invention utilise the disparity value diPolar line plan after high frequency information recovery
Performing reverse shearing operation to obtain a series of high angles under different parallaxesSpatial resolution epipolar plan view:
wherein the content of the first and second substances,
to use the parallax value diThe reverse shearing operation of (1).
In step S108, a plurality of high-angle spatial resolution epipolar plane maps under different parallaxes are fused by the epipolar plane map to the corresponding parallax map to obtain a high-spatial angular resolution epipolar plane map.
It can be understood that the embodiment of the invention utilizes the parallax map corresponding to the extreme plan view to perform a series of polar plan views after the anti-shearing operation
Fusing to obtain polar line plan E with high spatial angular resolutionH。
Further, in an embodiment of the present invention, a plurality of high-angle spatial resolution epipolar plane graphs under different parallaxes are fused by a fusion formula, wherein the fusion formula is:
wherein, WiIn order to fuse the weights needed for the fusion,
the limiting plane map, i ∈ 1, 2.
That is, embodiments of the present invention provide for a series of polar line plan views after a reverse shearing operation
The fusion method comprises the following steps:
wherein, WiTo fuse the required weights, when the floor plan is extreme
When the value in the disparity map corresponding to a certain pixel p is 1, Wi(p) 1, otherwise Wi(p)＝0。
In step S109, the high angular resolution light field is output.
It will be appreciated that the fused high angular resolution limit plan is output as a high angular resolution lightfield.
Specifically, the step is the reverse of step S101, and is performed by comparing the extracted two-dimensional epipolar line plan Ey*,t*(x, S) performing angular super-resolution as described in steps S102 to S108, and then subjecting the two-dimensional epipolar plane graph Ex*,s*(y, t) the angular super-resolution as described in step S102 to step S108 is performed. To two-dimensional polar line plan Ex*,s*(y, t) performing angle super-resolution, recovering to the generated new viewing angle, extracting a two-dimensional polar plane diagram, and performing angle super-resolution as described in the step S102 to the step S108, thereby completing the angle super-resolution of the whole four-dimensional light field.
In summary, the extracted two-dimensional polar plane map is firstly subjected to angle super-resolution as described in the steps B to H, and then the two-dimensional polar plane map is extracted from the new view angle generated after the angle super-resolution, and the angle super-resolution as described in the steps B to H is performed, so that the angle super-resolution of the whole light field is completed. The embodiment of the invention can perform angle super-resolution on the light field input with the sparse angle resolution and the larger parallax between the adjacent visual angles, thereby obtaining the light field with the high spatial angle resolution.
According to the light field angle super-resolution method based on the shearing two-dimensional polar plane diagram, the polar plane diagram is utilized to perform angle super-resolution on the light field with low angular resolution, and the space and angle information of the light field can be utilized simultaneously; shearing the epipolar plane graph through a series of parallax candidate values, so that the aliasing phenomenon caused by sparse angle sampling and large parallax can be reduced; the position relation of the polar line plane graph after the angle super-resolution can be restored by utilizing the reverse shearing operation related to the angle super-resolution multiplying power; by fusing a series of polar line plane graphs after super-resolution, the optimal effect can be achieved on each parallax value, so that a large parallax light field with certain sparse angular resolution can achieve an excellent angle super-resolution effect, the robustness is very high, and the robustness is also very high for noise in the parallax graph corresponding to the polar line plane graph.
Next, a light field angle super-resolution device based on a sheared two-dimensional epipolar plane diagram according to an embodiment of the present invention will be described with reference to the drawings.
FIG. 3 is a schematic structural diagram of a light field angle super-resolution device based on a sheared two-dimensional epipolar plane diagram according to an embodiment of the present invention.
As shown in fig. 3, the light field angle super-resolution device 10 based on the shearing two-dimensional epipolar plane diagram includes: the device comprises an extraction module 100, an acquisition module 200, a shearing module 300, a sampling module 400, a reconstruction module 500, a recovery module 600, an inverse shearing module 700, a fusion module 800 and an output module 900.
Wherein the extraction module 100 is configured to extract a plurality of two-dimensional epipolar plane maps from the low angular resolution light field. The acquisition module 200 is configured to acquire spatial low frequency information of each of the plurality of two-dimensional epipolar plane maps through a one-dimensional gaussian kernel function. The shearing module 300 is configured to perform a shearing operation on each two-dimensional epipolar plane map by using a preset disparity value to obtain a plurality of low-frequency epipolar plane maps after the shearing operation.
The sampling module 400 is configured to perform angular and dimensional upsampling on the low-frequency epipolar plane maps after the multiple shearing operations by a double-tri-interpolation method to achieve a target angular resolution. The reconstruction module 500 is configured to reconstruct the angle information from the upsampled low-frequency epipolar plane map by a convolutional neural network. The restoring module 600 is configured to restore spatial high-frequency information of a plurality of two-dimensional epipolar plane graphs through a non-blind deblurring operation to obtain an epipolar plane graph with high spatial angular resolution after angular super-resolution. The inverse shearing module 700 is configured to perform inverse shearing operation on the epipolar plane graph after the high-frequency information is recovered through a preset parallax value, so as to obtain a plurality of high-angle spatial resolution epipolar plane graphs under different parallaxes. The fusion module 800 is configured to fuse a plurality of high-angle spatial resolution epipolar plane maps under different parallaxes through the epipolar plane map pair corresponding to the parallax map, so as to obtain a high-spatial angular resolution epipolar plane map. The output module 900 is used to output the high angular resolution lightfield. The device 10 of the embodiment of the invention can achieve excellent angle super-resolution effect on a large parallax light field with certain sparse angle resolution, has strong robustness, and also has strong robustness on noise in a parallax image corresponding to a polar line plan.
Further, in an embodiment of the present invention, the clipping value used when the clipping operation is performed on the plurality of two-dimensional epipolar plane graphs is a parallax candidate value, wherein the clipped low-frequency epipolar plane graph is also a two-dimensional epipolar plane graph.
Further, in one embodiment of the present invention, the preset disparity value is obtained from a multiple of the angular super-resolution.
Further, in an embodiment of the present invention, a plurality of high-angle spatial resolution epipolar plane graphs under different parallaxes are fused by a fusion formula, wherein the fusion formula is:
wherein, WiIn order to fuse the weights needed for the fusion,
the limiting plane map, i ∈ 1, 2.
Further, in one embodiment of the present invention, the high angular spatial resolution epipolar plane plots at different parallaxes are:
wherein D isκThe non-blind deblurring operation is carried out,
to use the parallax value diF is a convolutional neural network operation, ELκ is a low frequency epipolar plan.
It should be noted that the foregoing explanation of the embodiment of the light field angle super-resolution method based on shearing a two-dimensional epipolar plane graph is also applicable to the light field angle super-resolution device based on shearing a two-dimensional epipolar plane graph in this embodiment, and is not repeated here.
According to the light field angle super-resolution device based on the shearing two-dimensional polar line plan, the polar line plan is used for carrying out angle super-resolution on the light field with low angular resolution, and the space and angle information of the light field can be simultaneously utilized; shearing the epipolar plane graph through a series of parallax candidate values, so that the aliasing phenomenon caused by sparse angle sampling and large parallax can be reduced; the position relation of the polar line plane graph after the angle super-resolution can be restored by utilizing the reverse shearing operation related to the angle super-resolution multiplying power; by fusing a series of polar line plane graphs after super-resolution, the optimal effect can be achieved on each parallax value, so that a large parallax light field with certain sparse angular resolution can achieve an excellent angle super-resolution effect, the robustness is very high, and the robustness is also very high for noise in the parallax graph corresponding to the polar line plane graph.
In the description of the present invention, it is to be understood that the terms "central," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," "counterclockwise," "axial," "radial," "circumferential," and the like are used in the orientations and positional relationships indicated in the drawings for convenience in describing the invention and to simplify the description, and are not intended to indicate or imply that the referenced devices or elements must have a particular orientation, be constructed and operated in a particular orientation, and are therefore not to be considered limiting of the invention.
Furthermore, the terms "first", "second" and "first" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present invention, "a plurality" means at least two, e.g., two, three, etc., unless specifically limited otherwise.
In the present invention, unless otherwise expressly stated or limited, the terms "mounted," "connected," "secured," and the like are to be construed broadly and can, for example, be fixedly connected, detachably connected, or integrally formed; can be mechanically or electrically connected; they may be directly connected or indirectly connected through intervening media, or they may be connected internally or in any other suitable relationship, unless expressly stated otherwise. The specific meanings of the above terms in the present invention can be understood by those skilled in the art according to specific situations.
In the present invention, unless otherwise expressly stated or limited, the first feature "on" or "under" the second feature may be directly contacting the first and second features or indirectly contacting the first and second features through an intermediate. Also, a first feature "on," "over," and "above" a second feature may be directly or diagonally above the second feature, or may simply indicate that the first feature is at a higher level than the second feature. A first feature being "under," "below," and "beneath" a second feature may be directly under or obliquely under the first feature, or may simply mean that the first feature is at a lesser elevation than the second feature.
In the description herein, references to the description of the term "one embodiment," "some embodiments," "an example," "a specific example," or "some examples," etc., mean that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. In this specification, the schematic representations of the terms used above are not necessarily intended to refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, various embodiments or examples and features of different embodiments or examples described in this specification can be combined and combined by one skilled in the art without contradiction.
Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention, and that variations, modifications, substitutions and alterations can be made to the above embodiments by those of ordinary skill in the art within the scope of the present invention.
Claims (10)
1. A super-resolution method for an optical field angle based on a shearing two-dimensional polar line plan is characterized by comprising the following steps:
extracting a plurality of two-dimensional epipolar plane graphs from the low angular resolution light field;
obtaining spatial low-frequency information of each two-dimensional epipolar plane map of the plurality of two-dimensional epipolar plane maps through a one-dimensional Gaussian kernel function;
performing shearing operation on each two-dimensional epipolar plane graph through a preset parallax value to obtain a plurality of low-frequency epipolar plane graphs after the shearing operation;
carrying out angle and dimension up-sampling on the low-frequency epipolar line plane graphs after the plurality of shearing operations by a double-tri-interpolation method so as to achieve a target angle resolution;
reconstructing angle information of the up-sampled low-frequency polar line plane graph through a convolutional neural network;
restoring the spatial high-frequency information of the plurality of two-dimensional polar line plane graphs through non-blind deblurring operation to obtain polar line plane graphs with high spatial angular resolution after angle super-resolution;
performing reverse shearing operation on the polar line plane graph after high-frequency information is recovered through the preset parallax value to obtain a plurality of polar line plane graphs with high angular spatial resolution under different parallaxes;
fusing the high-angle spatial resolution polar plane graphs under different parallaxes through the parallax graphs corresponding to the high-angle spatial resolution polar plane graphs under different parallaxes to obtain high-space angular resolution polar plane graphs; and
the high angular resolution lightfield is output.
2. The light field angle super-resolution method based on shearing of two-dimensional epipolar plane graph according to claim 1, wherein the shearing value used in the shearing operation of the plurality of two-dimensional epipolar plane graphs is a parallax candidate value, and wherein the sheared low-frequency epipolar plane graph is also a two-dimensional epipolar plane graph.
3. The light field angle super-resolution method based on shearing two-dimensional epipolar plane graph according to claim 1, wherein the preset disparity value is obtained according to a multiple of the angle super-resolution.
4. The light field angle super-resolution method based on shearing two-dimensional epipolar plane graph according to claim 1, wherein the plurality of high-angle spatial resolution epipolar plane graphs under different parallaxes are fused by a fusion formula, wherein the fusion formula is as follows:
wherein, WiIn order to fuse the weights needed for the fusion,
the limiting plane map, i ∈ 1, 2.
5. The light field angle super-resolution method based on shearing of two-dimensional epipolar plane graph according to any of claims 1-4, wherein the high angular spatial resolution epipolar plane graph under different parallaxes is:
wherein D isκThe non-blind deblurring operation is carried out,
to use the parallax value diF is a convolutional neural network operation, ELκ is a low-frequency polar plot,
to use the parallax value diThe reverse shearing operation of (1).
6. The utility model provides a light field angle super resolution device based on cut two-dimensional polar line plan which characterized in that includes:
the extraction module is used for extracting a plurality of two-dimensional polar line plane graphs according to the low-angle resolution light field;
an obtaining module, configured to obtain, through a one-dimensional gaussian kernel function, spatial low-frequency information of each two-dimensional epipolar plane map of the multiple two-dimensional epipolar plane maps;
the shearing module is used for carrying out shearing operation on each two-dimensional epipolar plane graph through a preset parallax value so as to obtain a plurality of low-frequency epipolar plane graphs after the shearing operation;
the sampling module is used for carrying out angle and dimension up-sampling on the low-frequency epipolar line plane graphs after the plurality of shearing operations through a double-tri-interpolation method so as to achieve a target angle resolution;
the reconstruction module is used for reconstructing angle information of the up-sampled low-frequency epipolar plane graph through a convolutional neural network;
the restoring module is used for restoring the spatial high-frequency information of the plurality of two-dimensional polar line plane graphs through non-blind deblurring operation so as to obtain polar line plane graphs with high spatial angular resolution after angle super-resolution;
the inverse shearing module is used for performing inverse shearing operation on the polar line plane graph after the high-frequency information is recovered through the preset parallax value so as to obtain a plurality of polar line plane graphs with high angular spatial resolution under different parallaxes;
the fusion module is used for fusing the high-angle spatial resolution polar plane graphs under different parallaxes through the parallax graphs corresponding to the high-angle spatial resolution polar plane graphs under different parallaxes to obtain high-space angular resolution polar plane graphs; and
and the output module is used for outputting the high-angle resolution light field.
7. The super-resolution device for light field angles based on the shearing two-dimensional epipolar plane graph according to claim 6, wherein the shearing values used in the shearing operations on the plurality of two-dimensional epipolar plane graphs are parallax candidate values, and the sheared low-frequency epipolar plane graph is also a two-dimensional epipolar plane graph.
8. The light field angle super-resolution device based on the sheared two-dimensional epipolar plane map of claim 6, wherein the preset disparity value is obtained according to a multiple of the angle super-resolution.
9. The light field angle super-resolution device based on the sheared two-dimensional epipolar plane graph according to claim 6, wherein the plurality of high-angle spatial resolution epipolar plane graphs under different parallaxes are fused by a fusion formula, wherein the fusion formula is as follows:
wherein, WiIn order to fuse the weights needed for the fusion,
the limiting plane map, i ∈ 1, 2.
10. The light field angle super-resolution device based on shearing two-dimensional epipolar plane graph according to any of claims 6-9, wherein the high-angle spatial resolution epipolar plane graph under different parallaxes is:
wherein D isκThe non-blind deblurring operation is carried out,
to use the parallax value diF is a convolutional neural network operation, ELκ is a low-frequency polar plot,
to use the parallax value diThe reverse shearing operation of (1).
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810315188.7A CN108615221B (en) | 2018-04-10 | 2018-04-10 | Light field angle super-resolution method and device based on shearing two-dimensional polar line plan |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810315188.7A CN108615221B (en) | 2018-04-10 | 2018-04-10 | Light field angle super-resolution method and device based on shearing two-dimensional polar line plan |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108615221A CN108615221A (en) | 2018-10-02 |
| CN108615221B true CN108615221B (en) | 2020-12-01 |
Family
ID=63660127
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810315188.7A Expired - Fee Related CN108615221B (en) | 2018-04-10 | 2018-04-10 | Light field angle super-resolution method and device based on shearing two-dimensional polar line plan |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108615221B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109447919B (en) * | 2018-11-08 | 2022-05-06 | 电子科技大学 | Light field super-resolution reconstruction method combining multi-view angle and semantic texture features |
| CN111932601B (en) * | 2019-09-27 | 2023-06-06 | 北京信息科技大学 | Dense depth reconstruction method based on YCbCr color space light field data |
| CN114066777B (en) * | 2021-11-30 | 2022-07-15 | 安庆师范大学 | Light field image angle reconstruction method |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105931190A (en) * | 2016-06-14 | 2016-09-07 | 西北工业大学 | High-angular-resolution light filed obtaining device and image generation method |
| CN106257454A (en) * | 2015-06-17 | 2016-12-28 | 汤姆逊许可公司 | For the method and apparatus carrying out data retrieval in light field data storehouse |
| CN106991648A (en) * | 2017-04-10 | 2017-07-28 | 中国石油大学(华东) | A kind of image super-resolution rebuilding method based on wavelet transformation and convolutional neural networks |
| CN107545586A (en) * | 2017-08-04 | 2018-01-05 | 中国科学院自动化研究所 | Based on the local depth acquisition methods of light field limit plane picture and system |
| CN107862698A (en) * | 2017-11-29 | 2018-03-30 | 首都师范大学 | Light field foreground segmentation method and device based on K mean cluster |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9179126B2 (en) * | 2012-06-01 | 2015-11-03 | Ostendo Technologies, Inc. | Spatio-temporal light field cameras |
-
2018
- 2018-04-10 CN CN201810315188.7A patent/CN108615221B/en not_active Expired - Fee Related
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106257454A (en) * | 2015-06-17 | 2016-12-28 | 汤姆逊许可公司 | For the method and apparatus carrying out data retrieval in light field data storehouse |
| CN105931190A (en) * | 2016-06-14 | 2016-09-07 | 西北工业大学 | High-angular-resolution light filed obtaining device and image generation method |
| CN106991648A (en) * | 2017-04-10 | 2017-07-28 | 中国石油大学(华东) | A kind of image super-resolution rebuilding method based on wavelet transformation and convolutional neural networks |
| CN107545586A (en) * | 2017-08-04 | 2018-01-05 | 中国科学院自动化研究所 | Based on the local depth acquisition methods of light field limit plane picture and system |
| CN107862698A (en) * | 2017-11-29 | 2018-03-30 | 首都师范大学 | Light field foreground segmentation method and device based on K mean cluster |
Non-Patent Citations (1)
| Title |
|---|
| Light Field Reconstruction Using Deep Convolutional Network on EPI;Gaochang Wu等;《2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)》;20170726;第1638-1646页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108615221A (en) | 2018-10-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Wu et al. | Light field reconstruction using deep convolutional network on EPI | |
| CN108074218B (en) | Image super-resolution method and device based on light field acquisition device | |
| CN106408524B (en) | Depth image enhancement method based on two-dimensional image assistance | |
| CN106780590B (en) | Method and system for acquiring depth map | |
| US9092890B2 (en) | Occlusion-aware reconstruction of three-dimensional scenes from light field images | |
| Bradley et al. | Accurate multi-view reconstruction using robust binocular stereo and surface meshing | |
| CN110503620B (en) | Image fusion method based on Fourier spectrum extraction | |
| CN108615221B (en) | Light field angle super-resolution method and device based on shearing two-dimensional polar line plan | |
| Malviya et al. | Image fusion of digital images | |
| CN111476242B (en) | Laser point cloud semantic segmentation method and device | |
| CN111145134B (en) | Block effect-based microlens light field camera full-focus image generation algorithm | |
| CN103440653A (en) | Binocular vision stereo matching method | |
| CN110738731B (en) | 3D reconstruction method and system for binocular vision | |
| CN104363369A (en) | Image restoration method and device for optical field camera | |
| CN110880162A (en) | Snapshot spectrum depth combined imaging method and system based on deep learning | |
| CN107392986B (en) | Image depth of field rendering method based on Gaussian pyramid and anisotropic filtering | |
| Bratsolis et al. | Comparison of optimized mathematical methods in the improvement of raster data and map display resolution of Sentinel-2 images | |
| CN115239836A (en) | Extreme sparse view angle CT reconstruction method based on end-to-end neural network | |
| Cho et al. | Depth map up-sampling using cost-volume filtering | |
| RU2716311C1 (en) | Device for reconstructing a depth map with searching for similar blocks based on a neural network | |
| Guðmundsson et al. | TOF-CCD image fusion using complex wavelets | |
| Yoo et al. | Bayesian approach for automatic joint parameter estimation in 3D image reconstruction from multi-focus microscope | |
| CN110895790A (en) | Scene image super-resolution method based on posterior degradation information estimation | |
| Rivadeneira et al. | Multi-Image Super-Resolution for Thermal Images. | |
| Liu et al. | Semi-global depth from focus |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20201201 |