CN108230223A - Light field angle super-resolution rate method and device based on convolutional neural networks - Google Patents
Light field angle super-resolution rate method and device based on convolutional neural networks Download PDFInfo
- Publication number
- CN108230223A CN108230223A CN201711462902.7A CN201711462902A CN108230223A CN 108230223 A CN108230223 A CN 108230223A CN 201711462902 A CN201711462902 A CN 201711462902A CN 108230223 A CN108230223 A CN 108230223A
- Authority
- CN
- China
- Prior art keywords
- resolution
- epipolar plane
- angle
- layer
- neural networks
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000013527 convolutional neural network Methods 0.000 title claims abstract description 56
- 238000000034 method Methods 0.000 title claims abstract description 40
- 238000005070 sampling Methods 0.000 claims abstract description 25
- 238000000605 extraction Methods 0.000 claims abstract description 16
- 238000011084 recovery Methods 0.000 claims abstract description 11
- 239000000284 extract Substances 0.000 claims description 17
- 238000003475 lamination Methods 0.000 claims description 9
- 230000006870 function Effects 0.000 description 49
- 238000012549 training Methods 0.000 description 13
- 230000000694 effects Effects 0.000 description 10
- 238000013528 artificial neural network Methods 0.000 description 6
- 230000000007 visual effect Effects 0.000 description 6
- 238000003384 imaging method Methods 0.000 description 5
- 230000003287 optical effect Effects 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 3
- 230000001629 suppression Effects 0.000 description 3
- 230000008901 benefit Effects 0.000 description 2
- 230000007547 defect Effects 0.000 description 2
- 239000000463 material Substances 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 230000001419 dependent effect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000009434 installation Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 230000011218 segmentation Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T1/00—General purpose image data processing
- G06T1/0007—Image acquisition
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/084—Backpropagation, e.g. using gradient descent
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/70—Denoising; Smoothing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/73—Deblurring; Sharpening
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Molecular Biology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Life Sciences & Earth Sciences (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a kind of light field angle super-resolution rate method and device based on convolutional neural networks, wherein, method includes：According to the multiple two-dimentional epipolar plane figures of low angle resolution light-field extraction；Space low-frequency information is extracted to each epipolar plane figure of multiple two-dimentional epipolar plane figures by one-dimensional gaussian kernel function；And pass through the up-sampling that bicubic interpolation method carries out angle dimension, reach desired angular resolution；Angle information is reconstructed to the epipolar plane figure after up-sampling by convolutional neural networks；By the spatial high-frequency information of non-blind deblurring operation recovery epipolar plane figure, the epipolar plane figure of the high spatial angular resolution after angle super-resolution rate is obtained；High angular resolution light field is exported according to the epipolar plane figure of the high spatial angular resolution after angle super-resolution rate.This method efficiently uses the texture information of two-dimentional epipolar plane figure, can improve the overall accuracy of light field super-resolution, has very strong robustness.
Description
Technical field
The present invention relates to technical field of computer vision, more particularly to a kind of light field angle based on convolutional neural networks surpasses
Resolution method and device.
Background technology
Optical field imaging is a kind of one of the most widely used means for capturing object scene three-dimensional information, is imaged with tradition
Unlike mode, light field not only records intensity of the light on some position, also records light and comes from certain on this position
Therefore the distribution situation of one angular range, is become the imaging of four dimensions from the imaging of two dimensions, including two spaces dimension and
Two angle dimensions.Early stage optical field acquisition device mainly has polyphaser array system and light field gantry system, needs customization expensive
Hardware facility.With the development of optical field imaging technology, one camera holds optical field acquisition equipment also to be occurred successively.But due to
The restriction of imaging sensor resolution ratio and the multi-dimensional data characteristic of light field, holding light-field camera, often there are space angles
The trade-off relationship of resolution ratio, that is, when needing higher spatial resolution, then camera angle resolution ratio is then relatively low, otherwise also
So.
Domestic and foreign scholars carry out light field angle super-resolution rate often through the light field using high spatial low angle resolution ratio
Or the method for visual angle difference obtains the light field of high spatial angular resolution, and can be according to being divided into two whether using depth information
Point.Generally required using the method for depth information and go out depth information of scene using input light field computation, recycle have visual angle and
Depth information renders new visual angle.Such method can be directed to larger parallax light field carry out angle super-resolution rate, but due to
The calculating of depth information is vulnerable to noise, object blocks and the influence of low texture region, and visual angle renders and is highly dependent on depth
The accuracy of information is spent, is often also easy to produce the defects of more apparent.And another kind of method is then generally basede on and utilizes limited sampling point
The principle estimated four-dimensional light field carries out light field angle super-resolution rate.Such method is accurate to light field angle super-resolution rate
Property it is higher, but tend to rely on more intensive visual angle sampling, and poor to the recovery effects of high-frequency information, have to be solved.
Invention content
The present invention is directed to solve at least some of the technical problems in related technologies.
For this purpose, an object of the present invention is to provide a kind of light field angle super-resolution rate sides based on convolutional neural networks
Method, this method this method are effectively utilized the texture information of two-dimentional epipolar plane figure, can improve the entirety of light field super-resolution
Accuracy has very strong robustness.
It is another object of the present invention to propose a kind of light field angle super-resolution rate device based on convolutional neural networks.
In order to achieve the above objectives, one aspect of the present invention embodiment proposes a kind of light field angle based on convolutional neural networks
Super-resolution method includes the following steps：According to the multiple two-dimentional epipolar plane figures of low angle resolution light-field extraction；By one-dimensional
Gaussian kernel function extracts space low-frequency information to each epipolar plane figure of the multiple two-dimentional epipolar plane figure；According to described
The space low-frequency information that each epipolar plane figure extracts carries out the up-sampling of angle dimension by bicubic interpolation method, makes it
Reach desired angular resolution；Angle information is reconstructed to the epipolar plane figure after up-sampling by convolutional neural networks；It is logical
The spatial high-frequency information of epipolar plane figure described in non-blind deblurring operation recovery is crossed, to obtain the high spatial after angle super-resolution rate
The epipolar plane figure of angular resolution；It is defeated according to the epipolar plane figure of the high spatial angular resolution after the angle super-resolution rate
Go out high angular resolution light field.
The light field angle super-resolution rate method based on convolutional neural networks of the embodiment of the present invention, by using epipolar plane
Figure carries out the light field of low angle resolution ratio angle super-resolution rate, and utilizes the space of light field and angle information simultaneously；One-dimensional height
This core is by extracting the low-frequency information of epipolar plane figure, it is possible to reduce the aliasing brought due to sparse angular samples；Profit
Only restore angle high-frequency information with residual error network, not only improve training speed, but also improve network overall accuracy；By known
The non-blind deblurring algorithm of kernel function restores spatial high-frequency information, can be optimal good recovery effects；And to certain dilute
Excellent angle super-resolution rate effect can be reached by dredging the light field of angular resolution, have very strong robustness.
In addition, the light field angle super-resolution rate method according to the above embodiment of the present invention based on convolutional neural networks may be used also
With with following additional technical characteristic：
Further, in one embodiment of the invention, the one-dimensional gaussian kernel function is：
Wherein, c is the scale adjustment parameter of kernel function, and σ is the shape adjustment parameter of kernel function, and x is space coordinate, and κ is
One-dimensional Gaussian convolution core.
Further, in one embodiment of the invention, the convolutional neural networks are residual error network, and the convolution
Neural network is：
f(E′L)=E 'L+R(E′L),
Wherein, E 'L=(EL* κ) ↑ it is epipolar plane figure after gaussian kernel function κ extracts angle low-frequency information, R is residual
Poor network.
Further, in one embodiment of the invention, after obtaining the angle super-resolution rate by the following formula
The epipolar plane figure of high spatial angular resolution：
Wherein, non-blind deblurring operation DκThe kernel function used is one-dimensional gaussian kernel function κ.
Further, in one embodiment of the invention, the residual error network includes three-layer coil lamination, and first rolls up layer by layer
Lamination L1Include the core of 64 1 × 9 × 9, the second convolutional layer L layer by layer2Include the core of 32 64 × 5 × 5, third convolutional layer layer by layer
L3Include the core of 1 32 × 5 × 5, be all connected with after each layer there are one correct linear unit.
In order to achieve the above objectives, another aspect of the present invention embodiment proposes a kind of light field angle based on convolutional neural networks
Super-resolution device is spent, including：First extraction module, for according to the multiple two-dimentional epipolar planes of low angle resolution light-field extraction
Figure；Second extraction module, for passing through each epipolar plane of the one-dimensional gaussian kernel function to the multiple two-dimentional epipolar plane figure
Figure extracts space low-frequency information；Sampling module, for the space low-frequency information extracted according to each epipolar plane figure
The up-sampling of angle dimension is carried out by bicubic interpolation method, reaches desired angular resolution；Module is rebuild, is used for
Angle information is reconstructed to the epipolar plane figure after up-sampling by convolutional neural networks；Acquisition module, for passing through non-blind go
Fuzzy operation restores the spatial high-frequency information of the epipolar plane figure, angle-resolved to obtain the high spatial after angle super-resolution rate
The epipolar plane figure of rate；Output module, for being put down according to the polar curve of the high spatial angular resolution after the angle super-resolution rate
Face figure output high angular resolution light field.
The light field angle super-resolution rate device based on convolutional neural networks of the embodiment of the present invention, by using epipolar plane
Figure carries out the light field of low angle resolution ratio angle super-resolution rate, and utilizes the space of light field and angle information simultaneously；One-dimensional height
This core is by extracting the low-frequency information of epipolar plane figure, it is possible to reduce the aliasing brought due to sparse angular samples；Profit
Only restore angle high-frequency information with residual error network, not only improve training speed, but also improve network overall accuracy；By known
The non-blind deblurring algorithm of kernel function restores spatial high-frequency information, can be optimal good recovery effects；And to certain dilute
Excellent angle super-resolution rate effect can be reached by dredging the light field of angular resolution, have very strong robustness.
In addition, the light field angle super-resolution rate device according to the above embodiment of the present invention based on convolutional neural networks may be used also
With with following additional technical characteristic：
Further, in one embodiment of the invention, the one-dimensional gaussian kernel function is：
Wherein, c is the scale adjustment parameter of kernel function, and σ is the shape adjustment parameter of kernel function, and x is space coordinate, and κ is
One-dimensional Gaussian convolution core.
Further, in one embodiment of the invention, the convolutional neural networks are residual error network, and the convolution
Neural network is：
f(E′L)=E 'L+R(E′L),
Wherein, E 'L=(EL* κ) ↑ it is epipolar plane figure after gaussian kernel function κ extracts angle low-frequency information, R is residual
Poor network.
Further, in one embodiment of the invention, after obtaining the angle super-resolution rate by the following formula
The epipolar plane figure of high spatial angular resolution：
Wherein, non-blind deblurring operation DκThe kernel function used is one-dimensional gaussian kernel function κ.
Further, in one embodiment of the invention, the residual error network includes three-layer coil lamination, and first rolls up layer by layer
Lamination L1Include the core of 64 1 × 9 × 9, the second convolutional layer L layer by layer2Include the core of 32 64 × 5 × 5, third convolutional layer layer by layer
L3Include the core of 1 32 × 5 × 5, be all connected with after each layer there are one correct linear unit.
The additional aspect of the present invention and advantage will be set forth in part in the description, and will partly become from the following description
It obtains significantly or is recognized by the practice of the present invention.
Description of the drawings
Above-mentioned and/or additional aspect and advantage of the invention will become from the following description of the accompanying drawings of embodiments
Significantly and it is readily appreciated that, wherein：
Fig. 1 is the flow according to the light field angle super-resolution rate method based on convolutional neural networks of the embodiment of the present invention
Figure；
Fig. 2 is the stream according to the light field angle super-resolution rate method based on convolutional neural networks of one embodiment of the invention
Cheng Tu；
Fig. 3 is is structure diagram according to the convolutional neural networks of one embodiment of the invention；
Fig. 4 is to be shown according to the structure of the light field angle super-resolution rate device based on convolutional neural networks of the embodiment of the present invention
It is intended to.
Specific embodiment
The embodiment of the present invention is described below in detail, the example of the embodiment is shown in the drawings, wherein from beginning to end
Same or similar label represents same or similar element or the element with same or like function.Below with reference to attached
The embodiment of figure description is exemplary, it is intended to for explaining the present invention, and is not considered as limiting the invention.
The light field angle oversubscription based on convolutional neural networks proposed according to embodiments of the present invention is described with reference to the accompanying drawings
Resolution method and device describes the light field based on convolutional neural networks proposed according to embodiments of the present invention with reference to the accompanying drawings first
Angle super-resolution rate method.
Fig. 1 is the flow chart of the light field angle super-resolution rate method based on convolutional neural networks of the embodiment of the present invention.
As shown in Figure 1, the light field angle super-resolution rate method based on convolutional neural networks of being somebody's turn to do includes the following steps：
In step S101, according to the multiple two-dimentional epipolar plane figures of low angle resolution light-field extraction.
It is understood that with reference to shown in Fig. 1 and Fig. 2, the embodiment of the present invention can extract low angle resolution light-field
Two-dimentional epipolar plane figure.
Specifically, the embodiment of the present invention is extracted by a space coordinate and an angle coordinate for fixed four-dimensional light field
Two-dimentional epipolar plane figure, wherein, L (x, y, s, t) can be expressed as to four-dimensional light field, wherein, x, y are that the two spaces of light field are tieed up
Degree, s, t are two angle dimensions of light field；And pass through and fix a space coordinate and an angle coordinate can extract out two class poles
Line plan view, a kind of is the two-dimentional epipolar plane figure extracted by fixed y and tAnother kind is by fixed x
The two-dimentional epipolar plane figure extracted with sEpipolar plane figure includes the one-dimensional space information of light field and one-dimensional angle
Information is spent, hereinafter to two kinds of two-dimentional epipolar plane figure abbreviation EL。
In step s 102, each epipolar plane figure of multiple two-dimentional epipolar plane figures is carried by one-dimensional gaussian kernel function
Take out space low-frequency information.
Wherein, in one embodiment of the invention, one-dimensional gaussian kernel function is：
Wherein, c is the scale adjustment parameter of kernel function, and σ is the shape adjustment parameter of kernel function, and x is space coordinate, and κ is
One-dimensional Gaussian convolution core.
It is understood that the embodiment of the present invention can utilize one-dimensional gaussian kernel function κ to each epipolar plane figure ELIt carries
Take out its space low-frequency information EL*κ。
Specifically, for spatial resolution, the light field angular resolution inputted is relatively low, that is, the angle of light field
There are lack sampling situations for degree sampling, and visual angle difference is carried out often in object edge there are larger defect using depth information, and
Aliasing can be occurred by carrying out angle super-resolution rate to it using non-depth reconstruction method, and weight is generated in the New Century Planned Textbook of generation
Shadow.
In addition, the embodiment of the present invention extracts space low-frequency information by using Gauss, effect is equivalent to epipolar plane
Figure carries out anti-aliasing processing, and using one-dimensional gaussian kernel function κ to only extracting epipolar plane figure ELSpace low-frequency information, Gauss
Kernel function is：
Wherein, scale adjustment parameters of the c for kernel function, shape adjustment parameters of the σ for kernel function, x ∈ [- 4 σ, 4 σ], core letter
Number is discrete function, that is, x only rounding numerical value, and scale adjustment parameter c makes value in section of kernel function and equal to 1, shape
Shape adjustment parameter σ is adjusted according to the maximum disparity of adjacent view in input light field, and the more big then σ of parallax is bigger, for example, working as light
When parallax is 4 between the adjacent view of field, σ=1.5 are taken, the size of kernel function is 13 pixels.
In step s 103, the space low-frequency information extracted according to each epipolar plane figure passes through bicubic interpolation method
The up-sampling of angle dimension is carried out, reaches desired angular resolution.
It is understood that the embodiment of the present invention can be to the low frequency polar curve plan view E that extractsL* κ is inserted using bicubic
Value method carries out the up-sampling of angle dimension, reaches desired angular resolution, is expressed as (EL* κ) ↑, wherein, it is general on
It is 2 to 4 to sample multiplying power.
In step S104, angle information is reconstructed to the epipolar plane figure after up-sampling by convolutional neural networks.
Further, in one embodiment of the invention, convolutional neural networks are residual error network, and convolutional neural networks
For：
f(E′L)=E 'L+R(E′L),
Wherein, E 'L=(EL* κ) ↑ it is epipolar plane figure after gaussian kernel function κ extracts angle low-frequency information, R is residual
Poor network.
Further, in one embodiment of the invention, residual error network includes three-layer coil lamination, the first convolutional layer layer by layer
L1Include the core of 64 1 × 9 × 9, the second convolutional layer L layer by layer2Include the core of 32 64 × 5 × 5, third convolutional layer L layer by layer3Packet
Include the core of 1 32 × 5 × 5, be all connected with after each layer there are one correct linear unit.
It is understood that the embodiment of the present invention can utilize convolutional neural networks to the low frequency epipolar plane after up-sampling
Scheme (EL* κ) ↑ angle information is reconstructed, it is expressed as f ((EL* κ) ↑), wherein, f is operated for convolutional neural networks, and network structure is such as
Shown in Fig. 3.
Specifically, convolutional neural networks used in the embodiment of the present invention can be residual error network：
f(EL')=EL′+R(EL'),
Wherein, E 'L=(EL* κ) ↑ it is epipolar plane figure after gaussian kernel function κ extracts angle low-frequency information, R is residual
Poor network, wherein, residual error network includes three-layer coil lamination, first layer L1Include the core of 64 1 × 9 × 9, second layer L2Including 32
A 64 × 5 × 5 core, third layer L3Include the core of 1 32 × 5 × 5, be all connected with after each layer there are one correct linear unit.
In addition, in the training of residual error network, enable and it is expected residual error r=E '-E 'L,
Wherein, E ' is by the expectation high angular resolution limit plan view of one-dimensional kernel function extraction low-frequency information, training
Loss function be：
Wherein, n is the quantity of training limit plan view.When being trained to residual error network, each training limit is put down
Face figure is divided into the limit plane subgraph that size is 17 × 17 pixels, and the step-length of segmentation is 14 pixels；Every 64 limit planes
Subgraph is trained as a batch；For over-fitting is avoided to impact training result, the embodiment of the present invention is increased using data
Training data is handled by force, wherein, the Gaussian noise for being 0 including overturning, space down-sampling and addition mean value.
It is understood that the embodiment of the present invention can be 0 side using mean value to the weights of each layer of core in residual error network
Difference is initialized for 0.001 Gaussian Profile；The iterations of residual error network training are 8 × 105It is secondary；Initially learning rate is
0.01, every 2.5 × 105Secondary iteration be reduced to before 1/10；When by 5 × 105After secondary training, learning rate by declining twice
Become 0.0001；The momentum of residual error network is 0.9；Residual error network is trained and is used a network for light field angle oversubscription
Resolution carries out under the Y channels of the YCbCr color spaces of limit plan view, and other two channels are without processing.It utilizes
After network carries out the limit plan view under Y channels the reconstruction of angle information, then by it as other two channel synthesizes finally
The limit plan view of output.
In step S105, by the spatial high-frequency information of non-blind deblurring operation recovery epipolar plane figure, to obtain angle
Spend the epipolar plane figure of the high spatial angular resolution after super-resolution.
Further, in one embodiment of the invention, the high-altitude after angle super-resolution rate is obtained by the following formula
Between angular resolution epipolar plane figure：
Wherein, non-blind deblurring operation DκThe kernel function used is one-dimensional gaussian kernel function κ.
It is understood that the embodiment of the present invention can utilize non-blind deblurring to operate DκRestore the space of epipolar plane figure
High-frequency information obtains the epipolar plane figure of the high spatial angular resolution after angle super-resolution rate：
Wherein, non-blind deblurring operation DκThe kernel function used is one-dimensional gaussian kernel function κ.It is put down due to being used to extracting polar curve
The kernel function κ of the low-frequency information of face figure is artificial setting, and step S103 to S104 does not destroy kernel function structure, therefore, this hair
Bright embodiment can preferably restore the high-frequency information removed by gaussian kernel function κ using the operation of non-blind deblurring.
In step s 106, the angle of elevation is exported according to the epipolar plane figure of the high spatial angular resolution after angle super-resolution rate
Spend resolution light-field.
It is understood that the limit plan view output after progress angle super-resolution rate can be height by the embodiment of the present invention
Angular resolution light field.
Specifically, which is the inverse process of step S101, passes through the two-dimentional epipolar plane figure to extracting
Carry out such as described angle super-resolution rates of step S102 to step S105, then to two-dimentional epipolar plane figureIt carries out such as
The described angle super-resolution rates of step S102 to step S105；To two-dimentional epipolar plane figureCarry out angle super-resolution
The New Century Planned Textbook that generation is reverted to after rate extracts two-dimentional epipolar plane figure, then carries out as step S102 is described to step S105
Angle super-resolution rate, so as to complete the angle super-resolution rate to entire four-dimensional light field.
The light field angle super-resolution rate method based on convolutional neural networks proposed according to embodiments of the present invention, by using
Epipolar plane figure carries out angle super-resolution rate to the light field of low angle resolution ratio, and is believed simultaneously using the space of light field and angle
Breath；One-dimensional Gaussian kernel is by extracting the low-frequency information of epipolar plane figure, it is possible to reduce due to sparse angular samples bring it is mixed
Folded phenomenon；Only restore angle high-frequency information using residual error network, not only improve training speed, but also it is integrally accurate to improve network
Degree；Spatial high-frequency information is restored by the non-blind deblurring algorithm of known kernel function, good recovery effects can be optimal；And
And excellent angle super-resolution rate effect can be reached to the light field of certain sparse angular resolution ratio, there is very strong robustness,
And due to the natural noise suppression effect of neural network, also there is robustness to noisy input.
The light field angle oversubscription based on convolutional neural networks proposed according to embodiments of the present invention referring next to attached drawing description
Resolution device.
Fig. 4 is the structural representation of the light field angle super-resolution rate device based on convolutional neural networks of the embodiment of the present invention
Figure.
Include as shown in figure 4, being somebody's turn to do the light field angle super-resolution rate device 10 based on convolutional neural networks：First extraction module
100th, the second extraction module 200, sampling module 300, reconstruction module 400, acquisition module 500 and output module 600.
Wherein, the first extraction module 100 is used for according to the multiple two-dimentional epipolar plane figures of low angle resolution light-field extraction.The
Two extraction modules 200 are used to extract each epipolar plane figure of multiple two-dimentional epipolar plane figures by one-dimensional gaussian kernel function
Space low-frequency information.The space low-frequency information that sampling module 300 is used to be extracted according to each epipolar plane figure passes through bicubic
Interpolation method carries out the up-sampling of angle dimension, reaches desired angular resolution.Module 400 is rebuild for passing through convolution
Neural network reconstructs angle information to the epipolar plane figure after up-sampling.Acquisition module 500 is used to grasp by non-blind deblurring
Make the spatial high-frequency information of recovery epipolar plane figure, put down with the polar curve for obtaining the high spatial angular resolution after angle super-resolution rate
Face figure.Output module 600 is used to export the angle of elevation according to the epipolar plane figure of the high spatial angular resolution after angle super-resolution rate
Spend resolution light-field.The device 10 of the embodiment of the present invention not only improves training speed, but also improves network overall accuracy, has
Very strong robustness, and due to the natural noise suppression effect of neural network, also there is robustness to noisy input.
Further, in one embodiment of the invention, one-dimensional gaussian kernel function is：
Wherein, c is the scale adjustment parameter of kernel function, and σ is the shape adjustment parameter of kernel function, and x is space coordinate, and κ is
One-dimensional Gaussian convolution core.
Further, in one embodiment of the invention, convolutional neural networks are residual error network, and convolutional neural networks
For：
f(E′L)=E 'L+R(E′L),
Wherein, E 'L=(EL* κ) ↑ it is epipolar plane figure after gaussian kernel function κ extracts angle low-frequency information, R is residual
Poor network.
Further, in one embodiment of the invention, the high-altitude after angle super-resolution rate is obtained by the following formula
Between angular resolution epipolar plane figure：
Wherein, non-blind deblurring operation DκThe kernel function used is one-dimensional gaussian kernel function κ.
Further, in one embodiment of the invention, residual error network includes three-layer coil lamination, the first convolutional layer layer by layer
L1Include the core of 64 1 × 9 × 9, the second convolutional layer L layer by layer2Include the core of 32 64 × 5 × 5, third convolutional layer L layer by layer3Packet
Include the core of 1 32 × 5 × 5, be all connected with after each layer there are one correct linear unit.
It should be noted that the aforementioned explanation to the light field angle super-resolution rate embodiment of the method based on convolutional neural networks
Illustrate to be also applied for the light field angle super-resolution rate device based on convolutional neural networks of the embodiment, details are not described herein again.
The light field angle super-resolution rate device based on convolutional neural networks proposed according to embodiments of the present invention, by using
Epipolar plane figure carries out angle super-resolution rate to the light field of low angle resolution ratio, and is believed simultaneously using the space of light field and angle
Breath；One-dimensional Gaussian kernel is by extracting the low-frequency information of epipolar plane figure, it is possible to reduce due to sparse angular samples bring it is mixed
Folded phenomenon；Only restore angle high-frequency information using residual error network, not only improve training speed, but also it is integrally accurate to improve network
Degree；Spatial high-frequency information is restored by the non-blind deblurring algorithm of known kernel function, good recovery effects can be optimal；And
And excellent angle super-resolution rate effect can be reached to the light field of certain sparse angular resolution ratio, there is very strong robustness,
And due to the natural noise suppression effect of neural network, also there is robustness to noisy input.
In the description of the present invention, it is to be understood that term " " center ", " longitudinal direction ", " transverse direction ", " length ", " width ",
" thickness ", " on ", " under ", "front", "rear", "left", "right", " vertical ", " level ", " top ", " bottom " " interior ", " outer ", " up time
The orientation or position relationship of the instructions such as needle ", " counterclockwise ", " axial direction ", " radial direction ", " circumferential direction " be based on orientation shown in the drawings or
Position relationship is for only for ease of the description present invention and simplifies description rather than instruction or imply that signified device or element must
There must be specific orientation, with specific azimuth configuration and operation, therefore be not considered as limiting the invention.
In addition, term " first ", " second " are only used for description purpose, and it is not intended that instruction or hint relative importance
Or the implicit quantity for indicating indicated technical characteristic.Define " first " as a result, the feature of " second " can be expressed or
Implicitly include at least one this feature.In the description of the present invention, " multiple " are meant that at least two, such as two, three
It is a etc., unless otherwise specifically defined.
In the present invention unless specifically defined or limited otherwise, term " installation ", " connected ", " connection ", " fixation " etc.
Term should be interpreted broadly, for example, it may be being fixedly connected or being detachably connected or integral；Can be that machinery connects
It connects or is electrically connected；It can be directly connected, can also be indirectly connected by intermediary, can be in two elements
The connection in portion or the interaction relationship of two elements, unless otherwise restricted clearly.For those of ordinary skill in the art
For, the concrete meaning of above-mentioned term in the present invention can be understood as the case may be.
In the present invention unless specifically defined or limited otherwise, fisrt feature can be with "above" or "below" second feature
It is that the first and second features are in direct contact or the first and second features pass through intermediary mediate contact.Moreover, fisrt feature exists
Second feature " on ", " top " and " above " but fisrt feature right over second feature or oblique upper or be merely representative of
Fisrt feature level height is higher than second feature.Fisrt feature second feature " under ", " lower section " and " below " can be
One feature is immediately below second feature or obliquely downward or is merely representative of fisrt feature level height less than second feature.
In the description of this specification, reference term " one embodiment ", " example ", " is specifically shown " some embodiments "
The description of example " or " some examples " etc. means specific features, structure, material or the spy for combining the embodiment or example description
Point is contained at least one embodiment of the present invention or example.In the present specification, schematic expression of the above terms are not
It must be directed to identical embodiment or example.Moreover, particular features, structures, materials, or characteristics described can be in office
It is combined in an appropriate manner in one or more embodiments or example.In addition, without conflicting with each other, the skill of this field
Art personnel can tie the different embodiments or examples described in this specification and the feature of different embodiments or examples
It closes and combines.
Although the embodiments of the present invention has been shown and described above, it is to be understood that above-described embodiment is example
Property, it is impossible to limitation of the present invention is interpreted as, those of ordinary skill in the art within the scope of the invention can be to above-mentioned
Embodiment is changed, changes, replacing and modification.
Claims (10)
- A kind of 1. light field angle super-resolution rate method based on convolutional neural networks, which is characterized in that include the following steps：According to the multiple two-dimentional epipolar plane figures of low angle resolution light-field extraction；Space low-frequency is extracted to each epipolar plane figure of the multiple two-dimentional epipolar plane figure by one-dimensional gaussian kernel function Information；Angle dimension is carried out by bicubic interpolation method according to the space low-frequency information that each epipolar plane figure extracts Up-sampling, reach desired angular resolution；Angle information is reconstructed to the epipolar plane figure after up-sampling by convolutional neural networks；By the spatial high-frequency information of epipolar plane figure described in non-blind deblurring operation recovery, after obtaining angle super-resolution rate The epipolar plane figure of high spatial angular resolution；AndHigh angular resolution light field is exported according to the epipolar plane figure of the high spatial angular resolution after the angle super-resolution rate.
- 2. the light field angle super-resolution rate method according to claim 1 based on convolutional neural networks, which is characterized in that institute Stating one-dimensional gaussian kernel function is：Wherein, c is the scale adjustment parameter of kernel function, and σ is the shape adjustment parameter of kernel function, and x is space coordinate, and κ is one-dimensional Gaussian convolution core.
- 3. the light field angle super-resolution rate method according to claim 1 based on convolutional neural networks, which is characterized in that institute Convolutional neural networks are stated as residual error network, and the convolutional neural networks are：f(E′L)=E 'L+R(E′L),Wherein, E 'L=(EL* κ) ↑ be by gaussian kernel function κ extract angle low-frequency information after epipolar plane figure, R be residual error net Network.
- 4. the light field angle super-resolution rate method according to claim 1 based on convolutional neural networks, which is characterized in that logical Cross the epipolar plane figure of the high spatial angular resolution after the following formula acquisition angle super-resolution rate：Wherein, non-blind deblurring operation DκThe kernel function used is one-dimensional gaussian kernel function κ.
- 5. the light field angle super-resolution rate method according to claim 4 based on convolutional neural networks, which is characterized in that institute It states residual error network and includes three-layer coil lamination, the first convolutional layer L layer by layer1Include the core of 64 1 × 9 × 9, the second convolutional layer L layer by layer2 Include the core of 32 64 × 5 × 5, third convolutional layer L layer by layer3Include the core of 1 32 × 5 × 5, be all connected with after each layer there are one Correct linear unit.
- 6. a kind of light field angle super-resolution rate device based on convolutional neural networks, which is characterized in that including：First extraction module, for according to the multiple two-dimentional epipolar plane figures of low angle resolution light-field extraction；Second extraction module, for passing through each epipolar plane of the one-dimensional gaussian kernel function to the multiple two-dimentional epipolar plane figure Figure extracts space low-frequency information；Sampling module, the space low-frequency information for being extracted according to each epipolar plane figure pass through bicubic interpolation method The up-sampling of angle dimension is carried out, reaches desired angular resolution；Module is rebuild, angle information is reconstructed to the epipolar plane figure after up-sampling for passing through convolutional neural networks；Acquisition module, for by the spatial high-frequency information of epipolar plane figure described in non-blind deblurring operation recovery, to obtain angle Spend the epipolar plane figure of the high spatial angular resolution after super-resolution；AndOutput module, for exporting the angle of elevation according to the epipolar plane figure of the high spatial angular resolution after the angle super-resolution rate Spend resolution light-field.
- 7. the light field angle super-resolution rate device according to claim 6 based on convolutional neural networks, which is characterized in that institute Stating one-dimensional gaussian kernel function is：Wherein, c is the scale adjustment parameter of kernel function, and σ is the shape adjustment parameter of kernel function, and x is space coordinate, and κ is one-dimensional Gaussian convolution core.
- 8. the light field angle super-resolution rate device according to claim 6 based on convolutional neural networks, which is characterized in that institute Convolutional neural networks are stated as residual error network, and the convolutional neural networks are：f(E′L)=E 'L+R(E′L),Wherein, EL'=(EL* κ) ↑ be by gaussian kernel function κ extract angle low-frequency information after epipolar plane figure, R be residual error net Network.
- 9. the light field angle super-resolution rate device according to claim 6 based on convolutional neural networks, which is characterized in that logical Cross the epipolar plane figure of the high spatial angular resolution after the following formula acquisition angle super-resolution rate：Wherein, non-blind deblurring operation DκThe kernel function used is one-dimensional gaussian kernel function κ.
- 10. the light field angle super-resolution rate device according to claim 9 based on convolutional neural networks, which is characterized in that The residual error network includes three-layer coil lamination, the first convolutional layer L layer by layer1Include the core of 64 1 × 9 × 9, the second convolutional layer layer by layer L2Include the core of 32 64 × 5 × 5, third convolutional layer L layer by layer3Include the core of 1 32 × 5 × 5, one is respectively connected with after each layer A amendment linear unit.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711462902.7A CN108230223A (en) | 2017-12-28 | 2017-12-28 | Light field angle super-resolution rate method and device based on convolutional neural networks |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711462902.7A CN108230223A (en) | 2017-12-28 | 2017-12-28 | Light field angle super-resolution rate method and device based on convolutional neural networks |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN108230223A true CN108230223A (en) | 2018-06-29 |
Family
ID=62646568
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711462902.7A Pending CN108230223A (en) | 2017-12-28 | 2017-12-28 | Light field angle super-resolution rate method and device based on convolutional neural networks |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108230223A (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109345449A (en) * | 2018-07-17 | 2019-02-15 | 西安交通大学 | A kind of image super-resolution based on converged network and remove non-homogeneous blur method |
| CN110441271A (en) * | 2019-07-15 | 2019-11-12 | 清华大学 | Light field high-resolution deconvolution method and system based on convolutional neural networks |
| CN110823094A (en) * | 2019-11-08 | 2020-02-21 | 北京理工大学 | Point light source three-dimensional coordinate measuring method and device |
| CN111818298A (en) * | 2020-06-08 | 2020-10-23 | 北京航空航天大学 | High-definition video monitoring system and method based on light field |
| CN111932601A (en) * | 2019-09-27 | 2020-11-13 | 北京信息科技大学 | Dense depth reconstruction method based on YCbCr color space light field data |
| CN113256772A (en) * | 2021-05-10 | 2021-08-13 | 华中科技大学 | Double-angle light field high-resolution reconstruction system and method based on visual angle conversion |
| CN113344800A (en) * | 2020-03-02 | 2021-09-03 | 通用汽车环球科技运作有限责任公司 | System and method for training a non-blind image deblurring module |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104899870A (en) * | 2015-05-15 | 2015-09-09 | 清华大学深圳研究生院 | Depth estimation method based on light-field data distribution |
-
2017
- 2017-12-28 CN CN201711462902.7A patent/CN108230223A/en active Pending
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104899870A (en) * | 2015-05-15 | 2015-09-09 | 清华大学深圳研究生院 | Depth estimation method based on light-field data distribution |
Non-Patent Citations (1)
| Title |
|---|
| GAOCHANG WU 等: "Light Field Reconstruction Using Deep Convolutional Network on EPI", 《2017 IEEE CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION(CVPR)》 * |
Cited By (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109345449A (en) * | 2018-07-17 | 2019-02-15 | 西安交通大学 | A kind of image super-resolution based on converged network and remove non-homogeneous blur method |
| CN109345449B (en) * | 2018-07-17 | 2020-11-10 | 西安交通大学 | Image super-resolution and non-uniform blur removing method based on fusion network |
| US11928792B2 (en) | 2018-07-17 | 2024-03-12 | Xi'an Jiaotong University | Fusion network-based method for image super-resolution and non-uniform motion deblurring |
| CN110441271A (en) * | 2019-07-15 | 2019-11-12 | 清华大学 | Light field high-resolution deconvolution method and system based on convolutional neural networks |
| CN111932601A (en) * | 2019-09-27 | 2020-11-13 | 北京信息科技大学 | Dense depth reconstruction method based on YCbCr color space light field data |
| CN111932601B (en) * | 2019-09-27 | 2023-06-06 | 北京信息科技大学 | Dense depth reconstruction method based on YCbCr color space light field data |
| CN110823094A (en) * | 2019-11-08 | 2020-02-21 | 北京理工大学 | Point light source three-dimensional coordinate measuring method and device |
| CN113344800A (en) * | 2020-03-02 | 2021-09-03 | 通用汽车环球科技运作有限责任公司 | System and method for training a non-blind image deblurring module |
| CN113344800B (en) * | 2020-03-02 | 2023-09-29 | 通用汽车环球科技运作有限责任公司 | System and method for training non-blind image deblurring module |
| CN111818298A (en) * | 2020-06-08 | 2020-10-23 | 北京航空航天大学 | High-definition video monitoring system and method based on light field |
| CN113256772A (en) * | 2021-05-10 | 2021-08-13 | 华中科技大学 | Double-angle light field high-resolution reconstruction system and method based on visual angle conversion |
| CN113256772B (en) * | 2021-05-10 | 2023-08-01 | 华中科技大学 | Double-angle light field high-resolution reconstruction system and method based on visual angle conversion |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108230223A (en) | Light field angle super-resolution rate method and device based on convolutional neural networks | |
| CN110570353A (en) | Dense connection generation countermeasure network single image super-resolution reconstruction method | |
| CN105069746B (en) | Video real-time face replacement method and its system based on local affine invariant and color transfer technology | |
| CN107358576A (en) | Depth map super resolution ratio reconstruction method based on convolutional neural networks | |
| Livny et al. | Automatic reconstruction of tree skeletal structures from point clouds | |
| CN108596961A (en) | Point cloud registration method based on Three dimensional convolution neural network | |
| CN102930518B (en) | Improved sparse representation based image super-resolution method | |
| CN105335972B (en) | Knitted fabric defect detection method based on small echo contourlet transform and vision significance | |
| CN110796694A (en) | Fruit three-dimensional point cloud real-time acquisition method based on KinectV2 | |
| Kasem et al. | Spatial transformer generative adversarial network for robust image super-resolution | |
| CN109000557A (en) | A kind of nuclear fuel rod pose automatic identifying method | |
| CN110363862B (en) | Three-dimensional grid sequence compression method based on human body template alignment | |
| CN107154038A (en) | A kind of visual fracture of rib aided diagnosis method of rib | |
| CN105139339A (en) | Polarization image super-resolution reconstruction method based on multi-level filtering and sample matching | |
| CN110009745B (en) | Method for extracting plane from point cloud according to plane element and model drive | |
| CN106204701A (en) | A kind of rendering intent based on light probe interpolation dynamic calculation indirect reference Gao Guang | |
| Kang et al. | Competitive learning of facial fitting and synthesis using uv energy | |
| CN108615221A (en) | Light field angle super-resolution rate method and device based on the two-dimentional epipolar plane figure of shearing | |
| Li | Image super-resolution using attention based densenet with residual deconvolution | |
| CN110288529A (en) | A kind of single image super resolution ratio reconstruction method being locally synthesized network based on recurrence | |
| Wolf et al. | Surface Reconstruction from Gaussian Splatting via Novel Stereo Views | |
| CN114693755B (en) | Non-rigid registration method and system for multimode image maximum moment and space consistency | |
| CN116152060A (en) | Double-feature fusion guided depth image super-resolution reconstruction method | |
| CN110264428A (en) | A kind of medical image denoising method based on the deconvolution of 3D convolution and generation confrontation network | |
| CN113486928B (en) | Multi-view image alignment method based on rational polynomial model differentiable tensor expression |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WD01 | Invention patent application deemed withdrawn after publication | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20180629 |