CN105118040B - File and picture distortion correction method based on structure laser rays - Google Patents
File and picture distortion correction method based on structure laser rays Download PDFInfo
- Publication number
- CN105118040B CN105118040B CN201510599577.3A CN201510599577A CN105118040B CN 105118040 B CN105118040 B CN 105118040B CN 201510599577 A CN201510599577 A CN 201510599577A CN 105118040 B CN105118040 B CN 105118040B
- Authority
- CN
- China
- Prior art keywords
- mrow
- laser rays
- msub
- file
- msup
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 54
- 238000012937 correction Methods 0.000 title claims abstract description 37
- 238000000605 extraction Methods 0.000 claims abstract description 10
- 239000000284 extract Substances 0.000 claims abstract description 9
- 238000004422 calculation algorithm Methods 0.000 claims description 18
- 238000005457 optimization Methods 0.000 claims description 3
- 238000005516 engineering process Methods 0.000 description 31
- 230000006870 function Effects 0.000 description 27
- 238000010586 diagram Methods 0.000 description 6
- 230000008901 benefit Effects 0.000 description 5
- 238000004590 computer program Methods 0.000 description 5
- 230000008569 process Effects 0.000 description 5
- 230000000007 visual effect Effects 0.000 description 5
- 230000008859 change Effects 0.000 description 4
- 230000007547 defect Effects 0.000 description 4
- 101710097688 Probable sphingosine-1-phosphate lyase Proteins 0.000 description 3
- 101710105985 Sphingosine-1-phosphate lyase Proteins 0.000 description 3
- 101710122496 Sphingosine-1-phosphate lyase 1 Proteins 0.000 description 3
- 238000001514 detection method Methods 0.000 description 3
- 238000000263 scanning probe lithography Methods 0.000 description 3
- 238000005452 bending Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000013507 mapping Methods 0.000 description 2
- 239000000523 sample Substances 0.000 description 2
- 230000002159 abnormal effect Effects 0.000 description 1
- 150000001875 compounds Chemical class 0.000 description 1
- 239000012141 concentrate Substances 0.000 description 1
- 238000007796 conventional method Methods 0.000 description 1
- 239000005357 flat glass Substances 0.000 description 1
- 230000014509 gene expression Effects 0.000 description 1
- 239000011521 glass Substances 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 239000011159 matrix material Substances 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000003825 pressing Methods 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 238000013139 quantization Methods 0.000 description 1
- 230000011218 segmentation Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
Landscapes
- Image Processing (AREA)
Abstract
The invention discloses a kind of file and picture distortion correction method based on structure laser rays, this method comprises at least：The inside and outside parameter of camera is demarcated, and calculates the plane equation where structure laser rays；Obtain file and picture；Extract laser rays；According to the plane equation and the laser rays of the extraction where the inside and outside parameter of the camera of demarcation, the structure laser rays, the 3D page directrixes of the file and picture are estimated；According to the 3D pages directrix, row interpolation is entered to developable surface；The developable surface of the interpolation is deployed；According to the developable surface of the expansion, pattern distortion is corrected.By the present invention, at least solves the technical problem for the geometric distortion for how correcting on-plane surface file and picture.
Description
Technical field
The present embodiments relate to Digital Image Processing and technical field of computer vision, swashs more particularly, to based on structure
The file and picture distortion correction method of light.
Background technology
Traditional flatbed scanner is when scanning thick bound documents (such as books and periodicals), because the spine region of document can not be complete
The open and flat glass plate surface for being close to scanner, the text image for ultimately resulting in obtain are abnormal in the significant image of spine region appearance
Change, non-homogeneous shade etc. scan defect.These scanning defects can have a strong impact on follow-up file and picture analysis work (such as document map
The Page Segmentation of picture, character recognition etc.).Therefore, as conventional file scanning instrument, traditional flatbed scanner is generally not
Fine scanning suitable for on-plane surface document works.
To realize the fine scanning of the documents such as books, it is curved to attempt to solve to have that people successively propose a series of technologies
The file and picture fine scanning problem of curved surface shape.These technologies can be roughly divided into two major classes：Method based on software and
Method based on hardware auxiliary.The former estimates to obtain the distortion letter of image first generally to the fault image of acquisition using algorithm
Number, the distortion distortion in correcting image is then converted by image；And the latter is then generally in image acquisition process, by certain
Hardware device aids in realizing the collection of the high quality of file and picture.
Method key based on software is the distortion function of correctly estimation file and picture, and conventional method includes being based on
The technology of image local deformation, the technology based on Shape-from-X, the technology based on page surface model, based on multiple views
Technology of image etc..Technology based on image local deformation, which is noticed, generally has substantial amounts of line of text in file and picture.It is and right
In distortionless image, these line of text are generally all horizontally oriented, are arranged in parallel.Therefore, this method is split and carried first
File and picture Chinese one's own profession is taken, and estimates line of text baseline corresponding to every line of text, then draws the line of text baseline of bending
Into a horizontal linear, and the word in line of text is rotated one by one and rearranged along the baseline of correction, so as to correct text
Capable bending.Image after sharp correction in this way, Text region precision can be significantly improved, but this method is corrected
Visual quality of images afterwards is not high, substantial amounts of distortion is still suffered from image, and can not correct the pattern distortion in non-textual region.
Recover the 3D shapes of document file page using certain visual cues in image based on shape-from-X technology,
So as to the distortion correction function of structural map picture.These visual cues include shade, profile, texture etc..In the prior art, with page
The shade curved surface in face is the most commonly used.The technology obtains the two by analyzing the restriction relation between page shade and page shape
The constraint equation met, solve the equation and obtain the 3D curved surfaces of the page.Finally piecture geometry fault is completed using the curved surface
Correction.The technology requires higher, the IMAQ being only applicable under light source controlled environment to page shade.
Technology based on page surface model then assumes that the curved shape of the page is certain certain types of curved surface, such as post
Face or developable surface, so as to constrain the estimation of page curved surface using the hypothesis.Among these, it is the most commonly used one that cylinder, which is assumed,
Kind is assumed.For estimation perspective transform under cylinder model parameter, the technology generally further limit camera shooting visual angle, away from
From, posture etc., and also need to be known a priori by the additional physical informations such as the size of document file page.These harsh conditions are made for practical application
Into many inconvenience.In addition, the parameter for accurate estimation cylinder model is, it is necessary to accurately split and extract horizontal in file and picture
Line of text, very high requirement is being proposed to the accurate extraction of still jejune line of text and fitting technique.Due to document layout class
The distortion that is introduced in the diversity and image acquisition procedures of type, noise etc. influence, and accurately split and extract horizontal line of text
Baseline is extremely difficult in practice, so as to seriously constrain the practical application of this method.
Developable surface assumes it is that another conventional page shape is assumed.Because document file page can always be expanded to plane
On, therefore, compared with cylinder is assumed, the hypothesis is more general.Notice that developable surface can be approached very well by planar chip one by one,
There is the newest proposition of researcher using burst thin-plate spline surface come approximately developable surfaces, the correction for piecture geometry fault.Should
Technology is corrected using plane perspective distortion correcting technology ripe at present, finally these is corrected piecewise to thin-plate spline surface
The correction of image is completed in result splicing afterwards.However, due to inevasible evaluated error, it is commonly present and splits in spliced image
The defects of stitching, be overlapping.These defects leverage the visual effect after image flame detection.Further, since perspective distortion with it is non-thread
Property geometric deformation distortion it is compound, the technology can not thoroughly geometric distortion present in correcting image, correction residual error be present
Greatly, the shortcomings of algorithm complexity.
Recently, researcher also proposed the file and picture distortion correction technology based on multi-view image.The technology passes through
Obtain the multiple image under same document different points of view, therefrom extract characteristics of image and carry out registration, so using multiple view geometry and
Epipolar-line constraint estimates the 3D shapes of the page.In order to accurately estimate the 3D shapes of the page, the technology is needed to being shot under different points of view
Image carry out accurate Characteristic points match.However, due to the influence of pattern distortion, in the serious local Feature Points Matching of distortion
Precision is very poor, and so as to cause the shape evaluated error in the region big, obtained deformation function can not effectively correct the region
Pattern distortion.
Different from the method based on software, the technology based on hardware auxiliary just introduces specific hardware in image acquisition phase
Equip to aid in the collection of book document image.Representational technology includes the technology based on 3D scanners, the book based on V-bed
Nationality scanning technique and the book scanning technology based on V-head.Technology based on 3D scanners directly obtains text using 3D scanners
The 3D depth informations of the shelves page, based on the 3D shapes of these signal reconstruct page curved surfaces, finally by a conformal projection and line
Reason mapping maps to the image of distortion in plane, carrys out the geometric warping of correcting image.This kind of method is generally to the geometry of image
Deformation Types are without restriction, can suitable for image polytype geometric distortion correction.However, due to being additionally required valency
The expensive 3D scanners assistant images correction of lattice, this kind of method in practice and are of little use.Also, due to geometry in depth data
Also there is the shortcomings of computationally intensive, numerical value is unstable in the missing of topological relation, the reconstruct of 3D page shapes, these are further limited
The application in practice of this kind of method.
Scanning technique based on V-bed devises a kind of base of V-arrangement easily to place the page of opening.It is placed on V
Page on shape base, is gently pressed by finger or covering pressure by means of transparent glass plate can prevent the page from significant shape occur
Become.When gathering page-images, the technology generally shoots the left and right page of books respectively using two high resolution cameras.The skill
Art can be prevented from producing in books image acquisition process and significantly distorted, but in order to obtain the page-images of high quality, scanning
During one book different-thickness part, the angle that constantly adjustment V-Bottom seat opens often is needed.In addition, the technology is prevented using pressing
Only book pages deform, and this technology belongs to contact type scanning method, are not suitable for the scanning of rapid wear ancient books.
Book scanning technology based on V-head is entered using the V-arrangement probe specially designed, and with reference to V-Bottom seat to books
Row scanning.This scanning technique is dedicated for the image scanning to the thick books of product.When scanning, one thick book is opened upward
It is positioned on V-Bottom seat, then V-arrangement probe is inserted the page and adjusts height compresses book pages, prevents the page from occurring
Flexural deformation.The technology is similar with the scanning technique based on V-bed, belongs to a kind of scanning technique of contact, is not suitable for easily
Damage the scanning of ancient books.In addition, the scanning technique based on V-head is specific to the scanning of book document and designed, it is impossible to uses
In the scanning of other documents, therefore, its application is very limited in practice.
In view of this, it is special to propose the present invention.
The content of the invention
The main purpose of the embodiment of the present invention is to provide a kind of file and picture distortion correction side based on structure laser rays
Method, it solves the technical problem for the geometric distortion for how correcting on-plane surface file and picture at least in part.
To achieve these goals, according to an aspect of the invention, there is provided following technical scheme：
A kind of file and picture distortion correction method based on structure laser rays, this method can at least include：
The inside and outside parameter of camera is demarcated, and calculates the plane equation where structure laser rays；
Obtain file and picture；
Extract laser rays；
According to swashing for the plane equation where the inside and outside parameter of the camera of demarcation, the structure laser rays and the extraction
Light, estimate the 3D page directrixes of the file and picture；
According to the 3D pages directrix, row interpolation is entered to developable surface；
The developable surface of the interpolation is deployed；
According to the developable surface of the expansion, pattern distortion is corrected.
Further, it is described that the inside and outside parameter of camera is demarcated, specifically include：
According to chessboard grid pattern calibrating block, calibrating block image is obtained, the intrinsic parameter of camera is calculated according to camera calibration algorithm；
World coordinate system is established on the calibrating block image, it is outer according to corresponding to the camera calibration algorithm calculates camera
Parameter.
Further, the plane equation calculated where structure laser rays, is specifically included：
According to the Camera extrinsic number and intrinsic parameter, calculate 3D of the point on the laser rays under world coordinate system and sit
Mark, plane fitting is carried out to the 3D coordinate points on same laser rays using least-squares algorithm.
Further, the acquisition file and picture specifically includes：
Document is irradiated using two a wordline laser devices, opens and closes laser power supply, while utilize camera shooting text
Shelves page-images.
Further, the extraction laser rays, is specifically included：
The red channel image of the file and picture is extracted, and calculates red channel image difference, obtains error image；Root
According to image threshold, it is higher than the pixel of specified threshold to image difference by column on error image, extraction maximum point is as institute
State the estimation of laser rays.
Further, the 3D page directrixes of the estimation file and picture, are specifically included：
According to the plane equation where the inside and outside parameter of the camera of the demarcation and the structure laser rays, document map is calculated
Every corresponding 3D coordinate on laser rays as described in, and the 3D points on the same laser rays are carried out using B-spline curves
Fitting.
Further, it is described that row interpolation is entered to developable surface, specifically include：
According to dynamic programming algorithm, solving-optimizing problem, the optimal respective function between two 3D page directrixes is obtained, its
In, the optimization problem is：
S.t. φ (0)=0, φ (l0)=l1,
Wherein,
C0- the first laser rays curve；
C1- Article 2 laser rays curve；
l0—C0The total length of curve；
l1—C1The total length of curve；
φ(s)—C0And C1Between respective function；
The arc length parameters of s-curve；
λ-specify parameter, for controlling the smooth degree of respective function to be asked；
A-specify parameter, 1 need to be more than；
B-specify parameter, 0 need to be less than；
The proportion function of σ (s)-curve corresponding points tangent line angle, is defined as follows：
Wherein,
α0(s)—C0The angle of tangent line at s points and corresponding straight edge line；
α1(s)—C1In the angle of tangent line and the corresponding straight edge line of corresponding point position.
Further, the developable surface to the interpolation deploys, and specifically includes：
Calculate two 3D directrixes on the developable surface and be expanded to the curvilinear equation corresponding to plane, utilize optimal corresponding letter
Number, the straight edge line of developable surface is constructed in the plane, curved surface is deployed.
Further, by solving ODE, two 3D directrixes on the developable surface is calculated and are expanded to plane institute
Corresponding curvilinear equation, wherein, the ODE is：
Wherein,
kgThe geodesic curvature function of-developable surface upper curve；
x0- treat the x coordinate of splay line starting point in the plane；
y0- treat the y-coordinate of splay line starting point in the plane；
μ0- treat the x coordinate of the expansion direction that splay line starting point is specified in the plane；
ν0- treat the y-coordinate of the expansion direction that splay line starting point is specified in the plane；
The ODE is solved by below equation：
Wherein,
Further, the developable surface according to the expansion, corrects to pattern distortion, specifically includes：
Regular grid is placed on the developable surface of the expansion, Surface Parameters corresponding to the mesh point is calculated and sits
Mark, and the inside and outside parameter of the camera is utilized, calculate the pixel coordinate that the mesh point corresponds to original image；
Using the corresponding relation of the pixel coordinate and grid point coordinates, by function interpolation structural map as deformation function,
Pattern distortion is corrected.
Compared with prior art, above-mentioned technical proposal at least has the advantages that：
The embodiment of the present invention obtained by using two-strip structure laser rays document to be bent (such as：Books) page 3D
Shape, estimate the 3D page directrixes of file and picture；According to 3D page directrixes, row interpolation is entered to developable surface, and by explicitly asking
ODE is solved to calculate the expansion of developable surface, and then constructs the rectification function and distortion correction is carried out to image.It is of the invention real
The distortion correction that example can be applied to the distortion book pages image of various shapes is applied, such as：Cylinder, plane, the conical surface and can splay
Face, have the advantages that computation complexity is low, precision is high, cost is cheap, non-contact scanning, full-automatic no interactions, applicability are wide.
The embodiment of the present invention can be applied to the high quality imaging of the distorted documents such as book pages, in document information digitlization, digital book
The fields such as shop structure, the protection of precious historical document ancient books and records have broad application prospects.
Certainly, any product for implementing the present invention is not necessarily required to realize all the above advantage simultaneously.
Other features and advantages of the present invention will be illustrated in the following description, also, partly becomes from specification
Obtain it is clear that or being understood by implementing the present invention.Objectives and other advantages of the present invention can be by the explanation write
Specifically noted method is realized and obtained in book, claims and accompanying drawing.
It should be noted that Summary is not intended to identify the essential features of claimed theme,
Also it is not the protection domain for determining claimed theme.Theme claimed is not limited to solve in background technology
In any or all shortcoming for referring to.
Brief description of the drawings
A part of the accompanying drawing as the application, for providing further understanding of the invention, of the invention is schematic
Embodiment and its illustrate to be used to explain the present invention, but do not form inappropriate limitation of the present invention.Obviously, drawings in the following description
Only some embodiments, to those skilled in the art, on the premise of not paying creative work, can be with
Other accompanying drawings are obtained according to these accompanying drawings.In the accompanying drawings：
Fig. 1 is the stream of the file and picture distortion correction method based on structure laser rays according to an exemplary embodiment
Cheng Tu；
Fig. 2 is that the structure of the book document scanning system based on structure laser rays according to an exemplary embodiment is shown
It is intended to；
Fig. 3 a are the file and picture for having laser rays according to an exemplary embodiment；
Fig. 3 b are the file and picture without laser rays according to an exemplary embodiment；
Fig. 4 a are the error image corresponding to Fig. 3 a and Fig. 3 b red channels according to an exemplary embodiment；
Fig. 4 b are the estimation of 3D directrix curves corresponding to two laser rays according to an exemplary embodiment；
Fig. 5 a are the schematic diagram of the optimal respective function according to an exemplary embodiment；
Fig. 5 b are the schematic diagram for the developable surface that the interpolation according to an exemplary embodiment obtains；
Fig. 6 is the schematic diagram of the form deployed to developable surface according to an exemplary embodiment；
Fig. 7 is the corresponding relation of regular grid and pixel in the expansion plane according to an exemplary embodiment
Schematic diagram；
Fig. 8 a are the schematic diagram of distorted document image；
Fig. 8 b are the schematic diagram of the file and picture after correction.
These accompanying drawings and word description are not intended as the concept limiting the invention in any way, but by reference to
Specific embodiment is that those skilled in the art illustrate idea of the invention.
Embodiment
Below in conjunction with the accompanying drawings and specific embodiment solved to the embodiment of the present invention technical problem, used technical side
Case and the technique effect of realization carry out clear, complete description.Obviously, described embodiment is only one of the application
Divide embodiment, be not whole embodiments.Based on the embodiment in the application, those of ordinary skill in the art are not paying creation
Property work on the premise of, the embodiment of all other equivalent or obvious modification obtained is all fallen within protection scope of the present invention.
The embodiment of the present invention can embody according to the multitude of different ways being defined and covered by claim.
It should be noted that in the following description, understand for convenience, give many details.It is but very bright
Aobvious, realization of the invention can be without these details.
It should be noted that in the case where not limiting clearly or not conflicting, each embodiment in the present invention and its
In technical characteristic can be mutually combined and form technical scheme.
Fig. 2 be according to an exemplary embodiment based on structure laser rays document (such as：Books) scanning system
Structural representation.The system is made up of two a wordline laser devices, a high resolution camera, support and base.
The present invention basic conception be:The page surface of book document to be captured is irradiated using two-strip structure laser rays, is led to
The laser rays crossed in the file and picture of extraction camera shooting, to estimate corresponding two 3D directrixes on page curved surface, and then construct
Interpolation is in the developable surface of this two directrixes.Finally, deployed by developable surface, geometric distortion present in image is rectified
Just, the undistorted file and picture of a panel height quality is obtained.
It should be noted that the embodiment of the present invention assumes that the distortion curved surface of document file page is smooth developable surface.The vacation
If merely to the present invention is better described, limiting the scope of the invention is not intended as.
In order to solve the technical problem for the geometric distortion for how correcting on-plane surface file and picture, the embodiment of the present invention provides one
File and picture distortion correction method of the kind based on structure laser rays.Fig. 1 is based on structure according to an exemplary embodiment
The flow chart of the file and picture distortion correction method of laser rays.As shown in figure 1, this method can at least include step S100 to step
Rapid S112.
S100：The inside and outside parameter of camera is demarcated, and calculates the plane equation where structure laser rays.
In this step, according to chessboard grid pattern calibrating block, calibrating block image is obtained, phase is calculated according to camera calibration algorithm
The intrinsic parameter of machine；World coordinate system is established on calibrating block image, the outer parameter according to corresponding to camera calibration algorithm calculates camera.
In actual applications, camera can from different perspectives, diverse location to chessboard grid pattern calibrating block be imaged, obtain one
Serial calibrating block image.Then, the intrinsic parameter of camera is calculated using camera calibration algorithm.Fixed camera, and by chessboard grid pattern
Calibrating block is positioned over below camera.It is imaged using camera, chooses coordinate origin and x, y-coordinate axle side manually on image
To, establish world coordinate system, and using camera calibration algorithm calculate outer parameter corresponding to camera (such as：Spin matrix and translation
Vector).Although term " x-axis " and " y-axis " are used to illustrate some directions in specific pattern herein, it should be appreciated that these arts
Language does not refer to absolute direction.In other words, " x-axis " can be any corresponding axle, and " y-axis " can be the specific axis different from x-axis.It is logical
Often, x-axis is perpendicular to y-axis.
In an optional embodiment, the plane equation where calculating structure laser rays specifically includes：
According to Camera extrinsic number and intrinsic parameter, 3D coordinate of the point on laser rays under world coordinate system is calculated, using most
Young waiter in a wineshop or an inn's multiplication algorithm carries out plane fitting to the 3D coordinate points on same laser rays.
In this embodiment, chessboard grid pattern calibrating block is placed away from the different distance of camera one after another, unlatching/closing swashs
Light device is simultaneously imaged with camera to it, and extracts laser rays therein.Then utilize corresponding to the calculating of camera calibration algorithm outside camera
Parameter, and then the 3D coordinates using the point on Camera extrinsic number and intrinsic parameter calculating laser rays under world coordinate system.Finally, it is sharp
Plane fitting is carried out to the 3D coordinate points on same laser rays with least-squares algorithm, obtained flat where counter structure laser rays
Face equation.
S102：Obtain file and picture.
In this step, the page surface of document (such as books) to be scanned is irradiated using two a wordline lasers.Pass through
Control device opens and closes laser power supply, while shoots the page-images of two width documents respectively using high resolution camera,
A wherein width has laser rays, and another width does not have laser rays.
S104：Extract laser rays.
As an example, two width that document scanning system shooting is given in Fig. 3 a and Fig. 3 b have laser rays and without laser rays
File and picture.The red channel image of two images is extracted respectively, and calculates red channel image difference, obtains error image.
Due to there are laser position on image and without laser position, difference is notable, therefore, on error image, it is easy to which extraction swashs
Light.Fig. 4 a give the error image corresponding to Fig. 3 a and Fig. 3 b red channels.When extracting laser rays on error image, need
A detection threshold value, and the order along image from left to right are manually set, by column to pixel value in error image higher than setting
The pixel of threshold value, maximum point corresponding to two parts detection above and below point.These maximum points can be used as laser rays point on the row
Estimation.
S106：According to the plane equation and the laser of extraction where the inside and outside parameter of the camera of demarcation, structure laser rays
Line, estimate the 3D page directrixes of file and picture.
In this step, using the plane equation where the inside and outside parameter of the camera demarcated and structure laser rays, figure is calculated
Every corresponding 3D coordinate on laser rays as in, and the 3D points on same laser rays are fitted using B- SPLs, obtain
To the estimation of document file page curved surface directrix.Wherein, in numerical analysis, B- SPLs are a kind of special expressions of SPL
Form.It is the linear combination of B- batten base curves.B- SPLs are a kind of vague generalization of Bezier curve.
As an example, Fig. 4 b give the estimation of 3D directrix curves corresponding to two laser rays.Laser is extracted from image
After line, to each pixel on laser rays, one can be constructed by camera photocentre and the straight line of the pixel.Calculate the straight line
With the focus of structure laser rays plane, 3D point coordinates corresponding to the pixel can be obtained, and then obtain corresponding to every laser rays
The 3D directrixes of curved surface.After obtaining 3D directrixes, further it is fitted using B-spline curves, removes the standard introduced by image quantization error
Line evaluated error.
S108：According to 3D page directrixes, row interpolation is entered to developable surface.
In this step, obtain on developable surface after two 3D directrixes, can construct interpolation in this two directrixes can splay
Face, as shown in Figure 5 b.For this reason, it may be necessary to calculate the optimal respective function between two directrixes.Solved using dynamic programming algorithm excellent
Change problem, the optimal respective function between two 3D page directrixes can be obtained, as shown in Figure 5 a.
Wherein, the optimization problem can be：
S.t. φ (0)=0, φ (l0)=l1,
Wherein,
C0- the first laser rays curve；
C1- Article 2 laser rays curve；
l0—C0The total length of curve；
l1—C1The total length of curve；
φ(s)—C0And C1Between respective function；
The arc length parameters of s-curve；
λ-specify parameter, for controlling the smooth degree of respective function to be asked；
A-specify parameter, 1 need to be more than；
B-specify parameter, 0 need to be less than；
The proportion function of σ (s)-curve corresponding points tangent line angle, is defined as follows：
Wherein,
α0(s)—C0The angle of tangent line at s points and corresponding straight edge line；
α1(s)—C1In the angle of tangent line and the corresponding straight edge line of corresponding point position.
S110：The developable surface of interpolation is deployed.
The step can specifically include：Calculate two 3D directrixes on the developable surface and be expanded to the curve corresponding to plane
Equation, using optimal respective function, the straight edge line of developable surface is constructed in the plane, curved surface is deployed.
In actual applications, after obtaining developable surface, by the way that developable surface is expanded in plane, pattern distortion can be constructed
The rectification function needed for correction.Developable surface deploys, and can be realized by corresponding two directrixes on Developing Surface.Pass through solution
ODE, a directrix on developable surface can be expanded in plane.The ODE is：
Wherein,
kgThe geodesic curvature function of-developable surface upper curve；
x0- treat the x coordinate of splay line starting point in the plane；
y0- treat the y-coordinate of splay line starting point in the plane；
μ0- treat the x coordinate of the expansion direction that splay line starting point is specified in the plane；
ν0- treat the y-coordinate of the expansion direction that splay line starting point is specified in the plane；
After the starting point coordinate of given curve expansion and initial deployment direction, the equation of developed curve can directly be counted by following formula
Calculate：
Wherein,
After 3D directrixes on two curved surfaces are expanded into plane, because developable surface expansion has conformality and protects away from property,
Therefore the respective function between two directrixes is utilized, can easily construct the straight edge line of developable surface in the plane, so as to
Obtain the expanded form of curved surface in the plane.Fig. 6 gives the form that ODE is deployed to developable surface that solves.
S112：According to the developable surface of expansion, pattern distortion is corrected.
The step can specifically include：Regular grid is placed on the developable surface of expansion, is calculated corresponding to mesh point
Surface Parameters coordinate, and the inside and outside parameter of camera is utilized, calculate the pixel coordinate that mesh point corresponds to original image；Sat using pixel
The corresponding relation of mark and grid point coordinates, by function interpolation, structural map is corrected as deformation function to pattern distortion.
In actual applications, in order to realize the correction to be distorted in image, it is also necessary to be configured to rectifying for pattern distortion correction
Positive function.Therefore, a regular grid is constructed first in Doubly curved surfaces plane.To each mesh point on regular grid,
Solve equation below group and calculate coordinate of the point corresponding to the point on developable surface：
Wherein,
Regular grid point coordinates in (u, v)-expansion plane；
- curve C0It is expanded to corresponding curve in plane
- curve C1It is expanded to corresponding curve in plane
Point coordinates corresponding to (s, τ)-regular grid (u, v) on curved surface.
Obtain on curved surface after corresponding point coordinates, also just obtained the 3D coordinates put corresponding to the mesh point on curved surface, entered
And the inside and outside parameter of camera is utilized, the pixel coordinate on image corresponding to the point can be calculated.So as to build regular grid in plane
One-to-one relationship in point coordinates and original image between pixel point coordinates.Finally, using Spline-Fitting, can construct as follows
Two binary functions：
Mapping of the two binary function implementation rule mesh points to image pixel point coordinates.Therefore, using the two two
Meta-function, for any point in expansion plane, it is convenient to calculate the pixel on image corresponding to the point.Fig. 7 gives
Calculate the corresponding relation of regular grid and pixel in expansion plane.
In order to verify the embodiment of the present invention, the books that are distorted using the document scanning system based on structure laser rays to a width
The page is imaged.By opening and closing laser, two images are shot respectively, and a width has laser rays, and another width is without laser
Line.Using this two images, by calculating the laser rays in image difference extraction image, and then curved surface corresponding to laser rays is estimated
3D directrixes, and interpolation is constructed in the developable surface of 3D directrixes, finally by developable surface deployment configuration pattern distortion the rectification function.
Fig. 8 a to Fig. 8 b are the file and pictures after the distorted document image and correction obtained.There it can be seen that the present invention can be effectively
The geometric distortion in file and picture is corrected, the line of text after correction in image is straight, character script size is homogeneous, the figure after correction
Picture and scan image are visually closely similar.
Each step is described in the way of above-mentioned precedence in the present embodiment, those skilled in the art can
To understand, in order to realize the effect of the present embodiment, performed between different steps not necessarily in such order, it can be simultaneously
Perform or execution order overturns, these simply change all within protection scope of the present invention.
The technical scheme provided above the embodiment of the present invention is described in detail.Although apply herein specific
Individual example to the present invention principle and embodiment be set forth, still, the explanation of above-described embodiment is only applicable to help and managed
Solve the principle of the embodiment of the present invention；Meanwhile to those skilled in the art, according to the embodiment of the present invention, it is being embodied
It can be made a change within mode and application.
It should be noted that：Mark in accompanying drawing is intended merely to be illustrated more clearly that the present invention, is not intended as protecting the present invention
Protect the improper restriction of scope.
Term " comprising ", "comprising" or any other like term are intended to including for nonexcludability, so that
Process, method, article or equipment/device including a series of elements not only include those key elements, but also including not bright
The other key elements really listed, either also include these processes, method, article or the intrinsic key element of equipment/device.Do not having
In the case of more limitations, the key element that is limited by sentence "including a ...", it is not excluded that the process including the key element,
Other key element in method, article or equipment/device also be present, i.e. the meaning of " including one " is also contemplated by " including another "
The meaning.
The present invention each step can be realized with general computing device, for example, they can concentrate on it is single
On computing device, such as：Personal computer, server computer, handheld device or portable set, laptop device or more
Processor device, it can also be distributed on the network that multiple computing devices are formed, they can be with different from order herein
Shown or described step is performed, they are either fabricated to each integrated circuit modules respectively or will be more in them
Individual module or step are fabricated to single integrated circuit module to realize.Therefore, the invention is not restricted to any specific hardware and soft
Part or its combination.
Method provided by the invention can be realized using PLD, and it is soft can also to be embodied as computer program
Part or program module (it include performing particular task or the routine for realizing particular abstract data type, program, object, component or
Data structure etc.), such as can be according to an embodiment of the invention a kind of computer program product, run the computer program
Product makes computer perform for demonstrated method.The computer program product includes computer-readable recording medium, should
Computer program logic or code section are included on medium, for realizing methods described.The computer-readable recording medium can
To be the removable medium (example that is mounted built-in medium in a computer or can be disassembled from basic computer
Such as：Using the storage device of hot plug technology).The built-in medium includes but is not limited to rewritable nonvolatile memory,
Such as：RAM, ROM, flash memory and hard disk.The removable medium includes but is not limited to：Optical storage media (such as：CD-
ROM and DVD), magnetic-optical storage medium (such as：MO), magnetic storage medium (such as：Tape or mobile hard disk), have it is built-in can
Rewrite nonvolatile memory media (such as：Storage card) and with built-in ROM media (such as：ROM boxes).
The present invention is not limited to above-mentioned embodiment, and in the case of without departing substantially from substantive content of the present invention, this area is common
Any deformation, improvement or the replacement that technical staff is contemplated that each fall within the scope of the present invention.
Although have been shown above, be described and pointed out basic novel feature of the invention suitable for various embodiments
Detailed description, it will be understood that not departing from the invention is intended in the case of, those skilled in the art can be to system
Form and details carry out various omissions, substitutions and changes.
Claims (9)
- A kind of 1. file and picture distortion correction method based on structure laser rays, it is characterised in that this method comprises at least：The inside and outside parameter of camera is demarcated, and calculates the plane equation where structure laser rays；Obtain file and picture；Extract laser rays；According to the plane equation and the laser rays of the extraction where the inside and outside parameter of the camera of demarcation, the structure laser rays, Estimate the 3D page directrixes of the file and picture；According to the 3D pages directrix, row interpolation is entered to developable surface；The developable surface of the interpolation is deployed；According to the developable surface of the expansion, pattern distortion is corrected；Wherein,It is described that row interpolation is entered to developable surface, specifically include：According to dynamic programming algorithm, solving-optimizing problem, the optimal respective function between two 3D page directrixes is obtained；The optimization problem is：<mrow> <munder> <mrow> <mi>m</mi> <mi>i</mi> <mi>n</mi> </mrow> <mrow> <mi>&phi;</mi> <mrow> <mo>(</mo> <mo>&CenterDot;</mo> <mo>)</mo> </mrow> </mrow> </munder> <msubsup> <mo>&Integral;</mo> <mn>0</mn> <msub> <mi>l</mi> <mn>0</mn> </msub> </msubsup> <mrow> <mo>(</mo> <mo>|</mo> <mo><</mo> <mfrac> <mrow> <msub> <mi>dC</mi> <mn>0</mn> </msub> </mrow> <mrow> <mi>d</mi> <mi>s</mi> </mrow> </mfrac> <mo>&times;</mo> <mfrac> <mrow> <msub> <mi>dC</mi> <mn>1</mn> </msub> </mrow> <mrow> <mi>d</mi> <mi>&phi;</mi> </mrow> </mfrac> <mo>,</mo> <msub> <mi>C</mi> <mn>1</mn> </msub> <mo>(</mo> <mrow> <mi>&phi;</mi> <mrow> <mo>(</mo> <mi>s</mi> <mo>)</mo> </mrow> </mrow> <mo>)</mo> <mo>-</mo> <msub> <mi>C</mi> <mn>0</mn> </msub> <mo>(</mo> <mi>s</mi> <mo>)</mo> <mo>></mo> <mo>|</mo> <mo>+</mo> <msup> <mi>&lambda;&phi;</mi> <mrow> <mo>&prime;</mo> <mn>2</mn> </mrow> </msup> <mo>(</mo> <mi>s</mi> <mo>)</mo> <mo>)</mo> </mrow> <mi>d</mi> <mi>s</mi> </mrow>S.t. φ (0)=0 φ (l0)=l1,<mrow> <mfrac> <mrow> <mo>-</mo> <mi>b</mi> <mi>&sigma;</mi> <mrow> <mo>(</mo> <mi>s</mi> <mo>)</mo> </mrow> </mrow> <mrow> <mn>1</mn> <mo>-</mo> <mi>b</mi> </mrow> </mfrac> <mo>&le;</mo> <msup> <mi>&phi;</mi> <mo>&prime;</mo> </msup> <mrow> <mo>(</mo> <mi>s</mi> <mo>)</mo> </mrow> <mo>&le;</mo> <mfrac> <mrow> <mi>a</mi> <mi>&sigma;</mi> <mrow> <mo>(</mo> <mi>s</mi> <mo>)</mo> </mrow> </mrow> <mrow> <mi>a</mi> <mo>-</mo> <mn>1</mn> </mrow> </mfrac> <mo>,</mo> </mrow>C0- the first laser rays curve；C1- Article 2 laser rays curve；l0—C0The total length of curve；l1—C1The total length of curve；φ(s)—C0And C1Between respective function；The arc length parameters of s-curve；λ-specify parameter, for controlling the smooth degree of respective function to be asked；A-specify parameter, 1 need to be more than；B-specify parameter, 0 need to be less than；The proportion function of σ (s)-curve corresponding points tangent line angle, is defined as follows：<mrow> <mi>&sigma;</mi> <mrow> <mo>(</mo> <mi>s</mi> <mo>)</mo> </mrow> <mo>=</mo> <mfrac> <mrow> <msub> <mi>sin&alpha;</mi> <mn>0</mn> </msub> <mrow> <mo>(</mo> <mi>s</mi> <mo>)</mo> </mrow> </mrow> <mrow> <msub> <mi>sin&alpha;</mi> <mn>1</mn> </msub> <mrow> <mo>(</mo> <mi>s</mi> <mo>)</mo> </mrow> </mrow> </mfrac> <mo>,</mo> </mrow>α0(s)—C0The angle of tangent line at s points and corresponding straight edge line；α1(s)—C1In the angle of tangent line and the corresponding straight edge line of corresponding point position.
- 2. the file and picture distortion correction method according to claim 1 based on structure laser rays, it is characterised in that described The inside and outside parameter of camera is demarcated, specifically included：According to chessboard grid pattern calibrating block, calibrating block image is obtained, the intrinsic parameter of camera is calculated according to camera calibration algorithm；World coordinate system is established on the calibrating block image, it is outer according to corresponding to the camera calibration algorithm calculates camera to join Number.
- 3. the file and picture distortion correction method according to claim 1 based on structure laser rays, it is characterised in that described The plane equation where structure laser rays is calculated, is specifically included：According to the Camera extrinsic number and intrinsic parameter, 3D coordinate of the point on the laser rays under world coordinate system is calculated, profit Plane fitting is carried out to the 3D coordinate points on same laser rays with least-squares algorithm.
- 4. the file and picture distortion correction method according to claim 1 based on structure laser rays, it is characterised in that described File and picture is obtained, is specifically included：Document is irradiated using two a wordline laser devices, opens and closes laser power supply, while utilize camera shooting documentation page Face image.
- 5. the file and picture distortion correction method according to claim 1 based on structure laser rays, it is characterised in that described Laser rays is extracted, is specifically included：The red channel image of the file and picture is extracted, and calculates red channel image difference, obtains error image；According to image threshold, it is higher than the pixel of specified threshold to image difference by column on error image, extracts maximum point Estimation as the laser rays.
- 6. the file and picture distortion correction method according to claim 1 based on structure laser rays, it is characterised in that described Estimate the 3D page directrixes of the file and picture, specifically include：According to the plane equation where the inside and outside parameter of the camera of the demarcation and the structure laser rays, calculate in file and picture Every corresponding 3D coordinate on the laser rays, and the 3D points on the same laser rays are fitted using B-spline curves.
- 7. the file and picture distortion correction method according to claim 1 based on structure laser rays, it is characterised in that described The developable surface of the interpolation is deployed, specifically included：Calculate two 3D directrixes on the developable surface and be expanded to the curvilinear equation corresponding to plane, using optimal respective function, The straight edge line of developable surface is constructed in the plane, and curved surface is deployed.
- 8. the file and picture distortion correction method according to claim 7 based on structure laser rays, it is characterised in thatBy solving ODE, calculate two 3D directrixes on the developable surface and be expanded to the curve side corresponding to plane Journey, wherein, the ODE is：<mfenced open = "{" close = ""> <mtable> <mtr> <mtd> <msup> <mi>x</mi> <mo>&prime;</mo> </msup> <msup> <mi>y</mi> <mrow> <mo>&prime;</mo> <mo>&prime;</mo> </mrow> </msup> <mo>-</mo> <msup> <mi>x</mi> <mrow> <mo>&prime;</mo> <mo>&prime;</mo> </mrow> </msup> <msup> <mi>y</mi> <mo>&prime;</mo> </msup> <mo>=</mo> <msub> <mi>k</mi> <mi>g</mi> </msub> </mtd> </mtr> <mtr> <mtd> <mrow> <msup> <mi>x</mi> <mrow> <mo>&prime;</mo> <mn>2</mn> </mrow> </msup> <mo>+</mo> <msup> <mi>y</mi> <mrow> <mo>&prime;</mo> <mn>2</mn> </mrow> </msup> <mo>=</mo> <mn>1</mn> </mrow> </mtd> </mtr> <mtr> <mtd> <mi>x</mi> <mo>(</mo> <mn>0</mn> <mo>)</mo> <mo>=</mo> <msub> <mi>x</mi> <mn>0</mn> </msub> <mo>,</mo> <mi>y</mi> <mo>(</mo> <mn>0</mn> <mo>)</mo> <mo>=</mo> <msub> <mi>y</mi> <mn>0</mn> </msub> <mo>,</mo> <msup> <mi>x</mi> <mo>&prime;</mo> </msup> <mo>(</mo> <mn>0</mn> <mo>)</mo> <mo>=</mo> <msub> <mi>&mu;</mi> <mn>0</mn> </msub> <mo>,</mo> <msup> <mi>y</mi> <mo>&prime;</mo> </msup> <mo>(</mo> <mn>0</mn> <mo>)</mo> <mo>=</mo> <msub> <mi>v</mi> <mn>0</mn> </msub> </mtd> </mtr> </mtable> </mfenced>Wherein,kgThe geodesic curvature function of-developable surface upper curve；x0- treat the x coordinate of splay line starting point in the plane；y0- treat the y-coordinate of splay line starting point in the plane；μ0- treat the x coordinate of the expansion direction that splay line starting point is specified in the plane；ν0- treat the y-coordinate of the expansion direction that splay line starting point is specified in the plane；The ODE is solved by below equation：Wherein,
- 9. the file and picture distortion correction method according to claim 1 based on structure laser rays, it is characterised in that described According to the developable surface of the expansion, pattern distortion is corrected, specifically included：Regular grid is placed on the developable surface of the expansion, calculates Surface Parameters coordinate corresponding to the mesh point, and Using the inside and outside parameter of the camera, the pixel coordinate that the mesh point corresponds to original image is calculated；Using the pixel coordinate and the corresponding relation of grid point coordinates, by function interpolation structural map as deformation function, to figure Image distortion is corrected.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510599577.3A CN105118040B (en) | 2015-09-18 | 2015-09-18 | File and picture distortion correction method based on structure laser rays |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510599577.3A CN105118040B (en) | 2015-09-18 | 2015-09-18 | File and picture distortion correction method based on structure laser rays |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN105118040A CN105118040A (en) | 2015-12-02 |
| CN105118040B true CN105118040B (en) | 2017-12-01 |
Family
ID=54666015
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510599577.3A Active CN105118040B (en) | 2015-09-18 | 2015-09-18 | File and picture distortion correction method based on structure laser rays |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105118040B (en) |
Families Citing this family (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106991649A (en) * | 2016-01-20 | 2017-07-28 | 富士通株式会社 | The method and apparatus that the file and picture captured to camera device is corrected |
| CN105979117B (en) * | 2016-04-28 | 2018-11-27 | 大连成者科技有限公司 | Bending page image flattening method based on laser rays |
| CN108073924B (en) * | 2016-11-17 | 2021-06-22 | 富士通株式会社 | Image processing method and device |
| CN109211264B (en) * | 2017-07-06 | 2021-04-30 | 杭州海康机器人技术有限公司 | Calibration method and device of measurement system, electronic equipment and readable storage medium |
| CN110274597B (en) * | 2019-06-13 | 2022-09-16 | 大连理工大学 | Method for solving problem of 'particle binding frame' when indoor robot is started at any point |
| CN110493520B (en) * | 2019-08-20 | 2022-09-13 | 北京千寻未来科技有限公司 | Imaging method, imaging device and electronic equipment |
| CN110533769B (en) * | 2019-08-20 | 2023-06-02 | 福建捷宇电脑科技有限公司 | Flattening method and terminal for open book image |
| CN111260586B (en) | 2020-01-20 | 2023-07-04 | 北京百度网讯科技有限公司 | Correction method and device for distorted document image |
| CN111272102A (en) * | 2020-05-06 | 2020-06-12 | 中国空气动力研究与发展中心低速空气动力研究所 | Line laser scanning three-dimensional measurement calibration method |
| CN112541506B (en) * | 2020-12-09 | 2023-11-28 | 上海肇观电子科技有限公司 | Text image correction method, device, equipment and medium |
| CN112671993A (en) * | 2020-12-31 | 2021-04-16 | 北京千寻未来科技有限公司 | Information acquisition device, flattenable flexible scanner and information acquisition method |
| CN114422653B (en) * | 2022-01-19 | 2024-01-30 | 大连成者科技有限公司 | Book flattening line laser rapid calibration method and laser scanner |
| CN115082939B (en) * | 2022-05-12 | 2023-11-17 | 吉林省吉林祥云信息技术有限公司 | System and method for correcting distortion table in image based on arc differentiation |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102208025A (en) * | 2011-05-27 | 2011-10-05 | 中国科学院自动化研究所 | Method for correcting geometric distortion of text image |
| CN104537631A (en) * | 2015-02-05 | 2015-04-22 | 哈尔滨工业大学 | PLIF image correction method based on digital image registration technique |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR100644687B1 (en) * | 2005-03-18 | 2006-11-10 | 삼성전자주식회사 | Method and apparatus for correcting image of MFP |
-
2015
- 2015-09-18 CN CN201510599577.3A patent/CN105118040B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102208025A (en) * | 2011-05-27 | 2011-10-05 | 中国科学院自动化研究所 | Method for correcting geometric distortion of text image |
| CN104537631A (en) * | 2015-02-05 | 2015-04-22 | 哈尔滨工业大学 | PLIF image correction method based on digital image registration technique |
Non-Patent Citations (1)
| Title |
|---|
| Active Flattening of Curved Document Images via Two Structured Beams;Gaofeng Meng et al.;《IEEE Conference on Computer Vision & Pattern Recognition》;20140628;第II部分 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN105118040A (en) | 2015-12-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105118040B (en) | File and picture distortion correction method based on structure laser rays | |
| US10783613B2 (en) | Content-based detection and three dimensional geometric reconstruction of objects in image and video data | |
| US20210027431A1 (en) | Content-based object detection, 3d reconstruction, and data extraction from digital images | |
| You et al. | Multiview rectification of folded documents | |
| Li et al. | Document rectification and illumination correction using a patch-based CNN | |
| US9779296B1 (en) | Content-based detection and three dimensional geometric reconstruction of objects in image and video data | |
| Tian et al. | Rectification and 3D reconstruction of curved document images | |
| US10289924B2 (en) | System and method for scanned document correction | |
| Liang et al. | Geometric rectification of camera-captured document images | |
| CN102790841B (en) | Method of detecting and correcting digital images of books in the book spine area | |
| Ulges et al. | Document image dewarping using robust estimation of curled text lines | |
| CN109271890B (en) | Method and apparatus for automatic fingerprint image extraction | |
| US9754187B2 (en) | Data capture from images of documents with fixed structure | |
| Yerushalmy et al. | Digital image forgery detection based on lens and sensor aberration | |
| US10198661B2 (en) | System for determining alignment of a user-marked document and method thereof | |
| Meng et al. | Active flattening of curved document images via two structured beams | |
| CN110827371B (en) | Certificate generation method and device, electronic equipment and storage medium | |
| CN109059868A (en) | A kind of binocular distance measuring method based on Adaptive matching window | |
| CN110321750A (en) | Two-dimensional code identification method and system in a kind of picture | |
| CN110120013A (en) | A kind of cloud method and device | |
| Safonov et al. | Document Image Processing for Scanning and Printing | |
| WO2017173368A1 (en) | Content-based detection and three dimensional geometric reconstruction of objects in image and video data | |
| Meng et al. | Detecting photographic cropping based on vanishing points | |
| CN107622514A (en) | Video camera convex lens model Auto-calibration method | |
| CN114422653A (en) | Book flattening line laser rapid calibration method and laser scanner |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |