CN103873758B - The method, apparatus and equipment that panorama sketch generates in real time - Google Patents
The method, apparatus and equipment that panorama sketch generates in real time Download PDFInfo
- Publication number
- CN103873758B CN103873758B CN201210548920.8A CN201210548920A CN103873758B CN 103873758 B CN103873758 B CN 103873758B CN 201210548920 A CN201210548920 A CN 201210548920A CN 103873758 B CN103873758 B CN 103873758B
- Authority
- CN
- China
- Prior art keywords
- panorama sketch
- image
- video camera
- projection
- camera
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Abstract
A kind of method that panorama sketch generates in real time is proposed, is included the following steps：Calibrating camera intrinsic parameter establishes coordinate system and creates plane of projection of cylinder；Obtain video image；By projecting the sequence of calculation by the projecting video image to the plane of projection of cylinder, panorama sketch is created.The embodiment of the present invention also proposed a kind of device that panorama sketch generates in real time.According to scheme proposed by the present invention, can in real time be generated using video flowing, live preview panorama sketch, while having the characteristics that easy to operate, hardware requirement is low.According to scheme proposed by the present invention, can in real time be generated in the mobile devices such as the mobile phone of current main-stream, preview panoramic picture.
Description
Technical field
The present invention relates to technical field of image processing, specifically, the method generated in real time the present invention relates to panorama sketch, dress
It sets and equipment.
Background technology
The field range that the shooting unit of digital imaging apparatus can capture, the generally much smaller than field range of human eye.
The project that panorama picture formation technology is image procossing, is studied extensively in computer vision field, in large scene displaying, takes the photograph
The numerous areas such as shadow, production of film and TV, training or even military affairs have important application.
For example, the application of Publication No. CN 102469259A in the prior art, panoramic picture generating means and complete are provided
Scape image generating method.In the case where generating panoramic picture by the multiple static images for constituting moving image, can generate extreme
Ground is horizontally long or the panoramic picture of lengthwise.Synthesis constitutes multiple static images of moving image and generates panoramic picture, determines from institute
The panoramic picture of generation cuts out the size of display panoramic picture, while detecting concern camera positions from panoramic picture.Also, root
Concern camera positions according to detection and identified size cut out display panoramic picture from panoramic picture.This method is rear
Processing stage generates panorama sketch.
In existing panorama picture formation technology, it can usually be attributed to the processing mode of following two major classes：
One kind is the matched device and method of feature based point.The feature of the captured image of such device and method extraction,
Such as SIFT（Scale-invariant feature transform）Feature etc. matches the characteristic point of each image, presses
Matching result carries out splicing, to build large-scale panoramic picture.Such method in specific implementation often will be by one
Fixed track carries out Image Acquisition to scene, and panorama sketch is generated by way of post-processing after completing to acquire.
Another kind of is the tracking for being carried out based on hardware positioning device camera motion posture.Common hardware device is to accelerate
Degree meter, gyroscope, GPS etc..
It is found after analysis that the scheme of panoramic picture is realized in the matching of feature based point, it is disadvantageous in that photographer
Shooting skill it is more demanding, action slightly deviation the quality of panorama sketch will be produced bigger effect.In addition, as a result of
The mode of post-processing can not real-time preview shooting result, be not easy to photographer and control shooting process.
The existing scheme that panoramic picture is realized based on hardware positioning is disadvantageous in that the precision of hardware device and higher
Cost.If precision be not enough difficult to obtain better quality as a result, the big of cost can be brought again using high-precision hardware device
Width increases.In addition, in current low and middle-end mobile device, it is many without configuring relevant hardware positioning equipment, it limits significantly
The use of such methods.
Therefore, it is necessary to propose effective technical solution, the real-time generation of panorama sketch is realized.
Invention content
The purpose of the present invention aims to solve at least one of above-mentioned technological deficiency, on the one hand the embodiment of the present invention proposes one kind
The method that panorama sketch generates in real time, includes the following steps：
Calibrating camera intrinsic parameter establishes coordinate system and creates plane of projection of cylinder；
Obtain video image；
By projecting the sequence of calculation by the projecting video image to the plane of projection of cylinder, panorama sketch is created.
On the other hand the embodiment of the present invention also proposed a kind of method for the figure that pans, include the following steps：
Start video camera and shoots video image；
According to the method that panorama sketch as described above generates in real time, the video image of shooting is produced into panorama sketch；
The panorama sketch of formation is shown on the screen.
On the other hand the embodiment of the present invention also proposed a kind of real-time generating means of panorama sketch, including：
Unit is demarcated, calibrating camera intrinsic parameter is used for；
Video acquisition unit, for obtaining video image；
Projecting cell projects the sequence of calculation by institute for establishing coordinate system and establishment plane of projection of cylinder, and for passing through
Projecting video image is stated to the plane of projection of cylinder, creates panorama sketch
On the other hand the embodiment of the present invention also proposed a kind of terminal device, including：
Starting module shoots video image for starting video camera；
The real-time generating means of panorama sketch as described above, the video image production panorama sketch for that will shoot；
Display module, the panorama sketch for showing formation on the screen.
It according to scheme proposed by the present invention, can in real time be generated, live preview panorama sketch, while had easily using video flowing
The low feature of operation, hardware requirement.It, can be real in the mobile devices such as the mobile phone of current main-stream according to scheme proposed by the present invention
Shi Shengcheng, preview panoramic picture.Said program proposed by the present invention does not interfere with system to the change very little of existing system
Compatibility, and realize simple, efficient.
The additional aspect of the present invention and advantage will be set forth in part in the description, these will become from the following description
Obviously, or practice through the invention is recognized.
Description of the drawings
Above-mentioned and/or additional aspect and advantage of the invention will become from the following description of the accompanying drawings of embodiments
Obviously and it is readily appreciated that, wherein：
Fig. 1 is the method flow diagram that panorama sketch of the embodiment of the present invention generates in real time；
Fig. 2 is the flow chart that panorama sketch generates specific example in real time；
Fig. 3 is to establish world coordinate system schematic diagram；
Fig. 4 is to establish camera coordinate system schematic diagram；
Fig. 5 is to create plane of projection of cylinder schematic diagram；
Fig. 6 is the discrete image schematic diagram for turning to corresponding resolution in project cylindrical face；
Fig. 7 is the schematic diagram for recording the corresponding mask of row pixel；
Fig. 8 is the schematic diagram for calculating the coverage area that present image projects to cylindrical surface；
Fig. 9 is the structural schematic diagram of panoramic picture generating means；
Figure 10 makes a living into the current video image that video camera obtains during panorama sketch；
Figure 11 makes a living into the splicing state of current panorama figure during panorama sketch；
Figure 12 for by the camera of terminal be aligned the figure of being panned scene；
Figure 13 is that will work as the image completion that takes of preceding camera to for generating the schematic diagram in panorama sketch painting canvas；
Figure 14 is the schematic diagram spliced to the video image that camera under each position takes；
Figure 15 is the schematic diagram that Panoramagram montage finishes.
Specific implementation mode
The embodiment of the present invention is described below in detail, examples of the embodiments are shown in the accompanying drawings, wherein from beginning to end
Same or similar label indicates same or similar element or element with the same or similar functions.Below with reference to attached
The embodiment of figure description is exemplary, and is only used for explaining the present invention, and is not construed as limiting the claims.
In order to realize on the one hand the purpose of the present invention, the embodiment of the present invention provide one kind in real time, be based on dynamic video
Panorama image generation method, including：
Calibrating camera intrinsic parameter establishes coordinate system and creates plane of projection of cylinder；
Obtain video image；
By projecting the sequence of calculation by the projecting video image to the plane of projection of cylinder, panorama sketch is created.
As shown in Figure 1, the method flow diagram generated in real time for panorama sketch of the embodiment of the present invention, including step S110 is to step
S130：
S110：Calibrating camera intrinsic parameter establishes coordinate system and creates plane of projection of cylinder.
In step s 110, it is necessary first to establish coordinate system, calibrating camera intrinsic parameter, be created justify according to intrinsic parameter thereafter
Column perspective plane.
As shown in Fig. 2, generating the flow chart of specific example in real time for panorama sketch.After beginning, calibrating camera internal reference first
It counts and establishes coordinate system and create plane of projection of cylinder.
Specifically, Calibration of camera intrinsic parameters, for example, by using " California Institute of Technology camera calibration tool box
（Caltech camera calibration toolbox）" or other common camera marking methods to the inside of video camera
Parameter is demarcated.In general, intrinsic parameters of the camera has following well known representation：
Wherein, K indicates intrinsic parameters of the camera matrix, fx, fyIndicate that camera lens focal length is in the horizontal direction and perpendicular respectively
Histogram to scale factor, u0, v0Image coordinate of the expression equivalent optical center of camera lens in image plane subpoint respectively.
Establishing coordinate system includes：
Using the equivalent optical center position of video camera initial attitude as coordinate origin, world coordinate system is established.As shown in figure 3, O is
The equivalent optical center of video camera initial attitude, AB are the optical axis of video camera initial attitude, Xw, Yw, ZwRespectively the three of world coordinate system
A reference axis, three axes are vertical two-by-two, and constitute right-handed system, Xw, ZwOn horizontal plane, Yw, ZwIt is located at optical axis AB same
On vertical plane.
Using the equivalent optical center of video camera as coordinate origin, camera coordinate system is established.As shown in figure 4, O is the initial appearance of video camera
The equivalent optical center of state, AB are the optical axis of video camera initial attitude, Xc, Yc, ZcRespectively three reference axis of world coordinate system, three
Reference axis is vertical two-by-two, and constitutes right-handed system, ZcIt is overlapped with optical axis AB, Yc, ZcIt is located on same vertical plane with optical axis AB.Wherein,
The primary optical axis of the initial attitude of video camera and the angle alpha of horizontal plane can be obtained by location hardware, can also be by estimating
Meter obtains approximation.For the point in three dimensions, under the coordinate vector Xc and world coordinate system under camera coordinate system
Coordinate vector Xw has formula（2）Shown relationship,
Xc=R*Xw+T （2）
Wherein, R, T indicate that the coordinate Xc under above-mentioned three-dimensional point camera coordinate system is transformed under world coordinate system respectively
Rotational transformation matrix and the translation transformation vector of coordinate Xw.Also, R, T determine the corresponding video camera posture of first frame video image
Matrix, concrete form such as formula（3）It is shown：
Plane of projection of cylinder is created to include the following steps：
Using the equivalent optical center of video camera as geometric center, plane of projection of cylinder is created；It is according to intrinsic parameter that project cylindrical face is discrete
The image of corresponding resolution is turned to, and carries out subgraph division on this image.
Specifically, as shown in figure 5, using the equivalent optical center of video camera as geometric center, plane of projection of cylinder is created.
As shown in fig. 6, according to intrinsic parameter by the discrete image for turning to corresponding resolution in project cylindrical face, and on this image
Carry out subgraph division.The selection mode of the resolution ratio of plane of projection of cylinder is as follows：
Wherein Dx, DyThe resolution ratio of plane of projection of cylinder horizontal direction and vertical direction, f are indicated respectivelyx, fyThe same formula of meaning
（1）, W, H indicate to obtain the width and height of single frame video image, the panorama sketch visual field height of h expression settings, μ (*) expressions pair respectively
" * " takes 2 minimum power side's value more than " * " value（For example, " * " is 5 powers that 31, μ (*) is taken as 2, i.e., 32）.Subgraph
Width and height be taken as 2 corresponding power side's value, be typically chosen as 32, the 64 equal numerical value convenient for processing.
The process for creating plane of projection of cylinder further includes that mask is arranged line by line to the project cylindrical face of above-mentioned discretization, to
Identify initial position and the final position of the pixel having been filled with.The initial value of each row mask is all set to sky, in Projection Iteration process
Middle continuous renewal.
S120：Obtain video image.
After executing the step S110, in the step s 120, video is obtained by video capture devices such as video camera, cameras
Video image frame is projected to plane of projection of cylinder by image thereafter in step s 130.
S130：It is projected video images onto on plane of projection of cylinder by projecting the sequence of calculation, creates panorama sketch.
In step s 130, each video frame is calculated by projection on sequential projection to plane of projection of cylinder, creates panorama
Figure, specifically, including to the progress conic projection of each frame video image：
Using video camera current pose matrix, the coverage area that present image projects to cylindrical surface is calculated；
It searches in coverage area and is not filled by pixel, and be filled with the projection value of present image.
I.e. as shown in Fig. 2, needing to calculate projection coverage area and filling perspective plane pixel to each video frame.
Wherein, in projecting filling process, the mask for updating projected image is needed, is often gone in projected image for identifying
The position range of filler pixels.Wherein the number of mask is equal to the line number on the perspective plane of discretization in above-mentioned steps, each mask
Identify initial position and the final position of the pixel being filled in its corresponding row.As shown in fig. 7, the row pixel is corresponding
Mask is { (C2, C4), (C7, C11), (C13, C18) }.
The coverage area that present image projects to cylindrical surface is calculated using video camera current pose matrix, as shown in Figure 8.
Dash area is drop shadow spread of the current camera on plane of projection of cylinder, is to calculate institute by attitude matrix
.According to image mask as shown in Figure 7, search and be not filled by pixel in coverage area, and with the projection value of present image into
Row filling.
Wherein, for first frame image, initial attitude matrix is obtained by the transformation relation of world coordinate system and camera coordinate system
It arrives, such as formula（2）,（3）It is shown, using initial attitude matrix, calculate its drop shadow spread on cylindrical surface, filling projection model
Enclose interior pixel.
In order to handle next video frame images, further include：
Image mask is changed after filling, changes being filled of being recorded in each row mask in the plane of projection of cylinder line by line
The initial position of pixel and final position；
The subgraph that newly fills up is searched, the subgraph newly filled up is carried out multistage down-sampled, and to subgraph and at different levels is adopted
Sampled images carry out FAST feature point extractions.
As shown in Fig. 2, when panorama sketch establishment is finished and is judged as NO, it is also necessary to more new projection plane mask and extraction subgraph
Characteristic point.
In addition, as shown in Fig. 2, for be not first frame image video data, carry out calculate projection coverage area before,
Further include：
Using the corresponding each video camera attitude matrix of each frame in front, the movement velocity of video camera is estimated, and further estimate
Count the approximation of the corresponding video camera attitude matrix of present frame；
It, will be on perspective plane on down-sampled obtained subgraphs at different levels according to the approximation of the video camera current pose matrix
Projecting characteristic points to image plane on；
The pixel in projection and current image plane by features described above point carries out characteristic matching, obtains one group of match point
It is right.
Optimize current camera according to above-mentioned matching double points using the approximation of above-mentioned current pose matrix as initial value
Attitude matrix obtains the exact value of final current pose matrix.
Specifically, using the corresponding video camera posture of each frame in front, the movement velocity of video camera is estimated, and then estimate
The approximation of the attitude matrix of current camera.Formula may be used in speed（6）It calculates：
vT=(T1-T0)/t（6）
vR=(φ1-φ0)/t
Wherein, vT, vRThe speed of video camera displacement and rotation, T are indicated respectively0, T1, front cross frame video camera is indicated respectively
Motion vector, φ0, φ1Indicate that the Eulerian angles vector of front cross frame video camera, t indicate the time interval of front cross frame respectively.It takes the photograph
The current location of camera can use formula（7）It calculates：
Wherein Tcur, Rcur, the current motion vector and spin matrix of estimation is indicated respectively,Indicate former frame and current
The time interval of frame, ξ（*）It indicates to construct spin matrix, v by Eulerian anglesT, vR, T1, φ1The same formula of meaning（6）.
Using obtained video camera current pose matrix by the projecting characteristic points on the subgraphs at different levels of perspective plane to image plane
On.Characteristic point therein can use " FAST " characteristic point, but be not limited to " FAST " characteristic point.
The pixel in projection and current image plane by features described above point carries out characteristic matching, obtains one group of match point
It is right.Matching algorithm therein uses the common " NormalizedCross-Correlation of image processing field（NCC）”
Method.
Optimize current camera according to above-mentioned matching double points using the approximation of above-mentioned current pose matrix as initial value
Attitude matrix obtains the exact value of final current pose matrix.Specific method is：With the optimization for minimizing re-projection error
Method is calculated, and first according to the matching result of lowest resolution sampled images characteristic point, calculates current video camera posture
Matrix, and the initial value of calculating is optimized in this, as the matching result using high rank sampled images, until highest is differentiated
The image of rate obtains the exact value of the attitude matrix of final Current projection calculating cycle.Wherein, the object function of optimization is such as
Formula（8）It is shown：
Wherein, reproj[R|T](Xi) indicate characteristic point XiProject to the subpoint of image plane, miIndicate image plane on
Characteristic point XiMatched picture point, ρ（*）Indicate Euclidean distance, [R | T] indicate the video camera attitude matrix that optimization is calculated.
Further include the initial attitude square for the value modification video camera for adjusting above-mentioned alpha in addition, after the completion of panorama sketch structure
Battle array, is corrected the panorama sketch.
For example, the error of alpha shown in Fig. 4 can make the panorama sketch of generation generate inclined effect, taken the photograph by modification
The initial attitude parameter alpha of camera, it is specific can to correct image tilt phenomenon caused by the error because of posture initial estimate
Implementing method can be realized by adjusting software interface parameter, after interface parameter adjusts, according to new parameter according to aforementioned step
Suddenly panoramic picture is recalculated.
As shown in figure 9, another aspect of the present invention provides a kind of real-time panoramic picture generation dress based on dynamic video
Set 100, including：Demarcate unit 110, video acquisition unit 120 and projecting cell 130.Wherein：
Unit 110 is demarcated, calibrating camera intrinsic parameter is used for.
Video acquisition unit 120, for obtaining video image.
Projecting cell 130, for establish coordinate system and create plane of projection of cylinder, and for by project the sequence of calculation will
Projecting video image creates panorama sketch to plane of projection of cylinder.
As the embodiment of panoramic picture generating means 100, projecting cell 130 is further used for video camera initial attitude
Equivalent optical center position be coordinate origin, establish world coordinate system；And it is additionally operable to using the equivalent optical center of video camera as coordinate original
Point, establishes camera coordinate system, for the point in three dimensions, the coordinate X under camera coordinate systemcUnder world coordinate system
Coordinate XwWith following relationship：Xc=R*Xw+ T, wherein R, T indicate the coordinate under above-mentioned three-dimensional point camera coordinate system respectively
XcTransform to the coordinate X under world coordinate systemwRotational transformation matrix and translation transformation vector.
As the embodiment of panoramic picture generating means 100, projecting cell 130 is further used for：
Using the equivalent optical center of video camera as geometric center, plane of projection of cylinder is created；
According to intrinsic parameter by the discrete image for turning to corresponding resolution in project cylindrical face, and subgraph is carried out on this image
It divides, the selection mode of corresponding resolution is：
Wherein, Dx, DyThe resolution ratio of plane of projection of cylinder horizontal direction and vertical direction, f are indicated respectivelyx, fyIt indicates to take the photograph respectively
Camera lens focal length in the horizontal direction with the scale factor of vertical direction, W, H indicate respectively obtain single frame video image width and
Height, h indicate that the panorama sketch visual field height of setting, μ (*) indicate to take " * " 2 minimum power side's value more than " * " value；
Mask is arranged line by line to plane of projection of cylinder, to identify initial position and the final position of the pixel having been filled with.
As the embodiment of panoramic picture generating means 100, projecting cell 130 further includes computing unit 131,
Include that computing unit 131 regards each frame by projecting the sequence of calculation to project video images onto plane of projection of cylinder
Frequency image carries out cylindrical surface projection and calculates：
Using video camera current pose matrix, the coverage area that present image projects to cylindrical surface is calculated；
It searches in coverage area and is not filled by pixel, and be filled with the projection value of present image.
As the embodiment of panoramic picture generating means 100, projecting cell 130 further includes that fills unit 132 and feature carry
Unit 133 is taken,
Fills unit 132 is changed in plane of projection of cylinder recorded in each row mask line by line for changing image mask after filling
The pixel being filled initial position and final position；
Feature extraction unit 133 carries out multistage drop to the subgraph newly filled up and adopts for searching the subgraph newly filled up
Sample, and image characteristic point extraction is carried out to subgraph and sampled images at different levels.
As the embodiment of panoramic picture generating means 100, projecting cell 130 further includes resetting bit location 134, reorientation
Unit 134 is used for：
Using the corresponding each video camera attitude matrix of each frame in front, the movement velocity of video camera is estimated, and further estimate
Count the approximation of the corresponding video camera attitude matrix of present frame；
According to the approximation of video camera current pose matrix, by the projecting characteristic points on the subgraphs at different levels of perspective plane to as flat
On face；
The pixel in projection and current image plane by features described above point carries out characteristic matching, obtains one group of match point
It is right；
Using the approximation of current pose matrix as initial value, according to matching double points, optimize the attitude matrix of current camera,
Obtain the value of accurate current pose matrix.
Further include correction unit 140, correction unit is with 140 in adjustment as the embodiment of panoramic picture generating means 100
The corresponding video camera attitude matrix of the first frame video image of parameter modification on interface, is corrected panorama sketch.
In addition, as accessory, for example, it is also possible to include storage unit, it to be used for the storage of panoramic picture；Display is single
Member, the generation result for real-time display current video image and panoramic picture.It should be appreciated that in a particular application, panorama
Video generation device 100 can become individual equipment, can also be presented as and be integrated among terminal device.
" terminal " used herein above, " terminal device " had both included the wireless signal receiver for only having non-emissive ability
Equipment, and include the equipment with the reception and transmitting hardware that can carry out two-way communication on bidirectional communication link.It is this to set
It is standby to may include：Honeycomb with and without multi-line display or other communication equipments；Can with combine voice and data processing,
The PCS Personal Communications System of fax and/or communication ability（PCS）；May include radio frequency receiver and pager, internet/interior
Networking access, web browser, notepad, calendar and/or global positioning system（GPS）The personal digital assistant of receiver
（PDA）；And/or conventional laptop including radio frequency receiver and/or palmtop computer or other equipment.It is used herein above
" terminal ", " terminal device " can be portable, can transport, be mounted on the vehicles（Aviation, sea-freight and/or land）In,
Or it is suitable for and/or is configured to transport in any other of the earth and/or space position in local runtime and/or with distribution form
Row.
It is of the invention for ease of understanding, the above method or equipment are described further with reference to specific processing procedure.
For example, flow chart according to fig. 2：
Calibration of camera intrinsic parameters.Using Caltech camera calibration toolbox in video camera
Portion's parameter is demarcated.F in this examplex, fy, u0, v0Calibration result be 669.43519,667.52837 respectively,
318.96791,233.78367.
Using the equivalent optical center of video camera as coordinate origin, world coordinate system is established.
Using the equivalent optical center of video camera as coordinate origin, initial camera coordinate system is established.Wherein, the initial attitude of video camera
In the angles alpha（As shown in Figure 4）It is set as 0.
Using the equivalent optical center of video camera as geometric center, plane of projection of cylinder is created, it is according to intrinsic parameter that project cylindrical face is discrete
The image of appropriate resolution ratio is turned to, and carries out subgraph division on this image.In this example, the single frame video that video camera obtains
Resolution ratio is 640 × 480, according to formula（4）、（5）It calculates, the resolution ratio of panoramic picture is 4096 × 1024.Point of subgraph
Resolution is taken as 64 × 64.
The video image for generating panoramic picture is obtained by video capture devices such as video camera, cameras.
The mask of projected image is set, and to identify, often row has been filled with the position range of pixel in projected image.
Panoramic picture generating process calculates Sequence composition by projection.In each projected image calculating cycle, including it is following
Calculating process：
Using the corresponding each video camera attitude matrix of each frame in front, the movement velocity of video camera is estimated, and further estimate
Count the approximation of the corresponding video camera attitude matrix of present frame；
According to the approximation of the video camera current pose matrix, the projecting characteristic points on the subgraphs at different levels of perspective plane are arrived
In image plane；
The pixel in projection and current image plane by features described above point carries out characteristic matching, obtains one group of match point
It is right；
Optimize current camera according to the matching double points using the approximation of the current pose matrix as initial value
Attitude matrix obtains the value of accurate current pose matrix.
The coverage area that present image projects to cylindrical surface is calculated with obtained video camera current pose matrix.According to image
Mask is searched in coverage area and is not filled by pixel, and is filled with the projection value of present image.
Image mask is changed after filling, and searches the subgraph newly filled up, the subgraph newly filled up is carried out multistage
It is down-sampled, and FAST feature extractions are carried out to subgraph and sampled images at different levels.
By changing the initial attitude parameter alpha of video camera, can correct produced by the error because of posture initial estimate
Scalloping.
For example, during Figure 10 makes a living into panorama sketch, the current video image of video camera acquisition.Figure 11 makes a living into panorama
During figure, the splicing state of current panorama figure.Rectangle frame interior zone wherein on image indicates the throwing of current camera
Shadow zone domain.
According to the method that panorama sketch disclosed above generates in real time, in specific application, the embodiment of the present invention also proposes
A kind of method for the figure that pans, includes the following steps：
Start video camera and shoots video image；
According to the method that panorama sketch disclosed above generates in real time, the video image of shooting is produced into panorama sketch；
The panorama sketch of formation is shown on the screen.
As shown in figure 12, for by the camera of terminal be aligned the figure of being panned scene.
Specifically, further including：When starting video camera shooting video image, video camera can move in any direction, lead to
It crosses rotation camera to shoot scene, and shows the video image taken on the screen, real-time update video image pair
The panorama sketch answered.
For example, when user carries out panorama sketch shooting, to the style of shooting of the first-class video acquisition unit of user's dollying
Without limitation.User can shoot scene with the movement of upper and lower, left and right any direction, rotation camera, obtain video image.
In addition, when the corresponding panorama sketch of real-time update video image, the splicing of Dynamic Announce panorama sketch on the screen.
E.g., including Panoramagram montage status display, user can shield simultaneously during carrying out panorama sketch shooting
Observe that view field of the current camera on panorama sketch can be observed in the splicing state of panoramic picture, user on curtain.For example,
On the screen when the splicing of Dynamic Announce panorama sketch, throwing of the video image of current camera shooting on panorama sketch is shown
Shadow zone domain.As shown in figure 13, for image completion that preceding camera takes will be worked as to for generating the signal in panorama sketch painting canvas
Figure, wherein the region that stain surrounds in the screen of lower half portion is view field of the video image of diagram on panorama sketch.Such as figure
Shown in 14, the schematic diagram that the video image to be taken to camera under each position is spliced is black in the screen of lower half portion
The region that point surrounds is view field of the video image of diagram on panorama sketch.In addition, during Panoramagram montage, use
Family adjusts by interactive mode and/or terminates splicing, for example, user adjusts the first frame video of parameter modification on adjustment interface
The corresponding video camera attitude matrix of image, is corrected panorama sketch；For example, when user is satisfied with the panorama sketch of shooting, pass through
It clicks End keys and stops shooting.As shown in figure 15, it is the schematic diagram that finishes of Panoramagram montage, stain surrounds in the screen of lower half portion
Region be diagram view field of the video image on panorama sketch.
According to the real-time generating means of panorama sketch disclosed above or method, in specific application, the embodiment of the present invention is also
A kind of terminal device is proposed, including：
Starting module shoots video image for starting video camera；
The real-time generating means of panorama sketch disclosed above, the video image production panorama sketch for that will shoot；Alternatively, generating
The video image of shooting is produced panorama sketch by module, the method for being generated in real time according to panorama sketch disclosed above；
Display module, the panorama sketch for showing formation on the screen.
Specifically, when starting video camera shooting video image, video camera can move in any direction, be taken the photograph by rotation
As head shoots scene；Display module is additionally operable to show the video image taken on the screen, real-time update video figure
As corresponding panorama sketch.
In addition, when the corresponding panorama sketch of real-time update video image, it is complete that display module is additionally operable to Dynamic Announce on the screen
The splicing of scape figure.
Terminal device includes but not limited to the mobile devices such as mobile phone, tablet computer, is described as follows by taking mobile phone as an example：
Mobile phone camera is directed to the scene for the figure of being panned, as shown in figure 12, for the camera of terminal to be aligned
The scene for the figure of being panned.
The start button " Start " on interface is clicked, by the image completion taken when preceding camera to for generating entirely
In scape picture cloth.For example, on the screen the splicing of Dynamic Announce panorama sketch when, display current camera shooting video figure
As the view field on panorama sketch.As shown in figure 13, for image completion that preceding camera takes will be worked as to complete for generating
Schematic diagram in scape picture cloth, wherein the region that stain surrounds in the screen of lower half portion is the video image of diagram in panorama sketch
On view field.On mobile phone display screen, the video image of mobile phone camera current shooting, lower half is shown in top half
The splicing state of current panorama figure is shown in part, and the rectangle frame interior zone wherein on image indicates the throwing when preceding camera
Shadow zone domain, the image of rectangle frame interior zone are the video image of camera current shooting.
Upper and lower, left and right any direction is mobile, rotation mobile phone camera shoots scene.In shooting process, to each
The video image that camera takes under a position is spliced.As shown in figure 14, to be taken to camera under each position
The schematic diagram that is spliced of video image, the region that stain surrounds in the screen of lower half portion be the video image that illustrates in panorama
View field on figure.
In addition, during Panoramagram montage, user adjusts by interactive mode and/or terminates splicing, for example,
User adjusts the corresponding video camera attitude matrix of the first frame video image of parameter modification on adjustment interface, and school is carried out to panorama sketch
Just；For example, when user is satisfied with the panorama sketch of shooting, stop shooting by clicking End keys.When Panoramagram montage finishes, or use
When conclusion button " End " is clicked at family, by the panoramic picture spliced storage to storage device, terminate whole process.Such as Figure 15
Schematic diagram that is shown, finishing for Panoramagram montage, the region that stain surrounds in the screen of lower half portion be that the video image illustrated exists
View field on panorama sketch.
Those skilled in the art are appreciated that realize all or part of step that above-described embodiment method carries
Suddenly it is that relevant hardware can be instructed to complete by program, the program can be stored in a kind of computer-readable storage medium
In matter, which includes the steps that one or a combination set of embodiment of the method when being executed.
In addition, each functional unit in each embodiment of the present invention can be integrated in a processing module, it can also
That each unit physically exists alone, can also two or more units be integrated in a module.Above-mentioned integrated mould
The form that hardware had both may be used in block is realized, can also be realized in the form of software function module.The integrated module is such as
Fruit is realized in the form of software function module and when sold or used as an independent product, can also be stored in a computer
In read/write memory medium.
Storage medium mentioned above can be read-only memory, disk or CD etc..
The above is only some embodiments of the present invention, it is noted that for the ordinary skill people of the art
For member, various improvements and modifications may be made without departing from the principle of the present invention, these improvements and modifications are also answered
It is considered as protection scope of the present invention.
Claims (24)
1. a kind of method that panorama sketch generates in real time, which is characterized in that include the following steps：
Calibrating camera intrinsic parameter establishes coordinate system and creates plane of projection of cylinder；
Obtain video image；
By projecting the sequence of calculation by the projecting video image to the plane of projection of cylinder, panorama sketch is created；
The initial attitude matrix of video camera is changed by adjusting the primary optical axis of the initial attitude of video camera and the angle of horizontal plane, it is right
The panorama sketch is corrected.
2. the method that panorama sketch as described in claim 1 generates in real time, which is characterized in that establishing coordinate system includes：
Using the equivalent optical center position of video camera initial attitude as coordinate origin, world coordinate system is established；
Using the equivalent optical center of video camera as coordinate origin, camera coordinate system is established, for the point in three dimensions, video camera is sat
Coordinate X under mark systemcWith the coordinate X under world coordinate systemwWith following relationship：Xc=R*Xw+ T, wherein on R, T are indicated respectively
State the coordinate X under three-dimensional point camera coordinate systemcTransform to the coordinate X under world coordinate systemwRotational transformation matrix and translation become
Commutation amount.
3. the method that panorama sketch as described in claim 1 generates in real time, which is characterized in that it includes following to create plane of projection of cylinder
Step：
Using the equivalent optical center of video camera as geometric center, plane of projection of cylinder is created；
According to the intrinsic parameter by the discrete image for turning to corresponding resolution in project cylindrical face, and subgraph is carried out on this image
It divides, the selection mode of the corresponding resolution is：
Wherein, Dx, DyThe resolution ratio of plane of projection of cylinder horizontal direction and vertical direction, f are indicated respectivelyx, fyVideo camera is indicated respectively
In the horizontal direction with the scale factor of vertical direction, W, H indicate to obtain the width and height of single frame video image, h respectively lens focus
Indicate that the panorama sketch visual field height of setting, μ (*) indicate to take " * " 2 minimum power side's value more than " * " value；
Mask is arranged line by line to the plane of projection of cylinder, to identify initial position and the final position of the pixel having been filled with.
4. the method that panorama sketch as described in claim 1 generates in real time, which is characterized in that will be described by projecting the sequence of calculation
Projecting video image includes that carrying out cylindrical surface projection to each frame video image calculates to the plane of projection of cylinder：
Using video camera current pose matrix, the coverage area that present image projects to cylindrical surface is calculated；
It searches in coverage area and is not filled by pixel, and be filled with the projection value of present image.
5. the method that panorama sketch as claimed in claim 4 generates in real time, which is characterized in that further include：
Image mask is changed after filling, changes the pixel being filled recorded in each row mask in the plane of projection of cylinder line by line
Initial position and final position；
The subgraph newly filled up is searched, it is multistage down-sampled to the subgraph progress newly filled up, and to subgraph and sample graph at different levels
As carrying out image characteristic point extraction.
6. the method that panorama sketch as claimed in claim 5 generates in real time, which is characterized in that when the video image of projection is not first
When frame image, further include：
Using the corresponding each video camera attitude matrix of each frame in front, the movement velocity of video camera is estimated, and further estimation is worked as
The approximation of the corresponding video camera attitude matrix of previous frame；
According to the approximation of the video camera current pose matrix, by the projecting characteristic points on the subgraphs at different levels of perspective plane to as flat
On face；
The pixel in projection and current image plane by features described above point carries out characteristic matching, obtains one group of matching double points；
Using the approximation of the current pose matrix as initial value, according to the matching double points, optimize the posture of current camera
Matrix obtains the value of accurate current pose matrix.
7. the method that panorama sketch as claimed in claim 4 generates in real time, which is characterized in that described by adjusting the first of video camera
The initial attitude matrix of the primary optical axis of beginning posture and the angle modification video camera of horizontal plane, to the corrected step of the panorama sketch
Suddenly, including：
The corresponding video camera attitude matrix of the first frame video image of parameter modification on software interface is adjusted, the panorama sketch is carried out
Correction.
8. a kind of method for the figure that pans, which is characterized in that include the following steps：
Start video camera and shoots video image；
According to the method that the panorama sketch as described in claim 1 to 7 is one of arbitrary generates in real time, by the video image of shooting
Produce panorama sketch；
The panorama sketch of formation is shown on the screen.
9. the method for the figure as claimed in claim 8 that pans, which is characterized in that further include：
When starting video camera shooting video image, the video camera can move in any direction, by rotating camera to field
Scape is shot；
Show the video image taken on the screen, the corresponding panorama sketch of video image described in real-time update.
10. the method for the figure as claimed in claim 9 that pans, which is characterized in that video image described in real-time update corresponds to
The panorama sketch when, the splicing of panorama sketch described in Dynamic Announce on the screen.
11. the method for the figure as claimed in claim 10 that pans, which is characterized in that further include：
On the screen when the splicing of panorama sketch described in Dynamic Announce, the video image of display current camera shooting is in panorama
View field on figure.
12. the method for the figure as claimed in claim 10 that pans, which is characterized in that further include：
On the screen in the splicing of panorama sketch described in Dynamic Announce, after the adjustment and/or END instruction that detect triggering, adjust
The whole and/or end splicing.
13. a kind of real-time generating means of panorama sketch, which is characterized in that including：
Unit is demarcated, calibrating camera intrinsic parameter is used for；
Video acquisition unit, for obtaining video image；
Projecting cell is regarded for establishing coordinate system and creating plane of projection of cylinder, and for passing through the projection sequence of calculation by described in
Frequency image projection creates panorama sketch to the plane of projection of cylinder；
Unit is corrected, the first of video camera is changed for the primary optical axis of the initial attitude by adjusting video camera and the angle of horizontal plane
Beginning attitude matrix is corrected the panorama sketch.
14. the real-time generating means of panorama sketch as claimed in claim 13, which is characterized in that the projecting cell is further used for
Using the equivalent optical center position of video camera initial attitude as coordinate origin, world coordinate system is established；And it is additionally operable to video camera etc.
Effect optical center is coordinate origin, establishes camera coordinate system, for the point in three dimensions, the coordinate X under camera coordinate systemc
With the coordinate X under world coordinate systemwWith following relationship：Xc=R*Xw+ T, wherein R, T indicate above-mentioned three-dimensional point video camera respectively
Coordinate X under coordinate systemcTransform to the coordinate X under world coordinate systemwRotational transformation matrix and translation transformation vector.
15. the real-time generating means of panorama sketch as claimed in claim 13, which is characterized in that the projecting cell is further used
In：
Using the equivalent optical center of video camera as geometric center, plane of projection of cylinder is created；
According to the intrinsic parameter by the discrete image for turning to corresponding resolution in project cylindrical face, and subgraph is carried out on this image
It divides, the selection mode of the corresponding resolution is：
Wherein, Dx, DyThe resolution ratio of plane of projection of cylinder horizontal direction and vertical direction, f are indicated respectivelyx, fyVideo camera is indicated respectively
In the horizontal direction with the scale factor of vertical direction, W, H indicate to obtain the width and height of single frame video image, h respectively lens focus
Indicate that the panorama sketch visual field height of setting, μ (*) indicate to take " * " 2 minimum power side's value more than " * " value；
Mask is arranged line by line to the plane of projection of cylinder, to identify initial position and the final position of the pixel having been filled with.
16. the real-time generating means of panorama sketch as claimed in claim 13, which is characterized in that the projecting cell further includes calculating
Unit,
Include that the computing unit is to each to the plane of projection of cylinder by the projecting video image by projecting the sequence of calculation
Frame video image carries out cylindrical surface projection and calculates：
Using video camera current pose matrix, the coverage area that present image projects to cylindrical surface is calculated；
It searches in coverage area and is not filled by pixel, and be filled with the projection value of present image.
17. the real-time generating means of panorama sketch as claimed in claim 16, which is characterized in that the projecting cell further includes filling
Unit and feature extraction unit,
The fills unit is changed in the plane of projection of cylinder and is remembered in each row mask for changing image mask after filling line by line
The initial position for the pixel of record being filled and final position；
The feature extraction unit, it is multistage down-sampled to the subgraph progress newly filled up for searching the subgraph newly filled up, and
Image characteristic point extraction is carried out to subgraph and sampled images at different levels.
18. the real-time generating means of panorama sketch as claimed in claim 17, which is characterized in that the projecting cell further includes resetting
Bit location, the bit location that resets are used for：
Using the corresponding each video camera attitude matrix of each frame in front, the movement velocity of video camera is estimated, and further estimation is worked as
The approximation of the corresponding video camera attitude matrix of previous frame；
According to the approximation of the video camera current pose matrix, by the projecting characteristic points on the subgraphs at different levels of perspective plane to as flat
On face；
The pixel in projection and current image plane by features described above point carries out characteristic matching, obtains one group of matching double points；
Using the approximation of the current pose matrix as initial value, according to the matching double points, optimize the posture of current camera
Matrix obtains the value of accurate current pose matrix.
19. the real-time generating means of panorama sketch as claimed in claim 16, which is characterized in that the correction unit is for passing through tune
The initial attitude matrix of the primary optical axis of the initial attitude of whole video camera and the angle modification video camera of horizontal plane, to the panorama sketch
It is corrected, including：
The corresponding video camera attitude matrix of the first frame video image of parameter modification on software interface is adjusted, the panorama sketch is carried out
Correction.
20. a kind of terminal device, which is characterized in that including：
Starting module shoots video image for starting video camera；
The real-time generating means of panorama sketch as described in claim 13 to 19 is one of arbitrary, the video image for that will shoot
Produce panorama sketch；
Display module, the panorama sketch for showing formation on the screen.
21. terminal device as claimed in claim 20, which is characterized in that described to take the photograph when starting video camera shooting video image
Camera shoots scene by rotating camera for moving in any direction；
The display module is additionally operable to show the video image taken on the screen, and video image is corresponding described in real-time update
The panorama sketch.
22. terminal device as claimed in claim 21, which is characterized in that video image described in real-time update is corresponding described complete
When scape figure, the display module is additionally operable to the splicing of panorama sketch described in Dynamic Announce on the screen.
23. terminal device as claimed in claim 22, which is characterized in that further include：
On the screen when the splicing of panorama sketch described in Dynamic Announce, the display module is additionally operable to show on the screen current
View field of the video image of video camera shooting on panorama sketch.
24. terminal device as claimed in claim 22, which is characterized in that further include：
On the screen in the splicing of panorama sketch described in Dynamic Announce, after the adjustment and/or END instruction that detect triggering, adjust
The whole and/or end splicing.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210548920.8A CN103873758B (en) | 2012-12-17 | 2012-12-17 | The method, apparatus and equipment that panorama sketch generates in real time |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210548920.8A CN103873758B (en) | 2012-12-17 | 2012-12-17 | The method, apparatus and equipment that panorama sketch generates in real time |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN103873758A CN103873758A (en) | 2014-06-18 |
| CN103873758B true CN103873758B (en) | 2018-09-21 |
Family
ID=50911851
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201210548920.8A Active CN103873758B (en) | 2012-12-17 | 2012-12-17 | The method, apparatus and equipment that panorama sketch generates in real time |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103873758B (en) |
Families Citing this family (19)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9883101B1 (en) * | 2014-07-23 | 2018-01-30 | Hoyos Integrity Corporation | Providing a real-time via a wireless communication channel associated with a panoramic video capture device |
| CN104238884B (en) * | 2014-09-12 | 2018-09-18 | 北京诺亚星云科技有限责任公司 | Multidate information based on digital panoramic is presented and user interactive system and equipment |
| CN105635651B (en) * | 2014-10-29 | 2019-05-24 | 浙江大华技术股份有限公司 | A kind of ball machine localization method and device |
| CN105635551B (en) * | 2014-10-29 | 2019-03-26 | 浙江大华技术股份有限公司 | A kind of ball-shaped camera generates the method and ball-shaped camera of panoramic picture |
| KR20160133230A (en) * | 2015-05-12 | 2016-11-22 | 엘지전자 주식회사 | Mobile terminal |
| EP3378221B1 (en) * | 2015-11-16 | 2022-01-12 | Google LLC | Stabilization based on accelerometer data |
| CN105554450B (en) * | 2015-12-14 | 2018-10-12 | 深圳大学 | Distributed video panorama display system |
| US10282814B2 (en) * | 2016-01-07 | 2019-05-07 | Mediatek Inc. | Method and apparatus of image formation and compression of cubic images for 360 degree panorama display |
| CN105809701B (en) * | 2016-03-25 | 2019-03-08 | 成都易瞳科技有限公司 | Panoramic video posture scaling method |
| CN105915794A (en) * | 2016-05-27 | 2016-08-31 | 武汉理工大学 | Panoramic scene making method using high-speed ball camera |
| DE102017009145A1 (en) * | 2016-10-14 | 2018-04-19 | Avago Technologies General Ip (Singapore) Pte. Ltd. | Capture and playback 360-degree video |
| CN107993276B (en) | 2016-10-25 | 2021-11-23 | 杭州海康威视数字技术股份有限公司 | Panoramic image generation method and device |
| CN106815869B (en) * | 2016-10-28 | 2020-06-19 | 北京鑫洋泉电子科技有限公司 | Optical center determining method and device of fisheye camera |
| US10410349B2 (en) * | 2017-03-27 | 2019-09-10 | Microsoft Technology Licensing, Llc | Selective application of reprojection processing on layer sub-regions for optimizing late stage reprojection power |
| CN107147910B (en) * | 2017-05-31 | 2019-11-22 | 上海国茂数字技术有限公司 | A kind of panoramic video hexagon method of sampling and device |
| CN109840897A (en) * | 2017-11-28 | 2019-06-04 | 深圳市航盛电子股份有限公司 | Vehicle panoramic method for correcting image and vehicle panoramic system |
| CN109978760B (en) * | 2017-12-27 | 2023-05-02 | 杭州海康威视数字技术股份有限公司 | Image stitching method and device |
| TWI700000B (en) * | 2019-01-29 | 2020-07-21 | 威盛電子股份有限公司 | Image stabilization method and apparatus for panoramic video, and method for evaluating image stabilization algorithm |
| CN113496520A (en) * | 2020-04-02 | 2021-10-12 | 北京四维图新科技股份有限公司 | Method and device for turning top view of camera and storage medium |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101710416A (en) * | 2009-12-07 | 2010-05-19 | 中国科学院新疆生态与地理研究所 | Processing method for multiple-target remote sensing image clouds |
| CN101895693A (en) * | 2010-06-07 | 2010-11-24 | 北京高森明晨信息科技有限公司 | Method and device for generating panoramic image |
-
2012
- 2012-12-17 CN CN201210548920.8A patent/CN103873758B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101710416A (en) * | 2009-12-07 | 2010-05-19 | 中国科学院新疆生态与地理研究所 | Processing method for multiple-target remote sensing image clouds |
| CN101895693A (en) * | 2010-06-07 | 2010-11-24 | 北京高森明晨信息科技有限公司 | Method and device for generating panoramic image |
Non-Patent Citations (1)
| Title |
|---|
| 《柱面全景图自动生成方法研究》;童宇;《中国优秀硕士学位论文全文数据库》;20060531;1.3，2.1-2.2,3.1-3.3,5.3,5.5.1,6.1 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN103873758A (en) | 2014-06-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103873758B (en) | The method, apparatus and equipment that panorama sketch generates in real time | |
| US10495461B2 (en) | Surveying system | |
| CN110799921A (en) | Shooting method and device and unmanned aerial vehicle | |
| US20060078214A1 (en) | Image processing based on direction of gravity | |
| US20120293607A1 (en) | Panorama Processing | |
| US20060078215A1 (en) | Image processing based on direction of gravity | |
| CN107646126A (en) | Camera Attitude estimation for mobile device | |
| US9799136B2 (en) | System, method and apparatus for rapid film pre-visualization | |
| CN206563985U (en) | 3-D imaging system | |
| JP2007226761A (en) | Locus image composition device for image object, locus image display device for image object, and program therefor | |
| WO2017112800A1 (en) | Macro image stabilization method, system and devices | |
| WO2022242395A1 (en) | Image processing method and apparatus, electronic device and computer-readable storage medium | |
| CN113450253B (en) | Image processing method, image processing device, electronic equipment and computer readable storage medium | |
| US20090059018A1 (en) | Navigation assisted mosaic photography | |
| CN113450254B (en) | Image processing method, image processing device, electronic equipment and computer readable storage medium | |
| WO2023241495A1 (en) | Photographic method and apparatus | |
| CN109547689A (en) | Automatically snap control method, device and computer readable storage medium | |
| CN114022562A (en) | Panoramic video stitching method and device capable of keeping integrity of pedestrians | |
| CN114339029B (en) | Shooting method and device and electronic equipment | |
| CN114332244A (en) | Camera positioning method and system for virtual studio | |
| CN109688306A (en) | Automatically snap control method, device, system, equipment and computer readable storage medium | |
| CN109688305A (en) | Automatically snap control method, device and computer readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |