CN105094739B - A kind of multiple terminals screen method for automatically split-jointing based on mobile terminal front camera - Google Patents
A kind of multiple terminals screen method for automatically split-jointing based on mobile terminal front camera Download PDFInfo
- Publication number
- CN105094739B CN105094739B CN201510527082.XA CN201510527082A CN105094739B CN 105094739 B CN105094739 B CN 105094739B CN 201510527082 A CN201510527082 A CN 201510527082A CN 105094739 B CN105094739 B CN 105094739B
- Authority
- CN
- China
- Prior art keywords
- mobile terminal
- front camera
- screen
- camera
- coordinate
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
Landscapes
- Studio Devices (AREA)
Abstract
The present invention is a kind of multiple terminals screen method for automatically split-jointing based on mobile terminal front camera, and this method carries out camera calibration so as to obtain the intrinsic parameter of each video camera first against the front camera of each mobile terminal；Secondly the spatial relationship of mobile terminal screen and its front camera is determined using accurate measuring method；Then during online, outer parameter of each front camera relative to same external markers is obtained by the method for camera calibration；Finally, with reference to the outer parameter and screen of front camera and the spatial relationship of front camera and consider that the resolution ratio of each terminal screen determines final display area.It is an object of the present invention to provide a kind of automatic multi-screen joining method based on computer vision using mobile terminal front camera, then shows the screen of spliced bigger for content.
Description
Technical field
The invention belongs to computer man-machine interacting technical fields.
Background technology
The important topic that content is computer man-machine interacting field is provided a user in a manner of clearer.Wherein,
Computer display screens are as providing a user the main equipment of content in the important role of field of human-computer interaction performer.With
The development of information technology, high-resolution display equipment is progressed into people’s lives brings good vision for user
Experience.However as mobile terminal devices such as an important branch of computing device, mobile phone, tablets in this side of screen resolution
Face existing inborn deficiency, is limited to the size of mobile terminal device, resolution sizes, which are compared to traditional PC, to be had
Larger gap.Since current mobile terminal device generally has mobile light, display bezel narrow or even Rimless etc.
Feature, researcher have carried out the research of multi-mobile-terminal screen splicing technical aspect.But current splicing is being supported
Special-shaped terminal and display area determine aspect, and there are still have the problem of to be solved：
(1) for supporting in terms of special-shaped terminal, the wide variety and different model terminal of current mobile terminal are in screen
Resolution ratio, physical size etc. be easy to cause when splicing that content is discontinuous, there are larger aberration etc. there is larger difference
Problem, how to overcome the difference of terminal room that a splicing interface with good visual effect is presented to user is this technology
Problem.
(2) display area determines aspect, due to being put when otherness possessed between mobile terminal and multiple terminals splicing
Mode is not fixed the problem caused in determining display area.How multiple mobile terminals are accurately obtained by effective mode
Spatial relationship between screen is to determine the core content of display area.In order to enable the connecting method supported is diversified as possible, institute
The technology of use must be able to carry out the real-time update of display area according to practical situation, this proposes new for this technology again
Problem and challenge.
Presently, there are research work in, the regularly placing of multiple same model terminal rooms is only supported to splice mostly, and need
The display area of each terminal is artificially determined according to terminal quantity and connecting method.For multiple special-shaped terminals it is irregular from
Dynamic Bonding Problem, such method cannot preferably complete splicing task.
Invention content
The purpose of the present invention：The front camera having using current mobile terminal, research one kind are regarded based on computer
The automatic multi-mobile-terminal screen splicing method felt efficiently uses the space that front camera is obtained between mobile terminal screen and closes
System, structure one have multiple screen splicings into have it is larger display area logical screen, to need the one of large screen display
A little applications provide infrastructural support.
Multi-mobile-terminal screen method for automatically split-jointing proposed by the present invention based on mobile terminal front camera, it is main
Step is as follows：
Step (1), the front camera for each mobile terminal carry out camera calibration, so that it is determined that each preposition camera shooting
The intrinsic parameter of head (including parameters such as focal length, imaging centers)；
Step (2) determines that the space between mobile terminal screen and its front camera is closed using accurate measuring method
System；
Step (3), it is online during, with reference to front camera intrinsic parameter obtained by calibrating in (1), measure it is each it is mobile eventually
Hold outer parameter of the front camera relative to same extrinsic calibration object；
Step (4), the Camera extrinsic number with reference to acquisition in the spatial relationship and (3) measured in (2), will by wireless network
These data transmissions of every mobile terminal are uniformly processed to a server, are considering mobile terminal different resolution
Under the conditions of determine the final display area of each mobile terminal.
Further, the particular content of the calibration of front camera intrinsic parameter is as follows in the step (1)：
Step (A1), print a known grid length of side checkerboard pattern be fixed on a smooth tablet, fixation take the photograph
As the focal length of head shoots several images containing complete checkerboard pattern from different angles；
Step (A2) detects tessellated characteristic point in image, and the real space for establishing checkerboard pattern characteristic point is sat
Correspondence between mark and its image coordinate brings the imaging equation of front camera into；
Step (A3), the intrinsic parameter that every mobile terminal is acquired by solving the linear equation (overdetermination) obtained in (A2).
The step (2) determines that the step of screen is with front camera spatial relationship is specific as follows：
Step (B1), the length and width that mobile terminal screen is measured with accurate physical measuring methods；
Step (B2), the focal length direction for assuming video camera use accurate object perpendicular to plane where mobile terminal screen
It is long relative to screen is respectively parallel to using the screen upper left corner as coordinate origin X-axis and Y-axis that reason measuring method measures front camera
With the coordinate value of wide coordinate system；
Step (B3), on the basis of (B2), by coordinate origin move in front camera focal length institute on straight line, so as to
Obtain displacement of four vertex of mobile terminal screen relative to front camera.
Each mobile terminal front camera is calculated in the step (3) relative in the outer parameter of same extrinsic calibration object
The step of it is as follows：
The extrinsic calibration object of known dimensions is fixed on a smooth tablet by step (C1)；
Step (C2), the multiple mobile terminal screens of adjustment position reach the display demand of user, then adjust
(C1) position of middle plateform to demarcate object simultaneously and completely appears in the visual zone of all mobile terminal front cameras
Interior, then each acquisition for mobile terminal one opens the image currently captured；
Step (C3), obtained by the method for computer vision each mobile terminal capture image characteristic point, establish
Article coordinate and the correspondence of image coordinate are demarcated, each mobile terminal front camera is obtained relative to outer by solving equation
The Camera extrinsic number of coordinate system where portion's calibration object；
The step (4) determines that the step of each mobile terminal final display area is as follows：
Step (D1), the connection that each mobile terminal and same server are established by wireless network；
Step (D2), by the screen obtained in the camera obtained in step (2) outer parameter, (2) and front camera
Spatial relationship and the resolution ratio of screen are transferred to server；
Step (D3), server determine the final display of each mobile terminal after the data for obtaining each mobile terminal
Region and total display area.
The advantage of the present invention compared with prior art：
Compared to existing multi-screen splicing technology, the present invention is provided how mobile whole based on mobile terminal front camera
Hold screen splicing method that there is following advantage.First, The present invention gives a kind of automatic screen splicing method, in reality
Use during by obtaining the image containing specific calibration object and passing through the method for computer vision automatically determine each terminal
Respective display area；Secondly, the present invention has higher accuracy, since present invention uses the sides based on computer vision
Method is therefore compared to manually mode with more accuracy objectively；Finally, the present invention can adapt to different terminals in screen
Resolution ratio, physical size and some other differences are so as to realizing the splicing of special-shaped mobile terminal.
Description of the drawings
Fig. 1 is the overall process of the multi-mobile-terminal screen splicing method based on mobile terminal front camera of the present invention
Schematic diagram；
Fig. 2 is calibration object used in the external parameters calibration stage of the present invention.
Specific embodiment
The present invention is described in further detail with example below in conjunction with the accompanying drawings：
Implementation process of the present invention includes four key steps：The calibration of mobile terminal front camera intrinsic parameter determines screen
Curtain and front camera spatial relationship demarcate outer parameter of each front camera relative to same calibration object, determine each move
The final display area of dynamic terminal screen.Fig. 1 illustrates the overall process schematic diagram of the present invention.
Step 1：The calibration of mobile terminal front camera intrinsic parameter：
First stage：Obtain a certain number of images containing complete calibration object：
Intrinsic parameter calibration phase is mainly to establish the world coordinates and its coordinate in image coordinate system of calibration object characteristic point
Correspondence.First, one piece will be fixed to higher smooth for the gridiron pattern of the known grid size of calibration calibration object
On the tablet of degree, the flatness of tablet is directly related to the accuracy of final calibration result；Secondly, turning-off mobile terminal system is needed
The supported automatic focusing function of system, then obtaining the image containing complete calibration object from different angles, several are (each mobile
Terminal at least obtains 5).The purpose in the stage is that provide clarity for subsequent calibration process sufficiently high and with larger difference
Image several, to improve the robustness of the accuracy of intrinsic parameter calibration result and calibration process.
Second stage：Establish the equation group containing intrinsic parameter：
First, the every image first stage obtained carries out binaryzation, and therefrom detects tessellated characteristic point；Its
It is secondary, world coordinates and figure are established with the image coordinate of intersection point detected with reference to the real space coordinate of checkerboard pattern characteristic point
As the correspondence of coordinate, the equation group that the inside and outside parameter containing camera is obtained in the imaging equation of front camera is carried it into.
Phase III：Solve system of linear equations：
Step 2：Determine front camera and screen space relationship：
First stage：Measure the length and width of mobile terminal screen：
Use the long h and width w for measuring each mobile terminal screen respectively with high-precision vernier caliper, the result of measurement
It is accurate to millimeter.
Second stage：Measure the position relationship of camera and screen：
First using the upper left corner of screen as coordinate origin, the broadside of the parallel screen of X-axis, the long side that Y-axis is parallel to screen is built
Vertical rectangular coordinate system in space.Vernier caliper is aided with the tools such as right angled triangle used by using the first stage, before measuring respectively
Put distance dx and dy of the center of camera to X-axis and to Y-axis, then coordinate of the camera in this coordinate system for (dx,
dy).For the ease of measure, in this stage in method assumes that camera focal length where straight line perpendicular to mobile terminal screen
Curtain.
Phase III：Calculate displacement of the screen vertex relative to front camera：
The vertex translations for the rectangular coordinate system that second stage is established on straight line, are established to front camera focal length institute
New coordinate system C ', then mutually under this coordinate system four vertex of mobile terminal screen coordinate be respectively (- dx ,-dy),
(w-dx ,-dy), (- dx ,-dy-h) and (w-dx ,-dy-h).
Step 3：Calculate outer parameter of each camera relative to same extrinsic calibration object：
First stage：Make extrinsic calibration tablet：
There is central symmetry and axisymmetric attribute in view of gridiron pattern calibration object before, in practical calibration process
Ambiguity is easily caused, therefore this stage this method has used calibration object as shown in Figure 2.It is incited somebody to action after determining the size of calibration object
It is printed on an A4 paper and fixed on one piece of smooth tablet.
Second stage：Capture is containing reference object image：
Multiple mobile terminals are put according to the display demand of user in approximately the same plane first, then mobile upper single order
The tablet that section makes causes the front camera of each mobile terminal that can capture complete calibration object, and then fixed calibration is flat
The camera function that each camera is opened in the position of plate obtains the image containing complete calibration object.
Phase III：Calculate the outer parameter matrix of camera：
First, the intrinsic parameter of each front camera obtained before is read from storage；Then, pass through computer vision
Method extraction image acceptance of the bid earnest characteristic point, then identify the direction of image acceptance of the bid earnest, it is special finally to establish calibration object
Levy the correspondence between the world coordinates of point and image coordinate；Finally, intrinsic parameter and correspondence is brought into imaging equation to acquire
The outer parameter matrix Ai of each camera.
Step 4：Determine the final display area of mobile terminal：
First stage：The connection of each mobile terminal and same server is established by wireless network：
The server each mobile terminal being connected to for each terminal data to be uniformly processed is established by wireless network,
Server in the present invention is used to that data to be uniformly processed and result is sent to corresponding terminal.
Second stage：Data transmission；
For each mobile terminal, this stage needs the data transmitted to include four outer parameter matrix Ai, screen vertex
Relative to the position of front camera and the resolution information of screen.
Phase III：Determine the respective display area of each terminal and total display area：
Current invention assumes that all screens be in it is same it is unilateral on, based in the hope of relative to same extraneous object of demarcating
Outer parameter matrix, passes through AiAj-1Can be the seat under i-th of camera coordinates system by the coordinate transformation under j-th of camera coordinates system
Mark.Based on this, the selected mobile terminal of the present invention is used as with reference to terminal, and all coordinate systems are converted to this camera sits
Coordinate under mark system.After four apex coordinates of screen represent same referential, acquired by way of seeking bounding box
Total display area finally determines the subregion that each mobile terminal screen is respectively shown.
Claims (1)
- A kind of 1. multiple terminals screen method for automatically split-jointing based on mobile terminal front camera, it is characterised in that this method step It is as follows：Step (1), the front camera for each mobile terminal carry out camera calibration, so that it is determined that each front camera Intrinsic parameter；Step (2) determines spatial relationship between mobile terminal screen and its front camera；Step (3), it is online during, with reference to front camera intrinsic parameter obtained by calibrating in (1), before measuring each mobile terminal Put Camera extrinsic number of the camera relative to same extrinsic calibration object；Step (4), the Camera extrinsic number with reference to acquisition in the spatial relationship and step (3) determined in step (2), are considering to move The final display area of each mobile terminal is determined under conditions of terminal different resolution；The step (2) determines that the step of screen is with front camera spatial relationship is specific as follows：Step (B1), the length and width that mobile terminal screen is measured with physical measuring methods；Step (B2), the focal length direction for assuming video camera use physical measuring methods perpendicular to plane where mobile terminal screen It measures front camera and is respectively parallel to screen length and wide coordinate relative to by coordinate origin X-axis and Y-axis of the screen upper left corner The coordinate value of system；Step (B3), on the basis of (B2), coordinate origin is moved in front camera focal length institute on straight line, so as to obtain Four vertex of mobile terminal screen relative to front camera displacement；The particular content of the calibration of front camera intrinsic parameter is as follows in the step (1)：Step (A1), print a known grid length of side checkerboard pattern be fixed on a smooth tablet, fixing camera Focal length shoot several images containing complete checkerboard pattern from different angles；Step (A2) detects tessellated characteristic point in image, establish checkerboard pattern characteristic point real space coordinate and Correspondence between its image coordinate brings the imaging equation of front camera into；Step (A3), the intrinsic parameter that every mobile terminal is acquired by solving the imaging equation obtained in (A2)；Step of each mobile terminal front camera relative to the outer parameter in same extrinsic calibration object is calculated in the step (3) It is rapid as follows：The extrinsic calibration object of known dimensions is fixed on smooth tablet by step (C1)；Step (C2), user are according to oneself display demand and before ensureing that extrinsic calibration object completely can appear in mobile terminal Multiple mobile terminals are put on same plane under the premise of putting camera, then the position of set-up procedure (C1) middle plateform makes Extrinsic calibration object is appeared in simultaneously and completely in the visual zones of all mobile terminal front cameras, it is then each mobile Terminal obtains an image currently captured；Step (C3), obtained by the method for computer vision each mobile terminal capture image characteristic point, establish calibration The correspondence of article coordinate and image coordinate solves and obtains each mobile terminal front camera relative to where extrinsic calibration object The Camera extrinsic number of coordinate system；The step (4) determines that the step of each mobile terminal final display area is as follows：Step (D1), the connection that each mobile terminal and same server are established by wireless network；Step (D2), by the screen obtained in the camera obtained in step (3) outer parameter, (2) and the space of front camera Relationship and the resolution ratio of screen are transferred to server；Step (D3), server determine the final display area of each mobile terminal after the data for obtaining each mobile terminal The coordinate on each four vertex of mobile terminal screen and total display area i.e. under unified coordinate system.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510527082.XA CN105094739B (en) | 2015-08-25 | 2015-08-25 | A kind of multiple terminals screen method for automatically split-jointing based on mobile terminal front camera |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510527082.XA CN105094739B (en) | 2015-08-25 | 2015-08-25 | A kind of multiple terminals screen method for automatically split-jointing based on mobile terminal front camera |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN105094739A CN105094739A (en) | 2015-11-25 |
| CN105094739B true CN105094739B (en) | 2018-06-12 |
Family
ID=54575291
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510527082.XA Expired - Fee Related CN105094739B (en) | 2015-08-25 | 2015-08-25 | A kind of multiple terminals screen method for automatically split-jointing based on mobile terminal front camera |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105094739B (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107783741A (en) * | 2016-08-25 | 2018-03-09 | 中兴通讯股份有限公司 | Control the method, device and mobile terminal of multiple mobile terminal screen tiled displays |
| CN108304148B (en) * | 2017-01-11 | 2020-10-16 | 南京中兴新软件有限责任公司 | Multi-screen splicing display method and device |
| CN108052299A (en) * | 2017-10-31 | 2018-05-18 | 西安万像电子科技有限公司 | terminal device, multi-screen splicing system and method |
| CN112583814A (en) * | 2020-12-07 | 2021-03-30 | 三星电子（中国）研发中心 | Multi-terminal equipment merging playing method |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101419705A (en) * | 2007-10-24 | 2009-04-29 | 深圳华为通信技术有限公司 | Video camera demarcating method and device |
| CN101674364A (en) * | 2009-09-28 | 2010-03-17 | 深圳华为通信技术有限公司 | Wireless screen splicing display method, mobile communication terminal and device |
| CN101710932A (en) * | 2009-12-21 | 2010-05-19 | 深圳华为通信技术有限公司 | Image stitching method and device |
| CN101866482A (en) * | 2010-06-21 | 2010-10-20 | 清华大学 | Panorama splicing method based on camera self-calibration technology, and device thereof |
| CN102855114A (en) * | 2012-08-20 | 2013-01-02 | 北京小米科技有限责任公司 | Multi-mobile-terminal screen splicing method, screen splicing server and mobile terminal |
| CN104731539A (en) * | 2013-12-24 | 2015-06-24 | 中兴通讯股份有限公司 | Multi-screen joint displaying method and device |
| CN104835143A (en) * | 2015-03-31 | 2015-08-12 | 中国航空无线电电子研究所 | Rapid projector system parameter calibration method |
-
2015
- 2015-08-25 CN CN201510527082.XA patent/CN105094739B/en not_active Expired - Fee Related
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101419705A (en) * | 2007-10-24 | 2009-04-29 | 深圳华为通信技术有限公司 | Video camera demarcating method and device |
| CN101674364A (en) * | 2009-09-28 | 2010-03-17 | 深圳华为通信技术有限公司 | Wireless screen splicing display method, mobile communication terminal and device |
| CN101710932A (en) * | 2009-12-21 | 2010-05-19 | 深圳华为通信技术有限公司 | Image stitching method and device |
| CN101866482A (en) * | 2010-06-21 | 2010-10-20 | 清华大学 | Panorama splicing method based on camera self-calibration technology, and device thereof |
| CN102855114A (en) * | 2012-08-20 | 2013-01-02 | 北京小米科技有限责任公司 | Multi-mobile-terminal screen splicing method, screen splicing server and mobile terminal |
| CN104731539A (en) * | 2013-12-24 | 2015-06-24 | 中兴通讯股份有限公司 | Multi-screen joint displaying method and device |
| CN104835143A (en) * | 2015-03-31 | 2015-08-12 | 中国航空无线电电子研究所 | Rapid projector system parameter calibration method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN105094739A (en) | 2015-11-25 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10984554B2 (en) | Monocular vision tracking method, apparatus and non-volatile computer-readable storage medium | |
| CN105094739B (en) | A kind of multiple terminals screen method for automatically split-jointing based on mobile terminal front camera | |
| CN106441138B (en) | The deformation monitoring method of view-based access control model measurement | |
| JP2020187358A (en) | Projection system, projection apparatus and calibrating method for displayed image thereof | |
| CN109218524B (en) | Mobile phone APP and method for generating house type graph through house measurement based on photographing and video recording | |
| CN105282513A (en) | Device and method for detecting operation state of ultra-high-voltage transformer in transformer substation based on 3D infrared panoramic image | |
| CN109922255A (en) | For generating the dual camera systems of real-time deep figure | |
| CN104567680A (en) | Measuring system for electric wire structure | |
| WO2020220832A1 (en) | Method and apparatus for achieving projection picture splicing, and projection system | |
| CN104833342B (en) | Mobile terminal and method of establishing stereoscopic model through multipoint distance measurement | |
| CN105812790A (en) | Evaluation method for perpendicularity of image sensor light-sensitive surface and optical axis and optical test card | |
| CN109859155A (en) | Image distortion detection method and system | |
| WO2012136002A1 (en) | Method,device,system,television and stereo glasses for adjusting stereo image | |
| CN108282607A (en) | Panoramic picture obtains equipment and its panoramic picture acquisition module | |
| CN104048603A (en) | System and method for measuring array laser reticle and area array CCD high-resolution large visual field | |
| EP3547024B1 (en) | Mass production method and system for panorama camera | |
| CN104864855A (en) | Single-camera omnidirectional stereoscopic vision sensor and design method thereof | |
| CN107534202B (en) | A kind of method and apparatus measuring antenna attitude | |
| JP2005322128A (en) | Calibration method for stereo three-dimensional measurement and three-dimensional position calculating method | |
| CN103278104B (en) | A kind of scaling board of the dual camera systems for DIC measurement and scaling method thereof | |
| CN107783741A (en) | Control the method, device and mobile terminal of multiple mobile terminal screen tiled displays | |
| CN104516170A (en) | Three-dimensional focusing method and system thereof | |
| CN113411564A (en) | Method, device, medium and system for measuring human eye tracking parameters | |
| CN204346368U (en) | A kind of measuring system of electric wire structure | |
| CN105078404B (en) | Fully automatic eye movement tracking distance measuring calibration instrument based on laser algorithm and use method of calibration instrument |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20180612 Termination date: 20210825 |
|
| CF01 | Termination of patent right due to non-payment of annual fee |