Disclosure of Invention
In view of the above, the present invention provides a calibration method and system for an MR hybrid photography camera,
in order to achieve the purpose, the invention adopts the following technical scheme:
on one hand, the invention discloses a calibration method of an MR hybrid photography camera, which is characterized by comprising the following steps:
s1, acquiring pixel coordinates in a plurality of pairs of physical cameras and coordinates in a corresponding virtual camera coordinate system;
s2, solving a matrix G according to the pixel coordinates in the multiple pairs of physical cameras and the coordinates in the corresponding virtual coordinate system;
s3, decomposing the matrix G to obtain a rotation matrix, a displacement vector and an internal reference matrix;
and S4, setting the pose and the internal reference of the virtual camera according to the rotation matrix, the displacement vector and the internal reference matrix.
Preferably, in S1, the pixel coordinates in the multiple pairs of physical cameras and the coordinates in the corresponding virtual camera coordinate system are obtained by using a positioning base station and two trackers.
Preferably, the number of the positioning base stations is determined by the tracked physical space size.
Preferably, in S1, the acquiring step includes:
s11, establishing a world coordinate system according to the positioning base station, wherein the first tracker serves as a point in a three-dimensional world, and the second tracker serves as a virtual camera and keeps a fixed position with the physical camera;
s12, obtaining a virtual camera coordinate system according to the world coordinate system;
s13, converting the coordinate of the tracker I in the world coordinate system into the coordinate in the virtual coordinate system;
s14, obtaining the pixel coordinates of the tracker I through a physical camera.
S15, moving the tracker I, and repeating the steps S11-S14 to obtain pixel coordinates in a plurality of pairs of physical cameras and coordinates in a corresponding virtual coordinate system.
Preferably, there are at least 6 pairs of pixel coordinates in the plurality of pairs of physical cameras and coordinates in the corresponding virtual camera coordinate system.
In another aspect, the present invention discloses a calibration system of an MR hybrid photographing camera for performing the calibration method of the MR hybrid photographing camera according to any one of claims 1 to 5, comprising:
the data acquisition module is used for acquiring pixel coordinates in a plurality of pairs of physical cameras and coordinates in a corresponding virtual camera coordinate system;
the data processing module is used for solving a rotation matrix, a displacement vector and an internal reference matrix according to the pixel coordinates in the physical cameras and the coordinates in the corresponding virtual camera coordinate system;
and the virtual camera parameter updating module is used for setting the pose and the internal parameters of the virtual camera according to the rotation matrix, the displacement vector and the internal parameter matrix.
Preferably, the data acquisition module comprises positioning base stations and two trackers, and the number of the positioning base stations is determined by the tracked physical space size.
Compared with the prior art, the calibration method and the calibration system disclosed by the invention have the advantages that the pose and the internal reference of the physical camera can be obtained, the virtual camera is arranged to have the pose and the internal reference consistent with those of the physical camera, so that the calibration efficiency and the calibration precision of the virtual camera and the physical camera are improved, the coordinate system of the virtual camera is coincided with the coordinate system of the physical camera, the image in the virtual camera is coincided with the image in the physical camera, the pictures shot by the two cameras are fully aligned, and the shot images are perfectly embedded together.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be obtained by a person skilled in the art without making any creative effort based on the embodiments in the present invention, belong to the protection scope of the present invention.
The embodiment of the invention discloses a calibration method and a calibration system of an MR hybrid photographic camera, and the calibration method and the calibration system can ensure that a coordinate system of a virtual camera is superposed with a coordinate system of a physical camera and an image in the virtual camera is superposed with an image in the physical camera, so that pictures shot by the two cameras are fully aligned, the shot images are perfectly embedded together, and a life-like effect is achieved.
In order to make the aforementioned objects, features and advantages of the present invention comprehensible, embodiments accompanied with figures are described in further detail below.
On one hand, the invention discloses a calibration method of an MR hybrid photography camera, which is characterized by comprising the following steps: (see FIG. 1)
S1, acquiring pixel coordinates in a plurality of pairs of physical cameras and coordinates in a corresponding virtual camera coordinate system;
in the invention, all indoor positioning systems meeting a certain precision can be used for acquiring coordinate data, in one embodiment, a cheaper and common VR positioning system, namely a Lighthouse-based indoor positioning system is used for calibrating the MR hybrid photography camera,
specifically, the equipment that indoor positioning system of Lighthouse needs includes: a positioning base station, a tracker I and a tracker II, in the embodiment of the invention, a tracker1 and a tracker2 are used as trackers,
the number of the positioning base stations is different from 2 to 4 according to the size of physical space to be tracked; the other two trackers are used for MR camera calibration, specifically, a tracker1 is used as a locator and can also be understood as a point in a three-dimensional world in the following algorithm, a tracker2 is used as a virtual camera and is fixed on a physical camera, and in the whole calibration process, the positions (rotation and displacement) of the two are always ensured to be fixed and unchanged, as shown in fig. 2;
then, a world coordinate system is established by using the positioning base station, and a virtual coordinate system of the virtual camera is established according to the world coordinate system, at this time, the virtual camera is actually an HTC tracker, the system provides the pose of each tracker connected to the base station, including the world coordinate and the world rotation,
further, the data acquisition flow is shown in FIG. 3,
opening an HTC five system, acquiring a position coordinate of a positioning tracker1 (which is regarded as a point in a three-dimensional world without rotation), namely a coordinate in a world coordinate system, and further automatically converting the tracker position serving as a positioner into a virtual camera coordinate system by the HTC five system to obtain the coordinate of the positioner in the virtual coordinate system;
simultaneously opening the physical camera, shooting a physical picture comprising the positioning tracker1, and obtaining the pixel coordinates of the physical tracker1 in the picture, namely the pixel coordinates of the positioner in the physical camera; the pixel coordinate is obtained through a two-dimensional pixel coordinate system of the physical camera, specifically, the coordinate system takes the upper left corner of a picture shot by the physical camera as the origin of coordinates, the right side is the positive direction of an x axis, and the right side is the positive direction of a y axis;
through the steps, the mapping relation from the virtual three-dimensional coordinates of a group of positioning tracker to the two-dimensional pixel coordinates can be obtained, then the positioning tracker is moved to change the position of the positioning tracker so as to enable the positioning tracker to be uniformly distributed in the picture, the steps are repeated to obtain enough point pairs (more than 6 pairs are ensured),
s2, solving a matrix G according to the pixel coordinates in the multiple pairs of physical cameras and the coordinates in the corresponding virtual coordinate system;
if the image of the virtual camera and the image of the physical camera are overlapped, it is required that the representation of a certain three-dimensional world point in the virtual camera coordinate system should be overlapped with the representation in the physical camera coordinate system after the pose transformation, that is, the corresponding relationship is as follows:
P I =GP M
wherein, P I =[x i ,y i ] T Defined as the coordinates of the pixels of the tracker under the physical camera,
P M =[x M ,y M ,z M ] T a representation of the coordinates of a tracker defined as a three-dimensional world in a virtual camera coordinate system;
solving a matrix G according to the corresponding relation and the point pair combination obtained in the step 1;
specifically, the image of the virtual camera and the image of the physical camera are overlapped, two steps are involved, wherein one step is the pose of a rigid body including displacement and rotation, and the other step is the setting of internal parameters of the virtual camera;
the displacement and rotation of the rigid body are formed by a rotation matrix R and a displacement vector T, and the pose of the rigid body obtained by the method can be expressed by the following formula:
for camera internal reference, an internal reference matrix of a camera is obtained according to a pinhole imaging principle, and a point is projected from a three-dimensional coordinate (under a camera coordinate system) to a pixel coordinate system, so that the following formula can be obtained:
wherein, the internal reference matrix can be further expressed as a homogeneous form:
from the above analysis, G = T can be obtained proj T pose ;
Further, P can be obtained I =T proj T pose P M Known as P M Is a representation of the coordinates of the tracker of the three-dimensional world under the virtual camera coordinate system; we assume the virtual camera coordinate system A, the world coordinate system of tracker P w The coordinates P of tracker in the virtual camera coordinate system according to mathematical definition M =A -1 P w ,
Thus, there is P I =T proj T pose A -1 P w ,
The purpose of the present application is to find a shift matrix T for shifting the virtual camera coordinate system to the physical camera coordinate system pose Therefore, also has P I =T proj (AT pose -1 ) -1 P w ,
In summary, according to T pose The physical camera coordinate system AT can be obtained pose -1 With the physical camera coordinate system, we can first get the tracker from the world coordinate system P w Convert to physical camera coordinate system and then apply projective transformation to get points on pixels!
Specifically, the matrix G is:
in one embodiment, one 3*4 matrix for G is:
[[247.6831,410.4999,0.00088,-2130.42580],
[1.4157488351899853e-05,142.9999,-247.6826,-8431.482037271988],
[5.8509535853e-07,0.999999,1.8749893022852892e-06,-6.999949]]
it can be seen that the matrix G is actually a 3x3 matrix formed by the camera internal reference matrix K and the rotation matrix R and a 3x1 vector formed by the camera internal reference matrix K and the displacement vector T, and is spliced to form a 3x4 matrix, i.e. G = [ KR | KT ],
that is, the matrix G contains 12 unknowns, according to the formula P I =GP M Each pair of points can result in two independent equations,
if the representation of matrix G is:
then the two independent equations obtained for each point pair are:
x i (g 31 x M +g 32 y M +g 33 z M +g 34 )=g 11 x M +g 12 y M +g 13 z M +g 14
y i (g 31 x M +g 32 y M +g 33 z M +g 34 )=g 21 x M +g 22 y M +g 23 z M +g 24
as described above, at least 6 sets of data point pairs are required, and all unknowns in the matrix G can be obtained.
S3, decomposing the matrix G to obtain a rotation matrix, a displacement vector and an internal reference matrix;
the first three columns of the matrix G are the result of the matrix multiplication KR, the form of K is an upper triangular matrix, the first three columns of the matrix G are subjected to RQ decomposition, the first obtained matrix is a 3x3 upper triangular matrix which accords with the configuration of the internal reference matrix K, the second matrix is a rotation matrix R naturally, the last column of G is the result of the matrix multiplication KT, and after the K is obtained, the displacement vector T is obtained by multiplying the inverse of the K by the last column of the G.
And S4, setting the pose and the internal parameters of the virtual camera according to the rotation matrix, the displacement vector and the internal parameter matrix.
The R and the T are applied to the pose of the virtual camera, and can be shifted to the pose of the physical camera, so that the aim of coinciding the coordinate system of the virtual camera with the coordinate system of the physical camera is fulfilled;
applying the internal reference matrix K to the virtual camera may make the image of the positioning tracker in the virtual camera completely coincide with the image in the physical camera.
On the other hand, the embodiment of the invention discloses a calibration system of an MR hybrid photography camera, which is the calibration method of the MR hybrid photography camera, and is characterized by comprising the following steps:
the data acquisition module is used for acquiring pixel coordinates in a plurality of pairs of physical cameras and coordinates in a corresponding virtual camera coordinate system; the data acquisition module comprises a positioning base station and two trackers, wherein the number of the positioning base station is determined by the size of tracked physical space.
The data processing module is used for solving a rotation matrix, a displacement vector and an internal reference matrix according to the pixel coordinates in the physical cameras and the coordinates in the corresponding virtual camera coordinate system;
and the virtual camera parameter updating module is used for setting the pose and the internal parameters of the virtual camera according to the rotation matrix, the displacement vector and the internal parameter matrix.
In the present specification, the embodiments are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other. The device disclosed by the embodiment corresponds to the method disclosed by the embodiment, so that the description is simple, and the relevant points can be referred to the method part for description.
The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.