CN112819898A - Thermal imager-RGB camera-IMU space combined calibration method, system and storage medium - Google Patents
Thermal imager-RGB camera-IMU space combined calibration method, system and storage medium Download PDFInfo
- Publication number
- CN112819898A CN112819898A CN202110076591.0A CN202110076591A CN112819898A CN 112819898 A CN112819898 A CN 112819898A CN 202110076591 A CN202110076591 A CN 202110076591A CN 112819898 A CN112819898 A CN 112819898A
- Authority
- CN
- China
- Prior art keywords
- thermal imager
- imu
- rgb
- rgb camera
- thermal
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
- 238000000034 method Methods 0.000 title claims abstract description 32
- 230000001133 acceleration Effects 0.000 claims abstract description 31
- 238000009826 distribution Methods 0.000 claims abstract description 13
- 239000011159 matrix material Substances 0.000 claims description 31
- 230000009466 transformation Effects 0.000 claims description 27
- 239000000243 solution Substances 0.000 claims description 15
- 238000005286 illumination Methods 0.000 claims description 12
- 238000006243 chemical reaction Methods 0.000 claims description 7
- 239000007983 Tris buffer Substances 0.000 claims description 6
- 238000013519 translation Methods 0.000 claims description 4
- 230000006870 function Effects 0.000 description 23
- 230000004927 fusion Effects 0.000 description 5
- 230000008447 perception Effects 0.000 description 4
- 238000005259 measurement Methods 0.000 description 3
- 230000000295 complement effect Effects 0.000 description 2
- 238000001514 detection method Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000004044 response Effects 0.000 description 2
- 238000006467 substitution reaction Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 230000036544 posture Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 238000001931 thermography Methods 0.000 description 1
- 230000001720 vestibular Effects 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10024—Color image
Landscapes
- Engineering & Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Studio Devices (AREA)
Abstract
The invention discloses a thermal imager-RGB camera-IMU space combined calibration method, a system and a storage medium, wherein the method comprises two steps of thermal imager image feature positioning and thermal imager-RGB camera-IMU combined calibration, the method comprises the steps of firstly adopting a mixed Gaussian distribution model to position the heat source center position of an LED array, then constructing a target function by using the reprojection error of a target point in a thermal imager image, the reprojection error of an RGB image corner point and the acceleration and angular velocity error of an IMU, and calculating the spatial relation among three sensor coordinate systems by minimizing the target function. The method is suitable for combined calibration of three sensors, namely a thermal imager, an RGB camera and an IMU, and has the advantages of stability and accuracy.
Description
Technical Field
The invention belongs to the technical field of image processing technology and multivariate information fusion, and particularly relates to a thermal imager-RGB camera-IMU space combined calibration method, system and storage medium.
Background
The multi-modal sensing platform integrated with the thermal imager, the RGB camera and the Inertial Measurement Unit (IMU) can enable the system to have the function of environmental sensing under complex and severe conditions, such as detection of vehicles and pedestrians driving at night; objects self-perceive under a violent motion background, etc.
The thermal imager is a device which converts an image displaying the temperature distribution of an object into a visual image by utilizing a thermal imaging technology, and has the advantages of good concealment, high safety, strong detection capability, long acting distance and the like. The RGB camera can collect rich scene information and is widely applied in the fields of positioning and navigation. The IMU is a device for measuring the three-axis angular velocity and the acceleration of an object, provides self inertial information similar to the human internal vestibular system, and has important application value in navigation. The thermal imager, the RGB camera and the IMU have complementary advantages in the aspects of image modality, perception accuracy and response speed, and the effective fusion of the thermal imager, the RGB camera and the IMU is widely concerned by robots and students in the field of automation. The effective and accurate calibration among the thermal imager, the RGB camera and the IMU is the premise of realizing multi-modal perception and fusion of the thermal imager, the RGB camera and the IMU, so the invention provides a thermal imager-RGB camera-IMU combined calibration method.
Disclosure of Invention
The invention mainly aims to overcome the defects in the prior art, and provides a thermal imager-RGB camera-IMU space combined calibration method, system and storage medium to realize effective fusion of three sensors.
In order to achieve the purpose, the invention adopts the following technical scheme:
the invention discloses a thermal imager-RGB camera-IMU space combined calibration method, which comprises the following steps:
positioning the image characteristics of the thermal imager, and searching the position of a target point of a calibration plate of the thermal imager;
the method comprises the following steps of thermal imager-RGB camera-IMU combined calibration, integrating thermal imager images, RGB images and IMU information in a unified framework, jointly estimating spatial relations among three coordinate systems of the RGB camera, the IMU and the thermal imager, including thermal imager and RGB camera spatial calibration, thermal imager and IMU spatial calibration and RGB camera and IMU spatial calibration, and then calculating parameters to be solved through a minimized objective function:
wherein, TrtRepresenting a transformation matrix of the thermal imager coordinate system relative to the RGB camera coordinate system; t isriA transformation matrix representing the inertial coordinate system relative to the RGB camera coordinate system; t istiRepresenting a transformation matrix, U, of the inertial frame relative to the thermal imager framerepro,rgb,thermalRepresenting the reprojection error, U, of the thermal imager image heat source center resolved by RGB camera motionrepro,thermal,rgbRepresenting the reprojection error, U, of the RGB image heat source center obtained by thermal imager motion solutionrepro,thermal,imuRepresenting a reprojection error of the thermal imager image heat source center obtained by IMU motion resolving; u shapeα,ω,rgbRepresenting the sum of the error terms of inertial acceleration and angular velocity of the IMU resulting from the RGB camera motion solution, Uα,ω,thermalRepresenting the sum of error terms of inertial acceleration and angular velocity of the IMU obtained by thermal imager motion calculation;
Trt,Tri,Ttithe following constraints are satisfied:
Tti·Tri -1·Trt=Ttr·Trt=I
wherein I is the identity matrix, (T)ti·Tri -1) Describing the conversion relationship from the RGB camera coordinate system to the thermal imager coordinate system, Tti·Tri -1·Trt=Ttr·TrtAdd to objective function T ═ Irt,Tri,TtiAnd obtaining a spatial relationship between the thermal imager, the RGB camera and the IMU coordinate system.
As a preferred technical scheme, in the thermal imager image feature positioning, a mixed gaussian distribution model is used to find the position of a target point of a thermal imager calibration plate, and an illumination area is modeled into a plurality of gaussian distributions, which are described as follows:
wherein A isiIs the amplitude of the ith heat source center; lambda [ alpha ]iAnd muiRespectively representing the position in the ith characteristic region and the centroid of the corresponding illumination region; sigmaiA covariance matrix representing the i-th region gaussian distribution; estimation of unknown parameter set by optimizing objective functioni,μi,Σi}:
Wherein I (x, y) represents a pixel value of (x, y); omegaiRepresenting a candidate set of pixels in an ith illumination region; even in the case of partial bulb area loss, the illumination area center can be accurately located by optimizing the objective function.
As a preferred technical scheme, in the thermal imager-RGB camera-IMU combined calibration, the objective function includes: the method comprises the following steps of re-projecting errors of target points of images of the thermal imager, re-projecting errors of corner points in RGB images, re-projecting errors of target points of images of the thermal imager, acceleration and angular velocity errors of IMUs, re-projecting errors of corner points in RGB images and acceleration and angular velocity errors of the IMUs.
As an optimal technical scheme, the thermal imager-RGB camera space calibration is adopted to describe the U part of the target function by adopting the reprojection errorrepro,rgb,thermalAnd Urepro,thermal,rgb,Urepro,rgb,thermalIs represented as:
wherein,andthe first pixel positions of the RGB image and the thermal imager image at the moment are respectively, t is more than or equal to 1 and less than or equal to K, and i is more than or equal to 1 and less than or equal to M; ctAnd CrThermal imager and RGB camera models, respectively; t istrIs the transformation relation of the RGB camera coordinate system relative to the thermal imager coordinate system; srtThe weight coefficient is the center re-projection error of the thermal imager image heat source;
Urepro,thermal,rgbis represented as:
wherein s istrIs the weight coefficient of the center reprojection error of the RGB image heat source.
As a preferred technical scheme, the thermal imager and the IMU space calibration specifically comprises:
the transformation from the IMU coordinate system to the world coordinate system is defined as:
wherein R iswi(t) and twi(t) represents rotation and translation from the inertial frame to the world frame at time t, respectively;
the error terms of inertial acceleration and angular velocity are:
wherein s isαIs a weight coefficient corresponding to the acceleration error; sωIs a weight coefficient corresponding to the angular velocity error;
the ith projected onto the thermal imager image at time tFor pixel locationRepresents; u shaperepro,thermal,imuGiven by:
wherein s istiThe weight coefficient of the ith heat source center on the thermal imager image at the time t; ctIs a thermal imager projection model.
As a preferred technical solution, the RGB camera and IMU space calibration specifically includes:
the error terms for acceleration and angular velocity are given by:
the ith projected onto RGB image at time tBy the pixel position ofRepresents, Urepro,rgb,imuGiven by:
wherein s istiThe weight coefficient is the center reprojection error of the ith heat source on the RGB image at the time t; crIs an RGB camera projection model.
The invention also discloses a thermal imager-RGB camera-IMU space combined calibration system, which is applied to the thermal imager-RGB camera-IMU space combined calibration method and comprises a thermal imager image feature positioning module and a thermal imager-RGB camera-IMU combined calibration module;
the thermal imager image feature positioning module is used for positioning the thermal imager image features and searching the target point position of the thermal imager calibration plate;
the thermal imager-RGB camera-IMU combined calibration module is used for thermal imager-RGB camera-IMU combined calibration, images of the thermal imager, RGB images and IMU information are integrated in a unified framework, spatial relations among three coordinate systems of the RGB camera, the IMU and the thermal imager are jointly estimated, the spatial relations include thermal imager and RGB camera spatial calibration, thermal imager and IMU spatial calibration and RGB camera and IMU spatial calibration, and then parameters to be solved are calculated through a minimized objective function:
wherein, TrtRepresenting a transformation matrix of the thermal imager coordinate system relative to the RGB camera coordinate system; t isriA transformation matrix representing the inertial coordinate system relative to the RGB camera coordinate system; t istiRepresenting a transformation matrix, U, of the inertial frame relative to the thermal imager framerepro,rgb,thermalRepresenting the reprojection error, U, of the thermal imager image heat source center resolved by RGB camera motionrepro,thermal,rgbRepresenting the reprojection error, U, of the RGB image heat source center obtained by thermal imager motion solutionrepro,thermal,imuRepresenting a reprojection error of the thermal imager image heat source center obtained by IMU motion resolving; u shapeα,ω,rgbRepresenting the sum of the error terms of inertial acceleration and angular velocity of the IMU resulting from the RGB camera motion solution, Uα,ω,thermalRepresenting the sum of error terms of inertial acceleration and angular velocity of the IMU obtained by thermal imager motion calculation;
Trt,Tri,Ttithe following constraints are satisfied:
Tti·Tri -1·Trt=Ttr·Trt=I
wherein I is the identity matrix, (T)ti·Tri -1) Describing the conversion relationship from the RGB camera coordinate system to the thermal imager coordinate system, Tti·Tri -1·Trt=Ttr·TrtAdd to objective function T ═ Irt,Tri,TtiAnd obtaining a spatial relationship between the thermal imager, the RGB camera and the IMU coordinate system.
The invention further provides a storage medium which stores a program, and when the program is executed by a processor, the thermal imager-RGB camera-IMU space joint calibration method is realized.
Compared with the prior art, the invention has the following advantages and beneficial effects:
the invention provides a thermal imager-RGB camera-IMU space combined calibration method. Firstly, the center position of the heat source of the LED array is positioned by adopting the mixed Gaussian distribution model, the technical problem that the thermal imager cannot perform characteristic positioning due to uneven distribution of the light source is solved, and the technical effect that the position estimation is still accurately performed under the condition that the LED light source is partially lost is achieved. Then, the invention adopts the technical scheme of establishing a target optimization function and optimizing the heat source center re-projection error extracted from the thermal imager image, the chessboard angular point re-projection error extracted from the RGB image and the re-projection error of the target point in the thermal imager image with the acceleration and angular speed errors, solves the technical problem of inconsistent joint calibration of the thermal imager, the RGB camera and the IMU, and achieves the technical effect of accurately calibrating the spatial position relation among the three sensor coordinate systems.
Drawings
FIG. 1 is a flow chart of a thermal imager-RGB camera-IMU space joint calibration method according to an embodiment of the invention;
FIG. 2 is a schematic structural diagram of a thermal imager-RGB camera-IMU spatial joint calibration system according to an embodiment of the invention;
fig. 3 is a schematic structural diagram of a storage medium according to an embodiment of the present invention.
Detailed Description
In order to make the technical solutions better understood by those skilled in the art, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. It is to be understood that the embodiments described are only a few embodiments of the present application and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
Examples
Thermal imagers, RGB cameras, and Inertial Measurement Units (IMU) have heterogeneous and complementary characteristics in terms of image modality, perception accuracy, and response speed, and the effective integration of these three elements is receiving increasing attention in the fields of robotics and computer vision. Effective and accurate calibration among the thermal imager, the RGB camera and the IMU is the premise for realizing multi-modal perception and fusion of the thermal imager, the RGB camera and the IMU. Therefore, the invention provides a spatial joint calibration method of the thermal imager, the RGB camera and the IMU, which is used for calibrating the relative positions and postures of the three sensors.
As shown in fig. 1, the thermal imager-RGB camera-IMU spatial joint calibration method of the present embodiment specifically includes the following steps:
s1, positioning the image characteristics of the thermal imager;
more specifically, the invention proposes to use a mixed gaussian distribution model to find the position of the target point of the calibration plate of the thermal imager, which models the illumination area as a plurality of gaussian distributions, mathematically described as:
wherein A isiIs the amplitude of the ith central region; lambda [ alpha ]iAnd muiRespectively representing the ith characteristic position and the centroid of the corresponding illumination area; sigmaiA covariance matrix representing the i-th region gaussian distribution. Estimation of unknown parameter set by optimizing objective functioni,μi,Σi}:
Wherein I (x, y) represents a pixel value of (x, y); omegaiRepresenting the candidate set of pixels in the ith illumination region, the proposed model is able to pinpoint the illumination region center by optimizing the objective function even in the event of bulb partial area loss.
Step 2: and (3) jointly calibrating the thermal imager, the RGB camera and the IMU, constructing a target function by using the reprojection error of a target point in an image of the thermal imager, the reprojection error of an angular point of the RGB image and the acceleration and angular speed error of the IMU, and calculating the spatial relationship among the three sensor coordinate systems by minimizing the target function.
Further, in this embodiment, the thermal imager image, the RGB image, and the inertial measurement information are integrated in a unified frame, the spatial relationship among the three coordinate systems of the RGB camera, the IMU, and the thermal imager is jointly estimated, and the parameters to be solved are calculated by minimizing the objective function:
wherein, TrtA transformation matrix (rotation and translation) representing the thermal imager coordinate system relative to the RGB camera coordinate system; t isriA transformation matrix representing the inertial coordinate system relative to the RGB camera coordinate system; t istiRepresenting a transformation matrix of the inertial coordinate system relative to the thermal imager coordinate system. T isrt,Tri,TtiThe following constraints are satisfied:
Tti·Tri -1·Trt=Ttr·Trt=I
wherein I is a 3 × 3 identity matrix; (T)ti·Tri -1) The conversion relationship from the RGB camera coordinate system to the thermal imager coordinate system is described. Will Tti·Tri -1·Trt=Ttr·TrtAdd to objective function T ═ Irt,Tri,TtiAnd obtaining the spatial relationship between the thermal imager-RGB camera-IMU coordinate system.
It can be understood that the objective function mainly includes three parts, which are respectively a reprojection error of a target point of the thermal imager image and a reprojection error of a corner point in the RGB image, a reprojection error of a target point of the thermal imager image and an acceleration and angular velocity error of the IMU, a reprojection error of a corner point in the RGB image and an acceleration and angular velocity error of the IMU, and they are respectively obtained through the following three steps:
s2.1, calibrating a thermal imager-RGB camera space;
the embodiment adopts the reprojection error to describe the objective function part Urepro,rgb,thermalAnd Urepro,thermal,rgb。Urepro,rgb,thermalRepresenting the reprojection error of the thermal imager image heat source center resolved by the RGB camera motion, which is expressed as:
wherein,andthe pixel positions of the RGB image and the thermal imager image at the ith (i is more than or equal to 1 and less than or equal to M) pixel position at the moment t (t is more than or equal to 1 and less than or equal to K) are respectively; ctAnd CrThermal imager and RGB camera models, respectively; t istrIs a coordinate system of RGB cameraA transformation relationship with respect to a thermal imager coordinate system; srtAnd the weight coefficient is the center re-projection error of the thermal imager image heat source.
Similarly, Urepro,thermal,rgbRepresenting the reprojection error of the RGB image heat source center resolved by the thermal imager motion, which is expressed as:
wherein s istrIs the weight coefficient of the center reprojection error of the RGB image heat source.
S2.2, calibrating a thermal imager and an IMU space;
the transformation from the IMU coordinate system to the world coordinate system is defined as:
wherein R iswi(t) and twi(t) represents rotation and translation, respectively, from the inertial frame to the world frame at time t.
The error terms of inertial acceleration and angular velocity are:
wherein s isαIs a weight coefficient corresponding to the acceleration error; sωIs a weight coefficient corresponding to the angular velocity error.
The ith projected onto the thermal imager image at time tFor pixel locationRepresents, Urepro,thermal,imuRepresenting the reprojection error of the thermal imager image heat source center resolved by the IMU motion, given by:
wherein s istiThe weight coefficient of the ith heat source center on the thermal imager image at the time t; ctIs that
And (5) projecting the model by the thermal imager.
S2.3, calibrating the RGB camera and the IMU space;
similar to the mutual calibration of the thermal imager coordinate system and the inertial coordinate system, the error terms of acceleration and angular velocity are given by:
the ith projected onto RGB image at time tBy the pixel position ofAnd (4) showing. U shaperepro,rgb,imuRepresenting the reprojection error of the RGB image heat source center resolved by IMU motion, given by:
wherein s istiThe weight coefficient is the center reprojection error of the ith heat source on the RGB image at the time t; crIs an RGB camera projection model.
As shown in fig. 2, in another embodiment, a thermal imager-RGB camera-IMU spatial joint calibration system is provided, which includes a thermal imager image feature positioning module and a thermal imager-RGB camera-IMU joint calibration module;
the thermal imager image feature positioning module is used for positioning the thermal imager image features and searching the target point position of the thermal imager calibration plate;
the thermal imager-RGB camera-IMU combined calibration module is used for thermal imager-RGB camera-IMU combined calibration, images of the thermal imager, RGB images and IMU information are integrated in a unified framework, spatial relations among three coordinate systems of the RGB camera, the IMU and the thermal imager are jointly estimated, the spatial relations include thermal imager and RGB camera spatial calibration, thermal imager and IMU spatial calibration and RGB camera and IMU spatial calibration, and then parameters to be solved are calculated through a minimized objective function:
wherein, TrtRepresenting a transformation matrix of the thermal imager coordinate system relative to the RGB camera coordinate system; t isriA transformation matrix representing the inertial coordinate system relative to the RGB camera coordinate system; t istiRepresenting a transformation matrix, U, of the inertial frame relative to the thermal imager framerepro,rgb,thermalRepresenting the reprojection error, U, of the thermal imager image heat source center resolved by RGB camera motionrepro,thermal,rgbRepresenting the reprojection error, U, of the RGB image heat source center obtained by thermal imager motion solutionrepro,thermal,imuRepresenting a reprojection error of the thermal imager image heat source center obtained by IMU motion resolving; u shapeα,ω,rgbRepresenting phases from RGBSum of error terms of inertial acceleration and angular velocity of the IMU obtained by machine motion solution, Uα,ω,thermalAnd representing the sum of error terms of inertial acceleration and angular speed of the IMU obtained by thermal imager motion calculation.
Trt,Tri,TtiThe following constraints are satisfied:
Tti·Tri -1·Trt=Ttr·Trt=I
wherein I is the identity matrix, (T)ti·Tri -1) Describing the conversion relationship from the RGB camera coordinate system to the thermal imager coordinate system, Tti·Tri -1·Trt=Ttr·TrtAdd to objective function T ═ Irt,Tri,TtiAnd obtaining a spatial relationship between the thermal imager, the RGB camera and the IMU coordinate system.
It should be noted that the system provided in the above embodiment is only illustrated by the division of the functional modules, and in practical applications, the function allocation may be completed by different functional modules according to needs, that is, the internal structure is divided into different functional modules to complete all or part of the functions described above.
As shown in fig. 3, in another embodiment of the present application, a storage medium is further provided, where the storage medium stores a program, and when the program is executed by a processor, the method for implementing a thermal imager-RGB camera-IMU spatial joint calibration is specifically:
positioning the image characteristics of the thermal imager, and searching the position of a target point of a calibration plate of the thermal imager;
the method comprises the following steps of thermal imager-RGB camera-IMU combined calibration, integrating thermal imager images, RGB images and IMU information in a unified framework, jointly estimating spatial relations among three coordinate systems of the RGB camera, the IMU and the thermal imager, including thermal imager and RGB camera spatial calibration, thermal imager and IMU spatial calibration and RGB camera and IMU spatial calibration, and then calculating parameters to be solved through a minimized objective function:
wherein, TrtRepresenting a transformation matrix of the thermal imager coordinate system relative to the RGB camera coordinate system; t isriA transformation matrix representing the inertial coordinate system relative to the RGB camera coordinate system; t istiRepresenting a transformation matrix, U, of the inertial frame relative to the thermal imager framerepro,rgb,thermalRepresenting the reprojection error, U, of the thermal imager image heat source center resolved by RGB camera motionrepro,thermal,rgbRepresenting the reprojection error, U, of the RGB image heat source center obtained by thermal imager motion solutionrepro,thermal,imuRepresenting a reprojection error of the thermal imager image heat source center obtained by IMU motion resolving; u shapeα,ω,rgbRepresenting the sum of the error terms of inertial acceleration and angular velocity of the IMU resulting from the RGB camera motion solution, Uα,ω,thermalAnd representing the sum of error terms of inertial acceleration and angular speed of the IMU obtained by thermal imager motion calculation.
Trt,Tri,TtiThe following constraints are satisfied:
Tti·Tri -1·Trt=Ttr·Trt=I
wherein I is the identity matrix, (T)ti·Tri -1) Describing the conversion relationship from the RGB camera coordinate system to the thermal imager coordinate system, Tti·Tri -1·Trt=Ttr·TrtAdd to objective function T ═ Irt,Tri,TtiAnd obtaining a spatial relationship between the thermal imager, the RGB camera and the IMU coordinate system.
It should be understood that portions of the present application may be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, the various steps or methods may be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any one or combination of the following techniques, which are known in the art, may be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application specific integrated circuit having an appropriate combinational logic gate circuit, a Programmable Gate Array (PGA), a Field Programmable Gate Array (FPGA), or the like.
The above embodiments are preferred embodiments of the present invention, but the present invention is not limited to the above embodiments, and any other changes, modifications, substitutions, combinations, and simplifications which do not depart from the spirit and principle of the present invention should be construed as equivalents thereof, and all such changes, modifications, substitutions, combinations, and simplifications are intended to be included in the scope of the present invention.
Claims (8)
1. The thermal imager-RGB camera-IMU space combined calibration method is characterized by comprising the following steps of:
positioning the image characteristics of the thermal imager, and searching the position of a target point of a calibration plate of the thermal imager;
the method comprises the following steps of thermal imager-RGB camera-IMU combined calibration, integrating thermal imager images, RGB images and IMU information in a unified framework, jointly estimating spatial relations among three coordinate systems of the RGB camera, the IMU and the thermal imager, including thermal imager and RGB camera spatial calibration, thermal imager and IMU spatial calibration and RGB camera and IMU spatial calibration, and then calculating parameters to be solved through a minimized objective function:
wherein, TrtRepresenting a transformation matrix of the thermal imager coordinate system relative to the RGB camera coordinate system; t isriA transformation matrix representing the inertial coordinate system relative to the RGB camera coordinate system; t istiRepresenting a transformation matrix, U, of the inertial frame relative to the thermal imager framerepro,rgb,thermalRepresenting the reprojection error, U, of the thermal imager image heat source center resolved by RGB camera motionrepro,thermal,rgbRepresenting the reprojection error, U, of the RGB image heat source center obtained by thermal imager motion solutionrepro,thermal,imuRepresentation resolution by IMU motionObtaining a reprojection error of the thermal imager image heat source center; u shapeα,ω,rgbRepresenting the sum of the error terms of inertial acceleration and angular velocity of the IMU resulting from the RGB camera motion solution, Uα,ω,thermalRepresenting the sum of error terms of inertial acceleration and angular velocity of the IMU obtained by thermal imager motion calculation;
Trt,Tri,Ttithe following constraints are satisfied:
Tti·Tri -1·Trt=Ttr·Trt=I
wherein I is the identity matrix, (T)ti·Tri -1) Describing the conversion relationship from the RGB camera coordinate system to the thermal imager coordinate system, Tti·Tri -1·Trt=Ttr·TrtAdd to objective function T ═ Irt,Tri,TtiAnd obtaining a spatial relationship between the thermal imager, the RGB camera and the IMU coordinate system.
2. The thermal imager-RGB camera-IMU spatial joint calibration method of claim 1, wherein the thermal imager image feature location uses a mixed gaussian distribution model to find the target point position of the thermal imager calibration plate, modeling the illumination area as a plurality of gaussian distributions, described as follows:
wherein A isiIs the amplitude of the ith heat source center; lambda [ alpha ]iAnd muiRespectively representing the position in the ith characteristic region and the centroid of the corresponding illumination region; sigmaiA covariance matrix representing the i-th region gaussian distribution; estimation of unknown parameter set by optimizing objective functioni,μi,Σi}:
Wherein I (x, y) represents a pixel value of (x, y); omegaiRepresenting a candidate set of pixels in an ith illumination region; even in the case of partial bulb area loss, the illumination area center can be accurately located by optimizing the objective function.
3. The thermal imager-RGB camera-IMU spatial joint calibration method of claim 1, wherein in the thermal imager-RGB camera-IMU spatial joint calibration, the objective function includes: the method comprises the following steps of re-projecting errors of target points of images of the thermal imager, re-projecting errors of corner points in RGB images, re-projecting errors of target points of images of the thermal imager, acceleration and angular velocity errors of IMUs, re-projecting errors of corner points in RGB images and acceleration and angular velocity errors of the IMUs.
4. The thermal imager-RGB camera-IMU space joint calibration method as claimed in claim 1, wherein thermal imager-RGB camera space calibration adopts re-projection error to describe the U of the objective function partrepro,rgb,thermalAnd Urepro,thermal,rgb,Urepro,rgb,thermalIs represented as:
wherein,andthe first pixel positions of the RGB image and the thermal imager image at the moment are respectively, t is more than or equal to 1 and less than or equal to K, and i is more than or equal to 1 and less than or equal to M; ctAnd CrThermal imager and RGB camera models, respectively; t istrIs the transformation relation of the RGB camera coordinate system relative to the thermal imager coordinate system; srtThe weight coefficient is the center re-projection error of the thermal imager image heat source;
Urepro,thermal,rgbquilt watchShown as follows:
wherein s istrIs the weight coefficient of the center reprojection error of the RGB image heat source.
5. The thermal imager-RGB camera-IMU spatial joint calibration method according to claim 1, wherein the thermal imager and IMU spatial calibration specifically is:
the transformation from the IMU coordinate system to the world coordinate system is defined as:
wherein R iswi(t) and twi(t) represents rotation and translation from the inertial frame to the world frame at time t, respectively;
the error terms of inertial acceleration and angular velocity are:
wherein s isαIs a weight coefficient corresponding to the acceleration error; sωIs a weight coefficient corresponding to the angular velocity error;
the ith projected onto the thermal imager image at time tFor pixel locationRepresents; u shaperepro,thermal,imuGiven by:
wherein s istiThe weight coefficient of the ith heat source center on the thermal imager image at the time t; ctIs a thermal imager projection model.
6. The thermal imager-RGB camera-IMU spatial joint calibration method according to claim 1, wherein the RGB camera and IMU spatial calibration specifically is:
the error terms for acceleration and angular velocity are given by:
the ith projected onto RGB image at time tBy the pixel position ofRepresents, Urepro,rgb,imuGiven by:
wherein s istiThe weight coefficient is the center reprojection error of the ith heat source on the RGB image at the time t; crIs an RGB camera projection model.
7. The thermal imager-RGB camera-IMU space combined calibration system is characterized by being applied to the thermal imager-RGB camera-IMU space combined calibration method of any one of claims 1-6, and comprising a thermal imager image feature positioning module and a thermal imager-RGB camera-IMU combined calibration module;
the thermal imager image feature positioning module is used for positioning the thermal imager image features and searching the target point position of the thermal imager calibration plate;
the thermal imager-RGB camera-IMU combined calibration module is used for thermal imager-RGB camera-IMU combined calibration, images of the thermal imager, RGB images and IMU information are integrated in a unified framework, spatial relations among three coordinate systems of the RGB camera, the IMU and the thermal imager are jointly estimated, the spatial relations include thermal imager and RGB camera spatial calibration, thermal imager and IMU spatial calibration and RGB camera and IMU spatial calibration, and then parameters to be solved are calculated through a minimized objective function:
wherein, TrtRepresenting a transformation matrix of the thermal imager coordinate system relative to the RGB camera coordinate system; t isriA transformation matrix representing the inertial coordinate system relative to the RGB camera coordinate system; t istiRepresenting a transformation matrix, U, of the inertial frame relative to the thermal imager framerepro,rgb,thermalRepresenting the reprojection error, U, of the thermal imager image heat source center resolved by RGB camera motionrepro,thermal,rgbRepresenting the reprojection error, U, of the RGB image heat source center obtained by thermal imager motion solutionrepro,thermal,imuRepresentation is solved by IMU motionCalculating the obtained re-projection error of the thermal imager image heat source center; u shapeα,ω,rgbRepresenting the sum of the error terms of inertial acceleration and angular velocity of the IMU resulting from the RGB camera motion solution, Uα,ω,thermalRepresenting the sum of error terms of inertial acceleration and angular velocity of the IMU obtained by thermal imager motion calculation;
Trt,Tri,Ttithe following constraints are satisfied:
Tti·Tri -1·Trt=Ttr·Trt=I
wherein I is the identity matrix, (T)ti·Tri -1) Describing the conversion relationship from the RGB camera coordinate system to the thermal imager coordinate system, Tti·Tri -1·Trt=Ttr·TrtAdd to objective function T ═ Irt,Tri,TtiAnd obtaining a spatial relationship between the thermal imager, the RGB camera and the IMU coordinate system.
8. A storage medium storing a program, wherein the program, when executed by a processor, implements the thermal imager-RGB camera-IMU spatial joint calibration method of any one of claims 1-6.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202110076591.0A CN112819898A (en) | 2021-01-20 | 2021-01-20 | Thermal imager-RGB camera-IMU space combined calibration method, system and storage medium |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202110076591.0A CN112819898A (en) | 2021-01-20 | 2021-01-20 | Thermal imager-RGB camera-IMU space combined calibration method, system and storage medium |
Publications (1)
Publication Number | Publication Date |
---|---|
CN112819898A true CN112819898A (en) | 2021-05-18 |
Family
ID=75858801
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202110076591.0A Withdrawn CN112819898A (en) | 2021-01-20 | 2021-01-20 | Thermal imager-RGB camera-IMU space combined calibration method, system and storage medium |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN112819898A (en) |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN114004901A (en) * | 2022-01-04 | 2022-02-01 | 南昌虚拟现实研究院股份有限公司 | Multi-camera calibration method and device, terminal equipment and readable storage medium |
CN116736227A (en) * | 2023-08-15 | 2023-09-12 | 无锡聚诚智能科技有限公司 | Method for jointly calibrating sound source position by microphone array and camera |
Citations (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN108090921A (en) * | 2016-11-23 | 2018-05-29 | 中国科学院沈阳自动化研究所 | Monocular vision and the adaptive indoor orientation method of IMU fusions |
-
2021
- 2021-01-20 CN CN202110076591.0A patent/CN112819898A/en not_active Withdrawn
Patent Citations (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN108090921A (en) * | 2016-11-23 | 2018-05-29 | 中国科学院沈阳自动化研究所 | Monocular vision and the adaptive indoor orientation method of IMU fusions |
Non-Patent Citations (1)
Title |
---|
王鑫等: "热像仪-RGB相机-IMU传感器的空间联合标定方法", 《仪器仪表学报2020》 * |
Cited By (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN114004901A (en) * | 2022-01-04 | 2022-02-01 | 南昌虚拟现实研究院股份有限公司 | Multi-camera calibration method and device, terminal equipment and readable storage medium |
CN114004901B (en) * | 2022-01-04 | 2022-03-18 | 南昌虚拟现实研究院股份有限公司 | Multi-camera calibration method and device, terminal equipment and readable storage medium |
CN116736227A (en) * | 2023-08-15 | 2023-09-12 | 无锡聚诚智能科技有限公司 | Method for jointly calibrating sound source position by microphone array and camera |
CN116736227B (en) * | 2023-08-15 | 2023-10-27 | 无锡聚诚智能科技有限公司 | Method for jointly calibrating sound source position by microphone array and camera |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN110009681B (en) | IMU (inertial measurement unit) assistance-based monocular vision odometer pose processing method | |
JP7054803B2 (en) | Camera parameter set calculation device, camera parameter set calculation method and program | |
CN111156998B (en) | Mobile robot positioning method based on RGB-D camera and IMU information fusion | |
JP6830140B2 (en) | Motion vector field determination method, motion vector field determination device, equipment, computer readable storage medium and vehicle | |
US20190371003A1 (en) | Monocular vision tracking method, apparatus and non-volatile computer-readable storage medium | |
CN111210478B (en) | Common-view-free multi-camera system external parameter calibration method, medium and system | |
US20180075618A1 (en) | Measurement system and method for measuring multi-dimensions | |
WO2020140431A1 (en) | Camera pose determination method and apparatus, electronic device and storage medium | |
CN111476106B (en) | Monocular camera-based straight road relative gradient real-time prediction method, system and device | |
US20180075614A1 (en) | Method of Depth Estimation Using a Camera and Inertial Sensor | |
KR20220025028A (en) | Method and device for building beacon map based on visual beacon | |
CN112819898A (en) | Thermal imager-RGB camera-IMU space combined calibration method, system and storage medium | |
CN110675455B (en) | Natural scene-based self-calibration method and system for vehicle body looking-around camera | |
JP2018190402A (en) | Camera parameter set calculation device, camera parameter set calculation method, and program | |
CN112734765A (en) | Mobile robot positioning method, system and medium based on example segmentation and multi-sensor fusion | |
CN113888639B (en) | Visual odometer positioning method and system based on event camera and depth camera | |
CN116205947A (en) | Binocular-inertial fusion pose estimation method based on camera motion state, electronic equipment and storage medium | |
CN111899276A (en) | SLAM method and system based on binocular event camera | |
CN112631431B (en) | Method, device and equipment for determining pose of AR (augmented reality) glasses and storage medium | |
CN111798521A (en) | Calibration method, calibration device, storage medium and electronic equipment | |
CN114897988B (en) | Multi-camera positioning method, device and equipment in hinge type vehicle | |
CN114663463A (en) | Method, system, device, electronic device and storage medium for measuring joint mobility | |
Qian et al. | Survey on fish-eye cameras and their applications in intelligent vehicles | |
CN113701750A (en) | Fusion positioning system of underground multi-sensor | |
CN116442707B (en) | System and method for estimating vertical and pitching motion information of vehicle body based on binocular vision |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
WW01 | Invention patent application withdrawn after publication | ||
WW01 | Invention patent application withdrawn after publication |
Application publication date: 20210518 |