CN114812610A - Parameter calibration method and device for visual inertial system, electronic equipment and medium - Google Patents
Parameter calibration method and device for visual inertial system, electronic equipment and medium Download PDFInfo
- Publication number
- CN114812610A CN114812610A CN202210414649.2A CN202210414649A CN114812610A CN 114812610 A CN114812610 A CN 114812610A CN 202210414649 A CN202210414649 A CN 202210414649A CN 114812610 A CN114812610 A CN 114812610A
- Authority
- CN
- China
- Prior art keywords
- sensor
- visual
- inertial
- parameter
- parameters
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 230000000007 visual effect Effects 0.000 title claims abstract description 529
- 238000000034 method Methods 0.000 title claims abstract description 88
- 238000013519 translation Methods 0.000 claims description 98
- 238000006243 chemical reaction Methods 0.000 claims description 27
- 230000003287 optical effect Effects 0.000 claims description 24
- 238000003860 storage Methods 0.000 claims description 20
- 230000001133 acceleration Effects 0.000 claims description 16
- 238000004590 computer program Methods 0.000 claims description 16
- 230000000977 initiatory effect Effects 0.000 claims description 6
- 230000003068 static effect Effects 0.000 claims description 5
- 238000012545 processing Methods 0.000 description 22
- 239000011159 matrix material Substances 0.000 description 20
- 238000010586 diagram Methods 0.000 description 16
- 230000000694 effects Effects 0.000 description 14
- 238000005516 engineering process Methods 0.000 description 14
- 238000004891 communication Methods 0.000 description 10
- 230000006870 function Effects 0.000 description 9
- 238000013507 mapping Methods 0.000 description 8
- 239000006185 dispersion Substances 0.000 description 6
- 230000008569 process Effects 0.000 description 6
- 230000006872 improvement Effects 0.000 description 5
- 238000010295 mobile communication Methods 0.000 description 5
- 230000003190 augmentative effect Effects 0.000 description 4
- 230000005540 biological transmission Effects 0.000 description 4
- 230000005236 sound signal Effects 0.000 description 4
- 230000009286 beneficial effect Effects 0.000 description 3
- 230000008859 change Effects 0.000 description 3
- 238000009795 derivation Methods 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 239000000835 fiber Substances 0.000 description 2
- 238000003384 imaging method Methods 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 230000001902 propagating effect Effects 0.000 description 2
- 239000004065 semiconductor Substances 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- 230000009466 transformation Effects 0.000 description 2
- RYGMFSIKBFXOCR-UHFFFAOYSA-N Copper Chemical compound [Cu] RYGMFSIKBFXOCR-UHFFFAOYSA-N 0.000 description 1
- 230000009471 action Effects 0.000 description 1
- 230000006978 adaptation Effects 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 230000000295 complement effect Effects 0.000 description 1
- 150000001875 compounds Chemical class 0.000 description 1
- 229910052802 copper Inorganic materials 0.000 description 1
- 239000010949 copper Substances 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000006073 displacement reaction Methods 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 230000004807 localization Effects 0.000 description 1
- 230000007774 longterm Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 229910044991 metal oxide Inorganic materials 0.000 description 1
- 150000004706 metal oxides Chemical class 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 238000005295 random walk Methods 0.000 description 1
- 230000002441 reversible effect Effects 0.000 description 1
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
- G01C25/00—Manufacturing, calibrating, cleaning, or repairing instruments or devices referred to in the other groups of this subclass
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
- G01C21/00—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00
- G01C21/10—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 by using measurements of speed or acceleration
- G01C21/12—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 by using measurements of speed or acceleration executed aboard the object being navigated; Dead reckoning
- G01C21/16—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 by using measurements of speed or acceleration executed aboard the object being navigated; Dead reckoning by integrating acceleration or speed, i.e. inertial navigation
- G01C21/165—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 by using measurements of speed or acceleration executed aboard the object being navigated; Dead reckoning by integrating acceleration or speed, i.e. inertial navigation combined with non-inertial navigation instruments
Landscapes
- Engineering & Computer Science (AREA)
- Radar, Positioning & Navigation (AREA)
- Remote Sensing (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Automation & Control Theory (AREA)
- Manufacturing & Machinery (AREA)
- Gyroscopes (AREA)
- Navigation (AREA)
Abstract
The disclosure relates to a parameter calibration method and device of a visual inertial system, electronic equipment and a medium. The method comprises the following steps: acquiring state data of the visual inertial system at the previous moment; estimating the state data of the visual inertial system at the current moment according to the state data of the visual inertial system at the previous moment, the coordinates of the feature points in the image acquired by the visual sensor at the current moment and the output data of the inertial sensor at the current moment; and calibrating the parameters to be calibrated of the visual inertial system according to the state data of the visual inertial system at the current moment.
Description
The application is a divisional application of Chinese patent application with the title of 'parameter calibration method and device of visual inertial system, electronic device and medium', which is filed by China patent office on 16/11/2020 and has the application number of 202011281318.3.
Technical Field
The present disclosure relates to the field of computer vision technologies, and in particular, to a method and an apparatus for calibrating parameters of a visual inertial system, an electronic device, and a medium.
Background
With the development of computer vision and image processing technology, sensing the surrounding environment for positioning through machine vision is widely applied. The visual sensor and the inertial sensor are fused to obtain higher-precision positioning information. A Visual-Inertial navigation And Mapping (VI-SLAM) system fused with a Visual sensor And an Inertial sensor is widely applied to the aspects of virtual reality, augmented reality, mixed reality, unmanned aerial vehicles, mobile robots, unmanned vehicles And the like. By fusing the data of the visual sensor and the inertial sensor, the visual inertial system can be positioned at high precision, and an accurate three-dimensional environment map can be constructed. The sensor parameters of the visual inertial system are calibrated, and the method has important significance for the visual inertial system.
Disclosure of Invention
The present disclosure provides a parameter calibration technical scheme of a visual inertial system.
According to an aspect of the present disclosure, there is provided a parameter calibration method of a visual inertial system, the visual inertial system including a visual sensor and an inertial sensor, the method including:
acquiring state data of the visual inertial system at the previous moment;
estimating the state data of the visual inertial system at the current moment according to the state data of the visual inertial system at the previous moment, the coordinates of the feature points in the image acquired by the visual sensor at the current moment and the output data of the inertial sensor at the current moment;
and calibrating the parameters to be calibrated of the visual inertial system according to the state data of the visual inertial system at the current moment.
The disclosed embodiment obtains the state data of the visual inertial system at the previous moment, and according to the state data of the visual inertial system at the previous moment, the coordinates of the feature points in the image acquired by the visual sensor at the current moment, and the output data of the inertial sensor at the current moment, estimating the state data of the visual inertial system at the current moment, and according to the state data of the visual inertial system at the current moment, the parameters to be calibrated of the visual inertial system are calibrated, so that the calibration process does not need to be assisted by external equipment such as a calibration reference object, a calibration plate, a turntable and the like, the parameters to be calibrated are calibrated according to the state of the visual inertial system, so that the self-calibration of the sensor parameters of the visual inertial system can be realized, and more accurate parameters can be obtained.
In one possible implementation form of the method,
the state data of the visual inertial system at the previous time comprises: the state data of the visual sensor at the previous moment, the state data of the inertial sensor at the previous moment and the numerical value of the parameter to be calibrated at the previous moment;
the state data of the visual inertial system at the current time includes: the state data of the visual sensor at the current moment, the state data of the inertial sensor at the current moment and the numerical value of the parameter to be calibrated at the current moment.
The sensor parameters of the visual inertial system are self-calibrated based on the state data of the visual inertial system in the implementation mode, and accurate parameters can be obtained.
In one possible implementation form of the method,
the status data of the vision sensor at the previous time comprises: a translation parameter of the vision sensor at the previous time and a rotation parameter of the vision sensor at the previous time;
the status data of the vision sensor at the current time includes: a translation parameter of the vision sensor at the current time and a rotation parameter of the vision sensor at the current time.
The sensor parameters of the visual inertial system are self-calibrated based on the state data of the visual sensor in the implementation mode, and the calibration effect is improved.
In one possible implementation of the method according to the invention,
the state data of the inertial sensor at the previous time comprises: the velocity of the inertial sensor at the previous time;
the state data of the inertial sensor at the current time comprises: a speed of the inertial sensor at the current time.
The self-calibration of the sensor parameters of the visual inertial system is carried out based on the state data of the inertial sensor in the implementation mode, and the calibration effect is improved.
In one possible implementation form of the method,
the state data of the inertial sensor at the previous time further includes: zero offset of the accelerometer of the inertial sensor at the previous time instant and zero offset of the gyroscope of the inertial sensor at the previous time instant;
the state data of the inertial sensor at the current time further includes: a zero offset of the accelerometer of the inertial sensor at the current time and a zero offset of the gyroscope of the inertial sensor at the current time.
The self-calibration of the sensor parameters of the visual inertial system is carried out based on the state data of the inertial sensor in the implementation mode, and the calibration effect of the visual inertial system adopting the consumption-level inertial sensor is further improved.
In a possible implementation manner, the parameter to be calibrated includes a time offset, wherein the time offset represents a difference between a clock of the visual sensor and a clock of the inertial sensor at the same time;
the calibrating the parameters to be calibrated of the visual inertial system according to the state data of the visual inertial system at the current moment comprises the following steps:
and in the state data of the visual inertial system at the current moment, starting calibration of the time offset under the condition that the zero offset of the accelerometer of the inertial sensor meets a first preset stability condition and the zero offset of the gyroscope of the inertial sensor meets a second preset stability condition.
In this implementation, the first preset stability condition is a preset stability condition of zero offset of the accelerometer, and the second preset stability condition is a preset stability condition of zero offset of the gyroscope. When the zero offset of the accelerometer of the inertial sensor meets a first preset stability condition and the zero offset of the gyroscope of the inertial sensor meets a second preset stability condition, the stability of the accelerometer and the stability of the gyroscope of the inertial sensor are both high. And starting the calibration of the time offset at the moment, which is beneficial to improving the convergence rate of the parameters to be calibrated of the visual inertial system and improving the precision and robustness of parameter calibration. In addition, by calibrating the time offset, the time stamps of the visual sensor and the inertial sensor can be aligned, so that the data of the visual sensor and the inertial sensor in the visual inertial system are synchronous.
In a possible implementation manner, the parameter to be calibrated includes an internal parameter of the visual sensor;
the calibrating the parameters to be calibrated of the visual inertial system according to the state data of the visual inertial system at the current moment comprises the following steps:
and in the state data of the visual inertial system at the current moment, under the condition that the zero offset of the accelerometer of the inertial sensor meets a third preset stability condition, starting calibration of internal parameters of the visual sensor, wherein the stability of the zero offset of the accelerometer corresponding to the third preset stability condition is higher than that of the accelerometer corresponding to the first preset stability condition.
In this implementation manner, calibration of the internal parameters of the visual sensor is started when the zero offset of the accelerometer of the inertial sensor in the state data of the visual inertial system at the current time meets a third preset stability condition, which is helpful for improving the convergence rate of the parameters to be calibrated of the visual inertial system and improving the precision and robustness of parameter calibration. In addition, under the condition that the zero offset of the accelerometer of the inertial sensor meets a third preset stability condition, the stability of the visual inertial system is high, and the calibration of the internal reference of the visual sensor is started at the moment, so that the current visual inertial system can be kept in a stable state, and a good calibration effect can be obtained.
In one possible implementation, the internal parameters of the vision sensor include a focal length and an optical center of the vision sensor;
prior to the initiating calibration of the internal reference of the vision sensor, the method further comprises:
under the condition that the field angle of the vision sensor is obtained, determining an initial value of the focal length of the vision sensor according to the width of an image collected by the vision sensor and the field angle of the vision sensor;
or,
and under the condition that the field angle of the vision sensor cannot be acquired, determining an initial value of the focal length of the vision sensor according to the width and the height of an image acquired by the vision sensor.
The parameter calibration of the visual inertial system is carried out according to the initial value of the focal length of the visual sensor determined by the implementation mode, and the calibration precision is improved.
In one possible implementation, the internal reference of the vision sensor includes an optical center coordinate of the vision sensor;
prior to the initiating calibration of the internal reference of the vision sensor, the method further comprises:
and determining an initial value of the optical center coordinate of the vision sensor according to the width and the height of the image acquired by the vision sensor.
The calibration of the parameters of the visual inertial system is carried out according to the initial value of the optical center coordinates of the visual sensor determined by the implementation mode, and the calibration precision is improved.
In a possible implementation manner, the parameter to be calibrated includes a coordinate conversion parameter between the inertial sensor and the visual sensor, where the coordinate conversion parameter includes at least one of a translation parameter and a rotation parameter between the inertial sensor and the visual sensor;
the calibrating the parameters to be calibrated of the visual inertial system according to the state data of the visual inertial system at the current moment comprises the following steps:
and in the state data of the visual inertial system at the current moment, under the condition that the zero offset of the accelerometer of the inertial sensor meets a fourth preset stability condition, starting calibration of the coordinate conversion parameter, wherein the stability of the zero offset of the accelerometer corresponding to the fourth preset stability condition is higher than that of the accelerometer corresponding to the first preset stability condition.
In this implementation manner, calibration of the coordinate conversion parameter is started when the zero offset of the accelerometer of the inertial sensor in the state data of the visual inertial system at the current time meets a fourth preset stability condition, which is helpful for improving the convergence rate of the parameter to be calibrated of the visual inertial system and improving the precision and robustness of parameter calibration. In addition, under the condition that the zero offset of the accelerometer of the inertial sensor meets a fourth preset stability condition, the stability of the visual inertial system is high, and calibration of coordinate conversion parameters between the inertial sensor and the visual sensor is started at the moment, so that the current visual inertial system can be kept in a stable state, and a good calibration effect can be obtained.
In a possible implementation manner, the parameter to be calibrated includes a distortion parameter of the visual sensor;
the calibrating the parameters to be calibrated of the visual inertial system according to the state data of the visual inertial system at the current moment comprises the following steps:
and under the condition that the internal parameters of the visual sensor meet a fifth preset stability condition according to the state data of the visual inertial system at the current moment, starting calibration of distortion parameters of the visual sensor.
In this implementation manner, calibration of distortion parameters of the visual sensor is started under the condition that it is determined that the internal parameters of the visual sensor meet a fifth preset stability condition according to the state data of the visual inertial system at the current time, which is helpful for improving the convergence speed of the to-be-calibrated parameters of the visual inertial system and improving the precision and robustness of parameter calibration. In this implementation, the internal parameters of the vision sensor are calibrated prior to the distortion parameters of the vision sensor. And under the condition that the internal reference of the vision sensor meets a fifth preset stability condition, the internal reference of the vision sensor is converged. And the calibration of the distortion parameters of the visual sensor is started at the moment, so that the calibration effect of the internal parameters of the visual sensor is not influenced.
In one possible implementation, the method further includes:
and determining the parameters to be calibrated which are kept fixed at the current moment according to the motion state of the visual inertial system.
In this implementation manner, the parameter to be calibrated, which is kept fixed at the current time, is determined according to the motion state of the visual inertial system, so that only the parameter to be calibrated, which is observable at the current time, is calibrated at the current time, and the parameter to be calibrated, which is not observable at the current time, is kept fixed, so that the effectiveness of the parameter is not reduced due to the motion state of the visual inertial system, and the precision of the calibrated parameter can be improved, thereby facilitating the improvement of the precision of the positioning, mapping, navigation and the like of the visual inertial system.
In a possible implementation manner, the determining, according to the motion state of the visual inertial system, the parameter to be calibrated that remains fixed at the current time includes at least one of:
under the condition that the vision inertial system is static, determining parameters to be calibrated which are kept fixed at the current moment to comprise time offset, translation parameters between the inertial sensor and the vision sensor and rotation parameters between the inertial sensor and the vision sensor;
under the condition that the translation parameter of the visual sensor or the inertial sensor is 0 and the rotation parameter is not 0, determining that the parameter to be calibrated which is kept fixed at the current moment comprises the translation parameter between the inertial sensor and the visual sensor;
determining that the parameter to be calibrated which is kept fixed at the current moment comprises a parameter corresponding to a rotation axis in the translation parameters between the inertial sensor and the visual sensor under the condition that the visual sensor or the inertial sensor rotates around one axis of an x axis, a y axis and a z axis;
under the condition that the visual sensor or the inertial sensor moves around any coordinate axis at a constant angular speed, determining that the parameters to be calibrated which are kept fixed at the current moment comprise time offset and parameters corresponding to the coordinate axis in translation parameters between the inertial sensor and the visual sensor;
under the condition that the visual sensor or the inertial sensor moves along any coordinate axis at a constant speed, determining that the parameters to be calibrated which are kept fixed at the current moment comprise time offset and parameters corresponding to the coordinate axis in translation parameters between the inertial sensor and the visual sensor;
under the condition that the visual sensor or the inertial sensor moves around any coordinate axis at a constant angular acceleration, determining that the parameters to be calibrated which are kept fixed at the current moment comprise time offset and parameters corresponding to the coordinate axis in translation parameters between the inertial sensor and the visual sensor.
According to the implementation mode, the parameters to be calibrated, which are not observable at the current moment, can be kept fixed, so that the effectiveness of the parameters cannot be reduced due to the motion state of the visual inertial system, and the precision of the calibrated parameters can be improved.
In one possible implementation, the method further includes:
and determining the parameters to be calibrated which are kept fixed at the current moment according to the tracking state of the visual inertial system.
In this implementation manner, the parameter to be calibrated, which is kept fixed at the current time, is determined according to the tracking state of the visual inertial system, so that only the parameter to be calibrated, which is observable at the current time, is calibrated at the current time, and the parameter to be calibrated, which is not observable at the current time, is kept fixed, so that the effectiveness of the parameter is not reduced due to the tracking state of the visual inertial system, and the precision of the calibrated parameter can be improved, thereby facilitating the improvement of the precision of the positioning, mapping, navigation and the like of the visual inertial system.
In a possible implementation manner, the determining, according to the tracking state of the visual inertial system, the parameter to be calibrated that remains fixed at the current time includes:
in the case of loss of tracking of the visual inertial system, determining parameters to be calibrated that remain fixed at the current time comprise a time offset, a translation parameter between the inertial sensor and the visual sensor, and a rotation parameter between the inertial sensor and the visual sensor;
and/or the presence of a gas in the gas,
and under the condition that the number of the characteristic points currently tracked by the vision inertial system is less than a preset number, determining that the parameters to be calibrated which are kept fixed at the current moment comprise time offset, a translation parameter between the inertial sensor and the vision sensor and a rotation parameter between the inertial sensor and the vision sensor.
According to the implementation mode, the parameters to be calibrated, which are not observable at the current moment, can be kept fixed, so that the effectiveness of the parameters cannot be reduced due to the tracking state of the visual inertial system, and the precision of the calibrated parameters can be improved.
According to an aspect of the present disclosure, there is provided a parameter calibration apparatus of a visual inertial system, the visual inertial system including a visual sensor and an inertial sensor, the apparatus including:
the acquisition module is used for acquiring the state data of the visual inertial system at the last moment;
the estimation module is used for estimating the state data of the visual inertial system at the current moment according to the state data of the visual inertial system at the previous moment, the coordinates of the feature points in the image acquired by the visual sensor at the current moment and the output data of the inertial sensor at the current moment;
and the calibration module is used for calibrating the parameters to be calibrated of the visual inertial system according to the state data of the visual inertial system at the current moment.
In one possible implementation form of the method,
the state data of the visual inertial system at the previous time comprises: the state data of the visual sensor at the previous moment, the state data of the inertial sensor at the previous moment and the numerical value of the parameter to be calibrated at the previous moment;
the state data of the visual inertial system at the current time includes: the state data of the visual sensor at the current moment, the state data of the inertial sensor at the current moment and the numerical value of the parameter to be calibrated at the current moment.
In one possible implementation form of the method,
the status data of the vision sensor at the previous time comprises: a translation parameter of the vision sensor at the previous time and a rotation parameter of the vision sensor at the previous time;
the status data of the vision sensor at the current time includes: a translation parameter of the vision sensor at the current time and a rotation parameter of the vision sensor at the current time.
In one possible implementation form of the method,
the state data of the inertial sensor at the previous time comprises: the velocity of the inertial sensor at the previous time;
the state data of the inertial sensor at the current time includes: a speed of the inertial sensor at the current time.
In one possible implementation form of the method,
the state data of the inertial sensor at the previous time further includes: zero offset of the accelerometer of the inertial sensor at the previous time instant and zero offset of the gyroscope of the inertial sensor at the previous time instant;
the state data of the inertial sensor at the current time further includes: a zero offset of the accelerometer of the inertial sensor at the current time and a zero offset of the gyroscope of the inertial sensor at the current time.
In a possible implementation manner, the parameter to be calibrated includes a time offset, wherein the time offset represents a difference between a clock of the visual sensor and a clock of the inertial sensor at the same time;
the calibration module is used for:
and in the state data of the visual inertial system at the current moment, starting calibration of the time offset under the condition that the zero offset of the accelerometer of the inertial sensor meets a first preset stability condition and the zero offset of the gyroscope of the inertial sensor meets a second preset stability condition.
In a possible implementation manner, the parameter to be calibrated includes an internal parameter of the visual sensor;
the calibration module is used for:
and in the state data of the visual inertial system at the current moment, under the condition that the zero offset of the accelerometer of the inertial sensor meets a third preset stability condition, starting calibration of the internal reference of the visual sensor, wherein the stability of the zero offset of the accelerometer corresponding to the third preset stability condition is higher than that of the accelerometer corresponding to the first preset stability condition.
In one possible implementation, the internal parameters of the vision sensor include a focal length and an optical center of the vision sensor;
the device further comprises:
the first determining module is used for determining an initial value of the focal length of the vision sensor according to the width of an image acquired by the vision sensor and the field angle of the vision sensor under the condition that the field angle of the vision sensor is obtained;
or,
and the second determining module is used for determining an initial value of the focal length of the vision sensor according to the width and the height of the image acquired by the vision sensor under the condition that the field angle of the vision sensor cannot be acquired.
In one possible implementation, the internal reference of the vision sensor includes an optical center coordinate of the vision sensor;
the device further comprises:
and the third determining module is used for determining the initial value of the optical center coordinate of the vision sensor according to the width and the height of the image acquired by the vision sensor.
In a possible implementation manner, the parameter to be calibrated includes a coordinate conversion parameter between the inertial sensor and the visual sensor, where the coordinate conversion parameter includes at least one of a translation parameter and a rotation parameter between the inertial sensor and the visual sensor;
the calibration module is used for:
and in the state data of the visual inertial system at the current moment, under the condition that the zero offset of the accelerometer of the inertial sensor meets a fourth preset stability condition, starting calibration of the coordinate conversion parameter, wherein the stability of the zero offset of the accelerometer corresponding to the fourth preset stability condition is higher than that of the accelerometer corresponding to the first preset stability condition.
In a possible implementation manner, the parameter to be calibrated includes a distortion parameter of the visual sensor;
the calibration module is used for:
and under the condition that the internal parameters of the visual sensor meet a fifth preset stability condition according to the state data of the visual inertial system at the current moment, starting calibration of distortion parameters of the visual sensor.
In one possible implementation, the apparatus further includes:
and the fourth determination module is used for determining the parameters to be calibrated which are kept fixed at the current moment according to the motion state of the visual inertial system.
In one possible implementation, the fourth determining module is configured to at least one of:
determining, with the visual inertial system stationary, parameters to be calibrated that remain fixed at the current time include a time offset, a translation parameter between the inertial sensor and the visual sensor, and a rotation parameter between the inertial sensor and the visual sensor;
under the condition that the translation parameter of the visual sensor or the inertial sensor is 0 and the rotation parameter is not 0, determining that the parameter to be calibrated which is kept fixed at the current moment comprises the translation parameter between the inertial sensor and the visual sensor;
determining that the parameter to be calibrated which is kept fixed at the current moment comprises a parameter corresponding to a rotation axis in the translation parameters between the inertial sensor and the visual sensor under the condition that the visual sensor or the inertial sensor rotates around one axis of an x axis, a y axis and a z axis;
under the condition that the visual sensor or the inertial sensor moves around any coordinate axis at a constant angular speed, determining that the parameters to be calibrated which are kept fixed at the current moment comprise time offset and parameters corresponding to the coordinate axis in translation parameters between the inertial sensor and the visual sensor;
under the condition that the visual sensor or the inertial sensor moves along any coordinate axis at a constant speed, determining that the parameters to be calibrated which are kept fixed at the current moment comprise time offset and parameters corresponding to the coordinate axis in translation parameters between the inertial sensor and the visual sensor;
under the condition that the visual sensor or the inertial sensor moves around any coordinate axis at a constant angular acceleration, determining that the parameters to be calibrated which are kept fixed at the current moment comprise time offset and parameters corresponding to the coordinate axis in translation parameters between the inertial sensor and the visual sensor.
In one possible implementation, the apparatus further includes:
and the fifth determining module is used for determining the parameters to be calibrated which are kept fixed at the current moment according to the tracking state of the visual inertial system.
In one possible implementation manner, the fifth determining module is configured to:
in the case of loss of tracking of the visual inertial system, determining parameters to be calibrated that remain fixed at the current time comprise a time offset, a translation parameter between the inertial sensor and the visual sensor, and a rotation parameter between the inertial sensor and the visual sensor;
and/or the presence of a gas in the gas,
when the number of feature points currently tracked by the visual inertial system is less than a preset number, determining that the parameters to be calibrated, which are kept fixed at the current moment, include a time offset, a translation parameter between the inertial sensor and the visual sensor, and a rotation parameter between the inertial sensor and the visual sensor.
According to an aspect of the present disclosure, there is provided an electronic device including: one or more processors; a memory for storing executable instructions; wherein the one or more processors are configured to invoke the memory-stored executable instructions to perform the above-described method.
According to an aspect of the present disclosure, there is provided a computer readable storage medium having stored thereon computer program instructions which, when executed by a processor, implement the above-described method.
The embodiment of the disclosure obtains the state data of the visual inertial system at the previous moment, and according to the state data of the visual inertial system at the previous moment, the coordinates of the feature point in the image acquired by the visual sensor at the current moment, and the output data of the inertial sensor at the current moment, estimating the state data of the visual inertial system at the current moment, and according to the state data of the visual inertial system at the current moment, the parameters to be calibrated of the visual inertial system are calibrated, so that the calibration process does not need to be assisted by external equipment such as a calibration reference object, a calibration plate, a turntable and the like, the parameters to be calibrated are calibrated according to the state of the visual inertial system, so that the self-calibration of the sensor parameters of the visual inertial system can be realized, and more accurate parameters can be obtained.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.
Other features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments, which proceeds with reference to the accompanying drawings.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the disclosure.
Fig. 1 shows a flowchart of a parameter calibration method of a visual inertial system according to an embodiment of the present disclosure.
Fig. 2 shows a block diagram of a parameter calibration apparatus of a visual inertial system provided in an embodiment of the present disclosure.
Fig. 3 illustrates a block diagram of an electronic device 800 provided by an embodiment of the disclosure.
Fig. 4 shows a block diagram of an electronic device 1900 provided by an embodiment of the disclosure.
Detailed Description
Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. In the drawings, like reference numbers can indicate functionally identical or similar elements. While the various aspects of the embodiments are presented in drawings, the drawings are not necessarily drawn to scale unless specifically indicated.
The word "exemplary" is used exclusively herein to mean "serving as an example, embodiment, or illustration. Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments.
The term "and/or" herein is merely an association describing an associated object, meaning that three relationships may exist, e.g., a and/or B, may mean: a exists alone, A and B exist simultaneously, and B exists alone. In addition, the term "at least one" herein means any one of a plurality or any combination of at least two of a plurality, for example, including at least one of A, B, C, and may mean including any one or more elements selected from the group consisting of A, B and C.
Furthermore, in the following detailed description, numerous specific details are set forth in order to provide a better understanding of the present disclosure. It will be understood by those skilled in the art that the present disclosure may be practiced without some of these specific details. In some instances, methods, means, elements and circuits that are well known to those skilled in the art have not been described in detail so as not to obscure the subject matter of the present disclosure.
In the related art, the vision inertial system needs to adopt an off-line calibration mode to calibrate parameters of the sensor in advance. This calibration method requires expensive equipment and skilled technicians to configure the calibrated parameters into the visual inertial system after calibration is complete. However, the visual inertial system is calibrated in a specific environment, and the calibrated parameters are difficult to adapt to different working environments. In addition, the parameters calibrated off-line often have errors, and cannot be adjusted on-line in real time. Meanwhile, when the mechanical structure of the equipment where the visual inertial system is located changes, or the focal length of the visual sensor is changeable, or the visual sensor and the inertial sensor are not rigidly fixed, the parameters calibrated offline in advance often cannot meet the current requirements. Under the condition that offline calibrated parameters cannot be provided, a visual inertia system in the related technology cannot be implemented in application scenes such as virtual reality, augmented reality, mixed reality, unmanned aerial vehicles, mobile robots and unmanned vehicles.
In order to solve technical problems similar to those described above, embodiments of the present disclosure provide a method and an apparatus for calibrating parameters of a visual inertial system, an electronic device, and a medium, which can perform self-calibration on parameters to be calibrated of the visual inertial system. When the vision inertia system has no off-line calibrated parameters, the parameters to be calibrated can be estimated on line in the processes of positioning, drawing or navigation and the like; when the vision inertia system has the parameters calibrated offline, the parameters calibrated offline can be optimized, and the precision of positioning, drawing or navigation and the like is improved.
Fig. 1 shows a flowchart of a parameter calibration method of a visual inertial system according to an embodiment of the present disclosure. The visual inertial system includes a visual sensor and an inertial sensor. The executive body of the parameter calibration method of the visual inertial system can be a parameter calibration device of the visual inertial system. For example, the parameter calibration device of the visual inertial system may be any device including the visual inertial system. For example, the parameter calibration method of the visual inertial system may be executed by a terminal device or a server or other processing device. The terminal device may be a User Equipment (UE), a mobile device, a User terminal, a cellular phone, a cordless phone, a Personal Digital Assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, or a wearable device. In some possible implementations, the parameter calibration method of the visual inertial system may be implemented by a processor calling computer-readable instructions stored in a memory. As shown in fig. 1, the parameter calibration method of the visual inertial system includes steps S11 to S13.
In step S11, state data of the visual inertial system at the previous time is acquired.
In the disclosed embodiments, the visual inertial system may represent a system that combines visual sensors and inertial sensors for displacement and attitude estimation for at least one of positioning, mapping, navigation, and the like. The Visual Inertial System may be referred to as a Visual-Inertial odometer (VIO), a Visual-Inertial Navigation System (VINS), a Visual-Inertial SLAM (VI-SLAM, which uses a Visual sensor And an Inertial sensor to perform Simultaneous Localization And Mapping), And the like. The visual sensor may refer to an apparatus for acquiring image information of an external environment using an optical element and an imaging device. The vision sensor may be a monocular vision sensor, a binocular vision sensor, an RGBD (Red, Green, Blue, Deep) vision sensor, etc. For example, the vision sensor may employ a monocular camera, a binocular camera, a multi-view camera, an RGBD camera, or the like. The inertial sensor may refer to a sensor that measures using an inertial force of a sensing mass. The inertial sensors may include a three-axis accelerometer and a three-axis gyroscope. The Inertial sensor may also be referred to as an Inertial Measurement Unit (IMU). For example, the inertial sensor may employ a consumer IMU including an accelerometer and a gyroscope, and may also employ a high precision inertial navigation system, a strapdown inertial navigation system, or the like.
In the embodiment of the present disclosure, the state data of the visual inertial system at the previous time may represent data representing a state of the visual inertial system at the previous time obtained by estimating the state of the visual inertial system at the previous time.
In step S12, the state data of the visual inertial system at the current time is estimated based on the state data of the visual inertial system at the previous time, the coordinates of the feature point in the image acquired by the visual sensor at the current time, and the output data of the inertial sensor at the current time.
In embodiments of the present disclosure, a time difference between two adjacent time instants (e.g., the current time instant and the previous time instant) may be greater than or equal to an inverse of a frequency with which the vision sensor acquires images. For example, the time difference between two adjacent time instants may be equal to the inverse of the frequency with which the vision sensor acquires images. In this example, the state data of the visual inertial system may be re-estimated each time a new image is acquired by the visual sensor. As another example, the time difference between two adjacent time instants may be equal to K times the inverse of the frequency with which the vision sensor acquires images, where K is an integer greater than 1. In one example, the previous time may be denoted as an ith time, and the current time may be denoted as an i +1 th time.
In the embodiment of the present disclosure, as at least one of the position, the posture, the motion state, and the like of the visual inertial system changes, the image acquired by the visual sensor and the data output by the inertial sensor will change. The feature points in the image acquired by the vision sensor may include corner points, key points, and the like in the image acquired by the vision sensor, which is not limited herein. The data output by the inertial sensor may include data output by an accelerometer of the inertial sensor and/or data output by a gyroscope of the inertial sensor. In the embodiment of the present disclosure, the state data of the visual inertial system at the current time may be estimated by using the coordinates of a plurality of feature points in the image acquired by the visual sensor at the current time and the data output by the inertial sensor. The state data of the visual inertial system at the current time may represent data representing the state of the visual inertial system at the current time, which is obtained by estimating the state of the visual inertial system at the current time.
In step S13, calibrating the parameters to be calibrated of the visual inertial system according to the state data of the visual inertial system at the current time.
In the embodiment of the present disclosure, the parameter to be calibrated of the visual inertial system may include at least one of an internal parameter of the visual sensor, a distortion parameter of the visual sensor, a coordinate transformation parameter between the inertial sensor and the visual sensor, a time offset between the visual sensor and the inertial sensor, and the like.Wherein the internal reference of the vision sensor may comprise a focal length f of the vision sensor x And f y And optical center coordinates c of said vision sensor x And c y . The distortion parameter of the vision sensor may comprise a radial distortion parameter k of the vision sensor 1 And k 2 And the tangential distortion parameter p of the vision sensor 1 And p 2 . The coordinate conversion parameters between the inertial sensor and the vision sensor may include a translation parameter between the inertial sensor and the vision sensor and/or a rotation parameter between the inertial sensor and the vision sensor. For example, the coordinate conversion parameters between the inertial sensor and the vision sensor may comprise translation parameters of the inertial sensor to the vision sensor and/or rotation parameters of the inertial sensor to the vision sensor, or the coordinate conversion parameters between the inertial sensor and the vision sensor may comprise translation parameters of the vision sensor to the inertial sensor and/or rotation parameters of the vision sensor to the inertial sensor. For example, the coordinate conversion parameter between the inertial sensor and the vision sensor may include a translation parameter p of the inertial sensor to the vision sensor extri And a rotation parameter q from the inertial sensor to the visual sensor extri . Wherein the translation parameters of the inertial sensor to the vision sensor may represent a translation component of a motion converted from the coordinate system of the inertial sensor to the camera coordinate system of the vision sensor; the rotation parameters of the inertial sensor to the vision sensor may represent a rotation component of a camera coordinate system motion converted from the coordinate system of the inertial sensor to the vision sensor. The time offset between the visual sensor and the inertial sensor can be denoted as t d It may represent the difference between the clock of the visual sensor and the clock of the inertial sensor at the same instant.
The embodiment of the disclosure obtains the state data of the visual inertial system at the previous moment, and according to the state data of the visual inertial system at the previous moment, the coordinates of the feature point in the image acquired by the visual sensor at the current moment, and the output data of the inertial sensor at the current moment, estimating the state data of the visual inertial system at the current moment, and according to the state data of the visual inertial system at the current moment, the parameters to be calibrated of the visual inertial system are calibrated, so that the calibration process does not need to be assisted by external equipment such as a calibration reference object, a calibration plate, a turntable and the like, the parameters to be calibrated are calibrated according to the state of the visual inertial system, so that the self-calibration of the sensor parameters of the visual inertial system can be realized, and more accurate parameters can be obtained.
In one possible implementation, the state data of the visual inertial system at the previous time instant includes: the state data of the visual sensor at the previous moment, the state data of the inertial sensor at the previous moment and the numerical value of the parameter to be calibrated at the previous moment; the state data of the visual inertial system at the current time includes: the state data of the visual sensor at the current moment, the state data of the inertial sensor at the current moment and the numerical value of the parameter to be calibrated at the current moment. In this implementation, the state data of the vision sensor may represent data that can embody a motion state of the vision sensor. Accordingly, the state data of the visual sensor at the previous moment can represent data capable of representing the motion state of the visual sensor at the previous moment; the state data of the visual sensor at the current time may represent data capable of representing a motion state of the visual sensor at the current time. The state data of the inertial sensor may represent data that can represent the state of the inertial sensor. Accordingly, the state data of the inertial sensor at the previous time may represent data capable of representing the state of the inertial sensor at the previous time; the state data of the inertial sensor at the current time may represent data capable of representing the state of the inertial sensor at the current time. The sensor parameters of the visual inertial system are self-calibrated based on the state data of the visual inertial system in the implementation mode, and accurate parameters can be obtained.
As an example of this implementation, the status data of the visual sensor at the previous time includes: a translation parameter of the vision sensor at the previous time and a rotation parameter of the vision sensor at the previous time; the status data of the vision sensor at the current time includes: a translation parameter of the vision sensor at the current time and a rotation parameter of the vision sensor at the current time. In this example, the status data of the vision sensor includes a translation parameter and a rotation parameter of the vision sensor. The sensor parameters of the visual inertial system are self-calibrated based on the state data of the visual sensor in the example, and the calibration effect is improved.
Of course, those skilled in the art can flexibly select the status data of the visual sensor according to the requirements of the actual application scenario. For example, the status data of the vision sensor may include only translation parameters of the vision sensor, or may include only rotation parameters of the vision sensor, or may include other parameters of the vision sensor, which is not limited herein.
As an example of this implementation, the state data of the inertial sensor at the previous time includes: the velocity of the inertial sensor at the previous time; the state data of the inertial sensor at the current time includes: a speed of the inertial sensor at the current time. In this example, the state data of the inertial sensor includes a velocity of the inertial sensor. The self-calibration of the sensor parameters of the visual inertial system is carried out based on the state data of the inertial sensor in the example, which is beneficial to improving the calibration effect.
In one example, the state data of the inertial sensor at the previous time further includes: zero offset of the accelerometer of the inertial sensor at the previous time instant and zero offset of the gyroscope of the inertial sensor at the previous time instant; the state data of the inertial sensor at the current time further includes: a zero offset of the accelerometer of the inertial sensor at the current time and a zero offset of the gyroscope of the inertial sensor at the current time. In this example, the state data of the inertial sensor includes a velocity of the inertial sensor, a zero offset of the accelerometer, and a zero offset of the gyroscope. The zero offset of the accelerometer represents the output of the accelerometer when the input of the accelerometer is 0, and the zero offset of the gyroscope represents the output of the gyroscope when the input of the gyroscope is 0. The self-calibration of the sensor parameters of the visual inertial system based on the state data of the inertial sensor in this example contributes to further improving the calibration effect of the visual inertial system using the consumer-grade inertial sensor.
Of course, those skilled in the art can flexibly select the state data of the inertial sensor according to the requirements of the actual application scenario. For example, the state data of the inertial sensor may include the velocity of the inertial sensor and the zero offset of the accelerometer, not including the zero offset of the gyroscope. As another example, the state data of the inertial sensor may include a velocity of the inertial sensor and a zero offset of a gyroscope, excluding a zero offset of an accelerometer. As another example, the state data of the inertial sensor may include only a zero offset of an accelerometer or a zero offset of a gyroscope.
Of course, a person skilled in the art may flexibly set the data included in the state data of the visual inertial system according to the requirements of the actual application scenario, which is not limited by the embodiment of the present disclosure. In the state data of the visual inertial system, for example, the translation parameter of the inertial sensor may be used to replace the translation parameter of the visual sensor, or, for example, the rotation parameter of the inertial sensor may be used to replace the rotation parameter of the visual sensor, or, for example, the speed of the visual sensor may be used to replace the speed of the inertial sensor, or, for example, other parameters measuring the stability and/or accuracy of the inertial sensor may be used to replace the zero offset of the accelerometer of the inertial sensor and the zero offset of the gyroscope of the inertial sensor.
In a possible implementation manner, the state data of the visual inertial sensor may include a translation parameter of the visual sensor, a rotation parameter of the visual sensor, a speed of the inertial sensor, a zero offset of an accelerometer of the inertial sensor, a zero offset of a gyroscope of the inertial sensor, and a parameter to be calibrated. For example, for a consumer-grade inertial sensor in a device such as a cell phone, the zero offset of an accelerometer and the zero offset of a gyroscope may be added to the state data of the visual inertial sensor.
In another possible implementation manner, the state data of the visual inertial sensor may include a translation parameter of the visual sensor, a rotation parameter of the visual sensor, a speed of the inertial sensor, and a parameter to be calibrated. For example, for high precision inertial sensors, the zero offset of the accelerometer and the zero offset of the gyroscope may not be considered.
In one possible implementation, the state data X of the visual inertial system can be constructed as shown in equation 1:
wherein, (.) C Camera coordinate system that can represent a vision sensor, (-) I The coordinate system of the inertial sensor may be represented. The visual sensor and the inertial sensor can be regarded as rigid bodies, and the motion of the rigid bodies can be in six degrees of freedom, namely translational motion in three degrees of freedom and rotational motion in three degrees of freedom;
a translation parameter indicative of a visual sensor,it may be a three-dimensional parameter that,may represent the translation parameters of the vision sensor at the last instant,may represent a translation parameter of the visual sensor at the current time;
a parameter indicative of the rotation of the vision sensor,it may be a three-dimensional parameter that,may represent the rotation parameters of the visual sensor at the last moment,may represent a rotation parameter of the visual sensor at the current time;
is indicative of the velocity of the inertial sensor,it may be a three-dimensional parameter that,may represent the velocity of the inertial sensor at the last moment,may represent the velocity of the inertial sensor at the current time;
representing the zero offset of the accelerometer of the inertial sensor,it may be a three-dimensional parameter that,may represent the zero offset of the accelerometer at the last moment of the inertial sensor,may represent the zero offset of the accelerometer of the inertial sensor at the current time; zero bias (bias) may also be referred to as random walk;
representing the zero offset of the gyroscope of the inertial sensor,it may be a three-dimensional parameter that,may represent the zero offset of the gyroscope of the inertial sensor at the last moment,may represent the zero offset of the gyroscope of the inertial sensor at the current time;
X calib representing the parameter to be calibrated.
As an example of this implementation, X calib Parameters may be included as shown in equation 2:
X calib =[f x f y c x c y k 1 k 2 p 1 p 2 p extri q extri t d ]in the formula (2), the first and second groups,
wherein f is x 、f y 、c x 、c y 、k 1 、k 2 、p 1 And p 2 Respectively a one-dimensional parameter, p extri And q is extri Can be a three-dimensional parameter, t d Is a one-dimensional parameter.
In one example, t d Can adoptDetermined by equation 3:
t d =t C -t I in the formula 3, the first step is,
wherein, t C Clock representing vision sensor, t I Representing the clock of the inertial sensor.
Those skilled in the art can determine which parameters to be specifically included in the parameters to be calibrated according to the requirements of the actual application scenario, which is not limited herein. In the embodiment of the present disclosure, in the case of having a parameter calibrated offline, a value of the parameter calibrated offline may be used as an initial value of the parameter, and the value of the parameter may be updated by self-calibration; in the case of no offline calibrated parameters, the initial values of the parameters to be calibrated may be set.
In one possible implementation, the internal reference of the vision sensor includes a focal length of the vision sensor; prior to the initiating calibration of the internal reference of the vision sensor, the method further comprises: under the condition that the field angle of the vision sensor is obtained, determining an initial value of the focal length of the vision sensor according to the width of an image collected by the vision sensor and the field angle of the vision sensor; or determining an initial value of the focal length of the vision sensor according to the width and the height of an image acquired by the vision sensor under the condition that the field angle of the vision sensor cannot be acquired. The parameter calibration of the visual inertial system is carried out according to the initial value of the focal length of the visual sensor determined by the implementation mode, and the calibration precision is improved.
For example, equation 4 may be employed to determine an initial value for the focal length of the vision sensor:
wherein W and H represent the width and height, respectively, of the image acquired by the vision sensor, and C fov Indicating the field angle of the vision sensor.
In one possible implementation, the internal reference of the vision sensor includes an optical center coordinate of the vision sensor; prior to the initiating calibration of the internal reference of the vision sensor, the method further comprises: and determining an initial value of the optical center coordinate of the vision sensor according to the width and the height of the image acquired by the vision sensor. The calibration of the parameters of the visual inertial system is carried out according to the initial value of the optical center coordinates of the visual sensor determined by the implementation mode, and the calibration precision is improved.
For example, the initial value of the optical center coordinates of the vision sensor can be determined using equations 5 and 6:
in one possible implementation, the distortion parameter k of the vision sensor 1 ,k 2 ,p 1 And p 2 May be set to 0.
In one possible implementation, the time offset t d May be set to 0.
In one possible implementation, the translation parameter p from the inertial sensor to the visual sensor extri May be set to an initial value of [ 000 ]] T 。
In one possible implementation, the rotation parameter q from the inertial sensor to the visual sensor extri May be set to be I 3×3 Wherein I represents an identity matrix.
It should be noted that, although the above implementation manner describes the setting manner of the initial value of the parameter to be calibrated, a person skilled in the art can understand that the disclosure should not be limited thereto. Those skilled in the art can flexibly set the initial value of the parameter to be calibrated according to the requirements and/or experience of the actual application scenario, which is not limited herein.
In one possible implementation, visual sensingThe three-dimensional coordinate of the feature point in the image acquired by the device under the camera coordinate system of the vision sensor can be recorded as P C =[X C Y C Z C ] T 。P C Corresponding two-dimensional coordinates in image plane coordinate systemP C The corresponding coordinates in the pixel coordinate system can be determined using equation 7:
from the pinhole vision sensor model, equations 8 to 11 can be obtained:
k r =1+k 1 r 2 +k 2 r 4 in the formula (9) shown below,
p C coordinate p after removing distortion C ′=[x C ′ y C ′] T ,x C ' and y C ' can be obtained by using formula 12 and formula 13, respectively:
x C ′=x C ×k r +p x in the formula (12), the first and second groups,
y C ′=y C ×k r +p y and (3) formula 13.
In one possible implementation, as shown in equation 14, the translation parameters of the vision sensorCan be based on the translation parameters of the inertial sensorAnd inertial sensor to visual sensor translation parameter p extri Determining:
as shown in equation 15, the rotation parameter of the vision sensorCan be based on the rotation parameters of the inertial sensorAnd rotation parameter q from inertial sensor to visual sensor extri Determining:
as shown in equations 16 through 18, the translational parameters of the inertial sensorRotation parameterAnd velocityCan pass productCalculating:
wherein (·) (i) and (·) (i +1) respectively represent data of an i-th time (i.e., a previous time) and an i + 1-th time (i.e., a current time),anddata representing the accelerometer and gyroscope outputs respectively,andnoise terms representing data output by the accelerometer and gyroscope, respectively.
wherein, a I And w I Real data representing an accelerometer and a gyroscope, respectively;is white noise, can be obtained by looking up a table, can default to 0, and can also be usedThe derivation yields the result of, for example,is white noise, can be obtained by looking up a table, can default to 0, and can also be usedThe derivation yields the result of, for example,
in one possible implementation, the jacobian matrix corresponding to the state data of the visual inertial system can be represented by equation 21:
wherein 15 represents J p 、J q ,J v ,J ba And J bg M represents J calib The dimension of the parameter to be calibrated (i.e. the number of the parameter to be calibrated). J. the design is a square p Jacobian matrix, J, representing translation parameters of a vision sensor q Jacobian matrix, J, representing the correspondence of rotation parameters of a vision sensor v Jacobian matrix representing velocity correspondences of inertial sensors, J ba Jacobian matrix, J, representing zero-offset correspondences of accelerometers of an inertial sensor bg Jacobian matrix, J, representing zero-offset correspondences of gyroscopes of inertial sensors calib To representAnd the Jacobian matrix corresponding to the parameters to be calibrated. J. the design is a square p 、J q ,J v ,J ba And J bg Can be determined using equations 22 through 26, respectively:
wherein,representing the residual error corresponding to the translation parameter of the inertial sensor,representing the residual error corresponding to the rotation parameter of the inertial sensor,the residual error corresponding to the velocity of the inertial sensor is represented, g represents the gravitational acceleration, and Δ t represents the reciprocal of the frequency of the inertial sensor output data.
In one possible implementation, the internal parameter f of the vision sensor can be obtained by using equations 27 to 30 x 、f y 、c x And c y Corresponding jacobian matrix:
in one possible implementation, the distortion parameter k of the vision sensor can be obtained by using equations 31 to 34 1 、k 2 、p 1 And p 2 Corresponding jacobian matrix:
in one possible implementation, P can be obtained by using equations 35 to 44 C =[X C Y C Z C ] T A Jacobian matrix corresponding to the reprojection, wherein the reprojection represents that points on one frame image are projected onto another frame image through a planar homography matrix and a projection matrix in computer vision。
In one possible implementation, after initialization of the visual inertial system, equations 45 through 48 may be used to determineVariance of zero offset, Var, of an accelerometer of an inertial sensor acc Variance of zero offset Var of gyroscope and inertial sensor gyro :
Where w represents the size of the sliding window. w may be determined empirically, e.g., w may be equal to 11, 3, etc.
In the disclosed embodiments, by estimating the state data of the visual inertial system, the zero-bias of the accelerometer and the zero-bias of the gyroscope will gradually settle, e.g., the variance Var of the zero-bias of the accelerometer acc Variance of zero offset Var of the sum gyroscope gyro Will continue to decrease to a converged value.
In a possible implementation manner, before starting calibration of any parameter to be calibrated, the parameter to be calibrated may be fixed to an initial value.
In a possible implementation manner, the parameter to be calibrated includes a time offset, wherein the time offset represents a difference between a clock of the visual sensor and a clock of the inertial sensor at the same time; the calibrating the parameters to be calibrated of the visual inertial system according to the state data of the visual inertial system at the current moment comprises the following steps: and in the state data of the visual inertial system at the current moment, starting calibration of the time offset under the condition that the zero offset of the accelerometer of the inertial sensor meets a first preset stability condition and the zero offset of the gyroscope of the inertial sensor meets a second preset stability condition.
In this implementation, the stability of the zero offset of the accelerometer can be measured according to the discrete degree of the zero offset of the accelerometer, and the stability of the zero offset of the gyroscope can be measured according to the discrete degree of the zero offset of the gyroscope. The higher the dispersion degree of the zero offset of the accelerometer is, the lower the stability of the zero offset of the accelerometer is; the lower the dispersion degree of the zero offset of the accelerometer is, the higher the stability of the zero offset of the accelerometer is; the higher the dispersion degree of the zero offset of the gyroscope is, the lower the stability of the zero offset of the gyroscope is; the lower the degree of dispersion of the zero bias of the gyroscope, the higher the stability of the zero bias of the gyroscope. For example, the stability of the zero offset of the accelerometer can be measured according to the variance, standard deviation, etc. of the zero offset of the accelerometer, and the stability of the zero offset of the gyroscope can be measured according to the variance, standard deviation, etc. of the zero offset of the gyroscope. The larger the variance or standard deviation of the zero offset of the accelerometer is, the lower the stability of the zero offset of the accelerometer is; the smaller the variance or standard deviation of the zero offset of the accelerometer, the higher the stability of the zero offset of the accelerometer; the larger the variance or standard deviation of the zero offset of the gyroscope is, the lower the stability of the zero offset of the gyroscope is; the smaller the variance or standard deviation of the gyroscope's null, the higher the stability of the gyroscope's null.
In this implementation, the first preset stability condition is a preset stability condition of zero offset of the accelerometer, and the second preset stability condition is a preset stability condition of zero offset of the gyroscope. When the zero offset of the accelerometer of the inertial sensor meets a first preset stability condition and the zero offset of the gyroscope of the inertial sensor meets a second preset stability condition, the stability of the accelerometer of the inertial sensor and the stability of the gyroscope of the inertial sensor are both high. And starting the calibration of the time offset at the moment, which is beneficial to improving the convergence rate of the parameters to be calibrated of the visual inertial system and improving the precision and robustness of parameter calibration. In addition, by calibrating the time offset, the time stamps of the visual sensor and the inertial sensor can be aligned, so that the data of the visual sensor and the inertial sensor in the visual inertial system are synchronous.
As an example of this implementation, the meeting of the zero offset of the accelerometer to the first preset stability condition includes: the variance of the zero offset of the accelerometer is smaller than a first preset value; the condition that the zero offset of the gyroscope meets the second preset stability condition comprises the following steps: and the variance of the zero offset of the gyroscope is smaller than a second preset value. For example, the first preset value may be 0.08 and the second preset value may be 0.001. Of course, the first preset value may also be slightly larger or slightly smaller than 0.08, and the second preset value may also be slightly larger or slightly smaller than 0.001, and those skilled in the art can flexibly determine the first preset value and the second preset value according to experience.
In a possible implementation manner, the parameter to be calibrated includes an internal parameter of the visual sensor; the calibrating the parameters to be calibrated of the visual inertial system according to the state data of the visual inertial system at the current moment comprises the following steps: and in the state data of the visual inertial system at the current moment, under the condition that the zero offset of the accelerometer of the inertial sensor meets a third preset stability condition, starting calibration of the internal reference of the visual sensor, wherein the stability of the zero offset of the accelerometer corresponding to the third preset stability condition is higher than that of the accelerometer corresponding to the first preset stability condition. In this implementation, the third preset stability condition is a preset zero offset stability condition of the accelerometer. The stability of the zero offset of the accelerometer corresponding to the third preset stability condition is higher than that of the accelerometer corresponding to the first preset stability condition, and the requirement of the third preset stability condition on the stability of the zero offset of the accelerometer is higher than that of the first preset stability condition. In this implementation manner, calibration of the internal parameters of the visual sensor is started when the zero offset of the accelerometer of the inertial sensor in the state data of the visual inertial system at the current time meets a third preset stability condition, which is helpful for improving the convergence rate of the parameters to be calibrated of the visual inertial system and improving the precision and robustness of parameter calibration. In addition, under the condition that the zero offset of the accelerometer of the inertial sensor meets a third preset stability condition, the stability of the visual inertial system is high, and the calibration of the internal reference of the visual sensor is started at the moment, so that the current visual inertial system can be kept in a stable state, and a good calibration effect can be obtained.
As an example of this implementation, the accelerometer having a zero offset that satisfies a third preset stability condition includes: and the variance of the zero offset of the accelerometer is smaller than a third preset value, wherein the third preset value is smaller than the first preset value. For example, the third preset value may be 0.008. Of course, the third preset value may also be slightly larger or smaller than 0.008, and those skilled in the art can flexibly determine the third preset value according to experience.
In a possible implementation manner, the parameter to be calibrated includes a coordinate conversion parameter between the inertial sensor and the visual sensor, where the coordinate conversion parameter includes at least one of a translation parameter and a rotation parameter between the inertial sensor and the visual sensor; the calibrating the parameters to be calibrated of the visual inertial system according to the state data of the visual inertial system at the current moment comprises the following steps: and in the state data of the visual inertial system at the current moment, under the condition that the zero offset of the accelerometer of the inertial sensor meets a fourth preset stability condition, starting calibration of the coordinate conversion parameter, wherein the stability of the zero offset of the accelerometer corresponding to the fourth preset stability condition is higher than that of the accelerometer corresponding to the first preset stability condition. In this implementation, the fourth preset stability condition is a preset zero-offset stability condition of the accelerometer. The stability of the zero offset of the accelerometer corresponding to the fourth preset stability condition is higher than that of the accelerometer corresponding to the first preset stability condition, and the requirement of the fourth preset stability condition on the stability of the zero offset of the accelerometer is higher than that of the first preset stability condition. In this implementation manner, calibration of the coordinate conversion parameter is started when the zero offset of the accelerometer of the inertial sensor in the state data of the visual inertial system at the current time meets a fourth preset stability condition, which is helpful for improving the convergence rate of the parameter to be calibrated of the visual inertial system and improving the precision and robustness of parameter calibration. In addition, under the condition that the zero offset of the accelerometer of the inertial sensor meets a fourth preset stability condition, the stability of the visual inertial system is high, and calibration of coordinate conversion parameters between the inertial sensor and the visual sensor is started at the moment, so that the current visual inertial system can be kept in a stable state, and a good calibration effect can be obtained.
As an example of this implementation, the accelerometer having a zero offset that satisfies a fourth preset stability condition includes: and the variance of the zero offset of the accelerometer is smaller than a fourth preset value, wherein the fourth preset value is smaller than the first preset value. The fourth preset value may be equal to the third preset value, or slightly larger or slightly smaller than the third preset value. For example, the fourth preset value may be 0.008. Of course, the fourth preset value may also be slightly larger or smaller than 0.008, and those skilled in the art can flexibly determine the fourth preset value according to experience.
In one possible implementation, the calibration time offset may continue after the calibration of the internal parameters of the vision sensor and the calibration of the coordinate conversion parameters are turned on.
In one possible implementation, f may be determined using equations 49 through 56 x Variance of (2)f y Variance of (2)c x Variance of (2)And c y Variance of (2)
In a possible implementation manner, the parameter to be calibrated includes a distortion parameter of the visual sensor; the calibrating the parameters to be calibrated of the visual inertial system according to the state data of the visual inertial system at the current moment comprises the following steps: and under the condition that the internal parameters of the visual sensor meet a fifth preset stability condition according to the state data of the visual inertial system at the current moment, starting calibration of distortion parameters of the visual sensor. In this implementation, the stability of the internal reference of the vision sensor may be measured in terms of the degree of dispersion of the internal reference of the vision sensor. Wherein, the higher the discrete degree of the internal reference of the vision sensor, the lower the stability of the internal reference of the vision sensor; the lower the degree of dispersion of the internal reference of the vision sensor, the higher the stability of the internal reference of the vision sensor. For example, the stability of the internal reference of the vision sensor may be measured in terms of variance, standard deviation, etc. of the internal reference of the vision sensor. The larger the variance or standard deviation of the internal parameters of the vision sensor is, the lower the stability of the internal parameters of the vision sensor is; the smaller the variance or standard deviation of the internal reference of the vision sensor, the higher the stability of the internal reference of the vision sensor. In this implementation, the fifth preset stability condition is a preset stability condition of an internal parameter of the vision sensor. And when the internal reference of the vision sensor meets the fifth preset stability condition, the stability of the internal reference of the vision sensor is higher. In this implementation manner, calibration of distortion parameters of the visual sensor is started under the condition that it is determined that the internal parameters of the visual sensor meet a fifth preset stability condition according to the state data of the visual inertial system at the current time, which is helpful for improving the convergence speed of the to-be-calibrated parameters of the visual inertial system and improving the precision and robustness of parameter calibration. In this implementation, the internal parameters of the vision sensor are calibrated prior to the distortion parameters of the vision sensor. And under the condition that the internal reference of the vision sensor meets a fifth preset stability condition, the internal reference of the vision sensor is converged. And the calibration of the distortion parameters of the visual sensor is started at the moment, so that the calibration effect of the internal parameters of the visual sensor is not influenced.
As an example of this implementation, the meeting of the fifth preset stability condition by the internal reference of the vision sensor includes: and the variance of the internal reference of the vision sensor is smaller than a fifth preset value. For example, the fifth preset value may be 0.001. Of course, the fifth preset value may also be slightly larger or smaller than 0.001, and those skilled in the art can flexibly determine the fifth preset value according to experienceA preset value. In this example, the variance of the internal parameters of the vision sensor is less than a fifth preset value, which may be f x 、f y 、c x And c y Are respectively smaller than a fifth preset value. Of course, the person skilled in the art can set the conditions for starting calibration of the distortion parameters of the vision sensor flexibly according to experience. For example, it may also be at f x 、f y 、c x And c y And under the condition that the variance of one item, two items or three items is less than a fifth preset value, starting the calibration of the distortion parameter of the visual sensor.
In one possible implementation, the calibration of the time offset, the internal reference of the vision sensor, and the coordinate transformation parameters may continue after the calibration of the distortion parameters of the vision sensor is turned on.
In one possible implementation, the method further includes: and determining the parameters to be calibrated which are kept fixed at the current moment according to the motion state of the visual inertial system. In this implementation, the motion state of the visual sensor may be taken as the motion state of the visual inertial system to improve the reliability of the acquired motion state of the visual inertial system. Of course, in practical applications, the motion state of the inertial sensor may be used as the motion state of the visual inertial system, and is not limited herein. In this implementation manner, according to the motion state of the visual inertial system, the parameter to be calibrated, which is not observable at the current time, may be determined as the parameter to be calibrated, which remains fixed at the current time. In this implementation manner, the parameter to be calibrated, which is kept fixed at the current time, is determined according to the motion state of the visual inertial system, so that only the parameter to be calibrated, which is observable at the current time, is calibrated at the current time, and the parameter to be calibrated, which is not observable at the current time, is kept fixed, so that the effectiveness of the parameter is not reduced due to the motion state of the visual inertial system, and the precision of the calibrated parameter can be improved, thereby facilitating the improvement of the precision of the positioning, mapping, navigation and the like of the visual inertial system.
As an example of this implementation, the determining, according to the motion state of the visual inertial system, the parameter to be calibrated that remains fixed at the current time includes: in the case where the visual inertial system is stationary, determining the parameters to be calibrated that remain fixed at the current time includes a time offset, a translation parameter between the inertial sensor and the visual sensor, and a rotation parameter between the inertial sensor and the visual sensor. In this example, if the motion state of the visual sensor is taken as the motion state of the visual inertial system, it may be determined that the visual inertial system is stationary in the case where the translation parameter of the visual sensor is 0 and the rotation parameter is 0; if the motion state of the inertial sensor is taken as the motion state of the visual inertial system, the visual inertial system may be determined to be stationary when the translation parameter of the inertial sensor is 0 and the rotation parameter is 0. In this example, with the visual inertial system stationary, the time offset, the translation parameter between the inertial sensor and the visual sensor, and the rotation parameter between the inertial sensor and the visual sensor are kept fixed, and the calibration is performed on the other parameters to be calibrated that have been started.
As an example of this implementation, the determining, according to the motion state of the visual inertial system, the parameter to be calibrated that remains fixed at the current time includes: and under the condition that the translation parameter of the visual sensor or the inertial sensor is 0 and the rotation parameter is not 0, determining that the parameter to be calibrated which is kept fixed at the current moment comprises the translation parameter between the inertial sensor and the visual sensor. In this example, in the case where the translation parameter of the visual sensor or the inertial sensor is 0 and the rotation parameter is not 0, it may be determined that the visual inertial system is operating in pure rotation. In this example, if the motion state of the visual sensor is taken as the motion state of the visual inertial system, the translation parameter between the inertial sensor and the visual sensor may be kept fixed and other to-be-calibrated parameters which have started calibration may be calibrated under the condition that the translation parameter of the visual sensor is 0 and the rotation parameter is not 0; if the motion state of the inertial sensor is taken as the motion state of the visual inertial system, the translation parameter between the inertial sensor and the visual sensor can be kept fixed under the condition that the translation parameter of the inertial sensor is 0 and the rotation parameter is not 0, and other to-be-calibrated parameters which are started to be calibrated are calibrated.
As an example of this implementation, the determining, according to the motion state of the visual inertial system, the parameter to be calibrated that remains fixed at the current time includes: in the case where the visual sensor or the inertial sensor rotates around one of an x-axis, a y-axis, and a z-axis, determining the parameter to be calibrated that remains fixed at the current time includes a parameter corresponding to a rotation axis among the translational parameters between the inertial sensor and the visual sensor. In this example, if the motion state of the visual sensor is taken as the motion state of the visual inertial system, in the case that the visual sensor rotates around one of the x axis, the y axis and the z axis, the parameter corresponding to the rotation axis in the translation parameters between the inertial sensor and the visual sensor is kept fixed, and the calibration is performed on other to-be-calibrated parameters for which the calibration is started; if the motion state of the inertial sensor is taken as the motion state of the visual inertial system, the parameters corresponding to the rotation axis in the translation parameters between the inertial sensor and the visual sensor can be kept fixed under the condition that the inertial sensor rotates around one of the x axis, the y axis and the z axis, and other parameters to be calibrated, which are started to be calibrated, are calibrated.
As an example of this implementation, the determining, according to the motion state of the visual inertial system, the parameter to be calibrated that remains fixed at the current time includes: in the case where the visual sensor or the inertial sensor moves around any one coordinate axis at a constant angular velocity, determining that the parameter to be calibrated, which is kept fixed at the current time, includes a time offset and a parameter corresponding to the coordinate axis among the translational parameters between the inertial sensor and the visual sensor. In this example, if the motion state of the visual sensor is taken as the motion state of the visual inertial system, the parameters corresponding to any coordinate axis in the time offset and the translation parameters between the inertial sensor and the visual sensor can be kept fixed and other started parameters to be calibrated can be calibrated under the condition that the visual sensor moves around any coordinate axis at a constant angular velocity; if the motion state of the inertial sensor is taken as the motion state of the visual inertial system, the parameters corresponding to any coordinate axis in the time offset and the translation parameters between the inertial sensor and the visual sensor can be kept fixed under the condition that the inertial sensor moves around any coordinate axis at a constant angular velocity, and other parameters to be calibrated, which are started to be calibrated, are calibrated. In this example, it may be determined that the vision sensor moves at a constant angular velocity around any one of the coordinate axes in a case where the vision sensor moves at the same angular velocity around the coordinate axes at the current time and the previous time; it may be determined that the inertial sensor moves at a constant angular velocity around any one of the coordinate axes in a case where the inertial sensor moves at the same angular velocity around the coordinate axes at the present time and the previous time. In practical applications, an error may be allowed, and for example, the same angular velocity may be determined in the case where the difference between the angular velocities is smaller than a predetermined threshold value.
As an example of this implementation, the determining, according to the motion state of the visual inertial system, the parameter to be calibrated that remains fixed at the current time includes: under the condition that the visual sensor or the inertial sensor moves along any coordinate axis at a constant speed, determining that the parameters to be calibrated which are kept fixed at the current moment comprise time offset and parameters corresponding to the coordinate axis in translation parameters between the inertial sensor and the visual sensor. In this example, if the motion state of the visual sensor is taken as the motion state of the visual inertial system, the parameters corresponding to any coordinate axis in the time offset and the translation parameters between the inertial sensor and the visual sensor can be kept fixed and other parameters to be calibrated, which are started to be calibrated, can be calibrated under the condition that the visual sensor moves along any coordinate axis at a constant speed; if the motion state of the inertial sensor is taken as the motion state of the visual inertial system, the time offset and parameters corresponding to the coordinate axes in the translation parameters between the inertial sensor and the visual sensor can be kept fixed under the condition that the inertial sensor moves along any coordinate axis at a constant speed, and other parameters to be calibrated which are started to be calibrated are calibrated. In this example, it may be determined that the visual sensor moves at a constant speed around any one of the coordinate axes in a case where the visual sensor moves at the same speed around the coordinate axes at the present time and the previous time; it may be determined that the inertial sensor moves at a constant speed around any one of the coordinate axes in a case where the inertial sensor moves at the same speed around the coordinate axes at the present time and the previous time. In practical applications, errors may be allowed, for example, the same speed may be determined in case the difference between the speeds is smaller than a predetermined threshold.
As an example of this implementation, the determining, according to the motion state of the visual inertial system, the parameter to be calibrated that remains fixed at the current time includes: under the condition that the visual sensor or the inertial sensor moves around any coordinate axis at a constant angular acceleration, determining that the parameters to be calibrated which are kept fixed at the current moment comprise time offset and parameters corresponding to the coordinate axis in translation parameters between the inertial sensor and the visual sensor. In this example, if the motion state of the visual sensor is taken as the motion state of the visual inertial system, the parameters corresponding to any coordinate axis in the time offset and the translation parameters between the inertial sensor and the visual sensor can be kept fixed and other started and calibrated parameters to be calibrated can be calibrated under the condition that the visual sensor moves around any coordinate axis at a constant angular acceleration; if the motion state of the inertial sensor is taken as the motion state of the visual inertial system, the time offset and the parameters corresponding to the coordinate axes in the translation parameters between the inertial sensor and the visual sensor can be kept fixed under the condition that the inertial sensor moves around any coordinate axis at a constant angular acceleration, and other parameters to be calibrated which are started to be calibrated are calibrated. In this example, it may be determined that the visual sensor moves at a constant angular acceleration around any one of the coordinate axes in a case where the visual sensor moves at the same angular acceleration around the coordinate axes at the present time and the previous time; it may be determined that the inertial sensor moves at a constant angular acceleration around any one of the coordinate axes in a case where the inertial sensor moves at the same angular acceleration around the coordinate axes at the present time and the previous time. In practical applications, errors may be allowed, for example, the same angular acceleration may be determined in the case where the difference between the angular accelerations is smaller than a predetermined threshold value.
In one possible implementation, the method further includes: and determining the parameters to be calibrated which are kept fixed at the current moment according to the tracking state of the visual inertial system. In this implementation manner, the parameter to be calibrated, which is kept fixed at the current time, is determined according to the tracking state of the visual inertial system, so that only the parameter to be calibrated, which is observable at the current time, is calibrated at the current time, and the parameter to be calibrated, which is not observable at the current time, is kept fixed, so that the effectiveness of the parameter is not reduced due to the tracking state of the visual inertial system, and the precision of the calibrated parameter can be improved, thereby facilitating the improvement of the precision of the positioning, mapping, navigation and the like of the visual inertial system.
As an example of this implementation, the determining, according to the tracking state of the visual inertial system, the parameter to be calibrated that remains fixed at the current time includes: in the case of loss of tracking of the visual inertial system, determining parameters to be calibrated that remain fixed at the current time include a time offset, a translation parameter between the inertial sensor and the visual sensor, and a rotation parameter between the inertial sensor and the visual sensor. In this example, a loss of tracking may indicate that the two images most recently acquired by the vision sensor have no correlation and/or a repositioning failure. In this example, in the case of loss of tracking of the visual inertial system, the time offset, the translation parameter between the inertial sensor and the visual sensor, and the rotation parameter between the inertial sensor and the visual sensor are kept fixed, and the other to-be-calibrated parameters whose calibration has been started are calibrated.
As an example of this implementation, the determining, according to the tracking state of the visual inertial system, the parameter to be calibrated that remains fixed at the current time includes: and under the condition that the number of the characteristic points currently tracked by the vision inertial system is less than a preset number, determining that the parameters to be calibrated which are kept fixed at the current moment comprise time offset, a translation parameter between the inertial sensor and the vision sensor and a rotation parameter between the inertial sensor and the vision sensor. For example, the preset number may be 20. Of course, a person skilled in the art can flexibly set the preset number according to the requirements of the actual application scenario, and is not limited herein. In this example, when the number of feature points currently tracked by the visual inertial system is less than a preset number, the time offset, the translation parameter between the inertial sensor and the visual sensor, and the rotation parameter between the inertial sensor and the visual sensor are kept fixed, and other parameters to be calibrated, for which calibration is started, are calibrated.
In one possible implementation, after starting calibration of the parameter to be calibrated, the parameter to be calibrated may be updated once per frame according to equation 57, or may be updated once every several frames:
X calib (i+1)=ΔX calib +X calib (i) in the formula (57), the compound represented by the formula (57),
wherein, as shown in formula 58, Δ X calib The estimate of the state data of the state of the visual inertial system can be solved from a Square-Root Inverse Sliding Window Filter (SR-ISWF)Obtaining:
wherein,to representOf the optimum value, Δ X calib Represents the variation, DeltaX, of the parameter to be calibrated between the ith time and the (i +1) th time calib By calculatingThus obtaining the product. By updating the jacobian matrix (shown in formula 21) corresponding to the state data of the visual inertial system, the blackplug matrix corresponding to the visual inertial system can be obtained. Wherein R is i An upper triangular matrix being the square root of the black-plug matrix, an inverse square-root sliding window filter solutionWhen the optimal value is obtained, a Jacobian matrix of an upper triangle is obtained by adopting a least square method; r is i Is the upper triangular matrix of the residual of the blackplug matrix, is the square root inverse sliding window filter solutionThe residual term at the optimum value of (a), i.e., the error term. When in useInfinite approximation to r i When the temperature of the water is higher than the set temperature,and (4) optimizing. The square root inverse sliding window filter uses an inverse filter to iterate and then linearize.
In a possible implementation manner, the embodiments of the present disclosure may be applied to an AR (Augmented Reality) system and/or a VR (Virtual Reality) system of a mobile terminal (e.g., a mobile phone), so as to implement self-calibration of sensor parameters of a visual inertial system in the AR system and/or the VR system.
In a possible implementation manner, when the visual inertial system has offline calibrated parameters, online optimization can be performed on the offline calibrated parameters by using the embodiment of the present disclosure, so as to improve the accuracy of positioning, mapping or navigation performed by the visual inertial system.
In a possible implementation manner, in a case that the visual inertial system does not have offline calibrated parameters, the embodiment of the present disclosure may be adopted to perform self-calibration on sensor parameters of the visual inertial system, so as to improve the adaptability and the universality of the visual inertial system. For example, on a mobile phone without offline calibrated parameters, adaptation can still be achieved by using the embodiments of the present disclosure.
The parameter calibration method of the visual inertial system provided by the embodiment of the disclosure can be applied to the application fields of virtual reality, augmented reality, mixed reality, unmanned aerial vehicles, mobile robots, unmanned vehicles and the like, and is not limited herein.
It is understood that the above-mentioned method embodiments of the present disclosure can be combined with each other to form a combined embodiment without departing from the logic of the principle, which is limited by the space, and the detailed description of the present disclosure is omitted. Those skilled in the art will appreciate that in the above methods of the specific embodiments, the specific order of execution of the steps should be determined by their function and possibly their inherent logic.
In addition, the present disclosure also provides a parameter calibration apparatus of a visual inertial system, an electronic device, a computer-readable storage medium, and a program, which can be used to implement any one of the parameter calibration methods of a visual inertial system provided by the present disclosure, and corresponding technical solutions and technical effects can be referred to in corresponding descriptions of the method sections and are not described again.
Fig. 2 shows a block diagram of a parameter calibration apparatus of a visual inertial system provided in an embodiment of the present disclosure. The visual inertial system includes a visual sensor and an inertial sensor, as shown in fig. 2, the parameter calibration apparatus of the visual inertial system includes:
an obtaining module 21, configured to obtain state data of the visual inertial system at a previous time;
an estimation module 22, configured to estimate state data of the visual inertial system at the current time according to the state data of the visual inertial system at the previous time, coordinates of feature points in an image acquired by the visual sensor at the current time, and output data of the inertial sensor at the current time;
and the calibration module 23 is configured to calibrate a parameter to be calibrated of the visual inertial system according to the state data of the visual inertial system at the current time.
In one possible implementation form of the method,
the state data of the visual inertial system at the previous time comprises: the state data of the visual sensor at the previous moment, the state data of the inertial sensor at the previous moment and the numerical value of the parameter to be calibrated at the previous moment;
the state data of the visual inertial system at the current time includes: the state data of the visual sensor at the current moment, the state data of the inertial sensor at the current moment and the numerical value of the parameter to be calibrated at the current moment.
In one possible implementation form of the method,
the status data of the vision sensor at the previous time comprises: a translation parameter of the vision sensor at the previous time and a rotation parameter of the vision sensor at the previous time;
the status data of the vision sensor at the current time includes: a translation parameter of the vision sensor at the current time and a rotation parameter of the vision sensor at the current time.
In one possible implementation form of the method,
the state data of the inertial sensor at the previous time comprises: the velocity of the inertial sensor at the previous time;
the state data of the inertial sensor at the current time includes: a speed of the inertial sensor at the current time.
In one possible implementation form of the method,
the state data of the inertial sensor at the previous time further includes: zero offset of the accelerometer of the inertial sensor at the previous time instant and zero offset of the gyroscope of the inertial sensor at the previous time instant;
the state data of the inertial sensor at the current time further includes: a zero offset of the accelerometer of the inertial sensor at the current time and a zero offset of the gyroscope of the inertial sensor at the current time.
In a possible implementation manner, the parameter to be calibrated includes a time offset, wherein the time offset represents a difference between a clock of the visual sensor and a clock of the inertial sensor at the same time;
the calibration module 23 is configured to:
and in the state data of the visual inertial system at the current moment, starting calibration of the time offset under the condition that the zero offset of the accelerometer of the inertial sensor meets a first preset stability condition and the zero offset of the gyroscope of the inertial sensor meets a second preset stability condition.
In a possible implementation manner, the parameter to be calibrated includes an internal parameter of the visual sensor;
the calibration module 23 is configured to:
and in the state data of the visual inertial system at the current moment, under the condition that the zero offset of the accelerometer of the inertial sensor meets a third preset stability condition, starting calibration of the internal reference of the visual sensor, wherein the stability of the zero offset of the accelerometer corresponding to the third preset stability condition is higher than that of the accelerometer corresponding to the first preset stability condition.
In one possible implementation, the internal parameters of the vision sensor include a focal length and an optical center of the vision sensor;
the device further comprises:
the first determining module is used for determining an initial value of the focal length of the vision sensor according to the width of an image acquired by the vision sensor and the field angle of the vision sensor under the condition that the field angle of the vision sensor is obtained;
or,
and the second determining module is used for determining an initial value of the focal length of the vision sensor according to the width and the height of the image acquired by the vision sensor under the condition that the field angle of the vision sensor cannot be acquired.
In one possible implementation, the internal reference of the vision sensor includes an optical center coordinate of the vision sensor;
the device further comprises:
and the third determining module is used for determining the initial value of the optical center coordinate of the vision sensor according to the width and the height of the image acquired by the vision sensor.
In a possible implementation manner, the parameter to be calibrated includes a coordinate conversion parameter between the inertial sensor and the visual sensor, where the coordinate conversion parameter includes at least one of a translation parameter and a rotation parameter between the inertial sensor and the visual sensor;
the calibration module 23 is configured to:
and in the state data of the visual inertial system at the current moment, under the condition that the zero offset of the accelerometer of the inertial sensor meets a fourth preset stability condition, starting calibration of the coordinate conversion parameter, wherein the stability of the zero offset of the accelerometer corresponding to the fourth preset stability condition is higher than that of the accelerometer corresponding to the first preset stability condition.
In a possible implementation manner, the parameter to be calibrated includes a distortion parameter of the visual sensor;
the calibration module 23 is configured to:
and under the condition that the internal parameters of the visual sensor meet a fifth preset stability condition according to the state data of the visual inertial system at the current moment, starting calibration of distortion parameters of the visual sensor.
In one possible implementation, the apparatus further includes:
and the fourth determination module is used for determining the parameters to be calibrated which are kept fixed at the current moment according to the motion state of the visual inertial system.
In one possible implementation, the fourth determining module is configured to at least one of:
under the condition that the vision inertial system is static, determining parameters to be calibrated which are kept fixed at the current moment to comprise time offset, translation parameters between the inertial sensor and the vision sensor and rotation parameters between the inertial sensor and the vision sensor;
under the condition that the translation parameter of the visual sensor or the inertial sensor is 0 and the rotation parameter is not 0, determining that the parameter to be calibrated which is kept fixed at the current moment comprises the translation parameter between the inertial sensor and the visual sensor;
determining that the parameter to be calibrated which is kept fixed at the current moment comprises a parameter corresponding to a rotation axis in the translation parameters between the inertial sensor and the visual sensor under the condition that the visual sensor or the inertial sensor rotates around one axis of an x axis, a y axis and a z axis;
under the condition that the visual sensor or the inertial sensor moves around any coordinate axis at a constant angular speed, determining that the parameters to be calibrated which are kept fixed at the current moment comprise time offset and parameters corresponding to the coordinate axis in translation parameters between the inertial sensor and the visual sensor;
under the condition that the visual sensor or the inertial sensor moves along any coordinate axis at a constant speed, determining that the parameters to be calibrated which are kept fixed at the current moment comprise time offset and parameters corresponding to the coordinate axis in translation parameters between the inertial sensor and the visual sensor;
under the condition that the visual sensor or the inertial sensor moves around any coordinate axis at a constant angular acceleration, determining that the parameters to be calibrated which are kept fixed at the current moment comprise time offset and parameters corresponding to the coordinate axis in translation parameters between the inertial sensor and the visual sensor.
In one possible implementation, the apparatus further includes:
and the fifth determining module is used for determining the parameters to be calibrated which are kept fixed at the current moment according to the tracking state of the visual inertial system.
In one possible implementation manner, the fifth determining module is configured to:
in the case of loss of tracking of the visual inertial system, determining parameters to be calibrated that remain fixed at the current time comprise a time offset, a translation parameter between the inertial sensor and the visual sensor, and a rotation parameter between the inertial sensor and the visual sensor;
and/or the presence of a gas in the gas,
and under the condition that the number of the characteristic points currently tracked by the vision inertial system is less than a preset number, determining that the parameters to be calibrated which are kept fixed at the current moment comprise time offset, a translation parameter between the inertial sensor and the vision sensor and a rotation parameter between the inertial sensor and the vision sensor.
The embodiment of the disclosure obtains the state data of the visual inertial system at the previous moment, and according to the state data of the visual inertial system at the previous moment, the coordinates of the feature point in the image acquired by the visual sensor at the current moment, and the output data of the inertial sensor at the current moment, estimating the state data of the visual inertial system at the current moment, and according to the state data of the visual inertial system at the current moment, the parameters to be calibrated of the visual inertial system are calibrated, so that the calibration process does not need to be assisted by external equipment such as a calibration reference object, a calibration plate, a turntable and the like, the parameters to be calibrated are calibrated according to the state of the visual inertial system, so that the self-calibration of the sensor parameters of the visual inertial system can be realized, and more accurate parameters can be obtained.
In some embodiments, functions or modules included in the apparatus provided in the embodiments of the present disclosure may be used to execute the method described in the above method embodiments, and specific implementations and technical effects thereof may refer to the description of the above method embodiments, which are not described herein again for brevity.
Embodiments of the present disclosure also provide a computer-readable storage medium having stored thereon computer program instructions, which when executed by a processor, implement the above-described method. The computer readable storage medium may be a non-volatile computer readable storage medium, or may be a volatile computer readable storage medium.
The embodiment of the present disclosure also provides a computer program, which includes computer readable code, and when the computer readable code runs in an electronic device, a processor in the electronic device executes the computer program to implement the method described above.
The embodiments of the present disclosure also provide another computer program product for storing computer readable instructions, which when executed, cause a computer to perform the operations of the parameter calibration method of a visual inertial system provided in any of the embodiments.
An embodiment of the present disclosure further provides an electronic device, including: one or more processors; a memory for storing executable instructions; wherein the one or more processors are configured to invoke the memory-stored executable instructions to perform the above-described method.
The electronic device may be provided as a terminal, server, or other form of device.
Fig. 3 illustrates a block diagram of an electronic device 800 provided by an embodiment of the disclosure. For example, the electronic device 800 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, or the like terminal.
Referring to fig. 3, electronic device 800 may include one or more of the following components: processing component 802, memory 804, power component 806, multimedia component 808, audio component 810, input/output (I/O) interface 812, sensor component 814, and communication component 816.
The processing component 802 generally controls overall operation of the electronic device 800, such as operations associated with display, telephone calls, data communications, camera operations, and recording operations. The processing components 802 may include one or more processors 820 to execute instructions to perform all or a portion of the steps of the methods described above. Further, the processing component 802 can include one or more modules that facilitate interaction between the processing component 802 and other components. For example, the processing component 802 can include a multimedia module to facilitate interaction between the multimedia component 808 and the processing component 802.
The memory 804 is configured to store various types of data to support operations at the electronic device 800. Examples of such data include instructions for any application or method operating on the electronic device 800, contact data, phonebook data, messages, pictures, videos, and so forth. The memory 804 may be implemented by any type or combination of volatile or non-volatile memory devices such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic or optical disks.
The power supply component 806 provides power to the various components of the electronic device 800. The power components 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the electronic device 800.
The multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and a user. In some embodiments, the screen may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive an input signal from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front facing camera and/or a rear facing camera. The front camera and/or the rear camera may receive external multimedia data when the electronic device 800 is in an operation mode, such as a shooting mode or a video mode. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.
The audio component 810 is configured to output and/or input audio signals. For example, the audio component 810 includes a Microphone (MIC) configured to receive external audio signals when the electronic device 800 is in an operational mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals may further be stored in the memory 804 or transmitted via the communication component 816. In some embodiments, audio component 810 also includes a speaker for outputting audio signals.
The I/O interface 812 provides an interface between the processing component 802 and peripheral interface modules, which may be keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to: a home button, a volume button, a start button, and a lock button.
The sensor assembly 814 includes one or more sensors for providing various aspects of state assessment for the electronic device 800. For example, the sensor assembly 814 may detect an open/closed state of the electronic device 800, the relative positioning of components, such as a display and keypad of the electronic device 800, the sensor assembly 814 may also detect a change in the position of the electronic device 800 or a component of the electronic device 800, the presence or absence of user contact with the electronic device 800, orientation or acceleration/deceleration of the electronic device 800, and a change in the temperature of the electronic device 800. Sensor assembly 814 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 814 may also include a light sensor, such as a Complementary Metal Oxide Semiconductor (CMOS) or Charge Coupled Device (CCD) image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.
The communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 may access a wireless network based on a communication standard, such as a wireless network (Wi-Fi), a second generation mobile communication technology (2G), a third generation mobile communication technology (3G), a fourth generation mobile communication technology (4G)/long term evolution of universal mobile communication technology (LTE), a fifth generation mobile communication technology (5G), or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 further includes a Near Field Communication (NFC) module to facilitate short-range communications. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID) technology, infrared data association (IrDA) technology, Ultra Wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.
In an exemplary embodiment, the electronic device 800 may be implemented by one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), controllers, micro-controllers, microprocessors, or other electronic components for performing the above-described methods.
In an exemplary embodiment, a non-transitory computer-readable storage medium, such as the memory 804, is also provided that includes computer program instructions executable by the processor 820 of the electronic device 800 to perform the above-described methods.
Fig. 4 shows a block diagram of an electronic device 1900 provided by an embodiment of the disclosure. For example, the electronic device 1900 may be provided as a server. Referring to fig. 4, electronic device 1900 includes a processing component 1922 further including one or more processors and memory resources, represented by memory 1932, for storing instructions, e.g., applications, executable by processing component 1922. The application programs stored in memory 1932 may include one or more modules that each correspond to a set of instructions. Further, the processing component 1922 is configured to execute instructions to perform the above-described method.
The electronic device 1900 may also include a power component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input/output (I/O) interface 1958. The electronic device 1900 may operate based on an operating system, such as the Microsoft Server operating system (Windows Server), stored in the memory 1932 TM ) Apple Inc. of the present application based on the graphic user interface operating System (Mac OS X) TM ) Multi-user, multi-process computer operating system (Unix) TM ) Free and open native code Unix-like operating System (Linux) TM ) Open native code Unix-like operating System (FreeBSD) TM ) Or the like.
In an exemplary embodiment, a non-transitory computer readable storage medium, such as the memory 1932, is also provided that includes computer program instructions executable by the processing component 1922 of the electronic device 1900 to perform the above-described methods.
The present disclosure may be systems, methods, and/or computer program products. The computer program product may include a computer-readable storage medium having computer-readable program instructions embodied thereon for causing a processor to implement various aspects of the present disclosure.
The computer readable storage medium may be a tangible device that can hold and store the instructions for use by the instruction execution device. The computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, semiconductor memory device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a Static Random Access Memory (SRAM), a portable compact disc read-only memory (CD-ROM), a Digital Versatile Disc (DVD), a memory stick, a floppy disk, a mechanical coding device, such as punch cards or in-groove projection structures having instructions stored thereon, and any suitable combination of the foregoing. Computer-readable storage media as used herein is not to be construed as transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., optical pulses through a fiber optic cable), or electrical signals transmitted through electrical wires.
The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to a respective computing/processing device, or to an external computer or external storage device via a network, such as the internet, a local area network, a wide area network, and/or a wireless network. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers and/or edge servers. The network adapter card or network interface in each computing/processing device receives computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in the respective computing/processing device.
The computer program instructions for carrying out operations of the present disclosure may be assembler instructions, Instruction Set Architecture (ISA) instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider). In some embodiments, the electronic circuitry that can execute the computer-readable program instructions implements aspects of the present disclosure by utilizing the state information of the computer-readable program instructions to personalize the electronic circuitry, such as a programmable logic circuit, a Field Programmable Gate Array (FPGA), or a Programmable Logic Array (PLA).
Various aspects of the present disclosure are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the disclosure. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer-readable program instructions.
These computer-readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks. These computer-readable program instructions may also be stored in a computer-readable storage medium that can direct a computer, programmable data processing apparatus, and/or other devices to function in a particular manner, such that the computer-readable medium storing the instructions comprises an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.
The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer, other programmable apparatus or other devices implement the functions/acts specified in the flowchart and/or block diagram block or blocks.
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The computer program product may be embodied in hardware, software or a combination thereof. In an alternative embodiment, the computer program product is embodied in a computer storage medium, and in another alternative embodiment, the computer program product is embodied in a Software product, such as a Software Development Kit (SDK), or the like.
Having described embodiments of the present disclosure, the foregoing description is intended to be exemplary, not exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen in order to best explain the principles of the embodiments, the practical application, or improvements made to the technology in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.
Claims (17)
1. A method for calibrating parameters of a visual inertial system, wherein the visual inertial system comprises a visual sensor and an inertial sensor, the method comprising:
acquiring state data of the visual inertial system at the previous moment;
estimating the state data of the visual inertial system at the current moment according to the state data of the visual inertial system at the previous moment, the coordinates of the feature points in the image acquired by the visual sensor at the current moment and the output data of the inertial sensor at the current moment;
calibrating the parameters to be calibrated of the visual inertial system according to the state data of the visual inertial system at the current moment;
the parameter to be calibrated comprises a coordinate conversion parameter between the inertial sensor and the visual sensor, wherein the coordinate conversion parameter comprises at least one of a translation parameter and a rotation parameter between the inertial sensor and the visual sensor;
the calibrating the parameters to be calibrated of the visual inertial system according to the state data of the visual inertial system at the current moment comprises the following steps:
and in the state data of the visual inertial system at the current moment, under the condition that the zero offset of the accelerometer of the inertial sensor meets a fourth preset stability condition, starting calibration of the coordinate conversion parameter, wherein the stability of the zero offset of the accelerometer corresponding to the fourth preset stability condition is higher than that of the accelerometer corresponding to the first preset stability condition.
2. The method of claim 1,
the state data of the visual inertial system at the previous time comprises: the state data of the visual sensor at the previous moment, the state data of the inertial sensor at the previous moment and the numerical value of the parameter to be calibrated at the previous moment;
the state data of the visual inertial system at the current time includes: the state data of the visual sensor at the current moment, the state data of the inertial sensor at the current moment and the numerical value of the parameter to be calibrated at the current moment.
3. The method of claim 2,
the status data of the vision sensor at the previous time comprises: a translation parameter of the vision sensor at the previous time and a rotation parameter of the vision sensor at the previous time;
the status data of the vision sensor at the current time includes: a translation parameter of the vision sensor at the current time and a rotation parameter of the vision sensor at the current time.
4. The method according to claim 2 or 3,
the state data of the inertial sensor at the previous time comprises: the velocity of the inertial sensor at the previous time;
the state data of the inertial sensor at the current time includes: a speed of the inertial sensor at the current time.
5. The method of claim 4,
the state data of the inertial sensor at the previous time further includes: zero offset of the accelerometer of the inertial sensor at the previous time instant and zero offset of the gyroscope of the inertial sensor at the previous time instant;
the state data of the inertial sensor at the current time further includes: a zero offset of the accelerometer of the inertial sensor at the current time and a zero offset of the gyroscope of the inertial sensor at the current time.
6. The method according to any one of claims 1 to 5, wherein the parameter to be calibrated comprises a time offset, wherein the time offset represents a difference between a clock of the visual sensor and a clock of the inertial sensor at the same time;
the calibrating the parameters to be calibrated of the visual inertial system according to the state data of the visual inertial system at the current moment comprises the following steps:
and in the state data of the visual inertial system at the current moment, starting calibration of the time offset under the condition that the zero offset of the accelerometer of the inertial sensor meets a first preset stability condition and the zero offset of the gyroscope of the inertial sensor meets a second preset stability condition.
7. The method according to any one of claims 1 to 6, wherein the parameter to be calibrated comprises an internal parameter of the visual sensor;
the calibrating the parameters to be calibrated of the visual inertial system according to the state data of the visual inertial system at the current moment comprises the following steps:
and in the state data of the visual inertial system at the current moment, under the condition that the zero offset of the accelerometer of the inertial sensor meets a third preset stability condition, starting calibration of the internal reference of the visual sensor, wherein the stability of the zero offset of the accelerometer corresponding to the third preset stability condition is higher than that of the accelerometer corresponding to the first preset stability condition.
8. The method of claim 7, wherein the internal parameters of the vision sensor include a focal length and an optical center of the vision sensor;
prior to the initiating calibration of the internal reference of the vision sensor, the method further comprises:
under the condition that the field angle of the vision sensor is obtained, determining an initial value of the focal length of the vision sensor according to the width of an image collected by the vision sensor and the field angle of the vision sensor;
or,
and under the condition that the field angle of the vision sensor cannot be acquired, determining an initial value of the focal length of the vision sensor according to the width and the height of an image acquired by the vision sensor.
9. The method of claim 7 or 8, wherein the internal reference of the vision sensor comprises an optical center coordinate of the vision sensor;
prior to the initiating calibration of the internal reference of the vision sensor, the method further comprises:
and determining an initial value of the optical center coordinate of the vision sensor according to the width and the height of the image acquired by the vision sensor.
10. The method according to any one of claims 1 to 9, wherein the parameter to be calibrated comprises a distortion parameter of the vision sensor;
the calibrating the parameters to be calibrated of the visual inertial system according to the state data of the visual inertial system at the current moment comprises the following steps:
and under the condition that the internal parameters of the visual sensor meet a fifth preset stability condition according to the state data of the visual inertial system at the current moment, starting calibration of distortion parameters of the visual sensor.
11. The method according to any one of claims 1 to 10, further comprising:
and determining the parameters to be calibrated which are kept fixed at the current moment according to the motion state of the visual inertial system.
12. The method according to claim 11, wherein determining the parameter to be calibrated that remains fixed at the current time, based on the motion state of the visual inertial system, comprises at least one of:
under the condition that the vision inertial system is static, determining parameters to be calibrated which are kept fixed at the current moment to comprise time offset, translation parameters between the inertial sensor and the vision sensor and rotation parameters between the inertial sensor and the vision sensor;
under the condition that the translation parameter of the visual sensor or the inertial sensor is 0 and the rotation parameter is not 0, determining that the parameter to be calibrated which is kept fixed at the current moment comprises the translation parameter between the inertial sensor and the visual sensor;
determining that the parameter to be calibrated which is kept fixed at the current moment comprises a parameter corresponding to a rotation axis in the translation parameters between the inertial sensor and the visual sensor under the condition that the visual sensor or the inertial sensor rotates around one axis of an x axis, a y axis and a z axis;
under the condition that the visual sensor or the inertial sensor moves around any coordinate axis at a constant angular speed, determining that the parameters to be calibrated which are kept fixed at the current moment comprise time offset and parameters corresponding to the coordinate axis in translation parameters between the inertial sensor and the visual sensor;
under the condition that the visual sensor or the inertial sensor moves along any coordinate axis at a constant speed, determining that the parameters to be calibrated which are kept fixed at the current moment comprise time offset and parameters corresponding to the coordinate axis in translation parameters between the inertial sensor and the visual sensor;
under the condition that the visual sensor or the inertial sensor moves around any coordinate axis at a constant angular acceleration, determining that the parameters to be calibrated which are kept fixed at the current moment comprise time offset and parameters corresponding to the coordinate axis in translation parameters between the inertial sensor and the visual sensor.
13. The method according to any one of claims 1 to 12, further comprising:
and determining the parameters to be calibrated which are kept fixed at the current moment according to the tracking state of the visual inertial system.
14. The method according to claim 13, wherein said determining the parameter to be calibrated that remains fixed at the current time according to the tracking state of the visual inertial system comprises:
in the case of loss of tracking of the visual inertial system, determining parameters to be calibrated that remain fixed at the current time comprise a time offset, a translation parameter between the inertial sensor and the visual sensor, and a rotation parameter between the inertial sensor and the visual sensor;
and/or the presence of a gas in the gas,
and under the condition that the number of the characteristic points currently tracked by the vision inertial system is less than a preset number, determining that the parameters to be calibrated which are kept fixed at the current moment comprise time offset, a translation parameter between the inertial sensor and the vision sensor and a rotation parameter between the inertial sensor and the vision sensor.
15. A parameter calibration device for a visual inertial system, wherein the visual inertial system comprises a visual sensor and an inertial sensor, the device comprising:
the acquisition module is used for acquiring the state data of the visual inertial system at the last moment;
the estimation module is used for estimating the state data of the visual inertial system at the current moment according to the state data of the visual inertial system at the previous moment, the coordinates of the feature points in the image acquired by the visual sensor at the current moment and the output data of the inertial sensor at the current moment;
the calibration module is used for calibrating the parameters to be calibrated of the visual inertial system according to the state data of the visual inertial system at the current moment;
the parameter to be calibrated comprises a coordinate conversion parameter between the inertial sensor and the visual sensor, wherein the coordinate conversion parameter comprises at least one of a translation parameter and a rotation parameter between the inertial sensor and the visual sensor;
the calibration module is used for: and in the state data of the visual inertial system at the current moment, under the condition that the zero offset of the accelerometer of the inertial sensor meets a fourth preset stability condition, starting calibration of the coordinate conversion parameter, wherein the stability of the zero offset of the accelerometer corresponding to the fourth preset stability condition is higher than that of the accelerometer corresponding to the first preset stability condition.
16. An electronic device, comprising:
one or more processors;
a memory for storing executable instructions;
wherein the one or more processors are configured to invoke the memory-stored executable instructions to perform the method of any one of claims 1 to 14.
17. A computer readable storage medium having computer program instructions stored thereon, which when executed by a processor implement the method of any one of claims 1 to 14.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202210414649.2A CN114812610B (en) | 2020-11-16 | 2020-11-16 | Parameter calibration method and device of visual inertial system, electronic equipment and medium |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202011281318.3A CN112229424B (en) | 2020-11-16 | 2020-11-16 | Parameter calibration method and device for visual inertial system, electronic equipment and medium |
CN202210414649.2A CN114812610B (en) | 2020-11-16 | 2020-11-16 | Parameter calibration method and device of visual inertial system, electronic equipment and medium |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202011281318.3A Division CN112229424B (en) | 2020-11-16 | 2020-11-16 | Parameter calibration method and device for visual inertial system, electronic equipment and medium |
Publications (2)
Publication Number | Publication Date |
---|---|
CN114812610A true CN114812610A (en) | 2022-07-29 |
CN114812610B CN114812610B (en) | 2024-07-16 |
Family
ID=74123467
Family Applications (5)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202210414649.2A Active CN114812610B (en) | 2020-11-16 | 2020-11-16 | Parameter calibration method and device of visual inertial system, electronic equipment and medium |
CN202210398549.5A Pending CN114812608A (en) | 2020-11-16 | 2020-11-16 | Parameter calibration method and device for visual inertial system, electronic equipment and medium |
CN202210398676.5A Pending CN114812609A (en) | 2020-11-16 | 2020-11-16 | Parameter calibration method and device for visual inertial system, electronic equipment and medium |
CN202011281318.3A Active CN112229424B (en) | 2020-11-16 | 2020-11-16 | Parameter calibration method and device for visual inertial system, electronic equipment and medium |
CN202210398679.9A Active CN114791291B (en) | 2020-11-16 | 2020-11-16 | Parameter calibration method and device of visual inertial system, electronic equipment and medium |
Family Applications After (4)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202210398549.5A Pending CN114812608A (en) | 2020-11-16 | 2020-11-16 | Parameter calibration method and device for visual inertial system, electronic equipment and medium |
CN202210398676.5A Pending CN114812609A (en) | 2020-11-16 | 2020-11-16 | Parameter calibration method and device for visual inertial system, electronic equipment and medium |
CN202011281318.3A Active CN112229424B (en) | 2020-11-16 | 2020-11-16 | Parameter calibration method and device for visual inertial system, electronic equipment and medium |
CN202210398679.9A Active CN114791291B (en) | 2020-11-16 | 2020-11-16 | Parameter calibration method and device of visual inertial system, electronic equipment and medium |
Country Status (2)
Country | Link |
---|---|
CN (5) | CN114812610B (en) |
WO (1) | WO2022100189A1 (en) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN114812608A (en) * | 2020-11-16 | 2022-07-29 | 浙江商汤科技开发有限公司 | Parameter calibration method and device for visual inertial system, electronic equipment and medium |
Families Citing this family (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN113029134B (en) * | 2021-03-02 | 2022-04-08 | 浙江商汤科技开发有限公司 | Visual inertial system initialization method and device, electronic equipment and storage medium |
CN112989625B (en) * | 2021-04-02 | 2024-04-05 | 上海有个机器人有限公司 | Method for eliminating abnormal value of UWB sensor |
CN113932805B (en) * | 2021-10-12 | 2024-02-23 | 天翼数字生活科技有限公司 | Method for improving positioning accuracy and speed of AR virtual object |
CN115752442B (en) * | 2022-12-07 | 2024-03-12 | 运来智能装备(无锡)有限公司 | Monocular vision-based auxiliary inertial positioning method |
Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US9031809B1 (en) * | 2010-07-14 | 2015-05-12 | Sri International | Method and apparatus for generating three-dimensional pose using multi-modal sensor fusion |
WO2020087846A1 (en) * | 2018-10-31 | 2020-05-07 | 东南大学 | Navigation method based on iteratively extended kalman filter fusion inertia and monocular vision |
CN111156997A (en) * | 2020-03-02 | 2020-05-15 | 南京航空航天大学 | Vision/inertia combined navigation method based on camera internal parameter online calibration |
CN111551191A (en) * | 2020-04-28 | 2020-08-18 | 浙江商汤科技开发有限公司 | Sensor external parameter calibration method and device, electronic equipment and storage medium |
Family Cites Families (17)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN104021554B (en) * | 2014-04-23 | 2017-03-01 | 北京大学深圳研究生院 | Camera inertial sensor scaling method based on operative sensor information |
US10012504B2 (en) * | 2014-06-19 | 2018-07-03 | Regents Of The University Of Minnesota | Efficient vision-aided inertial navigation using a rolling-shutter camera with inaccurate timestamps |
EP3158293B1 (en) * | 2015-05-23 | 2019-01-23 | SZ DJI Technology Co., Ltd. | Sensor fusion using inertial and image sensors |
US10578457B2 (en) * | 2016-09-15 | 2020-03-03 | Syracuse University | Robust and stable autonomous vision-inertial navigation system for unmanned vehicles |
CN106679648B (en) * | 2016-12-08 | 2019-12-10 | 东南大学 | Visual inertia combination SLAM method based on genetic algorithm |
CN107564061B (en) * | 2017-08-11 | 2020-11-20 | 浙江大学 | Binocular vision mileage calculation method based on image gradient joint optimization |
CN108489482B (en) * | 2018-02-13 | 2019-02-26 | 视辰信息科技(上海)有限公司 | The realization method and system of vision inertia odometer |
CN109029433B (en) * | 2018-06-28 | 2020-12-11 | 东南大学 | Method for calibrating external parameters and time sequence based on vision and inertial navigation fusion SLAM on mobile platform |
CN109685852B (en) * | 2018-11-22 | 2020-08-21 | 上海肇观电子科技有限公司 | Calibration method, system, equipment and storage medium for camera and inertial sensor |
CN109443355B (en) * | 2018-12-25 | 2020-10-27 | 中北大学 | Visual-inertial tight coupling combined navigation method based on self-adaptive Gaussian PF |
CN110246194A (en) * | 2019-06-25 | 2019-09-17 | 中国人民解放军国防科技大学 | Method for quickly calibrating rotation relation between camera and inertia measurement unit |
CN110411476B (en) * | 2019-07-29 | 2021-03-23 | 视辰信息科技(上海)有限公司 | Calibration adaptation and evaluation method and system for visual inertial odometer |
CN111308415B (en) * | 2019-11-01 | 2022-09-02 | 华为技术有限公司 | Online pose estimation method and equipment based on time delay |
CN110987021B (en) * | 2019-12-25 | 2021-08-24 | 湖北航天技术研究院总体设计所 | Inertial vision relative attitude calibration method based on rotary table reference |
CN111275769B (en) * | 2020-01-17 | 2023-10-24 | 联想(北京)有限公司 | Monocular vision parameter correction method and device |
CN111429524B (en) * | 2020-03-19 | 2023-04-18 | 上海交通大学 | Online initialization and calibration method and system for camera and inertial measurement unit |
CN114812610B (en) * | 2020-11-16 | 2024-07-16 | 浙江商汤科技开发有限公司 | Parameter calibration method and device of visual inertial system, electronic equipment and medium |
-
2020
- 2020-11-16 CN CN202210414649.2A patent/CN114812610B/en active Active
- 2020-11-16 CN CN202210398549.5A patent/CN114812608A/en active Pending
- 2020-11-16 CN CN202210398676.5A patent/CN114812609A/en active Pending
- 2020-11-16 CN CN202011281318.3A patent/CN112229424B/en active Active
- 2020-11-16 CN CN202210398679.9A patent/CN114791291B/en active Active
-
2021
- 2021-08-17 WO PCT/CN2021/113113 patent/WO2022100189A1/en active Application Filing
Patent Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US9031809B1 (en) * | 2010-07-14 | 2015-05-12 | Sri International | Method and apparatus for generating three-dimensional pose using multi-modal sensor fusion |
WO2020087846A1 (en) * | 2018-10-31 | 2020-05-07 | 东南大学 | Navigation method based on iteratively extended kalman filter fusion inertia and monocular vision |
CN111156997A (en) * | 2020-03-02 | 2020-05-15 | 南京航空航天大学 | Vision/inertia combined navigation method based on camera internal parameter online calibration |
CN111551191A (en) * | 2020-04-28 | 2020-08-18 | 浙江商汤科技开发有限公司 | Sensor external parameter calibration method and device, electronic equipment and storage medium |
Non-Patent Citations (3)
Title |
---|
孙伟;宋如意;王宇航;: "视觉/惯性组合导航中的SWF与MSCKF对比研究", 中国矿业大学学报, no. 01, 15 January 2020 (2020-01-15) * |
杨东方: "一种新的惯性/视觉组合***初始对准的时域约束方法", 仪器仪表学报, no. 2014, 15 May 2014 (2014-05-15), pages 788 - 793 * |
王晨琳;刘海颖;蒋鑫;: "一种单目视觉信息辅助惯性导航算法", 导航定位学报, no. 04, 29 November 2018 (2018-11-29) * |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN114812608A (en) * | 2020-11-16 | 2022-07-29 | 浙江商汤科技开发有限公司 | Parameter calibration method and device for visual inertial system, electronic equipment and medium |
CN114812609A (en) * | 2020-11-16 | 2022-07-29 | 浙江商汤科技开发有限公司 | Parameter calibration method and device for visual inertial system, electronic equipment and medium |
Also Published As
Publication number | Publication date |
---|---|
WO2022100189A1 (en) | 2022-05-19 |
CN114791291B (en) | 2024-06-07 |
CN112229424B (en) | 2022-04-22 |
CN114812608A (en) | 2022-07-29 |
CN114812609A (en) | 2022-07-29 |
CN112229424A (en) | 2021-01-15 |
CN114812610B (en) | 2024-07-16 |
CN114791291A (en) | 2022-07-26 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN112229424B (en) | Parameter calibration method and device for visual inertial system, electronic equipment and medium | |
CN111551191B (en) | Sensor external parameter calibration method and device, electronic equipment and storage medium | |
US20210158560A1 (en) | Method and device for obtaining localization information and storage medium | |
CN109584362B (en) | Three-dimensional model construction method and device, electronic equipment and storage medium | |
CN114814872A (en) | Pose determination method and device, electronic equipment and storage medium | |
CN111625764B (en) | Mobile data calibration method, device, electronic equipment and storage medium | |
CN113074726A (en) | Pose determination method and device, electronic equipment and storage medium | |
CN110986930A (en) | Equipment positioning method and device, electronic equipment and storage medium | |
JP7182020B2 (en) | Information processing method, device, electronic device, storage medium and program | |
CN112146645A (en) | Method and device for aligning coordinate system, electronic equipment and storage medium | |
WO2022134475A1 (en) | Point cloud map construction method and apparatus, electronic device, storage medium and program | |
CN111860373B (en) | Target detection method and device, electronic equipment and storage medium | |
CN112432637A (en) | Positioning method and device, electronic equipment and storage medium | |
CN112945207A (en) | Target positioning method and device, electronic equipment and storage medium | |
CN113703519A (en) | Method and device for determining posture of folding screen and storage medium | |
CN106875450B (en) | Training set optimization method and device for camera reorientation | |
CN111928861B (en) | Map construction method and device | |
CN114170308A (en) | All-in-one machine pose true value calculating method and device, electronic equipment and storage medium | |
CN113345000A (en) | Depth detection method and device, electronic equipment and storage medium | |
CN112767541A (en) | Three-dimensional reconstruction method and device, electronic equipment and storage medium | |
CN112837372A (en) | Data generation method and device, electronic equipment and storage medium | |
CN112432636A (en) | Positioning method and device, electronic equipment and storage medium | |
WO2022110777A1 (en) | Positioning method and apparatus, electronic device, storage medium, computer program product, and computer program | |
CN113608616A (en) | Virtual content display method and device, electronic equipment and storage medium | |
CN113192139A (en) | Positioning method and device, electronic equipment and storage medium |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant |