WO2022100189A1 - Method and apparatus for calibrating parameters of visual-inertial system, and electronic device and medium - Google Patents
Method and apparatus for calibrating parameters of visual-inertial system, and electronic device and medium Download PDFInfo
- Publication number
- WO2022100189A1 WO2022100189A1 PCT/CN2021/113113 CN2021113113W WO2022100189A1 WO 2022100189 A1 WO2022100189 A1 WO 2022100189A1 CN 2021113113 W CN2021113113 W CN 2021113113W WO 2022100189 A1 WO2022100189 A1 WO 2022100189A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- sensor
- visual
- inertial
- parameters
- current moment
- Prior art date
Links
- 238000000034 method Methods 0.000 title claims abstract description 83
- 230000000007 visual effect Effects 0.000 claims abstract description 545
- 238000013519 translation Methods 0.000 claims description 102
- 230000033001 locomotion Effects 0.000 claims description 59
- 230000003287 optical effect Effects 0.000 claims description 29
- 238000003860 storage Methods 0.000 claims description 26
- 238000006243 chemical reaction Methods 0.000 claims description 22
- 238000004590 computer program Methods 0.000 claims description 20
- 230000001133 acceleration Effects 0.000 claims description 16
- 230000000875 corresponding effect Effects 0.000 description 60
- 238000012545 processing Methods 0.000 description 25
- 239000011159 matrix material Substances 0.000 description 18
- 238000010586 diagram Methods 0.000 description 16
- 230000000694 effects Effects 0.000 description 14
- 238000004891 communication Methods 0.000 description 13
- 238000005516 engineering process Methods 0.000 description 10
- 230000006870 function Effects 0.000 description 10
- 238000013507 mapping Methods 0.000 description 9
- 230000002829 reductive effect Effects 0.000 description 7
- 239000006185 dispersion Substances 0.000 description 6
- 230000008569 process Effects 0.000 description 6
- 230000003190 augmentative effect Effects 0.000 description 5
- 238000010276 construction Methods 0.000 description 5
- 230000004807 localization Effects 0.000 description 5
- 230000005540 biological transmission Effects 0.000 description 4
- 230000005236 sound signal Effects 0.000 description 4
- 230000009466 transformation Effects 0.000 description 4
- 230000009471 action Effects 0.000 description 3
- 230000003068 static effect Effects 0.000 description 3
- 238000003491 array Methods 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 239000000835 fiber Substances 0.000 description 2
- 230000004927 fusion Effects 0.000 description 2
- 238000003384 imaging method Methods 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 239000004973 liquid crystal related substance Substances 0.000 description 2
- 238000005259 measurement Methods 0.000 description 2
- 238000010295 mobile communication Methods 0.000 description 2
- 230000001902 propagating effect Effects 0.000 description 2
- 239000004065 semiconductor Substances 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- 238000011179 visual inspection Methods 0.000 description 2
- RYGMFSIKBFXOCR-UHFFFAOYSA-N Copper Chemical compound [Cu] RYGMFSIKBFXOCR-UHFFFAOYSA-N 0.000 description 1
- 230000006978 adaptation Effects 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 230000000295 complement effect Effects 0.000 description 1
- 229910052802 copper Inorganic materials 0.000 description 1
- 239000010949 copper Substances 0.000 description 1
- 230000002596 correlated effect Effects 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000006073 displacement reaction Methods 0.000 description 1
- 230000005484 gravity Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000000670 limiting effect Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000005055 memory storage Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 230000036961 partial effect Effects 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 238000005295 random walk Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 230000002441 reversible effect Effects 0.000 description 1
- 230000001052 transient effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
- G01C25/00—Manufacturing, calibrating, cleaning, or repairing instruments or devices referred to in the other groups of this subclass
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
- G01C21/00—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00
- G01C21/10—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 by using measurements of speed or acceleration
- G01C21/12—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 by using measurements of speed or acceleration executed aboard the object being navigated; Dead reckoning
- G01C21/16—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 by using measurements of speed or acceleration executed aboard the object being navigated; Dead reckoning by integrating acceleration or speed, i.e. inertial navigation
- G01C21/165—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 by using measurements of speed or acceleration executed aboard the object being navigated; Dead reckoning by integrating acceleration or speed, i.e. inertial navigation combined with non-inertial navigation instruments
Definitions
- the present disclosure relates to the technical field of computer vision, and in particular, to a method and device for calibrating parameters of a visual inertial system, an electronic device and a medium.
- Visual-Inertial Simultaneous Localization And Mapping VI-SLAM
- VI-SLAM Visual-Inertial Simultaneous Localization And Mapping
- the visual-inertial system can be positioned with high precision, and an accurate three-dimensional environment map can be constructed. It is of great significance for the visual inertial system to calibrate the sensor parameters of the visual inertial system.
- the embodiments of the present disclosure provide a technical solution for parameter calibration of a visual inertial system.
- a method for calibrating parameters of a visual-inertial system includes a visual sensor and an inertial sensor, and the method includes:
- the coordinates of the feature points in the image collected by the visual sensor at the current moment, and the output data of the inertial sensor at the current moment estimate the State data of the visual inertial system at the current moment
- the to-be-calibrated parameters of the visual-inertial system are calibrated according to the state data of the visual-inertial system at the current moment.
- the state data of the visual inertial system at the last moment includes: the state data of the visual sensor at the last moment, the state data of the inertial sensor at the last moment status data and the value of the parameter to be calibrated at the previous moment;
- the status data of the visual inertial system at the current moment includes: the status data of the visual sensor at the current moment, the inertial sensor at the current moment The state data at the current moment and the value of the parameter to be calibrated at the current moment.
- the state data of the vision sensor at the last moment includes: translation parameters of the vision sensor at the last moment and rotation of the vision sensor at the last moment parameter;
- the state data of the visual sensor at the current moment includes: the translation parameter of the visual sensor at the current moment and the rotation parameter of the visual sensor at the current moment.
- the state data of the inertial sensor at the last moment includes: the speed of the inertial sensor at the last moment; the state data of the inertial sensor at the current moment includes: : The speed of the inertial sensor at the current moment.
- the state data of the inertial sensor at the last moment further includes: the zero offset of the accelerometer of the inertial sensor at the last moment and the zero offset of the inertial sensor at the last moment The zero offset of the gyroscope at a moment;
- the state data of the inertial sensor at the current moment further includes: the zero offset of the accelerometer of the inertial sensor at the current moment and the zero offset of the inertial sensor at the current moment The zero offset of the gyroscope.
- the parameter to be calibrated includes a time offset, wherein the time offset represents a difference between the clock of the visual sensor and the clock of the inertial sensor at the same moment;
- the step of calibrating the parameters to be calibrated of the visual inertial system according to the state data of the visual inertial system at the current moment includes: in the state data of the visual inertial system at the current moment, the inertial When the zero offset of the accelerometer of the sensor satisfies the first preset stability condition, and the zero offset of the gyroscope of the inertial sensor satisfies the second preset stability condition, the calibration of the time offset is enabled.
- the first preset stability condition is a preset stability condition of the bias of the accelerometer
- the second preset stability condition is a preset stability condition of the bias of the gyroscope.
- the parameters to be calibrated include internal parameters of the visual sensor; and the parameters to be calibrated of the visual inertial system are performed according to the state data of the visual inertial system at the current moment.
- Calibration includes: in the state data of the visual inertial system at the current moment, in the case that the zero offset of the accelerometer of the inertial sensor satisfies a third preset stability condition, turning on the internal reference of the visual sensor The calibration of the accelerometer, wherein the stability of the bias of the accelerometer corresponding to the third preset stability condition is higher than the stability of the bias of the accelerometer corresponding to the first preset stability condition.
- the stability of the visual inertial system is relatively high, and the calibration of the internal parameters of the visual sensor is started at this time, which can make the current visual
- the inertial system maintains a relatively stable state, so that a better calibration effect can be obtained.
- the internal parameters of the visual sensor include the focal length and the optical center of the visual sensor; before starting the calibration of the internal parameters of the visual sensor, the method further includes: after obtaining the obtained In the case of the visual field angle of the visual sensor, the initial value of the focal length of the visual sensor is determined according to the width of the image collected by the visual sensor and the visual field angle of the visual sensor; In the case of the field angle of the vision sensor, the initial value of the focal length of the vision sensor is determined according to the width and height of the image collected by the vision sensor.
- the parameter calibration of the visual inertial system is performed according to the initial value of the focal length of the visual sensor determined in this implementation manner, which helps to improve the calibration accuracy.
- the internal reference of the visual sensor includes the optical center coordinate of the visual sensor; before the enabling calibration of the internal reference of the visual sensor, the method further includes: according to the visual sensor The width and height of the image collected by the sensor determine the initial value of the optical center coordinates of the vision sensor. The parameters of the visual inertial system are calibrated according to the initial value of the optical center coordinate of the vision sensor determined by this implementation, which helps to improve the calibration accuracy.
- the parameter to be calibrated includes a coordinate conversion parameter between the inertial sensor and the visual sensor, wherein the coordinate conversion parameter includes a parameter between the inertial sensor and the visual sensor at least one of the translation parameters and rotation parameters of the visual inertial system; the calibration of the parameters to be calibrated of the visual inertial system according to the state data of the visual inertial system at the current moment, including: in the visual inertial system In the state data at the current moment, when the zero offset of the accelerometer of the inertial sensor satisfies a fourth preset stability condition, the calibration of the coordinate conversion parameter is enabled, wherein the fourth preset stability condition is The stability of the zero offset of the accelerometer corresponding to the stability condition is higher than the stability of the zero offset of the accelerometer corresponding to the first preset stability condition.
- the adjustment of the coordinate system is enabled.
- the calibration of the conversion parameters helps to improve the convergence speed of the to-be-calibrated parameters of the visual inertial system, and improves the accuracy and robustness of parameter calibration.
- the stability of the visual inertial system is relatively high, and at this timing, the communication between the inertial sensor and the visual sensor is turned on.
- the calibration of the coordinate transformation parameters can keep the current visual inertial system in a relatively stable state, so that a better calibration effect can be obtained.
- the parameter to be calibrated includes a distortion parameter of the visual sensor; the parameter to be calibrated for the visual inertial system according to the state data of the visual inertial system at the current moment
- the calibration includes: in the case where it is determined that the internal parameter of the visual sensor satisfies the fifth preset stability condition according to the state data of the visual inertial system at the current moment, turning on the distortion parameter of the visual sensor. Calibration.
- the distortion of the visual sensor is enabled when it is determined that the internal parameter of the visual sensor meets the fifth preset stability condition according to the state data of the visual inertial system at the current moment.
- the calibration of parameters helps to improve the convergence speed of the parameters to be calibrated in the visual inertial system, and improves the accuracy and robustness of parameter calibration.
- the internal reference of the vision sensor is calibrated prior to the distortion parameter of the vision sensor. In the case that the internal reference of the visual sensor satisfies the fifth preset stability condition, the internal reference of the visual sensor has converged. Turning on the calibration of the distortion parameters of the vision sensor at this time will not affect the calibration effect of the internal parameters of the vision sensor.
- the method further includes: determining the constant to-be-calibrated parameter at the current moment according to the motion state of the visual inertial system.
- determining the parameters to be calibrated at the current moment that remain fixed according to the motion state of the visual inertial system by determining the parameters to be calibrated at the current moment that remain fixed according to the motion state of the visual inertial system, only the parameters to be calibrated that are observable at the current moment are calibrated at the current moment, and the parameters to be calibrated that are observable at the current moment are calibrated.
- the parameters to be calibrated that cannot be observed at the current moment are kept fixed, so that the validity of the parameters will not be reduced due to the motion state of the visual inertial system, and the accuracy of the calibrated parameters can be improved, thereby helping to improve the positioning and construction of the visual inertial system. Accuracy of graphics, navigation, etc.
- the determining a parameter to be calibrated that remains fixed at the current moment according to the motion state of the visual-inertial system includes at least one of the following: when the visual-inertial system is stationary , determine that the parameters to be calibrated that remain fixed at the current moment include time offset, translation parameters between the inertial sensor and the visual sensor, and rotation parameters between the inertial sensor and the visual sensor; In the case where the translation parameter of the visual sensor or the inertial sensor is 0 and the rotation parameter is not 0, it is determined that the parameter to be calibrated that remains fixed at the current moment includes the translation parameter between the inertial sensor and the visual sensor ; In the case where the visual sensor or the inertial sensor rotates around one of the x-axis, the y-axis and the z-axis, it is determined that the parameters to be calibrated that remain fixed at the current moment include the inertial sensor and the The parameter corresponding to the rotation axis in the translation parameters between the visual sensors; in the case that the visual sensor or the iner
- the unobservable parameters to be calibrated at the current moment can be kept fixed, so that the validity of the parameters will not be reduced due to the motion state of the visual inertial system, and the accuracy of the calibrated parameters can be improved.
- the method further includes: determining the constant to-be-calibrated parameter at the current moment according to the tracking state of the visual-inertial system.
- determining the constant to-be-calibrated parameters at the current moment according to the tracking state of the visual-inertial system by determining the constant to-be-calibrated parameters at the current moment according to the tracking state of the visual-inertial system, only the to-be-calibrated parameters that are observable at the current moment are calibrated at the current moment, and the The parameters to be calibrated that cannot be observed at the current moment are kept fixed, so that the effectiveness of the parameters will not be reduced due to the tracking state of the visual inertial system, and the accuracy of the calibrated parameters can be improved, thereby helping to improve the positioning and construction of the visual inertial system. Accuracy of graphics, navigation, etc.
- the determining the constant to-be-calibrated parameter at the current moment according to the tracking state of the visual-inertial system includes: in the case that the tracking of the visual-inertial system is lost, determining when the visual-inertial system is lost.
- the parameters to be calibrated that are kept fixed at the current moment include time offset, translation parameters between the inertial sensor and the visual sensor, and rotation parameters between the inertial sensor and the visual sensor; and/or, in In the case where the number of feature points currently tracked by the visual inertial system is less than the preset number, it is determined that the parameters to be calibrated that remain fixed at the current moment include the time offset, the time offset between the inertial sensor and the visual sensor.
- Translation parameters and rotation parameters between the inertial sensor and the vision sensor According to this implementation, the parameters to be calibrated that cannot be observed at the current moment can be kept fixed, so that the effectiveness of the parameters will not be reduced due to the tracking state of the visual inertial system, and the accuracy of the calibrated parameters can be improved.
- a parameter calibration device for a visual inertial system includes a visual sensor and an inertial sensor, and the device includes:
- an acquisition module configured to acquire the state data of the visual inertial system at the last moment
- an estimation module configured to be based on the state data of the visual inertial system at the last moment, the coordinates of the feature points in the image collected by the visual sensor at the current moment, and the output of the inertial sensor at the current moment data, estimating the state data of the visual-inertial system at the current moment;
- the calibration module is configured to calibrate the parameters to be calibrated of the visual inertial system according to the state data of the visual inertial system at the current moment.
- an electronic device comprising: one or more processors; a memory configured to store executable instructions; wherein the one or more processors are configured to invoke the memory storage executable instructions to perform the above method.
- a computer-readable storage medium having computer program instructions stored thereon, the computer program instructions implementing the above method when executed by a processor.
- a computer program comprising computer-readable code, when the computer-readable code is executed in an electronic device, a processor in the electronic device executes and is configured to implement the above method .
- the characteristics of the feature points in the image collected by the visual sensor at the current moment are coordinates, and the output data of the inertial sensor at the current moment, estimate the state data of the visual inertial system at the current moment, and according to the state data of the visual inertial system at the current moment, the
- the parameters to be calibrated of the visual inertial system are calibrated. Therefore, in the calibration process, there is no need to use external equipment such as calibration reference objects, calibration plates, and turntables for assistance.
- the sensor parameters of the system are self-calibrated, and more accurate parameters can be obtained.
- FIG. 1 shows a flowchart of a method for calibrating parameters of a visual inertial system provided by an embodiment of the present disclosure.
- FIG. 2 shows a block diagram of a parameter calibration device for a visual inertial system provided by an embodiment of the present disclosure.
- FIG. 3 shows a block diagram of an electronic device provided by an embodiment of the present disclosure.
- FIG. 4 shows a block diagram of an electronic device provided by an embodiment of the present disclosure.
- the term "at least one” herein refers to any combination of any one of the plurality or at least two of the plurality, for example, including at least one of A, B, and C, and may mean including from A, B, and C. Any one or more elements selected from the set of B and C.
- numerous details are given in the following embodiments. It will be understood by those skilled in the art that the present disclosure may be practiced without certain details. In some instances, methods, means, components and circuits well known to those skilled in the art have not been described in detail so as not to obscure the subject matter of the present disclosure.
- the visual inertial system needs to use an offline calibration method to calibrate the parameters of the sensor in advance.
- This calibration method needs to rely on expensive equipment and professional technicians, and configure the calibrated parameters into the visual inertial system after the calibration is completed.
- the calibrated parameters are difficult to adapt to different operating environments.
- the parameters calibrated offline often have errors and cannot be adjusted online in real time.
- the mechanical structure of the equipment where the visual inertial system is located changes, or the focal length of the visual sensor can be changed, or the visual sensor and the inertial sensor are not rigidly fixed
- the pre-calibrated parameters usually cannot meet the current needs.
- the visual-inertial system in related technologies will not be able to be implemented in application scenarios such as virtual reality, augmented reality, mixed reality, drones, mobile robots, and unmanned vehicles.
- Embodiments of the present disclosure provide a method and device for calibrating parameters of a visual inertial system, an electronic device, and a medium, which can self-calibrate parameters to be calibrated in the visual inertial system.
- the parameters to be calibrated can be estimated online in the process of positioning, mapping or navigation; when the visual inertial system has offline calibrated parameters, it can be calibrated offline. Good parameters are optimized to improve the accuracy of positioning, mapping or navigation.
- FIG. 1 shows a flowchart of a method for calibrating parameters of a visual inertial system provided by an embodiment of the present disclosure.
- the visual-inertial system includes a visual sensor and an inertial sensor.
- the execution subject of the parameter calibration method of the visual inertial system may be a parameter calibration device of the visual inertial system.
- the parameter calibration device of the visual-inertial system may be any device including the visual-inertial system.
- the parameter calibration method of the visual inertial system may be executed by a terminal device or a server or other processing device.
- the terminal device may be a user equipment (User Equipment, UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (Personal Digital Assistant, PDA), a handheld device, a computing device, a vehicle-mounted device, or a wearable devices, etc.
- the parameter calibration method of the visual-inertial system may be implemented by the processor calling computer-readable instructions stored in the memory. As shown in FIG. 1 , the parameter calibration method of the visual inertial system includes steps S11 to S13.
- step S11 the state data of the visual inertial system at the last moment is acquired.
- a visual-inertial system may represent a system that combines visual sensors and inertial sensors to estimate displacement and attitude, thereby performing at least one of positioning, mapping, navigation, and the like.
- Visual-inertial systems can sometimes be called visual-inertial odometry (VIO), visual-inertial navigation systems (VINS), visual-inertial SLAM (Visual-Inertial Simultaneous Localization And Mapping, VI- SLAM, simultaneous localization and map construction using fusion of visual and inertial sensors) systems, etc.
- the visual sensor may refer to an instrument that uses optical elements and imaging devices to obtain image information of the external environment.
- the vision sensor may be a monocular vision sensor, a binocular vision sensor, a multi-eye vision sensor, an RGBD (Red, Red; Green, Green; Blue, Blue; Deep, depth) vision sensor and the like.
- the visual sensor may use a monocular camera, a binocular camera, a multi-eye camera, or an RGBD camera.
- the inertial sensor may refer to a sensor that uses the inertial force of the sensing mass to measure.
- the inertial sensor may include a three-axis accelerometer and a three-axis gyroscope.
- the inertial sensor may also be referred to as an inertial measurement unit (Inertial Measurement Unit, IMU).
- IMU inertial Measurement Unit
- the inertial sensor may use a consumer-grade IMU including an accelerometer and a gyroscope, and may also use a high-precision inertial navigation system, a strapdown inertial navigation system, and the like.
- the state data of the visual-inertial system at the last moment may indicate that the state of the visual-inertial system is estimated at the last moment, and the obtained state data indicates that the visual-inertial system is at the The data of the state at the last moment.
- step S12 according to the state data of the visual inertial system at the last moment, the coordinates of the feature points in the image collected by the visual sensor at the current moment, and the output of the inertial sensor at the current moment data, and estimate the state data of the visual-inertial system at the current moment.
- the time difference between two adjacent moments may be greater than or equal to the inverse of the frequency at which the vision sensor collects images.
- the time difference between two adjacent moments may be equal to the inverse of the frequency at which the vision sensor collects images.
- the state data of the visual-inertial system can be re-estimated each time a new image is acquired by the vision sensor.
- the time difference between two adjacent moments may be equal to K times the inverse of the frequency at which the vision sensor collects images, where K is an integer greater than 1.
- the last moment may be recorded as the i-th moment
- the current moment may be recorded as the i+1-th moment.
- the image collected by the visual sensor and the data output by the inertial sensor will change.
- the feature points in the image collected by the vision sensor may include corner points, key points, etc. in the image collected by the vision sensor, which are not limited herein.
- the data output by the inertial sensor may include data output by an accelerometer of the inertial sensor and/or data output by a gyroscope of the inertial sensor.
- the coordinates of multiple feature points in the image collected by the visual sensor at the current moment and the data output by the inertial sensor can be used to determine the state of the visual-inertial system at the current moment. data are estimated.
- the state data of the visual-inertial system at the current moment may indicate that the state of the visual-inertial system is estimated at the current moment, and the obtained state data representing the state of the visual-inertial system at the current moment data.
- step S13 the parameters to be calibrated of the visual inertial system are calibrated according to the state data of the visual inertial system at the current moment.
- the parameters to be calibrated of the visual inertial system may include internal parameters of the visual sensor, distortion parameters of the visual sensor, coordinate conversion parameters between the inertial sensor and the visual sensor, and all parameters of the visual sensor. at least one of a time offset between the visual sensor and the inertial sensor, and the like.
- the internal parameters of the visual sensor may include the focal lengths f x and f y of the visual sensor, and the optical center coordinates c x and cy of the visual sensor.
- the distortion parameters of the vision sensor may include radial distortion parameters k 1 and k 2 of the vision sensor, and tangential distortion parameters p 1 and p 2 of the vision sensor.
- the coordinate conversion parameters between the inertial sensor and the visual sensor may include translation parameters between the inertial sensor and the visual sensor and/or rotation parameters between the inertial sensor and the visual sensor.
- the coordinate conversion parameter between the inertial sensor and the visual sensor may include a translation parameter from the inertial sensor to the visual sensor and/or a rotation parameter from the inertial sensor to the visual sensor, or, the The coordinate conversion parameters between the inertial sensor and the visual sensor may include translation parameters from the visual sensor to the inertial sensor and/or rotation parameters from the visual sensor to the inertial sensor.
- the coordinate conversion parameter between the inertial sensor and the visual sensor may include a translation parameter (p extri ) from the inertial sensor to the visual sensor and a rotation parameter (q) from the inertial sensor to the visual sensor extri ).
- the translation parameter from the inertial sensor to the visual sensor may represent the translational component of the movement from the coordinate system of the inertial sensor to the camera coordinate system of the visual sensor; the translation parameter from the inertial sensor to the visual sensor
- the rotation parameter may represent the rotational component of the motion transformed from the coordinate system of the inertial sensor to the camera coordinate system of the vision sensor.
- the time offset between the visual sensor and the inertial sensor can be denoted as t d , which can represent the difference between the clock of the visual sensor and the clock of the inertial sensor at the same time.
- the embodiment of the present disclosure obtains the state data of the visual inertial system at the last moment, according to the state data of the visual inertial system at the last moment, the coordinates of the feature points in the image collected by the visual sensor at the current moment, and the inertial sensor at the current moment.
- the output data of the current moment estimate the state data of the visual inertial system at the current moment, and calibrate the parameters to be calibrated of the visual inertial system according to the state data of the visual inertial system at the current moment, Therefore, in the calibration process, there is no need to use external equipment such as calibration reference objects, calibration plates, turntables, etc. to assist, and the parameters to be calibrated are calibrated according to the state of the visual inertial system, so that the self-calibration of the sensor parameters of the visual inertial system can be realized, and can Get more precise parameters.
- the state data of the visual inertial system at the last moment includes: the state data of the visual sensor at the last moment, the state data of the inertial sensor at the last moment status data and the value of the parameter to be calibrated at the previous moment;
- the status data of the visual inertial system at the current moment includes: the status data of the visual sensor at the current moment, the inertial sensor at the current moment The state data at the current moment and the value of the parameter to be calibrated at the current moment.
- the state data of the visual sensor may represent data that can reflect the motion state of the visual sensor.
- the state data of the vision sensor at the last moment may represent data that can reflect the motion state of the vision sensor at the last moment; the state data of the vision sensor at the current moment may represent Data that can reflect the motion state of the visual sensor at the current moment.
- the state data of the inertial sensor may represent data capable of representing the state of the inertial sensor.
- the state data of the inertial sensor at the last moment may represent data that can reflect the state of the inertial sensor at the last moment; the state data of the inertial sensor at the current moment may represent data that can reflect the state of the inertial sensor at the current moment.
- Data representing the state of the inertial sensor at the current moment Based on the state data of the visual inertial system in this implementation manner, self-calibration of the sensor parameters of the visual inertial system can be performed, and more accurate parameters can be obtained.
- the state data of the vision sensor at the last moment includes: translation parameters of the vision sensor at the last moment and rotation parameters of the vision sensor at the last moment ;
- the state data of the vision sensor at the current moment includes: translation parameters of the vision sensor at the current moment and rotation parameters of the vision sensor at the current moment.
- the state data of the vision sensor includes translation parameters and rotation parameters of the vision sensor.
- the self-calibration of the sensor parameters of the visual-inertial system based on the state data of the visual sensor in this example helps to improve the calibration effect.
- the state data of the vision sensor may only include translation parameters of the vision sensor, or may only include rotation parameters of the vision sensor, or may include other parameters of the vision sensor, which are not limited herein.
- the state data of the inertial sensor at the last moment includes: the speed of the inertial sensor at the last moment; the state data of the inertial sensor at the current moment includes: The speed of the inertial sensor at the current moment.
- the state data of the inertial sensor includes the speed of the inertial sensor.
- the state data of the inertial sensor at the last moment further includes: the zero offset of the accelerometer of the inertial sensor at the last moment and the gyro of the inertial sensor at the last moment
- the state data of the inertial sensor at the current moment further includes: the zero offset of the accelerometer of the inertial sensor at the current moment and the zero offset of the gyroscope of the inertial sensor at the current moment partial.
- the inertial sensor state data includes the inertial sensor's velocity, the accelerometer's bias, and the gyroscope's bias.
- the zero offset of the accelerometer represents the output of the accelerometer when the input of the accelerometer is 0, and the zero offset of the gyroscope represents the output of the gyroscope when the input of the gyroscope is 0.
- the self-calibration of the sensor parameters of the visual inertial system based on the state data of the inertial sensor in this example is helpful to further improve the calibration effect of the visual inertial system using consumer-grade inertial sensors.
- the state data of the inertial sensor may include the speed of the inertial sensor and the bias of the accelerometer, but not the bias of the gyroscope.
- the state data of the inertial sensor may include the speed of the inertial sensor and the bias of the gyroscope, but not the bias of the accelerometer.
- the state data of the inertial sensor may only include the zero offset of the accelerometer or the zero offset of the gyroscope.
- the translation parameter of the inertial sensor may be used to replace the translation parameter of the visual sensor
- the rotation parameter of the inertial sensor may be used to replace the rotation of the visual sensor parameters
- the speed of the visual sensor may be used to replace the speed of the inertial sensor
- other parameters that measure the stability and/or accuracy of the inertial sensor may be used to replace the speed of the accelerometer of the inertial sensor.
- the state data of the visual inertial sensor may include translation parameters of the visual sensor, rotation parameters of the visual sensor, speed of the inertial sensor, zero offset of the accelerometer of the inertial sensor, and gyroscope of the inertial sensor.
- the zero offset and the parameters to be calibrated For example, for consumer-grade inertial sensors in mobile phones and other devices, the zero bias of the accelerometer and the zero bias of the gyroscope can be added to the state data of the visual inertial sensor.
- the state data of the visual inertial sensor may include translation parameters of the visual sensor, rotation parameters of the visual sensor, speed of the inertial sensor, and parameters to be calibrated. For example, for a high-precision inertial sensor, the zero offset of the accelerometer and the zero offset of the gyroscope can be ignored.
- the state data of the visual-inertial system can be constructed as shown in Equation 1:
- ( ⁇ )C can represent the camera coordinate system of the vision sensor
- ( ⁇ )I can represent the coordinate system of the inertial sensor.
- Both the visual sensor and the inertial sensor can be regarded as a rigid body, and the motion of the rigid body can be six degrees of freedom, namely translational motion with three degrees of freedom and rotational motion with three degrees of freedom;
- represents the translation parameter of the vision sensor can be a three-dimensional parameter, can represent the translation parameters of the vision sensor at the previous moment, It can represent the translation parameters of the vision sensor at the current moment;
- represents the rotation parameter of the vision sensor can be a three-dimensional parameter, It can represent the rotation parameter of the vision sensor at the last moment, It can represent the rotation parameters of the vision sensor at the current moment;
- represents the velocity of the inertial sensor can be a three-dimensional parameter, It can represent the speed of the inertial sensor at the last moment, It can represent the speed of the inertial sensor at the current moment; represents the zero offset of the inertial sensor's accelerometer, can be
- f x , f y , c x , cy , k 1 , k 2 , p 1 and p 2 are one-dimensional parameters respectively, f x and f y are the focal length of the vision sensor, and c x and c y are the visual
- the optical center coordinates of the sensor, k 1 and k 2 are the radial distortion of the vision sensor, p 1 and p 2 are the tangential distortion of the vision sensor;
- p extri and q extri can be three-dimensional parameters, respectively, from the inertial sensor to the vision sensor
- the translation parameter and rotation parameter of the sensor; t d is a one-dimensional parameter, which represents the difference between the visual sensor clock and the inertial sensor clock of the visual inertial system at the same time, that is, the time offset between the visual sensor and the inertial sensor. shift.
- t d can be determined using Equation 3:
- t C represents the clock of the visual sensor and t I represents the clock of the inertial sensor.
- the value of the parameter calibrated offline can be used as the initial value of the parameter, and the value of the parameter can be updated through self-calibration; In the case of parameters, you can set the initial value of the parameter to be calibrated.
- the internal reference of the visual sensor includes the focal length of the visual sensor; before the enabling calibration of the internal reference of the visual sensor, the method further includes: after obtaining the visual sensor In the case of the visual field angle of the sensor, the initial value of the focal length of the visual sensor is determined according to the width of the image collected by the visual sensor and the visual field angle of the visual sensor; or, if the visual sensor cannot be obtained In the case of an angle of view of , the initial value of the focal length of the vision sensor is determined according to the width and height of the image collected by the vision sensor.
- the parameter calibration of the visual inertial system is performed according to the initial value of the focal length of the visual sensor determined in this implementation manner, which helps to improve the calibration accuracy. For example, Equation 4 can be used to determine the initial values of the focal lengths f x and f y of the vision sensor:
- W and H represent the width and height of the image collected by the vision sensor, respectively, and C fov represents the field of view of the vision sensor.
- the internal reference of the visual sensor includes the optical center coordinates of the visual sensor; before the enabling calibration of the internal reference of the visual sensor, the method further includes: according to the visual sensor The width and height of the image collected by the sensor determine the initial value of the optical center coordinates of the vision sensor.
- the parameters of the visual inertial system are calibrated according to the initial value of the optical center coordinate of the vision sensor determined by this implementation, which helps to improve the calibration accuracy. For example, Equation 5 and Equation 6 can be used to determine the initial values of the optical center coordinates c x and cy of the vision sensor:
- the initial values of the distortion parameters k 1 , k 2 , p 1 and p 2 of the vision sensor may be set to 0.
- the initial value of the time offset t d may be set to 0.
- the initial value of the translation parameter p extri from the inertial sensor to the vision sensor can be set to [0 0 0] T .
- the initial value of the rotation parameter q extri from the inertial sensor to the vision sensor can be set to I 3 ⁇ 3 , where I represents the identity matrix.
- the corresponding two - dimensional coordinates of PC in the image plane coordinate system The coordinate ⁇ (P C ) corresponding to P C in the pixel coordinate system can be determined by formula 7:
- Equations 8 to 11 can be obtained:
- x c , y c are the corresponding two-dimensional coordinates of the feature point in the image plane coordinate system; k 1 , k 2 are the radial distortion of the vision sensor; p 1 and p 2 are the tangential distortion of the vision sensor; r is the The original distance from the feature point to the image plane coordinate system; k r , p x , and py are the distortion coefficients of the vision sensor.
- the translation parameter of the vision sensor can be based on the translation parameters of the inertial sensor and the translation parameter p extri from the inertial sensor to the vision sensor is determined:
- the rotation parameters of the vision sensor can be based on the rotational parameters of the inertial sensor and the rotational parameter q extri from the inertial sensor to the vision sensor is determined:
- ( ⁇ )(i) and ( ⁇ )(i+1) represent the data at the ith moment (ie the previous moment) and the i+1th moment (ie the current moment), respectively, and represent the data output by the accelerometer and gyroscope, respectively, and represent the noise terms of the data output by the accelerometer and gyroscope, respectively. in, and It can be determined by formula 19 and formula 20 respectively:
- a I and w I represent the real data of the accelerometer and gyroscope, respectively; It is white noise, which can be obtained by looking up the table, or it can be 0 by default, or it can be used Derive to get, for example, It is white noise, which can be obtained by looking up the table, or it can be 0 by default, or it can be used Derive to get, for example,
- Equation 21 the Jacobian matrix corresponding to the state data of the visual-inertial system can be expressed as Equation 21:
- 15 represents the sum of the dimensions of J p , J q , J v , J ba and J bg , and m represents the dimension of the parameters to be calibrated in J calib (ie the number of parameters to be calibrated).
- J p represents the Jacobian matrix corresponding to the translation parameter of the visual sensor
- J q represents the Jacobian matrix corresponding to the rotation parameter of the visual sensor
- J v represents the Jacobian matrix corresponding to the velocity of the inertial sensor
- Jba represents the accelerometer of the inertial sensor.
- J bg represents the Jacobian matrix corresponding to the zero offset of the gyroscope of the inertial sensor
- J calib represents the Jacobian matrix corresponding to the parameter to be calibrated.
- J p , J q , J v , J ba and J bg can be determined using Equations 22 to 26, respectively:
- Equation 27 to Equation 30 can be used to obtain the internal parameters of the vision sensor, such as the focal length coordinates f x , f y and the Jacobian matrix corresponding to the optical center coordinates c x and cy :
- the Jacobian matrix corresponding to the distortion parameters k 1 , k 2 , p 1 and p 2 of the vision sensor can be obtained by using Equation 31 to Equation 34:
- the response and projection matrices project points on one image onto another.
- equations 45 to 48 can be used to determine the variance Var acc of the zero bias of the accelerometer of the inertial sensor and the variance Var gyro of the zero bias of the gyroscope of the inertial sensor:
- w represents the size of the sliding window.
- w can be determined empirically, for example, w can be equal to 11, 3, etc.
- the bias of the accelerometer and the bias of the gyroscope will gradually become stable, for example, the variance of the bias of the accelerometer and the bias of the gyroscope will continue to decrease to a convergent value.
- the parameter to be calibrated before starting the calibration of any parameter to be calibrated, the parameter to be calibrated may be fixed as an initial value.
- the parameter to be calibrated includes a time offset, wherein the time offset represents a difference between the clock of the visual sensor and the clock of the inertial sensor at the same moment; the The step of calibrating the parameters to be calibrated of the visual inertial system according to the state data of the visual inertial system at the current moment includes: in the state data of the visual inertial system at the current moment, the inertial When the zero offset of the accelerometer of the sensor satisfies the first preset stability condition, and the zero offset of the gyroscope of the inertial sensor satisfies the second preset stability condition, the calibration of the time offset is enabled.
- the stability of the bias of the accelerometer can be measured according to the degree of dispersion of the bias of the accelerometer, and the stability of the bias of the gyroscope can be measured according to the degree of dispersion of the bias of the gyroscope.
- the stability of the zero bias of the accelerometer can be measured according to the variance and standard deviation of the zero bias of the accelerometer
- the stability of the zero bias of the gyroscope can be measured according to the variance and standard deviation of the zero bias of the gyroscope.
- the larger the variance or standard deviation of the accelerometer's bias the lower the stability of the accelerometer's bias
- the smaller the variance or standard deviation of the accelerometer's bias the more stable the accelerometer's bias is.
- the first preset stability condition is a preset stability condition of the bias of the accelerometer
- the second preset stability condition is a preset stability condition of the bias of the gyroscope.
- the fact that the zero bias of the accelerometer satisfies the first preset stability condition includes: the variance of the zero bias of the accelerometer is smaller than the first preset value; the zero bias of the gyroscope satisfies the first preset stability condition
- the two preset stability conditions include: the variance of the zero bias of the gyroscope is smaller than the second preset value.
- the first preset value may be 0.08
- the second preset value may be 0.001.
- the first preset value can also be slightly larger or smaller than 0.08
- the second preset value can also be slightly larger or smaller than 0.001.
- Those skilled in the art can flexibly determine the first preset value and the second preset value based on experience .
- the parameters to be calibrated include internal parameters of the visual sensor; and the parameters to be calibrated of the visual inertial system are performed according to the state data of the visual inertial system at the current moment.
- Calibration includes: in the state data of the visual inertial system at the current moment, in the case that the zero offset of the accelerometer of the inertial sensor satisfies a third preset stability condition, turning on the internal reference of the visual sensor The calibration of the accelerometer, wherein the stability of the zero offset of the accelerometer corresponding to the third preset stability condition is higher than the stability of the zero offset of the accelerometer corresponding to the first preset stability condition.
- the third preset stability condition is a preset stability condition of the zero offset of the accelerometer.
- the stability of the zero bias of the accelerometer corresponding to the third preset stability condition is higher than the stability of the zero bias of the accelerometer corresponding to the first preset stability condition, indicating that the third preset stability condition affects the zero bias of the accelerometer.
- the requirement for the stability of the bias is higher than the requirement for the stability of the zero bias of the accelerometer by the first preset stability condition.
- the calibration of the internal parameters of the sensor helps to improve the convergence speed of the to-be-calibrated parameters of the visual inertial system, and improves the accuracy and robustness of parameter calibration.
- the stability of the visual inertial system is relatively high, and the calibration of the internal parameters of the visual sensor is started at this time, which can make the current visual
- the inertial system maintains a relatively stable state, so that a better calibration effect can be obtained.
- the fact that the zero offset of the accelerometer meets the third preset stability condition includes: the variance of the zero offset of the accelerometer is less than a third preset value, wherein the third preset value less than the first preset value.
- the third preset value may be 0.008.
- the third preset value may also be slightly larger or smaller than 0.008, and those skilled in the art can flexibly determine the third preset value according to experience.
- the parameter to be calibrated includes a coordinate conversion parameter between the inertial sensor and the visual sensor, wherein the coordinate conversion parameter includes a parameter between the inertial sensor and the visual sensor at least one of the translation parameters and rotation parameters of the visual inertial system; the calibration of the parameters to be calibrated of the visual inertial system according to the state data of the visual inertial system at the current moment, including: in the visual inertial system In the state data at the current moment, when the zero offset of the accelerometer of the inertial sensor satisfies a fourth preset stability condition, the calibration of the coordinate conversion parameter is enabled, wherein the fourth preset stability condition is The stability of the zero offset of the accelerometer corresponding to the stability condition is higher than the stability of the zero offset of the accelerometer corresponding to the first preset stability condition.
- the fourth preset stability condition is a preset stability condition of the zero offset of the accelerometer.
- the stability of the zero offset of the accelerometer corresponding to the fourth preset stability condition is higher than the stability of the zero offset of the accelerometer corresponding to the first preset stability condition, indicating that the fourth preset stability condition affects the stability of the accelerometer.
- the requirement for the stability of the zero offset is higher than the requirement for the stability of the zero offset of the accelerometer by the first preset stability condition.
- the calibration of the conversion parameters helps to improve the convergence speed of the to-be-calibrated parameters of the visual inertial system, and improves the accuracy and robustness of parameter calibration.
- the stability of the visual inertial system is relatively high, and at this timing, the communication between the inertial sensor and the visual sensor is turned on.
- the calibration of the coordinate transformation parameters can keep the current visual inertial system in a relatively stable state, so that a better calibration effect can be obtained.
- the fact that the zero offset of the accelerometer satisfies the fourth preset stability condition includes: the variance of the zero offset of the accelerometer is less than a fourth preset value, wherein the fourth preset value less than the first preset value.
- the fourth preset value may be equal to the third preset value, and may also be slightly larger or smaller than the third preset value.
- the fourth preset value may be 0.008.
- the fourth preset value may also be slightly larger or smaller than 0.008, and those skilled in the art can flexibly determine the fourth preset value according to experience.
- the time offset can be continued to be calibrated.
- the variance of the focal length coordinate f x can be determined by using Equation 49 to Equation 56 Variance of focal length coordinates f y The variance of the optical center coordinate c x and the variance of the optical center coordinates c y
- the parameter to be calibrated includes a distortion parameter of the visual sensor; the parameter to be calibrated for the visual inertial system according to the state data of the visual inertial system at the current moment
- the calibration includes: in the case where it is determined that the internal parameter of the visual sensor satisfies the fifth preset stability condition according to the state data of the visual inertial system at the current moment, turning on the distortion parameter of the visual sensor. Calibration.
- the stability of the internal parameters of the visual sensor can be measured according to the discrete degree of the internal parameters of the visual sensor.
- the stability of the internal parameters of the visual sensor can be measured according to the variance, standard deviation, etc. of the internal parameters of the visual sensor.
- the fifth preset stability condition is a preset stability condition of an internal reference of the visual sensor.
- the distortion of the visual sensor is enabled when it is determined that the internal parameter of the visual sensor meets the fifth preset stability condition according to the state data of the visual inertial system at the current moment.
- the calibration of parameters helps to improve the convergence speed of the parameters to be calibrated in the visual inertial system, and improves the accuracy and robustness of parameter calibration.
- the internal reference of the vision sensor is calibrated prior to the distortion parameter of the vision sensor. In the case that the internal reference of the visual sensor satisfies the fifth preset stability condition, the internal reference of the visual sensor has converged. Turning on the calibration of the distortion parameters of the vision sensor at this time will not affect the calibration effect of the internal parameters of the vision sensor.
- the fact that the internal parameter of the visual sensor satisfies the fifth preset stability condition includes: the variance of the internal parameter of the visual sensor is smaller than the fifth preset value.
- the fifth preset value may be 0.001.
- the fifth preset value may also be slightly larger or smaller than 0.001, and those skilled in the art can flexibly determine the fifth preset value according to experience.
- the variance of the internal parameters of the visual sensor is smaller than the fifth preset value, which may be that the variances of the focal length coordinates f x , f y and the optical center coordinates c x and cy are respectively smaller than the fifth preset value.
- the distortion parameter for the vision sensor can also be turned on calibration.
- the time offset, the internal parameters of the vision sensor and the coordinate transformation parameters may be continuously calibrated.
- the method further includes: determining the constant to-be-calibrated parameter at the current moment according to the motion state of the visual inertial system.
- the motion state of the visual sensor may be used as the motion state of the visual inertial system, so as to improve the reliability of the acquired motion state of the visual inertial system.
- the motion state of the inertial sensor can also be used as the motion state of the visual inertial system, which is not limited here.
- the unobservable parameter to be calibrated at the current moment may be determined as the parameter to be calibrated that remains fixed at the current moment according to the motion state of the visual inertial system.
- the determining the parameter to be calibrated that is kept fixed at the current moment according to the motion state of the visual inertial system includes: when the visual inertial system is stationary,
- the parameters to be calibrated that remain fixed at the current moment include time offset, translation parameters between the inertial sensor and the visual sensor, and rotation parameters between the inertial sensor and the visual sensor.
- the motion state of the visual sensor is used as the motion state of the visual inertial system, it can be determined that the visual inertial system is stationary when the translation parameter of the visual sensor is 0 and the rotation parameter is 0; As the motion state of the visual inertial system, it can be determined that the visual inertial system is stationary when the translation parameter of the inertial sensor is 0 and the rotation parameter is 0.
- the time offset, the translation parameter between the inertial sensor and the vision sensor, and the rotation parameter between the inertial sensor and the vision sensor are combined Keep it fixed, and calibrate other parameters to be calibrated whose calibration has been turned on.
- the determining the parameter to be calibrated that remains fixed at the current moment according to the motion state of the visual inertial system includes: when the translation parameter of the visual sensor or the inertial sensor is 0 And if the rotation parameter is not 0, it is determined that the parameter to be calibrated that remains fixed at the current moment includes the translation parameter between the inertial sensor and the visual sensor.
- the rotation parameter is not 0, it can be determined that the visual-inertial system operates purely rotationally.
- the inertial sensor and the visual sensor can be combined when the translation parameter of the visual sensor is 0 and the rotation parameter is not 0.
- the translation parameters between the two remain fixed, and the other parameters to be calibrated that have been calibrated are calibrated; if the motion state of the inertial sensor is used as the motion state of the visual inertial system, the translation parameter of the inertial sensor can be 0 and the rotation parameter is not. In the case of 0, the translation parameters between the inertial sensor and the visual sensor are kept fixed, and other parameters to be calibrated that have been calibrated are calibrated.
- the determining the parameters to be calibrated that remain fixed at the current moment according to the motion state of the visual-inertial system includes: when the visual sensor or the inertial sensor is around the x-axis, When one of the y-axis and the z-axis rotates, it is determined that the parameter to be calibrated that remains fixed at the current moment includes a parameter corresponding to the rotation axis among the translation parameters between the inertial sensor and the visual sensor.
- the motion state of the visual sensor is used as the motion state of the visual inertial system
- the inertial when the visual sensor rotates around one of the x-axis, the y-axis and the z-axis, the inertial The parameter corresponding to the rotation axis in the translation parameters between the sensor and the visual sensor is kept fixed, and other parameters to be calibrated that have been calibrated are calibrated;
- the motion state of the inertial sensor is used as the motion state of the visual inertial system, it can be In the case where the inertial sensor rotates around one of the x-axis, the y-axis and the z-axis, the parameter corresponding to the rotation axis among the translation parameters between the inertial sensor and the visual sensor is kept fixed, and the Other parameters to be calibrated whose calibration has been turned on are calibrated.
- the determining the parameter to be calibrated that remains fixed at the current moment according to the motion state of the visual inertial system includes: when the visual sensor or the inertial sensor revolves around any coordinate axis
- the parameters to be calibrated that are determined to remain fixed at the current moment include a time offset and a parameter corresponding to the coordinate axis in the translation parameters between the inertial sensor and the visual sensor.
- the motion state of the visual sensor is taken as the motion state of the visual inertial system
- the time offset and the inertial sensor The parameters corresponding to the coordinate axes in the translation parameters between the visual sensors are kept fixed, and other parameters to be calibrated that have been calibrated are calibrated;
- the motion state of the inertial sensor is used as the motion state of the visual inertial system, then In the case where the inertial sensor moves at a constant angular velocity around any coordinate axis, the time offset and the parameter corresponding to the coordinate axis in the translation parameters between the inertial sensor and the visual sensor can be kept fixed, And calibrate other parameters to be calibrated that have been opened for calibration.
- the vision sensor moves at a constant angular velocity around any coordinate axis when the vision sensor moves at the same angular velocity around the coordinate axis at the current moment and the previous moment;
- the current moment and the previous moment move around any coordinate axis at the same angular velocity
- the inertial sensor moves around the coordinate axis at a constant angular velocity.
- errors may be allowed, for example, the same angular velocity may be determined when the difference between the angular velocities is smaller than a predetermined threshold.
- the determining the parameters to be calibrated that remain fixed at the current moment according to the motion state of the visual-inertial system includes: when the visual sensor or the inertial sensor is along any coordinate axis
- the parameters to be calibrated that are determined to remain fixed at the current moment include a time offset and a parameter corresponding to the coordinate axis in the translation parameters between the inertial sensor and the visual sensor.
- the motion state of the visual sensor is used as the motion state of the visual inertial system
- the time offset and the inertial sensor The parameters corresponding to the coordinate axes in the translation parameters between the visual sensors are kept fixed, and other parameters to be calibrated that have been calibrated are calibrated;
- the motion state of the inertial sensor is used as the motion state of the visual inertial system, then In the case where the inertial sensor moves at a constant speed along any coordinate axis, the time offset and the parameter corresponding to the coordinate axis among the translation parameters between the inertial sensor and the visual sensor can be kept fixed, And calibrate other parameters to be calibrated that have been opened for calibration.
- the vision sensor moves at a constant speed around any coordinate axis under the condition that the vision sensor moves at the same speed around any coordinate axis at the current moment and the previous moment;
- the current moment and the previous moment move around any coordinate axis at the same speed
- the inertial sensor moves at a constant speed around the coordinate axis.
- errors may be allowed, for example, the same speed may be determined when the difference between the speeds is less than a predetermined threshold.
- the determining the parameter to be calibrated that remains fixed at the current moment according to the motion state of the visual inertial system includes: when the visual sensor or the inertial sensor revolves around any coordinate axis In the case of moving at a constant angular acceleration, it is determined that the parameters to be calibrated that remain fixed at the current moment include time offset and parameters corresponding to the coordinate axes in the translation parameters between the inertial sensor and the visual sensor.
- the motion state of the visual sensor is used as the motion state of the visual inertial system
- the time offset and the inertial The parameter corresponding to the coordinate axis in the translation parameters between the sensor and the visual sensor is kept fixed, and other parameters to be calibrated that have been calibrated are calibrated; if the motion state of the inertial sensor is used as the motion state of the visual inertial system, Then, when the inertial sensor moves around any coordinate axis at a constant angular acceleration, the time offset and the translation parameters between the inertial sensor and the visual sensor can be kept in the parameters corresponding to the coordinate axis.
- the method further includes: determining the constant to-be-calibrated parameter at the current moment according to the tracking state of the visual-inertial system.
- determining the constant to-be-calibrated parameters at the current moment according to the tracking state of the visual-inertial system by determining the constant to-be-calibrated parameters at the current moment according to the tracking state of the visual-inertial system, only the to-be-calibrated parameters that are observable at the current moment are calibrated at the current moment, and the The parameters to be calibrated that cannot be observed at the current moment are kept fixed, so that the effectiveness of the parameters will not be reduced due to the tracking state of the visual inertial system, and the accuracy of the calibrated parameters can be improved, thereby helping to improve the positioning and construction of the visual inertial system. Accuracy of graphics, navigation, etc.
- the determining the parameters to be calibrated that remain fixed at the current moment according to the tracking state of the visual-inertial system includes: in the case that the tracking of the visual-inertial system is lost, determining the The parameters to be calibrated that are kept fixed at the current moment include a time offset, a translation parameter between the inertial sensor and the visual sensor, and a rotation parameter between the inertial sensor and the visual sensor.
- the loss of tracking may indicate that the two most recent images acquired by the vision sensor are not correlated and/or that the relocation failed.
- the time offset, the translation parameters between the inertial sensor and the vision sensor, and the rotation between the inertial sensor and the vision sensor are converted The parameters remain fixed, and other parameters to be calibrated whose calibration has been turned on are calibrated.
- the determining the parameters to be calibrated that remain fixed at the current moment according to the tracking state of the visual-inertial system includes: the number of feature points currently tracked by the visual-inertial system is less than In the case of a preset number, it is determined that the parameters to be calibrated that are kept fixed at the current moment include time offset, translation parameters between the inertial sensor and the visual sensor, and the distance between the inertial sensor and the visual sensor. rotation parameters.
- the preset number may be 20. Of course, those skilled in the art can flexibly set the preset number according to actual application scenario requirements, which is not limited here.
- the parameters to be calibrated can be updated every frame according to Equation 57, or the parameters to be calibrated can be updated every few frames:
- ⁇ X calib can solve the estimated value of the state data of the visual inertial system state according to the square-root inverse sliding window filter (Square-Root Inverse Sliding Window Filter, SR-ISWF) get:
- ⁇ X calib represents the change of the parameter to be calibrated between the i-th time and the i+1-th time
- ⁇ X calib is obtained by calculating get.
- R i is the upper triangular matrix of the square root of the Hessian matrix, and is the square root inverse sliding window filter solution
- the upper triangular Jacobian matrix obtained by the least squares method is used to obtain the optimal value of ;
- r i is the upper triangular matrix of the residual error of the Hessian matrix, which is the square root inverse sliding window filter solution
- the residual term when the optimal value of that is, the error term. when When infinitely close to r i , optimal.
- the square root inverse sliding window filter uses the inverse filter to iterate and then linearize.
- the embodiments of the present disclosure propose various sequences and timings of parameter self-calibration, which can realize rapid convergence of the calibrated parameters and improve the accuracy and robustness of parameter calibration.
- the corresponding calibration parameters are fixed according to the observability state of the system to ensure the validity of the system calibration parameters. The validity of the parameters will not be reduced due to the motion state of the system, thereby ensuring the accuracy of positioning and mapping.
- the embodiments of the present disclosure may be applied to an AR (Augmented Reality, augmented reality) system and/or a VR (Virtual Reality, virtual reality) system of a mobile terminal (such as a mobile phone), so as to implement an AR system And/or self-calibration of sensor parameters of visual-inertial systems in VR systems.
- AR Augmented Reality, augmented reality
- VR Virtual Reality, virtual reality
- the embodiment of the present disclosure can perform online optimization on the parameters calibrated offline, so as to improve the positioning of the visual inertial system. , mapping or navigation accuracy.
- the embodiments of the present disclosure can be used to self-calibrate the sensor parameters of the visual inertial system, so as to improve the adaptability of the visual inertial system. Compatibility and versatility. For example, on a mobile phone without offline-calibrated parameters, adaptation can still be achieved by using the embodiments of the present disclosure.
- the parameters of offline calibration in the related art often have errors and cannot be adjusted online in real time.
- the mechanical structure of the equipment where the visual inertial system is located changes, the focal length of the visual sensor changes, or the visual sensor and the inertial sensor are not rigidly fixed, the pre-offline calibration cannot meet the current needs.
- the parameter calibration method proposed in the embodiment of the present disclosure can perform online self-calibration without offline calibration parameters, and adapt to more devices; and can also use offline calibrated parameters as initial values when there are offline calibrated parameters. Online adjustments to improve positioning and mapping quality.
- the parameter calibration method proposed in the embodiments of the present disclosure can fix the corresponding calibration parameters according to the motion state, so as to ensure the validity of the system calibration parameters, and will not reduce the validity of the parameters due to the motion state of the system.
- the parameter calibration method of the visual inertial system provided by the embodiment of the present disclosure can be applied to application fields such as virtual reality, augmented reality, mixed reality, unmanned aerial vehicle, mobile robot, and unmanned vehicle, which is not limited herein.
- the foregoing method embodiments mentioned in the present disclosure can be combined with each other to form a combined embodiment without violating the principle and logic.
- the actual execution sequence of each step should be determined by its function and possible internal logic.
- the present disclosure also provides a parameter calibration device, electronic device, computer-readable storage medium, and program for a visual inertial system, all of which can be used to implement any parameter calibration method for a visual inertial system provided by the present disclosure, and the corresponding technical solutions and The technical effects can be found in the corresponding records in the Methods section.
- FIG. 2 shows a block diagram of a parameter calibration device for a visual inertial system provided by an embodiment of the present disclosure.
- the visual inertial system includes a visual sensor and an inertial sensor.
- the parameter calibration device of the visual inertial system includes:
- the acquisition module 21 is configured to acquire the state data of the visual inertial system at the last moment;
- the estimation module 22 is configured to collect the visual sensor at the current moment according to the state data of the visual inertial system at the last moment The coordinates of the feature points in the image, and the output data of the inertial sensor at the current moment, estimate the state data of the visual inertial system at the current moment;
- the calibration module 23 is configured to be based on the visual inertial system. In the state data at the current moment, the parameters to be calibrated of the visual inertial system are calibrated.
- the state data of the visual inertial system at the last moment includes: the state data of the visual sensor at the last moment, the state data of the inertial sensor at the last moment status data and the value of the parameter to be calibrated at the previous moment;
- the status data of the visual inertial system at the current moment includes: the status data of the visual sensor at the current moment, the inertial sensor at the current moment The state data at the current moment and the value of the parameter to be calibrated at the current moment.
- the state data of the vision sensor at the last moment includes: translation parameters of the vision sensor at the last moment and rotation of the vision sensor at the last moment parameter;
- the state data of the visual sensor at the current moment includes: the translation parameter of the visual sensor at the current moment and the rotation parameter of the visual sensor at the current moment.
- the state data of the inertial sensor at the last moment includes: the speed of the inertial sensor at the last moment; the state data of the inertial sensor at the current moment includes: : The speed of the inertial sensor at the current moment.
- the state data of the inertial sensor at the last moment further includes: the zero offset of the accelerometer of the inertial sensor at the last moment and the zero offset of the inertial sensor at the last moment The zero offset of the gyroscope at a moment;
- the state data of the inertial sensor at the current moment further includes: the zero offset of the accelerometer of the inertial sensor at the current moment and the zero offset of the inertial sensor at the current moment The zero offset of the gyroscope.
- the parameter to be calibrated includes a time offset, wherein the time offset represents a difference between the clock of the visual sensor and the clock of the inertial sensor at the same moment;
- the The calibration module 23 is configured to: in the state data of the visual inertial system at the current moment, the zero offset of the accelerometer of the inertial sensor satisfies the first preset stability condition, and the gyroscope of the inertial sensor In the case that the zero offset of satisfies the second preset stability condition, the calibration of the time offset is enabled.
- the parameters to be calibrated include internal parameters of the visual sensor; the calibration module 23 is configured to: in the state data of the visual inertial system at the current moment, the inertial sensor When the zero bias of the accelerometer meets the third preset stability condition, the calibration of the internal parameters of the visual sensor is enabled, wherein the stability of the zero bias of the accelerometer corresponding to the third preset stability condition The stability of the zero offset of the accelerometer is higher than that corresponding to the first preset stability condition.
- the internal parameters of the visual sensor include the focal length and the optical center of the visual sensor; the device further includes: a first determining module, configured to obtain the field of view angle of the visual sensor after acquiring In this case, the initial value of the focal length of the vision sensor is determined according to the width of the image collected by the vision sensor and the field of view of the vision sensor; or, the second determination module is configured to In the case of the field angle of the sensor, the initial value of the focal length of the vision sensor is determined according to the width and height of the image collected by the vision sensor.
- the internal reference of the vision sensor includes the optical center coordinates of the vision sensor; the apparatus further includes: a third determination module configured to be based on the width and height of the image collected by the vision sensor , and determine the initial value of the optical center coordinates of the vision sensor.
- the parameter to be calibrated includes a coordinate conversion parameter between the inertial sensor and the visual sensor, wherein the coordinate conversion parameter includes a parameter between the inertial sensor and the visual sensor at least one of the translation parameters and rotation parameters of
- the calibration of the coordinate conversion parameters is enabled, wherein the stability of the zero bias of the accelerometer corresponding to the fourth preset stability condition is higher than that corresponding to the first preset stability condition. The stability of the zero offset of the accelerometer.
- the parameter to be calibrated includes a distortion parameter of the visual sensor; the calibration module 23 is configured to: determine the When the internal parameter of the vision sensor meets the fifth preset stability condition, the calibration of the distortion parameter of the vision sensor is started.
- the apparatus further includes: a fourth determination module, configured to determine the constant to-be-calibrated parameter at the current moment according to the motion state of the visual inertial system.
- the fourth determining module is configured to at least one of the following: in the case that the visual inertial system is stationary, determine that the parameters to be calibrated that remain fixed at the current moment include time offset, Translation parameters between the inertial sensor and the vision sensor and rotation parameters between the inertial sensor and the vision sensor;
- the parameter to be calibrated that remains fixed at the current moment includes the difference between the inertial sensor and the visual sensor.
- Translation parameter when the visual sensor or the inertial sensor rotates around one of the x-axis, the y-axis, and the z-axis, determining that the parameter to be calibrated that remains fixed at the current moment includes the inertial sensor and the The parameter corresponding to the rotation axis among the translation parameters between the visual sensors; in the case that the visual sensor or the inertial sensor moves around any coordinate axis at a constant angular velocity, it is determined that the current moment remains fixed to be
- the calibration parameters include time offsets and parameters corresponding to the coordinate axes in the translation parameters between the inertial sensor and the visual sensor; when the visual sensor or the inertial sensor moves at a constant speed along any coordinate axis In the case of , it is determined that the parameters to be calibrated that remain fixed at
- the apparatus further includes: a fifth determination module, configured to determine the constant to-be-calibrated parameter at the current moment according to the tracking state of the visual inertial system.
- the fifth determining module is configured to: in the case that the tracking of the visual inertial system is lost, determine that the parameters to be calibrated that remain fixed at the current moment include a time offset, the inertial The translation parameter between the sensor and the visual sensor and the rotation parameter between the inertial sensor and the visual sensor; and/or, the number of feature points currently tracked by the visual-inertial system is less than a preset number of
- the parameters to be calibrated that are determined to remain fixed at the current moment include a time offset, a translation parameter between the inertial sensor and the visual sensor, and a rotation parameter between the inertial sensor and the visual sensor.
- the embodiment of the present disclosure obtains the state data of the visual inertial system at the last moment, according to the state data of the visual inertial system at the last moment, the coordinates of the feature points in the image collected by the visual sensor at the current moment, and the inertial sensor at the current moment.
- the output data of the current moment estimate the state data of the visual inertial system at the current moment, and according to the state data of the visual inertial system at the current moment, carry out the calibration parameters of the visual inertial system.
- the functions or modules included in the apparatus provided in the embodiments of the present disclosure may be configured to execute the methods described in the above method embodiments, and for implementation and technical effects, reference may be made to the above method embodiments.
- Embodiments of the present disclosure further provide a computer-readable storage medium, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the foregoing method is implemented.
- the computer-readable storage medium may be a non-volatile computer-readable storage medium, or may be a volatile computer-readable storage medium.
- An embodiment of the present disclosure also provides a computer program, including computer-readable code, when the computer-readable code is executed in an electronic device, a processor in the electronic device executes and is configured to implement the above method.
- Embodiments of the present disclosure further provide another computer program product configured to store computer-readable instructions, which, when executed, cause the computer to execute the operations of the method for calibrating parameters of a visual inertial system provided by any of the foregoing embodiments.
- Embodiments of the present disclosure further provide an electronic device, including: one or more processors; a memory for storing executable instructions; wherein the one or more processors are configured to invoke executable instructions stored in the memory instruction to execute the above method.
- the electronic device may be provided as a terminal, server or other form of device.
- FIG. 3 shows a block diagram of an electronic device 800 according to an embodiment of the present disclosure.
- electronic device 800 may be a mobile phone, computer, digital broadcast terminal, messaging device, game console, tablet device, medical device, fitness device, personal digital assistant, etc. terminal.
- the electronic device 800 may include one or more of the following components: a processing component 802, a memory 804, a power supply component 806, a multimedia component 808, an audio component 810, an input/output (I/O) interface 812 , sensor component 814 and communication component 816 .
- the processing component 802 generally controls the overall operations of the electronic device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations.
- the processing component 802 can include one or more processors 820 to execute instructions to perform all or some of the steps of the methods described above.
- processing component 802 may include one or more modules that facilitate interaction between processing component 802 and other components.
- processing component 802 may include a multimedia module to facilitate interaction between multimedia component 808 and processing component 802.
- Memory 804 is configured to store various types of data to support operation at electronic device 800 . Examples of such data include instructions for any application or method operating on electronic device 800, contact data, phonebook data, messages, pictures, videos, and the like.
- Memory 804 may be implemented by any type of volatile or non-volatile storage device or combination thereof, such as Static Random-Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM) , Static Random-Access Memory), Erasable Programmable Read-Only Memory (EPROM, Electrically Erasable Programmable Read-Only Memory), Programmable Read-Only Memory (PROM, Programmable Read-Only Memory), Read-Only Memory (ROM, Read Only Memory), magnetic memory, flash memory, magnetic disk or optical disk.
- SRAM Static Random-Access Memory
- EEPROM Electrically Erasable Programmable Read-Only Memory
- EPROM Electrically Erasable Programmable Read-Only Memory
- PROM Programmable Read-Only Memory
- Read-Only Memory Read-
- Power supply assembly 806 provides power to various components of electronic device 800 .
- Power supply components 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 800 .
- Multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user.
- the screen may include a liquid crystal display (LCD, Liquid Crystal Display) and a touch panel (TP, Touch Panel). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user.
- the touch panel includes one or more touch sensors to sense touch, swipe, and gestures on the touch panel. The touch sensor may not only sense the boundaries of a touch or swipe action, but also detect the duration and pressure associated with the touch or swipe action.
- multimedia component 808 includes a front-facing camera and/or a rear-facing camera. When the electronic device 800 is in an operation mode, such as a shooting mode or a video mode, the front camera and/or the rear camera may receive external multimedia data. Each of the front and rear cameras can be a fixed optical lens system or have focal length and optical zoom capability.
- Audio component 810 is configured to output and/or input audio signals.
- the audio component 810 includes a microphone (MIC, Microphone), which is configured to receive external audio signals when the electronic device 800 is in an operating mode, such as a calling mode, a recording mode, and a voice recognition mode.
- the received audio signal may be further stored in memory 804 or transmitted via communication component 816 .
- audio component 810 also includes a speaker for outputting audio signals.
- the I/O interface 812 provides an interface between the processing component 802 and a peripheral interface module, which may be a keyboard, a click wheel, a button, or the like. These buttons may include, but are not limited to: home button, volume buttons, start button, and lock button.
- Sensor assembly 814 includes one or more sensors for providing status assessment of various aspects of electronic device 800 .
- the sensor component 814 can detect the open/closed state of the electronic device 800, the relative positioning of components, such as the display and the keypad of the electronic device 800, and the sensor component 814 can also detect the electronic device 800 or one of the electronic devices 800. Changes in the position of components, presence or absence of user contact with the electronic device 800 , orientation or acceleration/deceleration of the electronic device 800 and changes in the temperature of the electronic device 800 .
- Sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects in the absence of any physical contact.
- the sensor assembly 814 may also include a light sensor, such as a Complementary Metal-Oxide-Semiconductor (CMOS) or Charge Coupled Device (CCD) image sensor, for use in imaging applications.
- CMOS Complementary Metal-Oxide-Semiconductor
- CCD Charge Coupled Device
- the sensor assembly 814 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.
- Communication component 816 is configured to facilitate wired or wireless communication between electronic device 800 and other devices.
- the electronic device 800 can access a wireless network based on a communication standard, such as a wireless network (Wi-Fi, Wireless Fidelity), a second-generation mobile communication technology (2G, The 2nd Generation) or a third-generation mobile communication technology (3G, The 3nd Generation) Generation), or their combination.
- the communication component 816 receives broadcast signals or broadcast related information from an external broadcast management system via a broadcast channel.
- the communication component 816 further includes a Near Field Communication (NFC, Near Field Communication) module to facilitate short-range communication.
- NFC Near Field Communication
- the NFC module can be based on Radio Frequency Identification (RFID, Radio Frequency Identification) technology, Infrared Data Association (IrDA, Infrared Data Association) technology, Ultra Wide Band (UWB, Ultra Wide Band) technology, Bluetooth (BT, Blue Tooth) technology and other technologies to achieve.
- RFID Radio Frequency Identification
- IrDA Infrared Data Association
- UWB Ultra Wide Band
- Bluetooth Bluetooth
- electronic device 800 may be implemented by one or more Application Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD) Processing Device), Programmable Logic Device (PLD, Programmable Logic Device), Field Programmable Gate Array (FPGA, Field Programmable Gate Array), controller, microcontroller, microprocessor, or other electronic component implementation for performing the above method.
- ASIC Application Specific Integrated Circuit
- DSP Digital Signal Processor
- DSPD Digital Signal Processing Device
- PLD Programmable Logic Device
- FPGA Field Programmable Gate Array
- controller microcontroller, microprocessor, or other electronic component implementation for performing the above method.
- microcontroller microprocessor
- FIG. 4 shows a block diagram of an electronic device 1900 according to an embodiment of the present disclosure.
- the electronic device 1900 may be provided as a server.
- electronic device 1900 includes processing component 1922, which may include one or more processors, and memory resources represented by memory 1932 for storing instructions executable by processing component 1922, such as applications.
- An application program stored in memory 1932 may include one or more modules, each corresponding to a set of instructions.
- the processing component 1922 is configured to execute instructions to perform the above-described methods.
- the electronic device 1900 may also include a power supply assembly 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input output (I/O) interface 1958 .
- the electronic device 1900 can operate based on an operating system stored in the memory 1932, such as a Microsoft server operating system (Windows ServerTM), a graphical user interface based operating system (Mac OS XTM) introduced by Apple, a multi-user multi-process computer operating system (UnixTM). ), a free and open source Unix-like operating system (LinuxTM), an open source Unix-like operating system (FreeBSDTM) or similar systems.
- a non-volatile computer-readable storage medium is also provided, such as memory 1932 comprising computer program instructions executable by processing component 1922 of electronic device 1900 to perform the above-described method.
- Embodiments of the present disclosure may be systems, methods and/or computer program products.
- the computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the embodiments of the present disclosure.
- a computer-readable storage medium may be a tangible device that can hold and store instructions for use by the instruction execution device.
- the computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing.
- Computer-readable storage media may include: portable computer disks, hard disks, random access memory (RAM, Random Access Memory), read-only memory, erasable programmable read-only memory (EPROM or flash memory), static random access memory, Portable Compact Disc Read-Only Memory (CD-ROM, Compact Disc Read-Only Memory), Digital Versatile Disc (DVD, Digital Video Disc), memory stick, floppy disk, mechanical coding device, such as a punch card on which instructions are stored Or the protruding structure in the groove, and any suitable combination of the above.
- RAM Random Access Memory
- EPROM or flash memory erasable programmable read-only memory
- static random access memory Portable Compact Disc Read-Only Memory
- CD-ROM Compact Disc Read-Only Memory
- DVD Digital Versatile Disc
- memory stick floppy disk
- mechanical coding device such as a punch card on which instructions are stored Or the protruding structure in the groove, and any suitable combination of the above.
- Computer-readable storage media are not to be construed as transient signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (eg, light pulses through fiber optic cables), or through electrical wires transmitted electrical signals.
- the computer readable program instructions described herein may be downloaded to various computing/processing devices from a computer readable storage medium, or to an external computer or external storage device over a network such as the Internet, a local area network, a wide area network, and/or a wireless network.
- the network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and/or edge servers.
- a network adapter card or network interface in each computing/processing device receives computer-readable program instructions from a network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in each computing/processing device .
- Computer program instructions for carrying out operations of the present disclosure may be assembly instructions, Industry Standard Architecture (ISA) instructions, machine instructions, machine-dependent instructions, pseudocode, firmware instructions, state setting data, or in one or more Source or object code written in any combination of programming languages, including object-oriented programming languages such as Smalltalk, C++, etc., and conventional procedural programming languages such as the "C" language or similar programming languages.
- the computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server implement.
- the remote computer may be connected to the user's computer through any kind of network, including a Local Area Network (LAN) or Wide Area Network (WAN), or may be connected to an external computer (eg, using Internet service provider to connect via the Internet).
- electronic circuits such as programmable logic circuits, field programmable gate arrays, or programmable logic arrays, that can execute computer readable program instructions are personalized by utilizing state information of computer readable program instructions , thereby implementing various aspects of the present disclosure. Aspects of the present disclosure are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the disclosure. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer readable program instructions.
- These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer or other programmable data processing apparatus to produce a machine that causes the instructions when executed by the processor of the computer or other programmable data processing apparatus , means implementing the functions/acts specified in one or more blocks of the flowcharts and/or block diagrams.
- These computer readable program instructions can also be stored in a computer readable storage medium, these instructions cause a computer, programmable data processing apparatus and/or other equipment to operate in a specific manner, so that the computer readable medium on which the instructions are stored includes An article of manufacture comprising instructions for implementing various aspects of the functions/acts specified in one or more blocks of the flowchart and/or block diagrams.
- Computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other equipment to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other equipment to produce a computer-implemented process , thereby causing instructions executing on a computer, other programmable data processing apparatus, or other device to implement the functions/acts specified in one or more blocks of the flowcharts and/or block diagrams.
- each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more functions for implementing the specified logical function(s) executable instructions.
- the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved.
- each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations can be implemented in dedicated hardware-based systems that perform the specified functions or actions , or can be implemented in a combination of dedicated hardware and computer instructions.
- the computer program product can be implemented in hardware, software or a combination thereof.
- the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (Software Development Kit, SDK) and the like.
- the calibration reference object, calibration plate, turntable and other external devices are not needed to assist in the calibration process, and the parameters to be calibrated are calibrated according to the state of the visual inertial system, so that the self-calibration of the sensor parameters of the visual inertial system can be realized. , and more accurate parameters can be obtained.
Landscapes
- Engineering & Computer Science (AREA)
- Radar, Positioning & Navigation (AREA)
- Remote Sensing (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Automation & Control Theory (AREA)
- Manufacturing & Machinery (AREA)
- Gyroscopes (AREA)
- Navigation (AREA)
Abstract
Description
Claims (33)
- 一种视觉惯性***的参数标定方法,所述视觉惯性***包括视觉传感器和惯性传感器,所述方法包括:A method for calibrating parameters of a visual inertial system, the visual inertial system comprising a visual sensor and an inertial sensor, the method comprising:获取所述视觉惯性***在上一时刻的状态数据;Obtain the state data of the visual inertial system at the last moment;根据所述视觉惯性***在所述上一时刻的状态数据,所述视觉传感器在当前时刻采集的图像中的特征点的坐标,以及所述惯性传感器在所述当前时刻的输出数据,估计所述视觉惯性***在所述当前时刻的状态数据;According to the state data of the visual inertial system at the last moment, the coordinates of the feature points in the image collected by the visual sensor at the current moment, and the output data of the inertial sensor at the current moment, estimate the State data of the visual inertial system at the current moment;根据所述视觉惯性***在所述当前时刻的状态数据,对所述视觉惯性***的待标定参数进行标定。The to-be-calibrated parameters of the visual-inertial system are calibrated according to the state data of the visual-inertial system at the current moment.
- 根据权利要求1所述的方法,其中,所述视觉惯性***在所述上一时刻的状态数据包括:所述视觉传感器在所述上一时刻的状态数据、所述惯性传感器在所述上一时刻的状态数据以及所述待标定参数在所述上一时刻的数值;所述视觉惯性***在所述当前时刻的状态数据包括:所述视觉传感器在所述当前时刻的状态数据、所述惯性传感器在所述当前时刻的状态数据以及所述待标定参数在所述当前时刻的数值。The method according to claim 1, wherein the state data of the visual inertial system at the last moment comprises: the state data of the visual sensor at the last moment, the inertial sensor at the last moment The state data at the moment and the value of the parameter to be calibrated at the previous moment; the status data of the visual inertial system at the current moment includes: the state data of the visual sensor at the current moment, the inertial The state data of the sensor at the current moment and the value of the parameter to be calibrated at the current moment.
- 根据权利要求2所述的方法,其中,所述视觉传感器在所述上一时刻的状态数据包括:所述视觉传感器在所述上一时刻的平移参数和所述视觉传感器在所述上一时刻的旋转参数;所述视觉传感器在所述当前时刻的状态数据包括:所述视觉传感器在所述当前时刻的平移参数和所述视觉传感器在所述当前时刻的旋转参数。The method according to claim 2, wherein the state data of the vision sensor at the last moment comprises: translation parameters of the vision sensor at the last moment and the vision sensor at the last moment The rotation parameter of the visual sensor at the current moment includes: the translation parameter of the visual sensor at the current moment and the rotation parameter of the visual sensor at the current moment.
- 根据权利要求2或3所述的方法,其中,The method according to claim 2 or 3, wherein,所述惯性传感器在所述上一时刻的状态数据包括:所述惯性传感器在所述上一时刻的速度;The state data of the inertial sensor at the last moment includes: the speed of the inertial sensor at the last moment;所述惯性传感器在所述当前时刻的状态数据包括:所述惯性传感器在所述当前时刻的速度。The state data of the inertial sensor at the current moment includes: the speed of the inertial sensor at the current moment.
- 根据权利要求4所述的方法,其中,所述惯性传感器在所述上一时刻的状态数据还包括:所述惯性传感器在所述上一时刻的加速度计的零偏以及所述惯性传感器在所述上一时刻的陀螺仪的零偏;所述惯性传感器在所述当前时刻的状态数据还包括:所述惯性传感器在所述当前时刻的加速度计的零偏以及所述惯性传感器在所述当前时刻的陀螺仪的零偏。The method according to claim 4, wherein the state data of the inertial sensor at the last moment further comprises: zero offset of the accelerometer of the inertial sensor at the last moment and the inertial sensor at the last moment. The zero offset of the gyroscope at the previous moment; the state data of the inertial sensor at the current moment also includes: the zero offset of the accelerometer of the inertial sensor at the current moment and the inertial sensor at the current moment The zero offset of the gyroscope at the moment.
- 根据权利要求1至5中任意一项所述的方法,其中,所述待标定参数包括时间偏移,其中,所述时间偏移表示在同一时刻所述视觉传感器的时钟与所述惯性传感器的时钟之间的差值;The method according to any one of claims 1 to 5, wherein the parameter to be calibrated includes a time offset, wherein the time offset represents the difference between the clock of the visual sensor and the inertial sensor at the same time. difference between clocks;所述根据所述视觉惯性***在所述当前时刻的状态数据,对所述视觉惯性***的待标定参数进行标定,包括:The calibration of the to-be-calibrated parameters of the visual-inertial system according to the state data of the visual-inertial system at the current moment includes:在所述视觉惯性***在所述当前时刻的状态数据中,所述惯性传感器的加速度计的零偏满足第一预设稳定性条件,且所述惯性传感器的陀螺仪的零偏满足第二预设稳定性条件的情况下,开启对所述时间偏移的标定。In the state data of the visual inertial system at the current moment, the zero offset of the accelerometer of the inertial sensor satisfies the first preset stability condition, and the zero offset of the gyroscope of the inertial sensor satisfies the second preset stability condition In the case of setting stability conditions, the calibration of the time offset is enabled.
- 根据权利要求1至6中任意一项所述的方法,其中,所述待标定参数包括所述视觉传感器的内参;所述根据所述视觉惯性***在所述当前时刻的状态数据,对所述视觉惯性***的待标定参数进行标定,包括:The method according to any one of claims 1 to 6, wherein the parameters to be calibrated include internal parameters of the visual sensor; The parameters to be calibrated of the visual inertial system are calibrated, including:在所述视觉惯性***在所述当前时刻的状态数据中,所述惯性传感器的加速度计的零偏满足第三预设稳定性条件的情况下,开启对所述视觉传感器的内参的标定,其中,所述第三预设稳定性条件对应的加速度计的零偏的稳定性,高于第一预设稳定性条件对应的加速度计的零偏的稳定性。In the state data of the visual inertial system at the current moment, when the zero offset of the accelerometer of the inertial sensor satisfies the third preset stability condition, the calibration of the internal parameters of the visual sensor is started, wherein , the stability of the zero offset of the accelerometer corresponding to the third preset stability condition is higher than the stability of the zero offset of the accelerometer corresponding to the first preset stability condition.
- 根据权利要求7所述的方法,其中,所述视觉传感器的内参包括所述视觉传感器的焦距和光心;在所述开启对所述视觉传感器的内参的标定之前,所述方法还包括:The method according to claim 7, wherein the internal parameters of the visual sensor include the focal length and optical center of the visual sensor; before the enabling calibration of the internal parameters of the visual sensor, the method further comprises:在获取得到所述视觉传感器的视场角的情况下,根据所述视觉传感器采集的图像的宽度以及所述视觉传感器的视场角,确定所述视觉传感器的焦距的初始值;In the case of obtaining the field of view angle of the visual sensor, determine the initial value of the focal length of the visual sensor according to the width of the image collected by the visual sensor and the field of view angle of the visual sensor;或者,在获取不到所述视觉传感器的视场角的情况下,根据所述视觉传感器采集的图像的宽度和高度,确定所述视觉传感器的焦距的初始值。Alternatively, in the case that the field of view of the vision sensor cannot be obtained, the initial value of the focal length of the vision sensor is determined according to the width and height of the image collected by the vision sensor.
- 根据权利要求7或8所述的方法,其中,所述视觉传感器的内参包括所述视觉传感器的光心坐标;在所述开启对所述视觉传感器的内参的标定之前,所述方法还包括:The method according to claim 7 or 8, wherein the internal reference of the visual sensor includes the optical center coordinate of the visual sensor; before the enabling calibration of the internal reference of the visual sensor, the method further comprises:根据所述视觉传感器采集的图像的宽度和高度,确定所述视觉传感器的光心坐标的初始值。According to the width and height of the image collected by the vision sensor, the initial value of the optical center coordinate of the vision sensor is determined.
- 根据权利要求1至9中任意一项所述的方法,其中,所述待标定参数包括所述惯性传感器 与所述视觉传感器之间的坐标转换参数,其中,所述坐标转换参数包括所述惯性传感器与所述视觉传感器之间的平移参数和旋转参数中的至少之一;所述根据所述视觉惯性***在所述当前时刻的状态数据,对所述视觉惯性***的待标定参数进行标定,包括:The method according to any one of claims 1 to 9, wherein the parameter to be calibrated comprises a coordinate conversion parameter between the inertial sensor and the visual sensor, wherein the coordinate conversion parameter comprises the inertial sensor at least one of translation parameters and rotation parameters between the sensor and the visual sensor; the visual inertial system is calibrated according to the state data of the visual inertial system at the current moment, and the to-be-calibrated parameters are calibrated, include:在所述视觉惯性***在所述当前时刻的状态数据中,所述惯性传感器的加速度计的零偏满足第四预设稳定性条件的情况下,开启对所述坐标转换参数的标定,其中,所述第四预设稳定性条件对应的加速度计的零偏的稳定性,高于第一预设稳定性条件对应的加速度计的零偏的稳定性。In the state data of the visual inertial system at the current moment, when the zero offset of the accelerometer of the inertial sensor satisfies the fourth preset stability condition, the calibration of the coordinate conversion parameter is started, wherein, The stability of the zero offset of the accelerometer corresponding to the fourth preset stability condition is higher than the stability of the zero offset of the accelerometer corresponding to the first preset stability condition.
- 根据权利要求1至10中任意一项所述的方法,其中,所述待标定参数包括所述视觉传感器的畸变参数;所述根据所述视觉惯性***在所述当前时刻的状态数据,对所述视觉惯性***的待标定参数进行标定,包括:The method according to any one of claims 1 to 10, wherein the parameter to be calibrated includes a distortion parameter of the visual sensor; and according to the state data of the visual inertial system at the current moment, the The parameters to be calibrated of the visual inertial system are calibrated, including:在根据所述视觉惯性***在所述当前时刻的状态数据,确定所述视觉传感器的内参满足第五预设稳定性条件的情况下,开启对所述视觉传感器的畸变参数的标定。When it is determined according to the state data of the visual inertial system at the current moment that the internal parameter of the visual sensor satisfies the fifth preset stability condition, the calibration of the distortion parameter of the visual sensor is started.
- 根据权利要求1至11中任意一项所述的方法,其中,所述方法还包括:The method according to any one of claims 1 to 11, wherein the method further comprises:根据所述视觉惯性***的运动状态,确定所述当前时刻的保持固定的待标定参数。According to the motion state of the visual inertial system, the constant to-be-calibrated parameter at the current moment is determined.
- 根据权利要求12所述的方法,其中,所述根据所述视觉惯性***的运动状态,确定在所述当前时刻保持固定的待标定参数,包括以下至少之一:The method according to claim 12, wherein the determining, according to the motion state of the visual inertial system, the parameters to be calibrated that remain fixed at the current moment, comprising at least one of the following:在所述视觉惯性***静止的情况下,确定在所述当前时刻保持固定的待标定参数包括时间偏移、所述惯性传感器与所述视觉传感器之间的平移参数以及所述惯性传感器与所述视觉传感器之间的旋转参数;In the case that the visual inertial system is stationary, it is determined that the parameters to be calibrated that remain fixed at the current moment include a time offset, a translation parameter between the inertial sensor and the visual sensor, and the inertial sensor and the Rotation parameters between vision sensors;在所述视觉传感器或者所述惯性传感器的平移参数为0且旋转参数不为0的情况下,确定在所述当前时刻保持固定的待标定参数包括所述惯性传感器与所述视觉传感器之间的平移参数;In the case that the translation parameter of the visual sensor or the inertial sensor is 0 and the rotation parameter is not 0, it is determined that the parameter to be calibrated that remains fixed at the current moment includes the difference between the inertial sensor and the visual sensor. translation parameter;在所述视觉传感器或者所述惯性传感器绕着x轴、y轴和z轴中的一个轴旋转的情况下,确定在所述当前时刻保持固定的待标定参数包括所述惯性传感器与所述视觉传感器之间的平移参数中旋转轴对应的参数;In the case that the visual sensor or the inertial sensor rotates around one of the x-axis, the y-axis and the z-axis, it is determined that the parameters to be calibrated that remain fixed at the current moment include the inertial sensor and the visual sensor The parameters corresponding to the rotation axis in the translation parameters between sensors;在所述视觉传感器或者所述惯性传感器绕着任一坐标轴以恒定的角速度运动的情况下,确定在所述当前时刻保持固定的待标定参数包括时间偏移以及所述惯性传感器与所述视觉传感器之间的平移参数中所述坐标轴对应的参数;In the case that the visual sensor or the inertial sensor moves around any coordinate axis at a constant angular velocity, it is determined that the parameters to be calibrated that remain fixed at the current moment include a time offset and the inertial sensor and the visual sensor The parameters corresponding to the coordinate axes in the translation parameters between;在所述视觉传感器或者所述惯性传感器沿着任一坐标轴以恒定的速度运动的情况下,确定在所述当前时刻保持固定的待标定参数包括时间偏移以及所述惯性传感器与所述视觉传感器之间的平移参数中所述坐标轴对应的参数;In the case that the visual sensor or the inertial sensor moves at a constant speed along any coordinate axis, it is determined that the parameters to be calibrated that remain fixed at the current moment include a time offset and the inertial sensor and the visual sensor The parameters corresponding to the coordinate axes in the translation parameters between;在所述视觉传感器或者所述惯性传感器绕着任一坐标轴以恒定的角加速度运动的情况下,确定在所述当前时刻保持固定的待标定参数包括时间偏移以及所述惯性传感器与所述视觉传感器之间的平移参数中所述坐标轴对应的参数。In the case that the visual sensor or the inertial sensor moves around any coordinate axis with a constant angular acceleration, it is determined that the parameters to be calibrated that remain fixed at the current moment include a time offset and the relationship between the inertial sensor and the visual sensor. The parameter corresponding to the coordinate axis in the translation parameter between sensors.
- 根据权利要求1至13中任意一项所述的方法,其中,所述方法还包括:The method according to any one of claims 1 to 13, wherein the method further comprises:根据所述视觉惯性***的跟踪状态,确定所述当前时刻的保持固定的待标定参数。According to the tracking state of the visual inertial system, the constant to-be-calibrated parameter at the current moment is determined.
- 根据权利要求14所述的方法,其中,所述根据所述视觉惯性***的跟踪状态,确定所述当前时刻的保持固定的待标定参数,包括:The method according to claim 14, wherein the determining the constant to-be-calibrated parameter at the current moment according to the tracking state of the visual-inertial system comprises:在所述视觉惯性***跟踪丢失的情况下,确定在所述当前时刻保持固定的待标定参数包括时间偏移、所述惯性传感器与所述视觉传感器之间的平移参数以及所述惯性传感器与所述视觉传感器之间的旋转参数;和/或,在所述视觉惯性***当前跟踪到的特征点的数量小于预设数量的情况下,确定在所述当前时刻保持固定的待标定参数包括时间偏移、所述惯性传感器与所述视觉传感器之间的平移参数以及所述惯性传感器与所述视觉传感器之间的旋转参数。In the case that the tracking of the visual inertial system is lost, it is determined that the parameters to be calibrated that remain fixed at the current moment include a time offset, a translation parameter between the inertial sensor and the visual sensor, and the inertial sensor and the and/or, when the number of feature points currently tracked by the visual inertial system is less than a preset number, it is determined that the parameters to be calibrated that remain fixed at the current moment include a time offset translation, translation parameters between the inertial sensor and the vision sensor, and rotation parameters between the inertial sensor and the vision sensor.
- 一种视觉惯性***的参数标定装置,所述视觉惯性***包括视觉传感器和惯性传感器,所述装置包括:A parameter calibration device for a visual inertial system, the visual inertial system includes a visual sensor and an inertial sensor, and the device includes:获取模块,配置为获取所述视觉惯性***在上一时刻的状态数据;an acquisition module, configured to acquire the state data of the visual inertial system at the last moment;估计模块,配置为根据所述视觉惯性***在所述上一时刻的状态数据,所述视觉传感器在当前时刻采集的图像中的特征点的坐标,以及所述惯性传感器在所述当前时刻的输出数据,估计所述视觉惯性***在所述当前时刻的状态数据;an estimation module, configured to be based on the state data of the visual inertial system at the last moment, the coordinates of the feature points in the image collected by the visual sensor at the current moment, and the output of the inertial sensor at the current moment data, estimating the state data of the visual-inertial system at the current moment;标定模块,配置为根据所述视觉惯性***在所述当前时刻的状态数据,对所述视觉惯性***的待标定参数进行标定。The calibration module is configured to calibrate the parameters to be calibrated of the visual inertial system according to the state data of the visual inertial system at the current moment.
- 根据权利要求16所述的装置,其中,所述视觉惯性***在所述上一时刻的状态数据包括:所述视觉传感器在所述上一时刻的状态数据、所述惯性传感器在所述上一时刻的状态数据以及所述 待标定参数在所述上一时刻的数值;所述视觉惯性***在所述当前时刻的状态数据包括:所述视觉传感器在所述当前时刻的状态数据、所述惯性传感器在所述当前时刻的状态数据以及所述待标定参数在所述当前时刻的数值。The device according to claim 16, wherein the state data of the visual inertial system at the last moment comprises: the state data of the visual sensor at the last moment, the inertial sensor at the last moment The state data at the moment and the value of the parameter to be calibrated at the previous moment; the status data of the visual inertial system at the current moment includes: the state data of the visual sensor at the current moment, the inertial The state data of the sensor at the current moment and the value of the parameter to be calibrated at the current moment.
- 根据权利要求17所述的装置,其中,所述视觉传感器在所述上一时刻的状态数据包括:所述视觉传感器在所述上一时刻的平移参数和所述视觉传感器在所述上一时刻的旋转参数;所述视觉传感器在所述当前时刻的状态数据包括:所述视觉传感器在所述当前时刻的平移参数和所述视觉传感器在所述当前时刻的旋转参数。The device according to claim 17, wherein the state data of the vision sensor at the last moment comprises: translation parameters of the vision sensor at the last moment and the vision sensor at the last moment The rotation parameter of the visual sensor at the current moment includes: the translation parameter of the visual sensor at the current moment and the rotation parameter of the visual sensor at the current moment.
- 根据权利要求17或18所述的装置,其中,所述惯性传感器在所述上一时刻的状态数据包括:所述惯性传感器在所述上一时刻的速度;所述惯性传感器在所述当前时刻的状态数据包括:所述惯性传感器在所述当前时刻的速度。The device according to claim 17 or 18, wherein the state data of the inertial sensor at the last moment comprises: the speed of the inertial sensor at the last moment; the inertial sensor at the current moment The status data includes: the speed of the inertial sensor at the current moment.
- 根据权利要求19所述的装置,其中,所述惯性传感器在所述上一时刻的状态数据还包括:所述惯性传感器在所述上一时刻的加速度计的零偏以及所述惯性传感器在所述上一时刻的陀螺仪的零偏;所述惯性传感器在所述当前时刻的状态数据还包括:所述惯性传感器在所述当前时刻的加速度计的零偏以及所述惯性传感器在所述当前时刻的陀螺仪的零偏。The device according to claim 19, wherein the state data of the inertial sensor at the last moment further comprises: the zero offset of the accelerometer of the inertial sensor at the last moment and the zero offset of the inertial sensor at the last moment The zero offset of the gyroscope at the previous moment; the state data of the inertial sensor at the current moment further includes: the zero offset of the inertial sensor at the current moment and the zero offset of the inertial sensor at the current moment The zero offset of the gyroscope at the moment.
- 根据权利要求16至20中任一项所述的装置,其中,所述待标定参数包括时间偏移,其中,所述时间偏移表示在同一时刻所述视觉传感器的时钟与所述惯性传感器的时钟之间的差值;21. The apparatus of any one of claims 16 to 20, wherein the parameter to be calibrated includes a time offset, wherein the time offset represents the difference between the clock of the visual sensor and the inertial sensor at the same time. difference between clocks;所述标定模块,配置为在所述视觉惯性***在所述当前时刻的状态数据中,所述惯性传感器的加速度计的零偏满足第一预设稳定性条件,且所述惯性传感器的陀螺仪的零偏满足第二预设稳定性条件的情况下,开启对所述时间偏移的标定。The calibration module is configured to, in the state data of the visual inertial system at the current moment, the zero offset of the accelerometer of the inertial sensor satisfies a first preset stability condition, and the gyroscope of the inertial sensor In the case that the zero offset of satisfies the second preset stability condition, the calibration of the time offset is enabled.
- 根据权利要求16至21所述的装置,其中,所述待标定参数包括所述视觉传感器的内参;所述标定模块,配置为在所述视觉惯性***在所述当前时刻的状态数据中,所述惯性传感器的加速度计的零偏满足第三预设稳定性条件的情况下,开启对所述视觉传感器的内参的标定,其中,所述第三预设稳定性条件对应的加速度计的零偏的稳定性,高于第一预设稳定性条件对应的加速度计的零偏的稳定性。The device according to claims 16 to 21, wherein the parameter to be calibrated includes an internal parameter of the visual sensor; the calibration module is configured to, in the state data of the visual inertial system at the current moment, the When the zero bias of the accelerometer of the inertial sensor satisfies the third preset stability condition, the calibration of the internal parameters of the visual sensor is enabled, wherein the zero bias of the accelerometer corresponding to the third preset stability condition The stability of the accelerometer is higher than the stability of the zero bias of the accelerometer corresponding to the first preset stability condition.
- 根据权利要求22所述的装置,其中,所述视觉传感器的内参包括所述视觉传感器的焦距和光心;所述装置还包括:第一确定模块,配置为在获取得到所述视觉传感器的视场角的情况下,根据所述视觉传感器采集的图像的宽度以及所述视觉传感器的视场角,确定所述视觉传感器的焦距的初始值;或者,第二确定模块,配置为在获取不到所述视觉传感器的视场角的情况下,根据所述视觉传感器采集的图像的宽度和高度,确定所述视觉传感器的焦距的初始值。The device according to claim 22, wherein the internal parameters of the visual sensor include the focal length and the optical center of the visual sensor; the device further comprises: a first determining module, configured to obtain a field of view of the visual sensor after acquiring In the case of angle, the initial value of the focal length of the visual sensor is determined according to the width of the image collected by the visual sensor and the field of view of the visual sensor; In the case of the field angle of the vision sensor, the initial value of the focal length of the vision sensor is determined according to the width and height of the image collected by the vision sensor.
- 根据权利要求22或23所述的装置,其中,所述视觉传感器的内参包括所述视觉传感器的光心坐标,所述装置还包括:第三确定模块,配置为根据所述视觉传感器采集的图像的宽度和高度,确定所述视觉传感器的光心坐标的初始值。The apparatus according to claim 22 or 23, wherein the internal reference of the visual sensor includes the optical center coordinates of the visual sensor, and the apparatus further comprises: a third determination module configured to be configured according to the image collected by the visual sensor The width and height determine the initial values of the optical center coordinates of the vision sensor.
- 根据权利要求16至24中任一项所述的装置,其中,所述待标定参数包括所述惯性传感器与所述视觉传感器之间的坐标转换参数,其中,所述坐标转换参数包括所述惯性传感器与所述视觉传感器之间的平移参数和旋转参数中的至少之一;所述标定模块,配置为在所述视觉惯性***在所述当前时刻的状态数据中,所述惯性传感器的加速度计的零偏满足第四预设稳定性条件的情况下,开启对所述坐标转换参数的标定,其中,所述第四预设稳定性条件对应的加速度计的零偏的稳定性,高于第一预设稳定性条件对应的加速度计的零偏的稳定性。The apparatus according to any one of claims 16 to 24, wherein the parameter to be calibrated comprises a coordinate conversion parameter between the inertial sensor and the visual sensor, wherein the coordinate conversion parameter comprises the inertial sensor at least one of translation parameters and rotation parameters between the sensor and the visual sensor; the calibration module is configured to, in the state data of the visual-inertial system at the current moment, the accelerometer of the inertial sensor In the case where the zero offset of 2 satisfies the fourth preset stability condition, the calibration of the coordinate conversion parameter is enabled, wherein the stability of the zero offset of the accelerometer corresponding to the fourth preset stability condition is higher than that of the fourth preset stability condition. The stability of the zero bias of the accelerometer corresponding to a preset stability condition.
- 根据权利要求16至25中任一项所述的装置,其中,所述待标定参数包括所述视觉传感器的畸变参数,所述标定模块,配置为在根据所述视觉惯性***在所述当前时刻的状态数据,确定所述视觉传感器的内参满足第五预设稳定性条件的情况下,开启对所述视觉传感器的畸变参数的标定。The device according to any one of claims 16 to 25, wherein the parameter to be calibrated includes a distortion parameter of the visual sensor, and the calibration module is configured to perform the calibration at the current moment according to the visual inertial system When it is determined that the internal parameter of the visual sensor meets the fifth preset stability condition, the calibration of the distortion parameter of the visual sensor is started.
- 根据权利要求16至26中任一项所述的装置,其中,所述装置还包括:第四确定模块,配置为根据所述视觉惯性***的运动状态,确定所述当前时刻的保持固定的待标定参数。The device according to any one of claims 16 to 26, wherein the device further comprises: a fourth determination module, configured to determine, according to the motion state of the visual inertial system, the current moment to remain fixed Calibration parameters.
- 根据权利要求27所述的装置,其中,所述第四确定模块配置为至少一些之一:The apparatus of claim 27, wherein the fourth determination module is configured to be at least one of:在所述视觉惯性***静止的情况下,确定在所述当前时刻保持固定的待标定参数包括时间偏移、所述惯性传感器与所述视觉传感器之间的平移参数以及所述惯性传感器与所述视觉传感器之间的旋转参数;In the case that the visual inertial system is stationary, it is determined that the parameters to be calibrated that remain fixed at the current moment include a time offset, a translation parameter between the inertial sensor and the visual sensor, and the inertial sensor and the Rotation parameters between vision sensors;在所述视觉传感器或者所述惯性传感器的平移参数为0且旋转参数不为0的情况下,确定在所述当前时刻保持固定的待标定参数包括所述惯性传感器与所述视觉传感器之间的平移参数;In the case that the translation parameter of the visual sensor or the inertial sensor is 0 and the rotation parameter is not 0, it is determined that the parameter to be calibrated that remains fixed at the current moment includes the difference between the inertial sensor and the visual sensor. translation parameter;在所述视觉传感器或者所述惯性传感器绕着x轴、y轴和z轴中的一个轴旋转的情况下,确定在所述当前时刻保持固定的待标定参数包括所述惯性传感器与所述视觉传感器之间的平移参数中旋转 轴对应的参数;In the case that the visual sensor or the inertial sensor rotates around one of the x-axis, the y-axis and the z-axis, it is determined that the parameters to be calibrated that remain fixed at the current moment include the inertial sensor and the visual sensor The parameters corresponding to the rotation axis in the translation parameters between sensors;在所述视觉传感器或者所述惯性传感器绕着任一坐标轴以恒定的角速度运动的情况下,确定在所述当前时刻保持固定的待标定参数包括时间偏移以及所述惯性传感器与所述视觉传感器之间的平移参数中所述坐标轴对应的参数;In the case that the visual sensor or the inertial sensor moves around any coordinate axis at a constant angular velocity, it is determined that the parameters to be calibrated that remain fixed at the current moment include a time offset and the inertial sensor and the visual sensor The parameters corresponding to the coordinate axes in the translation parameters between;在所述视觉传感器或者所述惯性传感器沿着任一坐标轴以恒定的速度运动的情况下,确定在所述当前时刻保持固定的待标定参数包括时间偏移以及所述惯性传感器与所述视觉传感器之间的平移参数中所述坐标轴对应的参数;In the case that the visual sensor or the inertial sensor moves at a constant speed along any coordinate axis, it is determined that the parameters to be calibrated that remain fixed at the current moment include a time offset and the inertial sensor and the visual sensor The parameters corresponding to the coordinate axes in the translation parameters between;在所述视觉传感器或者所述惯性传感器绕着任一坐标轴以恒定的角加速度运动的情况下,确定在所述当前时刻保持固定的待标定参数包括时间偏移以及所述惯性传感器与所述视觉传感器之间的平移参数中所述坐标轴对应的参数。In the case that the visual sensor or the inertial sensor moves around any coordinate axis with a constant angular acceleration, it is determined that the parameters to be calibrated that remain fixed at the current moment include a time offset and the relationship between the inertial sensor and the visual sensor. The parameter corresponding to the coordinate axis in the translation parameter between sensors.
- 根据权利要求16至23中任一项所述的装置,其中,所述装置还包括第五确定模块,配置为根据所述视觉惯性***的跟踪状态,确定所述当前时刻的保持固定的待标定参数。The device according to any one of claims 16 to 23, wherein the device further comprises a fifth determination module, configured to determine, according to the tracking state of the visual-inertial system, the to-be-calibrated remaining fixed at the current moment parameter.
- 根据权利要求29所述的装置,其中,所述第五确定模块配置为在所述视觉惯性***跟踪丢失的情况下,确定在所述当前时刻保持固定的待标定参数包括时间偏移、所述惯性传感器与所述视觉传感器之间的平移参数以及所述惯性传感器与所述视觉传感器之间的旋转参数;和/或,The apparatus according to claim 29, wherein the fifth determining module is configured to determine the parameters to be calibrated that remain fixed at the current moment in the case that the visual-inertial system tracking is lost, including a time offset, the Translation parameters between the inertial sensor and the vision sensor and rotation parameters between the inertial sensor and the vision sensor; and/or,在所述视觉惯性***当前跟踪到的特征点的数量小于预设数量的情况下,确定在所述当前时刻保持固定的待标定参数包括时间偏移、所述惯性传感器与所述视觉传感器之间的平移参数以及所述惯性传感器与所述视觉传感器之间的旋转参数。In the case that the number of feature points currently tracked by the visual inertial system is less than a preset number, it is determined that the parameters to be calibrated that remain fixed at the current moment include a time offset, a difference between the inertial sensor and the visual sensor The translation parameters of and the rotation parameters between the inertial sensor and the vision sensor.
- 一种电子设备,包括:An electronic device comprising:一个或多个处理器;用于存储可执行指令的存储器;one or more processors; memory for storing executable instructions;其中,所述一个或多个处理器被配置为调用所述存储器存储的可执行指令,以执行权利要求1至15中任意一项所述的方法。wherein the one or more processors are configured to invoke executable instructions stored in the memory to perform the method of any one of claims 1-15.
- 一种计算机可读存储介质,其上存储有计算机程序指令,所述计算机程序指令被处理器执行时实现权利要求1至15中任意一项所述的方法。A computer-readable storage medium having computer program instructions stored thereon, the computer program instructions implementing the method of any one of claims 1 to 15 when executed by a processor.
- 一种计算机程序,包括计算机可读代码,当所述计算机可读代码在电子设备中运行时,所述电子设备中的处理器执行配置为实现权利要求1至15中任意一项所述的方法。A computer program comprising computer readable code, when the computer readable code is run in an electronic device, a processor in the electronic device executes a method configured to implement any one of claims 1 to 15 .
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202011281318.3 | 2020-11-16 | ||
CN202011281318.3A CN112229424B (en) | 2020-11-16 | 2020-11-16 | Parameter calibration method and device for visual inertial system, electronic equipment and medium |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2022100189A1 true WO2022100189A1 (en) | 2022-05-19 |
Family
ID=74123467
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/CN2021/113113 WO2022100189A1 (en) | 2020-11-16 | 2021-08-17 | Method and apparatus for calibrating parameters of visual-inertial system, and electronic device and medium |
Country Status (2)
Country | Link |
---|---|
CN (5) | CN114812610B (en) |
WO (1) | WO2022100189A1 (en) |
Families Citing this family (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN114812610B (en) * | 2020-11-16 | 2024-07-16 | 浙江商汤科技开发有限公司 | Parameter calibration method and device of visual inertial system, electronic equipment and medium |
CN113029134B (en) * | 2021-03-02 | 2022-04-08 | 浙江商汤科技开发有限公司 | Visual inertial system initialization method and device, electronic equipment and storage medium |
CN112989625B (en) * | 2021-04-02 | 2024-04-05 | 上海有个机器人有限公司 | Method for eliminating abnormal value of UWB sensor |
CN113932805B (en) * | 2021-10-12 | 2024-02-23 | 天翼数字生活科技有限公司 | Method for improving positioning accuracy and speed of AR virtual object |
CN115752442B (en) * | 2022-12-07 | 2024-03-12 | 运来智能装备(无锡)有限公司 | Monocular vision-based auxiliary inertial positioning method |
Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20180164124A1 (en) * | 2016-09-15 | 2018-06-14 | Syracuse University | Robust and stable autonomous vision-inertial navigation system for unmanned vehicles |
CN109443355A (en) * | 2018-12-25 | 2019-03-08 | 中北大学 | Vision based on adaptive Gauss PF-inertia close coupling Combinated navigation method |
CN110411476A (en) * | 2019-07-29 | 2019-11-05 | 视辰信息科技(上海)有限公司 | Vision inertia odometer calibration adaptation and evaluation method and system |
CN111156997A (en) * | 2020-03-02 | 2020-05-15 | 南京航空航天大学 | Vision/inertia combined navigation method based on camera internal parameter online calibration |
CN111551191A (en) * | 2020-04-28 | 2020-08-18 | 浙江商汤科技开发有限公司 | Sensor external parameter calibration method and device, electronic equipment and storage medium |
CN112229424A (en) * | 2020-11-16 | 2021-01-15 | 浙江商汤科技开发有限公司 | Parameter calibration method and device for visual inertial system, electronic equipment and medium |
Family Cites Families (15)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US9031809B1 (en) * | 2010-07-14 | 2015-05-12 | Sri International | Method and apparatus for generating three-dimensional pose using multi-modal sensor fusion |
CN104021554B (en) * | 2014-04-23 | 2017-03-01 | 北京大学深圳研究生院 | Camera inertial sensor scaling method based on operative sensor information |
US10012504B2 (en) * | 2014-06-19 | 2018-07-03 | Regents Of The University Of Minnesota | Efficient vision-aided inertial navigation using a rolling-shutter camera with inaccurate timestamps |
EP3158293B1 (en) * | 2015-05-23 | 2019-01-23 | SZ DJI Technology Co., Ltd. | Sensor fusion using inertial and image sensors |
CN106679648B (en) * | 2016-12-08 | 2019-12-10 | 东南大学 | Visual inertia combination SLAM method based on genetic algorithm |
CN107564061B (en) * | 2017-08-11 | 2020-11-20 | 浙江大学 | Binocular vision mileage calculation method based on image gradient joint optimization |
CN108489482B (en) * | 2018-02-13 | 2019-02-26 | 视辰信息科技(上海)有限公司 | The realization method and system of vision inertia odometer |
CN109029433B (en) * | 2018-06-28 | 2020-12-11 | 东南大学 | Method for calibrating external parameters and time sequence based on vision and inertial navigation fusion SLAM on mobile platform |
CN109376785B (en) * | 2018-10-31 | 2021-09-24 | 东南大学 | Navigation method based on iterative extended Kalman filtering fusion inertia and monocular vision |
CN109685852B (en) * | 2018-11-22 | 2020-08-21 | 上海肇观电子科技有限公司 | Calibration method, system, equipment and storage medium for camera and inertial sensor |
CN110246194A (en) * | 2019-06-25 | 2019-09-17 | 中国人民解放军国防科技大学 | Method for quickly calibrating rotation relation between camera and inertia measurement unit |
CN111308415B (en) * | 2019-11-01 | 2022-09-02 | 华为技术有限公司 | Online pose estimation method and equipment based on time delay |
CN110987021B (en) * | 2019-12-25 | 2021-08-24 | 湖北航天技术研究院总体设计所 | Inertial vision relative attitude calibration method based on rotary table reference |
CN111275769B (en) * | 2020-01-17 | 2023-10-24 | 联想(北京)有限公司 | Monocular vision parameter correction method and device |
CN111429524B (en) * | 2020-03-19 | 2023-04-18 | 上海交通大学 | Online initialization and calibration method and system for camera and inertial measurement unit |
-
2020
- 2020-11-16 CN CN202210414649.2A patent/CN114812610B/en active Active
- 2020-11-16 CN CN202210398549.5A patent/CN114812608A/en active Pending
- 2020-11-16 CN CN202210398676.5A patent/CN114812609A/en active Pending
- 2020-11-16 CN CN202011281318.3A patent/CN112229424B/en active Active
- 2020-11-16 CN CN202210398679.9A patent/CN114791291B/en active Active
-
2021
- 2021-08-17 WO PCT/CN2021/113113 patent/WO2022100189A1/en active Application Filing
Patent Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20180164124A1 (en) * | 2016-09-15 | 2018-06-14 | Syracuse University | Robust and stable autonomous vision-inertial navigation system for unmanned vehicles |
CN109443355A (en) * | 2018-12-25 | 2019-03-08 | 中北大学 | Vision based on adaptive Gauss PF-inertia close coupling Combinated navigation method |
CN110411476A (en) * | 2019-07-29 | 2019-11-05 | 视辰信息科技(上海)有限公司 | Vision inertia odometer calibration adaptation and evaluation method and system |
CN111156997A (en) * | 2020-03-02 | 2020-05-15 | 南京航空航天大学 | Vision/inertia combined navigation method based on camera internal parameter online calibration |
CN111551191A (en) * | 2020-04-28 | 2020-08-18 | 浙江商汤科技开发有限公司 | Sensor external parameter calibration method and device, electronic equipment and storage medium |
CN112229424A (en) * | 2020-11-16 | 2021-01-15 | 浙江商汤科技开发有限公司 | Parameter calibration method and device for visual inertial system, electronic equipment and medium |
Also Published As
Publication number | Publication date |
---|---|
CN114791291B (en) | 2024-06-07 |
CN112229424B (en) | 2022-04-22 |
CN114812608A (en) | 2022-07-29 |
CN114812609A (en) | 2022-07-29 |
CN112229424A (en) | 2021-01-15 |
CN114812610B (en) | 2024-07-16 |
CN114791291A (en) | 2022-07-26 |
CN114812610A (en) | 2022-07-29 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
WO2022100189A1 (en) | Method and apparatus for calibrating parameters of visual-inertial system, and electronic device and medium | |
JP7236565B2 (en) | POSITION AND ATTITUDE DETERMINATION METHOD, APPARATUS, ELECTRONIC DEVICE, STORAGE MEDIUM AND COMPUTER PROGRAM | |
CN111551191B (en) | Sensor external parameter calibration method and device, electronic equipment and storage medium | |
US20150304652A1 (en) | Device orientation correction method for panorama images | |
CN109186592B (en) | Method and device for visual and inertial navigation information fusion and storage medium | |
CN113825055B (en) | Head-to-headphone rotation transform estimation for head pose tracking in spatial audio applications | |
US20210158560A1 (en) | Method and device for obtaining localization information and storage medium | |
JP7182020B2 (en) | Information processing method, device, electronic device, storage medium and program | |
US11042984B2 (en) | Systems and methods for providing image depth information | |
CN111625764B (en) | Mobile data calibration method, device, electronic equipment and storage medium | |
WO2022134475A1 (en) | Point cloud map construction method and apparatus, electronic device, storage medium and program | |
CN113074726A (en) | Pose determination method and device, electronic equipment and storage medium | |
CN112146645A (en) | Method and device for aligning coordinate system, electronic equipment and storage medium | |
JP7477596B2 (en) | Method, depth estimation system, and computer program for depth estimation | |
CN112767541A (en) | Three-dimensional reconstruction method and device, electronic equipment and storage medium | |
Henrik | Fusion of IMU and Monocular-SLAM in a Loosely Coupled EKF | |
CN118135042A (en) | Image generation method, device, electronic equipment and storage medium | |
CN112308878A (en) | Information processing method and device, electronic equipment and storage medium | |
CN118138881A (en) | Image processing method, device, electronic equipment and storage medium | |
CN112837361A (en) | Depth estimation method and device, electronic equipment and storage medium |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 21890718 Country of ref document: EP Kind code of ref document: A1 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 21890718 Country of ref document: EP Kind code of ref document: A1 |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 21890718 Country of ref document: EP Kind code of ref document: A1 |
|
32PN | Ep: public notification in the ep bulletin as address of the adressee cannot be established |
Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 28/11/2023) |