CN113140039A

CN113140039A - Multi-sensor fusion underground coal mine digital positioning and map construction system

Info

Publication number: CN113140039A
Application number: CN202110454473.9A
Authority: CN
Inventors: 南柄飞; 叶晨曦; 郭志杰; 王凯; 荣耀; 李森; 李首滨
Original assignee: Beijing Tiandi Marco Electro Hydraulic Control System Co Ltd; Beijing Meike Tianma Automation Technology Co Ltd
Current assignee: Beijing Tiandi Marco Electro Hydraulic Control System Co Ltd; Beijing Meike Tianma Automation Technology Co Ltd
Priority date: 2021-04-26
Filing date: 2021-04-26
Publication date: 2021-07-20

Abstract

The application discloses colliery of multisensor integration is digital location and map construction system in pit, this system includes: the device comprises a processing assembly and an explosion-proof assembly; the processing assembly comprises a spatial data acquisition unit, a visual image acquisition unit arranged below the spatial data acquisition unit and a data processing terminal arranged below the visual image acquisition unit; the explosion-proof assembly comprises an explosion-proof cover and an explosion-proof shell, wherein the spatial data acquisition unit is positioned in the covering range of the explosion-proof cover; the flameproof housing comprises a light-transmitting area and a light-proof area, and the light-transmitting area corresponds to the visual angles of the spatial data acquisition unit and the visual image acquisition unit. Therefore, the sensing capability complementation is carried out through the multiple sensors, the stability of the system is improved, and the robustness of the system is ensured.

Description

Multi-sensor fusion underground coal mine digital positioning and map construction system

Technical Field

The application relates to the technical field of coal mining, in particular to a coal mine underground digital positioning and map building system.

Background

Coal is the main energy in China, and plays an important role in promoting national industrial development, national economic progress and the like. In recent years, along with the rapid improvement of the intelligent level of mining equipment, the coal yield of coal mines in China is greatly improved. However, in the coal mine excavation process, due to the fact that the underground scene environment of the coal mine is severe, the operation equipment and the associated target object are complex and intricate, and the operators are in the space environment with high danger coefficients for long time to operate. Therefore, in order to guarantee the life safety of underground personnel and realize intelligent unmanned, green and efficient mining of coal mines, the digital positioning and scene reconstruction of underground coal mine space become one of important technical development directions.

In the related art, a signal-based communication mode is usually adopted for positioning, however, because the signal strength is extremely depended on, a large number of base stations are usually required to be arranged underground, and the technical problems of complex operation and high cost exist; meanwhile, based on the characteristics of underground operation of the coal mine, the positioning accuracy is low due to the fact that related signals are affected by the underground special scene environment (such as coal wall absorption). Therefore, how to generate a high-precision coal mine underground global space map so as to realize more accurate positioning, eliminate potential safety hazards and ensure personnel safety becomes a problem to be solved urgently.

Disclosure of Invention

The present application is directed to solving, at least to some extent, one of the technical problems in the related art.

Therefore, the first purpose of the application is to provide a method for positioning and mapping underground coal mine space by multi-sensor information fusion, which is used for solving the technical problems of poor positioning accuracy and potential safety hazards caused by low mapping precision of underground coal mine global space in the prior art.

In order to achieve the above object, an embodiment of the present application provides a method for positioning and mapping a coal mine underground space with multi-sensor information fusion, where the method includes: acquiring 1 st to Nth frames of visual images and point cloud data corresponding to the visual images, which are sequentially acquired by a sensor, wherein N is a positive integer; acquiring visual pose information according to the visual image; obtaining IMU data of an inertial measurement unit IMU to obtain IMU pose information according to the IMU data; acquiring target fusion pose information according to the visual pose information and the IMU pose information; correcting the target fusion pose information according to the point cloud data to obtain final target pose information; and generating a global space semantic map according to the final target pose information.

In addition, according to the method for positioning and mapping the underground coal mine space with the multi-sensor information fusion, which is disclosed by the embodiment of the application, the method can further have the following additional technical characteristics:

according to an embodiment of the application, the acquiring visual pose information according to the visual image includes: extracting visual features of the visual image according to the visual image; after the visual features of the visual image are extracted, camera initialization is carried out, and whether the camera initialization is successful or not is judged; and responding to the successful initialization of the camera, and performing visual feature matching and tracking to obtain visual pose information.

According to an embodiment of the present application, the acquiring target fusion pose information according to the visual pose information and the IMU pose information includes: performing IMU initialization and judging whether the IMU initialization is successful or not; responding to the successful IMU initialization, acquiring fusion pose information according to the visual pose information and the IMU pose information; and optimizing the fusion pose information based on a sliding window nonlinear optimization algorithm to obtain the target fusion pose information.

According to an embodiment of the present application, the performing, in response to successful initialization of the camera, visual feature matching and tracking to obtain visual pose information includes: respectively determining the number of the matched feature points and the number of the matched feature points in the 1 st to Nth frames of visual images, and judging whether visual feature matching and tracking are successful or not; confirming that the visual feature matching and tracking are successful in response to the fact that the number of the matched feature points reaches a first preset number threshold value, so as to obtain visual feature matching and tracking information; and acquiring the visual pose information according to the visual feature matching and tracking information.

According to an embodiment of the present application, after performing visual feature matching and tracking to obtain visual pose information in response to successful camera initialization, the method further includes: acquiring the visual pose information of adjacent frames; and selecting a visual key frame image according to the visual pose information of the adjacent frames.

According to an embodiment of the present application, the sliding window based nonlinear optimization algorithm optimizing the fusion pose information to obtain the target fusion pose information includes: optimizing the fusion pose information based on a sliding window nonlinear optimization algorithm to obtain the optimized fusion pose information; according to the visual key frame image, acquiring the feature similarity of the current visual key frame image and the historical visual key frame image; judging whether a closed loop state exists according to the feature similarity; and responding to the existence of the closed loop state, acquiring closed loop visual key frame images, and performing global closed loop nonlinear optimization on the fusion pose information corresponding to all the visual key frame images to obtain the target fusion pose information.

According to an embodiment of the present application, the correcting the target fusion pose information according to the point cloud data to obtain final target pose information includes: according to the target fusion pose information, carrying out distortion removal operation on the point cloud data to obtain accurate point cloud data; extracting target point cloud characteristics according to the accurate point cloud data; and matching and tracking the target point cloud characteristics in a nonlinear optimization mode by taking the target fusion pose information as an initial value to obtain the final target pose information.

According to an embodiment of the present application, the generating a global spatial semantic map according to the final target pose information includes: generating a global space map according to the final target pose information; and performing point cloud coloring processing on the global space map to generate the global space semantic map.

According to the method for positioning and constructing the underground coal mine space through multi-sensor information fusion, fusion processing can be performed on various sensor information, generation of the global space map with scene rich semantic information is achieved, real-time digital positioning and three-dimensional space construction of the underground coal mine scene are achieved, and accuracy of underground space positioning and three-dimensional construction is improved.

In order to achieve the above object, an embodiment of a second aspect of the present application provides a coal mine underground space positioning and mapping device with multi-sensor information fusion, the device including: the system comprises a first acquisition module, a second acquisition module and a third acquisition module, wherein the first acquisition module is used for acquiring 1 st to Nth frames of visual images which are sequentially acquired by a sensor and point cloud data corresponding to the visual images, and N is a positive integer; the second acquisition module is used for acquiring visual pose information according to the visual image; the third acquisition module is used for acquiring IMU data of the inertial measurement unit IMU so as to acquire IMU pose information according to the IMU data; the fourth acquisition module is used for acquiring target fusion pose information according to the visual pose information and the IMU pose information; the determining module is used for correcting the target fusion pose information according to the point cloud data to obtain final target pose information; and the generating module is used for generating a global space semantic map according to the final target pose information.

In addition, according to the coal mine underground space positioning and mapping device with multi-sensor information fusion of the embodiment of the application, the device can also have the following additional technical characteristics:

according to an embodiment of the present application, the second obtaining module is further configured to: extracting visual features of the visual image according to the visual image; after the visual features of the visual image are extracted, camera initialization is carried out, and whether the camera initialization is successful or not is judged; and responding to the successful initialization of the camera, and performing visual feature matching and tracking to obtain visual pose information.

According to an embodiment of the present application, the fourth obtaining module is further configured to: performing IMU initialization and judging whether the IMU initialization is successful or not; responding to the successful IMU initialization, acquiring fusion pose information according to the visual pose information and the IMU pose information; and optimizing the fusion pose information based on a sliding window nonlinear optimization algorithm to obtain the target fusion pose information.

According to an embodiment of the present application, the fourth obtaining module is further configured to: respectively determining the number of the matched feature points and the number of the matched feature points in the 1 st to Nth frames of visual images, and judging whether visual feature matching and tracking are successful or not; confirming that the visual feature matching and tracking are successful in response to the fact that the number of the matched feature points reaches a first preset number threshold value, so as to obtain visual feature matching and tracking information; and acquiring the visual pose information according to the visual feature matching and tracking information.

According to an embodiment of the present application, the fourth obtaining module is further configured to: acquiring the visual pose information of adjacent frames; and selecting a visual key frame image according to the visual pose information of the adjacent frames.

According to an embodiment of the present application, the fourth obtaining module is further configured to: optimizing the fusion pose information based on a sliding window nonlinear optimization algorithm to obtain the optimized fusion pose information; according to the visual key frame image, acquiring the feature similarity of the current visual key frame image and the historical visual key frame image; judging whether a closed loop state exists according to the feature similarity; and responding to the existence of the closed loop state, acquiring closed loop visual key frame images, and performing global closed loop nonlinear optimization on the fusion pose information corresponding to all the visual key frame images to obtain the target fusion pose information.

According to an embodiment of the present application, the determining module is further configured to: according to the target fusion pose information, carrying out distortion removal operation on the point cloud data to obtain accurate point cloud data; extracting target point cloud characteristics according to the accurate point cloud data; and matching and tracking the target point cloud characteristics in a nonlinear optimization mode by taking the target fusion pose information as an initial value to obtain the final target pose information.

According to an embodiment of the present application, the generating module is further configured to: generating a global space map according to the final target pose information; and performing point cloud coloring processing on the global space map to generate the global space semantic map.

The coal mine underground space positioning and mapping device with the multi-sensor information fusion, provided by the embodiment of the second aspect of the application, can realize generation of a global space map with scene rich semantic information based on fusion processing of various sensor information, realize real-time digital positioning and three-dimensional space construction of a coal mine underground scene, and improve the accuracy of underground space positioning and three-dimensional construction.

In order to achieve the above object, an embodiment of a third aspect of the present application provides a multi-sensor fused coal mine underground digital positioning and mapping system, including: the device comprises a processing assembly and an explosion-proof assembly; the processing assembly comprises a spatial data acquisition unit, a visual image acquisition unit arranged below the spatial data acquisition unit and a data processing terminal arranged below the visual image acquisition unit; the explosion-proof assembly comprises an explosion-proof cover and an explosion-proof shell, wherein the spatial data acquisition unit is positioned in the covering range of the explosion-proof cover; the flameproof housing comprises a light-transmitting area and a light-proof area, and the light-transmitting area corresponds to the visual angles of the spatial data acquisition unit and the visual image acquisition unit.

In addition, according to the multi-sensor fused coal mine underground digital positioning and mapping system of the embodiment of the application, the system can also have the following additional technical characteristics:

according to an embodiment of the application, the processing assembly further comprises: an Inertial Measurement Unit (IMU) disposed below the spatial data acquisition unit.

According to an embodiment of the present application, the data processing terminal includes: the parameter calibration unit is used for calibrating parameters of the visual image acquisition unit, calibrating joint parameters of the visual image acquisition unit and the IMU, calibrating joint parameters of the visual image acquisition unit and the spatial data acquisition unit, registering timestamps of the visual image acquisition unit, the spatial data acquisition unit and the IMU, and performing related intelligent analysis and processing on the data of the visual image acquisition unit, the spatial data acquisition unit and the IMU.

According to one embodiment of the application, the visual image acquisition unit is a monocular or binocular visual camera and is used for acquiring the underground visual image of the coal mine.

According to one embodiment of the application, the spatial data acquisition unit is a laser radar sensor or a millimeter wave radar sensor and is used for acquiring point cloud data corresponding to the underground visual image of the coal mine.

According to one embodiment of the present application, the opaque region is a metal housing.

According to an embodiment of the application, the processing assembly further comprises: and the anti-shaking cloud platform is arranged above the data processing terminal.

According to an embodiment of the present application, further comprising: and the moving assembly is arranged at the bottom of the explosion-proof assembly and is used for bearing the explosion-proof assembly to move.

According to an embodiment of the present application, further comprising: and the memory is used for storing the underground visual image of the coal mine, point cloud data corresponding to the underground visual image of the coal mine and the underground global space map of the coal mine output by the data processing terminal.

According to an embodiment of the present application, further comprising: and the wireless transmission unit is used for transmitting the underground coal mine visual image, the point cloud data corresponding to the underground coal mine image and the underground coal mine global space map output by the data processing terminal to the terminal equipment.

The embodiment of the third aspect of the application provides a multi-sensor fused coal mine underground digital positioning and mapping system, which can complement the sensing ability through multiple sensors, improve the stability of the system and ensure the robustness of the system.

In order to achieve the above object, an embodiment of a fourth aspect of the present application provides an electronic device, including: the device comprises a memory, a processor and a computer program which is stored on the memory and can run on the processor, wherein when the processor executes the program, the method for positioning and mapping the underground coal mine space with the multi-sensor information fusion is realized according to any one of the embodiments of the first aspect of the application.

In order to achieve the above object, a fifth embodiment of the present application provides a computer-readable storage medium, and the program is executed by a processor to implement the method for positioning and mapping the underground coal mine space with multi-sensor information fusion according to any one of the first embodiment of the present application.

Drawings

Fig. 1 is a schematic diagram of a method for positioning and mapping a coal mine underground space based on multi-sensor information fusion, which is disclosed in one embodiment of the present application.

Fig. 2 is a schematic diagram of a method for positioning and mapping an underground coal mine space based on multi-sensor information fusion, which is disclosed in another embodiment of the present application.

Fig. 3 is a schematic diagram of a method for positioning and mapping an underground coal mine space by multi-sensor information fusion, according to another embodiment of the present application.

Fig. 4 is a schematic diagram of a method for positioning and mapping an underground coal mine space by multi-sensor information fusion, according to another embodiment of the present application.

Fig. 5 is a schematic diagram of a method for positioning and mapping an underground coal mine space by multi-sensor information fusion, according to another embodiment of the present application.

Fig. 6 is a schematic diagram of a method for positioning and mapping an underground coal mine space by multi-sensor information fusion, according to another embodiment of the present application.

Fig. 7 is a schematic diagram of a method for positioning and mapping an underground coal mine space based on multi-sensor information fusion, according to another embodiment of the present application.

Fig. 8 is a schematic diagram of a method for positioning and mapping an underground coal mine space based on multi-sensor information fusion, according to another embodiment of the present application.

Fig. 9 is a schematic diagram of a method for positioning and mapping an underground coal mine space by multi-sensor information fusion, according to another embodiment of the present application.

Fig. 10 is a schematic structural diagram of a multi-sensor fused coal mine underground digital positioning and mapping system disclosed in an embodiment of the present application.

Fig. 11 is a schematic structural diagram of a processing component of a multi-sensor fused coal mine underground digital positioning and mapping system according to an embodiment of the present application.

Fig. 12 is a schematic structural diagram of a processing component of a multi-sensor fused coal mine underground digital positioning and mapping system according to another embodiment of the present application.

Fig. 13 is a schematic diagram of a data stream timestamp registration process disclosed in an embodiment of the present application.

Fig. 14 is a schematic structural diagram of a processing component of a multi-sensor fused coal mine underground digital positioning and mapping system according to another embodiment of the present application.

Fig. 15 is a schematic diagram of an overall framework of a multi-sensor fused coal mine underground digital positioning and mapping system disclosed in an embodiment of the present application.

Fig. 16 is a schematic diagram of the upper and middle layer main elements of the multi-sensor fused coal mine underground digital positioning and mapping system disclosed in one embodiment of the present application.

Fig. 17 is a schematic diagram of the main lower components of a multi-sensor fused coal mine underground digital positioning and mapping system according to an embodiment of the present application.

Fig. 18 is a schematic structural diagram of a coal mine underground space positioning and mapping device with multi-sensor information fusion, disclosed in an embodiment of the present application.

Detailed Description

For a better understanding of the above technical solutions, exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

It should be noted that the digital positioning and scene reconstruction of the underground space of the coal mine are one of the key supporting technologies for realizing intelligent unmanned mining, and have great application value and significance in the aspects of intelligent mining, digital monitoring management, early warning of danger, real-time positioning and perception of the environmental state of equipment and the like.

In the related technology, a coal mine underground global space map is generated mainly through the following two ways, so that the positioning of the coal mine underground space is realized: signal-based communication and sensing that relies on a single sensor.

Typical technologies for signal-based communication systems include UWB (Ultra-Wide Band), GPS (Global Positioning System), AGPS (Assisted Global Positioning System), ZigBEE (ZigBEE protocol), and the like. Implementations of such methods generally suffer from the following deficiencies. On one hand, the technical method is very dependent on the signal strength and the condition, a large number of base stations are usually required to be arranged underground, the operation is complex, the cost is high, and potential risks exist; on the other hand, the technical method has low precision in the underground environment space due to the effect of the underground special scene environment (such as coal wall absorption) on the signals, and is difficult to realize high-precision positioning.

For the way of sensing by a single sensor, the related sensors include various visual cameras, wheel speed meters, IMU (Inertial Measurement Unit), laser radars, and the like. However, most of the current technical solutions only use a single sensor, which often results in insufficient positioning accuracy of the system and also has the problem of excessive cost.

Therefore, the coal mine underground space positioning and mapping method based on multi-sensor information fusion can realize generation of a global space map with scene rich semantic information based on fusion processing of various sensor information, realize real-time digital positioning and three-dimensional space construction of a coal mine underground scene, and improve accuracy of underground space positioning and three-dimensional construction.

The method and the device for positioning and establishing the underground coal mine space with the multi-sensor information fusion according to the embodiment of the application are described below with reference to the attached drawings.

Fig. 1 is a schematic flow chart of a method for positioning and mapping a coal mine underground space with multi-sensor information fusion according to an embodiment disclosed in the present application.

As shown in fig. 1, the method for positioning and constructing a map of a coal mine underground space with multi-sensor information fusion provided by the embodiment of the application specifically includes the following steps:

s101, acquiring visual images of frames 1 to N and point cloud data corresponding to the visual images, wherein the visual images are sequentially acquired by a sensor, and N is a positive integer.

In the embodiment of the application, a scene picture, namely a visual image, acquired by a visual camera in a preset driving process of any road section can be received.

The number of the visual images uploaded by the visual camera which is attempted to be acquired can be set according to actual conditions. For example, 60 visual images sequentially acquired by the camera during the road segment driving process may be acquired, that is, N may be set to 60.

It should be noted that the visual camera may collect visual images at preset intervals, and upload the visual images collected in sequence. The preset interval may be a preset distance interval or a preset time interval. For example, the preset interval may be set to 17 meters, 20 meters, or the like; the preset interval may also be set to 10 seconds, 15 seconds, or the like.

And S102, acquiring visual pose information according to the visual image.

It should be noted that, in the present application, the specific manner for acquiring the visual pose information according to the visual image is not limited, and may be selected according to the actual situation.

Alternatively, the visual image may be input into a pre-trained decoder-encoder, and the visual features of the visual image may be output through multi-layer convolution, deconvolution, or the like. And further, acquiring visual pose information according to the visual features.

S103, obtaining IMU data of the inertial measurement unit IMU so as to obtain IMU pose information according to the IMU data.

The IMU is a device for measuring the three-axis attitude angle (or angular velocity) and acceleration of an object. Generally, an IMU includes three single-axis accelerometers and three single-axis gyroscopes, the accelerometers measure acceleration data of the rigid body in three independent axes of a carrier coordinate system, and the gyroscopes measure angular velocity data of the carrier relative to a navigation coordinate system, so that IMU attitude measurement data of the rigid body, including angular velocities and accelerations of the rigid body in three axes of a three-dimensional space, can be measured. Further, IMU pose information can be acquired according to the IMU data.

The IMU has different models, the acquisition frequency adopted by each model is possibly different, and the models of the IMU are not limited in the application.

It should be noted that, when the rigid body carries the IMU to move, the IMU may obtain the acceleration and the angular velocity during the movement through measurement, and calculate the to-be-fused attitude data of the IMU according to the standard acceleration offset, the standard angular velocity offset, the standard acceleration noise, and the standard angular velocity noise originally set by the IMU.

The method includes the steps that a standard acceleration offset, a standard angular velocity offset, a standard acceleration noise and a standard angular velocity noise are known, under a general condition, an IMU hardware manufacturer tests factory-delivered IMU equipment, and the standard acceleration offset, the standard angular velocity offset, the standard acceleration noise and the standard angular velocity noise of the IMU are determined through a large amount of test data.

And S104, acquiring target fusion pose information according to the visual pose information and the IMU pose information.

In the embodiment of the application, the fusion pose information can be acquired according to the visual pose information and the IMU pose information. Further, the fusion pose information can be optimized to obtain the optimized target fusion pose information.

It should be noted that, in the present application, a specific manner for optimizing the fusion pose information is not limited, and may be selected according to an actual situation. Optionally, the fusion pose information is optimized by a sliding window-based nonlinear optimization algorithm to obtain optimized target fusion pose information.

And S105, correcting the target fusion pose information according to the point cloud data to obtain final target pose information.

It should be noted that, in the present application, a specific manner for correcting the target fusion pose information is not limited, and may be selected according to an actual situation. Optionally, the target fusion pose information may be corrected in a nonlinear optimization manner to obtain final target pose information.

And S106, generating a global space semantic map according to the final target pose information.

According to the embodiment of the application, the global space semantic map can be generated according to the final target pose information. Further, the global spatial map may be subjected to a point cloud coloring process to generate a global spatial semantic map.

In the present application, when attempting to acquire visual pose information from a visual image, initialization of the visual front end may be performed first.

As a possible implementation manner, as shown in fig. 2, the method specifically includes the following steps:

s201, extracting visual characteristics of the visual image according to the visual image.

S202, after the visual features of the visual image are extracted, camera initialization is carried out, and whether the camera initialization is successful or not is judged.

In the present application, a sequential frame image (i.e., the 1 st to nth frame visual images) input to the visual camera is grayed, and then a pyramid image is constructed at multiple scales. And carrying out feature point detection, feature point description and feature point distortion correction on the basis of the construction of the pyramid image under the multi-scale.

In order to better adapt to the detection of key feature points of a visual image of an underground scene of a coal mine, a GPU (Graphics Processing Unit) is used for accelerating the extraction and screening operation of the key feature points, so that the stable robust detection of the key feature points in a sequence frame image is ensured.

Further, since the camera in the present application is preferably a binocular camera, in the implementation process, it is also necessary to perform visual image feature matching and scene depth estimation for the left and right cameras based on camera external parameters.

Alternatively, after the camera is initialized, the stable detection of the feature points may be performed on the visual image frame to obtain the key feature points. Further, the number of the detected key feature points may be compared with a second preset number threshold, and if the number reaches the second preset number threshold, the initialization of the recognition camera is successful; and if the number does not reach a second preset number threshold value, identifying that the initialization of the camera is unsuccessful.

The second preset number threshold may be set according to an actual situation.

And S203, responding to the successful initialization of the camera, and performing visual feature matching and tracking to obtain visual pose information.

As a possible implementation manner, as shown in fig. 3, a specific manner of performing visual feature matching and tracking in step S203 to obtain visual pose information includes the following steps:

s301, determining the number of the matched feature points and the number of the matched feature points in the 1 st to Nth frames of visual images respectively, and judging whether visual feature matching and tracking are successful.

As a possible implementation manner, as shown in fig. 4, a specific manner of determining the matching feature points and the number of matching feature points in the 1 st to nth frames of visual images in the step S301 includes the following steps:

s401, obtaining characteristic point information of the 1 st to Nth frames of visual images.

S402, screening the feature point information based on a preset screening strategy to determine the matched feature point.

Optionally, after the initialization of the visual front end is completed, feature point matching and tracking of a frame image of the visual front end are performed, so as to realize pose initial estimation based on visual information. Firstly, aiming at the feature matching of the feature points in the continuous frame images at the visual front end, the stable tracking of the feature points in the continuous frame images is realized.

S302, in response to the fact that the number of the matched feature points reaches a first preset number threshold, confirming that the visual feature matching and tracking are successful, and obtaining visual feature matching and tracking information.

Optionally, the number of stably matched feature points may be compared with a first preset number threshold, and if the number reaches the first preset number threshold, the stable tracking is successfully identified; and if the number does not reach the first preset number threshold value, identifying that the stable tracking is unsuccessful.

The first preset number threshold may be set according to an actual situation.

And S303, acquiring visual pose information according to the visual feature matching and tracking information.

Optionally, the pose of the previous frame may be used to perform initial inference on the pose of the current frame according to the specific model, and the current frame and the previous frame are matched with the feature point information, so as to complete the solution of the visual pose information of the current frame. In the implementation process, the initial pose estimation of the visual front end is specifically implemented according to the formula (1).

Wherein R.epsilon.SO (3) and

is an optimized target variable. For all matching feature points i ∈ X, XⁱIs the position of the feature point in the world coordinate system, xⁱTo match the location of the feature point on the pixel plane, π is the projection function of the world coordinate system onto the pixel plane, θ () is the robust kernel function, and Σ is the covariance matrix.

It should be noted that, in the present application, a specific manner for obtaining the visual pose information is not limited, and may be selected according to an actual situation. Optionally, the visual pose information can be obtained through a pre-trained optimization model; alternatively, a Motion recovery Structure algorithm (SFM) may be used to estimate a three-dimensional Structure from the acquired 1 st to nth frames of visual images (two-dimensional image sequences) collected in sequence, so as to obtain visual pose information.

Further, in response to the successful initialization of the camera, visual feature matching and tracking are performed to obtain visual pose information, and then a visual key frame image can be selected.

As a possible implementation manner, as shown in fig. 5, the method specifically includes the following steps:

and S501, acquiring visual pose information of adjacent frames.

And S502, selecting the visual key frame image according to the visual pose information of the adjacent frames.

It should be noted that, in the present application, when trying to acquire target fusion pose information according to the visual pose information and the IMU pose information, initialization of the IMU may be performed first.

As a possible implementation manner, as shown in fig. 6, the method specifically includes the following steps:

s601, initializing the IMU, and judging whether the IMU is initialized successfully.

It should be noted that, in the visual front end initialization process, the key feature point stable detection and feature description of the visual front end frame image are mainly realized. In a specific implementation process, when key feature points can be stably detected in a visual image frame and the detected number reaches a certain preset number, the current visual image frame is constructed into an initial key frame image at the front end of the vision. The constructed information may include a key frame ID, a key frame pose, and common frame image information viewed in common therewith. Meanwhile, the initial pose of the visual front end, i.e., the visual pose information, may be set to an identity matrix, i.e., T ═ I0. And after the key frame image is initialized, calculating to obtain the depth information of the key feature points in the key frame image, and simultaneously constructing the 3D map points of all the key feature points.

And estimating the pose of the IMU based on the IMU data while primarily estimating the visual pose. And performing pre-integration operation calculation on IMU data to obtain pre-integration increment of adjacent video frames registered with the IMU data, and corresponding Jacobian matrix and covariance value, thereby realizing estimation of IMU pose and speed.

The IMU initialization process mainly realizes parameter estimation of IMU sensor offset, scale information and gravity acceleration. Firstly, the initial estimation of IMU offset is realized according to IMU rotation increment between adjacent video frames registered with the IMU data sequence and IMU rotation increment estimated by the visual front-end pose based on pose and speed obtained by IMU data sequence calculation. And then calibrating external parameters based on the combination of the visual camera and the IMU, and calculating by utilizing SVD (singular value decomposition) according to the translation amounts of the visual camera and the IMU in the world coordinate system respectively to obtain the initial estimation of the scale and the gravitational acceleration. And finally, further estimating the offset, the scale information and the gravity acceleration parameters of the IMU sensor based on nonlinear optimization, thereby completing the initialization process of the IMU.

And S602, acquiring fusion pose information according to the visual pose information and IMU pose information in response to successful IMU initialization.

Optionally, after the IMU initialization process is completed, pose estimation of the visual front end is performed based on the IMU pose information. Firstly, matching and tracking key feature points of a current frame according to pose information of a previous frame in a sequence frame image in a visual front end and IMU pose information matched with the current frame. And then, solving the pose of the current frame on the basis of successfully tracking the key feature points in the continuous frame images at the front end of the vision to obtain the initial pose estimation of IMU information and visual information fusion, namely obtaining fusion pose information.

And S603, optimizing the fusion pose information based on a sliding window nonlinear optimization algorithm to obtain target fusion pose information.

It should be noted that, in the present application, in order to make full use of the visual semantic information to perform constraint optimization for the global pose, the accuracy of spatial positioning is improved. While high-frequency pose estimation based on IMU and visual information fusion is carried out, the global pose needs to be continuously optimized by using a closed-loop detection mode.

As a possible implementation manner, as shown in fig. 7, the specific process of optimizing the fusion pose information based on the sliding window nonlinear optimization algorithm in step S603 to obtain the target fusion pose information includes the following steps:

s701, optimizing the fusion pose information based on a sliding window nonlinear optimization algorithm to obtain the optimized fusion pose information.

S702, according to the visual key frame images, obtaining the feature similarity between the current visual key frame images and the historical visual key frame images.

And S703, judging whether a closed loop state exists or not according to the feature similarity.

In the embodiment of the application, similar key frame retrieval is carried out in the visual key frame image set according to the bag-of-words characteristics, and closed-loop visual key frames are selected according to certain standards.

In the embodiment of the application, only when the similarity of the bag-of-words features of the candidate closed-loop key frame and the current visual key frame reaches a certain standard and passes the continuity detection of the closed-loop key frame, the closed-loop state is considered to possibly occur.

Optionally, the feature similarity may be compared with a preset feature similarity threshold, and if the feature similarity reaches the feature similarity threshold, a closed loop state is identified; and if the feature similarity does not reach the feature similarity threshold, identifying that no closed loop state exists.

Optionally, in order to further improve the accuracy of the determination result of whether the closed-loop state exists, feature point matching between the current visual key frame and the candidate closed-loop key frame and the relative pose between the current visual key frame and the candidate closed-loop key frame may be obtained in a Sim3 similarity transformation manner. And when the number of the matched feature points is more than a certain number, the closed-loop detection is successfully identified, otherwise, the closed-loop detection is failed.

And S704, responding to the existence of the closed loop state, acquiring closed loop visual key frame images, and performing global closed loop nonlinear optimization on fusion pose information corresponding to all the visual key frame images to obtain target fusion pose information.

After initial estimation of the IMU information and visual information fusion pose is completed, further optimization calibration of the fusion pose is carried out through nonlinear optimization, and high-frequency pose estimation based on IMU and visual information fusion is completed. Because the underground scenes of the coal mine are mostly characterized by long straight corridors, and the high complexity of optimization calculation based on the common-view relation is considered, in the implementation process of the technical scheme, in order to reduce the real-time performance of the optimization scale guarantee system, the optimization calibration of the fusion pose is carried out in a sliding window mode. In the nonlinear pose optimization process of the fusion pose, pose estimation optimization is carried out by utilizing IMU pose constraint, visual observation constraint and prior pose information constraint corresponding to a specific key frame, and an optimization objective function is shown as a formula (2).

Where X is a set of optimization variables, r_priorFor prior pose information constraints by marginalization operations, sigma_priorIs the covariance to which it corresponds.

Residual constraints on IMU measurements for key frame images of k frame to k +1 frame, Σ_IIs the covariance to which it corresponds.

Passing the feature point i and the image key frame j through a projection function

Resulting reprojection residual constraint, Σ_CIs the covariance to which it corresponds. B is the set of all IMU measurement data within the sliding window and P is the set of all visual observations within the sliding window.

As mahalanobis distance related to the covariance matrix, i.e.

On the basis of successful closed-loop detection, firstly searching and updating the common-view relationship between the current frame and other visual key frames, then acquiring corrected poses of other key frames and space coordinate information of corresponding feature points through pose propagation, and updating the common-view relationship of the corresponding frames through conflict check of feature points corresponding to the current frame, the closed-loop key frames and adjacent key frames. And finally, carrying out global closed loop nonlinear optimization according to the corrected space coordinate information of the feature points and the updated common-view relationship between the frames to realize the revision of the poses of all the key frames, thereby obtaining the optimized target fusion pose information.

Further, in the present application, when trying to correct the target fusion pose information according to the point cloud data to obtain the final target pose information, distortion removal operation may be performed on the point cloud data first.

As a possible implementation manner, as shown in fig. 8, the method specifically includes the following steps:

s801, distortion removing operation is carried out on the point cloud data according to the target fusion pose information to obtain accurate point cloud data.

And S802, extracting target point cloud characteristics according to the accurate point cloud data.

And S803, matching and tracking the target point cloud characteristics in a nonlinear optimization mode by taking the target fusion pose information as an initial value to obtain final target pose information.

In the embodiment of the application, feature extraction can be realized by utilizing a plane activeness mode for each frame of point cloud data according to the clock registration relation between the video sequence frame and the point cloud data sequence. Further, point cloud data distortion removal operation is carried out on the basis of realization of high-frequency pose estimation by fusing the IMU and the visual information. And (4) performing feature matching of continuous point cloud data frames in a nonlinear optimization mode by taking high-frequency pose information obtained by fusing the IMU and the visual information as an initial value, and realizing low-frequency pose optimization correction to obtain final target pose information.

Further, in the application, when an attempt is made to generate the global space semantic map according to the final target pose information, the global space semantic map with rich scene semantic information can be generated by performing point cloud coloring processing on the global space map.

As a possible implementation manner, as shown in fig. 9, the method specifically includes the following steps:

and S901, generating a global space map according to the final target pose information.

And S902, carrying out point cloud coloring processing on the global space map to generate a global space semantic map.

In the embodiment of the application, the point cloud data registered with the visual frame image can be projected into the visual plane based on the external reference calibration result between the laser radar and the visual camera, the visual information of the corresponding projection pixel point is obtained, then the obtained visual information is utilized to perform coloring processing on the point cloud data, and the digital semantic map of the global space is generated.

The embodiment of the application provides a multi-sensor fused coal mine underground digital positioning and mapping system.

Fig. 10 is a schematic structural diagram of a multi-sensor fused coal mine underground digital positioning and mapping system according to an embodiment of the present disclosure.

As shown in fig. 10, the system 1000 for digital positioning and mapping in a coal mine with multiple sensors integrated according to the embodiment of the present application includes: a processing component 110 and an explosion-proof component 120.

As shown in fig. 11, the processing component 110 includes a spatial data acquisition unit 111, a visual image acquisition unit 112 disposed below the spatial data acquisition unit 111, and a data processing terminal 113 disposed below the visual image acquisition unit 112.

The spatial data acquisition unit 111 may be a laser radar sensor or a microwave sensor, and is configured to acquire spatial depth information corresponding to a visual image in a coal mine, and acquire point cloud information corresponding to 1 st to nth frames of visual images sequentially acquired by the sensor, where N is a positive integer.

The visual image acquisition unit 112 may be a binocular infrared visual camera, and is used for a sensor to sequentially acquire the 1 st to nth frame visual images of the underground coal mine image; the visual image acquisition unit 112 may also be a camera device such as a panoramic camera, and in this case, after the underground video of the coal mine is acquired, the 1 st to nth frames of visual images acquired in sequence may be acquired through processing such as frame extraction.

A data processing terminal 113 for extracting visual features of the visual image according to the visual image; acquiring target fusion pose information according to the visual characteristics and the IMU pose information; correcting the target fusion pose information according to the point cloud information to obtain final target pose information; and generating a global space semantic map according to the final target pose information.

The explosion-proof assembly 120 comprises an explosion-proof cover 121 and an explosion-proof shell 122, wherein the spatial data acquisition unit 111 is located in the coverage range of the explosion-proof cover 121; flameproof housing 122 includes light transmissive area 1221 and light opaque area 1222.

The light-transmitting area 1221 corresponds to the viewing angles of the visual image acquisition unit 112 and the spatial data acquisition unit 111, so as to ensure that the visual image acquisition unit 112 can smoothly acquire visual images and ensure that the spatial data acquisition unit 111 can smoothly acquire spatial data; the opaque region 1222 is a metal housing.

Therefore, the coal mine underground space positioning and map building system with multi-sensor information fusion provided by the embodiment of the application can realize underground coal mine space positioning and real-time three-dimensional scene building by arranging the space data acquisition unit, the visual image acquisition unit and the data processing terminal. The system body structure has an integrated intrinsic safety design, so that sensing capability complementation is performed through multiple sensors, the stability of the system is improved, and the robustness of the system is ensured. Meanwhile, the problem of positioning drift of the spatial data acquisition unit in the underground space is solved to a certain extent, the visual perception characteristic of the visual image acquisition unit is fully utilized, and the semantic information of the spatial scene is acquired, so that the spatial point cloud data acquired by the spatial data acquisition unit has semantic information with rich scenes, and the accuracy of underground space positioning and three-dimensional construction is improved.

In some embodiments, as shown in fig. 12, the processing component 110, further comprises: an inertial measurement unit IMU 114.

The IMU114 is disposed below the spatial data acquisition unit 111, and is connected to the visual image acquisition unit 112, for acquiring IMU data, so as to acquire IMU pose information according to the IMU data.

In some embodiments, the data processing terminal 113 includes: a parameter calibration unit 1131.

The parameter calibration unit 1131 is configured to perform parameter calibration on the visual image acquisition unit, perform joint parameter calibration on the visual image acquisition unit and the IMU, perform joint parameter calibration on the visual image acquisition unit and the spatial data acquisition unit, perform registration on timestamps of data of the visual image acquisition unit, the spatial data acquisition unit, and the IMU, and perform related intelligent analysis and processing on the data of the three.

It should be noted that, in practical applications, before the coal mine underground digital positioning and mapping system 1000 with multiple sensors integrated provided by the present application leaves a factory, parameter calibration may be performed.

The following explains the process of time stamp registration of data of the visual image acquisition unit, the spatial data acquisition unit and the IMU respectively aiming at parameter calibration of the visual image acquisition unit, joint parameter calibration of the visual image acquisition unit and the IMU, joint parameter calibration of the visual image acquisition unit and the spatial data acquisition unit, and the like.

For parameter calibration of the visual image acquisition unit, it should be noted that, regarding type selection of the visual camera, a monocular visual camera, a binocular infrared camera, a depth camera and the like can be selected according to actual requirements. In the implementation process of the technical scheme of the invention, the binocular infrared vision camera is adopted according to the characteristics of the underground scene of the coal mine, so that the texture vision characteristics of the underground scene can be fully acquired, and the robust detection of the tracking characteristic points is effectively ensured.

Taking a binocular infrared vision camera as an example, the calibration of the vision camera is divided into an internal reference calibration part and an external reference calibration part. During the calibration process, an open source calibration tool such as kalibr is used for calibration. Meanwhile, in order to ensure the calibration accuracy of the vision camera, a special calibration plate made of aluminum oxide materials (or ceramics) with a backlight plate is adopted for calibrating the vision image acquisition. In the process of calibrating the visual image acquisition, the camera needs to be fully excited according to each shooting angle (the shooting angle of the camera relative to the calibration plate), and meanwhile, the acquired visual image is ensured not to have a fuzzy phenomenon caused by movement, so that the calibration accuracy of the visual camera is ensured.

The combined parameter calibration is carried out on the visual image acquisition unit and the IMU, and the combined calibration of the visual camera and the IMU is required after the calibration of the visual camera is finished. The visual camera-IMU combination is calibrated using the open source calibration tool such as kalibr (similar open source tool also having calibration tool such as opencv calibration) and the above-mentioned proprietary calibration board. Before calibration, the relative positions of the camera and the IMU need to be fixed, and simultaneously, an accelerometer and a gyroscope in the IMU need to be fully excited, and the calibrated image picture of the visual camera is ensured not to have a fuzzy phenomenon caused by movement.

Aiming at carrying out combined parameter calibration on a visual image acquisition unit and a spatial data acquisition unit, the combined calibration of a laser radar-visual camera is calibrated by using a calibrtToolKit module in a public automatic tool. Before calibration, the relative positions of the laser radar and the vision camera need to be fixed, and a special calibration plate in a large-specification checkerboard form is manufactured. For example, a proprietary calibration plate of 120cm by 80cm gauge may be fabricated and used. In the calibration process, the angle of the calibration plate needs to be changed to acquire calibration data of a specific angle, so that the precision of the combined calibration of the laser radar and the vision camera is ensured.

It should be noted that, in practical applications, two ways may be adopted to obtain calibration data, one way is to fix the calibration plate and obtain calibration data at different angles by sufficiently moving the sensor system device. One way is to fix the sensor system device and obtain calibration data for different viewing angles by moving the calibration plate to change the viewing angle.

For the registration of the time stamps of the data of the visual image acquisition unit, the spatial data acquisition unit and the IMU, the IMU provides a high data stream, typically at a frequency between 200Hz and 400Hz, whereas the data stream of the visual camera and the lidar sensor is at a low frequency. Typically, the video camera data stream has a frequency between 15Hz and 60Hz, and the lidar data stream has a frequency as low as 5Hz to 15 Hz. Therefore, the time stamps of the sensor system data need to be registered, so that the consistency of the sensor system data stream in time series is realized.

For the time stamp registration of the high data stream of the IMU and the low data stream of the visual camera, in the technical scheme of the application, the time difference between the IMU and the sequence data frame of the visual camera is set as an optimization variable, and then the time difference is solved through nonlinear optimization to realize the registration of the clock. The residual equation of the nonlinear optimization is constructed based on the reprojection. For the visual camera and the lidar, a soft trigger and a hard trigger are combined to perform clock registration of the visual camera and the lidar data stream, and the flow of data stream timestamp registration is shown in fig. 13. The trigger signal of the soft trigger process is initiated by system Software, and is soft triggered using, for example, an API (Application Programming Interface) Interface provided by a visual camera SDK (Software Development Kit). The implementation of hard triggering requires first setting a specific device (e.g., an MCU (Micro-Controller Unit)) outside the sensor, and then connecting the specific device with an I/O Interface (I/O Interface) of each sensor in the system, so as to implement hard triggering based on Strobe signals.

As can be seen from fig. 13, the left camera may be first soft triggered using the API of the vision camera (here, the binocular vision camera, with left and right cameras), and then the right camera and the lidar may be hard triggered using the Strobe signal returned by the left camera. Where the trigger frequency for the right camera is an amount between 15Hz and 60Hz (i.e. triggered once every 1 Strobe signal received) and the trigger frequency for the lidar is an amount between 5Hz and 15Hz (i.e. triggered once every 3 Strobe signals received). Wherein, the Strobe signal is a signal generated when the visual camera exposure occurs, and the high level is a valid bit. In the implementation process, because it is considered that the Strobe signal generated by the system soft trigger has a certain delay when being received by the system, the system needs to perform pre-output processing on the Strobe signal generated by the soft trigger. The left camera is normally subjected to exposure delay processing so that the exposure generation time is synchronized with the time of the Strobe signal received by the system.

In some embodiments, as shown in fig. 14, the processing component 110 further includes: anti-shake cloud platform 115.

The anti-shake cradle head 115 is arranged above the data processing terminal 113 and used for balancing shake in the movement process of the coal mine underground space positioning and image building system 1000.

In some embodiments, the multi-sensor fused coal mine underground digital positioning and mapping system 1000 further includes: a mobile component 130, a memory 140, and a wireless transmission unit 150.

The moving assembly 130 is arranged at the bottom of the flameproof assembly 120 and used for carrying the flameproof assembly 120 to move.

The memory 140 is configured to store the underground coal mine visual image, the point cloud data information corresponding to the underground coal mine visual image, and an underground coal mine global space map output by the data processing terminal.

The wireless transmission unit 140 is configured to send the coal mine underground visual image, the point cloud data information corresponding to the coal mine underground image, and the coal mine underground global space map output by the data processing terminal to the intelligent analysis processing terminal device.

In summary, as shown in fig. 15(a), in the multi-sensor fused coal mine underground digital positioning and mapping system 1000, the overall frame can be divided into an inner layer and an outer layer, as shown in fig. 15(b), the outer layer is the explosion-proof component 120, and the inner layer is the processing component 110 formed by each sensor element.

The processing assembly 110 is integrally divided into three parts, as shown in fig. 16, a spatial data acquisition unit 111, such as a lidar sensor, is disposed on the uppermost layer, a visual image acquisition unit 112 and an IMU114, such as a visual camera and an IMU device, are disposed on the middle layer, and as shown in fig. 17, a data processing terminal 113 and an anti-shake pan-tilt head 115 are disposed on the lower layer.

The outer part of the explosion-proof shell 122 is integrally divided into two parts, the top part of the explosion-proof shell 121 is an explosion-proof cover of a laser radar sensor, for example, the other part of the explosion-proof shell 122 is an explosion-proof shell, and the explosion-proof shell 122 can be divided into a light-transmitting area and a light-proof area; wherein, the light transmission area has a light-transmittable surface exclusive for visual camera visual angle and a light-transmittable surface exclusive for laser radar visual angle, and the light-tight area is an anti-explosion surface of non-light-transmittable metal.

Therefore, the multi-sensor fused coal mine underground digital positioning and mapping system provided by the embodiment of the application can realize the rapid calibration of the sensor system and the timestamp consistency registration function of each sensor data realized based on a soft and hard combination mode.

The embodiment of the multi-sensor information fusion underground coal mine space positioning and mapping method is also suitable for the multi-sensor information fusion underground coal mine space positioning and mapping device provided by the embodiment, and detailed description is omitted in the embodiment.

Fig. 18 is a schematic structural diagram of a multi-sensor information fusion coal mine underground space positioning and mapping device according to an embodiment of the application.

As shown in fig. 18, the coal mine underground space positioning and map building device 2000 includes: a first obtaining module 210, a second obtaining module 220, a third obtaining module 230, a fourth obtaining module 240, a determining module 250, and a generating module 260. Wherein:

a first obtaining module 210, configured to obtain 1 st to nth frames of visual images sequentially acquired by a sensor and point cloud data corresponding to the visual images, where N is a positive integer;

a second obtaining module 220, configured to obtain visual pose information according to the visual image;

a third obtaining module 230, configured to obtain IMU data of an inertial measurement unit IMU, so as to obtain IMU pose information according to the IMU data;

a fourth obtaining module 240, configured to obtain target fusion pose information according to the visual pose information and the IMU pose information;

a determining module 250, configured to modify the target fusion pose information according to the point cloud data to obtain final target pose information;

and the generating module 260 is configured to generate a global spatial semantic map according to the final target pose information.

According to an embodiment of the present application, the second obtaining module 220 is further configured to: extracting visual features of the visual image according to the visual image; after the visual features of the visual image are extracted, camera initialization is carried out, and whether the camera initialization is successful or not is judged; and responding to the successful initialization of the camera, and performing visual feature matching and tracking to obtain visual pose information.

According to an embodiment of the present application, the fourth obtaining module 240 is further configured to: performing IMU initialization and judging whether the IMU initialization is successful or not; responding to the successful IMU initialization, acquiring fusion pose information according to the visual pose information and the IMU pose information; and optimizing the fusion pose information based on a sliding window nonlinear optimization algorithm to obtain the target fusion pose information.

According to an embodiment of the present application, the fourth obtaining module 240 is further configured to: respectively determining the number of the matched feature points and the number of the matched feature points in the 1 st to Nth frames of visual images, and judging whether visual feature matching and tracking are successful or not; confirming that the visual feature matching and tracking are successful in response to the fact that the number of the matched feature points reaches a first preset number threshold value, so as to obtain visual feature matching and tracking information; and acquiring the visual pose information according to the visual feature matching and tracking information.

According to an embodiment of the present application, the fourth obtaining module 240 is further configured to: acquiring the visual pose information of adjacent frames; and selecting a visual key frame image according to the visual pose information of the adjacent frames.

According to an embodiment of the present application, the fourth obtaining module 240 is further configured to: optimizing the fusion pose information based on a sliding window nonlinear optimization algorithm to obtain the optimized fusion pose information; according to the visual key frame image, acquiring the feature similarity of the current visual key frame image and the historical visual key frame image; judging whether a closed loop state exists according to the feature similarity; and responding to the existence of the closed loop state, acquiring closed loop visual key frame images, and performing global closed loop nonlinear optimization on the fusion pose information corresponding to all the visual key frame images to obtain the target fusion pose information.

According to an embodiment of the application, the determining module 250 is further configured to: according to the target fusion pose information, carrying out distortion removal operation on the point cloud data to obtain accurate point cloud data; extracting target point cloud characteristics according to the accurate point cloud data; and matching and tracking the target point cloud characteristics in a nonlinear optimization mode by taking the target fusion pose information as an initial value to obtain the final target pose information.

According to an embodiment of the application, the generating module 260 is further configured to: generating a global space map according to the final target pose information; and performing point cloud coloring processing on the global space map to generate the global space semantic map.

According to the coal mine underground space positioning and map building device with multi-sensor information fusion, fusion processing can be performed on various sensor information, global space map generation with rich scene semantic information is achieved, real-time digital positioning and three-dimensional space building of coal mine underground scenes are achieved, and accuracy of underground space positioning and three-dimensional building is improved.

In the description of the present application, it is to be understood that the terms "central," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," "counterclockwise," "axial," "radial," "circumferential," and the like are used in the orientations and positional relationships indicated in the drawings for convenience in describing the present application and to simplify the description, and are not intended to indicate or imply that the referenced devices or elements must have a particular orientation, be constructed and operated in a particular orientation, and are therefore not to be considered limiting of the present application.

Furthermore, the terms "first", "second" and "first" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of the present application, "a plurality" means two or more unless specifically limited otherwise.

In this application, unless expressly stated or limited otherwise, the terms "mounted," "connected," "secured," and the like are to be construed broadly and can include, for example, fixed connections, removable connections, or integral parts; can be mechanically or electrically connected; either directly or indirectly through intervening media, either internally or in any other relationship. The specific meaning of the above terms in the present application can be understood by those of ordinary skill in the art as appropriate.

In this application, unless expressly stated or limited otherwise, the first feature "on" or "under" the second feature may be directly contacting the first and second features or indirectly contacting the first and second features through intervening media. Also, a first feature "on," "over," and "above" a second feature may be directly or diagonally above the second feature, or may simply indicate that the first feature is at a higher level than the second feature. A first feature being "under," "below," and "beneath" a second feature may be directly under or obliquely under the first feature, or may simply mean that the first feature is at a lesser elevation than the second feature.

In the description herein, reference to the description of the term "one embodiment," "some embodiments," "an example," "a specific example," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the application. In this specification, the schematic representations of the terms used above are not necessarily intended to refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, various embodiments or examples and features of different embodiments or examples described in this specification can be combined and combined by one skilled in the art without contradiction.

Although embodiments of the present application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present application, and that variations, modifications, substitutions and alterations may be made to the above embodiments by those of ordinary skill in the art within the scope of the present application.

Claims

1. The utility model provides a colliery of multisensor integration is digital location and map construction system in pit which characterized in that includes: the device comprises a processing assembly and an explosion-proof assembly; wherein the content of the first and second substances,

the processing assembly comprises a spatial data acquisition unit, a visual image acquisition unit arranged below the spatial data acquisition unit and a data processing terminal arranged below the visual image acquisition unit;

the explosion-proof assembly comprises an explosion-proof cover and an explosion-proof shell, wherein the spatial data acquisition unit is positioned in the covering range of the explosion-proof cover; the flameproof housing comprises a light-transmitting area and a light-proof area, and the light-transmitting area corresponds to the visual angles of the spatial data acquisition unit and the visual image acquisition unit.

2. The coal mine underground digital positioning and mapping system of claim 1, wherein the processing assembly further comprises:

an Inertial Measurement Unit (IMU) disposed below the spatial data acquisition unit.

3. The system for digital underground coal mine positioning and mapping according to claims 1-2, wherein the data processing terminal comprises:

the parameter calibration unit is used for calibrating parameters of the visual image acquisition unit, calibrating joint parameters of the visual image acquisition unit and the IMU, calibrating joint parameters of the visual image acquisition unit and the spatial data acquisition unit, registering timestamps of the visual image acquisition unit, the spatial data acquisition unit and the IMU, and performing related intelligent analysis and processing on the data of the visual image acquisition unit, the spatial data acquisition unit and the IMU.

4. The coal mine underground digital positioning and mapping system according to claim 1, wherein the visual image acquisition unit is a monocular or binocular visual camera for acquiring the coal mine underground visual image.

5. The coal mine underground digital positioning and map building system according to claim 1, wherein the spatial data acquisition unit is a laser radar sensor or a millimeter wave radar sensor and is used for acquiring point cloud data corresponding to a coal mine underground visual image.

6. The coal mine underground digital positioning and mapping system of claim 1, wherein the opaque region is a metal housing.

7. The coal mine underground digital positioning and mapping system of claim 1, wherein the processing assembly further comprises:

and the anti-shaking cloud platform is arranged above the data processing terminal.

8. The coal mine underground digital positioning and mapping system according to claim 1, further comprising:

and the moving assembly is arranged at the bottom of the explosion-proof assembly and is used for bearing the explosion-proof assembly to move.

9. The coal mine digital positioning and mapping system of claim 1, further comprising:

and the memory is used for storing the underground visual image of the coal mine, point cloud data corresponding to the underground visual image of the coal mine and the underground global space map of the coal mine output by the data processing terminal.

10. The coal mine underground digital positioning and mapping system according to claim 1, further comprising:

and the wireless transmission unit is used for transmitting the underground coal mine visual image, the point cloud data corresponding to the underground coal mine image and the underground coal mine global space map output by the data processing terminal to the terminal equipment.