WO2022262152A1 - 地图构建方法及装置、电子设备、存储介质和计算机程序产品 - Google Patents

地图构建方法及装置、电子设备、存储介质和计算机程序产品 Download PDF

Info

Publication number
WO2022262152A1
WO2022262152A1 PCT/CN2021/122715 CN2021122715W WO2022262152A1 WO 2022262152 A1 WO2022262152 A1 WO 2022262152A1 CN 2021122715 W CN2021122715 W CN 2021122715W WO 2022262152 A1 WO2022262152 A1 WO 2022262152A1
Authority
WO
WIPO (PCT)
Prior art keywords
feature point
image
mobile device
grid
scene
Prior art date
Application number
PCT/CN2021/122715
Other languages
English (en)
French (fr)
Inventor
张展鹏
邓文钧
龙有炼
赵扬波
于行尧
成慧
Original Assignee
深圳市商汤科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 深圳市商汤科技有限公司 filed Critical 深圳市商汤科技有限公司
Publication of WO2022262152A1 publication Critical patent/WO2022262152A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
    • G06T17/05—Geographic models
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics
    • G06T19/003—Navigation within 3D models or images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/60—Analysis of geometric attributes
    • G06T7/62—Analysis of geometric attributes of area, perimeter, diameter or volume
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
    • G06T7/85—Stereo camera calibration
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/46—Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features
    • G06V10/462—Salient features, e.g. scale invariant feature transforms [SIFT]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10004—Still image; Photographic image
    • G06T2207/10012—Stereo images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10016—Video; Image sequence
    • G06T2207/10021—Stereoscopic video; Stereoscopic image sequence
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30244—Camera pose

Definitions

  • Embodiments of the present disclosure relate to but are not limited to the field of computer technology, and in particular, relate to a map construction method and device, electronic equipment, storage media, and computer program products.
  • Sweeping robots are used more and more in indoor cleaning.
  • sweeping robots can usually use their own sensors for self-positioning and map building to achieve autonomous navigation.
  • autonomous navigation can be understood as using the constructed map and self-positioning information to perform path planning and obstacle analysis in advance, so that the intelligent sweeping robot can perform mobile cleaning.
  • the present disclosure at least provides a method and device for constructing a map, electronic equipment, a storage medium, and a computer program product.
  • An embodiment of the present disclosure provides a map construction method, including: acquiring a scene image and a depth image of a target scene collected by a mobile device, and acquiring the relative pose of the mobile device at the current moment; The depth image, determining the first spatial feature point of the target scene at a previous moment and the second spatial feature point at the current moment, the time sequence of the previous moment is before the current moment, and the first spatial feature point and the The second spatial feature point is matched; according to the first spatial feature point and the second spatial feature point, the relative pose is optimized to obtain the positioning information of the mobile device at the current moment; according to The positioning information and the depth image determine a grid map corresponding to the target scene, the grid map is used to indicate the drivable area in the target scene, so that the mobile device can Maps for navigation.
  • the scene image is collected by an image acquisition component set on the mobile device
  • the depth image is collected by a depth sensing component set on the mobile device
  • determining a first spatial feature point of the target scene at a previous moment and a second spatial feature point at a current moment includes: extracting feature points from a scene image collected at a previous moment, Obtain the first image feature point; according to the first image feature point, determine the second image feature point in the scene image collected at the current moment that matches the first image feature point; according to the scene image and the depth image
  • the positioning information of the device at the current moment includes: for any relative pose, transforming the first spatial feature point into a third spatial feature point at the current moment according to the relative pose; The first error between the second spatial feature point and the third spatial feature point, and/or the second error between the relative pose and other relative poses, optimize the relative pose, and obtain the optimized relative poses, wherein the other relative poses include relative poses other than the relative poses among the multiple relative poses; according to the optimized relative poses, determine that the mobile device is at the current moment
  • determining the grid map corresponding to the target scene according to the positioning information and the depth image includes: determining a two-dimensional plane corresponding to the target scene, and Perform grid division on the plane to obtain a grid plane; determine the position information of the obstacle in the target scene according to the positioning information and the depth image; determine and mark the obstacle according to the position information of the obstacle A grid map corresponding to the target scene is obtained for the grid belonging to the grid plane.
  • the determining the grid map corresponding to the target scene according to the positioning information and the depth image includes: determining the target scene according to the positioning information and the depth image The initial grid map corresponding to the scene; identifying the scene image, determining the scene category of the target scene and the obstacle category of obstacles in the target scene; according to the scene category, the obstacle category and the The initial grid map is used to generate the grid map.
  • the method further includes: according to the obstacle category corresponding to the non-drivable area in the target scene, setting an adjustment coefficient for the grid representing the non-drivable area in the grid map and A preset constant, wherein the adjustment coefficient is used to indicate the decay rate or growth rate of the preset constant; according to the adjustment coefficient and the preset constant, the grid map is updated to obtain an updated grid map, so that the mobile device performs navigation operations according to the updated grid map.
  • updating the grid map according to the adjustment coefficient and the preset constant to obtain the updated grid map includes: For any grid in the area, decrement the preset constant of the grid according to the decay speed indicated by the adjustment coefficient of the grid, or increase the grid according to the growth rate indicated by the adjustment coefficient of the grid A preset constant; when the preset constant decreases to less than the first preset threshold, or the preset constant increases to more than the second preset threshold, the grid is set as a drivable area, and The updated raster map.
  • the relative pose includes: the relative position and heading angle of the mobile device at the current moment relative to the previous moment; the image acquisition component, the depth sensing component, and the inertial
  • the sensing components form an information collection module, the information collection module is arranged on the front end of the mobile device, and the orientation of the information collection module is consistent with the moving direction of the mobile device; the wheeled mileage of the mobile device
  • the gauge is set at the bottom of the mobile device.
  • An embodiment of the present disclosure provides a map construction device, including: an acquisition module configured to acquire a scene image and a depth image of a target scene collected by a mobile device, and acquire the relative pose of the mobile device at the current moment; determine A module configured to determine, according to the scene image and the depth image, a first spatial feature point at a previous moment of the target scene and a second spatial feature point at a current moment, the time sequence of the previous moment is in the Before the current moment, the first spatial feature point matches the second spatial feature point; the optimization module is configured to calculate the relative position according to the first spatial feature point and the second spatial feature point pose optimization to obtain the positioning information of the mobile device at the current moment; the building module is configured to determine a grid map corresponding to the target scene according to the positioning information and the depth image, and the grid map It is used to indicate the drivable area in the target scene, so that the mobile device can perform a navigation operation according to the grid map.
  • the scene image is collected by an image acquisition component set on the mobile device
  • the depth image is collected by a depth sensing component set on the mobile device
  • the determining module includes: an extracting submodule configured to extract feature points from the scene image collected at a previous moment to obtain a first image feature point; a matching submodule is configured to determine the current feature point according to the first image feature point A second image feature point that matches the first image feature point in the scene image collected at any time; the mapping submodule is configured to determine the second image feature point that matches the first image feature point according to the mapping relationship between the scene image and the depth image.
  • the optimization module includes: a transformation submodule configured to, for any relative pose, transform the first relative pose according to the relative pose A spatial feature point is transformed into a third spatial feature point at the current moment; the optimization submodule is configured to be based on a first error between the second spatial feature point and the third spatial feature point, and/or Or the second error between the relative pose and other relative poses, optimize the relative pose to obtain the optimized relative pose, wherein the other relative poses include multiple relative poses except all relative poses A relative pose other than the above-mentioned relative pose; the positioning information determining submodule is configured to determine the positioning information of the mobile device at the current moment according to the optimized relative pose; wherein, the plurality of relative poses The pose includes the relative pose measured by the inertial sensing component set on the mobile device, and the relative pose measured by the wheel odometer set on the mobile device.
  • the building module includes: a grid plane determination submodule configured to determine a two-dimensional plane corresponding to the target scene, and perform grid division on the two-dimensional plane to obtain a grid plane; the location information determination submodule is configured to determine the location information of obstacles in the target scene according to the positioning information and the depth image; the first grid map determination submodule is configured to determine the location information according to the obstacles position information of the object, determine and mark the grid to which the obstacle belongs in the grid plane, and obtain the grid map corresponding to the target scene.
  • the construction module includes: an initial grid map determination submodule configured to determine an initial grid map corresponding to the target scene according to the positioning information and the depth image;
  • the identification submodule is configured to identify the scene image, determine the scene category of the target scene and the obstacle category of obstacles in the target scene;
  • the second grid map determination submodule is configured to The scene category, the obstacle category, and the initial grid map are used to generate the grid map.
  • the device further includes: a setting module configured to, according to the obstacle category corresponding to the non-drivable area in the target scene, set the Grid setting adjustment coefficients and preset constants, wherein the adjustment coefficients are used to indicate the decay speed or growth speed of the preset constants; the update module is configured to update The grid map obtains an updated grid map, so that the mobile device performs navigation operations according to the updated grid map.
  • the update module includes: an adjustment submodule configured to, for any grid representing the non-drivable area in the grid map, according to the adjustment coefficient indicated by the grid
  • the decay speed is to decrease the preset constant of the grid, or to increase the preset constant of the grid according to the growth rate indicated by the adjustment coefficient of the grid;
  • the update submodule is configured to When the constant decreases to less than the first preset threshold, or the preset constant increases to greater than the second preset threshold, the grid is set as a drivable area to obtain an updated grid map.
  • the relative pose includes: the relative position and heading angle of the mobile device at the current moment relative to the previous moment; the image acquisition component, the depth sensing component, and the inertial
  • the sensing components form an information collection module, the information collection module is arranged on the front end of the mobile device, and the orientation of the information collection module is consistent with the moving direction of the mobile device; the wheeled mileage of the mobile device
  • the gauge is set at the bottom of the mobile device.
  • An embodiment of the present disclosure provides an electronic device, including: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to invoke the instructions stored in the memory to execute the above method.
  • An embodiment of the present disclosure provides a computer-readable storage medium, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the foregoing method is implemented.
  • An embodiment of the present disclosure also provides a computer program product, including computer-readable codes, or a non-volatile computer-readable storage medium bearing computer-readable codes, when the computer-readable codes are stored in a processor of an electronic device During operation, the processor in the electronic device executes the steps of the above map construction method.
  • the spatial feature points determined by the scene image and the depth image can be used to optimize the relative pose of the mobile device at the current moment, so that higher-precision positioning information can be obtained;
  • the grid map determined by the positioning information and the depth image can accurately indicate the feasible area in the target scene. It can not only meet the needs of navigation operations such as path planning and obstacle analysis, but also effectively improve the efficiency of map construction, which is helpful for mobile devices. Perform reasonable and efficient autonomous navigation operations.
  • Fig. 1a is a schematic diagram of a side view of a mobile device provided by an embodiment of the present disclosure.
  • Fig. 1b is a schematic diagram of a top view of a mobile device provided by an embodiment of the present disclosure.
  • Fig. 1c is a schematic diagram of a map construction method provided by an embodiment of the present disclosure.
  • FIG. 2 is a schematic diagram of an implementation flow of a map construction method provided by an embodiment of the present disclosure.
  • FIG. 3 is a schematic diagram of a plane coordinate system provided by an embodiment of the present disclosure.
  • FIG. 4 is a schematic diagram of a plane coordinate system of a grid plane provided by an embodiment of the present disclosure.
  • Fig. 5 is a schematic diagram of the composition and structure of a map construction device provided by an embodiment of the present disclosure.
  • FIG. 6 is a schematic diagram of the composition and structure of an electronic device provided by an embodiment of the present disclosure.
  • the sweeping robot is usually equipped with a variety of sensors, such as lidar, monocular camera, inertial measurement unit (Inertial Measurement Unit, IMU), wheel odometer, etc., to comprehensively utilize various sensors for self-positioning and map building. etc. to realize autonomous navigation.
  • sensors such as lidar, monocular camera, inertial measurement unit (Inertial Measurement Unit, IMU), wheel odometer, etc., to comprehensively utilize various sensors for self-positioning and map building. etc. to realize autonomous navigation.
  • Fig. 1a is a schematic diagram of a side view of a mobile device provided by an embodiment of the present disclosure
  • Fig. 1b is a schematic diagram of a top view of a mobile device provided by an embodiment of the present disclosure.
  • a depth camera-inertial measurement unit Red Green Blue-Depth-Inertial Measurement Unit, RGBD-IMU
  • RGBD-IMU Red Green Blue-Depth-Inertial Measurement Unit
  • wheel odometer 12 can be set on the mobile device, wherein the RGBD-IMU module Group includes RGB camera, depth (Depth) camera, inertial measurement unit.
  • the RGB camera is used to collect the scene image of the scene where the mobile device is located;
  • the depth camera is used to collect the depth image of the scene where the mobile device is located;
  • the inertial measurement unit and the wheel odometer are used to measure the relative pose of the mobile device.
  • the RGBD-IMU module 11 can be arranged on the front end of the mobile device, and the orientation of the RGBD-IMU module 11 is consistent with the moving direction of the mobile device, so that it is convenient for the RGB camera to collect scene images, the depth camera Collect depth images; and facilitate the inertial measurement unit to measure the relative pose of the mobile device, and also facilitate time synchronization between the scene image, the depth image and the relative pose;
  • the wheel odometer 12 can be arranged at the bottom of the mobile device, In order to facilitate the wheel odometer to measure the relative pose of the mobile device.
  • the scene image collected by the RGB camera, the depth image collected by the depth camera, and the relative pose measured by the inertial measurement unit and the wheel odometer can be used comprehensively to realize self-positioning, obstacle analysis, map construction, path planning, etc.
  • Fig. 1c is a schematic diagram of a map construction method provided by an embodiment of the present disclosure. As shown in Fig. 1c, the method can be applied to the above-mentioned mobile device, and the method includes:
  • Scene recognition is performed on the RGB image collected by the RGBD-IMU module 11 set on the mobile device to obtain the semantic information of the scene where the mobile device is located, and the semantic information includes the scene category of the scene and the obstacle category of obstacles in the scene; where, for example Scene recognition can be performed on RGB images through a neural network, and the embodiments of the present disclosure do not limit the network structure, network type and training method of the neural network;
  • the drivable area analysis (that is, obstacle analysis) is performed to obtain the drivable area (or obstacle position) of the scene; it can be understood that the depth image contains The depth information of the obstacle, the location information of the obstacle in the scene can be obtained according to the depth information of the obstacle, and then the usable area in the scene can be obtained according to the position information of the obstacle, that is, the analysis of the usable area can be realized;
  • the above semantic information, the drivable area (or obstacle position) of the scene, the self-positioning information and the grid map of the scene are fused together to obtain a grid map with semantics.
  • various information collected by the RGBD-IMU module and the wheel odometer can be integrated to construct a grid map with semantics, so that the above-mentioned mobile devices can avoid Autonomous navigation operations such as obstacles and path planning.
  • lidar is usually installed on mobile devices to collect 3D point cloud data of the scene, but the introduction of lidar will make the mechanical structure of mobile devices (such as intelligent sweeping robots) protrude to a certain height, affecting the mobile device. Accessibility (for example, some furniture cannot be drilled under or is easy to get stuck).
  • the mobile device can be designed to be shorter and more passable in the scene where it is located; and the depth image collected by the depth camera It can be used to avoid relatively low obstacles; it can also use the images taken by the RGB camera facing forward to perform semantic analysis on the scene, which can realize more intelligent autonomous navigation operations.
  • map building is a key technology for enabling autonomous navigation operations.
  • 3D map construction technology is usually used, such as Structure from motion (SFM) technology, to construct a 3D map of the scene according to the scene image, and use the relative Position and pose for self-positioning; and then combine self-positioning information with 3D maps for obstacle analysis, path planning and other navigation operations.
  • SFM Structure from motion
  • the amount of calculation and storage required to construct a 3D map is large. Since the scene where the mobile device is located will change in real time, the map construction needs to be executed multiple times in a loop. For some processors and memory capabilities Limited mobile devices may not be able to effectively cope with the high real-time map construction requirements; and, the relative pose measured by the inertial measurement unit and the wheel odometer will produce a large cumulative error as the mobile device continues to operate. The self-positioning accuracy of the mobile device is reduced.
  • the spatial feature points determined by the scene image collected by the RGB camera and the depth image collected by the depth camera can be used to perform the relative pose measured by the inertial measurement unit and/or the wheel odometer. Optimization is beneficial to reduce the impact of cumulative errors on self-positioning accuracy, and obtain higher-precision positioning information; then, the grid map determined based on higher-precision positioning information and depth images can accurately indicate the driving position in the target scene. Areas can not only meet the needs of navigation operations such as obstacle analysis and path planning, but also effectively improve the efficiency of map construction, enabling mobile devices to perform reasonable and efficient autonomous navigation.
  • the map construction method according to the embodiment of the present disclosure can be executed by a mobile device, and the mobile device may include: various types of mobile robots such as intelligent sweeping robots, intelligent cleaning robots, intelligent lawn mowing robots, and intelligent delivery and transportation robots, and self-driving vehicles that require autonomy. equipment for navigation. In this manner, the mobile device can perform more efficient and reasonable autonomous navigation operations based on higher-precision positioning information and grid maps.
  • various types of mobile robots such as intelligent sweeping robots, intelligent cleaning robots, intelligent lawn mowing robots, and intelligent delivery and transportation robots, and self-driving vehicles that require autonomy. equipment for navigation.
  • the mobile device can perform more efficient and reasonable autonomous navigation operations based on higher-precision positioning information and grid maps.
  • the map construction method according to the embodiments of the present disclosure may also be implemented by a processor (such as a processor of a mobile device) invoking computer-readable instructions stored in a memory.
  • a processor such as a processor of a mobile device
  • the RGB camera, depth camera, inertial measurement unit, and/or wheel odometer set on the mobile device can communicate with the processor, so as to collect data such as scene images, depth images, and relative poses, sent to the processor and/or memory, so as to implement the map construction method of the embodiment of the present disclosure.
  • Fig. 2 is a schematic diagram of the implementation process of a map construction method provided by an embodiment of the present disclosure. As shown in Fig. 2, the map construction method includes:
  • step S11 the scene image and the depth image of the target scene collected by the mobile device are obtained, and the relative pose of the mobile device at the current moment is obtained.
  • the mobile device may be provided with an image acquisition component, a depth sensing component, an inertial sensing component and/or a wheel odometer.
  • the scene image of the target scene can be collected by the image acquisition component set on the mobile device; the depth image is collected by the depth sensing component set on the mobile device; the relative pose can be collected by the inertial sensing component set on the mobile device and/or wheel odometer measurements.
  • the image acquisition component may include: any one of a monocular RGB camera and a binocular RGB camera;
  • the depth sensing component includes a depth camera, which may also be called a depth sensor, a time of flight (Time of flight, TOF) camera, etc.;
  • the inertial sensing assembly includes an inertial measurement unit (IMU), also referred to as an inertial sensor.
  • IMU inertial measurement unit
  • the image acquisition component, the depth sensing component and the inertial sensing component can form an information acquisition module, and the information acquisition module can be set at the front end of the mobile device, and the orientation of the information acquisition module is the same as that of the mobile device.
  • the moving direction of the mobile device is consistent, and the wheel odometer can be arranged at the bottom of the mobile device, so that each component and the wheel odometer can collect relevant information respectively, and at the same time, it is beneficial to reduce the mechanical height of the mobile device.
  • the above image acquisition component, depth sensing component, inertial sensing component and wheel odometer are set up on the mobile device, which is an implementation mode provided by the embodiments of the present disclosure. Personnel can design the location layout of each component and the wheel odometer on the mobile device according to actual needs, which is not limited by the embodiments of the present disclosure.
  • step S12 according to the scene image and the depth image, determine the first spatial feature point of the target scene at the previous moment and the second spatial feature point at the current moment, the sequence of the previous moment is before the current moment, the first spatial feature point and the Match the feature points in the second space.
  • the spatial feature point may be understood as a three-dimensional feature point in space, that is, a three-dimensional feature point in the target scene.
  • the depth image may contain depth information of the object in the target scene, and the position information of the three-dimensional feature points of the object in the scene may be obtained by using the depth information of the object in the depth image and the pixel position information of the object in the scene image.
  • the scene image and the depth image may be data collected in real time, and for the scene image and the depth image collected at any time, the scene image and the depth image collected at the time may be used to obtain the three-dimensional point cloud at the time, Based on this, feature point matching can be performed on the 3D point cloud at the current moment and the 3D point cloud at the previous moment to obtain the matching 3D feature point pairs in the two 3D point clouds, that is, to obtain the first spatial feature point at the previous moment , and the second spatial feature point at the current moment.
  • formula (1) can be used to realize the scene image and depth image collected at any time, and use the scene image and depth image collected at this time to obtain the 3D point cloud at this time.
  • (M x , M y , M z ) represent the three-dimensional coordinates of the three-dimensional point cloud
  • (m x , my y ) represent the pixel coordinates of the pixel points in the scene image respectively
  • d represents the depth value corresponding to the pixel point in the depth image
  • f x , f y respectively represent the focal length of the image acquisition component in the direction of the horizontal axis x and the vertical axis y of the imaging plane.
  • performing feature point matching on the 3D point cloud at the current moment and the 3D point cloud at the previous moment may include, for example: extracting multiple first spatial feature points in the 3D point cloud at the current moment, and a plurality of second spatial feature points in the three-dimensional point cloud at the previous moment; for any first spatial feature point, calculate the distance between the first spatial feature point and each second spatial feature point; The corresponding second spatial feature point is used as a spatial feature point matching the first spatial feature point.
  • feature point extraction methods known in the art can be used, for example, point feature histogram (Point Feature Histogram, PFH), simple point feature histogram (Simple Point Feature Histograms, SPFH) to realize each three-dimensional point cloud
  • point feature histogram Point Feature Histogram
  • simple point feature histogram Simple Point Feature Histograms, SPFH
  • SPFH Simple Point Feature Histograms
  • first spatial feature point matches the second spatial feature point
  • first spatial feature point at the previous moment and the second spatial feature point at the current moment jointly indicate the object on the same object in the target scene. same point.
  • poses of the mobile device are different at different times, and the first spatial feature point and the second spatial feature point are based on spatial feature points determined under different poses of the mobile device.
  • step S13 the relative pose is optimized according to the first spatial feature point and the second spatial feature point, and the positioning information of the mobile device at the current moment is obtained.
  • the relative pose may include: the relative position and heading angle of the mobile device at the current moment relative to the previous moment. That is, the relative pose can be understood as the relative value between the position of the mobile device at the current moment and the position of the mobile device at the previous moment, and the heading angle at the current moment.
  • the first spatial feature point at the previous moment matches the second spatial feature point at the current moment, then theoretically, based on the relative pose at the current moment, the first spatial feature point is obtained by mapping transformation
  • the feature points in the space should coincide with the feature points in the second space, and there is an error between the relative pose measured by the inertial sensing component and/or the wheel odometer and the actual pose of the mobile device. Therefore, for the first space There is also an error between the spatial feature points obtained by mapping and transforming the feature points and the second spatial feature points.
  • optimizing the relative pose according to the first spatial feature point and the second spatial feature point to obtain the positioning information of the mobile device at the current moment may include: The spatial feature points are mapped and transformed to obtain the third spatial feature point; according to the error between the third spatial feature point and the second spatial feature point, the relative pose is optimized to obtain the positioning information of the mobile device at the current moment.
  • the relative pose at any time can be obtained through step S13, and the pose (ie, position and heading angle), or the pose relative to the initial position (such as the origin of the spatial coordinate system of the target scene), that is, to obtain the actual pose of the mobile device in the target scene.
  • the position of the mobile device can be represented by two-dimensional coordinates
  • the attitude of the mobile device can be represented by the heading angle, which can be understood as the moving orientation of the mobile device based on the horizontal direction.
  • FIG. 3 is a schematic diagram of a plane coordinate system provided by an embodiment of the present disclosure.
  • the plane coordinate system shown in Figure 3 is used as an example to illustrate a way to determine the positioning information of a mobile device.
  • the relative pose at time t1 includes Heading angle ⁇ 1 and relative position (x 1 , y 1 )
  • the relative pose at time t2 includes heading angle ⁇ 2 and relative position (x 2 , y 2 )
  • the actual pose of the mobile device at time t1 is also the
  • the relative pose at time t1 and the actual pose of the mobile device at time t2 include: position (x 1 +x 2 , y 1 +y 2 ) and heading angle ⁇ 2 .
  • the position of the mobile device can also be represented by three-dimensional coordinates
  • the posture of the mobile device can be represented by a three-degree-of-freedom attitude angle, which can be determined according to the actual situation of the target scene, which is not limited by the embodiments of the present disclosure.
  • the ground is relatively flat, and the two-dimensional coordinates and heading angle can be used as the pose of the mobile device to meet the self-positioning requirements; for some outdoor mobile robots, the ground has large fluctuations, and three-dimensional coordinates and three-freedom may be required degree attitude angle to characterize the pose of the mobile device.
  • step S14 the grid map corresponding to the target scene is determined according to the positioning information and the depth image, and the grid map is used to indicate the drivable area in the target scene, so that the mobile device can perform navigation operations according to the grid map.
  • the objects in the target scene can be understood as obstacles of the mobile device, and the area occupied by the obstacles in the target scene can be called a non-driving area.
  • the area in the target scene other than the non-drivable area may be called the drivable area of the mobile device, that is, the mobile device can move within the drivable area.
  • the target scene is, for example, a living room, a bedroom, etc.
  • the area and shape of the target scene (such as indoor area and shape) are usually fixed.
  • the mobile device when it enters the target scene for the first time, it can scan the target scene in all directions, for example, collect a panoramic depth image, and use the panoramic depth image to determine the overall outline of the target scene (such as The overall outline of the room) or the area range of the target scene; and then the determined overall outline can be used to obtain a two-dimensional plane corresponding to the target scene. It should be understood that the area and shape of the two-dimensional plane can be consistent with the target scene.
  • the two-dimensional plane of the target scene may be grid-divided according to a certain grid size to obtain a grid plane.
  • the grid size can be set according to actual needs, and the granularity and number of grids in the grid plane are related to the grid size and the area of the target scene, which is not limited by the embodiments of the present disclosure.
  • the depth image includes depth information of the object in the target scene, and the depth information can also be understood as the distance between the object in the target scene and the mobile device. Then, according to the positioning information of the mobile device and the depth information of the obstacles in the target scene contained in the depth image, the position information of the obstacles in the target scene can be obtained; In the grid plane, the grid of the obstacle is determined and marked to obtain the grid map of the target scene.
  • the depth image may contain depth information of obstacles of various heights.
  • the obstacle avoidance capability of the mobile device can be increased, especially for Obstacle avoidance for low obstacles such as a sock dropped to the ground.
  • the grid map of the target scene will change.
  • the above-mentioned map construction method of steps S11 to S14 may be executed cyclically, so as to update the grid map of the target scene in real time.
  • the navigation logic for navigation operations based on the grid map can be deployed on the mobile device, that is, the navigation operations such as planning the moving path and analyzing obstacles based on the grid map can be realized on the mobile device;
  • the grid map can also be sent to the back-end server, that is, the navigation logic is deployed in the server, and the server realizes planning the moving path and analyzing obstacles according to the grid map, and returns the planning results and analysis results to the mobile device to control
  • the mobile device moves, which is not limited by the embodiments of the present disclosure.
  • the spatial feature points determined by the scene image and the depth image can be used to optimize the relative pose of the mobile device at the current moment, so that higher-precision positioning information can be obtained;
  • the grid map determined by the positioning information and the depth image can accurately indicate the feasible area in the target scene.
  • the grid map is a two-dimensional map, it can not only meet the requirements of path planning and obstacle detection for mobile devices moving on a plane. It can also effectively reduce the amount of calculation and storage required for map construction, improve the efficiency of map construction, and enable mobile devices to perform reasonable and efficient autonomous navigation operations. It is also conducive to reducing the mechanical height of mobile devices and increasing the size of mobile devices. mobility and accessibility.
  • the grid map can also be a three-dimensional map.
  • the mobile device when it enters the target scene for the first time, it can scan the target scene in all directions, for example, collect a panoramic depth image, and use the panoramic depth image to determine the overall outline of the target scene (such as the overall outline of the room) or The range of the spatial area of the target scene; and then the determined overall outline can be used to obtain the three-dimensional space corresponding to the target scene. It should be understood that the volume and shape of the three-dimensional space can be consistent with the target scene.
  • the three-dimensional space of the target scene may be grid-divided according to a certain three-dimensional grid size (ie, length, width, and height) to obtain a three-dimensional grid space.
  • a certain three-dimensional grid size ie, length, width, and height
  • the size of the 3D grid can be set according to actual needs, and the granularity and quantity of the 3D grid in the 3D grid space are related to the size of the 3D grid and the volume of the target scene, which is not limited in this embodiment of the present disclosure.
  • the depth image includes depth information of the object in the target scene, and the depth information can also be understood as the distance between the object in the target scene and the mobile device.
  • determining the grid map corresponding to the target scene may include: according to the positioning information of the mobile device, and the obstacle in the target scene contained in the depth image The depth information of the object can be used to obtain the three-dimensional position information of the obstacle in the target scene; then, using the three-dimensional position information of the obstacle, in the three-dimensional grid space of the above target scene, the grid to which the obstacle belongs is determined and marked, and the target grid is obtained.
  • a raster map of the scene may include: according to the positioning information of the mobile device, and the obstacle in the target scene contained in the depth image The depth information of the object can be used to obtain the three-dimensional position information of the obstacle in the target scene; then, using the three-dimensional position information of the obstacle, in the three-dimensional grid space of the above target scene, the grid to which the obstacle belongs is determined and marked, and the target grid is obtained.
  • the three-dimensional grid map can indicate the three-dimensional drivable area in the scene, so that the mobile device can perform more reasonable navigation operations based on the three-dimensional drivable area, for example, the planning can be more reasonable and accurate path of movement.
  • the scene image is collected by the image acquisition component set on the mobile device
  • the depth image is collected by the depth sensing component set on the mobile device.
  • step S12 according to The scene image and the depth image determine the first spatial feature point of the target scene at the previous moment and the second spatial feature point of the current moment, which may include:
  • Step S121 Extracting feature points from the scene image collected at the previous moment to obtain feature points of the first image.
  • image feature point extraction techniques known in the art can be used, for example, scale-invariant feature transform (Scale-invariant feature transform, SIFT) algorithm, fast feature point extraction and description (Oriented FAST and Rotated BRIEF, ORB) algorithm, etc. , to implement feature point extraction on the scene image collected at the previous moment, which is not limited in this embodiment of the present disclosure.
  • scale-invariant feature transform Scale-invariant feature transform, SIFT
  • SIFT Scale-invariant feature transform
  • FAST and Rotated BRIEF ORB
  • Step S122 According to the first image feature point, determine the second image feature point matching the first image feature point in the scene image collected at the current moment.
  • tracking algorithms known in the art can be used, for example, feature point tracking algorithm (that is, use the similarity of feature points to track), optical flow tracking algorithm (that is, use the motion trajectory of feature points to track), to achieve the first
  • the image feature point is to determine the second image feature point matching the first image feature point in the scene image collected at the current moment, which is not limited in this embodiment of the present disclosure.
  • Step S123 According to the mapping relationship between the scene image and the depth image, determine a first depth value corresponding to the feature point of the first image, and a second depth value corresponding to the feature point of the second image.
  • the position of the image acquisition component (such as a monocular RGB camera) and the depth sensing component (such as a depth camera) on the mobile device is fixed, so that the distance between the image acquisition component and the depth sensing component can be known. Based on the relative positional relationship, the mapping relationship between the image coordinate system (ie scene image) of the image acquisition component and the image coordinate system (and depth image) of the depth sensing component can be obtained.
  • mapping relationship may be represented by a mapping matrix, and the mapping matrix may include a rotation matrix and a translation matrix, which are not limited in this embodiment of the present disclosure.
  • the depth image or the scene image collected at any time can be mapped and transformed, so as to transform the depth image and the scene image into the same image coordinate system, for example , transforming the depth image into the image coordinate system corresponding to the scene image; furthermore, in the same image coordinate system, the corresponding depth value can be determined from the corresponding depth image according to the position of any image feature point in the scene image.
  • the first depth value is determined from the depth image at the previous moment, and the position of the second image feature point in the scene image at the current moment is determined. , determine the second depth value from the depth image at the current moment.
  • Step S124 Determine the first spatial feature point according to the first image feature point, the first depth value, and the internal camera parameters of the image acquisition component.
  • the internal camera parameters may include focal lengths f x , f y in the directions of the horizontal axis x and the vertical axis y .
  • the above formula (1) can be used to determine the first spatial feature point according to the first image feature point, the first depth value, and the camera internal reference of the image acquisition component, and also That is, taking the position of the first image feature point as (m x , m y ) and the first depth value as d, the three-dimensional coordinates of the first spatial feature point can be obtained.
  • Step S125 Determine the second spatial feature point according to the second image feature point, the second depth value, and the internal camera parameters of the image acquisition component.
  • the intrinsic camera parameters may include focal lengths f x , f y in the directions of the horizontal axis x and the vertical axis y .
  • the above formula (1) can be used to determine the second spatial feature point according to the second image feature point, the second depth value, and the camera internal reference of the image acquisition component.
  • the matching image feature points in the scene image and the depth image can be efficiently determined, and then the corresponding matching first spatial feature points and second spatial feature points can be determined.
  • the mobile device can be equipped with an inertial sensing component and a wheel odometer, and the inertial sensing component and the wheel odometer can each measure the relative pose of the mobile device, since the inertial sensing component and the wheel odometer measure the Relative to the pose, there will be a problem of low positioning accuracy due to cumulative errors.
  • the above-mentioned first spatial feature points and second spatial feature points, as well as inertial sensing components and wheel The error between the two relative poses measured by the formula odometer is used to optimize any one of the relative poses to obtain higher-precision positioning information.
  • the multiple relative poses may include relative poses measured by inertial sensing components set on mobile devices, and wheel odometers set on mobile devices. relative pose.
  • the inertial sensing component (that is, the inertial measurement unit) includes an accelerometer and a gyroscope, and the angular velocity and acceleration of the mobile device can be measured through the acceleration and gyroscope, and then the angular velocity and acceleration are pre-integrated, From the previous moment to the current moment, the moving distance (ie relative position) and moving direction (ie heading angle) of the mobile device can be calculated to obtain the relative pose measured by the inertial sensing component.
  • the wheel odometer can measure the rotational speed of the driving wheel of the mobile device (ie, the angular velocity and the linear velocity of the driving wheel), and then according to the rotational speed of the driving wheel and the mechanical structure information of the driving wheel (such as the driving wheel Radius and wheelbase), can calculate the moving distance (that is, the relative position) and the moving direction (that is, the heading angle) of the mobile device from the previous moment to the current moment, and obtain the relative pose measured by the wheel odometer.
  • the rotational speed of the driving wheel of the mobile device ie, the angular velocity and the linear velocity of the driving wheel
  • the mechanical structure information of the driving wheel such as the driving wheel Radius and wheelbase
  • step S13 according to the first spatial feature point and the second spatial feature point, the relative pose is optimized to obtain the positioning information of the mobile device at the current moment, including:
  • Step S131 For any relative pose, transform the first spatial feature point to the third spatial feature point at the current moment according to the relative pose;
  • the relative pose measured by the wheel odometer and the relative pose measured by the inertial sensing component can both represent the relative pose of the mobile device at the current moment, thus, any one of the relative poses can be selected as The object of optimization, that is, as the initial value of optimization.
  • the relative pose measured by the wheel odometer may be selected as the optimization object, which is not limited in this embodiment of the present disclosure.
  • converting the first spatial feature point to the third spatial feature point at the current moment according to the relative pose may include: performing mapping transformation on the first spatial feature point according to the relative pose to obtain the third spatial feature point.
  • formula (2) can be used to realize the mapping transformation of the first spatial feature point according to the relative pose.
  • X represents the first spatial feature point
  • X * represents the third spatial feature point
  • R and t represent the relative pose
  • R represents the rotation matrix (that is, the matrix determined by the heading angle)
  • t represents the translation matrix (that is, the relative position determined matrix).
  • Step S132 According to the first error between the second spatial feature point and the third spatial feature point, and/or the second error between the relative pose and other relative poses, optimize the relative pose to obtain the optimized relative pose, wherein the other relative poses include relative poses other than the relative pose among the plurality of relative poses.
  • the first spatial feature point at the previous moment matches the second spatial feature point at the current moment, then theoretically, according to the relative pose at the current moment, the first spatial feature point is obtained by mapping transformation
  • the spatial feature point should coincide with the second spatial feature point, but there is an error between the relative pose measured by the inertial sensing component and the wheel odometer and the actual pose of the mobile device, thus, the first spatial feature point
  • an iterative optimization algorithm known in the art such as Gauss-Newton algorithm, Levenberg-Marquardt algorithm, etc., can be used to realize the optimization according to the first error and/or the second error Relative pose, the optimized relative pose is obtained.
  • optimizing the relative pose according to the first error and the second error can make the precision of the optimized relative pose higher.
  • the relative pose is optimized by the Gauss-Newton algorithm, for example, including: using the relative pose as the initial value to be optimized; bringing the relative pose, the first error and the second error into the Taylor series expansion to obtain a nonlinear regression model ;Use the least squares method to solve the nonlinear regression model to obtain the correction coefficient; correct the initial value through the correction coefficient to obtain the correction value; then use the correction value as the initial value to be optimized, and execute multiple rounds iteratively according to the above process, until The correction value converges, and the optimized relative pose is obtained.
  • Step S133 Determine the location information of the mobile device at the current moment according to the optimized relative pose.
  • the determination method of the positioning information disclosed in the above-mentioned embodiments of the present disclosure can be referred to, for example, the determination method of the positioning information shown in FIG. 3 , so as to determine the positioning information of the mobile device at the current moment according to the optimized relative pose.
  • the error between the first spatial feature point and the second spatial feature point, and the two relative poses measured by the inertial sensor component and the wheel odometer can be used comprehensively, and any relative The pose is optimized to obtain higher-precision positioning information.
  • step S14 according to the positioning information and the depth image, the grid map corresponding to the target scene is determined, including:
  • Step S141 Determine the two-dimensional plane corresponding to the target scene, and perform grid division on the two-dimensional plane to obtain the grid plane;
  • the mobile device when it enters the target scene for the first time, it can scan the target scene in all directions, for example, collect a panoramic depth image, and use the panoramic depth image to determine the overall outline of the target scene (such as the overall Outline) or the area range of the target scene; and then the determined overall outline can be used to obtain a two-dimensional plane corresponding to the target scene. It should be understood that the area and shape of the two-dimensional plane can be consistent with the target scene.
  • the two-dimensional plane of the target scene may be grid-divided according to a certain grid size to obtain a grid plane.
  • the grid size can be set according to actual needs, and the granularity and number of grids in the grid plane are related to the grid size and the area of the target scene, which is not limited by the embodiments of the present disclosure.
  • Step S142 Determine the position information of the obstacle in the target scene according to the positioning information and the depth image
  • the depth image includes depth information of the object in the target scene, and the depth information can also be understood as the distance between the object in the target scene and the mobile device. Then, according to the positioning information of the mobile device and the depth information of the obstacle in the target scene contained in the depth image, the position information of the obstacle in the target scene can be obtained.
  • FIG. 4 is a schematic diagram of a plane coordinate system of a grid plane provided by an embodiment of the present disclosure.
  • the positioning information of the mobile device is represented by two-dimensional coordinates (x, y) and heading angle ⁇
  • the depth information of obstacles is represented by d 0
  • the position information of obstacles is represented by (x 0 , y 0 )
  • the location information of obstacles can be obtained through simultaneous equations (3).
  • Step S143 According to the location information of the obstacle, determine and mark the grid to which the obstacle belongs in the grid plane, and obtain the grid map corresponding to the target scene.
  • the grid plane corresponds to the target scene, and according to the position information of the obstacle in the target scene, the grid to which the obstacle belongs in the grid plane can be obtained, and then according to the preset marking method, the grid where the obstacle belongs can be obtained.
  • Marking in the grid such as marking numbers, filling colors, etc., to indicate that the grid is the grid to which the obstacle belongs, and also means that the area indicated by the grid is a non-driving area.
  • the embodiment of the present disclosure does not limit the marking manner of the grid to which the obstacle belongs in the grid map.
  • the grid map of the target scene can be generated conveniently and effectively, and the generated grid map can not only meet the needs of navigation operations such as path planning and obstacle analysis, but also effectively reduce the amount of calculation required for map construction and storage capacity, improving map construction efficiency.
  • the obstacles in different target scenes are also different, for example, the object is different from the obstacle in the bedroom.
  • obstacles in the target scene can be identified to obtain the semantic information of obstacles in the target scene, and then the semantic information of obstacles and the above-mentioned grid map can be combined to make the mobile device execute More intelligent navigation operations, for example, for intelligent sweeping robots, can more intelligently formulate reasonable cleaning routes.
  • step S14 according to the positioning information and the depth image, the grid map corresponding to the target scene is determined, including:
  • Step S144 Determine the initial grid map corresponding to the target scene according to the positioning information and the depth image.
  • the initial grid map in the embodiment of the present disclosure may be obtained by referring to the above-mentioned process of generating the grid map from step S141 to step S143.
  • Step S145 Recognize the scene image, and determine the scene category of the target scene and the obstacle category of obstacles in the target scene.
  • the image recognition technology known in the art can be used, for example, a neural network for image recognition, to recognize the scene image, to obtain the scene category of the target scene and the obstacle category of obstacles in the target scene, that is, to obtain the target Semantic information of the scene and semantic information of obstacles.
  • the scene category of the target scene can represent different target scenes, such as bedroom, living room, kitchen, etc.
  • the obstacle category of obstacles can represent different obstacles, such as people, pets, sofas, tables, stools, etc.
  • Step S146 Generate a grid map according to the scene category, the obstacle category and the initial grid map.
  • generating a grid map according to the scene category, obstacle category, and initial grid map may include: filling the scene category as an attribute value into one or more grids of the initial grid map Middle; fill in the obstacle category as an attribute value in the grid corresponding to the obstacle.
  • different scene categories and different obstacle categories can be represented by arbitrary characters, numbers, etc., so that characters, numbers, etc. that can represent different scene categories and different obstacle categories can be used as attribute values to fill in the grid.
  • the disclosed embodiments are not limiting.
  • each grid in the grid map can represent: whether the area indicated by the grid is a drivable area; if it is a non-drivable area, what types of obstacles are contained in the undrivable area, such as unknown obstacles (such as the outline of a room), or a specific type of obstacle (such as a pet), etc.; and the type of scene that the grid or the grid map belongs to, for example, whether it belongs to the living room or the bedroom.
  • the mobile device can not only perform more intelligent navigation tasks but also make some intelligent navigation decisions based on the grid map containing rich semantic information. For example, for intelligent sweeping robots, it can be based on The above-mentioned grid map formulates the cleaning route; it can also be based on the semantic information in the grid map. For example, if there are people in the room, the room will not be cleaned first; if there are shoes on the moving path, the shoes will be avoided and the user will be prompted to pack the shoes.
  • the generated grid map can contain rich semantic information in the target scene, so that the mobile device can perform more intelligent navigation operations.
  • the position of the obstacle in the target scene where the mobile device is located may change in real time, and correspondingly, the grid map may also change at any time.
  • the mobile device can move in different target scenes, and can also move in different areas in the same target scene.
  • the grid in view of the position change of the obstacle where the mobile device is in the target scene, or the obstacle in the target scene is within the field of view of the mobile device, the grid can be adjusted in real time according to the determined obstacle position, obstacle type, etc.
  • the map is updated.
  • the mobile device is already outside the target scene, for example, when the intelligent sweeping robot moves from the bedroom to the living room, or the obstacle in the target scene is already outside the mobile device's field of view, it cannot perceive that it is outside the mobile device's field of view.
  • the position changes of the obstacles are already outside the target scene, for example, when the intelligent sweeping robot moves from the bedroom to the living room, or the obstacle in the target scene is already outside the mobile device's field of view.
  • the method further include:
  • Step S16 According to the obstacle category corresponding to the non-drivable area, set an adjustment coefficient and a preset constant for the grid representing the non-drivable area in the grid map, wherein the adjustment coefficient is used to indicate the decay rate or growth rate of the preset constant.
  • different adjustment coefficients can be preset according to the obstacle type, or the corresponding relationship between the obstacle type and the adjustment coefficient can be preset, and then the adjustment coefficient can be set for the grid representing the non-driving area according to the obstacle type .
  • the preset constants may be different constants according to obstacle types, or the same constant may be set, which is not limited in this embodiment of the present disclosure.
  • the corresponding relationship between obstacle types and adjustment coefficients can be preset. For example, for obstacles with a higher probability of moving positions, a larger adjustment coefficient can be set; For an obstacle with a small probability of moving position, a small adjustment coefficient may be set, which is not limited in this embodiment of the present disclosure.
  • the adjustment coefficient is used to indicate the decay speed or growth speed of the preset constant, where the adjustment coefficient can be set to be positively correlated with the decay speed or growth speed, for example, the larger the adjustment coefficient, the faster the decay speed or growth speed of the preset constant The faster the value, the higher the probability that the obstacle will change its position in real time; on the contrary, the smaller the adjustment coefficient, the slower the attenuation or growth rate of the preset constant, and the lower the possibility that the obstacle will change its position in real time.
  • the adjustment coefficient may also be set to be negatively correlated with the decay speed or the growth speed, which is not limited in this embodiment of the present disclosure.
  • an adjustment coefficient and a preset constant can be set for the grid;
  • an adjustment coefficient and a preset constant are set for the grid representing the non-drivable area, which is not limited in this embodiment of the present disclosure.
  • the non-driving area in the target scene is outside the field of view of the mobile device, that is, the obstacle in the target scene is outside the field of view of the mobile device; the obstacle is outside the field of view of the mobile device, which can be understood as an obstacle
  • the object is outside the shooting range (field angle) of the image acquisition component and the depth sensing component, that is, the scene image and the depth image collected by the image capture component and the depth sensing component do not contain the obstacle.
  • the possibility of changing the position of obstacles can be reflected by using the adjustment coefficient and the preset constant, or the position change of obstacles can be predicted, so as to update the grid map, especially for those already in the field of view of mobile devices
  • the grid corresponding to the external obstacles is updated.
  • Step S17 Update the grid map according to the adjustment coefficient and the preset constant to obtain the updated grid map, so that the mobile device can perform navigation operations according to the updated grid map.
  • the grid map is updated according to the adjustment coefficient and the preset constant to obtain the updated grid map, which includes: for any grid in the grid map that represents an undrivable area, according to the grid
  • the attenuation rate indicated by the adjustment coefficient of the grid decreases the preset constant of the grid, or, according to the growth rate indicated by the adjustment coefficient of the grid, the preset constant of the grid is increased; when the preset constant decreases to less than the first preset threshold, Or when the preset constant is increased to be greater than the second preset threshold, the grid is set as the drivable area to obtain an updated grid map. In this manner, it is possible to efficiently update the grid map according to the adjustment coefficient and the preset constant.
  • the values of the first preset threshold and the second preset threshold may be set according to values of preset constants and adjustment coefficients, historical experiences, etc., which are not limited in this embodiment of the present disclosure.
  • the preset constant decreases to be less than the first preset threshold, or the preset constant increases to be greater than the second preset threshold, it can be considered that the obstacle has left the original occupied position as time goes by.
  • the grid to which the obstacle belongs can be set as the drivable area, for example, the number used to mark the undrivable area is changed to the number used to mark the drivable area, etc., to obtain an updated grid map.
  • updating the grid map according to the adjustment coefficient and the preset constant is a prediction of the position change of the obstacle, and does not reflect the actual position change of the obstacle. If for any grid representing an obstacle, the preset constant of the grid decreases to less than the first preset threshold, or the preset constant increases to greater than the second preset threshold, but the obstacle is always moving within the field of view of the device; or, the obstacle is again within the field of view from outside the field of view, at this time, the grid map is still updated with the position of the obstacle actually determined in the above-mentioned embodiments of the present disclosure.
  • the mobile device can perform intelligent navigation operations and navigation decisions based on the updated grid map.
  • the preset constant corresponding to the person decays to less than the first preset threshold with the adjustment coefficient, it can be considered that the person is no longer in the room at this time, and the moving path to clean the room can be planned according to the updated grid map ; or, when there are shoes on the floor of the room, the shoes can be avoided for cleaning.
  • the preset constant corresponding to the shoes increases with the adjustment coefficient to be greater than the second preset threshold, it can be considered that the shoes are no longer in the original position. Then, according to the updated grid map, the movement path for cleaning the area originally occupied by the shoes can be planned.
  • the prediction of obstacles outside the field of view of the mobile device can be achieved.
  • Position changes which in turn enable mobile devices to achieve more intelligent navigation operations, navigation decisions, etc.
  • the mobile device (such as an intelligent sweeping robot) not only has the advantages of the existing planned mobile device, but also does not need to introduce a laser radar structure, thereby increasing the passability of the mobile device in the scene; it can also Increase the obstacle avoidance ability of mobile devices to avoid low obstacles; introduce a forward-facing depth camera and RGB camera, which can shoot indoor objects and scenes, combined with visual perception algorithms (recognition of scene categories and obstacles in scene images) category) to realize the semantic understanding of the scene, so that the mobile device can perform more intelligent navigation tasks.
  • the depth image is obtained by the TOF camera.
  • the accuracy of the information is higher and the computational overhead is lower.
  • the RGBD-IMU is an integral module, the RGB image and the depth image can be synchronized in good time, so the depth information in the depth image can also be used in the self-positioning of the mobile device (that is, to determine the location of the mobile device). location information).
  • the mobile device can be designed to be shorter and more passable in the scene where it is located; and the depth image collected by the depth camera can be It is used to avoid low obstacles; compared with a mobile device with an upward-facing camera and no lidar, it can use the images collected by the forward-facing RGB camera to perform semantic analysis on the scene; compared with only Compared with mobile devices with depth cameras, in addition to obstacle avoidance and path planning, it can also perform semantic analysis of the scene in which it is located, so as to perform more intelligent navigation tasks.
  • this disclosure also provides map construction devices, electronic equipment, computer-readable storage media, and programs, all of which can be used to implement any of the map construction methods provided by this disclosure, and refer to the corresponding technical solutions and descriptions in the method section. .
  • Fig. 5 is a schematic diagram of the composition and structure of a map construction device provided by an embodiment of the present disclosure. As shown in Fig. 5, the device includes:
  • the acquiring module 101 is configured to acquire the scene image and the depth image of the target scene collected by the mobile device, and acquire the relative pose of the mobile device at the current moment;
  • the determining module 102 is configured to determine the first spatial feature point of the target scene at a previous moment and the second spatial feature point at the current moment according to the scene image and the depth image, and the time sequence of the previous moment is Before the current moment, the first spatial feature point matches the second spatial feature point;
  • the optimization module 103 is configured to optimize the relative pose according to the first spatial feature point and the second spatial feature point, and obtain the positioning information of the mobile device at the current moment;
  • the construction module 104 is configured to determine a grid map corresponding to the target scene according to the positioning information and the depth image, and the grid map is used to indicate the drivable area in the target scene, so that the The mobile device performs navigation operations according to the grid map.
  • the scene image is collected by an image acquisition component set on the mobile device
  • the depth image is collected by a depth sensing component set on the mobile device
  • the determining module 102 includes: an extracting submodule configured to extract feature points from scene images collected at a previous moment to obtain first image feature points; a matching submodule is configured to determine according to the first image feature points The second image feature point matched with the first image feature point in the scene image collected at the current moment; the mapping submodule is configured to determine the feature point corresponding to the depth image according to the mapping relationship between the scene image and the depth image The first depth value corresponding to the feature point of the first image, and the second depth value corresponding to the feature point of the second image; the first determining submodule is configured to according to the feature point of the first image, the first depth value The depth value and the internal reference of the camera of the image acquisition component determine the first spatial feature point; the second determining submodule is configured to be based on the second image feature point, the second depth value and the image
  • the optimization module 103 includes: a transformation submodule configured to, for any relative pose, according to the relative pose, transform the The first spatial feature point is converted to the third spatial feature point at the current moment; the optimization submodule is configured to be based on the first error between the second spatial feature point and the third spatial feature point, and /or the second error between the relative pose and other relative poses, optimize the relative pose to obtain the optimized relative pose, wherein the other relative poses include multiple relative poses except The relative pose other than the relative pose; the positioning information determination submodule is configured to determine the positioning information of the mobile device at the current moment according to the optimized relative pose; wherein, the plurality of relative The pose includes the relative pose measured by the inertial sensing component set on the mobile device, and the relative pose measured by the wheel odometer set on the mobile device.
  • the construction module 104 includes: a grid plane determination submodule configured to determine a two-dimensional plane corresponding to the target scene, and perform grid division on the two-dimensional plane to obtain a grid plane grid plane; the location information determination submodule is configured to determine the location information of obstacles in the target scene according to the positioning information and the depth image; the first grid map determination submodule is configured to determine the location information according to the For the position information of the obstacle, determine and mark the grid to which the obstacle belongs in the grid plane, and obtain the grid map corresponding to the target scene.
  • the construction module 104 includes: an initial grid map determination submodule configured to determine the initial grid map corresponding to the target scene according to the positioning information and the depth image
  • the identification submodule is configured to identify the scene image, determine the scene category of the target scene and the obstacle category of obstacles in the target scene;
  • the second grid map determination submodule is configured according to The scene category, the obstacle category, and the initial grid map generate the grid map.
  • the device further includes: a setting module configured to, according to the obstacle category corresponding to the non-drivable area in the target scene, set the Grid setting adjustment coefficients and preset constants, wherein the adjustment coefficients are used to indicate the decay speed or growth speed of the preset constants; the update module is configured to update The grid map obtains an updated grid map, so that the mobile device performs navigation operations according to the updated grid map.
  • the update module includes: an adjustment submodule configured to, for any grid representing the non-drivable area in the grid map, according to the adjustment coefficient indicated by the grid
  • the decay speed is to decrease the preset constant of the grid, or to increase the preset constant of the grid according to the growth rate indicated by the adjustment coefficient of the grid;
  • the update submodule is configured to When the constant decreases to less than the first preset threshold, or the preset constant increases to greater than the second preset threshold, the grid is set as a drivable area to obtain an updated grid map.
  • the relative pose includes: the relative position and heading angle of the mobile device at the current moment relative to the previous moment; the image acquisition component, the depth sensing component, and the inertial
  • the sensing components form an information collection module, the information collection module is arranged on the front end of the mobile device, and the orientation of the information collection module is consistent with the moving direction of the mobile device; the wheeled mileage of the mobile device
  • the gauge is set at the bottom of the mobile device.
  • the spatial feature points determined by the scene image and the depth image can be used to optimize the relative pose of the mobile device at the current moment, so that higher-precision positioning information can be obtained;
  • the grid map determined by the positioning information and the depth image can accurately indicate the feasible area in the target scene, which can not only meet the needs of navigation operations such as path planning and obstacle analysis, but also effectively improve the efficiency of map construction and make mobile devices perform reasonably. Efficient autonomous navigation operations.
  • the functions or modules included in the apparatus provided by the embodiments of the present disclosure may be used to execute the methods described in the above method embodiments, and for specific implementation, refer to the descriptions of the above method embodiments.
  • Embodiments of the present disclosure also provide a computer-readable storage medium, on which computer program instructions are stored, and the above-mentioned method is implemented when the computer program instructions are executed by a processor.
  • Computer readable storage media may be volatile or nonvolatile computer readable storage media.
  • An embodiment of the present disclosure also proposes an electronic device, including: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to invoke the instructions stored in the memory to execute the above method.
  • An embodiment of the present disclosure also provides a computer program product, including computer-readable codes, or a non-volatile computer-readable storage medium carrying computer-readable codes, when the computer-readable codes are stored in a processor of an electronic device When running in the electronic device, the processor in the electronic device executes the above method.
  • Electronic devices may be provided as terminal devices, mobile devices or other forms of devices.
  • FIG. 6 is a schematic diagram of the composition and structure of an electronic device 800 provided by an embodiment of the present disclosure.
  • the electronic device 800 may be a mobile device, or a terminal device such as a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, or a personal digital assistant.
  • electronic device 800 may include one or more of the following components: processing component 802, memory 804, power supply component 806, multimedia component 808, audio component 810, input/output (I/O) interface 812, sensor component 814 , and the communication component 816.
  • the processing component 802 generally controls the overall operations of the electronic device 800, such as those associated with display, telephone calls, data communications, camera operations, and recording operations.
  • the processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above method. Additionally, processing component 802 may include one or more modules that facilitate interaction between processing component 802 and other components. For example, processing component 802 may include a multimedia module to facilitate interaction between multimedia component 808 and processing component 802 .
  • the memory 804 is configured to store various types of data to support operations at the electronic device 800 . Examples of such data include instructions for any application or method operating on the electronic device 800, contact data, phonebook data, messages, pictures, videos, and the like.
  • the memory 804 can be realized by any type of volatile or non-volatile storage device or their combination, such as Static Random-Access Memory (Static Random-Access Memory, SRAM), Electrically Erasable Programmable Read-Only Memory (Electrically Erasable Programmable Read Only Memory, EEPROM), Erasable Programmable Read-Only Memory (Erasable Programmable Read-Only Memory, EPROM), Programmable Read-Only Memory (Programmable Read-Only Memory, PROM), Read-Only Memory (Read-Only Memory, ROM), magnetic memory, flash memory, magnetic disk or optical disk.
  • Static Random-Access Memory SRAM
  • Electrically Erasable Programmable Read-Only Memory Electrically Erasable Programmable Read-Only Memory (Electrically Erasable
  • the power supply component 806 provides power to various components of the electronic device 800 .
  • Power components 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for electronic device 800 .
  • the multimedia component 808 includes a screen providing an output interface between the electronic device 800 and the user.
  • the screen may include a liquid crystal display (Liquid Crystal Display, LCD) and a touch panel (Touch Panel, TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user.
  • the touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensor may not only sense a boundary of a touch or swipe action, but also detect duration and pressure associated with the touch or swipe action.
  • the multimedia component 808 includes a front camera and/or a rear camera. When the electronic device 800 is in an operation mode, such as a shooting mode or a video mode, the front camera and/or the rear camera can receive external multimedia data. Each front camera and rear camera can be a fixed optical lens system or have focal length and optical zoom capability.
  • the audio component 810 is configured to output and/or input audio signals.
  • the audio component 810 includes a microphone (Microphone, MIC), and when the electronic device 800 is in an operation mode, such as a calling mode, a recording mode and a voice recognition mode, the microphone is configured to receive an external audio signal. Received audio signals may be further stored in memory 804 or sent via communication component 816 .
  • the audio component 810 also includes a speaker for outputting audio signals.
  • the I/O interface 812 provides an interface between the processing component 802 and a peripheral interface module, which may be a keyboard, a click wheel, a button, and the like. These buttons may include, but are not limited to: a home button, volume buttons, start button, and lock button.
  • Sensor assembly 814 includes one or more sensors for providing status assessments of various aspects of electronic device 800 .
  • the sensor component 814 can detect the open/closed state of the electronic device 800, the relative positioning of components, such as the display and the keypad of the electronic device 800, the sensor component 814 can also detect the electronic device 800 or a Changes in position of components, presence or absence of user contact with electronic device 800 , electronic device 800 orientation or acceleration/deceleration and temperature changes in electronic device 800 .
  • Sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects in the absence of any physical contact.
  • the sensor assembly 814 may also include an optical sensor, such as a Complementary Metal Oxide Semiconductor (CMOS) or Charge-coupled Device (CCD) image sensor, for use in imaging applications.
  • CMOS Complementary Metal Oxide Semiconductor
  • CCD Charge-coupled Device
  • the sensor component 814 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor or a temperature sensor.
  • the communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices.
  • the electronic device 800 can access a wireless network based on a communication standard, such as a wireless network (Wireless Fidelity, WiFi), a second-generation mobile communication technology (2nd-Generation, 2G) or a third-generation mobile communication technology (3rd-Generation, 3G) , or a combination of them.
  • a communication standard such as a wireless network (Wireless Fidelity, WiFi), a second-generation mobile communication technology (2nd-Generation, 2G) or a third-generation mobile communication technology (3rd-Generation, 3G) , or a combination of them.
  • the communication component 816 receives broadcast signals or broadcast related information from an external broadcast management system via a broadcast channel.
  • the communication component 816 also includes a Near Field Communication (NFC) module to facilitate short-range communication.
  • NFC Near Field Communication
  • the NFC module can be based on Radio Frequency Identification (RFID) technology, Infrared Data Association (Infrared Data Association, IrDA) technology, Ultra Wide Band (Ultra Wide Band, UWB) technology, Bluetooth (Blue Tooth, BT) technology and other techniques to achieve.
  • RFID Radio Frequency Identification
  • IrDA Infrared Data Association
  • UWB Ultra Wide Band
  • Bluetooth Bluetooth
  • the electronic device 800 may be implemented by one or more application-specific integrated circuits (Application Specific Integrated Circuit, ASIC), digital signal processors (Digital Signal Process, DSP), digital signal processing devices (Digital Signal Process Device , DSPD), Programmable Logic Device (Programmable Logic Device, PLD), Field Programmable Gate Array (Field Programmable Gate Array, FPGA), controller, microcontroller, microprocessor or other electronic components to implement the above method.
  • ASIC Application Specific Integrated Circuit
  • DSP Digital Signal Process
  • DSPD digital signal processing devices
  • PLD Programmable Logic Device
  • FPGA Field Programmable Gate Array
  • controller microcontroller, microprocessor or other electronic components to implement the above method.
  • a non-volatile computer-readable storage medium such as the memory 804 including computer program instructions, which can be executed by the processor 820 of the electronic device 800 to implement the above method.
  • the present disclosure can be a system, method and/or computer program product.
  • a computer program product may include a computer readable storage medium having computer readable program instructions thereon for causing a processor to implement various aspects of the present disclosure.
  • a computer readable storage medium may be a tangible device that can retain and store instructions for use by an instruction execution device.
  • a computer readable storage medium may be, for example, but is not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing.
  • Computer-readable storage media include: portable computer disks, hard disks, Random Access Memory (RAM), Read-Only Memory (ROM), computer Erasable programmable read-only memory (EPROM or flash memory), static random-access memory (Static Random-Access Memory, SRAM), portable compact disk read-only memory (Compact Disc Read-Only Memory, CD-ROM), digital multifunction Disk (Digital Video Disc, DVD), memory stick, floppy disk, mechanically encoded device, such as a punched card with instructions stored thereon or a raised structure in a groove, and any suitable combination of the foregoing.
  • RAM Random Access Memory
  • ROM Read-Only Memory
  • EPROM or flash memory computer Erasable programmable read-only memory
  • SRAM static random-access memory
  • SRAM Static Random-Access Memory
  • portable compact disk read-only memory Compact Disc Read-Only Memory
  • CD-ROM Compact Disc Read-Only Memory
  • DVD digital multifunction Disk
  • memory stick floppy disk
  • mechanically encoded device
  • computer-readable storage media are not to be construed as transient signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., pulses of light through fiber optic cables), or transmitted electrical signals.
  • Computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to a respective computing/processing device, or downloaded to an external computer or external storage device over a network, such as the Internet, a local area network, a wide area network, and/or a wireless network.
  • the network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and/or edge servers.
  • a network adapter card or a network interface in each computing/processing device receives computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in each computing/processing device .
  • Computer program instructions for performing the operations of the present disclosure may be assembly instructions, Instruction Set Architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or in the form of one or more source or object code written in any combination of programming languages, including object-oriented programming languages—such as Smalltalk, C++, etc., and conventional procedural programming languages—such as the “C” language or similar programming languages.
  • Computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server implement.
  • the remote computer can be connected to the user's computer through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or it can be connected to an external computer such as use an Internet service provider to connect via the Internet).
  • electronic circuits such as programmable logic circuits, field programmable gate arrays (FPGAs) or programmable logic arrays (ProgrammableLogicArrays, PLAs), are personalized and customized by using state information of computer-readable program instructions. Circuitry may execute computer readable program instructions to implement various aspects of the present disclosure.
  • These computer-readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine such that when executed by the processor of the computer or other programmable data processing apparatus , producing an apparatus for realizing the functions/actions specified in one or more blocks in the flowchart and/or block diagram.
  • These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause computers, programmable data processing devices and/or other devices to work in a specific way, so that the computer-readable medium storing instructions includes An article of manufacture comprising instructions for implementing various aspects of the functions/acts specified in one or more blocks in flowcharts and/or block diagrams.
  • each block in a flowchart or block diagram may represent a module, a portion of a program segment, or an instruction that includes one or more Executable instructions.
  • the functions noted in the block may occur out of the order noted in the figures. For example, two blocks in succession may, in fact, be executed substantially concurrently, or they may sometimes be executed in the reverse order, depending upon the functionality involved.
  • each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations can be implemented by a dedicated hardware-based system that performs the specified function or action , or may be implemented by a combination of dedicated hardware and computer instructions.
  • the computer program product can be specifically realized by means of hardware, software or a combination thereof.
  • the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (Software Development Kit, SDK) etc. Wait.
  • a software development kit Software Development Kit, SDK
  • the embodiment of the present disclosure discloses a map construction method and device, an electronic device, a storage medium, and a computer program product, wherein the map construction method includes: obtaining a scene image and a depth image of a target scene collected by a mobile device, and obtaining Relative pose at the current moment; according to the scene image and the depth image, determine the first spatial feature point of the target scene at the previous moment and the second spatial feature point at the current moment, and the timing of the previous moment is before the current moment , the first spatial feature point matches the second spatial feature point; according to the first spatial feature point and the second spatial feature point, the relative pose is optimized to obtain the positioning information of the mobile device at the current moment; according to the positioning The information and the depth image determine the grid map corresponding to the target scene, and the grid map is used to indicate the drivable area in the target scene, so that the mobile device can perform navigation operations according to the grid map.
  • the embodiment of the present disclosure utilizes the spatial feature points determined by the scene image and the depth image to optimize the relative pose of the mobile device at the current moment, so that higher-precision positioning information can be obtained; furthermore, the higher-precision positioning information and depth
  • the grid map determined by the image can accurately indicate the feasible area in the target scene, which can not only meet the needs of navigation operations such as path planning and obstacle analysis, but also effectively improve the efficiency of map construction and help mobile devices to perform reasonable and efficient tasks. Autonomous navigation operations.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Software Systems (AREA)
  • Geometry (AREA)
  • Remote Sensing (AREA)
  • Computer Graphics (AREA)
  • Multimedia (AREA)
  • Radar, Positioning & Navigation (AREA)
  • Computer Hardware Design (AREA)
  • General Engineering & Computer Science (AREA)
  • Navigation (AREA)

Abstract

本公开涉及一种地图构建方法及装置、电子设备、存储介质和计算机程序产品。其中，方法包括：获取移动设备采集的目标场景的场景图像以及深度图像，并获取移动设备在当前时刻下的相对位姿（S11）；根据场景图像与深度图像，确定目标场景在先时刻的第一空间特征点以及当前时刻的第二空间特征点，在先时刻的时序在当前时刻之前，第一空间特征点与第二空间特征点相匹配（S12）；根据第一空间特征点与第二空间特征点，对相对位姿进行优化，得到移动设备在当前时刻下的定位信息（S13）；根据定位信息以及深度图像，确定目标场景对应的栅格地图（S14），栅格地图用于指示目标场景中的可行驶区域，以使移动设备根据栅格地图进行导航作业。

Description

地图构建方法及装置、电子设备、存储介质和计算机程序产品
相关申请的交叉引用
本公开实施例基于申请号为202110678480.7、申请日为2021年06月18日的中国专利申请提出，并要求该中国专利申请的优先权，该中国专利申请的全部内容在此引入本公开作为参考。
技术领域
本公开实施例涉及但不限于计算机技术领域，尤其涉及一种地图构建方法及装置、电子设备、存储介质和计算机程序产品。
背景技术
扫地机器人在室内清扫中运用得越来越多，目前，扫地机器人通常可利用自身设置的传感器进行自定位、建图等，以实现自主导航。其中，自主导航可理解为，利用构建的地图以及自定位信息提前进行路径规划、障碍物分析等，使智能扫地机器人进行移动清扫。
发明内容
本公开至少提供一种地图构建方法及装置、电子设备、存储介质和计算机程序产品。
本公开实施例提供了一种地图构建方法，包括：获取移动设备采集的目标场景的场景图像以及深度图像，并获取所述移动设备在当前时刻下的相对位姿；根据所述场景图像与所述深度图像，确定所述目标场景在先时刻的第一空间特征点以及当前时刻的第二空间特征点，所述在先时刻的时序在所述当前时刻之前，所述第一空间特征点与所述第二空间特征点相匹配；根据所述第一空间特征点与所述第二空间特征点，对所述相对位姿进行优化，得到所述移动设备在当前时刻下的定位信息；根据所述定位信息以及所述深度图像，确定所述目标场景对应的栅格地图，所述栅格地图用于指示所述目标场景中的可行驶区域，以使所述移动设备根据所述栅格地图进行导航作业。
在一种可能的实现方式中，所述场景图像是由所述移动设备上设置的图像采集组件采集的，所述深度图像是由所述移动设备上设置的深度传感组件采集的，其中，根据所述场景图像与所述深度图像，确定所述目标场景在先时刻的第一空间特征点以及当前时刻的第二空间特征点，包括：对在先时刻采集的场景图像进行特征点提取，得到第一图像特征点；根据所述第一图像特征点，确定当前时刻采集的场景图像中与所述第一图像特征点匹配的第二图像特征点；根据所述场景图像与所述深度图像之间的映射关系，确定与所述第一图像特征点对应的第一深度值，以及与所述第二图像特征点对应的第二深度值；根据所述第一图像特征点、所述第一深度值以及所述图像采集组件的相机内参，确定所述第一空间特征点；根据所述第二图像特征点、所述第二深度值以及所述图像采集组件的相机内参，确定所述第二空间特征点。
在一种可能的实现方式中，所述相对位姿为多个，所述根据所述第一空间特征点与所述第二空间特征点，对所述相对位姿进行优化，得到所述移动设备在当前时刻下的定位信息，包括：针对任一相对位姿，根据所述相对位姿，将所述第一空间特征点变换至所述当前时刻下的第三空间特征点；根据所述第二空间特征点与所述第三空间特征点之间的第一误差，和/或所述相对位姿与其它相对位姿之间的第二误差，优化所述相对位姿，得到优化后的相对位姿，其中，所述其它相对位姿包括多个相对位姿中除所述相对位姿以外的相对位姿；根据所述优化后的相对位姿，确定所述移动设备在当前时刻下的定位信息；其中，所述多个相对位姿包括所述移动设备上设置的惯性传感组件测量的相对位姿，以及所述移动设备上设置的轮式里程计测量的相对位姿。
在一种可能的实现方式中，根据所述定位信息以及所述深度图像，确定所述目标场景对应的栅格地图，包括：确定所述目标场景对应的二维平面，并对所述二维平面进行栅格划分，得 到栅格平面；根据所述定位信息以及所述深度图像，确定所述目标场景中障碍物的位置信息；根据所述障碍物的位置信息，确定并标记所述障碍物在所述栅格平面中的所属栅格，得到所述目标场景对应的栅格地图。
在一种可能的实现方式中，所述根据所述定位信息以及所述深度图像，确定所述目标场景对应的栅格地图，包括：根据所述定位信息以及所述深度图像，确定所述目标场景对应的初始栅格地图；对所述场景图像进行识别，确定所述目标场景的场景类别以及所述目标场景中障碍物的障碍物类别；根据所述场景类别、所述障碍物类别以及所述初始栅格地图，生成所述栅格地图。
在一种可能的实现方式中，所述方法还包括：根据所述目标场景中不可行驶区域对应的障碍物类别，对所述栅格地图中表征所述不可行驶区域的栅格设置调整系数以及预设常数，其中，所述调整系数用于指示所述预设常数的衰减速度或增长速度；根据所述调整系数以及所述预设常数，更新所述栅格地图，得到更新后的栅格地图，以使所述移动设备根据所述更新后的栅格地图进行导航作业。
在一种可能的实现方式中，根据所述调整系数以及所述预设常数，更新所述栅格地图，得到更新后的栅格地图，包括：针对所述栅格地图中表征所述不可行驶区域的任一栅格，根据所述栅格的调整系数指示的衰减速度，递减所述栅格的预设常数，或者，根据所述栅格的调整系数指示的增长速度，递增所述栅格的预设常数；在所述预设常数递减至小于第一预设阈值，或所述预设常数递增至大于第二预设阈值的情况下，将所述栅格设置为可行驶区域，得到更新后的栅格地图。
在一种可能的实现方式中，所述相对位姿包括：所述移动设备在当前时刻相对于在先时刻的相对位置以及航向角；所述移动设备的图像采集组件、深度传感组件以及惯性传感组件组成信息采集模组，所述信息采集模组设置在所述移动设备的前端，所述信息采集模组的朝向与所述移动设备的移动方向一致；所述移动设备的轮式里程计设置在所述移动设备的底端。
本公开实施例提供了一种地图构建装置，包括：获取模块，被配置为获取移动设备采集的目标场景的场景图像以及深度图像，并获取所述移动设备在当前时刻下的相对位姿；确定模块，被配置为根据所述场景图像与所述深度图像，确定所述目标场景在先时刻的第一空间特征点以及当前时刻的第二空间特征点，所述在先时刻的时序在所述当前时刻之前，所述第一空间特征点与所述第二空间特征点相匹配；优化模块，被配置为根据所述第一空间特征点与所述第二空间特征点，对所述相对位姿进行优化，得到所述移动设备在当前时刻下的定位信息；构建模块，被配置为根据所述定位信息以及所述深度图像，确定所述目标场景对应的栅格地图，所述栅格地图用于指示所述目标场景中的可行驶区域，以使所述移动设备根据所述栅格地图进行导航作业。
在一种可能的实现方式中，所述场景图像是由所述移动设备上设置的图像采集组件采集的，所述深度图像是由所述移动设备上设置的深度传感组件采集的，其中，确定模块，包括：提取子模块，被配置为对在先时刻采集的场景图像进行特征点提取，得到第一图像特征点；匹配子模块，被配置为根据所述第一图像特征点，确定当前时刻采集的场景图像中与所述第一图像特征点匹配的第二图像特征点；映射子模块，被配置为根据所述场景图像与所述深度图像之间的映射关系，确定与所述第一图像特征点对应的第一深度值，以及与所述第二图像特征点对应的第二深度值；第一确定子模块，被配置为根据所述第一图像特征点、所述第一深度值以及所述图像采集组件的相机内参，确定所述第一空间特征点；第二确定子模块，被配置为根据所述第二图像特征点、所述第二深度值以及所述图像采集组件的相机内参，确定所述第二空间特征点。
在一种可能的实现方式中，所述相对位姿为多个，所述优化模块，包括：变换子模块，被配置为针对任一相对位姿，根据所述相对位姿，将所述第一空间特征点变换至所述当前时刻下的第三空间特征点；优化子模块，被配置为根据所述第二空间特征点与所述第三空间特征点之间的第一误差，和/或所述相对位姿与其它相对位姿之间的第二误差，优化所述相对位姿，得到优化后的相对位姿，其中，所述其它相对位姿包括多个相对位姿中除所述相对位姿以外的相对 位姿；定位信息确定子模块，被配置为根据所述优化后的相对位姿，确定所述移动设备在当前时刻下的定位信息；其中，所述多个相对位姿包括所述移动设备上设置的惯性传感组件测量的相对位姿，以及所述移动设备上设置的轮式里程计测量的相对位姿。
在一种可能的实现方式中，构建模块，包括：栅格平面确定子模块，被配置为确定所述目标场景对应的二维平面，并对所述二维平面进行栅格划分，得到栅格平面；位置信息确定子模块，被配置为根据所述定位信息以及所述深度图像，确定所述目标场景中障碍物的位置信息；第一栅格地图确定子模块，被配置为根据所述障碍物的位置信息，确定并标记所述障碍物在所述栅格平面中的所属栅格，得到所述目标场景对应的栅格地图。
在一种可能的实现方式中，所述构建模块，包括：初始栅格地图确定子模块，被配置为根据所述定位信息以及所述深度图像，确定所述目标场景对应的初始栅格地图；识别子模块，被配置为对所述场景图像进行识别，确定所述目标场景的场景类别以及所述目标场景中障碍物的障碍物类别；第二栅格地图确定子模块，被配置为根据所述场景类别、所述障碍物类别以及所述初始栅格地图，生成所述栅格地图。
在一种可能的实现方式中，所述装置还包括：设置模块，被配置为根据所述目标场景中不可行驶区域对应的障碍物类别，对所述栅格地图中表征所述不可行驶区域的栅格设置调整系数以及预设常数，其中，所述调整系数用于指示所述预设常数的衰减速度或增长速度；更新模块，被配置为根据所述调整系数以及所述预设常数，更新所述栅格地图，得到更新后的栅格地图，以使所述移动设备根据所述更新后的栅格地图进行导航作业。
在一种可能的实现方式中，更新模块，包括：调整子模块，被配置为针对所述栅格地图中表征所述不可行驶区域的任一栅格，根据所述栅格的调整系数指示的衰减速度，递减所述栅格的预设常数，或者，根据所述栅格的调整系数指示的增长速度，递增所述栅格的预设常数；更新子模块，被配置为在所述预设常数递减至小于第一预设阈值，或所述预设常数递增至大于第二预设阈值的情况下，将所述栅格设置为可行驶区域，得到更新后的栅格地图。
在一种可能的实现方式中，所述相对位姿包括：所述移动设备在当前时刻相对于在先时刻的相对位置以及航向角；所述移动设备的图像采集组件、深度传感组件以及惯性传感组件组成信息采集模组，所述信息采集模组设置在所述移动设备的前端，所述信息采集模组的朝向与所述移动设备的移动方向一致；所述移动设备的轮式里程计设置在所述移动设备的底端。
本公开实施例提供了一种电子设备，包括：处理器；用于存储处理器可执行指令的存储器；其中，所述处理器被配置为调用所述存储器存储的指令，以执行上述方法。
本公开实施例提供了一种计算机可读存储介质，其上存储有计算机程序指令，所述计算机程序指令被处理器执行时实现上述方法。
本公开实施例还提供一种计算机程序产品，包括计算机可读代码，或者承载有计算机可读代码的非易失性计算机可读存储介质，当所述计算机可读代码在电子设备的处理器中运行时，所述电子设备中的处理器执行上述地图构建方法的步骤。
在本公开实施例中，能够利用场景图像与深度图像确定出的空间特征点，对移动设备当前时刻下的相对位姿进行优化，从而可以得到较高精度的定位信息；进而利用较高精度的定位信息以及深度图像确定出的栅格地图，能够精准地指示目标场景中的可行使区域，不仅能够满足路径规划、障碍物分析等导航作业需求，还有效提高地图构建效率，有助于移动设备执行合理高效的自主导航作业。
应当理解的是，以上的一般描述和后文的细节描述仅是示例性和解释性的，而非限制本公开。根据下面参考附图对示例性实施例的详细说明，本公开的其它特征及方面将变得清楚。
附图说明
此处的附图被并入说明书中并构成本说明书的一部分，这些附图示出了符合本公开的实施例，并与说明书一起用于说明本公开的技术方案。
图1a为本公开实施例提供的一种移动设备侧视图的示意图。
图1b为本公开实施例提供的一种移动设备俯视图的示意图。
图1c为本公开实施例提供的一种地图构建方法的示意图。
图2为本公开实施例提供的一种地图构建方法的实现流程示意图。
图3为本公开实施例提供的一种平面坐标系的示意图。
图4为本公开实施例提供的一种栅格平面的平面坐标系的示意图。
图5为本公开实施例提供的一种地图构建装置的组成结构示意图。
图6为本公开实施例提供的一种电子设备的组成结构示意图。
具体实施方式
以下将参考附图详细说明本公开的各种示例性实施例、特征和方面。附图中相同的附图标记表示功能相同或相似的元件。尽管在附图中示出了实施例的各种方面，但是除非特别指出，不必按比例绘制附图。
在这里专用的词“示例性”意为“用作例子、实施例或说明性”。这里作为“示例性”所说明的任何实施例不必解释为优于或好于其它实施例。
本文中术语“和/或”，仅仅是一种描述关联对象的关联关系，表示可以存在三种关系，例如，A和/或B，可以表示：单独存在A，同时存在A和B，单独存在B这三种情况。另外，本文中术语“至少一种”表示多种中的任意一种或多种中的至少两种的任意组合，例如，包括A、B、C中的至少一种，可以表示包括从A、B和C构成的集合中选择的任意一个或多个元素。
另外，为了更好地说明本公开，在下文的具体实施方式中给出了众多的具体细节。本领域技术人员应当理解，没有某些具体细节，本公开同样可以实施。在一些实例中，对于本领域技术人员熟知的方法、手段、元件和电路未作详细描述，以便于凸显本公开的主旨。
相关技术中，扫地机器人上通常设置有多种传感器，如激光雷达、单目相机、惯性测量单元(Inertial Measurement Unit，IMU)、轮式里程计等，以综合利用各个传感器进行自定位、建图等，以实现自主导航。
然而相关技术中，利用惯性测量单元、轮式里程计等进行自定位的精度不高，利用单目相机采集的图像数据结合三维地图构建技术，构建三维地图的实时性和效率不高。
针对以上方案所存在的缺陷，均是发明人在经过实践并仔细研究后得出的结果，因此，上述问题的发现过程以及下文中本公开实施例针对上述问题所提出的解决方案，都应该是发明人在本公开过程中对本公开做出的贡献。
图1a为本公开实施例提供的一种移动设备侧视图的示意图，图1b为本公开实施例提供的一种移动设备俯视图的示意图。如图1a和1b所示，移动设备上可设置深度摄像头-惯性测量单元(Red Green Blue-Depth-Inertial Measurement Unit，RGBD-IMU)模组11和轮式里程计12，其中，RGBD-IMU模组包括RGB相机、深度(Depth)相机、惯性测量单元。
其中，RGB相机用于采集移动设备所在场景的场景图像；深度相机用于采集移动设备所在场景的深度图像；惯性测量单元和轮式里程计用于测量移动设备的相对位姿。通过该方式，不仅可实现移动设备的自主导航，同时可以满足移动设备的成本控制需求。
如图1a和图1b所示，RGBD-IMU模组11可设置在移动设备的前端，该RGBD-IMU模组11的朝向与移动设备的移动方向一致，从而便于RGB相机采集场景图像、深度相机采集深度图像；以及便于惯性测量单元测量移动设备的相对位姿，也可便于实现场景图像、深度图像与相对位姿之间的时间同步；轮式里程计12可设置在移动设备的底端，以便于轮式里程计测量移动设备的相对位姿。
其中，可综合利用RGB相机采集的场景图像、深度相机采集的深度图像以及惯性测量单元和轮式里程计测量的相对位姿，实现移动设备针对所在场景的自定位、障碍物分析、地图构建、路径规划等作业。
图1c为本公开实施例提供的一种地图构建方法的示意图，如图1c所示，所述方法可应用于上述移动设备，所述方法包括：
对移动设备上设置的RGBD-IMU模组11采集的RGB图像进行场景识别，得到移动设备所处场景的语义信息，语义信息包括场景的场景类别和场景中障碍物的障碍物类别；其中，例如可通过神经网络对RGB图像进行场景识别，对于神经网络的网络结构、网络类型以及训练方式，本公开实施例不作限制；
根据RGBD-IMU模组11采集的深度图像，进行可行使区域分析(也即障碍物分析)，得到所处场景的可行驶区域(或障碍物位置)；可理解的是，深度图像包含场景中障碍物的深度信息，根据障碍物的深度信息可得到场景中障碍物的位置信息，进而根据障碍物的位置信息可得到场景中的可行使区域，也即实现可行使区域分析；
根据RGBD-IMU模组11采集的RGB图像、深度图像、IMU信息(即惯性测量单元测量的相对位姿)，以及移动设备上设置的轮式里程计12测量的里程计信息(即轮式里程计测量的相对位姿)，进行自定位和建图，得到自定位信息和所处场景的栅格地图；
将上述语义信息、所处场景的可行驶区域(或障碍物位置)、自定位信息以及所处场景的栅格地图进行信息融合，得到带语义的栅格地图。
在本公开实施例中，能够融合RGBD-IMU模组和轮式里程计采集的各种信息，构建带语义的栅格地图，使上述移动设备基于带语义的栅格地图进行更智能化地避障、路径规划等自主导航作业。
相关技术中，通常在移动设备上设置激光雷达，以采集场景的三维点云数据，但激光雷达的引入会使得移动设备(如智能扫地机器人)的机械结构上突出一定的高度，影响移动设备的可通过性(例如有些家具下面钻不进去或者容易被卡住)。在本公开实施例中，与含激光雷达的移动设备相比，由于未引入激光雷达，因此移动设备可以设计得更矮小，在所处场景中的通过性更高；而且深度相机采集的深度图像可以用于避开较为低矮的障碍物；还可利用朝前设置的RGB相机拍摄的图像，对所处场景进行语义分析，能够实现更智能化的自主导航作业。
应理解的是，地图构建是实现自主导航作业的一项关键技术。相关技术中通常是利用三维地图构建技术，例如运动恢复结构(Structure from motion，SFM)技术，根据场景图像，构建出场景的三维地图，并利用惯性测量单元和/或轮式里程计测量的相对位姿进行自定位；进而结合自定位信息与三维地图进行障碍物分析、路径规划等导航作业。
可理解的是，该种方式中，构建三维地图的所需的运算量和存储量较大，由于移动设备所处场景会实时变化，地图构建需要循环多次执行，对于一些处理器和存储器能力有限的移动设备，可能无法有效应对实时性较高的地图构建需求；以及，惯性测量单元和轮式里程计测量的相对位姿，会随着移动设备的持续运行，产生较大的累计误差，使得移动设备的自定位精度降低。
根据本公开实施例的地图构建方法，能够利用RGB相机采集的场景图像与深度相机采集的深度图像所确定出的空间特征点，对惯性测量单元和/或轮式里程计测量的相对位姿进行优化，有利于降低累计误差对自定位精度的影响，得到较高精度的定位信息；进而根据较高精度的定位信息与深度图像确定出的栅格地图，能够精准地指示目标场景中的可行驶区域，不仅能够满足障碍物分析、路径规划等导航作业需求，还有效提高地图构建效率，使移动设备执行合理高效的自主导航。
根据本公开实施例的地图构建方法，可由移动设备执行，移动设备可包括：智能扫地机器人、智能清洁机器人、智能割草机器人、智能送货运输机器人等各类移动机器人、自动驾驶车辆等需要自主导航的设备。通过该方式，能够使移动设备基于更高精度的定位信息以及栅格地图，进行更高效合理地自主导航作业。
根据本公开实施例的地图构建方法，还可由处理器(如移动设备的处理器)，调用存储器中存储的计算机可读指令的方式来实现。可理解的是，移动设备上设置的RGB相机、深度相机、惯性测量单元和/或轮式里程计等可与处理器进行通信，以将采集的场景图像、深度图像以及相对位姿等数据，发送至处理器和/或存储器中，从而实现本公开实施例的地图构建方法。
图2为本公开实施例提供的一种地图构建方法的实现流程示意图，如图2所示，所述地图 构建方法包括：
在步骤S11中，获取移动设备采集的目标场景的场景图像以及深度图像，并获取移动设备在当前时刻下的相对位姿。
移动设备上可设置有图像采集组件、深度传感组件、惯性传感组件和/或轮式里程计。目标场景的场景图像可是由移动设备上设置的图像采集组件采集的；深度图像是由所述移动设备上设置的深度传感组件采集的；相对位姿可是由移动设备上设置的惯性传感组件和/或轮式里程计测量的。
其中，图像采集组件可包括：单目RGB相机、双目RGB相机中的任意一种；深度传感组件包括深度相机，也可称为深度传感器、飞行时间(Time of flight，TOF)相机等；惯性传感组件包括惯性测量单元IMU，也可称为惯性传感器。
在一种可能的实现方式中，图像采集组件、深度传感组件以及惯性传感组件可组成信息采集模组，信息采集模组可设置在移动设备的前端，信息采集模组的朝向与移动设备的移动方向一致，轮式里程计可设置在所述移动设备的底端，以便于各个组件和轮式里程计各自采集相关信息，同时有利于降低移动设备的机械高度。
需要说明的是，以上图像采集组件、深度传感组件、惯性传感组件以及轮式里程计在移动设备上的设置方式，是本公开实施例提供的一种实现方式，实际上，本领域技术人员可根据实际需求，设计各个组件和轮式里程计在移动设备上的位置布局，对此本公开实施例不作限制。
在步骤S12中，根据场景图像与深度图像，确定目标场景在先时刻的第一空间特征点以及当前时刻的第二空间特征点，在先时刻的时序在当前时刻之前，第一空间特征点与第二空间特征点相匹配。
其中，空间特征点可理解为空间中的三维特征点，也即目标场景中的三维特征点。可理解的是，深度图像中可包含目标场景中物体的深度信息，利用深度图像中物体的深度信息与场景图像中物体的像素位置信息，可得到场景中物体的三维特征点的位置信息。
可理解的是，场景图像与深度图像可以是实时采集的数据，针对任一时刻采集的场景图像与深度图像，可利用该时刻采集的场景图像与深度图像，得到该时刻下的三维点云，基于此，可对当前时刻的三维点云与在先时刻的三维点云进行特征点匹配，得到两三维点云中相匹配的三维特征点对，也即得到在先时刻的第一空间特征点，以及当前时刻的第二空间特征点。
在一种可能的实现方式中，例如可通过公式(1)实现针对任一时刻采集的场景图像与深度图像，利用该时刻采集的场景图像与深度图像，得到该时刻下的三维点云。
Figure PCTCN2021122715-appb-000001
其中，(M x，M y，M z)代表三维点云的三维坐标，(m x、m y)分别代表场景图像中像素点的像素坐标，d代表深度图像中与像素点对应的深度值，f x、f y分别代表图像采集部件在成像平面的横轴x和纵轴y方向上的焦距。
在一种可能的实现方式中，对当前时刻的三维点云与在先时刻的三维点云进行特征点匹配，例如可包括：提取当前时刻的三维点云中的多个第一空间特征点，以及在先时刻的三维点云中的多个第二空间特征点；针对任一第一空间特征点，计算该第一空间特征点与各第二空间特征点之间的距离；将距离最小所对应的第二空间特征点作为与第一空间特征点相匹配的空间特征点。
应理解的是，可采用本领域已知的特征点提取方式，例如，点特征直方图(Point Feature Histogram，PFH)、简单点特征直方图(Simple Point Feature Histograms，SPFH)实现对各三维点云进行特征点提取，得到各三维点云的空间特征点，对此本公开实施例不作限制。
其中，第一空间特征点与第二空间特征点相匹配，可理解为，在先时刻下的第一空间特征点与当前时刻下的第二空间特征点，共同指示目标场景中同一物体上的同一点。应理解的是，不同时刻下移动设备的位姿不同，第一空间特征点与第二空间特征点是基于移动设备不同位姿下所确定出的空间特征点。
在步骤S13中，根据第一空间特征点与第二空间特征点，对相对位姿进行优化，得到移动设备在当前时刻下的定位信息。
其中，相对位姿可包括：移动设备在当前时刻相对于在先时刻的相对位置以及航向角。也即，相对位姿，可理解为当前时刻下移动设备的位置与在先时刻下移动设备的位置之间的相对值，以及当前时刻下的航向角。
如上所述，在先时刻的第一空间特征点与当前时刻的第二空间特征点相匹配，那么理论上，基于当前时刻下的相对位姿，对第一空间特征点进行映射变换所得到的空间特征点，应该与第二空间特征点重合，而由于惯性传感组件和/或轮式里程计测量的相对位姿，与移动设备的实际位姿之间存在误差，因此，对第一空间特征点进行映射变换所得到的空间特征点，与第二空间特征点之前也存在误差。
在一种可能的实现方式中，根据第一空间特征点与第二空间特征点，对相对位姿进行优化，得到移动设备在当前时刻下的定位信息，可包括：根据相对位姿对第一空间特征点进行映射变换得到第三空间特征点；根据第三空间特征点与第二空间特征点之前的误差，对相对位姿进行优化，得到移动设备在当前时刻下的定位信息。
可理解的是，通过步骤S13可得到任意时刻下的相对位姿，利用任意时刻下的相对位姿，可得到移动设备相对于初始时刻(例如扫地机器人开始作业的时刻)的位姿(即位置与航向角)，或者说相对于初始位置(例如目标场景的空间坐标系的原点)的位姿，也即得到移动设备在目标场景中的实际位姿。其中，考虑到移动设备通常在地面移动，移动设备的位置可用二维坐标表征，移动设备的姿态可用航向角表征，航向角可理解为移动设备基于水平方向的移动方位。
图3为本公开实施例提供的一种的平面坐标系的示意图。为便于理解，以图3示出的平面坐标系，举例说明确定移动设备的定位信息的一种方式，如图3所示，假设坐标系的原点为初始位置，t1时刻下的相对位姿包括航向角θ 1和相对位置(x 1，y 1)，t2时刻下的相对位姿包括航向角θ 2和相对位置(x 2，y 2)，t1时刻下移动设备的实际位姿也即该t 1时刻下的相对位姿，t2时刻下移动设备的实际位姿包括：位置(x 1+x 2，y 1+y 2)和航向角θ 2。
应理解的是，也可以用三维坐标表征移动设备的位置，用三自由度的姿态角表征移动设备的姿态，具体可根据目标场景的实际情况决定，对此本公开实施例不作限制，例如，针对室内的智能扫地机器人，地面较为平整，二维坐标与航向角作为移动设备的位姿即可满足自定位需求；针对某些室外的移动机器人，地面起伏较大，可能需要三维坐标与三自由度的姿态角来表征移动设备的位姿。
在步骤S14中，根据定位信息以及深度图像，确定目标场景对应的栅格地图，栅格地图用于指示目标场景中的可行驶区域，以使移动设备根据栅格地图进行导航作业。
其中，对于目标场景中物体，大部分可理解为移动设备的障碍物，障碍物在目标场景中所占的区域，可称为不可行驶区域。相应地，目标场景中除不可行驶区域以外的区域，可称为移动设备的可行驶区域，也即移动设备可以在可行驶区域内移动。
可理解的是，目标场景例如客厅、卧室等，目标场景的面积、形状(如室内面积、形状)通常是固定的。在一种可能的实现方式中，可在移动设备首次进入目标场景的情况下，对目标场景进行全方位扫描，例如采集全景的深度图像，利用全景的深度图像，确定目标场景的整体轮廓(例如室内的整体轮廓)或者说目标场景的区域范围；进而可利用确定出的整体轮廓，得到目标场景对应的二维平面，应理解的是，二维平面的面积与形状可与目标场景一致。
其中，可按照一定的栅格尺寸，对目标场景的二维平面进行栅格划分，得到栅格平面。应理解的是，栅格尺寸可根据实际需求设置，栅格平面中栅格的粒度与数量，与栅格尺寸以及目标场景的面积相关，对此本公开实施例不作限制。
如上所述，深度图像中包含目标场景中物体的深度信息，该深度信息也可理解为目标场景中物体与移动设备之间的距离。那么根据移动设备的定位信息，与深度图像中包含的目标场景中障碍物的深度信息，可得到障碍物在目标场景中的位置信息；进而可利用障碍物的位置信息，在上述目标场景的栅格平面中，确定并标记障碍物的所属栅格，得到目标场景的栅格地图。
应理解的是，深度图像中可包含各类高度的障碍物的深度信息，通过该方式，在基于深度图像生成的栅格地图进行导航作业时，能够增加移动设备的避障能力，尤其是针对低矮障碍物(如掉到地面上的袜子)的避障能力。
考虑到，目标场景中的障碍物可是会发生变化的，例如，随时移动的人和宠物，室内桌椅板凳的位置调整等，因此，目标场景的栅格地图是会发生变化的。基于此，可循环执行上述步骤S11至S14的地图构建方法，从而实时更新目标场景的栅格地图。
在一种可能的实现方式中，可将基于栅格地图进行导航作业的导航逻辑部署在移动设备上，也即可在移动设备上实现根据栅格地图规划移动路径、分析障碍物等导航作业；还可将栅格地图发送至后端服务器，也即将导航逻辑部署在服务器中，在服务器实现根据栅格地图规划移动路径、分析障碍物，并将规划结果和分析结果返回给移动设备，以控制移动设备移动，对此本公开实施例不作限制。
在本公开实施例中，能够利用场景图像与深度图像确定出的空间特征点，对移动设备当前时刻下的相对位姿进行优化，从而可以得到较高精度的定位信息；进而利用较高精度的定位信息以及深度图像确定出的栅格地图，能够精准地指示目标场景中的可行使区域，同时由于栅格地图是二维地图，针对在平面移动的移动设备，不仅能够满足路径规划、障碍物分析等导航作业需求，还有效降低地图构建所需的运算量和存储量，提高地图构建效率，使移动设备执行合理高效的自主导航作业，且有利于降低移动设备的机械高度，增大移动设备的可移动区域和可通过性。
可理解的是，栅格地图还可是三维地图。如上所述，可在移动设备首次进入目标场景时，对目标场景进行全方位扫描，例如采集全景的深度图像，利用全景的深度图像，确定目标场景的整体轮廓(例如室内的整体轮廓)或者说目标场景的空间区域范围；进而可利用确定出的整体轮廓，得到目标场景对应的三维空间，应理解的是，三维空间的体积与形状可与目标场景一致。
其中，可按照一定的三维栅格尺寸(即长宽高)，对目标场景的三维空间进行栅格划分，得到三维栅格空间。应理解的是，三维栅格尺寸可根据实际需求设置，三维栅格空间中三维栅格的粒度与数量，与三维栅格尺寸以及目标场景的体积相关，对此本公开实施例不作限制。
如上所述，深度图像中包含目标场景中物体的深度信息，该深度信息也可理解为目标场景中物体与移动设备之间的距离。在一种可能的实现方式中，在步骤S14中，根据定位信息以及深度图像，确定目标场景对应的栅格地图，可包括：根据移动设备的定位信息，与深度图像中包含的目标场景中障碍物的深度信息，可得到障碍物在目标场景中的三维位置信息；进而利用障碍物的三维位置信息，在上述目标场景的三维栅格空间中，确定并标记障碍物的所属栅格，得到目标场景的栅格地图。
在本公开实施例中，三维的栅格地图能够指示场景中三维立体的可行驶区域，从而使移动设备能够基于三维立体的可行驶区域，执行更合理的导航作业，例如，可规划更为合理和准确的移动路径。
如上所述，场景图像是由移动设备上设置的图像采集组件采集的，深度图像是由移动设备上设置的深度传感组件采集的，在一种可能的实现方式中，在步骤S12中，根据场景图像与深度图像，确定目标场景在先时刻的第一空间特征点以及当前时刻的第二空间特征点，可包括：
步骤S121：对在先时刻采集的场景图像进行特征点提取，得到第一图像特征点。
其中，可采用本领域已知的图像特征点提取技术，例如，尺度不变特征变换(Scale-invariant feature transform，SIFT)算法、快速特征点提取和描述(Oriented FAST and Rotated BRIEF，ORB)算法等，实现对在先时刻采集的场景图像进行特征点提取，对此本公开实施例不作限制。
步骤S122：根据第一图像特征点，确定当前时刻采集的场景图像中与第一图像特征点匹配的第二图像特征点。
其中，可采用本领域已知的跟踪算法，例如，特征点跟踪算法(即利用特征点的相似度进行跟踪)、光流跟踪算法(即利用特征点的运动轨迹进行跟踪)，实现根据第一图像特征点，确 定当前时刻采集的场景图像中与第一图像特征点匹配的第二图像特征点，对此本公开实施例不作限制。
步骤S123：根据场景图像与深度图像之间的映射关系，确定与第一图像特征点对应的第一深度值，以及与第二图像特征点对应的第二深度值。
应理解的是，图像采集组件(如单目RGB相机)与深度传感组件(如深度相机)，在移动设备上的位置是固定的，由此可知晓图像采集组件与深度传感组件之间的相对位置关系，基于该相对位置关系，可得到图像采集组件的图像坐标系(即场景图像)与深度传感组件的图像坐标系(及深度图像)之间的映射关系。
其中，映射关系可用映射矩阵表征，映射矩阵可包括旋转矩阵与平移矩阵，对此本公开实施例不作限制。
应理解的是，根据场景图像与深度图像之间的映射关系，可对任一时刻下采集的深度图像或场景图像进行映射变换，以将深度图像与场景图像变换至同一图像坐标系中，例如，将深度图像变换至场景图像对应的图像坐标系中；进而可在同一图像坐标系中，实现根据场景图像中任一图像特征点的位置，从对应的深度图像中确定出对应的深度值。
也即，实现根据在先时刻的场景图像中第一图像特征点的位置，从在先时刻的深度图像中确定出第一深度值，以及根据当前时刻的场景图像中第二图像特征点的位置，从当前时刻的深度图像中确定出第二深度值。
步骤S124：根据第一图像特征点、第一深度值以及图像采集组件的相机内参，确定第一空间特征点。
可知晓的是，相机内参可包括在横轴x和纵轴y方向上的焦距f x、f y。其中，在不考虑相机畸变与光心偏移基础上，可采用上述公式(1)实现根据第一图像特征点、第一深度值以及图像采集组件的相机内参，确定第一空间特征点，也即，将第一图像特征点的位置作为(m x、m y)，将第一深度值作为d，可得到第一空间特征点的三维坐标。
步骤S125：根据第二图像特征点、第二深度值以及图像采集组件的相机内参，确定第二空间特征点。
如上所述，相机内参可包括在横轴x和纵轴y方向上的焦距f x、f y。同样的，在不考虑相机畸变与光心偏移基础上，可采用上述公式(1)实现根据第二图像特征点、第二深度值以及图像采集组件的相机内参，确定第二空间特征点。
在本公开实施例中，能够高效地确定出场景图像与深度图像中相匹配的图像特征点，进而确定出对应匹配的第一空间特征点和第二空间特征点。
如上所述，移动设备可设置惯性传感组件和轮式里程计，惯性传感组件和轮式里程计各自可测量移动设备的相对位姿，由于惯性传感组件和轮式里程计各自测量的相对位姿，都会存在累计误差导而致定位精度不高的问题，在一种可能的实现方式中，可综合利用上述第一空间特征点与第二空间特征点、以及惯性传感组件和轮式里程计分别测量的两个相对位姿之间的误差，对其中任一相对位姿进行优化，得到更高精度的定位信息。
在一种可能的实现方式中，相对位姿可为多个，多个相对位姿可包括移动设备上设置的惯性传感组件测量的相对位姿，以及移动设备上设置的轮式里程计测量的相对位姿。
其中，可知晓的是，惯性传感组件(即惯性测量单元)中包括加速度计和陀螺仪，通过加速度及和陀螺仪，可测量移动设备的角速度和加速度，进而对角速度和加速度进行预积分，可计算出从在先时刻到当前时刻，移动设备的移动距离(也即相对位置)和移动方向(也即航向角)，得到惯性传感组件测量的相对位姿。
其中，可知晓的是，轮式里程计可测量移动设备的驱动轮的转动速度(即驱动轮的角速度、线速度)，进而根据驱动轮的转动速度和驱动轮的机械结构信息(如驱动轮的半径和轴距)，可以计算出从在先时刻到当前时刻，移动设备的移动距离(也即相对位置)和移动方向(也即航向角)，得到轮式里程计测量的相对位姿。
在一种可能的实现方式中，在步骤S13中，根据第一空间特征点与第二空间特征点，对相 对位姿进行优化，得到移动设备在当前时刻下的定位信息，包括：
步骤S131：针对任一相对位姿，根据相对位姿，将第一空间特征点变换至当前时刻下的第三空间特征点；
可理解的是，轮式里程计测量的相对位姿和惯性传感组件测量的相对位姿，均可表征当前时刻下移动设备的相对位姿，由此，可选择其中任一相对位姿作为优化对象，也即作为优化的初始值。例如，可选择轮式里程计测量的相对位姿作为优化对象，对此本公开实施例不作限制。
其中，根据相对位姿，将第一空间特征点变换至当前时刻下的第三空间特征点，可包括：根据相对位姿，对第一空间特征点进行映射变换，得到第三空间特征点。例如可通过公式(2)实现根据相对位姿对第一空间特征点进行映射变换。
X *＝RX+t      (2)；
其中，X代表第一空间特征点，X *代表第三空间特征点，R和t代表相对位姿，R代表旋转矩阵(即由航向角确定的矩阵)，t代表平移矩阵(即由相对位置确定的矩阵)。
步骤S132：根据第二空间特征点与第三空间特征点之间的第一误差，和/或相对位姿与其它相对位姿之间的第二误差，优化相对位姿，得到优化后的相对位姿，其中，其它相对位姿包括多个相对位姿中除该相对位姿以外的相对位姿。
可理解的是，若选择轮式里程计测量的相对位姿作为优化对象，则其它相对位姿可是惯性传感组件测量的相对位姿；相应地，若选择惯性传感组件测量的相对位姿作为优化对象，则其它相对位姿可是轮式里程计测量的相对位姿。
如上所述，在先时刻的第一空间特征点与当前时刻的第二空间特征点相匹配，那么理论上，根据当前时刻下的相对位姿，对第一空间特征点进行映射变换所得到的空间特征点，应该与第二空间特征点重合，而由于惯性传感组件和轮式里程计测量的相对位姿，与移动设备的实际位姿之间存在误差，由此，对第一空间特征点进行映射变换所得到的空间特征点，与第二空间特征点之前也存在误差。
可理解的是，由于轮式里程计与惯性传感组件测量相对位姿的原理不同、精度不同等，轮式里程计测量的相对位姿和惯性传感组件测量的相对位姿之间也存在误差。
在一种可能的实现方式中，可利用本领域已知的迭代优化算法，例如高斯牛顿算法、列文伯格-马夸尔特算法等，实现根据第一误差和/或第二误差，优化相对位姿，得到优化后的相对位姿。其中，根据第一误差和第二误差优化相对位姿，能够使优化后的相对位姿的精度更高。
其中，通过高斯牛顿算法优化相对位姿，例如包括：将相对位姿作为待优化的初始值；将相对位姿、第一误差和第二误差带入泰勒级数展开式，得到非线性回归模型；利用最小二乘法，求解非线性回归模型，得到修正系数；通过修正系数对初始值进行修正，得到修正值；可再将修正值作为待优化的初始值，按照上述过程迭代执行多轮，至修正值收敛，得到优化后的相对位姿。
应理解的是，采用不同的迭代优化算法，迭代优化过程可不同，本领域技术人员可选择任意已知的迭代优化算法，只要是根据第一误差和/或第二误差优化相对位姿，均在本公开的保护范围内。
步骤S133：根据优化后的相对位姿，确定移动设备在当前时刻下的定位信息。
其中，可参照上述本公开实施例中公开的定位信息的确定方式，例如图3示出的定位信息的确定方式，实现根据优化后的相对位姿，确定移动设备在当前时刻下的定位信息。
在本公开实施例中，能够综合利用第一空间特征点与第二空间特征点、以及惯性传感组件和轮式里程计分别测量的两个相对位姿之间的误差，对其中任一相对位姿进行优化，得到更高精度的定位信息。
在一种可能的实现方式中，在步骤S14中，根据定位信息以及深度图像，确定目标场景对应的栅格地图，包括：
步骤S141：确定目标场景对应的二维平面，并对二维平面进行栅格划分，得到栅格平面；
如上本公开实施例所述，可在移动设备首次进入目标场景时，对目标场景进行全方位扫描， 例如采集全景的深度图像，利用全景的深度图像，确定目标场景的整体轮廓(例如室内的整体轮廓)或者说目标场景的区域范围；进而可利用确定出的整体轮廓，得到目标场景对应的二维平面，应理解的是，二维平面的面积与形状可与目标场景一致。
其中，可按照一定的栅格尺寸，对目标场景的二维平面进行栅格划分，得到栅格平面。应理解的是，栅格尺寸可根据实际需求设置，栅格平面中栅格的粒度与数量，与栅格尺寸以及目标场景的面积相关，对此本公开实施例不作限制。
步骤S142：根据定位信息以及深度图像，确定目标场景中障碍物的位置信息；
如上所述，深度图像中包含目标场景中物体的深度信息，该深度信息也可理解为目标场景中物体与移动设备之间的距离。那么根据移动设备的定位信息，与深度图像中包含的目标场景中障碍物的深度信息，可得到障碍物在目标场景中的位置信息。
举例来说，图4为本公开实施例提供的栅格平面的平面坐标系的示意图。如图4所示，移动设备的定位信息用二维坐标(x,y)和航向角θ表征，障碍物的深度信息用d 0表征，障碍物的位置信息用(x 0,y 0)表征，可通过联立方程组(3)得到障碍物的位置信息。
Figure PCTCN2021122715-appb-000002
步骤S143：根据障碍物的位置信息，确定并标记障碍物在栅格平面中的所属栅格，得到目标场景对应的栅格地图。
可理解的是，栅格平面对应于目标场景，根据障碍物在目标场景中的位置信息，可得到障碍物在栅格平面中的所属栅格，进而可按照预设的标记方式，在该所属栅格中进行标记，如可标记数字、填充颜色等，以表征该栅格是障碍物的所属栅格，也代表该栅格指示的区域是不可行驶区域。其中，对于栅格地图中障碍物所属栅格的标记方式，本公开实施例不作限制。
在本公开实施例中，能够便捷有效地生成目标场景的栅格地图，并且生成的栅格地图，不仅能够满足路径规划、障碍物分析等导航作业需求，还有效降低地图构建所需的运算量和存储量，提高地图构建效率。
考虑到，目标场景包含多种障碍物，例如随时移动的人和宠物，不经常移动的家具等，不同目标场景中障碍物也不同，例如客体与卧室中的障碍物不同。为了使移动设备执行更智能的导航作业，可对目标场景中的障碍物进行识别，得到目标场景中障碍物的语义信息，进而可结合障碍物的语义信息和上述栅格地图，使移动设备执行更智能化的导航作业，例如，针对智能扫地机器人，可以更加智能化地制定合理的清扫路线。
在一种可能的实现方式中，在步骤S14中，根据定位信息以及深度图像，确定目标场景对应的栅格地图，包括：
步骤S144：根据定位信息以及深度图像，确定目标场景对应的初始栅格地图。
其中，可参照上述步骤S141至步骤S143生成栅格地图的过程，得到本公开实施例中的初始栅格地图。
步骤S145：对场景图像进行识别，确定目标场景的场景类别以及目标场景中障碍物的障碍物类别。
其中，可采用本领域已知的图像识别技术，例如，用于图像识别的神经网络，对场景图像进行识别，得到目标场景的场景类别以及目标场景中障碍物的障碍物类别，也即得到目标场景的语义信息和障碍物的语义信息。
其中，目标场景的场景类别可表征不同的目标场景，例如，卧室、客厅、厨房等；障碍物的障碍物类别可表征不同的障碍物，例如，人、宠物、沙发、桌子、凳子等。
步骤S146：根据场景类别、障碍物类别以及初始栅格地图，生成栅格地图。
在一种可能的实现方式中，根据场景类别、障碍物类别以及初始栅格地图，生成栅格地图，可包括：将场景类别作为一个属性值填写到初始栅格地图的一个或多个栅格中；将障碍物类别作为一个属性值填写到对应障碍物的所属栅格中。
其中，例如可通过任意字符、数字等，表征不同场景类别和不同障碍物类别，从而可用表 征不同场景类别和不同障碍物类别的字符、数字等，作为属性值填写到栅格中，对此本公开实施例不作限制。
通过该方式，栅格地图中的每个栅格可表征：该栅格指示的区域是否为可行驶区域；若为不可行驶区域，该不可行驶区域包含什么类别的障碍物，例如是未知障碍物(如房间轮廓)，还是特定类型障碍物(如宠物)等；以及还可知晓该栅格或该栅格地图属于的场景类型，例如是属于客厅，还是卧室等。
在一种可能的实现方式中，移动设备根据包含丰富语义信息的栅格地图，不仅可执行更智能化的导航作业，还可进行一些智能化的导航决策，例如，针对智能扫地机器人，可根据上述栅格地图制定清扫路线；还可根据栅格地图中的语义信息例如发现房间有人，则先不清扫该房间；发现移动路径上有鞋子，则避开鞋子并提示用户收拾鞋子等。
在本公开实施例中，能够使生成的栅格地图包含目标场景中丰富的语义信息，以便于移动设备执行更智能化的导航作业。
应理解的是，移动设备所处的目标场景中障碍物的位置可能会实时变化，相应地，栅格地图也是会随时变化。同时，移动设备可在不同目标场景中移动，也可在同一目标场景中的不同区域内移动。
基于此，针对移动设备处于目标场景内，或者说目标场景中障碍物处于移动设备的视野范围内的障碍物的位置变化情况，可实时根据确定出的障碍物位置、障碍物类别等对栅格地图进行更新。而针对移动设备已处于目标场景外，例如智能扫地机器人从卧室移动到客厅，或者说目标场景中障碍物已处于移动设备的视野范围外的情况下，便无法感知已处于移动设备的视野范围外的障碍物的位置变化情况。
为了使移动设备能够在目标场景中障碍物已处于移动设备的视野范围外的情况下，也可实现更智能化的导航作业、导航决策等，在一种可能的实现方式中，所述方法还包括：
步骤S16：根据不可行驶区域对应的障碍物类别，对栅格地图中表征不可行驶区域的栅格设置调整系数以及预设常数，其中，调整系数用于指示预设常数的衰减速度或增长速度。
其中，可根据障碍物类别预设不同的调整系数，或者说可以预设障碍物类别与调整系数之间的对应关系，进而可实现根据障碍物类别，对表征不可行驶区域的栅格设置调整系数。预设常数可根据障碍物类别设置不同的常数，也可设置同一个常数，对此本公开实施例不作限制。
可理解的是，不同障碍物移动位置的概率不同，例如，针对人和宠物，可认为移动位置的概率较大；针对椅子、鞋子等，可认为移动位置的概率适中；而针对沙发、床、书柜等，可认为移动位置的概率偏小。由此，可根据不同障碍物移动位置的概率，预设障碍物类别与调整系数之间的对应关系，例如，针对移动位置的概率较大的障碍物，可设置较大的调整系数；而针对移动位置的概率较小的障碍物，可设置较小的调整系数，对此本公开实施例不作限制。
如上所述，调整系数用于指示预设常数的衰减速度或增长速度，其中，可以设置调整系数与衰减速度或增长速度正相关，例如，调整系数越大，预设常数的衰减速度或增长速度越快，代表障碍物实时变化位置的可能性越高；反之，调整系数越小，预设常数的衰减速度或增长速度越慢，代表障碍物实时变化位置的可能性越低。当然，也可设置调整系数与衰减速度或增长速度负相关，对此本公开实施例不作限制。
在一种可能的实现方式中，可以在栅格地图中的栅格被首次标记为不可行驶区域时，对该栅格设置调整系数以及预设常数；还可在目标场景中的不可行驶区域已处于移动设备的视野范围外时，对表征该不可行驶区域的栅格设置调整系数以及预设常数，对此本公开实施例不作限制。
其中，目标场景中的不可行驶区域处于移动设备的视野范围外，也即，目标场景中的障碍物处于移动设备的视野范围外；该障碍物处于移动设备的视野范围外，可理解为，障碍物处于图像采集部件和深度传感部件的拍摄范围(视场角)外，也即，图像采集部件和深度传感部件采集的场景图像和深度图像中已不包含该障碍物。
通过该方式，利用调整系数以及预设常数，能够反映障碍物位置变化的可能性，或者说能 够对障碍物的位置变化进行预测，以便于更新栅格地图，尤其针对已处于移动设备的视野范围外的障碍物所对应栅格的进行更新。
步骤S17：根据调整系数以及预设常数，更新栅格地图，得到更新后的栅格地图，以使移动设备根据更新后的栅格地图进行导航作业。
在一种可能的实现方式中，根据调整系数以及预设常数，更新栅格地图，得到更新后的栅格地图，包括：针对栅格地图中表征不可行驶区域的任一栅格，根据栅格的调整系数指示的衰减速度，递减栅格的预设常数，或者，根据栅格的调整系数指示的增长速度，递增栅格的预设常数；在预设常数递减至小于第一预设阈值，或预设常数递增至大于第二预设阈值的情况下，将栅格设置为可行驶区域，得到更新后的栅格地图。通过该方式，能够实现高效地根据调整系数以及预设常数更新栅格地图。
其中，可根据预设常数和调整系数的数值，以及历史经验等，设置第一预设阈值及第二预设阈值的数值，对此本公开实施例不作限制。
其中，预设常数递减至小于第一预设阈值，或预设常数递增至大于第二预设阈值，可认为是，随着时间的流逝，障碍物已离开原来的占据的位置。此时，可将该障碍物的所属栅格设置为可行驶区域，例如，将用于标记不可行驶区域的数字，修改为用于标记可行驶区域的数字等，得到更新后的栅格地图。
可理解的是，通过根据调整系数以及预设常数更新栅格地图的方式，是对障碍物位置变化的预测，并不体现实际障碍物的位置变化。若针对表征障碍物的任一栅格，该栅格的预设常数递减至小于第一预设阈值，或预设常数递增至大于第二预设阈值的情况发生，但该障碍物一直处于移动设备的视野范围内；或者，该障碍物从视野范围外又重新处于视野范围内，此时，仍以上述本公开实施例实际确定的障碍物位置，更新栅格地图。
在一种可能的实现方式中，移动设备可根据更新后的栅格地图进行智能化的导航作业，导航决策，例如，针对智能扫地机器人，若发现某房间有人，则先不清扫该房间，那么当人对应的预设常数随着调整系数衰减至小于第一预设阈值时，此时可认为人已不在该房间中，则可根据更新后的栅格地图，规划去该房间清扫的移动路径；或者，当房间的地面上有鞋子，可避开该鞋子进行清扫，当鞋子对应的预设常数随着调整系数递增至大于第二预设阈值时，此时可认为鞋子已不在原来位置，则可根据更新后的栅格地图，规划清扫鞋子原来所占区域的移动路径等。
在本公开实施例，通过根据调整系数以及预设常数更新栅格地图，尤其针对目标场景中的障碍物处于移动设备的视野范围外的情况，能够实现预测处于移动设备的视野范围外障碍物的位置变化，进而可使移动设备实现更智能化的导航作业、导航决策等。
根据本公开的实施例，移动设备(如智能扫地机器人)既具有现有规划式移动设备的优势，同时不需要引入激光雷达结构，从而增加移动设备在所处场景中的可通过性；还可增加移动设备的避障能力，实现对低矮障碍物的规避；引入朝前的深度相机与RGB相机，可以对室内物体以及场景进行拍摄，结合视觉感知算法(识别场景图像的场景类别和障碍物类别)，实现对所处场景的语义理解，从而使得移动设备可以进行更智能化的导航任务。
根据本公开的实施例，移动设备上设置的信息采集模组(即RGBD-IMU模组)中，深度图像由TOF相机获得，相较于使用双目相机获取深度信息的方式，TOF相机获取深度信息的精度更高，计算开销更低。并且由于RGBD-IMU是一个整体的模组，RGB图像以及深度图像可以进行良好的时间同步，因此，可将深度图像中的深度信息也用到移动设备的自定位中(也即确定移动设备的定位信息)。
根据本公开的实施例，与含激光雷达的移动设备相比，由于未引入激光雷达，因此移动设备可以设计得更矮小，在所处场景中通过性更高；而且深度相机采集的深度图像可以用于避开低矮的障碍物；与含朝上设置摄像头且不含激光雷达的移动设备相比，可利用朝前设置的RGB相机所采集的图像，对所处场景进行语义分析；与仅含深度相机的移动设备相比，除了避障、路径规划等作业，还可对所处场景进行语义分析，从而进行更智能化的导航任务。
可以理解，本公开提及的上述各个方法实施例，在不违背原理逻辑的情况下，均可以彼此相互结合形成结合后的实施例。本领域技术人员可以理解，在具体实施方式的上述方法中，各步骤的具体执行顺序应当以其功能和可能的内在逻辑确定。
此外，本公开还提供了地图构建装置、电子设备、计算机可读存储介质、程序，上述均可用来实现本公开提供的任一种地图构建方法，相应技术方案和描述和参见方法部分的相应记载。
图5为本公开实施例提供的一种地图构建装置的组成结构示意图，如图5所示，所述装置包括：
获取模块101，被配置为获取移动设备采集的目标场景的场景图像以及深度图像，并获取所述移动设备在当前时刻下的相对位姿；
确定模块102，被配置为根据所述场景图像与所述深度图像，确定所述目标场景在先时刻的第一空间特征点以及当前时刻的第二空间特征点，所述在先时刻的时序在所述当前时刻之前，所述第一空间特征点与所述第二空间特征点相匹配；
优化模块103，被配置为根据所述第一空间特征点与所述第二空间特征点，对所述相对位姿进行优化，得到所述移动设备在当前时刻下的定位信息；
构建模块104，被配置为根据所述定位信息以及所述深度图像，确定所述目标场景对应的栅格地图，所述栅格地图用于指示所述目标场景中的可行驶区域，以使所述移动设备根据所述栅格地图进行导航作业。
在一种可能的实现方式中，所述场景图像是由所述移动设备上设置的图像采集组件采集的，所述深度图像是由所述移动设备上设置的深度传感组件采集的，其中，确定模块102，包括：提取子模块，被配置为对在先时刻采集的场景图像进行特征点提取，得到第一图像特征点；匹配子模块，被配置为根据所述第一图像特征点，确定当前时刻采集的场景图像中与所述第一图像特征点匹配的第二图像特征点；映射子模块，被配置为根据所述场景图像与所述深度图像之间的映射关系，确定与所述第一图像特征点对应的第一深度值，以及与所述第二图像特征点对应的第二深度值；第一确定子模块，被配置为根据所述第一图像特征点、所述第一深度值以及所述图像采集组件的相机内参，确定所述第一空间特征点；第二确定子模块，被配置为根据所述第二图像特征点、所述第二深度值以及所述图像采集组件的相机内参，确定所述第二空间特征点。
在一种可能的实现方式中，所述相对位姿为多个，所述优化模块103，包括：变换子模块，被配置为针对任一相对位姿，根据所述相对位姿，将所述第一空间特征点变换至所述当前时刻下的第三空间特征点；优化子模块，被配置为根据所述第二空间特征点与所述第三空间特征点之间的第一误差，和/或所述相对位姿与其它相对位姿之间的第二误差，优化所述相对位姿，得到优化后的相对位姿，其中，所述其它相对位姿包括多个相对位姿中除所述相对位姿以外的相对位姿；定位信息确定子模块，被配置为根据所述优化后的相对位姿，确定所述移动设备在当前时刻下的定位信息；其中，所述多个相对位姿包括所述移动设备上设置的惯性传感组件测量的相对位姿，以及所述移动设备上设置的轮式里程计测量的相对位姿。
在一种可能的实现方式中，构建模块104，包括：栅格平面确定子模块，被配置为确定所述目标场景对应的二维平面，并对所述二维平面进行栅格划分，得到栅格平面；位置信息确定子模块，被配置为根据所述定位信息以及所述深度图像，确定所述目标场景中障碍物的位置信息；第一栅格地图确定子模块，被配置为根据所述障碍物的位置信息，确定并标记所述障碍物在所述栅格平面中的所属栅格，得到所述目标场景对应的栅格地图。
在一种可能的实现方式中，所述构建模块104，包括：初始栅格地图确定子模块，被配置为根据所述定位信息以及所述深度图像，确定所述目标场景对应的初始栅格地图；识别子模块，被配置为对所述场景图像进行识别，确定所述目标场景的场景类别以及所述目标场景中障碍物的障碍物类别；第二栅格地图确定子模块，被配置为根据所述场景类别、所述障碍物类别以及所述初始栅格地图，生成所述栅格地图。
在一种可能的实现方式中，所述装置还包括：设置模块，被配置为根据所述目标场景中不 可行驶区域对应的障碍物类别，对所述栅格地图中表征所述不可行驶区域的栅格设置调整系数以及预设常数，其中，所述调整系数用于指示所述预设常数的衰减速度或增长速度；更新模块，被配置为根据所述调整系数以及所述预设常数，更新所述栅格地图，得到更新后的栅格地图，以使所述移动设备根据所述更新后的栅格地图进行导航作业。
在一种可能的实现方式中，更新模块，包括：调整子模块，被配置为针对所述栅格地图中表征所述不可行驶区域的任一栅格，根据所述栅格的调整系数指示的衰减速度，递减所述栅格的预设常数，或者，根据所述栅格的调整系数指示的增长速度，递增所述栅格的预设常数；更新子模块，被配置为在所述预设常数递减至小于第一预设阈值，或所述预设常数递增至大于第二预设阈值的情况下，将所述栅格设置为可行驶区域，得到更新后的栅格地图。
在一种可能的实现方式中，所述相对位姿包括：所述移动设备在当前时刻相对于在先时刻的相对位置以及航向角；所述移动设备的图像采集组件、深度传感组件以及惯性传感组件组成信息采集模组，所述信息采集模组设置在所述移动设备的前端，所述信息采集模组的朝向与所述移动设备的移动方向一致；所述移动设备的轮式里程计设置在所述移动设备的底端。
在本公开实施例中，能够利用场景图像与深度图像确定出的空间特征点，对移动设备当前时刻下的相对位姿进行优化，从而可以得到较高精度的定位信息；进而利用较高精度的定位信息以及深度图像确定出的栅格地图，能够精准地指示目标场景中的可行使区域，不仅能够满足路径规划、障碍物分析等导航作业需求，还有效提高地图构建效率，使移动设备执行合理高效的自主导航作业。
在一些实施例中，本公开实施例提供的装置具有的功能或包含的模块可以用于执行上文方法实施例描述的方法，其具体实现可以参照上文方法实施例的描述。
本公开实施例还提出一种计算机可读存储介质，其上存储有计算机程序指令，所述计算机程序指令被处理器执行时实现上述方法。计算机可读存储介质可以是易失性或非易失性计算机可读存储介质。
本公开实施例还提出一种电子设备，包括：处理器；用于存储处理器可执行指令的存储器；其中，所述处理器被配置为调用所述存储器存储的指令，以执行上述方法。
本公开实施例还提供了一种计算机程序产品，包括计算机可读代码，或者承载有计算机可读代码的非易失性计算机可读存储介质，当所述计算机可读代码在电子设备的处理器中运行时，所述电子设备中的处理器执行上述方法。
电子设备可以被提供为终端设备、移动设备或其它形态的设备。
图6为本公开实施例提供的一种电子设备800的组成结构示意图。例如，电子设备800可以是移动设备，或移动电话，计算机，数字广播终端，消息收发设备，游戏控制台，平板设备，医疗设备，健身设备，个人数字助理等终端设备。
参照图6，电子设备800可以包括以下一个或多个组件：处理组件802，存储器804，电源组件806，多媒体组件808，音频组件810，输入/输出(I/O)的接口812，传感器组件814，以及通信组件816。
处理组件802通常控制电子设备800的整体操作，诸如与显示，电话呼叫，数据通信，相机操作和记录操作相关联的操作。处理组件802可以包括一个或多个处理器820来执行指令，以完成上述的方法的全部或部分步骤。此外，处理组件802可以包括一个或多个模块，便于处理组件802和其他组件之间的交互。例如，处理组件802可以包括多媒体模块，以方便多媒体组件808和处理组件802之间的交互。
存储器804被配置为存储各种类型的数据以支持在电子设备800的操作。这些数据的示例包括用于在电子设备800上操作的任何应用程序或方法的指令，联系人数据，电话簿数据，消息，图片，视频等。存储器804可以由任何类型的易失性或非易失性存储设备或者它们的组合实现，如静态随机存取存储器(Static Random-Access Memory，SRAM)，电可擦除可编程只读存储器(Electrically Erasable Programmable Read Only Memory，EEPROM)，可擦除可编程只读存储器(Erasable Programmable Read-Only Memory，EPROM)，可编程只读存储器(Programmable  Read-Only Memory，PROM)，只读存储器(Read-Only Memory，ROM)，磁存储器，快闪存储器，磁盘或光盘。
电源组件806为电子设备800的各种组件提供电力。电源组件806可以包括电源管理***，一个或多个电源，及其他与为电子设备800生成、管理和分配电力相关联的组件。
多媒体组件808包括在所述电子设备800和用户之间的提供一个输出接口的屏幕。在一些实施例中，屏幕可以包括液晶显示器(Liquid Crystal Display，LCD)和触摸面板(Touch Panel，TP)。如果屏幕包括触摸面板，屏幕可以被实现为触摸屏，以接收来自用户的输入信号。触摸面板包括一个或多个触摸传感器以感测触摸、滑动和触摸面板上的手势。所述触摸传感器可以不仅感测触摸或滑动动作的边界，而且还检测与所述触摸或滑动操作相关的持续时间和压力。在一些实施例中，多媒体组件808包括一个前置摄像头和/或后置摄像头。当电子设备800处于操作模式，如拍摄模式或视频模式时，前置摄像头和/或后置摄像头可以接收外部的多媒体数据。每个前置摄像头和后置摄像头可以是一个固定的光学透镜***或具有焦距和光学变焦能力。
音频组件810被配置为输出和/或输入音频信号。例如，音频组件810包括一个麦克风(Microphone，MIC)，当电子设备800处于操作模式，如呼叫模式、记录模式和语音识别模式时，麦克风被配置为接收外部音频信号。所接收的音频信号可以被进一步存储在存储器804或经由通信组件816发送。在一些实施例中，音频组件810还包括一个扬声器，用于输出音频信号。
I/O接口812为处理组件802和***接口模块之间提供接口，上述***接口模块可以是键盘，点击轮，按钮等。这些按钮可包括但不限于：主页按钮、音量按钮、启动按钮和锁定按钮。
传感器组件814包括一个或多个传感器，用于为电子设备800提供各个方面的状态评估。例如，传感器组件814可以检测到电子设备800的打开/关闭状态，组件的相对定位，例如所述组件为电子设备800的显示器和小键盘，传感器组件814还可以检测电子设备800或电子设备800一个组件的位置改变，用户与电子设备800接触的存在或不存在，电子设备800方位或加速/减速和电子设备800的温度变化。传感器组件814可以包括接近传感器，被配置用来在没有任何的物理接触时检测附近物体的存在。传感器组件814还可以包括光传感器，如互补金属氧化物半导体(Complementary Metal Oxide Semiconductor，CMOS)或电荷耦合装置(Charge-coupled Device，CCD)图像传感器，用于在成像应用中使用。在一些实施例中，该传感器组件814还可以包括加速度传感器，陀螺仪传感器，磁传感器，压力传感器或温度传感器。
通信组件816被配置为便于电子设备800和其他设备之间有线或无线方式的通信。电子设备800可以接入基于通信标准的无线网络，如无线网络(Wireless Fidelity，WiFi)，第二代移动通信技术(2nd-Generation，2G)或第三代移动通信技术(3rd-Generation，3G)，或它们的组合。在一个示例性实施例中，通信组件816经由广播信道接收来自外部广播管理***的广播信号或广播相关信息。在一个示例性实施例中，所述通信组件816还包括近场通信(Near Field Communication，NFC)模块，以促进短程通信。例如，在NFC模块可基于射频识别(Radio Frequency Identification，RFID)技术，红外数据协会(Infrared Data Association，IrDA)技术，超宽带(Ultra Wide Band，UWB)技术，蓝牙(Blue Tooth，BT)技术和其他技术来实现。
在示例性实施例中，电子设备800可以被一个或多个应用专用集成电路(Application Specific Integrated Circuit，ASIC)、数字信号处理器(Digital Signal Process，DSP)、数字信号处理设备(Digital Signal Process Device，DSPD)、可编程逻辑器件(Programmable Logic Device，PLD)、现场可编程门阵列(Field Programmable Gate Array，FPGA)、控制器、微控制器、微处理器或其他电子元件实现，用于执行上述方法。
在示例性实施例中，还提供了一种非易失性计算机可读存储介质，例如包括计算机程序指令的存储器804，上述计算机程序指令可由电子设备800的处理器820执行以完成上述方法。
本公开可以是***、方法和/或计算机程序产品。计算机程序产品可以包括计算机可读存储介质，其上载有用于使处理器实现本公开的各个方面的计算机可读程序指令。
计算机可读存储介质可以是可以保持和存储由指令执行设备使用的指令的有形设备。计算 机可读存储介质例如可以是(但不限于)电存储设备、磁存储设备、光存储设备、电磁存储设备、半导体存储设备或者上述的任意合适的组合。计算机可读存储介质的更具体的例子(非穷举的列表)包括：便携式计算机盘、硬盘、随机存取存储器(Random Access Memory，RAM)、只读存储器(Read-Only Memory，ROM)、可擦式可编程只读存储器(EPROM或闪存)、静态随机存取存储器(Static Random-Access Memory，SRAM)、便携式压缩盘只读存储器(Compact Disc Read-Only Memory，CD-ROM)、数字多功能盘(Digital Video Disc，DVD)、记忆棒、软盘、机械编码设备、例如其上存储有指令的打孔卡或凹槽内凸起结构、以及上述的任意合适的组合。这里所使用的计算机可读存储介质不被解释为瞬时信号本身，诸如无线电波或者其他自由传播的电磁波、通过波导或其他传输媒介传播的电磁波(例如，通过光纤电缆的光脉冲)、或者通过电线传输的电信号。
这里所描述的计算机可读程序指令可以从计算机可读存储介质下载到各个计算/处理设备，或者通过网络、例如因特网、局域网、广域网和/或无线网下载到外部计算机或外部存储设备。网络可以包括铜传输电缆、光纤传输、无线传输、路由器、防火墙、交换机、网关计算机和/或边缘服务器。每个计算/处理设备中的网络适配卡或者网络接口从网络接收计算机可读程序指令，并转发该计算机可读程序指令，以供存储在各个计算/处理设备中的计算机可读存储介质中。
用于执行本公开操作的计算机程序指令可以是汇编指令、指令集架构(Instruction Set Architecture，ISA)指令、机器指令、机器相关指令、微代码、固件指令、状态设置数据、或者以一种或多种编程语言的任意组合编写的源代码或目标代码，所述编程语言包括面向对象的编程语言—诸如Smalltalk、C++等，以及常规的过程式编程语言—诸如“C”语言或类似的编程语言。计算机可读程序指令可以完全地在用户计算机上执行、部分地在用户计算机上执行、作为一个独立的软件包执行、部分在用户计算机上部分在远程计算机上执行、或者完全在远程计算机或服务器上执行。在涉及远程计算机的情形中，远程计算机可以通过任意种类的网络—包括局域网(Local Area Network，LAN)或广域网(Wide Area Network，WAN)—连接到用户计算机，或者，可以连接到外部计算机(例如利用因特网服务提供商来通过因特网连接)。在一些实施例中，通过利用计算机可读程序指令的状态信息来个性化定制电子电路，例如可编程逻辑电路、现场可编程门阵列(FPGA)或可编程逻辑阵列(ProgrammableLogicArray，PLA)，该电子电路可以执行计算机可读程序指令，从而实现本公开的各个方面。
这里参照根据本公开实施例的方法、装置(***)和计算机程序产品的流程图和/或框图描述了本公开的各个方面。应当理解，流程图和/或框图的每个方框以及流程图和/或框图中各方框的组合，都可以由计算机可读程序指令实现。
这些计算机可读程序指令可以提供给通用计算机、专用计算机或其它可编程数据处理装置的处理器，从而生产出一种机器，使得这些指令在通过计算机或其它可编程数据处理装置的处理器执行时，产生了实现流程图和/或框图中的一个或多个方框中规定的功能/动作的装置。也可以把这些计算机可读程序指令存储在计算机可读存储介质中，这些指令使得计算机、可编程数据处理装置和/或其他设备以特定方式工作，从而，存储有指令的计算机可读介质则包括一个制造品，其包括实现流程图和/或框图中的一个或多个方框中规定的功能/动作的各个方面的指令。
也可以把计算机可读程序指令加载到计算机、其它可编程数据处理装置、或其它设备上，使得在计算机、其它可编程数据处理装置或其它设备上执行一系列操作步骤，以产生计算机实现的过程，从而使得在计算机、其它可编程数据处理装置、或其它设备上执行的指令实现流程图和/或框图中的一个或多个方框中规定的功能/动作。
附图中的流程图和框图显示了根据本公开的多个实施例的***、方法和计算机程序产品的可能实现的体系架构、功能和操作。在这点上，流程图或框图中的每个方框可以代表一个模块、程序段或指令的一部分，所述模块、程序段或指令的一部分包含一个或多个用于实现规定的逻辑功能的可执行指令。在有些作为替换的实现中，方框中所标注的功能也可以以不同于附图中所标注的顺序发生。例如，两个连续的方框实际上可以基本并行地执行，它们有时也可以按相反的顺序执行，这依所涉及的功能而定。也要注意的是，框图和/或流程图中的每个方框、以及 框图和/或流程图中的方框的组合，可以用执行规定的功能或动作的专用的基于硬件的***来实现，或者可以用专用硬件与计算机指令的组合来实现。
该计算机程序产品可以具体通过硬件、软件或其结合的方式实现。在一个可选实施例中，所述计算机程序产品具体体现为计算机存储介质，在另一个可选实施例中，计算机程序产品具体体现为软件产品，例如软件开发包(Software Development Kit，SDK)等等。
以上已经描述了本公开的各实施例，上述说明是示例性的，并非穷尽性的，并且也不限于所披露的各实施例。在不偏离所说明的各实施例的范围和精神的情况下，对于本技术领域的普通技术人员来说许多修改和变更都是显而易见的。本文中所用术语的选择，旨在最好地解释各实施例的原理、实际应用或对市场中的技术的改进，或者使本技术领域的其它普通技术人员能理解本文披露的各实施例。
工业实用性
本公开实施例公开了一种地图构建方法及装置、电子设备、存储介质和计算机程序产品，其中，地图构建方法包括：获取移动设备采集的目标场景的场景图像以及深度图像，并获取移动设备在当前时刻下的相对位姿；根据场景图像与所述深度图像，确定目标场景在先时刻的第一空间特征点以及当前时刻的第二空间特征点，在先时刻的时序在所述当前时刻之前，第一空间特征点与所述第二空间特征点相匹配；根据第一空间特征点与第二空间特征点，对相对位姿进行优化，得到移动设备在当前时刻下的定位信息；根据定位信息以及深度图像，确定目标场景对应的栅格地图，栅格地图用于指示目标场景中的可行驶区域，以使移动设备根据栅格地图进行导航作业。本公开实施例利用场景图像与深度图像确定出的空间特征点，对移动设备当前时刻下的相对位姿进行优化，从而可以得到较高精度的定位信息；进而利用较高精度的定位信息以及深度图像确定出的栅格地图，能够精准地指示目标场景中的可行使区域，不仅能够满足路径规划、障碍物分析等导航作业需求，还有效提高地图构建效率，有助于移动设备执行合理高效的自主导航作业。

Claims (19)

  1. 一种地图构建方法，包括：
    获取移动设备采集的目标场景的场景图像以及深度图像，并获取所述移动设备在当前时刻下的相对位姿；
    根据所述场景图像与所述深度图像，确定所述目标场景在先时刻的第一空间特征点以及当前时刻的第二空间特征点，所述在先时刻的时序在所述当前时刻之前，所述第一空间特征点与所述第二空间特征点相匹配；
    根据所述第一空间特征点与所述第二空间特征点，对所述相对位姿进行优化，得到所述移动设备在当前时刻下的定位信息；
    根据所述定位信息以及所述深度图像，确定所述目标场景对应的栅格地图，所述栅格地图用于指示所述目标场景中的可行驶区域，以使所述移动设备根据所述栅格地图进行导航作业。
  2. 根据权利要求1所述的方法，其中，所述场景图像是由所述移动设备上设置的图像采集组件采集的，所述深度图像是由所述移动设备上设置的深度传感组件采集的，
    其中，根据所述场景图像与所述深度图像，确定所述目标场景在先时刻的第一空间特征点以及当前时刻的第二空间特征点，包括：
    对在先时刻采集的场景图像进行特征点提取，得到第一图像特征点；
    根据所述第一图像特征点，确定当前时刻采集的场景图像中与所述第一图像特征点匹配的第二图像特征点；
    根据所述场景图像与所述深度图像之间的映射关系，确定与所述第一图像特征点对应的第一深度值，以及与所述第二图像特征点对应的第二深度值；
    根据所述第一图像特征点、所述第一深度值以及所述图像采集组件的相机内参，确定所述第一空间特征点；
    根据所述第二图像特征点、所述第二深度值以及所述图像采集组件的相机内参，确定所述第二空间特征点。
  3. 根据权利要求1或2所述的方法，其中，所述相对位姿为多个，所述根据所述第一空间特征点与所述第二空间特征点，对所述相对位姿进行优化，得到所述移动设备在当前时刻下的定位信息，包括：
    针对任一相对位姿，根据所述相对位姿，将所述第一空间特征点变换至所述当前时刻下的第三空间特征点；
    根据所述第二空间特征点与所述第三空间特征点之间的第一误差，和/或所述相对位姿与其它相对位姿之间的第二误差，优化所述相对位姿，得到优化后的相对位姿，其中，所述其它相对位姿包括多个相对位姿中除所述相对位姿以外的相对位姿；
    根据所述优化后的相对位姿，确定所述移动设备在当前时刻下的定位信息；
    其中，所述多个相对位姿包括所述移动设备上设置的惯性传感组件测量的相对位姿，以及所述移动设备上设置的轮式里程计测量的相对位姿。
  4. 根据权利要求1至3任一项所述的方法，其中，根据所述定位信息以及所述深度图像，确定所述目标场景对应的栅格地图，包括：
    确定所述目标场景对应的二维平面，并对所述二维平面进行栅格划分，得到栅格平面；
    根据所述定位信息以及所述深度图像，确定所述目标场景中障碍物的位置信息；
    根据所述障碍物的位置信息，确定并标记所述障碍物在所述栅格平面中的所属栅格，得到所述目标场景对应的栅格地图。
  5. 根据权利要求1至4任一项所述的方法，其中，所述根据所述定位信息以及所述深度图像，确定所述目标场景对应的栅格地图，包括：
    根据所述定位信息以及所述深度图像，确定所述目标场景对应的初始栅格地图；
    对所述场景图像进行识别，确定所述目标场景的场景类别以及所述目标场景中障碍物的障碍物类别；
    根据所述场景类别、所述障碍物类别以及所述初始栅格地图，生成所述栅格地图。
  6. 根据权利要求1至5任一项所述的方法，其中，所述方法还包括：
    根据所述目标场景中不可行驶区域对应的障碍物类别，对所述栅格地图中表征所述不可行驶区域的栅格设置调整系数以及预设常数，其中，所述调整系数用于指示所述预设常数的衰减速度或增长速度；
    根据所述调整系数以及所述预设常数，更新所述栅格地图，得到更新后的栅格地图，以使所述移动设备根据所述更新后的栅格地图进行导航作业。
  7. 根据权利要求6所述的方法，其中，根据所述调整系数以及所述预设常数，更新所述栅格地图，得到更新后的栅格地图，包括：
    针对所述栅格地图中表征所述不可行驶区域的任一栅格，根据所述栅格的调整系数指示的衰减速度，递减所述栅格的预设常数，或者，根据所述栅格的调整系数指示的增长速度，递增所述栅格的预设常数；
    在所述预设常数递减至小于第一预设阈值，或所述预设常数递增至大于第二预设阈值的情况下，将所述栅格设置为可行驶区域，得到更新后的栅格地图。
  8. 根据权利要求1至6任一项所述的方法，其中，
    所述相对位姿包括：所述移动设备在当前时刻相对于在先时刻的相对位置以及航向角；
    所述移动设备的图像采集组件、深度传感组件以及惯性传感组件组成信息采集模组，所述信息采集模组设置在所述移动设备的前端，所述信息采集模组的朝向与所述移动设备的移动方向一致；
    所述移动设备的轮式里程计设置在所述移动设备的底端。
  9. 一种地图构建装置，包括：
    获取模块，被配置为获取移动设备采集的目标场景的场景图像以及深度图像，并获取所述移动设备在当前时刻下的相对位姿；
    确定模块，被配置为根据所述场景图像与所述深度图像，确定所述目标场景在先时刻的第一空间特征点以及当前时刻的第二空间特征点，所述在先时刻的时序在所述当前时刻之前，所述第一空间特征点与所述第二空间特征点相匹配；
    优化模块，被配置为根据所述第一空间特征点与所述第二空间特征点，对所述相对位姿进行优化，得到所述移动设备在当前时刻下的定位信息；
    构建模块，被配置为根据所述定位信息以及所述深度图像，确定所述目标场景对应的栅格地图，所述栅格地图用于指示所述目标场景中的可行驶区域，以使所述移动设备根据所述栅格地图进行导航作业。
  10. 根据权利要求9所述的装置，其中，所述场景图像是由所述移动设备上设置的图像采集组件采集的，所述深度图像是由所述移动设备上设置的深度传感组件采集的；
    所述确定模块，包括：提取子模块，被配置为对在先时刻采集的场景图像进行特征点提取，得到第一图像特征点；
    匹配子模块，被配置为根据所述第一图像特征点，确定当前时刻采集的场景图像中与所述第一图像特征点匹配的第二图像特征点；
    映射子模块，被配置为根据所述场景图像与所述深度图像之间的映射关系，确定与所述第一图像特征点对应的第一深度值，以及与所述第二图像特征点对应的第二深度值；
    第一确定子模块，被配置为根据所述第一图像特征点、所述第一深度值以及所述图像采集组件的相机内参，确定所述第一空间特征点；
    第二确定子模块，被配置为根据所述第二图像特征点、所述第二深度值以及所述图像采集组件的相机内参，确定所述第二空间特征点。
  11. 根据权利要求9或10所述的装置，其中，所述相对位姿为多个，所述优化模块，包括：变换子模块，被配置为针对任一相对位姿，根据所述相对位姿，将所述第一空间特征点变换至 所述当前时刻下的第三空间特征点；优化子模块，被配置为根据所述第二空间特征点与所述第三空间特征点之间的第一误差，和/或所述相对位姿与其它相对位姿之间的第二误差，优化所述相对位姿，得到优化后的相对位姿，其中，所述其它相对位姿包括多个相对位姿中除所述相对位姿以外的相对位姿；定位信息确定子模块，被配置为根据所述优化后的相对位姿，确定所述移动设备在当前时刻下的定位信息；其中，所述多个相对位姿包括所述移动设备上设置的惯性传感组件测量的相对位姿，以及所述移动设备上设置的轮式里程计测量的相对位姿。
  12. 根据权利要求9至11任一项所述的装置，其中，构建模块，包括：
    栅格平面确定子模块，被配置为确定所述目标场景对应的二维平面，并对所述二维平面进行栅格划分，得到栅格平面；
    位置信息确定子模块，被配置为根据所述定位信息以及所述深度图像，确定所述目标场景中障碍物的位置信息；
    第一栅格地图确定子模块，被配置为根据所述障碍物的位置信息，确定并标记所述障碍物在所述栅格平面中的所属栅格，得到所述目标场景对应的栅格地图。
  13. 根据权利要求9至12任一项所述的装置，其中，所述构建模块，包括：
    初始栅格地图确定子模块，被配置为根据所述定位信息以及所述深度图像，确定所述目标场景对应的初始栅格地图；
    识别子模块，被配置为对所述场景图像进行识别，确定所述目标场景的场景类别以及所述目标场景中障碍物的障碍物类别；第二栅格地图确定子模块，被配置为根据所述场景类别、所述障碍物类别以及所述初始栅格地图，生成所述栅格地图。
  14. 根据权利要求9至13任一项所述的装置，其中，所述装置还包括：
    设置模块，被配置为根据所述目标场景中不可行驶区域对应的障碍物类别，对所述栅格地图中表征所述不可行驶区域的栅格设置调整系数以及预设常数，其中，所述调整系数用于指示所述预设常数的衰减速度或增长速度；
    更新模块，被配置为根据所述调整系数以及所述预设常数，更新所述栅格地图，得到更新后的栅格地图，以使所述移动设备根据所述更新后的栅格地图进行导航作业。
  15. 根据权利要求14所述的装置，其中，更新模块，包括：
    调整子模块，被配置为针对所述栅格地图中表征所述不可行驶区域的任一栅格，根据所述栅格的调整系数指示的衰减速度，递减所述栅格的预设常数，或者，根据所述栅格的调整系数指示的增长速度，递增所述栅格的预设常数；
    更新子模块，被配置为在所述预设常数递减至小于第一预设阈值，或所述预设常数递增至大于第二预设阈值的情况下，将所述栅格设置为可行驶区域，得到更新后的栅格地图。
  16. 根据权利要求9至14任一项所述的装置，其中，所述相对位姿包括：所述移动设备在当前时刻相对于在先时刻的相对位置以及航向角；所述移动设备的图像采集组件、深度传感组件以及惯性传感组件组成信息采集模组，所述信息采集模组设置在所述移动设备的前端，所述信息采集模组的朝向与所述移动设备的移动方向一致；所述移动设备的轮式里程计设置在所述移动设备的底端。
  17. 一种电子设备，包括：
    处理器；
    用于存储处理器可执行指令的存储器；
    其中，所述处理器被配置为调用所述存储器存储的指令，以执行权利要求1至8中任意一项所述的方法。
  18. 一种计算机可读存储介质，其上存储有计算机程序指令，所述计算机程序指令被处理器执行时实现权利要求1至8中任意一项所述的方法。
  19. 一种计算机程序产品，包括计算机可读代码，或者承载有计算机可读代码的非易失性计算机可读存储介质，当所述计算机可读代码在电子设备的处理器中运行时，所述电子设备中的处理器执行权利要求1至8中任意一项所述的方法。
PCT/CN2021/122715 2021-06-18 2021-10-08 地图构建方法及装置、电子设备、存储介质和计算机程序产品 WO2022262152A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202110678480.7 2021-06-18
CN202110678480.7A CN113284240B (zh) 2021-06-18 2021-06-18 地图构建方法及装置、电子设备和存储介质

Publications (1)

Publication Number Publication Date
WO2022262152A1 true WO2022262152A1 (zh) 2022-12-22

Family

ID=77284994

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/122715 WO2022262152A1 (zh) 2021-06-18 2021-10-08 地图构建方法及装置、电子设备、存储介质和计算机程序产品

Country Status (2)

Country Link
CN (1) CN113284240B (zh)
WO (1) WO2022262152A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117132590A (zh) * 2023-10-24 2023-11-28 威海天拓合创电子工程有限公司 一种基于图像的多板卡缺陷检测方法及装置

Families Citing this family (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113284240B (zh) * 2021-06-18 2022-05-31 深圳市商汤科技有限公司 地图构建方法及装置、电子设备和存储介质
CN113848943B (zh) * 2021-10-18 2023-08-08 追觅创新科技(苏州)有限公司 栅格地图的修正方法及装置、存储介质及电子装置
CN114061586A (zh) * 2021-11-10 2022-02-18 北京有竹居网络技术有限公司 用于生成电子设备的导航路径的方法和产品
CN116148879B (zh) * 2021-11-22 2024-05-03 珠海一微半导体股份有限公司 一种机器人提升障碍物标注精度的方法
CN114187384A (zh) * 2021-12-17 2022-03-15 深圳Tcl数字技术有限公司 地图构建方法、装置、电子设备及存储介质
CN114004874B (zh) * 2021-12-30 2022-03-25 贝壳技术有限公司 占用栅格地图的获取方法和装置
WO2024005707A1 (en) * 2022-07-01 2024-01-04 Grabtaxi Holdings Pte. Ltd. Method, device and system for detecting dynamic occlusion
CN115205461B (zh) * 2022-07-15 2023-11-14 小米汽车科技有限公司 场景重建方法、装置、可读存储介质及车辆
CN114935341B (zh) * 2022-07-25 2022-11-29 深圳市景创科技电子股份有限公司 一种新型slam导航计算视频识别方法及装置
CN115752481B (zh) * 2022-12-09 2023-09-01 广东车卫士信息科技有限公司 基于图像识别的ar导航方法、ar眼镜、介质、设备
CN116342827B (zh) * 2023-05-26 2023-08-08 厦门精图信息技术有限公司 一种地图元素三维呈现***及方法
CN116687274B (zh) * 2023-06-19 2024-04-16 深圳市毫准科技有限公司 一种基于手机的可插拔式扫地机器人及扫地清洁控制方法
CN116704074A (zh) * 2023-08-03 2023-09-05 松灵机器人(深圳)有限公司 边界地图构建方法、装置、***和可读存储介质
CN117068207B (zh) * 2023-10-17 2024-01-12 合肥焕智科技有限公司 一种基于激光slam导航的车载控制***
CN117804449B (zh) * 2024-02-29 2024-05-28 锐驰激光(深圳)有限公司 割草机地面感知方法、装置、设备及存储介质

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20140198962A1 (en) * 2013-01-17 2014-07-17 Canon Kabushiki Kaisha Information processing apparatus, information processing method, and storage medium
US20170161546A1 (en) * 2015-12-08 2017-06-08 Mitsubishi Electric Research Laboratories, Inc. Method and System for Detecting and Tracking Objects and SLAM with Hierarchical Feature Grouping
CN107369183A (zh) * 2017-07-17 2017-11-21 广东工业大学 面向mar的基于图优化slam的跟踪注册方法及***
CN107845114A (zh) * 2017-11-10 2018-03-27 北京三快在线科技有限公司 地图的构建方法、装置及电子设备
CN109993793A (zh) * 2019-03-29 2019-07-09 北京易达图灵科技有限公司 视觉定位方法及装置
CN110264509A (zh) * 2018-04-27 2019-09-20 腾讯科技（深圳）有限公司 确定图像捕捉设备的位姿的方法、装置及其存储介质
CN113284240A (zh) * 2021-06-18 2021-08-20 深圳市商汤科技有限公司 地图构建方法及装置、电子设备和存储介质

Family Cites Families (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105045263B (zh) * 2015-07-06 2016-05-18 杭州南江机器人股份有限公司 一种基于Kinect深度相机的机器人自定位方法
CN110376594B (zh) * 2018-08-17 2022-02-01 北京京东叁佰陆拾度电子商务有限公司 一种基于拓扑图的智能导航的方法和***
CN111486855B (zh) * 2020-04-28 2021-09-14 武汉科技大学 一种具有物体导航点的室内二维语义栅格地图构建方法
CN111881239B (zh) * 2020-07-17 2023-07-28 上海高仙自动化科技发展有限公司 构建方法、构建装置、智能机器人及可读存储介质
CN112284393B (zh) * 2020-10-23 2022-12-23 苏州大学 一种智能移动机器人全局路径规划方法和***
CN112327888B (zh) * 2021-01-07 2021-03-30 中智行科技有限公司 路径规划方法、装置、电子设备和存储介质

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20140198962A1 (en) * 2013-01-17 2014-07-17 Canon Kabushiki Kaisha Information processing apparatus, information processing method, and storage medium
US20170161546A1 (en) * 2015-12-08 2017-06-08 Mitsubishi Electric Research Laboratories, Inc. Method and System for Detecting and Tracking Objects and SLAM with Hierarchical Feature Grouping
CN107369183A (zh) * 2017-07-17 2017-11-21 广东工业大学 面向mar的基于图优化slam的跟踪注册方法及***
CN107845114A (zh) * 2017-11-10 2018-03-27 北京三快在线科技有限公司 地图的构建方法、装置及电子设备
CN110264509A (zh) * 2018-04-27 2019-09-20 腾讯科技（深圳）有限公司 确定图像捕捉设备的位姿的方法、装置及其存储介质
CN109993793A (zh) * 2019-03-29 2019-07-09 北京易达图灵科技有限公司 视觉定位方法及装置
CN113284240A (zh) * 2021-06-18 2021-08-20 深圳市商汤科技有限公司 地图构建方法及装置、电子设备和存储介质

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117132590A (zh) * 2023-10-24 2023-11-28 威海天拓合创电子工程有限公司 一种基于图像的多板卡缺陷检测方法及装置
CN117132590B (zh) * 2023-10-24 2024-03-01 威海天拓合创电子工程有限公司 一种基于图像的多板卡缺陷检测方法及装置

Also Published As

Publication number Publication date
CN113284240B (zh) 2022-05-31
CN113284240A (zh) 2021-08-20

Similar Documents

Publication Publication Date Title
WO2022262152A1 (zh) 地图构建方法及装置、电子设备、存储介质和计算机程序产品
JP7236565B2 (ja) 位置姿勢決定方法、装置、電子機器、記憶媒体及びコンピュータプログラム
US11165959B2 (en) Connecting and using building data acquired from mobile devices
US9886774B2 (en) Photogrammetric methods and devices related thereto
CN110268225B (zh) 一种多设备之间协同操作的方法、服务端及电子设备
EP2951788B1 (en) Real-time 3d reconstruction with power efficient depth sensor usage
US20190096089A1 (en) Enabling use of three-dimensonal locations of features with two-dimensional images
CN109313810A (zh) 用于对环境进行测绘的***和方法
US20120113274A1 (en) Augmented reality interface for video tagging and sharing
CN113205549B (zh) 深度估计方法及装置、电子设备和存储介质
CN105074691A (zh) 上下文感知定位、映射和跟踪
US11290705B2 (en) Rendering augmented reality with occlusion
CA3069813C (en) Capturing, connecting and using building interior data from mobile devices
CN112105892A (zh) 使用运动数据和面元数据识别地图特征
Xie et al. Event-based stereo matching using semiglobal matching
US9188444B2 (en) 3D object positioning in street view
WO2023088127A1 (zh) 室内导航方法、服务器、装置和终端
KR20220169472A (ko) 센서 캘리브레이트 방법 및 장치, 전자 기기와 저장 매체
CN114550086A (zh) 一种人群定位方法及装置、电子设备和存储介质
JP7354528B2 (ja) 自律移動装置、自律移動装置のレンズの汚れ検出方法及びプログラム
US20240196107A1 (en) System and method for enhancing a collaborative camera installation experience
WO2022153910A1 (ja) 検出システム、検出方法、及びプログラム
Shuai et al. Multi-sensor Fusion for Autonomous Positioning of Indoor Robots
WO2024138851A1 (zh) 基于深度学习感知的多层级语义地图构建方法和装置
KR20240057297A (ko) 신경망 모델을 학습시키는 방법 및 전자 장치

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21945732

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 05.04.2024)