Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The autonomous positioning and map building method and device provided by the invention are suitable for the application scene shown in fig. 1, wherein the application scene comprises a robot 10, the robot 10 is a movable robot, and the robot refers to a machine with some artificial intelligence, such as a sweeping robot, a humanoid robot, an automatic driving automobile and the like. The robot 10 may need to move in an unknown environment in order to accomplish a user's task or otherwise. In order to realize autonomous positioning and navigation in the process of movement, an incremental map needs to be built, and the position of the incremental map is estimated.
For convenience, referring to fig. 2, a continuous motion of the robot 10 is divided into discrete time instants t 1, … k, each time instant being represented by xnIndicating the position of the robot 10 itself, i.e. x1,x2…xkWhich constitutes the motion trajectory of the robot 10. Suppose a map consists of many landmarks, such as y in the figure1、y2And y3At each time, the robot 10 observes a portion of the landmarks resulting in their range observation z (i.e., the distance between x and y). Positioning of the robot 10, i.e. estimating the position (x) of the robot 10 in a map, the robot 10 can passAnd estimating the position of the distance observation value of the landmark in the existing map. Map building, i.e., estimating, the location (y) of the landmark in the map, which may be obtained from the location (x) of the robot 10 and the distance observation (z). The positioning and mapping of the robot 10 is a continuous process, and as the position of the robot 10 changes, the robot 10 observes new landmarks and continuously adds new landmarks to the map.
Such as walls, windows, pillars, trees, buildings, tables, cabinets, flowers, signs, people, pets, vehicles, etc., among others. In some embodiments, the mobile nature of walls, windows, pillars, trees, buildings, etc. may be defined as "stationary objects," the mobile nature of tables, cabinets, flowers, signs, etc. as "movable objects," and the mobile nature of people, pets, vehicles, etc. as "moving objects. The robot 10 only adds the road signs belonging to the fixed objects into the map, and because the road signs are not easy to change, the situation that the road signs change when the robot 10 positions according to the road signs can be avoided, the influence of the surrounding environment on the robot positioning and map building can be reduced, and the positioning of the robot and the calculation of the road signs in the map are more accurate. It should be noted that the definition of the movement attribute may be defined in advance according to the application scenario of the robot 10, and not absolutely, the same object may be a "movable object" in some application scenarios, and may be a "fixed object" in other application scenarios.
Fig. 3 is a schematic flow chart of an autonomous positioning and mapping method provided by an embodiment of the present invention, which may be executed by the robot 10 in fig. 1, as shown in fig. 3, and the method includes:
101: and acquiring a distance observation value of the robot to the landmark point.
The distance observation of the robot 10 to the road signs may be based on a vision method, for example, obtaining an image in front of the robot vision by using a binocular camera or a depth camera, and then obtaining the distance between the robot 10 and each road sign by obtaining depth information of each pixel point in the image. In other embodiments, the robot may measure the distance between the robot 10 and the road sign by other methods.
Taking an example of obtaining a distance observation value by a binocular camera based on a vision method, referring to fig. 4, the robot 10 obtains a distance observation value for a road sign, and includes:
1011: and respectively acquiring a first image and a second image in a visual range through a binocular camera.
That is, a first image is obtained by a camera located at the left side of the robot 10, and a second image is obtained by a camera located at the right side of the robot 10, wherein the left camera and the right camera may be disposed at the left eye and the right eye of the robot 10, respectively.
1012: and respectively carrying out image recognition on the first image and the second image, recognizing the category of each region in the image, determining the movement attribute according to the category, and marking the category and the movement attribute for each region, wherein the movement attribute comprises a moving object, a movable object and a fixed object.
Specifically, the first image and the second image are subjected to image recognition, and for example, a neural network model based on deep learning can be used for recognition, so as to identify the category of each object in the image. Meanwhile, the movement attribute of each object in the image can be determined according to the movement attribute defined in advance by the object type, and the type and the movement attribute (such as a moving object, a movable object and a fixed object) of the pixel point of the corresponding region of the object are marked. For example, it is recognized through image recognition that the object types in the first image are respectively a table, a person, a wall, and the like, and according to the definition of the table, the person, and the wall in advance, the movement attributes thereof are respectively a movable object, a moving object, and a fixed object, then the pixel point label type and the movement attribute corresponding to the region of the table in the image are "table" and "movable object", the pixel point label type and the movement attribute corresponding to the region of the person in the image are "person" and "moving object", and the pixel point label type and the movement attribute corresponding to the region of the wall in the image are "wall" and "fixed object". It will be appreciated by those skilled in the art that in the actual application of labeling pixel points with categories and movement attributes, computer symbols representing actual categories and actual movement attributes may be employed rather than actual categories and actual movement attributes themselves. Wherein the movement attribute of each object may be defined in advance according to the application scenario of the robot 10.
1013: feature points are extracted based on the first image and the second image, and feature points belonging to a moving object are removed.
Specifically, feature points are extracted from each pixel point of the first image and the second image, and an algorithm such as SIFT or ORB may be adopted. The feature points are generally some "stable points" in the image, and will not disappear due to the change of the viewing angle, the change of the illumination, and the interference of noise, such as corner points, edge points, bright points in dark areas, dark points in bright areas, and the like. After the feature point extraction, the feature points marked as moving objects in the feature points may be removed. Because the moving objects have a high probability of moving, if the robot 10 is used as a reference for positioning, the robot 10 will have inaccurate position. Therefore, in this step, the signposts whose movement attribute is a moving object can be rejected.
In other embodiments, after each region in the image is identified, the region of the moving object to which the moving attribute belongs may be directly masked. In this way, when feature points are extracted, feature points located in the masked region are not extracted, so that the extracted feature points do not include feature points whose moving attributes are moving objects.
1014: and performing feature point matching on the basis of the first image and the second image after the feature points are removed, wherein the feature point matching is performed among the feature points with the same category, so as to obtain a distance observation value of a landmark point with a moving attribute of a non-moving object in the image.
Specifically, feature point matching may be performed based on, for example, a stereo matching algorithm, and matching may be performed between feature points having the same category. For example, feature point matching is performed between the category and a feature point belonging to "table", or between the category and a feature point belonging to "wall". Therefore, the matching range can be reduced, and the effectiveness of the matching result is improved. After matching the feature points, the matching result may be used to calculate the parallax of a point on the first image and the second image according to the triangulation principle to determine the depth of the feature point, i.e., the distance from the robot 10 to the feature point.
102: and acquiring the position of the robot in a map.
103: and adding a new road mark point belonging to a fixed object in the road mark points into the map, and obtaining the pose of the new road mark point according to the position and the distance observation value.
Referring to fig. 2, when the robot 10 starts a movement, the starting point (x) of the movement of the robot 10 may be set1) Arranged as dots, where the robot can observe the landmark y1And y2Suppose road sign y1And y2All the moving attributes of the road sign are fixed objects, namely the road sign y1And y2Added to the map. Since the robot 10 is now at a circular point, there is a road sign y according to the robot's alignment to the road sign1And y2Is observed value z of1And z2The road sign y can be obtained1And y2Pose in the map.
When the robot 10 moves to position x2In time, the robot 10 is facing the road sign y1And y2A distance observed value of z1' and z2', according to z1' and z2' on map (x)1A map of location acquisition) for location searching (i.e., positioning). When a certain position in the map is searched, the distance between the position and each landmark in the map is called a positioning distance, the distance observation value of the robot 10 actually corresponding to each corresponding landmark is called an observation distance, and if the coincidence degree of the positioning distance conforming to the observation distance is greater than a preset threshold value, the position is considered as x2An estimated location of the location.
Only road sign y is shown in fig. 21And y2Actually, each landmark includes a plurality of feature points (landmark points). The above-described matching degree is actually calculated to a degree that the positioning distance of each landmark point in the position or map matches the actual observation distance of the robot 10 for each corresponding landmark point. Example (b)If the preset threshold is 70, the landmark point with the positioning distance meeting the observation distance is marked as 1, and the landmark point with the positioning distance not meeting the observation distance is marked as 0. If more than 70 waypoints match the actual observed distance from the location, the location may be considered to be the location of the robot 10 in the map. Otherwise, a new location needs to be searched again.
In which determining whether the positioning distance of the landmark point meets the observation distance needs to be based on the corresponding landmark point, and therefore, before calculating the degree of the meeting, the correspondence between the landmark point observed by the robot 10 and the landmark point in the map needs to be matched. Feature point matching may be performed on landmark points observed by the robot 10 and landmark points in the map to determine corresponding landmark points in the map for landmark points observed by the robot. In some embodiments, when adding each landmark point into the map, the category to which each landmark point belongs may be marked, and the feature point matching may be performed between feature points of the same category, so as to reduce the matching range and improve the validity of the matching result.
In some embodiments, when adding each landmark point to the map, a moving weight value may also be marked for each landmark point. For example, although trees and building movement attributes are fixed objects as road signs in a map, buildings are relatively less prone to movement, and therefore, the movement weight value of a building may be set to be greater than that of a tree. For example, the movement weight value of a building is set to 3, and the movement weight value of a tree is set to 2. Correspondingly, the matching degree can also be obtained by combining the moving weight value corresponding to each landmark point in the map. In some embodiments, the degree of conformity and the mobile weight value are positively correlated. Also as explained above, assume road sign y1Has a moving weight value of 3, signpost y2Is 1, the preset threshold is again 70. If the signpost y1In which there are 20 characteristic points whose locating distance is in accordance with observation distance, signpost y2If there are 15 feature points whose location distance matches the observation distance, the matching degree is 20 × 3+15 × 1 ═ 75>70, the position is successfully located. If the signpost y1In which 8 positioning distances coincide with observationCharacteristic points of distance, road sign y245 feature points whose locating distance matches the observation distance have coincidence degree of 8 × 3+45 × 1 ═ 69<This position location fails and a new position needs to be searched in the map again 70.
Wherein, in some embodiments, in order to reduce the search range, the operation amount is reduced. It is also possible to estimate the displacement of the robot 10 by a detection device such as a sensor, estimate the position of the robot 10, and search for a position within a certain range around the estimated position on the map. Specifically, the displacement of the robot 10 between the first time and the second time is estimated, feature point matching may be performed on landmark point depth maps obtained at the first time and the second time, and the displacement of the robot 10 may be obtained by the depth of the same feature point in different depth maps. The displacement can be further filtered and fused with attitude information provided by an Inertial Measurement Unit (IMU).
When the robot 10 moves to position x4When the robot observes a new landmark y at this position3If the signpost y3If it also belongs to a fixed object, the road sign y will be marked3Added to the map. Road sign y3X obtained from position location of pose in map4Position of and the robot 10 is facing the landmark y at this position3Is obtained.
Because each road sign consists of a plurality of characteristic points, the characteristic points added into the map and corresponding to the road sign are pieced together to form the map.
According to the autonomous positioning and map building method provided by the embodiment of the invention, only the landmark points belonging to the fixed object are added into the map, so that the situation that the robot changes the landmark when referring to the landmark is avoided, and the influence of the surrounding environment on the positioning of the robot and the building of the map is reduced, so that the positioning of the robot and the calculation of the landmark in the map are more accurate.
In other embodiments, referring to fig. 5, the autonomous positioning and map building method further includes, in addition to the steps 101 and 103, the steps 104:
and if the position is coincident with the historical position, correcting the position and the position of the landmark point obtained between the historical position and the position according to the position of the landmark point obtained from the historical position.
In order to correct an error due to drift, a map having uniform information is obtained. The robot 10 will periodically check whether the current position is a history position visited before, i.e. perform loop back detection. If the current position coincides with the historical position, the current position and a map obtained from the historical position to the current position are corrected based on the map obtained at the historical position. Since the influence of the change of the road sign is eliminated when the robot 10 is positioned, the robot 10 can more easily find the historical position at the time of the loop detection.
Accordingly, an embodiment of the present invention further provides an autonomous positioning and map building apparatus, where the autonomous positioning and map building apparatus is used for the robot 10 shown in fig. 1, and as shown in fig. 6, the autonomous positioning and map building apparatus 600 includes:
an observation distance obtaining module 601, configured to obtain a distance observation value of the robot for a landmark point;
a positioning module 602, configured to obtain a position of the robot in a map;
and the map building module 603 is configured to add a new landmark point belonging to a fixed object in the landmark points to the map, and obtain a pose of the new landmark point according to the position and the distance observation value.
The autonomous positioning and map building device provided by the embodiment of the invention only adds the road signs belonging to the fixed objects into the map, so that the condition that the road signs change when the robot refers to the road signs is avoided, and the influence of the surrounding environment on the positioning and map building of the robot is reduced, thereby ensuring that the positioning of the robot and the calculation of the road signs in the map are more accurate.
In some embodiments of the autonomous positioning and map building apparatus 600, the observation distance obtaining module 601 is specifically configured to:
respectively acquiring a first image and a second image in a visual range through a binocular camera device;
respectively carrying out image recognition on the first image and the second image, recognizing the category of each region in the images, determining a movement attribute according to the category, and marking the category and the movement attribute for each region, wherein the movement attribute comprises a moving object, a movable object and a fixed object;
extracting feature points based on the first image and the second image, and removing the feature points belonging to a moving object;
and performing feature point matching on the basis of the first image and the second image after the feature points are removed, wherein the feature point matching is performed among the feature points with the same category, so as to obtain a distance observation value of a landmark point with a moving attribute of a non-moving object in the image.
Specifically, in some embodiments, the mapping module 603 is specifically configured to:
and adding a new road mark point belonging to a fixed object in the road mark points into the map, and marking the category and the mobile weight value of the new road mark point.
Specifically, in some embodiments, the positioning module 602 is specifically configured to:
carrying out feature point matching on the landmark points observed by the robot at the second moment and the landmark points in the map at the first moment, and determining the landmark points corresponding to the landmark points observed by the robot at the second moment in the map at the first moment, wherein the feature point matching is carried out among the feature points with the same category;
searching the position in the map at the first moment according to the distance observation value of the robot to the landmark point at the second moment;
and combining the moving weight value of the landmark points in the map to obtain the coincidence degree of the positioning distance and the observation distance of the robot at a certain position in the map, if the coincidence degree exceeds a preset threshold value, positioning the position as the position of the robot in the map, wherein the positioning distance is the distance between the position and each landmark point in the map, and the observation distance is the distance observation value of each corresponding landmark point of the robot.
In other embodiments of the autonomous positioning and mapping apparatus 600, please refer to fig. 7, which further comprises:
a loop detection module 604, configured to correct the position according to the landmark point pose obtained at the historical position and the landmark point pose obtained between the historical position and the position if the position coincides with the historical position.
It should be noted that the autonomous positioning and map building apparatus can execute the autonomous positioning and map building method provided by the embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method. For technical details that are not described in detail in the embodiment of the autonomous positioning and map building apparatus, reference may be made to the autonomous positioning and map building method provided in the embodiment of the present invention.
Fig. 8 is a schematic diagram of a hardware structure of a robot 10 according to an embodiment of the present invention, and as shown in fig. 8, the robot 10 includes:
one or more processors 11 and a memory 12, with one processor 11 being an example in fig. 8.
The processor 11 and the memory 12 may be connected by a bus or other means, and fig. 8 illustrates the connection by a bus as an example.
The memory 12, as a non-volatile computer-readable storage medium, may be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as program instructions/modules corresponding to the autonomous positioning and map building method in the embodiment of the present invention (for example, the observed distance obtaining module 601, the positioning module 602, and the map building module 603 shown in fig. 6). The processor 11 executes various functional applications of the robot and data processing, i.e., the autonomous positioning and map building method of the above-described method embodiment, by running non-volatile software programs, instructions, and modules stored in the memory 12.
The memory 12 may include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created from use of the autonomous positioning and map building apparatus, and the like. Further, the memory 12 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid state storage device. In some embodiments, the memory 12 optionally includes memory located remotely from the processor 11, which may be connected to the autonomous positioning and mapping apparatus via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The one or more modules are stored in the memory 12 and, when executed by the one or more processors 11, perform the autonomous positioning and mapping method of any of the above-described method embodiments, e.g., performing the method steps 101-103 of fig. 3, 1011-1014 of fig. 4, 101-104 of fig. 5, described above; the functions of the modules 601 and 603 in fig. 6 and the modules 601 and 604 in fig. 7 are realized.
The product can execute the method provided by the embodiment of the invention, and has corresponding functional modules and beneficial effects of the execution method. For technical details that are not described in detail in this embodiment, reference may be made to the method provided by the embodiment of the present invention.
Embodiments of the present invention provide a non-transitory computer-readable storage medium storing computer-executable instructions, which are executed by one or more processors, such as one processor 11 in fig. 8, to enable the one or more processors to perform the autonomous positioning and map building method in any of the above method embodiments, such as performing the above-described method steps 101 to 103 in fig. 3, method steps 1011 to 1014 in fig. 4, and method steps 101 to 104 in fig. 5; the functions of the modules 601 and 603 in fig. 6 and the modules 601 and 604 in fig. 7 are realized.
The above-described embodiments of the apparatus are merely illustrative, and the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment.
Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented by software plus a general hardware platform, and certainly can also be implemented by hardware. It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware related to instructions of a computer program, which can be stored in a computer readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. The storage medium may be a magnetic disk, an optical disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), or the like.
Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; within the idea of the invention, also technical features in the above embodiments or in different embodiments may be combined, steps may be implemented in any order, and there are many other variations of the different aspects of the invention as described above, which are not provided in detail for the sake of brevity; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and the modifications or the substitutions do not make the essence of the corresponding technical solutions depart from the scope of the technical solutions of the embodiments of the present invention.