CN115830095A

CN115830095A - Point cloud generation method, point cloud generation device, electronic equipment, point cloud generation system and computer storage medium

Info

Publication number: CN115830095A
Application number: CN202111082258.7A
Authority: CN
Inventors: 沈俊杰
Original assignee: Zte Nanjing Co ltd; ZTE Corp
Current assignee: Zte Nanjing Co ltd; ZTE Corp
Priority date: 2021-09-15
Filing date: 2021-09-15
Publication date: 2023-03-21
Also published as: WO2023040433A1

Abstract

The present disclosure provides a point cloud generating method, comprising: determining new image information to be updated according to the current image information and the historical image information of the area to be updated; the historical mapping information is used for generating a historical point cloud of an area including the area to be updated; determining current mapping information according to the mapping information to be updated and the historical mapping information; and generating point cloud according to the current mapping information. The method can save manpower and material resources and improve the point cloud generation efficiency. The present disclosure also provides a point cloud generating device, an electronic apparatus, a point cloud generating system, and a computer storage medium.

Description

Point cloud generation method, point cloud generation device, electronic equipment, point cloud generation system and computer storage medium

Technical Field

The present disclosure relates to the field of computer vision technologies, and in particular, to a point cloud generating method, a point cloud generating apparatus, an electronic device, a point cloud generating system, and a computer storage medium.

Background

Motion Structure From Motion (SFM) is an important link in image-based three-dimensional reconstruction technology, and a visual point cloud model of a scene can be recovered through three-dimensional vision, multi-view geometry and other technologies. By utilizing the image and the point cloud and solving a N-point perspective problem (PnP), the effective estimation of the camera pose can be realized, the specific coordinate and orientation information of the camera held by the user in the scene can be acquired, and the method has wide application in the field of Augmented Reality (AR). However, after a large scene (such as an airport, a railway station, a commercial complex, etc.) is reconstructed, the scene is changed due to certain irresistible factors (such as changing shops, painting walls, etc.), and at this time, a lot of manpower and material resources are consumed when the point cloud of the area including the scene is updated.

Disclosure of Invention

In view of the above-mentioned shortcomings in the prior art, the present disclosure provides a point cloud generating method, a point cloud generating device, an electronic apparatus, a point cloud generating system, and a computer storage medium.

In a first aspect, an embodiment of the present disclosure provides a point cloud generating method, including:

determining new image information to be updated according to the current image information and the historical image information of the area to be updated; the historical mapping information is used for generating a historical point cloud of an area including the area to be updated;

determining current mapping information according to the mapping information to be updated and the historical mapping information;

and generating point cloud according to the current mapping information.

Optionally, the historical mapping information includes a historical key frame set, the mapping information to be updated includes a key frame set to be updated, the current mapping information includes a current key frame set, key frames in the key frame set are arranged according to a time sequence, and the key frames are all image frames selected from the image information according to a preset key frame selection strategy; the determining the current mapping information according to the to-be-updated mapping information and the historical mapping information comprises:

determining two key frames which are respectively matched with the key frame at the first preset position and the key frame at the second preset position in the key frame set to be updated from the historical key frame set;

and replacing the key frame between the two determined key frames in the historical key frame set with the key frame in the key frame set to be updated to obtain the current key frame set.

Optionally, a key frame in a first preset position in the key frame set to be updated is a first key frame in the key frame set to be updated, and a key frame in a second preset position in the key frame set to be updated is a last key frame in the key frame set to be updated.

Optionally, the determining, from the historical keyframe set, two keyframes respectively matched with the keyframe in the to-be-updated keyframe set at the first preset position and the keyframe in the second preset position includes:

extracting features of each key frame in the historical key frame set, the key frame in the first preset position in the key frame set to be updated and the key frame in the second preset position in the key frame set to be updated to obtain a feature description subset of each key frame in the historical key frame set, a feature description subset of the key frame in the first preset position in the key frame set to be updated and a feature description subset of the key frame in the second preset position in the key frame set to be updated;

performing feature matching according to a feature description subset of each key frame in the historical key frame set, a feature description subset of a key frame at a first preset position in the to-be-updated key frame set and a feature description subset of a key frame at a second preset position in the to-be-updated key frame set to determine key frames of which feature descriptors meet a first preset condition and key frames of which feature descriptors meet a second preset condition in the historical key frame set;

and determining key frames matched with key frames positioned at a first preset position in the key frame set to be updated from the key frames of which the feature descriptors accord with a first preset condition, and determining key frames matched with key frames positioned at a second preset position in the key frame set to be updated from the key frames of which the feature descriptors accord with a second preset condition.

Optionally, the key frames with the feature descriptors meeting the first preset condition include N key frames with the maximum number of feature descriptor matches with the key frames located at the first preset position in the to-be-updated key frame set, and the key frames with the feature descriptors meeting the second preset condition include N key frames with the maximum number of feature descriptor matches with the key frames located at the second preset position in the to-be-updated key frame set.

Optionally, the determining, from the key frames whose feature descriptors meet a first preset condition, a key frame that matches a key frame located at a first preset position in the key frame set to be updated, and determining, from the key frames whose feature descriptors meet a second preset condition, a key frame that matches a key frame located at a second preset position in the key frame set to be updated includes:

performing feature matching on the key frames of which the feature descriptors meet first preset conditions and the key frames located at a first preset position in the key frame set to be updated, performing feature matching on the key frames of which the feature descriptors meet second preset conditions and the key frames located at a second preset position in the key frame set to be updated, and determining initial matching feature points matched with feature points in the key frames located at the first preset position in the key frame set to be updated in the key frames of which the feature descriptors meet the first preset conditions and initial matching feature points matched with feature points in the key frames located at the second preset position in the key frame set to be updated in the key frames of which the feature descriptors meet the second preset conditions;

filtering the initial matching feature points to eliminate the initial matching feature points which are mismatched and determine the matching feature points in the key frames of which the feature descriptors accord with a first preset condition and the matching feature points in the key frames of which the feature descriptors accord with a second preset condition;

determining the key frame with the most matched feature points in the key frames with the feature descriptors meeting the first preset condition as the key frame matched with the key frame positioned at the first preset position in the key frame set to be updated, and determining the key frame with the most matched feature points in the key frames with the feature descriptors meeting the second preset condition as the key frame matched with the key frame positioned at the second preset position in the key frame set to be updated.

Optionally, the current mapping information further includes pose information of a key frame to be updated relative to a mapping software coordinate system, and the generating a point cloud according to the current mapping information includes:

extracting the characteristics of the key frames in the current key frame set to obtain Scale Invariant Feature Transform (SIFT) characteristic points of the key frames in the current key frame set;

performing feature matching according to the SIFT feature points of each key frame in the current key frame set, and determining matched SIFT feature points;

and carrying out triangularization processing according to the matched SIFT feature points and the pose information of the key frame to be updated relative to a mapping software coordinate system to generate point cloud.

Optionally, the performing feature matching according to the SIFT feature points of each key frame in the current key frame set includes:

and performing feature matching on any two key frames with the interframe space smaller than a preset threshold value in the current key frame set according to the SIFT feature points of each key frame in the current key frame set.

Optionally, the historical mapping information further includes historical mapping software mapping information, and the mapping information to be updated further includes current mapping information of the mapping software and pose information of the keyframe to be updated relative to a coordinate system of the mapping software;

the determining of the mapping information to be updated according to the current image information and the historical mapping information of the area to be updated comprises:

and determining the key frame set to be updated, the current map information of the mapping software and the pose information of the key frame to be updated relative to a mapping software coordinate system according to the current image information of the area to be updated and the historical map information of the mapping software.

Optionally, before determining the mapping information to be updated according to the current image information and the historical mapping information of the area to be updated, the method further includes the step of obtaining the current image information of the area to be updated:

acquiring the video file of the area to be updated from the starting position to the end position; the starting point position and the ending point position are both positioned in the area including the area to be updated and outside the area to be updated, and the vertical distance between the starting point position and the ending point position and the boundary of the area to be updated is larger than a preset threshold value;

and converting the video file into the current image information.

In another aspect, an embodiment of the present disclosure provides a point cloud generating apparatus, including:

the image information processing module to be updated is used for determining image information to be updated according to the current image information and the historical image information of the area to be updated; the historical mapping information is used for generating a historical point cloud of an area including the area to be updated;

the current mapping information processing module is used for determining current mapping information according to the mapping information to be updated and the historical mapping information;

and the point cloud generating module is used for generating a point cloud according to the current mapping information.

In another aspect, an embodiment of the present disclosure provides an electronic device, including:

one or more processors;

a storage device having one or more programs stored thereon;

the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the point cloud generation method as previously described.

In another aspect, an embodiment of the present disclosure provides a point cloud generating system, including:

the image acquisition device is used for acquiring current image information of the area to be updated; and

an electronic device as hereinbefore described.

In yet another aspect, the disclosed embodiments provide a computer storage medium having a computer program stored thereon, wherein the program when executed implements the point cloud generation method as described above.

According to the point cloud generating method provided by the embodiment of the disclosure, only current image information of an area to be updated is obtained through scanning, the image information to be updated of the area to be updated is determined by referring to historical image information, only a part corresponding to the area to be updated in the historical image information is updated by using the image information to be updated so as to obtain the current image information, and finally point cloud is generated directly according to the current image information. The currently generated point cloud has been updated in part corresponding to the region to be updated as compared to the historical point cloud. That is to say, the point cloud generating method provided by the embodiment of the disclosure can avoid rescanning the whole area including the area to be updated, thereby avoiding the point cloud from being generated again according to the image information obtained by rescanning, saving manpower and material resources, and improving the point cloud generating efficiency.

Drawings

Fig. 1 is a schematic flow diagram of a point cloud generation method provided by the present disclosure;

FIG. 2 is a flow chart diagram illustrating a method of determining current mapping information provided by the present disclosure;

FIG. 3 is a first flowchart illustrating a method for determining two keyframes according to the present disclosure;

FIG. 4 is a first flowchart illustrating a method for determining two keyframes according to the present disclosure;

FIG. 5 is a schematic flow chart diagram illustrating a method for generating a point cloud according to current mapping information provided by the present disclosure;

FIG. 6 is a second block diagram of a point cloud generating apparatus provided by the present disclosure;

FIG. 7 is a second block diagram of a point cloud generating apparatus provided by the present disclosure;

fig. 8 is a schematic structural diagram of an electronic device provided by the present disclosure;

FIG. 9 is a schematic structural diagram of a point cloud generation system provided by the present disclosure;

fig. 10 is a schematic diagram of a computer-readable storage medium provided by the present disclosure.

Detailed Description

Example embodiments will be described more fully hereinafter with reference to the accompanying drawings, but which may be embodied in different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

As used herein, the term "and/or" includes any and all combinations of one or more of the associated listed items.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and/or "comprising," 8230; \8230 "; when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.

Embodiments described herein may be described with reference to plan and/or cross-sectional views in idealized representations of the present disclosure. Accordingly, the example illustrations can be modified in accordance with manufacturing techniques and/or tolerances. Accordingly, the embodiments are not limited to the embodiments shown in the drawings, but include modifications of configurations formed based on a manufacturing process. Thus, the regions illustrated in the figures have schematic properties, and the shapes of the regions shown in the figures illustrate specific shapes of regions of elements, but are not intended to be limiting.

Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and the present disclosure, and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

At present, when a scene in a large area changes, if the point cloud of the large area needs to be updated, the whole large area needs to be rescanned, and then the point cloud of the large area is regenerated according to image information obtained by rescanning. The method consumes a large amount of manpower and material resources, and the point cloud generation efficiency is low. For this reason, the embodiments of the present disclosure propose to implement updating of the point cloud only according to the current image information of the area to be updated in the large area. Specifically, since the historical map creation information can be necessarily obtained when the historical point cloud of the large area is generated, the map information to be updated can be directly obtained by combining the current image information of the area to be updated on the basis of the historical map creation information, the current map creation information can be determined by comparing the map information to be updated with the historical map creation information, and finally the point cloud can be directly generated according to the current map creation information without rescanning the whole large area or re-generating the point cloud of the large area according to the image information of the whole large area obtained by rescanning, so that the purposes of saving manpower and material resources and improving the point cloud generation efficiency can be achieved.

As shown in fig. 1, an embodiment of the present disclosure provides a point cloud generating method, which may include the following steps:

in step S1, determining new map information to be updated according to current image information and historical map information of an area to be updated; the historical mapping information is used for generating historical point clouds of regions including the region to be updated.

In step S2, current mapping information is determined according to the mapping information to be updated and the historical mapping information.

In step S3, point clouds are generated according to the current mapping information.

The historical mapping information is generated when generating a historical point cloud of an area including an area to be updated (the area including the area to be updated is hereinafter referred to as a "large area"), and the historical mapping information of the large area can be acquired in advance when generating a current point cloud of the large area according to the embodiment of the disclosure. The historical mapping information may be obtained by image processing of the historical image information of the large area through mapping software, and may be understood as a project file related to the image processing.

According to the point cloud generating method provided by the embodiment of the disclosure, only current image information of a region to be updated is obtained through scanning, the to-be-updated map creating information of the region to be updated is determined by referring to the historical map creating information, then only a part, corresponding to the region to be updated, in the historical map creating information is updated by using the to-be-updated map creating information so as to obtain the current map creating information, and finally, the point cloud is directly generated according to the current map creating information, and compared with the historical point cloud, the part, corresponding to the region to be updated, of the currently generated point cloud is updated. That is to say, the point cloud generating method provided by the embodiment of the present disclosure can avoid rescanning the entire region including the region to be updated, thereby avoiding generating the point cloud again according to the image information obtained by rescanning, which not only saves manpower and material resources, but also improves the point cloud generating efficiency.

The historical map building information carries historical key frames of a large area, the large area comprises an area to be updated, and the historical map building information also necessarily carries the historical key frames of the area to be updated. And the information of the graph to be updated carries the current key frame of the area to be updated. Because the area to be updated has changed, the current key frame of the area to be updated is necessarily greatly different from the historical key frame, so that the embodiment of the present disclosure provides that only the portion, corresponding to the area to be updated, in the historical key frame of the large area can be updated, specifically, the historical key frame of the area to be updated in the historical key frame of the large area is replaced by the current key frame of the area to be updated, the current key frame of the large area is obtained, and thus the current mapping information of the large area is obtained.

Correspondingly, in some embodiments, the historical mapping information includes a historical key frame set, the mapping information to be updated includes a key frame set to be updated, the current mapping information includes a current key frame set, the key frames in the key frame set are arranged according to a time sequence, and the key frames are all image frames selected from the image information according to a preset key frame selection strategy. As shown in fig. 2, the determining of the current mapping information according to the to-be-updated mapping information and the historical mapping information (i.e., step S2) may include the following steps:

in step S21, two pieces of key frames respectively matched with the key frame located at the first preset position and the key frame located at the second preset position in the key frame set to be updated are determined from the historical key frame set.

In step S22, a key frame between two determined key frames in the historical key frame set is replaced by a key frame in the key frame set to be updated, so as to obtain a current key frame set.

The historical key frame set is a set of historical key frames of a large area, and the current key frame set is a set of current key frames of the large area. The key frame set to be updated comprises current key frames (namely, key frames carrying objects in the region to be updated) of the region to be updated, the key frames in the key frame set to be updated are arranged according to a time sequence, and the current key frames of the region to be updated are all positioned between the key frame at the first preset position and the key frame at the second preset position in the key frame set to be updated.

As described above, the historical mapping information may be obtained after image processing is performed on the historical image information of the large area by mapping software, and the embodiment of the present disclosure may use the same mapping software to process the current image information and the historical mapping information of the area to be updated to obtain the mapping information to be updated. Different mapping software has corresponding key frame selection strategies, and common key frame selection strategies may include that the number of feature points tracked by a current image frame is small, map points tracked in a previous key frame in the current image frame are few, and the like.

KF0 is used for representing a historical key frame set, KF1 is used for representing a key frame set to be updated, KF1-A is used for representing a key frame positioned at a first preset position in KF1, KF1-B is used for representing a key frame positioned at a second preset position in KF1, KF0-A is used for representing a key frame matched with KF1-A in KF0, KF0-B is used for representing a key frame matched with KF1-B in KF0, KF2 is used for representing a current key frame set, key frames in KF0, KF1 and KF2 are arranged according to a time sequence, all key frames positioned between KF0-A and KF0-B in KF0 are integrally replaced by all key frames in KF1, and KF2 can be obtained.

Two key frames respectively matched with the key frame positioned at a first preset position and the key frame positioned at a second preset position in the key frame set to be updated are found from the historical key frame set, and then the key frame positioned between the two determined key frames in the historical key frame set is replaced by the key frame in the key frame set to be updated, so that the current key frame set comprising the historical key frame of the region which is not to be updated and the current key frame of the region to be updated is obtained. The current mapping information can be obtained only by updating the part of the historical key frame set corresponding to the area to be updated, so that the method is simple and convenient, and is beneficial to improving the point cloud generation efficiency.

Because the area to be updated changes, the key frames in the key frame set to be updated usually have a larger difference compared with the key frames at the same corresponding positions in the historical key frame set, if it is determined that the key frame of the object in the area to be updated begins to appear first and the key frame of the object in the area to be updated appears last in the key frame set to be updated, and then each key frame in the historical key frame set is respectively matched with the key frame of the object in the area to be updated beginning to appear first and the key frame of the object in the area to be updated appearing last in the key frame set to be updated, then it is likely that the matching fails and the operation is burdensome. In the embodiment of the present disclosure, scanning is usually started from outside the region to be updated to obtain the current image information of the region to be updated, and then, in addition to the current key frames of the region to be updated, there are also current key frames of regions not to be updated in the frame set to be updated, and these current key frames of regions not to be updated can be matched with the key frames in the historical key frame set successfully. Therefore, in order to match the key frames in the historical key frame set with the key frames in the key frame set to be updated successfully, and also in order to simplify the operation and save the time, the key frames in the historical key frame set can be directly matched with the first key frame and the last key frame in the key frame set to be updated.

Correspondingly, in some embodiments, a key frame in a first preset position in the set of key frames to be updated is a first key frame in the set of key frames to be updated, and a key frame in a second preset position in the set of key frames to be updated is a last key frame in the set of key frames to be updated.

In order to determine two key frames respectively matched with a key frame at a first preset position and a key frame at a second preset position in a key frame set to be updated from a historical key frame set, each key frame in the historical key frame set is generally required to be matched with the key frame at the first preset position or the key frame at the second preset position one by one.

Accordingly, in some embodiments, as shown in fig. 3, the determining two key frames respectively matching the key frame located at the first preset position and the key frame located at the second preset position in the to-be-updated key frame set from the historical key frame set (i.e., step S21) may include the following steps:

in step S211, feature extraction is performed on each key frame in the historical key frame set, the key frame located at the first preset position in the to-be-updated key frame set, and the key frame located at the second preset position in the to-be-updated key frame set, so as to obtain a feature description subset of each key frame in the historical key frame set, a feature description subset of the key frame located at the first preset position in the to-be-updated key frame set, and a feature description subset of the key frame located at the second preset position in the to-be-updated key frame set.

In step S212, feature matching is performed according to the feature description subset of each key frame in the historical key frame set, the feature description subset of the key frame located at the first preset position in the to-be-updated key frame set, and the feature description subset of the key frame located at the second preset position in the to-be-updated key frame set, so as to determine the key frame whose feature descriptor meets the first preset condition and the key frame whose feature descriptor meets the second preset condition in the historical key frame set.

In step S213, a key frame matching the key frame located at the first preset position in the key frame set to be updated is determined from the key frames whose feature descriptors meet the first preset condition, and a key frame matching the key frame located at the second preset position in the key frame set to be updated is determined from the key frames whose feature descriptors meet the second preset condition.

The feature extraction refers to extracting image information and determining whether each point of an image belongs to an image feature. The feature descriptor is an array vector for describing feature points in an image, and generally describes pixel information around the feature points. Feature points, i.e. points or blocks with scale invariance, can represent an image or object in an identical or at least very similar invariant form in other similar images containing the same scene or object. Feature matching refers to an image matching method in which features extracted from an image are used as conjugate entities, extracted feature attributes or description parameters (actually, features of the features, which may also be considered as features of the image) are used as matching entities, and similarity measures between the matching entities are calculated to realize registration of the conjugate entities.

Firstly, feature extraction can be performed on each key frame, KF1-A and KF1-B in KF0 by using a feature extraction algorithm to obtain a feature description subset of each key frame, a feature description subset of KF1-A and a feature description subset of KF1-B in KF0, wherein the feature description subset comprises SIFT (Scale-invariant feature transform) feature points and descriptors. And then, SIFT feature points and descriptors of KF1-A and SIFT feature points and descriptors of KF1-B can be respectively matched with SIFT feature points and descriptors of each key frame in KF0, and as the SIFT feature points and descriptors of the key frames matched with KF1-A or KF1-B in KF0 certainly meet certain conditions, preliminary screening can be performed through preset conditions, namely, the key frames with SIFT feature points and descriptors meeting first preset conditions and the key frames with SIFT feature points and descriptors meeting second preset conditions are determined from KF 0. However, the key frames in the KF0 whose SIFT feature points and descriptors meet the preset condition are not necessarily the key frames that match the key frames located at the preset position in the KF1-a or KF1-B, and therefore, it is also necessary to determine the key frame that matches the key frame located at the first preset position from the key frames whose feature descriptors meet the first preset condition, and determine the key frame that matches the key frame located at the second preset position from the key frames whose feature descriptors meet the second preset condition.

The method comprises the steps of directly extracting a feature description subset of each key frame in a historical key frame set, a feature description subset of the key frame located at a first preset position and a feature description subset of the key frame located at a second preset position, performing feature matching according to the feature description subset, preliminarily screening out key frames of which feature descriptors meet first preset conditions and key frames of which feature descriptors meet second preset conditions from the historical key frame set, and further determining key frames matched with the key frames located at the first preset position and key frames matched with the key frames located at the second preset position from the key frames of which feature descriptors meet the first preset conditions and the key frames of which feature descriptors meet the second preset conditions.

The more the number of matched SIFT feature points and descriptors between the key frame in the KF0 and the KF1-A or the KF1-B is, the higher the matching possibility between the key frame and the KF1-A or the KF1-B is, and therefore, the preliminary screening condition can be determined according to the matching number of the feature descriptors. Correspondingly, in some embodiments, the key frames whose feature descriptors meet the first preset condition include N key frames whose feature descriptors match the key frames located at the first preset position in the key frame set to be updated, and the key frames whose feature descriptors meet the second preset condition include N key frames whose feature descriptors match the key frames located at the second preset position in the key frame set to be updated.

The embodiment of the present disclosure does not specifically limit how to specifically determine the N frames of key frames with the largest number of matching feature descriptors, for example, the matching number of the feature descriptors may be counted by a statistical histogram, the matching number of the feature descriptors is sorted in a descending order, the matching number values of the feature descriptors in the top N bits of the ranking are obtained, and the key frames corresponding to the matching number values of the feature descriptors in the top N bits of the ranking are determined. The matching number of the feature descriptors can be sorted from small to large, the feature descriptor matching number value ranked at the last N bits is obtained, and the key frames corresponding to the feature descriptor matching number value at the last N bits are determined. The matching number of the feature descriptors can be counted by using a pie statistical chart, the first N feature descriptor matching number values with the largest sector area are obtained, and key frames corresponding to the first N feature descriptor matching number values are determined. It should be noted that, in the embodiment of the present disclosure, the value of N is not specifically limited, for example, N may be 2, 3, 4, and the like. However, the value of N may be set to 2 for the purpose of further reducing the amount of computation.

After the key frames with the feature descriptors meeting the preset conditions are preliminarily screened from the historical key frame set, because the number of the key frames with the feature descriptors meeting the preset conditions is small, the key frames with the feature descriptors meeting the preset conditions and the key frames at the preset positions in the key frame set to be updated can be directly subjected to feature matching at this moment, so as to determine feature points (namely initial matching feature points) which are matched with the feature points in the key frames at the preset positions in the key frame set to be updated in the key frames with the feature descriptors meeting the preset conditions. Whether the key frames are matched is generally determined according to the number of matched feature points among the key frames, and in order to further improve the accuracy of a matching result, wrong initial matched feature points can be filtered.

Accordingly, in some embodiments, as shown in fig. 4, the determining, from the key frames whose feature descriptors meet the first preset condition, a key frame that matches a key frame located at a first preset position in the key frame set to be updated, and determining, from the key frames whose feature descriptors meet the second preset condition, a key frame that matches a key frame located at a second preset position in the key frame set to be updated (i.e., step S213) may include the following steps:

in step S2131, feature matching is performed on the keyframes with the feature descriptors meeting the first preset condition and the keyframes located at the first preset position in the set of keyframes to be updated, and feature matching is performed on the keyframes with the feature descriptors meeting the second preset condition and the keyframes located at the second preset position in the set of keyframes to be updated, and an initial matching feature point, which is matched with the feature point in the keyframe located at the first preset position in the set of keyframes to be updated in the keyframes with the feature descriptors meeting the first preset condition, and an initial matching feature point, which is matched with the feature point in the keyframe located at the second preset position in the set of keyframes to be updated in the feature descriptors meeting the second preset condition, are determined.

In step S2132, the initial matching feature points are filtered to eliminate the initial matching feature points that are mismatched by mistake and determine matching feature points in the keyframes whose feature descriptors satisfy the first preset condition and matching feature points in the keyframes whose feature descriptors satisfy the second preset condition.

In step S2133, the keyframe with the most matched feature points in the keyframes whose feature descriptors meet the first preset condition is determined as the keyframe matched with the keyframe located at the first preset position in the set of keyframes to be updated, and the keyframe with the most matched feature points in the keyframes whose feature descriptors meet the second preset condition is determined as the keyframe matched with the keyframe located at the second preset position in the set of keyframes to be updated.

Firstly, determining initial matching feature points in a key frame with a feature descriptor conforming to a first preset condition and initial matching feature points in a key frame with a feature descriptor conforming to a second preset condition, and then filtering the initial matching feature points by using a RANSAC (Random Sample Consensus) algorithm to remove wrong feature points in the initial matching feature points, wherein the remaining initial matching feature points are all matching feature points. And finally, the key frame with the most matched feature points in the key frames with the feature descriptors meeting the first preset condition is the key frame matched with the key frame positioned at the first preset position, and the key frame with the most matched feature points in the key frames with the feature descriptors meeting the second preset condition is the key frame matched with the key frame positioned at the second preset position. The method for determining the matched key frame from the key frames of which the feature descriptors meet the preset conditions can further improve the accuracy of the matching result.

Feature matching is a necessary step in the process of generating a point cloud according to mapping information, feature points are used for describing features in an image frame, commonly used feature points include SIFT feature points and ORB (Oriented FAST and Rotated) feature points, the feature description capability of the SIFT feature points is stronger than that of the ORB feature points, and the computing speed is higher when the ORB feature points are used for computing, so that the real-time performance is guaranteed. In the embodiment of the disclosure, mapping software is used when determining the mapping information to be updated according to the current image information and the historical mapping information of the area to be updated, the mapping software has a high requirement on real-time performance, so ORB feature points are selected, but three-dimensional reconstruction software is used when generating point clouds according to the current mapping information, and the three-dimensional reconstruction software has a low requirement on real-time performance, so that SIFT feature points of key frames in a current key frame set can be extracted for feature matching, so as to improve the accuracy of matching results. When the mapping software is used for processing the current image information and the historical mapping information of the area to be updated, besides the key frame set to be updated, the pose information of the key frame to be updated relative to the mapping software coordinate system is also determined, the matched SIFT feature points and the pose information of the key frame to be updated relative to the mapping software coordinate system are directly used for triangularization, and then 3D (3 Dimensions, three-dimensional) map points (namely point clouds) can be restored, and the generated point clouds have reliability, and the key frame does not need to be triangulated and then optimized by using bundle adjustment so as to improve the reliability of the point clouds.

Accordingly, in some embodiments, the current mapping information further includes pose information of the keyframe to be updated with respect to the mapping software coordinate system, and as shown in fig. 5, the generating a point cloud according to the current mapping information (i.e., step S3) may include the following steps:

in step S31, feature extraction is performed on the key frames in the current key frame set to obtain SIFT feature points of each key frame in the current key frame set.

In step S32, feature matching is performed according to the SIFT feature points of each key frame in the current key frame set, and matched SIFT feature points are determined.

In step S33, triangularization is performed according to the matched SIFT feature points and the pose information of the keyframe to be updated relative to the mapping software coordinate system, so as to generate a point cloud.

The pose information is information for describing the position and the posture of the object, and may include a rotation matrix, a translation matrix, and the like. Triangulation is an image processing method that recovers the 3D coordinates of feature points from their projection under a camera. The point cloud generation according to the current mapping information in the embodiment of the present disclosure is an SFM (Structure from Motion) process, and mainly includes three steps of feature extraction, feature matching, and Motion Structure restoration with a known pose. The SIFT feature points are extracted for feature matching, and the matched SIFT feature points and pose information of a key frame to be updated relative to a mapping software coordinate system are utilized for triangularization, so that the accuracy of a matching result and the reliability of point cloud can be improved.

At present, in the SFM process, a full Match (extreme Match) mode is generally adopted to Match key frames, specifically, if N key frames are supposed, the key frame of the 1 st frame needs to be matched with the key frame of the 2 nd frame and the key frame of the 3 rd frame respectively (8230); the key frame of the Nth frame needs to be matched with the key frame of the 2 nd frame and the key frame of the 4 th frame respectively (82308230); the key frame of the 823030of the 3 rd frame and the key frame of the 4 th frame and the key frame of the 5 th frame respectively (82308230); the key frame of the N8230), therefore, when the number N of the frames is large, the full Match mode has extremely low efficiency. In the embodiment of the invention, the current key frame set comprises current key frames of a large area, the number of the key frames is large, the key frames are obviously not suitable for matching the key frames in a full matching mode, and the key frames in the key frame set are arranged according to the time sequence, so that a sequence matching (Sequential Match) mode can be selected for matching the key frames when the feature matching is carried out, namely, the feature matching can be carried out on any two key frames with the inter-frame interval meeting a certain condition in the current key frame set.

Accordingly, in some embodiments, the performing feature matching according to the SIFT feature points of each key frame in the current key frame set (i.e. step S32) may include the following steps: and performing feature matching on any two key frames with the interframe interval smaller than a preset threshold value in the current key frame set according to the SIFT feature points of the key frames in the current key frame set.

Specifically, assuming that the preset threshold is 3, the two keyframes in the current keyframe set whose interframe space is less than or equal to the preset threshold may include: any two key frames with the inter-frame interval of 0, 1 and 2. The method can match the 1 st key frame in the current key frame with the 2 nd key frame, the 3 rd key frame and the 4 th key frame respectively, match the 2 nd key frame with the 1 st key frame, the 3 rd key frame, the 4 th key frame and the 5 th key frame respectively \8230, match the k th key frame with the k-3 rd key frame, the k-2 nd key frame, the k-1 th key frame, the k +2 th key frame and the k +3 th key frame respectively (k > 3). This way the efficiency of feature matching can be greatly improved.

The key frames are image frames selected from image information by using mapping software according to a preset key frame selection strategy, sufficient time sequence intervals are met among the key frames, and a good positioning effect is achieved.

The embodiment of the present disclosure does not specifically limit how the preset number is specifically set, for example, the preset number may be 3, 4, 5, and so on.

Only feature matching is performed on any two key frames with the interframe space being a preset number of frames in the current image information in the current key frame set, matching of key frames with an interframe space being too small is avoided, unnecessary operations are reduced, the matching times can be effectively reduced, and the matching duration can be shortened.

The mapping software can be used for directly processing the current image information of the area to be updated to obtain the mapping information to be updated, but the obtained mapping information to be updated and the historical mapping information of the large area cannot be kept under the same reference system, even if the mapping information to be updated is obtained, the mapping information to be updated and the historical mapping information cannot be simultaneously processed subsequently, and the point cloud of the whole large area cannot be directly generated only according to the mapping information to be updated. Therefore, the historical map information (map information of the mapping software, namely a map for realizing positioning formed by a key point sequence automatically generated by the mapping software) output by the mapping software at the same time of outputting the historical key frame set can be directly initialized according to the historical map information, and the map information to be updated is generated according to the current image information of the area to be updated on the basis of the historical map information, so that the map information to be updated and the historical map information can be ensured to be processed simultaneously subsequently.

In some embodiments, the historical mapping information further comprises historical mapping information of mapping software, and the mapping information to be updated further comprises current mapping information of the mapping software and pose information of a key frame to be updated relative to a coordinate system of the mapping software; the determining of the map information to be updated according to the current image information and the historical map information of the area to be updated (i.e. step S1) may include the following steps: and determining a key frame set to be updated, current map information of the mapping software and pose information of the key frame to be updated relative to a mapping software coordinate system according to the current image information of the area to be updated and the historical map information of the mapping software.

The historical mapping information can comprise a historical key frame set, historical mapping information of mapping software and pose information of historical key frames relative to a mapping software coordinate system, and the mapping information to be updated can comprise a key frame set to be updated, current map information of the mapping software and pose information of key frames to be updated relative to the mapping software coordinate system. The mapping software can be image processing software under the framework of vSLAM (Visual SLAM, visual simultaneous localization and mapping) and the like. After the image information is input into mapping software for processing, the mapping software can output information such as feature points, descriptors of the feature points, matching pairs among the descriptors, 3D map points restored according to the matching pairs, common views and the like in the key frames besides outputting key frame sets, pose information and mapping software map information, and the information can be used for visual positioning, scene structure restoration and the like. When determining the map information to be updated from the current image information and the history map information of the area to be updated using the SLAM framework, the program may select the positioning mode. The historical mapping information is obtained after processing the historical image information of a large area, in the process of determining the historical mapping information according to the historical image information by using an SLAM frame, a program usually selects a tracking mode, so that the historical image information can be processed by using self-carried time sequence attributes, a proper inter-frame interval to be tracked can be set according to the speed of a scanning area, a larger inter-frame interval can be selected to reduce the accumulation of calculation errors when the scanning speed is lower, and a smaller inter-frame interval can be selected to avoid the condition that the scene change is too large to cause the incapability of tracking when the scanning speed is higher.

It should be understood that mapping software used to determine map information to be updated according to current image information and historical map information of an area to be updated should maintain consistency of influencing factors such as version, setting parameters and the like with mapping software used to determine historical map information when generating historical point clouds of a large area.

Since the area to be updated is an area where a change has occurred, there is usually a large difference in image information of the area to be updated compared to before the change has occurred. When the current image information of the area to be updated is acquired, scanning can be started from the outside of the area to be updated, and the scanning can be performed on other areas except the area to be updated and the whole area to be updated, so that incomplete scanning of the area to be updated is avoided, and the complete image information of the area to be updated is acquired.

Correspondingly, in some embodiments, before determining the mapping information to be updated according to the current image information and the historical mapping information of the area to be updated, the method further includes the step of acquiring the current image information of the area to be updated: acquiring a video file of a region to be updated from a starting position to an end position; the starting point position and the end point position are both positioned in an area including an area to be updated and outside the area to be updated, and the vertical distance between the starting point position and the end point position and the boundary of the area to be updated is larger than a preset threshold value; and converting the video file into current image information.

The preset threshold may be 15 meters, 20 meters, and the like, which is not specifically limited in this disclosure. The area to be updated can be scanned by using an image acquisition device, the image acquisition device can be a camera, and in order to improve the image building speed, the image acquisition device can be a monocular camera such as a mobile phone camera, a single lens reflex camera, a panoramic camera with symmetrical fisheyes and the like. The regions with more non-pure colors and more shapes in the region to be updated have more abundant textures, and the regions with abundant textures can be emphatically scanned. When the area to be updated is scanned, the image acquisition device may be controlled to move forward along a curved route and keep shooting, and after a certain distance, the image acquisition device starts to move back to a certain point in the distance and starts to move back from the certain point. This may also ensure a complete scan of the area to be updated.

It should be understood that the image capturing device used to scan the area to be updated should maintain consistency of the model, setting parameters, shooting parameters (such as angle, speed, etc.), and other influencing factors with the image capturing device used to scan the large area when generating the historical point cloud of the large area.

Based on the same technical concept, as shown in fig. 6, an embodiment of the present disclosure further provides a point cloud generating apparatus, which may include:

the image creating information processing module 101 is configured to determine image creating information to be updated according to current image information and historical image creating information of an area to be updated; the historical mapping information is used for generating historical point clouds of regions including the region to be updated.

And the current mapping information processing module 102 is configured to determine current mapping information according to-be-updated mapping information and historical mapping information.

And the point cloud generating module 103 is used for generating a point cloud according to the current mapping information.

In some embodiments, the historical mapping information includes a historical key frame set, the mapping information to be updated includes a key frame set to be updated, the current mapping information includes a current key frame set, key frames in the key frame set are arranged according to a time sequence, and the key frames are all image frames selected from the image information according to a preset key frame selection strategy; the current mapping information processing module 102 is configured to:

In some embodiments, a key frame in a first preset position in the set of key frames to be updated is a first key frame in the set of key frames to be updated, and a key frame in a second preset position in the set of key frames to be updated is a last key frame in the set of key frames to be updated.

In some embodiments, the current mapping information processing module 102 is configured to:

feature extraction is carried out on each key frame in the historical key frame set, the key frame located at a first preset position in the key frame set to be updated and the key frame located at a second preset position in the key frame set to be updated, so that a feature description subset of each key frame in the historical key frame set, a feature description subset of the key frame located at the first preset position in the key frame set to be updated and a feature description subset of the key frame located at the second preset position in the key frame set to be updated are obtained;

performing feature matching according to a feature descriptor subset of each key frame in the historical key frame set, a feature descriptor subset of a key frame at a first preset position in the key frame set to be updated and a feature descriptor subset of a key frame at a second preset position in the key frame set to be updated, so as to determine a key frame of which the feature descriptor meets a first preset condition and a key frame of which the feature descriptor meets a second preset condition in the historical key frame set;

and determining key frames matched with key frames positioned at a first preset position in the key frame set to be updated from the key frames of which the feature descriptors accord with the first preset condition, and determining key frames matched with key frames positioned at a second preset position in the key frame set to be updated from the key frames of which the feature descriptors accord with the second preset condition.

In some embodiments, the key frames with the feature descriptors meeting the first preset condition include N key frames with the largest number of feature descriptor matches with the key frames in the set of key frames to be updated at the first preset position, and the key frames with the feature descriptors meeting the second preset condition include N key frames with the largest number of feature descriptor matches with the key frames in the set of key frames to be updated at the second preset position.

performing feature matching on key frames of which feature descriptors accord with a first preset condition and key frames at a first preset position in a key frame set to be updated, performing feature matching on key frames of which feature descriptors accord with a second preset condition and key frames at a second preset position in the key frame set to be updated, and determining initial matching feature points, matched with feature points in the key frames at the first preset position in the key frame set to be updated, in the key frames of which feature descriptors accord with the first preset condition and initial matching feature points matched with feature points in the key frames at the second preset position in the key frame set to be updated in the key frames of which feature descriptors accord with the second preset condition;

filtering the initial matching feature points to eliminate the initial matching feature points which are mismatched and determine the matching feature points in the key frames of which the feature descriptors accord with the first preset condition and the matching feature points in the key frames of which the feature descriptors accord with the second preset condition;

In some embodiments, the current mapping information further includes pose information of the keyframe to be updated with respect to a mapping software coordinate system, and the point cloud generating module 103 is configured to:

performing feature matching according to SIFT feature points of each key frame in the current key frame set, and determining matched SIFT feature points;

and carrying out triangularization treatment according to the matched SIFT feature points and the pose information of the key frame to be updated relative to the mapping software coordinate system to generate point cloud.

In some embodiments, the point cloud generation module 103 is configured to: and performing feature matching on any two key frames with the interframe interval smaller than a preset threshold value in the current key frame set according to the SIFT feature points of the key frames in the current key frame set.

In some embodiments, the historical mapping information further comprises historical mapping information of mapping software, and the mapping information to be updated further comprises current mapping information of the mapping software and pose information of a key frame to be updated relative to a coordinate system of the mapping software; the to-be-updated map building information processing module 101 is configured to: and determining a key frame set to be updated, current map information of the mapping software and pose information of the key frame to be updated relative to a mapping software coordinate system according to the current image information of the area to be updated and the historical map information of the mapping software.

In some embodiments, as shown in fig. 7, the point cloud generating apparatus further includes an image information obtaining module 104, where the image information obtaining module 104 is configured to:

acquiring a video file of a region to be updated from a starting position to an end position; the starting point position and the end point position are both positioned in an area including an area to be updated and outside the area to be updated, and the vertical distance between the starting point position and the end point position and the boundary of the area to be updated is larger than a preset threshold value;

the video file is converted into current image information.

In addition, as shown in fig. 8, an embodiment of the present disclosure also provides an electronic device, including:

one or more processors 201;

a storage 202 on which one or more programs are stored;

the one or more programs, when executed by the one or more processors 201, cause the one or more processors 201 to implement the point cloud generation method provided by the embodiments as described above.

In addition, as shown in fig. 9, an embodiment of the present disclosure further provides a point cloud generating system, including:

an electronic device as hereinbefore described.

Furthermore, as shown in fig. 10, the embodiment of the present disclosure further provides a computer storage medium, on which a computer program is stored, wherein the program, when executed, implements the point cloud generating method provided by the foregoing embodiments.

It will be understood by those of ordinary skill in the art that all or some of the steps of the methods, functional modules/units in the apparatus, disclosed above may be implemented as software, firmware, hardware, and suitable combinations thereof. In a hardware implementation, the division between functional modules/units mentioned in the above description does not necessarily correspond to the division of physical components; for example, one physical component may have multiple functions, or one function or step may be performed by several physical components in cooperation. Some or all of the physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application specific integrated circuit. Such software may be distributed on computer readable media, which may include computer storage media (or non-transitory media) and communication media (or transitory media). The term computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data, as is well known to those of ordinary skill in the art. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital Versatile Disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can accessed by a computer. In addition, communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media as known to those skilled in the art.

Example embodiments have been disclosed herein, and although specific terms are employed, they are used and should be interpreted in a generic and descriptive sense only and not for purposes of limitation. In some instances, features, characteristics and/or elements described in connection with a particular embodiment may be used alone or in combination with features, characteristics and/or elements described in connection with other embodiments, unless expressly stated otherwise, as would be apparent to one skilled in the art. Accordingly, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the scope of the disclosure as set forth in the appended claims.

Claims

1. A point cloud generation method, comprising:

and generating point cloud according to the current mapping information.

2. The point cloud generation method according to claim 1, wherein the historical mapping information includes a historical key frame set, the mapping information to be updated includes a key frame set to be updated, the current mapping information includes a current key frame set, key frames in the key frame set are arranged in a time sequence, and the key frames are all image frames selected from the image information according to a preset key frame selection strategy; the determining the current mapping information according to the to-be-updated mapping information and the historical mapping information comprises:

3. The point cloud generation method of claim 2, wherein a key frame in the set of key frames to be updated at a first preset position is a first key frame in the set of key frames to be updated, and a key frame in the set of key frames to be updated at a second preset position is a last key frame in the set of key frames to be updated.

4. The point cloud generation method of claim 2, wherein the determining two keyframes from the historical keyframe set that respectively match the keyframe in the to-be-updated keyframe set at a first preset position and the keyframe in a second preset position comprises:

5. The point cloud generation method according to claim 4, wherein the key frames with feature descriptors meeting a first preset condition include N key frames with the largest number of feature descriptor matches with the key frames at a first preset position in the key frame set to be updated, and the key frames with feature descriptors meeting a second preset condition include N key frames with the largest number of feature descriptor matches with the key frames at a second preset position in the key frame set to be updated.

6. The point cloud generation method according to claim 4, wherein the determining, from the key frames whose feature descriptors meet a first preset condition, a key frame that matches a key frame located at a first preset position in the set of key frames to be updated, and determining, from the key frames whose feature descriptors meet a second preset condition, a key frame that matches a key frame located at a second preset position in the set of key frames to be updated includes:

performing feature matching on the key frames of which the feature descriptors meet a first preset condition and the key frames at a first preset position in the key frame set to be updated, performing feature matching on the key frames of which the feature descriptors meet a second preset condition and the key frames at a second preset position in the key frame set to be updated, and determining initial matching feature points, which are matched with feature points in the key frames at the first preset position in the key frame set to be updated, in the key frames of which the feature descriptors meet the first preset condition and initial matching feature points, which are matched with feature points in the key frames at the second preset position in the key frame set to be updated, in the key frames of which the feature descriptors meet the second preset condition;

7. The point cloud generation method of claim 2, wherein the current mapping information further includes pose information of a keyframe to be updated relative to a mapping software coordinate system, the generating a point cloud from the current mapping information comprising:

8. The point cloud generation method of claim 7, wherein the feature matching according to the SIFT feature points of each keyframe in the current set of keyframes comprises:

and performing feature matching on any two key frames with the interframe interval smaller than a preset threshold value in the current key frame set according to the SIFT feature points of the key frames in the current key frame set.

9. The point cloud generation method of claim 7, wherein the historical mapping information further includes mapping software historical map information, and the mapping information to be updated further includes mapping software current map information and pose information of the keyframe to be updated relative to a mapping software coordinate system;

10. The point cloud generation method according to any one of claims 1 to 9, wherein before determining the mapping information to be updated according to the current image information and the historical mapping information of the area to be updated, the method further comprises a step of acquiring the current image information of the area to be updated:

and converting the video file into the current image information.

11. A point cloud generating apparatus comprising:

12. An electronic device, comprising:

one or more processors;

a storage device having one or more programs stored thereon;

the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the point cloud generation method of any of claims 1-10.

13. A point cloud generation system, comprising:

the electronic device of claim 12.

14. A computer storage medium having stored thereon a computer program, wherein the program when executed implements the point cloud generation method of any of claims 1-10.