CN117128985A

CN117128985A - Point cloud map updating method and equipment

Info

Publication number: CN117128985A
Application number: CN202310477578.5A
Authority: CN
Inventors: 刘小伟; 陈讯; 黄韦维
Original assignee: Honor Device Co Ltd
Current assignee: Honor Device Co Ltd
Priority date: 2023-04-27
Filing date: 2023-04-27
Publication date: 2023-11-28
Anticipated expiration: 2043-04-27
Also published as: CN117128985B

Abstract

The application provides a method and equipment for updating a point cloud map, and relates to the field of positioning and image processing. The method comprises the following steps: acquiring image information, wherein the image information comprises an image of a first area and corresponding position information; according to the position information corresponding to the image, the positioning result of the image is obtained through the VPS, the image with accurate positioning result is a normal positioning image, and the image with inaccurate positioning result is an abnormal positioning image; acquiring whether a point cloud map corresponding to the first area needs to be updated or not according to the number of the abnormal positioning images; when the updating is needed, a scene graph corresponding to the first area is obtained, nodes of the scene graph are images, and edges are the number of matched features between the images; and updating the point cloud map of the first area according to the number of the normal positioning images and the number of the abnormal positioning images in the scene graph. According to the method provided by the application, the accuracy of the point cloud map updating time can be intelligently improved, and the freshness of the point cloud map is ensured.

Description

Point cloud map updating method and equipment

Technical Field

The present application relates to the field of positioning and image processing, and in particular, to a method and apparatus for updating a point cloud map.

Background

The visual positioning system (visual positioning system, VPS) is a technique for positioning a terminal device based on images acquired by the terminal device. For example, the terminal equipment acquires a surrounding environment to obtain a first image, and obtains a second image with highest matching degree with the first image in the database by matching the first image with a pre-stored image in the database, and determines the position of the current terminal equipment according to the shooting position of the second image.

It can be seen that the VPS positioning technology uses the environment image as the matching granularity to perform positioning, and along with the adjustment of the time, frequency, angle and the like of image shooting, the image can timely and accurately reflect the change of the real world environment. The timely and accurate environment change data is the key data for map updating. Therefore, how to apply the VPS positioning technology to map updating and improve the accuracy and timeliness of map updating becomes a concern.

Disclosure of Invention

The embodiment of the application provides a method and equipment for updating a point cloud map, which solve the problem of low updating efficiency of the point cloud map by means of a feature matching relation between a time dimension and new and old environment images and quantity information of different types of images.

In a first aspect, a method for updating a point cloud map is provided, and the method is applied to a cloud, and includes:

acquiring image information sent by a terminal device, wherein the image information comprises an image of a first area and position information corresponding to the image of the first area;

according to the position information corresponding to the image of the first area, a positioning result of the image is obtained through a Visual Positioning System (VPS), wherein the image with accurate positioning result is a normal positioning image, and the image with inaccurate positioning result is an abnormal positioning image;

acquiring whether a point cloud map corresponding to the first area needs to be updated or not according to the number of the abnormal positioning images;

when a point cloud map corresponding to the first area needs to be updated, a scene graph corresponding to the first area is obtained, nodes of the scene graph are the normal positioning image and the abnormal positioning image which have characteristic matching relations, and edges of the scene graph are the number of matched characteristics among the images corresponding to the nodes;

and updating the point cloud map corresponding to the first area according to the number of the normal positioning images and the number of the abnormal positioning images in the scene graph.

According to the point cloud map updating method provided by the implementation mode, the image of the built map area is acquired through interaction between the cloud and the terminal equipment, and the necessity of point cloud map updating is judged based on the VPS positioning technology, so that the cost of map updating is saved greatly, the accuracy of point cloud map updating time is improved more intelligently, and the freshness of the point cloud map is guaranteed. In addition, the method judges the fusion of the point cloud maps of the new part and the old part based on the constructed scene graph before updating, can ensure that the updated part is better fused into the original offline three-dimensional model, and obtains the point cloud map with better fusion effect.

With reference to the first aspect, in certain implementation manners of the first aspect, the method further includes:

storing an offline three-dimensional model, wherein the area corresponding to the offline three-dimensional model is a built map area; wherein,

the offline three-dimensional model comprises a first part and other parts, wherein the first part is an unexplored point cloud map corresponding to the first area, and the other parts are point cloud maps corresponding to other areas remained in the constructed drawing area.

In one implementation, the remaining other regions refer to regions of the constructed map other than the first GPS region, such as may correspond to other regions of the map other than the GPS region indicated by the arrow in fig. 2 below.

In one implementation, the cloud divides the constructed region into a plurality of tiles of GPS regions, including the first GPS region. The cloud can take the GPS areas of the small blocks as a processing unit for updating the point cloud map, so that the point cloud map updating data with larger scale can be prevented from being processed at one time, and the flexibility and the stability of data processing are improved.

With reference to the first aspect, in certain implementation manners of the first aspect, the image information further includes an image of the other area and position information corresponding to the image of the other area, and the method further includes:

And carrying out image classification according to the position information in the image information and the position of the first area, and acquiring an image belonging to the first area.

With reference to the first aspect, in some implementations of the first aspect, the acquiring, according to the number of abnormal positioning images, whether updating the point cloud map corresponding to the first area is required specifically includes:

and if the number of the abnormal positioning images is equal to or greater than a first threshold value, determining that the point cloud map corresponding to the first area needs to be updated.

and if the number of the abnormal positioning images is equal to or greater than a first threshold value and the duty ratio of the abnormal positioning images is equal to or greater than a second threshold value after a certain abnormal time, determining that the point cloud map corresponding to the first area needs to be updated.

If the number of the abnormal positioning images is equal to or greater than a first threshold value; the method comprises the steps of,

after a certain abnormal time, the duty ratio of the abnormal positioning image is equal to or greater than a second threshold value; the method comprises the steps of,

and if the semantic category difference between the normal positioning image and the abnormal positioning image after semantic segmentation is larger than a preset difference, determining that the point cloud map corresponding to the first area needs to be updated.

With reference to the first aspect, in some implementations of the first aspect, when the point cloud map corresponding to the first area needs to be updated, obtaining a scene graph corresponding to the first area specifically includes:

performing common view judgment on the normal positioning image of the first area before the abnormal time and the abnormal positioning image after the abnormal time;

and taking the images with common characteristics with other images after common view judgment as the nodes, taking the number of the characteristics matched between every two images as the edges between the corresponding nodes of every two images, and acquiring the scene graph.

With reference to the first aspect, in some implementations of the first aspect, the updating the point cloud map corresponding to the first area according to the number of the normal positioning images and the number of the abnormal positioning images in the scene graph specifically includes:

Counting whether the duty ratio of the normal positioning images included in the scene graph is equal to or greater than a fifth threshold value when the number of the normal positioning images included in the scene graph is equal to or greater than a third threshold value and the number of the abnormal positioning images included in the scene graph is equal to or greater than a fourth threshold value; and if the duty ratio of the normal positioning image included in the scene graph is equal to or greater than a fifth threshold value, updating the point cloud map corresponding to the first area.

With reference to the first aspect, in some implementations of the first aspect, the updating the point cloud map corresponding to the first area specifically includes:

the same position included in the normal positioning image and the abnormal positioning image corresponding to the first area is used as a characteristic point;

performing triangulation processing on the characteristic points to obtain three-dimensional point cloud information corresponding to the characteristic points;

and in the offline three-dimensional model, updating a point cloud map corresponding to the first area based on the three-dimensional point cloud information.

With reference to the first aspect, in certain implementations of the first aspect, the image information further includes time information;

the step of classifying the image according to the position information in the image information and the position of the first area to obtain the image belonging to the first area, and the step of:

And performing image classification according to the time information in the image information to acquire images belonging to the first area and within the same time period.

In a second aspect, there is provided an apparatus comprising:

one or more processors;

one or more memories;

the one or more memories store one or more computer programs comprising instructions that, when executed by the one or more processors, cause the apparatus to perform the method of any of the first aspects above.

In a third aspect, there is provided a computer readable storage medium storing computer executable program instructions which, when run on a computer, cause the computer to perform the method of any one of the first aspects.

In a fourth aspect, there is provided a computer program product comprising computer program code which, when run on a computer, causes the computer to perform the method according to any of the preceding implementation forms of the first aspect.

Drawings

Fig. 1A and fig. 1B are schematic diagrams of an area live-action schematic diagram and a three-dimensional point cloud model respectively provided in an embodiment of the present application.

Fig. 2 is a schematic diagram of dividing a GPS area into constructed areas according to an embodiment of the present application.

Fig. 3 is a schematic diagram of a system architecture to which a method for updating a point cloud map according to an embodiment of the present application is applicable.

Fig. 4 is a schematic structural diagram of a terminal device 100 according to an embodiment of the present application.

Fig. 5 is a block diagram of a software structure of a terminal device 100 according to an embodiment of the present application.

Fig. 6 is a schematic structural diagram of a cloud 200 according to an embodiment of the present application.

Fig. 7 is a schematic diagram of a position corresponding to a first GPS area according to an embodiment of the present application.

Fig. 8A to fig. 8D are schematic diagrams of scenes that may be involved in an implementation process of some map updating methods according to embodiments of the present application.

Fig. 9 is a schematic flowchart of a method for updating a point cloud map according to an embodiment of the present application.

Fig. 10 is a schematic diagram of a model corresponding to a scene graph according to an embodiment of the present application.

Fig. 11 is a schematic flowchart of a method for updating a point cloud map according to an embodiment of the present application.

Detailed Description

It should be noted that the terms used in the implementation section of the embodiment of the present application are only used to explain the specific embodiment of the present application, and are not intended to limit the present application. In the description of the embodiments of the present application, unless otherwise indicated, "/" means or, for example, a/B may represent a or B; "and/or" herein is merely one association relationship describing an associated obstacle, meaning that there may be three relationships, e.g., a and/or B, may represent: a exists alone, A and B exist together, and B exists alone. In addition, in the description of the embodiments of the present application, unless otherwise indicated, "a plurality" means two or more, and "at least one", "one or more" means one, two or more.

The terms "first" and "second" are used below for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a definition of "a first", "a second" feature may explicitly or implicitly include one or more of such features.

Reference in the specification to "one embodiment" or "some embodiments" or the like means that a particular feature, structure, or characteristic described in connection with the embodiment is included in one or more embodiments of the application. Thus, appearances of the phrases "in one embodiment," "in some embodiments," "in other embodiments," and the like in the specification are not necessarily all referring to the same embodiment, but mean "one or more but not all embodiments" unless expressly specified otherwise. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless expressly specified otherwise.

It is well known that over time, our living environment will change. The advertising board is added to a building, and road signs are added beside roads; as large as new landmark buildings, road re-planning, etc. The real world environment in which we live changes almost every moment.

In many cases, the map has become an important tool for assisting people in traveling, and the accuracy of the map influences the traveling efficiency of people. If a change has occurred in the real environment, but the map is not updated, the user experience is seriously affected. At present, a point cloud map (or a three-dimensional (3 d) point cloud map) product is generally updated by manually checking a map on site after a real environment is changed, that is, a professional needs to be arranged to acquire environment information again, and then the point cloud map is updated. However, this way of updating the point cloud map not only consumes manpower, but also seriously affects the updating efficiency.

In view of the above, the embodiment of the application provides a method and a device for updating a point cloud map. According to the method, VPS positioning is carried out on an image of a built image area reported by terminal equipment, feature matching is carried out on a query image and an image of a corresponding position of an established offline three-dimensional model (namely, a model corresponding to a point cloud map), and an image with accurate positioning (hereinafter referred to as a normal positioning image) and an image with inaccurate positioning (hereinafter referred to as an abnormal positioning image) are obtained. And judging whether the point cloud map needs to be updated according to a preset first condition. If the point cloud map is required to be updated, constructing a scene graph by utilizing common view judgment in the updating process, and judging whether the updating part can be better integrated into the original offline three-dimensional model according to a preset second condition.

Unlike the existing manual on-site inspection map, in the method for updating the point cloud map provided by the embodiment of the application, improvement on one aspect comprises: the cloud terminal autonomously judges whether the cloud point map needs to be updated or not according to a first condition; the improvement of another aspect comprises: and constructing a scene graph by carrying out common view judgment on images before and after the environmental change, and judging the blending effect of the updated part and the original point cloud map by a second condition. Wherein the first condition may be associated with the number of abnormally located images, the change in the duty cycle of the abnormally located images over time, and the semantic category gap of the abnormally located images and the normally located images; the second condition may be associated with a number of normally located images in the scene graph, a number of abnormally located images, and a duty cycle of the normally located images. The details of the first condition and the second condition will be described in the following examples, respectively, and will not be described in detail here.

According to the point cloud map updating method provided by the embodiment of the application, the image of the built map area is acquired through interaction between the cloud and the terminal equipment, and the necessity of updating the point cloud map is judged based on the quantity information of the abnormal positioning images acquired by the VPS positioning technology, so that the situation that excessive labor consumption is not required can be realized, the map updating cost is greatly saved, the accuracy of the map updating time is improved more intelligently, and the map freshness is maintained. In addition, before the point cloud map is finally updated, the point cloud map updating fusion performance is judged based on the constructed scene graph, so that the updated part can be better fused into the original offline three-dimensional model, and the point cloud map with better fusion effect can be obtained.

It should be noted that the method for updating the point cloud map provided by the embodiment of the application can be applied to various scenes. Exemplary, for example, a scene of updating an arbitrary point cloud map, or a scene of updating a high-precision map constructed based on a 3D model, or the like. The method can also be applied to point cloud map updating in Virtual Reality (VR)/augmented reality (augmented reality, AR) scenes or point cloud maps involved in intelligent transportation (such as unmanned) and intelligent navigation. The embodiment of the application does not limit the scene of the application of the point cloud map updating method.

In order to better understand the method for updating the point cloud map provided by the embodiment of the application, some technical terms or definitions which may appear in the text are first described below.

1. Constructed map area

In the embodiment of the application, for certain areas, modeling can be performed in advance according to the morphology, layout and the like of the areas, a three-dimensional model corresponding to the areas is constructed, and the three-dimensional model is stored in a cloud. The three-dimensional model may be an offline three-dimensional model, and may correspond to a point cloud map to be updated in the present application.

2. Visual positioning system VPS

VPS refers to a system that accomplishes positioning tasks through machine vision. For example, in the process of performing live-action navigation, images acquired in real time are matched with a map stored in a cloud to obtain accurate positioning. In the embodiment of the application, the user can acquire the image through the terminal equipment with the VPS function and upload the image to the cloud. For the images (hereinafter referred to as query images) acquired by the cloud, the cloud can search the corresponding image information in the offline three-dimensional model based on the position information of the query images, and then perform feature matching on the query images and the searched images to acquire the positioning result of the query images.

3. Abnormal positioning image, normal positioning image

In the embodiment of the application, after the cloud performs feature matching on the query image based on the VPS technology, two types of images can be obtained, one is an image with accurate positioning result, and the application defines the image as a normal positioning image; the other is an image with inaccurate positioning results, which is defined as an abnormal positioning image by the application.

The accurate positioning result can mean that the image matching degree of the query image and the corresponding image in the offline three-dimensional model is equal to or greater than a certain preset first numerical value (such as 75%, 80%, etc.); or, the number of feature matches between the query image and the corresponding image in the offline three-dimensional model is larger, for example, the number of feature matches is equal to or larger than a preset second value. The inaccurate result can mean that the image matching degree of the query image and the corresponding image in the offline three-dimensional model is smaller than a preset first numerical value (such as 75%, 80%, etc.); or, the number of feature matches between the query image and the corresponding image in the offline three-dimensional model is smaller, for example, the number of feature matches is smaller than a preset second numerical value.

It will be appreciated that the normal localization image may represent the correspondence of its corresponding real environment with the modeling of the region. If all the query images acquired by the cloud end are matched with the images respectively corresponding to the offline three-dimensional model, namely the query images are normal positioning images, the real environment is not changed, and at the moment, the point cloud map of the area is not required to be updated. The abnormal positioning image may indicate that the corresponding real environment is inconsistent with modeling the area, which indicates that the real environment of the area changes, and the point cloud map of the area may need to be updated.

4. Global positioning system (global positioning system, GPS) area/grid area

The GPS area (or grid area) is an area where a map is constructed and is divided into a plurality of small areas in order to avoid processing the point cloud map update data on a large scale at a time. The cloud can take the GPS areas of the small blocks as a processing unit, so that the data volume of one-time processing is reduced. Typically, the constructed areas may be divided according to GPS information, so these small areas are defined as GPS areas (or grid areas) in the present application.

The implementation process of the method for updating the point cloud map provided by the embodiment of the application is described in a scene by taking a certain established map area as an example in the following with reference to the attached drawings.

Exemplary, as shown in fig. 1A and fig. 1B, a schematic view of an area live-action and a schematic view of a corresponding three-dimensional model provided by an embodiment of the present application are respectively shown.

In some embodiments, the cloud may pre-obtain a three-dimensional model corresponding to the region of the built map. The cloud end is not limited in the way of acquiring the three-dimensional model, for example, the cloud end can model the constructed region by itself, or the cloud end can acquire the three-dimensional model from a third party. The three-dimensional model may be obtained in an existing manner, for example, by collecting three-dimensional point cloud data in the environment through a laser radar, and generating the three-dimensional model based on the three-dimensional point cloud data, which is not limited by the embodiment of the present application.

In some embodiments, the cloud may store the three-dimensional model to a database, obtaining an offline three-dimensional model.

Exemplary, as shown in fig. 2, a schematic diagram of dividing a GPS area into constructed areas is provided in an embodiment of the present application.

In some embodiments, the cloud may divide the created map area into multiple grid areas based on GPS information. For example, in a GPS map of a map-built area, it is divided into a plurality of GPS areas according to GPS positions. The cloud may store location information of each GPS region, such as a GPS range value, a vertex position, a size, and the like, corresponding to the GPS region.

Exemplary, as shown in fig. 3, a schematic diagram of a system architecture to which the method for updating a point cloud map according to the embodiment of the present application is applicable is shown. The system architecture includes at least one terminal device 100 and a cloud 200.

As an example, fig. 3 is presented taking the example that the terminal device 100 is a mobile phone. In practical applications, however, the terminal device 100 may be a plurality of types of electronic devices having VPS functions and communication functions. Such as tablet computers, wearable devices, digital cameras, netbooks, personal digital assistants (personal digital assistant, PDAs), etc. The embodiment of the present application does not limit the type of the terminal device 100.

In some embodiments, the terminal device 100 may be used to acquire images in the mapped region. In addition, the terminal device 100 may also acquire position information, such as position coordinates through GPS. Optionally, the terminal device 100 may also acquire time information corresponding to the acquired image, e.g., the terminal device 100 may add a time stamp to the image information. For example, the time information corresponding to the image may be, for example, a system time when the terminal device 100 collects the image, or a timer time of the terminal device 100, or a time when the terminal device 100 obtains from a third party device, which is not limited in the embodiment of the present application.

In some embodiments, the terminal device 100 may implement information interaction with the cloud 200 through a wireless communication link, such as uploading image information to the cloud. The image information may include an image (i.e., a query image) acquired by the terminal device 100 through the VPS function, time information corresponding to the image, position information corresponding to the image, and the like. The network type corresponding to the wireless communication link between the terminal device 100 and the cloud 200 may be a wireless network based on the existing communication technology standard, such as a long term evolution (long term evolution, LTE) wireless network, a car networking (vehical to everything, V2X), a fifth generation mobile communication technology (the 5th generation,5G), and other types of wireless networks that may occur in the future, such as a sixth generation mobile communication technology (the 6th generation,6G), and so on.

In some embodiments, cloud 200 may be one server or a server cluster including multiple servers. The cloud 200 has various capabilities, such as wireless communication capabilities, storage capabilities, and VPS-based computing capabilities, among others. For example, the cloud 200 may be configured to store the query image uploaded by the terminal device, perform feature matching on the image and the offline three-dimensional model, and then perform an operation of determining whether the point cloud map needs to be updated based on the matching result, perform an update procedure of the point cloud map, and so on. In addition, the cloud 200 may also classify the received query image. For example, classifying images according to position information, dividing images in the same GPS area into the same class; for another example, images are classified according to time information, such as classifying images in the same GPS area by season.

Exemplary, as shown in fig. 4, a schematic block diagram of a terminal device 100 according to an embodiment of the present application is provided.

The terminal device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (universal serial bus, USB) interface 130, a charge management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, keys 190, a motor 191, an indicator 192, a camera 193, a display 194, and a subscriber identity module (subscriber identification module, SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyro sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, and the like.

It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the terminal device 100. In other embodiments of the application, terminal device 100 may include more or less components than illustrated, or certain components may be combined, or certain components may be split, or different arrangements of components. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

The processor 110 may include one or more processing units, such as: the processor 110 may include an application processor (application processor, AP), a modem processor, a graphics processor (graphics processing unit, GPU), an image signal processor (image signal processor, ISP), a controller, a memory, a video codec, a digital signal processor (digital signal processor, DSP), a baseband processor, and/or a neural network processor (neural-network processing unit, NPU), etc. Wherein the different processing units may be separate devices or may be integrated in one or more processors.

The controller may be a neural center and a command center of the terminal device 100. The controller can generate operation control signals according to the instruction operation codes and the time sequence signals to finish the control of instruction fetching and instruction execution.

A memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may hold instructions or data that the processor 110 has just used or recycled. If the processor 110 needs to reuse the instruction or data, it can be called directly from the memory. Repeated accesses are avoided and the latency of the processor 110 is reduced, thereby improving the efficiency of the system.

In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an integrated circuit (inter-integrated circuit, I2C) interface, an integrated circuit built-in audio (inter-integrated circuit sound, I2S) interface, a pulse code modulation (pulse code modulation, PCM) interface, a universal asynchronous receiver transmitter (universal asynchronous receiver/transmitter, UART) interface, a mobile industry processor interface (mobile industry processor interface, MIPI), a general-purpose input/output (GPIO) interface, a subscriber identity module (subscriber identity module, SIM) interface, and/or a universal serial bus (universal serial bus, USB) interface, among others.

The I2C interface is a bi-directional synchronous serial bus comprising a serial data line (SDA) and a serial clock line (derail clock line, SCL). In some embodiments, the processor 110 may contain multiple sets of I2C buses. The processor 110 may be coupled to the touch sensor 180K, charger, flash, camera 193, etc., respectively, through different I2C bus interfaces. For example: the processor 110 may be coupled to the touch sensor 180K through an I2C interface, so that the processor 110 and the touch sensor 180K communicate through an I2C bus interface to implement a touch function of the terminal device 100.

The I2S interface may be used for audio communication. In some embodiments, the processor 110 may contain multiple sets of I2S buses. The processor 110 may be coupled to the audio module 170 via an I2S bus to enable communication between the processor 110 and the audio module 170. In some embodiments, the audio module 170 may transmit an audio signal to the wireless communication module 160 through the I2S interface, to implement a function of answering a call through the bluetooth headset.

PCM interfaces may also be used for audio communication to sample, quantize and encode analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 may be coupled through a PCM bus interface. In some embodiments, the audio module 170 may also transmit audio signals to the wireless communication module 160 through the PCM interface to implement a function of answering a call through the bluetooth headset. Both the I2S interface and the PCM interface may be used for audio communication.

The UART interface is a universal serial data bus for asynchronous communications. The bus may be a bi-directional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, a UART interface is typically used to connect the processor 110 with the wireless communication module 160. For example: the processor 110 communicates with a bluetooth module in the wireless communication module 160 through a UART interface to implement a bluetooth function. In some embodiments, the audio module 170 may transmit an audio signal to the wireless communication module 160 through a UART interface, to implement a function of playing music through a bluetooth headset.

The MIPI interface may be used to connect the processor 110 to peripheral devices such as a display 194, a camera 193, and the like. The MIPI interfaces include camera serial interfaces (camera serial interface, CSI), display serial interfaces (display serial interface, DSI), and the like. In some embodiments, processor 110 and camera 193 communicate through a CSI interface to implement the photographing function of terminal device 100. The processor 110 and the display 194 communicate via a DSI interface to implement the display function of the terminal device 100.

The GPIO interface may be configured by software. The GPIO interface may be configured as a control signal or as a data signal. In some embodiments, a GPIO interface may be used to connect the processor 110 with the camera 193, the display 194, the wireless communication module 160, the audio module 170, the sensor module 180, and the like. The GPIO interface may also be configured as an I2C interface, an I2S interface, a UART interface, an MIPI interface, etc.

The USB interface 130 is an interface conforming to the USB standard specification, and may specifically be a Mini USB interface, a Micro USB interface, a USB Type C interface, or the like. The USB interface 130 may be used to connect a charger to charge the terminal device 100, or may be used to transfer data between the terminal device 100 and a peripheral device. And can also be used for connecting with a headset, and playing audio through the headset. The interface may also be used to connect other terminals, such as AR devices, etc.

It should be understood that the interfacing relationship between the modules illustrated in the embodiment of the present application is only illustrative, and does not constitute a structural limitation of the terminal device 100. In other embodiments of the present application, the terminal device 100 may also use different interfacing manners, or a combination of multiple interfacing manners in the foregoing embodiments.

The charge management module 140 is configured to receive a charge input from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charge management module 140 may receive a charging input of a wired charger through the USB interface 130. In some wireless charging embodiments, the charge management module 140 may receive wireless charging input through a wireless charging coil of the terminal device 100. The charging management module 140 may also supply power to the terminal through the power management module 141 while charging the battery 142.

The power management module 141 is used for connecting the battery 142, and the charge management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and/or the charge management module 140 and provides power to the processor 110, the internal memory 121, the external memory, the display 194, the camera 193, the wireless communication module 160, and the like. The power management module 141 may also be configured to monitor battery capacity, battery cycle number, battery health (leakage, impedance) and other parameters. In other embodiments, the power management module 141 may also be provided in the processor 110. In other embodiments, the power management module 141 and the charge management module 140 may be disposed in the same device.

The wireless communication function of the terminal device 100 can be implemented by the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, a modem processor, a baseband processor, and the like.

The antennas 1 and 2 are used for transmitting and receiving electromagnetic wave signals. Each antenna in the terminal device 100 may be used to cover a single or multiple communication bands. Different antennas may also be multiplexed to improve the utilization of the antennas. For example: the antenna 1 may be multiplexed into a diversity antenna of a wireless local area network. In other embodiments, the antenna may be used in conjunction with a tuning switch.

The mobile communication module 150 may provide a solution including 2G/3G/4G/5G wireless communication applied to the terminal device 100. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (low noise amplifier, LNA), etc. The mobile communication module 150 may receive electromagnetic waves from the antenna 1, perform processes such as filtering, amplifying, and the like on the received electromagnetic waves, and transmit the processed electromagnetic waves to the modem processor for demodulation. The mobile communication module 150 can amplify the signal modulated by the modem processor, and convert the signal into electromagnetic waves through the antenna 1 to radiate. In some embodiments, at least some of the functional modules of the mobile communication module 150 may be disposed in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 may be provided in the same device as at least some of the modules of the processor 110.

The modem processor may include a modulator and a demodulator. The modulator is used for modulating the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used for demodulating the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low frequency baseband signal to the baseband processor for processing. The low frequency baseband signal is processed by the baseband processor and then transferred to the application processor. The application processor outputs sound signals through an audio device (not limited to the speaker 170A, the receiver 170B, etc.), or displays images or video through the display screen 194. In some embodiments, the modem processor may be a stand-alone device. In other embodiments, the modem processor may be provided in the same device as the mobile communication module 150 or other functional module, independent of the processor 110.

The wireless communication module 160 may provide solutions for wireless communication including wireless local area network (wireless local area networks, WLAN) (e.g., wireless fidelity (wireless fidelity, wi-Fi) network), bluetooth (BT), global navigation satellite system (global navigation satellite system, GNSS), frequency modulation (frequency modulation, FM), near field wireless communication technology (near field communication, NFC), infrared technology (IR), etc., applied to the terminal device 100. The wireless communication module 160 may be one or more devices that integrate at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the electromagnetic wave signals, filters the electromagnetic wave signals, and transmits the processed signals to the processor 110. The wireless communication module 160 may also receive a signal to be transmitted from the processor 110, frequency modulate it, amplify it, and convert it to electromagnetic waves for radiation via the antenna 2.

In some embodiments, antenna 1 and mobile communication module 150 of terminal device 100 are coupled, and antenna 2 and wireless communication module 160 are coupled, such that terminal device 100 may communicate with a network and other devices via wireless communication techniques. The wireless communication techniques may include the Global System for Mobile communications (global system for mobile communications, GSM), general packet radio service (general packet radio service, GPRS), code division multiple access (code division multiple access, CDMA), wideband code division multiple access (wideband code division multiple access, WCDMA), time division code division multiple access (time-division code division multiple access, TD-SCDMA), long term evolution (long term evolution, LTE), BT, GNSS, WLAN, NFC, FM, and/or IR techniques, among others. The GNSS may include a global satellite positioning system (global positioning system, GPS), a global navigation satellite system (global navigation satellite system, GLONASS), a beidou satellite navigation system (beidou navigation satellite system, BDS), a quasi zenith satellite system (quasi-zenith satellite system, QZSS) and/or a satellite based augmentation system (satellite based augmentation systems, SBAS).

The terminal device 100 implements display functions through a GPU, a display screen 194, an application processor, and the like. The display screen 194 is used to display images, videos, and the like.

The terminal device 100 may implement a photographing function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, an application processor, and the like.

The digital signal processor is used for processing digital signals, and can process other digital signals besides digital image signals. For example, when the terminal device 100 selects a frequency bin, the digital signal processor is used to fourier transform the frequency bin energy, or the like. Video codecs are used to compress or decompress digital video. The NPU is a neural-network (NN) computing processor, and can rapidly process input information by referencing a biological neural network structure, for example, referencing a transmission mode between human brain neurons, and can also continuously perform self-learning.

The external memory interface 120 may be used to connect an external memory card, such as a Micro SD card, to realize expansion of the memory capability of the terminal device 100. The external memory card communicates with the processor 110 through an external memory interface 120 to implement data storage functions. For example, files such as music, video, etc. are stored in an external memory card. The internal memory 121 may be used to store computer executable program code including instructions.

The terminal device 100 may implement audio functions through an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, an application processor, and the like. Such as music playing, recording, etc.

The pressure sensor 180A is used to sense a pressure signal, and may convert the pressure signal into an electrical signal. The gyro sensor 180B may be used to determine a motion gesture of the terminal device 100. The magnetic sensor 180D includes a hall sensor. The terminal device 100 can detect the opening and closing of the flip cover using the magnetic sensor 180D. The acceleration sensor 180E can detect the magnitude of acceleration of the terminal device 100 in various directions (typically three axes). The magnitude and direction of gravity may be detected when the terminal device 100 is stationary. The method can also be used for identifying the gesture of the terminal, and is applied to the applications such as horizontal and vertical screen switching, pedometers and the like. The proximity light sensor 180G may include, for example, a Light Emitting Diode (LED) and a light detector, such as a photodiode. The light emitting diode may be an infrared light emitting diode. The terminal device 100 emits infrared light outward through the light emitting diode. The ambient light sensor 180L is used to sense ambient light level. The terminal device 100 may adaptively adjust the brightness of the display 194 based on the perceived ambient light level. The fingerprint sensor 180H is used to collect a fingerprint. The temperature sensor 180J is for detecting temperature. The touch sensor 180K, also referred to as a "touch panel". The touch sensor 180K may be disposed on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, which is also called a "touch screen". The touch sensor 180K is for detecting a touch operation acting thereon or thereabout. The bone conduction sensor 180M may acquire a vibration signal.

In addition, the terminal device 100 further includes an air pressure sensor 180C and a distance sensor 180F. Wherein the air pressure sensor 180C is used to measure air pressure. In some embodiments, the terminal device 100 calculates altitude from barometric pressure values measured by the barometric pressure sensor 180C, aiding in positioning and navigation.

A distance sensor 180F for measuring a distance. The terminal device 100 may measure the distance by infrared or laser. In some embodiments, the terminal device 100 may range using the distance sensor 180F to achieve fast focusing.

For example, the software system of the terminal device 100 may employ a layered architecture, an event driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. In the embodiment of the application, taking an Android system with a layered architecture as an example, a software structure of the terminal device 100 is illustrated. Fig. 5 is a software configuration block diagram of the terminal device 100 of the embodiment of the present application.

The layered architecture divides the software into several layers, each with distinct roles and branches. The layers communicate with each other through a software interface. In some embodiments, the Android system is divided into four layers, from top to bottom, an application layer, an application framework layer, an Zhuoyun row (Android run), a system library, a kernel layer, a hardware abstraction layer (hardware abstraction layer, HAL), and a hardware layer.

The application layer may include a series of application packages. As shown in fig. 5, the application package may include camera, calendar, map, WLAN, music, short message, gallery, call, navigation, bluetooth, video, etc. applications.

The application framework layer provides an application programming interface (application programming interface, API) and programming framework for application programs of the application layer. The application framework layer includes a number of predefined functions. As shown in FIG. 5, the application framework layer may include a window manager, a content provider, a telephony manager, a resource manager, a notification manager, a view system, and the like.

The window manager is used for managing window programs. The window manager can acquire the size of the display screen, judge whether a status bar exists, lock the screen, intercept the screen and the like.

The content provider is used to store and retrieve data and make such data accessible to applications. The data may include video, images, audio, calls made and received, browsing history and bookmarks, phonebooks, etc.

The view system includes visual controls, such as controls to display text, controls to display pictures, and the like. The view system may be used to build applications. The display interface may be composed of one or more views. For example, a display interface including a text message notification icon may include a view displaying text and a view displaying a picture.

The telephony manager is used to provide the communication functions of the terminal device 100. Such as the management of call status (including on, hung-up, etc.).

The resource manager provides various resources for the application program, such as localization strings, icons, pictures, layout files, video files, and the like.

The notification manager allows the application to display notification information in a status bar, can be used to communicate notification type messages, can automatically disappear after a short dwell, and does not require user interaction. Such as notification manager is used to inform that the download is complete, message alerts, etc. The notification manager may also be a notification in the form of a chart or scroll bar text that appears on the system top status bar, such as a notification of a background running application, or a notification that appears on the screen in the form of a dialog window. For example, a text message is prompted in a status bar, a prompt tone is emitted, the terminal vibrates, and an indicator light blinks.

Android run time includes a core library and virtual machines. Android run time is responsible for scheduling and management of the Android system.

The core library consists of two parts: one part is a function which needs to be called by java language, and the other part is a core library of android.

The application layer and the application framework layer run in a virtual machine. The virtual machine executes java files of the application program layer and the application program framework layer as binary files. The virtual machine is used for executing the functions of obstacle life cycle management, stack management, thread management, security and abnormality management, garbage collection and the like.

The system library may include a plurality of functional modules. For example: surface manager (surface manager), media Libraries (Media Libraries), three-dimensional graphics processing Libraries (e.g., openGL ES), 2D graphics engines (e.g., SGL), etc.

The surface manager is used to manage the display subsystem and provides a fusion of 2D and 3D layers for multiple applications.

Media libraries support a variety of commonly used audio, video format playback and recording, still image files, and the like. The media library may support a variety of audio and video encoding formats, such as MPEG4, h.264, MP3, AAC, AMR, JPG, PNG, etc.

The three-dimensional graphic processing library is used for realizing three-dimensional graphic drawing, image rendering, synthesis, layer processing and the like.

The 2D graphics engine is a drawing engine for 2D drawing.

The kernel layer is a layer between hardware and software. The inner core layer at least comprises a display driver, a camera sensor driver, an audio driver and a sensor driver.

Exemplary, as shown in fig. 6, a schematic structural diagram of a cloud 200 according to an embodiment of the present application is shown.

In some embodiments, the cloud 200 may include an acquisition module, a database module, a classification module, a common view determination module, a map update module, and a condition determination module. The condition judging module can also comprise a quantity counting sub-module and a time counting sub-module. Optionally, the condition judgment module may further include a semantic judgment sub-module.

In some embodiments, the acquisition module may correspond to a wireless communication module of the cloud 200, and may be used to wirelessly communicate with the terminal device 100. Specifically, the acquiring module may receive, through a communication link between the cloud 200 and the terminal device 100, an image uploaded by the terminal device 100, and time information, position information, and the like corresponding to the image.

In some embodiments, the database module may be used to store an offline three-dimensional model of the created map area acquired by the cloud. In addition, the database module may be further configured to store images uploaded by the terminal device 100 during a certain historical period, for example, may store image information of a statistical period, such as a week, a month, a quarter, etc. Alternatively, the database module may be located locally to the cloud 200 (for example, a local memory space of the cloud 200) or may be an external memory space of the cloud 200, which is not limited in the embodiment of the present application.

Optionally, the cloud 200 may clean up invalid image information in the database. The invalid image information here may include image information received in a previous statistical period after the point cloud map update, that is, old image information which is no longer used in the map update flow. Specifically, the cloud end operation of clearing the old image information may be performed by an image clearing module (not shown in the figure).

In some embodiments, the classification module may be configured to classify the obtained query image according to location information, or according to location information and time information. Taking the classification of the query image according to the location information as an example, as shown in fig. 7, the location diagram corresponding to the first GPS area provided in the embodiment of the present application is shown. With reference to the figures, it can be known that the first GPS area is a GPS area (X ₁₁ ,Y ₁₁ )、(X ₁₂ ,Y ₁₂ )、(X ₁₃ ,Y ₁₃ )、(X ₁₄ ,Y ₁₄ ) A rectangular region having four coordinate positions as vertices, and the region includes a rectangular region having coordinates (x _a ,y _a ) And (d) position ofThe coordinates are (x) _b ,y _b ) Is a position of (c). If the cloud acquires two query images, the location information is (x _a ,y _a ) And (x) _b ,y _b ) The cloud end can judge that the two query images belong to the first GPS area according to the coordinates of the query images and the position of the first GPS area, and then divide the information of the two query images into categories corresponding to the first GPS area.

In some embodiments, the GPS regions divided for the mapped region may be any shape, such as rectangular (as shown in FIG. 2 or FIG. 7), circular, or the like. For easy understanding, please refer to table 1, the embodiment of the present application takes the shape of the GPS area as an example, and provides an example of classification results for classifying the query image according to the location information.

TABLE 1

Note that the classification results shown in table 1 are only examples. In practical applications, the classification result may further include other more information items, such as time information corresponding to the image, seasons corresponding to the image (see table 2 below), and the like, which is not limited in the embodiment of the present application.

In some embodiments, the condition determining module may determine, based on the query image acquired by the cloud 200, whether the corresponding GPS area meets a first condition for updating the point cloud map, that is, whether the GPS area needs to be updated. If the point cloud map needs to be updated, a subsequent point cloud map updating process can be executed on the GPS area. The condition judgment module may specifically include a quantity statistics sub-module, a time statistics sub-module, and a semantic judgment sub-module. The number counting sub-module can be used for counting the number of the abnormal locating images or counting the number of the abnormal locating images and the number of the normal locating images. The time statistics sub-module may be used to count the change of the duty cycle of the anomaly localization image over time. The semantic judgment sub-module can be used for carrying out semantic segmentation on the normal positioning image and the abnormal positioning image in the same GPS region, counting significant semantic categories and judging the difference of the semantic categories of the two types of images.

In some embodiments, whether to update the point cloud map may be determined according to output results of the number statistics sub-module and the time statistics sub-module; or judging whether to update the point cloud map according to the output results of the quantity counting sub-module, the time counting sub-module and the semantic judging sub-module.

In some embodiments, the common view judging module may be configured to perform feature matching on all query images (including a normal positioning image and an abnormal positioning image) of the GPS area, and construct a scene graph (scene graph). In addition, the common view judging module can be used for judging the connectivity of the scene graph and the number of the normal positioning images so as to ensure that the map part needing to be updated can be fused into the origin cloud map better. The specific processes and related principles will be described in the following examples, which are not described in detail herein.

Note that, the structure of the cloud 200 shown in fig. 6 is only an example. In practical applications, the cloud 200 may further include other more or fewer modules, or the modules included in the cloud 200 may also be used to perform other more functions, which is not limited in the embodiment of the present application.

The following describes an implementation scenario possibly related to the point cloud map updating method provided by the application with reference to the accompanying drawings. As shown in fig. 8A to 8D, exemplary, some scene diagrams that may be involved in the implementation process may be provided for the method for updating a map according to the embodiment of the present application.

Referring first to fig. 8A, a schematic diagram of a first GPS area is shown. In some scenarios, after the user enters the first GPS area, the terminal device (hereinafter, a cell phone is taken as an example) may turn on the VPS positioning function. For example, there may be various ways to turn on the VPS location function, such as: mode 1, after entering the GPS area, a user can manually start the VPS positioning function of the mobile phone; in mode 2, based on obtaining authorization from the user to use the VPS function by the mobile phone, when the user is detected to enter the first GPS area (e.g., based on the GPS location of the mobile phone), the mobile phone may automatically turn on the VPS positioning function.

In some embodiments, the VPS location function may be a function that the handset leaves the factory and may also be a function implemented by a third party application (App). One handset may take multiple images of the same GPS area from different angles, as shown in fig. 8A, and the handset may take the same building from different angles, as shown in the first, second and third angles. Images at different angles may be characteristic of different parts of the building. For example, the image corresponding to the first angle may be as shown in fig. 8B; the image corresponding to the second angle may be as shown in fig. 8C; the image corresponding to the third angle may be as shown in fig. 8D.

And then, uploading the shot image to the cloud end by the mobile phone, and uploading the shot image together with the image, wherein the shot image also comprises position information, time information and the like corresponding to the image. In some embodiments, the manner of uploading the image information to the cloud may be various, for example: in the mode (1), the mobile phone can upload image information to the cloud in real time; in the mode (2), the mobile phone can upload image information to the cloud under the manual operation of a user; the method (3), the mobile phone can periodically upload image information to the cloud (such as uploading at a fixed time every day); in mode (4), the handset may upload image information to the cloud when connected to a particular type of network, such as a wireless fidelity (wireless fidelity, wi-Fi) network. The specific mode of uploading the image information to the cloud is not limited in the embodiment of the application.

In some embodiments, after the cloud acquires the image, the uploaded image may be classified according to the location information. And then, the cloud end locates the images of the same GPS area through the VPS, and obtains a normal locating image and an abnormal locating image in the GPS area. And then, the cloud end can judge whether to trigger the subsequent flow of updating the point cloud map according to the first condition. If the subsequent procedure is executed, the cloud end can construct a scene graph through common view judgment, and judge whether the updating part can be better integrated into the original offline three-dimensional model/the original point cloud map according to the second condition. If the second condition is met, the cloud end can perform three-dimensional transformation on the characteristic points of the two types of images in the panoramic image of the GPS area to obtain three-dimensional point cloud data corresponding to the part, and then update the corresponding part in the original offline three-dimensional model by utilizing the three-dimensional point cloud data. The second condition may be used to determine connectivity of the scene graph and fusion effects with other parts of the original point cloud model.

It is to be noted that the cloud end extracts the features of the building under different angles through the images of different angles, so that the cloud end can conveniently obtain the comprehensive appearance of the building, and the accuracy of matching the image features with the three-dimensional model can be improved. In other words, the accuracy of the point cloud map update is related to the number of images acquired by the cloud and the diversity of angles. If the number of images of a certain GPS area acquired by the cloud is larger and the angles are more diversified, the accuracy of map updating of the cloud for the GPS area may be higher.

According to the method for updating the point cloud map, provided by the embodiment of the application, the image information of the built map area can be automatically acquired through interaction between the cloud and the terminal equipment, and the updating operation of the point cloud map is triggered at an accurate time according to the acquired image information. The whole point cloud map updating process does not need to excessively consume manpower, so that the cost of map updating can be greatly saved, the accuracy of map updating time can be improved more intelligently, and the freshness of the map is maintained.

The above describes the structure, system architecture, application scenario and visual implementation flow of the device that may be involved in the method for updating the point cloud map provided by the embodiment of the present application. For a better understanding of the method provided by the embodiments of the present application, the following description describes the underlying implementation logic of the method with reference to the accompanying drawings.

Exemplary, as shown in fig. 9, a schematic flowchart of a method for updating a point cloud map is provided in an embodiment of the present application. The process can be executed by using the terminal equipment (still taking a mobile phone as an example) and the cloud as main bodies, and specifically can include the following steps:

s901, the user uses VPS positioning function in the created map area.

The built map area herein may refer to an area where an offline three-dimensional model is built in advance, for example, when the built map area is a mall, it is indicated that the cloud end has stored an offline three-dimensional model corresponding to the mall, that is, a point cloud map.

In some embodiments, the user may actively use the VPS location function in the mapped region. For example, the management background of some of the mapped areas may inform the user that the VPS location function may be used in the area by pushing, notifying, etc., based on which the user may actively turn on the function or authorize the handset to use the function.

Under a possible condition, the mobile phone can also report the GPS position of the user to the cloud, and when the cloud detects that the user enters the GPS range of the established map area, the mobile phone can be instructed to start the VPS positioning function, or the mobile phone prompts the user to agree to start the VPS positioning function, or the mobile phone prompts the user to manually start the VPS positioning function. Thereafter, the handset may turn on the VPS location function either automatically or via user authorized use or manual operation by the user.

In some embodiments, after the cell phone turns on the VPS positioning function, the image in the environment may be acquired automatically, or the image in the environment may be acquired under manual operation by the user. Alternatively, the handset may perform periodic acquisitions, such as 3 seconds, 10 seconds, etc., when automatically acquiring images.

S902, uploading the images to the cloud by the mobile phone, and storing all query images positioned by using the VPS, time information and position information of the query images by the cloud.

The query image is an image of a certain GPS area uploaded by the mobile phone, which is acquired by the cloud. In some descriptions, the query image is also directly described as an image in conjunction with the contextual meaning. In practical applications, the query image may be an RGB image or a sequence of pictures.

In some embodiments, when the mobile phone sends an image to the cloud, the mobile phone may also send location information (such as a GPS location) corresponding to the image to the cloud. In addition, time information corresponding to the image can be sent to the cloud end, for example, a time stamp is added in the image information. Alternatively, the time information may be, for example, a system time of the mobile phone when the image is acquired, or a system time when the mobile phone sends the image to the cloud.

S903, the cloud classifies the uploaded images according to the position information, and judges whether the VPS positioning is successful or not.

In some embodiments, the cloud may classify according to the GPS location to which the image corresponds. By way of example, the process may include, for example: the cloud pre-divides a large-area constructed map area into a plurality of GPS areas with smaller areas; each GPS area may correspond to a size range value (e.g., 10m x 10 m) and a GPS range value, such as the rectangular GPS area shown in fig. 7, which may be determined based on the four vertices of the rectangle. And then, the cloud terminal judges which GPS area the query image belongs to according to the GPS position of the query image, and classifies the query images in the same GPS area into the same category. For example, the results after classification may be found in table 1 above.

Alternatively, the shape corresponding to the GPS area may be set to a circular shape or other shapes. If the GPS area is circular, the position of the GPS area may be determined according to the position and radius of the center of the circle, for example.

It should be noted that, because the field of view of the camera is limited and the range displayed by each image is limited, the whole large environment is divided into a plurality of small GPS areas, so that the point cloud map updating process can be performed one by one for the small GPS areas, or the point cloud map updating process can be performed in parallel for several GPS areas, without performing one-time large-scale process on the whole built map area, and the flexibility and stability of the map updating process are improved.

In some embodiments, the cloud end determining whether the VPS positioning is successful may determine according to an image matching degree of the query image and a corresponding image in the offline three-dimensional model. For example, if the image matching degree of the query image and the image at the corresponding position in the offline three-dimensional model is equal to or greater than a certain preset first numerical value (such as 75%, 80%, etc.), the positioning result corresponding to the query image is considered to be accurate; or, if the feature matching number of the query image and the corresponding image in the offline three-dimensional model is equal to or greater than a preset second value, the positioning result corresponding to the query image is considered to be accurate. The matching degree of the Query image and the corresponding image in the offline three-dimensional model is smaller than a preset first numerical value (such as 75%, 80%, etc.), and the positioning information corresponding to the Query image is considered to be inaccurate; or if the feature matching number of the query image and the corresponding image in the offline three-dimensional model is smaller than a preset second numerical value, the positioning information corresponding to the query image is considered to be inaccurate. Optionally, after the cloud determines the accuracy of the positioning result of the query image, positioning result information can be fed back to the mobile phone to prompt that the current positioning is accurate; or, indicate that the current position is inaccurate.

It can be understood that if the positioning result of the query image is accurate, this means that the current real environment of the region has a smaller environmental difference with the offline three-dimensional model, and the environment is not significantly changed compared with the modeling period, so that the point cloud map in this case is less likely to need to be updated. If the positioning result of the query image is inaccurate, the current real environment of the region has a larger environmental difference with the offline three-dimensional model, and compared with the modeling period, the environment may be obviously changed, so that the point cloud map in the situation has a larger possibility of needing to be updated.

It should be noted that the feature used to obtain the image matching degree in the VPS positioning process of the present application may be an image descriptor (image descriptor). Briefly, the output of an image descriptor is a feature vector, which is an abstraction of the image itself, which may be a list of numbers used to represent the image. Taking the vector of local aggregate descriptors (vector of locally aggregated descriptors, vlad) as an example, the process of extracting the image descriptors may include: inputting an image to be extracted (namely a query image) into a convolutional neural network, and acquiring a feature matrix of H, W and D (H is the height of the image to be extracted, W is the width of the image to be extracted, and D is the feature dimension); inputting the feature matrix into a convolutional network layer, and obtaining a clustering center and the residual distribution of the local features of the image to be extracted and the clustering center through convolution and normalization index functions; and finally, outputting an image descriptor of the image to be extracted through normalization operation by utilizing the residual error distribution.

In some embodiments, after extracting the image descriptor of the query image, the image matching result may also be obtained by matching with a violent matching algorithm or a fast nearest neighbor algorithm (fast libary for approximate nearest neighbors, FLANN), and the like. Optionally, in the foregoing image matching process, in order to make the result more accurate, the matching result may be optimized by a cross matching algorithm or the like, and the specific implementation flow may refer to the prior art, which is not described in detail in the present application.

S904, the cloud end judges abnormality, namely judges whether the point cloud map of the GPS area needs to be updated.

In some embodiments, the cloud end may determine whether to update the point cloud map of the GPS area according to a preset first condition. The judging method mainly comprises the following steps of: and the cloud end judges whether the point cloud map of the GPS area needs to be updated according to the number of the abnormal positioning images and/or the change of the duty ratio of the abnormal positioning images along with time.

For example, the first condition for determining whether to update the point cloud map in the embodiment of the present application may be set to different contents according to needs. For example, the first condition may be set as: and if the number of the abnormal positioning images in the GPS area is equal to or greater than a first threshold value, determining that the point cloud map corresponding to the GPS area needs to be updated. For another example, the first condition may be set as: if the number of the abnormal locating images in the GPS area is equal to or greater than a first threshold value, and the duty ratio of the abnormal locating images in the GPS area is equal to or greater than a second threshold value at a certain time (defined as abnormal time), determining that the point cloud map corresponding to the GPS area needs to be updated. For another example, the first condition may be further set to: if the number of the abnormal positioning images in the GPS area is equal to or greater than a first threshold value, the ratio of the abnormal positioning images in the GPS area is equal to or greater than a second threshold value, and after the normal positioning images and the abnormal positioning images are subjected to semantic segmentation, the semantic class difference is greater than a preset difference, determining that the point cloud map corresponding to the GPS area needs to be updated. The step of determining whether to update the point cloud map based on the semantic class gap results of the normal positioning image and the abnormal positioning image may be optional.

Here, the term "ratio of the abnormality localization images" refers to the ratio of the number of abnormality localization images to the number of the same class of query images. The same category refers to a category of classifying the query image by the cloud, such as the same GPS region, the same season, and the like.

For the first condition of the first type, the judging process may include the steps of:

s904 (1) a, the cloud counts the number of the abnormal location images acquired by the cloud, and the counting mode may be periodic, such as weekly, monthly or quarterly.

S904 (1) b, if the number of abnormal positioning images in the GPS area is equal to or greater than the first threshold in the statistical period, the cloud performs the update procedure of the map, that is, step S905 and step S906 below.

For the above second type of first condition, the judging process may include the steps of:

s904 (2) a, the cloud counts the number of the abnormal location images acquired by the cloud, and the counting mode may be periodic, such as weekly, monthly or quarterly.

S904 (2) b, if the number of abnormal positioning images of the GPS area is equal to or greater than the first threshold in the statistical period, counting whether the duty ratio of the abnormal positioning images is equal to or greater than the second threshold at the abnormal time in the period.

It should be noted that, considering that factors such as pedestrian activity and season replacement in the GPS area may bring unavoidable influence to accuracy of the real environment presented by the image, for example, the pedestrian blocks a portion of the market layout, the season replacement causes a change in tree morphology, and the like, under normal conditions, the duty ratio of the abnormally located image is usually fixed near a certain ratio (for example, at 15%). However, if the GPS area changes substantially, such as adding or subtracting buildings, road changes, billboard replacement, etc., the duty cycle of the anomalous positioning image increases significantly after a certain time (i.e., anomalous time). Thus, the problem that the unavoidable factors bring errors to updating the point cloud map can be solved through the step S904 (2) b.

In some embodiments, the cloud may further classify the query image according to the season in which the time stamp of the image is located, taking into account the effect of seasonal variations on the real environment. Exemplary, as shown in table 2 below, another query image classification result example provided in an embodiment of the present application is shown.

TABLE 2

It should be noted that, in the embodiment of the present application, image classification according to seasons may be used as an alternative. It is easy to understand that in practical application, whether to classify according to image time or not can be flexibly selected according to other time division modes (such as weekly or monthly as a period) according to different regions, and the classification is not required to be performed according to the traditional four-season division mode. For example, in a region with clear four seasons (such as a region near a yellow river basin), the classification can be performed according to the manner of table 2; however, if the region is not well-known in the four seasons (such as the region near the equator), it may be unnecessary to classify the images by seasons.

Alternatively, the image classification may be performed according to time information in an outdoor environment, while the image classification may not be performed according to time information in an indoor environment. In some embodiments, the cloud may determine whether the environment corresponding to the image is indoor or outdoor through image recognition, for example, through the duty cycle of the sky in the image.

S904 (2) c, if the duty ratio of the abnormal positioning image is equal to or greater than the second threshold value in the period, the cloud performs the update procedure of the map, that is, the following steps S905 and S906 are performed.

For the above third type of first condition, the judging process may include the steps of:

s904 (3) a, counting the number of the obtained abnormal positioning images by the cloud.

S904 (3) b, if the number of abnormal positioning images of the GPS area is equal to or greater than the first threshold value in the statistical period, counting whether the duty ratio of the abnormal positioning images is equal to or greater than the second threshold value at the abnormal time in the period.

S904 (3) c, if the duty ratio of the abnormal positioning image is equal to or greater than the second threshold value at the abnormal time in the period, counting whether the difference between the semantic categories of the abnormal positioning image and the normal positioning image in the GPS region reaches the target difference.

In some embodiments, the cloud may perform semantic segmentation on the abnormal positioning image and the normal positioning image in the GPS region, count significant semantic categories, and then determine whether the semantic categories of the abnormal positioning image and the normal positioning image differ by a target gap. The manner of semantic segmentation of the abnormal locating image and the normal locating image by the cloud can be based on a traditional method and/or a method based on a convolutional neural network (convolutional neural networks, CNN). Traditional semantic segmentation methods include, for example, statistical-based methods and geometric-based methods; CNN-based methods include, for example, methods based on semantically partitioning the CNN model, or methods based on the full convolution network (fully convolutional networks, FCN) model, and so forth. The detailed flow of semantic segmentation can be referred to as the existing flow, and the embodiment of the present application will not be described in detail.

S904 (3) d, if the difference between the semantic categories of the abnormal positioning image and the normal positioning image of the GPS area reaches the target difference, the cloud executes the update procedure of the map, that is, the following steps S905 and S906 are executed.

S905, the cloud determines that the newly acquired image can be used for updating the map through common view judgment.

The newly acquired image may refer to a query image of the GPS area acquired by the cloud in a statistical period.

For ease of understanding, the process involved in step S905 may be divided into three phases: stage 1, constructing a scene graph by a cloud; stage 2, determining scene graph communication by the cloud; and 3, counting the number of the normal positioning images in the scene graph to be up to or exceeding a certain number.

First, for stage 1, it should be noted that the images in the same scene graph have a common view relationship. The principle of constructing a scene graph can be understood that if two images have a common view relationship and features which can be matched, two nodes can be connected by taking the number of feature matching as an edge, and the two nodes can correspond to the two images. Similarly, images with a common view relationship in the GPS area can be used as nodes, the images are connected by two by taking the feature number which can be matched by the images as edges, and finally, the nodes are all complete scene graphs with the connection relationship. For example, the model corresponding to the scene graph may be as shown in FIG. 10.

For example, the specific process of constructing the scene graph by the cloud may include: (1) The cloud performs feature extraction on the normal positioning image of the GPS region before the abnormal time, and the abnormal positioning image after the abnormal time, wherein the abnormal time is the time when the duty ratio of the abnormal positioning image of the GPS region is obviously increased, that is, the time when the duty ratio of the one-field positioning image of the GPS region is equal to or greater than the second threshold value. The feature may be, for example, any point feature with a high specificity, such as an image point feature obtained by using a scale-invariant feature transform (SIFT) algorithm. (2) The SIFT algorithm is based on appearance matching, geometric relations among images are not considered, corresponding features cannot be mapped to the same three-dimensional scene point, so that the cloud end can calculate a base matrix and a homography matrix of every two images (a normal positioning image and an abnormal positioning image) by using a RANSAC method, and the matching pairs with higher reliability are screened out according to the geometric relations, and outer points are removed. (3) The cloud end outputs a complete scene graph corresponding to the GPS region, nodes of the scene graph are images, and edges are the number of image feature matching.

For stage 2, the process of determining scene graph connectivity by the cloud may include: (1) The statistical scene graph includes the number of two types of images (i.e., normally located images and abnormally located images). (2) If the number of normally positioned images and the number of abnormally positioned images included in the scene graph reach a preset number, respectively, for example, the number of normally positioned images included in the scene graph is equal to or greater than a third threshold value and the number of abnormally positioned images included in the scene graph is equal to or greater than a fourth threshold value, this means that there is a certain number of features in common between the normally positioned images and the abnormally positioned images, instead of all being the same type of image connection, or one type of image connection number is far greater than the connection number of two different types of images.

The connectivity of the scene graph is judged to ensure that after the real environment changes, the abnormal positioning images and the original normal positioning images have certain relevance, so that the abnormal positioning images are acquired in the GPS area, the environment changes do occur in the GPS area, and the images of other environments are not misused for analyzing the map changes of the current GPS area.

For stage 3, it should be noted that the cloud needs to count whether the normal positioning image duty ratio in the scene graph reaches or exceeds a certain ratio (such as the fifth threshold), because, in practical situations, even if the real environment of a certain GPS area changes, the portion that changes will be generally less or far less than the portion that does not change, from the point of view, the operation of stage 3 can help prove whether the data is reliable. Furthermore, the key purpose of this operation is to ensure that the new part can eventually better replace the original corresponding part in the offline three-dimensional model, i.e. better blend into the original point cloud map, rather than add in cleanly. In other words, in the embodiment of the present application, the scene graph corresponds to a GPS area, and the GPS area may only correspond to a small portion of the offline three-dimensional model, however, the final purpose of the point cloud map update is to fuse the changed portion of the GPS area into the complete offline three-dimensional model corresponding to the established scene graph area, so if the proportion of the normal positioning image in the scene graph is too small, there is a large error in final fusion, and a better fusion effect cannot be obtained.

The judgment process in the stage 2 and the stage 3 is based on the second condition. That is, the second condition may include: whether the number of the two types of images (i.e., the normal positioning image and the abnormal positioning image) included in the scene graph reaches a preset number, respectively, such as the number of the normal positioning images included in the scene graph is equal to or greater than a third threshold value, and the number of the abnormal positioning images included in the scene graph is equal to or greater than a fourth threshold value; and whether the duty cycle of the normally positioned image in the scene graph reaches or exceeds a certain ratio, such as the duty cycle of the normally positioned image in the scene graph is equal to or greater than a fifth threshold.

S906, the cloud end updates the point cloud map corresponding to the GPS area.

In some embodiments, the process of map updating by the cloud may include: and taking the same position of the image corresponding to the GPS region under a plurality of different view angles as a characteristic point, performing triangularization processing on the characteristic point, converting the characteristic point information into three-dimensional point cloud information, and then updating the corresponding part in the offline three-dimensional model based on the three-dimensional point cloud information to obtain an updated point cloud map.

According to the point cloud map updating method provided by the embodiment of the application, the image of the built map area is acquired through interaction between the cloud and the terminal equipment, and the necessity of the point cloud map updating is judged based on the VPS positioning technology, so that the cost of map updating is greatly saved, the accuracy of map updating time is improved more intelligently, and the freshness of the map is maintained. In addition, the method carries out judgment on the update fusion performance of the point cloud map based on the constructed scene graph before updating, and can ensure that the updated part is better fused into the original offline three-dimensional model, and the point cloud map with better fusion effect is obtained.

Exemplary, as shown in fig. 11, a schematic flowchart of a method for updating a point cloud map is provided in an embodiment of the present application. The method can take the cloud as an execution main body and specifically comprises the following steps:

s1101, acquiring image information sent by a terminal device, where the image information includes an image of a first area and position information corresponding to the image of the first area.

Wherein the first region may correspond to a GPS region in the above embodiments, such as the first GPS region.

The present application is mainly described by taking GPS as an example, but in practical application, the position information in the method may also be obtained by other manners, such as by a beidou satellite positioning system, etc., which is not limited by the embodiment of the present application.

S1102, according to the position information corresponding to the image of the first area, acquiring a positioning result of the image through the VPS, wherein the image with accurate positioning result is a normal positioning image, and the image with inaccurate positioning result is an abnormal positioning image.

S1103, acquiring whether the point cloud map corresponding to the first area needs to be updated according to the number of the abnormal positioning images.

S1104, when the point cloud map corresponding to the first area needs to be updated, a scene graph corresponding to the first area is obtained, nodes of the scene graph are normal positioning images and abnormal positioning images which have characteristic matching relations with other images, and edges of the scene graph are the number of matched characteristics among the images corresponding to the nodes.

S1105, updating the point cloud map corresponding to the first area according to the number of normal positioning images and the number of abnormal positioning images in the scene graph.

In some embodiments, the method further comprises: storing an offline three-dimensional model, wherein the area corresponding to the offline three-dimensional model is a built map area; the offline three-dimensional model comprises a first part and other parts, wherein the first part is a point cloud map corresponding to the first area which is not updated, and the other parts are point cloud maps corresponding to other areas remained in the constructed area.

In some embodiments, the image information further includes an image of the other region and position information corresponding to the image of the other region, and the method further includes: and carrying out image classification according to the position information in the image information and the position of the first area, and acquiring an image belonging to the first area.

In some embodiments, the acquiring, according to the number of the abnormal positioning images, whether the point cloud map corresponding to the first area needs to be updated specifically includes: and if the number of the abnormal positioning images is equal to or greater than a first threshold value, determining that the point cloud map corresponding to the first area needs to be updated.

In some embodiments, the acquiring, according to the number of the abnormal positioning images, whether the point cloud map corresponding to the first area needs to be updated specifically includes: and if the number of the abnormal positioning images is equal to or greater than a first threshold value and the duty ratio of the abnormal positioning images is equal to or greater than a second threshold value after a certain abnormal time, determining that the point cloud map corresponding to the first area needs to be updated.

In some embodiments, the acquiring, according to the number of the abnormal positioning images, whether the point cloud map corresponding to the first area needs to be updated specifically includes: if the number of the abnormal positioning images is equal to or greater than a first threshold value; and after a certain anomaly time, the duty cycle of the anomaly localization image is equal to or greater than a second threshold; and determining that the point cloud map corresponding to the first area needs to be updated if the semantic category difference between the normal positioning image and the abnormal positioning image after semantic segmentation is larger than a preset difference.

In some embodiments, when the point cloud map corresponding to the first area needs to be updated, acquiring the scene map corresponding to the first area specifically includes: performing common view judgment on the normal positioning image of the first area before the abnormal time and the abnormal positioning image after the abnormal time; and taking the images with common characteristics with other images after common view judgment as the nodes, taking the number of the characteristics matched between every two images as the edges between the corresponding nodes of every two images, and acquiring the scene graph.

In some embodiments, the updating the point cloud map corresponding to the first area according to the number of the normal positioning images and the number of the abnormal positioning images in the scene graph specifically includes: counting whether the duty ratio of the normal positioning images included in the scene graph is equal to or greater than a fifth threshold value when the number of the normal positioning images included in the scene graph is equal to or greater than a third threshold value and the number of the abnormal positioning images included in the scene graph is equal to or greater than a fourth threshold value; and if the duty ratio of the normal positioning image included in the scene graph is equal to or greater than a fifth threshold value, updating the point cloud map corresponding to the first area.

In some embodiments, the updating the point cloud map corresponding to the first area specifically includes: the same position included in the normal positioning image and the abnormal positioning image corresponding to the first area is used as a characteristic point; performing triangulation processing on the characteristic points to obtain three-dimensional point cloud information corresponding to the characteristic points; and in the offline three-dimensional model, updating the point cloud map corresponding to the first area based on the three-dimensional point cloud information.

In some embodiments, the image information further includes time information; the step of classifying the image according to the position information in the image information and the position of the first area to obtain the image belonging to the first area, and the step of: and performing image classification according to the time information in the image information to acquire images belonging to the first area and within the same time period.

Based on the same technical concept, the embodiment of the application also provides cloud equipment, which comprises one or more processors; one or more memories; the one or more memories store one or more computer programs comprising instructions that, when executed by the one or more processors, cause the computer or processor to perform one or more steps of any of the methods described above.

Based on the same technical concept, the embodiment of the application also provides terminal equipment, which comprises one or more processors; one or more memories; the one or more memories store one or more computer programs comprising instructions that, when executed by the one or more processors, cause the computer or processor to perform one or more steps of any of the methods described above.

Based on the same technical idea, the embodiments of the present application further provide a computer-readable storage medium having stored therein computer-executable program instructions, which when executed on a computer, cause the computer or processor to perform one or more steps of any of the methods described above.

Based on the same technical idea, an embodiment of the present application also provides a computer program product containing instructions, the computer program product comprising computer program code which, when run on a computer, causes the computer or processor to perform one or more steps of any one of the methods described above.

In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, produces a flow or function in accordance with embodiments of the present application, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions may be stored in or transmitted across a computer-readable storage medium. The computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by a wired (e.g., coaxial cable, fiber optic, digital subscriber line), or wireless (e.g., infrared, wireless, microwave, etc.). The computer readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that contains an integration of one or more available media. The usable medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a Solid State Disk (SSD)), or the like.

Those of ordinary skill in the art will appreciate that implementing all or part of the above-described method embodiments may be accomplished by a computer program to instruct related hardware, the program may be stored in a computer readable storage medium, and the program may include the above-described method embodiments when executed. And the aforementioned storage medium includes: ROM or random access memory RAM, magnetic or optical disk, etc.

The foregoing is merely a specific implementation of the embodiment of the present application, but the protection scope of the embodiment of the present application is not limited to this, and any changes or substitutions within the technical scope disclosed in the embodiment of the present application should be covered in the protection scope of the embodiment of the present application. Therefore, the protection scope of the embodiments of the present application shall be subject to the protection scope of the claims.

Claims

1. A method for updating a point cloud map, applied to a cloud, the method comprising:

when a point cloud map corresponding to the first area needs to be updated, a scene graph corresponding to the first area is obtained, nodes of the scene graph are the normal positioning image and the abnormal positioning image which have characteristic matching relations with other images, and edges of the scene graph are the number of matched characteristics between the images corresponding to the nodes;

2. The method according to claim 1, wherein the method further comprises:

the offline three-dimensional model comprises a first part and other parts, wherein the first part corresponds to a point cloud map corresponding to the first area which is not updated, and the other parts correspond to point cloud maps corresponding to other areas remained in the constructed drawing area.

3. The method of claim 2, wherein the image information further includes an image of the other region and position information corresponding to the image of the other region, the method further comprising:

4. The method of claim 1, wherein the obtaining, according to the number of abnormal positioning images, whether the point cloud map corresponding to the first area needs to be updated specifically includes:

5. The method of claim 1, wherein the obtaining, according to the number of abnormal positioning images, whether the point cloud map corresponding to the first area needs to be updated specifically includes:

and after a certain abnormal time, determining that the point cloud map corresponding to the first area needs to be updated if the duty ratio of the abnormal positioning image is equal to or greater than a second threshold value.

6. The method of claim 1, wherein the obtaining, according to the number of abnormal positioning images, whether the point cloud map corresponding to the first area needs to be updated specifically includes:

7. The method according to any one of claims 1-6, wherein when the point cloud map corresponding to the first area needs to be updated, obtaining a scene graph corresponding to the first area specifically includes:

8. The method according to claim 1, wherein updating the point cloud map corresponding to the first area according to the number of the normal positioning images and the number of the abnormal positioning images in the scene graph specifically includes:

Counting whether the duty ratio of the normal positioning images included in the scene graph is equal to or greater than a fifth threshold value when the number of the normal positioning images included in the scene graph is equal to or greater than a third threshold value and the number of the abnormal positioning images included in the scene graph is equal to or greater than a fourth threshold value;

and if the duty ratio of the normal positioning image included in the scene graph is equal to or greater than a fifth threshold value, updating the point cloud map corresponding to the first area.

9. The method according to any one of claims 1-8, wherein the updating the point cloud map corresponding to the first area specifically includes:

10. A method according to claim 3, wherein the image information further comprises time information;

And performing image classification according to the time information in the image information to acquire images belonging to the first area and the same time period.

11. An apparatus, comprising:

one or more processors;

one or more memories;

the one or more memories store one or more computer programs comprising instructions that, when executed by the one or more processors, cause the apparatus to perform the method of any of claims 1-10.

12. A computer readable storage medium storing computer executable program instructions which, when run on a computer, cause the computer to perform the method of any one of claims 1 to 10.