CN114565906A - Obstacle detection method, obstacle detection device, electronic device, and storage medium - Google Patents

Obstacle detection method, obstacle detection device, electronic device, and storage medium Download PDF

Info

Publication number
CN114565906A
CN114565906A CN202210209173.9A CN202210209173A CN114565906A CN 114565906 A CN114565906 A CN 114565906A CN 202210209173 A CN202210209173 A CN 202210209173A CN 114565906 A CN114565906 A CN 114565906A
Authority
CN
China
Prior art keywords
target
grid
position information
processed
image
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202210209173.9A
Other languages
Chinese (zh)
Inventor
李�浩
许舒恒
李元骏
许新玉
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Jingdong Kunpeng Jiangsu Technology Co Ltd
Original Assignee
Jingdong Kunpeng Jiangsu Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Jingdong Kunpeng Jiangsu Technology Co Ltd filed Critical Jingdong Kunpeng Jiangsu Technology Co Ltd
Priority to CN202210209173.9A priority Critical patent/CN114565906A/en
Publication of CN114565906A publication Critical patent/CN114565906A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
    • G06F16/58—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
    • G06F16/583—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
    • G06F16/58—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
    • G06F16/587—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using geographical or spatial information, e.g. location

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Library & Information Science (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Traffic Control Systems (AREA)
  • Image Analysis (AREA)

Abstract

The embodiment of the invention discloses a method and a device for detecting obstacles, electronic equipment and a storage medium, wherein the method comprises the following steps: determining at least one target pixel point in a target image to be processed and target position information of the at least one target pixel point; determining at least one historical image to be processed associated with a target image to be processed, and calling historical position information of each historical image to be processed; determining a target grid corresponding to the target vehicle and target grid attribute information of each sub-grid region in the target grid according to the vehicle position information, the target position information and each historical position information of the target vehicle; and determining the target barrier according to the attribute information of each target grid. According to the technical scheme of the embodiment of the invention, through the pixel-level processing process, the technical effect of accurately and comprehensively detecting various obstacles on the road is realized.

Description

Obstacle detection method, obstacle detection device, electronic device, and storage medium
Technical Field
The embodiment of the invention relates to the technical field of unmanned vehicles, in particular to a method and a device for detecting obstacles, electronic equipment and a storage medium.
Background
At present, in a perception algorithm of an unmanned delivery vehicle, a deep neural network is mostly used for detecting common obstacles, and optionally, the common obstacles can be vehicles, pedestrians, non-motor vehicles and the like; or, a laser radar is arranged on the unmanned distribution vehicle, and the corresponding obstacle is determined by collecting point cloud data in the space.
When the present invention is implemented based on the above-described embodiments, the inventors have found that the following problems occur:
the deep neural network can only be used for detecting common obstacles, and for uncommon obstacles such as bricks, steel pipes, plastic boxes and the like on the ground, the deep neural network cannot determine the types of objects, so that corresponding detection results cannot be output; when the laser radar is used for detecting the obstacles, the acquired point cloud data are sparse, so that the unmanned vehicle cannot detect the obstacles which are close to the road surface, low in height and small in size. Therefore, the two detection methods have the condition of false detection or missed detection, so that the unmanned vehicle cannot accurately and comprehensively detect the obstacles on the road, and the problem of poor driving effect of the unmanned vehicle is caused.
Disclosure of Invention
The invention provides a method and a device for detecting obstacles, electronic equipment and a storage medium, which are used for realizing the technical effect of accurately and comprehensively detecting various obstacles on a road.
In a first aspect, an embodiment of the present invention provides an obstacle detection method, which is applied to an unmanned vehicle, and includes:
determining at least one target pixel point in a target image to be processed and target position information of the at least one target pixel point;
determining at least one historical image to be processed associated with the target image to be processed, and calling historical position information of each historical image to be processed;
determining a target grid corresponding to a target vehicle and target grid attribute information of each sub-grid region in the target grid according to vehicle position information of the target vehicle, the target position information and each historical position information;
and determining the target barrier according to the attribute information of each target grid.
In a second aspect, an embodiment of the present invention further provides an obstacle detection apparatus, where the apparatus includes:
the target position information determining module is used for determining at least one target pixel point in the target image to be processed and the target position information of the at least one target pixel point;
the historical position information determining module is used for determining at least one historical image to be processed related to the target image to be processed and calling the historical position information of each historical image to be processed;
the target grid attribute information determining module is used for determining a target grid corresponding to a target vehicle and target grid attribute information of each sub-grid area in the target grid according to vehicle position information of the target vehicle, the target position information and each historical position information;
and the target obstacle determining module is used for determining the target obstacles according to the attribute information of each target grid.
In a third aspect, an embodiment of the present invention further provides an electronic device, where the electronic device includes:
one or more processors;
a storage device for storing one or more programs,
when the one or more programs are executed by the one or more processors, the one or more processors implement the obstacle detection method according to any one of the embodiments of the present invention.
In a fourth aspect, embodiments of the present invention further provide a storage medium containing computer-executable instructions, which when executed by a computer processor, are configured to perform the obstacle detection method according to any one of the embodiments of the present invention.
The technical scheme of the embodiment of the invention comprises the steps of firstly determining at least one target pixel point in a target image to be processed and target position information of the at least one target pixel point, then determining at least one historical image to be processed associated with the target image to be processed, and calling historical position information of each historical image to be processed, namely determining position information corresponding to the pixel points at multiple moments; furthermore, according to the vehicle position information, the target position information and the historical position information of the target vehicle, the target grid corresponding to the target vehicle and the target grid attribute information of each sub-grid area in the target grid are determined, the target barrier is determined according to the target grid attribute information, a convenient barrier detection mode is provided for the unmanned vehicle, meanwhile, the non-difference detection of various types of barriers is achieved through pixel-level processing, therefore, the barrier detection result is more accurate and comprehensive, and the problem of false detection or missing detection of the barriers in the existing detection mode is solved.
Drawings
In order to more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, a brief description is given below of the drawings used in describing the embodiments. It should be clear that the described figures are only views of some of the embodiments of the invention to be described, not all, and that for a person skilled in the art, other figures can be derived from these figures without inventive effort.
Fig. 1 is a schematic flow chart (a) of an obstacle detection method according to an embodiment of the present invention;
fig. 2 is a schematic flow chart (ii) of an obstacle detection method according to an embodiment of the present invention;
fig. 3 is a schematic diagram (iii) of an obstacle detection method according to an embodiment of the present invention;
fig. 4 is a block diagram (iv) of a structure of an obstacle detection apparatus according to an embodiment of the present invention;
fig. 5 is a schematic structural diagram (v) of an electronic device according to an embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting of the invention. It should be further noted that, for the convenience of description, only some of the structures related to the present invention are shown in the drawings, not all of the structures.
Fig. 1 is a schematic flow chart (a) of an obstacle detection method according to an embodiment of the present invention, where the present embodiment is applicable to a situation where an unmanned vehicle detects various types of obstacles on a road, and the method may be executed by an obstacle detection apparatus, where the apparatus may be implemented in the form of software and/or hardware, and the hardware may be an electronic device, such as a mobile terminal, a PC terminal, or a server.
As shown in fig. 1, the method specifically includes the following steps:
s110, determining at least one target pixel point in the target image to be processed and target position information of the at least one target pixel point.
The unmanned vehicle can be provided with the camera device, and correspondingly, an image directly shot by the camera device can be used as an image to be processed. For example, during the driving process of the unmanned vehicle, the driving environment information can be recorded based on the camera device, and each recorded video frame is taken as an image to be processed. Specifically, information of a plurality of elements exists in the image to be processed, and mainly includes information of a driving road, such as a road surface of a road, a mark on the road, an obstacle, and the like, and meanwhile, pedestrians, vehicles, sky, buildings, and the like may also be included in the image to be processed. It is to be understood that the image to be processed serves as a data basis for the unmanned vehicle to determine the obstacle on the road, and although including the above-described various types of elements, only the element that is on or about to be on the road on which the unmanned vehicle is traveling, and that may collide with the unmanned vehicle, is the obstacle that it needs to detect.
In this embodiment, when the unmanned vehicle needs to determine an obstacle on the current road, the to-be-processed image captured by the imaging device at the current time may be used as the target processing image, and the to-be-processed image captured by the imaging device when the unmanned vehicle is at the current position may also be used as the target to-be-processed image. For example, when an unmanned vehicle needs to determine an obstacle on a road at a certain time, a front road image captured by the image capturing device at the certain time may be taken as a target image to be processed; when the unmanned vehicle is at a certain position and an obstacle on the road needs to be determined, an image taken by the image pickup device at the position may be taken as a target image to be processed.
Since there may be many obstacles with different shapes, colors, sizes, and other features on the road, and these obstacles cannot be detected accurately and comprehensively by using the conventional deep learning model or laser radar, in this embodiment, the target image to be processed needs to be processed from the pixel level first.
Specifically, at least one target pixel point needs to be determined in the target image to be processed, and target position information of the at least one target pixel point needs to be determined. The target pixel points may be one or more pixel points corresponding to the obstacle in the target image to be processed, for example, the photographed target image to be processed is displayed, a carton is placed in the center of a front road, and if the unmanned vehicle continues to travel forward, the carton may collide with the carton, and at this time, in the target image to be processed, the pixel points corresponding to the carton serving as the obstacle are the target pixel points.
Further, after the target pixel point is determined, the coordinate value of the target pixel point in the two-dimensional coordinate system can be used as target position information. Continuing with the above example, after a plurality of target pixel points corresponding to the carton are determined, a two-dimensional coordinate system associated with the target image to be processed can be determined, and coordinate values of the pixel points in the two-dimensional coordinate system are used as target position information.
S120, determining at least one historical image to be processed associated with the target image to be processed, and calling historical position information of each historical image to be processed.
In this embodiment, due to the limitation of the shooting range of the shooting device and the two-dimensional image, a three-dimensional obstacle cannot be determined only by one target image to be processed, and meanwhile, one target image to be processed cannot accurately and comprehensively describe various types of obstacles existing on the current road. Therefore, after the target image to be processed is determined, at least one historical image to be processed needs to be determined from a plurality of images to be processed taken by the shooting device.
After the target image to be processed is determined, a plurality of images within a preset time period before the target image to be processed can be selected as the historical image to be processed based on a target moment corresponding to the target image to be processed; in another mode, after the target image to be processed is determined, a video segment to which the image belongs may be further determined, so that a plurality of video frames in the video segment before the target image to be processed are used as historical images to be processed.
Further, in order to implement comprehensive detection of the obstacle on the road, it is further required to determine a pixel point corresponding to each obstacle in each historical image to be processed according to a manner of determining a target pixel point, and further determine coordinate values of the pixel points in a corresponding two-dimensional coordinate system associated with the image, and use the coordinate values as historical position information.
And S130, determining a target grid corresponding to the target vehicle and target grid attribute information of each sub-grid region in the target grid according to the vehicle position information, the target position information and each historical position information of the target vehicle.
The target vehicle is a vehicle that is performing obstacle detection, and the vehicle position information is information associated with the current position of the target vehicle, and may be, for example, coordinate values in a three-dimensional space coordinate system where the target vehicle is currently located.
It will be appreciated by those skilled in the art that the process of detecting obstacles on a roadway by an unmanned vehicle is also essentially the process of determining where these obstacles are located relative to the vehicle itself. Based on this, in the present embodiment, after the vehicle position information, the target position information, and the respective historical position information of the target vehicle are determined, it is necessary to convert these information (e.g., specific coordinate values) into the same image.
In particular, the above information may be transformed into the same raster map. First, a target grid corresponding to a target vehicle is determined. The target grid may be a grid image, i.e. an image that has been discretized in both space and intensity, and is made up of at least a plurality of sub-grids. When the target grid is a matrix, any one of the elements may correspond to a point in the image, and the value of the element corresponds to the gray scale level of the point in the image.
Meanwhile, each sub-grid is also associated with corresponding target grid attribute information, and it can be understood that when the vehicle position information, the target position information and each historical position information are converted into the same target grid, corresponding attribute values can be given to the sub-grids as target grid attribute information, so that the sub-grids can reflect corresponding entities in the image to be processed. For example, when the value reflecting the gray level of the pixel point in the image is used as the target grid attribute information of the sub-grid, the shape of the obstacle on the road can be constructed through the gray values of the plurality of images in the target grid.
And S140, determining the target barrier according to the attribute information of each target grid.
In this embodiment, after determining the attribute information of each target grid in the target grid, the target obstacle on the current road can be determined according to the attribute information. For example, when a value reflecting the gray level of a pixel point in an image is used as the target grid attribute information, multiple gray values of the same sub-grid may be counted, and whether an obstacle exists at a position corresponding to the sub-grid is determined according to the statistical result, for example, when ten pieces of gray value information are determined for the same sub-grid based on one target image to be processed and nine historical images to be processed, the ten pieces of gray values are all 255, and it may be determined that an obstacle exists at a position corresponding to the sub-grid on a road based on the ten pieces of gray value information.
Further, in order to specify the overall shape of the obstacle in the target grid, it is necessary to construct a structure based on the detection results of the respective sub-grids, for example, by integrating a plurality of sub-grids at adjacent positions indicating the presence of the obstacle, and the obtained two-dimensional integration result indicates the shape of the obstacle, thereby realizing the overall detection of the obstacle on the road. It will be appreciated that the accuracy of the shape of the constructed obstacle is related to the size of the sub-grid in the target grid, the smaller the sub-grid, the greater the density of the target grid, and the greater the accuracy of the shape of the constructed obstacle.
According to the technical scheme of the embodiment, at least one target pixel point in a target image to be processed and target position information of the at least one target pixel point are determined, at least one historical image to be processed associated with the target image to be processed is determined, historical position information of each historical image to be processed is called, and position information corresponding to the pixel points at multiple moments is determined; furthermore, according to the vehicle position information, the target position information and the historical position information of the target vehicle, the target grid corresponding to the target vehicle and the target grid attribute information of each sub-grid area in the target grid are determined, the target barrier is determined according to the target grid attribute information, a convenient barrier detection mode is provided for the unmanned vehicle, meanwhile, the non-difference detection of various types of barriers is achieved through pixel-level processing, therefore, the barrier detection result is more accurate and comprehensive, and the problem of false detection or missing detection of the barriers in the existing detection mode is solved.
Fig. 2 is a schematic flow diagram (ii) of an obstacle detection method according to an embodiment of the present invention, where on the basis of the foregoing embodiment, the position information of each target pixel is converted into a world coordinate system, and meanwhile, a mapping table representing an association relationship between a two-dimensional coordinate value and a three-dimensional coordinate value of each target pixel is constructed based on the conversion result, so that when a three-dimensional coordinate value of a target pixel is subsequently needed, the three-dimensional coordinate values are called in real time by table lookup; after the target grid and each sub-grid area are determined, the target grid attribute information of each sub-grid area is determined, meanwhile, the attribute information of the sub-grid area in the blind area range is kept unchanged, and the situation that an unmanned vehicle mistakenly takes an undetected blind area as a travelable road is avoided by enabling the sub-grid to continuously output the attribute of the obstacle; and finally, clustering the sub-grid areas to be aggregated based on the target grid attribute information, so as to determine the area occupied by the target barrier in the target grid. The specific implementation manner can be referred to the technical scheme of the embodiment. The technical terms that are the same as or corresponding to the above-mentioned embodiments are not described in detail herein.
As shown in fig. 2, the method specifically includes the following steps:
s210, determining at least one target pixel point in the target image to be processed and target position information of the at least one target pixel point.
In this embodiment, in the process of determining at least one target pixel point in the target image to be processed, optionally, based on an image semantic segmentation algorithm, performing classification marking on each pixel point in the target image to be processed to obtain a classification marking of each pixel point; and determining at least one target pixel point in the target image to be processed based on each category mark.
The elements in the unmanned vehicle driving road or the environment to which the road belongs can be classified in advance, and optionally, the elements include sky, roads, buildings, pedestrians and the like. In this embodiment, the category of each element may also be preset, and on this basis, the image semantic segmentation algorithm may determine, according to the category of the element marked in advance, a category marker corresponding to each pixel point in the target image to be processed.
In this embodiment, in order to detect an obstacle that may collide with the unmanned vehicle, among the multiple determined category labels, other category labels other than the road category label may be determined first, and further, pixel points corresponding to other category labels in the target image to be processed are taken as target pixel points. In the practical application process, in order to reflect the pixel points corresponding to the non-road category labels in a simpler form, the image can be subjected to semantic segmentation, and then binarization processing can be performed on the obtained attributes of each pixel point, namely, the pixel value of the pixel point which can be driven by the unmanned vehicle and carries the road category label is set to be 0, and the pixel value of the pixel point which does not carry the road category label is set to be 1.
It can be understood that, in this way, instead of directly identifying and detecting the obstacle image in the conventional way, pixel points that do not belong to the road on which the unmanned vehicle can travel are directly determined, that is, the region reflected by the determined pixel points is the region in which the obstacle exists.
In the process of determining the target position information of at least one target pixel point, optionally, the target position information of each target pixel point is determined according to a mapping relation between a pixel point coordinate and a world coordinate which is established in advance.
The target position information refers to a three-dimensional coordinate value of a target pixel point in a world coordinate system, the mapping relation is determined according to a predetermined transformation matrix, and the transformation matrix is determined based on an internal reference calibration matrix and an external reference calibration matrix of a camera device arranged on a target vehicle, the vertical distance between the camera device and a horizontal plane and the distance of the coordinate system. The coordinate system distance can be the distance between the coordinate system of the camera device and the coordinate system of the world, and the distance value can be a fixed value or a variable value, and the value can be determined in real time in the process of practical application.
It should be noted that the installation position of the camera device on the unmanned vehicle is fixed, and accordingly, the vertical distance between the camera device and the horizontal plane is also determined. Meanwhile, the internal reference calibration matrix and the external reference calibration matrix of the camera device can also be predetermined, and optionally, the internal reference calibration matrix and the external reference calibration matrix can be factory parameters of the camera device, and can also be parameters determined after the debugging of workers is finished.
Exemplarily, a coordinate of a target pixel point in a two-dimensional coordinate system is known as PiDetermining the target position information of the point according to the mapping relation between the pixel point coordinates and the world coordinates which are established in advance, namely determining the three-dimensional coordinates P of the point in the world coordinate systemw＝(xw，yw，zw) The process of (1). Wherein z iswKnown as being in three dimensionsThe height coordinate of the camera in the coordinate system, i.e. the vertical distance of the camera from the horizontal plane. P isiAnd PwThe following equation is satisfied,
Figure BDA0003532414170000101
wherein M isICalibrating a matrix (known) for camera internal parameters, MEFor an external reference calibration matrix (known) of the camera device and a three-dimensional coordinate system (world coordinate system), ZcSubstituting the parameters for the distance between the coordinate system of the camera and the three-dimensional coordinate system can obtain the following formula:
Figure BDA0003532414170000111
through the calculation, the three-dimensional coordinate value of each target pixel point in the world coordinate system, namely the target position information, can be obtained.
Meanwhile, it should be noted that, because the coordinate conversion process reflected by the above formula is mainly related to the internal reference calibration matrix and the external reference calibration matrix of the camera device, in the practical application process, in order to reduce the time consumption of running the above algorithm on line, when the coordinates of each target pixel point in the target image to be processed are obtained, the corresponding three-dimensional coordinate values can be directly calculated and obtained, and meanwhile, a mapping table representing the association relationship between the two-dimensional coordinate values and the three-dimensional coordinate values of each target pixel point is constructed based on the conversion result, and based on this, the three-dimensional coordinate values can be called in real time in the subsequent mode of table lookup when the three-dimensional coordinate values of the target pixel points are needed.
S220, determining at least one historical image to be processed associated with the target image to be processed according to the generation time of the target image to be processed and a preset time interval; and calling historical position information corresponding to the historical to-be-processed image.
In this embodiment, since the historical to-be-processed image and the target to-be-processed image have a correlation in the parameter of time, when one target to-be-processed image is determined, in order to further determine the historical to-be-processed image, the generation time of the target to-be-processed image needs to be determined. It is understood that time stamp information generated when the image pickup device picks up the target to-be-processed image is acquired.
Further, according to a preset time interval, a historical image to be processed can be determined, specifically, after the target image to be processed generation time t is determined, a plurality of timestamps which are before the time and within a preset time interval δ t can be determined, and then multi-frame images corresponding to the timestamps are used as the historical image to be processed. That is to say, the timestamp information in the [ t- δ t, t ] interval is determined, and then the image corresponding to each timestamp in the interval is taken as the set of the images to be processed, which can be understood that the set of the images to be processed includes at least one historical image to be processed and one target image to be processed.
In this embodiment, after obtaining at least one historical to-be-processed image, historical position information corresponding to the historical to-be-processed image may also be retrieved. Specifically, each pixel point in the historical image to be processed can be determined according to the mode of determining the target pixel point, the pixel value of the pixel point which can be driven by the unmanned vehicle and carries the road category mark is set to be 0, the pixel value of the pixel point which does not carry the road category mark is set to be 1, and the pixel points except the road category mark are determined to be used as the historical target pixel points in the historical image to be processed through binarization processing. Further, the corresponding two-dimensional coordinates of the history target pixel points in the history to-be-processed image are converted to obtain three-dimensional coordinate values of the history target pixel points in a world coordinate system, and each three-dimensional coordinate value is used as the history position information in the embodiment. And finally, constructing a set T based on all the historical position information (namely three-dimensional coordinate values) carrying the non-road attributes in the historical to-be-processed image and all the target position information (namely three-dimensional coordinate values) carrying the non-road attributes in the target to-be-processed image so as to represent the set of the barrier points on the road at the time T.
It should be noted that, in order to ensure that the two-dimensional coordinates of the pixel points in the multiple images can be converted into a unified world coordinate system, for any historical image to be processed and a unique target image to be processed, the historical position information and the target position information corresponding to each pixel point are determined based on the same coordinate system.
Because a shooting blind area may exist in the camera device, and meanwhile, the problem of false detection caused by the image semantic segmentation model is inevitable, and a single frame image cannot completely reflect obstacles on a road, so that after a target image to be processed is determined, a historical image to be processed associated with the target image to be processed is further determined, and the situation can be understood.
And S230, determining the target grid and each sub-grid region in the target grid according to the vehicle position information, the side length information of the sub-grid region and the covering length and the covering width of the covering target vehicle.
The vehicle position information may be a coordinate value corresponding to a current position of the unmanned vehicle in a world coordinate system. After the vehicle position information is determined, a two-dimensional target grid can be constructed by taking the position as the center. In practical application, the coordinates of the vehicle in the constructed target grid can be defined as (x)center，ycenter)。
In this embodiment, since the target grid is composed of a plurality of sub-grids, while determining the target grid, each sub-grid region may be determined based on the following formula:
Figure BDA0003532414170000131
the grid _ length is the length of the target grid, the grid _ width is the width of the target grid, and the coverage ranges of the target grid in the x-axis direction and the y-axis direction are respectively roi _ x and roi _ y. In practical applications, since each sub-grid in the target grid may be square, the determined side length of the sub-grid region is represented by grid _ size.
S240, determining target grid attribute information of each sub-grid area according to each historical position information of each historical image to be processed and the target position information of the target image to be processed.
In this embodiment, after the target grid is constructed according to the vehicle position information and each sub-grid region is determined, in order to enable the unmanned vehicle to detect the obstacle on the road, the position information corresponding to each pixel point in the image to be processed needs to be integrated into the target grid. It is understood that each of the historical positional information (two-dimensional coordinate values) of the historical to-be-processed image and the target positional information (two-dimensional coordinate values) in the target to-be-processed image is made to correspond to each of the sub grids in the target grid.
Optionally, for each image to be processed, the target sub-grid region to which each position information belongs is determined according to the horizontal and vertical coordinates of each position information in the current image to be processed, the side length information of the sub-grid region, and the horizontal and vertical coordinates of the vehicle position information.
Specifically, when the historical position information is retrieved, in order to represent the set of obstacle points on the road at time T, the set T has been constructed based on all the historical position information (i.e., three-dimensional coordinate values) carrying the non-road attribute in the historical to-be-processed image and all the target position information (i.e., three-dimensional coordinate values) carrying the non-road attribute in the target to-be-processed image. Therefore, it can be understood that the target to-be-processed image and each historical to-be-processed image are included in the to-be-processed image, and the target position information and the historical position information are included in the position information. Further, after determining the set of the position information in the image to be processed, a target sub-grid region corresponding to the position information may be determined in the target grid based on the following formula:
Figure BDA0003532414170000141
wherein, grid _ size is the side length of each sub-grid of the square; (x)center，ycenter) Coordinates for the vehicle in the constructed target grid, corresponding to (x)grid，ygrid) Then the coordinates of each target pixel point in the constructed target grid are obtained through conversion; and (x, y) is the coordinate of each target pixel point in the coordinate system before coordinate conversion.
In practical applications, in order to fuse the sets of obstacle points in consecutive time periods, (x, y) may be coordinate values obtained by integrating each piece of position information into the same local coordinate system (i.e., a local coordinate system). The local coordinate system may be a coordinate system established with a starting point on the vehicle travel route as an origin. It can be understood that after the position information of each pixel point is integrated into the local coordinate system, the position of the static obstacle on the road will not change along with the driving of the vehicle, and meanwhile, the vehicle position information (the three-dimensional coordinate value of the vehicle in the world coordinate system) can be converted into the local coordinate system by using a preset transformation matrix, which is not described herein again in the embodiments of the present disclosure.
After the target sub-grid regions corresponding to the position information are determined in the target grid, because each sub-grid is associated with corresponding attribute information, the target sub-grid regions corresponding to each image to be processed are also required to be marked as barrier attributes, and non-target sub-grid regions are required to be marked as free attributes, so that a grid attribute label sequence corresponding to each sub-grid region is obtained. Further, determining target grid attribute information according to each grid attribute label sequence.
Specifically, since each piece of position information is a coordinate value of a pixel point reflecting an obstacle, the position information is associated with a sub-grid, and then attribute information of the sub-grid can be marked, for example, the sub-grid is marked as an obstacle in the attribute information, and attribute information of another sub-grid which cannot be associated with the position information is marked as a free attribute, indicating that no obstacle exists at a position corresponding to the sub-grid. It can be understood that, since each piece of position information is from the target image to be processed and at least one historical image to be processed, the same sub-grid can be associated with a plurality of pieces of position information, so as to obtain a plurality of corresponding attribute information, and for one sub-grid, a grid attribute tag sequence can be constructed based on the plurality of attribute information.
After the embodiment, the target grid attribute information of each sub-grid can be determined after the grid attribute tag sequence of each sub-grid is obtained, and optionally, the target grid attribute information of the corresponding sub-grid region is determined according to the frequency of the barrier attribute and the free attribute in each grid attribute tag sequence.
It can be understood that each grid is associated with a tag sequence with a length δ t, tag information at each time in a [ t- δ t, t ] interval is recorded, and the tag information is integrated to determine the final attribute of the sub-grid. Based on the mode-finding strategy, for any sub-grid, the frequency of occurrence of two attributes (an obstacle attribute and a free attribute) in the plurality of label information can be determined, that is, whether an obstacle exists at a position corresponding to the sub-grid can be determined.
For example, for one sub-grid in the target grid, all ten pieces of tag information in the attribute tag sequence are the attributes of the obstacle, and based on this, it may be determined that the obstacle exists at the position corresponding to the sub-grid; if eight pieces of information of the ten pieces of label information of the attribute label sequence are barrier attributes, and the other two pieces of information are free attributes, the situation that barriers exist in the position corresponding to the sub grid is indicated, and if all the ten pieces of information of the attribute label sequence are free attributes, the situation that no barriers exist in the position corresponding to the sub grid is indicated.
If it is determined that the position information is within the range of the blind zone of the field of view of the target vehicle based on the horizontal and vertical coordinates of the position information, the attribute information of the sub-grid region to which the position information belongs is kept unchanged. It can be understood that, because a blind area may exist in the shooting process of the camera device on the unmanned vehicle, after the two-dimensional coordinates of the pixel point are obtained, if the pixel point is determined to be in the shooting blind area of the camera device through the two-dimensional coordinates, in the process of constructing the sub-grid attribute tag sequence, the existing attributes of the sub-grid attribute tag sequence are maintained (for example, the attribute corresponding to the blind area is marked as the attribute of the obstacle and is maintained constantly), and multi-frame accumulation is not performed, so that the sub-grid can be ensured to continuously output the attribute of the obstacle. In this way, the situation that an unmanned vehicle mistakenly takes an undetected blind area as a travelable road is avoided.
S250, determining at least one to-be-aggregated sub-grid area of the barrier attribute according to the attribute information of each target grid; and determining the target barrier by clustering at least one sub-grid area to be aggregated.
Specifically, after the target grid attribute information of each sub-grid in the target grid is obtained, the sub-grids in each row (or each column) may be traversed, so as to determine the sub-grid region to be aggregated, which is marked as the attribute of the obstacle. Further, a set B is constructed based on the sub-grid areas to be aggregated, nearest neighbor clustering processing is carried out on the points in the set B, at least one clustering result can be generated, and the convex polygon constructed based on each clustering result represents the obstacle on the road, namely, the shape of the obstacle is represented in the target grid.
In order to improve the accuracy of the detection result, in the multiple coordinate conversion processes, only the point which is on the ground plane or close to the ground plane in the world coordinate system (i.e., three-dimensional space) may be taken, and on the basis that each clustering result reflects the actual top view of the obstacle, the unmanned vehicle can plan the subsequent driving path based on the top view of the obstacle in the target grid.
According to the technical scheme of the embodiment, the position information of each target pixel point is converted into a world coordinate system, meanwhile, a mapping table representing the association relation between two-dimensional coordinate values and three-dimensional coordinate values of each target pixel point is constructed based on the conversion result, and therefore when the three-dimensional coordinate values of the target pixel points are needed in the follow-up process, the three-dimensional coordinate values are called in real time in a table look-up mode; after the target grid and each sub-grid area are determined, the target grid attribute information of each sub-grid area is determined, meanwhile, the attribute information of the sub-grid area in the blind area range is kept unchanged, and the situation that an unmanned vehicle mistakenly takes an undetected blind area as a travelable road is avoided by enabling the sub-grid to continuously output the attribute of the obstacle; and finally, clustering the sub-grid areas to be aggregated based on the target grid attribute information, so as to determine the area occupied by the target barrier in the target grid.
As an alternative embodiment of the above embodiment, fig. 3 is a schematic diagram (iii) of an obstacle detection method according to an embodiment of the present invention. For the purpose of clearly describing the technical solution of the present embodiment, the application scenario is a situation where an unmanned vehicle detects various types of obstacles on a road, but the present invention is not limited to the above scenario and may be applied to various scenarios where obstacles on a road need to be detected.
Referring to fig. 3, in order to implement pixel-level processing in the process of detecting an obstacle by an unmanned vehicle without performing truth value labeling on the characteristics of the obstacle or identifying the obstacle in an image, it is first required to extract pixel points in a non-travelable area of the unmanned vehicle. Specifically, the original image acquired by the camera device may be processed based on an image semantic segmentation technology, so as to determine semantic labels of each pixel in the image, and the semantic labels are used as attributes of the pixel points, such as road labels and non-road labels. Further, binarization processing is carried out on the semantic segmentation result, the pixel value of the pixel point of the attribute of the drivable road is set to be 0, and the pixel values of the pixel points of the other attributes are set to be 1.
With continued reference to fig. 3, since the pixels in the two-dimensional image cannot be applied to obstacle detection of the unmanned vehicle, it is also necessary to convert the coordinates of the pixels in the two-dimensional image into a three-dimensional spatial coordinate system, wherein the three-dimensional spatial coordinate system may be a spatial coordinate system established based on the unmanned vehicle. Specifically, the height z of the camera device on the unmanned vehicle relative to the ground in the three-dimensional space coordinate system can be determinedwAnd realizing the conversion of the pixel point coordinates based on the following formula:
Figure BDA0003532414170000181
wherein M isICalibrating a matrix (known) for camera internal parameters, MEFor an external reference calibration matrix (known) of the camera device and a three-dimensional coordinate system (world coordinate system), ZcThe distance of the coordinate system of the camera device relative to the three-dimensional coordinate system; meanwhile, the coordinate of the target pixel point in the two-dimensional coordinate system is Pi(u, v) with a coordinate P in three-dimensional spacew＝(xw，yw，zw) Based on this, substituting the above parameters can obtain the following formula:
Figure BDA0003532414170000182
through the calculation, the coordinates of each pixel point in the two-dimensional image in the three-dimensional space can be obtained.
With reference to fig. 3, because the points of the obstacle reflected by the single-frame image are not complete enough, when the unmanned vehicle detects the obstacle on the road at a certain time, the images of multiple frames before the certain time need to be fused for processing, and it can be understood that the pixel points in the history image acquired by the camera device are determined based on the above manner, and the two-dimensional coordinates of each pixel point in the image are converted to obtain the coordinates of the pixel point in the three-dimensional space.
With reference to fig. 3, after obtaining the three-dimensional coordinates of the pixel points in the multi-frame image, in order to merge the sets of the obstacle points in the continuous time, the three-dimensional coordinates need to be integrated into the same local coordinate system, where the local coordinate system may be a three-dimensional coordinate system established by using a preset transformation matrix with the driving starting point of the unmanned vehicle as the origin, and it can be understood that after the coordinate values of the pixel points in the three-dimensional space coordinate system in the image are integrated into the local coordinate system in a unified manner, the position information of the static obstacle does not change along with the driving of the vehicle.
Continuing to refer to fig. 3, after the vehicle position information and all the pixel points are converted into the local coordinate system, a grid map may be established with the vehicle as the center, each sub-grid in the grid map stores the attribute information of the grid, and the formula related to establishing the grid map is as follows:
Figure BDA0003532414170000191
wherein, grid _ size is the side length of each sub-grid of the square; (x)center，ycenter) Coordinates in the constructed grid map for the unmanned vehicle, corresponding to (x)grid，ygrid) The coordinates of each pixel point in the constructed grid graph are obtained through conversion; and (x, y) is the coordinate of each pixel point in the local coordinate system before coordinate transformation.
Meanwhile, other parameters of the grid map also satisfy:
Figure BDA0003532414170000192
grid _ length is the length of the grid graph, grid _ width is the width of the grid graph, and the coverage ranges of the grid graph in the x-axis direction and the y-axis direction are roi _ x and roi _ y, respectively.
Continuing to refer to fig. 3, after the coordinates of the vehicle and each pixel point are all converted into the grid map, the attribute of the sub-grid can be marked, for example, when the pixel value of the pixel point is 0, the pixel point represents the road area in the image corresponding to the pixel point, and therefore, the attribute label of the sub-grid corresponding to the pixel point can also be marked as a free attribute; when the pixel value of a pixel point is 1, it indicates that the pixel point corresponds to an obstacle in the image, and therefore, the attribute label of the sub-grid corresponding to the pixel point can also be marked as an obstacle attribute. Since the sub-grids in the grid map are associated with the pixels in the plurality of images, there are a plurality of attribute labels corresponding to the sub-grids.
Continuing to refer to fig. 3, after determining a plurality of attribute tags of a sub-grid, a tag sequence corresponding to the sub-grid may be constructed, and a final attribute of the sub-grid may be obtained by integrating the tag sequence, for example, when the frequency of occurrence of a free attribute in the tag sequence is high, a road in an image corresponding to the sub-grid may be determined, and when the frequency of occurrence of an obstacle attribute in the tag sequence is high, an obstacle in the image corresponding to the sub-grid may be determined.
It should be noted that, in this embodiment, since the computation of a large number of pixels is involved in the process of computing the raster map and each sub-raster tag sequence, in order to improve the computation efficiency, a Unified computing Device Architecture (CUDA) may be used to perform parallelization acceleration, and this way may optimize the time consumption of the whole computation and the occupation of CPU resources, thereby implementing real-time detection of the obstacle.
With reference to fig. 3, after the final attributes of the sub-grids in the grid map are determined, traversal may be performed on each row (or each column) of sub-grids, so as to determine boundary sub-grids between the obstacle attribute sub-grid and the free attribute sub-grid, further, a set B is constructed based on the boundary sub-grids, nearest neighbor clustering is performed on the grids in the set B, that is, at least one clustering result may be obtained, and then a corresponding convex polygon is constructed according to each clustering result. It is understood that the convex polygon is the detected obstacle actually present on the road.
The beneficial effects of the above technical scheme are: the method has the advantages that a convenient obstacle detection mode is provided for the unmanned vehicle, simultaneously, the undifferentiated detection of various types of obstacles is realized through pixel-level processing, so that the obstacle detection result is more accurate and comprehensive, and the problem of obstacle false detection or missing detection in the existing detection mode is avoided.
Fig. 4 is a block diagram (iv) of a structure of an obstacle detection apparatus according to an embodiment of the present invention, which is capable of executing the obstacle detection method according to any embodiment of the present invention, and has corresponding functional modules and beneficial effects. As shown in fig. 4, the apparatus specifically includes: a target location information determination module 310, a historical location information determination module 320, a target grid attribute information determination module 330, and a target obstacle determination module 340.
The target position information determining module 310 is configured to determine at least one target pixel point in the target image to be processed, and target position information of the at least one target pixel point.
A historical position information determining module 320, configured to determine at least one historical to-be-processed image associated with the target to-be-processed image, and retrieve a historical position confidence of each historical to-be-processed image.
The target grid attribute information determining module 330 is configured to determine a target grid corresponding to a target vehicle and target grid attribute information of each sub-grid region in the target grid according to vehicle position information of the target vehicle, the target position information, and each piece of historical position information.
And a target obstacle determining module 340, configured to determine a target obstacle according to the target grid attribute information.
On the basis of the above technical solutions, the target location information determining module 310 includes a classification marking unit, a target pixel point determining unit, and a target location information determining unit.
And the classification marking unit is used for performing classification marking on each pixel point in the target image to be processed based on an image semantic segmentation algorithm to obtain a classification mark of each pixel point.
And the target pixel point determining unit is used for determining at least one target pixel point in the target image to be processed based on each category mark.
The target position information determining unit is used for determining the target position information of each target pixel point according to the mapping relation between the pixel point coordinates and the world coordinates which are established in advance; wherein the mapping relationship is determined according to a predetermined transformation matrix, and the transformation matrix is determined based on an internal reference calibration matrix and an external reference calibration matrix of a camera device arranged on the target vehicle, a vertical distance between the camera device and a horizontal plane, and a coordinate system distance.
On the basis of the above technical solutions, the historical position information determining module 320 includes a historical to-be-processed image determining unit and a historical position information determining unit.
And the historical to-be-processed image determining unit is used for determining at least one historical to-be-processed image associated with the target to-be-processed image according to the generation time of the target to-be-processed image and a preset time interval.
A historical position information determining unit for retrieving historical position information corresponding to the historical image to be processed; wherein the historical location information and the target location information are determined based on the same coordinate system.
On the basis of the above technical solutions, the target grid attribute information determining module 330 includes a target grid determining unit and a target grid attribute information determining unit.
And the target grid determining unit is used for determining the target grid and each sub-grid region in the target grid according to the vehicle position information, the side length information of the sub-grid region, and the covering length and the covering width for covering the target vehicle.
And the target grid attribute information determining unit is used for determining the target grid attribute information of each sub-grid area according to each historical position information of each historical image to be processed and the target position information of the target image to be processed.
Optionally, the target grid attribute information determining unit is further configured to determine, for each to-be-processed image, a target sub-grid region to which each piece of position information belongs according to a horizontal coordinate and a vertical coordinate of each piece of position information in the current to-be-processed image, side length information of the sub-grid region, and a horizontal coordinate and a vertical coordinate of the vehicle position information; the image to be processed comprises a target image to be processed and historical images to be processed, and the position information comprises target position information and historical position information; marking a target sub-grid area corresponding to each image to be processed as an obstacle attribute, and marking a non-target sub-grid area as a free attribute to obtain a grid attribute label sequence corresponding to each sub-grid area; and determining the target grid attribute information according to each grid attribute label sequence.
Optionally, the target grid attribute information determining unit is further configured to determine the target grid attribute information of the corresponding sub-grid region according to the frequency of the obstacle attribute and the free attribute in each grid attribute tag sequence.
On the basis of the technical solutions, the obstacle detection device further includes an attribute information holding module.
And the attribute information holding module is used for holding the attribute information of the sub-grid area to which the position information belongs unchanged if the position information is determined to be positioned in the range of the visual field blind area of the target vehicle according to the horizontal and vertical coordinates of the position information.
Optionally, the target obstacle determining module 340 is further configured to determine at least one to-be-aggregated sub-grid region of the obstacle attribute according to the information of each target grid attribute; and determining the target obstacle by clustering the at least one sub-grid area to be aggregated.
According to the technical scheme provided by the embodiment, at least one target pixel point in a target image to be processed and target position information of the at least one target pixel point are determined, at least one historical image to be processed associated with the target image to be processed is determined, historical position information of each historical image to be processed is called, and position information corresponding to the pixel points at multiple moments is determined; furthermore, the target grid corresponding to the target vehicle and the target grid attribute information of each sub-grid area in the target grid are determined according to the vehicle position information, the target position information and the historical position information of the target vehicle, and the target obstacle is determined according to each target grid attribute information, so that a convenient obstacle detection mode is provided for the unmanned vehicle, meanwhile, the undifferentiated detection of various types of obstacles is realized through pixel-level processing, the obstacle detection result is more accurate and comprehensive, and the problem of obstacle false detection or missing detection in the existing detection mode is solved.
The obstacle detection device provided by the embodiment of the invention can execute the obstacle detection method provided by any embodiment of the invention, and has corresponding functional modules and beneficial effects of the execution method.
It should be noted that, the units and modules included in the apparatus are merely divided according to functional logic, but are not limited to the above division as long as the corresponding functions can be implemented; in addition, specific names of the functional units are only for convenience of distinguishing from each other, and are not used for limiting the protection scope of the embodiment of the invention.
Fig. 5 is a schematic structural diagram (v) of an electronic device according to an embodiment of the present invention. FIG. 5 illustrates a block diagram of an exemplary electronic device 40 suitable for use in implementing embodiments of the present invention. The electronic device 40 shown in fig. 5 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiment of the present invention.
As shown in fig. 5, electronic device 40 is embodied in the form of a general purpose computing device. The components of the electronic device 40 may include, but are not limited to: one or more processors or processing units 401, a system memory 402, and a bus 403 that couples the various system components (including the system memory 402 and the processing unit 401).
Bus 403 represents one or more of any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. By way of example, such architectures include, but are not limited to, Industry Standard Architecture (ISA) bus, micro-channel architecture (MAC) bus, enhanced ISA bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus.
Electronic device 40 typically includes a variety of computer system readable media. Such media may be any available media that is accessible by electronic device 40 and includes both volatile and nonvolatile media, removable and non-removable media.
The system memory 402 may include computer system readable media in the form of volatile memory, such as Random Access Memory (RAM)404 and/or cache memory 405. The electronic device 40 may further include other removable/non-removable, volatile/nonvolatile computer system storage media. By way of example only, storage system 406 may be used to read from and write to non-removable, nonvolatile magnetic media (not shown in FIG. 5, commonly referred to as a "hard drive"). Although not shown in FIG. 5, a magnetic disk drive for reading from and writing to a removable, nonvolatile magnetic disk (e.g., a "floppy disk") and an optical disk drive for reading from or writing to a removable, nonvolatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to the bus 403 by one or more data media interfaces. Memory 402 may include at least one program product having a set (e.g., at least one) of program modules that are configured to carry out the functions of embodiments of the invention.
A program/utility 408 having a set (at least one) of program modules 407 may be stored, for example, in memory 402, such program modules 407 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which examples or some combination thereof may comprise an implementation of a network environment. Program modules 407 generally perform the functions and/or methods of the described embodiments of the invention.
The electronic device 40 may also communicate with one or more external devices 409 (e.g., keyboard, pointing device, display 410, etc.), one or more devices that enable a user to interact with the electronic device 40, and/or any devices (e.g., network card, modem, etc.) that enable the electronic device 40 to communicate with one or more other computing devices. Such communication may be through input/output (I/O) interface 411. Also, the electronic device 40 may communicate with one or more networks (e.g., a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network, such as the internet) via the network adapter 412. As shown, the network adapter 412 communicates with the other modules of the electronic device 40 over the bus 403. It should be appreciated that although not shown in FIG. 5, other hardware and/or software modules may be used in conjunction with electronic device 40, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, among others.
The processing unit 401 executes various functional applications and data processing by executing programs stored in the system memory 402, for example, to implement the obstacle detection method provided by the embodiment of the present invention.
Embodiments of the present invention also provide a storage medium (vi) containing computer-executable instructions which, when executed by a computer processor, are operable to perform a method of obstacle detection.
The method comprises the following steps:
determining at least one target pixel point in a target image to be processed and target position information of the at least one target pixel point;
determining at least one historical image to be processed associated with the target image to be processed, and calling historical position information of each historical image to be processed;
determining a target grid corresponding to a target vehicle and target grid attribute information of each sub-grid region in the target grid according to vehicle position information of the target vehicle, the target position information and each historical position information;
and determining the target barrier according to the attribute information of each target grid.
Computer storage media for embodiments of the invention may employ any combination of one or more computer-readable media. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
A computer readable signal medium may include a propagated data signal with computer readable item code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
The item code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Computer project code for carrying out operations for embodiments of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The project code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present invention and the technical principles employed. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in greater detail by the above embodiments, the present invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims (12)

1. An obstacle detection method, applied to an unmanned vehicle, comprising:
determining at least one target pixel point in a target image to be processed and target position information of the at least one target pixel point;
determining at least one historical image to be processed associated with the target image to be processed, and calling historical position information of each historical image to be processed;
determining a target grid corresponding to a target vehicle and target grid attribute information of each sub-grid region in the target grid according to vehicle position information of the target vehicle, the target position information and each historical position information;
and determining the target barrier according to the attribute information of each target grid.
2. The method of claim 1, wherein the determining at least one target pixel point in the target to-be-processed image comprises:
classifying and marking each pixel point in the target image to be processed based on an image semantic segmentation algorithm to obtain a classification mark of each pixel point;
and determining at least one target pixel point in the target image to be processed based on each category mark.
3. The method of claim 1, wherein determining destination location information for the at least one destination pixel point comprises:
determining target position information of each target pixel point according to a mapping relation between a pixel point coordinate and a world coordinate which is established in advance;
wherein the mapping relationship is determined according to a predetermined transformation matrix, the transformation matrix being determined based on an internal reference calibration matrix and an external reference calibration matrix of a camera device provided on the subject vehicle, a vertical distance between the camera device and a horizontal plane, and a coordinate system distance.
4. The method according to claim 1, wherein the determining at least one historical to-be-processed image associated with the target to-be-processed image and retrieving historical position information of each historical to-be-processed image comprises:
determining at least one historical image to be processed associated with the target image to be processed according to the generation time of the target image to be processed and a preset time interval;
calling historical position information corresponding to the historical to-be-processed image; wherein the historical location information and the target location information are determined based on the same coordinate system.
5. The method of claim 1, wherein determining the target grid corresponding to the target vehicle and the target grid attribute information of each sub-grid region in the target grid according to the vehicle position information of the target vehicle, the target position information, and each of the historical position information comprises:
determining the target grid and each sub-grid region in the target grid according to the vehicle position information, the side length information of the sub-grid regions, and the covering length and the covering width for covering the target vehicle;
and determining the target grid attribute information of each sub-grid region according to the historical position information of each historical image to be processed and the target position information of the target image to be processed.
6. The method according to claim 4, wherein the determining the target grid attribute information of each sub-grid region according to each historical position information of each historical image to be processed and the target position information of the target image to be processed comprises:
aiming at each image to be processed, determining a target sub-grid region to which each position information belongs according to the horizontal and vertical coordinates of each position information in the current image to be processed, the side length information of the sub-grid region and the horizontal and vertical coordinates of the vehicle position information; the image to be processed comprises a target image to be processed and historical images to be processed, and the position information comprises target position information and historical position information;
marking a target sub-grid area corresponding to each image to be processed as an obstacle attribute, and marking a non-target sub-grid area as a free attribute to obtain a grid attribute label sequence corresponding to each sub-grid area;
and determining the target grid attribute information according to each grid attribute label sequence.
7. The method of claim 6, wherein determining the target grid attribute information from each grid attribute tag sequence comprises:
and determining target grid attribute information of the corresponding sub-grid region according to the frequency of the barrier attribute and the free attribute in each grid attribute tag sequence.
8. The method of claim 6, further comprising:
and if the position information is determined to be located in the range of the blind area of the visual field of the target vehicle according to the horizontal and vertical coordinates of the position information, keeping the attribute information of the sub-grid area to which the position information belongs unchanged.
9. The method of claim 1, wherein determining a target obstacle based on each target grid attribute information comprises:
determining at least one to-be-aggregated sub-grid area of the barrier attribute according to the attribute information of each target grid;
and determining the target obstacle by clustering the at least one sub-grid area to be aggregated.
10. An obstacle detection device, comprising:
the target position information determining module is used for determining at least one target pixel point in the target image to be processed and the target position information of the at least one target pixel point;
the historical position information determining module is used for determining at least one historical image to be processed related to the target image to be processed and calling the historical position information of each historical image to be processed;
the target grid attribute information determining module is used for determining a target grid corresponding to a target vehicle and target grid attribute information of each sub-grid area in the target grid according to vehicle position information of the target vehicle, the target position information and each historical position information;
and the target obstacle determining module is used for determining the target obstacles according to the attribute information of each target grid.
11. An electronic device, characterized in that the electronic device comprises:
one or more processors;
a storage device for storing one or more programs,
when executed by the one or more processors, cause the one or more processors to implement the obstacle detection method of any of claims 1-9.
12. A storage medium containing computer executable instructions for performing the obstacle detection method of any one of claims 1-9 when executed by a computer processor.
CN202210209173.9A 2022-03-04 2022-03-04 Obstacle detection method, obstacle detection device, electronic device, and storage medium Pending CN114565906A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210209173.9A CN114565906A (en) 2022-03-04 2022-03-04 Obstacle detection method, obstacle detection device, electronic device, and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210209173.9A CN114565906A (en) 2022-03-04 2022-03-04 Obstacle detection method, obstacle detection device, electronic device, and storage medium

Publications (1)

Publication Number Publication Date
CN114565906A true CN114565906A (en) 2022-05-31

Family

ID=81717964

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210209173.9A Pending CN114565906A (en) 2022-03-04 2022-03-04 Obstacle detection method, obstacle detection device, electronic device, and storage medium

Country Status (1)

Country Link
CN (1) CN114565906A (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116311095A (en) * 2023-03-16 2023-06-23 广州市衡正工程质量检测有限公司 Pavement detection method based on region division, computer equipment and storage medium

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116311095A (en) * 2023-03-16 2023-06-23 广州市衡正工程质量检测有限公司 Pavement detection method based on region division, computer equipment and storage medium
CN116311095B (en) * 2023-03-16 2024-01-02 广州市衡正工程质量检测有限公司 Pavement detection method based on region division, computer equipment and storage medium

Similar Documents

Publication Publication Date Title
US11037305B2 (en) Method and apparatus for processing point cloud data
CN111652097B (en) Image millimeter wave radar fusion target detection method
Bai et al. Infrastructure-based object detection and tracking for cooperative driving automation: A survey
CN110956137A (en) Point cloud data target detection method, system and medium
CN113761999A (en) Target detection method and device, electronic equipment and storage medium
US20220044558A1 (en) Method and device for generating a digital representation of traffic on a road
CN113963330A (en) Obstacle detection method, obstacle detection device, electronic device, and storage medium
CN115273039B (en) Small obstacle detection method based on camera
CN116310679A (en) Multi-sensor fusion target detection method, system, medium, equipment and terminal
CN111967396A (en) Processing method, device and equipment for obstacle detection and storage medium
CN114140592A (en) High-precision map generation method, device, equipment, medium and automatic driving vehicle
CN115719436A (en) Model training method, target detection method, device, equipment and storage medium
WO2022166606A1 (en) Target detection method and apparatus
CN115100741A (en) Point cloud pedestrian distance risk detection method, system, equipment and medium
Zhou et al. Street-view imagery guided street furniture inventory from mobile laser scanning point clouds
CN114565906A (en) Obstacle detection method, obstacle detection device, electronic device, and storage medium
CN113763438B (en) Point cloud registration method, device, equipment and storage medium
CN113255555A (en) Method, system, processing equipment and storage medium for identifying Chinese traffic sign board
CN114494323A (en) Obstacle detection method, device, equipment and storage medium
CN116977963A (en) Automatic driving multi-mode collaborative sensing method and system based on BEV visual angle
CN114429631B (en) Three-dimensional object detection method, device, equipment and storage medium
CN115457505A (en) Small obstacle detection method, device and equipment for camera and storage medium
Luo et al. Complete trajectory extraction for moving targets in traffic scenes that considers multi-level semantic features
CN114820931A (en) Virtual reality-based CIM (common information model) visual real-time imaging method for smart city
CN113570622A (en) Obstacle determination method and device, electronic equipment and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination