CN112348293A

CN112348293A - Method and device for predicting track of obstacle

Info

Publication number: CN112348293A
Application number: CN202110018259.9A
Authority: CN
Inventors: 徐一; 樊明宇; 任冬淳; 夏华夏; 代亚暄; 钱德恒; 朱炎亮
Original assignee: Beijing Sankuai Online Technology Co Ltd
Current assignee: Beijing Sankuai Online Technology Co Ltd
Priority date: 2021-01-07
Filing date: 2021-01-07
Publication date: 2021-02-09

Abstract

The specification provides a method and a device for predicting a track of an obstacle, wherein interaction weights between every two obstacles are determined according to motion features extracted from historical motion tracks of the obstacles, corresponding motion features are weighted according to the determined interaction weights, weighted space interaction features are obtained, a recurrent neural sub-network obtains space-time interaction features of the obstacles according to the input space interaction features, a pre-trained second model determines a predicted motion track of the obstacles according to the space-time interaction features of the obstacles, the interaction weights reflect the degree of influence between the obstacles and each other obstacle, therefore, the space interaction features depict interaction between the obstacles on the basis of the motion features, and the predicted future track precision of the obstacles is higher.

Description

Method and device for predicting track of obstacle

Technical Field

The present disclosure relates to the field of computer technologies, and in particular, to a method and an apparatus for predicting a trajectory of an obstacle.

Background

Currently, in the field of unmanned driving technology, a reference trajectory for a future period of time is usually planned for an unmanned device, so that the unmanned device can travel along the reference trajectory.

Due to the existence of the obstacle on the road, the reference track planned for the unmanned equipment needs to ensure that the unmanned equipment can avoid the obstacle to run. For an obstacle which can participate in traffic and the position of the obstacle can change continuously with time, in order to enable the unmanned equipment to accurately avoid the obstacle, the future track of the obstacle is generally required to be predicted.

In the prior art, the future track of each obstacle is often predicted only according to the historical state information of the obstacle. However, in the real world, when considering how each obstacle will travel in the future, not only considering itself (for example, how to arrive at the destination at the shortest distance) but also considering avoiding other obstacles, that is, there is an objective mutual influence between obstacles, so that a method of predicting only based on historical state information of obstacles cannot depict the interaction between obstacles, and the predicted trajectory precision of the obstacles is poor.

Disclosure of Invention

The embodiment of the present specification provides a method and an apparatus for predicting a trajectory of an obstacle, so as to partially solve the problems in the prior art.

The embodiment of the specification adopts the following technical scheme:

a method of trajectory prediction of an obstacle, comprising:

determining historical movement tracks of all obstacles in history;

for each obstacle, inputting a historical motion track of the obstacle within a preset time into a pre-trained first model, and extracting a motion characteristic of the obstacle through a characteristic extraction sub-network of the first model;

inputting the motion characteristics of each obstacle into the attention subnetwork of the first model to obtain the interaction weight between each two obstacles;

weighting the motion characteristics of each obstacle according to the determined interaction weight, and taking the weighted motion characteristics of each obstacle as the space interaction characteristics of each obstacle;

inputting the space interaction characteristics of the obstacles into the recurrent neural subnetwork of the first model to obtain the space-time interaction characteristics of the obstacles;

and inputting the space-time interactive characteristics output by the first model into a pre-trained second model, and determining the predicted movement locus of each obstacle through the second model.

Optionally, the extracting the motion feature of the obstacle specifically includes:

extracting the position characteristic and the speed characteristic of the obstacle;

inputting the motion characteristics of each obstacle into the attention subnetwork of the first model, specifically comprising:

inputting the location features of each obstacle into the attention subnetwork of the first model;

inputting the spatial interaction characteristics of the obstacles into the recurrent neural subnetwork of the first model to obtain the space-time interaction characteristics of the obstacles, which specifically comprises:

inputting the space interaction characteristics of each obstacle into the recurrent neural sub-network of the first model to obtain space-time interaction coarse characteristics of each obstacle;

and determining the space-time interaction characteristics of the obstacles according to the position characteristics, the speed characteristics and the space-time interaction coarse characteristics of the obstacles.

Optionally, the recurrent neural sub-network is a first LSTM long-short term memory network;

determining the space-time interaction characteristics of each obstacle according to the position characteristics, the speed characteristics and the space-time interaction coarse characteristics of each obstacle, and specifically comprising the following steps:

determining position space-time interaction characteristics according to the position characteristics and the space-time interaction coarse characteristics of each obstacle; determining speed space-time interaction characteristics according to the speed characteristics and the space-time interaction coarse characteristics of each obstacle;

and determining the space-time interaction characteristics of each obstacle according to the position space-time interaction characteristics and the speed space-time interaction characteristics.

Optionally, the second model specifically includes: an encoding end and a decoding end;

inputting the space-time interactive features output by the first model into a pre-trained second model, and determining the predicted movement locus of each obstacle through the second model, wherein the method specifically comprises the following steps:

inputting the space-time interaction characteristics output by the first model into the coding end to obtain the coding characteristics output by the coding end;

and inputting the coding characteristics into the decoding end, and predicting the track points of each obstacle at the target moment through the decoding end.

Optionally, the historical motion trajectory of each obstacle is a track point of each obstacle at each historical time before the reference time;

the target time is a future time after the reference time;

and the space-time interaction characteristics output by the first model are the space-time interaction characteristics at each historical moment.

Optionally, the encoding end is a second LSTM;

inputting the space-time interaction characteristics output by the first model into the coding end to obtain the coding characteristics output by the coding end, and the method specifically comprises the following steps:

and inputting the space-time interaction characteristics of each historical moment into the second LSTM to obtain hidden layer characteristics output by the hidden layer in the second LSTM according to the space-time interaction characteristics of each historical moment, wherein the hidden layer characteristics are used as the output coding characteristics of the coding end.

Optionally, inputting the encoding characteristics into the decoding end, and predicting, by the decoding end, trajectory points of each obstacle at a target time, specifically including:

and inputting the coding characteristics and the space-time interaction characteristics of the last historical moment of the reference moment into the decoding end, and predicting the track points of each obstacle at the target moment through the decoding end.

Optionally, the decoding end is a third LSTM;

after predicting the trajectory point of each obstacle at the target time, the method further comprises:

re-determining the reference time, and re-determining the target time according to the re-determined reference time;

and inputting the hidden layer characteristics output by the decoding end at the last historical moment of the redetermined reference moment and the space-time interactive characteristics output by the decoding end at the last historical moment of the redetermined reference moment into the decoding end so as to predict the track points of each obstacle at the redetermined target moment through the decoding end.

Optionally, predicting, by the decoding end, a trajectory point of each obstacle at the target time includes:

for each obstacle, splicing the space-time interaction vector corresponding to the obstacle in the space-time interaction characteristics of each obstacle at the last historical moment of the reference moment with a plurality of predetermined random noises respectively, and inputting the spliced space-time interaction vectors into the decoding end;

and aiming at each spliced space-time interaction vector, determining the track point of the barrier at the target moment through the decoding end according to the space-time interaction vector and the coding characteristics output by the coding end.

Optionally, the pre-training of the first model and the second model specifically includes:

determining each sample obstacle and a sample track corresponding to each sample obstacle;

according to a preset reference time, regarding each sample obstacle, regarding a track before the reference time as an initial track of the sample obstacle and regarding a track after the reference time as a labeling track of the sample obstacle in sample tracks corresponding to the sample obstacle;

inputting the initial track of each sample obstacle into a first model, and extracting the motion characteristics of the sample obstacle through a characteristic extraction sub-network of the first model;

inputting the motion characteristics of each sample obstacle into the attention subnetwork of the first model to obtain the interaction weight of each sample obstacle;

weighting the motion characteristics of each sample obstacle according to the determined interaction weight, and taking the weighted motion characteristics of each sample obstacle as the space interaction characteristics of each sample obstacle;

inputting the space interaction characteristics of each sample obstacle into the recurrent neural sub-network of the first model to obtain the space-time interaction characteristics of each sample obstacle;

inputting the space-time interaction characteristics output by the first model into a second model, and determining the predicted track of each sample obstacle through the second model;

and adjusting parameters in the first model and the second model by taking the minimum difference between the predicted track and the marked track of each sample obstacle as a target.

Optionally, determining the predicted trajectory of each sample obstacle through the second model specifically includes:

determining, for each sample obstacle, a number of predicted trajectories for the sample obstacle through the second model;

the method for adjusting the parameters in the first model and the second model by taking the minimum difference between the predicted track and the marked track of each sample obstacle as a target specifically comprises the following steps:

determining a predicted track with the minimum difference with the labeling track of each sample obstacle as a target track of the sample obstacle;

and adjusting parameters in the first model and the second model by taking the minimum difference between the target track and the labeling track of each sample obstacle as a target.

The present specification provides an obstacle trajectory prediction apparatus including:

the track determining module is used for determining historical movement tracks of all obstacles in history;

the feature extraction module is used for inputting the historical motion track of each obstacle within a preset time into a pre-trained first model, and extracting the motion feature of the obstacle through a feature extraction sub-network of the first model;

the attention weight module is used for inputting the motion characteristics of each obstacle into the attention subnetwork of the first model to obtain the interaction weight between each two obstacles;

the space characteristic module is used for weighting the motion characteristics of each obstacle according to the determined interaction weight and taking the weighted motion characteristics of each obstacle as the space interaction characteristics of each obstacle;

the space-time characteristic module is used for inputting the space interaction characteristics of the obstacles into the recurrent neural subnetwork of the first model to obtain the space-time interaction characteristics of the obstacles;

and the track prediction module is used for inputting the space-time interaction characteristics output by the first model into a pre-trained second model and determining the predicted motion track of each obstacle through the second model.

The present specification provides a computer-readable storage medium storing a computer program which, when executed by a processor, implements the above-described trajectory prediction method for an obstacle.

The present specification provides an unmanned device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the above-mentioned method for predicting the trajectory of an obstacle.

The technical scheme adopted by the specification can achieve the following beneficial effects:

according to the motion characteristics extracted from the historical motion trail of each obstacle, the interaction weight between every two obstacles is determined, the corresponding motion characteristics are weighted according to the determined interaction weight, the weighted space interaction characteristics are obtained, the recurrent neural sub-network obtains the space-time interaction characteristics of each obstacle according to the input space interaction characteristics, the pre-trained second model determines the predicted motion trail of each obstacle according to the space-time interaction characteristics of each obstacle, wherein the interaction weight reflects the influence degree between each obstacle and each other obstacle, therefore, the space interaction characteristics further depict the interaction between the obstacles on the basis of the motion characteristics, and the predicted future trajectory accuracy of the obstacles is higher.

Drawings

The accompanying drawings, which are included to provide a further understanding of the specification and are incorporated in and constitute a part of this specification, illustrate embodiments of the specification and together with the description serve to explain the specification and not to limit the specification in a non-limiting sense. In the drawings:

fig. 1 is a schematic flow chart of a method for predicting a trajectory of an obstacle in the present specification;

FIG. 2 is a schematic diagram of a trajectory prediction model according to the present disclosure;

FIG. 3 is a schematic diagram illustrating a second model of a trajectory prediction model according to the present disclosure;

FIG. 4 is a schematic diagram of a method for re-determining a target time in the present specification;

FIG. 5 is a schematic diagram of a method for selecting hidden layer features in a decoding end according to the present disclosure;

FIGS. 6A and 6B are schematic diagrams of two methods for predicting a plurality of movement trajectories for an obstacle in this specification;

FIG. 7 is a schematic flow chart of a trajectory prediction model training method of the present disclosure;

fig. 8 is a schematic diagram of a trajectory prediction device for an obstacle provided in the present specification;

fig. 9 is a schematic view of the drone corresponding to fig. 1 provided by the present description.

Detailed Description

In order to make the objects, technical solutions and advantages of the present disclosure more clear, the technical solutions of the present disclosure will be clearly and completely described below with reference to the specific embodiments of the present disclosure and the accompanying drawings. It is to be understood that the embodiments described are only a few embodiments of the present disclosure, and not all embodiments. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments in the present specification without any creative effort belong to the protection scope of the present specification.

In the field of unmanned driving technology, in order to guide decision making, planning and control of unmanned equipment, future tracks of obstacles around the unmanned equipment need to be predicted, so that the unmanned equipment is helped to avoid the obstacles according to the predicted future tracks of the obstacles.

The method for predicting the track of the obstacle provided by the specification is realized by adopting a corresponding track prediction model, and the track prediction model outputs the predicted motion track of each obstacle according to the input historical motion track of each obstacle.

The technical solutions provided by the embodiments of the present description are described in detail below with reference to the accompanying drawings.

The first embodiment is as follows:

fig. 1 is a schematic flow chart of a method for predicting a trajectory of an obstacle in this specification, which specifically includes the following steps:

s100: and determining historical movement tracks of all obstacles in history.

The trajectory prediction method provided in this specification may be executed by an unmanned vehicle (hereinafter, abbreviated as an unmanned vehicle), or may be executed by an electronic device capable of performing information transmission with or controlling the unmanned vehicle, such as a notebook computer, a mobile phone, a server, and the like, which is not limited in this specification. For convenience of description, the track prediction method provided by the specification is exemplarily described by taking an unmanned vehicle as an execution subject.

The unmanned vehicle described in this specification may include an autonomous vehicle and a vehicle having a driving assistance function. The unmanned vehicle may be a delivery vehicle applied to the delivery field.

In planning the trajectory of the unmanned vehicle, the obstacles considered are generally obstacles around the position where the unmanned vehicle is located, for example, obstacles 20 meters away from the unmanned vehicle, although any distance range may be considered as the surroundings of the unmanned vehicle. However, since the driving tendency of the unmanned vehicle is known, only an obstacle in the driving tendency direction of the unmanned vehicle (for example, an obstacle in front of the unmanned vehicle) may be considered, and an obstacle in the environment where the unmanned vehicle is located may be regarded as an obstacle around the unmanned vehicle, for example, each obstacle on the same road as the unmanned vehicle. It can be seen that there are many existing methods for selecting the obstacle, and this specification does not limit this.

On the other hand, on the one hand, the unmanned vehicle runs under the guidance of the track planned by the unmanned vehicle, so that the track of the unmanned vehicle is known without prediction, and the unmanned vehicle can not be used as an obstacle; on the other hand, since the unmanned vehicle also participates in traffic, the movement of the unmanned vehicle may affect the movement of other obstacles around the unmanned vehicle in the future, that is, interact with other obstacles, so that the unmanned vehicle itself may also be used as an obstacle, which is not limited in this specification.

The obstacles can be divided into static obstacles and dynamic obstacles according to whether the obstacles can actively move or not, wherein the static obstacles usually do not actively move, the positions of the static obstacles do not change along with time, such as telegraph poles, street lamps and the like, the dynamic obstacles refer to obstacles which can participate in traffic, such as vehicles, pedestrians and the like, and the movement of one dynamic obstacle can influence the movement of other dynamic obstacles because the positions of the dynamic obstacles can change along with time and the dynamic obstacles can interact with each other.

In the method for predicting the trajectory of the obstacle provided by the present specification, based on the above features of the dynamic obstacle, the obstacle whose trajectory is to be predicted inevitably includes the dynamic obstacle, and for the static obstacle, on one hand, the position of the static obstacle does not change with time, that is, there is no movement trajectory, and on the other hand, the presence of the static obstacle unilaterally affects the movement of the dynamic obstacle, that is, the dynamic obstacle actively avoids the static obstacle to travel, so that the static obstacle can be regarded as the obstacle referred to in the present specification, and the fixed position of the static obstacle is regarded as the trajectory point of the static obstacle, which does not change with time; the influence of the static obstacle on the dynamic obstacle may also be considered separately, for example, the predicted movement track of the dynamic obstacle does not pass through the position where the static obstacle is located, but the predicted movement track of the static obstacle is not output.

Since each obstacle is located around the unmanned vehicle, the sensor devices mounted on the unmanned vehicle can acquire the motion information of each obstacle in a period of time in the history, and of course, the sensor devices mounted at other positions can acquire the motion information of each obstacle in the history, and the corresponding terminal device or server sends the motion information to the unmanned vehicle.

Historical movement tracks of the obstacles in the history can be determined according to historical movement information of the obstacles, and the historical movement tracks can not only comprise tracks of the obstacles, but also comprise when, in what state and where along the historical movement tracks of the obstacles. Further, if the historical motion trajectory is not a smooth curve, but is a line formed by connecting a plurality of sequential trajectory points, the historical motion trajectory may include: each track point of the historical motion track, the time when the corresponding obstacle passes by each track point, and the driving state (such as speed, acceleration and the like) of the corresponding obstacle when passing by each track point.

The following steps S102-S110 can be realized by a trajectory prediction model as shown in FIG. 2.

S102: and for each obstacle, inputting the historical motion track of the obstacle in a preset time into a pre-trained first model, and extracting the motion characteristics of the obstacle through a characteristic extraction sub-network of the first model.

Because the acquired time spans of the historical movement tracks of the obstacles may not be the same, for comparison, the historical movement tracks of the obstacles within the same preset time duration may be intercepted, so as to ensure that the historical movement tracks occur within the same historical time period, and provide possibility for representing interaction of the obstacles. For example, a reference time may be determined, and then a historical movement track within a preset time period before the reference time is intercepted, and generally speaking, the current time when the unmanned vehicle executes the method may be used as the reference time. The method provided in the present specification will be described below by taking the historical motion trajectory of the input trajectory prediction model as an example of the historical motion trajectory within a preset time period before the reference time.

Inputting the intercepted historical movement track of each obstacle into a pre-trained first model, and obtaining the movement characteristics of each obstacle through a characteristic extraction sub-network of the first model, wherein the movement characteristics can represent the movement state of each obstacle within the historical preset time. It is understood that, for simplicity of calculation, the input historical motion trajectory may not be smooth and continuous, but track points corresponding to a plurality of historical moments within a preset time period before the reference moment are input, in this case, the feature extraction sub-network may output motion features of obstacles corresponding to the historical moments, and for convenience of description, the description will be given below by taking this as an example.

Generally speaking, due to the immediate requirement for unmanned vehicle control and the general requirement for the method, the motion information collected by the sensing device may be preprocessed before being input into the feature extraction sub-network, so that the trajectory prediction method provided by the present specification can be executed at different times and when the unmanned vehicle travels to different positions with a small amount of computation. For example, for the position information, when the position coordinates acquired by the sensing device are in a geodetic coordinate system, the position coordinates of each obstacle in the geodetic coordinate system may be converted into coordinates in a relative coordinate system which is more convenient to calculate, for example, the offset of each obstacle from a central point may be used as the coordinates of each obstacle, the central point may be the position coordinates of the unmanned vehicle at the corresponding historical time, or may be the average coordinates of each obstacle at each historical time, of course, if the coordinates of each obstacle acquired by the sensing device are the coordinates with a small calculation amount, for example, the coordinates of each obstacle acquired by the unmanned vehicle are the coordinates of each obstacle from the unmanned vehicle, and at this time, preprocessing before the motion information is input into the feature extraction sub-network is not required.

S104: and inputting the motion characteristics of each obstacle into the attention subnetwork of the first model to obtain the interaction weight between each two obstacles.

S106: and weighting the motion characteristics of each obstacle according to the determined interaction weight, and taking the weighted motion characteristics of each obstacle as the space interaction characteristics of each obstacle.

The motion characteristics of the input attention sub-network can reflect the interaction between the obstacles, so that the attention sub-network can determine the interaction weight of each obstacle according to the input motion characteristics to reflect the common influence degree between every two obstacles, namely the interaction compactness degree. This is because the obstacle itself may be influenced by other obstacles to adjust its position, i.e., the interaction described above. For example, the closer the obstacle is, the greater the possibility of collision with the obstacle is, so the obstacle will usually preferentially avoid the closer obstacle, in this case, the closer the interaction between the obstacle and the closer obstacle is, the greater the attention is paid, and the greater the influence is.

Specifically, from what aspect the input motion characteristic reflects the motion of the obstacle, the determined interaction weight also reflects the closeness of the obstacle interaction from the same aspect. For example, when the position characteristics of each obstacle at each historical time are input, the position characteristics can show which obstacles have interaction, specifically, because the position of each obstacle with the extracted position characteristics is around the unmanned vehicle, interaction exists between the obstacles, and the interaction weight determined by the attention subnetwork can further measure the closeness degree of interaction between each two obstacles according to the distance between the obstacles, generally speaking, the closer the distance between two obstacles is, the stronger the decision of how to act on the obstacle is, that is, the closer the interaction is, the higher the attention is, the greater the interaction weight between the two obstacles is determined. For brevity, the following description will take the example where the input to the attention subnetwork is a location characteristic of each obstacle at each historical time.

The motion characteristics of each obstacle at each historical time are weighted, and the weighted motion characteristics are the motion characteristics reflecting the degree of interaction between the obstacle and each other obstacle, so that the motion characteristics of each obstacle at the historical time are weighted, and the obtained weighted motion characteristics of each obstacle can be considered as the motion characteristics reflecting the degree of interaction between the obstacles at the historical time, namely the spatial interaction characteristics of each obstacle at the historical time.

S108: and inputting the space interaction characteristics of the obstacles into the recurrent neural subnetwork of the first model to obtain the space-time interaction characteristics of the obstacles.

The spatial interaction feature of each obstacle can reflect the interaction feature of each obstacle at the position of the historical time at each historical time, but it is difficult to accurately predict the future movement locus of each obstacle only from the isolated spatial interaction features of each obstacle at the past historical times. Therefore, in the present specification, the spatial interaction characteristics of each obstacle at each historical time can be input into the recurrent neural sub-network, and when the recurrent neural network processes the sequence information, the output at the previous time is fed back into the hidden layer of the recurrent neural network at the next time, and the characteristics of the output at the next time are affected. Specifically, the spatial interaction features of the obstacles at the historical moments in the time series can be input into the recurrent neural network, and the features output by the recurrent neural network at each historical moment actually include the features in the spatial interaction information at all the previous historical moments, that is, the features represent the relationship of the spatial interaction features in the time series and can be used as the space-time interaction features of the obstacles before the corresponding historical moments. Any recurrent neural network can be selected as a recurrent neural subnetwork in this specification, such as a Long Short-Term Memory (LSTM), which is not limited in this specification.

Furthermore, when only the position features are input into the attention network, namely the space-time interaction features are determined only according to the position information of the obstacles at each historical moment, the information represented by the space-time interaction features can be enriched according to other features extracted from the track information. Specifically, other features, such as speed features, included in the historical motion trajectory may be extracted, the spatio-temporal interaction features output by the recurrent neural sub-network are used as spatio-temporal interaction coarse features, the represented information of the spatio-temporal interaction coarse features is enriched according to the extracted other features, and the enriched spatio-temporal interaction coarse features are used as the spatio-temporal interaction features. The method comprises the steps of determining position space-time interaction characteristics according to position characteristics and space-time interaction coarse characteristics of obstacles, determining speed space-time interaction characteristics according to speed characteristics and space-time interaction coarse characteristics of the obstacles, and determining space-time interaction characteristics of the obstacles according to the position space-time interaction characteristics and the speed space-time interaction characteristics.

The recurrent neural network can output the space-time interactive characteristics of the corresponding historical time according to the input space interactive characteristics of each historical time, and the space-time interactive characteristics reflect the change of the space interactive characteristics of each obstacle with time before the historical time aiming at the output space-time interactive characteristics of each historical time.

S110: and inputting the space-time interactive characteristics output by the first model into a pre-trained second model, and determining the predicted movement locus of each obstacle through the second model.

And inputting the space-time interaction characteristics of each historical moment into a pre-trained second model, and determining the predicted motion trail of each obstacle through the second model. The following description will be given taking the spatiotemporal interaction characteristics output from the first model to the second model as spatiotemporal interaction characteristics of each obstacle at each historical time as an example.

As shown in fig. 1, the present specification provides a method for predicting a trajectory of an obstacle, which determines an interaction weight between each two obstacles according to a motion feature extracted from a historical motion trajectory of each obstacle, weights a corresponding motion feature with the determined interaction weight to obtain a weighted spatial interaction feature, obtains a spatial-temporal interaction feature of each obstacle according to the input spatial interaction feature by a recurrent neural sub-network, and determines a predicted motion trajectory of each obstacle according to the spatial-temporal interaction feature of each obstacle by a pre-trained second model, wherein the interaction weight reflects an influence degree between each obstacle and each other obstacle, so that the spatial interaction feature describes interactions between obstacles on the basis of the motion feature, and the predicted future trajectory of each obstacle has higher accuracy.

Furthermore, the second model shown in fig. 3 may be used to determine the predicted movement locus of each obstacle according to the input space-time interaction characteristics.

As shown in fig. 3, the second model includes an encoding end and a decoding end, the space-time interaction feature is input into the encoding end, the encoding end encodes the received space-time interaction feature, the encoding feature output by the encoding end is input into the decoding end, and the decoding end predicts the trajectory point of the obstacle at the target time according to the input encoding feature. The target time is a future time after the reference time, that is, the second model determines a future predicted movement locus of each obstacle so as to predict the track point of the target time. The output result of the trajectory prediction model may be the predicted trajectory point at the target time directly, or may be the predicted travel information (for example, speed and speed direction) between the previous historical time of each obstacle at the reference time and the target time, and the trajectory point of each obstacle at the target time is determined based on the acquired position information of each obstacle at the previous historical time of the reference time. The output prediction result is in any form, and the description is not limited.

Specifically, the encoding end may be an LSTM, and the encoding end encodes the spatiotemporal interaction features at each historical time to obtain the encoding features, where the spatiotemporal interaction features output by the first model are spatiotemporal interaction features at each historical time of each obstacle in the time sequence. Specifically, after each historical time is sequentially input to the encoding terminal in time series, the spatio-temporal interaction characteristics of the historical time closest to the reference time are input, and the obtained output of the encoding terminal can be considered as including the information of the spatio-temporal interaction characteristics of each historical time, so that the output can be used as the encoding characteristics output by the encoding terminal. Of course, the hidden layer feature output by the encoding end, which is obtained by using the temporal-spatial interaction feature of the history time closest to the reference time as an input, may be considered to carry information of the temporal-spatial interaction feature of each history time, and therefore, the hidden layer feature may also be used as the encoding feature output by the encoding end.

Generally, when the unmanned vehicle executes the method shown in fig. 1, the track points of each obstacle at several target moments can be predicted, so as to obtain the predicted movement track of each obstacle in a future period. Specifically, after the track point of each obstacle at the target time is predicted at the decoding end each time, the reference time may be re-determined, and by using the method in steps S100 to S110 in fig. 1, according to the predicted track point at the previous history time of the re-determined reference time, the space-time interaction feature at the previous history time of the re-determined reference time is determined, and the track point of each obstacle at the re-determined target time is predicted according to the space-time interaction feature, and this is repeated, and the predicted track of each obstacle at each track point predicted by the route is determined.

As shown in fig. 4, the decoding side firstly uses the coding features output by the coding side and the space-time interaction features of the historical time 2 as input, predicts the track points of each obstacle at the target time 1, then re-determines the reference time, and after the reference time is re-determined, the target time 1 becomes the historical time 3, determines the space-time interaction features of the historical time 3 by adopting the method in steps S100 to S110 in fig. 1, and predicts the track points of each obstacle at the target time 2 according to the space-time interaction features of the historical time 3.

When the decoding end performs the method shown in fig. 1 once, the decoding end may use the hidden layer feature output by the encoding end as the hidden layer feature initialized by the decoding end when predicting the trace point of the first target time, and predict the trace point of each obstacle at the first target time by using the space-time interaction feature of the history time closest to the reference time as an input. After the track point of the first target time is predicted, the decoding end can transmit an output hidden layer feature on the decoding end to the prediction of the track point of the newly determined target time, and predict the track point of each obstacle at the newly determined target time by taking the space-time interaction feature of the previous historical time of the newly determined reference time as input.

Taking fig. 5 as an example, in fig. 5, when the spatio-temporal interaction feature x1 at the previous historical time of the reference time is input to the decoding end, and the decoding end predicts the track point at the first target time, the encoding feature h0 (i.e., the hidden layer feature output by the encoding end) output by the encoding end is used as the hidden layer feature initialized by the decoding end, so that the decoding end predicts the track point of each obstacle at the first target time according to h0 and x1 (the predicted track point is not shown in fig. 5), and outputs the hidden layer feature h1 at the first target time. After the reference time is determined again, when the decoding end predicts the track point of each obstacle at the second target time, the hidden layer feature is the hidden layer feature h1 of the first target time output by the decoding end, and so on.

Due to various driving tendencies of the obstacles in reality, a predicted track obviously cannot depict various movement tendencies of the obstacles when the obstacles face complex environments with different paths. Therefore, in the trajectory prediction method of an obstacle provided in the present specification, several trajectories can be predicted for each obstacle. Specifically, a mode of splicing the space-time interaction characteristic input by the decoding end with random noise can be adopted, so that the decoding end can output a plurality of track points according to one input space-time interaction characteristic. Fig. 6A and 6B illustrate two methods of predicting the trajectory points of an obstacle at two target times when the method shown in fig. 1 is performed once, so as to obtain 4 predicted motion trajectories of the obstacle, wherein t0 is the last historical time of a reference time, and t1 and t2 are two target times of the predicted trajectory points, respectively:

in fig. 6A, when predicting the trajectory point of each obstacle at the time t1 based on the spatio-temporal interaction features of the obstacle at the time t0, the decoding end may respectively splice the spatio-temporal interaction features at the time t0 with 4 random noises determined in advance, and respectively predict 4 trajectory points p1 to p4 of the obstacle at the time t1 based on the spatio-temporal interaction features at the time t0 after the splice, whereas when predicting the trajectory point of the obstacle at the time t2, the decoding end may splice the random noises without the spatio-temporal interaction features at the time t1 corresponding to the 4 trajectory points at the time t1, and may obtain the predicted 4 trajectory points p5 to p8 of the obstacle at the time t2, thereby predicting 4 predicted trajectories of the obstacle at the time t0 to t 2.

In fig. 6B, when predicting the trajectory point of each obstacle at time t1 based on the spatio-temporal interaction characteristics of each obstacle at time t0, the decoding end may respectively splice the spatio-temporal interaction characteristics at time t0 with 2 random noises determined in advance, and respectively predict 2 trajectory points p1 and p2 of the obstacle at time t1 based on the spatio-temporal interaction characteristics at time t0 after the splice, and when predicting the trajectory point of the obstacle at time t2, respectively splice the spatio-temporal interaction characteristics at time t1 corresponding to p1 and p2 with two random noises, so that the spatio-temporal interaction characteristics at time t1 corresponding to p1 and p2 after the splice the random noises are used to predict 4 trajectory points p3 to p6 of the obstacle at time t2, thereby predicting 4 predicted trajectories of the obstacle at time t0 to t 2.

It can be seen that the number of predicted trajectories that can be determined for each obstacle is determined by the number and time of the spliced random noise, and the random noise can be spliced into any space-time interaction feature by any existing method, which is not described herein again.

Example two:

fig. 7 is a schematic flowchart of a process for training the first model and the second model shown in fig. 2, which is provided in an embodiment of the present disclosure, and includes:

s700: and determining each sample obstacle and a sample track corresponding to each sample obstacle.

S702: and according to a preset reference time, regarding each sample obstacle, regarding the track before the reference time as the initial track of the sample obstacle and regarding the track after the reference time as the marked track of the sample obstacle in the sample tracks corresponding to the sample obstacle.

Generally, the sample trajectory is a real historical trajectory of each obstacle collected in advance, specifically, the sensing device may be arranged in a real environment in advance, the collected obstacle is selected as the sample obstacle, and the real trajectory of the collected sample obstacle is used as the sample trajectory corresponding to each sample obstacle. Since the sample trajectory is usually the real acquired trajectory within the same target time period, it is often not necessary to intercept the acquired trajectory within a preset time period.

Dividing the sample track of each sample obstacle in a target time interval by taking a preset reference time as a boundary, wherein the track before the reference time is taken as the initial track of the sample obstacle, and the track after the reference time is taken as the labeled track of the sample obstacle.

S704: and for each sample obstacle, inputting the initial track of the sample obstacle into a first model, and extracting the motion characteristics of the sample obstacle through a characteristic extraction sub-network of the first model.

S706: and inputting the motion characteristics of each sample obstacle into the attention subnetwork of the first model to obtain the interaction weight of each sample obstacle.

S708: and weighting the motion characteristics of each sample obstacle by the determined interaction weight, and taking the weighted motion characteristics of each sample obstacle as the space interaction characteristics of each sample obstacle.

S710: and inputting the space interaction characteristics of the sample obstacles into the recurrent neural sub-network of the first model to obtain the space-time interaction characteristics of the sample obstacles.

S712: and inputting the space-time interaction characteristics output by the first model into a second model, and determining the predicted track of each sample obstacle through the second model.

The initial trajectory of each obstacle is input to a trajectory prediction model as shown in fig. 2, and the trajectory pre-storage model outputs a predicted trajectory of each sample obstacle, in the same manner as in fig. 1.

S714: and adjusting parameters in the first model and the second model by taking the minimum difference between the predicted track and the marked track of each sample obstacle as a target.

The method comprises the steps of taking the minimum difference between a predicted track and an annotated track of each sample obstacle as a target, adjusting parameters in a first model and a second model, further, when a plurality of predicted tracks of each sample obstacle are output by a track prediction model, the minimum difference between all predicted tracks and annotated tracks of each sample obstacle can be taken as a target adjustment parameter, or selecting a predicted track closest to the annotated track of each sample obstacle from the predicted tracks determined by the sample obstacles aiming at each sample obstacle, taking the predicted track as a target track, and adjusting parameters in the first model and the second model by taking the minimum difference between the target track and the annotated track of each sample obstacle as a target when parameters are adjusted.

The above description provides an obstacle trajectory prediction method exemplarily provided for the present specification, and based on the same idea, the present specification further provides a corresponding apparatus, a storage medium, and an unmanned device.

Fig. 8 is a schematic structural diagram of an obstacle trajectory prediction apparatus provided in an embodiment of the present disclosure, where the apparatus includes:

a track determining module 800, configured to determine historical movement tracks of the obstacles in history;

a feature extraction module 802, configured to, for each obstacle, input a pre-trained first model into a historical motion trajectory of the obstacle within a preset time period, and extract a motion feature of the obstacle through a feature extraction sub-network of the first model;

an attention weight module 804, configured to input the motion characteristics of each obstacle into the attention subnetwork of the first model, to obtain an interaction weight between each two obstacles;

a spatial feature module 806, configured to weight the motion feature of each obstacle by using the determined interaction weight, and use the weighted motion feature of each obstacle as a spatial interaction feature of each obstacle;

a space-time feature module 808, configured to input the space interaction features of the obstacles into the recurrent neural subnetwork of the first model, so as to obtain the space-time interaction features of the obstacles;

and the track prediction module 810 is used for inputting the space-time interaction characteristics output by the first model into a pre-trained second model, and determining the predicted motion track of each obstacle through the second model.

Optionally, the feature extraction module 802 is specifically configured to extract a position feature and a speed feature of the obstacle; the attention weighting module 804 is specifically configured to input the location characteristics of each obstacle into the attention subnetwork of the first model; the space-time feature module 808 is specifically configured to input the space interaction features of the obstacles into the recurrent neural subnetwork of the first model to obtain space-time interaction coarse features of the obstacles; and determining the space-time interaction characteristics of the obstacles according to the position characteristics, the speed characteristics and the space-time interaction coarse characteristics of the obstacles.

Optionally, the recurrent neural sub-network is a first LSTM long-short term memory network; the space-time characteristic module 808 is specifically configured to determine a position space-time interaction characteristic according to the position characteristic and the space-time interaction coarse characteristic of each obstacle; determining speed space-time interaction characteristics according to the speed characteristics and the space-time interaction coarse characteristics of each obstacle; and determining the space-time interaction characteristics of each obstacle according to the position space-time interaction characteristics and the speed space-time interaction characteristics.

Optionally, the second model specifically includes: an encoding end and a decoding end; the trajectory prediction module 810 is specifically configured to input the space-time interaction feature output by the first model to the coding end, so as to obtain an output coding feature of the coding end; and inputting the coding characteristics into the decoding end, and predicting the track points of each obstacle at the target moment through the decoding end.

Optionally, the historical motion trajectory of each obstacle is a track point of each obstacle at each historical time before the reference time; the target time is a future time after the reference time; and the space-time interaction characteristics output by the first model are the space-time interaction characteristics at each historical moment.

Optionally, the encoding end is a second LSTM; the trajectory prediction module 810 is specifically configured to input the temporal-spatial interaction feature of each historical time into the second LSTM, obtain a hidden layer feature output by a hidden layer in the second LSTM according to the temporal-spatial interaction feature of each historical time, and use the hidden layer feature as the encoding-end output encoding feature.

Optionally, the trajectory prediction module 810 is specifically configured to input the coding features and the space-time interaction features of the previous historical time of the reference time into the decoding end, and predict, through the decoding end, trajectory points of each obstacle at the target time.

Optionally, the decoding end is a third LSTM; the feature extraction module 802 is specifically configured to re-determine the reference time, and re-determine the target time according to the re-determined reference time; the trajectory prediction module 810 is specifically configured to input the hidden layer feature output by the decoding end at the previous historical time of the redetermined reference time and the space-time interaction feature at the previous historical time of the redetermined reference time into the decoding end, so as to predict, through the decoding end, trajectory points of each obstacle at the redetermined target time.

Optionally, the feature extraction module 802 is specifically configured to, for each obstacle, splice a space-time interaction vector corresponding to the obstacle in the space-time interaction features of the obstacle at the previous historical time of the reference time with a plurality of predetermined random noises, and input the spliced space-time interaction vectors to the decoding end; aiming at each spliced space-time interaction vector, determining the track point of the barrier at the target moment through the decoding end according to the space-time interaction vector and the coding characteristics output by the coding end

The present specification also provides a computer-readable storage medium storing a computer program, which is operable to execute the trajectory prediction method of an obstacle provided in fig. 1 above.

This specification also provides a schematic block diagram of the drone shown in figure 9. As shown in fig. 9, at the hardware level, the page loading device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, but may also include hardware required for other services. The processor reads a corresponding computer program from the non-volatile memory into the memory and then runs the computer program to implement the obstacle trajectory prediction method described in fig. 1 above. Of course, besides the software implementation, the present specification does not exclude other implementations, such as logic devices or a combination of software and hardware, and the like, that is, the execution subject of the following processing flow is not limited to each logic unit, and may be hardware or logic devices.

In the 90 s of the 20 th century, improvements in a technology could clearly distinguish between improvements in hardware (e.g., improvements in circuit structures such as diodes, transistors, switches, etc.) and improvements in software (improvements in process flow). However, as technology advances, many of today's process flow improvements have been seen as direct improvements in hardware circuit architecture. Designers almost always obtain the corresponding hardware circuit structure by programming an improved method flow into the hardware circuit. Thus, it cannot be said that an improvement in the process flow cannot be realized by hardware physical modules. For example, a Programmable Logic Device (PLD), such as a Field Programmable Gate Array (FPGA), is an integrated circuit whose Logic functions are determined by programming the Device by a user. A digital system is "integrated" on a PLD by the designer's own programming without requiring the chip manufacturer to design and fabricate application-specific integrated circuit chips. Furthermore, nowadays, instead of manually making an Integrated Circuit chip, such Programming is often implemented by "logic compiler" software, which is similar to a software compiler used in program development and writing, but the original code before compiling is also written by a specific Programming Language, which is called Hardware Description Language (HDL), and HDL is not only one but many, such as abel (advanced Boolean Expression Language), ahdl (alternate Hardware Description Language), traffic, pl (core universal Programming Language), HDCal (jhdware Description Language), lang, Lola, HDL, laspam, hardward Description Language (vhr Description Language), vhal (Hardware Description Language), and vhigh-Language, which are currently used in most common. It will also be apparent to those skilled in the art that hardware circuitry that implements the logical method flows can be readily obtained by merely slightly programming the method flows into an integrated circuit using the hardware description languages described above.

The controller may be implemented in any suitable manner, for example, the controller may take the form of, for example, a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro) processor, logic gates, switches, an Application Specific Integrated Circuit (ASIC), a programmable logic controller, and an embedded microcontroller, examples of which include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320, the memory controller may also be implemented as part of the control logic for the memory. Those skilled in the art will also appreciate that, in addition to implementing the controller as pure computer readable program code, the same functionality can be implemented by logically programming method steps such that the controller is in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers, embedded microcontrollers and the like. Such a controller may thus be considered a hardware component, and the means included therein for performing the various functions may also be considered as a structure within the hardware component. Or even means for performing the functions may be regarded as being both a software module for performing the method and a structure within a hardware component.

The systems, devices, modules or units illustrated in the above embodiments may be implemented by a computer chip or an entity, or by a product with certain functions. One typical implementation device is a computer. In particular, the computer may be, for example, a personal computer, a laptop computer, a cellular telephone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

For convenience of description, the above devices are described as being divided into various units by function, and are described separately. Of course, the functions of the various elements may be implemented in the same one or more software and/or hardware implementations of the present description.

As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.

The memory may include forms of volatile memory in a computer readable medium, Random Access Memory (RAM) and/or non-volatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of a computer-readable medium.

Computer-readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, a computer readable medium does not include a transitory computer readable medium such as a modulated data signal and a carrier wave.

It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.

As will be appreciated by one skilled in the art, embodiments of the present description may be provided as a method, system, or computer program product. Accordingly, the description may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the description may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

This description may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including memory storage devices.

The embodiments in the present specification are described in a progressive manner, and the same and similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for the system embodiment, since it is substantially similar to the method embodiment, the description is simple, and for the relevant points, reference may be made to the partial description of the method embodiment.

The above description is only an example of the present specification, and is not intended to limit the present specification. Various modifications and alterations to this description will become apparent to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present specification should be included in the scope of the claims of the present specification.

Claims

1. A method for predicting a trajectory of an obstacle, comprising:

determining historical movement tracks of all obstacles in history;

2. The method of claim 1, wherein extracting the motion characteristic of the obstacle specifically comprises:

3. The method of claim 2, where the recurrent neural subnetwork is a first LSTM long-short term memory network;

4. The method of claim 1, wherein the second model specifically comprises: an encoding end and a decoding end;

5. The method of claim 4, wherein the historical motion trajectory of each obstacle is a trajectory point of each obstacle at each historical time before the reference time;

the target time is a future time after the reference time;

6. The method of claim 5, wherein the encoding side is a second LSTM;

and inputting the space-time interaction characteristics of each historical moment into the second LSTM to obtain hidden layer characteristics output by the hidden layer in the second LSTM according to the space-time interaction characteristics of each historical moment, wherein the hidden layer characteristics are used as the coding characteristics output by the coding end.

7. The method according to claim 6, wherein the encoding features are input into the decoding end, and the track points of each obstacle at the target time are predicted by the decoding end, specifically comprising:

8. The method of claim 7, wherein the decoding side is a third LSTM;

9. The method according to claim 5, wherein predicting, by the decoding side, a trajectory point of each obstacle at a target time comprises:

10. The method of any of claims 1 to 9, wherein pre-training the first model and the second model specifically comprises:

11. The method of claim 10, wherein determining, via the second model, the predicted trajectory for each sample obstacle comprises:

12. An obstacle trajectory prediction device, comprising:

13. A computer-readable storage medium, characterized in that the storage medium stores a computer program which, when executed by a processor, implements the method of any of the preceding claims 1 to 11.

14. An unmanned aerial vehicle comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor when executing the program implements the method of any of claims 1 to 11.