WO2022022721A1 - 轨迹预测方法、装置、设备、存储介质及程序 - Google Patents
轨迹预测方法、装置、设备、存储介质及程序 Download PDFInfo
- Publication number
- WO2022022721A1 WO2022022721A1 PCT/CN2021/109871 CN2021109871W WO2022022721A1 WO 2022022721 A1 WO2022022721 A1 WO 2022022721A1 CN 2021109871 W CN2021109871 W CN 2021109871W WO 2022022721 A1 WO2022022721 A1 WO 2022022721A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- time
- information
- series
- position information
- neural network
- Prior art date
Links
- 238000000034 method Methods 0.000 title claims abstract description 90
- 238000004590 computer program Methods 0.000 claims abstract description 9
- 238000013528 artificial neural network Methods 0.000 claims description 112
- 230000004927 fusion Effects 0.000 claims description 76
- 230000007613 environmental effect Effects 0.000 claims description 25
- 230000015654 memory Effects 0.000 claims description 21
- 238000012549 training Methods 0.000 claims description 16
- 230000008569 process Effects 0.000 description 19
- 230000002457 bidirectional effect Effects 0.000 description 16
- 238000010586 diagram Methods 0.000 description 16
- 230000006870 function Effects 0.000 description 14
- 239000011159 matrix material Substances 0.000 description 13
- 230000008859 change Effects 0.000 description 12
- 239000013598 vector Substances 0.000 description 12
- 238000004891 communication Methods 0.000 description 11
- 238000012545 processing Methods 0.000 description 8
- 230000002123 temporal effect Effects 0.000 description 6
- 230000003068 static effect Effects 0.000 description 5
- 238000013527 convolutional neural network Methods 0.000 description 4
- 230000000007 visual effect Effects 0.000 description 4
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 241001465754 Metazoa Species 0.000 description 2
- 230000006399 behavior Effects 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000011156 evaluation Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000008447 perception Effects 0.000 description 2
- 230000002441 reversible effect Effects 0.000 description 2
- 230000006403 short-term memory Effects 0.000 description 2
- KLDZYURQCUYZBL-UHFFFAOYSA-N 2-[3-[(2-hydroxyphenyl)methylideneamino]propyliminomethyl]phenol Chemical compound OC1=CC=CC=C1C=NCCCN=CC1=CC=CC=C1O KLDZYURQCUYZBL-UHFFFAOYSA-N 0.000 description 1
- 240000004050 Pentaglottis sempervirens Species 0.000 description 1
- 235000004522 Pentaglottis sempervirens Nutrition 0.000 description 1
- 230000001133 acceleration Effects 0.000 description 1
- 230000009471 action Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000013480 data collection Methods 0.000 description 1
- 201000001098 delayed sleep phase syndrome Diseases 0.000 description 1
- 208000033921 delayed sleep phase type circadian rhythm sleep disease Diseases 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 230000001815 facial effect Effects 0.000 description 1
- 230000010365 information processing Effects 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 230000000366 juvenile effect Effects 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
Images
Classifications
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- B60W60/00—Drive control systems specially adapted for autonomous road vehicles
- B60W60/001—Planning or execution of driving tasks
- B60W60/0027—Planning or execution of driving tasks using trajectory prediction for other traffic participants
- B60W60/00274—Planning or execution of driving tasks using trajectory prediction for other traffic participants considering possible movement changes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/044—Recurrent networks, e.g. Hopfield networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/049—Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- B60W2554/00—Input parameters relating to objects
- B60W2554/40—Dynamic objects, e.g. animals, windblown objects
- B60W2554/402—Type
- B60W2554/4023—Type large-size vehicles, e.g. trucks
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- B60W2554/00—Input parameters relating to objects
- B60W2554/40—Dynamic objects, e.g. animals, windblown objects
- B60W2554/402—Type
- B60W2554/4029—Pedestrians
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- B60W2554/00—Input parameters relating to objects
- B60W2554/40—Dynamic objects, e.g. animals, windblown objects
- B60W2554/404—Characteristics
- B60W2554/4041—Position
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- B60W2554/00—Input parameters relating to objects
- B60W2554/40—Dynamic objects, e.g. animals, windblown objects
- B60W2554/404—Characteristics
- B60W2554/4045—Intention, e.g. lane change or imminent movement
Definitions
- the embodiments of the present disclosure relate to the technical field of intelligent driving, and relate to, but are not limited to, a trajectory prediction method, apparatus, device, storage medium, and program.
- the intrinsic correlation of the historical motion of pedestrian or vehicle trajectories is mainly considered, such as using the historical trajectory position information of pedestrians or vehicles to make trajectory prediction in the future.
- Embodiments of the present disclosure provide a trajectory prediction method, apparatus, device, storage medium, and program.
- An embodiment of the present disclosure provides a trajectory prediction method, the method is executed by an electronic device, and the method includes:
- the time-series location information is the location information of the object at different time points within a preset time period
- the time-series attitude information is the The posture information of the object at different time points within a preset duration
- the posture information at the different time points includes the orientation information of the multiple parts of the object at the different time points;
- the future trajectory of the object is determined according to the time-series position information, the time-series posture information, and the motion intention.
- the motion intention of the object can be determined more accurately; then, the future trajectory of the object is predicted based on the estimated motion intention, time-series position information and time-series attitude information as input, and in the predicted
- the orientation information of the object is used in the process; in this way, by combining the time-series position information, the time-series pose information and the motion intention, and considering the orientation information of the object, the accuracy of predicting the future trajectory of the object can be effectively improved.
- An embodiment of the present disclosure provides a trajectory prediction device, the device includes:
- an intention determination module configured to determine the motion intention of the object according to the time-series position information and time-series attitude information of the object; wherein, the time-series position information is the position information of the object at different time points within a preset time period, and the The time sequence posture information is the posture information of the object at different time points within a preset duration; the posture information at different time points includes orientation information of multiple parts of the object at the different time points;
- the future trajectory determination module is configured to determine the future trajectory of the object according to the time-series position information, the time-series attitude information and the motion intention.
- An embodiment of the present disclosure provides a computer storage medium, where computer-executable instructions are stored thereon, and after the computer-executable instructions are executed, the above-mentioned trajectory prediction method can be implemented.
- An embodiment of the present disclosure provides a computer device, where the computer device includes a memory and a processor, where computer-executable instructions are stored in the memory, and the processor can implement the above-mentioned when executing the computer-executable instructions in the memory.
- the trajectory prediction method described above is a technique that uses the trajectory to predict the trajectory of the computer-executable instructions to implement the above-mentioned when executing the computer-executable instructions in the memory.
- Embodiments of the present disclosure further provide a computer program, the computer program includes computer-readable codes, and when the computer-readable codes are executed in an electronic device, the processor of the electronic device executes the code to implement the above-mentioned The trajectory prediction method described above.
- Embodiments of the present disclosure provide a trajectory prediction method, apparatus, device, storage medium, and program, which use the time-series position information and time-series attitude information of the object as input to estimate the motion intention of the object.
- the future trajectory of the object can be predicted, and the orientation information of the relevant object is used in the prediction process.
- the accuracy of predicting the future trajectory of the object can be effectively improved.
- FIG. 1 is a schematic diagram of an implementation flowchart of a trajectory prediction method according to an embodiment of the present disclosure
- FIG. 2 is a schematic diagram of a system architecture to which the trajectory prediction method according to the embodiment of the present disclosure can be applied;
- FIG. 3A is a schematic flowchart of another implementation of a trajectory prediction method according to an embodiment of the present disclosure
- 3B is a schematic flowchart of another implementation of the trajectory prediction method according to an embodiment of the present disclosure.
- 4A is a schematic diagram of the distribution of objects in the data set according to an embodiment of the present disclosure and the distribution of intentions of each object type;
- FIG. 4B is a schematic diagram of another distribution of objects in the data set and the intent of each object type according to an embodiment of the present disclosure
- FIG. 4C is another schematic diagram of the distribution of objects in the data set and the intent of each object type according to an embodiment of the present disclosure
- FIG. 4D is another schematic diagram of distribution of objects in the data set and the intent of each object type according to an embodiment of the present disclosure
- FIG. 5 is a schematic framework diagram of a trajectory prediction system provided by an embodiment of the present disclosure.
- FIG. 6 is a structural diagram of an implementation framework of a trajectory prediction method according to an embodiment of the present disclosure.
- FIG. 7 is a schematic structural composition diagram of a trajectory prediction apparatus according to an embodiment of the present disclosure.
- FIG. 8 is a schematic structural diagram of a computer device according to an embodiment of the present disclosure.
- a trajectory prediction method is applied to a computer device.
- the computer device may include objects or non-objects.
- the functions implemented by the method may be implemented by calling a program code by a processor in the computer device.
- the program code may be stored in In the computer storage medium, it can be seen that the computer device includes at least a processor and a storage medium.
- FIG. 1 is a schematic diagram of an implementation flowchart of a trajectory prediction method according to an embodiment of the present disclosure, as shown in FIG. 1 , and is described in conjunction with the method shown in FIG. 1 :
- Step S101 Determine the motion intention of the object according to the time-series position information and the time-series posture information of the object.
- the time-series position information is the position information of the object at different time points within a preset time period
- the time-series attitude information is the attitude information of the object at different time points within the preset time period
- the objects are movable objects in the traffic environment, including human objects, such as pedestrians or cyclists. It also includes non-human objects, including but not limited to at least one of the following: vehicles with various functions (such as trucks, cars, motorcycles, bicycles, etc.), vehicles with various wheel numbers (such as four-wheeled vehicles) Vehicles, two-wheeled vehicles, etc.) and any movable equipment, such as robots, aircraft, blind guides, smart toys, toy cars, etc.
- the posture information at different time points includes orientation information of one or more parts of the human object at the different time points.
- Step S102 Determine the future trajectory of the object according to the time-series position information, the time-series posture information, and the motion intention.
- the movement intention is the movement tendency of the object in the future period, for example, the object is a pedestrian, and the movement intention is whether he intends to pass a traffic light or whether he intends to go straight in the future period.
- the time-series position information, the time-series pose information and the motion intention are combined and input into the neural network as a whole to predict the future trajectory of the object.
- the time-series position information and the time-series attitude information are spliced together in a preset manner as a fusion feature, and the fusion feature and motion intention are jointly referenced to predict the future trajectory of the object.
- the time-series position information and time-series pose information of the object are used to estimate the pedestrian's intention (eg, whether to cross the road, etc.), thus, by considering the more abundant time-series location of the moving object information and timing pose information, can more accurately determine the motion intention of the moving object; then, predict the future trajectory of the object based on the estimated object intention and the output of the learning model, and use multiple The time series information of the direction of each part in the part; in this way, by combining the time series information of the position and posture with the motion intention, the future trajectory of the moving object can be predicted, which can effectively improve the accuracy of the future trajectory prediction.
- FIG. 2 shows a schematic diagram of a system architecture to which a trajectory prediction method according to an embodiment of the present disclosure can be applied; as shown in FIG. 2 , the system architecture includes an acquisition terminal 201 , a network 202 and a trajectory prediction terminal 203 .
- the acquisition terminal 201 reports the time series position information and time series attitude information of the object to the trajectory prediction terminal 203 through the network 202 .
- the trajectory prediction terminal 203 in response to the time-series position information and time-series attitude information of the object, firstly determines the motion intention of the object according to the time-series location information and time-series attitude information of the object; then, according to the time-series location information, the time-series attitude information information and the motion intention to determine the future trajectory of the object. At the same time, the trajectory prediction terminal 203 uploads the future trajectory of the object to the network 202 and sends it to the acquisition terminal 201 through the network 202 .
- the acquisition terminal 201 may include an image acquisition device, and the trajectory prediction terminal 203 may include a visual processing device or a remote server with visual information processing capability.
- Network 202 may employ wired or wireless connections.
- the trajectory prediction terminal 203 is a visual processing device
- the acquisition terminal 201 can be connected to the visual processing device through a wired connection, such as data communication through a bus;
- the trajectory prediction terminal 203 is a remote server, the acquisition terminal 201 can Data exchange with remote server through wireless network.
- the acquisition terminal 201 may be a vision processing device with a video capture module, or a host with a camera.
- the trajectory prediction method of the embodiment of the present disclosure may be executed by the acquisition terminal 201 , and the above-mentioned system architecture may not include the network 202 and the trajectory prediction terminal 203 .
- Step S101 can be implemented by the following steps, as shown in FIG. 3A , combined with FIG. 3A Make the following instructions:
- Step S11 according to the time-series position information and the time-series attitude information, obtain the environment information of the environment where the object is located.
- the environmental information includes at least one of the following: road information, pedestrian information, or traffic light information.
- the time sequence position information of the object and the orientation information in the time sequence attitude information the world map is intercepted to obtain the local map area of the environment where the object is located, so as to obtain the local map information of the object, and the local map information is determined as the environmental information.
- the time-series position information and time-series attitude information of an object at a historical moment can be obtained through the following process: first, determine at least two historical moments whose duration from the current moment is less than or equal to a preset duration; then, obtain the object at least two historical moments Time series position information and time series attitude information.
- the acquired time-series position information and time-series attitude information of multiple historical moments whose duration from the current moment is less than the preset duration can be improved.
- the current time is 10:05:20
- the time-series position information and time-series attitude information of objects less than 5 seconds away from the current time are obtained, that is, from 10:05:15 to 10:05:20 between the temporal position information and temporal pose information of the object.
- the time series position information and the time series attitude information are related to the attributes of the object.
- the time-series position information and the time-series attitude information include at least: the person's time-series location information, body orientation and face orientation; if a set of time-series locations is acquired every 1 second between this historical period information and time-series pose information, for example, if the time-series position information and the time-series pose information include the object's body orientation, face orientation, and the position of the object, then determine the object's body orientation, face The orientation and location of the object.
- time 10:05:15 to 10:05:20 obtain a set of time-series position information and time-series attitude information every 1 second, that is, there are 5 time point distances, then determine the body orientation, face orientation and the location of the object.
- the time-series position information and the time-series attitude information include at least: time-series location information of the moving device, head orientation of the device, and driving instruction information of the moving device.
- the time-series position information and the time-series attitude information include: the time-series position of the vehicle, the heading of the vehicle, and the driving instruction information of the vehicle; wherein, the driving instruction information includes but is not limited to at least one of the following: driving direction, driving speed and The status of the headlights (for example, the status of the turn signals), etc.
- the obtained rich time-series position information and time-series attitude information are used as the basis for intercepting the world map, and the environment information of the environment where the object is located is obtained. That is to say, the environmental information can be intercepted from the world map by using the position information of the object and the orientation information of the object in the time series position information and the time series attitude information to determine the road structure and sidewalk in the local map where the object is currently located. information and traffic light information on the road; in this way, by obtaining the rich time-series location information and time-series attitude information of the object, to predict the environmental information such as the road structure where the object is currently located, the accuracy of the map division can be improved. Even when there are few observation points (even only one frame of observation data), it can still give reasonable prediction results.
- Step S12 fuse the environment information, the time-series position information and the time-series attitude information to obtain a fusion feature.
- the time-series position information and the time-series pose information include: body orientation, face orientation, and the position of the object; separate the body orientation, face orientation, and the position of the object; input three independent In the first neural network, time-series position information and time-series attitude information used to indicate the changes of body orientation, face orientation and the position of the object in time are obtained respectively; the time-series location information and time-series attitude information are input into the second neural network , obtain the adjusted time-series position information and adjusted time-series attitude information; input multiple different distances into the third neural network (for example, a fully connected model) to obtain the body orientation, face orientation and the position of the object at the distance Corresponding weight; multiply the weight by the adjusted time-series position information and adjusted time
- the time-series position information, the time-series attitude information, and the environment information are acquired at the same time point. For example, all of them are for 5 time points in the historical period, so splicing the multiplication result with the environmental information obtained after encoding the local map area can be achieved by the following methods: combining the matrix representing the multiplication result with the environmental information representing the environment The matrix of information is spliced together according to the row or column to form a matrix, that is, the fusion feature is obtained.
- the matrix representing the multiplication result is a matrix with 3 rows and 5 columns
- the matrix representing the environmental information is a matrix with 6 rows and 5 columns
- the two matrices are spliced together according to the columns, and a matrix with 9 rows and 5 columns is obtained, that is, the fusion feature is obtained. .
- Step S13 Determine the motion intention of the object according to the fusion feature.
- the movement intention can be understood as: the movement tendency of the object during the movement process.
- the intention classification includes but is not limited to one or more of the following: turn left, turn right, go straight , stand still, make a U-turn, accelerate, decelerate, cross the road, wait for a red light, and walk backwards, etc.
- the intent classification includes but is not limited to one or more of the following: turn left, turn right, go straight, stand still, change lanes left, change lanes right, accelerate, decelerate, overtake, reverse, and wait for red lights, etc. .
- the fusion feature is decoded by using a fully connected layer network to obtain the probability that the fusion feature is each category in the preset category library, and the category with the highest probability is regarded as the most probable fusion feature.
- category, and predicting the motion intent of an object based on the most probable category can improve the accuracy of predicting intent.
- step S102 may be implemented in the following manner:
- Step S14 Determine the future trajectory of the object according to the fusion feature and the motion intention.
- the future trajectory of the object in the future period can be predicted by fusing the feature and the motion intention; it is also possible not to predict the motion intention of the object, and only the first neural network can be used to perform multiple iterations on the fused feature, Predict the future trajectory of an object over a future period. For example, by decoding the second adjusted time-series position information and time-series attitude information, the future trajectory of the predicted object can be obtained; in this way, trajectory prediction can be performed by using various time-series location information and time-series attitude information, even if there are few observation points (or even only One frame of observation data), or in scenarios such as sudden acceleration, deceleration, and sudden turns of the object, the accuracy of future trajectory prediction can still be guaranteed.
- the map information is integrated into the time series position information and the time series attitude information to predict the motion intention, which can improve the accuracy of the prediction of the motion intention. Then, the future trajectory of the object is predicted based on the motion intention. Improve the accuracy of trajectory prediction.
- the world map may be intercepted through the location information of the object and the orientation information of the object to determine the local map area of the current environment of the object, that is, step S11 may This is achieved through the following process:
- Step S111 intercepting the world map according to the position information and orientation information of the object at the historical moment to obtain a local map area of the environment where the object is located.
- the orientation information and the position information in the time series attitude information appear in pairs, that is, the position information of the object and the orientation information at the position are determined at a certain historical moment.
- the object is a human body (for example, a pedestrian or a cyclist)
- the current road structure where the person is located is determined according to the person's position information and the person's body orientation, so as to intercept the world map to determine the current road structure of the pedestrian.
- the local map area in which it is located is located.
- the object is a moving device such as a vehicle
- the current road where the vehicle is located is determined according to the location information of the vehicle and the heading of the vehicle, so as to intercept the world map to determine the local map area where the vehicle is currently located.
- the interception of the world map can be realized in the following manner: according to each position in the plurality of time series position information and the orientation of the object when the position is located, determine the local map area of the environment where the object is located, and obtain a plurality of local maps area.
- the orientation of the object when it is in this position can be understood as the orientation of multiple parts when the object is in this position. In this way, by referring to the orientations of multiple parts of the object at one location, the local map area of the object can be delineated, which can improve the accuracy of the determined environmental information, thereby improving the accuracy of future trajectory prediction.
- a local map area of the environment where the object is located is delineated in the world map. For example, taking this position as the center and along the facing direction, a rectangular area is delineated as a local map area of the environment where the object is located. In this way, multiple locations and multiple orientation information under each location can determine multiple local map areas.
- the multiple local map areas are encoded to obtain multiple encoded maps, that is, environmental information. In this way, taking the location as the center and referring to the orientation information, the local map area is demarcated, so that the map information included in the demarcated local map area has a high correlation with the object, that is, the effectiveness of the environmental information can be improved.
- Step S112 Encode the elements in the local map area to obtain the environment information.
- each element represents map information of a corresponding area
- the map information includes at least one of the following: road structure information, sidewalks, or road traffic lights.
- the elements of this local map area are encoded as masks, and each codeword represents the map information of the corresponding area.
- the environmental information is a matrix including 1 and 0, where 1 represents a sidewalk, 0 represents a road danger zone, and so on.
- the multiple environmental information and the corresponding time series position information and time series attitude information are fused to obtain multiple sets of fusion features, and the motion intention of the object is predicted by classifying the fusion features.
- the structure of the first neural network is not limited, including but not limited to a convolutional neural network, a long short-term memory network (Long Short-Term Memory, LSTM), etc.
- LSTM Long Short-Term Memory
- input the time-series position information and time-series attitude information of multiple historical moments for example, taking the object as a pedestrian as an example, multiple body orientations, multiple face orientations, and multiple positions of the object respectively
- the bidirectional LSTM network Obtain the time-series position information and time-series attitude information which are used to indicate the time-series position information and time-series attitude information, respectively
- the distance is input into the fully connected model, and the weight corresponding to the body orientation, face orientation and the position of the object under the distance is obtained; the weight is multiplied by the adjusted time-series position information and time-series attitude information to obtain multiple
- a local map area of the object is defined according to the position and orientation of the object; and mask coding is performed on the local map area to obtain environmental information, and each codeword represents the map information of the area.
- the time-series position information and time-series attitude information of the object are combined with the encoded map to predict the intention of the object, and then predict the future trajectory of the object, which can improve the accuracy of the obtained future trajectory.
- time-series modeling is performed respectively to obtain the time-series changes of each time-series location information and time-series attitude information, and then each time-series location
- the time-series position information of the information and the time-series attitude information and the time-series attitude information and the environment information are fused to obtain the fusion feature, that is, step S12, which can be realized through the following process, as shown in FIG.
- FIG. Another schematic flow chart of implementation is described below in conjunction with the steps shown in FIG. 3A and FIG. 3B :
- Step S201 predicting time-series location information and time-series attitude information in a future period according to the time-series location information and time-series attitude information through a first neural network.
- step S201 can be performed by The following process is implemented:
- the time-series position information and time-series attitude information (that is, multiple time-series location information and time-series attitude information) of each historical moment are arranged in time sequence; then, the arranged multiple time-series location information and time-series attitude information, Input the first neural network to obtain multiple time-series position information and time-series attitude information.
- the first neural network may be a bidirectional LSTM network, and the number of the first neural networks matches the types contained in the time-series position information and the time-series attitude information.
- the time-series position information and time-series pose information include: the object's body orientation, face orientation, and the location of the object; then, the first neural network is three independent bidirectional LSTM networks.
- the time-series position information and the time-series attitude information include: the head orientation of the object, the state of the headlights, and the location of the object; then, the first neural network is three independent bidirectional LSTM networks.
- a plurality of time series position information and time series attitude information are input into the bidirectional LSTM network to obtain corresponding time series position information and time series attitude information. For example, if the object is a pedestrian, input the pedestrian's body orientation, face orientation and the position of the pedestrian at different times into three independent bidirectional LSTM networks, respectively, and obtain multiple time series position information corresponding to the body orientation at different times.
- time series posture information indicating the change of body orientation in time
- multiple time series position information and time series posture information corresponding to the face orientation at different times indicating the change in face orientation in time
- the pedestrian is at different times
- a plurality of time-series position information and time-series attitude information corresponding to the location of the object indicating the temporal change of the location of the object.
- the object is a vehicle
- Corresponding multiple time-series position information and time-series attitude information (indicating the change of the head orientation in time), multiple time-series location information and time-series attitude information corresponding to the state of the lights at different times (indicating the change of the state of the lights in time) situation) and a plurality of time-series position information and time-series attitude information corresponding to the position of the vehicle at different times (indicating the change of the position of the vehicle in time).
- the first neural network is a trained neural network, which can be obtained by training in the following manner:
- the time-series location information and time-series attitude information of the object at historical moments are used as the input of the first neural network, and based on each set of time-series location information and time-series attitude information, it is predicted that the object corresponds to the future time period
- the predicted time-series position information and time-series attitude information are obtained, so as to obtain the predicted time-series location information and time-series attitude information.
- the object here can be understood as a sample object. For example, pedestrians or animals in the sample images of the preset dataset.
- the preset data set at least includes time-series position information and time-series attitude information of the sample object in the sample image.
- the preset data set at least includes the body orientation, face orientation of the sample object or the position of the sample object in the sample image.
- time-series position information and time-series attitude information in the future period are fused with the environment information of the environment where the object is located to obtain the fusion prediction feature.
- the time-series position information and time-series attitude information predicted by the first neural network to be trained and the environment information are fused to obtain the fusion prediction feature.
- future trajectories of objects in future time periods are predicted, at least according to the fused predicted features.
- the first neural network is used to iterate the fusion prediction feature, thereby predicting the future trajectory of the object in the future time period.
- the trained fully-connected network is used for classification to predict the motion intention of the object, and the motion intention and the fusion prediction feature are combined to predict the future trajectory of the object.
- the first prediction loss of the first neural network to be trained with respect to the future trajectory is determined.
- the first prediction loss is determined based on the first neural network, the future trajectory, and the ground truth trajectory of the object.
- the first prediction loss includes at least one of the following: the average number of failed predictions of future trajectories whose length is greater than a preset threshold, the success rate of future trajectories under error thresholds corresponding to different distances, or the end position and true value of future trajectories The error between the end positions of the trajectories.
- the average number of failed predictions of future trajectories whose length is greater than the preset threshold can be understood as: for future trajectories whose length is greater than the preset threshold (for example, predicting the future trajectories in the future 5s); for each moment in the future trajectories All predictions are made, the historical trajectory of the first 5 seconds of the moment is used as input, and the future trajectory of the next 5 seconds is predicted; then, the motion prediction trajectory needs to be predicted multiple times, so as to obtain the results of multiple predictions; statistics of multiple predictions The number of failures in the result; then divide the number of failures by the length of the future trajectory to achieve normalization; since there are many future trajectories with a trajectory length greater than a preset threshold, divide the number of failed predictions in each trajectory by The length of the future trajectory, multiple normalized values are obtained; finally, these multiple normalized values are averaged to obtain the average number of failure predictions for each trajectory.
- the success rate of the predicted future trajectory under the error thresholds corresponding to different distances can be understood as different error thresholds are preset for different distances. For example, the larger the distance, the larger the set error threshold. If the error of the obtained future trajectory is less than the error threshold at a certain distance, it is determined that the prediction is successful. In this way, the performance of the predicted future trajectory under different error thresholds can be characterized, and based on this, the detail effect of the neural network can be improved.
- the error between the end position of the future trajectory and the end position of the true value trajectory can be understood as the difference between the end point of the future trajectory and the end point of the true value trajectory.
- the network parameters of the first neural network are adjusted to train the first neural network.
- the first prediction loss may be directly used to adjust the network parameters. For example, the average number of failed predictions of predicted future trajectories with a length greater than a preset threshold, the success rate of predicted future trajectories under error thresholds corresponding to different distances, or the difference between the end position of the future trajectory and the end position of the true value trajectory At least one of the errors of the network parameters is adjusted.
- the performance of the first neural network obtained by training is better.
- the above-mentioned reference adjustment process can also be implemented in the following ways. First, determine the size of the success rate and the average number of failure predictions, and when the success rate is less than the average number of failure predictions, determine all of the predictions for this time. The future trajectory fails; then, the network parameters of the neural network are adjusted using at least one of the average position error, the average number of failure predictions, the success rate, or the error. In this way, the predicted future trajectory in the training process is evaluated through multiple evaluation criteria, so that the network parameters of the neural network can be adjusted more accurately, so that the future trajectory predicted by the adjusted first neural network is more accurate.
- Step S202 splicing the time-series position information, time-series attitude information and the environment information in the future period according to a preset method to obtain the fusion feature.
- the time-series position information and time-series attitude information and the corresponding local map can be understood as the time-series location information and time-series attitude information belonging to a group of time-series location information and time-series attitude information and according to this group of time-series
- the location information and orientation information in the location information and the time series attitude information are intercepted from the local map.
- a plurality of time-series position information and time-series attitude information are spliced with the local map in a one-to-one correspondence according to a preset method to obtain a fusion feature; the preset method may be according to the sequence of inputting the time-series location information and the time-series attitude information into the neural network.
- the three kinds of time-series position information and time-series attitude information are sequentially input into the neural network (for example, LSTM network); then in the order from the pedestrian's body orientation, face orientation to the pedestrian's position, the time-series position information and time-series pose information and the corresponding local map are spliced to obtain fusion features. Then, a fully connected network is used to decode the fusion features to predict the pedestrian's motion intention, that is, whether the pedestrian wants to turn left, turn right, go straight, stand still, or turn around.
- the neural network For example, LSTM network
- the time-series position information and the time-series attitude information include: vehicle front time-series location information and time-series attitude information, position time-series location information and time-series attitude information, and vehicle lamp state time-series location Information and timing attitude information, the three timing position information and timing attitude information are in the order of the front timing position information and timing attitude information, the position timing position information and timing attitude information, and the light state timing position information and timing attitude information.
- the neural network for example, LSTM network
- the neural network in turn; then according to the sequence from the front timing position information and timing attitude information, position timing position information and timing attitude information to the light state timing position information and timing attitude information, the timing position information
- the fusion features are obtained by splicing with the time series pose information and the corresponding local map.
- a fully connected network is used to decode the fused features to predict the motion intention of the vehicle, that is, whether the vehicle wants to turn left, turn right, go straight, stand still, change lanes left, change lanes right, overtake or reverse, etc.
- the above-mentioned steps S201 and S202 provide a method for realizing "merging the environment information with the time-series position information and the time-series attitude information to obtain fusion features".
- Step S203 the second neural network is used to determine the confidence level that the fusion feature is each intent category in the intent category library.
- the second neural network may be a fully connected network for classifying fused features. For example, by using a fully connected network to predict the possibility that the fusion feature is each intent category in the intent category library, the confidence level of each intent category can be obtained.
- the corresponding intent category library includes: turn left, turn right, go straight, stand still or turn around, etc.; a fully connected network is used to predict that the fusion features may be left turn, Confidence for each intent category in right turn, straight, stationary or U-turn, etc., eg, probability for each intent category.
- the second neural network is a trained neural network, which can be obtained by training in the following manner:
- the fusion feature is input into the second neural network to be trained, and the motion intention of the object is predicted as the confidence level of each intention category in the intention category library.
- the second neural network to be trained may be a fully connected network to be trained, and the fusion feature is input into the second neural network to be trained to predict the motion intention of the object as the probability of each category in the category library.
- the object may be a sample object, and the fusion feature of the sample object is input into the second neural network to be trained to classify the motion intention of the sample object.
- a second prediction loss of the second neural network's confidence about each intent category is determined based on the object's ground-truth intent.
- the second prediction loss may be a categorical cross-entropy loss function.
- the network parameters of the second neural network to be trained are adjusted to train the second neural network to be trained to obtain the second neural network.
- the network parameters of the second neural network to be trained are adjusted by using the classified cross-entropy loss function, so as to train the second neural network to be trained, and the trained second neural network is obtained.
- the loss function is the sum of the first prediction loss and the second prediction loss.
- Step S204 Determine the motion intention of the object according to the intention category with the highest confidence.
- the category with the highest probability is selected, and the category with the highest probability is determined as the movement intention of the object. For example, using a fully connected network to predict that the fusion feature may be left turn, right turn, straight, stationary or U-turn, etc.
- the probability of each category is: 0.1, 0.2, 0.2, 0.1 and 0.4, then the category with the highest probability is U-turn , indicating that the most probable movement intention of the object is to turn around, so that it is finally determined that the movement intention of the object is to turn around.
- the neural network can accurately predict the most probable motion intention by classifying the intention category of the fusion feature.
- the above steps S203 and S204 provide a method for realizing "determining the motion intention of the object according to the fusion feature".
- the fusion feature is classified by using a fully connected network, so that accurate prediction can be achieved.
- Step S205 Determine the iterative step size according to the length of the future period.
- the length of the future period is 3 seconds, and the iteration step is determined to be 0.3 seconds.
- Step S206 use the first neural network to iterate the motion intention and the fusion feature to obtain the coordinates of the object under each iteration step size.
- the number of iterations required is determined first according to the iteration step size and the length of the future period, and then the first neural network is used to iterate the motion intent and the fusion feature to obtain the coordinates of each iteration.
- the length of the future period is 3 seconds
- the iterative step is determined to be 0.3 seconds
- the number of iterations required is 10
- the first neural network is used to perform successive iterations on the motion intent and the fusion feature
- Step S207 Determine the future trajectory according to the coordinates of the object under each iteration step.
- the intention prediction of the object and the trajectory prediction are combined into a system, and the coordinates under each step length are obtained through step-by-step iteration, and the future trajectory is predicted, so that the efficiency and prediction of the final predicted future trajectory can be improved. Effect.
- the following processes are further included:
- LSTM networks are used to adjust each time-series position information and time-series attitude information to obtain the first adjusted time-series location information and time-series attitude information.
- a bidirectional LSTM network or a model of a fully connected layer may be used to adjust the time-series position information and the time-series attitude information; each time-series location information and The time series attitude information is input into the bidirectional LSTM network or the model of the fully connected layer, and a weight matrix is obtained. Then, the weight matrix is divided into parts of the same type as the time series position information and the time series attitude information, and each part corresponds to the time series position. Each time-series position information and time-series attitude information in the information and the time-series attitude information are multiplied to obtain a plurality of first adjusted time-series location information and time-series attitude information.
- the time-series position information and the time-series posture information include: the object's body orientation, face orientation, and the position of the object; input these three features into three
- three kinds of time-series position information and time-series attitude information corresponding to the three features are obtained;
- the order of the positions of is input into the second neural network in turn, and a weight matrix is obtained;
- the weight matrix is divided into three parts, the first part is multiplied by the time-series position information and time-series attitude information at different times, and the second part is combined with The time-series position information and time-series attitude information at different times are multiplied, and the third part is multiplied with the time-series location information and time-series attitude information of objects at different times to obtain the first adjusted time-series location information and time-series attitude information including three characteristics.
- a weight vector is obtained, and the weight vector is used to adjust each first adjusted time-series location information and time-series attitude information , and obtain the second adjusted timing position information and timing attitude information.
- a fully connected model is adopted, and for multiple input distances, a weight vector corresponding to each time-series position information and time-series attitude information at the multiple positions is output. And multiply the obtained weight vector corresponding to each time-series position information and time-series attitude information with the first adjusted time-series location information and time-series attitude information corresponding to this kind of time-series location information and time-series attitude information to obtain the second adjusted time-series location. information and time series attitude information, so as to obtain the second adjusted time series position information and time series attitude information.
- the second adjusted time-series position information and time-series attitude information are spliced with the environment information to obtain the fusion feature.
- the second adjustment time series position information and the time series attitude information in the second adjustment time series position information and the time series attitude information are spliced with the plurality of coded maps according to a preset method to obtain Fusion features.
- the three kinds of time-series position information and time-series posture information are sequentially input into the neural network (for example, LSTM network) in the order of the pedestrian's body orientation, face orientation and the position of the object.
- the obtained second adjustment time series position information and time series attitude information also include these three characteristics, according to the order from the pedestrian's body orientation, face orientation, the position of the pedestrian to the local map, the second adjustment time series position
- the information and time series pose information are spliced with the corresponding local maps to obtain fusion features.
- a fully connected network is used to decode the fusion features to predict the pedestrian's motion intention, that is, whether the pedestrian wants to turn left, turn right, go straight, stand still, or turn around.
- the embodiments of the present disclosure provide a trajectory prediction method.
- a vehicle, a pedestrian, or a non-motor vehicle may have complex behaviors, such as sudden turning, sudden left or right turn, or walking.
- complex behavior cannot be easily predicted or expected from the historical trajectories of vehicles, pedestrians or non-motor vehicles alone.
- autonomous systems with sensing capabilities can naturally extract richer information to make more informed decisions.
- the embodiments of the present disclosure use the orientation of the object to describe the motion of the object and the local map area to describe the surrounding static environment.
- the position is represented as a point (x, y) in the horizontal plane, while the volume and face orientations are extracted from the corresponding Red Green Blue (RGB) image and then projected onto the horizontal plane, represented as a unit vector (d x , dy ).
- the local map area is obtained from the high-definition map and contains multiple road information, such as crosswalks, lane lines, intersections or sidewalks.
- Embodiments of the present disclosure use a data collection vehicle to collect object trajectory data in an urban driving scene.
- the car is equipped with cameras, 64-line LiDAR, radar, Global Positioning System (GPS) or Inertial Measurement Unit (IMU).
- GPS Global Positioning System
- IMU Inertial Measurement Unit
- the embodiment of the present disclosure utilizes the marked high-definition map to detect, analyze and track the future trajectory of the generated object through the perception function.
- the embodiments of the present disclosure provide future trajectories of pedestrians and raw data at 10 hertz (HZ), where the raw data includes raw images, point cloud points, vehicle poses of the ego vehicle and a high-definition map.
- HZ hertz
- the embodiments of the present disclosure use a first neural network and a second neural network (wherein the first neural network and the second neural network can be implemented by using a model of a deep neural network algorithm) to get the output.
- the preset data set provided by the embodiment of the present disclosure includes: the face orientation, body orientation and position of the pedestrian, vehicle light information, vehicle head orientation information, and the like of the pedestrian. In this way, using the dataset containing such rich information to train the first neural network and the second neural network makes the trained first neural network and the second neural network more generalizable.
- Embodiments of the present disclosure collect raw sensor data at a frequency of 10 Hz, including front-view RGB images (800 ⁇ 1762), LiDAR point clouds, and ego vehicle pose and motion information.
- the embodiments of the present disclosure provide semantic annotation of road categories (ie, lane lines, intersections, crosswalks, sidewalks, etc.) for bird's-eye view High Definition Maps (HDMap).
- Road categories are represented as polygons or lines with no overlapping areas.
- the HDMap is cropped and aligned with the ego car for each data frame. With the help of perception function, the running trajectories of objects can be generated through detection and tracking.
- the trajectories are sampled to 0.3 seconds per frame.
- Embodiments of the present disclosure collect over 12,000 minutes of raw data and sample over 300,000 different trajectories for vehicles, pedestrians, and cyclists.
- embodiments of the present disclosure manually annotate the objects in the collected trajectories with semantic attributes and intents.
- Embodiments of the present disclosure use different property settings for each object class to better capture its functionality.
- the embodiments of the present disclosure will indicate the age group (adult/juvenile), gender (female/male) ), face orientation (angle) and body orientation; for the vehicle, the embodiment of the present disclosure marks the turn signal status (turn left/turn right/brake) and the heading direction.
- Intent can be understood as the future action of the object after a specific time (1s in the setting of the embodiment of the present disclosure) after the observation point. Similar to this attribute, embodiments of the present disclosure define different intent spaces for vehicles, pedestrians, and cyclists, as shown in FIGS. 4A to 4D , wherein: FIG. 4A represents different objects, namely vehicle 401 , pedestrian 402 and cyclist Among the cyclists 403, the number of vehicles 401 is 334,696, accounting for 58%, the number of pedestrians 402 is 178,343, accounting for 31%, and the number of cyclists 403 is 61,934, accounting for 11%.
- FIG. 4B shows the result of the intention prediction for the vehicle, in which the straight 421 occupies 38.9% (that is, the intention of the vehicle to make a straight is 38.9%), the left turn 422 occupies 2%, the right turn 423 occupies 1%, and the left lane change 424 1.6%, right lane change 425 2%, left overtaking 426 0.1%, right overtaking 427 0.1%, stationary 428 54%, and other 429 0.2%.
- Figure 4C shows the result of the intention prediction for pedestrians, in which 431 go straight 48.6%, 432 turn left 16.8%, 433 turn right 23.6%, stand still 434 occupied 6.8%, turn 435 occupied 0.4%, and other 436 occupied 3.7% %.
- Figure 4D shows the results of intent prediction for cyclists, where straight 441 occupies 37.5%, left turn 442 occupies 13.5%, right turn 443 occupies 17.9%, stationary 444 occupies 24%, U-turn occupies 0.1%, and other 445 occupy 7%.
- the datasets of the embodiments of the present disclosure cover more object categories and provide rich contextual annotations, including road information and attribute annotations.
- the datasets of embodiments of the present disclosure use broader intent definitions and have larger data scales.
- a unified framework is adopted to jointly predict the future trajectory and potential intention of objects.
- At least one of the first neural network and the second neural network used in the embodiments of the present disclosure may include, but is not limited to, an LSTM-based encoder-decoder architecture, and based on the first neural network and the second neural network At least one of them can improve the immediacy and generality of the framework.
- an encoder is used to extract object features from historical motion trajectories and rich contextual information of objects, including semantic object attributes and local road structures.
- the decoder is used to estimate the intent distribution and regress to future positions.
- FIG. 5 is a schematic diagram of the framework of a trajectory prediction system provided by an embodiment of the present disclosure, and the following description is made with reference to FIG. 5 :
- a time series model is established for each time series position information and time series attitude information, that is, each time series position information and time series attitude information are input into the first neural network (here the first neural network can be implemented by LSTM network 506), and the corresponding timing characteristics.
- the location information 502 is input into the LSTM network 506 to obtain the location time sequence feature
- the body orientation 503 is input into the LSTM network 506 to obtain the body orientation time sequence feature
- the face orientation 504 is input into the LSTM network 506 to obtain the face orientation time sequence feature
- the road structure 505 is input into the second neural network (here, the second neural network can be implemented by using the CNN network 507 ) to encode the road structure to obtain road time-series position information and time-series attitude information.
- the road time sequence position information, time sequence attitude information and time sequence features are fused to obtain fusion features, and the fusion features are input into the first neural network (here, the first neural network can be implemented by the MLP network 508 ), and the intention is predicted to obtain The result of the intent prediction is crossing the road 509.
- the result of the intention prediction crossed the road 509 and the fusion feature are combined and input into the LSTM network 506, and multiple iterations are performed to predict the running trajectory of the pedestrian, and the predicted future trajectory 510 is obtained; in FIG. 5, by comparing the pedestrian 501 It can be seen that the accuracy of the predicted future trajectory 510 obtained by using the trajectory prediction method provided by the embodiment of the present disclosure is very high.
- a set of LSTM or CNN networks are used to encode the object’s motion history and multimodal contextual input, depending on the specific form of each data item. After the encoded features are concatenated into fused features, they are fed into the decoder to jointly predict future trajectories and latent intents.
- the observation result of the ith object is expressed as in, is the location information, is contextual information.
- T is the last observation time (for example, the value of T can be greater than 0 and less than 5 minutes)
- n, m are the observation period and the prediction period respectively (for example, the value of n, m can be greater than 0 and less than 5 real number of minutes).
- Embodiments of the present disclosure use a set of bidirectional LSTM networks as the first neural network to encode multi-source input data.
- the historical trajectory of the object pT-m:T is directly fed into the LSTM to obtain the hidden state at time T (denoted as ) as a motion history feature.
- Contextual information is processed according to its specific form.
- semantic attributes such as face orientation and headlight status are closely related to object intent and future motion, and reflect intrinsic properties of objects that cannot be obtained from motion history.
- Local maps provide road structures to standardize trajectory predictions.
- the sequence of directions ie face, body and vehicle heading
- the sequence of lamp states are respectively directly input into independent bidirectional LSTMs.
- the embodiment of the present disclosure uses the local map once within the observation time T to reduce redundancy.
- the original map is first rasterized, and then the rasterized map is input into the CNN model to extract time-series position information and time-series attitude information of the map.
- all encoded vectors are concatenated as fused features embedded at time T, as in Equation (1):
- ⁇ represents the transform function of the entire encoder.
- Embodiments of the present disclosure model intent prediction as a classification problem. Among them, the model predicts the posterior probability distribution over a finite set of intents based on the fused features e T of a given object.
- the embodiment of the present disclosure uses a Multilayer perceptron (MLP), and connects the softmax layer as an intent classifier.
- MLP Multilayer perceptron
- the embodiment of the present disclosure minimizes the loss of cross-entropy, as shown in formula (2):
- Embodiments of the present disclosure treat trajectory prediction as a sequence generation task and employ an LSTM decoder to predict object motion at each future time step.
- the features embedded in eT are fed into the decoder at the beginning.
- embodiments of the present disclosure determine intent embedding features by passing the output of the intent classifier through another fully connected layer And the intent embedding feature is used as the auxiliary input of the trajectory decoder, which provides a good condition for trajectory prediction.
- the embodiments of the present disclosure minimize the Gaussian-like loss function during the training process:
- the following description takes the object as a pedestrian as an example:
- Table 1 shows the accuracy of body orientation and face orientation acquired at different acquisition distances. It can be seen from Table 1 that the pedestrian's position, body orientation and face orientation are used to represent the dynamic situation of the pedestrian, while the local map area is used to represent the static surrounding environment.
- the position, body orientation, and face orientation that is, the time-series position information and time-series posture information of pedestrians may be regarded as dynamic features, and the local map area may be regarded as static features.
- the accuracy of the face direction (Face direction) and the body direction (Body direction) is related to the distance from the pedestrian to the ego vehicle.
- Embodiments of the present disclosure use the embedding function ⁇ to express this relationship:
- Wd represents the input-to-output transformation parameter in the second neural network
- Pedestrians follow basic traffic rules, which are related to their corresponding local road structures.
- the local map area is the basic static environment for the future trajectory prediction of pedestrians.
- FIG. 6 is a structural diagram of an implementation framework of a trajectory prediction method according to an embodiment of the present disclosure. As shown in FIG. 6 , first, the time-series position information and time-series attitude information of the pedestrian 61 are extracted from the images 601 to 60n, for example, the facial orientation body orientation and the location of pedestrian 61 and local map areas based on body orientation and location
- first neural networks 62, 63 and 64 for example, bidirectional LSTM network
- time-series features used to indicate body orientation ie time-series position information and time-series pose information
- face-orientation time-series features and the time-series features of the temporal changes of the position of the sample object and then input the time-series features into another second neural network 65 (for example, a bidirectional LSTM network) to obtain the first adjusted time-series features.
- second neural network 65 for example, a bidirectional LSTM network
- the coding map 602 is expanded into a one-dimensional feature vector, the one-dimensional feature vector is encoded, and another bidirectional LSTM network, namely the first neural network 66, is input to obtain the time series feature corresponding to the feature vector; then, the The time sequence feature is used as an auxiliary feature of the time sequence feature corresponding to the time sequence position information and time sequence attitude information of the pedestrian 61, and these features are spliced to obtain a fusion feature; then the fusion feature is decoded through the decoded neural network 67 to obtain the predicted pedestrian The future trajectory, that is, the dotted line 69 ; the solid line 70 is the true future trajectory of the pedestrian 61 . It can be seen that the prediction result of the network model adopted in the embodiment of the present disclosure is very accurate.
- Embodiments of the present disclosure employ mask encoding for local map regions, resulting in an encoded map 602, where each codeword is populated with a specific integer associated with its semantic road structure class.
- each codeword is populated with a specific integer associated with its semantic road structure class.
- the local map regions are then uniformly discretized into grids, where each grid is represented by a structure-specific number of major semantic road structure classes. For example, “crosswalk” and “sidewalk” are represented as number “1”, “dangerous place” is represented as “-1”, and others are represented as number "0", that is, a grid 603 for dividing dangerous or safe areas is obtained.
- the coded dynamic features ie, the temporal position information and temporal pose information of pedestrians
- the coded static features ie, local map regions
- the preset data set of the historical data provided by the embodiments of the present disclosure is a large-scale and information-based trajectory data set, so as to facilitate the task of pedestrian trajectory prediction in automatic driving.
- there are multiple evaluation criteria in the data set such as the average number of failed predictions of future trajectories whose length is greater than the preset threshold, the success rate of future trajectories under error thresholds corresponding to different distances, or the end position of the future trajectory and the end point of the true value trajectory.
- the error between the positions is used to evaluate the accuracy and robustness of the prediction model; thus, even in very complex scenarios, the neural network can still accurately predict the future trajectories of pedestrians.
- FIG. 7 is a schematic structural diagram of the trajectory prediction device according to an embodiment of the present disclosure.
- the device 700 includes:
- the intention determination module 701 is configured to determine the motion intention of the object according to the time series position information and the time series attitude information of the object; wherein, the time series position information is the position information of the object at different time points within a preset time period, so The time sequence attitude information is the attitude information of the object at different time points within a preset duration; the attitude information at the different time points includes the orientation information of the object at the different time points;
- the future trajectory determination module 702 is configured to determine the future trajectory of the object according to the time-series position information, the time-series posture information, and the motion intention.
- the intent determination module 701 includes: a map interception sub-module, configured to obtain environmental information of the environment where the object is located according to the time-series position information and the time-series attitude information; a feature fusion sub-module, configured as The environmental information, the time-series position information and the time-series attitude information are fused to obtain a fusion feature; an intention prediction sub-module is configured to determine the motion intention of the object according to the fusion feature; the future trajectory determination module 702 , comprising: a trajectory prediction sub-module, configured to determine the future trajectory of the object according to the fusion feature and the motion intention.
- the object includes at least one of a human object and a non-human object
- the posture information at different time points includes: the position of the human object
- the orientation information at the different time points, the part includes at least one of the following: limbs, face; in the case that the object includes the non-human body object, the non-human body object includes at least one of the following: a vehicle , animal, and movable device;
- the posture information at different time points includes: orientation information and driving instruction information of the non-human object at the different time points.
- the device further includes: a historical moment determination module configured to determine at least two historical moments whose duration from the current moment is less than or equal to a specific duration; a feature information acquisition module configured to acquire the object in at least two Time-series position information and time-series attitude information at historical moments.
- the map interception sub-module includes: a map interception unit configured to determine the environment information according to the position information and orientation information of the object at any historical moment; wherein the environment information includes at least the following At least one of: road information, pedestrian information, or traffic light information.
- the map intercepting unit is further configured to: center on the position information and according to the orientation information, delimit a local map area of the environment where the object is located in the world map; The elements in the area are encoded to obtain the environmental information.
- the feature fusion sub-module includes: a unit for determining time-series position information and time-series attitude information, configured to predict the time-series in a future period according to the time-series location information and time-series attitude information through a first neural network location information and time sequence attitude information; a feature splicing unit, configured to splicing the time sequence location information, time sequence attitude information and the environment information in the future period in a preset manner to obtain the fusion feature.
- the intent prediction sub-module includes: a confidence level determination unit, configured to determine, through the second neural network, the confidence level that the fusion feature is each intent category in the intent category library; the intent prediction unit, configured as The intent category with the highest confidence is used to determine the motion intent of the object.
- the trajectory prediction sub-module includes: an iterative step unit, configured to determine an iterative step according to the length of the future period; and a feature iteration unit, configured to use the iterative step according to the The first neural network iterates the motion intention and the fusion feature to obtain the coordinates of the object under each iteration step; the future trajectory determination unit is configured to coordinates to determine the future trajectory.
- the device further includes a first training module configured to train the first neural network
- the first training module includes: a prediction sub-module for predicting time-series position information and time-series attitude information, configured to input the time-series location information and time-series attitude information of the object into the first neural network to be trained, and predict that the object is in the Time-series position information and time-series attitude information in a future period; a prediction feature fusion sub-module, configured to fuse the time-series location information and time-series attitude information in the future period with the environmental information of the environment where the object is located to obtain a fusion prediction feature a predicting future trajectory submodule, configured to predict the future trajectory of the object in the future period at least according to the fusion prediction feature; a first prediction loss determination submodule, configured to be based on the true value trajectory of the object, Determine the first prediction loss of the first neural network to be trained with respect to the future trajectory; the first neural network parameter adjustment sub-module is configured to The parameters are adjusted to obtain the first neural network.
- a prediction sub-module for predicting time-serie
- the device further includes a second training module configured to train a second neural network
- the second training module includes: a category confidence determination sub-module, configured to input the fusion feature into the second neural network to be trained, and predict the motion intent of the object as the confidence level of each intent category in the intent category library; 2.
- a prediction loss determination sub-module configured to determine a second prediction loss of the confidence of the second neural network to be trained with respect to each intention category according to the true intention of the object; a second neural network parameter adjustment sub-module The module is configured to adjust the network parameters of the second neural network to be trained according to the second prediction loss to obtain the second neural network.
- an embodiment of the present disclosure further provides a computer program product, wherein the computer program product includes computer-executable instructions, and after the computer-executable instructions are executed, the trajectory prediction method provided by the embodiment of the present disclosure can be implemented.
- an embodiment of the present disclosure further provides a computer storage medium, where computer-executable instructions are stored thereon, and when the computer-executable instructions are executed by a processor, the trajectory prediction method provided by the foregoing embodiments is implemented.
- Embodiments of the present disclosure further provide a computer program, where the computer program includes computer-readable codes.
- the computer-readable codes are executed in an electronic device, the processor of the electronic device executes the above-mentioned implementation.
- the trajectory prediction method provided by the example.
- FIG. 8 is a schematic structural diagram of a computer device according to an embodiment of the present disclosure.
- the device 800 includes: a processor 801 , at least one communication bus, a communication Interface 802 , at least one external communication interface and memory 803 .
- the communication interface 802 is configured to realize the connection communication between these components.
- the communication interface 802 may include a display screen, and the external communication interface may include a standard wired interface and a wireless interface.
- the processor 801 is configured to execute an image processing program in the memory, so as to implement the trajectory prediction method provided by the above embodiment.
- the above-mentioned memory may be a volatile memory (volatile memory), such as a random access memory (Random Access Memory, RAM); or a non-volatile memory (non-volatile memory), such as a read-only memory (Read Only Memory) -Only Memory, ROM), flash memory (flash memory), hard disk (Hard Disk Drive, HDD) or solid-state drive (Solid-State Drive, SSD); or a combination of the above types of memory, and provide instructions and data.
- volatile memory such as a random access memory (Random Access Memory, RAM)
- non-volatile memory such as a read-only memory (Read Only Memory) -Only Memory, ROM), flash memory (flash memory), hard disk (Hard Disk Drive, HDD) or solid-state drive (Solid-State Drive, SSD); or a combination of the above types of memory, and provide instructions and data.
- the above processor can be an application specific integrated circuit (ASIC), a digital signal processor (Digital Signal Processor, DSP), a digital signal processing device (Digital Signal Processor Device, DSPD), a programmable logic device (Programmable Logic Device) , PLD), at least one of field programmable gate array (Field Programmable Gate Array, FPGA), central processing unit (Central Processing Unit, CPU), controller, microcontroller, microprocessor.
- ASIC application specific integrated circuit
- DSP digital signal processor
- DSPD digital signal processing device
- DSPD Digital Signal Processor Device
- PLD programmable logic device
- FPGA Field Programmable Gate Array
- CPU Central Processing Unit
- controller microcontroller
- the disclosed apparatus and method may be implemented in other manners.
- the device embodiments described above are only illustrative.
- the division of the units is only a logical function division. In actual implementation, there may be other division methods.
- multiple units or components may be combined, or Can be integrated into another system, or some features can be ignored, or not implemented.
- the coupling, or direct coupling, or communication connection between the components shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be electrical, mechanical or other forms. of.
- the unit described above as a separate component may or may not be physically separated, and the component displayed as a unit may or may not be a physical unit; it may be located in one place or distributed to multiple network units; Some or all of the units may be selected according to actual needs to achieve the purpose of the solution in this embodiment.
- each functional unit in each embodiment of the present disclosure may be all integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above integration
- the unit can be implemented either in the form of hardware or in the form of hardware plus software functional units.
- the above-mentioned integrated units of the present disclosure are implemented in the form of software functional modules and sold or used as independent products, they may also be stored in a computer-readable storage medium.
- the technical solutions of the embodiments of the present disclosure essentially or the parts that make contributions to the prior art can be embodied in the form of a software product, and the computer software product is stored in a storage medium and includes several instructions for A computer device (which may be a personal computer, a server, or a network device, etc.) is caused to execute all or part of the methods described in the various embodiments of the present disclosure.
- the aforementioned storage medium includes various media that can store program codes, such as a removable storage device, a ROM, a magnetic disk, or an optical disk.
- Embodiments of the present disclosure provide a trajectory prediction method, device, device, storage medium, and program, wherein the motion intention of the object is determined according to the time-series position information and time-series attitude information of the object; wherein the time-series location information is all The position information of the object at different time points within the preset time period, and the time sequence attitude information is the attitude information of the object at different time points within the preset time period; wherein, the attitude information of the different time points includes the position information of the object.
- the orientation information of the multiple parts at the different time points; the future trajectory of the object is determined according to the time-series position information, the time-series posture information, and the motion intention.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Computing Systems (AREA)
- General Physics & Mathematics (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Life Sciences & Earth Sciences (AREA)
- General Engineering & Computer Science (AREA)
- Artificial Intelligence (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Automation & Control Theory (AREA)
- Human Computer Interaction (AREA)
- Transportation (AREA)
- Mechanical Engineering (AREA)
- Traffic Control Systems (AREA)
- Image Analysis (AREA)
Abstract
Description
Claims (15)
- 一种轨迹预测方法,所述方法由电子设备执行,所述方法包括:根据对象的时序位置信息和时序姿态信息,确定所述对象的运动意图;其中,所述时序位置信息为所述对象在预设时长内不同时间点的位置信息,所述时序姿态信息为所述对象在预设时长内不同时间点的姿态信息;所述不同时间点的姿态信息包括所述对象在所述不同时间点的朝向信息;根据所述时序位置信息、所述时序姿态信息以及所述运动意图,确定所述对象的未来轨迹。
- 根据权利要求1所述的方法,其中,所述根据对象的时序位置信息和时序姿态信息,确定所述对象的运动意图,包括:根据所述时序位置信息和所述时序姿态信息,获取所述对象所处环境的环境信息;将所述环境信息、所述时序位置信息和时序姿态信息进行融合,得到融合特征;根据所述融合特征,确定所述对象的运动意图;所述根据所述时序位置信息、所述时序姿态信息以及所述运动意图,确定所述对象的未来轨迹,包括:根据所述融合特征和所述运动意图,确定所述对象的未来轨迹。
- 根据权利要求1或2所述的方法,其中,所述对象包括人体对象和非人体对象中的至少之一;在所述对象包括所述人体对象的情况下,所述不同时间点的姿态信息包括:所述人体对象的部位的在所述不同时间点的朝向信息,所述部位包括以下至少之一:肢体、面部;在所述对象包括所述非人体对象的情况下,所述非人体对象包括以下至少之一:车辆、可移动设备;所述不同时间点的姿态信息包括:所述非人体对象在所述不同时间点的朝向信息和行驶指示信息。
- 根据权利要求1或2所述的方法,其中,所述根据对象的时序位置信息和时序姿态信息,确定所述对象的运动意图之前,所述方法还包括:确定距离当前时刻的时长小于等于特定时长的至少两个历史时刻;获取所述对象在所述至少两个历史时刻的时序位置信息和时序姿态信息。
- 根据权利要求2至4任一所述的方法,其中,所述根据所述时序位置信息和所述时序姿态信息,获取所述对象所处环境的环境信息,包括:根据所述对象在任一历史时刻的位置信息和朝向信息,确定所述环境信息;其中,所述环境信息至少包括下列中的至少一个:道路信息、行人信息或交通灯信息。
- 根据权利要求5所述的方法,其中,所述根据所述对象在任一历史时刻的位置信息和朝向信息,确定所述环境信息,包括:以所述位置信息为中心,按照所述朝向信息,在世界地图中划定所述对象所在环境的局部地图区域;对所述局部地图区域中的元素进行编码,得到所述环境信息。
- 根据权利要求2、5和6任一所述的方法,其中,所述将所述环境信息、所述时序位置信息和时序姿态信息进行融合,得到融合特征,包括:通过第一神经网络,根据所述时序位置信息和时序姿态信息,预测在未来时段内的时序位置信息和时序姿态信息;将所述未来时段内的时序位置信息、时序姿态信息和所述环境信息,按照预设方式进行拼接,得到所述融合特征。
- 根据权利要求2、5至7任一所述的方法,其中,所述根据所述融合特征,确定所述对象的运动意图,包括:通过第二神经网络确定所述融合特征为意图类别库中每一意图类别的置信度;将置信度最大的意图类别,确定所述对象的运动意图。
- 根据权利要求2、5至8任一所述的方法,其中,所述根据所述融合特征和所述运动意图,确定所述对象的未来轨迹,包括:根据所述未来时段的长度,确定迭代步长;按照所述迭代步长,采用所述第一神经网络对所述运动意图和所述融合特征进行迭代,得到所述对象在每一迭代步长下的坐标;根据所述对象在每一迭代步长下的坐标,确定所述未来轨迹。
- 根据权利要求7至9任一所述的方法,其中,所述第一神经网络的训练方法,包括:将所述对象的时序位置信息和时序姿态信息输入待训练第一神经网络中,预测所述对象在所述未来时段内的时序位置信息和时序姿态信息;将所述未来时段内的时序位置信息、时序姿态信息与所述环境信息进行融合,得到融合预测特征;至少根据所述融合预测特征,预测所述对象在所述未来时段内的未来轨迹;根据所述对象的真值轨迹,确定所述待训练第一神经网络关于所述未来轨迹的第一预测损失;根据所述第一预测损失,对所述待训练第一神经网络的网络参数进行调整,得到所述第一神经网络。
- 根据权利要求8至10任一所述的方法,其中,所述第二神经网络的训练方法,包括:将所述融合特征输入待训练第二神经网络,预测所述对象的运动意图为意图类别库中每一意图类别的置信度;根据所述对象的真值意图,确定所述待训练第二神经网络关于所述每一意图类别的置信度的第二预测损失;根据所述第二预测损失,对所述待训练第二神经网络的网络参数进行调整,得到所述第二神经网络。
- 一种轨迹预测装置,所述装置包括:意图确定模块,配置为根据对象的时序位置信息和时序姿态信息,确定所述对象的运动意图;其中,所述时序位置信息为所述对象在预设时长内不同时间点的位置信息,所述时序姿态信息为所述对象在预设时长内不同时间点的姿态信息;所述不同时间点的姿态信息包括所述对象在所述不同时间点的朝向信息;未来轨迹确定模块,配置为根据所述时序位置信息、所述时序姿态信息以及所述运动意图,确定所述对象的未来轨迹。
- 一种计算机存储介质,所述计算机存储介质上存储有计算机可执行指令,该计算机可执行指令被执行后,能够实现权利要求1至11任一项所述的轨迹预测方法。
- 一种计算机设备,所述计算机设备包括存储器和处理器,所述存储器上存储有计算机可执行指令,所述处理器运行所述存储器上的计算机可执行指令时可实现权利要求1至11任一项所述的轨迹预测方法。
- 一种计算机程序,所述计算机程序包括计算机可读代码,在所述计算机可读代码在电子设备中运行的情况下,所述电子设备的处理器执行用于实现如权利要求1至11任 一所述的轨迹预测方法。
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2022546580A JP7513726B2 (ja) | 2020-07-31 | 2021-07-30 | 軌道予測方法、装置、機器、記憶媒体およびプログラム |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202010763409.4A CN111942407B (zh) | 2020-07-31 | 2020-07-31 | 轨迹预测方法、装置、设备及存储介质 |
CN202010763409.4 | 2020-07-31 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2022022721A1 true WO2022022721A1 (zh) | 2022-02-03 |
Family
ID=73337954
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/CN2021/109871 WO2022022721A1 (zh) | 2020-07-31 | 2021-07-30 | 轨迹预测方法、装置、设备、存储介质及程序 |
Country Status (3)
Country | Link |
---|---|
JP (1) | JP7513726B2 (zh) |
CN (1) | CN111942407B (zh) |
WO (1) | WO2022022721A1 (zh) |
Cited By (12)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN114463687A (zh) * | 2022-04-12 | 2022-05-10 | 北京云恒科技研究院有限公司 | 一种基于大数据的移动轨迹预测方法 |
CN114516336A (zh) * | 2022-02-24 | 2022-05-20 | 重庆长安汽车股份有限公司 | 一种考虑道路约束条件的车辆轨迹预测方法 |
CN114550297A (zh) * | 2022-02-25 | 2022-05-27 | 北京拙河科技有限公司 | 一种行人的意图分析方法及*** |
CN114997297A (zh) * | 2022-05-26 | 2022-09-02 | 哈尔滨工业大学 | 一种基于多级区域划分的目标运动意图推理方法及*** |
CN115562332A (zh) * | 2022-09-01 | 2023-01-03 | 北京普利永华科技发展有限公司 | 一种无人机机载记录数据的高效处理方法及*** |
CN115564803A (zh) * | 2022-12-06 | 2023-01-03 | 腾讯科技(深圳)有限公司 | 一种动画处理方法、装置、设备、存储介质及产品 |
CN115690160A (zh) * | 2022-11-16 | 2023-02-03 | 南京航空航天大学 | 一种低帧率视频行人轨迹预测方法与*** |
CN116602663A (zh) * | 2023-06-02 | 2023-08-18 | 深圳市震有智联科技有限公司 | 一种基于毫米波雷达的智能监测方法及*** |
CN116723616A (zh) * | 2023-08-08 | 2023-09-08 | 杭州依森匠能数字科技有限公司 | 一种灯光亮度控制方法及*** |
CN116778101A (zh) * | 2023-06-26 | 2023-09-19 | 北京道仪数慧科技有限公司 | 基于营运载具的地图生成方法及*** |
CN117560638A (zh) * | 2024-01-10 | 2024-02-13 | 山东派蒙机电技术有限公司 | 应用于移动端通信***的融合通信方法、装置及设备 |
CN117799641A (zh) * | 2024-02-08 | 2024-04-02 | 北京科技大学 | 多车干扰下智能物流车高效节能驾驶优化控制方法及装置 |
Families Citing this family (14)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN111942407B (zh) * | 2020-07-31 | 2022-09-23 | 商汤集团有限公司 | 轨迹预测方法、装置、设备及存储介质 |
CN112364997B (zh) * | 2020-12-08 | 2021-06-04 | 北京三快在线科技有限公司 | 一种障碍物的轨迹预测方法及装置 |
CN113033364B (zh) * | 2021-03-15 | 2024-06-14 | 商汤集团有限公司 | 轨迹预测方法、行驶控制方法、装置、电子设备及存储介质 |
CN113029154B (zh) * | 2021-04-01 | 2022-07-12 | 北京深睿博联科技有限责任公司 | 一种盲人导航方法及装置 |
CN113316788A (zh) * | 2021-04-20 | 2021-08-27 | 深圳市锐明技术股份有限公司 | 行人运动轨迹的预测方法、装置、电子设备和存储介质 |
CN112859883B (zh) * | 2021-04-25 | 2021-09-07 | 北京三快在线科技有限公司 | 一种无人驾驶设备的控制方法及控制装置 |
CN113157846A (zh) * | 2021-04-27 | 2021-07-23 | 商汤集团有限公司 | 意图及轨迹预测方法、装置、计算设备和存储介质 |
CN113382304B (zh) * | 2021-06-07 | 2023-07-18 | 北博(厦门)智能科技有限公司 | 一种基于人工智能技术的视频拼接方法 |
CN113658214B (zh) * | 2021-08-16 | 2022-08-09 | 北京百度网讯科技有限公司 | 轨迹预测方法、碰撞检测方法、装置、电子设备及介质 |
CN114312831B (zh) * | 2021-12-16 | 2023-10-03 | 浙江零跑科技股份有限公司 | 一种基于空间注意力机制的车辆轨迹预测方法 |
CN114663982B (zh) * | 2022-04-21 | 2024-06-25 | 湖南大学 | 一种基于多特征融合的人手轨迹预测与意图识别方法 |
CN114872735B (zh) * | 2022-07-10 | 2022-10-04 | 成都工业职业技术学院 | 基于神经网络算法的自动驾驶物流车辆决策方法和装置 |
CN115628736A (zh) * | 2022-09-23 | 2023-01-20 | 北京智行者科技股份有限公司 | 行人轨迹的预测方法、设备、移动装置和存储介质 |
CN115790606B (zh) * | 2023-01-09 | 2023-06-27 | 深圳鹏行智能研究有限公司 | 轨迹预测方法、装置、机器人及存储介质 |
Citations (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
EP1223567A1 (en) * | 2001-01-11 | 2002-07-17 | Siemens Aktiengesellschaft | A method for inter-vehicle communication of individualized vehicle data |
CN1847817A (zh) * | 2005-04-15 | 2006-10-18 | ***通信集团公司 | 一种使用移动通信方式为汽车提供服务的***和方法 |
US20180126951A1 (en) * | 2016-11-07 | 2018-05-10 | Nio Usa, Inc. | Method and system for authentication in autonomous vehicles |
CN109801508A (zh) * | 2019-02-26 | 2019-05-24 | 百度在线网络技术(北京)有限公司 | 路口处障碍物的运动轨迹预测方法及装置 |
CN110210417A (zh) * | 2019-06-05 | 2019-09-06 | 深圳前海达闼云端智能科技有限公司 | 一种行人运动轨迹的预测方法、终端及可读存储介质 |
CN111095380A (zh) * | 2017-09-20 | 2020-05-01 | 本田技研工业株式会社 | 车辆控制装置、车辆控制方法、及程序 |
CN111942407A (zh) * | 2020-07-31 | 2020-11-17 | 商汤集团有限公司 | 轨迹预测方法、装置、设备及存储介质 |
Family Cites Families (14)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10345815B2 (en) | 2016-09-14 | 2019-07-09 | Qualcomm Incorporated | Motion planning and intention prediction for autonomous driving in highway scenarios via graphical model-based factorization |
DE102016217770A1 (de) * | 2016-09-16 | 2018-03-22 | Audi Ag | Verfahren zum Betrieb eines Kraftfahrzeugs |
KR101946940B1 (ko) * | 2016-11-09 | 2019-02-12 | 엘지전자 주식회사 | 차량에 구비된 차량 제어 장치 및 차량의 제어방법 |
CN107423679A (zh) * | 2017-05-31 | 2017-12-01 | 深圳市鸿逸达科技有限公司 | 一种行人意图检测方法和*** |
JP6833630B2 (ja) * | 2017-06-22 | 2021-02-24 | 株式会社東芝 | 物体検出装置、物体検出方法およびプログラム |
US11367354B2 (en) * | 2017-06-22 | 2022-06-21 | Apollo Intelligent Driving Technology (Beijing) Co., Ltd. | Traffic prediction based on map images for autonomous driving |
US11104334B2 (en) * | 2018-05-31 | 2021-08-31 | Tusimple, Inc. | System and method for proximate vehicle intention prediction for autonomous vehicles |
JP7125286B2 (ja) * | 2018-06-22 | 2022-08-24 | 本田技研工業株式会社 | 行動予測装置及び自動運転装置 |
US11302197B2 (en) | 2018-09-17 | 2022-04-12 | Nissan Motor Co., Ltd. | Vehicle behavior prediction method and vehicle behavior prediction device |
CN110059598B (zh) * | 2019-04-08 | 2021-07-09 | 南京邮电大学 | 基于姿态关节点的长时程快慢网络融合的行为识别方法 |
CN110147743B (zh) * | 2019-05-08 | 2021-08-06 | 中国石油大学(华东) | 一种复杂场景下的实时在线行人分析与计数***及方法 |
CN110796856B (zh) * | 2019-10-16 | 2022-03-25 | 腾讯科技(深圳)有限公司 | 车辆变道意图预测方法及变道意图预测网络的训练方法 |
CN111401233A (zh) * | 2020-03-13 | 2020-07-10 | 商汤集团有限公司 | 轨迹预测方法、装置、电子设备及介质 |
CN111402632B (zh) * | 2020-03-18 | 2022-06-07 | 五邑大学 | 一种交叉口行人运动轨迹的风险预测方法 |
-
2020
- 2020-07-31 CN CN202010763409.4A patent/CN111942407B/zh active Active
-
2021
- 2021-07-30 JP JP2022546580A patent/JP7513726B2/ja active Active
- 2021-07-30 WO PCT/CN2021/109871 patent/WO2022022721A1/zh active Application Filing
Patent Citations (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
EP1223567A1 (en) * | 2001-01-11 | 2002-07-17 | Siemens Aktiengesellschaft | A method for inter-vehicle communication of individualized vehicle data |
CN1847817A (zh) * | 2005-04-15 | 2006-10-18 | ***通信集团公司 | 一种使用移动通信方式为汽车提供服务的***和方法 |
US20180126951A1 (en) * | 2016-11-07 | 2018-05-10 | Nio Usa, Inc. | Method and system for authentication in autonomous vehicles |
CN111095380A (zh) * | 2017-09-20 | 2020-05-01 | 本田技研工业株式会社 | 车辆控制装置、车辆控制方法、及程序 |
CN109801508A (zh) * | 2019-02-26 | 2019-05-24 | 百度在线网络技术(北京)有限公司 | 路口处障碍物的运动轨迹预测方法及装置 |
CN110210417A (zh) * | 2019-06-05 | 2019-09-06 | 深圳前海达闼云端智能科技有限公司 | 一种行人运动轨迹的预测方法、终端及可读存储介质 |
CN111942407A (zh) * | 2020-07-31 | 2020-11-17 | 商汤集团有限公司 | 轨迹预测方法、装置、设备及存储介质 |
Cited By (21)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN114516336B (zh) * | 2022-02-24 | 2023-09-26 | 重庆长安汽车股份有限公司 | 一种考虑道路约束条件的车辆轨迹预测方法 |
CN114516336A (zh) * | 2022-02-24 | 2022-05-20 | 重庆长安汽车股份有限公司 | 一种考虑道路约束条件的车辆轨迹预测方法 |
CN114550297A (zh) * | 2022-02-25 | 2022-05-27 | 北京拙河科技有限公司 | 一种行人的意图分析方法及*** |
CN114550297B (zh) * | 2022-02-25 | 2022-09-27 | 北京拙河科技有限公司 | 一种行人的意图分析方法及*** |
CN114463687A (zh) * | 2022-04-12 | 2022-05-10 | 北京云恒科技研究院有限公司 | 一种基于大数据的移动轨迹预测方法 |
CN114997297A (zh) * | 2022-05-26 | 2022-09-02 | 哈尔滨工业大学 | 一种基于多级区域划分的目标运动意图推理方法及*** |
CN114997297B (zh) * | 2022-05-26 | 2024-05-03 | 哈尔滨工业大学 | 一种基于多级区域划分的目标运动意图推理方法及*** |
CN115562332A (zh) * | 2022-09-01 | 2023-01-03 | 北京普利永华科技发展有限公司 | 一种无人机机载记录数据的高效处理方法及*** |
CN115562332B (zh) * | 2022-09-01 | 2023-05-16 | 北京普利永华科技发展有限公司 | 一种无人机机载记录数据的高效处理方法及*** |
CN115690160A (zh) * | 2022-11-16 | 2023-02-03 | 南京航空航天大学 | 一种低帧率视频行人轨迹预测方法与*** |
CN115690160B (zh) * | 2022-11-16 | 2023-12-15 | 南京航空航天大学 | 一种低帧率视频行人轨迹预测方法与*** |
CN115564803A (zh) * | 2022-12-06 | 2023-01-03 | 腾讯科技(深圳)有限公司 | 一种动画处理方法、装置、设备、存储介质及产品 |
CN116602663B (zh) * | 2023-06-02 | 2023-12-15 | 深圳市震有智联科技有限公司 | 一种基于毫米波雷达的智能监测方法及*** |
CN116602663A (zh) * | 2023-06-02 | 2023-08-18 | 深圳市震有智联科技有限公司 | 一种基于毫米波雷达的智能监测方法及*** |
CN116778101A (zh) * | 2023-06-26 | 2023-09-19 | 北京道仪数慧科技有限公司 | 基于营运载具的地图生成方法及*** |
CN116778101B (zh) * | 2023-06-26 | 2024-04-09 | 北京道仪数慧科技有限公司 | 基于营运载具的地图生成方法及*** |
CN116723616B (zh) * | 2023-08-08 | 2023-11-07 | 杭州依森匠能数字科技有限公司 | 一种灯光亮度控制方法及*** |
CN116723616A (zh) * | 2023-08-08 | 2023-09-08 | 杭州依森匠能数字科技有限公司 | 一种灯光亮度控制方法及*** |
CN117560638A (zh) * | 2024-01-10 | 2024-02-13 | 山东派蒙机电技术有限公司 | 应用于移动端通信***的融合通信方法、装置及设备 |
CN117560638B (zh) * | 2024-01-10 | 2024-03-22 | 山东派蒙机电技术有限公司 | 应用于移动端通信***的融合通信方法、装置及设备 |
CN117799641A (zh) * | 2024-02-08 | 2024-04-02 | 北京科技大学 | 多车干扰下智能物流车高效节能驾驶优化控制方法及装置 |
Also Published As
Publication number | Publication date |
---|---|
CN111942407A (zh) | 2020-11-17 |
JP2023511765A (ja) | 2023-03-22 |
CN111942407B (zh) | 2022-09-23 |
JP7513726B2 (ja) | 2024-07-09 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
WO2022022721A1 (zh) | 轨迹预测方法、装置、设备、存储介质及程序 | |
Chen et al. | A review of vision-based traffic semantic understanding in ITSs | |
US11682137B2 (en) | Refining depth from an image | |
US11814039B2 (en) | Vehicle operation using a dynamic occupancy grid | |
Yang et al. | Crossing or not? Context-based recognition of pedestrian crossing intention in the urban environment | |
US20210216077A1 (en) | Method, apparatus and computer storage medium for training trajectory planning model | |
US20230142676A1 (en) | Trajectory prediction method and apparatus, device, storage medium and program | |
JP2022516288A (ja) | 階層型機械学習ネットワークアーキテクチャ | |
CN111316286A (zh) | 轨迹预测方法及装置、存储介质、驾驶***与车辆 | |
US20190374151A1 (en) | Focus-Based Tagging Of Sensor Data | |
US11887324B2 (en) | Cross-modality active learning for object detection | |
Rezaei et al. | Traffic-Net: 3D traffic monitoring using a single camera | |
Zhang et al. | Gc-net: Gridding and clustering for traffic object detection with roadside lidar | |
CN115393677A (zh) | 使用融合图像的端到端***训练 | |
Ghahremannezhad et al. | Object detection in traffic videos: A survey | |
Azfar et al. | Deep learning-based computer vision methods for complex traffic environments perception: A review | |
CN117372991A (zh) | 基于多视角多模态融合的自动驾驶方法及*** | |
US20230105331A1 (en) | Methods and systems for semantic scene completion for sparse 3d data | |
Mehtab | Deep neural networks for road scene perception in autonomous vehicles using LiDARs and vision sensors | |
Ke | Real-time video analytics empowered by machine learning and edge computing for smart transportation applications | |
Nishida et al. | Environment Recognition from A Spherical Camera Image Based on DeepLab v3+ | |
Foster | Object detection and sensor data processing for off-road autonomous vehicles | |
Sarkar et al. | Real-Time Risk Prediction at Signalized Intersection Using Graph Neural Network [Webinar] | |
US20240220787A1 (en) | Neuromorphic computing system for edge computing | |
Sirmacek et al. | Sequential image processing methods for improving semantic video segmentation algorithms |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 21848661 Country of ref document: EP Kind code of ref document: A1 |
|
ENP | Entry into the national phase |
Ref document number: 2022546580 Country of ref document: JP Kind code of ref document: A |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
32PN | Ep: public notification in the ep bulletin as address of the adressee cannot be established |
Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 28/04/2023) |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 21848661 Country of ref document: EP Kind code of ref document: A1 |