US20200142433A1 - System for autonomous operation of uavs - Google Patents
System for autonomous operation of uavs Download PDFInfo
- Publication number
- US20200142433A1 US20200142433A1 US16/532,910 US201916532910A US2020142433A1 US 20200142433 A1 US20200142433 A1 US 20200142433A1 US 201916532910 A US201916532910 A US 201916532910A US 2020142433 A1 US2020142433 A1 US 2020142433A1
- Authority
- US
- United States
- Prior art keywords
- uavs
- data
- uav
- plan
- ground
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 claims abstract description 41
- 238000012544 monitoring process Methods 0.000 claims description 17
- 238000003032 molecular docking Methods 0.000 claims description 8
- 230000008859 change Effects 0.000 claims description 5
- 230000004044 response Effects 0.000 claims description 5
- 238000005265 energy consumption Methods 0.000 claims 2
- 238000004891 communication Methods 0.000 abstract description 14
- 230000009471 action Effects 0.000 description 11
- 230000006870 function Effects 0.000 description 11
- 238000007689 inspection Methods 0.000 description 7
- 238000010586 diagram Methods 0.000 description 6
- 230000002123 temporal effect Effects 0.000 description 6
- 239000003795 chemical substances by application Substances 0.000 description 4
- 230000003993 interaction Effects 0.000 description 4
- 238000005457 optimization Methods 0.000 description 4
- 230000004913 activation Effects 0.000 description 3
- 238000013473 artificial intelligence Methods 0.000 description 3
- 230000008901 benefit Effects 0.000 description 3
- 238000001514 detection method Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 239000013598 vector Substances 0.000 description 3
- 101150013568 US16 gene Proteins 0.000 description 2
- 230000001133 acceleration Effects 0.000 description 2
- 230000003213 activating effect Effects 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 2
- 230000007613 environmental effect Effects 0.000 description 2
- 238000011156 evaluation Methods 0.000 description 2
- 230000002068 genetic effect Effects 0.000 description 2
- 230000002045 lasting effect Effects 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 238000003462 Bender reaction Methods 0.000 description 1
- UFHFLCQGNIYNRP-UHFFFAOYSA-N Hydrogen Chemical compound [H][H] UFHFLCQGNIYNRP-UHFFFAOYSA-N 0.000 description 1
- 230000002411 adverse Effects 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000013481 data capture Methods 0.000 description 1
- 238000013480 data collection Methods 0.000 description 1
- 238000000354 decomposition reaction Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 230000003203 everyday effect Effects 0.000 description 1
- 238000010304 firing Methods 0.000 description 1
- 238000009472 formulation Methods 0.000 description 1
- 239000000446 fuel Substances 0.000 description 1
- 239000007789 gas Substances 0.000 description 1
- 239000001257 hydrogen Substances 0.000 description 1
- 229910052739 hydrogen Inorganic materials 0.000 description 1
- 238000003780 insertion Methods 0.000 description 1
- 230000037431 insertion Effects 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 230000001404 mediated effect Effects 0.000 description 1
- 239000000203 mixture Substances 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 238000005295 random walk Methods 0.000 description 1
- 230000002787 reinforcement Effects 0.000 description 1
- 238000002922 simulated annealing Methods 0.000 description 1
- 239000000779 smoke Substances 0.000 description 1
- 239000007921 spray Substances 0.000 description 1
- 239000000126 substance Substances 0.000 description 1
- CCEKAJIANROZEO-UHFFFAOYSA-N sulfluramid Chemical group CCNS(=O)(=O)C(F)(F)C(F)(F)C(F)(F)C(F)(F)C(F)(F)C(F)(F)C(F)(F)C(F)(F)F CCEKAJIANROZEO-UHFFFAOYSA-N 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G05—CONTROLLING; REGULATING
- G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G05D1/00—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
- G05D1/10—Simultaneous control of position or course in three dimensions
- G05D1/101—Simultaneous control of position or course in three dimensions specially adapted for aircraft
- G05D1/104—Simultaneous control of position or course in three dimensions specially adapted for aircraft involving a plurality of aircrafts, e.g. formation flying
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B64—AIRCRAFT; AVIATION; COSMONAUTICS
- B64C—AEROPLANES; HELICOPTERS
- B64C25/00—Alighting gear
- B64C25/32—Alighting gear characterised by elements which contact the ground or similar surface
- B64C25/34—Alighting gear characterised by elements which contact the ground or similar surface wheeled type, e.g. multi-wheeled bogies
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B64—AIRCRAFT; AVIATION; COSMONAUTICS
- B64C—AEROPLANES; HELICOPTERS
- B64C39/00—Aircraft not otherwise provided for
- B64C39/02—Aircraft not otherwise provided for characterised by special use
- B64C39/024—Aircraft not otherwise provided for characterised by special use of the remote controlled vehicle type, i.e. RPV
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B64—AIRCRAFT; AVIATION; COSMONAUTICS
- B64U—UNMANNED AERIAL VEHICLES [UAV]; EQUIPMENT THEREFOR
- B64U50/00—Propulsion; Power supply
- B64U50/30—Supply or distribution of electrical power
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
- G06Q10/063—Operations research, analysis or management
- G06Q10/0631—Resource planning, allocation, distributing or scheduling for enterprises or organisations
- G06Q10/06311—Scheduling, planning or task assignment for a person or group
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/08—Logistics, e.g. warehousing, loading or distribution; Inventory or stock management
-
- G—PHYSICS
- G08—SIGNALLING
- G08G—TRAFFIC CONTROL SYSTEMS
- G08G5/00—Traffic control systems for aircraft, e.g. air-traffic control [ATC]
- G08G5/0004—Transmission of traffic-related information to or from an aircraft
-
- G—PHYSICS
- G08—SIGNALLING
- G08G—TRAFFIC CONTROL SYSTEMS
- G08G5/00—Traffic control systems for aircraft, e.g. air-traffic control [ATC]
- G08G5/0004—Transmission of traffic-related information to or from an aircraft
- G08G5/0013—Transmission of traffic-related information to or from an aircraft with a ground station
-
- G—PHYSICS
- G08—SIGNALLING
- G08G—TRAFFIC CONTROL SYSTEMS
- G08G5/00—Traffic control systems for aircraft, e.g. air-traffic control [ATC]
- G08G5/0017—Arrangements for implementing traffic-related aircraft activities, e.g. arrangements for generating, displaying, acquiring or managing traffic information
- G08G5/0026—Arrangements for implementing traffic-related aircraft activities, e.g. arrangements for generating, displaying, acquiring or managing traffic information located on the ground
-
- G—PHYSICS
- G08—SIGNALLING
- G08G—TRAFFIC CONTROL SYSTEMS
- G08G5/00—Traffic control systems for aircraft, e.g. air-traffic control [ATC]
- G08G5/003—Flight plan management
- G08G5/0034—Assembly of a flight plan
-
- G—PHYSICS
- G08—SIGNALLING
- G08G—TRAFFIC CONTROL SYSTEMS
- G08G5/00—Traffic control systems for aircraft, e.g. air-traffic control [ATC]
- G08G5/003—Flight plan management
- G08G5/0039—Modification of a flight plan
-
- G—PHYSICS
- G08—SIGNALLING
- G08G—TRAFFIC CONTROL SYSTEMS
- G08G5/00—Traffic control systems for aircraft, e.g. air-traffic control [ATC]
- G08G5/0047—Navigation or guidance aids for a single aircraft
- G08G5/0069—Navigation or guidance aids for a single aircraft specially adapted for an unmanned aircraft
-
- G—PHYSICS
- G08—SIGNALLING
- G08G—TRAFFIC CONTROL SYSTEMS
- G08G5/00—Traffic control systems for aircraft, e.g. air-traffic control [ATC]
- G08G5/0073—Surveillance aids
- G08G5/0082—Surveillance aids for monitoring traffic from a ground station
-
- G—PHYSICS
- G08—SIGNALLING
- G08G—TRAFFIC CONTROL SYSTEMS
- G08G5/00—Traffic control systems for aircraft, e.g. air-traffic control [ATC]
- G08G5/0073—Surveillance aids
- G08G5/0091—Surveillance aids for monitoring atmospheric conditions
-
- B64C2201/042—
-
- B64C2201/141—
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B64—AIRCRAFT; AVIATION; COSMONAUTICS
- B64U—UNMANNED AERIAL VEHICLES [UAV]; EQUIPMENT THEREFOR
- B64U2201/00—UAVs characterised by their flight controls
- B64U2201/10—UAVs characterised by their flight controls autonomous, i.e. by navigating independently from ground or air stations, e.g. by using inertial navigation systems [INS]
Definitions
- One or more embodiments of the present disclosure generally relate to Unmanned Aerial Vehicles (UAVs), or “drones”, and specifically to the autonomous control of UAVs.
- UAVs Unmanned Aerial Vehicles
- drones drones
- UAVs are unmanned aircraft that can carry sensors, video cameras, communications equipment, or other payloads, and are typically remotely piloted individually by an operator or controlled by an automated system.
- Embodiments described herein include an automated system for controlling a plurality of unmanned aerial vehicles (UAVs).
- the automated system can include a navigational system and/or a scheduling system.
- a navigational system and a scheduling system can collectively be referred to as a planning system.
- the system includes a receiver, a transmitter, and a processor in communication with a memory. The receiver receives a first telemetric data from the plurality of UAVs, and the transmitter is configured to transmit control data to the plurality of UAVs.
- the processor-issuable instructions can include instructions to: substantially simultaneously determine a plurality of plans (e.g., navigational, scheduling and/or other plans) for each of the plurality of UAVs based at least on the first telemetric data; and iteratively revise the plurality of plans.
- the plurality of plans can include an instruction for at least one of the UAVs of the plurality of UAVs to travel to a recharging station.
- Recharging stations of the present disclosure can be passive devices or can be in wireless communication with the automated system and/or one or more UAVs of the plurality of UAVs.
- At least one of the first telemetric data and the updated telemetric data includes position data and/or battery charge data.
- iteratively revising the plurality of plans is based on updated telemetric data received, via the receiver, from the plurality of UAVs.
- At least one UAV of the plurality of UAVs can be an unmanned ground aerial vehicle (UGAV) comprising wheels for ground movement (which may also be referred to as a “hybrid” UAV), and the control data transmitted to the at least one UAV (i.e., to the UGAV) can comprise both flight navigation data and ground navigation data.
- UGAV unmanned ground aerial vehicle
- hybrid hybrid
- the plans for each of the plurality of UAVs can be coordinated so as to at least one of: avoid collisions between UAVs of the plurality of UAVs; avoid collisions between UAVs and other objects or regions specified on a geographical navigation model; avoid interaction of the UAVs with events specified on a temporal model; synchronize the planned movements of the UAVs and the activation of their sensors in order to guarantee continuance of data acquisition of a particular region or event for an arbitrarily long amount of time; ensure that the plans for the UAVs allows the data acquisition specified as objective of the plan; and ensure that each UAV of the plurality of UAVs maintains a battery charge that is above a predetermined level, which could be different for each UAV.
- the plans can be coordinated to ensure that at least one UAV is present in each of a predetermined plurality of geographic zones.
- the plans can be coordinated to ensure that no more than one UAV is present in each of a predetermined plurality of geographic zones.
- the updated telemetric data received from the plurality of UAVs can include a UAV fault notification.
- At least one of the first telemetric data and the updated telemetric data can include meteorological data.
- the updated telemetric data can include trigger event data.
- control data can include at least one of: ground navigation data, flight navigation data, data acquisition data, communication data, take-off data, landing data, docking data and recharging data.
- the plans are coordinated so as to run substantially concurrently with one another.
- a method of controlling a plurality of unmanned aerial vehicles includes: (1) substantially simultaneously determining a plurality of plans for each of the plurality of UAVs based at least on the first telemetric data; and, subsequently, (2) iteratively revising the plurality of plans. Iteratively revising the plurality of plans can be performed at predetermined time intervals (e.g., at pre-specified times of day) and/or can be based on a decision made by a configured software program.
- the scheduling system includes a receiver, a transmitter, and a processor in communication with a memory.
- the receiver and/or transmitter might be the same in the scheduling system and/or the navigation system.
- the receiver can receive a first telemetric data from one or more external devices (e.g., UAVs), and the transmitter is configured to transmit control data to the external devices.
- the memory can include processor-issuable instructions to: substantially simultaneously determine a plurality of actions for the devices installed in each of a plurality of UAVs or in other pre-defined connected receivers that are not necessarily installed on UAVs (e.g., alarm systems, charging stations, etc.).
- FIG. 1 is a block diagram of an automated system for controlling a plurality of unmanned aerial vehicles (UAVs), according to an embodiment.
- UAVs unmanned aerial vehicles
- FIG. 2 is a block diagram of an unmanned ground aerial vehicle (UGAV) planner, according to an embodiment.
- UGAV unmanned ground aerial vehicle
- FIG. 3 is a block diagram of a planner that is subject to a supervisor function, according to an embodiment.
- FIG. 4 is a flowchart depicting a method of controlling a plurality of UAVs, according to an embodiment.
- FIG. 5 illustrates an alarm monitoring plan involving 3 UGAVs, over a rectangular navigation model, according to an embodiment.
- FIG. 6 illustrates the alarm monitoring plan of FIG. 5 over time.
- FIG. 7 illustrates a front-end graphical interface to configure some aspects of a scheduling system, according to an embodiment.
- Small helicopters and multi-copters including hybrid airplanes with vertical take-off and landing (VTOL) vehicles, are a fast expanding technology in the civil market. They are increasingly used in operations that would expose people to risk (e.g., in chemical plants, nuclear plants, etc.), activities carried out in highly inaccessible areas (e.g., viaducts, skyscrapers, furnaces, offshore rescue, etc.), and monitoring procedures requiring a high degree of precision (e.g., inspection of cracks in dams/bridges, airplane maintenance, oil tank keels, etc.).
- risk e.g., in chemical plants, nuclear plants, etc.
- activities carried out in highly inaccessible areas e.g., viaducts, skyscrapers, furnaces, offshore rescue, etc.
- monitoring procedures requiring a high degree of precision e.g., inspection of cracks in dams/bridges, airplane maintenance, oil tank keels, etc.
- VTOL vehicle technology ranges from industrial (e.g., in plants, machineries, etc.) to environmental safeguarding (e.g., volcanoes, parks, coasts, etc.), including the protection of sensitive areas (e.g., archaeological sites, territorial borders, etc.).
- industrial e.g., in plants, machineries, etc.
- environmental safeguarding e.g., volcanoes, parks, coasts, etc.
- sensitive areas e.g., archaeological sites, territorial borders, etc.
- ICAPS International Conference on Automated Planning and Scheduling
- AAMAS International Conference on Autonomous Agents and Multi-agent Systems
- Known autonomous navigation systems have relatively limited functionality. While they may be able to perform simple operations involving only one drone with a limited battery-time, they are typically unable to perform complex multi-drone operations continuously for an unlimited period of time. In addition, such systems may lack support for functionality such as: intelligent coordination of multiple UAVs at the same time, continuous 24/7 operations, automatic precise landing, automatic recharging, robustness with respect to unpredictability of the operations, and alarm integrations.
- UAVs may be equipped with a ground propulsion system that allows the UAV to translate and/or rotate on the ground, for instance by means of wheels of various nature, and to take off, fly and land by means for instance of propellers or wings.
- a ground propulsion system that allows the UAV to translate and/or rotate on the ground, for instance by means of wheels of various nature, and to take off, fly and land by means for instance of propellers or wings.
- Examples include multi-copters equipped with an attached ground propulsion system, or mobile ground robots (e.g. mini-rovers) equipped with an attachment providing them wings and/or vertical take-off and landing propulsion methods.
- UAV-attachable ground propulsion systems are described in International Patent Application No.: PCT/US16/26152, titled “Ground Movement System Plugin for Vertical Take Off and Landing Unmanned Aerial Vehicles,” filed on Apr. 6, 2016, the entire contents of which are hereby incorporated by reference in their entirety.
- Such systems consider planning/scheduling operations between only flying robots/
- Embodiments described herein are related to a system for simultaneously controlling and/or coordinating operations or activities of multiple UAVs (i.e., a “swarm” of UAVs) at the same time, that include piloting under control of software.
- UAVs i.e., a “swarm” of UAVs
- planeners e.g., a UAV planner
- the operations can consist of movement on the ground or on air, collectively referred to as “navigation” (e.g., at a particular time for a particular distance at a particular speed), as well as other operations such as data acquisition from an onboard sensor, communication with another UAV, use of actuators, take-off/landing, docking, monitoring a particular region of the navigation model for a particular time in a particular way (e.g., capturing landscape pictures of the area every 5 seconds, or continuous videos of a particular spot), communicating some data with a charging station (also referred to herein as a “UAV station”), docking into a specific UAV station, communicating via an external device, activating specific on-board sensors or equipment, activating specific actuators, taking-off vertically and reaching a specific altitude, landing in a specific trajectory and speed, recharging for a given amount of time.
- avigation e.g., at a particular time for a particular distance at a particular speed
- other operations such as data acquisition
- Control includes navigation as well as other forms of UAV control.
- Systems described herein are useful for the planning of missions using pre-computed and/or dynamic information, planning for guaranteed time-continuance without service disruption over an operational period that would consume more than the energy available in a single UAV, and for generating robust mission plans that can adapt to uncertainty and the occurrence of faults.
- Planners described herein can achieve efficient spatial and temporal coordination of UAVs in swarms.
- the problem of choice of the operation is referred to as “scheduling”, and the problem of considering scheduling together with navigation can be referred to as “planning.”
- the planner can include software that is directed by one or more deterministic automated planning and scheduling algorithms, or interface to an optimization/artificial intelligence/heuristic software system, or can interface with another software operated partially by one or more human operators.
- the flight of the UAVs can be controlled through some permanent or intermittent remote link/connection established by an onboard device (data transmission can be radio, cellular LTE or any other known mobile communication technology, including relaying through satellite, or with focused electromagnetic beams etc.).
- Navigation and operation instructions can be imparted commands such as application program interfaces (APIs) released by drone vendors or direct instructions to the autopilot of the UAV.
- APIs include those released by 3D Robotics, DJI or the MavLink protocol.
- FIG. 1 is a block diagram of an automated system 100 for controlling a plurality of unmanned aerial vehicles (UAVs) 110 A- 110 D, according to an embodiment.
- the system 100 includes a receiver 102 , a processor 104 a transmitter 108 and a memory 106 .
- the receiver is configured to receive signals from a plurality of UAVs (e.g., including UGAVs) that collectively may form a “cluster.”
- the receiver is also configured to receive signals from one or more non-UAV locations, such as external sensors (e.g., alarm systems) on the ground.
- UAV clusters can work together in a coordinated fashion to accomplish one or more tasks, such as an inspection of a large infrastructure or a surveillance operation of an extended region.
- Each UAV of the plurality UAVs can have the capability to fly (e.g., with the aid of one or more wings and/or propellers) as well as to move along the ground/terrain (e.g., with the aid of tracks, wheels, etc.).
- the signals received at the receiver 102 from the one or more UAVs can include telemetric data, such as: sensor data (e.g., collected by a corresponding UAV via one or more onboard sensors during flight and/or ground travel), position data, battery/energy level data, video data, captured static images, point-cloud data from lasers, structured or raw data processed by a on-board computer on the UAV, acceleration or velocity data useful for odometry, and/or the like.
- sensor data e.g., collected by a corresponding UAV via one or more onboard sensors during flight and/or ground travel
- position data e.g., collected by a corresponding UAV via one or more onboard sensors during flight and/or ground travel
- video data e.g., captured static images
- point-cloud data from lasers e.g., structured or raw data processed by a on-board computer on the UAV
- acceleration or velocity data useful for odometry e.g., acceleration or velocity data useful for odometry
- the memory 106 can be local to the system 100 and/or may comprise cloud-based storage.
- the memory 106 can include processor-readable instructions to: substantially simultaneously determine a plurality of navigational and/or scheduling plans for each of the plurality of UAVs, for example based at least on a first telemetric data received from the plurality of UAVs, and to revise the plurality of navigational and/or scheduling plans (e.g., iteratively). Iteratively revising the plurality of plans can be performed at predetermined time intervals (e.g., at pre-specified times of day) and/or can be based on a decision made by a configured software program.
- the memory can also store one or more maps (e.g., 2D and/or 3D terrain maps), meteorological weather data, GPS coordinate data, UAV navigational history data, historical or statistical data relevant for the mission planning, information on the hardware performance for some UAV and sensors, information on the behavior of different human or automated users authorized to configure or control the planning system, etc.
- maps e.g., 2D and/or 3D terrain maps
- the receiver 102 is operably coupled to, and in communication with, one or more processors 104 that are in communication with the memory 106 (e.g., a cloud-based database, or a local volatile or non-volatile memory), for example to retrieve instructions therefrom and/or to store data received from the one or more UAVs 110 A- 110 D.
- the memory 106 e.g., a cloud-based database, or a local volatile or non-volatile memory
- the processor 104 can be configured to retrieve data and/or instructions from the memory 106 , and to determine a plurality of navigational and/or scheduling plans for each of the plurality of UAVs. Determining the plurality of navigational and/or scheduling plans for each of the plurality of UAVs can involve adherence to one or more predetermined and/or actively changing plan constraints and objective functions.
- the plurality of plans can include an instruction for at least one of the UAVs of the plurality of UAVs to travel to a recharging station. Such an instruction can include the timing, location and/or duration of a recharging event.
- the plan constraints can be deterministic rules that influence the determination of a plan.
- the rules may force UAVs to recharge on the nearest available UAV station before their batteries are depleted.
- the rules can direct the UAVs to switch among themselves in a “relay mode” to perform an operation that lasts longer than the battery time of a single UAV.
- An example of such a rule would be that at a certain battery level, say 20%, a UAV needs to interrupt what it is doing in 30 seconds and reach a station within 2 minutes, while another UAV whose battery level is at least 80% and which can reach the point of the first UAV within 30 seconds needs to reach a position close to the first UAV in order to substitute it in its operation.
- the plan constraints can represent the functioning of the UAVs and enforce relations between the movements or actions performed by the UAVs and the state of other variables defined in the system. For example, a rule could determine that the battery level variable associated to an UAV needs to decrease of a certain amount depending on the nature and extent of a movement of the robot. Another rule determines the position of the UAV on the navigation model over time, under specified conditions.
- Plan constraints can take the form of probabilistic rules defined over intervals of time. For instance, a plan constraint could be that a certain area in the navigation model contains the presence of one UAV among the available UAV at least 50% of the total time allocated for all operations with the UAVs, during night hours.
- Plan constraints can be formulated in a formal action language typical of formal logic in mathematics computer science, robotics, or artificial intelligence, or a formalism designed to reasoning about dynamical situations. Examples include PDDL/STRIPS variants, TAEMS, Boolean SAT, Situation/Fluent Calculus languages such as GOLOG, Linear Temporal Logic formulas, etc.
- the processor 104 is operably coupled to the transmitter 108 , for example to provide data to the transmitter 108 for wireless transmission to one or more of the plurality of UAVs 110 A- 110 D, as shown in FIG. 1 .
- the data transmitted to the one or more UAVs via the transmitter 108 can include the plurality of navigational and/or scheduling plans. These plans can include control data such as ground navigation data, flight navigation data, data acquisition data, communication data, take-off data, landing data, docking data and recharging data.
- a navigational plan can include one or more of the following: itinerary of positional destinations (e.g. in the form of expected time of arrival at waypoints) for a specified period of time, change of value at prescribed times for all the status variables of the system, list of velocities and acceleration vectors for the prescribed trajectories, definition of the mean of navigation through the available propulsion methods (air or ground), list of orientation in space of the UAV at any given time during the calculated trajectories, etc.
- the navigational plans for each of the plurality of UAVs are coordinated so as to at least one of: avoid collisions between UAVs of the plurality of UAVs; avoid collisions between UAVs and other objects or regions specified on a geographical navigation model; avoid interaction of the UAVs with events specified on a temporal model; synchronize the planned movements of the UAVs and the activation of their sensors in order to guarantee continuance of data acquisition of a particular region or event for an arbitrarily long amount of time; ensure that the navigation plans for the UAVs allows the data acquisition specified as objective of the plan; and ensure that each UAV of the plurality of UAVs maintains a battery charge that is above a predetermined level, which could be different for each UAV.
- the navigational plans can be coordinated to ensure that predetermined number (e.g., at least one, not more than one, two, etc.) of UAVs are present in each of a set of predetermined geographic zones, for example as defined by the label of the zone in the navigation model (e.g. zones with high probability of human presence).
- predetermined number e.g., at least one, not more than one, two, etc.
- a scheduling plan can include one or more of the following, in a time resolved manner: one or more sensor activation instructions, one or more data collection instructions, instructions that result in the movement of mechanical parts of devices such as doors, passages, conveyor belts, instructions of controlling the light or sounds emitted by external devices, instructions that make external device communicate or transmit over a wired or wireless channel, instructions to activate devices such as anti-burglary systems (e.g. sprays, or smoke generators), etc.
- One or more navigational and/or scheduling plans can be transmitted to the one or more UAVs, and these navigational and/or scheduling plans can be identical to one another, different from each of the others (e.g., customized), or any combination.
- the revision of the navigational and/or scheduling plans can be based on automatic predetermined triggers, such as at predetermined time intervals, or in response to one or more detected events (i.e., external triggers), such as the firing of an alarm, a change in weather conditions, a change in air-traffic-control policies, input from a human operator, the detection of a nearby vehicle, input from an external program that is modeling the happenings on the actionable environment, updated telemetric data received, via the receiver, from the plurality of UAVs, or input from a user that is in communication with the system 100 .
- detected events i.e., external triggers
- a trigger is a data packet with a predetermined information structure, meant to be detected and classified by the processor 104 (e.g., a UAV or UGAV planner, as described in deatil below) to be representative of the event that generated the trigger.
- the transmitter 108 can transmit the plurality of navigational and/or scheduling plans to one or more of the plurality of UAVs 110 A- 110 D simultaneously or substantially simultaneously.
- the plurality of navigational and/or scheduling plans can be coordinated so as to run substantially concurrently (e.g., by being defined by the processor to be run at overlapping or concurrent time periods).
- the updated telemetric data received from the plurality of UAVs comprises a UAV fault notification and/or trigger event data.
- an anomaly can be a GPS link failure, or the battery status of an UAV is not following the predicted models, or the movement trajectories are different than the ones instructed.
- the updated telemetric data can include one or more of: meteorological data, sensor data, position data, battery life data, etc.
- one or more UAVs of the plurality of UAVs 110 A- 110 D is in wireless communication with another UAV of the plurality of UAVs 110 A- 110 D and transmits, for example, data including meteorological data, sensor data, position data and/or battery life data to the other UAV.
- the receiver 102 , processor 104 , memory 106 and transmitter 108 are co-located within a common enclosure or device.
- FIG. 2 is a block diagram of an unmanned ground aerial vehicle (UGAV) planner 200 , or planning system, according to an embodiment.
- FIG. 2 represents a global description of how an embodiment of the system could be connected to the external word.
- the UGAV planner 200 can be implemented in many forms of software, firmware and hardware, as long as the actual software code or algorithms or control hardware for the implementation is not limiting the uses and capabilities of the UGAV planner 200 as described herein.
- the UGAV planner 200 generates a plan that will try to satisfy predetermined and/or actively changing plan constraints and objective functions using all resources available (e.g., UAVs, UGAVs) and all relevant information available to represent the status of the system at the time of execution of the planning algorithms. As shown in FIG.
- the UGAV planner 200 can receive data from one or more alarm sensors 222 (e.g., on the ground/not airborne), a meteorological (weather) station via one or more weather sensors 224 , and/or from an external trigger or stimulus, such as a button that can be pressed by a human in a graphical user interface (GUI) control room 226 , or in any other user interaction (UI) system in a human-in-the-loop mode.
- an external trigger can allow a human operator to take complete control of one or more UAV's navigation, for example through a joystick.
- the one or more weather sensors 224 provide environmental information to the UGAV planner 200 via a data feed (e.g., including a geographical reference).
- the UGAV planner 200 can receive data from one or more charging stations 220 , for example indicating whether or not a UAV is presently charging there, the status of their mechanical components, etc., as if they were external sensors.
- the UGAV planner 200 can be wirelessly connected to, and receive telemetric data from, the UGAVs 210 .
- the UGAV Planner 200 may be informed of the telemetric status of all the UGAVs, without notice, or at a regular intervals of time.
- Access of some or all information to the UGAV Planner 200 can be direct or mediated by a database or a third software.
- the UGAV planner 200 can perform planning operations and generate navigational and/or scheduling plans (subject to any plan constraints) to control several UAVs 210 (e.g., UGAVs) to which the UGAV planner 200 is wirelessly connected, for example based on telemetry data received from the UAVs.
- Navigational and/or scheduling plans can be determined for a given interval of time (which may be finite, semi-infinite or open-ended) that respects any plan constraints, and returns information on the violation of the constraints, as well as information of plan quality if soft constraints are present, given enough computational resources and time to process on at least one processor.
- Methods to determine the navigational and/or scheduling plans include known methods in the domain of Planning and Scheduling and Combinatorial Optimization.
- Examples of computational techniques or solvers are: Branch and Bound, Dynamic Programming, Forward-Search, Heuristic searches, Trial and errors, Greedy methods, SAT-based formulations, Markov Decision Processes such as POMDP methods, Mixed-Integer-Linear-Programming solvers such as CPLEX or Gurobi, Iterative methods, Genetic Algorithms, Monte-Carlo Tree searches and Random Walks, multi-level heuristic techniques, Ant Colony optimization methods, Simulated annealing, Master-Agent decomposition software architectures, Disjunctive graphs methods, Benders Decompositions, Constraint Satisfaction Programming methods, Hierarchical Task Networks, Machine Learning Methods, including Unsupervised and Supervised learning, reinforcement learning, Deep Learning Methods, non-digital methods such as the use of analog or neuromorphic computers, and electronic, biological or quantum computing devices.
- the UGAV planner 200 can be in communication with one or more databases ( 206 A- 206 B) that store data relevant to the UGAV planning operations.
- the UGAV planner 200 can access status data (e.g., position data, weather data, speed/velocity data, battery life data, etc.) corresponding to one or more of the UAVs 210 through the one or more databases 206 A- 206 B.
- the UGAV planner 200 can access geographical data through the one or more databases 206 A- 206 B.
- the UGAV planner 200 may have access to geographical data for an area where UGAV operations take place, and may be in permanent or intermittent connection with the UGAVs through a telecommunications link.
- the connection to the UGAVs may be such that the planning system can control the navigation of the UGAVs and the data acquisition of the sensors mounted on the UGAV, for example by transmitting the navigational and/or scheduling plans (e.g., including one or more scheduled recharging events) to the UGAVs 210 .
- the navigational and/or scheduling plans can include future position data for one or more of the plurality of UGAVs.
- the future position data can specify precise locations for the one or more of the plurality of UGAVs, or may specify ranges of position within which the one or more of the plurality of UGAVs are permitted to be at any given time.
- some plan constraints of the UAV planner can be that under certain conditions some of the UAV needs to be forbidden ground movement operations, or flight operations, in all or some of the prescribed area of operation. More generally, any restriction on the possible movement of the UAV can be formulated, even in a collective form. For example, it can be enforced that all the movements inside buildings or all the paths that cross regions where it is likely to be human presence needs to be performed on the ground.
- the plan constraints can be configured during initialization of the UGAV planner 200 , or can be injected in the UGAV planner 200 at any time, e.g., automatically by a packaged module available to the UGAV planner 200 and/or by a user through a graphical or command line User Interface.
- the initialization or the update of the UGAV planner 200 includes cryptographic authentication, such as the digital signature of a patch update or the insertion of a user password (e.g., including multiple factor authentications).
- plan constraints can be defined as “soft constraints” or “objective functions,” which can serve as a figure-of-merit for the UGAV planner 200 to distinguish among multiple different plans based on quality.
- These functions' outputs can be represented by a scalar, a vector or a tensor, or in general by any n-tuple of real numbers, provided a distance metric to determine the level of optimality is also defined.
- an objective function could be to minimize the total amount of energy consumed in the plan, or to maximize the number of concurrent operations with all the available UAV.
- a plan generated by the UGAV planner 200 includes an instruction for a recharging station to increase the power of a recharging unit of a docked UAV.
- a recharging station to increase the power of a recharging unit of a docked UAV.
- Another example is the opening of a door allowing a UAV to enter a building, as discussed further below with reference to FIG. 5 .
- the automated plan constraints and objective functions are converted in a form to be represented as a Boolean Satisfiability Problem.
- the problem can be formulated as a variant of job-shop-scheduling, or an unconstrained Mixed-Integer Programming problem, linear or non-linear.
- the generated navigation plans can be modified by one or more obstacle avoidance algorithms (e.g., algorithms based on Vector Field Histograms (VFH)) or collision avoidance algorithms.
- obstacle avoidance algorithms e.g., algorithms based on Vector Field Histograms (VFH)
- collision avoidance algorithms e.g., algorithms based on Vector Field Histograms (VFH)
- a panner has absolute temporal information (e.g., it can be synchronized with external sensors) and can access on demand the following information within a specific time interval, given as input: (A) the values of the variables representing the status of the system (e.g. the position of the UAVs and their battery sensor status); (B) some pre-defined rules or “plan constraints” that depend on (A) and also on additional static conditions, such as the time and date of the day of operation, and the geographical information of the area where the operation takes place; (C) some other plan constraints that depend on (A) and (B) and also on additional dynamic conditions, such as the occurrence of the triggers by external sensors or human interaction.
- A the values of the variables representing the status of the system (e.g. the position of the UAVs and their battery sensor status)
- B some pre-defined rules or “plan constraints” that depend on (A) and also on additional static conditions, such as the time and date of the day of operation, and the geographical information of the area where the operation takes place
- C some other
- FIG. 3 is a block diagram of a planner 300 (e.g., a UGAV planner) that is subject to a supervisor function, according to an embodiment. More specifically, FIG. 3 shows a planner 300 that is encapsulated as a module of a supervisor 330 (e.g., an artificial intelligence system).
- the planner 300 and/or the supervisor 330 can be implemented in many forms of software, firmware and hardware, as long as the actual software code or algorithms or control hardware for the implementation is not limiting the uses and capabilities of the planner 300 and/or the supervisor 330 as described herein.
- the planner 300 and/or the supervisor 330 generates a plan that will try to satisfy predetermined and/or actively changing plan constraints and objective functions using all resources available (e.g., UAVs, UGAVs) and all relevant information available to represent the status of the system at the time of execution of the planning algorithms.
- the supervisor 330 and/or the planner 300 can receive data from one or more alarm sensors 322 (e.g., on the ground/not airborne), a meteorological (weather) station via one or more weather sensors 324 , and/or from an external trigger or stimulus, such as a button that can be pressed by a human in a GUI control room 326 .
- the supervisor 330 and/or the planner 300 can also receive data from a plurality of UAVs 310 and/or a plurality of UAV recharging stations 320 , and the planner 300 can be configured to wirelessly communicate with the plurality of UAVs 310 and/or the plurality of UAV recharging stations 320 .
- the supervisor 330 and/or the planner 300 can be in communication with one or more databases ( 306 A- 306 B) that store data relevant to planning operations.
- the planner 300 can perform planning operations and create navigational and/or scheduling plans to control several UAVs 210 (e.g., UGAVs) to which the planner 300 is wirelessly connected.
- the supervisor 330 can use the planner 300 to generate plans, schedules and/or actions for the plurality of UAVs 310 and one or more recharging stations 320 , and to update the databases 306 A- 306 B, however unlike in FIG. 2 , the supervisor 330 has control over when the planner 300 is invoked (e.g., control over when the planner 300 transmits the plans, schedules and/or action data to the UAVs 310 ).
- the planner 300 generates plans, schedules and/or actions for the plurality of UAVs 310 and one or more recharging stations 320 , and the supervisor 330 has control over when the planner 300 is invoked (e.g., control over when the planner 300 transmits the plans, schedules and/or action data to the UAVs 310 ).
- the supervisor 330 can query the planner 300 at regular intervals of time (e.g., about 10 seconds) with parameters representing projected models of reality based on the current status and/or on forecasting algorithms.
- the supervisor 330 can have its own internal policies regarding when and under what conditions execute a plan generated by the planner 300 .
- the supervisor 330 program has memory access (data stored in a database 306 A, 306 B) of all generated and executed plans by the UAVs 310 and of the status variables over time, for example as communicated by the external sensors and by telemetry.
- the supervisor 330 program queries the planner 300 when a certain threshold of difference between the status of at least one of a UAV telemetry feed and a projected status predicted by the plan in execution computed by the planner 300 is reached. For example, the supervisor 330 could decide to “re-plan” when the integrated sum of the absolute values of the differences between all trajectories of UAVs 310 and their predicted model reaches a value of, for example, about 10 square meters, or when the sum of the absolute value of the time-shifts with respect to the predicted time of arrivals at specific waypoints is superior to 10 seconds.
- the supervisor 330 program is entirely or partially under the direction of a human operator, who determines or influences when a plan is generated by the planner 300 and when it is executed by the UAVs 310 .
- the planner 300 is called to generate one or more plans in order to perform photography and/or video recordings of specific geographic sectors within a certain amount of time elapsed from the receipt of a trigger event from an alarm 322 installed on an industrial site.
- the planner 300 could be called by a human-configured supervisor 330 to generate a plan lasting 15 minutes, with the objective of monitoring the sector were the alarm went off, as well as the perimeter of several buildings of an industrial campus along with the main gates and access roads, with specific pre-defined probabilities over the 15 minutes period.
- the planner may generate a plan that will try to satisfy the plan constraints and the objective functions using all resources available (e.g., UAVs or UGAVs in a swarm) and all information available to represent the status of the system at the time of execution of the planning algorithms.
- resources available e.g., UAVs or UGAVs in a swarm
- the planner 300 is called to generate one or more plans in order to perform inspections of machinery at specific times during the day, or at the earliest available time after the scheduled operation assuming that there are adverse atmospheric events affecting the quality of the data capture.
- the planner 300 could be invoked at 11 am every day to inspect and map the status of a dam of height 100 m and width 600 m.
- the planner 300 running planning algorithms subject to these plan constraints could, for instance, generate plans that continuously use 8 UAVs in relay mode, with 4 UAVs active at any given time, for an operation lasting approximately 6 hours.
- the planner 300 is called at pre-defined times during the day, and the actions suggested are directly executed by the UAVs and/or other devices controlled by the planner 300 .
- the planning can be invoked to generate solutions to other use cases.
- An example, non-exhaustive list of such use cases includes: video surveillance of urban areas, intrusion detection at the interior of warehouses, national border patrol, inspection of oil & gas facilities, inspection of bridges, inspection of the exterior of power plants, detection of fires or poaching in wilderness areas, detecting leakages over pipelines, harbor video surveillance and tracking of cargo, monitoring the activity of a volcano, 3D mapping of a construction site, and video and/or photo documentation of an area subject to a natural disaster, such as an earthquake.
- a combination of open-source and proprietary solvers can be used to perform the planning.
- an optimization software package such as IBM ILOG CPLEX Optimization Studio (CPLEX) could be used to solve the scheduling problem, while a genetic algorithm among those published in the robotics literature could suggest a navigation plan for the multi-agent system.
- CPLEX IBM ILOG CPLEX Optimization Studio
- the resulting plans can be concurrently run by the supervisor 330 software on different solvers in order to have alternative solutions and choose to execute one according to a given policy configuration.
- FIG. 4 is a flowchart depicting a method 440 of controlling a plurality of UAVs (e.g., UGAVs), according to an embodiment.
- the method 440 which is compatible with the systems shown in FIGS. 1-3 , includes reading the status variables (e.g., docking positions, waypoint data, labels associated with UAVs and their status (e.g., is it flying or charging), descriptional labels, and/or the like) of UAVs, sensors and/or other external controlled devices 441 , followed by determining (e.g., via the processor 104 of FIG. 1 or the UGAV planner of FIG.
- the status variables e.g., docking positions, waypoint data, labels associated with UAVs and their status (e.g., is it flying or charging), descriptional labels, and/or the like
- determining e.g., via the processor 104 of FIG. 1 or the UGAV planner of FIG.
- Update conditions can include the passage of a predetermined period of time, the detected movement of one or more of the plurality UAVs beyond a predetermined geographical window or volume, the receipt (e.g., at a receiver or at a processor) of an external trigger or stimulus, etc.
- the method proceeds to 452 , a “wait” period, and returns to evaluation 446 either autonomously (e.g., after a predetermined wait period), in response to one or more external triggers, or both. If it is determined that an update condition is met at 446 , then the method proceeds to 448 , where one or more revised navigational plans and/or scheduling plans are determined. The one or more revised navigational plans and/or scheduling plans are transmitted to one or more of the plurality of UAVs at 450 .
- the determination of revised navigational plans and/or scheduling plans at 448 includes determining revised navigational plans and/or scheduling plans for all UAVs of the plurality of UAVs and, correspondingly, a plurality of revised navigational plans and/or scheduling plans are transmitted—one to each of the plurality of UAVs.
- the method then proceeds to wait, at 452 , until a predetermined wait period has elapsed, or until one or more external triggers are received, before again evaluating whether a subsequent update condition has been met (at 446 ).
- the method of FIG. 4 further includes the use of a 2D map and/or a 3D map as an input for a navigation model (e.g., to generate the navigational plan at 442 ).
- a navigation model e.g., to generate the navigational plan at 442 .
- Such maps may be representative of the terrain and/or status of the region of operation.
- the maps can include data on one or more charging stations and/or data on an initial position of UAVs of the plurality of UAVs at a point in time.
- the navigation model can be created by a human operator, or be a default model available, for example, from a library of models.
- the navigation model can include a reference to the location of docking/landing/recharging stations (a “UAV Station”).
- UAV Stations can comprise devices or locations on the ground (which may or not be mobile) where the UAVs can autonomously dock, either through landing directly on them or through entering them by means of a ground accessible opening, in order to recharge/swap their batteries (or energy supply units, e.g., hydrogen fuel cells), be protected from the outside environment, and/or update, upload or download the data stored in their companion computer system on board.
- batteries or energy supply units, e.g., hydrogen fuel cells
- FIG. 5 illustrates an alarm monitoring plan involving 3 UGAVs, over a rectangular navigation model, according to an embodiment.
- the alarm monitoring plan generated for example by a system similar to those described above with reference to FIGS. 1-3 and/or according to a method similar to the method of FIG. 4 , has a duration of 26 minutes, and has an objective of continuously monitoring an alarm sector (defined, at least in part, by the chart area) for at least 15 minutes (a plan “constraint”).
- UGAV-1 was in sector F4 at the beginning of the scheduled time, and was subsequently sent to check an alarm in sector L3, which had just gone off and which triggered the generation of the plan. Then, the UGAV-1 was sent to a station in C9 to recharge.
- a second UGAV-2 is sent to move on the ground from a station in Q9 where it was recharging towards a region to monitor inside a building (Q3).
- This region to monitor could be optional (and, for example, specified as a soft constraint).
- the planner e.g., similar to the processor 104 of FIG. 1 , the UGAV planner of FIG. 2 , or the planner 300 of FIG. 3 ) coordinates the opening and closure of doors in Q6.
- FIG. 6 illustrates the alarm monitoring plan of FIG. 5 from a temporal profile point of view, through a Gantt chart. For each action described in the alarm monitoring plan, a corresponding task bar is drawn and defined. Tasks that are vertically aligned are executed synchronously.
- FIG. 7 illustrates an embodiment of a graphical interface allowing a human operator to configure the planning system and/or the supervisor software.
- Form 701 is a form via which an operator can set up a time duration in minutes of a specific plan called “Fire Alarm” that is for example being called when a trigger of a fire alarm sensor goes off.
- the user can specify the importance of each specific sector (to be defined on a navigation model) to be set as a parameter of plan constraints.
- a “Sector 1 patrol” plan is configured, in a box, for example to be starting each day at 6:00 AM and to last until 12:30 PM.
- An “All sectors patrol” plan for the scheduling of a patrol operation of all sectors in the map is shown at 703 .
- a button 704 is included to submit the values of the form boxes to the planner/supervisor that is now configured.
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Aviation & Aerospace Engineering (AREA)
- Business, Economics & Management (AREA)
- Human Resources & Organizations (AREA)
- Remote Sensing (AREA)
- Radar, Positioning & Navigation (AREA)
- Economics (AREA)
- Strategic Management (AREA)
- Entrepreneurship & Innovation (AREA)
- Marketing (AREA)
- Operations Research (AREA)
- Quality & Reliability (AREA)
- Development Economics (AREA)
- Tourism & Hospitality (AREA)
- General Business, Economics & Management (AREA)
- Theoretical Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Educational Administration (AREA)
- Game Theory and Decision Science (AREA)
- Life Sciences & Earth Sciences (AREA)
- Atmospheric Sciences (AREA)
- Automation & Control Theory (AREA)
- Chemical & Material Sciences (AREA)
- Combustion & Propulsion (AREA)
- Mechanical Engineering (AREA)
- Control Of Position, Course, Altitude, Or Attitude Of Moving Bodies (AREA)
- Traffic Control Systems (AREA)
Abstract
Description
- This application claims benefit of the earlier filing date of U.S. Provisional Patent Application No.: 62/183,597, titled “System for Autonomous Operation of Multiple Unmanned Aerial Vehicles Supported by Recharging Stations to Perform Services,” filed on Jun. 23, 2015, and International Patent Application No.: PCT/US16/26152, titled “Ground Movement System Plugin for Vertical Take Off and Landing Unmanned Aerial Vehicles,” filed on Apr. 6, 2016, the entire contents of which are hereby incorporated by reference in their entirety.
- All patents, patent applications and publications cited herein are hereby incorporated by reference in their entirety in order to more fully describe the state of the art as known to those skilled therein as of the date of the invention described herein.
- One or more embodiments of the present disclosure generally relate to Unmanned Aerial Vehicles (UAVs), or “drones”, and specifically to the autonomous control of UAVs.
- UAVs are unmanned aircraft that can carry sensors, video cameras, communications equipment, or other payloads, and are typically remotely piloted individually by an operator or controlled by an automated system.
- Embodiments described herein include an automated system for controlling a plurality of unmanned aerial vehicles (UAVs). The automated system can include a navigational system and/or a scheduling system. A navigational system and a scheduling system can collectively be referred to as a planning system. In some such embodiments, the system includes a receiver, a transmitter, and a processor in communication with a memory. The receiver receives a first telemetric data from the plurality of UAVs, and the transmitter is configured to transmit control data to the plurality of UAVs. The processor-issuable instructions can include instructions to: substantially simultaneously determine a plurality of plans (e.g., navigational, scheduling and/or other plans) for each of the plurality of UAVs based at least on the first telemetric data; and iteratively revise the plurality of plans. The plurality of plans can include an instruction for at least one of the UAVs of the plurality of UAVs to travel to a recharging station. Recharging stations of the present disclosure can be passive devices or can be in wireless communication with the automated system and/or one or more UAVs of the plurality of UAVs.
- In some embodiments, at least one of the first telemetric data and the updated telemetric data includes position data and/or battery charge data.
- In some embodiments, iteratively revising the plurality of plans is based on updated telemetric data received, via the receiver, from the plurality of UAVs.
- At least one UAV of the plurality of UAVs can be an unmanned ground aerial vehicle (UGAV) comprising wheels for ground movement (which may also be referred to as a “hybrid” UAV), and the control data transmitted to the at least one UAV (i.e., to the UGAV) can comprise both flight navigation data and ground navigation data.
- In some embodiments, the plans for each of the plurality of UAVs can be coordinated so as to at least one of: avoid collisions between UAVs of the plurality of UAVs; avoid collisions between UAVs and other objects or regions specified on a geographical navigation model; avoid interaction of the UAVs with events specified on a temporal model; synchronize the planned movements of the UAVs and the activation of their sensors in order to guarantee continuance of data acquisition of a particular region or event for an arbitrarily long amount of time; ensure that the plans for the UAVs allows the data acquisition specified as objective of the plan; and ensure that each UAV of the plurality of UAVs maintains a battery charge that is above a predetermined level, which could be different for each UAV.
- In some embodiments, the plans can be coordinated to ensure that at least one UAV is present in each of a predetermined plurality of geographic zones.
- In some embodiments, the plans can be coordinated to ensure that no more than one UAV is present in each of a predetermined plurality of geographic zones.
- In some embodiments, the updated telemetric data received from the plurality of UAVs can include a UAV fault notification.
- In some embodiments, at least one of the first telemetric data and the updated telemetric data can include meteorological data.
- In some embodiments, the updated telemetric data can include trigger event data.
- In some embodiments, the control data can include at least one of: ground navigation data, flight navigation data, data acquisition data, communication data, take-off data, landing data, docking data and recharging data.
- In some embodiments, the plans are coordinated so as to run substantially concurrently with one another.
- In some embodiments, a method of controlling a plurality of unmanned aerial vehicles (UAVs) includes: (1) substantially simultaneously determining a plurality of plans for each of the plurality of UAVs based at least on the first telemetric data; and, subsequently, (2) iteratively revising the plurality of plans. Iteratively revising the plurality of plans can be performed at predetermined time intervals (e.g., at pre-specified times of day) and/or can be based on a decision made by a configured software program.
- In some embodiments, the scheduling system includes a receiver, a transmitter, and a processor in communication with a memory. The receiver and/or transmitter might be the same in the scheduling system and/or the navigation system. The receiver can receive a first telemetric data from one or more external devices (e.g., UAVs), and the transmitter is configured to transmit control data to the external devices. The memory can include processor-issuable instructions to: substantially simultaneously determine a plurality of actions for the devices installed in each of a plurality of UAVs or in other pre-defined connected receivers that are not necessarily installed on UAVs (e.g., alarm systems, charging stations, etc.).
- In order to illustrate the way in which the above mentioned advantages and features of the embodiments can be obtained, a brief description of the components is provided in the appended drawings. It needs to be underlined that the figures are not drawn to scale, and similar elements, in terms of function or shape, are indicated through reference numerals; the drawings should be considered as illustrations of typical embodiments without limiting of the scope.
-
FIG. 1 is a block diagram of an automated system for controlling a plurality of unmanned aerial vehicles (UAVs), according to an embodiment. -
FIG. 2 is a block diagram of an unmanned ground aerial vehicle (UGAV) planner, according to an embodiment. -
FIG. 3 is a block diagram of a planner that is subject to a supervisor function, according to an embodiment. -
FIG. 4 is a flowchart depicting a method of controlling a plurality of UAVs, according to an embodiment. -
FIG. 5 illustrates an alarm monitoring plan involving 3 UGAVs, over a rectangular navigation model, according to an embodiment. -
FIG. 6 illustrates the alarm monitoring plan ofFIG. 5 over time. -
FIG. 7 illustrates a front-end graphical interface to configure some aspects of a scheduling system, according to an embodiment. - Small helicopters and multi-copters, including hybrid airplanes with vertical take-off and landing (VTOL) vehicles, are a fast expanding technology in the civil market. They are increasingly used in operations that would expose people to risk (e.g., in chemical plants, nuclear plants, etc.), activities carried out in highly inaccessible areas (e.g., viaducts, skyscrapers, furnaces, offshore rescue, etc.), and monitoring procedures requiring a high degree of precision (e.g., inspection of cracks in dams/bridges, airplane maintenance, oil tank keels, etc.). In addition to industrial and governmental critical infrastructure applications, potential applications of VTOL vehicle technology range from industrial (e.g., in plants, machineries, etc.) to environmental safeguarding (e.g., volcanoes, parks, coasts, etc.), including the protection of sensitive areas (e.g., archaeological sites, territorial borders, etc.).
- Some operations, such as 24/7 video surveillance, alarm response, monitoring and inspections, can benefit from an automatic navigation system. Research systems for planning and scheduling of autonomous robotic systems have been proposed, for example at international conferences such as the International Conference on Automated Planning and Scheduling (ICAPS) and the International Conference on Autonomous Agents and Multi-agent Systems (AAMAS). The terms “navigation,” “planning” and “scheduling,” as used herein, are defined according to their common usage in the relevant art, for example, as typified by ICAPS and AAMAS research.
- Known autonomous navigation systems have relatively limited functionality. While they may be able to perform simple operations involving only one drone with a limited battery-time, they are typically unable to perform complex multi-drone operations continuously for an unlimited period of time. In addition, such systems may lack support for functionality such as: intelligent coordination of multiple UAVs at the same time, continuous 24/7 operations, automatic precise landing, automatic recharging, robustness with respect to unpredictability of the operations, and alarm integrations.
- In addition, some UAVs may be equipped with a ground propulsion system that allows the UAV to translate and/or rotate on the ground, for instance by means of wheels of various nature, and to take off, fly and land by means for instance of propellers or wings. Examples include multi-copters equipped with an attached ground propulsion system, or mobile ground robots (e.g. mini-rovers) equipped with an attachment providing them wings and/or vertical take-off and landing propulsion methods. UAV-attachable ground propulsion systems are described in International Patent Application No.: PCT/US16/26152, titled “Ground Movement System Plugin for Vertical Take Off and Landing Unmanned Aerial Vehicles,” filed on Apr. 6, 2016, the entire contents of which are hereby incorporated by reference in their entirety. Such systems consider planning/scheduling operations between only flying robots/aircrafts/UAVs or ground robots as separate entities, which can collaborate in some cases, but they are not designed to control robots that are capable both of flying and of actively moving on the ground.
- As such, there is currently a need for systems and methods to direct autonomous operations, on the ground and/or in the air, of swarms of robots (such as UAVs) that are air and ground-capable, in indoor and/or outdoor environments, in a dynamic and reliable way, taking into account the options for the robots to automatically refuel their energy supply.
- Embodiments described herein are related to a system for simultaneously controlling and/or coordinating operations or activities of multiple UAVs (i.e., a “swarm” of UAVs) at the same time, that include piloting under control of software. Such systems are referred to herein as “planners” (e.g., a UAV planner). The operations can consist of movement on the ground or on air, collectively referred to as “navigation” (e.g., at a particular time for a particular distance at a particular speed), as well as other operations such as data acquisition from an onboard sensor, communication with another UAV, use of actuators, take-off/landing, docking, monitoring a particular region of the navigation model for a particular time in a particular way (e.g., capturing landscape pictures of the area every 5 seconds, or continuous videos of a particular spot), communicating some data with a charging station (also referred to herein as a “UAV station”), docking into a specific UAV station, communicating via an external device, activating specific on-board sensors or equipment, activating specific actuators, taking-off vertically and reaching a specific altitude, landing in a specific trajectory and speed, recharging for a given amount of time. Control includes navigation as well as other forms of UAV control. Systems described herein are useful for the planning of missions using pre-computed and/or dynamic information, planning for guaranteed time-continuance without service disruption over an operational period that would consume more than the energy available in a single UAV, and for generating robust mission plans that can adapt to uncertainty and the occurrence of faults. Planners described herein can achieve efficient spatial and temporal coordination of UAVs in swarms.
- The problem of choice of the operation is referred to as “scheduling”, and the problem of considering scheduling together with navigation can be referred to as “planning.” The planner can include software that is directed by one or more deterministic automated planning and scheduling algorithms, or interface to an optimization/artificial intelligence/heuristic software system, or can interface with another software operated partially by one or more human operators.
- The flight of the UAVs can be controlled through some permanent or intermittent remote link/connection established by an onboard device (data transmission can be radio, cellular LTE or any other known mobile communication technology, including relaying through satellite, or with focused electromagnetic beams etc.). Navigation and operation instructions can be imparted commands such as application program interfaces (APIs) released by drone vendors or direct instructions to the autopilot of the UAV. Example APIs include those released by 3D Robotics, DJI or the MavLink protocol.
-
FIG. 1 is a block diagram of anautomated system 100 for controlling a plurality of unmanned aerial vehicles (UAVs) 110A-110D, according to an embodiment. As shown inFIG. 1 , thesystem 100 includes areceiver 102, a processor 104 atransmitter 108 and amemory 106. The receiver is configured to receive signals from a plurality of UAVs (e.g., including UGAVs) that collectively may form a “cluster.” In some embodiments, the receiver is also configured to receive signals from one or more non-UAV locations, such as external sensors (e.g., alarm systems) on the ground. UAV clusters can work together in a coordinated fashion to accomplish one or more tasks, such as an inspection of a large infrastructure or a surveillance operation of an extended region. Each UAV of the plurality UAVs can have the capability to fly (e.g., with the aid of one or more wings and/or propellers) as well as to move along the ground/terrain (e.g., with the aid of tracks, wheels, etc.). - The signals received at the
receiver 102 from the one or more UAVs can include telemetric data, such as: sensor data (e.g., collected by a corresponding UAV via one or more onboard sensors during flight and/or ground travel), position data, battery/energy level data, video data, captured static images, point-cloud data from lasers, structured or raw data processed by a on-board computer on the UAV, acceleration or velocity data useful for odometry, and/or the like. - The
memory 106 can be local to thesystem 100 and/or may comprise cloud-based storage. Thememory 106 can include processor-readable instructions to: substantially simultaneously determine a plurality of navigational and/or scheduling plans for each of the plurality of UAVs, for example based at least on a first telemetric data received from the plurality of UAVs, and to revise the plurality of navigational and/or scheduling plans (e.g., iteratively). Iteratively revising the plurality of plans can be performed at predetermined time intervals (e.g., at pre-specified times of day) and/or can be based on a decision made by a configured software program. The memory can also store one or more maps (e.g., 2D and/or 3D terrain maps), meteorological weather data, GPS coordinate data, UAV navigational history data, historical or statistical data relevant for the mission planning, information on the hardware performance for some UAV and sensors, information on the behavior of different human or automated users authorized to configure or control the planning system, etc. - The
receiver 102 is operably coupled to, and in communication with, one ormore processors 104 that are in communication with the memory 106 (e.g., a cloud-based database, or a local volatile or non-volatile memory), for example to retrieve instructions therefrom and/or to store data received from the one ormore UAVs 110A-110D. - As discussed above, the
processor 104 can be configured to retrieve data and/or instructions from thememory 106, and to determine a plurality of navigational and/or scheduling plans for each of the plurality of UAVs. Determining the plurality of navigational and/or scheduling plans for each of the plurality of UAVs can involve adherence to one or more predetermined and/or actively changing plan constraints and objective functions. The plurality of plans can include an instruction for at least one of the UAVs of the plurality of UAVs to travel to a recharging station. Such an instruction can include the timing, location and/or duration of a recharging event. - The plan constraints can be deterministic rules that influence the determination of a plan. The rules, for example, may force UAVs to recharge on the nearest available UAV station before their batteries are depleted. The rules can direct the UAVs to switch among themselves in a “relay mode” to perform an operation that lasts longer than the battery time of a single UAV. An example of such a rule would be that at a certain battery level, say 20%, a UAV needs to interrupt what it is doing in 30 seconds and reach a station within 2 minutes, while another UAV whose battery level is at least 80% and which can reach the point of the first UAV within 30 seconds needs to reach a position close to the first UAV in order to substitute it in its operation.
- The plan constraints can represent the functioning of the UAVs and enforce relations between the movements or actions performed by the UAVs and the state of other variables defined in the system. For example, a rule could determine that the battery level variable associated to an UAV needs to decrease of a certain amount depending on the nature and extent of a movement of the robot. Another rule determines the position of the UAV on the navigation model over time, under specified conditions.
- Plan constraints can take the form of probabilistic rules defined over intervals of time. For instance, a plan constraint could be that a certain area in the navigation model contains the presence of one UAV among the available UAV at least 50% of the total time allocated for all operations with the UAVs, during night hours.
- Plan constraints can be formulated in a formal action language typical of formal logic in mathematics computer science, robotics, or artificial intelligence, or a formalism designed to reasoning about dynamical situations. Examples include PDDL/STRIPS variants, TAEMS, Boolean SAT, Situation/Fluent Calculus languages such as GOLOG, Linear Temporal Logic formulas, etc.
- The
processor 104 is operably coupled to thetransmitter 108, for example to provide data to thetransmitter 108 for wireless transmission to one or more of the plurality ofUAVs 110A-110D, as shown inFIG. 1 . The data transmitted to the one or more UAVs via thetransmitter 108 can include the plurality of navigational and/or scheduling plans. These plans can include control data such as ground navigation data, flight navigation data, data acquisition data, communication data, take-off data, landing data, docking data and recharging data. - A navigational plan can include one or more of the following: itinerary of positional destinations (e.g. in the form of expected time of arrival at waypoints) for a specified period of time, change of value at prescribed times for all the status variables of the system, list of velocities and acceleration vectors for the prescribed trajectories, definition of the mean of navigation through the available propulsion methods (air or ground), list of orientation in space of the UAV at any given time during the calculated trajectories, etc. In some embodiments, the navigational plans for each of the plurality of UAVs are coordinated so as to at least one of: avoid collisions between UAVs of the plurality of UAVs; avoid collisions between UAVs and other objects or regions specified on a geographical navigation model; avoid interaction of the UAVs with events specified on a temporal model; synchronize the planned movements of the UAVs and the activation of their sensors in order to guarantee continuance of data acquisition of a particular region or event for an arbitrarily long amount of time; ensure that the navigation plans for the UAVs allows the data acquisition specified as objective of the plan; and ensure that each UAV of the plurality of UAVs maintains a battery charge that is above a predetermined level, which could be different for each UAV. In some embodiments, the navigational plans can be coordinated to ensure that predetermined number (e.g., at least one, not more than one, two, etc.) of UAVs are present in each of a set of predetermined geographic zones, for example as defined by the label of the zone in the navigation model (e.g. zones with high probability of human presence).
- A scheduling plan can include one or more of the following, in a time resolved manner: one or more sensor activation instructions, one or more data collection instructions, instructions that result in the movement of mechanical parts of devices such as doors, passages, conveyor belts, instructions of controlling the light or sounds emitted by external devices, instructions that make external device communicate or transmit over a wired or wireless channel, instructions to activate devices such as anti-burglary systems (e.g. sprays, or smoke generators), etc. One or more navigational and/or scheduling plans can be transmitted to the one or more UAVs, and these navigational and/or scheduling plans can be identical to one another, different from each of the others (e.g., customized), or any combination.
- The revision of the navigational and/or scheduling plans can be based on automatic predetermined triggers, such as at predetermined time intervals, or in response to one or more detected events (i.e., external triggers), such as the firing of an alarm, a change in weather conditions, a change in air-traffic-control policies, input from a human operator, the detection of a nearby vehicle, input from an external program that is modeling the happenings on the actionable environment, updated telemetric data received, via the receiver, from the plurality of UAVs, or input from a user that is in communication with the
system 100. As used herein, a trigger is a data packet with a predetermined information structure, meant to be detected and classified by the processor 104 (e.g., a UAV or UGAV planner, as described in deatil below) to be representative of the event that generated the trigger. When instructed by theprocessor 104, thetransmitter 108 can transmit the plurality of navigational and/or scheduling plans to one or more of the plurality ofUAVs 110A-110D simultaneously or substantially simultaneously. Also, the plurality of navigational and/or scheduling plans can be coordinated so as to run substantially concurrently (e.g., by being defined by the processor to be run at overlapping or concurrent time periods). - The updated telemetric data received from the plurality of UAVs comprises a UAV fault notification and/or trigger event data. For example, an anomaly can be a GPS link failure, or the battery status of an UAV is not following the predicted models, or the movement trajectories are different than the ones instructed. Alternatively or in addition, the updated telemetric data can include one or more of: meteorological data, sensor data, position data, battery life data, etc. In some embodiments, one or more UAVs of the plurality of
UAVs 110A-110D is in wireless communication with another UAV of the plurality ofUAVs 110A-110D and transmits, for example, data including meteorological data, sensor data, position data and/or battery life data to the other UAV. - The
receiver 102,processor 104,memory 106 andtransmitter 108 are co-located within a common enclosure or device. -
FIG. 2 is a block diagram of an unmanned ground aerial vehicle (UGAV)planner 200, or planning system, according to an embodiment.FIG. 2 represents a global description of how an embodiment of the system could be connected to the external word. Depending upon the embodiment, theUGAV planner 200 can be implemented in many forms of software, firmware and hardware, as long as the actual software code or algorithms or control hardware for the implementation is not limiting the uses and capabilities of theUGAV planner 200 as described herein. In all implementations set forth herein, theUGAV planner 200 generates a plan that will try to satisfy predetermined and/or actively changing plan constraints and objective functions using all resources available (e.g., UAVs, UGAVs) and all relevant information available to represent the status of the system at the time of execution of the planning algorithms. As shown inFIG. 2 , theUGAV planner 200 can receive data from one or more alarm sensors 222 (e.g., on the ground/not airborne), a meteorological (weather) station via one ormore weather sensors 224, and/or from an external trigger or stimulus, such as a button that can be pressed by a human in a graphical user interface (GUI)control room 226, or in any other user interaction (UI) system in a human-in-the-loop mode. In some embodiments, an external trigger can allow a human operator to take complete control of one or more UAV's navigation, for example through a joystick. In some embodiments, the one ormore weather sensors 224 provide environmental information to theUGAV planner 200 via a data feed (e.g., including a geographical reference). - The
UGAV planner 200 can receive data from one or more chargingstations 220, for example indicating whether or not a UAV is presently charging there, the status of their mechanical components, etc., as if they were external sensors. TheUGAV planner 200 can be wirelessly connected to, and receive telemetric data from, theUGAVs 210. In some embodiments, theUGAV Planner 200 may be informed of the telemetric status of all the UGAVs, without notice, or at a regular intervals of time. - Access of some or all information to the
UGAV Planner 200 can be direct or mediated by a database or a third software. - The
UGAV planner 200 can perform planning operations and generate navigational and/or scheduling plans (subject to any plan constraints) to control several UAVs 210 (e.g., UGAVs) to which theUGAV planner 200 is wirelessly connected, for example based on telemetry data received from the UAVs. Navigational and/or scheduling plans can be determined for a given interval of time (which may be finite, semi-infinite or open-ended) that respects any plan constraints, and returns information on the violation of the constraints, as well as information of plan quality if soft constraints are present, given enough computational resources and time to process on at least one processor. Methods to determine the navigational and/or scheduling plans include known methods in the domain of Planning and Scheduling and Combinatorial Optimization. Examples of computational techniques or solvers are: Branch and Bound, Dynamic Programming, Forward-Search, Heuristic searches, Trial and errors, Greedy methods, SAT-based formulations, Markov Decision Processes such as POMDP methods, Mixed-Integer-Linear-Programming solvers such as CPLEX or Gurobi, Iterative methods, Genetic Algorithms, Monte-Carlo Tree searches and Random Walks, multi-level heuristic techniques, Ant Colony optimization methods, Simulated annealing, Master-Agent decomposition software architectures, Disjunctive graphs methods, Benders Decompositions, Constraint Satisfaction Programming methods, Hierarchical Task Networks, Machine Learning Methods, including Unsupervised and Supervised learning, reinforcement learning, Deep Learning Methods, non-digital methods such as the use of analog or neuromorphic computers, and electronic, biological or quantum computing devices. - The
UGAV planner 200 can be in communication with one or more databases (206A-206B) that store data relevant to the UGAV planning operations. In some implementations, theUGAV planner 200 can access status data (e.g., position data, weather data, speed/velocity data, battery life data, etc.) corresponding to one or more of theUAVs 210 through the one ormore databases 206A-206B. Alternatively or in addition, theUGAV planner 200 can access geographical data through the one ormore databases 206A-206B. - The
UGAV planner 200 may have access to geographical data for an area where UGAV operations take place, and may be in permanent or intermittent connection with the UGAVs through a telecommunications link. The connection to the UGAVs may be such that the planning system can control the navigation of the UGAVs and the data acquisition of the sensors mounted on the UGAV, for example by transmitting the navigational and/or scheduling plans (e.g., including one or more scheduled recharging events) to theUGAVs 210. - The navigational and/or scheduling plans can include future position data for one or more of the plurality of UGAVs. The future position data can specify precise locations for the one or more of the plurality of UGAVs, or may specify ranges of position within which the one or more of the plurality of UGAVs are permitted to be at any given time.
- Some embodiments, some plan constraints of the UAV planner can be that under certain conditions some of the UAV needs to be forbidden ground movement operations, or flight operations, in all or some of the prescribed area of operation. More generally, any restriction on the possible movement of the UAV can be formulated, even in a collective form. For example, it can be enforced that all the movements inside buildings or all the paths that cross regions where it is likely to be human presence needs to be performed on the ground.
- The plan constraints can be configured during initialization of the
UGAV planner 200, or can be injected in theUGAV planner 200 at any time, e.g., automatically by a packaged module available to theUGAV planner 200 and/or by a user through a graphical or command line User Interface. In some embodiments, the initialization or the update of theUGAV planner 200 includes cryptographic authentication, such as the digital signature of a patch update or the insertion of a user password (e.g., including multiple factor authentications). - In some embodiments, plan constraints can be defined as “soft constraints” or “objective functions,” which can serve as a figure-of-merit for the
UGAV planner 200 to distinguish among multiple different plans based on quality. These functions' outputs can be represented by a scalar, a vector or a tensor, or in general by any n-tuple of real numbers, provided a distance metric to determine the level of optimality is also defined. For example, an objective function could be to minimize the total amount of energy consumed in the plan, or to maximize the number of concurrent operations with all the available UAV. - In some embodiments, a plan generated by the
UGAV planner 200 includes an instruction for a recharging station to increase the power of a recharging unit of a docked UAV. Another example is the opening of a door allowing a UAV to enter a building, as discussed further below with reference toFIG. 5 . - In one implementation of the
UGAV planner 200, the automated plan constraints and objective functions are converted in a form to be represented as a Boolean Satisfiability Problem. Alternatively, or concurrently, the problem can be formulated as a variant of job-shop-scheduling, or an unconstrained Mixed-Integer Programming problem, linear or non-linear. - In some implementations of the
UGAV planner 200, the generated navigation plans can be modified by one or more obstacle avoidance algorithms (e.g., algorithms based on Vector Field Histograms (VFH)) or collision avoidance algorithms. - In some embodiments, a panner has absolute temporal information (e.g., it can be synchronized with external sensors) and can access on demand the following information within a specific time interval, given as input: (A) the values of the variables representing the status of the system (e.g. the position of the UAVs and their battery sensor status); (B) some pre-defined rules or “plan constraints” that depend on (A) and also on additional static conditions, such as the time and date of the day of operation, and the geographical information of the area where the operation takes place; (C) some other plan constraints that depend on (A) and (B) and also on additional dynamic conditions, such as the occurrence of the triggers by external sensors or human interaction.
-
FIG. 3 is a block diagram of a planner 300 (e.g., a UGAV planner) that is subject to a supervisor function, according to an embodiment. More specifically,FIG. 3 shows aplanner 300 that is encapsulated as a module of a supervisor 330 (e.g., an artificial intelligence system). Theplanner 300 and/or thesupervisor 330 can be implemented in many forms of software, firmware and hardware, as long as the actual software code or algorithms or control hardware for the implementation is not limiting the uses and capabilities of theplanner 300 and/or thesupervisor 330 as described herein. In all implementations set forth herein, theplanner 300 and/or thesupervisor 330 generates a plan that will try to satisfy predetermined and/or actively changing plan constraints and objective functions using all resources available (e.g., UAVs, UGAVs) and all relevant information available to represent the status of the system at the time of execution of the planning algorithms. As shown inFIG. 3 , thesupervisor 330 and/or theplanner 300 can receive data from one or more alarm sensors 322 (e.g., on the ground/not airborne), a meteorological (weather) station via one ormore weather sensors 324, and/or from an external trigger or stimulus, such as a button that can be pressed by a human in aGUI control room 326. Thesupervisor 330 and/or theplanner 300 can also receive data from a plurality ofUAVs 310 and/or a plurality ofUAV recharging stations 320, and theplanner 300 can be configured to wirelessly communicate with the plurality ofUAVs 310 and/or the plurality ofUAV recharging stations 320. Thesupervisor 330 and/or theplanner 300 can be in communication with one or more databases (306A-306B) that store data relevant to planning operations. Theplanner 300 can perform planning operations and create navigational and/or scheduling plans to control several UAVs 210 (e.g., UGAVs) to which theplanner 300 is wirelessly connected. In some embodiments, thesupervisor 330 can use theplanner 300 to generate plans, schedules and/or actions for the plurality ofUAVs 310 and one ormore recharging stations 320, and to update thedatabases 306A-306B, however unlike inFIG. 2 , thesupervisor 330 has control over when theplanner 300 is invoked (e.g., control over when theplanner 300 transmits the plans, schedules and/or action data to the UAVs 310). In other embodiments, theplanner 300 generates plans, schedules and/or actions for the plurality ofUAVs 310 and one ormore recharging stations 320, and thesupervisor 330 has control over when theplanner 300 is invoked (e.g., control over when theplanner 300 transmits the plans, schedules and/or action data to the UAVs 310). - The
supervisor 330 can query theplanner 300 at regular intervals of time (e.g., about 10 seconds) with parameters representing projected models of reality based on the current status and/or on forecasting algorithms. Thesupervisor 330 can have its own internal policies regarding when and under what conditions execute a plan generated by theplanner 300. - In some embodiments, the
supervisor 330 program has memory access (data stored in adatabase UAVs 310 and of the status variables over time, for example as communicated by the external sensors and by telemetry. - In some embodiments, the
supervisor 330 program queries theplanner 300 when a certain threshold of difference between the status of at least one of a UAV telemetry feed and a projected status predicted by the plan in execution computed by theplanner 300 is reached. For example, thesupervisor 330 could decide to “re-plan” when the integrated sum of the absolute values of the differences between all trajectories ofUAVs 310 and their predicted model reaches a value of, for example, about 10 square meters, or when the sum of the absolute value of the time-shifts with respect to the predicted time of arrivals at specific waypoints is superior to 10 seconds. - In some embodiments, the
supervisor 330 program is entirely or partially under the direction of a human operator, who determines or influences when a plan is generated by theplanner 300 and when it is executed by theUAVs 310. - In one embodiment, the
planner 300 is called to generate one or more plans in order to perform photography and/or video recordings of specific geographic sectors within a certain amount of time elapsed from the receipt of a trigger event from analarm 322 installed on an industrial site. For example, theplanner 300 could be called by a human-configuredsupervisor 330 to generate a plan lasting 15 minutes, with the objective of monitoring the sector were the alarm went off, as well as the perimeter of several buildings of an industrial campus along with the main gates and access roads, with specific pre-defined probabilities over the 15 minutes period. The planner may generate a plan that will try to satisfy the plan constraints and the objective functions using all resources available (e.g., UAVs or UGAVs in a swarm) and all information available to represent the status of the system at the time of execution of the planning algorithms. - In one embodiment, the
planner 300 is called to generate one or more plans in order to perform inspections of machinery at specific times during the day, or at the earliest available time after the scheduled operation assuming that there are adverse atmospheric events affecting the quality of the data capture. For example, in a hydroelectric power plant, theplanner 300 could be invoked at 11 am every day to inspect and map the status of a dam of height 100 m and width 600 m. Theplanner 300, running planning algorithms subject to these plan constraints could, for instance, generate plans that continuously use 8 UAVs in relay mode, with 4 UAVs active at any given time, for an operation lasting approximately 6 hours. - In some embodiments, the
planner 300 is called at pre-defined times during the day, and the actions suggested are directly executed by the UAVs and/or other devices controlled by theplanner 300. - In other embodiments, the planning can be invoked to generate solutions to other use cases. An example, non-exhaustive list of such use cases includes: video surveillance of urban areas, intrusion detection at the interior of warehouses, national border patrol, inspection of oil & gas facilities, inspection of bridges, inspection of the exterior of power plants, detection of fires or poaching in wilderness areas, detecting leakages over pipelines, harbor video surveillance and tracking of cargo, monitoring the activity of a volcano, 3D mapping of a construction site, and video and/or photo documentation of an area subject to a natural disaster, such as an earthquake.
- In one implementation of the
planner 300, a combination of open-source and proprietary solvers can be used to perform the planning. For example, an optimization software package such as IBM ILOG CPLEX Optimization Studio (CPLEX) could be used to solve the scheduling problem, while a genetic algorithm among those published in the robotics literature could suggest a navigation plan for the multi-agent system. In some cases, the resulting plans can be concurrently run by thesupervisor 330 software on different solvers in order to have alternative solutions and choose to execute one according to a given policy configuration. -
FIG. 4 is a flowchart depicting amethod 440 of controlling a plurality of UAVs (e.g., UGAVs), according to an embodiment. As shown inFIG. 4 , themethod 440, which is compatible with the systems shown inFIGS. 1-3 , includes reading the status variables (e.g., docking positions, waypoint data, labels associated with UAVs and their status (e.g., is it flying or charging), descriptional labels, and/or the like) of UAVs, sensors and/or other external controlled devices 441, followed by determining (e.g., via theprocessor 104 ofFIG. 1 or the UGAV planner ofFIG. 2 ) a plurality of navigational plans and/or scheduling plans for a plurality of UAVs (442), e.g., based on the status variables. The plurality of navigational plans and/or scheduling plans is transmitted, at 444 (e.g., via a transmitter) to the plurality of UAVs. An evaluation is performed, at 446, of whether or not an update condition has been met. Update conditions can include the passage of a predetermined period of time, the detected movement of one or more of the plurality UAVs beyond a predetermined geographical window or volume, the receipt (e.g., at a receiver or at a processor) of an external trigger or stimulus, etc. If an update condition is not met, the method proceeds to 452, a “wait” period, and returns toevaluation 446 either autonomously (e.g., after a predetermined wait period), in response to one or more external triggers, or both. If it is determined that an update condition is met at 446, then the method proceeds to 448, where one or more revised navigational plans and/or scheduling plans are determined. The one or more revised navigational plans and/or scheduling plans are transmitted to one or more of the plurality of UAVs at 450. In some embodiments, the determination of revised navigational plans and/or scheduling plans at 448 includes determining revised navigational plans and/or scheduling plans for all UAVs of the plurality of UAVs and, correspondingly, a plurality of revised navigational plans and/or scheduling plans are transmitted—one to each of the plurality of UAVs. The method then proceeds to wait, at 452, until a predetermined wait period has elapsed, or until one or more external triggers are received, before again evaluating whether a subsequent update condition has been met (at 446). - In some embodiments, the method of
FIG. 4 further includes the use of a 2D map and/or a 3D map as an input for a navigation model (e.g., to generate the navigational plan at 442). Such maps may be representative of the terrain and/or status of the region of operation. Alternatively or in addition, the maps can include data on one or more charging stations and/or data on an initial position of UAVs of the plurality of UAVs at a point in time. The navigation model can be created by a human operator, or be a default model available, for example, from a library of models. The navigation model can include a reference to the location of docking/landing/recharging stations (a “UAV Station”). UAV Stations can comprise devices or locations on the ground (which may or not be mobile) where the UAVs can autonomously dock, either through landing directly on them or through entering them by means of a ground accessible opening, in order to recharge/swap their batteries (or energy supply units, e.g., hydrogen fuel cells), be protected from the outside environment, and/or update, upload or download the data stored in their companion computer system on board. -
FIG. 5 illustrates an alarm monitoring plan involving 3 UGAVs, over a rectangular navigation model, according to an embodiment. The alarm monitoring plan, generated for example by a system similar to those described above with reference toFIGS. 1-3 and/or according to a method similar to the method ofFIG. 4 , has a duration of 26 minutes, and has an objective of continuously monitoring an alarm sector (defined, at least in part, by the chart area) for at least 15 minutes (a plan “constraint”). In the alarm monitoring plan ofFIG. 5 , UGAV-1 was in sector F4 at the beginning of the scheduled time, and was subsequently sent to check an alarm in sector L3, which had just gone off and which triggered the generation of the plan. Then, the UGAV-1 was sent to a station in C9 to recharge. In order to dock into the station, the UAV flies to, and lands in, F8 and then moves along the ground to reach C9. Concurrently to all these actions, a second UGAV-2 is sent to move on the ground from a station in Q9 where it was recharging towards a region to monitor inside a building (Q3). This region to monitor could be optional (and, for example, specified as a soft constraint). In order to facilitate the robot's entry into the building, the planner (e.g., similar to theprocessor 104 ofFIG. 1 , the UGAV planner ofFIG. 2 , or theplanner 300 ofFIG. 3 ) coordinates the opening and closure of doors in Q6. After a brief monitoring period, UGAV-2 is sent to the sector L3 to replace the monitoring action of UGAV-1 which, as soon as UGAV-2 arrives, leaves sector L3. After 3.5 minutes, UGAV-2 leaves the sector, taking off and flying to sector N9, where it lands and it moves to dock into the recharging station in Q9 where it was hosted 21 minutes earlier. During all these operations, a UGAV-3 in C2 is recharging.FIG. 6 illustrates the alarm monitoring plan ofFIG. 5 from a temporal profile point of view, through a Gantt chart. For each action described in the alarm monitoring plan, a corresponding task bar is drawn and defined. Tasks that are vertically aligned are executed synchronously. -
FIG. 7 illustrates an embodiment of a graphical interface allowing a human operator to configure the planning system and/or the supervisor software.Form 701 is a form via which an operator can set up a time duration in minutes of a specific plan called “Fire Alarm” that is for example being called when a trigger of a fire alarm sensor goes off. In the form boxes, the user can specify the importance of each specific sector (to be defined on a navigation model) to be set as a parameter of plan constraints. At 702, a “Sector 1 patrol” plan is configured, in a box, for example to be starting each day at 6:00 AM and to last until 12:30 PM. An “All sectors patrol” plan for the scheduling of a patrol operation of all sectors in the map is shown at 703. Abutton 704 is included to submit the values of the form boxes to the planner/supervisor that is now configured. - Although the disclosed subject matter has been described and illustrated in the foregoing exemplary embodiments, it is understood that the present disclosure has been made only by way of example, and that numerous changes in the details of implementation of the disclosed subject matter may be made without departing from the spirit and scope of the disclosed subject matter.
Claims (23)
Priority Applications (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US16/532,910 US10976752B2 (en) | 2015-06-23 | 2019-08-06 | System for autonomous operation of UAVs |
US17/227,663 US11644851B2 (en) | 2015-06-23 | 2021-04-12 | System for autonomous operation of multiple hybrid unmanned aerial vehicles supported by recharging station to perform services |
Applications Claiming Priority (4)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US201562183597P | 2015-06-23 | 2015-06-23 | |
PCT/US2016/039055 WO2016210156A1 (en) | 2015-06-23 | 2016-06-23 | System for autonomous operation of multiple hybrid unmanned aerial vehicles supported by recharging stations to perform services |
US201715738236A | 2017-12-20 | 2017-12-20 | |
US16/532,910 US10976752B2 (en) | 2015-06-23 | 2019-08-06 | System for autonomous operation of UAVs |
Related Parent Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US15/738,236 Continuation US10372142B2 (en) | 2015-06-23 | 2016-06-23 | System for autonomous operation of multiple hybrid unmanned aerial vehicles supported by recharging stations to perform services |
PCT/US2016/039055 Continuation WO2016210156A1 (en) | 2015-06-23 | 2016-06-23 | System for autonomous operation of multiple hybrid unmanned aerial vehicles supported by recharging stations to perform services |
Related Child Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US17/227,663 Continuation US11644851B2 (en) | 2015-06-23 | 2021-04-12 | System for autonomous operation of multiple hybrid unmanned aerial vehicles supported by recharging station to perform services |
Publications (2)
Publication Number | Publication Date |
---|---|
US20200142433A1 true US20200142433A1 (en) | 2020-05-07 |
US10976752B2 US10976752B2 (en) | 2021-04-13 |
Family
ID=57586434
Family Applications (3)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US15/738,236 Active 2036-07-25 US10372142B2 (en) | 2015-06-23 | 2016-06-23 | System for autonomous operation of multiple hybrid unmanned aerial vehicles supported by recharging stations to perform services |
US16/532,910 Active US10976752B2 (en) | 2015-06-23 | 2019-08-06 | System for autonomous operation of UAVs |
US17/227,663 Active US11644851B2 (en) | 2015-06-23 | 2021-04-12 | System for autonomous operation of multiple hybrid unmanned aerial vehicles supported by recharging station to perform services |
Family Applications Before (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US15/738,236 Active 2036-07-25 US10372142B2 (en) | 2015-06-23 | 2016-06-23 | System for autonomous operation of multiple hybrid unmanned aerial vehicles supported by recharging stations to perform services |
Family Applications After (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US17/227,663 Active US11644851B2 (en) | 2015-06-23 | 2021-04-12 | System for autonomous operation of multiple hybrid unmanned aerial vehicles supported by recharging station to perform services |
Country Status (4)
Country | Link |
---|---|
US (3) | US10372142B2 (en) |
EP (1) | EP3314344B1 (en) |
CN (1) | CN107924190A (en) |
WO (1) | WO2016210156A1 (en) |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11400823B1 (en) * | 2018-04-11 | 2022-08-02 | Amazon Technolgies, Inc. | Dynamic scheduling of tasks and charging of an autonomous mobile device |
WO2022213125A1 (en) * | 2021-04-02 | 2022-10-06 | Asylon, Inc. | Integration between unmanned aerial system and unmanned ground robotic vehicle |
Families Citing this family (70)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US9629220B2 (en) * | 2013-08-05 | 2017-04-18 | Peter Panopoulos | Sensor-based controllable LED lighting system with repositionable components and method |
US10517234B2 (en) * | 2014-08-27 | 2019-12-31 | Ponsse Oyj | Method, arrangement and user interface for presenting data describing forest work unit operation |
CN107924190A (en) * | 2015-06-23 | 2018-04-17 | 阿肯技术公司 | For being supported by recharging station with the system of the autonomous operation of multiple mixing unmanned vehicles of execution service |
US9758246B1 (en) * | 2016-01-06 | 2017-09-12 | Gopro, Inc. | Systems and methods for adjusting flight control of an unmanned aerial vehicle |
US10303186B2 (en) * | 2016-03-29 | 2019-05-28 | Chengwen Chris WANG | Unmanned spatial vehicle performance |
US11226619B2 (en) | 2016-04-24 | 2022-01-18 | Flytrex Aviation Ltd. | Dynamically arming a safety mechanism on a delivery drone |
US12007764B2 (en) | 2016-04-24 | 2024-06-11 | Flytrex Aviation Ltd. | System and method for aerial traffic management of unmanned aerial vehicles |
US10191485B2 (en) | 2016-04-24 | 2019-01-29 | Flytrex Aviation Ltd. | Apparatus and method for centralized control of vehicles |
US11762384B2 (en) | 2016-04-24 | 2023-09-19 | Flytrex Aviation Ltd. | System and method for dynamically arming a failsafe on a delivery drone |
US11423493B2 (en) * | 2016-12-09 | 2022-08-23 | Schlumberger Technology Corporation | Combinatorial solver and logic solver planner |
CN108513560B (en) * | 2016-12-23 | 2019-07-05 | 瑞典爱立信有限公司 | Unmanned vehicle in control zone |
US10288439B2 (en) * | 2017-02-22 | 2019-05-14 | Robert D. Pedersen | Systems and methods using artificial intelligence for routing electric vehicles |
US10329017B2 (en) * | 2017-03-13 | 2019-06-25 | General Electric Company | System and method for integrating flight path and site operating data |
US9836049B1 (en) * | 2017-05-05 | 2017-12-05 | Pinnacle Vista, LLC | Relay drone system |
US10317905B2 (en) * | 2017-08-10 | 2019-06-11 | RavenOPS, Inc. | Autonomous robotic technologies for industrial inspection |
US10235642B2 (en) * | 2017-08-11 | 2019-03-19 | Tata Consultancy Services Limited | Method and system for optimally allocating warehouse procurement tasks to distributed robotic agents |
JP6561270B2 (en) * | 2017-08-30 | 2019-08-21 | 株式会社プロドローン | Unmanned mobile body and unmanned mobile system using the same |
US10761544B2 (en) | 2017-10-13 | 2020-09-01 | Deere & Company | Unmanned aerial vehicle (UAV)-assisted worksite operations |
US11308735B2 (en) | 2017-10-13 | 2022-04-19 | Deere & Company | Unmanned aerial vehicle (UAV)-assisted worksite data acquisition |
US10580311B2 (en) | 2017-10-26 | 2020-03-03 | Wing Aviation Llc | UAV group charging based on demand for UAV service |
US10496095B1 (en) * | 2017-11-07 | 2019-12-03 | United States Of America As Represented By The Secretary Of The Navy | Autonomous agent scheduling |
EP3508869B1 (en) * | 2017-12-21 | 2023-10-18 | The Boeing Company | Light-weight radar system |
US10974911B2 (en) * | 2017-12-22 | 2021-04-13 | Wing Aviation Llc | Replenishment station for aerial vehicle with robotic device and conveyor |
US11257184B1 (en) | 2018-02-21 | 2022-02-22 | Northrop Grumman Systems Corporation | Image scaler |
IT201800003849A1 (en) * | 2018-03-21 | 2019-09-21 | Vhive Tech Ltd | System and method for managing unmanned aerial systems (UAS) that perform an adaptive mission |
US11157003B1 (en) * | 2018-04-05 | 2021-10-26 | Northrop Grumman Systems Corporation | Software framework for autonomous system |
DE102018207043A1 (en) * | 2018-05-07 | 2019-11-07 | Audi Ag | Method and coordination device for coordinating charging processes of several motor vehicles to optimize the energy availability and the electricity costs |
CN110737212B (en) * | 2018-07-18 | 2021-01-01 | 华为技术有限公司 | Unmanned aerial vehicle control system and method |
US10845815B2 (en) * | 2018-07-27 | 2020-11-24 | GM Global Technology Operations LLC | Systems, methods and controllers for an autonomous vehicle that implement autonomous driver agents and driving policy learners for generating and improving policies based on collective driving experiences of the autonomous driver agents |
EP3608633B1 (en) * | 2018-08-07 | 2022-05-11 | General Electric Company | System and method for guiding a vehicle along a travel path |
CN109191004B (en) * | 2018-09-25 | 2022-04-01 | 中国人民解放军空军工程大学 | Multi-unmanned aerial vehicle surveying and mapping task allocation method and device |
US11392284B1 (en) | 2018-11-01 | 2022-07-19 | Northrop Grumman Systems Corporation | System and method for implementing a dynamically stylable open graphics library |
CN109774500B (en) * | 2019-01-04 | 2022-01-25 | 三峡大学 | Laser unmanned aerial vehicle orderly charging system |
EP3696744A1 (en) * | 2019-02-13 | 2020-08-19 | Robert Bosch GmbH | Safeguarding resources of physical entites in a shared environment |
KR102293287B1 (en) * | 2019-03-13 | 2021-08-24 | 중앙대학교 산학협력단 | The Method of Charge Scheduling for Multi-Drone Networks and the device thereof |
US11064337B2 (en) | 2019-03-28 | 2021-07-13 | At&T Intellectual Property I, L.P. | Beam as a service for 5G or other next generation network |
US10880836B2 (en) * | 2019-03-28 | 2020-12-29 | At&T Intellectual Property I, L.P. | Beam provisioning for sensory data collection for 5G or other next generation networks |
US11297554B2 (en) | 2019-03-28 | 2022-04-05 | At&T Intellectual Property I, L.P. | Candidate beam selection and control for 5G or other next generation network |
CN110334838B (en) * | 2019-04-11 | 2023-06-23 | 国网新疆电力有限公司营销服务中心(资金集约中心、计量中心) | AGV trolley cooperative scheduling method and system based on ant colony algorithm and genetic algorithm |
CN110058613B (en) * | 2019-05-13 | 2022-05-13 | 大连海事大学 | Multi-unmanned-aerial-vehicle multi-ant-colony collaborative target searching method |
US11410558B2 (en) | 2019-05-21 | 2022-08-09 | International Business Machines Corporation | Traffic control with reinforcement learning |
CN110209150B (en) * | 2019-07-09 | 2022-02-18 | 新疆大学 | Job shop scheduling scheme robustness measuring method based on multi-process fault influence |
US20230131370A1 (en) * | 2019-07-19 | 2023-04-27 | Sports Data Labs, Inc. | An unmanned aerial vehicle (uav)-based system for collecting and distributing animal data for monitoring |
US11026095B2 (en) | 2019-07-31 | 2021-06-01 | At&T Intellectual Property I, L.P. | Real-time network provisioning for distributed virtual zones of collaborative mobile devices for 5G or other next generation network |
EP4014217A1 (en) * | 2019-09-02 | 2022-06-22 | Skygrid, LLC | Utilizing visualization for managing an unmanned aerial vehicle |
WO2021046015A1 (en) | 2019-09-02 | 2021-03-11 | Skygrid, Llc | Flight path deconfliction among unmanned aerial vehicles |
US20210080954A1 (en) * | 2019-09-17 | 2021-03-18 | Travis Kunkel | Method for unmanned vehicle swapping |
US11869363B1 (en) * | 2019-09-17 | 2024-01-09 | Travis Kunkel | System and method for autonomous vehicle and method for swapping autonomous vehicle during operation |
US11705009B2 (en) | 2019-12-05 | 2023-07-18 | Rockwell Collins, Inc. | System and method for optimizing mission fulfillment by unmanned aircraft systems (UAS) via dynamic atmospheric modeling |
US11334725B2 (en) * | 2020-01-06 | 2022-05-17 | International Business Machines Corporation | Sensor data collection control based on natural language interaction |
US20210279570A1 (en) * | 2020-03-03 | 2021-09-09 | Hrl Laboratories, Llc | Method for proving or identifying counter-examples in neural network systems that process point cloud data |
CN111401745B (en) * | 2020-03-16 | 2024-02-09 | 合肥科大智能机器人技术有限公司 | 5G-based routing inspection robot cluster task allocation method and system |
US11440193B2 (en) * | 2020-03-30 | 2022-09-13 | Wipro Limited | Method, device, and system for managing collaboration amongst robots |
US11089118B1 (en) | 2020-06-19 | 2021-08-10 | Northrop Grumman Systems Corporation | Interlock for mesh network |
CN111880558B (en) * | 2020-07-06 | 2021-05-11 | 广东技术师范大学 | Plant protection unmanned aerial vehicle obstacle avoidance spraying method and device, computer equipment and storage medium |
US11443518B2 (en) | 2020-11-30 | 2022-09-13 | At&T Intellectual Property I, L.P. | Uncrewed aerial vehicle shared environment privacy and security |
US11726475B2 (en) | 2020-11-30 | 2023-08-15 | At&T Intellectual Property I, L.P. | Autonomous aerial vehicle airspace claiming and announcing |
US11797896B2 (en) | 2020-11-30 | 2023-10-24 | At&T Intellectual Property I, L.P. | Autonomous aerial vehicle assisted viewing location selection for event venue |
CN113218399B (en) * | 2021-05-12 | 2022-10-04 | 天津大学 | Maze navigation method and device based on multi-agent layered reinforcement learning |
US20220413500A1 (en) * | 2021-06-29 | 2022-12-29 | Ben Eazzetta | System and Method for Robotic Mission Planning & Routing |
CN113536689B (en) * | 2021-07-26 | 2023-08-18 | 南京邮电大学 | Multi-unmanned aerial vehicle task allocation execution control method based on hybrid genetic intelligent algorithm |
CN113591966B (en) * | 2021-07-27 | 2024-06-21 | 中国人民解放军国防科技大学 | Multi-unmanned system task decomposition method combining graph processing and logic description |
CN113689470B (en) * | 2021-09-02 | 2023-08-11 | 重庆大学 | Pedestrian motion trail prediction method under multi-scene fusion |
US20230144372A1 (en) * | 2021-09-17 | 2023-05-11 | Anduril Industries Inc. | Planning an operation for a set of assets |
CN113778119B (en) * | 2021-09-26 | 2022-10-21 | 江西理工大学 | Unmanned aerial vehicle control path optimization method |
CN113991785B (en) * | 2021-11-01 | 2024-04-05 | 南京理工大学 | Unmanned aerial vehicle-assisted sensor node charging method |
CN114237282B (en) * | 2021-11-29 | 2023-10-27 | 南京邮电大学 | Unmanned aerial vehicle flight path intelligent planning method for intelligent industrial park monitoring |
CN114973780A (en) * | 2022-07-27 | 2022-08-30 | 中国铁塔股份有限公司湖北省分公司 | Unmanned aerial vehicle shutdown data communication method, device, equipment and storage medium |
US20240151533A1 (en) * | 2022-11-07 | 2024-05-09 | Agency For Defense Development | Method and electronic device for determining mission of a device |
CN117270571B (en) * | 2023-10-30 | 2024-05-07 | 广东翼景信息科技有限公司 | Heterogeneous multi-unmanned aerial vehicle cluster operation oriented control method and device |
Family Cites Families (43)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US4944321A (en) | 1990-02-02 | 1990-07-31 | Moyet Ortiz Francisco | Portable vehicle garage and tent structure |
US7032861B2 (en) | 2002-01-07 | 2006-04-25 | Sanders Jr John K | Quiet vertical takeoff and landing aircraft using ducted, magnetic induction air-impeller rotors |
DE10231391A1 (en) | 2002-07-08 | 2004-02-12 | Alfred Kärcher Gmbh & Co. Kg | Tillage system |
US7149611B2 (en) | 2003-02-21 | 2006-12-12 | Lockheed Martin Corporation | Virtual sensor mast |
US7248018B2 (en) | 2004-04-20 | 2007-07-24 | Ufoz, Llc | Personal renewable-energy fueling and storage station for electric-powered vehicles |
DE102007003458A1 (en) | 2007-01-24 | 2008-07-31 | Diehl Bgt Defence Gmbh & Co. Kg | Power supply device for battery-operated small air-craft, has charging device provided for recharging rechargeable battery after implementing flight mission of small air-craft, and landing and loading platform attached to battery magazine |
WO2008147484A2 (en) | 2007-02-16 | 2008-12-04 | Donald Orval Shaw | Modular flying vehicle |
US20090314883A1 (en) | 2007-05-10 | 2009-12-24 | Arlton Paul E | Uav launch and recovery system |
US8255092B2 (en) | 2007-05-14 | 2012-08-28 | Irobot Corporation | Autonomous behaviors for a remote vehicle |
CN101382429B (en) * | 2008-10-17 | 2011-06-22 | 北京航空航天大学 | Earth-air special-shaped multi-robot searching and rescuing system |
US8205820B2 (en) | 2009-02-03 | 2012-06-26 | Honeywell International Inc. | Transforming unmanned aerial-to-ground vehicle |
KR101262968B1 (en) | 2009-09-02 | 2013-05-09 | 부산대학교 산학협력단 | Unmanned Aerial System Including Unmanned Aerial Vehicle Having Spherical Loading Portion And Unmanned Ground Vehicle Therefor |
US8511606B1 (en) | 2009-12-09 | 2013-08-20 | The Boeing Company | Unmanned aerial vehicle base station |
WO2011149544A1 (en) * | 2010-05-26 | 2011-12-01 | Aerovironment Inc. | Reconfigurable battery-operated vehicle system |
EP2681635B8 (en) | 2011-02-28 | 2022-07-20 | BAE Systems Australia Limited | Control computer for an unmanned vehicle |
US8738198B2 (en) | 2011-05-26 | 2014-05-27 | Foster-Miller, Inc. | Robot surveillance system and method |
US9555884B2 (en) | 2012-02-16 | 2017-01-31 | Borealis Technical Limited | Method for improving ground travel capability and enhancing stealth in unmanned aerial vehicles |
US8781650B2 (en) * | 2012-04-12 | 2014-07-15 | The Boeing Company | Aircraft navigation system |
US20140024999A1 (en) | 2012-07-17 | 2014-01-23 | Elwha LLC, a limited liability company of the State of Delaware | Unmanned device utilization methods and systems |
US8781727B1 (en) | 2013-01-15 | 2014-07-15 | Google Inc. | Methods and systems for performing flocking while executing a long-range fleet plan |
US8876571B2 (en) * | 2013-02-15 | 2014-11-04 | Disney Enterprises, Inc. | Aerial display system with marionettes articulated and supported by airborne devices |
US9102406B2 (en) | 2013-02-15 | 2015-08-11 | Disney Enterprises, Inc. | Controlling unmanned aerial vehicles as a flock to synchronize flight in aerial displays |
US9862503B2 (en) | 2013-09-16 | 2018-01-09 | Jose Cruz Chavez, JR. | Aircraft retrieval device |
CN103576055B (en) * | 2013-11-14 | 2016-05-11 | 浙江大学 | High voltage transmission line line walking detection system based on multi-aircraft and line walking detection method |
US20160122038A1 (en) | 2014-02-25 | 2016-05-05 | Singularity University | Optically assisted landing of autonomous unmanned aircraft |
US9671790B2 (en) * | 2014-05-20 | 2017-06-06 | Verizon Patent And Licensing Inc. | Scheduling of unmanned aerial vehicles for mission performance |
US9845165B2 (en) * | 2014-07-16 | 2017-12-19 | Airogistic, L.L.C. | Methods and apparatus for unmanned aerial vehicle landing and launch |
CN104168455B (en) * | 2014-08-08 | 2018-03-09 | 北京航天控制仪器研究所 | A kind of space base large scene camera system and method |
US9752878B2 (en) * | 2014-12-09 | 2017-09-05 | Sikorsky Aircraft Corporation | Unmanned aerial vehicle control handover planning |
CN204355274U (en) * | 2014-12-15 | 2015-05-27 | 中国航空工业经济技术研究院 | A kind of can the logistics unmanned gyroplane of ground running |
IL237130A0 (en) | 2015-02-05 | 2015-11-30 | Ran Krauss | Landing and charging system for drones |
CN104699102B (en) * | 2015-02-06 | 2017-07-18 | 东北大学 | A kind of unmanned plane and intelligent vehicle collaborative navigation and investigation monitoring system and method |
CN110027709B (en) | 2015-03-12 | 2022-10-04 | 奈庭吉尔智慧***公司 | Automatic unmanned aerial vehicle system |
WO2016154551A1 (en) | 2015-03-26 | 2016-09-29 | Matternet, Inc. | Route planning for unmanned aerial vehicles |
KR101566341B1 (en) | 2015-06-01 | 2015-11-05 | 원기연 | A multi- purpose fire fighting drone |
CN107924190A (en) * | 2015-06-23 | 2018-04-17 | 阿肯技术公司 | For being supported by recharging station with the system of the autonomous operation of multiple mixing unmanned vehicles of execution service |
CN108290633A (en) | 2015-11-10 | 2018-07-17 | 马特耐特公司 | The method and system transported using unmanned aviation carrier |
CA2929254C (en) | 2016-05-06 | 2018-12-11 | SKyX Limited | Unmanned aerial vehicle (uav) having vertical takeoff and landing (vtol) capability |
IL246358A0 (en) | 2016-06-20 | 2016-11-30 | Fox Yuval | Positioning and locking system and method for unmanned vehicles |
IL246357B (en) | 2016-06-20 | 2020-02-27 | Krauss Ran | Payload exchange facilitating connector |
US10507938B2 (en) | 2016-07-21 | 2019-12-17 | Percepto Robotics Ltd | Methods and systems of anchoring an unmanned aerial vehicle on a ground station |
US10551852B2 (en) | 2016-07-21 | 2020-02-04 | Percepto Robotics Ltd | Systems and methods for automated landing of a drone |
CA2963662A1 (en) | 2017-04-07 | 2018-10-07 | SKyX Limited | Autonomous flight vehicle capable of fixed wing flight and rotary wing flight |
-
2016
- 2016-06-23 CN CN201680047953.0A patent/CN107924190A/en active Pending
- 2016-06-23 WO PCT/US2016/039055 patent/WO2016210156A1/en unknown
- 2016-06-23 EP EP16815322.9A patent/EP3314344B1/en active Active
- 2016-06-23 US US15/738,236 patent/US10372142B2/en active Active
-
2019
- 2019-08-06 US US16/532,910 patent/US10976752B2/en active Active
-
2021
- 2021-04-12 US US17/227,663 patent/US11644851B2/en active Active
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11400823B1 (en) * | 2018-04-11 | 2022-08-02 | Amazon Technolgies, Inc. | Dynamic scheduling of tasks and charging of an autonomous mobile device |
WO2022213125A1 (en) * | 2021-04-02 | 2022-10-06 | Asylon, Inc. | Integration between unmanned aerial system and unmanned ground robotic vehicle |
Also Published As
Publication number | Publication date |
---|---|
US20180188747A1 (en) | 2018-07-05 |
US20220066476A1 (en) | 2022-03-03 |
US10372142B2 (en) | 2019-08-06 |
CN107924190A (en) | 2018-04-17 |
US10976752B2 (en) | 2021-04-13 |
EP3314344B1 (en) | 2021-02-10 |
EP3314344A1 (en) | 2018-05-02 |
WO2016210156A1 (en) | 2016-12-29 |
EP3314344A4 (en) | 2019-02-13 |
US11644851B2 (en) | 2023-05-09 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US11644851B2 (en) | System for autonomous operation of multiple hybrid unmanned aerial vehicles supported by recharging station to perform services | |
US9983584B2 (en) | Method and apparatus for developing a flight path | |
Maza et al. | Classification of multi-UAV architectures | |
US20190031346A1 (en) | System and method for controlling an unmanned vehicle and releasing a payload from the same | |
US20180233054A1 (en) | Method and apparatus for controlling agent movement in an operating space | |
Waharte et al. | Coordinated search with a swarm of UAVs | |
AU2013230737B2 (en) | Autonomous vehicle and method for coordinating the paths of multiple autonomous vehicles | |
US20180211547A1 (en) | Method and system to improve safety concerning drones | |
CN108227685B (en) | Real-time control system of hierarchical remote control unmanned vehicle | |
Islam et al. | A path planning algorithm for collective monitoring using autonomous drones | |
Bailon-Ruiz et al. | Real-time wildfire monitoring with a fleet of UAVs | |
Bailon-Ruiz et al. | Planning to monitor wildfires with a fleet of UAVs | |
Chakrabarty et al. | Vehicle to Vehicle (V2V) communication for Collision avoidance for Multi-copters flying in UTM–TCL4 | |
Hussein et al. | Key enabling technologies for drones | |
Millan-Romera et al. | A UTM simulator based on ROS and Gazebo | |
Ding et al. | Initial designs for an automatic forced landing system for safer inclusion of small unmanned air vehicles into the national airspace | |
Hussein et al. | Reference architecture specification for drone systems | |
Kumar et al. | Geo-fencing technique in unmanned aerial vehicles for post disaster management in the Internet of Things | |
Farooq et al. | Quantitative and qualitative assessment of indoor exploration algorithms for autonomous uavs | |
Kiam et al. | Multilateral mission planning in a time-varying vector field with dynamic constraints | |
AU2015201728B2 (en) | Autonomous vehicle and method for coordinating the paths of multiple autonomous vehicles | |
Hehtke et al. | An Autonomous Mission Management System to Assist Decision Making of a HALE Operator | |
Vannini et al. | Harpia: A hybrid system for agricultural UAV missions | |
Gomes et al. | An interconnected network of UAS as a system-of-systems | |
Kaugerand et al. | A system of systems solution for perimeter control: Combining unmanned aerial system with unattended ground sensor network |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
FEPP | Fee payment procedure |
Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: SMALL ENTITY |
|
FEPP | Fee payment procedure |
Free format text: ENTITY STATUS SET TO SMALL (ORIGINAL EVENT CODE: SMAL); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY Free format text: ENTITY STATUS SET TO SMALL (ORIGINAL EVENT CODE: SMAL); ENTITY STATUS OF PATENT OWNER: SMALL ENTITY |
|
AS | Assignment |
Owner name: ARCHON TECHNOLOGIES S.R.L., ITALY Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:VENTURELLI, DAVIDE;REEL/FRAME:052356/0178 Effective date: 20151215 |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: PUBLICATIONS -- ISSUE FEE PAYMENT RECEIVED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: PUBLICATIONS -- ISSUE FEE PAYMENT VERIFIED |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |
|
AS | Assignment |
Owner name: KIA CORPORATION, KOREA, REPUBLIC OF Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:ARCHON TECHNOLOGIES SRL;REEL/FRAME:059902/0012 Effective date: 20220502 Owner name: HYUNDAI MOTOR COMPANY, KOREA, REPUBLIC OF Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:ARCHON TECHNOLOGIES SRL;REEL/FRAME:059902/0012 Effective date: 20220502 |
|
FEPP | Fee payment procedure |
Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |