WO2019126755A1

WO2019126755A1 - Generating and classifying training data for machine learning functions

Info

Publication number: WO2019126755A1
Application number: PCT/US2018/067298
Authority: WO
Inventors: Dickie Andrew MARTIN; Chase John GAUDET
Original assignee: Fugro N.V.
Priority date: 2017-12-21
Filing date: 2018-12-21
Publication date: 2019-06-27

Abstract

Disclosed are systems, methods and computer readable mediums for generating and applying training data to training data computer learning systems and for autonomous operation of surface and sub-surface vessels. The systems, methods and computer readable mediums can include a data source, the data source providing one or more of vehicle data, operational data, and environmental data and can further include a processor, the processor analyzing at least data from the data source and generating one or more control outputs based at least in part on the analysis.

Description

GENERATING AND CLASSIFYING TRAINING DATA FOR MACHINE

LEARNING FUNCTIONS

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] The present application claims the benefit of U.S. Provisional Application No.

62/609,119 filed December 21, 2017, which is hereby incorporated by reference in its entirety.

TECHNICAL FIELD

[0002] The present technology pertains to computer learning systems, and more specifically to systems and methods for generating and applying training data for autonomous surface/sub surface vessels.

BACKGROUND

[0003] Machine learning is capable of analyzing tremendously large data sets at a scale that continues to increase. Using various machine learning techniques and frameworks, it is possible to analyze data sets to extract patterns and correlations that may otherwise have never been noticed when subject to human analysis alone. Using carefully tailored data inputs a machine learning system can be manipulated to learn a desired operation, function, or pattern. However, this training process is complicated by the fact that the machine learning system’s inner functionality remains largely opaque to the human observer and the fact that the training data set can easily be biased or too small, both of which result in faulty or otherwise insufficient training.

[0004] Machine learning is of particular interest in the field of controls, particularly autonomous and semi-autonomous controls. While self-driving car technology is becoming increasingly prevalent, corresponding technology for maritime vessel operations has proven to be far more difficult to solve, due at least in part to a number of fundamental operational differences that prevent crossover from the automotive world to the maritime world. Maritime vessel operations are characterized by extremely slow response times to control commands (on the order of minutes as compared to seconds for automobiles) and far more complex vessel dynamics (e.g., multiple deployed payloads or instrumentation packages, combined surface and sub-surface operations, etc.) that pose significant challenges when it comes to predicting future state or other dynamic change. While machine learning may be suitable to begin solving this problem, machine learning approaches are fundamentally limited by the extreme difficulty in obtaining a large volume of accurate and evenly distributed training data. Accordingly, it would be highly desirable to provide a solution to the difficulty of obtaining or otherwise creating such training data.

BRIEF DESCRIPTION OF THE DRAWINGS

[0005] In order to describe the manner in which the above-recited and other advantages and features of the disclosure can be obtained, a more particular description of the principles briefly described above will be rendered by reference to the appended drawings. Understanding that these drawings depict only exemplary embodiments of the disclosure and are not therefore to be considered to be limiting of its scope, the principles herein are described and explained with additional specificity and detail through the use of the accompanying drawings in which:

[0006] FIG. 1 illustrates an example data collection system of the present disclosure;

[0007] FIG. 2 illustrates an example single machine learning network training process of the present disclosure;

[0008] FIG. 3 illustrates an example modular vessel command system and a modular autonomous vessel control system;

[0009] FIG. 4 illustrates an example trained and deployed machine learning based maritime control system of the present disclosure;

[0010] FIG. 5 illustrates an example trained and deployed machine learning based maritime control system of the present disclosure;

[0011] FIG. 6 illustrates an example system for training and autonomous operation of a sub surface vessel; and

[0012] FIG. 7 illustrates an example of a system for implementing certain aspects of the present technology.

DETAILED DESCRIPTION

[0013] Various elements of the disclosure are discussed in detail below. While specific implementations are discussed, it should be understood that this is done for illustration purposes only. A person skilled in the relevant art will recognize that other components and configurations may be used without parting from the spirit and scope of the disclosure. Additional features and advantages of the disclosure will be set forth in the description which follows, and in part will be obvious from the description, or can be learned by practice of the herein disclosed principles. The features and advantages of the disclosure can be realized and obtained by means of the instruments and combinations particularly pointed out in the appended claims. These and other features of the disclosure will become more fully apparent from the following description and appended claims, or can be learned by the practice of the principles set forth herein.

[0014] The field of autonomous and semi-autonomous vehicle operation has recently experienced significant growth, driven largely by various attempts to successfully create and deploy a self-driving car. Much of this work has benefited from the fact that autonomous land vehicles (e.g., cars) are almost instantly responsive to control inputs and consequently stand to benefit from the large gap in reaction times between human drivers and autonomous systems. For example, a self-driving car might detect a foreign object in the roadway 100 feet ahead and instantly begin evasive maneuvers, whereas a human driver might collide with the foreign object, for example, due to a lapse in attention. In general, autonomous land-going vehicles can be characterized by the fact that they generally experience little in the way of latency between receiving a commanded state or position and achieving the commanded state or position. For example, this latency might be on the order of single-digit seconds for almost all land vehicle operations, including steering, accelerating, braking, parking, etc.

[0015] However, such near-instantaneous response times are not generally present with ocean going vessels, such as cruise ships or cargo and oil tankers, which oftentimes have an inherent and unavoidable command latency on the order of minutes. Consequently, the techniques developed for autonomous land-going vehicles oftentimes have little to no crossover to ocean going vessels - where land-based autonomous vehicles can be primarily reactive, ocean-going autonomous vehicles must be primarily predictive and operate across a plurality of possible future states. Further still, this latency issue becomes even more challenging when vessels have deployed payloads in the water, e.g., for surveying operations, fishing operations, construction operations, etc.

[0016] With such a deployed payload, it is necessary to pay in the tethering cable in response to various vessel maneuvers in order to prevent the payload from hitting the ocean floor or otherwise deviating from its desired height above the ocean floor. In some instances, even when high speed winches are employed, there can be an extensive delay between the time pay in begins and the time at which the payload reacts as the winch must first pull the catenary out of the cable before the payload will react. For example, a survey vessel may have sub- surface sensor instrumentation deployed at depths in excess of 4km and at a layback (distance behind vessel) in excess of l2km, in which case the delay between pay in and payload reaction can exceed several minutes. Accordingly, autonomous control systems for ocean-going vessels are designed not only subject to entirely different operational constraints as compared to land vehicles, but must be designed with fundamentally different control and sensing criteria as compared to land vehicles. As such, it would be highly desirable to provide an autonomous or semi-autonomous control system for ocean-going vessels that fully accounts for the control of any deployed payloads.

[0017] Accordingly, disclosed is a computer learning system that can be integrated with sensor, data access, and/or control systems already present on a given vehicle or vessel in order to thereby generate a plurality of training data sets for training one or more machine learning algorithms or networks in the autonomous or semi-autonomous operation of a vehicle or vessel. In certain ocean-going and maritime embodiments, it is contemplated that a vehicle or vessel can include ocean-going vessels, freshwater vessels, sub-surface vessels, and various other forms of floating or displacement watercraft as would be appreciated by one of ordinary skill in the art. Although specific reference is made herein to ocean-going and maritime embodiments, it is understood that aspects of the present disclosure can be applied to additional vehicle types, environments, and control systems and techniques, such that a given vehicle can be provided with machine learning to train and implement one or more obstacle detection and avoidance systems, systems which can function in either an independent fashion or as an underlying component of an overall vehicle navigation and control system. For example, the disclosed training data generation and machine learning process can be applied to train vehicles adapted for one or more of a sub- sea environment, a sea environment, a surface environment, a sub surface environment, an airborne environment, an atmospheric environment, and an outer space environment. These varied environmental applications are possible at least in part due to the fact that the disclosed training data generation and machine learning techniques exhibit far less of an environmental dependence than seen in conventional control systems, as the machine learning is able to adapt to the unique constraints and dynamics imparted upon the vehicle by a given environment whereas conventional control systems must often be re-designed from the ground up in order to function in different environments. For example, in implementing conventional control systems, the vehicle control and response dynamics for a maritime vessel such as an oil tanker are fundamentally distinct from the vehicle control and response dynamics for a robotic vacuum cleaner, and as such, given the same or similar sensor data input, the control systems for these two different vehicle types would produce vastly differing control outputs. However, if the disclosed training data generation and machine learning system were to be implemented, as will be described herein, then both an oil tanker and a robotic vacuum cleaner could be trained to perform a process of obstacle or risk detection to thereby map their surroundings for subsequent navigation. As an additional portion of the machine learning, or as a separate process, a navigational system can learn the corresponding control and response dynamics for each vehicle type, and based on the combination of the obstacle/risk detection and the learned control and response dynamics, the same system can be conditioned or trained to implement a suitable control and navigation system for both an oil tanker and a robotic vacuum cleaner, or a variety of other locomotion devices, machines, etc.

[0018] Additionally, it is understood that the referenced machine learning algorithms or networks can take various forms and implementations as would be appreciated by one of ordinary skill in the art and are not restricted to a singular type or construction. For the sake of clarity, the instant disclosure refers primarily to convolutional neural networks (CNNs), artificial neural networks (ANNs), and long short-term memory networks (LTSMs), although again, it is contemplated that the various machine learning networks as are known in the art may be utilized without departing from the scope of the instant disclosure. For example, machine learning techniques can be broadly viewed as employing feedback techniques to minimize a cost function, such that the cost function minimization causes the machine learning to converge on a final trained state. Minimization and feedback techniques include backward propagation, which evaluates the output of a machine learning system for a given training data input, and uses this evaluation as an input to the next training cycle of the machine learning system. Additional techniques beyond backward propagation can be employed in machine learning and training applications, including evolutionary strategy and genetic algorithms, which can be viewed as likewise performing a cost function minimization in order to train the machine learning system. The training data used to train a machine learning system can be adjusted or adapted for various categories of machine learning, including supervised learning, unsupervised learning, semi- supervised learning, reinforcement learning, etc. While general reference is made to autonomous operation or autonomous vessels, this term is not to be construed as limiting and instead is intended to encompass various lesser degrees of autonomy. For example, these lesser degrees of autonomy include remote control, wherein a vessel has supplemental or secondary (but not primary) autonomous decision making capabilities, and supervised autonomy, wherein a vessel has primary autonomous decision making capabilities but is subject to monitoring and intervention by a human operator (e.g., a captain) based on his or her own human judgment.

[0019] The disclosure turns now to FIG. 1, which depicts an example architecture 100 for acquiring and generating training data for use with one or more computer or machine learning networks. In particular, architecture 100 illustrates a system and method for generating or constructing training data for the autonomous or semi-autonomous operation of ocean-going vessels such as survey ships or oil field service ships, although it is appreciated that the training data generation may be tailored for other types of vessels without departing from the scope of the instant disclosure. As illustrated, the primary training data inputs are provided by an a priori data source 102, a sensor layer/acquisition system 104, and a metadata generation and classification system 120. Although represented as singular entities in example architecture 100, it is appreciated that one or more of these three data sources 102, 104, 120 may in fact be provided by a plurality of discrete sensors, sensor packages, databases, data sources, etc. and that the singular entities 102, 104, and 120 are instead more akin to a data processing or data collection layer provided in order to condense the disparate constituent data streams into a unified stream. No matter how the primary training data inputs are broken into streams, they are ultimately stored in a training database 190, which contains temporally indexed data 192, non- temporally indexed data 194, and metadata 196. Training database 190 might provide separate logical partitions for one or more of the three datasets 192, 194, and 196, or can store the datasets on a shared partition.

[0020] In some embodiments, a system or apparatus for training data generation can be installed in one or more vessels for generating training data as will be described below. For a given vessel, in the context of architecture 100, such an installed system might include a supervisor UI 140 provided in the control center of the vessel in order to receive inputs from the vessel’s captain or a training data supervisor. One or more of metadata generation and classification system 120 and mission plan 130 can be likewise provided in situ on the vessel alongside supervisor UI 140, or can be provided remotely, such as at a headquarters, operating base, or dock operation associated with the individual or company responsible for the vessel. Similarly, a priori data source 102 can be provided in situ on the vessel or remotely, as is desired - the nature of a priori data makes it such that the data source 102 can function normally as long as its communicative links and data channels are active. In some embodiments, all of the components seen in architecture 100 might be provided as a single unit or otherwise all provided on the vessel. In such embodiments, training data database 190 might be unique to just that given vessel, i.e. contains only training data generated in the course of operating the given vessel with or without payloads. A master training data database (not shown) could then be provided, either remotely (e.g., at a headquarters, operating base, dock operation, etc.) or in the cloud, such that the training data collected in training data database 190 could be streamed to the master training data database as it is generated, or could be offloaded in bulk to the master training data database at some later time after its collection, such as when the vessel returns to port.

[0021] With the above considerations in mind, the disclosure turns now to a discussion of the training data itself. Broadly, training data for machine learning algorithms and networks can be broken down into three categories. A priori data, provided in architecture 100 by a priori data source 102, is data that is generally static and can be characterized in advance. As such, a priori data commonly exhibits limited to no temporally dependency (and can thus be stored in training data database 190 as non-temporally indexed data 194) and is oftentimes data which can be obtained without deploying a vessel into the field. For purposes of illustration, a partial listing of the a priori data pertaining to the operation of an ocean floor survey vessel that may be collected by a priori data source 102 can include:

• Vessel Data ( a priori )

o Measurements and design data

^■ Length, registered length, model number, tonnage, displacement, draft (empty and with various loadings), cargo capacity, propeller characteristics, presence of bow or stern thrusters, center of gravity etc. o Capabilities and performance data

^■ Maximum rated power output, number of engines, fuel capacity, motion (roll, pitch, heading, etc.) response rates/curves, turning radius, turning speed, thrust at the propeller, thrust at bow and/or stern thrusters, stopping speed/distance, station keeping (ability to stay still), acceleration profile, winch torque/speed curves, etc.

o Constraints and failure data

^■ True maximum power output, maximum RPM/redline of engine(s), maximum hull speed, winch stall torques, maximum winch rate (given payload), etc.

• Payload Data ( a priori )

o Design characteristics

^■ Weight, dimensions, sub-components, sensor arrays, communication systems, stored position, deployed position, battery capacity, etc.

^■ Payload type

• Rigid Vessel Deployed Payload: OTSM (Over The Side Mount), Moon Pool, TMS (Tether Management System), etc.

• Towed Body: Surface, Sub-surface, etc.

• ROV (Remote Operated Vehicle): Powered tether, fly-by-wire, etc.

• AUV (Autonomous Underwater Vehicle): Single or multiple, etc. o Capabilities and performance data ( a priori )

^■ Detection capabilities of payload devices, measurement quality windows (e.g., High quality if instrument is dOm above ocean floor, Medium quality if instrument is >10m above ocean floor, Low quality if instrument is >50m above ocean floor), pay in time, time to configure for storage on vessel, time to configure for deployment from vessel, running time, etc. o Constraints and failure data ( a priori )

^■ Maximum depth, operational temperature range, maximum operational time, failsafe procedure/safe-mode, maximum cable or tether diameter, maximum cable or tether tension, maximum weather/meteorological conditions compatible with payload deployment, maximum pay-in speed, etc.

• Environment Data ( a priori )

o Bathymetry maps, seafloor topology, etc.

o Weather and Ocean Meteorology (Met Ocean) forecasts, etc. ^■ Surface currents, surface winds, sub-surface currents, sub-surface thermals

^■ Atmospheric weather forecast (visibility, wind speed, precipitation, storm activity, temperature, humidity, etc.) , etc.

• Target Data ( a priori )

o Charts - give safe navigation corridors, define‘no-go zones’, define shoal areas, define boundaries (regulator, ownership, etc.), locate infrastructure (rigs, platforms, buoys, etc.) etc.

^■ Third parties and/or governmental hydrographic/charting offices provide navigational and bathymetry data, point cloud, raster, or TIN (triangular irregular network) format, etc.

o GIS (Geographical Information System) data - government maintained geographic and spatial information, etc.

[0022] The above listing of a priori data is provided for purposes of example and is not intended to be limiting. Although the a priori data is often spatial or quantitative, some a priori data might have a temporal component. For example, Met Ocean forecasts have a temporal component in addition to a spatial component, indicating a time at which a given weather condition is expected in a given spatial location. Additionally, some spatial a priori data may gain a temporal component as it evolves or changes. For example, while bathymetry data alone is spatial, viewed in a larger context such bathymetry data has a temporal component as it changes over time (e.g., year to year). In general, a priori data source 102 can collect, retrieve, generate or otherwise acquire any a priori data pertaining to a vessel and its operation in or around various bodies of water as is desired and geographic/environmental data from a plurality of external sources where the vessel does or may operate. Furthermore, the fact that a priori data has a weak temporal dependence does not imply that the a priori data is inherently scalar data. Indeed, many a priori data factors can be characterized as functions of numerous other variables besides time. For example, the power output of a vessel can be a function of the number of engines operable, each engine’s operating power, fuel type, ambient temperature, etc. As an additional example, many a priori capabilities and performance data, whether for the vessel or a payload, will dynamically change dependent upon the vessel load. As such, although training data database 190 categorizes and indexes the received training data from a priori data source 102 as either temporally based or non-temporally based, it is contemplated that categorizations on other variables or factors can be employed without departing from the scope of the instant disclosure. In order to assist in indexing training data database 190 on the desired variable(s), in some embodiments a priori data source 102 can perform normalization operations prior to transmitting training data to training data database 190. For example, the a priori training data could be indexed on the desired variable(s) and normalized to one or more of a table structure, an XML structure, a conventional database file type structure, or various other formats known in the art. In some embodiments, training data database 190 may itself perform this normalization operation prior to or as a part of the write operation (e.g., database 190 is accessible through and receives data through one or more servers (e.g., cloud-based, etc.) over a communication network).

[0023] The second category of input training data is real-time data, which is provided in architecture 100 by sensor layer/acquisition system 104. For example, sensor layer/ acquisition system 104 acquires the real-time data from one or more sensors (or the like) attached to the vessel in a geographic area. Real-time data is generally dynamic and is generally collected or refreshed at some relatively small periodic interval, i.e. in substantially real-time. Accordingly, real-time data commonly exhibits a strong temporal dependency (and can thus be stored in training data database 190 as temporally indexed data 192) and is oftentimes data which can only be obtained from a vessel or sensor deployed into the field or area of interest. For purposes of illustration, a partial listing of real-time data pertaining to the operation of an ocean floor survey vessel can include:

• Vessel data {in situ )

o Performance Data

^■ Position, speed, motion, heave, engine metrics, lights/whistles/signaling, drivetrain metrics (how many engines running, engine RPM(s), propeller metrics, other drivetrain sub-component performance/metrics), etc.

• Payload Data {in situ )

o Positioning data (three-dimensional)

^■ Location and depth: Relative to surface vessel(s) or other deployed payload(s), absolute, changes over time, surface estimated position vs. payload estimated position, etc.

o Operational data ^■ Payload motion, communication tether status, safe-mode/offline status, mission plan progress/status, power consumption, control signal latency, remaining battery life/operational time, control surface (wings, aileron, rudder) metrics, deployed/stowed status, difference between desired and actual altitude above sea floor, winch count, etc.

• Environment Data (in situ )

o Meteorological/weather sensor data, yields updated/new Ocean Met forecasts o Updated in situ bathymetry maps/seafloor topology from sensors on vessel (or detected anomalies between in situ measurements v a priori maps), etc.

• Target Data/Collision Avoidance {in situ )

o AIS (Automatic Identification System) data, captured via satellite, onboard AIS transceiver, or third party subscription service, etc.

^■ IMO (International Maritime Organization) ID, vessel type, vessel size, beam length, girth, draft, destination, cargo on board, navigation status, rate of turn, speed, heading, bearing, relative azimuth, etc.

o Radar readout(s), can be correlated with AIS data to tag radar blips with ship’s info, etc.

o Lidar readout(s), point cloud data from laser scanning, etc.

o Imagery, camera or video, etc.

^■ Visible light, UV, infrared, etc.

^■ Can be integrated with computer vision for correlation with other in situ data and/or dynamic target identification, etc.

o Ship communications

^■ VHF (Very High Frequency) radio transmission, other communication links: can be digitized and converted to text, etc.

[0024] The above listing of real-time data is provided for purposes of example and is not intended to be limiting - in general, sensor layer/acquisition system 104 can collect, retrieve, generate or otherwise acquire any real-time data pertaining to the ongoing operation of a vessel in or around various bodies of waters as is desired. It is noted that although the real-time data is typically highly temporally dependent, this does not imply that real-time data is solely a function of time. For example, many real-time operational characteristics of both the vessel and the payload(s) are further dependent upon loading factors and other state variables and will dynamically change in light of such. As such, although training data database 190 categorizes and indexes the received training data from sensor layer/acquisition system 104 as either temporally based or non-temporally based, it is contemplated that categorizations on other variables or factors can be employed without departing from the scope of the instant disclosure. In order to assist in indexing training data database 190 on the desired variable(s), in some embodiments sensor layer/acquisition system 104 can perform normalization operations prior to transmitting training data to training data database 190. For example, the real-time training data could be indexed on the desired variable(s) and normalized to one or more of a table structure, an XML structure, a conventional database file type structure, or various other formats known in the art. In some embodiments, training data database 190 may itself perform this normalization operation prior to or as a part of the write operation.

[0025] The third category of input training data is labeled training examples, which are provided or obtained in architecture 100 by metadata generation and classification system 120. These labeled training examples can be used to drive or guide a machine learning algorithm or network as it learns from the raw or input training data contained in training data database 190. This labeled training example data generally comprises one or more input conditions (e.g., possible input conditions are enumerated in the a priori data, the real-time data, or a combination of the two) and a desired or recorded output. More particularly, the input conditions might be given by, based upon, or extracted from one or more of the detected deviations 131, from a priori or real time data determined to be associated with the detected deviations 131, an identified cause of the detected deviations 131 (wherein the identified cause can be drawn from any combination of a priori data and real-time data), a causal chain constructed by metadata system 120 comprising various combinations of a priori data and real-time data with the chain terminating in the detected deviations 131. Meanwhile, the output conditions of the labeled training example pair might be given by, based upon, or extracted from an analysis of real-time data indicative of control inputs made to the vessel in a time window immediately surrounding the deviation, a log of direct control inputs made into the vessel, or an external or separate input specifying, after the fact, the specific control inputs and series of control inputs that were made into the vessel in response to the deviation. In these and other manners, it is possible to utilize the already stored data and metadata contained within training data database 190 to gain insight into the underlying cause and response dynamics behind deviation 131 and any subsequent corrective action.

[0026] Although it is possible to manually create labeled data by carefully constructing scenarios of input conditions and the desired output response, such an approach is, generally, cumbersome and impractical due to the vast amounts of labeled data required for training machine learning networks, requiring what would be a nearly impossible amount of human capital in order to manually generate sufficient amounts of labeled data pairs. Advantageously, it is contemplated that the metadata generation and classification system 120 can be installed on a vessel under human control or supervision such that metadata (i.e. labels for the training example pairs, or simply labeled training examples) can be generated in a substantially automated fashion with minimal or occasional human input (e.g., to the supervisor UI 140 in response to one or more prompts 121 from metadata system 120), for example, only to explain why a specific action or set of actions was undertaken. In particular, metadata system 120 receives or is able to access a stored mission plan 130 corresponding to the planned operations of a given vessel at a given time. By comparing the stored mission plan 130 with the actual real-time vessel data obtained by sensor layer/acquisition system 104, metadata system 120 can automatically identify any deviations 131 from the mission plan 130.

[0027] In some embodiments, metadata system 120 may first attempt to determine the underlying cause or reason for the deviation 131 automatically. In order to do so, metadata system 120 receives from sensor layer 104 a selection of control data 105, which here includes VHF radio and other communications data, payload changes, and signaling events, that is most likely to indicate that deviation 131 was undertaken due to, for example, duress or an emergency and did not simply arise accidentally or erroneously. For example, metadata system 120 can analyze the selection of control data 105 for factors indicating an emergency (e.g., words and phrases associated with emergency extracted from the communications data, drastic or catastrophic state changes in the payload(s), signaling events associated with duress or emergency, etc.) and can oftentimes determine, with a reasonable level of confidence, whether or not deviation 131 occurred due to an emergency, and therefore, whether or not deviation 131 can be utilized as a labeled data pair for training purposes.

[0028] In some instances this automatic analysis and root cause determination fails and human intervention and input is necessary. In response to such a scenario, wherein one or more deviations 131 remain unsolved by metadata generation and classification system 120, the metadata system can transmit a prompt 121 to a supervisor user interface (UI) 140 that is provided on a computing device of the human supervisor (e.g., captain) of the vessel or on a computing device of the vessel under human supervision. The prompt 121 can present the identified one or more deviations 130 from stored mission plan 130 and request an input or response 141 from the captain of the vessel specifying why the deviation occurred. The input or response 141 can be entered free- form and subsequently analyzed to extract its meaning. In some embodiments, the input or response 141 can be selected from a pre-determined set of options or a series of linked questions such that metadata system 120 can leverage these pre-defined categorizations to more efficiently determine why a deviation occurred and either dismiss the deviation (in case of a false alarm or accidental deviation) or generate a labeled data pair consisting of all measured input conditions in some time period prior to the deviation and the reason why the deviation occurred. In some embodiments, the prompt 121 can include a set of uniquely generated selectable options, the answers to which can be utilized by metadata system 120 to generate a refined categorization of the previously unknown or difficult to categorize deviation. In some embodiments, the deviation categorization can be performed on the basis of the severity or expected severity of the deviation, wherein this severity based categorization can be performed in the initial algorithmic categorization attempt, in the refined categorization made in light of the input/response 141 received from the supervisor UI 140, or both. From the refined categorization, metadata system 120 should be able to either identify a root cause (e.g., because the refining process eliminated all other options) or a most likely root cause (e.g., the refining process eliminated most but not all other options), both of which can be used for labeling the input, output training example pair to indicate a correlation between the identified most likely cause (the input) and the determined control inputs taken in response to the deviation (the output). These labels can be stored in metadata store 196 of training data database 190, or can be stored in a separate partition (not shown). Regardless of their storage location, the combination of a labeled (input, output) pair can be combined with its corresponding a priori and real-time training data in order to form a complete training unit (e.g., supervised, etc.) for the presently contemplated machine learning processes.

[0029] In some embodiments, the input or response 141 can be solicited or otherwise received some period of time after the deviation occurred, as a serious deviation is expected to require the full attention of the captain in order to achieve resolution - once the deviation has been handled, the captain may then be prompted to provide input or response 141 to the prompt 121.

[0030] In some embodiments, a captain may be on board a vessel in a purely supervisory role for a semi-autonomous vessel guidance system. In such scenarios, the captain may monitor all operational conditions, traffic, etc. as he would normally, but with the sole purpose of identifying errors or undesirable actions output by the semi-autonomous vessel guidance system. If such an action is identified, then the captain can trigger a manual override and take control of the vessel until the issue has been satisfactorily resolved. These supervisor overrides 143 can optionally be collected by metadata generation and classification system 120, as such overrides can be immensely valuable in performing additional training on the machine learning network that made the error. If supervisor overrides 143 are received, metadata system 120 may transmit them to training data database 190 where the overrides 143 can be stored along with the existing metadata 196.

[0031] The above description of labeled training data is provided under the assumption that the machine learning algorithm or network is being trained in a supervised or semi- supervised manner, both of which have labeled pairs (e.g., input conditions, desired/correct output) in order to apply either a positive reinforcement (e.g., machine learning obtained desired/correct output) or a negative reinforcement (e.g., machine learning failed to obtain desired/correct output) during the back propagation step of the training process.

[0032] If the machine learning algorithm or network is unsupervised, then labeled training data is generally not required, as the machine learning builds its own pattern recognitions and correlations across the input training data set. In such scenarios, metadata generation and classification system 120 may be omitted, or the input/responses 141 received from the captain or human supervisor can be split and stored as temporally indexed training data 192 or non- temporally indexed training data 194 for use as additional training data for the unsupervised machine learning algorithm or network.

[0033] In general, the combination of a priori data source 102, sensor layer/acquisition system 104, and metadata generation system 120, it is contemplated that any data involved in the operation, navigation, or control of a vessel and its payload(s) can be collected across any number of disparate sources, classified in training data database 190 on the basis of one or more desired classification variables (temporal or non-temporal in the example above), and made available for use in training one or more machine learning algorithms or networks - while training data database 190 does not itself build correlations or causal links between different training data inputs or variables, database 190 is operable to provide a comprehensive collection of all data that might be needed for a machine learning network to build these correlations and causal links.

[0034] An example training process for a single machine learning network 201 is shown in diagram 200 of FIG. 2. A training data database 290 is provided, and as illustrated is divided into temporally indexed data 292, non-temporally indexed data 294, and metadata (or labeled data) 296. Training data database 290 is generally similar to training data database 190 of FIG. 1, although training data database 290 is contemplated to contain training data assembled across a much larger and more diverse selection of vessels, conditions, missions, etc. than the single vessel, single training data capture session that was discussed as an example with respect to FIG. 1. In other words, training data database 290 has collected sufficient training data and labeled data that it provides a massive, but balanced (i.e. is not biased towards certain conditions) training data set.

[0035] Training data database 290 can be utilized to train anywhere between one machine learning network to thousands of machine learning networks. For the sake of simplicity, only a single machine learning network 201 is depicted, although the additional branching arrows and ellipses are indicative of the fact that additional machine learning networks may also train from training data database 290, subsequent to or simultaneous with the training of machine learning network 201.

[0036] The example of FIG. 2 makes use of a convolutional neural network (CNN) 230, which is notable for its ability to learn and construct spatial correlations between various portions of received input data. However, in order to take advantage of this spatial correlation property, the training data stored within training data database 290 (which is stored as either temporal data 292 or non-temporal data 294) must be converted into a spatial format that is readable by CNN 230.

[0037] However, even before this conversion takes place, a determination is made as to which of the variables stored as temporal data 292 and non-temporal data 294 are to be classified as spatial data and which are to be classified as non-spatial data. As an aside, it is noted that the spatial non- spatial categorization could be performed when the training data is saved into the training data database (i.e., categorize the training data on spatial dependence in lieu of categorizing on temporality). It is noted that the raw training data is received in a format that is already either naturally temporal (e.g., real-time data, timestamped by a sensor or system that received the data) or naturally non-temporal (e.g., a priori data, a static quantity or property). Such a natural or clear delineation does not exist for the spatial v. non-spatial characterization. For example, the altitude of a deployed sensor payload off of the seafloor could easily be represented as a spatial variable (plot altitude as a third dimension at x-y coordinates where an altitude was recorded) or represented as a non-spatial variable (store a scalar value of the current altitude, and optionally, store the altitude at t-l, the altitude at t-2, etc.).

[0038] In other words, a complication arises because the determination of whether a variable is best represented in a spatial form or a non-spatial form for purposes of input to a machine learning network is largely opaque to the human mind. Thus, if a spatial v. non-spatial classification was to be performed at the moment of entry of the training data into the training data database 290, the classification would become fixed and the entire training data set would adopt this (e.g., likely arbitrary) spatial classification scheme. Advantageously, the disclosed approach of storing training data in training data database 290 based on temporal and non temporal indices avoids this problem, as different assignations of spatial and non-spatial representations of the training data variables can be made as desired. Accordingly, in some embodiments it is contemplated that a plurality of machine learning networks can be trained off of training data database 290 each receive a different assignation of variables to be represented spatially and variables to be represented non- spatially. In some embodiments, the spatial variable assignations can be random. In some embodiments, the spatial variable assignations can be constrained by various variables that are not suitable for representation in one of either a spatial form or a non-spatial form. For example, variables such as the number of engines running do not lend themselves to a spatial representation, while variables such as bathymetry maps do not lend themselves to a non-spatial representation. These constraints can be determined manually and input into the system as an existing condition, or could be previously learned constraints from earlier iterations of the machine learning network training cycle described below.

[0039] Returning now to FIG. 2, machine learning network 201 receives a first spatial variable assignation 202 (e.g., spatial vars., non-spatial vars.), which will be applied to each training data set that is input into machine learning network 201. Additional machine learning networks undergoing training (not shown) receive spatial variable assignations 204 (e.g., spatial vars. B, non-spatial vars. B) and 206 (e.g., spatial vars. C, non-spatial vars. C), thereby providing a greater probability that one or more of the machine learning networks being trained will generate outputs that are better than any of those that would be generated if only a single, fixed spatial variable assignation was used across all of the machine learning networks being trained.

[0040] As input, machine learning network 201 first receives its spatial variable assignation 202 and may additionally receive at least a first training data set selected from the temporally indexed training data 292 and the non- temporally indexed training data 294. In general, each training data set will be normalized at normalization system 210 in view of the spatial variable assignation 202. Those variables that are assigned to a spatial representation are converted into an appropriate spatial format. In some examples, this will be a raster (also known as a‘grid’ or‘bin’ representation), represented herein by spatial tensors 212. In some embodiments, a triangular irregular network (TIN) might be employed in order to provide a spatial representation of the desired variables.

[0041] In effect, one or more coordinate gridding systems with a desired number of spatial dimensions can be utilized for the spatially represented variables of a given training data set (e.g., spatial tensors 212 are represented by a two-dimensional x-y coordinate grid, while other spatial tensors or matrices might have a three-dimensional x-y-z coordinate grid), although of course each given training data set is not localized to the same portion of the coordinate gridding system, as it is the coordinate system convention or selection (e.g. choice of x-y, choice of x-y-z, Cartesian, polar, etc.) that may be shared or common. For example, a first training data set might correspond to a first vessel’s activities that were recorded in the Atlantic Ocean while a second training data set might correspond to a second vessel’s activities that were recorded in the Gulf of Mexico. Both training data sets can utilize an x-y-z coordinate grid, but the two grids are not necessarily normalized in any overlapping manner and are not necessarily correlated relative to one another or positioned in a master frame. In some embodiments, different training data sets can employ different coordinate systems, which can be converted or normalized if necessary or as desired.

[0042] For each given training data set, a new dimension is added to the existing coordinate grid for each spatially represented variable. It is noted that while the coordinate system itself may be common to the spatially represented variables, it is possible or even likely that different spatially represented variables will have a different resolution or minimum grid size. For example, a spatial variable for the altitude of a deployed sub- surface instrumentation payload might have a minimum grid size of lOm while a spatial variable for surface wind speeds might have a minimum grid size of 300m. In general, it is advantageous to use the best resolution available for any given variable, rather than normalizing all of the spatially represented variables to have the same or substantially the same resolution. Indeed, beyond just resolution differences, it is found that the variables selected for spatial representation need not be obviously correlated, or even correlated at all, as an advantage of machine learning networks and convolution neural nets (CNNs) in particular is their ability to identify and leverage correlations that would otherwise not be identified by the human brain.

[0043] Accordingly, as depicted in FIG. 2, for each given training data set, normalizer 210 generates normalized spatial tensors 212, which are subsequently fed into CNN 230 for training. Although a convolutional neural network is depicted, it is also contemplated that the functionality of CNN 230 can be provided by a convolutional Long Short-Term Memory (LSTM) network (or any other suitable neural network, learning algorithm, etc.) without departing from the scope of the instant disclosure. As illustrated, CNN 230 is configured with a weight matrix 232 (e.g., initialized with random values) that is suitably sized for matrix multiplication with the normalized spatial tensors 212. In each iteration of the CNN training process, normalized spatial tensors 212 are multiplied with the weight matrix 232 to yield a spatial CNN tensor output. In a supervised or semi- supervised training environment, the ultimate goal is to improve the accuracy of the weight matrix 232 by evaluating this spatial CNN tensor output against an actual or expected control action that is contained in labeled data corresponding to the given training data set. However, several more steps are often required before the output of CNN 230 is ready to be evaluated against the corresponding labeled data.

[0044] In some embodiments, normalizer 210 may normalize temporal data (or other non-spatial data) to a common time frame or temporal reference frame, as many machine learning techniques do not support the use of asynchronous data. For example, a convolutional neural network such as CNN 230 requires different grids/layers of temporal data to be correlated in time before they may be analyzed or manipulated, such that the convolutional neural network receives a series of rasters that are correlated to a common epoch. Generally, it can often be the case that temporal data is generated in an asynchronous fashion, e.g. temporal data can be generated at differing frequencies, timestamped based on an inaccurate clock, not timestamped at all, etc. For example, gyroscope data might be measured at 60 Hz, while vehicle position data might be generated or received at 40 Hz. Such a frequency offset can lead to timestamps where only one type of data is received, and not both. For example, at a time t = Os, both gyroscope and position data are received, but at a time t = ls, only gyroscope data is received and at a time t = 1.5s, only position data is received. In order to process temporal data in a machine learning network or system, the temporal data should be in a common epoch or time frame.

[0045] In some embodiments, an interpolation can be performed on training data retrieved from training data database 290 in order to align each individual retrieved sensor data to a common epoch or time frame suitable for use in a training update step of a machine learning process. For example, each individually retrieved sensor data is analyzed against a desired time. If a given sensor data was collected at the desired time, then this collected value can be used for the training data set. However, if a given sensor data was not collected at the desired time, then an interpolated value corresponding to the desired time must be calculated. This interpolation can be performed by normalizer 210 or by a dedicated interpolation system (not shown).

[0046] For example, given a desired time and a sensor data that was not collected at the desired time, two data points might be retrieved - the data point collected most immediately prior to the desired time and the data point collected most immediately subsequent to the desired time. These two data points can be thought of as bracketing the desired time. Various interpolation algorithms are then applied to generate an interpolated value or an interpolated function between the two bracketing data points. In some embodiments, the interpolation algorithms might analyze a greater portion of the sensor data in order to produce the interpolation between the two bracketing data points. In some embodiments, an interpolation function or process might be applied to training data before or as it is being stored in training data database 290, such that training data database 290 stores a plurality of interpolation function that receive as input a desired time and output an interpolated sensor data corresponding to the input time. Thus, from the interpolation, a data value at the desired time can be determined for any given sensor, even if the corresponding sensor data was not actually collected or retrieved at the desired time.

[0047] Accordingly, it is contemplated that interpolation and/or extrapolation can be performed in order to generate one or more intervening values for a data set (e.g., interpolate 30 Hz data into 60 Hz data). Additionally, asynchronous data can be converted into a synchronous form by downsampling higher frequency data into a common, lower frequency (e.g., downsample 60 Hz to 30 Hz). In some embodiments, some combination of interpolation and downsampling can be employed to convert asynchronous data into synchronous data with a common time step or a common collection frequency. The common collection frequency can be pre-defined, for example, to be a minimum collection frequency of the data set (to avoid interpolation and instead only downsample data), to be a maximum collection frequency of the data set (to avoid downsampling data and instead only interpolate lower frequency data), or to be the collection frequency that will minimize the number of downsampling and/or interpolation operations that must be performed.

[0048] Having traced the flow of training data assigned to a spatial representation, the disclosure turns now to the flow of training data assigned to a non-spatial representation. Based on spatial variable assignation 202, the variables that are assigned to take a non-spatial representation are normalized into a plurality of non-spatial tensors 214 that are compatible for input into a sensor fusion network 220, which in some embodiments can be provided by a neural network or an embedding layer. Sensor fusion network 220 takes as input the plurality of non-spatial tensors 214 and multiplies them with its stored or learned weight matrix 222 to yield as output a fused tensor that is in a machine learning readable/compatible format. Advantageously, the fused tensor output combines and weights the different non-spatial variables contained within the tensors 214, which may otherwise be un-related or not immediately obvious as combinable, into a single vector.

[0049] The output of CNN 230 and the output of sensor fusion network 220 are coupled into a 4D state estimator 240, which recombines and analyzes the spatially derived tensor output of CNN 230 and the non-spatially derived fused tensor output of sensor fusion network 220 to generate a 4D Risk Map (t) 242. As illustrated, 4D state estimator 240 can additionally receive as input a 4D Risk Map (t-1) 243, which is generated in the previous time step (t-1) by 4D state estimator 240. Advantageously, 4D Risk Map 242 can be human readable, whereas the two tensor inputs received at 4D state estimator 240 are typically not human readable. In some embodiments, 4D state estimator 240 is provided as another neural network, although it is appreciated that various other machine learning algorithms and networks may be utilized to without departing from the scope of the instant disclosure.

[0050] The 4D Risk Map 242 provides a representation of a 3D environment (e.g., the environment derived from or contained within the current training data set) that has been overlaid with future risk potential as a fourth dimension, wherein risk can be defined and/or quantified according to various different standards. In some embodiments, the risk calculation can be driven by the following goal hierarchy: 1) Safety of Life 2) Safety of Environment 3) Safety of Equipment 4) Continuation of Vessel Operations/Mission. Based on these or other criterion, risk or future risk potential can be quantified by 4D state estimator 240 and overlaid on the 3D environment in which the various future risks may present themselves.

[0051] In some examples, sufficient training is required before this desired functionality can be achieved. As illustrated, a back propagation process 282 can be provided by a first training module 280, which in some embodiments can utilize one or more portions of the training data and metadata 296 stored in training data database 290 in order to perform back propagation 282. Although specific reference is made herein to back propagation, it is appreciated that other processes and cost minimization functions can be employed in lieu of back propagation 282, such as, but not limited to the genetic or evolutionary algorithms described previously. In general, back propagation process 282 and first training module 280 can be thought of as governing or controlling the training of the machine learning system with respect to the 4D Risk Map 242 and its generation.

[0052] As illustrated, first training module 280 can receive as input the 4D Risk Map 242 and training data obtained from the metadata store 296 of training data database 290. From these inputs, first training module 280 generates back propagation feedback for injection into one or more of sensor fusion network 220 (e.g., to update weight matrix 222), CNN 230 (e.g., to update weight matrix 232), and 4D state estimator 240 (e.g., to update weigh matrix 241). In some embodiments, this back propagation process 282 can be performed prior to the action prediction of action prediction system 250, which also receives as input the 4D Risk Map 242. In some embodiments, back propagation process 282 can be performed simultaneously or asynchronously with the action prediction of action prediction system 250, as long as the injection and weight matrix updating process performed in view of the additional back propagation is completed before the ingestion of a next training data set, e.g. the t+1 training data set.

[0053] In some embodiments, back propagation 282 might utilize a separate training set in order to converge to the weight matrix 222 for the sensor fusion network 220, or weight matrix 222 might otherwise be stored in advance. In instances where the weight matrix 222 is not stored in advance, then it can be jointly trained with weight matrix 232 of CNN 230 and weight matrix

where the training data is obtained from, the weight matrices 222, 232, 241 comprise learned values that impart certain weights onto the various constituent factors or variables of the corresponding inputs received to sensor fusion network 220, CNN 230, and 4D state estimator 240. For example, weight matrix 222 is a learned weighting matrix that imparts a certain weight onto the various non-spatial tensors 214, e.g., the weight matrix 222 may have been trained such that vessel heading is weighted to be twice as important or influential as vessel heave. In other examples, the weighting can take into consideration or otherwise reflect correlations between the different variables and how they affect one another.

[0054] As mentioned previously, the 4D Risk Map//) 242 is input directly into an action prediction system 250 (which itself may also be provided by a machine learning algorithm, function, or network). The action prediction system 250 is operative to assess 4D Risk Map//)

242 in order to determine a best course of action, for the same time step t, to either avoid an imminent risk or reduce the probability or prevalence of future risks. This determined best course of action (or courses of action) can be generated as a human-readable vector 252 of actionable changes for the current time step, labeled here as‘Actions//)’. Some examples of actionable changes or control variables include the vessel’s rudder angle, the vessel’s throttle position (e.g. from 0% to 100%), signal to blow the whistle, etc.

[0055] The action prediction system 250 may be governed, at least in part, by the same four goals described above with respect to the risk calculation. In addition to the input of 4D Risk Map//) 242, action prediction system 250 receives an actionable changes vector Actions//- i ) 253, representing the actionable changes generated by action prediction system 250 in the previous time step (t-1). Action prediction system 250 further receives as input ColRegs (International Regulations for Preventing Collisions at Sea) 254 in either a standard computer-readable form or in a more specifically tailored machine learning network format. Although not illustrated, action prediction system 250 may additionally receive or access a stored copy of the current vessel mission plan, which can be utilized with respect to the final goal of the goal hierarchy - 4) Continuation of Vessel Operations/Mission. Given these inputs, action prediction system 250 is operable to generate the human-readable vector of actionable changes for the current time step, Actions//). [0056] Ultimately, once machine learning network 201 is trained and deployed on a vessel in either an autonomous or semi-autonomous control configuration, actionable changes vector Actions//,) 252 will be utilized to either control the vessel (in autonomous or semi-autonomous mode) or advise a human captain (semi- autonomous mode). However, while machine learning network 201 remains in training, the actionable changes vector Actions//) 252 is instead received at a second training module 260, which evaluates the action or actions contained in Actions//) 252 against the actual or expected action specified by the corresponding labeled training data for the training data set currently being processed. This corresponding labeled training data is retrieved from the metadata store 296 of training data database 290. The evaluation performed by second training module 260 can vary depending upon the type of training being utilized. Supervised training will provide the correct action for every training data set, semi- supervised training will provide the correct action for some but not all training data sets, and reinforced learning will provide a positive or negative reinforcement but no explicit indication of whether or not Actions//) was correct.

[0057] For example, in the case of supervised training, if the actionable changes vector Actions//) 252 matches the action specified by the labeled training data from metadata store 296, then back propagation 262 provides a positive reinforcement allowing one or more of the weight matrices 222, 232, 241, 251 to be adjusted to give incrementally more weight to the matrix values that produced the correct actionable changes vector Actions//) 252. On the other hand, if the actionable changes vector Actions//) 252 does not match the action specified by the labeled training data from metadata store 296, then back propagation 262 provides a negative reinforcement enabling one or more of the weight matrices 222, 232, 241, 251 to give incrementally less weight to the matrix values that produced the incorrect actionable changes vector Actions//) 252. In some embodiments, the negative reinforcement can specify a degree to which Actions//) 252 was incorrect and the size of the incremental weighting reduction can vary in accordance with the degree to which the actionable change vector was incorrect. If unsupervised training is employed, then second training module 260 will not provide any feedback regarding Actions//), and in some embodiments may itself not be present.

[0058] Notably, where first training module 280 pertained primarily to the generation of 4D Risk Map//) 242, second training module 260 pertains to the overall machine learning network 201, as second training module 260 provides a back propagation 262 to all four of the weight matrices 222, 232, 241, 251 present in the machine learning network. Accordingly, weight matrix 222 of sensor fusion network 220, weight matrix 232 of CNN 230, and weight matrix 241 of 4D state estimator 240 can each receive two back propagation inputs, and are thus driven to converge not only to a final weighting that produces suitable 4D risk maps, but a final weighting that also produces suitable actionable change vectors.

[0059] The above description is made with reference to a single time stamp t of training data being passed through a single machine learning network 201. In some embodiments, multiple machine learning networks can be trained in parallel, with one or more of the machine learning networks utilizing a different spatial and non-spatial variable assortment as compared to the other machine learning networks. Furthermore, the process described above for processing a single time stamp of a training data set is repeated for thousands, hundreds of thousands, or millions of different time stamps of training data, depending upon the number of training data sets that are available from training data database 290.

[0060] When machine learning network 201 analyzes multiple training data sets, no time dependence exists between different training data sets - while each training data set on its own is associated with some time period wherein the data was collected in one or more time stamps, the training data sets compared to one another are stateless, and it can in fact be beneficial for machine learning network 201 to analyze drastically different training data sets rather than training data sets that are temporally linked or collected by only a single vessel. In the training process, variety of training data sets is perhaps the most highly desired quality, which emphasizes the need for the robust and automated training data generation outlined with respect to FIG. 1. The training process can conclude when the training data is exhausted, the machine learning network 201 fails or cannot recover from predicting erratic actionable change vectors 252, or if it is determined that machine learning network 201 has been sufficiently trained and its training matrix 232 has suitably converged for deployment or live testing.

[0061] The disclosure turns now to FIG. 3, which depicts a first diagram 300a of a selection of a vessel’s primary command center system and a second diagram 300b of a machine learning autonomous operations system deployed on the vessel. The vessel’s primary command center system is split between an in situ bridge or command center 310 that is located on the vessel and a remote command center 320 that is communicatively linked (e.g., over a wireless network, cellular network, short-range wireless, satellite, etc.) with vessel command 310 but is not physically located on the vessel. In some embodiments, remote command center 320 might be located on land, such as near a port or headquarters of the company operating the vessel. In some embodiments, remote command center 320 might be located on another vessel, which may or may not be in close proximity to vessel command 310. As illustrated, vessel command 310 can be characterized by two primary operations. The first is captain/control operations 312, which involves the control, steering, and general running of the vessel. The second is survey/payload operations 314, which more broadly can be thought of as mission operations, which involve performing various operations, with or without the assistance of deployed payload devices and instrumentation, as specified by the Mission Plan for the vessel.

[0062] Turning now to the autonomous operations, diagram 300b presents the deployed machine learning autonomous operation system in a modular perspective similar to the manner in which discrete or separate computer systems might be interconnected on the vessel in order to implement the presently disclosed techniques. Broadly, the autonomous operations are split into two primary systems, an USV (unmanned surface vessel) command core 330 and a payload mechanics system 360. Note that these two primary autonomous systems parallel the two primary operations of the vessel command center 310.

[0063] As illustrated, an in-situ data collection system 332 transmits in-situ data to both USV command core 330 and to a collision avoidance system 336, wherein the collision avoidance system 336 may be an existing system of the vessel command center 310. Collision avoidance system 336 receives AIS (Automatic Identification System) data from an AIS system 338, which may be provided by an AIS transceiver, a satellite link, or other communications link, any of which may either be retrofit to the vessel or pre-existing on the vessel, whether in command center 310 or elsewhere. Collision avoidance system 336 further receives radar input from one or more radars 339, which in some embodiments may be split into at least a high-frequency radar (for tracking targets on the horizon) and a low-frequency radar (for tracking targets in the immediate vicinity). Although not illustrated, the one or more radars 339 may be supplemented with one or more Lidar units, which utilize laser scanning to further increase resolution and sensing capabilities in the close-range otherwise handled by the low-frequency radar. Finally, collision avoidance system 336 is illustrated as receiving input from an image classifier and computer vision system 337, which can be utilized to further assist in the dynamic target recognition that is performed by collision avoidance system 336. The computer vision system 337 may, in some embodiments, capture or receive raw photo or video data captured from on the vessel or in the vicinity immediately surrounding the vessel and subsequently process this raw photo or video data in order to identify and classify objects (e.g., other ships, buoys, rigs, etc.) or generate alerts corresponding to any unidentifiable objects. In order to perform this processing and classification, the raw photo or video data can be transformed or normalized from the camera space into a real-world or geographical coordinate space/system. This coordinate space can be similar to or shared with the coordinate space of one or more of the spatial data inputs that are normalized to a grid or raster format, such that the image classification data can itself be utilized as both a training data input and a machine learning input. In some embodiments, computer vision system 337 may be driven by a machine learning function. Collision avoidance system 336 receives and processes the aforementioned data and generates a collision warning profile or a listing of potential upcoming threats to USV command core 330.

[0064] USV command core 330 further receives as input a listing of a priori data, which here is illustrated as including a Mission Plan for the vessel, Meteorological Data, Bathymetry data, GIS data, and chart/no-go zone data, all of which are described previously. Using these three inputs (from in situ data source 332, collision avoidance system 336, and a priori data source 340), USV command core is configured to generate a 4D Risk Map 344, for example by utilizing the training data generated in accordance with FIG. 1 and the machine learning training process described with respect to FIG. 2. The 4D Risk Map 344 is then transmitted to a safe ops logic system 348, which in some instances is the same as the action prediction system 242 of FIG. 2. In some embodiments, safe ops logic system 348 can be supplemented with a variety of pre defined rules specific to the operation of a specific vessel, or specific to the current environment in which the vessel is operating (e.g., temporary restrictions in place, foreign port with different and unfamiliar rules and regulations, etc.). In some embodiments, it is contemplated that safe ops logic system 348 can be programmed to account for any deficiencies in the USV command core 330 and underlying machine learning that are not discovered until the vessel is already underway. In this manner, a captain or supervisor of the USV command core 330 can apply temporary updates to maintain a high level of vessel performance until the USV command core 330 can be adequately retrained or refreshed.

[0065] Supplementing the safe ops logic 348 is a ColRegs system 349, which is utilized to define the international maritime collision avoidance rules, regulations, and procedures and couples to safe ops logic 348 in order to provide a rule-based implementation of the ColRegs such that autonomous or semi-autonomous vessel actions will not violate ColRegs unless it is necessary in order to avoid impinging upon one of the four criteria of the goal hierarchy: 1) Safety of Life 2) Safety of Environment 3) Safety of Equipment 4) Continuation of Vessel Operations/Mission.

[0066] Safe ops logic 348 outputs an actionable or guidance control vector to payload control system 362, which is a sub-component of the payload mechanics system 360, which is the second primary system of the depicted autonomous operation modular architecture. This transmission between safe ops logic 348 and payload control system 362 is effected because the two control systems must work in conjunction. While vessel control provided by safe ops logic 348 takes precedence over payload control system 362 (which ties in only to goal 4) Continuation of Mission), it is nevertheless highly important that vessel operations and payload operations are not operating independently.

[0067] For example, from a vessel-centric point of view, the default response to a possible collision (e.g., detected by collision avoidance system 336 or via 4D Risk Map 344) is to stop or take quick evasive action (e.g., perform a tight turn). However, neither of these maneuvers would be acceptable in the vast majority of instances wherein a vessel has one or more payloads or instruments deployed into the water at the time when the possible collision is detected. For example, if the vessel is towing sub-surface sea floor measurement instrumentation, such instrumentation is almost always constrained to require some minimal altitude above the sea bed. As such, if the vessel was to stop immediately in order to avoid a potential collision, the towed instrumentation would almost certainly collide with the sea bed, causing undesired damage or even environmental harm.

[0068] Even if a loose communication exists between vessel command (USV command core 330) and payload mechanics system 360, this may still be insufficient. For example, if vessel command detects a possible upcoming collision event but is also aware that a towed instrument is currently deployed, a pay-in command may be immediately transmitted to begin winching in the towed instrument. However, absent a direct communicative link between USV command core 330 and payload mechanics system 360, the command core 330 will not have a real-time knowledge of where the towed instrument currently is located and whether or not it is safe to stop the vessel. The command core 330 will additionally not have any predictive knowledge or model of when the towed instrument is predicted to pass through the minimum clearance altitude above the sea bed required for the vessel to execute a full stop. As such, a great deal of uncertainty and inefficient and even dangerous operations can result due to a lack of communication and predictive modeling undertaken between safe ops logic 348 and payload control 362. As a further example, vessel operations and payload operations can interfere with one another even when there is no towed payload (which can present the most challenges with respect to combined vessel and payload control). Instead of a towed payload, a vessel may have one or more sensors hanging off the side of the vessel, with a tether length typically at least a full order of magnitude less than the length of the tow cable for a sub-surface instrument or sensor array. However, even with this short tether length, these deployed sensors interfere with the vessel’s minimum turning radius or maximum turning rate, as the cables will snap and/or the sensor will break if the vessel turns too quickly or sharply, such as might be commanded when performing an evasive collision avoidance maneuver. In this manner, all manner of deployed payloads are operable to impose dynamic constraints upon a vessel in terms of its operating capabilities and its safe operating regime. Consequently, the presently disclosed machine learning technique is operable to resolve this deficiency by enabling predictive, synchronous control of both the vessel and any of its payloads, whether they are deployed, stowed, or somewhere in between.

[0069] As depicted in diagram 300b, payload control 362 is supplemented with a payload status system 364, which can be a passive or active component that either receives or requests/determines various operations parameters and characteristics of the payload, such as those described with respect to the data parameters of FIG. 1. Payload status system 364 communicates with safe ops log 348, thereby completing the bi-directional communication channel between USV command core 330 (vessel control) and payload mechanics 360 (payload control).

[0070] FIG. 4 depicts a diagram 400 corresponding to the operation of a deployed autonomous or semi-autonomous machine learning vessel control system, for example, machine learning network 201 of FIG. 2 (once it has completed training). Because training has been completed, note that the weight matrix 422 of sensor fusion network 420, weight matrix 432 of CNN 430, weight matrix 441 of 4D state estimator 440, and weight matrix 451 of action predictor 450 are all fully trained, and thus, no training modules or back propagation processes are depicted nor required. In other words, where the weighting matrices and their constituent values were in flux and otherwise converging or being adjusted in the training process of FIG. 2, these weight matrix values are now held constant in FIG. 4, and the varying data is received from an operational database or data source 492, which provides a priori and real-time or in situ data, shown here as being indexed into temporal indices 493 and non-temporal indices 494. The a priori data can be received at some refresh interval (e.g., every 15+ minutes for weather forecast refresh which is substantially not qualified as real time data, or some data such as vessel length may never be refreshed) and the real-time or in situ data can be received at some known periodic rate (e.g., the measurement frequency of various real-time sensors, before or after any interpolation or downsampling is applied, etc.)

[0071] Although not shown, FIG. 4 can in some embodiments contain an additional in situ validation system, which receives as input real-time or in-situ data from the data source 492 and checks to see if any portion of the real-time data can be correlated with or compared against a portion of the a priori data received. For example, while a vessel performing survey operations may have loaded a priori chart, GIS, and/or bathymetry data before setting out to perform a seafloor measurement operation using a towed instrument, it can be desirable to obtain in situ measurements of the sea floor using a forward looking sensor mounted on the vessel. In this manner, the in situ data can be received and analyzed against the seafloor topography/profile contained in the a priori data such that the analysis is complete well before the towed instrument would need to be controlled in response to a discrepancy detected in the a priori data. In instances where a discrepancy is detected between existing a priori data and newly collected in situ data, the in situ validation system can be governed by a weighting or disparity policy, which stipulates how the conflicting data should be handled. In some embodiments, data might be associated with a reliability score, e.g., a priori data might be considered extremely reliable when obtained from a National Oceanic and Atmospheric Administration (NOAA) survey performed in the last 3 years whereas the in situ data might be considered moderately unreliable due to the optimal operating parameters of the in situ sensor being exceeded. In some embodiments, a conservative approach might be taken, wherein whichever data set that poses a greater threat or is the comparative worst case scenario is taken as the reference data set for the autonomous or semi-autonomous vehicle operations. No matter how this determination is to be made, normalization system 410 will either receive a priori data and real-time/m-szYw data directly from data source 492, or normalization system 410 will receive a priori data and real- timdin-situ data as they have been adjusted based on the conflict policy implemented at an optional in situ validation system.

[0072] As was the case with the training of machine learning network 201, the normalization system 410 divides the input data received from data source 492 into a spatial portion and a non- spatial portion, as stipulated by the (spatial vars., non-spatial vars.) distribution associated with the deployed CNN 430. From the spatial portion of data generated by normalization system 410, CNN 430 uses its trained weight matrix 432 to generate a CNN tensor output which is passed to 4D state estimator 440. Similarly, from the non-spatial portion of data generated by normalization system 410, sensor fusion network 420 uses its trained weight matrix 422 to generate a fused tensor output which is passed to 4D state estimator 440.

[0073] 4D state estimator 440 also works in much the same manner as was described with respect to the 4D state estimator 240 of FIG. 2. However, 4D state estimator is no longer coupled to a first training module to receive back propagation or otherwise update its machine learning function and weight matrix. Instead, no training is needed, as 4D state estimator 440 utilizes its trained weight matrix 441 to generate an output of 4D Risk Map//) 442 which displays a high degree of predictive accuracy for the time step t (e.g., assuming that the training process has been properly performed). By receiving as input 4D Risk Map//-/) 443, which is the 4D Risk Map that estimator 440 generated in the previous time step /-/, a far greater degree of predictive accuracy is obtained, as any unexpected or large fluctuations in risk calculations that appear for only a single time step are diminished and damped at least in part due to the feed forward nature of the 4D state estimator 440. As can be seen from FIG. 4, this feed forward state-driven dependency is also present in action prediction system 450, which receives as input at time step t the vector of actionable changes Actions//- i ) 453 from time step t-1 and the 4D Risk Map/ί) from time step t. These two inputs are further combined with ColRegs logic 456 in order to calculate the vector of actionable changes Actions//) 452 for the current time step t.

[0074] One or more of the 4D Risk Map//) 442 and the vector of actionable changes Actions//) 452 can be transmitted to an optional supervision system (not shown) which might be provided in semi-autonomous rather than fully autonomous deployments, or as a backup or failsafe system in fully autonomous deployments. It is noted that 4D Risk Map//) 442 and the vector of actionable changes Actions//) 452 are also stored in memory or in a buffer such that they can be fed into 4D state estimator 440 and action prediction system 450, respectively, at the next time step t+1. Returning to optional supervision system, one or more of 4D Risk Map//) 442 and the vector of actionable changes Actions//) 452 are analyzed to determine whether Actions//) 452 should be passed to a vessel and payload control system for execution or if an override command should be pushed to the vessel and payload control system instead of Actions//). In some embodiments, supervision system can be implemented in software, such as by a separate machine learning function or network, or by a non-learning algorithm or method that applies more conventional analyses. In some embodiments, supervision system can be manually implemented, or can be semi-autonomous with the expectation of manual intervention when needed, for example, a human operator (e.g., captain, machine learning supervisor) monitors the 4D Risk Map(s) and other vessel data in order to make a judgment call, based on his or her experience, as to whether or not the computed actionable changes vector Actions//) 452 is reasonable or should be overridden with a substitute command input. In some embodiments, deployed autonomous and semi-autonomous machine learning systems might include an onboard operations database which can be utilized to store a variety of operational records and data, including but not limited to, all override instances (can be used to refresh machine learning training), new raw data for generating new training data for future trainings, etc.

[0075] In some instances, the training data and trained machine learning systems, as discussed above, can be used to train a sub-surface vessel, for example, an ROV. That is, while the above disclosure generally discusses autonomous or semi-autonomous surface vessels, sub-surface vessels are also contemplated. In some examples, a sub-surface vessel can be autonomous (or semi-autonomous) and can also control the operation of a surface vessel, for example, as if the surface vessel was autonomous. An autonomous (or semi-autonomous) surface vessel can also control the operation of a sub- surface vessel.

[0076] FIG. 5 presents an alternate state diagram 500 for a machine learning operation according to aspects of the instant disclosure. Whereas previous figures presented a priori data and real time data as being processed each time step, in some embodiments it can be possible to only process the a priori data initially, and reuse the results of this calculation until an update to one or more variables of the a priori data is received. In some embodiments, the existing a priori data can be updated based on real-time observations and ensuing predictions. The a priori data processing begins in a pre-processing step seen at the far left, wherein the a priori data (either spatial or non-spatial, although labeled in the figure as only spatial data 502) are input into a convolutional network 504. This convolutional network 504 provides an initial state characterization of the a priori parameters that should remain valid until one or more of these parameters change. Additionally, as illustrated the a priori data 502 is processed using a convolutional network 504 that is different from the convolutional-LSTM 522 that is utilized at each time step to process real-time or current spatial data such as 5l2a and 5l2b. Convolutional network 504 might be trained and converged in a separate training process taking as input only the a priori training data contained in the system, and does not include the real-time data in the training set. Because a priori training data may likely be limited in comparison to real-time data, it is possible that a single convolutional network 504 is created from the available a priori training data and that this same network is then utilized to perform the pre-processing step for each discrete machine learning function or network that is deployed.

[0077] The two illustrated time steps, t=l and 1=2 are identical in their construction. At time step t=l, real-time spatial data 5l2a and real-time non-spatial data 5l4a are freshly received, and each is input into a respective machine learning function. The real-time spatial data 512 is input into a convolutional-LSTM 522 (comparable to the CNN of the previous discussion) and the real-time non-spatial data is input into a sensor fusion function (comparable to the sensor fusion network of the previous discussion). An intermediate ingestion system 526 receives the three data tensors (which are machine readable but not human readable) from the pre-processing convolutional network 504, the convolutional-LSTM 522, and sensor fusion 524. Intermediate ingestion system 526 transmits these data tensors to 4D state estimator 530, which at time step t=l receives no other inputs before generating a 4D Risk Map 532a, which is the machine learning network’s initial risk characterization of the data it has been presented. As was the case previously, the 4D Risk Map 532a (which is human readable) is input into an action prediction machine learning function 540, which still at time step t=l receives no other inputs before generating a human readable actionable changes vector 542a, depicted here as comprising various adjustments to factors such as vessel rudder angle, vessel/payload speed changes, vessel whistle actuation, vessel winch actuation, etc.

[0078] The next time step t=2 is identical to time step t=l with the primary exceptions (besides the refreshed spatial data 5l2b and refreshed non-spatial data 5l4b) being that the Convolutional-LSTM 522 now receives the additional input of the convolutional output vector from t=l, 4D state estimator 530 now receives the additional input of the 4D state estimation from t=l and action prediction system 540 now receives the additional input of the actionable changes from t=l, thereby beginning the feed-forward state processes discussed with respect to the previous figures.

[0079] In some instances, the training data and trained machine learning systems, as discussed above, can be used to train a sub-surface vessel, for example, an ROV. That is, while the above disclosure generally discusses autonomous or semi-autonomous surface vessels, sub-surface vessels are also contemplated. In some examples, a sub-surface vessel can be autonomous (or semi-autonomous) and can also control the operation of a surface vessel, for example, as if the surface vessel was autonomous. An autonomous (or semi-autonomous) surface vessel can also control the operation of a sub- surface vessel.

[0080] FIG. 6 illustrates an example system and method 600 for training and autonomous operation of a sub-surface vessel. For example, machine learning concepts, as discussed above and further illustrated below, can be applied to sub-surface vessels, in particular for structure inspections or routing around structural components. In some examples, an ROV can recognize from its video sensor(s) and sonar sensor(s) components of the structure, locate itself and proceed to perform inspections on appropriate structural elements.

[0081] At block 602, the system and method can construct one or more 3D virtual models. For example, a sub-surface vessel can have one or more 3D virtual models constructed, which render a variety of perspective views, orientations and observation points of the sub-surface vessel. In some examples, the 3D virtual models can be constructed to resemble images that would be viewed by sensors (e.g., video, acoustic, etc.) on the sub-surface vessel. In some examples, a plurality of views from one or more directions can be constructed.

[0082] At block 604 and 606, the 3D virtual models can be used to train one or more machine learning systems (e.g., neural networks, etc.), for example, as discussed above. For example, the trained systems can be trained to recognize specific components of the structure from portions of the 3D virtual models (e.g., video, images, acoustics, etc.).

[0083] Block 608 represents one or more video sensors (e.g., imaging devices, imaging cameras, thermal cameras, etc.) coupled to the sub-surface vessel. The orientation of the video sensors can be used to construct the one or more images/3D virtual models. Block 610 represents one or more sonar sensors (e.g., acoustic capture device, acoustic camera, sonar scanner, etc.). In some examples, the sound waves can be used to measure distance and location to objects and further, the images/3D virtual models can be represented in grey scale based on the acoustic reflectivity of the component. In some examples, steel structural member(s) can have a much brighter return than, for instance, a rope. These sonar sensors and accompanying processing components can produce a product that looks much like a grey scale camera image, but using acoustics rather than light.

[0084] Block 612 represents one or more trained machine learning system (614, 616). Trained machine learning system 614, 616 can be one or more neural network or learning function, as discussed above. Learning system 614 can, based on the observed images of block 608 and the training performed in block 604, classify and/or identify one or more members of the structure (or structural components) that are visible in the image(s). Learning system 616 can, based on the observed sonar sensors 610 and the training performed in block 606, classify/locate one or more members of the structure (or structural components) are visible in the image(s). In some examples, the trained machine learning systems can be configured to perform pattern recognition between the data received from sensors 608, 610 and known 3D virtual models (e.g., known 3D models previously presented to the machine learning systems).

[0085] At block 618, the orientation of the sub-surface vessel (e.g., ROV, etc.) relative to observed structures can be determined. Sub-surface vehicles are normally positioned via inertial systems and Ultra-Short Baseline (USBL) systems attached to a surface vessel. The accuracy of this configuration is limited and usually not sufficient for close up work. By utilizing the structural components identified at block 614 and the approximate position of the sub-surface vessel from its own positioning, the sub-surface vessel can refine its position relative to the observed structure (e.g., from the video sensors 608).

[0086] At block 620, the range/distance and orientation of the sub-surface vessel (e.g., ROV, etc.) relative to observed feature can be determined. In some examples, the range/distance can be determined by triangulation with, for example, by known dimensions of the observed feature. Sub-surface vehicles are normally positioned via inertial systems and USBL systems attached to a surface vessel. The accuracy of this configuration is limited and usually not sufficient for close up work. By utilizing the structural components identified at block 616 and the approximate position of the sub-surface vessel from its own positioning, the sub-surface vessel can refine its position relative to the observed structure (e.g., from the sonar sensors 610). In some examples, blocks 618 and 620 can be combined, in other instances they can be independent on one another. [0087] At block 622, range/distance and bearing information (from block 618 and 620) can be utilized to guide the sub-surface vessel to a desired structure/structural component. For example, by utilizing the refinements from block 618 and 620 along with knowledge of the 3D model of the structure, a safe path (e.g., free of objects, obstructions, etc.) from current position of the sub surface vessel to a target position can be charted.

[0088] At block 624, the surface vessel, to which the sub-surface vessel is in communication with can receive input of the safe path and follow the sub-surface vessel. In some instances, a surface vessel does not exist (e.g., UUV) and this block is ignored. In some instances, the surface vessel may be a manned vessel or it may be unmanned (USV). In some examples, the position of the sub-surface vessel (from block 622) and knowledge of the 3D virtual model can be utilized, by the surface vessel, to follow the sub-surface vessel (keeping as short as possible) and avoid collisions or avoidance areas around the structure/structural components. In some examples, the position can be determined based on triangulation with known dimensions of the 3D virtual model.

[0089] FIG. 7 shows an example of computing system 700 in which the components of the system are in communication with each other using connection 705. Connection 705 can be a physical connection via a bus, or a direct connection into processor 710, such as in a chipset architecture. Connection 705 can also be a virtual connection, networked connection, or logical connection.

[0090] In some embodiments computing system 700 is a distributed system in which the functions described in this disclosure can be distributed within a datacenter, multiple datacenters, a peer network, etc. In some embodiments, one or more of the described system components represents many such components each performing some or all of the function for which the component is described. In some embodiments, the components can be physical or virtual devices.

[0091] Example system 700 includes at least one processing unit (CPU or processor) 710 and connection 705 that couples various system components including system memory 715, such as read only memory (ROM) and random access memory (RAM) to processor 710. Computing system 700 can include a cache of high-speed memory connected directly with, in close proximity to, or integrated as part of processor 710. [0092] Processor 710 can include any general purpose processor and a hardware service or software service, such as services 732, 734, and 736 stored in storage device 730, configured to control processor 710 as well as a special-purpose processor where software instructions are incorporated into the actual processor design. Processor 710 may essentially be a completely self-contained computing system, containing multiple cores or processors, a bus, memory controller, cache, etc. A multi-core processor may be symmetric or asymmetric.

[0093] To enable user interaction, computing system 700 includes an input device 745, which can represent any number of input mechanisms, such as a microphone for speech, a touch- sensitive screen for gesture or graphical input, keyboard, mouse, motion input, speech, etc. Computing system 700 can also include output device 735, which can be one or more of a number of output mechanisms known to those of skill in the art. In some instances, multimodal systems can enable a user to provide multiple types of input/output to communicate with computing system 700. Computing system 700 can include communications interface 740, which can generally govern and manage the user input and system output. There is no restriction on operating on any particular hardware arrangement and therefore the basic features here may easily be substituted for improved hardware or firmware arrangements as they are developed.

[0094] Storage device 730 can be a non-volatile memory device and can be a hard disk or other types of computer readable media which can store data that are accessible by a computer, such as magnetic cassettes, flash memory cards, solid state memory devices, digital versatile disks, cartridges, random access memories (RAMs), read only memory (ROM), and/or some combination of these devices.

[0095] The storage device 730 can include software services, servers, services, etc., that when the code that defines such software is executed by the processor 710, it causes the system to perform a function. In some embodiments, a hardware service that performs a particular function can include the software component stored in a computer-readable medium in connection with the necessary hardware components, such as processor 710, connection 705, output device 735, etc., to carry out the function.

[0096] Methods according to the aforementioned description can be implemented using computer-executable instructions that are stored or otherwise available from computer readable media. Such instructions can comprise instructions and data which cause or otherwise configure a general purpose computer, special purpose computer, or special purpose processing device to perform a certain function or group of functions. Portions of computer resources used can be accessible over a network. The computer executable instructions may be binaries, intermediate format instructions such as assembly language, firmware, or source code. Computer-readable media that may be used to store instructions, information used, and/or information created during methods according to the aforementioned description include magnetic or optical disks, flash memory, USB devices provided with non-volatile memory, networked storage devices, and so on.

[0097] For clarity of explanation, in some instances the present technology may be presented as including individual functional blocks including functional blocks comprising devices, device components, steps or routines in a method embodied in software, or combinations of hardware and software.

[0098] The computer-readable storage devices, mediums, and memories can include a cable or wireless signal containing a bit stream and the like. However, when mentioned, non-transitory computer-readable storage media expressly exclude media such as energy, carrier signals, electromagnetic waves, and signals per se.

[0099] Devices implementing methods according to these disclosures can comprise hardware, firmware and/or software, and can take any of a variety of form factors. Such form factors can include laptops, smart phones, small form factor personal computers, personal digital assistants, rackmount devices, standalone devices, and so on. Functionality described herein also can be embodied in peripherals or add-in cards. Such functionality can also be implemented on a circuit board among different chips or different processes executing in a single device.

[0100] The instructions, media for conveying such instructions, computing resources for executing them, and other structures for supporting such computing resources are means for providing the functions described in these disclosures.

[0101] Although a variety of information was used to explain aspects within the scope of the appended claims, no limitation of the claims should be implied based on particular features or arrangements, as one of ordinary skill would be able to derive a wide variety of implementations. Further and although some subject matter may have been described in language specific to structural features and/or method steps, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to these described features or acts. Such functionality can be distributed differently or performed in components other than those identified herein. Rather, the described features and steps are disclosed as possible components of systems and methods within the scope of the appended claims. Moreover, claim language reciting "at least one of' a set indicates that one member of the set or multiple members of the set satisfy the claim.

Claims

CLAIMS We claim:

1. A system for controlling a vehicle, the system comprising:

a data source, the data source providing one or more of vehicle data, operational data, and environmental data; and

a processor, the processor analyzing at least data from the data source and generating one or more control outputs based at least in part on the analysis.

2. The system of claim 1, wherein the analyzing comprises, at least in part, machine learning or machine learning training.

3. The system of claim 1, wherein the data source further provides one or more of training data and training examples.

4. The system of claim 1, wherein the one or more control outputs are control actions corresponding to the vehicle.

5. The system of claim 1, wherein the processor analyzing at least data from the data source further comprises normalizing data from the data source.

6. The system of claim 1, wherein the vehicle is an automobile, truck, motorcycle, airplane, helicopter, vessel, boat, ship, barge, remote operate vehicle, or the like.

7. The system of claim 1, wherein the environmental data corresponds to one or more of a sub sea environment, a sea environment, a surface environment, a sub- surface environment, an airborne environment, an atmospheric environment, and an outer space environment.

8. A method for controlling a vehicle, the method comprising:

obtaining input data, the input data comprising one or more of vehicle data, operational data, and environmental data;

analyzing at least the input data; and generating one or more control outputs based at least in part on the analysis.

9. The method of claim 8, wherein the analyzing comprises, at least in part, machine learning or machine learning training.

10. The method of claim 8 , wherein the input data further comprises one or more of training data and training examples.

11. The method of claim 8, wherein the one or more control outputs are control actions corresponding to the vehicle.

12. The method of claim 8, wherein the analyzing further comprises normalizing data from the data source.

13. The method of claim 8, wherein the vehicle is an automobile, truck, motorcycle, airplane, helicopter, vessel, boat, ship, barge, remote operate vehicle, or the like.

14. The method of claim 8, wherein the environmental data corresponds to one or more of a sub sea environment, a sea environment, a surface environment, a sub- surface environment, an airborne environment, an atmospheric environment, and an outer space environment.

15. A non-transitory computer-readable storage medium having instructions stored therein which, when executed by one or more processors cause the one or more processors to:

obtain input data, the input data comprising one or more of vehicle data, operational data, and environmental data;

analyze at least the input data; and

generate one or more control outputs based at least in part on the analysis.

16. The non-transitory computer-readable storage medium of claim 15, wherein the analyzing comprises, at least in part, machine learning or machine learning training.

17. The non-transitory computer-readable storage medium of claim 15, wherein the input data further comprises one or more of training data and training examples.

18. The non-transitory computer-readable storage medium of claims 15, wherein the one or more control outputs are control actions corresponding to a vehicle associated with the vehicle data.

19. The non-transitory computer-readable storage medium of claim 15, wherein the analyzing further comprises normalizing data from the data source.

20. The non-transitory computer-readable storage medium of claim 15, wherein the one or more control outputs are configured to control a vehicle.

21. The non-transitory computer-readable storage medium of claim 15, wherein the vehicle is an automobile, truck, motorcycle, airplane, helicopter, vessel, boat, ship, barge, or the like.

22. The non-transitory computer-readable storage medium of claim 1, wherein the environmental data corresponds to one or more of a sub- sea environment, a sea environment, a surface environment, a sub- surface environment, an airborne environment, an atmospheric environment, and an outer space environment.