WO2016118672A2 - Real time machine vision and point-cloud analysis for remote sensing and vehicle control - Google Patents

Real time machine vision and point-cloud analysis for remote sensing and vehicle control Download PDF

Info

Publication number
WO2016118672A2
WO2016118672A2 PCT/US2016/014196 US2016014196W WO2016118672A2 WO 2016118672 A2 WO2016118672 A2 WO 2016118672A2 US 2016014196 W US2016014196 W US 2016014196W WO 2016118672 A2 WO2016118672 A2 WO 2016118672A2
Authority
WO
WIPO (PCT)
Prior art keywords
vehicle
data
map
information
assets
Prior art date
Application number
PCT/US2016/014196
Other languages
French (fr)
Other versions
WO2016118672A3 (en
Inventor
Fabien Chraim
Shanmukha PUTTAGUNTA
Anuj Gupta
Scott Harvey
Jason CREADORE
Graham Mills
Original Assignee
Solfice Research, Inc.
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Solfice Research, Inc. filed Critical Solfice Research, Inc.
Priority to EP16740714.7A priority Critical patent/EP3248140A4/en
Priority to JP2017556798A priority patent/JP2018508418A/en
Priority to CN201680006431.6A priority patent/CN107533630A/en
Publication of WO2016118672A2 publication Critical patent/WO2016118672A2/en
Publication of WO2016118672A3 publication Critical patent/WO2016118672A3/en

Links

Classifications

    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B61—RAILWAYS
    • B61L—GUIDING RAILWAY TRAFFIC; ENSURING THE SAFETY OF RAILWAY TRAFFIC
    • B61L23/00—Control, warning or like safety means along the route or between vehicles or trains
    • B61L23/34—Control, warning or like safety means along the route or between vehicles or trains for indicating the distance between vehicles or trains by the transmission of signals therebetween
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B61—RAILWAYS
    • B61L—GUIDING RAILWAY TRAFFIC; ENSURING THE SAFETY OF RAILWAY TRAFFIC
    • B61L23/00—Control, warning or like safety means along the route or between vehicles or trains
    • B61L23/04—Control, warning or like safety means along the route or between vehicles or trains for monitoring the mechanical state of the route
    • B61L23/041—Obstacle detection
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B61—RAILWAYS
    • B61L—GUIDING RAILWAY TRAFFIC; ENSURING THE SAFETY OF RAILWAY TRAFFIC
    • B61L25/00—Recording or indicating positions or identities of vehicles or trains or setting of track apparatus
    • B61L25/02—Indicating or recording positions or identities of vehicles or trains
    • B61L25/025—Absolute localisation, e.g. providing geodetic coordinates
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B61—RAILWAYS
    • B61L—GUIDING RAILWAY TRAFFIC; ENSURING THE SAFETY OF RAILWAY TRAFFIC
    • B61L25/00—Recording or indicating positions or identities of vehicles or trains or setting of track apparatus
    • B61L25/02—Indicating or recording positions or identities of vehicles or trains
    • B61L25/026—Relative localisation, e.g. using odometer
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B61—RAILWAYS
    • B61L—GUIDING RAILWAY TRAFFIC; ENSURING THE SAFETY OF RAILWAY TRAFFIC
    • B61L27/00—Central railway traffic control systems; Trackside control; Communication systems specially adapted therefor
    • B61L27/04—Automatic systems, e.g. controlled by train; Change-over to manual control
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B61—RAILWAYS
    • B61L—GUIDING RAILWAY TRAFFIC; ENSURING THE SAFETY OF RAILWAY TRAFFIC
    • B61L3/00—Devices along the route for controlling devices on the vehicle or train, e.g. to release brake or to operate a warning signal
    • B61L3/02—Devices along the route for controlling devices on the vehicle or train, e.g. to release brake or to operate a warning signal at selected places along the route, e.g. intermittent control simultaneous mechanical and electrical control
    • B61L3/08—Devices along the route for controlling devices on the vehicle or train, e.g. to release brake or to operate a warning signal at selected places along the route, e.g. intermittent control simultaneous mechanical and electrical control controlling electrically
    • B61L3/12—Devices along the route for controlling devices on the vehicle or train, e.g. to release brake or to operate a warning signal at selected places along the route, e.g. intermittent control simultaneous mechanical and electrical control controlling electrically using magnetic or electrostatic induction; using radio waves
    • B61L3/127—Devices along the route for controlling devices on the vehicle or train, e.g. to release brake or to operate a warning signal at selected places along the route, e.g. intermittent control simultaneous mechanical and electrical control controlling electrically using magnetic or electrostatic induction; using radio waves for remote control of locomotives
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
    • G01S17/88—Lidar systems specially adapted for specific applications
    • G01S17/93—Lidar systems specially adapted for specific applications for anti-collision purposes
    • G01S17/931—Lidar systems specially adapted for specific applications for anti-collision purposes of land vehicles
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/10—Terrestrial scenes
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/59—Context or environment of the image inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B61—RAILWAYS
    • B61L—GUIDING RAILWAY TRAFFIC; ENSURING THE SAFETY OF RAILWAY TRAFFIC
    • B61L2205/00—Communication or navigation systems for railway traffic
    • B61L2205/04—Satellite based navigation systems, e.g. global positioning system [GPS]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V2201/00—Indexing scheme relating to image or video recognition or understanding
    • G06V2201/02—Recognising information on displays, dials, clocks

Definitions

  • GPS Global Positioning System
  • supplemental sensing systems may be desirable, as well as highly detailed infrastructure and landmark maps, potentially including three-dimensional semantic maps.
  • Radio towers still require signaling equipment to be deployed in order for the radio communication to take place.
  • additional transponders have to be deployed along tracks for the train to reliably determine the position of the train and the track it is currently occupying.
  • ECS Error Control System
  • trackside equipment which relies on trackside equipment and a train-mounted control that reacts to the information related to the signaling. That system relies heavily on infrastructure that has not been deployed in the United States or in developing countries.
  • a solution that requires minimal deployment of wayside signaling equipment would be beneficial for establishing Positive Train Control throughout the United States and in the developing world.
  • Deploying millions of balises—the transponders used to detect and communicate the presence of trains and their location— every 1-15 km along tracks is less effective because balises are negatively affected by environmental conditions, theft, and require regular maintenance, and the data collected may not be used in real time.
  • Obtaining positional data through only trackside equipment is not a scalable solution considering the costs of utilizing balises throughout the entire railway network PTC.
  • train control and safety systems cannot rely solely on a global positioning system (GPS) as it not sufficiently accurate to distinguish between tracks, thereby requiring wayside signaling for position calibration.
  • GPS global positioning system
  • Local environment sensors which may include a machine vision system such as LiDAR, can be mounted on a vehicle.
  • a GPS receiver may also be included to provide a first geographical position of the vehicle.
  • a remote database and processor stores and processes data collected from multiple sources, and an on-board vehicle processor downloads data relevant for operation, safety, and/or control of the moving vehicle.
  • the local environmental sensors generate data describing a surrounding environment, such as point-cloud data generated by a LiDAR sensor. Collected data can be processed locally, on board the vehicle, or uploaded to a remote data system for storage, processing and analysis. Analysis mechanisms (on-board and/or
  • An exemplary embodiment of a system described herein includes a hardware component mounted on railroad or other vehicles, a remote database, and analysis components to process data collected regarding information about a transportation system, including moving and stationary vehicles, infrastructure, and transit pathway (e.g. rail or road) condition.
  • the system can accurately estimate the precise position of the vehicle traveling down the transit pathway, such as by comparing the location of objects detected in the vehicle's on-board sensors relative to the known location of objects. Additional attributes about the exemplary components are detailed herein and include the following:
  • the Hardware informs the movement of vehicles for safety, including: in railroad applications, identifying the track upon which they are traveling, obstructions, health of track and rail system, among other features; and in automotive applications, the lane upon which the vehicle is traveling, the texture and health of the road, the identification of assets in the vicinity, amongst other features.
  • the Remote Database contains information about assets, and which can be queried remotely to obtain additional asset information.
  • methods include machine vision data collected by the traveling vehicle itself, or by another vehicle (such as road-rail vehicles, track inspection vehicles, aerial vehicles, mobile mapping platforms, etc.). This data is then processed to generate the asset information (location, features, road/track health, among other information).
  • FIG. 1 is a representative flow diagram of a Train Control System
  • FIG. 2 is a representative flow diagram of the on board ecosystem
  • FIG. 3 is a representative flow diagram for obtaining positional information
  • FIG. 4 is an exemplary depiction of a train extrapolating the signal state
  • FIG. 5 is an exemplary depiction of the various interfaces available to the conductor as feedback
  • FIG. 6 is a representative flow diagram for obtaining the track ID occupied by the train
  • FIG. 7 is a representative flow diagram which describes the track ID algorithm
  • FIG. 8 is a representative flow diagram which describes the signal state algorithm
  • FIG. 9 is a representative flow diagram which depicts sensing and feedback.
  • FIG. 10 is a representative flow diagram of image stitching techniques for relative track positioning.
  • FIGS. 11 A and 1 IB are flow diagrams of point-cloud analysis processes.
  • FIG. 12 is a schematic block diagram of an apparatus for point-cloud analysis.
  • FIG. 13 is a flow diagram of a process for analyzing point-cloud data.
  • FIG. 14 is a further flow diagram of a process for analyzing point-cloud data.
  • FIG. 15 is a chart illustrating point cloud tile size and density distribution in an exemplary point-cloud survey.
  • FIG. 16 is a schematic block diagram of a point-cloud processing cluster.
  • FIG. 17 is a plot of characteristics for compression mechanisms usable with point- cloud data.
  • FIG. 18 is a plot of characteristics for compression mechanisms usable with point- cloud data.
  • FIG. 19 is a plot of characteristics for compression mechanisms usable with point- cloud data.
  • FIG. 20 is a flow diagram of a process for track detection.
  • FIG. 21 is a visualization of a point-cloud section with extracted rail information.
  • FIG. 22 is a histogram of point-cloud intensity levels in an exemplary point-cloud segment.
  • FIG. 23 is a visualization of track detection mechanism output.
  • FIG. 24 is a schematic block diagram of a map generation system utilizing supervised machine learning.
  • FIG. 25 is a schematic block diagram of a run-time system for automobile localization, automobile control and map auditing.
  • methods and apparatuses are provided for determining the position of one or more moving vehicles, e.g., trains or autonomous driving vehicles, without depending on balises/transponders distributed throughout the operating environment for accurate positional data.
  • embodiments are sometimes referred to herein as BVRVB-PTC, a PTC vision system, or a machine vision system.
  • railway embodiments can use a series of sensor fusion and data fusion techniques to obtain the track position with improved precision and reliability.
  • Such embodiments can also be used for auto-braking of trains for committing red light violations on the track, for optimizing fuel based on terrain, synchronizing train speeds to avoid red lights, anti-collision systems, and for preventative maintenance of not only the trains, but also the tracks, rails, and gravel substrate underlying the tracks.
  • Some embodiments may use a backend processing and storage component for keeping track of asset location and health information (accessible by the moving vehicle or by railroad operators through reports).
  • embodiments to take advantage of highly detailed infrastructure and landmark maps can be utilized to direct the flow of traffic in the real world and plan routes for vehicles to travel from source to destination.
  • the three-dimensional nature of the maps in addition to their accuracy in representing the physical world, assist the vehicles in anticipating events beyond their sensing range, foveating their sensors to the assets of interest, and localizing the vehicles in relation to the landmarks.
  • highly detailed three-dimensional (semantic) maps for the pseudo-static assets, the vehicle's resources are liberated to observe the dynamic objects around it.
  • the PTC vision system may include modules that handle communication, image capture, image processing, computational devices, data aggregation platforms that interface with the train signal bus and inertial sensors (including on-board and positional sensors).
  • FIG. 1 illustrates an exemplary flow operation of a Train Control System.
  • step SI 00 a train undergoes normal operation.
  • step SI 05 the train state is retrieved from the Data Aggregation Platform (described below).
  • step SI 10 the train position is refined.
  • step SI 15 semaphore signal states are identified from local environment sensor information.
  • step S120 feedback is applied.
  • the train speed can be adjusted (step S125), alarms and/or notifications can be raised (step S130). Further detail concerning of each of these steps is described hereinbelow.
  • a PTC vision system may include one or more of the following: Data Aggregation Platform (DAP) 215, Vision Apparatus (VA) 230, Positive Train Control Computer (PTCC) 210, Human Machine Interface (HMI) 205, GPS Receiver 225, and the Vehicular Communication Device (VCD) 220, typically communicating via LAN or WAN communications network 240.
  • DAP Data Aggregation Platform
  • VA Vision Apparatus
  • PTCC Positive Train Control Computer
  • HMI Human Machine Interface
  • GPS Receiver 225 GPS Receiver 225
  • VCD Vehicular Communication Device
  • the components e.g., VCD, HMI, PTCC, VA, DAP, GPS
  • Each component in the PTC vision system may have its own power supply or share one with the PTCC.
  • the power supplies used for the components in the PTC vision system may include non-interruptible components for power outages.
  • the PTCC module maintains the state of information passing in between the modules of the PTC vision system.
  • the PTCC communicates with the HMI, VA, VCD, GPS, and DAP. Communication may include providing information (e.g., data) and/or receiving information.
  • An interface e.g., bus, connection
  • Modules of the ecosystem may communicate with each other, a human operator, and/or a third party (e.g., another train, conductor, train operator) using any conventional communication protocol. Communication may be accomplished via wired and/or wireless communication link (e.g., channel).
  • the PTCC may be implemented using any conventional processing circuit including a microprocessor, a computer, a signal processor, memory, and/or buses.
  • a PTCC may perform any computation suitable for performing the functions of the PTC vision system.
  • the HMI module may receive information from the PTCC module.
  • Information received by the HMI module may include: Geolocation (e.g., GPS Latitude & Longitude coordinates); Time; Recommended speeds; Directional Heading (e.g., azimuth); Track ID;
  • the HMI module may provide information to the PTCC module.
  • Information provided to the PTCC may include information and/or requests from an operator.
  • the HMI may process (e.g., format, reduce, adjust, correlate) information prior to providing the information to an operator or the PTCC module.
  • the information provided by the HMI to the PTCC module may include: Conductor commands to slow down the train; Conductor requests to bypass certain parameters (e.g., speed restrictions); Conductor acknowledgement of messages (e.g., faults, state information); Conductor requests for additional information (e.g., diagnostic procedures, accidents along the railway track, or other points of interest along the railway track); and Any other information of interest relevant to a conductor's train operation.
  • the HMI provides a user interface (e.g., GUI) to a human user (e.g., conductor, operator).
  • a human user may operate controls (e.g., buttons, levers, knobs, touch screen, keyboard) of the HMI module to provide information to the HMI module or to request information from the vision system.
  • controls e.g., buttons, levers, knobs, touch screen, keyboard
  • An operator may wear the user interface to the HMI module.
  • the user interface may communicate with the HMI module via tactile operation, wired communication, and/or wireless communication.
  • Information provided to a user by the HMI module may include: Recommended speed, Present speed, Efficiency score or index, Driver profile, Wayside signaling state, Stations of interest, Map view of inertial metrics, Fault messages, Alarms, Conductor interface for actuation of locomotive controls, and Conductor interface for acknowledgement of messages or notifications.
  • the VCD module performs communication (e.g., wired, wireless).
  • the VCD module enables the PTC vision system to communicate with other devices on and off the train.
  • the VCD module may provide Wide Area Network ("WAN") and/or Local Area Network ("LAN") communications.
  • WAN communications may be performed using any conventional communication technology and/or protocol (e.g., cellular, satellite, dedicated channels).
  • LAN communications may be performed using any conventional communication technology and/or protocol (e.g., Ethernet, WiFi, Bluetooth, WirelessHART, low power WiFi, Bluetooth low energy, fibre optics, IEEE 802.15.4e).
  • Wireless communications may be performed using one or more antennas suitable to the frequency and/or protocols used.
  • the VCD module may receive information from the PTCC module.
  • the VCD may transmit information received from the PTCC module.
  • Information may be transmitted to headquarters (e.g., central location), wayside equipment, individuals, and/or other trains.
  • Information from the PTCC module may include: Packets addressed to other trains; Packets addressed to common backend server to inform operators of train location; Packets addressed to wayside equipment; Packets addressed to wayside personnel to communicate train location; Any node to node arbitrary payload; and Packets addressed to third party listeners of PTC vision system.
  • the VCD module may also provide information to the PTCC module. The VCD may receive information from any source to which the VCD may transmit information.
  • Information provided by the VCD to the PTCC may include: Packets addressed from other trains; Packets addressed from common backend server to give feedback to a conductor or a train; Packets addressed from wayside equipment; Packets addressed from wayside personnel to communicate personnel location; Any node to node arbitrary payload; and Packets addressed from third party listeners of PTC vision system.
  • the GPS modules may include a conventional global positioning system ("GPS") receiver.
  • the GPS module receives signals from GPS satellites and determines a geographical position of the receiver and time (e.g., UTC time) using the information provided by the signals.
  • the GPS module may include one or more antennas for receiving the signals from the satellites. The antennas may be arranged to reduce and/or detect multipath signals and/or error.
  • the GPS module may maintain a historical record of geographical position and/or time.
  • the GPS module may determine a speed and direction of travel of the train.
  • a GPS module may receive correction information (e.g., WAAS, differential) to improve the accuracy of the geographic coordinates determined by the GPS receiver.
  • the GPS module may provide information to PTCC module.
  • the information provided by the GPS module may include: Time (e.g., UTC, local); Geographic coordinates (e.g., latitude & longitude, northing & easting); Correction information (e.g., WAAS, differential); Speed; and Direction of travel.
  • Time e.g., UTC, local
  • Geographic coordinates e.g., latitude & longitude, northing & easting
  • Correction information e.g., WAAS, differential
  • Speed e.g., Speed
  • Direction of travel e.g., direction of travel.
  • the DAP may receive (e.g., determine, detect, request) information regarding a train, the systems (e.g., hardware, software) of a train, and/or a state of operation of a train (e.g., train state).
  • the DAP may receive information from the systems of a train regarding the speed of the train, train acceleration, train deceleration, braking effort (e.g., force applied), brake pressure, brake circuit status, train wheel traction, inertial metrics, fluid (e.g., oil, hydraulic) pressures, and energy consumption.
  • Information from a train may be provided via a signal bus used by the train to transport information regarding the state and operation of the systems of the train.
  • a signal bus includes one or more conventional signal busses such as Fieldbus (e.g., IEC 61158), Multifunction Vehicle Bus (“MVB”), wire train bus (“WTB”), controller area network bus (“CanBUS”), Train Communication Network (“TCN”) (e.g., IEC 61375), and Process Field Bus (“Profibus”).
  • a signal bus may include devices that perform wired and/or wireless (e.g., TTEthernet) communication using any conventional and/or proprietary protocol.
  • the DAP may further include any conventional sensor to detect information not provided by the systems of the train. Sensors may be deployed (e.g., attached, mounted) at any location on the train. Sensors may provide information to the DAP directly and/or via another device or bus (e.g., signal bus, vehicle control unit, wide train bus, multifunction vehicle bus). Sensors may detect any physical property (e.g., density, elasticity, electrical properties, flow, magnetic properties, momentum, pressure, temperature, tension, velocity, viscosity). The DAP may provide information regarding the train to the other modules of the PTC ecosystem via the PTCC module.
  • Sensors may be deployed (e.g., attached, mounted) at any location on the train. Sensors may provide information to the DAP directly and/or via another device or bus (e.g., signal bus, vehicle control unit, wide train bus, multifunction vehicle bus). Sensors may detect any physical property (e.g., density, elasticity, electrical properties, flow, magnetic properties, momentum, pressure, temperature, tension, velocity, viscosity).
  • the DAP may receive information from any module of the PTC ecosystem via the PTCC module.
  • the DAP may provide information received from any source to other modules of the PTC ecosystem via the PTCC module.
  • Other modules may use information provided by or through the DAP to perform their respective functions.
  • the DAP may store received data.
  • the DAP may access stored data.
  • the DAP may create a historical record of received data.
  • the DAP may relate data from one source to another source.
  • the DAP may relate data of one type to data of another type.
  • the DAP may process (e.g., format, manipulate, extrapolate) data.
  • the DAP may store data that may be used, at least in part, to derive a signal state of the track on which the train travels, geographic position of the train, and other information used for positive train control.
  • the DAP may receive information from the PTCC module.
  • Information received by the DAP from the PTCC module may include: Requests for train state data; Requests for braking interface state; Commands to actuate train behavior (speed, braking, traction effort); Requests for fault messages; Acknowledgement of fault messages; Requests to raise alarms in the train; Requests for notifications of alarms raised in the train; and Requests for wayside equipment state.
  • the DAP may provide information to the PTCC module.
  • Information provided by the DAP to the PTCC module may include: Data from the signal bus of the train regarding train state; Acknowledge of requests; Fault messages on train bus; and Wayside equipment state.
  • the VA module detects the environment around the train.
  • the VA module detects the environment through which a train travels.
  • the VA module may detect the tracks upon which the train travels, tracks adjacent to the tracks traveled by the train, the aspect (e.g., appearance) of wayside (e.g., along tracks) signals (semaphore, mechanical, light, position), infrastructure (e.g., bridges, overpasses, tunnels), and/or objects (e.g., people, animals, vehicles).
  • Additional examples include: PTC assets, ETCS assets, Tracks, Signals, Signal lights, Permanent speed restrictions, Catenary structures, Catenary wires, Speed limit Signs, Roadside safety structures, Crossings, Pavements at crossings, Clearance point locations for switches installed on the main and siding tracks, Clearance/ structure gauge/ kinematic envelope, Beginning and ending limits of track detection circuits in non-signaled territory, Sheds, Stations, Tunnels, Bridges, Turnouts, Cants, Curves, Switches, Ties, Ballast, Culverts, Drainage structures, Vegetation ingress, Frog (crossing point of two rails), Highway grade crossings, Integer mileposts, Interchanges,
  • the VA module may detect the environment using any type of conventional sensor that detects a physical property and/or a physical characteristic.
  • Sensors of the VA module may include cameras (e.g., still, video), remote sensors (e.g., Light Detection and Ranging), radar, infrared, motion, and range sensors.
  • Operation of the VA module may be in accordance with a geographic location of the train, track conditions, environmental conditions (e.g., weather), speed of the train. Operation of the VA may include the selection of sensors that collect information and the sampling rate of the sensors.
  • the VA module may receive information from the PTCC module.
  • Information provided by the PTCC module may provide parameters and/or settings to control the operation of the VA module.
  • the PTCC may provide information for controlling the sampling frequency of one or more sensors of the VA.
  • the information received by the VA from the PTCC module may include: The frequency of the sampling, The thresholds for the sensor data, and Sensor configurations for timing and processing.
  • the VA module may provide information to the PTCC module.
  • the information provided by the VA module to the PTCC module may include: Present sensor configuration parameters, Sensor operational status, Sensor capability (e.g., range, resolution, maximum operating parameters), Raw or processed sensor data, Processing capability, and Data formats.
  • Raw or processed sensor data may include a point cloud (e.g., two-dimensional, three-dimensional), an image (e.g., jpg), a sequence of images, a video sequence (e.g., live, recorded playback), scanned map (e.g., two-dimensional, three-dimensional), an image detected by Light Detection and Ranging (e.g., LIDAR), infrared image, and/or low light image (e.g., night vision).
  • the VA module may perform some processing of sensor data. Processing may include data reduction, data augmentation, data extrapolation, and object identification.
  • Sensor data may be processed, whether by the VA module and/or the PTCC module, to detect and/or identify: Track used by the train, Distance to tracks, objects and/or infrastructure, Wayside signal indication (e.g., meaning, message, instruction, state, status), Track condition (e.g., passable, substandard), Track curvature, Direction (e.g., turn, straight) of upcoming segment, Track deviation from horizontal (e.g., declivity, acclivity), Junctions, Crossings, Interlocking exchanges, Position of train derived from environmental information, and Track identity (e.g., track ID).
  • Wayside signal indication e.g., meaning, message, instruction, state, status
  • Track condition e.g., passable, substandard
  • Track curvature e.g., Direction (e.g., turn, straight) of upcoming segment
  • Track deviation from horizontal e.g., declivity, acclivity
  • Junctions e.g., Crossings
  • the VA module may be coupled (e.g., mounted) to the train.
  • the VA module may be coupled at any position on the train (e.g., top, inside, underneath).
  • the coupling may be fixed and/or adjustable.
  • An adjustable coupling permits the viewpoint of the sensors of the VA module to be moved with respect to the train and/or the environment. Adjustment of the position of the VA may be made manually or automatically. Adjustment may be made responsive to a geographic position of the train, track condition, environmental conditions around the train, and sensor operational status.
  • the PTCC utilizes its access to all subsystems (e.g., modules) of the PTC system to derive (e.g., determine, calculate, extrapolate) track ID and signal state from the sensor data obtained from the VA module.
  • the PTCC module may utilize the train operating state information, discussed above, and data from the GPS receiver to refine geographic position data.
  • the PTCC module may also use information from any module of the PTC environment, including the PTC vision system, to qualify and/or interpret sensor information provided by the VA module. For example, the PTCC may use geographic position information from the GPS module to determine whether the infrastructure or signaling data detected by the VA corresponds to a particular location.
  • Speed and heading (e.g., azimuth) information derived from video information provided by the VA module may be compared to the speed and heading information provided by the GPS module to verify accuracy or to determine likelihood of correctness.
  • the PTCC may use images provided by the VA module with position information from the GPS module to prepare map information provided to the operator via the user interface of the HMI module.
  • the PTCC may use present and historical data from the DAP to detect the position of the train using dead reckoning, position determination may be correlated to the location information provided by the VA module and/or GPS module.
  • the PTCC may receive communications from other trains or wayside radio transponders (e.g., balises) via the VCD module for position determination that may be correlated and/or corrected (e.g., refined) using position information from the VA module and/or the GPS module or even dead reckoning position information from the DAP. Further, track ID, signal state, or train position may be requested to be entered by the operator via the HMI user interface for further correlation and/or verification.
  • trains or wayside radio transponders e.g., balises
  • the VCD module for position determination that may be correlated and/or corrected (e.g., refined) using position information from the VA module and/or the GPS module or even dead reckoning position information from the DAP.
  • track ID, signal state, or train position may be requested to be entered by the operator via the HMI user interface for further correlation and/or verification.
  • the PTCC module may also provide information and calls to action (e.g., messages, warnings, suggested actions, commands) to a conductor via the HMI user interface.
  • action e.g., messages, warnings, suggested actions, commands
  • the PTCC may bypass the conductor and actuate a change in train behavior (e.g., function, operation) utilizing the integration with the braking interface or the traction interface to adjust the speed of the train.
  • PTCC handles the routing of information by describing the recipient(s) of interest, the payload, frequency, route and duration of the data stream to share the train state with third party listeners and devices.
  • the PTCC may also dispatch/receive packets of information automatically or through calls to action from the common backend server in the control room or from the railway operators or from the control room terminal or from the conductor or from wayside signaling or modules in the PTC vision system or other third party listeners subscribed to the data on the train.
  • the PTCC may also receive information concerning assets near the location of the moving vehicle.
  • the PTCC may use the VA to collect data concerning PTC and other assets.
  • the PTCC may also process the newly collected data (or forward it) to audit and augment the information in the backend database.
  • the Track Identification Algorithm depicted in FIGs. 6 - 7 determines which track the rolling stock is currently utilizing.
  • the TIA creates a superimposed feature dataset by overlaying the features from the 3D LIDAR scanners and FLIR Cameras onto the onboard camera frame buffer.
  • the superset of features allows for three orthogonal measurements and perspectives of the tracks.
  • Thermal features from the FLIR Camera may be used to identify (e.g., separate, locate, isolate) the thermal signature of the railway tracks to generate a region of interest (spatial & temporal filters) in the global feature vector.
  • Range information from the 3D LIDAR scanner's 3D point cloud dataset may be utilized to identify the elevation of the railway track to also generate a region of interest (spatial & temporal filters) in the global feature vector.
  • Line detection algorithms may be utilized on the onboard camera, FLIR cameras and 3D LIDAR scanner's 3D point cloud dataset to further increase confidence in identifying tracks.
  • Color information from the onboard camera and the FLIR cameras may be used to also create a region of interest (spatial & temporal filter) in the global feature vector.
  • the TIA may look for overlaps in the regions of interest from multiple orthogonal measurements on the global feature vector to increase redundancy and confidence in track identification data.
  • the TIA may utilize the region of interest data to filter out false positives when the regions of interest do not overlap in the global feature vector.
  • the TIA may process the feature vectors in a region of interest to identify the width, distance, and curvature of a track.
  • the TIA may examine the rate at which a railway track is converging towards a point to further validate the track identification process; furthermore the slope of a railway track may also be used to filter out noise in the global feature vector dataset.
  • the TIA may take into consideration the spatial and temporal consistency of feature vectors prior to identifying the relative offset position of a train amongst multiple railway tracks.
  • Directional heading may be obtained by sampling the GPS receiver multiple times to create a temporal profile of movement in geographic coordinates.
  • the list of potential absolute track IDs may be obtained through a query to a locally cached GIS dataset or a remotely hosted backend server.
  • the odometer and directional heading may be used to calculate the dead reckoning offset.
  • the TIA compares the relative offset position of the train among multiple railway tracks and references to the list of potential absolute track IDs to identify the absolute track ID that the train is utilizing.
  • the global feature vector samples may be annotated with the geolocation (e.g., geographic coordinate) information and track ID. This allows the TIA to utilize the global feature vector datasets to directly determine a track position in the future. This machine learning approach reduces the computational cost of searching for an absolute track ID.
  • the TIA may further match global feature vector samples from a local or backend database with spatial transforms.
  • the parameters of the spatial transform may be utilized to calculate an offset position from a reference position generated from the query match.
  • the TIA may utilize the global feature vectors to stitch together features from multiple points in space or from a single point in space using various image processing techniques (e.g., image stitching, geometric registration, image calibration, image blending). This results in a superset of feature data that has collated global feature vectors from multiple points or a single point in space.
  • image processing techniques e.g., image stitching, geometric registration, image calibration, image blending.
  • the TIA can normalize the offset position for a relative track ID prior to determining an absolute track ID. This is useful when there are tracks outside the range of the vision apparatus (VA). This functionality is depicted in FIG. 10.
  • the TIA is a core component in the PTC vision system that eliminates the need for wireless transponders, beacons or balises to obtain positional data. TIA may also enable railway operators to annotate newly constructed railway tracks for their network wide GIS datasets that are authoritative in mapping the wayside equipment and infrastructure assets.
  • the Signal State Algorithm determines the signal state of the track a train is currently utilizing.
  • the purpose of this component is to ensure a train's operation is in compliance with the expected operational parameters of the railway operators or modal control rooms or central control rooms.
  • the compliance of a train's inertial metrics along a railway track can be audited in a distributed environment many backend servers or a centralized environment with a common backend server.
  • a train's ability to obtain the absolute track ID is important for correlating the semaphore signal state to the track ID utilized by a train. Auditing signal compliance is possible once the correlation between the semaphore signal state and the absolute track ID is established. Placement of sensors is important for efficiently determining a semaphore signal state.
  • FIG. 4 depicts one example wherein the 3D LIDAR scanner is forward facing and mounted on top of a train's roof.
  • the SSA takes into account an absolute track ID utilized by a train in order to audit the signal compliance of the train. Once the correlation of a track to a semaphore signal is complete, the signal state from that semaphore signal may actuate calls to action as feedback to a train or conductor.
  • Correlation of a railway track to a semaphore signal state may be possible by analyzing the regulatory specifications for wayside signaling from a railway operator. Utilizing the regulatory documentation, the spatial-temporal consistency of a semaphore signal may be compared to the spatial-temporal consistency of a railway track. A scoring mechanism may be used to choose the best candidate semaphore signal for the current railway track utilized by the train.
  • a local or remote GIS dataset may be queried to confirm the geolocation of a semaphore signal.
  • a local or remote signaling server may be queried to confirm the signal state in the semaphore signal matches what the PTC vision system is extrapolating.
  • Areas wherein the signal state is available to the train via radio communication may be utilized to confirm the accuracy of the PTC vision system and additionally augment the feedback provided to a machine learning apparatus that helps tune the PTC vision system.
  • a 3D point cloud dataset obtained from a PTC vision system may be utilized to analyze the structure of the semaphore signal. If the structure of an object of interest matches the expected specifications as defined by the regulatory body for a semaphore signal in that rail corridor, the object of interest may be annotated and added as a candidate for the scoring mechanism referenced above.
  • An infrared image captured through an FLIR camera may be utilized to identify the light being emitted from a wayside semaphore signal.
  • a call to action will be dispatched to the HMI onboard the train for signal compliance.
  • a call to action will be dispatched directly to the braking interface onboard the train for signal compliance.
  • the color spectrum in an image captured through the PTC vision system may be segmented to compute centroids that are utilized to identify blobs that resemble signal green, red, yellow or double yellow lights.
  • a centroid's spatial coordinates and size of its blob may be utilized to validate the spatial-temporal consistency of the semaphore signal with specifications from a regulatory body.
  • a spatial-temporal consistency profile of a track may be created by analyzing the curvature of a track, spacing between the rails on a track, and rate of convergence of the track spacing towards a point on the horizon.
  • a spatial-temporal consistency profile of a semaphore signal may be created by analyzing the following components: the height of a semaphore signal, the relative spatial distance between points in space, and the orientation and distance with respect to a track a train is currently utilizing.
  • the backend server may be queried to inform a train of an expected semaphore signal state along a railway track segment that the train is currently utilizing.
  • the backend server may be queried to inform a train of an expected semaphore signal state along a railway track segment identified by an absolute track ID and geolocation coordinates.
  • the Position Refinement Algorithm provides a high confidence geolocation service onboard the train.
  • the purpose of this algorithm is to ensure that loss of geolocation services does not occur when a single sensor fails.
  • the PRA relies on redundant geolocation services to obtain the track position.
  • GPS or Differential GPS may be utilized to obtain fairly accurate geolocation coordinates.
  • Tachometer data along with directional heading information can be utilized to calculate an offset position.
  • a WiFi antenna may scan SSIDs along with signal strength of each SSID while
  • GPS is working and later use the Medium Access Control (MAC) addresses (or any unique identifier associated with an SSID) to quickly determine the geolocation coordinates.
  • MAC Medium Access Control
  • the signal strength of the SSID during the scan by a WiFi antenna may be utilized to calculate the position relative to the original point of measurement.
  • the PTC vision system may choose to insert the SSID profile (SSID name, MAC address, geolocation coordinates, signal strength) as a reference point into a database based on the confidence in the current train's geolocation.
  • Global feature vectors created by the PTC vision system may be utilized to lookup geolocation coordinates to further ensure accuracy of the geolocation coordinates.
  • a scoring mechanism that takes samples from all the components described above would filter out for inconsistent samples that might inhibit a train's ability to obtain geolocation information. Furthermore, the samples may carry different weightage based on the performance and accuracy of each subcomponent in the PRA.
  • the PTC vision system samples the train state from the various subsystems described above.
  • the train state is defined as a comprehensive overview of track, signal and on-board information.
  • the state consists of track ID, signal state of relevant signals, relevant on-board information, location information (pre- and post-refinement, reference PRA, TIA and SSA algorithms described above), and information obtained from backend servers.
  • These backend servers hold information pertaining to the railroad infrastructure.
  • a backend database of assets is accessed remotely by the moving vehicle as well as railroad operators and officers. The moving train and its conductor for example use this information to anticipate signals along the route. Operator and maintenance officers have access to track information for example.
  • These reports and notifications are relevant to signals and signs, structures, track features and assets, safety information.
  • the PTC vision system issues notifications (local or remote), possibly raises alarms on-board the train, and can automatically control the train's inertial metrics by interfacing with various subsystems on-board (e.g., traction interface, braking interface, traction slippage system).
  • notifications local or remote
  • subsystems on-board e.g., traction interface, braking interface, traction slippage system.
  • On-board data represents a unit where all the data extracted from the various train systems is collected and made available. This data usually includes but is not limited to: Time information, Diagnostics information from various onboard devices, Energy monitoring information, Brake interface information, Location information, Signaling state obtained from train interfaces to wayside equipment, Environmental state obtained through the VA devices on board or on other trains, and Any other data from components that would help in Positive Train Control.
  • This data is made available within the PTC vision system for other components and can be transmitted to remote servers, other trains, or wayside equipment.
  • Location data is strategic to ensure that trains are operating within a safety envelope that meets the Federal Railroad Administration's PTC criteria.
  • wayside equipment is currently being utilized by the industry to accurately determine vehicle position.
  • the output of location services described above e.g., TIA & SSA
  • TIA & SSA provides the relative track position based on computer vision algorithms.
  • the relative position can be obtained through using a single sensor or multiple sensors.
  • the position we obtain is returned as an offset position, usually denoted as a relative track number.
  • Directional heading can also be a factor in building a query to obtain the absolute position from the feedback to the train.
  • the absolute position can be obtained either from a cached local database, or cached local dataset, remote database, remote dataset, relative offset position using on board inertial metric data, GPS samples, Wi-Fi SSIDs and their respective signal strength or through synchronization with existing wayside signaling equipment.
  • the various types of datasets we use include but are not limited to: 3D point cloud datasets, FLIR imaging, Video buffer data from on-board cameras.
  • this information can be utilized to correlate signal state from wayside signaling to the corresponding track.
  • the location services can also be exposed to third party listeners.
  • the on board components defined in the PTC vision system can act as listeners to the location services.
  • the train can scan the MAC IDs of the networked devices in the surrounding areas and utilize MAC ID filtering for any application these networked devices are utilizing. This is useful for creating context aware applications that depend on the pairing the MAC ID of a third party device (e.g., mobile phones, laptops, tablets, station servers, and other computational devices) with a train's geolocation information.
  • a third party device e.g., mobile phones, laptops, tablets, station servers, and other computational devices
  • the track signal state is important for ensuring the train complies with the PTC safety envelope at all times.
  • the PTC vision system's functional scope includes extrapolating the signal value from wayside signaling (semaphore signal state).
  • the communication module or the vision apparatus may identify the signal values of the wayside equipment.
  • a central back end server can relay the information to the train as feedback.
  • this information can also augment the vision-based signal extrapolation algorithms (e.g., TIA & SSA).
  • Datasets are used at the discretion of the PTC vision system.
  • data necessary for updating asset information is generated by the vision apparatus. This data then gets processed to verify the integrity of certain asset information, as well as update other asset information. Missing assets, damaged assets or ones that have been tampered with can then be detected and reported. The status of the infrastructure can also be verified, and the operational safety can be assessed, every time a vehicle with the vision apparatus travels down the track. For example, clearance measurements are performed making sure that no obstacles block the path of trains. The volume of ballast supporting the track is estimated and monitored over time.
  • the backend component has many purposes. For one, it receives, annotates, stores and forwards the data from the trains and algorithms to the various local or remote subscribers.
  • the backend also hosts many processes for analyzing the data (in real-time or offline), then generating the correct output. This output is then sent directly to the train as feedback, or relayed to command and dispatch centers or train stations.
  • Some of the aforementioned processes can include: Algorithms to reduce headways between trains to optimize the flow on certain corridors; Algorithms that optimize the overall flow of the network by considering individual trains or corridors; and Collision avoidance algorithms that constantly monitor the location and behavior of the trains.
  • the backend also hosts the asset database queried by the moving train to obtain asset and infrastructure information, as required by rolling stock movement regulations.
  • This database holds the following assets with relevant information and features: PTC assets, ETCS assets, Tracks, Signals, Signal lights, Permanent speed restrictions, Catenary structures, Catenary wires, Speed limit Signs, Roadside safety structures, Crossings, Pavements at crossings, Clearance point locations for switches installed on the main and siding tracks, Clearance/ structure gauge/ kinematic envelope, Beginning and ending limits of track detection circuits in non-signaled territory, Sheds, Stations, Tunnels, Bridges, Turnouts, Cants, Curves, Switches, Ties, Ballast, Culverts, Drainage structures, Vegetation ingress, Frog (crossing point of two rails), Highway grade crossings, Integer mileposts, Interchanges, Interlocking/control point locations, Maintenance facilities, Milepost signs, and Other signs and signals.
  • the rolling stock vehicle utilizes the information queried from the database to refine the track identification algorithm, the position refinement algorithm and the signal state detection algorithm.
  • the train (or any other vehicle utilizing the machine vision apparatus) moving along/in close proximity to the track collects data necessary to populate, verify and update the information in the database.
  • the backend infrastructure also generates alerts and reports concerning the state of the assets for various railroad officers.
  • the train can be controlled using the PTC vision system (e.g., Applications in Fig 5).
  • the output of the sensory stage might trigger certain actions independently of the any other system. For example, upon the detection of a red-light violation, the braking interface might be triggered automatically to attempt to bring the train to a stop.
  • Certain control commands can also arrive to the train through its VCD.
  • the backend system can for example instruct the train to increase its speed thereby reducing the headway between trains.
  • Other train subsystems might also be actuated through the PTC vision system, as long as they are accessible on the locomotive itself.
  • Feedback can also reach the locomotive and conductor through alarms.
  • an alarm can be displayed on the HMI.
  • the alarms can accompany any automatic control or exist on its own.
  • the alarms can stop by being acknowledged or halt independently.
  • Feedback can be in the form of notifications to the conductor through the user interface of the HMI module. These notifications may describe the data sensed and collected locally through the PTC vision system, or data obtained from the backend systems through the VCD. These notifications may require listeners or may be permanently enabled. An example of a notification can be about speed recommendations for the conductor to follow.
  • the backend may have two modules: data aggregation and data processing.
  • Data aggregation is one module whose role is to aggregate and route information between trains and a central backend.
  • the data processing component is utilized to make recommendations to the trains.
  • the communication is bidirectional and this backend server can serve all of the various possible applications from the PTC vision system.
  • Possible applications for PTC vision system include the following: Signal detection; Track detection; Speed synchronization; Extrapolating interlocking state of track and relaying it back to other trains in the network; Fuel optimization; Anti-Collision system; Rail detection algorithms; Track fault detection or preventative derailment detection; Track performance metric; Image stitching algorithms to create comprehensive reference datasets using samples from multiple runs; Cross Train imaging for, e.g., Preventative maintenance, Fault detection, and/or Vibration signature of passerby trains; Imaging based geolocation or geofiltering services; SSID based geolocation or geofiltering; and Sensory fusion of GPS + Inertial Metrics + Computer Vision-based algorithms.
  • FIG. 25 is a schematic block diagram of an exemplary in-vehicle system for vehicle localization and/or control.
  • In-vehicle runtime engine (“IVRE") 2500 and vehicle decision engine 2510 are computation and control modules, typically microprocessor-based, implemented locally on board a vehicle.
  • Local 3D map cache 2530 stores map data associated with the area surrounding the vehicle's rough position, as determined by GPS and EVIU sensors 2520, and can be periodically or continuously updated from a remote map store via communications module 2540 (which may include, e.g., a cellular data transceiver).
  • Machine vision sensors 2550 may include one or more mechanisms for sensing a local environment proximate the vehicle, such as LiDAR, video cameras and/or radar.
  • IVRE 2500 implements vehicle localization by obtaining a rough vehicle position from onboard GPS and IMU sensors 2520.
  • Machine vision sensors 2550 generate environmental signatures indicative of the local environment surrounding the vehicle, which are passed to IVRE 2500.
  • IVRE 2500 queries local 3D map cache 2530 using
  • Detailed vehicle position and other observed or calculated information can be utilized to implement other functionality, such as vehicle control and/or map auditing.
  • data from machine vision sensors 2550 can be analyzed using graphs and other data analysis mechanisms, as described elsewhere herein, for IVRE 2500 to determine a centerline for a lane in which the vehicle is traveling.
  • IVRE 2500 can also operate to obtain semantics (such as events and triggers) along the vehicle's route.
  • Available compute resources can be used to audit centralized map data sources by comparing previously-observed asset information obtained from centralized maps (and, e.g., stored in local 3D map cache 2530) to asset information derived from real time data captured by machine vision sensors 2550.
  • IVRE 2500 can thereby identify errors of omission (i.e. observed assets omitted from centralized map data) as well as errors of commission (i.e. assets in centralized map data that are not observed by machine vision sensors 2550). Such errors can be stored in cache 2530, and subsequently communicated to a central map repository via communications module 2540.
  • auditing of map data by a local vehicle may be initiated by a centralized control server, communicating with the vehicle via communications module 2540. For example, if the time elapsed since last auditing of a map section exceeds a threshold, a centralized control server can request auditing from a local vehicle traveling through the target region. In another example, if one vehicle reports discrepancies between centralized map data and locally-observed conditions, the centralized control server may request confirmation auditing by one or more other vehicles moving within the area of the discrepancy. Auditing requests may pertain to various combinations of geographic regions and/or mapping layers.
  • vehicle decision engine 2510 can operate to control various other systems and functions of the vehicle. For example, in an autonomous driving implementation, vehicle decision engine 2510 may utilize lane center line information and precise vehicle position information in order to steer the vehicle and maintain a centered lane position. These and other vehicle control operations may be beneficially implemented using systems and processes described herein.
  • Maps are collections of objects, their location and their properties. Maps can be divided into layers, where each layer is a grouping of objects of the same type. The location of each object is defined, along with a geometric attribute (example: the location of a pole could be a point in three-dimensional space, whereas a signal can be located by drawing a polygon around it).
  • a map becomes "semantic" when the semantic associations between different objects and layers are also recorded. For example, a map composed of the centerlines of various lanes on a roadway as well as the signs located around the infrastructure is labeled semantic, when the associations between the various signs and centerlines are recorded.
  • the semanticization of a map creates more context for the vehicle or user consuming the map.
  • the semantic map can also be packaged with regulatory information from various transportation authorities.
  • Geometric features used to describe shapes include points, lines, polygons, and arcs. The features are typically in three dimensions, but they can be projected into two-dimensional spaces where depth/elevation is lost.
  • semantic maps can be recorded and delivered in different coordinate and reference frames. There are also transformations allowing to project maps from one coordinate reference frame to the next. These maps can be packaged and delivered in different formats. Common formats include GeoJSON, KML, shapefiles, and the like.
  • the geospatial data used for semantic map creation comes from LiDAR, visible spectrum cameras, infrared cameras, and other optical equipment.
  • the output is a set of data points in three dimensions, along with images and video feeds in the visible spectrum and other frequencies.
  • the collection vehicle is also variable (aerial, mobile, terrestrial).
  • the geospatial data is collected initially with the collection vehicle being the origin of the reference frame.
  • IMU Inertial Measurement Unit
  • GPS Global Positioning Systems
  • Semantic maps derived from point cloud survey data may provide a vehicle with high levels of detail and information regarding the vehicle's current or anticipated local environment, which may be used, for example, to assist in relative vehicle localization, or serve as input data to autonomous control decision-making systems (e.g. automated braking, steering, speed control, etc.). Additionally, or alternatively, point-cloud data measured by a vehicle may be compared to previously-measured point cloud data to detect conditions or changes in a local environment, such as a fallen tree, overgrown vegetation, changed signage, lane closures, track or roadway obstructions, or the like. The detected changes in the environment can be used to further update the semantic maps.
  • FIG. 11 A illustrates a typical prior art process for extracting asset information from point cloud data.
  • surveying procedures generate point cloud data sets, such as using a LiDAR surveying apparatus.
  • step SI 105 the raw point cloud data is visualized.
  • GIS Geographical Information Systems
  • the first step in the GIS analysts' process is to separate the terabytes of point cloud data into smaller manageable sections. This is due to the fact that contemporary personal computers are limited (memory/computational power) and are unable to manage the terabytes of LiDAR data at once.
  • the GIS analysts use 3D visualization software to traverse each of the smaller sections of point cloud. As they progress through their respective sections, the GIS analysts delineate and annotate the important assets. Finally, the annotated assets of each GIS analyst are combined into one map (step SI 110). Varying file formats and software systems can create additional difficulties in merging the separate datasets.
  • FIG. 1 IB illustrates an alternative approach to extracting asset information from raw point cloud data.
  • step SI 150 surveying is conducted to generate the raw point cloud data.
  • step SI 155 asset maps are generated directly from the raw point cloud data, without requiring visualization of the large, complex data set, or manual annotation of that data.
  • Figure 12 illustrates a computing apparatus for rapidly and efficiently extracting asset information from large point-cloud data sets.
  • Figure 13 illustrates a process for using the apparatus of Figure 12.
  • the components within the apparatus of Figure 12 are implemented using Internet-connected cloud computing resources, which may include one or more servers.
  • Front-End component 1200 includes data upload tool 1205, configuration tool 1210, and map retrieval tool 1215.
  • Front-End component 1200 provides a mechanism for end users to interact with and control the computing apparatus.
  • a user can upload LiDAR and other surveying data from a local data storage device to data storage component 1220 (step S1300).
  • Data storage component 1220 may implement a distributed file system (such as the Hadoop Distributed File System) or other mechanism for storing data.
  • Configuration tool 1210 can be accessed via a user's network-connected computing device (not shown), and enables a user to define the format of uploaded data as well as other survey details, and specify assets to search for and annotate (step SI 305). After a user interacts with configuration tool 1210 to select desired assets, the user is provided with various options to configure the output map format.
  • configuration tool 1210 then solicits a desired turnaround time from a configuring user, and presents the user with an estimated cost for the analysis (step S1310).
  • the cost estimate is determined based on, e.g., the size of the uploaded data set to be analyzed, the number (and complexity) of selected assets, the output format, and the selected turnaround time.
  • the user interacts with configuration tool 1210 to initiate an analysis job (step S1315).
  • the geospatial data uploaded through front end 1200 is tracked in database collections. This data is organized by category, geographic area, and other properties. As the data evolves through various stages of execution, the relevant database entries get updated.
  • Point-cloud data uploaded through the front-end tool is stored in a secure and replicated manner.
  • the data is tiled into different size tiles in a Cartesian coordinate system.
  • the tiles themselves are limited in two dimensions and namespaced accordingly.
  • tiles are limited in X and Y dimensions, and unlimited in a Z dimension that is vertical or parallel to the direction of the Earth's gravitational pull, such that a tile defines a columnar area, unlimited in height (i.e. limited only to the extent of available geospatial data) and having a rectangular cross-section.
  • tiles which are 1000m on the side (in the horizontal plane) can be utilized.
  • the files representing the tiles would then hold all the points which belong to the particular geographic area delimited by the tile, and no other.
  • tree structures (such as quadtrees and octrees) are implemented depending on the traversal style for the data.
  • Processing of the data to automatically extract semantic maps from geospatial data occurs on computation clusters, implemented within processing unit 1240 (embodiments of which are described further with reference to Figure 16, below). These have access to the point cloud and other data through the network accessible storage unit 1220. Intermediary results as well as finalized ones are stored similarly.
  • Figure 14 illustrates a process that may be performed by the apparatus of Figure
  • the point-cloud data is subdivided into chunks (step S1400) by data storage / preprocessing component 1220. These chunks can be subsets of tiles or combinations thereof, potentially selected to optimize for, e.g., the desired processing method, available memory and other runtime considerations. Individual nodes in the computation cluster (i.e. within processing unit 1240) are then capable of processing geospatial and other data associated with a given data chunk, i.e., selected subsets or combinations of tiles.
  • the density of the point-cloud may be an important factor in determining the number of tiles (or the size of tile subsets) to process within the same computation node.
  • Figure 15 illustrates the size of tiles with respect to the number of points within (represented by the diagonal line), as well as the distribution of tiles sizes for an exemplary dataset comprising LiDAR point-cloud data measured along a 2km section of railway (each tile represented by hatches across the diagonal line).
  • Data storage and preprocessing component 1220 performs tile aggregation, and/or subdivision, prior to feeding data to processing unit 1240, in order to optimize the analysis performance.
  • job scheduler 1225 creates a queue containing tasks pertaining to the job, as configured in steps S1305 and S1310.
  • Job scheduler 1225 associates one or more of analysis mechanisms 1250 (typically implementing various different data analysis algorithms) with the task (step S1405), and creates a cluster of machines within processing unit 1240 to process the data (step S1410).
  • the size of the cluster i.e. the number of computation nodes
  • job scheduler 1225 can initiate a cluster of 20 machines with four cores each, and process the same dataset in approximately 24 hours instead.
  • Processing unit 1240 is composed of a collection of compute clusters.
  • the size of the cluster depends on the number of jobs.
  • Figure 16 illustrates an exemplary compute cluster.
  • Each cluster contains: a master instance 1605, responsible for managing the cluster; a set number of principal computation nodes 1610, which also store data in data storage system 1220; and a variable number of "spot" instances 1620.
  • compute clusters consisting entirely of spot instances, or entirely of principal nodes, may be utilized.
  • data storage and preprocessor component 1220 directs a stream of data chunks (e.g. aggregations of tiles satisfying a desired data subset size) to processing unit 1240 (step S1415).
  • data chunks e.g. aggregations of tiles satisfying a desired data subset size
  • processing unit 1240 execute appropriate data analysis mechanisms 1250 to, e.g., extract asset or feature information from the 3D point-cloud tiles.
  • map generator 1230 combines the output of nodes within processing unit 1240 into semantic maps (step S1420). Reporting analytics can be derived from the semantic maps by running queries to analyze particular assets and their combinations.
  • Map generator 1230 may also include an annotation integrity verifier operating to verify the integrity of annotated datasets over time.
  • locations may be surveyed repeatedly at different times.
  • trains equipped with LiDAR or other railway surveying vehicles may periodically survey the same length of railway, such as to monitor the health or status of assets along a track.
  • LiDAR-equipped survey vehicles may travel along a given portion of road at different times.
  • data captured by LiDAR equipped automobiles, such as autonomous driving cars may be regularly analyzed, providing potentially frequent analyses of the local environment in a given location.
  • Each time a new map is generated by map generator 1230 concerning a given area asset or local feature information can be compared to such information contained in older maps. Alarms, notifications or events can be triggered when discrepancies are detected.
  • map generator 1230 The output of map generator 1230 is ultimately made available to the user, via front end 1200 and map retrieval tool 1215 (step S1425). Once a job is completed and a map is generated, scheduler 1225 (monitoring the status of tasks and jobs) generates notifications for the end user.
  • Feature maps (containing only the location, geometry and features of various assets), as well as semantic ones can also be stored in remotely accessible geodatabases.
  • the map data can be retrieved either directly or through a server to facilitate the querying and collection of results.
  • the maps can be retrieved in their entirety or by selecting a specific area of interest.
  • data upload step SI 300 employs end-to-end encryption (such as AES encryption) from the user data source to the cloud computing platform.
  • AES encryption may also be utilized for communications between a user's system and front-end 1200.
  • data storage component 1220 may include a compression mechanism to compress point-cloud data before storage.
  • LAZ LempelZivOberhumer
  • GZIP also based on LempelZiv methods
  • LASzip released by rapidlasso GmbH
  • LZO and GZIP are optimized for decompression, and therefore present a superior alternative to LAZ in terms of CPU time required for decompression.
  • Data analysis mechanisms 1250 are typically selected based on the nature of the information desired to be extracted from the point-cloud data. It may be desirable to design mechanisms 1250 with very low false positive rates, while maintaining acceptable detection rates. For added confidence in generated maps, in some applications, a subset of results may be verified manually by inspecting the original point-cloud and raw imaging data.
  • track detection may be an important first step. Track detection can be important because knowledge of the track position facilitates identification of assets, since regulations often assign specific locations for each asset in relation to the track.
  • Figure 20 illustrates a process for track detection and traversal that can be implemented by processing unit 1240, e.g. in step S1415 of Figure 14.
  • step S2000 a 100m x 100m section of point-cloud data is identified for analysis.
  • step S2010 the geometry of the 10,000m 2 point cloud section is analyzed to extract a subset of points which are associated with the track. Many techniques can be employed to achieve the desired result. In some
  • previously-classified tracks from similar data sets can be studied to identify properties of data in the vicinity of the tracks, with those properties serving as an indicia of track location in newly-analyzed data.
  • Other techniques include projecting points in two-dimensional space (based on, e.g., height or pulse intensity) and utilizing edge detection mechanisms and transforms to isolate regions belonging to the track.
  • the 10,000 m 2 point cloud section in step S2000 may consist of about 1 GB of data
  • the extracted track subset output in step S2010 may consist of about 1 MB of data.
  • Figure 21 is a visualization of the 10,000 m2 point cloud section input to step
  • Lines 2100 represent track that is visible in the point-cloud.
  • Line 2110 represent track that was obscured during the LiDAR data collection process, having a position that is estimated. This is typically the result of shadowing, a process which occurs when the object of interest is hidden from direct line of sight of the measuring instrument.
  • Dots 2120 correspond to problematic positioning of a LiDAR tripod system which resulted in some track sections being obstructed.
  • the location of the invisible track can be inferred by utilizing known spatial continuity properties of the infrastructure (such as spacing relative to other observed elements) (step S2020).
  • Geospatial data presents many dimensionalities that can be taken advantage of during asset extraction.
  • Imagery, infrared, video feeds and/or multispectral sensors can be combined to increase detection confidence and accuracy.
  • Most LiDAR systems include an intensity measurement for each point.
  • classification mechanisms and filters can be added to the system, for an increased track detection rate.
  • Figure 22 is a histogram of point-cloud intensity levels in an exemplary track detection implementation.
  • Figure 22a illustrates quantity of each measured intensity level in an analyzed body of point cloud data, as a whole.
  • Figure 22b illustrates the same histogram, for points within the point cloud identified as corresponding to track.
  • a simple band pass filter can be effective in some cases to further narrow a search space for points belonging to the rail.
  • Other classification methods can also be utilized.
  • Figure 23 is a visualization of a portion of the output of an implementation including a track detection mechanism and other asset detection mechanisms.
  • track segments 2300 are identified first, then for each track, centerline markers 2310 are established. Once the tracks and track centerlines are identified, subsequent analysis components can traverse the track within the point-cloud data, while enjoying a 360 degree view of high resolution point cloud data around each point in the centerline.
  • Other analysis mechanisms identify and locate other assets or features for inclusion in a sematic map. For example, an overhead wire detection mechanism identifies and locates overhead wires, and demarcates them with overhead wire centerline indicia 2320.
  • a pole detection mechanism identifies and locates trackside poles, and locates them with indicia 2330.
  • analysis mechanisms may be applied sequentially, with an output of one mechanism serving as an input to another mechanism.
  • assets and elements of the local environment regularly are replaced, added, removed or shifted. It may be desirable to regularly check clearance above and around a track to ensure safe operation, and that train cars do not come into contact with any obstructions.
  • a track detection mechanism such as that described above, may be implemented as part of a sequence of analysis mechanisms.
  • the output of a track detection mechanism that includes the track centerline may be subsequently used as an input to a track clearance check mechanism.
  • a bounding box is defined with respect to the track center line, and any objects that encroach within that bound are reported. The dimensions of the bounding box can be modified to fit various standards.
  • Overhead wires can be identified within point-cloud data. The height of the wire in comparison with the track is assessed. Areas with saggy lines are reported. By using pole location information, the catenary shape of the wire can also be assessed.
  • the automated extraction of maps can be achieved by combining computation blocks into directed acyclic graphs (hereafter referred to as "graphs").
  • the blocks contained in these graphs have a varying degree of complexity, ranging from simple averaging and thresholding to transforms, filters, decompositions, etc.
  • the output of one stage of the graph can feed into any other subsequent stage.
  • the stages need not run in sequence but can be parallelized given sufficient information per stage.
  • a graph is generally used to classify points within a point cloud belonging to the same category, or to vectorize.
  • Vectorization refers to the creation of an (often imaginary) line or polygon going through a set of points delimiting their center, boundary, location, etc.
  • computation graphs can be used to implement classifiers, clustering methods, fitting routings, neural networks and the like.
  • Rotations and projections are also used, often in conjunction with machine vision processing techniques.
  • the creation of semantic maps from geospatial data may be parallelized. There are many levels of parallelization that can be implemented. At the highest level, the survey data can be divided into regularly-shaped regions of interest which get streamed to different machines and CPU processes. The results coming from each area need to then be merged in a "reduce” step once all the processes finish, similarly to the process of Figure 14. Since boundary conditions arise, padding the regions of interest with extra data which is truncated at the end of the process usually removes those deformities near the edges. The size of the region of interest, as well as the padding thickness is determined by the graph extracting the assets or features.
  • parallelism can occur when processing is taking place along a pre-extracted vector. For example, when searching for signs in the vicinity of a railroad track, the data can be traversed by extracting regions around waypoints along the previously extracted track centerline. Multiple processes can then be used in parallel along different waypoints of the track.
  • each point when analyzing a particular region, each point can be considered individually.
  • a voxel surrounding that point is usually extracted and analyzed.
  • This process can also be made parallel, in those cases when the outcome of one point's operation does not affect that of any other point.
  • GPU graphics processing units
  • Geospatial data is not limited to point cloud, but extends to imagery, video feeds, multispectral data, RADAR, etc.
  • some embodiments may utilize any additional data sources that are available.
  • datasets can be combined in a pre-processing stage (e.g. step S1400), before feeding into the computation graphs. This approach provides computation graphs with data from multiple sources for processing.
  • one set of data may be used to generate a hypothesis concerning an asset and its properties; data from other sources can then be used to validate and/or augment the hypothesis via other analysis mechanisms.
  • Figure 24 illustrates an embodiment of a system implementing supervised machine learning, including training component 2400 and map generation component 2410.
  • Training component 2400 receives as inputs, raw point cloud data 2420 and sample output 2422.
  • sample output 2422 may be verified output data associated with approximately 1% of the total data set.
  • Sample output 2422 may include classified point cloud data (where points belonging to a particular asset category are grouped together), and/or a vectorized map (with points, lines and polygons drawn over assets of interest).
  • Training component output 2424 defines an optimized categorization mechanism, such as algorithm coefficients for an analysis mechanism comparable to mechanisms 1250 in the map generation system of Figure 12.
  • Training component output 2424 may also define a region of interest for the algorithms to be most effective, define functional blocks within a computation graph which should be utilized, and/or define features of interest for a particular asset under consideration. Training component output 2424 is fed into map generation component 2410, along with the full corpus of raw point cloud data 2420. Map generation component 2410 then operates to generate map output 2426.
  • Unsupervised methods can also be implemented for generating maps. Such processes can rely on scale-dependent features to describe contextual information for individual map points. They can also rely on deep learning to design feature transformations for use with map point features. Ensembles of feature transformations generated by deep learning are used to encode map point context information. Asset membership for points can then be based on features transformed by deep learning algorithms. Another method revolves around curriculum- based learning where assets are described in a curriculum, then learned in computation graphs. This method can be effective when the assets of interest are regular in shape and properties, and do not exhibit a lot of spatial complexity.
  • a neural network is often trained in a primary step, then applied to the remainder of the geospatial data for extraction of the map.
  • Machine learning techniques can therefore assist in optimizing and refining computation graphs. These graphs can be engineered manually or learned using the above methods.
  • a parameter search component is useful for accuracy improvements and reductions in false positives and negatives.
  • various parameters of the computation graph (from the region of interest, to the parameters of each function, to the number and nature of features used in a classifier) can all be modulated and the output monitored.
  • search methodologies the best performance combination of parameters can be found and applied to the remainder of the data. This step assumes the availability of previously annotated semantic maps.
  • computation graphs When computation graphs are refined to an acceptable performance level, they can be used directly in the vehicles. This would correspond to streaming of the intelligence from the cloud to the vehicles, as opposed to the more conventional streaming of data from local environments to cloud systems. With geospatial data, the sheer size of the sensor data can be prohibitive. Therefore, in some embodiments, locally-obtained sensor data (e.g. data obtain by vehicle-mounted sensors) is summarized via local computation resources, with only a subset of collected information and/or extracted content being sent back to remote data systems. For example, resources comparable to data storage / preprocessor component 1220, processing unit 1240 and data analysis mechanisms 1250, can be implemented in-vehicle to extract semantic map data from onboard sensor systems. Computation graphs analogous to those described above for implementation in a cloud-based processing structure, can be optimized and tested in a machine learning framework, while presenting an opportunity for local in-vehicle
  • Such embodiments can utilize the vehicles as a distributed computing platform, constantly updating the contents of a centrally-maintained map, while consuming most of the remotely-sensed data in place, rather than streaming all of it to a central, cloud-based system.
  • a simulation environment can be utilized.
  • maps are programmatically generated in large numbers of permutations of parameters, to replicate the variability of terrains and landmarks on the face of the planet.
  • Three dimensional models are then generated from the maps and raytraced to create a point cloud in as similar a way to real data collection as possible. Since the location of every asset is known a priori, a perfect map extracted from the point cloud is then available.
  • the variability of the data, and the fact that a perfect ground truth exists for each point cloud greatly increases the scope of the computation graphs and their accuracy. It also provides a mechanism to understand the limitations of the current computing paradigms.
  • Quality control can be performed in multiple ways. Similar to creating a semantic map, a GIS analyst can use conventional visualization tools and overlay the raw survey data with the automatically extracted map. Any discrepancies can then be identified and corrected. Another method for QC would be to crowd source the effort amongst multiple agents online. Since each one of those agents might not be entirely skilled in semantic map creation, the QC work would need to be replicated. Hypotheses can then be confirmed or denied by each QC result, and a final conclusion reached with enough trials.

Landscapes

  • Engineering & Computer Science (AREA)
  • Mechanical Engineering (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Electromagnetism (AREA)
  • Radar, Positioning & Navigation (AREA)
  • Remote Sensing (AREA)
  • Train Traffic Observation, Control, And Security (AREA)
  • Electric Propulsion And Braking For Vehicles (AREA)
  • Traffic Control Systems (AREA)
  • Image Analysis (AREA)

Abstract

Methods and apparatus for real time machine vision and point-cloud data analysis are provided, for remote sensing and vehicle control. Point cloud data can be analyzed via scalable, centralized, cloud computing systems for extraction of asset information and generation of semantic maps. A data storage / preprocessor (1220) subdivides a data set for streaming to a distributed processing unit (1240) and operation via data analysis mechanisms (1250). The output of the processing unit (1240) is aggregated by a map generator (1230). Machine learning components can optimize data analysis mechanisms to improve asset and feature extraction from sensor data. Optimized data analysis mechanisms can be downloaded to vehicles for use in on-board systems analyzing vehicle sensor data. Semantic map data can be used locally in vehicles, along with onboard sensors, to derive precise vehicle localization and provide input to vehicle to control systems.

Description

TITLE
Real Time Machine Vision And Point-Cloud Analysis For Remote Sensing And Vehicle Control
CROSS REFERENCE TO RELATED APPLICATIONS
[0001] The present claims the benefit of, priority to, and incorporates by reference, in its entirety, United States Provisional Patent Application No. 62/105,696, entitled A Scalable Approach To Point-Cloud Data Processing for Railroad Asset Location and Health Monitoring, filed January 20, 2015.
BACKGROUND
[0002] The automated localization of moving vehicles and machine-based remote sensing of vehicle local environment is becoming increasingly important in several different disciplines. One such discipline is automotive transportation. In recent years, many cars and trucks implement onboard Global Positioning System (GPS) receivers and navigation systems utilizing GPS data for driver guidance. However, as automobile manufacturers seek to implement more advanced driving automation, such as autonomous driving features, GPS-based location systems may not be able to provide sufficiently accurate vehicle localization, nor do they allow for real-time sensing of a vehicle's local environment. Therefore, supplemental sensing systems may be desirable, as well as highly detailed infrastructure and landmark maps, potentially including three-dimensional semantic maps.
[0003] Another application in which vehicle localization, sensing of a local environment and three-dimensional semantic maps may be desirable is in the operation of trains. The U.S. Congress passed the U.S. Rail Safety Improvement Act in 2008 to ensure all trains are monitored in real time to enable "Positive Train Control" (PTC). This law requires that all trains report their location information such that all train movements are tracked in real time. PTC is required to function both in signaled territories and dark territories.
[0004] In order to achieve this milestone, numerous companies have tried to implement various PTC systems. A reoccurring problem is that current PTC systems can only track a train when it passes by wayside transponders or signaling stations along a railway line, rendering the operators unaware of the status of the train in between wayside signals. Therefore, the distance between consecutive physical wayside signaling infrastructures determines the minimum safe distance required between trains (headway). Current signaling infrastructure also limits the scope of deploying wayside signaling equipment due to the cost and complexity of constructing and maintaining PTC infrastructure along the length of the railway network. The current
methodology for detecting trains the last time they passed near a wayside detector suffers from a lack of position information in-between transponders.
[0005] Certain companies went a step further to utilize radio towers along the length of the operator's track network to create virtual signals between trains, circumventing the need for wayside signaling equipment. Radio towers still require signaling equipment to be deployed in order for the radio communication to take place. However, for dependable location information, additional transponders have to be deployed along tracks for the train to reliably determine the position of the train and the track it is currently occupying.
[0006] One example of a PTC system in use is the European Train Control System
(ETCS) which relies on trackside equipment and a train-mounted control that reacts to the information related to the signaling. That system relies heavily on infrastructure that has not been deployed in the United States or in developing countries.
[0007] A solution that requires minimal deployment of wayside signaling equipment would be beneficial for establishing Positive Train Control throughout the United States and in the developing world. Deploying millions of balises— the transponders used to detect and communicate the presence of trains and their location— every 1-15 km along tracks is less effective because balises are negatively affected by environmental conditions, theft, and require regular maintenance, and the data collected may not be used in real time. Obtaining positional data through only trackside equipment is not a scalable solution considering the costs of utilizing balises throughout the entire railway network PTC. Moreover, train control and safety systems cannot rely solely on a global positioning system (GPS) as it not sufficiently accurate to distinguish between tracks, thereby requiring wayside signaling for position calibration.
[0008] As autonomous driving, train control and other vehicle operating systems evolve, these and other challenges may be addressed by systems and methods described hereinbelow. SUMMARY
[0009] In accordance with one aspect disclosed herein, systems and methods are described for localization and/or control of a vehicle, such as a train or automobile. Local environment sensors, which may include a machine vision system such as LiDAR, can be mounted on a vehicle. A GPS receiver may also be included to provide a first geographical position of the vehicle. A remote database and processor stores and processes data collected from multiple sources, and an on-board vehicle processor downloads data relevant for operation, safety, and/or control of the moving vehicle. The local environmental sensors generate data describing a surrounding environment, such as point-cloud data generated by a LiDAR sensor. Collected data can be processed locally, on board the vehicle, or uploaded to a remote data system for storage, processing and analysis. Analysis mechanisms (on-board and/or
implemented in remote data systems) can operate on the collected data to extract information from the sensor data, such as the identification and position of objects in the local environment.
[0010] An exemplary embodiment of a system described herein includes a hardware component mounted on railroad or other vehicles, a remote database, and analysis components to process data collected regarding information about a transportation system, including moving and stationary vehicles, infrastructure, and transit pathway (e.g. rail or road) condition. The system can accurately estimate the precise position of the vehicle traveling down the transit pathway, such as by comparing the location of objects detected in the vehicle's on-board sensors relative to the known location of objects. Additional attributes about the exemplary components are detailed herein and include the following:
[0011] The Hardware: informs the movement of vehicles for safety, including: in railroad applications, identifying the track upon which they are traveling, obstructions, health of track and rail system, among other features; and in automotive applications, the lane upon which the vehicle is traveling, the texture and health of the road, the identification of assets in the vicinity, amongst other features.
[0012] The Remote Database: contains information about assets, and which can be queried remotely to obtain additional asset information.
[0013] Database Population With Asset Information: methods include machine vision data collected by the traveling vehicle itself, or by another vehicle (such as road-rail vehicles, track inspection vehicles, aerial vehicles, mobile mapping platforms, etc.). This data is then processed to generate the asset information (location, features, road/track health, among other information).
[0014] Data Analysis Mechanisms: fuse together several data and information streams
(e.g. from the sensors, the database, wayside units, the vehicle's information bus, etc.) to result in an accurate estimate of the lane, track ID or other indicia of localization.
[0015] These and other aspects of the disclosure will be apparent in view of the text and drawings provided herein.
BRIEF DESCRIPTION OF THE DRAWINGS
[0016] Exemplary embodiments will now be further described with reference to the drawings, wherein like designations denote like elements, and:
[0017] FIG. 1 is a representative flow diagram of a Train Control System;
[0018] FIG. 2 is a representative flow diagram of the on board ecosystem;
[0019] FIG. 3 is a representative flow diagram for obtaining positional information;
[0020] FIG. 4 is an exemplary depiction of a train extrapolating the signal state;
[0021] FIG. 5 is an exemplary depiction of the various interfaces available to the conductor as feedback;
[0022] FIG. 6 is a representative flow diagram for obtaining the track ID occupied by the train;
[0023] FIG. 7 is a representative flow diagram which describes the track ID algorithm;
[0024] FIG. 8 is a representative flow diagram which describes the signal state algorithm;
[0025] FIG. 9 is a representative flow diagram which depicts sensing and feedback; and
[0026] FIG. 10 is a representative flow diagram of image stitching techniques for relative track positioning.
[0027] FIGS. 11 A and 1 IB are flow diagrams of point-cloud analysis processes.
[0028] FIG. 12 is a schematic block diagram of an apparatus for point-cloud analysis.
[0029] FIG. 13 is a flow diagram of a process for analyzing point-cloud data.
[0030] FIG. 14 is a further flow diagram of a process for analyzing point-cloud data. [0031] FIG. 15 is a chart illustrating point cloud tile size and density distribution in an exemplary point-cloud survey.
[0032] FIG. 16 is a schematic block diagram of a point-cloud processing cluster.
[0033] FIG. 17 is a plot of characteristics for compression mechanisms usable with point- cloud data.
[0034] FIG. 18 is a plot of characteristics for compression mechanisms usable with point- cloud data.
[0035] FIG. 19 is a plot of characteristics for compression mechanisms usable with point- cloud data.
[0036] FIG. 20 is a flow diagram of a process for track detection.
[0037] FIG. 21 is a visualization of a point-cloud section with extracted rail information.
[0038] FIG. 22 is a histogram of point-cloud intensity levels in an exemplary point-cloud segment.
[0039] FIG. 23 is a visualization of track detection mechanism output.
[0040] FIG. 24 is a schematic block diagram of a map generation system utilizing supervised machine learning.
[0041] FIG. 25 is a schematic block diagram of a run-time system for automobile localization, automobile control and map auditing.
DETAILED DESCRIPTION
[0042] In accordance with one embodiment, methods and apparatuses are provided for determining the position of one or more moving vehicles, e.g., trains or autonomous driving vehicles, without depending on balises/transponders distributed throughout the operating environment for accurate positional data. Some train-based implementations of such
embodiments are sometimes referred to herein as BVRVB-PTC, a PTC vision system, or a machine vision system.
[0043] Also disclosed are solutions to use that positional data to optimize vehicle control and operation, such as the operation of the trains within a rail system. Railway embodiments can use a series of sensor fusion and data fusion techniques to obtain the track position with improved precision and reliability. Such embodiments can also be used for auto-braking of trains for committing red light violations on the track, for optimizing fuel based on terrain, synchronizing train speeds to avoid red lights, anti-collision systems, and for preventative maintenance of not only the trains, but also the tracks, rails, and gravel substrate underlying the tracks. Some embodiments may use a backend processing and storage component for keeping track of asset location and health information (accessible by the moving vehicle or by railroad operators through reports).
[0044] In addition to localization, it may be desirable for autonomous driving
embodiments to take advantage of highly detailed infrastructure and landmark maps. These maps can be utilized to direct the flow of traffic in the real world and plan routes for vehicles to travel from source to destination. The three-dimensional nature of the maps, in addition to their accuracy in representing the physical world, assist the vehicles in anticipating events beyond their sensing range, foveating their sensors to the assets of interest, and localizing the vehicles in relation to the landmarks. By utilizing highly detailed three-dimensional (semantic) maps for the pseudo-static assets, the vehicle's resources are liberated to observe the dynamic objects around it.
[0045] The PTC vision system may include modules that handle communication, image capture, image processing, computational devices, data aggregation platforms that interface with the train signal bus and inertial sensors (including on-board and positional sensors).
[0046] Figure 1 illustrates an exemplary flow operation of a Train Control System. In step SI 00, a train undergoes normal operation. In step SI 05, the train state is retrieved from the Data Aggregation Platform (described below). In step SI 10, the train position is refined. In step SI 15, semaphore signal states are identified from local environment sensor information. In step S120, feedback is applied. The train speed can be adjusted (step S125), alarms and/or notifications can be raised (step S130). Further detail concerning of each of these steps is described hereinbelow.
[0047] Referring to FIG. 2, a PTC vision system may include one or more of the following: Data Aggregation Platform (DAP) 215, Vision Apparatus (VA) 230, Positive Train Control Computer (PTCC) 210, Human Machine Interface (HMI) 205, GPS Receiver 225, and the Vehicular Communication Device (VCD) 220, typically communicating via LAN or WAN communications network 240. [0048] The components (e.g., VCD, HMI, PTCC, VA, DAP, GPS) may be integrated into a single component or be modular in nature and may be virtual software or a physical hardware device. Each component in the PTC vision system may have its own power supply or share one with the PTCC. The power supplies used for the components in the PTC vision system may include non-interruptible components for power outages.
[0049] The PTCC module maintains the state of information passing in between the modules of the PTC vision system. The PTCC communicates with the HMI, VA, VCD, GPS, and DAP. Communication may include providing information (e.g., data) and/or receiving information. An interface (e.g., bus, connection) between any module of the ecosystem may include any conventional interface. Modules of the ecosystem may communicate with each other, a human operator, and/or a third party (e.g., another train, conductor, train operator) using any conventional communication protocol. Communication may be accomplished via wired and/or wireless communication link (e.g., channel).
[0050] The PTCC may be implemented using any conventional processing circuit including a microprocessor, a computer, a signal processor, memory, and/or buses. A PTCC may perform any computation suitable for performing the functions of the PTC vision system.
[0051] The HMI module may receive information from the PTCC module. Information received by the HMI module may include: Geolocation (e.g., GPS Latitude & Longitude coordinates); Time; Recommended speeds; Directional Heading (e.g., azimuth); Track ID;
Distance/headway between neighboring trains on the same track; Distance/headway between neighboring trains on adjacent tracks; Stations of interest, including Next station, Previous station, or Stations between origin and destination; State of virtual or physical semaphore for current track segment utilized by a train; State of virtual or physical semaphore for upcoming and previous track segments in a train's route; and State of virtual or physical semaphore for track segments which share track interlocks with current track.
[0052] The HMI module may provide information to the PTCC module. Information provided to the PTCC may include information and/or requests from an operator. The HMI may process (e.g., format, reduce, adjust, correlate) information prior to providing the information to an operator or the PTCC module. The information provided by the HMI to the PTCC module may include: Conductor commands to slow down the train; Conductor requests to bypass certain parameters (e.g., speed restrictions); Conductor acknowledgement of messages (e.g., faults, state information); Conductor requests for additional information (e.g., diagnostic procedures, accidents along the railway track, or other points of interest along the railway track); and Any other information of interest relevant to a conductor's train operation.
[0053] The HMI provides a user interface (e.g., GUI) to a human user (e.g., conductor, operator). A human user may operate controls (e.g., buttons, levers, knobs, touch screen, keyboard) of the HMI module to provide information to the HMI module or to request information from the vision system. An operator may wear the user interface to the HMI module. The user interface may communicate with the HMI module via tactile operation, wired communication, and/or wireless communication. Information provided to a user by the HMI module may include: Recommended speed, Present speed, Efficiency score or index, Driver profile, Wayside signaling state, Stations of interest, Map view of inertial metrics, Fault messages, Alarms, Conductor interface for actuation of locomotive controls, and Conductor interface for acknowledgement of messages or notifications.
[0054] The VCD module performs communication (e.g., wired, wireless). The VCD module enables the PTC vision system to communicate with other devices on and off the train. The VCD module may provide Wide Area Network ("WAN") and/or Local Area Network ("LAN") communications. WAN communications may be performed using any conventional communication technology and/or protocol (e.g., cellular, satellite, dedicated channels). LAN communications may be performed using any conventional communication technology and/or protocol (e.g., Ethernet, WiFi, Bluetooth, WirelessHART, low power WiFi, Bluetooth low energy, fibre optics, IEEE 802.15.4e). Wireless communications may be performed using one or more antennas suitable to the frequency and/or protocols used.
[0055] The VCD module may receive information from the PTCC module. The VCD may transmit information received from the PTCC module. Information may be transmitted to headquarters (e.g., central location), wayside equipment, individuals, and/or other trains.
Information from the PTCC module may include: Packets addressed to other trains; Packets addressed to common backend server to inform operators of train location; Packets addressed to wayside equipment; Packets addressed to wayside personnel to communicate train location; Any node to node arbitrary payload; and Packets addressed to third party listeners of PTC vision system. [0056] The VCD module may also provide information to the PTCC module. The VCD may receive information from any source to which the VCD may transmit information.
Information provided by the VCD to the PTCC may include: Packets addressed from other trains; Packets addressed from common backend server to give feedback to a conductor or a train; Packets addressed from wayside equipment; Packets addressed from wayside personnel to communicate personnel location; Any node to node arbitrary payload; and Packets addressed from third party listeners of PTC vision system.
[0057] The GPS modules may include a conventional global positioning system ("GPS") receiver. The GPS module receives signals from GPS satellites and determines a geographical position of the receiver and time (e.g., UTC time) using the information provided by the signals. The GPS module may include one or more antennas for receiving the signals from the satellites. The antennas may be arranged to reduce and/or detect multipath signals and/or error. The GPS module may maintain a historical record of geographical position and/or time. The GPS module may determine a speed and direction of travel of the train. A GPS module may receive correction information (e.g., WAAS, differential) to improve the accuracy of the geographic coordinates determined by the GPS receiver. The GPS module may provide information to PTCC module. The information provided by the GPS module may include: Time (e.g., UTC, local); Geographic coordinates (e.g., latitude & longitude, northing & easting); Correction information (e.g., WAAS, differential); Speed; and Direction of travel.
[0058] The DAP may receive (e.g., determine, detect, request) information regarding a train, the systems (e.g., hardware, software) of a train, and/or a state of operation of a train (e.g., train state). For example, the DAP may receive information from the systems of a train regarding the speed of the train, train acceleration, train deceleration, braking effort (e.g., force applied), brake pressure, brake circuit status, train wheel traction, inertial metrics, fluid (e.g., oil, hydraulic) pressures, and energy consumption. Information from a train may be provided via a signal bus used by the train to transport information regarding the state and operation of the systems of the train. A signal bus includes one or more conventional signal busses such as Fieldbus (e.g., IEC 61158), Multifunction Vehicle Bus ("MVB"), wire train bus ("WTB"), controller area network bus ("CanBUS"), Train Communication Network ("TCN") (e.g., IEC 61375), and Process Field Bus ("Profibus"). A signal bus may include devices that perform wired and/or wireless (e.g., TTEthernet) communication using any conventional and/or proprietary protocol.
[0059] The DAP may further include any conventional sensor to detect information not provided by the systems of the train. Sensors may be deployed (e.g., attached, mounted) at any location on the train. Sensors may provide information to the DAP directly and/or via another device or bus (e.g., signal bus, vehicle control unit, wide train bus, multifunction vehicle bus). Sensors may detect any physical property (e.g., density, elasticity, electrical properties, flow, magnetic properties, momentum, pressure, temperature, tension, velocity, viscosity). The DAP may provide information regarding the train to the other modules of the PTC ecosystem via the PTCC module.
[0060] The DAP may receive information from any module of the PTC ecosystem via the PTCC module. The DAP may provide information received from any source to other modules of the PTC ecosystem via the PTCC module. Other modules may use information provided by or through the DAP to perform their respective functions.
[0061] The DAP may store received data. The DAP may access stored data. The DAP may create a historical record of received data. The DAP may relate data from one source to another source. The DAP may relate data of one type to data of another type. The DAP may process (e.g., format, manipulate, extrapolate) data. The DAP may store data that may be used, at least in part, to derive a signal state of the track on which the train travels, geographic position of the train, and other information used for positive train control.
[0062] The DAP may receive information from the PTCC module. Information received by the DAP from the PTCC module may include: Requests for train state data; Requests for braking interface state; Commands to actuate train behavior (speed, braking, traction effort); Requests for fault messages; Acknowledgement of fault messages; Requests to raise alarms in the train; Requests for notifications of alarms raised in the train; and Requests for wayside equipment state.
[0063] The DAP may provide information to the PTCC module. Information provided by the DAP to the PTCC module may include: Data from the signal bus of the train regarding train state; Acknowledge of requests; Fault messages on train bus; and Wayside equipment state.
[0064] The VA module detects the environment around the train. The VA module detects the environment through which a train travels. The VA module may detect the tracks upon which the train travels, tracks adjacent to the tracks traveled by the train, the aspect (e.g., appearance) of wayside (e.g., along tracks) signals (semaphore, mechanical, light, position), infrastructure (e.g., bridges, overpasses, tunnels), and/or objects (e.g., people, animals, vehicles). Additional examples include: PTC assets, ETCS assets, Tracks, Signals, Signal lights, Permanent speed restrictions, Catenary structures, Catenary wires, Speed limit Signs, Roadside safety structures, Crossings, Pavements at crossings, Clearance point locations for switches installed on the main and siding tracks, Clearance/ structure gauge/ kinematic envelope, Beginning and ending limits of track detection circuits in non-signaled territory, Sheds, Stations, Tunnels, Bridges, Turnouts, Cants, Curves, Switches, Ties, Ballast, Culverts, Drainage structures, Vegetation ingress, Frog (crossing point of two rails), Highway grade crossings, Integer mileposts, Interchanges,
Interlocking/control point locations, Maintenance facilities, Milepost signs, and Other signs and signals.
[0065] The VA module may detect the environment using any type of conventional sensor that detects a physical property and/or a physical characteristic. Sensors of the VA module may include cameras (e.g., still, video), remote sensors (e.g., Light Detection and Ranging), radar, infrared, motion, and range sensors. Operation of the VA module may be in accordance with a geographic location of the train, track conditions, environmental conditions (e.g., weather), speed of the train. Operation of the VA may include the selection of sensors that collect information and the sampling rate of the sensors.
[0066] The VA module may receive information from the PTCC module. Information provided by the PTCC module may provide parameters and/or settings to control the operation of the VA module. For example, the PTCC may provide information for controlling the sampling frequency of one or more sensors of the VA. The information received by the VA from the PTCC module may include: The frequency of the sampling, The thresholds for the sensor data, and Sensor configurations for timing and processing.
[0067] The VA module may provide information to the PTCC module. The information provided by the VA module to the PTCC module may include: Present sensor configuration parameters, Sensor operational status, Sensor capability (e.g., range, resolution, maximum operating parameters), Raw or processed sensor data, Processing capability, and Data formats.
[0068] Raw or processed sensor data may include a point cloud (e.g., two-dimensional, three-dimensional), an image (e.g., jpg), a sequence of images, a video sequence (e.g., live, recorded playback), scanned map (e.g., two-dimensional, three-dimensional), an image detected by Light Detection and Ranging (e.g., LIDAR), infrared image, and/or low light image (e.g., night vision). The VA module may perform some processing of sensor data. Processing may include data reduction, data augmentation, data extrapolation, and object identification.
[0069] Sensor data may be processed, whether by the VA module and/or the PTCC module, to detect and/or identify: Track used by the train, Distance to tracks, objects and/or infrastructure, Wayside signal indication (e.g., meaning, message, instruction, state, status), Track condition (e.g., passable, substandard), Track curvature, Direction (e.g., turn, straight) of upcoming segment, Track deviation from horizontal (e.g., declivity, acclivity), Junctions, Crossings, Interlocking exchanges, Position of train derived from environmental information, and Track identity (e.g., track ID).
[0070] The VA module may be coupled (e.g., mounted) to the train. The VA module may be coupled at any position on the train (e.g., top, inside, underneath). The coupling may be fixed and/or adjustable. An adjustable coupling permits the viewpoint of the sensors of the VA module to be moved with respect to the train and/or the environment. Adjustment of the position of the VA may be made manually or automatically. Adjustment may be made responsive to a geographic position of the train, track condition, environmental conditions around the train, and sensor operational status.
[0071] The PTCC utilizes its access to all subsystems (e.g., modules) of the PTC system to derive (e.g., determine, calculate, extrapolate) track ID and signal state from the sensor data obtained from the VA module. In addition, the PTCC module may utilize the train operating state information, discussed above, and data from the GPS receiver to refine geographic position data. The PTCC module may also use information from any module of the PTC environment, including the PTC vision system, to qualify and/or interpret sensor information provided by the VA module. For example, the PTCC may use geographic position information from the GPS module to determine whether the infrastructure or signaling data detected by the VA corresponds to a particular location. Speed and heading (e.g., azimuth) information derived from video information provided by the VA module may be compared to the speed and heading information provided by the GPS module to verify accuracy or to determine likelihood of correctness. The PTCC may use images provided by the VA module with position information from the GPS module to prepare map information provided to the operator via the user interface of the HMI module. The PTCC may use present and historical data from the DAP to detect the position of the train using dead reckoning, position determination may be correlated to the location information provided by the VA module and/or GPS module. The PTCC may receive communications from other trains or wayside radio transponders (e.g., balises) via the VCD module for position determination that may be correlated and/or corrected (e.g., refined) using position information from the VA module and/or the GPS module or even dead reckoning position information from the DAP. Further, track ID, signal state, or train position may be requested to be entered by the operator via the HMI user interface for further correlation and/or verification.
[0072] The PTCC module may also provide information and calls to action (e.g., messages, warnings, suggested actions, commands) to a conductor via the HMI user interface. Using control algorithms, the PTCC may bypass the conductor and actuate a change in train behavior (e.g., function, operation) utilizing the integration with the braking interface or the traction interface to adjust the speed of the train. PTCC handles the routing of information by describing the recipient(s) of interest, the payload, frequency, route and duration of the data stream to share the train state with third party listeners and devices.
[0073] The PTCC may also dispatch/receive packets of information automatically or through calls to action from the common backend server in the control room or from the railway operators or from the control room terminal or from the conductor or from wayside signaling or modules in the PTC vision system or other third party listeners subscribed to the data on the train.
[0074] The PTCC may also receive information concerning assets near the location of the moving vehicle. The PTCC may use the VA to collect data concerning PTC and other assets. The PTCC may also process the newly collected data (or forward it) to audit and augment the information in the backend database.
[0075] Algorithms: The Track Identification Algorithm (TIA), depicted in FIGs. 6 - 7 determines which track the rolling stock is currently utilizing. The TIA creates a superimposed feature dataset by overlaying the features from the 3D LIDAR scanners and FLIR Cameras onto the onboard camera frame buffer. The superset of features (global feature vector) allows for three orthogonal measurements and perspectives of the tracks. [0076] Thermal features from the FLIR Camera may be used to identify (e.g., separate, locate, isolate) the thermal signature of the railway tracks to generate a region of interest (spatial & temporal filters) in the global feature vector.
[0077] Range information from the 3D LIDAR scanner's 3D point cloud dataset may be utilized to identify the elevation of the railway track to also generate a region of interest (spatial & temporal filters) in the global feature vector.
[0078] Line detection algorithms may be utilized on the onboard camera, FLIR cameras and 3D LIDAR scanner's 3D point cloud dataset to further increase confidence in identifying tracks.
[0079] Color information from the onboard camera and the FLIR cameras may be used to also create a region of interest (spatial & temporal filter) in the global feature vector.
[0080] The TIA may look for overlaps in the regions of interest from multiple orthogonal measurements on the global feature vector to increase redundancy and confidence in track identification data.
[0081] The TIA may utilize the region of interest data to filter out false positives when the regions of interest do not overlap in the global feature vector.
[0082] The TIA may process the feature vectors in a region of interest to identify the width, distance, and curvature of a track.
[0083] The TIA may examine the rate at which a railway track is converging towards a point to further validate the track identification process; furthermore the slope of a railway track may also be used to filter out noise in the global feature vector dataset.
[0084] The TIA may take into consideration the spatial and temporal consistency of feature vectors prior to identifying the relative offset position of a train amongst multiple railway tracks.
[0085] Directional heading may be obtained by sampling the GPS receiver multiple times to create a temporal profile of movement in geographic coordinates.
[0086] The list of potential absolute track IDs may be obtained through a query to a locally cached GIS dataset or a remotely hosted backend server.
[0087] In a situation wherein the GPS receiver loses synchronization with GPS satellites, the odometer and directional heading may be used to calculate the dead reckoning offset. [0088] The TIA compares the relative offset position of the train among multiple railway tracks and references to the list of potential absolute track IDs to identify the absolute track ID that the train is utilizing.
[0089] After the TIA obtains an absolute track ID, the global feature vector samples may be annotated with the geolocation (e.g., geographic coordinate) information and track ID. This allows the TIA to utilize the global feature vector datasets to directly determine a track position in the future. This machine learning approach reduces the computational cost of searching for an absolute track ID.
[0090] The TIA may further match global feature vector samples from a local or backend database with spatial transforms. The parameters of the spatial transform may be utilized to calculate an offset position from a reference position generated from the query match.
[0091] Furthermore, the TIA may utilize the global feature vectors to stitch together features from multiple points in space or from a single point in space using various image processing techniques (e.g., image stitching, geometric registration, image calibration, image blending). This results in a superset of feature data that has collated global feature vectors from multiple points or a single point in space.
[0092] Utilizing the superset of data, the TIA can normalize the offset position for a relative track ID prior to determining an absolute track ID. This is useful when there are tracks outside the range of the vision apparatus (VA). This functionality is depicted in FIG. 10.
[0093] The TIA is a core component in the PTC vision system that eliminates the need for wireless transponders, beacons or balises to obtain positional data. TIA may also enable railway operators to annotate newly constructed railway tracks for their network wide GIS datasets that are authoritative in mapping the wayside equipment and infrastructure assets.
[0094] The Signal State Algorithm (SSA), described in FIG. 8, determines the signal state of the track a train is currently utilizing. The purpose of this component is to ensure a train's operation is in compliance with the expected operational parameters of the railway operators or modal control rooms or central control rooms. The compliance of a train's inertial metrics along a railway track can be audited in a distributed environment many backend servers or a centralized environment with a common backend server. A train's ability to obtain the absolute track ID is important for correlating the semaphore signal state to the track ID utilized by a train. Auditing signal compliance is possible once the correlation between the semaphore signal state and the absolute track ID is established. Placement of sensors is important for efficiently determining a semaphore signal state. FIG. 4 depicts one example wherein the 3D LIDAR scanner is forward facing and mounted on top of a train's roof.
[0095] The SSA takes into account an absolute track ID utilized by a train in order to audit the signal compliance of the train. Once the correlation of a track to a semaphore signal is complete, the signal state from that semaphore signal may actuate calls to action as feedback to a train or conductor.
[0096] Correlation of a railway track to a semaphore signal state may be possible by analyzing the regulatory specifications for wayside signaling from a railway operator. Utilizing the regulatory documentation, the spatial-temporal consistency of a semaphore signal may be compared to the spatial-temporal consistency of a railway track. A scoring mechanism may be used to choose the best candidate semaphore signal for the current railway track utilized by the train.
[0097] A local or remote GIS dataset may be queried to confirm the geolocation of a semaphore signal.
[0098] A local or remote signaling server may be queried to confirm the signal state in the semaphore signal matches what the PTC vision system is extrapolating.
[0099] Areas wherein the signal state is available to the train via radio communication may be utilized to confirm the accuracy of the PTC vision system and additionally augment the feedback provided to a machine learning apparatus that helps tune the PTC vision system.
[0100] A 3D point cloud dataset obtained from a PTC vision system may be utilized to analyze the structure of the semaphore signal. If the structure of an object of interest matches the expected specifications as defined by the regulatory body for a semaphore signal in that rail corridor, the object of interest may be annotated and added as a candidate for the scoring mechanism referenced above.
[0101] An infrared image captured through an FLIR camera may be utilized to identify the light being emitted from a wayside semaphore signal. In a situation where the red light is emitting from a candidate semaphore signal that is correlated to a track the train is currently on, a call to action will be dispatched to the HMI onboard the train for signal compliance. Upon a train's failure to comply with a semaphore signal that is correlated to a track the train is currently on, a call to action will be dispatched directly to the braking interface onboard the train for signal compliance.
[0102] The color spectrum in an image captured through the PTC vision system may be segmented to compute centroids that are utilized to identify blobs that resemble signal green, red, yellow or double yellow lights. A centroid's spatial coordinates and size of its blob may be utilized to validate the spatial-temporal consistency of the semaphore signal with specifications from a regulatory body.
[0103] A spatial-temporal consistency profile of a track may be created by analyzing the curvature of a track, spacing between the rails on a track, and rate of convergence of the track spacing towards a point on the horizon. A spatial-temporal consistency profile of a semaphore signal may be created by analyzing the following components: the height of a semaphore signal, the relative spatial distance between points in space, and the orientation and distance with respect to a track a train is currently utilizing.
[0104] The backend server may be queried to inform a train of an expected semaphore signal state along a railway track segment that the train is currently utilizing.
[0105] The backend server may be queried to inform a train of an expected semaphore signal state along a railway track segment identified by an absolute track ID and geolocation coordinates.
[0106] The Position Refinement Algorithm, as depicted in Fig. 3, provides a high confidence geolocation service onboard the train. The purpose of this algorithm is to ensure that loss of geolocation services does not occur when a single sensor fails. The PRA relies on redundant geolocation services to obtain the track position.
[0107] GPS or Differential GPS may be utilized to obtain fairly accurate geolocation coordinates.
[0108] Tachometer data along with directional heading information can be utilized to calculate an offset position.
[0109] A WiFi antenna may scan SSIDs along with signal strength of each SSID while
GPS is working and later use the Medium Access Control (MAC) addresses (or any unique identifier associated with an SSID) to quickly determine the geolocation coordinates. The signal strength of the SSID during the scan by a WiFi antenna may be utilized to calculate the position relative to the original point of measurement. The PTC vision system may choose to insert the SSID profile (SSID name, MAC address, geolocation coordinates, signal strength) as a reference point into a database based on the confidence in the current train's geolocation.
[0110] Global feature vectors created by the PTC vision system may be utilized to lookup geolocation coordinates to further ensure accuracy of the geolocation coordinates.
[0111] A scoring mechanism that takes samples from all the components described above would filter out for inconsistent samples that might inhibit a train's ability to obtain geolocation information. Furthermore, the samples may carry different weightage based on the performance and accuracy of each subcomponent in the PRA.
[0112] PTC VISION SYSTEM HIGH LEVEL PROCESS DESCRIPTION
[0113] In this section, we refer to the flowchart shown in Fig. 9. The PTC vision system samples the train state from the various subsystems described above. The train state is defined as a comprehensive overview of track, signal and on-board information. In particular the state consists of track ID, signal state of relevant signals, relevant on-board information, location information (pre- and post-refinement, reference PRA, TIA and SSA algorithms described above), and information obtained from backend servers. These backend servers hold information pertaining to the railroad infrastructure. A backend database of assets is accessed remotely by the moving vehicle as well as railroad operators and officers. The moving train and its conductor for example use this information to anticipate signals along the route. Operator and maintenance officers have access to track information for example. These reports and notifications are relevant to signals and signs, structures, track features and assets, safety information.
[0114] After collecting this state, the PTC vision system issues notifications (local or remote), possibly raises alarms on-board the train, and can automatically control the train's inertial metrics by interfacing with various subsystems on-board (e.g., traction interface, braking interface, traction slippage system).
[0115] SENSORY STAGE
[0116] On-board data: The On-board data component represents a unit where all the data extracted from the various train systems is collected and made available. This data usually includes but is not limited to: Time information, Diagnostics information from various onboard devices, Energy monitoring information, Brake interface information, Location information, Signaling state obtained from train interfaces to wayside equipment, Environmental state obtained through the VA devices on board or on other trains, and Any other data from components that would help in Positive Train Control.
[0117] This data is made available within the PTC vision system for other components and can be transmitted to remote servers, other trains, or wayside equipment.
[0118] Location data is strategic to ensure that trains are operating within a safety envelope that meets the Federal Railroad Administration's PTC criteria. In this regard, wayside equipment is currently being utilized by the industry to accurately determine vehicle position. The output of location services described above (e.g., TIA & SSA) provides the relative track position based on computer vision algorithms.
[0119] The relative position can be obtained through using a single sensor or multiple sensors. The position we obtain is returned as an offset position, usually denoted as a relative track number. Directional heading can also be a factor in building a query to obtain the absolute position from the feedback to the train.
[0120] The absolute position can be obtained either from a cached local database, or cached local dataset, remote database, remote dataset, relative offset position using on board inertial metric data, GPS samples, Wi-Fi SSIDs and their respective signal strength or through synchronization with existing wayside signaling equipment.
[0121] The various types of datasets we use include but are not limited to: 3D point cloud datasets, FLIR imaging, Video buffer data from on-board cameras.
[0122] Once the location is known, this information can be utilized to correlate signal state from wayside signaling to the corresponding track. The location services can also be exposed to third party listeners. The on board components defined in the PTC vision system can act as listeners to the location services. In addition, the train can scan the MAC IDs of the networked devices in the surrounding areas and utilize MAC ID filtering for any application these networked devices are utilizing. This is useful for creating context aware applications that depend on the pairing the MAC ID of a third party device (e.g., mobile phones, laptops, tablets, station servers, and other computational devices) with a train's geolocation information.
[0123] The track signal state is important for ensuring the train complies with the PTC safety envelope at all times. The PTC vision system's functional scope includes extrapolating the signal value from wayside signaling (semaphore signal state). In this regard, the communication module or the vision apparatus may identify the signal values of the wayside equipment. In areas where the signal is not visible, a central back end server can relay the information to the train as feedback. When wayside equipment is equipped with radio communication, this information can also augment the vision-based signal extrapolation algorithms (e.g., TIA & SSA). Datasets are used at the discretion of the PTC vision system.
[0124] Utilizing datasets collected by the PTC vision system, one can identify the features of the track from the rest of the data in the apparatus and identify the relative track position. The relative track position along with directional heading information can be sent to a backend server to obtain the absolute track ID. The absolute track ID denotes the track identification as listed by the operator. This payload is arbitrary to the train, allowing seamless operations amongst multiple operators without having an operator specific software stack on the train. Operator agnostic software allows trains to operate with great interoperability, even if it is traveling through infrastructures from different rail operators. Since the payloads are arbitrary, the trains are intrinsically inter-operable even when switching between rail-operators. As the rolling stock travels along the track, data necessary for updating asset information is generated by the vision apparatus. This data then gets processed to verify the integrity of certain asset information, as well as update other asset information. Missing assets, damaged assets or ones that have been tampered with can then be detected and reported. The status of the infrastructure can also be verified, and the operational safety can be assessed, every time a vehicle with the vision apparatus travels down the track. For example, clearance measurements are performed making sure that no obstacles block the path of trains. The volume of ballast supporting the track is estimated and monitored over time.
[0125] Backend:
[0126] The backend component has many purposes. For one, it receives, annotates, stores and forwards the data from the trains and algorithms to the various local or remote subscribers. The backend also hosts many processes for analyzing the data (in real-time or offline), then generating the correct output. This output is then sent directly to the train as feedback, or relayed to command and dispatch centers or train stations.
[0127] Some of the aforementioned processes can include: Algorithms to reduce headways between trains to optimize the flow on certain corridors; Algorithms that optimize the overall flow of the network by considering individual trains or corridors; and Collision avoidance algorithms that constantly monitor the location and behavior of the trains. [0128] The backend also hosts the asset database queried by the moving train to obtain asset and infrastructure information, as required by rolling stock movement regulations. This database holds the following assets with relevant information and features: PTC assets, ETCS assets, Tracks, Signals, Signal lights, Permanent speed restrictions, Catenary structures, Catenary wires, Speed limit Signs, Roadside safety structures, Crossings, Pavements at crossings, Clearance point locations for switches installed on the main and siding tracks, Clearance/ structure gauge/ kinematic envelope, Beginning and ending limits of track detection circuits in non-signaled territory, Sheds, Stations, Tunnels, Bridges, Turnouts, Cants, Curves, Switches, Ties, Ballast, Culverts, Drainage structures, Vegetation ingress, Frog (crossing point of two rails), Highway grade crossings, Integer mileposts, Interchanges, Interlocking/control point locations, Maintenance facilities, Milepost signs, and Other signs and signals.
[0129] The rolling stock vehicle utilizes the information queried from the database to refine the track identification algorithm, the position refinement algorithm and the signal state detection algorithm. The train (or any other vehicle utilizing the machine vision apparatus) moving along/in close proximity to the track collects data necessary to populate, verify and update the information in the database. The backend infrastructure also generates alerts and reports concerning the state of the assets for various railroad officers.
[0130] FEEDBACK STAGE
[0131] Automatic Control:
[0132] There are several ways with which the train can be controlled using the PTC vision system (e.g., Applications in Fig 5). The output of the sensory stage might trigger certain actions independently of the any other system. For example, upon the detection of a red-light violation, the braking interface might be triggered automatically to attempt to bring the train to a stop.
[0133] Certain control commands can also arrive to the train through its VCD. As such, the backend system can for example instruct the train to increase its speed thereby reducing the headway between trains. Other train subsystems might also be actuated through the PTC vision system, as long as they are accessible on the locomotive itself.
[0134] Onboard Alarms:
[0135] Feedback can also reach the locomotive and conductor through alarms. In the case of a red-light violation for example, an alarm can be displayed on the HMI. The alarms can accompany any automatic control or exist on its own. The alarms can stop by being acknowledged or halt independently.
[0136] Notifications (Local/Remote):
[0137] Feedback can be in the form of notifications to the conductor through the user interface of the HMI module. These notifications may describe the data sensed and collected locally through the PTC vision system, or data obtained from the backend systems through the VCD. These notifications may require listeners or may be permanently enabled. An example of a notification can be about speed recommendations for the conductor to follow.
[0138] Backend architecture and data processing.
[0139] The backend may have two modules: data aggregation and data processing. Data aggregation is one module whose role is to aggregate and route information between trains and a central backend. The data processing component is utilized to make recommendations to the trains. The communication is bidirectional and this backend server can serve all of the various possible applications from the PTC vision system.
[0140] Possible applications for PTC vision system include the following: Signal detection; Track detection; Speed synchronization; Extrapolating interlocking state of track and relaying it back to other trains in the network; Fuel optimization; Anti-Collision system; Rail detection algorithms; Track fault detection or preventative derailment detection; Track performance metric; Image stitching algorithms to create comprehensive reference datasets using samples from multiple runs; Cross Train imaging for, e.g., Preventative maintenance, Fault detection, and/or Vibration signature of passerby trains; Imaging based geolocation or geofiltering services; SSID based geolocation or geofiltering; and Sensory fusion of GPS + Inertial Metrics + Computer Vision-based algorithms.
[0141] In accordance with other embodiments, remote sensing and localization features can be utilized to implement run-time systems in automotive vehicles, such as autonomously driving cars. Figure 25 is a schematic block diagram of an exemplary in-vehicle system for vehicle localization and/or control. In-vehicle runtime engine ("IVRE") 2500 and vehicle decision engine 2510 are computation and control modules, typically microprocessor-based, implemented locally on board a vehicle. Local 3D map cache 2530 stores map data associated with the area surrounding the vehicle's rough position, as determined by GPS and EVIU sensors 2520, and can be periodically or continuously updated from a remote map store via communications module 2540 (which may include, e.g., a cellular data transceiver). Machine vision sensors 2550 may include one or more mechanisms for sensing a local environment proximate the vehicle, such as LiDAR, video cameras and/or radar.
[0142] In operation, IVRE 2500 implements vehicle localization by obtaining a rough vehicle position from onboard GPS and IMU sensors 2520. Machine vision sensors 2550 generate environmental signatures indicative of the local environment surrounding the vehicle, which are passed to IVRE 2500. IVRE 2500 queries local 3D map cache 2530 using
environmental signatures received from machine vision sensors 2550, to match features or objects observed in the vehicle's local environment to known features or objects having known positions within 3D semantic maps stored in cache 2530. By comparing the vehicle's observed position relative to local features or objects, with the position of those features and objects on maps, the vehicle's position can be refined with significantly more accuracy than typically possible using GPS - with margin of error potentially measured in centimeters.
[0143] Detailed vehicle position and other observed or calculated information can be utilized to implement other functionality, such as vehicle control and/or map auditing. For example, data from machine vision sensors 2550 can be analyzed using graphs and other data analysis mechanisms, as described elsewhere herein, for IVRE 2500 to determine a centerline for a lane in which the vehicle is traveling. IVRE 2500 can also operate to obtain semantics (such as events and triggers) along the vehicle's route. Available compute resources can be used to audit centralized map data sources by comparing previously-observed asset information obtained from centralized maps (and, e.g., stored in local 3D map cache 2530) to asset information derived from real time data captured by machine vision sensors 2550. IVRE 2500 can thereby identify errors of omission (i.e. observed assets omitted from centralized map data) as well as errors of commission (i.e. assets in centralized map data that are not observed by machine vision sensors 2550). Such errors can be stored in cache 2530, and subsequently communicated to a central map repository via communications module 2540.
[0144] In some embodiments, auditing of map data by a local vehicle may be initiated by a centralized control server, communicating with the vehicle via communications module 2540. For example, if the time elapsed since last auditing of a map section exceeds a threshold, a centralized control server can request auditing from a local vehicle traveling through the target region. In another example, if one vehicle reports discrepancies between centralized map data and locally-observed conditions, the centralized control server may request confirmation auditing by one or more other vehicles moving within the area of the discrepancy. Auditing requests may pertain to various combinations of geographic regions and/or mapping layers.
[0145] In some embodiments, it may be desirable to utilize information such as precise vehicle position, assets and semantics, and navigation information, as inputs to vehicle decision engine 2510. Vehicle decision engine 2510 can operate to control various other systems and functions of the vehicle. For example, in an autonomous driving implementation, vehicle decision engine 2510 may utilize lane center line information and precise vehicle position information in order to steer the vehicle and maintain a centered lane position. These and other vehicle control operations may be beneficially implemented using systems and processes described herein.
[0146] SEMANTIC MAP CREATION USING GEOSPATIAL DATA
[0147] Maps are collections of objects, their location and their properties. Maps can be divided into layers, where each layer is a grouping of objects of the same type. The location of each object is defined, along with a geometric attribute (example: the location of a pole could be a point in three-dimensional space, whereas a signal can be located by drawing a polygon around it). A map becomes "semantic" when the semantic associations between different objects and layers are also recorded. For example, a map composed of the centerlines of various lanes on a roadway as well as the signs located around the infrastructure is labeled semantic, when the associations between the various signs and centerlines are recorded. This can be achieved by creating a mapping between the unique identifier of a sign and the unique identifiers of the lanes to which the sign is relevant. The semanticization of a map creates more context for the vehicle or user consuming the map. The semantic map can also be packaged with regulatory information from various transportation authorities.
[0148] Any asset's physical geometry can be described in a map. Geometric features used to describe shapes include points, lines, polygons, and arcs. The features are typically in three dimensions, but they can be projected into two-dimensional spaces where depth/elevation is lost. In general, semantic maps can be recorded and delivered in different coordinate and reference frames. There are also transformations allowing to project maps from one coordinate reference frame to the next. These maps can be packaged and delivered in different formats. Common formats include GeoJSON, KML, shapefiles, and the like. [0149] In some embodiments, the geospatial data used for semantic map creation comes from LiDAR, visible spectrum cameras, infrared cameras, and other optical equipment. The act of obtaining machine vision data for map creation, where this data is georeferenced to a particular location on the planet, is called surveying. The output is a set of data points in three dimensions, along with images and video feeds in the visible spectrum and other frequencies. There can be many different hardware platforms for data collection. The collection vehicle is also variable (aerial, mobile, terrestrial). The geospatial data is collected initially with the collection vehicle being the origin of the reference frame. By locating the vehicle throughout the survey (using, e.g., an Inertial Measurement Unit (IMU) and Global Positioning Systems (GPS)), the images, laser scans and video feeds are then registered to a fixed reference frame which which is georeferenced. The data generated in the survey can be streamed or saved locally for later consumption.
[0150] Some embodiments of the vehicle localization and local environment sensing systems described herein benefit from use of point cloud survey data. Semantic maps derived from point cloud survey data may provide a vehicle with high levels of detail and information regarding the vehicle's current or anticipated local environment, which may be used, for example, to assist in relative vehicle localization, or serve as input data to autonomous control decision-making systems (e.g. automated braking, steering, speed control, etc.). Additionally, or alternatively, point-cloud data measured by a vehicle may be compared to previously-measured point cloud data to detect conditions or changes in a local environment, such as a fallen tree, overgrown vegetation, changed signage, lane closures, track or roadway obstructions, or the like. The detected changes in the environment can be used to further update the semantic maps.
[0151] However, increasing levels of point cloud survey data detail can result in extremely large datasets, which may be costly or time consuming for a service provider to process, or for a vehicle to store or process. For example, LiDAR-based 3D railroad surveying systems traveling linearly along a rail track may generate over 20 GB of geospatial data for every kilometer of scanning. The raw point cloud data generated by LiDAR scanning typically then requires additional processing to extract useful asset information.
[0152] Three dimensional semantic maps are traditionally created from point cloud data and other geospatial data through the use of 3D visualization software. Figure 11 A illustrates a typical prior art process for extracting asset information from point cloud data. In step SI 100, surveying procedures generate point cloud data sets, such as using a LiDAR surveying apparatus. In step SI 105, the raw point cloud data is visualized. Typically, Geographical Information Systems (GIS) analysts use point-and-click methods to manually identify, annotate, and classify critical assets within the data. The first step in the GIS analysts' process is to separate the terabytes of point cloud data into smaller manageable sections. This is due to the fact that contemporary personal computers are limited (memory/computational power) and are unable to manage the terabytes of LiDAR data at once.
[0153] Subsequently, the GIS analysts use 3D visualization software to traverse each of the smaller sections of point cloud. As they progress through their respective sections, the GIS analysts delineate and annotate the important assets. Finally, the annotated assets of each GIS analyst are combined into one map (step SI 110). Varying file formats and software systems can create additional difficulties in merging the separate datasets.
[0154] Extracting value from point-cloud data is limited by both the prior art process and the infrastructure. Point-and-click annotation is manual, slow and prone to error. Additionally, conventional file-based systems prevent GIS developers and administrators from effectively managing the growing point cloud datasets.
[0155] Figure 1 IB illustrates an alternative approach to extracting asset information from raw point cloud data. In step SI 150, surveying is conducted to generate the raw point cloud data. In step SI 155, asset maps are generated directly from the raw point cloud data, without requiring visualization of the large, complex data set, or manual annotation of that data.
[0156] Figure 12 illustrates a computing apparatus for rapidly and efficiently extracting asset information from large point-cloud data sets. Figure 13 illustrates a process for using the apparatus of Figure 12. Preferably, the components within the apparatus of Figure 12 are implemented using Internet-connected cloud computing resources, which may include one or more servers. Front-End component 1200 includes data upload tool 1205, configuration tool 1210, and map retrieval tool 1215. Front-End component 1200 provides a mechanism for end users to interact with and control the computing apparatus.
[0157] Using data upload tool 1205, a user can upload LiDAR and other surveying data from a local data storage device to data storage component 1220 (step S1300). Data storage component 1220 may implement a distributed file system (such as the Hadoop Distributed File System) or other mechanism for storing data. Configuration tool 1210 can be accessed via a user's network-connected computing device (not shown), and enables a user to define the format of uploaded data as well as other survey details, and specify assets to search for and annotate (step SI 305). After a user interacts with configuration tool 1210 to select desired assets, the user is provided with various options to configure the output map format. Preferably, configuration tool 1210 then solicits a desired turnaround time from a configuring user, and presents the user with an estimated cost for the analysis (step S1310). The cost estimate is determined based on, e.g., the size of the uploaded data set to be analyzed, the number (and complexity) of selected assets, the output format, and the selected turnaround time. Finally, when configuration is complete, the user interacts with configuration tool 1210 to initiate an analysis job (step S1315).
[0158] The geospatial data uploaded through front end 1200 is tracked in database collections. This data is organized by category, geographic area, and other properties. As the data evolves through various stages of execution, the relevant database entries get updated.
[0159] Point-cloud data uploaded through the front-end tool is stored in a secure and replicated manner. To simplify retrieval, the data is tiled into different size tiles in a Cartesian coordinate system. The tiles themselves are limited in two dimensions and namespaced accordingly. Preferably, tiles are limited in X and Y dimensions, and unlimited in a Z dimension that is vertical or parallel to the direction of the Earth's gravitational pull, such that a tile defines a columnar area, unlimited in height (i.e. limited only to the extent of available geospatial data) and having a rectangular cross-section. In an exemplary implementation, tiles which are 1000m on the side (in the horizontal plane) can be utilized. The files representing the tiles would then hold all the points which belong to the particular geographic area delimited by the tile, and no other. In certain embodiments, tree structures (such as quadtrees and octrees) are implemented depending on the traversal style for the data.
[0160] Processing of the data to automatically extract semantic maps from geospatial data occurs on computation clusters, implemented within processing unit 1240 (embodiments of which are described further with reference to Figure 16, below). These have access to the point cloud and other data through the network accessible storage unit 1220. Intermediary results as well as finalized ones are stored similarly.
[0161] Figure 14 illustrates a process that may be performed by the apparatus of Figure
12 upon initiation of an analysis job. In order to simplify data processing, and enable implementation of a MapReduce data analysis framework, the point-cloud data is subdivided into chunks (step S1400) by data storage / preprocessing component 1220. These chunks can be subsets of tiles or combinations thereof, potentially selected to optimize for, e.g., the desired processing method, available memory and other runtime considerations. Individual nodes in the computation cluster (i.e. within processing unit 1240) are then capable of processing geospatial and other data associated with a given data chunk, i.e., selected subsets or combinations of tiles.
[0162] The density of the point-cloud may be an important factor in determining the number of tiles (or the size of tile subsets) to process within the same computation node. In an exemplary embodiment, Figure 15 illustrates the size of tiles with respect to the number of points within (represented by the diagonal line), as well as the distribution of tiles sizes for an exemplary dataset comprising LiDAR point-cloud data measured along a 2km section of railway (each tile represented by hatches across the diagonal line). Data storage and preprocessing component 1220 performs tile aggregation, and/or subdivision, prior to feeding data to processing unit 1240, in order to optimize the analysis performance.
[0163] Given the benefits of tile aggregation, as described above, having a reduced point- cloud density can result in reduced processing times. However, low densities generally make the feature detection process more difficult, and can result in higher rates of false positives. The richer the point-cloud data, the more accurate the detection process becomes.
[0164] Once processing is initiated, job scheduler 1225 creates a queue containing tasks pertaining to the job, as configured in steps S1305 and S1310. Job scheduler 1225 associates one or more of analysis mechanisms 1250 (typically implementing various different data analysis algorithms) with the task (step S1405), and creates a cluster of machines within processing unit 1240 to process the data (step S1410). The size of the cluster (i.e. the number of computation nodes) may be determined to satisfy the turnaround time requested in step S1310, given the previously-measured average time for a single node to implement the require data analysis mechanism(s) 1250 on a tile aggregation of known average size (e.g. 250MB). For example, consider a sample dataset submitted for processing, estimated to take about 240 hours of compute time on an eight-core desktop computer. Since data analysis mechanisms 1250 are preferably designed to run concurrently, job scheduler 1225 can initiate a cluster of 20 machines with four cores each, and process the same dataset in approximately 24 hours instead.
[0165] Processing unit 1240 is composed of a collection of compute clusters. The size of the cluster depends on the number of jobs. Figure 16 illustrates an exemplary compute cluster. Each cluster contains: a master instance 1605, responsible for managing the cluster; a set number of principal computation nodes 1610, which also store data in data storage system 1220; and a variable number of "spot" instances 1620. In some embodiments, it may be desirable to size principal instances 1610 to be capable of processing the entirety of the data and meeting the turnaround time requirement, with spot instances 1620 activated based on, e.g., their cost and/or job time constraints. In other embodiments, compute clusters consisting entirely of spot instances, or entirely of principal nodes, may be utilized.
[0166] Once an appropriately-configured compute cluster is generated, data storage and preprocessor component 1220 directs a stream of data chunks (e.g. aggregations of tiles satisfying a desired data subset size) to processing unit 1240 (step S1415). Principal nodes and spot instances within processing unit 1240 execute appropriate data analysis mechanisms 1250 to, e.g., extract asset or feature information from the 3D point-cloud tiles.
[0167] Once the dataset has been processed by processing unit 1240 and the desired information extracted, map generator 1230 is triggered. Map generator 1230 combines the output of nodes within processing unit 1240 into semantic maps (step S1420). Reporting analytics can be derived from the semantic maps by running queries to analyze particular assets and their combinations.
[0168] Map generator 1230 may also include an annotation integrity verifier operating to verify the integrity of annotated datasets over time. In some applications, locations may be surveyed repeatedly at different times. For example, in railway applications, trains equipped with LiDAR or other railway surveying vehicles may periodically survey the same length of railway, such as to monitor the health or status of assets along a track. In some roadway applications, LiDAR-equipped survey vehicles may travel along a given portion of road at different times. In other roadway applications, data captured by LiDAR equipped automobiles, such as autonomous driving cars, may be regularly analyzed, providing potentially frequent analyses of the local environment in a given location. Each time a new map is generated by map generator 1230 concerning a given area, asset or local feature information can be compared to such information contained in older maps. Alarms, notifications or events can be triggered when discrepancies are detected.
[0169] The output of map generator 1230 is ultimately made available to the user, via front end 1200 and map retrieval tool 1215 (step S1425). Once a job is completed and a map is generated, scheduler 1225 (monitoring the status of tasks and jobs) generates notifications for the end user.
[0170] Feature maps (containing only the location, geometry and features of various assets), as well as semantic ones can also be stored in remotely accessible geodatabases. The map data can be retrieved either directly or through a server to facilitate the querying and collection of results. The maps can be retrieved in their entirety or by selecting a specific area of interest.
[0171] Security, Compression and Integrity
[0172] The security of the data and maps may be an important aspect of many embodiments. Preferably, data upload step SI 300 employs end-to-end encryption (such as AES encryption) from the user data source to the cloud computing platform. Such encryption may also be utilized for communications between a user's system and front-end 1200.
[0173] In some embodiments, it may be desirable to store raw point cloud data within data storage component 1220 in a compressed format. For example, an exemplary distributed compute cluster having one terabyte of storage for every four central processing unit (CPU) cores, storing the 3D point-cloud data in its raw form may lead to slower processing times because the storage infrastructure would be I/O bound while the CPU cores sometimes sit idle. This means the CPUs would essentially wait for data to be read from storage, before processing it. Compressing the raw point-cloud data before storing it allows the system to spend less time reading and writing data to disk. Therefore, data storage component 1220 may include a compression mechanism to compress point-cloud data before storage.
[0174] However, by storing compressed raw point cloud data, processing time is increased, because the data must be decompressed by a decompression mechanism before applying data analysis mechanisms 1250. Typically, there is a positive relationship between the compression ratio of compressed data, and the amount of processing time required to compress and decompress the data. Therefore, it may be desirable in some embodiments to continually measure CPU time and modulate data compression ratios to balance, as closely as possible, the rate at which data can be read from storage component 1220, and the rate at which that data can be uncompressed and processed by processing unit 1240.
[0175] Many lossless data compression mechanisms may be utilized to treat large point- cloud datasets, as described herein. Examples include LempelZivOberhumer (LZO), GZIP (also based on LempelZiv methods), and LASzip (released by rapidlasso GmbH, and hereafter referred to as LAZ). Figures 17, 18 and 19 show a comparative analysis of these three compression mechanisms. In terms of compression, the LAZ method presents a constant CPU time across all compression levels (the higher the compression level, the smaller the compressed output file). This method is very attractive since it results in smaller file sizes when compared to LZO and GZIP. LZO and GZIP, however, are optimized for decompression, and therefore present a superior alternative to LAZ in terms of CPU time required for decompression. In some embodiments, it may be desirable to speed up data processing while minimizing storage requirements by selecting a compression mechanism from amongst multiple mechanisms having different characteristics, based on the nature of dataset and the characteristics (such as cost and availability) of available computing infrastructure.
[0176] MACHINE VISION ANALYSIS MECHANISMS
[0177] Data analysis mechanisms 1250 are typically selected based on the nature of the information desired to be extracted from the point-cloud data. It may be desirable to design mechanisms 1250 with very low false positive rates, while maintaining acceptable detection rates. For added confidence in generated maps, in some applications, a subset of results may be verified manually by inspecting the original point-cloud and raw imaging data.
[0178] Track Detection and Traversal
[0179] In embodiments processing railway point-cloud survey data, track detection may be an important first step. Track detection can be important because knowledge of the track position facilitates identification of assets, since regulations often assign specific locations for each asset in relation to the track.
[0180] Figure 20 illustrates a process for track detection and traversal that can be implemented by processing unit 1240, e.g. in step S1415 of Figure 14. In step S2000, a 100m x 100m section of point-cloud data is identified for analysis. In step S2010, the geometry of the 10,000m2 point cloud section is analyzed to extract a subset of points which are associated with the track. Many techniques can be employed to achieve the desired result. In some
embodiments, previously-classified tracks from similar data sets can be studied to identify properties of data in the vicinity of the tracks, with those properties serving as an indicia of track location in newly-analyzed data. Other techniques include projecting points in two-dimensional space (based on, e.g., height or pulse intensity) and utilizing edge detection mechanisms and transforms to isolate regions belonging to the track. In an exemplary use case, the 10,000 m2 point cloud section in step S2000 may consist of about 1 GB of data, while the extracted track subset output in step S2010 may consist of about 1 MB of data.
[0181] Figure 21 is a visualization of the 10,000 m2 point cloud section input to step
S2000, and the extracted rail data output in step S2010. Lines 2100 represent track that is visible in the point-cloud. Line 2110 represent track that was obscured during the LiDAR data collection process, having a position that is estimated. This is typically the result of shadowing, a process which occurs when the object of interest is hidden from direct line of sight of the measuring instrument. Dots 2120 correspond to problematic positioning of a LiDAR tripod system which resulted in some track sections being obstructed. The location of the invisible track can be inferred by utilizing known spatial continuity properties of the infrastructure (such as spacing relative to other observed elements) (step S2020).
[0182] Geospatial data presents many dimensionalities that can be taken advantage of during asset extraction. Imagery, infrared, video feeds and/or multispectral sensors can be combined to increase detection confidence and accuracy. Most LiDAR systems include an intensity measurement for each point. By analyzing the intensity of points both on and off the track, classification mechanisms and filters can be added to the system, for an increased track detection rate. Figure 22 is a histogram of point-cloud intensity levels in an exemplary track detection implementation. Figure 22a illustrates quantity of each measured intensity level in an analyzed body of point cloud data, as a whole. Figure 22b illustrates the same histogram, for points within the point cloud identified as corresponding to track. A simple band pass filter can be effective in some cases to further narrow a search space for points belonging to the rail. Other classification methods can also be utilized.
[0183] Figure 23 is a visualization of a portion of the output of an implementation including a track detection mechanism and other asset detection mechanisms. Via operation of the track detection mechanism, track segments 2300 are identified first, then for each track, centerline markers 2310 are established. Once the tracks and track centerlines are identified, subsequent analysis components can traverse the track within the point-cloud data, while enjoying a 360 degree view of high resolution point cloud data around each point in the centerline. [0184] Other analysis mechanisms identify and locate other assets or features for inclusion in a sematic map. For example, an overhead wire detection mechanism identifies and locates overhead wires, and demarcates them with overhead wire centerline indicia 2320. A pole detection mechanism identifies and locates trackside poles, and locates them with indicia 2330. These and other features may be included in semantic map output generated via the systems and methods described herein.
[0185] In some embodiments, analysis mechanisms may be applied sequentially, with an output of one mechanism serving as an input to another mechanism. For example, in railway applications, assets and elements of the local environment regularly are replaced, added, removed or shifted. It may be desirable to regularly check clearance above and around a track to ensure safe operation, and that train cars do not come into contact with any obstructions. In such an application, a track detection mechanism, such as that described above, may be implemented as part of a sequence of analysis mechanisms. The output of a track detection mechanism that includes the track centerline may be subsequently used as an input to a track clearance check mechanism. A bounding box is defined with respect to the track center line, and any objects that encroach within that bound are reported. The dimensions of the bounding box can be modified to fit various standards.
[0186] Determining the location of signs, signals, switches, wayside units, and the like is also possible using the detection framework. Once localized, the classification of these assets is rendered possible given the geometric features of each asset, according to manufacturer's specifications or other object definitions.
[0187] Another analysis mechanism that may be beneficially employed in a railway application is overhead line inspection. Overhead wires can be identified within point-cloud data. The height of the wire in comparison with the track is assessed. Areas with saggy lines are reported. By using pole location information, the catenary shape of the wire can also be assessed.
[0188] While certain analysis components are described in the context of railway track detection, it is contemplated and understood that similar analysis mechanisms and methods may be utilized to identify other types of assets, potentially in other applications. For example, mechanisms analogous to the track detection mechanism described herein may be useful in a roadway context for identifying lane markings and/or curbs. [0189] Computing Paradigms
[0190] The automated extraction of maps can be achieved by combining computation blocks into directed acyclic graphs (hereafter referred to as "graphs"). The blocks contained in these graphs have a varying degree of complexity, ranging from simple averaging and thresholding to transforms, filters, decompositions, etc. The output of one stage of the graph can feed into any other subsequent stage. The stages need not run in sequence but can be parallelized given sufficient information per stage. When creating feature maps, a graph is generally used to classify points within a point cloud belonging to the same category, or to vectorize. Vectorization refers to the creation of an (often imaginary) line or polygon going through a set of points delimiting their center, boundary, location, etc. As such, computation graphs can be used to implement classifiers, clustering methods, fitting routings, neural networks and the like.
Rotations and projections are also used, often in conjunction with machine vision processing techniques.
[0191] To take full advantage of distributed computing, the creation of semantic maps from geospatial data may be parallelized. There are many levels of parallelization that can be implemented. At the highest level, the survey data can be divided into regularly-shaped regions of interest which get streamed to different machines and CPU processes. The results coming from each area need to then be merged in a "reduce" step once all the processes finish, similarly to the process of Figure 14. Since boundary conditions arise, padding the regions of interest with extra data which is truncated at the end of the process usually removes those deformities near the edges. The size of the region of interest, as well as the padding thickness is determined by the graph extracting the assets or features.
[0192] At another level, parallelism can occur when processing is taking place along a pre-extracted vector. For example, when searching for signs in the vicinity of a railroad track, the data can be traversed by extracting regions around waypoints along the previously extracted track centerline. Multiple processes can then be used in parallel along different waypoints of the track.
[0193] Finally, when analyzing a particular region, each point can be considered individually. In this traversal method, a voxel surrounding that point is usually extracted and analyzed. This process can also be made parallel, in those cases when the outcome of one point's operation does not affect that of any other point. [0194] These are some of the traversal methodologies employed in the map creation process, and some of the ways in which data processing can be made parallel. In addition, the use of GPU (graphics processing units), in conjunction with the conventional CPUs also carries great speed improvements and can further assist in reducing turnaround times.
[0195] Geospatial data is not limited to point cloud, but extends to imagery, video feeds, multispectral data, RADAR, etc. For increased mapping accuracy and correctness, some embodiments may utilize any additional data sources that are available. Several techniques can be utilized for using data from different sources. In some embodiments, datasets can be combined in a pre-processing stage (e.g. step S1400), before feeding into the computation graphs. This approach provides computation graphs with data from multiple sources for processing. In other embodiments, one set of data may be used to generate a hypothesis concerning an asset and its properties; data from other sources can then be used to validate and/or augment the hypothesis via other analysis mechanisms.
[0196] Machine Learning:
[0197] Many machine learning techniques can be implemented to assist in the semantic map creation process. Existing annotated maps can be used to train graphs and optimize them, to automatically generate accurate semantic maps from geospatial data. The input data to the machine learning system is comprised of survey data, as well as the corresponding, annotated output maps. The output of the machine learning system is a refined graph, which can then be applied to more extensive survey data, in order to extract maps at scale. In some instances, classified point clouds (where a category is assigned to each point based on which asset it belongs to) are used to feed into the training process. In others, vectorized maps are used to learn the map creation process and tune the processing graphs. These methods fall under the supervised learning category, relying on evaluating performance (through error measurement) and reinforcement of desirable performance.
[0198] Figure 24 illustrates an embodiment of a system implementing supervised machine learning, including training component 2400 and map generation component 2410. Training component 2400 receives as inputs, raw point cloud data 2420 and sample output 2422. In some circumstances, sample output 2422 may be verified output data associated with approximately 1% of the total data set. Sample output 2422 may include classified point cloud data (where points belonging to a particular asset category are grouped together), and/or a vectorized map (with points, lines and polygons drawn over assets of interest). Training component output 2424 defines an optimized categorization mechanism, such as algorithm coefficients for an analysis mechanism comparable to mechanisms 1250 in the map generation system of Figure 12. Training component output 2424 may also define a region of interest for the algorithms to be most effective, define functional blocks within a computation graph which should be utilized, and/or define features of interest for a particular asset under consideration. Training component output 2424 is fed into map generation component 2410, along with the full corpus of raw point cloud data 2420. Map generation component 2410 then operates to generate map output 2426.
[0199] Unsupervised methods can also be implemented for generating maps. Such processes can rely on scale-dependent features to describe contextual information for individual map points. They can also rely on deep learning to design feature transformations for use with map point features. Ensembles of feature transformations generated by deep learning are used to encode map point context information. Asset membership for points can then be based on features transformed by deep learning algorithms. Another method revolves around curriculum- based learning where assets are described in a curriculum, then learned in computation graphs. This method can be effective when the assets of interest are regular in shape and properties, and do not exhibit a lot of spatial complexity.
[0200] With these learning schemes, a neural network is often trained in a primary step, then applied to the remainder of the geospatial data for extraction of the map.
[0201] Machine learning techniques can therefore assist in optimizing and refining computation graphs. These graphs can be engineered manually or learned using the above methods. A parameter search component is useful for accuracy improvements and reductions in false positives and negatives. In this step, various parameters of the computation graph (from the region of interest, to the parameters of each function, to the number and nature of features used in a classifier) can all be modulated and the output monitored. By using search methodologies, the best performance combination of parameters can be found and applied to the remainder of the data. This step assumes the availability of previously annotated semantic maps.
[0202] When computation graphs are refined to an acceptable performance level, they can be used directly in the vehicles. This would correspond to streaming of the intelligence from the cloud to the vehicles, as opposed to the more conventional streaming of data from local environments to cloud systems. With geospatial data, the sheer size of the sensor data can be prohibitive. Therefore, in some embodiments, locally-obtained sensor data (e.g. data obtain by vehicle-mounted sensors) is summarized via local computation resources, with only a subset of collected information and/or extracted content being sent back to remote data systems. For example, resources comparable to data storage / preprocessor component 1220, processing unit 1240 and data analysis mechanisms 1250, can be implemented in-vehicle to extract semantic map data from onboard sensor systems. Computation graphs analogous to those described above for implementation in a cloud-based processing structure, can be optimized and tested in a machine learning framework, while presenting an opportunity for local in-vehicle
implementation. Such embodiments can utilize the vehicles as a distributed computing platform, constantly updating the contents of a centrally-maintained map, while consuming most of the remotely-sensed data in place, rather than streaming all of it to a central, cloud-based system.
[0203] While machine learning implementations described herein can tremendously accelerate the development of new graphs to map new features and assets, learning exercises can sometimes suffer from a shortage of training data, and issues with respect to accuracy. The consequence of these issues can include over-fitting and performance ceilings. When the amount of training data is limited, the learning routines might skew the graph's performance heavily towards the little data which is available, making it prone to fail when new cases are introduced which have not been trained for. Concerning performance, the creation of maps for training data is typically a manual process which is prone to error. As such, when the training data itself is not entirely accurate, the resulting graph won't be accurate either. For example, if a GIS analyst achieved only 80% accuracy of assets in their manually generated map, then any graph which has been trained on that data will have a very hard time crossing the 80% threshold of accuracy.
[0204] To address these issues, a simulation environment can be utilized. In the simulation environment, maps are programmatically generated in large numbers of permutations of parameters, to replicate the variability of terrains and landmarks on the face of the planet. Three dimensional models are then generated from the maps and raytraced to create a point cloud in as similar a way to real data collection as possible. Since the location of every asset is known a priori, a perfect map extracted from the point cloud is then available. The variability of the data, and the fact that a perfect ground truth exists for each point cloud greatly increases the scope of the computation graphs and their accuracy. It also provides a mechanism to understand the limitations of the current computing paradigms.
[0205] However, no matter how much a graph is trained, and how many test cases it undergoes, an automated map extraction can never be ideal. For this reason, a manual quality control (QC) step can be introduced to help find any issues. To avoid having to perform QC over the entire map, a level of confidence can be generated during the map making process. This level represents how confident a graph was in extracting the desired features from a map. QC can then be performed on regions in the lowest percentiles of confidence.
[0206] Quality control can be performed in multiple ways. Similar to creating a semantic map, a GIS analyst can use conventional visualization tools and overlay the raw survey data with the automatically extracted map. Any discrepancies can then be identified and corrected. Another method for QC would be to crowd source the effort amongst multiple agents online. Since each one of those agents might not be entirely skilled in semantic map creation, the QC work would need to be replicated. Hypotheses can then be confirmed or denied by each QC result, and a final conclusion reached with enough trials.
[0207] It is important to garner the QC results to reinforce the computation graphs. When discrepancies are detected, newly simulated worlds can be utilized that include the problematic test case. Further retraining of the graphs may then account for the use case in future work.
[0208] While certain embodiments have been described herein in detail for purposes of clarity and understanding, the foregoing description and Figures merely explain and illustrate the present invention and the present invention is not limited thereto. It will be appreciated that those skilled in the art, having the present disclosure before them, will be able to make these and other modifications and variations to that disclosed herein without departing from the scope of any claims.

Claims

CLAIMS What is claimed is:
1. An apparatus for identifying assets within point-cloud survey data, the apparatus comprising:
a front end component accessible via a digital communications network for receiving a point-cloud dataset;
a data storage component, the data storage component storing the point-cloud dataset and subdividing the point-cloud dataset into a plurality of data chunks;
a processing unit comprising a compute cluster, the processing unit receiving streamed data chunks from the data storage component and applying one or more analysis mechanisms to each data chunk to extract asset information; and
a map generator combining asset information extracted from the data analysis mechanisms into an output map.
2. The apparatus of claim 1, in which each data chunk comprises one or more tiles of point- cloud data.
3. The apparatus of claim 2, in which each tile comprises a subset of point-cloud data within a rectangular column extending lengthwise along the Earth's gravity vector.
4. The apparatus of claim 3, in which each data chunk contains a number of contiguous tiles optimized to achieve a target data chunk size.
5. The apparatus of claim 1, in which the map generator further comprises an annotation integrity verifier comparing asset information in an output map with asset information in one or more prior output maps corresponding to a common local environment, to generate a notification when discrepancies are detected.
6. The apparatus of claim 1, further comprising:
a compression mechanism operating to compress the point-cloud data prior to storage within the data storage component; and a decompression mechanism operating to decompress the point-cloud data prior to application of the analysis mechanisms by the processing unit;
whereby the compression mechanism modulates its compression ratio to balance a data retrieval rate from the data storage component, with a data processing rate achievable by the processing unit.
7. A vehicle localization apparatus comprising:
a GPS receiver mounted to a vehicle, the GPS receiver providing a first geographical position of the vehicle;
a local map cache residing within the vehicle, the local map cache storing a local map of assets comprising, for each asset, a location, properties associated with the asset, and one or more relationships relative to other assets;
one or more local environment sensors mounted on the vehicle to enable collection of data associated with a local environment in the vicinity of the vehicle;
one or more vehicle computers, the vehicle computers receiving the first geographical position from the GPS receiver to retrieve, from the local map cache, records associated with assets previously mapped in the vicinity of the first geographical position;
a feature extraction component implemented by the vehicle computers, the feature extraction component receiving the local environment sensor data to identify and locate observed assets presently within the vicinity of the vehicle; and
a position refinement component implemented by the vehicle computers, the position refinement component comparing the identity and location of observed assets from the feature extraction component with asset information retrieved from the local map cache to determine a present state of the vehicle.
8. The vehicle localization apparatus of claim 7, in which the present state of the vehicle comprises a vehicle location.
9. The vehicle localization apparatus of claim 8, in which the present state of the vehicle further comprises a vehicle velocity and direction of travel.
10. The vehicle localization apparatus of claim 7, further comprising a wireless vehicular communication device via which the local map cache can download local map data from a remote database during vehicle operation.
11. The vehicle localization apparatus of claim 7, in which the one or more local
environment sensors comprise one or more of: a LIDAR sensor, a digital camera and a radar sensor.
12. The vehicle localization apparatus of claim 7, in which the local environment attributes comprise one or more of signs, roadside safety structures, and semaphores.
13. The vehicle localization apparatus of claim 7, in which the local environment sensor data associated with a local environment in the vicinity of the vehicle comprises three-dimensional point cloud data.
14. The vehicle localization apparatus of claim 7, in which the vehicle is a train.
15. The vehicle localization apparatus of claim 14, in which the present state of the vehicle comprises a track identification.
16. The vehicle localization apparatus of claim 7, further comprising an interface component through which the present state of the vehicle can be communicated to one or more vehicle control systems.
17. The vehicle localization apparatus of claim 10, further comprising:
a map audit component identifying differences between the local map of assets and the observed assets and outputting said differences to the vehicular communication device for transmission to the remote database.
18. The vehicle localization apparatus of claim 17, in which the map audit component comprises a missing asset detector identifying assets that are present within the observed assets and not present within the local map of assets, or that are not present within the observed assets and present within the local map of assets.
19. The vehicle localization apparatus of claim 17, in which the map audit component comprises an asset alteration detector identifying assets within the observed assets having characteristics indicative of damage or tampering that differ from characteristics associated with the asset within the local map of assets.
20. The vehicle localization apparatus of claim 10, in which the vehicle is adapted for travel on railway tracks; the apparatus further comprising:
a track clearance evaluation component receiving information from the feature extraction component indicating a location of a first asset, the track clearance evaluation component identifying the first asset as an obstruction and reporting the obstruction location to a backend server via the vehicular communication device.
21. A method for auditing map data by one or more network-connected servers maintaining maps within a database, the method comprising the steps of:
receiving a request for map data from a vehicle, the vehicle having local environment sensors and a local map cache;
transmitting map data to the vehicle in response to the request, the map data comprising asset information, the asset information comprising identification, features and location of one or more assets; and
receiving, from the vehicle, a report indicative of one or more differences between the map data and and information detected by the vehicle local environment sensors; and
updating the database based on information within the report.
22. The method of claim 21, in which the report comprises identification and location of an asset detected by the vehicle local environment sensors and not present within the database.
23. The method of claim 21, in which the report comprises identification of an asset present within the database but not detected by the vehicle local environment sensors.
24. The method of claim 21, in which the asset information comprises one or more asset characteristics; and in which the report comprises differences between asset characteristics with the database and asset characteristics detected by the vehicle local environment sensors.
25. The method of claim 21, in which the vehicle is a train, and the step of receiving, from the vehicle, a report indicative of one or more differences between the map data and and information detected by the vehicle local environment sensors comprises:
receiving a report indicative of obstruction clearance relative to the path of the train.
PCT/US2016/014196 2015-01-20 2016-01-20 Real time machine vision and point-cloud analysis for remote sensing and vehicle control WO2016118672A2 (en)

Priority Applications (3)

Application Number Priority Date Filing Date Title
EP16740714.7A EP3248140A4 (en) 2015-01-20 2016-01-20 Real time machine vision and point-cloud analysis for remote sensing and vehicle control
JP2017556798A JP2018508418A (en) 2015-01-20 2016-01-20 Real-time machine vision and point cloud analysis for remote sensing and vehicle control
CN201680006431.6A CN107533630A (en) 2015-01-20 2016-01-20 For the real time machine vision of remote sense and wagon control and put cloud analysis

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US201562105696P 2015-01-20 2015-01-20
US62/105,696 2015-01-20

Publications (2)

Publication Number Publication Date
WO2016118672A2 true WO2016118672A2 (en) 2016-07-28
WO2016118672A3 WO2016118672A3 (en) 2016-10-20

Family

ID=56417920

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2016/014196 WO2016118672A2 (en) 2015-01-20 2016-01-20 Real time machine vision and point-cloud analysis for remote sensing and vehicle control

Country Status (4)

Country Link
EP (1) EP3248140A4 (en)
JP (1) JP2018508418A (en)
CN (1) CN107533630A (en)
WO (1) WO2016118672A2 (en)

Cited By (41)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2018134535A1 (en) 2017-01-20 2018-07-26 Messulam Pierre Driver assistance system for rail vehicles
CN108571974A (en) * 2017-03-14 2018-09-25 福特全球技术公司 Use the vehicle location of video camera
WO2018191613A1 (en) * 2017-04-14 2018-10-18 Bayer Cropscience Lp Vegetation detection and alert method and system for a railway vehicle
CN109286946A (en) * 2018-11-13 2019-01-29 山东建筑大学 Based on without the mobile communication indoor method for optimizing wireless network and system for relying on positioning
EP3459812A1 (en) * 2017-09-22 2019-03-27 Aurora Flight Sciences Corporation Detection system for a vehicle
US10426393B2 (en) 2017-09-22 2019-10-01 Aurora Flight Sciences Corporation Systems and methods for monitoring pilot health
WO2019211302A1 (en) * 2018-05-03 2019-11-07 Thales High-integrity autonomous system for locating a train in a railway network reference system
WO2019226782A1 (en) * 2018-05-25 2019-11-28 Bayer Cropscience Lp System and method for vegetation management risk assessment and resolution
EP3589527A4 (en) * 2017-02-28 2020-02-19 Thales Canada Inc. Guideway mounted vehicle localization system
US10599138B2 (en) 2017-09-08 2020-03-24 Aurora Flight Sciences Corporation Autonomous package delivery system
CN111046765A (en) * 2019-12-02 2020-04-21 北京深测科技有限公司 Danger early warning method and system for high-speed rail
US10699135B2 (en) 2017-11-20 2020-06-30 Here Global B.V. Automatic localization geometry generator for stripe-shaped objects
CN111832548A (en) * 2020-06-29 2020-10-27 西南交通大学 Train positioning method
CN112147635A (en) * 2020-09-25 2020-12-29 北京亮道智能汽车技术有限公司 Detection system, method and device
CN112184869A (en) * 2020-10-09 2021-01-05 北京理工大学 Point cloud simplification method for keeping geometric features based on absolute Gaussian curvature estimation
CN112363837A (en) * 2020-11-19 2021-02-12 北京航天泰坦科技股份有限公司 Cluster environment-based point cloud data parallel processing method, device, equipment and storage medium
WO2021028082A1 (en) * 2019-08-09 2021-02-18 Siemens Mobility GmbH Rail recognition using lidar
WO2021067935A1 (en) * 2019-10-04 2021-04-08 Waymo Llc Spatio-temporal embeddings
CN112669461A (en) * 2021-01-07 2021-04-16 中煤航测遥感集团有限公司 Airport clearance safety detection method and device, electronic equipment and storage medium
WO2021076251A1 (en) * 2019-10-16 2021-04-22 Bnsf Railway Company Systems and methods for auditing assets
CN113039579A (en) * 2018-11-19 2021-06-25 华为技术有限公司 System, equipment and method for generating high-resolution and high-precision point cloud
EP3851872A1 (en) * 2020-01-16 2021-07-21 Outsight Object detection on a path of travel and obstacle detection on railway tracks using free space information
US11136120B2 (en) 2018-10-05 2021-10-05 Aurora Flight Sciences Corporation Ground operations for autonomous object pickup
CN113536883A (en) * 2021-03-23 2021-10-22 长沙智能驾驶研究院有限公司 Obstacle detection method, vehicle, device, and computer storage medium
CN113677583A (en) * 2020-01-21 2021-11-19 深圳元戎启行科技有限公司 Vehicle driving data processing method and device based on graph calculation and computer equipment
WO2021232160A1 (en) * 2020-05-22 2021-11-25 Profound Positioning Inc. Vehicle localization system and method
US11340080B2 (en) 2018-10-29 2022-05-24 Motional Ad Llc Automatic annotation of environmental features in a map during navigation of a vehicle
EP4002090A1 (en) * 2020-11-24 2022-05-25 GEOTAB Inc. Extrema-retentive data buffering and simplification
EP4002091A1 (en) * 2020-11-24 2022-05-25 GEOTAB Inc. Extrema-retentive data buffering and simplification
US20220165065A1 (en) * 2019-10-31 2022-05-26 Aptiv Technologies Limited Multi-Domain Neighborhood Embedding and Weighting of Sampled Data
IT202000030677A1 (en) * 2020-12-15 2022-06-15 Ecosud S R L DYNAMIC DEVICE AND RELATED METHOD FOR CARTOGRAPHIC INSPECTION AND FOR THE TRANSFER OF KNOW-HOW
EP4016231A1 (en) * 2020-12-16 2022-06-22 Siemens Mobility GmbH Method for controlling a device
US11556509B1 (en) 2020-07-31 2023-01-17 Geotab Inc. Methods and devices for fixed interpolation error data simplification processes for telematic
US11585664B2 (en) 2020-06-16 2023-02-21 Geotab Inc. Dataset simplification of n-dimensional signals captured for asset tracking
WO2023019709A1 (en) * 2021-08-19 2023-02-23 中国铁路设计集团有限公司 Automatic detection method of conductor height and pull-out value of overhead line system based on vehicle-mounted mobile laser point cloud
US11593329B2 (en) 2020-07-31 2023-02-28 Geotab Inc. Methods and devices for fixed extrapolation error data simplification processes for telematics
US11609888B2 (en) 2020-07-31 2023-03-21 Geotab Inc. Methods and systems for fixed interpolation error data simplification processes for telematics
US11623673B2 (en) 2019-04-12 2023-04-11 Thales Management & Services Deutschland Gmbh Method for safely and autonomously determining the position information of a train on a track
EP4129742A4 (en) * 2020-03-30 2023-05-24 Mitsubishi Electric Corporation Data extraction device, data extraction method, and data extraction program
WO2024068204A1 (en) * 2022-09-29 2024-04-04 GTS France Method for managing network reconfigurations of a hybrid on-board/ground-based communication system of a rolling rail vehicle during a journey
US11999396B2 (en) 2018-09-14 2024-06-04 Siemens Mobility GmbH Automated on-vehicle control system for a rail vehicle

Families Citing this family (26)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108416808B (en) * 2018-02-24 2022-03-08 斑马网络技术有限公司 Vehicle repositioning method and device
US10468062B1 (en) * 2018-04-03 2019-11-05 Zoox, Inc. Detecting errors in sensor data
US11050933B2 (en) * 2018-04-27 2021-06-29 Continenal Automotive Systems, Inc. Device and method for determining a center of a trailer tow coupler
CN108648150A (en) * 2018-05-10 2018-10-12 句容康泰膨润土有限公司 A kind of image split-joint method
US20200103233A1 (en) * 2018-10-01 2020-04-02 GM Global Technology Operations LLC Decentralized distributed map using blockchain
US20190061771A1 (en) * 2018-10-29 2019-02-28 GM Global Technology Operations LLC Systems and methods for predicting sensor information
CN111198904A (en) * 2018-11-16 2020-05-26 千寻位置网络有限公司 Data processing method and device and processing system
US11536844B2 (en) * 2018-12-14 2022-12-27 Beijing Voyager Technology Co., Ltd. Dynamic sensor range detection for vehicle navigation
CN109948661B (en) * 2019-02-27 2023-04-07 江苏大学 3D vehicle detection method based on multi-sensor fusion
CN111935642B (en) * 2019-05-13 2022-10-21 北京地平线机器人技术研发有限公司 Positioning method and device of movable equipment
US11581022B2 (en) * 2019-05-29 2023-02-14 Nokia Technologies Oy Method and apparatus for storage and signaling of compressed point clouds
KR102043405B1 (en) * 2019-06-20 2019-11-12 주식회사 첨단공간정보 System of image processing and editing based on GIS
CN112241129A (en) * 2019-07-17 2021-01-19 中车长春轨道客车股份有限公司 Control system, method, apparatus, computer device, and storage medium
JP7343153B2 (en) * 2019-08-08 2023-09-12 独立行政法人自動車技術総合機構 train position detection device
CN112441075A (en) * 2019-08-30 2021-03-05 比亚迪股份有限公司 Rail transit external environment sensing system and method and rail transit equipment
CN110481601B (en) * 2019-09-04 2022-03-08 深圳市镭神智能***有限公司 Track detection system
CN111145166B (en) * 2019-12-31 2023-09-01 北京深测科技有限公司 Security monitoring method and system
CN111223107A (en) * 2019-12-31 2020-06-02 武汉中海庭数据技术有限公司 Point cloud data set manufacturing system and method based on point cloud deep learning
CN111091700A (en) * 2020-03-23 2020-05-01 北京全路通信信号研究设计院集团有限公司 Comprehensive configuration intelligent terminal for track circuit outdoor monitoring equipment
CN111427985B (en) * 2020-03-25 2023-04-11 北京小马慧行科技有限公司 Map updating method and device, storage medium and processor
WO2021226786A1 (en) * 2020-05-11 2021-11-18 Mtr Corporation Limited On-board systems for trains and methods of determining safe speeds and locations of trains
EP3919344A1 (en) * 2020-06-05 2021-12-08 Mitsubishi Electric R&D Centre Europe B.V. Method for geolocating an interference source in a communication-based transport system
CN112132896B (en) * 2020-09-16 2024-05-10 北京埃福瑞科技有限公司 Method and system for detecting states of trackside equipment
US20220091266A1 (en) * 2020-09-18 2022-03-24 Denso International America, Inc. Systems and methods for enhancing outputs of a lidar
CN113514030B (en) * 2021-07-12 2022-08-23 石家庄铁道大学 Ballast bed section detection method and system
CN117311283B (en) * 2023-10-24 2024-03-19 风凯换热器制造(常州)有限公司 Workshop running control intelligent monitoring method and system for preassembly body in heat exchanger

Family Cites Families (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP4086994B2 (en) * 1999-02-08 2008-05-14 株式会社デンソー Image data supply device and image compression device
JP2006321277A (en) * 2005-05-17 2006-11-30 Hitachi Ltd Railroad facilities information providing system, its method, and its program
US8605947B2 (en) * 2008-04-24 2013-12-10 GM Global Technology Operations LLC Method for detecting a clear path of travel for a vehicle enhanced by object detection
JP2010202017A (en) * 2009-03-03 2010-09-16 Mitsubishi Electric Corp Data analysis device and method, and program
US20140379254A1 (en) * 2009-08-25 2014-12-25 Tomtom Global Content B.V. Positioning system and method for use in a vehicle navigation system
CN101697229B (en) * 2009-10-30 2012-06-13 宁波大学 Method for extracting region of interest of medical image
US8525835B1 (en) * 2010-02-24 2013-09-03 The Boeing Company Spatial data compression using implicit geometry
US20110216063A1 (en) * 2010-03-08 2011-09-08 Celartem, Inc. Lidar triangular network compression
EP2574207A1 (en) * 2010-03-31 2013-04-03 Ambercore Software Inc. System and method for extracting features from data having spatial coordinates
US9102341B2 (en) * 2012-06-15 2015-08-11 Transportation Technology Center, Inc. Method for detecting the extent of clear, intact track near a railway vehicle
CN102799898A (en) * 2012-06-28 2012-11-28 浙江大学 Efficient target identification processing method of high-resolution remote sensing image in context
US9221461B2 (en) * 2012-09-05 2015-12-29 Google Inc. Construction zone detection using a plurality of information sources
US9846025B2 (en) * 2012-12-21 2017-12-19 Wabtec Holding Corp. Track data determination system and method
JP2014185947A (en) * 2013-03-25 2014-10-02 Geo Technical Laboratory Co Ltd Image photographing method for three-dimensional restoration
DE102013104088A1 (en) * 2013-04-23 2014-10-23 Deutsches Zentrum für Luft- und Raumfahrt e.V. Method for automatically detecting characteristic elements, in particular a level crossing, and device therefor

Cited By (71)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
FR3062103A1 (en) * 2017-01-20 2018-07-27 Pierre Messulam SYSTEM FOR AIDING THE DRIVING OF RAILWAY VEHICLES
WO2018134535A1 (en) 2017-01-20 2018-07-26 Messulam Pierre Driver assistance system for rail vehicles
US11608097B2 (en) 2017-02-28 2023-03-21 Thales Canada Inc Guideway mounted vehicle localization system
EP3589527A4 (en) * 2017-02-28 2020-02-19 Thales Canada Inc. Guideway mounted vehicle localization system
CN108571974A (en) * 2017-03-14 2018-09-25 福特全球技术公司 Use the vehicle location of video camera
CN108571974B (en) * 2017-03-14 2023-12-22 福特全球技术公司 Vehicle positioning using a camera
US11046340B2 (en) 2017-04-14 2021-06-29 Bayer Cropscience Lp Vegetation detection and alert system for a railway vehicle
CN110662687B (en) * 2017-04-14 2021-12-28 拜耳作物科学有限合伙公司 Vegetation detection and alarm method and system for railway vehicle
EP4049914A3 (en) * 2017-04-14 2023-02-22 Bayer CropScience LP Object detection and alert method and system for a railway vehicle
US10370014B2 (en) 2017-04-14 2019-08-06 Bayer Cropscience Lp Vegetation detection and alert system for a railway vehicle
AU2018253281B2 (en) * 2017-04-14 2022-08-04 Discovery Purchaser Corporation Vegetation detection and alert method and system for a railway vehicle
CN110662687A (en) * 2017-04-14 2020-01-07 拜耳作物科学有限合伙公司 Vegetation detection and alarm method and system for railway vehicle
WO2018191613A1 (en) * 2017-04-14 2018-10-18 Bayer Cropscience Lp Vegetation detection and alert method and system for a railway vehicle
EP4095012A3 (en) * 2017-04-14 2023-05-03 Bayer CropScience LP Vegetation detection and alert method and system for a railway vehicle
AU2022263574B2 (en) * 2017-04-14 2023-06-15 Discovery Purchaser Corporation Vegetation detection and alert method and system for a railway vehicle
US10599138B2 (en) 2017-09-08 2020-03-24 Aurora Flight Sciences Corporation Autonomous package delivery system
US10426393B2 (en) 2017-09-22 2019-10-01 Aurora Flight Sciences Corporation Systems and methods for monitoring pilot health
EP3459812A1 (en) * 2017-09-22 2019-03-27 Aurora Flight Sciences Corporation Detection system for a vehicle
US10699135B2 (en) 2017-11-20 2020-06-30 Here Global B.V. Automatic localization geometry generator for stripe-shaped objects
FR3080823A1 (en) * 2018-05-03 2019-11-08 Thales SYSTEM FOR INTEGRATED AND INDEPENDENT LOCATION OF A TRAIN IN A RAILWAY NETWORK REFERENTIAL
WO2019211302A1 (en) * 2018-05-03 2019-11-07 Thales High-integrity autonomous system for locating a train in a railway network reference system
WO2019226782A1 (en) * 2018-05-25 2019-11-28 Bayer Cropscience Lp System and method for vegetation management risk assessment and resolution
US11010608B2 (en) 2018-05-25 2021-05-18 Bayer Cropscience Lp System and method for vegetation management risk assessment and resolution
AU2019275041B2 (en) * 2018-05-25 2022-03-24 Discovery Purchaser Corporation System and method for vegetation management risk assessment and resolution
US11999396B2 (en) 2018-09-14 2024-06-04 Siemens Mobility GmbH Automated on-vehicle control system for a rail vehicle
US11136120B2 (en) 2018-10-05 2021-10-05 Aurora Flight Sciences Corporation Ground operations for autonomous object pickup
US11774261B2 (en) 2018-10-29 2023-10-03 Motional Ad Llc Automatic annotation of environmental features in a map during navigation of a vehicle
US11340080B2 (en) 2018-10-29 2022-05-24 Motional Ad Llc Automatic annotation of environmental features in a map during navigation of a vehicle
CN109286946A (en) * 2018-11-13 2019-01-29 山东建筑大学 Based on without the mobile communication indoor method for optimizing wireless network and system for relying on positioning
CN113039579A (en) * 2018-11-19 2021-06-25 华为技术有限公司 System, equipment and method for generating high-resolution and high-precision point cloud
US11623673B2 (en) 2019-04-12 2023-04-11 Thales Management & Services Deutschland Gmbh Method for safely and autonomously determining the position information of a train on a track
WO2021028082A1 (en) * 2019-08-09 2021-02-18 Siemens Mobility GmbH Rail recognition using lidar
WO2021067935A1 (en) * 2019-10-04 2021-04-08 Waymo Llc Spatio-temporal embeddings
US11657291B2 (en) 2019-10-04 2023-05-23 Waymo Llc Spatio-temporal embeddings
WO2021076251A1 (en) * 2019-10-16 2021-04-22 Bnsf Railway Company Systems and methods for auditing assets
US11932290B2 (en) 2019-10-16 2024-03-19 Bnsf Railway Company Systems and methods for auditing assets
US11427232B2 (en) 2019-10-16 2022-08-30 Bnsf Railway Company Systems and methods for auditing assets
US20220165065A1 (en) * 2019-10-31 2022-05-26 Aptiv Technologies Limited Multi-Domain Neighborhood Embedding and Weighting of Sampled Data
US11693090B2 (en) * 2019-10-31 2023-07-04 Aptiv Technologies Limited Multi-domain neighborhood embedding and weighting of sampled data
CN111046765A (en) * 2019-12-02 2020-04-21 北京深测科技有限公司 Danger early warning method and system for high-speed rail
CN111046765B (en) * 2019-12-02 2023-07-14 北京深测科技有限公司 Dangerous early warning method and system for high-speed rail
EP3851872A1 (en) * 2020-01-16 2021-07-21 Outsight Object detection on a path of travel and obstacle detection on railway tracks using free space information
CN113677583B (en) * 2020-01-21 2023-08-15 深圳元戎启行科技有限公司 Graph calculation-based vehicle driving data processing method and device and computer equipment
CN113677583A (en) * 2020-01-21 2021-11-19 深圳元戎启行科技有限公司 Vehicle driving data processing method and device based on graph calculation and computer equipment
EP4129742A4 (en) * 2020-03-30 2023-05-24 Mitsubishi Electric Corporation Data extraction device, data extraction method, and data extraction program
WO2021232160A1 (en) * 2020-05-22 2021-11-25 Profound Positioning Inc. Vehicle localization system and method
US11585664B2 (en) 2020-06-16 2023-02-21 Geotab Inc. Dataset simplification of n-dimensional signals captured for asset tracking
US11867512B2 (en) 2020-06-16 2024-01-09 Geotab Inc. Dataset simplification of n-dimensional signals captured for asset tracking
CN111832548B (en) * 2020-06-29 2022-11-15 西南交通大学 Train positioning method
CN111832548A (en) * 2020-06-29 2020-10-27 西南交通大学 Train positioning method
US11556509B1 (en) 2020-07-31 2023-01-17 Geotab Inc. Methods and devices for fixed interpolation error data simplification processes for telematic
US11609888B2 (en) 2020-07-31 2023-03-21 Geotab Inc. Methods and systems for fixed interpolation error data simplification processes for telematics
US11593329B2 (en) 2020-07-31 2023-02-28 Geotab Inc. Methods and devices for fixed extrapolation error data simplification processes for telematics
CN112147635B (en) * 2020-09-25 2024-05-31 北京亮道智能汽车技术有限公司 Detection system, method and device
CN112147635A (en) * 2020-09-25 2020-12-29 北京亮道智能汽车技术有限公司 Detection system, method and device
CN112184869A (en) * 2020-10-09 2021-01-05 北京理工大学 Point cloud simplification method for keeping geometric features based on absolute Gaussian curvature estimation
CN112363837A (en) * 2020-11-19 2021-02-12 北京航天泰坦科技股份有限公司 Cluster environment-based point cloud data parallel processing method, device, equipment and storage medium
US11546395B2 (en) 2020-11-24 2023-01-03 Geotab Inc. Extrema-retentive data buffering and simplification
US11838364B2 (en) * 2020-11-24 2023-12-05 Geotab Inc. Extrema-retentive data buffering and simplification
EP4002091A1 (en) * 2020-11-24 2022-05-25 GEOTAB Inc. Extrema-retentive data buffering and simplification
US20220166834A1 (en) * 2020-11-24 2022-05-26 Geotab Inc. Extrema-retentive data buffering and simplification
EP4002090A1 (en) * 2020-11-24 2022-05-25 GEOTAB Inc. Extrema-retentive data buffering and simplification
IT202000030677A1 (en) * 2020-12-15 2022-06-15 Ecosud S R L DYNAMIC DEVICE AND RELATED METHOD FOR CARTOGRAPHIC INSPECTION AND FOR THE TRANSFER OF KNOW-HOW
EP4016231A1 (en) * 2020-12-16 2022-06-22 Siemens Mobility GmbH Method for controlling a device
CN112669461A (en) * 2021-01-07 2021-04-16 中煤航测遥感集团有限公司 Airport clearance safety detection method and device, electronic equipment and storage medium
CN112669461B (en) * 2021-01-07 2024-01-26 中煤航测遥感集团有限公司 Airport clearance safety detection method and device, electronic equipment and storage medium
CN113536883A (en) * 2021-03-23 2021-10-22 长沙智能驾驶研究院有限公司 Obstacle detection method, vehicle, device, and computer storage medium
CN113536883B (en) * 2021-03-23 2023-05-02 长沙智能驾驶研究院有限公司 Obstacle detection method, vehicle, apparatus, and computer storage medium
WO2023019709A1 (en) * 2021-08-19 2023-02-23 中国铁路设计集团有限公司 Automatic detection method of conductor height and pull-out value of overhead line system based on vehicle-mounted mobile laser point cloud
WO2024068204A1 (en) * 2022-09-29 2024-04-04 GTS France Method for managing network reconfigurations of a hybrid on-board/ground-based communication system of a rolling rail vehicle during a journey
FR3140506A1 (en) * 2022-09-29 2024-04-05 Thales Method for managing network reconfigurations of a hybrid on-board/ground communication system of a railway rolling vehicle during a journey

Also Published As

Publication number Publication date
JP2018508418A (en) 2018-03-29
WO2016118672A3 (en) 2016-10-20
EP3248140A2 (en) 2017-11-29
CN107533630A (en) 2018-01-02
EP3248140A4 (en) 2018-12-05

Similar Documents

Publication Publication Date Title
US10549768B2 (en) Real time machine vision and point-cloud analysis for remote sensing and vehicle control
WO2016118672A2 (en) Real time machine vision and point-cloud analysis for remote sensing and vehicle control
US11959740B2 (en) Three-dimensional data creation method and three-dimensional data creation device
US20180370552A1 (en) Real time machine vision system for vehicle control and protection
CN110832474B (en) Method for updating high-definition map
US11988518B2 (en) Updating high definition maps based on lane closure and lane opening
US20210004363A1 (en) Updating high definition maps based on age of maps
US10832502B2 (en) Calibration for autonomous vehicle operation
US11106218B2 (en) Adaptive mapping to navigate autonomous vehicles responsive to physical environment changes
JP6714688B2 (en) System and method for matching road data objects to generate and update an accurate road database
US11346682B2 (en) Augmented 3D map
JP2022065083A (en) Adaptive mapping for navigating autonomous vehicles responsive to physical environment change
CN108369775B (en) Adaptive mapping to navigate an autonomous vehicle in response to changes in a physical environment
JP2019501468A (en) Machine learning system and technique for optimizing teleoperation and / or planner decisions
JP2024025803A (en) Vehicle utilizing space information acquired using sensor, sensing device utilizing space information acquired using sensor, and server
WO2021198772A1 (en) Navigating a vehicle using an electronic horizon
US11961304B2 (en) Systems and methods for deriving an agent trajectory based on multiple image sources
US20220351526A1 (en) Multi-frame image segmentation
US11961241B2 (en) Systems and methods for deriving an agent trajectory based on tracking points within images
JP2020510941A (en) Highway system for connected self-driving car and method using the same
US20220234621A1 (en) Augmented 3d map
US20240104757A1 (en) Systems and methods for using image data to identify lane width
CA3154469A1 (en) Device and method for autonomously locating a mobile vehicle on a railway track
US20240183684A1 (en) Using robot observations
US20240104939A1 (en) Systems and methods for using image data to identify lane width

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 16740714

Country of ref document: EP

Kind code of ref document: A2

ENP Entry into the national phase

Ref document number: 2017556798

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

REEP Request for entry into the european phase

Ref document number: 2016740714

Country of ref document: EP

121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 16740714

Country of ref document: EP

Kind code of ref document: A2