Worldwide a substantial effort is spent on the development of collision avoidance systems for the maritime sector. These systems are deployed for on-board and onshore decision support and ultimately for autonomous ships. What is currently lacking is a structured, well-defined and broadly accepted methodology to evaluate the performance of these systems. It is MARIN’s objective, as an independent organisation, to jointly develop this standard, to further extend it and support it in the future.
In this Joint Industry Project the methodology will be developed and demonstrated for the performance evaluation of collision avoidance systems: a scenario generator in which realistic, but also extreme conditions are included to create distinctive tests. A scenario describes amongst others the weather, current, waves and shipping activity (no of ships and behaviour of ships); and an assessment module, that quantifies the performance of a C/A system based on the results of a set of scenarios using a defined set of criteria. The program will include a study to define the evaluation criteria and the influence of sensor performance on the C/A performance.
In addition, a simulation environment is required to which the C/A systems can be interfaced, tested and assessed. This simulation environment must support (sufficiently) true-to-life models of different types of ships, the environmental conditions (wind, waves, current). The development of this simulation environment is not a goalof this project but the minimum requirements will be defined. The test scenario generation and performance evaluation will be set up in such a way that they are independent from the simulation environment such that it remains interesting for the other partners in the JIP to use their own simulation environment.
The C/A system to be evaluated will be included as Hardware/Software in the Loop and interfaced with the simulated own ship using standard protocols. This will allow the C/A system to steer ownship in the simulator environment. For now, focus will be on real-time scenario simulation as it will require a different requirements for the C/A system to be tested.
An important delivery of the JIP is a representative set of maritime traffic scenarios. The scenarios will include situations that allow for an evaluation of all Collision Regulations (COLREGs) under various metocean conditions and in various areas with different vessel density. AIS data will be used as the basis to determine safe and COLREG compliant behaviour in these conditions.
A methodology will be defined to vary the scenario parameters in an intelligent way for the purpose of a sensitivity and robustness test. The JIP will study how the influence of sensor performance and sensor fusion algorithms can be taken into account into the robustness evaluation.
In general, the target vessels will behave in accordance with the COLREGs, but in some cases may also deviate to test the systems robustness. The sailed track of the target vessels will be predefined. Together with simulator sessions in which scenarios are studied by experts we will establish the criteria and their safe limit values for use in the testbed.
A dedicated Collision Avoidance Assessment Module will evaluate the performance of the own ship. Each scenario will describe a single ship encounter and will be assessed on safety and compliancy with the COLREGs. Using a set of algorithms the applicable rulesets and appropriate actions are determined. Each encounter scenario is evaluated against the criteria defined in this project using a system of rewards and penalties. At the end of all scenario simulations the system reports the overall score for safety and COLREGs compliance, the score is given per rule set, scenario, weather condition and complexity ranking to determine weaknesses and strengths of the tested C/A system
A uniform approach for testing collision avoidance systems based on the safety of ship encounters and COLREG compliancy. The C/A assessment can also be used on board or for the evaluation of trainees on a bridge-simulator.
A method for generating relevant test scenarios and realistic target behaviour.
A set of industry accepted criteria for COLREG compliance and safe behaviour for use in the testbed, based on AIS studies and expert sessions.
The availability of a test environment for autonomous systems.