Project S20 Co-Pilot Pyxis

Project Roadmap

Empathize

Maritime pilot transfers remain one of the most dangerous routine operations at sea, with ladder-based transfers accounting for a majority of reported incidents. Despite strict regulations, captains still rely heavily on visual judgment and experience when operating within sub-metre distances.

Through field observations and stakeholder discussions with Pyxis Maritime, we identified that limited visibility, lack of precise distance awareness, and high cognitive load are key challenges during close-range manoeuvres. Captains must simultaneously manage vessel positioning, monitor the pilot ladder, and coordinate with crew under dynamic sea conditions.

Existing maritime systems are designed for large-scale navigation and fail to provide real-time, close-proximity guidance, highlighting a critical gap for a solution that enhances situational awareness without disrupting existing workflows.

Define

Pilot transfer operations remain one of the most hazardous manoeuvres in maritime environments, requiring precise coordination between a pilot vessel and a moving cargo ship within sub-metre distances. Despite strict regulations, captains rely heavily on visual judgment and experience, often operating with limited visibility and no real-time instrumentation for distance or alignment. This creates a critical gap where human error, blind spots, and dynamic sea conditions significantly increase the risk of misalignment, collision, or unsafe boarding.

Solution

Pyxis Co-Pilot addresses this gap by introducing a real-time, onboard guidance system that enhances the captain’s situational awareness during pilot transfer operations. By integrating computer vision, LiDAR sensing, and a phase-aware dashboard, the system provides accurate distance measurements, ladder tracking, and alignment feedback in real time. Designed to operate fully on edge devices, it delivers low-latency, reliable guidance without requiring vessel modification or internet connectivity—augmenting human decision-making to enable safer, smoother, and more controlled manoeuvres.

How Pyxis Co-Pilot Works

1. System Setup & Activation
The system is installed onboard the pilot vessel using a combination of cameras, LiDAR sensors, and onboard computing units. Once activated, it operates fully on edge devices, requiring no internet connectivity or integration with existing vessel systems, ensuring fast deployment and reliability.

2. Real-Time Sensing & Detection
Computer vision continuously detects the pilot ladder, vessel structure, and surrounding environment, while LiDAR sensors provide accurate, sub-metre measurements of hull distance and spatial positioning. Together, these inputs create a precise, real-time understanding of the transfer environment.

3. Phase-Aware Guidance
As the vessel approaches the cargo ship, the system automatically transitions through operational phases (Approach, Zoning, Holding). Each phase delivers tailored guidance, helping the captain maintain alignment, control speed, and ensure safe positioning during critical moments.

4. Live Decision Support Dashboard
All data is consolidated into a single, intuitive dashboard that presents ladder tracking, vessel alignment, and proximity alerts. Visual indicators, colour-coded zones, and real-time metrics allow the captain to quickly interpret conditions and make informed decisions under pressure.

5. Safe & Reliable Operation
Designed for real-world maritime conditions, the system runs entirely on onboard hardware, ensuring low latency, high reliability, and continuous performance. It enhances human decision-making without disrupting existing workflows or requiring complex setup.

Pyxis Co-Pilot Main Features

Images show: Live View, Remote Control, Spatial View, Diagnostics. The dashboard delivers real-time situational awareness during pilot transfer operations. Live View: AI-powered ladder detection with real-time distance and safety indicators Remote: Simple interface for control, calibration, and mode switching Spatial View: Top-down LiDAR visualization for tracking

System Hardware Overview

Pyxis Co-Pilot enhances pilot transfer operations through a real-time, edge-powered system that combines AI, sensors, and intuitive visualization to support safer and more precise decision-making at sea.

System Flow

Hardware System

AI & Data Pipeline

Pyxis Co-Pilot operates as an integrated real-time guidance system onboard the pilot vessel. Once activated, cameras and LiDAR continuously capture environmental data while onboard computing units process inputs locally. The system automatically adapts across operational phases—Approach, Zoning, and Holding—providing context-aware guidance throughout the manoeuvre. All information is consolidated into a single dashboard, enabling the captain to monitor ladder position, vessel alignment, and proximity in real time for safer decision-making.

The system consists of a compact, fully standalone hardware setup designed for maritime environments. An NVIDIA Jetson processes computer vision models for ladder detection, while a Raspberry Pi integrates sensor data and controls the dashboard interface. LiDAR and IMU sensors provide accurate distance and orientation measurements, and dual cameras mounted on a stabilised gimbal ensure reliable tracking under motion. Powered by a portable battery system, Pyxis Co-Pilot operates independently without requiring vessel modification or internet connectivity.

Pyxis Co-Pilot leverages a custom-built computer vision pipeline tailored for maritime pilot transfer scenarios. Due to the absence of public datasets, the team developed a scalable workflow combining data collection, auto-labelling, and human validation. Models are trained to detect pilot ladders and adapt to different environments, achieving significant improvements in detection accuracy. This pipeline enables continuous iteration and deployment across new vessels, forming the foundation for a robust and transferable maritime AI system.

Acknowledgements

The team would like to express our deepest gratitude to Pyxis Maritime, whose partnership, guidance, and
operational access made this project possible. We are especially grateful to Mr Low Jin Yi and Mr Choon Huat
Lee, whose mentorship has gone far beyond technical guidance. Their patience, integrity, and sincere
commitment to our growth shaped not only the direction of the project, but also the way we approached
challenges as a team. We deeply appreciate their kindness, encouragement, and belief in our potential. We also
thank the Pyxis engineering and operations teams for their openness, patience, and continuous support
throughout our field visits, technical discussions, and early prototyping stages.

We would also like to thank Captain Widi, who generously shared his experience, operational insights, and
time to help us understand the realities of Pilot Transfer operations. His candour, professionalism, and
willingness to engage deeply with our ideas were invaluable in shaping a solution grounded in real-world
needs.

Our sincere appreciation goes to our faculty advisors: Prof. Freddy, Prof Geraldine and Prof. Bernard for their
steady guidance, expert feedback, and encouragement from the start of the Capstone through the Midterm and
Interim reviews. Their mentorship has been instrumental in helping us refine our thinking, strengthen our
analysis, and communicate our work effectively while navigating the complexities of a multidisciplinary
project.

We would also like to acknowledge the broader SUTD community, and in particular Hasmidah from the
Capstone Office, for her administrative support, as well as the many staff members who provided resources,
equipment, and logistical assistance throughout the term.
To all who contributed, directly or indirectly, to the development of our project, we offer our heartfelt thanks.