The problem
RoboSub is a competition where a submarine robot has to find and pass through gates, follow markers and hit targets with no human control. Our team gets a few hours of pool time a week. Every bug found in the pool is an expensive one, and every labelled training image used to be drawn by hand.
What I built
A simulator the real software can't tell from the robot
I rebuilt the team's Unity simulator as a digital twin: the same thrusters, the same sensors (cameras, depth, IMU, DVL, hydrophones), sending the same ROS 2 messages. The control, planning and vision code run unchanged against it. This is called software-in-the-loop testing, and it means a mission can be rehearsed on a laptop.
Training images without labelling
The simulator knows where every object is, so it can label its own images. With a teammate, I built the pipeline from simulator to trained detector. The first detectors trained only on simulated images failed on real ones. What fixed it was heavy image augmentation and modelling how water bends light at the camera housing (refraction), which changes where objects appear.
Seeing in 3D on the robot
I wrote the on-board perception pipeline in C++ and ROS 2. A neural network (accelerated with TensorRT) finds objects in each frame. Depth gives their position. A Kalman filter smooths each object's position over time, and Hungarian matching decides which detection belongs to which known object. The result is a map of what is where, updated 30 times a second on the Jetson.
Results
| Measure | Result |
|---|---|
| First detector, trained on zero real images, tested on 477 real pool images | 0.46 mAP50 |
| Final detector, tested on real pool images | 0.83 mAP50 |
| Hand-labelling saved at competition | 5+ hours |
| 3D object map on Jetson AGX Orin | 30 Hz |
| RoboSub 2026 | 6th overall |
The first number is what simulated images alone gave us, with no labelling at all. At competition, where the props were new, that meant a working detector hours before a hand-labelled one would have been ready. The final detector reached 0.83.
What went wrong: the model fit that was worse than guessing
We wanted model predictive control, which needs a model of how the robot moves through water: how much the water drags on it and how much water it has to push out of the way (added mass). I wrote an optimizer (CasADi with IPOPT) to fit those numbers to recorded pool runs.
It converged. Then I checked its predictions against data it hadn't seen, and it did worse than predicting the average every time.
The optimizer was fine. The data couldn't answer the question:
- The runs were recorded with the controller on. The controller pushes exactly as hard as needed to cancel drag, so drag and thrust always appear together and can't be separated.
- I had the commanded force, not the real one. The thrusters saturate and have a dead zone near zero.
- The tether pulled the robot forward with about 1.7 N, which the fit blamed on the model.
We used hand-tuned values for the season. I wrote a new test procedure for the pool: controller off, constant thrust until the speed settles (thrust then equals drag), then cut power and let it coast. I also turned the lesson into the modelling chapter of AUV Lab, the interactive course I built for new members.
What I took from it: look at how the data was collected before fitting anything to it. A fit that converges tells you about the optimizer.
Related
- AUV Lab: thruster allocation, PID, LQR and MPC explained interactively.
- ros2-unity-sensors: the simulator's sensors as a reusable package.
- auv-hydrodynamic-sysid: the fitting code and the pool procedure.