Frame 08Open sourceRobotics perceptionApril 2025
Two YOLO11 models, from binary PNG masks to a live ROS 2 topic
A ROS 2 Humble package with two rclpy nodes. They read a ZED2i left-camera stream, draw green boxes around pallets and a yellow overlay on the floor, and republish both as image topics. I trained both models on a laptop GPU, one from a folder of binary masks that first had to become polygon labels. The constraint that shaped the code: the test data was a rosbag, not a robot, and the subscriber had to match the bag's QoS or it would receive nothing.
- Stack
- ROS 2 Humble, rclpy, Ultralytics YOLO11 (YOLO11m detect, YOLO11n-seg), OpenCV, cv_bridge, PyTorch
- Hardware
- NVIDIA RTX 4070 Laptop GPU for training and timing; ZED2i camera via a recorded bag
- Scope
- Sole author
- Two nodes: a fine-tuned YOLO11m boxes pallets; a YOLO11n-seg model masks the ground. Both republish on
/inference/…image topics. - Segmenter: 0.966 mask mAP, about 5.0 ms per image on 133 validation images. Detector: 0.911 mAP50 after a 10-epoch fine-tune. Both from notebook outputs in the repo.
- Missing: any runtime check beyond a confidence threshold, and any engineering history. The repo is three hours of web uploads with hardcoded paths, weights on Google Drive and lint-only tests.
What could go wrong, and how we would know
A perception node like this has three ways to fail without an error message. The subscriber never fires: ROS 2 checks QoS at match time and a reliable subscriber will not match a best-effort publisher. Against a bag recorded best-effort, the default subscription starts cleanly and then never receives a frame.
The labels are wrong before training starts: turning a binary mask into a YOLO polygon is a hand-written step, and a bug there teaches the model the wrong shape while the validation score still looks fine, because the validation labels come from the same converter. And the mask lands on the wrong pixels: the segmenter runs at 640 by 640, the camera frame does not.
The QoS profile is explicit: BEST_EFFORT, KEEP_LAST, depth 10, with the comment "QoS to match bag settings" in inference_segment.py. Both nodes log every frame they receive, so a silent non-match shows up within seconds of playing the bag. Masks come back to the original frame size with nearest-neighbour interpolation. The label converter has no check of its own beyond the label plot Ultralytics writes at the start of training; that is a weak check and I say so below. At runtime there is no check at all: conf=0.1 on the detector and conf=0.7 on the segmenter are the only gates between the model and the topic.
What I built
The segmentation data arrived as 1,338 image and mask pairs. Training.ipynb shuffles them with a fixed seed and splits 80/10/10, which lands at 1070 train, 133 val and 135 test. Each mask is thresholded at 127, passed to cv2.findContours with RETR_EXTERNAL and CHAIN_APPROX_SIMPLE, and reduced to its largest contour by area. The points are divided by width and height and written as one line: class 0, then the normalised polygon. One polygon per image, by construction.
The segmenter starts from yolo11n-seg.pt and trains 100 epochs at batch 8, image size 640. The detector starts from an earlier YOLO11m checkpoint of mine and gets a 10-epoch fine-tune at batch 12 on a separate pallet dataset, 0.31 hours on the RTX 4070 Laptop GPU.
inference.py subscribes to the ZED2i left rectified image, runs the detector at conf=0.1, keeps class 1 only (the class map is {0: '-', 1: 'pallet'}), draws green rectangles and publishes /inference/pallet_detection. inference_segment.py resizes the frame to 640 by 640, runs the segmenter at conf=0.7, upsamples each mask to the original frame with INTER_NEAREST, paints it yellow and blends it in before publishing /inference/ground_segmentation. A launch file starts both.
| Chose | Over | Because | Cost |
|---|---|---|---|
| A best-effort subscriber | The rclpy default, reliable | The bag was recorded best-effort and a reliable subscriber will not match it | Dropped frames are invisible. Fine for a viewer, wrong for anything that counts |
| Largest external contour only | Every contour, or contours with holes | YOLO's segment format wants one polygon per instance and the floor is one region | A floor split in two by an obstacle loses the smaller island, and holes are filled in |
| Squash to 640 by 640, nearest-neighbour back | Letterboxing | Four lines of OpenCV | Aspect ratio distortion at inference, and the model never saw squashed frames in training |
| Publishing drawn images | vision_msgs detections plus a mask topic | I needed to see it in rqt | Nothing downstream can consume the boxes as data |
/inference/ground_segmentation, photographed off the monitor. The overlay covers the floor and bleeds onto the cart frame under the pallet.
/inference/pallet_detection in rqt Image View, same scene. Green boxes are the class 1 detections at conf=0.1.Verification
Two validation runs, both recorded as cell outputs in Training.ipynb. The segmenter's best.pt on the 133 val images: box mAP 0.976, mask mAP 0.966, at 1.0 ms preprocess, 3.3 ms inference and 0.7 ms postprocess per image. The detector, on its own 657-image validation split with 1,094 pallet instances: precision 0.818, recall 0.844, mAP50 0.911, mAP50-95 0.733 on the last epoch row (re-validating best.pt prints 0.732; noise). The timings are Ultralytics batched validation on the laptop GPU, not the latency of the ROS callback with its cv_bridge conversions and a log line per frame.
- 0.966mask mAP, ground segmenter, 133 val images
- 0.911mAP50, pallet detector, 657 val images
- 5.0 msper image, segmenter, RTX 4070 Laptop GPU
- 1,338image and mask pairs, split 1070 / 133 / 135
What I found
The segmentation val set has 133 images and exactly 133 instances. That is not a property of the floor. It is a property of my converter, which emits one polygon per mask, so the 0.966 is a score on a task I simplified before training: one connected ground region per frame, no holes. In the image above the overlay runs onto the cart frame under the pallet, which the labels never taught the model to leave out.
The detector's training log warned that the dataset had 7,804 segments for 8,086 boxes and that it would drop the segments and train on boxes only. I had exported a mixed detect-and-segment dataset. Harmless for a detector, but the polygon labels on that set did no work, and I only noticed because I read the log.
Outcome, and what it does not prove
Both nodes ran against the recorded ZED2i bag, and the two images above are what the output topics looked like on screen. That is the whole claim. It proves nothing about edge hardware: there is no Jetson, TensorRT or ONNX export anywhere in the repo. It proves little about generalisation: the val images come from the same seeded shuffle of the same 1,338 pairs, and the 135-image test split is never evaluated in the notebook. And the detector number is only partly reproducible, since it sits on a checkpoint whose training is not in the repo.
What I would do differently
- Model path and input topic as ROS parameters instead of a path under
/home/chandhan. - Hand the raw frame to
predictand let Ultralytics letterbox it, instead of squashing to 640 by 640 first. - Publish data, not pictures.
vision_msgsis listed as a dependency in the README and never imported. - One functional test that pushes a frame through each callback and asserts an image comes out at the input resolution.
- Run the 135-image test split, then hold out a different scene entirely.
- All 15 commits are same-day web uploads on 23 April 2025 ("Add files via upload", plus two directory deletes). There is no development history to read.
- Model paths are hardcoded to
/home/chandhan/…and the weights live on Google Drive, not in the repo. It does not run from a clone without edits. - The only tests are the
ros2 pkg createlint boilerplate. No CI, no LICENSE, andpackage.xmlstill says TODO. - The Requirements file pins ultralytics 8.0.224 and torch 2.1.0; the notebook ran 8.3.111 and 2.6.0, and 8.0 cannot load YOLO11 weights.
- The segmentation val set is 133 same-distribution images; the 135-image test split was never evaluated. The detector sits on a checkpoint whose training is not in the repo.
- The two images are photographs of a monitor, not saved frames.
Sources README Training.ipynb inference.py inference_segment.py Scope: I (sole author) Verified 3 Sep 2026