tips for debugging slow dds

I have a lot of nodes and a lot of topics running, some of these topics publish heavy pointclouds and images

My system runs maybe 10x faster on fastRTPS when compared to cyclonedds.

Are there any tools I could use to figure out why?

I once had a slowdown like this when one of my publishers was configured for reliable QoS and my subscriber was sensor QoS, are there any tools for detecting that kind of mismatch?

Asked by johnconn on 2020-02-12 19:02:45 UTC

Comments

My system runs maybe 10x faster on fastRTPS when compared to cyclonedds.

Wow. That would be the first time I hear this claimed by anyone.

Afaik tools go: I'm not aware of any tools for ROS 2 specifically. Many DDS vendors have tools available though. They should at least let you inspect the QoS side of things.

There is the ros2doctor. It's a work-in-progress, and I'm not sure how deep the analysis goes, but it may be worth a try.

Asked by gvdhoorn on 2020-02-13 03:36:42 UTC

I've had good success with cyclone in the past. I must be configuring something wrong

Asked by johnconn on 2020-02-13 08:11:46 UTC

I've had good success with cyclone in the past.

hence my surprised comment.

Asked by gvdhoorn on 2020-02-13 08:37:23 UTC

Answers

I kind of cross-posted this here: https://github.com/ros2/rmw_cyclonedds/issues/251#issuecomment-700490630

There is a lot of good cyclone specific info and more general info on that

The ros2 tuning guide was also a great help to me.

It turned out that the default buffers on ubuntu 18 are too small to achieve good performance on larger images. Once I increased buffer sizes and turned off multicasting after discovery, my performance was much improved.

Asked by johnconn on 2020-10-28 01:18:32 UTC