[ROS2] High load average (but little cpu and io)

edit

• Platform: Nvidia Jetson NX (ARM64)
• Ros distribution: humble
• Runtime env: Ubuntu22 docker container
• Host env: ubuntu20, jetpack 5.

We are experiencing performance issues related to high load. Most of our code has been migrated with ros1 with almost no change to the algorithm, thus most of the change is only related to ros2.

We have multiple ros nodes with a combination of cpp and python nodes. While the same software stack would not cause any issue in ros1, with ros2 the load average increases drastically and the system is at times unresponsive. I have seen similar discussions on performance issues in ros2 mainly related to underperformant executors or slow message marshalling.

We havent tried yet to use the custom EventExecutor or change our message and we would like to avoid doing so because:

• we are working on humble which does not yet support the experimental EventExecutor
• "flattening" our custom messages means a big refactoring of the existing code base

What I cannot grasp yet is why such performance drop is associated with little io and cpu. Hopefully, a more experienced user might point to a different system metric, related to ROS2, that could highlight where is the bottleneck.

As a side note, another performance issue we found is that, in this setup, just remotely opening RViz on a networked computer caused additional slowdown of some (already slower) nodes.

Thanks in advance for any insightful answer!