Ask Your Question
1

ROS2 Performance: rclpy is 30x-100x slower than rclcpp

asked 2021-04-09 12:08:36 -0600

Karl gravatar image

updated 2021-11-12 03:13:56 -0600

Hi there,

TL;DR: Publishing a typical 10MB Pointcloud on a very fast system takes only 2.8 ms in rclypp but 92 ms in rclpy.

We observed significant latency/performance problems when publishing large data like images or pointclouds via rclpy. Investigating the problem, we found that publishing large data of any kind is 30-100x slower in Python than in C++.

Corresponding issue on Github: https://github.com/ros2/rclpy/issues/763
Package for easy reproduction of the issue: https://github.com/karl-schulz/ros2_l...

The problem seems to be:

  • Independent of System or OS (we tried source build, binary build, Ubuntu 18, 20, ARM, x64, ..)
  • Not a middleware problem, as rclcpp is fast as expected and we tried both CycloneDDS and FastRTPS
  • Not an inherent Python problem, as copying messages is fast and the ROS1 Python API also works fine

This makes rclpy almost useless when working with high-frequency (more than 10 FPS) image or Pointcloud data, or for low-latency applications.

Did anybody observe the same problem? What could be the reason for this performance problem?

In the worst case: Is is expected behavior?

Help would be really appreciated :)

Edit: Good news! This problem was solved in the ROS2 Galactic (but you need to build from source). If you still want to use foxy you have to compile from source with this fix: https://github.com/ros2/rosidl_python...

edit retag flag offensive close merge delete

Comments

Corresponding issue on Github: https://github.com/ros2/rclpy/issues/763

what's your motivation for cross-posting to Github?

I'd normally immediately close duplicates like this, but perhaps you have a good reason, so if you could clarify?

gvdhoorn gravatar image gvdhoorn  ( 2021-04-10 04:57:54 -0600 )edit

Additionally: a quick search shows this is not a new issue. See #q362928, #q296715, #q331171 and #q323570 (and there are more).

They seem to report the same thing, and have links to issues on the various ROS 2 repository trackers also.

gvdhoorn gravatar image gvdhoorn  ( 2021-04-10 05:00:23 -0600 )edit

Hi @gvdhoorn, I crossposted here because I was curious if anybody else did encounter the same problem and in general to reach a larger audience than the maintainers of rclpy to ask for solutions to the problem, as not many people check the rclpy Github issues.

Karl gravatar image Karl  ( 2021-04-10 17:34:47 -0600 )edit

We spent multiple weeks looking for a corresponding issue/solution, and also saw the ones you mentioned: * #q362928 Here the latency is way higher (900ms) and the bottleneck is the data setter, not the publish(...) call * #q296715 The suggested PYTHONOPTIMIZE=0 did not improve the performance for us * #q331171 and #q323570 are different problems (publishing String, converting from CV2) and also don't have solutions posted.

Also the most issues date back to Bouncy/Crystal and I would assume Foxy should be is maturer and also has the claim to have a reasonably fast Python API. Also considering this is not a problem in ROS1.

If you think that this question doesn't belong here or is a duplicate to the github issue, you can of course close it. But I would not regard it as solved by the previous questions, except if we accept that ROS2 Python is not ...(more)

Karl gravatar image Karl  ( 2021-04-10 17:35:54 -0600 )edit

Thanks for all the additional comments. Forum etiquette encourages posters to mention the Q&As (and GH issues) they've seen. It would have been good to mention (for instance) the four I linked and you apparently had already found, as it would have at least prevented me from spending time finding those posts and commenting about them.

If you think that this question doesn't belong here or is a duplicate to the github issue, you can of course close it.

However you rationalise it, it is a duplicate of your GH issue.

But I would not regard it as solved by the previous questions, except if we accept that ROS2 Python is not really usable for high FPS image data

I don't see how this is relevant. The closure reason would be the duplication, not whether or not it's resolved or not.

gvdhoorn gravatar image gvdhoorn  ( 2021-04-11 01:54:35 -0600 )edit

I crossposted here [..] to reach a larger audience

yes, I understand. That's typically the motivation for doing it.

I don't expect any/too many commenters here, and you already have the maintainers 'on the case'.

Seeing as you already link to the GH issue in your OP, I suggest you close your post here: it will stay visible (so reaches "a wider audience"), but by closing it, we force/encourage discussion and diagnosis to happen on the GH issue (and thereby avoid duplication of effort and split discussions).

gvdhoorn gravatar image gvdhoorn  ( 2021-04-11 01:57:16 -0600 )edit

You are right, I should have mentioned the related issues! And I also think as we have a technical discussion in the Github issue, I will close this here and leave a link to the Github issue once it is resolved for anybody encountering the same issue.

Karl gravatar image Karl  ( 2021-04-11 04:12:45 -0600 )edit

Good news! Fixed by: https://github.com/ros2/rosidl_python... You can easily apply the changes to your local foxy build by replacing the fixed file in rosidl_python.

Karl gravatar image Karl  ( 2021-05-02 08:01:48 -0600 )edit

1 Answer

Sort by ยป oldest newest most voted
1

answered 2021-11-03 06:24:35 -0600

Karl gravatar image

updated 2021-11-12 03:14:49 -0600

Good news!

This problem was solved in ROS2 Galactic. However, you will need to build it from source. I will update this answer when it is also solved when installing via apt.

If you still want to use Foxy you have to compile from source with this fix: https://github.com/ros2/rosidl_python...

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Question Tools

3 followers

Stats

Asked: 2021-04-09 12:08:36 -0600

Seen: 467 times

Last updated: Nov 12 '21