ROS2 Performance: rclpy is 30x-100x slower than rclcpp
Hi there,
TL;DR: Publishing a typical 10MB Pointcloud on a very fast system takes only 2.8 ms in rclypp but 92 ms in rclpy.
We observed significant latency/performance problems when publishing large data like images or pointclouds via rclpy. Investigating the problem, we found that publishing large data of any kind is 30-100x slower in Python than in C++.
Corresponding issue on Github: https://github.com/ros2/rclpy/issues/763
Package for easy reproduction of the issue: https://github.com/karl-schulz/ros2_l...
The problem seems to be:
- Independent of System or OS (we tried source build, binary build, Ubuntu 18, 20, ARM, x64, ..)
- Not a middleware problem, as rclcpp is fast as expected and we tried both CycloneDDS and FastRTPS
- Not an inherent Python problem, as copying messages is fast and the ROS1 Python API also works fine
This makes rclpy almost useless when working with high-frequency (more than 10 FPS) image or Pointcloud data, or for low-latency applications.
Did anybody observe the same problem? What could be the reason for this performance problem?
In the worst case: Is is expected behavior?
Help would be really appreciated :)
Edit: Good news! This problem was solved in the ROS2 Galactic (but you need to build from source). If you still want to use foxy you have to compile from source with this fix: https://github.com/ros2/rosidl_python...
what's your motivation for cross-posting to Github?
I'd normally immediately close duplicates like this, but perhaps you have a good reason, so if you could clarify?
Additionally: a quick search shows this is not a new issue. See #q362928, #q296715, #q331171 and #q323570 (and there are more).
They seem to report the same thing, and have links to issues on the various ROS 2 repository trackers also.
Hi @gvdhoorn, I crossposted here because I was curious if anybody else did encounter the same problem and in general to reach a larger audience than the maintainers of rclpy to ask for solutions to the problem, as not many people check the rclpy Github issues.
We spent multiple weeks looking for a corresponding issue/solution, and also saw the ones you mentioned: * #q362928 Here the latency is way higher (900ms) and the bottleneck is the data setter, not the publish(...) call * #q296715 The suggested PYTHONOPTIMIZE=0 did not improve the performance for us * #q331171 and #q323570 are different problems (publishing String, converting from CV2) and also don't have solutions posted.
Also the most issues date back to Bouncy/Crystal and I would assume Foxy should be is maturer and also has the claim to have a reasonably fast Python API. Also considering this is not a problem in ROS1.
If you think that this question doesn't belong here or is a duplicate to the github issue, you can of course close it. But I would not regard it as solved by the previous questions, except if we accept that ROS2 Python is not ...(more)
Thanks for all the additional comments. Forum etiquette encourages posters to mention the Q&As (and GH issues) they've seen. It would have been good to mention (for instance) the four I linked and you apparently had already found, as it would have at least prevented me from spending time finding those posts and commenting about them.
However you rationalise it, it is a duplicate of your GH issue.
I don't see how this is relevant. The closure reason would be the duplication, not whether or not it's resolved or not.
yes, I understand. That's typically the motivation for doing it.
I don't expect any/too many commenters here, and you already have the maintainers 'on the case'.
Seeing as you already link to the GH issue in your OP, I suggest you close your post here: it will stay visible (so reaches "a wider audience"), but by closing it, we force/encourage discussion and diagnosis to happen on the GH issue (and thereby avoid duplication of effort and split discussions).
You are right, I should have mentioned the related issues! And I also think as we have a technical discussion in the Github issue, I will close this here and leave a link to the Github issue once it is resolved for anybody encountering the same issue.
Good news! Fixed by: https://github.com/ros2/rosidl_python... You can easily apply the changes to your local foxy build by replacing the fixed file in rosidl_python.