rclpy action server seems slowing down and consuming memory in long term
Hi, please help us.
We are testing ROS2 rclpy action server for long-term usage. The testing action server is almost same to the action_tutorials_py's fibonacci action server. We just change the client node to call the action repeatedly forever.
It looks fine in the beginning, however, when we continue the action calls for long term (over 1 day), the server response is obviously slowing down. In addition, it is getting consuming memory at some growing rate. For example, it consume 0.7% of the memory at the begining, but we can see it consumes 1.2 % after 10 hours and continue to grow. (Using top
command.)
We tested it and got same result with:
- Ubuntu 22.04(WSL2) + Humble(binary install)
- RaspberryPi OS(Debian bullseye) + Humble(build from source)
You can find the reproduction code here. To reproduce this problem, build it and just run the server and client in indivisual terminal as:
ros2 run action_tutorials_py fibonacci_action_server
ros2 run action_tutorials_py fibonacci_action_client
It keeps calling the action forever, and after several hours, you can see the server response is slowing down and the consuming memory is getting increasing.
I added objgraph.show_growth() in the server callback to check if the python objects are growing. It shows the number of some objects (Future, Fibonacci_GetResult_Response, Fibonacci_Result) is increasing one by one corresponding to the action called.
I doubt this can be a critical memory leak of rclpy, but I'm not fully confident on it. Can someone reproduce this problem? I want to report GitHub Issues when others can confirm the problem.
Thank you,
This PR can be a solution! https://github.com/ros2/rclpy/pull/1070
Not sure, but could be related to #q406766.
@longjie0723 you should make your comment the answer. A memory leak fix has been pushed to the repo.
Thank you!