ROS Resources: Documentation | Support | Discussion Forum | Index | Service Status | ros @ Robotics Stack Exchange
Ask Your Question
0

rclpy action server seems slowing down and consuming memory in long term

asked 2023-01-14 20:26:22 -0500

longjie0723 gravatar image

Hi, please help us.

We are testing ROS2 rclpy action server for long-term usage. The testing action server is almost same to the action_tutorials_py's fibonacci action server. We just change the client node to call the action repeatedly forever.

It looks fine in the beginning, however, when we continue the action calls for long term (over 1 day), the server response is obviously slowing down. In addition, it is getting consuming memory at some growing rate. For example, it consume 0.7% of the memory at the begining, but we can see it consumes 1.2 % after 10 hours and continue to grow. (Using top command.)

We tested it and got same result with:

  • Ubuntu 22.04(WSL2) + Humble(binary install)
  • RaspberryPi OS(Debian bullseye) + Humble(build from source)

You can find the reproduction code here. To reproduce this problem, build it and just run the server and client in indivisual terminal as:

ros2 run action_tutorials_py fibonacci_action_server
ros2 run action_tutorials_py fibonacci_action_client

It keeps calling the action forever, and after several hours, you can see the server response is slowing down and the consuming memory is getting increasing.

I added objgraph.show_growth() in the server callback to check if the python objects are growing. It shows the number of some objects (Future, Fibonacci_GetResult_Response, Fibonacci_Result) is increasing one by one corresponding to the action called.

Increasing objects

I doubt this can be a critical memory leak of rclpy, but I'm not fully confident on it. Can someone reproduce this problem? I want to report GitHub Issues when others can confirm the problem.

Thank you,

edit retag flag offensive close merge delete

Comments

This PR can be a solution! https://github.com/ros2/rclpy/pull/1070

longjie0723 gravatar image longjie0723  ( 2023-01-15 05:10:30 -0500 )edit

Not sure, but could be related to #q406766.

gvdhoorn gravatar image gvdhoorn  ( 2023-01-15 06:40:57 -0500 )edit

@longjie0723 you should make your comment the answer. A memory leak fix has been pushed to the repo.

Mike Scheutzow gravatar image Mike Scheutzow  ( 2023-01-22 09:11:48 -0500 )edit

Thank you!

longjie0723 gravatar image longjie0723  ( 2023-01-22 18:25:09 -0500 )edit

1 Answer

Sort by ยป oldest newest most voted
0

answered 2023-01-22 18:24:16 -0500

longjie0723 gravatar image

This was found to be due to a memory leak in rclpy. I have confirmed that this pull request fixes my problem. It will eventually be merged into the release. Thanks to all contributors!

edit flag offensive delete link more

Question Tools

3 followers

Stats

Asked: 2023-01-14 20:26:22 -0500

Seen: 337 times

Last updated: Jan 22 '23