ROS Resources: Documentation | Support | Discussion Forum | Index | Service Status | ros @ Robotics Stack Exchange
Ask Your Question

rosout high memory usage

asked 2019-11-06 11:28:09 -0600

ixil gravatar image

updated 2022-01-22 16:10:08 -0600

Evgeny gravatar image

I'm running melodic on a ubuntu 18.04 container (through systemd-nspawn inside Archlinux:

uname -a: Linux rsl-bionic 5.3.8-arch1-1 #1 SMP PREEMPT @1572357769 x86_64 x86_64 x86_64 GNU/Linux

Using only rosrun rosout rosout Uses upwards of 8.6Gb of Resident memory. I don't have the ponts to upload the traces, also I'm not really sure what I'm doing when I am trying to trace it output. But in one allocation 8.6Gb seems to be initialized in the XmlRpcServer in a

std::vector<pollfd, std::allocator<pollfd> > ::resize(unsigned long) in ?? (

Any pointers for fixing/tracing this? I'll probably have to give up and repartition my machine and install ubuntu natively, but I can't reason as to why/how it could be related.

edit retag flag offensive close merge delete

1 Answer

Sort by » oldest newest most voted

answered 2019-12-19 14:16:44 -0600

Ralino gravatar image

I am running the same setup and ran into the same problem but managed to fix it after creating a Debug build of the core ros libraries and analyzing the code of the XmlRpcServer where the massive memory allocation happens.

The XmlRpcServer in ros_comm sets up a vector for querying all possible file descriptors to check if it has run into the limit of opened files. When running a container on Arch however, this limit is 1073741816. With each file descriptor having a size of 8 Byte, thats 8GB for every ros node. You can check the current (hard) limit of the running shell with ulimit -Hn. If you reduce the limit with ulimit -Hn 524288, the memory usage should be fine again.

You can make this limit persistent by adding the file /etc/security/limits.d/30-nofilelimit.conf in the container with the following content:

#<domain>  <type>  <item>  <value>

*          soft    nofile  1024
*          hard    nofile  524288
edit flag offensive delete link more



This is some good detective work @Ralino.

I would strongly suggest you open an issue on ros/ros_comm about this. Please provide sufficient detail about how you found out and where the vector is created, and a link to this Q&A.

Seeing as the focus of development is on ROS 2 these days I cannot guarantee it will be fixed (or implemented differently), but it would be tremendously valuable to have a record of your findings on the issue tracker regardless.

gvdhoorn gravatar image gvdhoorn  ( 2019-12-20 03:36:07 -0600 )edit

@Ralino thanks for taking the time to post your work. I gave up in the end using the container approach. But relieved to know there is a solution

ixil gravatar image ixil  ( 2019-12-20 04:58:55 -0600 )edit

If anyone is having trouble with setting the hard limit: check out, you may also need to run ulimit -Sn 524288.

pumaking gravatar image pumaking  ( 2023-03-22 00:07:12 -0600 )edit

You can also use docker run ... --ulimit nofile=1024:524288 to set the soft and hard nofile limits. If you want this for all containers consider adding:

"default-ulimits": {
    "nofile": {
        "Name": "nofile",
        "Hard": 524288,
        "Soft": 1024

To your /etc/docker/daemon.json file.

danambrosio gravatar image danambrosio  ( 2023-03-27 22:09:53 -0600 )edit

Question Tools



Asked: 2019-11-06 11:28:09 -0600

Seen: 1,666 times

Last updated: Nov 07 '19