rosout high memory usage

asked 2019-11-06 11:28:09 -0500

ixil
23 ●1 ●2 ●6

updated 2022-01-22 16:10:08 -0500

Evgeny
26 ●6

I'm running melodic on a ubuntu 18.04 container (through systemd-nspawn inside Archlinux:

uname -a: Linux rsl-bionic 5.3.8-arch1-1 #1 SMP PREEMPT @1572357769 x86_64 x86_64 x86_64 GNU/Linux

Using only rosrun rosout rosout Uses upwards of 8.6Gb of Resident memory. I don't have the ponts to upload the traces, also I'm not really sure what I'm doing when I am trying to trace it output. But in one allocation 8.6Gb seems to be initialized in the XmlRpcServer in a

std::vector<pollfd, std::allocator<pollfd> > ::resize(unsigned long) in ?? (libroscpp.so)

Any pointers for fixing/tracing this? I'll probably have to give up and repartition my machine and install ubuntu natively, but I can't reason as to why/how it could be related.

answered 2019-12-19 14:16:44 -0500

Ralino
46 ●3 ●2

I am running the same setup and ran into the same problem but managed to fix it after creating a Debug build of the core ros libraries and analyzing the code of the XmlRpcServer where the massive memory allocation happens.

The XmlRpcServer in ros_comm sets up a vector for querying all possible file descriptors to check if it has run into the limit of opened files. When running a container on Arch however, this limit is 1073741816. With each file descriptor having a size of 8 Byte, thats 8GB for every ros node. You can check the current (hard) limit of the running shell with ulimit -Hn. If you reduce the limit with ulimit -Hn 524288, the memory usage should be fine again.

You can make this limit persistent by adding the file /etc/security/limits.d/30-nofilelimit.conf in the container with the following content:

#<domain>  <type>  <item>  <value>

*          soft    nofile  1024
*          hard    nofile  524288

edit flag offensive delete link

Comments

This is some good detective work @Ralino.

I would strongly suggest you open an issue on ros/ros_comm about this. Please provide sufficient detail about how you found out and where the vector is created, and a link to this Q&A.

Seeing as the focus of development is on ROS 2 these days I cannot guarantee it will be fixed (or implemented differently), but it would be tremendously valuable to have a record of your findings on the issue tracker regardless.

gvdhoorn ( 2019-12-20 03:36:07 -0500 )edit

@Ralino thanks for taking the time to post your work. I gave up in the end using the container approach. But relieved to know there is a solution

ixil ( 2019-12-20 04:58:55 -0500 )edit

If anyone is having trouble with setting the hard limit: check out https://askubuntu.com/questions/10615..., you may also need to run ulimit -Sn 524288.

pumaking ( 2023-03-22 00:07:12 -0500 )edit

You can also use docker run ... --ulimit nofile=1024:524288 to set the soft and hard nofile limits. If you want this for all containers consider adding:

"default-ulimits": {
    "nofile": {
        "Name": "nofile",
        "Hard": 524288,
        "Soft": 1024
    }
}

To your /etc/docker/daemon.json file.

danambrosio ( 2023-03-27 22:09:53 -0500 )edit

add a comment

1 Answer

Comments

Question Tools

Stats

Related questions

rosout high memory usage edit

1 Answer

Comments

Question Tools

Stats

Related questions

rosout high memory usage