Ask Your Question
0

rosout high memory usage

asked 2019-11-06 11:28:09 -0600

ixil gravatar image

updated 2019-11-07 03:32:54 -0600

I'm running melodic on a ubuntu 18.04 container (through systemd-nspawn inside Archlinux:

uname -a: Linux rsl-bionic 5.3.8-arch1-1 #1 SMP PREEMPT @1572357769 x86_64 x86_64 x86_64 GNU/Linux

Using only rosrun rosout rosout Uses upwards of 8.6Gb of Resident memory. I don't have the ponts to upload the traces, also I'm not really sure what I'm doing when I am trying to trace it output. But in one allocation 8.6Gb seems to be initialized in the XmlRpcServer in a

std::vector<pollfd, std::allocator<pollfd> > ::resize(unsigned long) in ?? (libroscpp.so)

Any pointers for fixing/tracing this? I'll probably have to give up and repartition my machine and install ubuntu natively, but I can't reason as to why/how it could be related.

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted
1

answered 2019-12-19 14:16:44 -0600

Ralino gravatar image

I am running the same setup and ran into the same problem but managed to fix it after creating a Debug build of the core ros libraries and analyzing the code of the XmlRpcServer where the massive memory allocation happens.

The XmlRpcServer in ros_comm sets up a vector for querying all possible file descriptors to check if it has run into the limit of opened files. When running a container on Arch however, this limit is 1073741816. With each file descriptor having a size of 8 Byte, thats 8GB for every ros node. You can check the current (hard) limit of the running shell with ulimit -Hn. If you reduce the limit with ulimit -Hn 524288, the memory usage should be fine again.

You can make this limit persistent by adding the file /etc/security/limits.d/30-nofilelimit.conf in the container with the following content:

#<domain>  <type>  <item>  <value>

*          soft    nofile  1024
*          hard    nofile  524288
edit flag offensive delete link more

Comments

This is some good detective work @Ralino.

I would strongly suggest you open an issue on ros/ros_comm about this. Please provide sufficient detail about how you found out and where the vector is created, and a link to this Q&A.

Seeing as the focus of development is on ROS 2 these days I cannot guarantee it will be fixed (or implemented differently), but it would be tremendously valuable to have a record of your findings on the issue tracker regardless.

gvdhoorn gravatar image gvdhoorn  ( 2019-12-20 03:36:07 -0600 )edit

@Ralino thanks for taking the time to post your work. I gave up in the end using the container approach. But relieved to know there is a solution

ixil gravatar image ixil  ( 2019-12-20 04:58:55 -0600 )edit

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Question Tools

2 followers

Stats

Asked: 2019-11-06 11:28:09 -0600

Seen: 216 times

Last updated: Nov 07 '19