ROS Resources: Documentation | Support | Discussion Forum | Index | Service Status | Q&A answers.ros.org
Ask Your Question
0

Messages drop on slow subscriber although queue not full / empty

asked 2021-11-04 04:22:50 -0500

belorenz gravatar image

updated 2021-11-04 07:04:47 -0500

Problem

Setup: ROS noetic, Ubuntu20.

I have a slow pose estimation node which subscribes to incoming images with a low frequency of only 2Hz. The images are published by a rosbag with 20Hz. The bag contains about 800 images (colored, 1280x1024px) , so they should easily fit into the 10k elements queue of the publisher (rosbag play --queue=10000) and subscriber (message_filters.TimeSynchronizer([image_sub, info_sub],10000)

I would expect that the subscriber queues the incoming images so that every single image can be processed one after another. But instead, already after 3 image messages the dropping of messages starts so that in the end ~60% of the messages are dropped.

While debugging the message_filter.TimeSynchronizer (synchronizes CameraInfo and Image subscriber) I could see that the image queue is never longer than 1, while the camera info queue grew proportional to the published messages.

edit retag flag offensive close merge delete

Comments

The problem I am describing here is already solved. Nevertheless I want to describe it and its solution here, because I think it is a common problem for which I could not find any solution online.

posting this here is perfect, so +100 for that, but could you please post the solution as an answer instead of a section in your question?

If you then accept the answer, it will be completely clear the issue was resolved.

Right now we only see a new and unanswered question on the index page.

gvdhoorn gravatar image gvdhoorn  ( 2021-11-04 05:03:10 -0500 )edit

Added an answer but was not able to accept it as correct since I have < 10 points.

belorenz gravatar image belorenz  ( 2021-11-04 07:03:37 -0500 )edit

1 Answer

Sort by ยป oldest newest most voted
0

answered 2021-11-04 07:02:28 -0500

belorenz gravatar image

Solution

While searching for a solution I learned that message transport in ROS happens with TCP/UDP. Through this hint to the TCP\UDP send buffer I could stop the dropping by increasing the initial and maximum size of the TCP read and write memory (tcp_rmem and tcp_wmem) with the help of this blog post.

The second number is the initial size and the third number is the maximum size of memory of the TCP buffer in bytes. I increased the initial and maximum value by adding the following lines to /etc/sysctl.conf:

net.ipv4.tcp_rmem= 10240 1258291200 2058291201  
net.ipv4.tcp_wmem= 10240 1258291200 2058291201

I choose arbitrary values which simply worked for me (28GB RAM), you might need to adjust these values to your problem.

After saving the file I applied the changes with

sudo sysctl -p
edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Question Tools

2 followers

Stats

Asked: 2021-11-04 04:22:50 -0500

Seen: 141 times

Last updated: Nov 04 '21