ROS Resources: Documentation | Support | Discussion Forum | Index | Service Status | ros @ Robotics Stack Exchange
Ask Your Question
0

Why Are Some Messages Being Buffered

asked 2020-12-22 01:42:07 -0500

kloonie gravatar image

updated 2020-12-28 01:16:57 -0500

I'm running into a bit of an issue where some messages aren't being delivered right away but appear to be getting buffered. Here's a quick overview of my setup ( I tried posting a block diagram but I don't seem to have enough points to attach an image)

                                                   UART
App Node -> Intermediate Node -> Serializer Node  <----->  Arduino

In essence, the intermediate node subscribes to a number of application nodes, amalgamates data from them and publishes it as a serial command. The serializer is responsible for sending the commands to the Arduino as well as receiving data from the Arduino and publishing it for the app nodes.

The issue is that sometimes I see a case where the serial commands are being buffered and then delivered all at once in quick succession. At first I thought it was the UART interface being busy, but the Serializer Node is still able to receive commands while the buffering is happening so that doesn't seem to be it.

I noticed that if I remove the Intermediate Node and publish Serial commands directly from the App Nodes, then the buffering doesn't happen.

Is there anything inherently wrong with having this double subscriber model? Any idea why the messages are being buffered? Is there any way to prevent messages from being buffered?

Platform: Raspberry Pi4 ROS Version: Kinetic Language: Python (Rospy)

Edit: Adding some more details

  • The intermediate node subscribes to multiple topics but I can reproduce this with a single subscriber
  • The App Node publishes inside a ROS timer running at 100Hz and I monitor the time between subscriber callbacks in both the intermediate and serializer nodes using Pytthon's datetime.now().
  • I don't see any missed callbacks in the intermediate node, but in the serializer node I often see a callback that is several 100ms after the last one, followed by lots of very tightly spaced callbacks (the delayed messages catching up).
edit retag flag offensive close merge delete

Comments

Could this be caused by Nagle's being active on the subscribers in your intermediate node?

gvdhoorn gravatar image gvdhoorn  ( 2020-12-23 14:01:34 -0500 )edit

Thanks for the suggestion, but that doesn't seem to have made a difference. Is there any way I can further debug this using the ROS tools or otherwise?

kloonie gravatar image kloonie  ( 2020-12-28 00:48:50 -0500 )edit

Please update your question with code showing how you configured your subscribers.

gvdhoorn gravatar image gvdhoorn  ( 2020-12-28 02:32:57 -0500 )edit

1 Answer

Sort by ยป oldest newest most voted
1

answered 2020-12-24 04:27:19 -0500

Panagiotis Grimanelis gravatar image

It would really help if you could also post your nodes!

To my understanding, your Intermediate node is subscribed to multiple topics. In order for your Intermediate node to publish a message, it will wait to receive a message from all the subscribed topics. If one of the topics is somehow delayed then the whole node will be delayed while the rest of the topics will start buffering.

Also check your Serializer node, in case it gets stuck in a loop somewhere and for a while everything comes to a stop.

In conclusion, check your loops, rates, buffers and queues. Anyways, there shouldn't be anything wrong with a double subscriber model.

edit flag offensive delete link more

Comments

Unfortunately I cant post the nodes but I really appreciate the help and will try to clarify some points.

  • The intermediate node subscribes to multiple topics but I can reproduce this with a single subscriber
  • The App Node publishes inside a ROS timer running at 100Hz and I monitor the time between subscriber callbacks in both the intermediate and serializer nodes using Pytthon's datetime.now().
  • I don't see any missed callbacks in the intermediate node, but in the serializer node I often see a callback that is several 100ms after the last one, followed by lots of very tightly spaced callbacks (the delayed messages catching up).

So basically I know the App node is publishing at 100Hz. I know the intermediate node is receiving subscriber callbacks and re-publishing at 100Hz inside that callback. However, the serializer is not consistently receiving the callbacks at 100Hz

kloonie gravatar image kloonie  ( 2020-12-28 01:15:40 -0500 )edit

My best guess would be that there is a loop in the Serializer node where it gets stuck for a while. If it is not important for you to receive the entirety of the messages in Arduino you can mess a bit with the queue and the buffer of the publisher in the Serializer. In that way you can skip the delayed messages and go directly to the latest. Depending on the size of the messages you are sending, you can reduce the queue of the publisher and get a better response but you will be losing all your buffered messages.

Panagiotis Grimanelis gravatar image Panagiotis Grimanelis  ( 2020-12-30 14:28:34 -0500 )edit
1

Thanks for the feedback again. I've created a barebones version of my application that doesn't seem to suffer from the same issue so I will start adding functionality until I see it breaking again. I was just hoping there was a more analytical approach to it, but this should at least lead me to the source of the issue.

kloonie gravatar image kloonie  ( 2020-12-30 14:53:53 -0500 )edit

Question Tools

1 follower

Stats

Asked: 2020-12-22 01:42:07 -0500

Seen: 282 times

Last updated: Dec 28 '20