Hi everyone,

We are using ROS as a basis for our robotics/AI project. It's been great so far with all the tools and code it provides, however today I ran into a small (big?) problem.

A small conceptual explanation of how our system is designed and how it works. We have some input on a socket port which is basically visual data. We have a ros node that reads this port out, does some processing, and publishes the data ready for consumption on a topic. This happens at 60Hz. We have another node that consumes this visual data and sends instructions out a serial port, the control node. This also happens at 60Hz. So far so good; this is standard ROS stuff and besides some minor caveats we've had no problems using this model.

Now here comes my problem. Some buffering takes place in the visual node when publishing the processed data. So at 60Hz we receive visual data from the socket, and at 60Hz we call publisher.publish(processedVisualData);. What sometimes happens is that new data from the socket arrives before the previously published data is actually sent to the control node. I know this for sure because sometimes this happens in the control node (function calls derived from print statements throughout my program):

• "Start spinning..."
• ros::spinOnce()
• visualUpdateCallback();
• "Ros spinning complete"
• runControl()
• "Control finished."
• "Start spinning..."
• ros::spinOnce()
• "Ros spinning complete"
• runControl()
• "Control finished."

So what happens here is because of the buffering in the visual node (or in the TCP of the subscriber? I've been looking at the ROS source code a lot lately and I've seen buffers in literally every class I look at) the control node sometimes runs without a new visual update, even though that visual update was given to the publisher in the visual node. So even though both nodes run at 60Hz (I've confirmed this with rostopic hz; the throughput of visual messages and control messages is almost 60Hz), the control nodes effectively runs at 30Hz, because every odd cycle it has to reuse "stale" data.

There are a few possible solutions I've found:

• Set the queue of the publisher in the visual node to one. That ensures that everytime a message is being sent it's guaranteed to be the most up-to-date. The buffering delay is somehow still in place however, so control still runs at 30Hz. (I tested this by only running the control function when new data arrived. It's output was 30 Hz even though the visual update frequency was 60Hz.)
• Produce more messages. I guess this can work, but there are 2 limitations
  • The visual input really runs strictly at 60Hz. I can't make it produce more data
  • Buffering is still in place. So even if I could crank the visual data up to 100Hz, then my control node will still run at a percentage of that. That's bad because it makes performance unpredictable: for some topics, if I send messages at 60Hz, I would like any other nodes to receive them at 60Hz, and not at some buffered 30Hz as it is now.
• Use nodelets. Arguably this is the best solution IMO, since it also removes overhead from TCP (which we've already had some performance issues with in the past w.r.t. many frequently accessed ROS parameters) and removes copying overhead, etc. However we are constrained by time and as far as I can tell this will be a pretty major rewrite that we can't afford.
• Change line 215 on http://docs.ros.org/jade/api/roscpp/html/transportsubscriberlink8cppsource.html from false to true. Disadvantages:
  • Will require me to build a custom version from ROS and merge it every once in a while. I don't want to add this complexity to our project since we already have enough of that.
  • Will make every topic in the whole project immediate/eager, while I actually like this performance for some topics (mostly debug ones).
  • Unsure about whether or not it will work, for two reasons:
    • Not sure if it even does what I think it does (i.e. transfer the message to TCP as soon as it appears. Maybe it only sends half the message and puts the rest in some other leftover buffer, which doens't solve the problem but is still immediate)
    • Even if it does that buffering might still be a problem because, as mentioned before, ROS uses buffers everywhere, so somehow I'd have to short-circuit all those buffers.

In the best case scenario there would be an option in ROS publishers/subscribers or in the roscore program to make some topics "real-time", i.e. to make sure that some pieces of data get pumped through the system A.S.A.P., and don't lie around for several milliseconds in some buffers. However I've googled quite a bit already and read a lot of ROS source, and there doesn't seem to be a button for that (except for the earlier thing on line 215, but that's not configurable nor reliable). I really hope I'm wrong though.

We can probably work around it, but the situation as it is now is highly undesirable.

EDIT: As suggested by @gvdhoorn tcpNoDelay can be set on subscribers. This improves my control throughput from 30 to 45 "fresh" cycles per second. The problem is still not solved but it's already better with very little effort. I have a feeling that if tcpNoDelay would also be present on publishers the problem would be solved; at the time of writing however this is not the case.