ROS Resources: Documentation | Support | Discussion Forum | Index | Service Status | Q&A
Ask Your Question

Lag in recieving messages to a subscriber

asked 2021-10-18 02:55:19 -0500

fayyaz gravatar image

updated 2021-10-18 08:27:45 -0500

gvdhoorn gravatar image

I am using foxy with cyclone dds and all the nodes are running in the same host.

I created a simple publisher and subscriber and measured time for the message to reach from the publisher to subscriber. On an average, it took around 0.35ms for a geometry_msgs/Twist data for a subscriber to receive the message published by the publisher.

I tried to use intra process communication to see how much improvement i can get from that. It gave around 0.09ms for the message to be passed to the subscriber.

Is there any way I can speed the message exchange as it seems weird that simple message passing takes this much time when the controller runs at a much faster speed.

I have tried the intra process according to this. I tried fastdds and also different qos profile but I couldn't find much difference in speed.

I could find that intraprocess is not available for service and it also takes around 1 ms for message passing. Is there any way I can improve the speed.


simple pub sub

intra pub sub

edit retag flag offensive close merge delete


Can you describe what you are trying to do where you consider 0.35 milliseconds is too slow? Maybe ros is not the correct middleware.

Mike Scheutzow gravatar image Mike Scheutzow  ( 2021-10-18 07:45:31 -0500 )edit

May be I shouldn't have... But however many of my work is now in ros2, it would be difficult to move away from ros now.. Do you know how i can speed the transmission further or is it the maximum you can have ?

fayyaz gravatar image fayyaz  ( 2021-10-18 08:13:19 -0500 )edit

Is 0.35ms correct? Is it not 0.35 seconds? That's a factor 1000. 0.35 milliseconds is 350 microseconds, or 350 millionth of a second ..

Same for 0.09ms. That's only 90 microseconds.

@fayyaz: could you describe your setup a little? 90 microseconds isn't all that much, but there are of course robots/processes for which it is too much. Especially if there are multiple intermediary pub-sub steps.

gvdhoorn gravatar image gvdhoorn  ( 2021-10-18 08:21:58 -0500 )edit

The plot is in milliseconds. So it is 0.35 ms and 0.09ms.

As there are multiple publisher and subscribers, the delay adds on and it starts to amount to a notable delay. Let's say the encoder data from wheels, imu and other sensors and a node of ekf to fuse them and then passing the pose to the navigation controller, the delay of 0.35ms slowly starts to become relevant eventhough we are running all the nodes in the same host.

I doubt if there might be something which i might have missed to make it faster further and hence i posted the question.

Also in case of intraprocess, I can have that only between two nodes and if I subscribe to the topic in another node, It rolls back to the same nature similar to interprocess.

Also as there is no intraprocess defined for services, I ...(more)

fayyaz gravatar image fayyaz  ( 2021-10-19 02:06:28 -0500 )edit

There are quite some organisations/people/companies using ROS 2 for the kinds of use-cases you describe, so it does surprise me you identify communication delays as problematic. I'm not dismissing your experiences though. It's very likely you have additional requirements and/or constraints which exacerbate this problem.

Typical mobile robots I know don't use ROS to transport encoder ticks or close the loop at that low of a level in the control hierarchy. An interface which publishes JointStates and accepts geometry_msgs/Twist is often seen. The embedded controller of the mobile base then takes care of the rest.

Hosting all of your nodes inside a single container, using intra-process communication can definitely help, but you'd still need to make sure to subscribe to the topics which are less sensitive to pub-sub delays (as you've noticed yourself).

I'm not sure how services are ...(more)

gvdhoorn gravatar image gvdhoorn  ( 2021-10-19 03:48:44 -0500 )edit

But to clarify: I can't give you a direct answer to:

I doubt if there might be something which i might have missed to make it faster

Perhaps someone with more experience using ROS for these kinds of low-level tasks could contribute something.

One last though: if using vanilla ROS 2 is not necessarily a requirement, you could see whether Micro-ROS could help here. That would still allow you to use a ROS 2 interface to your low-level control, while dealing with the low-level control itself in a more traditional embedded way.

gvdhoorn gravatar image gvdhoorn  ( 2021-10-19 03:51:03 -0500 )edit

Thank you for the clarification. It's just that the controller was able to solve an optimization problem in less than a millisecond, but it takes around the same time to just send the optimized solution to a different node. May be running in a realtime kernel might improve the communication faster i guess. Thanks a lot for answering the query quickly.

fayyaz gravatar image fayyaz  ( 2021-10-19 04:11:06 -0500 )edit

Using a middleware to communicate between individual processes is always going to incur a cost.

Intra-process (ie: shared memory) communication can mitigate this somewhat, but it's never going to be as fast as exchanging pointers between functions in a single process. That's just not how this works.

Note that this is not unique to ROS (1 or 2): it's a common characteristic of networking and the way processes work.

gvdhoorn gravatar image gvdhoorn  ( 2021-10-19 04:13:04 -0500 )edit

1 Answer

Sort by ยป oldest newest most voted

answered 2021-10-19 06:51:22 -0500

Mike Scheutzow gravatar image

I'm going to share my opinion: you're using the wrong tool for the job. More precisely, the wrong architecture. It sounds like you are doing some kind of motor control loop, with feedback from a sensor.

In this situation it makes no sense to use a modular, distributed multi-host software system. The overhead makes ROS a poor fit for this problem. You could increase performance by doing all work in a single linux process and eliminating the networking calls entirely.

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Question Tools



Asked: 2021-10-18 02:55:19 -0500

Seen: 408 times

Last updated: Oct 19 '21