Ask Your Question
0

Latency between nodelets

asked 2017-04-12 04:35:23 -0600

mkreddy477 gravatar image

Dear all,

I have implemented stereo_image_proc with customized algorithms. Now I am trying to measure the latency in the pipeline.

I have observed that each nodelet has inconsistency latency ranging form 0 to 60ms. I have measured the latency as follows Ros_latency = (nodelet1_publishing_time - nodelet2_subscribing_time)

How can I minimize these delays so that making the whole system consistent.

Thanks in advance

edit retag flag offensive close merge delete

Comments

1

I guess you mean nodelet2_callback_time or something similar? The subscribing time is not very meaningful in this investigation. How exactly do you measure the time? Do you call ros::Time::now() after publishing and at the beginning of the receiving callback?

cellard0or gravatar imagecellard0or ( 2017-04-12 08:57:27 -0600 )edit

Yes, I am referring to the nodelet2 callback time. I am using gettimeoftheday() to just before publishing in nodelet1 and just after the callback called in nodelet2. Also, I am using the exact sync polocy.

mkreddy477 gravatar imagemkreddy477 ( 2017-04-13 00:26:32 -0600 )edit

So your timing suggests that indeed the ROS system takes varying amounts of time to call your callback. How could that be not the case? ROS internal maintenance and callback queues/schedulers is not deterministic in the end. However, I am not sure if 60ms is maybe too much to call it Jitter.

cellard0or gravatar imagecellard0or ( 2017-04-13 02:18:01 -0600 )edit

2 Answers

Sort by ยป oldest newest most voted
0

answered 2017-04-12 15:12:48 -0600

tfoote gravatar image

To get a more specific answer you'll need to provide a lot more information about how to reproduce your issue. In particular you need to understand how any specific component is integrated into the larger system.

Questions that would be helpful for improving performance.

  • Are you running the nodelets in the same process?
  • Is there other nodelets running?
  • How many threads are available in the nodelet manager? How many do you expect used at any given time?
  • Are you CPU constrained?
  • What else is running on the system?
  • What else is publishing or subscribing in the node on the system?

If you can also get 0ms latency that suggests that it can go quickly. And then the question is why is it sometimes not delivering it quickly in other cases.

My speculation guessing at many of the answers to the above question is that you're callback queue is being starved for threads/processing time when you get the large latency numbers. When resources are exhausted we have many ways to gracefully degrade. Everything is queued etc, and if queues overflow content will get dropped. Adding a large processing load in other callbacks can starve other callbacks in the queue etc.

edit flag offensive delete link more

Comments

Yes, I am running the nodelets in the same process and all the nodelets are running. The nodelets are consuming not more than 20 threads at a time. Available threads to the manager are 88. Nothing else is running on the system except the roscore and rosbag play.

mkreddy477 gravatar imagemkreddy477 ( 2017-04-13 00:36:53 -0600 )edit
0

answered 2017-04-12 09:11:50 -0600

cellard0or gravatar image

updated 2017-04-18 01:29:01 -0600

In short: No, I think you will not be able to reduce the latency between the two nodelets. This is because it originates in the internal ROS system and tweaking it is most possibly overkill.
If there actually is a hard timing requirement, you have two options. Weakening the requirement or using a real time system.

In general, varying latencies between message publishing and receiving are expected and there is few one can do about it.

However, usually you do not need stable latencies. In most cases you are just interested in synchronizing multiple topics. If that is what you want, have a look at http://wiki.ros.org/message_filters , especially point 7, Policy-Based Synchronizer. There are example for synchronizing multiple topics either by exact or by approximate time.

edit flag offensive delete link more

Comments

I am not just interested in stabilizing them. I am interested more in reducing the whole latency. The whole system can afford < 70ms of latency. The peak latency I am observing was 130ms and the average latency was around 110ms.

mkreddy477 gravatar imagemkreddy477 ( 2017-04-13 00:38:03 -0600 )edit

If the 70 ms are a hard requirement you are in need for a real time system. Even if you manage to decrease the mean latency there can be no guarantee that it will not take longer at some point. Afaik timing assumptions are just not possible without RT.

cellard0or gravatar imagecellard0or ( 2017-04-13 02:06:13 -0600 )edit

In the overall system the latency observed was varying due to the latency between nodlet2 and nodelet3. This has spikes. Sometimes it was 0 ms delay and sometimes it was 50 ms delay. Is there a way to reduce this. That will be great if we can.

mkreddy477 gravatar imagemkreddy477 ( 2017-04-14 00:49:22 -0600 )edit

I think the answer is no. Updated my answer to reflect that.

cellard0or gravatar imagecellard0or ( 2017-04-18 01:29:27 -0600 )edit
1

On desktop hardware messaging back and forth can get into the kHz range for closed loop systems over loopback network. As such 60ms strongly suggests something is blocking or being starved for resources on an order of magnitude higher than the simple internal communication overhead.

tfoote gravatar imagetfoote ( 2017-04-18 01:56:18 -0600 )edit

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Question Tools

1 follower

Stats

Asked: 2017-04-12 04:35:23 -0600

Seen: 842 times

Last updated: Apr 18 '17