| ROS Resources: Documentation | Support | Discussion Forum | Index | Service Status | ros @ Robotics Stack Exchange |
Dear all,
I have implemented stereo_image_proc with customized algorithms. Now I am trying to measure the latency in the pipeline.
I have observed that each nodelet has inconsistency latency ranging form 0 to 60ms. I have measured the latency as follows Ros_latency = (nodelet1_publishing_time - nodelet2_subscribing_time)
How can I minimize these delays so that making the whole system consistent.
Thanks in advance
To get a more specific answer you'll need to provide a lot more information about how to reproduce your issue. In particular you need to understand how any specific component is integrated into the larger system.
Questions that would be helpful for improving performance.
If you can also get 0ms latency that suggests that it can go quickly. And then the question is why is it sometimes not delivering it quickly in other cases.
My speculation guessing at many of the answers to the above question is that you're callback queue is being starved for threads/processing time when you get the large latency numbers. When resources are exhausted we have many ways to gracefully degrade. Everything is queued etc, and if queues overflow content will get dropped. Adding a large processing load in other callbacks can starve other callbacks in the queue etc.
Yes, I am running the nodelets in the same process and all the nodelets are running. The nodelets are consuming not more than 20 threads at a time. Available threads to the manager are 88. Nothing else is running on the system except the roscore and rosbag play.
In short: No, I think you will not be able to reduce the latency between the two nodelets. This is because it originates in the internal ROS system and tweaking it is most possibly overkill.
If there actually is a hard timing requirement, you have two options. Weakening the requirement or using a real time system.
In general, varying latencies between message publishing and receiving are expected and there is few one can do about it.
However, usually you do not need stable latencies. In most cases you are just interested in synchronizing multiple topics. If that is what you want, have a look at http://wiki.ros.org/message_filters , especially point 7, Policy-Based Synchronizer. There are example for synchronizing multiple topics either by exact or by approximate time.
I am not just interested in stabilizing them. I am interested more in reducing the whole latency. The whole system can afford < 70ms of latency. The peak latency I am observing was 130ms and the average latency was around 110ms.
If the 70 ms are a hard requirement you are in need for a real time system. Even if you manage to decrease the mean latency there can be no guarantee that it will not take longer at some point. Afaik timing assumptions are just not possible without RT.
In the overall system the latency observed was varying due to the latency between nodlet2 and nodelet3. This has spikes. Sometimes it was 0 ms delay and sometimes it was 50 ms delay. Is there a way to reduce this. That will be great if we can.
I think the answer is no. Updated my answer to reflect that.
On desktop hardware messaging back and forth can get into the kHz range for closed loop systems over loopback network. As such 60ms strongly suggests something is blocking or being starved for resources on an order of magnitude higher than the simple internal communication overhead.
Asked: 2017-04-12 04:35:23 -0500
Seen: 1,423 times
Last updated: Apr 18 '17
ROS Answers is licensed under Creative Commons Attribution 3.0 Content on this site is licensed under a Creative Commons Attribution Share Alike 3.0 license.
I guess you mean nodelet2_callback_time or something similar? The subscribing time is not very meaningful in this investigation. How exactly do you measure the time? Do you call ros::Time::now() after publishing and at the beginning of the receiving callback?
Yes, I am referring to the nodelet2 callback time. I am using gettimeoftheday() to just before publishing in nodelet1 and just after the callback called in nodelet2. Also, I am using the exact sync polocy.
So your timing suggests that indeed the ROS system takes varying amounts of time to call your callback. How could that be not the case? ROS internal maintenance and callback queues/schedulers is not deterministic in the end. However, I am not sure if 60ms is maybe too much to call it Jitter.