Ensuring time synchronization on multi-machine setup
Hi!
I'm looking into time synchronization on multi-machine setup (that might or might not have access to internet while it's running). Let's assume 3 computers for the time being and no internet access.
The idea I've been playing with is to use one machine as NTP server (ideally the same that runs the ROS master) - that way I take the master's computer's time as system time.
With this setup I don't have any control over the boot time - sometimes the master machine might boot up after the slave machines are running. Since on the slave machines I'm waiting for roscore to launch on the master machine this slightly helps but not quite. If the time difference between machines is significant what can often happen is that the time gets step changed after all the slave nodes are running, which breaks the timers and what not.
Did anyone of you had an issue like this? How did you solve it? I'm using chrony, and on large time differences between server and client the skew doesn't seem to work and jumps tend to break things. Is there any solid way I can ensure clocks are synchronized before running any nodes?
Tbh this sounds like something that should be solved "outside of ROS". Chrony is already a good step. I'd try to look for solutions without ROS in mind for coordinating when programs get started. Perhaps a simple shell script which checks whether
chronyhas had a chance to run and blocks .... otherwise.
I agree that this would make sense. This could work nicely with robot_upstart given some development time.
In other news I noticed that in roscpp time shifting back and forth might break or not the timer callbacks depending how ROS is spinning. I created an issue here that describes my findings in more details: https://github.com/ros/ros_comm/issue...