Accelerating ROS simulation to faster than real time

asked 2017-12-20 05:42:19 -0600

shardator
43 ●4 ●6 ●12

updated 2017-12-20 09:21:06 -0600

Hi,

It's hard to believe, but according to my research I am the only one who would like to accelerate simulation of a ROS robot... and it seems to be an almost impossible task.

It looks like when I create a node that generates faster than real time clock topic, it totally messes up the simulation. Not even bags can be replayed faster, than real time.

The reason, as I found is that a huge part of the ROS system relies on WallTime and WallDurations, and not just by using these classes, but by doing waits and sleeps in wall times relevant in the simulation.

I tried to create an accelerated version of ROS:

Removed the definitions of WallTime and SteadyTime, replaced them with typedefs to Time
Removed the definition of WallDuration and replaced it with typedef to Duration
Scaled all timed waits with my desired acceleration factor
Instead of a clock topic I share the time from a timer process in shared memory, so all calls to Time::now() are in sync

But now I get stuck in various timing issues related to transformations, more specifically I get errors, like:

Lookup would require extrapolation into the future.  Requested time 1270387817.717999935 but the latest data is at time 1270387817.482000113

(Note that the timestamp is intentionally in the past, as it is an accelerated time, not the real time.)

Now in the stack trace, as I see, this is happening in waitForTransform() call, which is called with Time(0) by the simulation (stdr_simulator). And this is weird, semantically, as I would think (by the name of it) that waitForTransform() is intended to wait, until the transform is available, not throw an exception that it is not (yet).

Could please someone help here?

UPDATE I think the problem is somehow related to BufferCore::getLatestCommonTime(). Somehow the cached times does not match the cached transforms.

edit retag flag offensive close merge delete

Comments

Time(0) has special semantics in TF, it basically means "latest transform".

But in general if use_sim_time is set and there is a publisher on /clock, I would expect things to 'just work'. I've not done it myself, but what you're doing seems rather convoluted.

gvdhoorn ( 2017-12-20 10:04:37 -0600 )edit

No, it does not, because timer delays are expected to be real time. So if you set a timer to 10Hz, it will fire 10Hz real frequency, and not accelerated frequency. This is because WallTime is used, and pthread conditional waits are called with the nominal duration.

shardator ( 2017-12-20 10:11:32 -0600 )edit

Also, I know about the Time(0) semantics, it is just after a while, inside waitForTransform() is replaced by some common time of the two frames (which we are trying to obtain the transformation between). Now this common time is somehow incorrect, as no actual transformation exists with that.

shardator ( 2017-12-20 10:15:21 -0600 )edit

Using simulated time is done with Gazebo and some other sims as well, so your statements confuse me.

I'm not saying you're wrong, but I would just make sure what you're doing is absolutely necessary, as it would be unfortunate otherwise.

gvdhoorn ( 2017-12-20 10:20:08 -0600 )edit

If what you're doing is insidestdr, then that could be true. It may be that the stdr authors didn't (want to) consider FTR / STR cases and just use WallTime everywhere.

gvdhoorn ( 2017-12-20 10:21:29 -0600 )edit

No, I found that ROS core things fire at wrong times. So wrong, that my robot teleports off the map. It has nothing to do with the simulator, I have already excluded that option via debugging. I have recompiled the whole ROS infrastructure with my changes, as I found many places which depend on the

shardator ( 2017-12-20 10:25:29 -0600 )edit

speed of the clock...

shardator ( 2017-12-20 10:25:39 -0600 )edit

Then I would urge you to report that on the proper issue trackers.

If with "ROS core" you actually mean roscpp and the other pkgs in ros_comm, then the tracker would be ros/ros_comm/issues.

gvdhoorn ( 2017-12-20 10:27:35 -0600 )edit

see more comments

2 Answers

Sort by » oldest newest most voted

answered 2017-12-20 13:46:55 -0600

tfoote

58457 ●128 ●550 ●526 http://www.ros.org/

It is generally recommended to use the standard ROS Time by default. There are places which interact with hardware etc that WallTime is necessary since hardware timeouts etc cannot reflect accelerated or declerated time.

I know of many systems that are using simulated time both faster and slower than realtime and have used it successfully myself in both cases. If there are modules that are using WallTime incorrectly please ticket them directly.

And if you're having issues with transforms please provide a reproducible example and open a ticket here: https://github.com/ros/geometry2/issues

Reading your post again it sounds like you're publishing /clock from a node other than the simulator? "when I create a node that generates faster than real time clock topic, it totally messes up the simulation." That suggests that you will have two sources of a /clock and that will wreak havoc on your system. If you're using a simulator and want to speed up time you should tell the simulator to accelerate time not just publish it yourself from another node. With two conflicting /clock sources nodes will be getting whiplash jumping between the two timelines. That will cause immediate timeouts and I would not expect any time dependent algorithm to function correctly.

edit flag offensive delete link

Comments

Hi, The simulator does not provide a clock, I checked that. There is no clock publisher in that. Also, there is no way _not_ to use WallTime/SteadyTime as some parts of ROS utilize that. TimerManager in particular. And we definitely have timers.

shardator ( 2017-12-20 15:18:27 -0600 )edit

I don't know about stdr's implementation but both stage and gazebo ros packages implement the /clock API and can run faster than realtime.

tfoote ( 2017-12-20 15:36:51 -0600 )edit

Can you please provide an example how roscpp::TimerManager cannot work with accelerated realtime? Looking at the code it does use WallTime/SteadyTIme but only for collecting statistics, but otherwise it's templated to use the time source of the timer object that you passed into it.

tfoote ( 2017-12-20 15:40:08 -0600 )edit

Pardon me my ignorance, but I fail to see how. If I set a timer in my robot's own code, it will fire with physical, real frequency I've given. All timeouts, every mutex waits happen in real time, unaccelerated in the code of ROS core libs. How is it possible to run an equivalent sim then accelerated

shardator ( 2017-12-20 15:42:17 -0600 )edit

If you use the ROS Time abstraction and enable simulated time. The roscpp Timers will also follow the abstraction. Please try some small test nodes to see how things work

tfoote ( 2017-12-20 16:02:09 -0600 )edit

I did that. It simply does not work correctly for anything complex. Only for very simple scenarios. Do you happen to know, how timers work in simulated time conditions? I know for fact that SteadyTime is being used, which is not correct for acceleration.

shardator ( 2017-12-20 16:09:19 -0600 )edit

We even get incorrect timings in unaccelerated simulated time, which does not actually contain the current real time. It works, but with tiny, but unacceptable glitches, which ruin our sensitive mapping and routing algorithm. When clock contains real time values, it works.

shardator ( 2017-12-20 16:11:01 -0600 )edit

They work like this: https://github.com/ros/ros_comm/blob/...

Please provide a reproducible example of your "unacceptable glitches" otherwise I don't think there's much more I can help you with.

tfoote ( 2017-12-20 16:29:21 -0600 )edit

see more comments

answered 2017-12-22 04:59:53 -0600

Felix Widmaier
382 ●18 ●32 ●40

When using Gazebo for simulation, there is a simple way to accelerate time: In the right panel, there is a tab World. In this tab go to Physics and search for the value "real time update rate".

Simply increase this value to accelerate the simulation time (e.g. double it to accelerate by factor 2).

edit flag offensive delete link

Comments

Hi! Thanks. We need to do batch simulations (with perturbed parameters) for optimization, so we need like 20x accelerated, scripted simulation. Is it possible to do that with Gazebo? Is not Gazebo using too much resources (we just do 2D simulations, we don't need 3D...)

shardator ( 2017-12-22 06:24:26 -0600 )edit

You are right, Gazebo takes a lot of resources. I have successfully accelerated by factor 2 or 3 in the past but 20 is probably too much (at least for a normal computer).

Felix Widmaier ( 2017-12-22 07:04:18 -0600 )edit

:( We've explored simple options already. This looks like a difficult task. I think it would be nice to have a 10x accelerated simulation, but I can't even do that consistently.

shardator ( 2017-12-22 07:09:23 -0600 )edit

add a comment

Accelerating ROS simulation to faster than real time

Comments

2 Answers

Comments

Comments

Question Tools

Stats

Related questions

Accelerating ROS simulation to faster than real time edit

Comments

2 Answers

Comments

Comments

Question Tools

Stats

Related questions

Accelerating ROS simulation to faster than real time