rosbag record performance issues
Hello ROS community,
We are running our ROS system on a 6-core multiprocessor machine (8GB RAM, 32GB eMMC storage, 64bit ubuntu Linux) and experience significant load increase when turning on ROS recording.
In detail, without recording our system has roughly a load of 1 between 2. Once we turn on recording (rosbag record -a) our load climbs up to roughly 5 or even 6.
The amount of data we log is roughly 200KB/sec and is mostly comprised of float32 values. This is not insignificant, but I am still surprised that recording has such a high impact.
Since we do not fully max out our CPUs, I assume the load is mostly related to IO!?
My questions:
Given the amount of data we log, is this load profile "normal"? or are there any tips to improve this?
if this can hardly be improved would it make sense to isolate the recording process on a separate machine to protect our core robotics system from slowing down?
Thanks a lot for reading.
the first thing would probably be to do some profiling, then we know what is going on.
Take a look at rosbag/Commandline - record, have you investigated the effect of
buffsizeandchunksize?Additionaly - but without profiling this is really all speculative: network topology and other system characteristics are going to influence everything, so it might be good to describe that a bit.
Is the increased load only in the rosbag process? I've found that recording high-throughput topics with no other subscribers can add significant serialization overhead to the publisher (which was previously idle).
@gvdhoorn, I played with buff- & chunk-size, but this did not get big improvements. I only did some system profiling (iostat, netstat, etc.) but still need to profile the record node and others.
@Ed Venator, yep, this is true, I see other nodes' CPU rising when turning on recording.
@Ed Venator, I guess besides reducing messages sizes (which I have done as much as possible), reducing frequencies and not recording everything, there is not much to do to improve performance?
There's probably not much you can do. You could use topic_tools/throttle to reduce the rate of the topics going into the record node, but there's no way to completely isolate the effects of bagging.
Just to get an idea: if load is increased because
rosbagintroduces additional subscribers then it might be interesting to know what sort of nrs of nodes we're talking about here. 10, 100?rosbag record -aobviously records everything, so that could result in quite some additional subscriptions.Do you have any nodelets, or is everything only nodes? I don't know what the status is, but if you have nodelets then osrf/nodelet_rosbag could be interesting.