Ask Your Question
0

rosbag record performance issues

asked 2017-07-11 21:07:44 -0600

rob_ros gravatar image

updated 2017-07-11 22:44:33 -0600

Hello ROS community,

We are running our ROS system on a 6-core multiprocessor machine (8GB RAM, 32GB eMMC storage, 64bit ubuntu Linux) and experience significant load increase when turning on ROS recording.

In detail, without recording our system has roughly a load of 1 between 2. Once we turn on recording (rosbag record -a) our load climbs up to roughly 5 or even 6.

The amount of data we log is roughly 200KB/sec and is mostly comprised of float32 values. This is not insignificant, but I am still surprised that recording has such a high impact.

Since we do not fully max out our CPUs, I assume the load is mostly related to IO!?

My questions:

  • Given the amount of data we log, is this load profile "normal"? or are there any tips to improve this?

  • if this can hardly be improved would it make sense to isolate the recording process on a separate machine to protect our core robotics system from slowing down?

Thanks a lot for reading.

edit retag flag offensive close merge delete

Comments

I assume the load is mostly related to IO!?

the first thing would probably be to do some profiling, then we know what is going on.

Take a look at rosbag/Commandline - record, have you investigated the effect of buffsize and chunksize?

gvdhoorn gravatar imagegvdhoorn ( 2017-07-12 01:54:49 -0600 )edit

Additionaly - but without profiling this is really all speculative: network topology and other system characteristics are going to influence everything, so it might be good to describe that a bit.

gvdhoorn gravatar imagegvdhoorn ( 2017-07-12 01:56:39 -0600 )edit

Is the increased load only in the rosbag process? I've found that recording high-throughput topics with no other subscribers can add significant serialization overhead to the publisher (which was previously idle).

Ed Venator gravatar imageEd Venator ( 2017-07-12 21:17:29 -0600 )edit

@gvdhoorn, I played with buff- & chunk-size, but this did not get big improvements. I only did some system profiling (iostat, netstat, etc.) but still need to profile the record node and others.

@Ed Venator, yep, this is true, I see other nodes' CPU rising when turning on recording.

rob_ros gravatar imagerob_ros ( 2017-07-17 13:42:11 -0600 )edit

@Ed Venator, I guess besides reducing messages sizes (which I have done as much as possible), reducing frequencies and not recording everything, there is not much to do to improve performance?

rob_ros gravatar imagerob_ros ( 2017-07-17 13:44:05 -0600 )edit

There's probably not much you can do. You could use topic_tools/throttle to reduce the rate of the topics going into the record node, but there's no way to completely isolate the effects of bagging.

Ed Venator gravatar imageEd Venator ( 2017-07-18 19:45:49 -0600 )edit

Just to get an idea: if load is increased because rosbag introduces additional subscribers then it might be interesting to know what sort of nrs of nodes we're talking about here. 10, 100? rosbag record -a obviously records everything, so that could result in quite some additional subscriptions.

gvdhoorn gravatar imagegvdhoorn ( 2017-07-19 06:08:28 -0600 )edit

Do you have any nodelets, or is everything only nodes? I don't know what the status is, but if you have nodelets then osrf/nodelet_rosbag could be interesting.

gvdhoorn gravatar imagegvdhoorn ( 2017-07-19 06:10:26 -0600 )edit

2 Answers

Sort by » oldest newest most voted
0

answered 2019-03-11 08:02:02 -0600

Alexey gravatar image

also, it may be Linux writeback buffers issue: https://unix.stackexchange.com/questi...

I am not an expert but it seems like it is highly plausible as your setup has lots of memory

edit flag offensive delete link more
0

answered 2017-07-27 01:56:25 -0600

Procópio gravatar image

updated 2017-07-27 01:56:41 -0600

Did you specify the topic you want to record? The command shown in your question rosbag record -a will record ALL the topics present in your environment.

Regarding your questions:

  1. No, this is not normal. We have robots fully functional while recording topics which amount to as much as 1700KB/sec.

  2. Yes, it could help, but I really doubt that you need it with a topic as small as 200KB/sec. I believe you should try to find what is going on with more details and I would be that you are recording much more topics than needed.

What is the size and content of a recorded bag (rosbag info foo.bag) ?

edit flag offensive delete link more

Comments

We have robots fully functional while recording topics which amount to as much as 1700KB/sec.

data rate is not everything, msg rate is also important.

gvdhoorn gravatar imagegvdhoorn ( 2017-07-27 02:32:10 -0600 )edit

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Question Tools

2 followers

Stats

Asked: 2017-07-11 21:07:44 -0600

Seen: 1,188 times

Last updated: Jul 27 '17