How do you split a rosbag into several files without calling rosbag filter multiple times?

asked 2018-01-31 09:44:01 -0500

jborro gravatar image

I want to split a 100-GB rosbag into 100 1-GB bags. I tried using rosbag filter but it takes a long time as I have to run each filter manually and each time, it performs a scan of the full bag. Is there a better way to perform this split (either through command line or Python script)?

I've tried writing a Python script that reads through the rosbag message-by-message and writes each to a new bag (i.e. first GB of messages goes in bag 1, 2nd GB goes in bag 2, etc.). However, when I try to run the outputted bags, I get a "Expected CONNECTION op not found" error. What's the cause of this?

For reference, the python script is as follows:

import rosbag
num_msgs_per_bag = 1400
inputBag = 'Jan28.bag'
total_num_msgs_in_bag = rosbag.Bag(inputBag).get_message_count()

print (total_num_msgs_in_bag)

num_output_bags = total_num_msgs_in_bag / num_msgs_per_bag
bagCount = 1
msgCount = 0

outbag = rosbag.Bag('Jan28-'+str(bagCount)+'.bag', 'w')
for topic, msg, t in rosbag.Bag(inputBag).read_messages():
    outbag.write(topic, msg, t)
    print (msgCount)
    msgCount = msgCount + 1
    if msgCount % num_msgs_per_bag == 0:
        bagCount = bagCount + 1
        print ('******NEW BAG*******')
        outbag = rosbag.Bag('Jan28-'+str(bagCount)+'.bag', 'w')
edit retag flag offensive close merge delete

Comments

Well, I have no experience with the API, but you could also replay it and record everything with the --split command:

rosbag record -a --split --size=1024

mgruhler gravatar image mgruhler  ( 2018-01-31 09:50:52 -0500 )edit

@mig I tried that and it hangs partway through (maybe some issue with the buffer?). The rosbag which I'm trying to split is 360GB and I don't want to slow down the playback rate as that'll slow down the recorded rosbag.

jborro gravatar image jborro  ( 2018-01-31 23:12:19 -0500 )edit

360GB is actually huge. This then also depends on how fast writing to the HDD is. There are Parameters --buffsize and --chunksize with which you could Play around. But no idea, if this helps. The docu might help: http://wiki.ros.org/rosbag/Commandlin...

mgruhler gravatar image mgruhler  ( 2018-02-01 00:51:28 -0500 )edit