High rates of corrupt bags with rosbag python api?

asked 2018-01-30 13:10:43 -0500

8729 ●137 ●239 ●256 https://github.com/lucasw

updated 2018-01-30 15:03:19 -0500

I have a python action server where the callback creates a rosbag.Bag and then callback subscribing to various messages then write to that bag, and when the action is over the bag is closed and the callbacks then return without attempting to write.

It appears that about 30%-50% of the bags that come out of this say they need to be reindexed, and then after rosbag reindex a huge amount of the messages are lost- what ought to be hundreds of messages are now 10s or even zero- sometimes the reindexed bag is the same size as orig, other times smaller.

answered 2018-01-30 13:12:42 -0500

lucasw

8729 ●137 ●239 ●256 https://github.com/lucasw

updated 2018-01-31 13:28:42 -0500

I'm pretty sure that using threading Lock acquire-release around all bag writes will solve this, I'll update this once I have it working (this occurred to me as I was writing the question, but this should be useful for others that make the same mistake).

Note that the rosbag API's are not thread-safe for reading and writing any given bag file. Users of the APIs must ensure no concurrent input/output operations are performed on different threads.

http://wiki.ros.org/rosbag/Code%20API

This appears to be working, haven't seen a corrupt bag yet:

...
from threading import Lock

class Blah:
    def __init__(self):
        self.lock = Lock()
...  
        self.lock.acquire()
        self.bag = rosbag(filename, 'w')
        self.lock.release()
...
        self.lock.acquire()
        self.bag.close()
        self.lock.release()
...
    def foo_callback(self, msg):
        self.lock.acquire()
        self.bag.write("foo", msg)
        self.lock.release()
...