ROS Resources: Documentation | Support | Discussion Forum | Index | Service Status | ros @ Robotics Stack Exchange
Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

This turned out to be a limitation of the Python threading model, not the rosbag API. Python supports multi-threading, but all threads must execute on the same core Python interpreter process though the Global Interpreter Lock (GIL). Essentially a single Python process is restricted to a single CPU core regardless of the thread count. This is useful for I/O-limited tasks which spend a lot of time blocked in wait queues, but not very useful for CPU-limited tasks. For more info, see https://realpython.com/python-gil/

The best way around this (that I've found so far) is to use multiple Python processes. I modified my test script to use Python's multiprocessing library instead of the concurrent library and got the following results:

  • Number of workers: 8
  • Bag file count, total size (MB), avg size (MB): 81, 532.4, 6.6
  • Total time (s): 2.439
  • Time per file (s): 0.030
  • Throughput (MB/s): 218.3

I'm still not blown away by the read speed of the rosbag API and open to any suggestions for further improvements, but I think this answers my original question.