Deterministic replay and debugging
EDIT: Reposted here, with some responses.
I need to build a ROS system that can log its inputs and node communication such that all calculations can be reproduced exactly during analysis later. The system needs to have two modes:
Operation
- The system is in control of the physical robot
- All sensor data is recorded
- Sufficient data about the timing and data dependencies between the nodes is recorded.
Analysis
- The system is not connected to the physical robot
- The recorded sensor data is replayed to the system
- The relative timings and data dependencies between the nodes are preserved. This means other nodes may have to idle if one node is paused in a debugger.
For example there may be a system with this topology A --[/x]--> B --[/y]--> C where A,B,C are nodes and x,y are topics. All three nodes are implemented with a polling loop (using ros::Rate and ros::spinOnce).
During analysis the node B is stopped in a debugger. Now A needs to pause because otherwise the topic queue x would overflow and messages would be dropped that were not dropped during operation. Same with C: It needs to pause because it is not getting the messages from B that it did get during operation.
This seems like a standard problem, yet I cannot find anything in the ROS ecosystem that addresses this.
I have some ideas for developing this myself, but I'd rather use an existing solution.
rosbag appears to be unsuitable: The limitations of rosbag record/play
