Question

If a ros2 bag record is interrupted through a power cut, it is likely that the sqlite3 database will be corrupted, and attempts to repair the bagfile through ros2 bag reindex will fail with an sqlite3 error. Sort of like:

[ERROR] [1667965287.653354653] [rosbag2_storage]: Could not open 'bagfile_2022-11-08-17-46-05/bagfile_2022-11-08-17-46-05_0.db3' with 'sqlite3'. Error: Failed to setup storage. Error: Error when processing SQL statement. SQLite error (11): database disk image is malformed [ERROR] [1667965287.653386311] [rosbag2_storage]: Could not load/open plugin with storage id 'sqlite3'.

As explained here this is desired behaviour in ros2 bag for writing speed reasons. However, how do you recover such a bag?

Asked by PeterMilani on 2022-11-10 18:54:18 UTC

Answer 1

TLDR;

Assuming your bag file is called

Install the application sqlite3 through

apt install sqlite3

Check for issues in db3 file in bag.

sqlite3 /path/to/bag/bag.db3 "PRAGMA integrity_check;"

Now I have seen the above command return errors or ok with no change to the reindexing success for rosbag2. Make a backup of the database:

 cp bag/bag.db3 bag/bag.db.bak
 sqlite3 bag/bag.db3

At the prompt dump the raw contents of the db.

.mode insert
.output bag/dump_all.sql
.dump
.exit

Use grep to extract just the relevant transactions (remember I don't know sql)

cat bag/dump_all.sql | grep -v TRANSACTION | grep -v ROLLBACK | grep -v COMMIT >bag/dump_all_notrans.sql

now load back into a new database:

 sqlite3 bag/bag.db3 ".read bag/dump_all_notrans.sql"

check that the size of the database is no longer zero

ls -al bag/bag.db3

Now you should be able to reindex successfully:

ros2 bag reindex bag/

Note that for a performance penalty there is an argument --storage-preset-profile resilent that is supposed to record ros2 bags in a more recoverable way.

Asked by PeterMilani on 2022-11-10 18:57:06 UTC

ros2 bag reindex fails after bag not shutdown correctly

Comments

Answers

Comments