Recording Large Rosbags of Lidar & Camera Data

asked 2022-06-09 12:37:01 -0600

123 ●12 ●20 ●23

I have been trying to find some documentation on this but was unable to really see anyhting helpful.

We have 5 cameras and 3 Lidars and need to record data from all sensors. For now we have been recording only two cameras (Allied Vision G-319C) and one Lidar (Velodyne VLP-32C) and are already running into problems.

The data is published at 20 Hz (lidar) and 30 Hz (cameras). Recording only one image for each camera in compressed format and the /velodyne_points results in the buffer being overflown and frames/points being dropped. The data is recorded to an internal SSD but I also tried an external SSD which works slightly worse.

I am wondering what the best practice would be in this case such as recording multiple rosbags at the same time, for example record one rosbag for each camera or something like that to distribute the load - or not even using rosbags but some other tool that i am unaware of.

(I have played around with different buffer sizes (setting it to 0 results in the rosbag automatically shutting down quickly) and splitting the rosbags at different sizes (500 MB, 5000 MB) without much improvement.)

Hardware: Astuff Spectra/Neuosys 6109GC with 32 GB of ram, two internal 265 GB and 1 TB SSDs, external 2 TB SSD, XEON E2278G 8th Gen

Thanks

edit retag flag offensive close merge delete

add a comment

2

answered 2022-06-20 11:20:17 -0600

JoloGermanAg

123 ●12 ●20 ●23

The solution for our problem was to reduce the image sizes using image_proc/resize. We are now able to record at full frequency. Thanks for all your answers!

edit flag offensive delete link

add a comment

2

answered 2022-06-13 12:04:03 -0600

GeorgNo

183 ●7 ●23 ●22

We are recording our sensor data (3 x 3D LiDAR, 3 x 2k rgb camera, IMU, GPS) in the following way and not running into any issues:

rosbag record -O "$NAME"_"$cnt" --split --size=4096 /topics

Although we had some problems (frames dropping) with our network card of the computer since our cameras and lidars are ethernet based. We bought a 10Gbe to thunderbolt 4 adapter and all is wokring fine that way

edit flag offensive delete link

Comments

Thank you, I will try this as well!

JoloGermanAg ( 2022-06-13 13:16:32 -0600 )edit

add a comment

2

answered 2022-06-10 22:41:51 -0600

fergs

13902 ●59 ●157 ●196 http://www.robotandchisel.com

There are a couple tricks out there that can be used:

Do you really need full frame rate for the sensors? If not, you can insert something like the throttle node from topic_tools in between your actual sensors and the rosbag recorder to downsample the frame rate to a more manageable amount of data.
I've personally always had better luck compressing the bagfiles as a separate step AFTER recording (so, record uncompressed, then when the bagfile finishes or splits you compress it).

edit flag offensive delete link

Comments

I suppose you are right in that it's not necessary to record at the full frequency - i have tested this and it does work at around 10 Hz for everything. I also realized that our images are rather large so resizing them may help too. Thank you for your answer!

JoloGermanAg ( 2022-06-13 13:16:05 -0600 )edit

add a comment

Recording Large Rosbags of Lidar & Camera Data

3 Answers

Comments

Comments

Question Tools

Stats

Related questions

Recording Large Rosbags of Lidar & Camera Data edit

3 Answers

Comments

Comments

Question Tools

Stats

Related questions

Recording Large Rosbags of Lidar & Camera Data