Recording a bag drops lots of frames

edit

Hello I have a Jetson Xavier NX, and 4 cameras plugged in, 2 CSI-2 and 1 USB. The idea is to record frames to a NVME SSD card using ROS. For testing purposes I always launch only 1 camera and record that same camera. For the CSI-2 cameras I use gscam to publish the frames to a topic, but for the USB camera I had to write a node in c++ to publish the frames. The CSI-2 cameras publishes at 50 hz while the USB camera at 68hz. My ROS version is melodic.

Now here's the problem, the first time I tested each camera everything went fine, the frame drop was minimal or non-existent. But suddenly 2 days ago each rosbag I recorded started to give me lots of the same warning:

[ WARN] [1614948238.606295288]: rosbag record buffer exceeded. Dropping oldest queued message.

Checking the rosbags I realized that each one is dropping at least 50% of the total frames from any camera. The only big change I made to the Jetson was to install the compressed_image_transport plugin, and even so I specified that I only wanted to record the topic for the raw frames. I also uninstalled the plugin just in case, but the results are the same. Just in case I checked the temperature for both the Jetson (40°C) and the SSD (30°C) card.

Checking my ssd card I found somthing odd, nvme list says that my card is full when I know this isn't the case since df -h says there's still 100GB available.

Node             SN                   Model                Namespace Usage                      Format           FW Rev  
---------------- -------------------- -------------------- --------- --------------------- ---------------- --------
/dev/nvme0n1     2014E299BEBD         CT250P2SSD8          1         250,06  GB / 250,06  GB    512   B +  0 B P2CR010

Here's my smart log with nvme smart-log /dev/nvme0n1, I don't think there's something wrong with the SSD card.

Smart Log for NVME device:nvme0n1 namespace-id:ffffffff
critical_warning                    : 0
temperature                         : 30 C
available_spare                     : 100%
available_spare_threshold           : 5%
percentage_used                     : 0%
data_units_read                     : 600.617
data_units_written                  : 2.655.492
host_read_commands                  : 2.760.498
host_write_commands                 : 7.390.077
controller_busy_time                : 513
power_cycles                        : 146
power_on_hours                      : 2.163
unsafe_shutdowns                    : 88
media_errors                        : 0
num_err_log_entries                 : 8
Warning Temperature Time            : 0
Critical Composite Temperature Time : 0
Thermal Management T1 Trans Count   : 0
Thermal Management T2 Trans Count   : 0
Thermal Management T1 Total Time    : 0
Thermal Management T2 Total Time    : 0

If you are wondering how I launch and record a camera, here's how:

CSI-2 camera:

1) Launch camera in /dev/video0

roslaunch gscam v4l0.launch

2) Check that camera is publishing to the topic at the desired rate

rostopic hz /v4l0/camera/image_raw

average rate: 50.400
    min: 0.016s max: 0.024s std dev: 0.00121s window: 47
average rate: 50.224
    min: 0.016s max: 0.024s std dev: 0.00088s window: 97
average rate: 50.152
    min: 0.016s max: 0.024s std dev: 0.00078s window: 147

3) Record bag

rosbag record /v4l0/camera/image_raw -o bag.bag __name:=my_bag

For the USB camera is basically the ... (more)