Simulated /clock stops during playback of rosbags recorded using the API

edit

I'm using data files with known timestamp alongside the rosbag API to create rosbags. For example I have pointcloud data converted from PCD to PointCloud2 msgs with the following headers.

seq: 158
stamp: 
  secs: 1625374470
  nsecs: 167207956
frame_id: "points_raw"
---
seq: 159
stamp: 
  secs: 1625374470
  nsecs: 267851829
frame_id: "points_raw"

I can visualize the data just fine in RVIZ using

roscore
rosparam set use_sim_time true
rosbag play data.bag --clock

I can echo the /clock topic and get something that makes sense.

Then, suddenly, the /clock stops and playback becomes laggy & confused. The rosbag still seems to be playing though the time doesn't increment properly..but it eventually ends. The problem goes away when I play the whole rosbag once; on second playback, /clock works throughout and everything looks fine. Reindexing the rosbag also solves the issue, though I cannot see any real differences between the rosbags using rosbag info and rqt_bag.

Probably something weird going on due to how I'm writing the rosbag but I can't find any documentation in the rosbag API about this except being sure to close the rosbag after writing. Any ideas? I'd rather avoid running rosbag reindex.