The timestamps in ROS bag files, and by extension your original CSV files, are stored in integer nanoseconds. Therefore, I think your problem is that you are assuming the timestamps are in microseconds when you convert them to a double representation. I think if you divide by 1e9 instead of 1e6 it will fix it.