Robotics StackExchange | Archived questions

tf_old_data warning by rtabmap - no map created

Hi, I have stereolabs ZED camera with roszedwrapper and rtabmap_ros running on a ubuntu 16.04 kinetic installation. Both packages were built from source because rtabmapviz was crashing before.

Currently only the demo launches for RGBD Mapping are in use. (http://wiki.ros.org/rtabmap_ros/Tutorials/HandHeldMapping)

Start of ZED

// B) With zed odometry:
$ export ROS_NAMESPACE=camera
$ roslaunch zed_wrapper zed_camera.launch

Start of RTABMAP

// B) With zed odometry
$ roslaunch rtabmap_ros rtabmap.launch rtabmap_args:="--delete_db_on_start" depth_topic:=/camera/depth/depth_registered frame_id:=zed_camera_center approx_sync:=false visual_odometry:=false odom_topic:=/camera/odom

Yesterday, everything was fine. Had several test runs to record a few databases with environment information.

Today I receive from both nodes warning messages regarding TFOLDDATA and no database is created. In rtabmapviz the loop closure detection view is empty.

ZED Warning:

Warning: TF_OLD_DATA ignoring data from the past for frame odom at time 1.53669e+09 according to authority unknown_publisher
Possible reasons are listed at http://wiki.ros.org/tf/Errors%20explained
         at line 277 in /tmp/binarydeb/ros-kinetic-tf2-0.5.18/src/buffer_core.cpp

Rtabmap Warning:

[ WARN] [1536688768.585123648]: TF_OLD_DATA ignoring data from the past for frame odom at time 1.53669e+09 according to authority unknown_publisher
...
[ WARN] [1536690433.251155322]: Could not get transform from odom to zed_camera_center after 0.200000 seconds (for stamp=1536690419.817974)! Error="Lookup would require extrapolation into the past.  Requested time 1536690419.817974107 but the earliest data is at time 1536690429.499749414, when looking up transform from frame [zed_camera_center] to frame [odom]. canTransform returned after 0.201604 timeout was 0.2.".
[ WARN] [1536690433.251314047]: We received odometry message, but we cannot get the corresponding TF odom->zed_camera_center at data stamp 1536690419.817974s (odom msg stamp is 1536690419.817974s). Make sure TF of odometry is also published to get more accurate pose estimation. This warning is only printed once.

The only thing I changed was the time / timezone setting of ubuntu. Changed from ? to Berlin. I tried to change the timezone again, but nothing changes.

Reboot etc. was tested of course.

Any hints?

UPDATE: If i start zedroswrapper only and run command:

 rosrun tf tf_monitor zed_camera_center odom

I get zero delay and everything seems ok with tf.

Then i start rtabmap and everything gets wrong. warning from above are posted and rosrun tf... output is:

//RESULTS: for zed_camera_center to odom
Chain is: zed_camera_center -> odom
Net delay     avg = 0.11676: max = 0.25259

Frames:
Frame: odom published by unknown_publisher Average Delay: -2.6757e+08 Max Delay: 0.209381
Frame: zed_camera_center published by unknown_publisher Average Delay: 0.155448 Max Delay: 0.209376

All Broadcasters:
Node: unknown_publisher 62.4184 Hz, Average Delay: -1.7838e+08 Max Delay: 0.209381
Node: unknown_publisher(static) 1e+08 Hz, Average Delay: 0 Max Delay: 0
Warning: TF_OLD_DATA ignoring data from the past for frame odom at time 1.53675e+09 according to authority unknown_publisher
Possible reasons are listed at http://wiki.ros.org/tf/Errors%20explained
         at line 277 in /tmp/binarydeb/ros-kinetic-tf2-0.5.18/src/buffer_core.cpp
Warning: TF_OLD_DATA ignoring data from the past for frame odom at time 1.53675e+09 according to authority unknown_publisher
Possible reasons are listed at http://wiki.ros.org/tf/Errors%20explained
         at line 277 in /tmp/binarydeb/ros-kinetic-tf2-0.5.18/src/buffer_core.cpp
Warning: TF_OLD_DATA ignoring data from the past for frame odom at time 1.53675e+09 according to authority unknown_publisher
Possible reasons are listed at http://wiki.ros.org/tf/Errors%20explained
         at line 277 in /tmp/binarydeb/ros-kinetic-tf2-0.5.18/src/buffer_core.cpp
Warning: TF_OLD_DATA ignoring data from the past for frame odom at time 1.53675e+09 according to authority unknown_publisher
Possible reasons are listed at http://wiki.ros.org/tf/Errors%20explained
         at line 277 in /tmp/binarydeb/ros-kinetic-tf2-0.5.18/src/buffer_core.cpp

Sometimes a closed loop is detected and written to the database. But most times the warning is shown.

UPDATE: I downloaded the brand new version of ZED SDK and recompiled zed ros wrapper. Restarted everything. the warning are only shown during start 6 or 7 times. In terminal the last message is rtabmap 0.17.5 started ... But there is no map created. In RTABMAPVIZ there is no loop closure detected. All required topics are published.

How can this be analysed further?

*UPDATE * Without ZED odom

// A) With rtabmap odometry:
$ export ROS_NAMESPACE=camera
$ roslaunch zed_wrapper zed_camera.launch publish_tf:=false

// A) With rtabmap odometry
 $ roslaunch rtabmap_ros rtabmap.launch rtabmap_args:="--delete_db_on_start" depth_topic:=/camera/depth/depth_registered frame_id:=zed_camera_center approx_sync:=false

With this start scenario I receive an error during rtabmap_ros start, followed by a info that rtabmap is started. After this severial warnings that no data is recieved.

/rtabmap/rtabmapviz subscribed to (exact sync):
   /rtabmap/odom,
   /camera/rgb/image_rect_color,
   /camera/depth/depth_registered,
   /camera/rgb/camera_info,
   /rtabmap/odom_info
[ INFO] [1536866712.835186013]: rtabmapviz started.
[ INFO] [1536866712.925913078]: Odom: quality=334, std dev=0.021413m|0.049059rad, update time=2018274524.000000s
[ INFO] [1536866713.201391283]: Odom: quality=341, std dev=0.027108m|0.047597rad, update time=1006524082.000000s
[ INFO] [1536866713.437458216]: Odom: quality=316, std dev=0.019121m|0.044490rad, update time=993062068.000000s
terminate called after throwing an instance of 'std::runtime_error'
  what():  Duration is out of dual 32-bit range
[rtabmap/rgbd_odometry-1] process has died [pid 32046, exit code -6, cmd /home/nvidia/catkin_ws/devel/lib/rtabmap_ros/rgbd_odometry --delete_db_on_start rgb/image:=/camera/rgb/image_rect_color depth/image:=/camera/depth/depth_registered rgb/camera_info:=/camera/rgb/camera_info rgbd_image:=rgbd_image_relay odom:=odom __name:=rgbd_odometry __log:=/home/nvidia/.ros/log/a894dea8-b78a-11e8-8272-00044ba5b25b/rtabmap-rgbd_odometry-1.log].
log file: /home/nvidia/.ros/log/a894dea8-b78a-11e8-8272-00044ba5b25b/rtabmap-rgbd_odometry-1*.log
[ INFO] [1536866716.218308343]: RTAB-Map detection rate = 1.000000 Hz
[ INFO] [1536866716.218983124]: rtabmap: Deleted database "/home/nvidia/.ros/rtabmap.db" (--delete_db_on_start or -d are set).
[ INFO] [1536866716.219170515]: rtabmap: Using database from "/home/nvidia/.ros/rtabmap.db" (0 MB).
[ INFO] [1536866716.771743784]: rtabmap: Database version = "0.17.5".
[ INFO] [1536866716.954319798]: /rtabmap/rtabmap: queue_size    = 10
[ INFO] [1536866716.954419990]: /rtabmap/rtabmap: rgbd_cameras = 1
[ INFO] [1536866716.954473589]: /rtabmap/rtabmap: approx_sync   = false
[ INFO] [1536866716.954728596]: Setup depth callback
[ INFO] [1536866717.116803714]: 
/rtabmap/rtabmap subscribed to (exact sync):
   /rtabmap/odom,
   /camera/rgb/image_rect_color,
   /camera/depth/depth_registered,
   /camera/rgb/camera_info,
   /rtabmap/odom_info
[ INFO] [1536866717.155583598]: rtabmap 0.17.5 started...
[ WARN] [1536866722.117344470]: /rtabmap/rtabmap: Did not receive data since 5 seconds! Make sure the input topics are published ("$ rostopic hz my_topic") and the timestamps in their header are set. If topics are coming from different computers, make sure the clocks of the computers are synchronized ("ntpdate"). Parameter "approx_sync" is false, which means that input topics should have all the exact timestamp for the callback to be called.
/rtabmap/rtabmap subscribed to (exact sync):
   /rtabmap/odom,
   /camera/rgb/image_rect_color,
   /camera/depth/depth_registered,
   /camera/rgb/camera_info,
   /rtabmap/odom_info

I have a doubt that depth_registered is published correctly. In RVIZ (without rtabmap) most times only a black image is shown. Only in some less areas a depth image is shown. I also tried to run through the calibration process of ZED, nothing changed. But on the other hand, the depth image is shown correctly if I use ZED Depth Viewer.

I created two sample depth images with rqtimageview. Not sure if they are ok. I'm wondering about the large black area of the second one. (that are windows - no bright light outside - in 8-10m distance)

image description image description

Please find attach to tf tree pdfs. One with zed node only and a second one with zed and rtab node.

image description image description

Here you will see the rosgraph of rtabmap with zed and rtabmap odometry. As well as the corresponding tf tree.

image description

image description

Parallel rostopic hz are showing this:

rostopic hz /camera/rgb/image_rect_color
subscribed to [/camera/rgb/image_rect_color]
average rate: 61.031
    min: 0.008s max: 0.027s std dev: 0.00373s window: 42
average rate: 42.053
    min: 0.008s max: 0.324s std dev: 0.03480s window: 78
average rate: 42.353
    min: 0.008s max: 0.324s std dev: 0.02792s window: 121


nvidia@ros-jetson:~$ rostopic hz /camera/rgb/camera_info
subscribed to [/camera/rgb/camera_info]
average rate: 42.668
    min: 0.022s max: 0.026s std dev: 0.00086s window: 42
average rate: 42.843
    min: 0.021s max: 0.027s std dev: 0.00104s window: 85
average rate: 43.516
    min: 0.021s max: 0.028s std dev: 0.00124s window: 130
average rate: 43.147
    min: 0.021s max: 0.028s std dev: 0.00125s window: 172
average rate: 42.562
    min: 0.021s max: 0.028s std dev: 0.00136s window: 212

nvidia@ros-jetson:~$ rostopic hz /camera/depth/depth_registered
subscribed to [/camera/depth/depth_registered]
average rate: 42.375
    min: 0.016s max: 0.029s std dev: 0.00260s window: 26
average rate: 42.633
    min: 0.016s max: 0.029s std dev: 0.00169s window: 69
average rate: 42.830
    min: 0.015s max: 0.031s std dev: 0.00227s window: 112
average rate: 43.317
    min: 0.015s max: 0.031s std dev: 0.00247s window: 156
average rate: 42.976

Asked by Boregard on 2018-09-11 13:32:03 UTC

Comments

Berlin is CEST (ie: GMT+2 at the moment). If the ZED has its own internal clock and it doesn't take daylight saving into account, it could be at GMT+1, or even GMT. That would put it at -1 or -2 hours, which could explain the "old data".

I don't have ZED. so you'd have to check.

Asked by gvdhoorn on 2018-09-11 14:57:56 UTC

Can you try the approach without zed odometry?

Asked by matlabbe on 2018-09-13 14:12:29 UTC

Sure, I updated the question. Thx.

Asked by Boregard on 2018-09-13 15:02:25 UTC

With rtabmap odometry, the odometry node is crashing after processing some images: terminate called after throwing an instance of 'std::runtime_error' what(): Duration is out of dual 32-bit range, so it is normal that rtabmap node is not working (it is not receiving any odometry).

Asked by matlabbe on 2018-09-13 15:25:31 UTC

I'll test with latest zed sdk if I have the same error here...

Asked by matlabbe on 2018-09-13 15:26:15 UTC

Well, both approaches (with zed odometry or with rtabmap odometry) are fine here. What does /camera/depth/depth_registered in rqt_image_view look like?

Asked by matlabbe on 2018-09-13 15:44:14 UTC

I would advise to start by looking at the tf tree and see which transformation are off in time from each other.rosrun tf2_tools view_frames.py creates a pdf where you can see latest timestamps. If you detect which transformation is wrongly dated, it will be easier to find an error.

Asked by MRWRWK on 2018-09-13 16:25:17 UTC

thx. I attached the images and tf tree abve. I think there is the latest SDK version installed. Updated a few days ago, but I will double check this evening.

Asked by Boregard on 2018-09-14 00:37:17 UTC

The second tf tree doesn't look like the one with rtabmap odometry, as rtabmap_ros/rgbd_odometry would be the node publishing /odom -> /zed_camera_center. Can you show rqt_graph too?

Asked by matlabbe on 2018-09-14 12:30:35 UTC

Thanks a lot for your support. I attached both. The tf tree from yesterday was with zed odometry. sorry. Latest SDK was already installed.

Asked by Boregard on 2018-09-14 14:13:48 UTC

Is your camera connected to an USB 3.0 port? If on USB2, RGB and depth streams may struggle to be published. In rtqgraph, the image topics are not shown. Can you do rostopic hz on all /camera/rgb/image_rect_color, /camera/depth/depth_registered, /camera/rgb/camera_info at the same time?

Asked by matlabbe on 2018-09-14 14:19:39 UTC

I connected to a jetson TX2 dev board. There should be a USB 3. But I will check in detail. may be there is a driver issue or similar.

rostopics seem to be published.

Asked by Boregard on 2018-09-14 14:30:51 UTC

USB 3 seems to be fine. ZED Diagnostic reports no error, too. ZED is connected directly, no hub, no other equipment on the port.

Asked by Boregard on 2018-09-14 14:52:13 UTC

This could be an issue with cv_bridge and multiple OpenCV versions installed. Make sure you have only one opencv version installed. If you installed OpenCV4Tegra from Jetpack, make sure to remove opencv installed by kinetic and rebuild vision_opencv

Asked by matlabbe on 2018-09-14 20:32:24 UTC

I updated rtabmap_ros installation instructions specifically for TX2, see https://github.com/introlab/rtabmap_ros#build-from-source-for-nvidia-jetson-with-opencv-4-tegra

Asked by matlabbe on 2018-09-14 20:33:27 UTC

RTABMAP gets now created. Thanks a lot. There is only a single version of openCV on my jetson. Jetpack 3.3 includes opencv 3.3.1. Anyway I went through your updated docu and compiled vision_opencv from source. Now, rtabmap is created as one week before :) I suppose that there was any ubuntu update

Asked by Boregard on 2018-09-15 12:33:02 UTC

Answers