tf_old_data warning by rtabmap - no map created
Hi, I have stereolabs ZED camera with roszedwrapper and rtabmap_ros running on a ubuntu 16.04 kinetic installation. Both packages were built from source because rtabmapviz was crashing before.
Currently only the demo launches for RGBD Mapping are in use. (http://wiki.ros.org/rtabmap_ros/Tutorials/HandHeldMapping)
Start of ZED
// B) With zed odometry:
$ export ROS_NAMESPACE=camera
$ roslaunch zed_wrapper zed_camera.launch
Start of RTABMAP
// B) With zed odometry
$ roslaunch rtabmap_ros rtabmap.launch rtabmap_args:="--delete_db_on_start" depth_topic:=/camera/depth/depth_registered frame_id:=zed_camera_center approx_sync:=false visual_odometry:=false odom_topic:=/camera/odom
Yesterday, everything was fine. Had several test runs to record a few databases with environment information.
Today I receive from both nodes warning messages regarding TFOLDDATA and no database is created. In rtabmapviz the loop closure detection view is empty.
ZED Warning:
Warning: TF_OLD_DATA ignoring data from the past for frame odom at time 1.53669e+09 according to authority unknown_publisher
Possible reasons are listed at http://wiki.ros.org/tf/Errors%20explained
at line 277 in /tmp/binarydeb/ros-kinetic-tf2-0.5.18/src/buffer_core.cpp
Rtabmap Warning:
[ WARN] [1536688768.585123648]: TF_OLD_DATA ignoring data from the past for frame odom at time 1.53669e+09 according to authority unknown_publisher
...
[ WARN] [1536690433.251155322]: Could not get transform from odom to zed_camera_center after 0.200000 seconds (for stamp=1536690419.817974)! Error="Lookup would require extrapolation into the past. Requested time 1536690419.817974107 but the earliest data is at time 1536690429.499749414, when looking up transform from frame [zed_camera_center] to frame [odom]. canTransform returned after 0.201604 timeout was 0.2.".
[ WARN] [1536690433.251314047]: We received odometry message, but we cannot get the corresponding TF odom->zed_camera_center at data stamp 1536690419.817974s (odom msg stamp is 1536690419.817974s). Make sure TF of odometry is also published to get more accurate pose estimation. This warning is only printed once.
The only thing I changed was the time / timezone setting of ubuntu. Changed from ? to Berlin. I tried to change the timezone again, but nothing changes.
Reboot etc. was tested of course.
Any hints?
UPDATE: If i start zedroswrapper only and run command:
rosrun tf tf_monitor zed_camera_center odom
I get zero delay and everything seems ok with tf.
Then i start rtabmap and everything gets wrong. warning from above are posted and rosrun tf... output is:
//RESULTS: for zed_camera_center to odom
Chain is: zed_camera_center -> odom
Net delay avg = 0.11676: max = 0.25259
Frames:
Frame: odom published by unknown_publisher Average Delay: -2.6757e+08 Max Delay: 0.209381
Frame: zed_camera_center published by unknown_publisher Average Delay: 0.155448 Max Delay: 0.209376
All Broadcasters:
Node: unknown_publisher 62.4184 Hz, Average Delay: -1.7838e+08 Max Delay: 0.209381
Node: unknown_publisher(static) 1e+08 Hz, Average Delay: 0 Max Delay: 0
Warning: TF_OLD_DATA ignoring data from the past for frame odom at time 1.53675e+09 according to authority unknown_publisher
Possible reasons are listed at http://wiki.ros.org/tf/Errors%20explained
at line 277 in /tmp/binarydeb/ros-kinetic-tf2-0.5.18/src/buffer_core.cpp
Warning: TF_OLD_DATA ignoring data from the past for frame odom at time 1.53675e+09 according to authority unknown_publisher
Possible reasons are listed at http://wiki.ros.org/tf/Errors%20explained
at line 277 in /tmp/binarydeb/ros-kinetic-tf2-0.5.18/src/buffer_core.cpp
Warning: TF_OLD_DATA ignoring data from the past for frame odom at time 1.53675e+09 according to authority unknown_publisher
Possible reasons are listed at http://wiki.ros.org/tf/Errors%20explained
at line 277 in /tmp/binarydeb/ros-kinetic-tf2-0.5.18/src/buffer_core.cpp
Warning: TF_OLD_DATA ignoring data from the past for frame odom at time 1.53675e+09 according to authority unknown_publisher
Possible reasons are listed at http://wiki.ros.org/tf/Errors%20explained
at line 277 in /tmp/binarydeb/ros-kinetic-tf2-0.5.18/src/buffer_core.cpp
Sometimes a closed loop is detected and written to the database. But most times the warning is shown.
UPDATE: I downloaded the brand new version of ZED SDK and recompiled zed ros wrapper. Restarted everything. the warning are only shown during start 6 or 7 times. In terminal the last message is rtabmap 0.17.5 started ... But there is no map created. In RTABMAPVIZ there is no loop closure detected. All required topics are published.
How can this be analysed further?
*UPDATE * Without ZED odom
// A) With rtabmap odometry:
$ export ROS_NAMESPACE=camera
$ roslaunch zed_wrapper zed_camera.launch publish_tf:=false
// A) With rtabmap odometry
$ roslaunch rtabmap_ros rtabmap.launch rtabmap_args:="--delete_db_on_start" depth_topic:=/camera/depth/depth_registered frame_id:=zed_camera_center approx_sync:=false
With this start scenario I receive an error during rtabmap_ros start, followed by a info that rtabmap is started. After this severial warnings that no data is recieved.
/rtabmap/rtabmapviz subscribed to (exact sync):
/rtabmap/odom,
/camera/rgb/image_rect_color,
/camera/depth/depth_registered,
/camera/rgb/camera_info,
/rtabmap/odom_info
[ INFO] [1536866712.835186013]: rtabmapviz started.
[ INFO] [1536866712.925913078]: Odom: quality=334, std dev=0.021413m|0.049059rad, update time=2018274524.000000s
[ INFO] [1536866713.201391283]: Odom: quality=341, std dev=0.027108m|0.047597rad, update time=1006524082.000000s
[ INFO] [1536866713.437458216]: Odom: quality=316, std dev=0.019121m|0.044490rad, update time=993062068.000000s
terminate called after throwing an instance of 'std::runtime_error'
what(): Duration is out of dual 32-bit range
[rtabmap/rgbd_odometry-1] process has died [pid 32046, exit code -6, cmd /home/nvidia/catkin_ws/devel/lib/rtabmap_ros/rgbd_odometry --delete_db_on_start rgb/image:=/camera/rgb/image_rect_color depth/image:=/camera/depth/depth_registered rgb/camera_info:=/camera/rgb/camera_info rgbd_image:=rgbd_image_relay odom:=odom __name:=rgbd_odometry __log:=/home/nvidia/.ros/log/a894dea8-b78a-11e8-8272-00044ba5b25b/rtabmap-rgbd_odometry-1.log].
log file: /home/nvidia/.ros/log/a894dea8-b78a-11e8-8272-00044ba5b25b/rtabmap-rgbd_odometry-1*.log
[ INFO] [1536866716.218308343]: RTAB-Map detection rate = 1.000000 Hz
[ INFO] [1536866716.218983124]: rtabmap: Deleted database "/home/nvidia/.ros/rtabmap.db" (--delete_db_on_start or -d are set).
[ INFO] [1536866716.219170515]: rtabmap: Using database from "/home/nvidia/.ros/rtabmap.db" (0 MB).
[ INFO] [1536866716.771743784]: rtabmap: Database version = "0.17.5".
[ INFO] [1536866716.954319798]: /rtabmap/rtabmap: queue_size = 10
[ INFO] [1536866716.954419990]: /rtabmap/rtabmap: rgbd_cameras = 1
[ INFO] [1536866716.954473589]: /rtabmap/rtabmap: approx_sync = false
[ INFO] [1536866716.954728596]: Setup depth callback
[ INFO] [1536866717.116803714]:
/rtabmap/rtabmap subscribed to (exact sync):
/rtabmap/odom,
/camera/rgb/image_rect_color,
/camera/depth/depth_registered,
/camera/rgb/camera_info,
/rtabmap/odom_info
[ INFO] [1536866717.155583598]: rtabmap 0.17.5 started...
[ WARN] [1536866722.117344470]: /rtabmap/rtabmap: Did not receive data since 5 seconds! Make sure the input topics are published ("$ rostopic hz my_topic") and the timestamps in their header are set. If topics are coming from different computers, make sure the clocks of the computers are synchronized ("ntpdate"). Parameter "approx_sync" is false, which means that input topics should have all the exact timestamp for the callback to be called.
/rtabmap/rtabmap subscribed to (exact sync):
/rtabmap/odom,
/camera/rgb/image_rect_color,
/camera/depth/depth_registered,
/camera/rgb/camera_info,
/rtabmap/odom_info
I have a doubt that depth_registered is published correctly. In RVIZ (without rtabmap) most times only a black image is shown. Only in some less areas a depth image is shown. I also tried to run through the calibration process of ZED, nothing changed. But on the other hand, the depth image is shown correctly if I use ZED Depth Viewer.
I created two sample depth images with rqtimageview. Not sure if they are ok. I'm wondering about the large black area of the second one. (that are windows - no bright light outside - in 8-10m distance)
Please find attach to tf tree pdfs. One with zed node only and a second one with zed and rtab node.
Here you will see the rosgraph of rtabmap with zed and rtabmap odometry. As well as the corresponding tf tree.
Parallel rostopic hz are showing this:
rostopic hz /camera/rgb/image_rect_color
subscribed to [/camera/rgb/image_rect_color]
average rate: 61.031
min: 0.008s max: 0.027s std dev: 0.00373s window: 42
average rate: 42.053
min: 0.008s max: 0.324s std dev: 0.03480s window: 78
average rate: 42.353
min: 0.008s max: 0.324s std dev: 0.02792s window: 121
nvidia@ros-jetson:~$ rostopic hz /camera/rgb/camera_info
subscribed to [/camera/rgb/camera_info]
average rate: 42.668
min: 0.022s max: 0.026s std dev: 0.00086s window: 42
average rate: 42.843
min: 0.021s max: 0.027s std dev: 0.00104s window: 85
average rate: 43.516
min: 0.021s max: 0.028s std dev: 0.00124s window: 130
average rate: 43.147
min: 0.021s max: 0.028s std dev: 0.00125s window: 172
average rate: 42.562
min: 0.021s max: 0.028s std dev: 0.00136s window: 212
nvidia@ros-jetson:~$ rostopic hz /camera/depth/depth_registered
subscribed to [/camera/depth/depth_registered]
average rate: 42.375
min: 0.016s max: 0.029s std dev: 0.00260s window: 26
average rate: 42.633
min: 0.016s max: 0.029s std dev: 0.00169s window: 69
average rate: 42.830
min: 0.015s max: 0.031s std dev: 0.00227s window: 112
average rate: 43.317
min: 0.015s max: 0.031s std dev: 0.00247s window: 156
average rate: 42.976
Asked by Boregard on 2018-09-11 13:32:03 UTC
Comments
Berlin is CEST (ie: GMT+2 at the moment). If the ZED has its own internal clock and it doesn't take daylight saving into account, it could be at GMT+1, or even GMT. That would put it at -1 or -2 hours, which could explain the "old data".
I don't have ZED. so you'd have to check.
Asked by gvdhoorn on 2018-09-11 14:57:56 UTC
Can you try the approach without zed odometry?
Asked by matlabbe on 2018-09-13 14:12:29 UTC
Sure, I updated the question. Thx.
Asked by Boregard on 2018-09-13 15:02:25 UTC
With rtabmap odometry, the odometry node is crashing after processing some images:
terminate called after throwing an instance of 'std::runtime_error' what(): Duration is out of dual 32-bit range
, so it is normal that rtabmap node is not working (it is not receiving any odometry).Asked by matlabbe on 2018-09-13 15:25:31 UTC
I'll test with latest zed sdk if I have the same error here...
Asked by matlabbe on 2018-09-13 15:26:15 UTC
Well, both approaches (with zed odometry or with rtabmap odometry) are fine here. What does
/camera/depth/depth_registered
inrqt_image_view
look like?Asked by matlabbe on 2018-09-13 15:44:14 UTC
I would advise to start by looking at the tf tree and see which transformation are off in time from each other.
rosrun tf2_tools view_frames.py
creates a pdf where you can see latest timestamps. If you detect which transformation is wrongly dated, it will be easier to find an error.Asked by MRWRWK on 2018-09-13 16:25:17 UTC
thx. I attached the images and tf tree abve. I think there is the latest SDK version installed. Updated a few days ago, but I will double check this evening.
Asked by Boregard on 2018-09-14 00:37:17 UTC
The second tf tree doesn't look like the one with rtabmap odometry, as rtabmap_ros/rgbd_odometry would be the node publishing /odom -> /zed_camera_center. Can you show rqt_graph too?
Asked by matlabbe on 2018-09-14 12:30:35 UTC
Thanks a lot for your support. I attached both. The tf tree from yesterday was with zed odometry. sorry. Latest SDK was already installed.
Asked by Boregard on 2018-09-14 14:13:48 UTC
Is your camera connected to an USB 3.0 port? If on USB2, RGB and depth streams may struggle to be published. In rtqgraph, the image topics are not shown. Can you do
rostopic hz
on all/camera/rgb/image_rect_color
,/camera/depth/depth_registered
,/camera/rgb/camera_info
at the same time?Asked by matlabbe on 2018-09-14 14:19:39 UTC
I connected to a jetson TX2 dev board. There should be a USB 3. But I will check in detail. may be there is a driver issue or similar.
rostopics seem to be published.
Asked by Boregard on 2018-09-14 14:30:51 UTC
USB 3 seems to be fine. ZED Diagnostic reports no error, too. ZED is connected directly, no hub, no other equipment on the port.
Asked by Boregard on 2018-09-14 14:52:13 UTC
This could be an issue with cv_bridge and multiple OpenCV versions installed. Make sure you have only one opencv version installed. If you installed OpenCV4Tegra from Jetpack, make sure to remove opencv installed by kinetic and rebuild vision_opencv
Asked by matlabbe on 2018-09-14 20:32:24 UTC
I updated rtabmap_ros installation instructions specifically for TX2, see https://github.com/introlab/rtabmap_ros#build-from-source-for-nvidia-jetson-with-opencv-4-tegra
Asked by matlabbe on 2018-09-14 20:33:27 UTC
RTABMAP gets now created. Thanks a lot. There is only a single version of openCV on my jetson. Jetpack 3.3 includes opencv 3.3.1. Anyway I went through your updated docu and compiled vision_opencv from source. Now, rtabmap is created as one week before :) I suppose that there was any ubuntu update
Asked by Boregard on 2018-09-15 12:33:02 UTC