ROS Resources: Documentation | Support | Discussion Forum | Index | Service Status | ros @ Robotics Stack Exchange
Ask Your Question
0

How to Compare the SLAM trajectory with Simulation Ground Truth

asked 2019-09-19 10:07:24 -0500

al389660 gravatar image

Hello, I am working with Gazebo simulator. I have successfully created a map of the environment using rtabmap_ros. Now I want to compare (plot) the trajectory of the robot as generated by the navigation set-up with the actual position of the robot in the Gazebo.

I don't know how to export each topic separately to plot them on the same scale later for comparison purposes. please, explain the steps and topics involved to plot the comparison.

  1. how to export and plot the actual position of the robot in the Gazebo.
  2. how to export and plot the trajectory of the robot as calculated by the navigation setup.

Thank in advance

edit retag flag offensive close merge delete

Comments

You want to get ground truth to compare it to the odometry you are getting using rtabmap?

Choco93 gravatar image Choco93  ( 2019-09-20 01:36:00 -0500 )edit

yes, I want to plot them both. After getting a goal command, the robot navigates autonomously by generating the trajectory using rtabmap_ros. I want to plot that trajectory with the actual position of the robot in Gazebo (ground Truth) to compare the accuracy of the rtabmap_ros. I just do not know how to do that and which topics are involved.

al389660 gravatar image al389660  ( 2019-09-20 16:30:53 -0500 )edit

You can take a look at this answer. And for plotting you can use rqt_plot I guess.

Choco93 gravatar image Choco93  ( 2019-09-21 10:21:30 -0500 )edit

1 Answer

Sort by ยป oldest newest most voted
0

answered 2019-09-22 18:43:03 -0500

matlabbe gravatar image

updated 2019-09-29 18:47:12 -0500

As @Choco93 said, you could add the following to your urdf file:

<gazebo>
    <plugin name="p3d_base_controller" filename="libgazebo_ros_p3d.so">
      <alwaysOn>true</alwaysOn>
      <updateRate>50.0</updateRate>
      <bodyName>base_footprint</bodyName>
      <topicName>ground_truth/state</topicName>
      <gaussianNoise>0.01</gaussianNoise>
      <frameName>world</frameName>
      <xyzOffsets>0 0 0</xyzOffsets>
      <rpyOffsets>0 0 0</rpyOffsets>
    </plugin>
</gazebo>

To feed ground truth to rtabmap, we need to convert the /ground_truth/state topic to TF, and change the base_footprint frame to a fake one like base_footprint_gt. This can be done with odom_msg_to_tf node:

$ rosrun rtabmap_ros odom_msg_to_tf odom:=ground_truth/state _frame_id:=base_footprint_gt
$ rosrun tf static_transform_publisher 0 0 0 0 0 0 world map 100

The TF world -> base_footprint_gt will be published. I added world -> map static transform just for convenience, to be able to see both base_footprint and base_footprint_gt in RVIZ under world global frame. Last thing to do is to setup ground truth parameters of rtabmap node:

<node pkg="rtabmap_ros" type="rtabmap" name="rtabmap" args="-d">
  ...
  <param name="ground_truth_frame_id" value="world"/>
  <param name="ground_truth_base_frame_id" value="base_footprint_gt"/>
</node>

With rtabmapviz, you will see the ground truth path in gray, and the estimated path in blue. In View->Statistics->GT panel, you will see ATE (absolute trajectory error) statistics.

EDIT When opening the database in rtabmap-databaseViewer, you can show the Statistics panel to see the computed errors under Gt tab. You can plot the error over time by clicking the button next to the statistic (in screenshot below we can see that a loop closure occurred just before 170, causing a drop of error). Under the Graph view, you should see a gray path representing the ground truth along with the blue path representing the estimated trajectory. For convenience, you can also see the same RMSE computed under it for the current optimization iteration selected. image description

Another tool is rtabmap-report to export estimated and ground truth poses in RGB-D dataset format:

$ rtabmap-report --poses rtabmap.db 
Database: rtabmap.db
    (164, s=1.000): error lin=0.020m (max=0.035m, odom=0.036m) ang=1.1deg, slam: avg=59ms (max=197ms) loops=36, odom: avg=70ms (max=100ms), camera: avg=19ms, map=285MB

This will create rtabmap_odom.txt, rtabmap_slam.txt and rtabmap_gt.txt that can be fed to RGB-D dataset's evaluation tool:

$ python evaluate_ate.py rtabmap_slam.txt rtabmap_gt.txt --plot ate.pdf
$ python evaluate_rpe.py rtabmap_odom.txt rtabmap_gt.txt --plot rpe.pdf --fixed_delta

EDIT2 The databases you shared in the comments are very difficult for visual loop closure detection as the same texture is seen everywhere, thus we cannot distinguish a place from another. No loop closures can be detected. Also the biggest problem is that you never revisit the same area with the same camera orientation. image description

For the second map, you could however enable ICP refining to correct your odometry.

Before: image description

After: image description

Parameters (rtabmap should be built with libpointmatcher for those ICP parameters):

Icp/PM=true
Icp/MaxTranslation=2.00
Icp/MaxCorrespondenceDistance=0.75
Icp/PMOutlierRatio=0.9
Icp/PointToPlane=true
Reg/Strategy=1
RGBD/NeighborLinkRefining=true
RGBD/ProximityPathMaxNeighbors=5 ...
(more)
edit flag offensive delete link more

Comments

Thanks, I followed your instructions and I am getting ground-truth path in gray, and the estimated path in blue. But the problem is that I am cannot find "View->Statistics->GT panel, you will see ATE (absolute trajectory error) statistics."

Here is what I did,

  1. I changed my robot's urdf and added the recommended plugin.
  2. I then added the recommended parameters in my roadmap launch file under rtabmap node.
  3. I ran my simulations (gazebo and rtabmap_ros) and also executed the nodes above as given in step 2 of your response. I then moved my robot in an environment and built a map to see if everything is working fine.
  4. Finally, I stopped my simulations and opened the rtabmap database file in a standalone rtabmap application.

Here is what I got,

  1. I could see the ground-truth in grey and estimated path in blue under the "window->show view->map graph".

but I could not ...(more)

al389660 gravatar image al389660  ( 2019-09-24 15:38:43 -0500 )edit

Please let me know how can I plot the error. thanks

al389660 gravatar image al389660  ( 2019-09-24 15:44:38 -0500 )edit

In standalone, when reopening the database, the GT results woudl not appear in Statistics panel. Can you try opening the database in rtabmap-databaseViewer, and open the Statistics panel to see it results are there? Another way to export the poses (estimated and gt) is to use rtabmap-report tool.

matlabbe gravatar image matlabbe  ( 2019-09-24 17:26:43 -0500 )edit

I just tried rtabmap-databaeViewer. However, it doesnot give me the error. Another thing is that, I want to plot the error over time like for every pose (as a continuous graph). How can I do that?. because as I understand the GT panel Statistics will only give me a number to represent an obsolute error, not the whole set of errors over entire path of robot movement.

al389660 gravatar image al389660  ( 2019-09-24 18:24:06 -0500 )edit

I updated the answer!

matlabbe gravatar image matlabbe  ( 2019-09-24 18:54:06 -0500 )edit

thank you very much for your prompt response and help that I am getting from you. I am able to edit the database files in rtabmap-databaseViewer. I would appreciate if you could either explain the GT tab its component in general or provide the link to the document. I mean what each of them represent, like you explained the drop in error is caused by loop closure. Thank you very much.

al389660 gravatar image al389660  ( 2019-09-24 19:55:52 -0500 )edit

I can plot both paths to draw a comparison but the only problem is that when I plot the error using GT panel of rtabmap-databaseViewer, it shows that error increases with the time.

My question is that the error should drop to zero at some point whenever a loop closure is detected. in order to make sure that a loop closure situation should arise, i deliberately map the same area twice. But it does not affect the error plot. I think the loop closure detection is not working at all. Please, explain what I am missing here. here is the launch file for rtabmap https://gofile.io/?c=wmjV5K

al389660 gravatar image al389660  ( 2019-09-25 16:43:16 -0500 )edit

See https://vision.in.tum.de/data/dataset... for explanation about ATE. The Gt panel show only ATE errors (in general only translational RMSE is used in papers for comparison), not RPE as in the reference. For interpretation of ATE over time, see figure 13 in this paper: https://introlab.3it.usherbrooke.ca/m...

Maybe no loop closures have been detected, can you share the database?

matlabbe gravatar image matlabbe  ( 2019-09-25 18:42:12 -0500 )edit

here is the link to the database https://gofile.io/?c=P9mEpJ. if no loop closure is detected, why is that?. I have already shared my launch file in a previous comment.

al389660 gravatar image al389660  ( 2019-09-26 02:01:49 -0500 )edit

updated answer!

matlabbe gravatar image matlabbe  ( 2019-09-26 11:15:03 -0500 )edit

I have tried to make changes to the database as recommended in your previous comment but could not produce the same results as you have shown after processing.

Here is what I did,

I opened the database (wall structure database) in rtabmap application and modified the parameters as recommended in your previous response. I was able to set all parameters except the following,

Icp/PM=true

Icp/PMOutlierRatio=0.9

Another point is that in my rtabmap application libpointmatcher doesnot allow me to enable it. after applying these changes I pushed the start button to start mapping again. but I could not produce the same results as you have produced.

Secondly, I also modified my launch file and incorporated all the parameters as suggested. and I ran the simulation all over again. but still the results were almost the same.

One more question

How to build rtabmap with libpointmatcher for ...(more)

al389660 gravatar image al389660  ( 2019-09-26 18:45:29 -0500 )edit

The ICP parameters I've shown are tuned for libpointmatcher. Install libpointmatcher, then build/install rtabmap. cmake should be able to detect the installed libpointmatcher.

matlabbe gravatar image matlabbe  ( 2019-09-26 21:10:39 -0500 )edit

Thanks for the support, I tried to explore rtabmap-report tool that you recommended for storing the text file for evaluation purposes.

Here is what I did,

I have stored my database files in a folder in Downloads, I entered to that directory using cd command, and executed rtabmap-report [DB_NAME].db.

Here is the input and output of this endeavor,

input: /Downloads/RTABMAP Databases for Post Processing$ rtabmap-report rtabmapcompare1.db

Output: Database: rtabmapcompare1.db (241, s=1,000): error lin=1,051m (max=1,051m, odom=1,051m) ang=7,7deg, slam: avg=258ms (max=521ms) loops=0, odom: avg=0ms (max=0ms), camera: avg=0ms, map=-1MB

The problem is I cannot locate any [DB_NAME]_poses.txt and [DB_NAME]_gt.txt files. where can I find those files?

I have also tried executing the same command by placing the database files in .ros folder. Still could not find any text ...(more)

al389660 gravatar image al389660  ( 2019-09-27 17:16:39 -0500 )edit

Sorry I forgot to add --poses option to rtabmap-report.

matlabbe gravatar image matlabbe  ( 2019-09-27 19:46:19 -0500 )edit

thank for your response. I am trying to use online version of the evaluation tool for calculating RPE. but it does not give me desired output rather it says,

Command: evaluate_rpe.py rgbd_dataset_freiburg2_360_kidnap-groundtruth.txt rtabmapcompare1_poses.txt --max_pairs 10000 --fixed_delta --delta 1 --delta_unit s --plot figure.png --offset 0 --scale 1 --verbose

Output: Traceback (most recent call last): File "/usr/demos/cvpr/rgbd/benchmark/rgbd_benchmark_tools/src/rgbd_benchmark_tools/evaluate_rpe.py", line 223, in float(args.scale)) File "/usr/demos/cvpr/rgbd/benchmark/rgbd_benchmark_tools/src/rgbd_benchmark_tools/evaluate_rpe.py", line 127, in evaluate_trajectory raise Exception("Number of overlap in the timestamps is too small. Did you run the evaluation on the right files?") Exception: Number of overlap in the timestamps is too small. Did you run the evaluation on the right files?

Please help me in using this tool. thanks

al389660 gravatar image al389660  ( 2019-09-28 14:40:47 -0500 )edit

also note that I upload [DataBaseName.db]_poses.txt file for evaluation purposes.

al389660 gravatar image al389660  ( 2019-09-28 14:42:58 -0500 )edit

The evaluation tool is comparing against RGB-D dataset ground truth, not your ground truth. It is likely that the timestamps won't match, as you are not comparing the same trajectories. To compare with a custom ground truth, you should download the script.

matlabbe gravatar image matlabbe  ( 2019-09-28 19:33:32 -0500 )edit

I have tried to calculate RPE using evaluate_rpe.py script.

Here is what I did,

I downloaded the evaluate_rpe.py script and changed it to executable file using "chmod +x" command. I then ran evaluate_rpe.py file with "fileName_gt.txt" and "fileName_poses.txt" arguments. It does not execute successfully.

I am getting this exception,

Traceback (most recent call last): File "./evaluate_rpe.py", line 339, in <module> float(args.scale)) File "./evaluate_rpe.py", line 240, in evaluate_trajectory raise Exception("Number of overlap in the timestamps is too small. Did you run the evaluation on the right files?") Exception: Number of overlap in the timestamps is too small. Did you run the evaluation on the right files?

Please let me know what is that I am missing here?

al389660 gravatar image al389660  ( 2019-09-29 00:43:32 -0500 )edit

Can you post the 2 trajectory files?

matlabbe gravatar image matlabbe  ( 2019-09-29 08:11:45 -0500 )edit

please find the requested files here,

https://gofile.io/?c=VZUjYM
al389660 gravatar image al389660  ( 2019-09-29 09:23:55 -0500 )edit

The problem is that report tool exported poses in raw format, not in RGB-D dataset format. I updated the code so that poses are exported by default in RGB-D dataset format (to be compatible with RGB-D dataset evaluation scripts). I also added a new export called [DBNAME]_odom.txt, renamed the [DBNAME]_poses.txt one to [DBNAME]_slam.txt. The odom one is needed to compare with evaluate_rpe.txt.

Example:

$ rtabmap-report --poses rtabmap.db
$ python evaluate_ate.py rtabmap_slam.txt rtabmap_gt.txt --plot ate.pdf
$ python evaluate_rpe.py rtabmap_odom.txt rtabmap_gt.txt --plot rpe.pdf --fixed_delta
matlabbe gravatar image matlabbe  ( 2019-09-29 18:45:33 -0500 )edit

How can I update my rtabmap code. I have installed rtabmap_ros package.

I tried to update my rtabmap code using the following instructions as suggested on introlab/rtabmap website

$ cd /opt/ros/kinetic/share/rtabmap

$ git pull origin master

$ cd build

$ cmake ..

$ make -j4

$ sudo make install

please note that my rtabmap directory is /opt/ros/kinetic/share/rtabmap.

After changing directory to /opt/ros/kinetic/share/rtabmap, I execute the "git pull origin master" and get the following error.

"fatal: Not a git repository (or any of the parent directories): .git"

I would appreciate if you could let me know how can I update my code to get the desired results.

Thanks

al389660 gravatar image al389660  ( 2019-09-29 20:16:03 -0500 )edit

You cannot do that in the binaries. Follow instructions on this page (you can ignore the Optional ones): https://github.com/introlab/rtabmap_r... You should uninstall rtabmap:

sudo apt remove ros-kinetic-rtabmap

then build from source.

matlabbe gravatar image matlabbe  ( 2019-09-30 08:33:25 -0500 )edit

After removing rtabmap_ros, I built it from source as recommended. but still rtabmap-report generates the same files.

please guide what is that I am missing. Thanks

al389660 gravatar image al389660  ( 2019-09-30 16:44:34 -0500 )edit

make sure you are starting the right rtabmap-report:

$ which rtabmap-report

You should have to rebuild also rtabmap library.

matlabbe gravatar image matlabbe  ( 2019-09-30 17:07:34 -0500 )edit

still having the same problem that rtabmap-report generates DatabaseName._poses.txt and DatabaseName_gt.text files, no odom file. the output of the query "which rtabmap-report" results,

/opt/ros/kinetic/bin/rtabmap-report

which means there is only one rtabmap-report. I have followed all instruction on the https://github.com/introlab/rtabmap_r... to reinstall the package. How can I rebuild rtabmap separately please elaborate. Thanks.

al389660 gravatar image al389660  ( 2019-09-30 20:21:08 -0500 )edit

You forgot to remove rtabmap binaries (the one under /opt/ros/kinetic is the binary!), like in my second previous comment:

sudo apt remove ros-kinetic-rtabmap

When building rtabmap from source, if you git clone in your home, the app would be there:

~/rtabmap/bin/rtabmap-report

If you make install it, it would be in /usr/local/bin/rtabmap-report!

matlabbe gravatar image matlabbe  ( 2019-09-30 20:42:28 -0500 )edit

Thanks for your support. I want to save the 2D map ( using rtabmap) of the environment and plug it into the ROS navigation Stack.

How can I do the following,

  1. save the 2D map of the environment using rtabmap.

  2. plug the saved 2D map to the ROS navigation stack.

  3. perform autonomous navigation.

Please guide me. thanks

al389660 gravatar image al389660  ( 2019-10-06 12:11:26 -0500 )edit

Question Tools

2 followers

Stats

Asked: 2019-09-19 10:07:24 -0500

Seen: 2,216 times

Last updated: Sep 29 '19