ROS Resources: Documentation | Support | Discussion Forum | Index | Service Status | ros @ Robotics Stack Exchange
Ask Your Question

measuring metrics, performance, etc for ROS robot?

asked 2021-06-03 13:57:32 -0600

mugetsu gravatar image

updated 2022-02-20 18:58:32 -0600

lucasw gravatar image

Hi, I'm interested to learn more about best practices and existing tools for measuring performance and metrics of an autonomous vehicle. I've consulted the tutorials from but haven't seen something like this mentioned. By metrics, I'm talking about how to best measure localization, map accuracy, latency, response to dynamic objects, essentially any aspect of an AV stack that can be quantified to verify the performance of the entire system.

Are there any books, publications, or other sources to learn what other people are doing? Or anyone open to contributing their own thoughts?

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted

answered 2021-07-18 21:31:33 -0600

The answer for this question covers a subjectively large scope of study. There are various modules that we may have in autonomous vehicle. For example, if we are talking about camera perception engine, then the subsequent question is what end result(s) that we want to evaluate: is it depth map, is it object detection, is it lane detection as segmentation, is it lane detection as LDWS, etc.

However, if the question is focused only on (for instance) localization from Autoware.Auto, you may need to consider few aspects to focus in:

  • metrics to evaluate from general perspective (e.g. runtime, complexity, iterations),
  • metrics to evaluate from algorithm perspective (e.g. translation accuracy, rotational accuracy)
  • metrics to evaluate from embedded system implementation perspective (e.g. binary size, memory requirement)

If you are looking to focus on general perspective, you can look into:

If you are looking to focus on algorithm perspective, you can search relevant journal publications or any open-source qualitative evaluation depending on which one you want to evaluate (I don't think there is all-in-one source for this).

edit flag offensive delete link more

Question Tools



Asked: 2021-06-03 13:57:32 -0600

Seen: 221 times

Last updated: Jul 18 '21