measuring metrics, performance, etc for ROS robot?
Hi, I'm interested to learn more about best practices and existing tools for measuring performance and metrics of an autonomous vehicle. I've consulted the tutorials from Autoware.auto but haven't seen something like this mentioned. By metrics, I'm talking about how to best measure localization, map accuracy, latency, response to dynamic objects, essentially any aspect of an AV stack that can be quantified to verify the performance of the entire system.
Are there any books, publications, or other sources to learn what other people are doing? Or anyone open to contributing their own thoughts?
Asked by mugetsu on 2021-06-03 13:57:32 UTC
Answers
The answer for this question covers a subjectively large scope of study. There are various modules that we may have in autonomous vehicle. For example, if we are talking about camera perception engine, then the subsequent question is what end result(s) that we want to evaluate: is it depth map, is it object detection, is it lane detection as segmentation, is it lane detection as LDWS, etc.
However, if the question is focused only on (for instance) localization from Autoware.Auto, you may need to consider few aspects to focus in:
- metrics to evaluate from general perspective (e.g. runtime, complexity, iterations),
- metrics to evaluate from algorithm perspective (e.g. translation accuracy, rotational accuracy)
- metrics to evaluate from embedded system implementation perspective (e.g. binary size, memory requirement)
If you are looking to focus on general perspective, you can look into: https://www.hanselman.com/blog/content/binary/NDepend%20metrics%20placemats%201.1.pdf
If you are looking to focus on algorithm perspective, you can search relevant journal publications or any open-source qualitative evaluation depending on which one you want to evaluate (I don't think there is all-in-one source for this).
Asked by peterchondro on 2021-07-18 21:31:33 UTC
Comments