ROS Resources: Documentation | Support | Discussion Forum | Index | Service Status | ros @ Robotics Stack Exchange
Ask Your Question

Logs processing good practices and recommendations

asked 2022-04-05 09:35:40 -0500

Iony_M gravatar image

Hello Dear Community,

I'm currently tasked with designing the way our company will process the log files outputted by our system (A mobile off-road robot) in order to answer some questions about the robot's performance and identify easier if there were issues, where in the system and when it occurred and so on. I should say that some of the data is time based, for example:

  1. Offset from the intended path
  2. Status throughout the drive execution

I'm currently doing some online research on tool that will help me process and create this understanding from the files which so far are just formatted text files. We have decided to distance ourselves from the logging capability of ROS and use SPDlog in C++ and the logging library in Python.

Some initial online search suggests to do string parsing for each line and take out of the string structure the relevant data, but this seems very fragile and dependent of the string structure, which most certainly will change and therefore feels that it is not the solution. I though also to make a dedicated topic for each relevant variable, status or diagnostic state I'd like to keep track of and then leverage the fact that the data is already serialized, stamped and isolated form other information and then join them together.

I would very much appreciate past experiences and suggestions on the matter.

Shoudl some of the suggestions depend on the setup currently used, this is it:

  • OS: Ubuntu 18.04
  • Release: Ros 1

Thank you a lot

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted

answered 2022-04-06 13:02:07 -0500

shonigmann gravatar image

depends a bit on the data rates you intend to capture, and there are pros and cons to sticking with ROS vs directly writing to a database. If you do go the ros topic route, it opens up a few options that might be interesting, e.g.:

You could also consider implementing your own "aggregator" node that subscribes to (or has a service interface for) updates from other published data, and periodically publishes an aggregate status message. I agree with the sentiment of avoiding string comprehension if possible.

As a completely separate note, if you are unsure how the messages you care about might change in the future, taking a page from the web-dev world could be interesting. I've heard of groups using google protobuf to keep things flexible in the face of changing definitions / content. Not something I've played with personally, but still potentially interesting.

edit flag offensive delete link more



In addition to the packages you mention, I've always found wiki/mongodb_store very interesting.

gvdhoorn gravatar image gvdhoorn  ( 2022-04-06 14:01:17 -0500 )edit

Question Tools



Asked: 2022-04-05 09:35:40 -0500

Seen: 99 times

Last updated: Apr 06 '22