Logs processing good practices and recommendations

edit

Hello Dear Community,

I'm currently tasked with designing the way our company will process the log files outputted by our system (A mobile off-road robot) in order to answer some questions about the robot's performance and identify easier if there were issues, where in the system and when it occurred and so on. I should say that some of the data is time based, for example:

1. Offset from the intended path
2. Status throughout the drive execution

I'm currently doing some online research on tool that will help me process and create this understanding from the files which so far are just formatted text files. We have decided to distance ourselves from the logging capability of ROS and use SPDlog in C++ and the logging library in Python.

Some initial online search suggests to do string parsing for each line and take out of the string structure the relevant data, but this seems very fragile and dependent of the string structure, which most certainly will change and therefore feels that it is not the solution. I though also to make a dedicated topic for each relevant variable, status or diagnostic state I'd like to keep track of and then leverage the fact that the data is already serialized, stamped and isolated form other information and then join them together.

I would very much appreciate past experiences and suggestions on the matter.

Shoudl some of the suggestions depend on the setup currently used, this is it:

• OS: Ubuntu 18.04
• Release: Ros 1

Thank you a lot