ROS Logging improvements ?
ROS provides a way to log messages from any node to console (output="screen") OR to file (output="log"), out of the box.
However when working on a complex industrial product, I find this logging facility not sufficient.
I am wondering if there is any "success story" about how one should go to extend the existing logging system, to provide a full logging feature coverage for running ROS robots.
I am thinking something along the lines of :
- logging all messages from each nodes into separate (rotated&compressed) files, maybe even syslog.
- transferring some of these messages to rosout (logs propagation between nodes, rosout just happens to be the root logging node)
- storing all rosout messages into a file (same as for any node)
- showing some of the rosout messages to the terminal open with roslaunch (rosout special case, since it seems other nodes don't get access to the terminal when launched under roslaunch).
- using a unified API (not mylogger.warning here and rospy.logwarn there...)
- reordering rosout messages based on log event occurring order (using some vector clock or so), not depending on when rosout receive the message (network delay is really a thing). Probably not doable without extensive modifications though...
Any hints / pointers ?
Maybe also a tool that allows to "merge" multiple log files in a unified logtext viewer... based on timestamp ?
Maybe also a proper /rosout/mysupervisor/mynode topic hierarchy, with customizable propagation rules, with rosout managing all these topics, and logging their messages appropriately, following his own configured handlers...
Afaik,
rosconsoleuses log4cxx, which should allow you to configure asyslogappender (like this fi). That should allow you to make use of all the filtering/aggregation/etc featuressysloghas.It would probably be good for
rosoutto stay what it is today: the aggregatingstdoutof live ROS nodes. All the features that you list are nice, but I'm not sure whether they should be added torosout(or the ROS logging system itself). Separation of concerns seems like a good thing .... and other tools further down the pipeline could filter and persist according to any preference you might have.
rqt_console would seem to support some of the things you are looking for already (especially the ordering one), are you looking for infrastructure support for all of your list, or would tools also be ok?
The current problem is that (at least where I work) ROS developers are used to watch the terminal to check if a problem appears on the console. This is obviously not good enough when a robot breaks without someone watching. I actually don't mind how it is solved...
I am currently investigating the different tools/libraries ROS has for logging, and how I can extend them, or what I need to do from scratch... But I find them usually lacking compared to what is already available for monitoring general IT systems, so I might roll out some combination of these...
Piping ros log output to a system facility would allow for:
of log messages.
You mean tools like
nagiosand others? I've seen systems that combine ....
rosouttosyslog/systemdwith ros diagnostics to monitoring tool X proprietary format to get at similar functionality.Oh, any link that I can have a look at ? In the end I'll need something that can fit on a robot, with, eventually, backup on a server... But yes it seems having the possibility to easily log to syslog would be a good first step...
Did you end up settling on a good solution here?