log running nodes
Hi,
is there an easy way to know deduce from logs which nodes are running and when? Or a way to monitor it and to record it mannually (for example publishing the start and stop of nodes into some topic)?
In the end, what I would like to obtain, is something like
timestamp1 nodeA started
timestamp2 nodeB started
timestamp3 nodeA stoped
timestamp4 nodeC started
timestamp5 nodeB stoped
timestamp6 nodeC stoped
if instead of "stoped" I can know if it is a clean exit or a crash, it's even better.
If you have a solution to get this info from existing logs, it would be nice (right now, I have logs from a "crash" I couldn't find the cause yet), but if there is something to do beforehand, I'm also find with it.
A very ugly solution would be a bash script running "rosnode list" at regular intervals, then doing the diff between 2 successive runs and saving it to a file along with timestamp. But I would be surprised if there isn't any better solution
Thanks a lot in advance
Felix
Asked by felixN on 2021-02-09 05:11:47 UTC
Answers
How accurate do you need the data to be? You can find out when each node (un)registers for subscriptions and publications in master.log
, which is usually in ~/.ros/log/latest
. Here's an example of some output:
[rosmaster.master][INFO] 2021-02-09 09:18:53,983: +PUB [/rosout] /node1 http://computer:34533/
[rosmaster.master][INFO] 2021-02-09 09:18:53,984: +PUB [/topic1] /node1 http://computer:34533/
[rosmaster.master][INFO] 2021-02-09 09:18:53,984: +PUB [/topic2] /node1 http://computer:34533/
[rosmaster.master][INFO] 2021-02-09 09:18:53,988: +PARAM [/node1/param1] by /node1
[rosmaster.master][INFO] 2021-02-09 09:18:53,988: +PARAM [/node1/param2] by /node1
[rosmaster.master][INFO] 2021-02-09 09:19:44,372: -SUB [/clock] /node1 http://computer:34533/
[rosmaster.master][INFO] 2021-02-09 09:19:44,372: -SUB [/tf] /node1 http://computer:34533/
[rosmaster.master][INFO] 2021-02-09 09:19:44,372: -SUB [/tf_static] /node1 http://computer:34533/
...
[rosmaster.master][INFO] 2021-02-09 09:19:44,369: -PUB [/rosout] /node1 http://computer:34533/
[rosmaster.master][INFO] 2021-02-09 09:19:44,369: -PUB [/topic1] /node1 http://computer:34533/
[rosmaster.master][INFO] 2021-02-09 09:19:44,369: -PUB [/topic2] /node1 http://computer:34533/
[rosmaster.master][INFO] 2021-02-09 09:19:44,372: -SUB [/clock] /node1 http://computer:34533/
[rosmaster.master][INFO] 2021-02-09 09:19:44,372: -SUB [/tf] /node1 http://computer:34533/
[rosmaster.master][INFO] 2021-02-09 09:19:44,372: -SUB [/tf_static] /node1 http://computer:34533/
Also, you can find info, warnings, and errors in rosout.log
. There will be other files in the same folder, depending on your logging configuration. I'm not sure if the standard logs will contain all the information you need, but you can customize the logging: C++ wiki, Python wiki.
If you're using launch files, each of their logs may indicate more precisely what you want. An example:
...
[roslaunch][INFO] 2021-02-09 09:18:48,568: process[node1-6]: started with pid [16366]
[roslaunch][INFO] 2021-02-09 09:18:48,568: ... successfully launched [node1-6]
...
[roslaunch][INFO] 2021-02-09 09:19:44,010: [node1-6] killing on exit
[roslaunch][INFO] 2021-02-09 09:19:44,011: process[node1-6]: killing os process with pid[16366] pgid[16366]
[roslaunch][INFO] 2021-02-09 09:19:44,011: [node1-6] sending SIGINT to pgid [16366]
[roslaunch][INFO] 2021-02-09 09:19:44,012: [node1-6] sent SIGINT to pgid [16366]
[roslaunch][INFO] 2021-02-09 09:19:44,413: process[node1-6]: SIGINT killed with return value 0
...
Asked by tryan on 2021-02-09 09:40:50 UTC
Comments
Thanks a lot! It's not exactly what I was looking for, but it does the job. I had a look into master.log before, but there was too much "trash". but doing a grep on PUB and/or SUB enables not get something reasonnably readable.
Asked by felixN on 2021-02-09 09:56:10 UTC
I updated my answer with something that may be a little more on target. I agree; it can be hard to sift through the logs if you don't know exactly what to look for. grep
is a good idea.
Asked by tryan on 2021-02-09 12:59:03 UTC
Comments