ROS Communication segfault issue with python
Hi ROS community,
Recently I encountered a very weird segfault with ros communication in python.
Here is the brief description about the problem of interest and the communication structure.
I am sending commands from local PC to control Fetch Robot via wifi connection to local router.
The local PC runs mainly motion planner and deep learning stuff and send high level command to the robot to execute.
The main local programs (experienced segfault) are written in python (2.7). Both local PC and the Fetch robot are running in Ubuntu 14.04, ROS indigo.
the weird thing about the segfault is that it did not occur consistently (with good luck, segfault does not appear for hours, with bad luck, it shows up a few times within half an hour). Also, the segfault does not occur in the same line of codes but mainly happens in two places: one dealing with a forward pass through a deep neural network which has nothing to do with ros and another dealing with calling moveit to plan a path for the manipulator.
To debug the segfault, I use python faulthandler and below are two instances of segfault happening in different lines of codes
Instance 1 when planning a path
Instance 2 when deep learning computation takes place
As you can see, even though they happened in different places, it all complains something related to tcpros and threading, but it tells me very little about what potential issues are (sorry I am not so familiar with network or communication stuff)
I am suspecting it could be due to the following 1. network connectivity issues 2. ros communication issues
Hence, I will much appreciate if anyone can help narrow down the issues from which I can debug further. Thanks a lot in advance!
If
pythonisSEGFAULTing, wouldn't it make more sense to run thepythonbinary/binaries ingdb? All the Python stacktraces show things I would expect (when using a networking middleware): lots of subscriptions waiting on new data, callbacks being called, etc. That does not mean .... the issue is in
tcpros_*.py, just that it's being used a lot (note that it could of course be that there is a problem in those classes/files, but seeing the file mentioned a lot does not imply causation).thanks for the comments @gvdhoorn! I did try to use debug with
gdbby following this tutorialbut I didn't get useful information from gdb, hence i resorted to faulthandler.
I agree with you that observing
tcprosbeing mentioned many times does not mean it is the cause of the problem. now I am more suspicious in my deep neural network with pytorch which potentially cause the problem.SEGFAULTs are (nearly) impossible to debug without backtraces, so you'll have to get one in some way.gdbis the easiest.Python typically (read: in my experience) doesn't
SEGFAULT, unless you're doing some strange things interacting with native code.