Hello all,

Recently, I am running into the following problem:

• The machine runs for several hours a set of ROS nodes, then at some point nodes cannot communicate, the commands rostopic list or rosnode list hang without response on terminal
• I went to the logs and found that one node tried to spawn it's own master, but failed because the 11311 is already occupied --> For me, this means that the other roscore/rosmaster did not died but cannot be found anymore

Any ideas why would the system run for several hours and suddenly face this problem?

The node where I spotted the problem (trying to spawn it's own master) get's triggered every 15 minutes. This means most of the day is working fine, but suddenly does not find master.

I leave the log file and my bashrc file. The reason I defined all the environment variables is because another machine in the network logs some topics.

source /opt/ros/noetic/setup.bash
source /home/bossminion/bossminionSystem/devel/setup.bash

export ROS_MASTER_URI=http://bossminion:11311/
export ROS_HOSTNAME=bossminion
export ROS_IP=10.8.0.21

And what is on the log

auto-starting new master
process[master]: started with pid [2401]
ERROR: Unable to start XML-RPC server, port 11311 is already in use
Unhandled exception in thread started by <bound method XmlRpcNode.run of <rosgraph.xmlrpc.XmlRpcNode object at 0x7f38a45133d0>>
Traceback (most recent call last):
  File "/opt/ros/noetic/lib/python3/dist-packages/rosgraph/xmlrpc.py", line 215, in run
    self._run()
  File "/opt/ros/noetic/lib/python3/dist-packages/rosgraph/xmlrpc.py", line 284, in _run
    self._run_init()
  File "/opt/ros/noetic/lib/python3/dist-packages/rosgraph/xmlrpc.py", line 234, in _run_init
    self.server = ThreadingXMLRPCServer((bind_address, port), log_requests)
  File "/opt/ros/noetic/lib/python3/dist-packages/rosgraph/xmlrpc.py", line 115, in __init__
    SimpleXMLRPCServer.__init__(self, addr, SilenceableXMLRPCRequestHandler, log_requests)
  File "/usr/lib/python3/SimpleXMLRPCServer.py", line 593, in __init__
    SocketServer.TCPServer.__init__(self, addr, requestHandler, bind_and_activate)
  File "/usr/lib/python3/SocketServer.py", line 417, in __init__
    self.server_bind()
  File "/usr/lib/python3/SocketServer.py", line 431, in server_bind
    self.socket.bind(self.server_address)
  File "/usr/lib/python3/socket.py", line 228, in meth
    return getattr(self._sock,name)(*args)
socket.error: [Errno 98] Address already in use
ERROR: could not contact master [http://localhost:11311]