Hundreds of outbound connections to rosout, OS out of sockets
This is the output of rosnode info of one of the nodes running in my computer, all nodes have a similar output.
Applications that usually do not use many resources (like robot_monitor) are consuming 100% of the CPU.
$ rosnode info /robot_monitor_8970_1329732898963
--------------------------------------------------------------------------------
Node [/robot_monitor_8970_1329732898963]
Publications:
* /rosout [rosgraph_msgs/Log]
Subscriptions:
* /clock [rosgraph_msgs/Clock]
* /diagnostics_agg [diagnostic_msgs/DiagnosticArray]
Services: None
contacting node http://denmark:40743/ ...
Pid: 8970
Connections:
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /rosout
* to: /rosout
* direction: outbound
* transport: TCPROS
* topic: /clock
* to: http://denmark:51220/
* direction: inbound
* transport: TCPROS
* topic: /clock
* to: http://denmark:50871/
* direction: inbound
* transport: TCPROS
* topic: /clock
* to: http://denmark:60632/
* direction: inbound
* transport: TCPROS
* topic: /diagnostics_agg
* to: http://denmark:43778/
* direction: inbound
* transport: TCPROS
Any idea of why is this happening? This has happened to some of my colleagues, and a similar issue might have been reported here https://code.ros.org/trac/ros/ticket/3808 and here https://code.ros.org/trac/ros/ticket/3419 but there's no definitive answer.
When this happens, most applications stop working, for instance rosbag started reporting this continuously:
[ERROR] [1329738581.938630911, 8672.295000000]: accept() on socket [7] failed with error [Too many open files]
[ERROR] [1329738581.938804866, 8672.295000000]: accept() on socket [7] failed with error [Too many open files]
[ERROR] [1329738698.288456836, 8672.295000000]: accept() on socket [7] failed with error [Too many open files]
Rosbridge this:
Problem Problem Problem Problem Problem Problem Problem Problem Problem Problem
Problem Problem Problem Problem Problem Problem Problem Problem Problem Problem
Traceback (most recent call last):
File "/opt/ros/diamondback/stacks/rosbridge/bin/rosbridge.py", line 422, in <module>
serveForever(handleFrameFactory(handleMsgFactory, sub, passfile, ros), handleOutput, loop, host, port, 1.0/(hz*2.0))
File "/opt/ros/diamondback/stacks/rosbridge/bin/terribleWSS/ws.py", line 201, in serveForever
serverSocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
File "/usr/lib/python2.6/socket.py", line 182, in __init__
_sock = _realsocket(family, type, proto)
error: [Errno 24] Too many open files
When this situation was ongoing, netstat reported over 32200 sockets in use. Upon closing all ros-related applications, the socket count went down to 700.
Logs from robot_monitor (I was unable to upload it, kept getting an error when uploading the file):
[rospy.client][INFO] 2012-02-20 11:14:58,966: init_node, name[/robot_monitor_8970_1329732898963], pid[8970]
[xmlrpc][INFO] 2012-02-20 11:14:58,967: XML-RPC server binding to 0.0.0.0
[xmlrpc][INFO] 2012-02-20 11:14:58,967: Started XML-RPC server [http://denmark:40743/]
[rospy.init][INFO] 2012-02-20 11:14:58,967: ROS Slave URI: [http://denmark:40743/]
[rospy.impl.masterslave][INFO] 2012-02-20 11:14:58,967: _ready: http://denmark:40743/
[xmlrpc][INFO] 2012-02-20 11:14:58,973: xml rpc node: starting XML-RPC server
[rospy.registration][INFO] 2012-02-20 11:14:58,973: Registering with master node http://localhost:11311
[rospy.init][INFO] 2012-02-20 11:14:59,068: registered with master
[rospy.rosout][INFO] 2012-02-20 11:14:59,068: initializing /rosout core topic
[rospy.rosout][INFO] 2012-02-20 11:14:59,073: connected to core topic /rosout
[rospy.simtime][INFO] 2012-02-20 11:14:59,088: initializing /clock core topic
[rospy.simtime][INFO] 2012-02-20 11:14:59,090: connected to core topic /clock
[xmlrpc][ERROR] 2012-02-20 11:15:10,255: Traceback (most recent call last):
File "/usr/lib/python2.6/SocketServer.py", line 558, in process_request_thread
self.finish_request(request, client_address)
File "/usr/lib/python2.6/SocketServer.py", line 320, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "/usr/lib/python2.6/SocketServer.py", line 615, in __init__
self.handle()
File "/usr/lib/python2.6/BaseHTTPServer.py", line 329, in handle
self.handle_one_request()
File "/usr/lib/python2.6/BaseHTTPServer.py", line 323, in handle_one_request
method()
File "/usr/lib/python2.6/SimpleXMLRPCServer.py", line 487, in do_POST
self.send_response(200)
File "/usr/lib/python2.6/BaseHTTPServer.py", line 384, in send_response
self.send_header('Server', self.version_string())
File "/usr/lib/python2.6/BaseHTTPServer.py", line 390, in send_header
self.wfile.write("%s: %s\r\n" % (keyword, value))
File "/usr/lib/python2.6/socket.py", line 300, in write
self.flush()
File "/usr/lib/python2.6/socket.py", line 286, in flush
self._sock.sendall(buffer)
error: [Errno 32] Broken pipe
[xmlrpc][ERROR] 2012-02-20 11:15:10,258: Traceback (most recent call last):
File "/usr/lib/python2.6/SocketServer.py", line 558, in process_request_thread
self.finish_request(request, client_address)
File "/usr/lib/python2.6/SocketServer.py", line 320, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "/usr/lib/python2.6/SocketServer.py", line 615, in __init__
self.handle()
File "/usr/lib/python2.6/BaseHTTPServer.py", line 329, in handle
self.handle_one_request()
File "/usr/lib/python2.6/BaseHTTPServer.py", line 323, in handle_one_request
method()
File "/usr/lib/python2.6/SimpleXMLRPCServer.py", line 487, in do_POST
self.send_response(200)
File "/usr/lib/python2.6/BaseHTTPServer.py", line 384, in send_response
self.send_header('Server', self.version_string())
File "/usr/lib/python2.6/BaseHTTPServer.py", line 390, in send_header
self.wfile.write("%s: %s\r\n" % (keyword, value))
File "/usr/lib/python2.6/socket.py", line 300, in write
self.flush()
File "/usr/lib/python2.6/socket.py", line 286, in flush
self._sock.sendall(buffer)
error: [Errno 32] Broken pipe
[xmlrpc][ERROR] 2012-02-20 11:15:10,259: Traceback (most recent call last):
File "/usr/lib/python2.6/SocketServer.py", line 558, in process_request_thread
self.finish_request(request, client_address)
File "/usr/lib/python2.6/SocketServer.py", line 320, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "/usr/lib/python2.6/SocketServer.py", line 615, in __init__
self.handle()
File "/usr/lib/python2.6/BaseHTTPServer.py", line 329, in handle
self.handle_one_request()
File "/usr/lib/python2.6/BaseHTTPServer.py", line 323, in handle_one_request
method()
File "/usr/lib/python2.6/SimpleXMLRPCServer.py", line 487, in do_POST
self.send_response(200)
File "/usr/lib/python2.6/BaseHTTPServer.py", line 384, in send_response
self.send_header('Server', self.version_string())
File "/usr/lib/python2.6/BaseHTTPServer.py", line 390, in send_header
self.wfile.write("%s: %s\r\n" % (keyword, value))
File "/usr/lib/python2.6/socket.py", line 300, in write
self.flush()
File "/usr/lib/python2.6/socket.py", line 286, in flush
self._sock.sendall(buffer)
error: [Errno 32] Broken pipe
[rospy.internal][WARNING] 2012-02-20 11:16:07,043: Unknown error initiating TCP/IP socket to denmark:58360 (http://denmark:50871/): Traceback (most recent call last):
File "/opt/ros/diamondback/stacks/ros_comm/clients/rospy/src/rospy/impl/tcpros_base.py", line 474, in connect
self.read_header()
File "/opt/ros/diamondback/stacks/ros_comm/clients/rospy/src/rospy/impl/tcpros_base.py", line 526, in read_header
self._validate_header(read_ros_handshake_header(self.socket, self.read_buff, self.protocol.buff_size))
File "/opt/ros/diamondback/ros/core/roslib/src/roslib/network.py", line 350, in read_ros_handshake_header
d = sock.recv(buff_size)
error: [Errno 104] Connection reset by peer
[rosout][WARNING] 2012-02-20 11:16:13,611: Inbound TCP/IP connection failed: connection from sender terminated before handshake header received. 0 bytes were received. Please check sender for additional details.
[rospy.internal][ERROR] 2012-02-20 11:16:13,615: Inbound TCP/IP connection failed:
Traceback (most recent call last):
File "/opt/ros/diamondback/stacks/ros_comm/clients/rospy/src/rospy/impl/tcpros_base.py", line 298, in _tcp_server_callback
header = read_ros_handshake_header(sock, cStringIO.StringIO(), buff_size)
File "/opt/ros/diamondback/ros/core/roslib/src/roslib/network.py", line 352, in read_ros_handshake_header
raise ROSHandshakeException("connection from sender terminated before handshake header received. %s bytes were received. Please check sender for additional details."%b.tell())
ROSHandshakeException: connection from sender terminated before handshake header received. 0 bytes were received. Please check sender for additional details.
[rosout][WARNING] 2012-02-20 11:16:14,717: Inbound TCP/IP connection failed: connection from sender terminated before handshake header received. 0 bytes were received. Please check sender for additional details.
[rospy.internal][ERROR] 2012-02-20 11:16:14,758: Inbound TCP/IP connection failed:
Traceback (most recent call last):
File "/opt/ros/diamondback/stacks/ros_comm/clients/rospy/src/rospy/impl/tcpros_base.py", line 298, in _tcp_server_callback
header = read_ros_handshake_header(sock, cStringIO.StringIO(), buff_size)
File "/opt/ros/diamondback/ros/core/roslib/src/roslib/network.py", line 352, in read_ros_handshake_header
raise ROSHandshakeException("connection from sender terminated before handshake header received. %s bytes were received. Please check sender for additional details."%b.tell())
ROSHandshakeException: connection from sender terminated before handshake header received. 0 bytes were received. Please check sender for additional details.
[rosout][WARNING] 2012-02-20 11:19:11,204: Inbound TCP/IP connection failed: connection from sender terminated before handshake header received. 0 bytes were received. Please check sender for additional details.
[rospy.internal][ERROR] 2012-02-20 11:19:11,214: Inbound TCP/IP connection failed:
Traceback (most recent call last):
File "/opt/ros/diamondback/stacks/ros_comm/clients/rospy/src/rospy/impl/tcpros_base.py", line 298, in _tcp_server_callback
header = read_ros_handshake_header(sock, cStringIO.StringIO(), buff_size)
File "/opt/ros/diamondback/ros/core/roslib/src/roslib/network.py", line 352, in read_ros_handshake_header
raise ROSHandshakeException("connection from sender terminated before handshake header received. %s bytes were received. Please check sender for additional details."%b.tell())
ROSHandshakeException: connection from sender terminated before handshake header received. 0 bytes were received. Please check sender for additional details.
[rosout][WARNING] 2012-02-20 11:19:22,275: Inbound TCP/IP connection failed: connection from sender terminated before handshake header received. 0 bytes were received. Please check sender for additional details.
[rospy.internal][ERROR] 2012-02-20 11:19:22,376: Inbound TCP/IP connection failed:
Traceback (most recent call last):
File "/opt/ros/diamondback/stacks/ros_comm/clients/rospy/src/rospy/impl/tcpros_base.py", line 298, in _tcp_server_callback
header = read_ros_handshake_header(sock, cStringIO.StringIO(), buff_size)
File "/opt/ros/diamondback/ros/core/roslib/src/roslib/network.py", line 352, in read_ros_handshake_header
raise ROSHandshakeException("connection from sender terminated before handshake header received. %s bytes were received. Please check sender for additional details."%b.tell())
ROSHandshakeException: connection from sender terminated before handshake header received. 0 bytes were received. Please check sender for additional details.
[rosout][WARNING] 2012-02-20 11:21:28,812: Inbound TCP/IP connection failed: connection from sender terminated before handshake header received. 0 bytes were received. Please check sender for additional details.
[rospy.internal][ERROR] 2012-02-20 11:21:28,824: Inbound TCP/IP connection failed:
Traceback (most recent call last):
File "/opt/ros/diamondback/stacks/ros_comm/clients/rospy/src/rospy/impl/tcpros_base.py", line 298, in _tcp_server_callback
header = read_ros_handshake_header(sock, cStringIO.StringIO(), buff_size)
File "/opt/ros/diamondback/ros/core/roslib/src/roslib/network.py", line 352, in read_ros_handshake_header
raise ROSHandshakeException("connection from sender terminated before handshake header received. %s bytes were received. Please check sender for additional details."%b.tell())
ROSHandshakeException: connection from sender terminated before handshake header received. 0 bytes were received. Please check sender for additional details.
[rosout][WARNING] 2012-02-20 11:21:52,633: Inbound TCP/IP connection failed: connection from sender terminated before handshake header received. 0 bytes were received. Please check sender for additional details.
[rospy.internal][ERROR] 2012-02-20 11:21:52,720: Inbound TCP/IP connection failed:
Traceback (most recent call last):
File "/opt/ros/diamondback/stacks/ros_comm/clients/rospy/src/rospy/impl/tcpros_base.py", line 298, in _tcp_server_callback
header = read_ros_handshake_header(sock, cStringIO.StringIO(), buff_size)
File "/opt/ros/diamondback/ros/core/roslib/src/roslib/network.py", line 352, in read_ros_handshake_header
raise ROSHandshakeException("connection from sender terminated before handshake header received. %s bytes were received. Please check sender for additional details."%b.tell())
ROSHandshakeException: connection from sender terminated before handshake header received. 0 bytes were received. Please check sender for additional details.
[rosout][WARNING] 2012-02-20 11:23:23,451: Inbound TCP/IP connection failed: connection from sender terminated before handshake header received. 0 bytes were received. Please check sender for additional details.
[rospy.internal][ERROR] 2012-02-20 11:23:23,471: Inbound TCP/IP connection failed:
Traceback (most recent call last):
File "/opt/ros/diamondback/stacks/ros_comm/clients/rospy/src/rospy/impl/tcpros_base.py", line 298, in _tcp_server_callback
header = read_ros_handshake_header(sock, cStringIO.StringIO(), buff_size)
File "/opt/ros/diamondback/ros/core/roslib/src/roslib/network.py", line 352, in read_ros_handshake_header
raise ROSHandshakeException("connection from sender terminated before handshake header received. %s bytes were received. Please check sender for additional details."%b.tell())
ROSHandshakeException: connection from sender terminated before handshake header received. 0 bytes were received. Please check sender for additional details.
[rosout][WARNING] 2012-02-20 11:25:22,490: Inbound TCP/IP connection failed: connection from sender terminated before handshake header received. 0 bytes were received. Please check sender for additional details.
[rospy.internal][ERROR] 2012-02-20 11:25:22,555: Inbound TCP/IP connection failed:
Traceback (most recent call last):
File "/opt/ros/diamondback/stacks/ros_comm/clients/rospy/src/rospy/impl/tcpros_base.py", line 298, in _tcp_server_callback
header = read_ros_handshake_header(sock, cStringIO.StringIO(), buff_size)
File "/opt/ros/diamondback/ros/core/roslib/src/roslib/network.py", line 352, in read_ros_handshake_header
raise ROSHandshakeException("connection from sender terminated before handshake header received. %s bytes were received. Please check sender for additional details."%b.tell())
ROSHandshakeException: connection from sender terminated before handshake header received. 0 bytes were received. Please check sender for additional details.
[rosout][WARNING] 2012-02-20 11:25:24,808: Inbound TCP/IP connection failed: connection from sender terminated before handshake header received. 0 bytes were received. Please check sender for additional details.
[rospy.internal][ERROR] 2012-02-20 11:25:24,902: Inbound TCP/IP connection failed:
Traceback (most recent call last):
File "/opt/ros/diamondback/stacks/ros_comm/clients/rospy/src/rospy/impl/tcpros_base.py", line 298, in _tcp_server_callback
header = read_ros_handshake_header(sock, cStringIO.StringIO(), buff_size)
File "/opt/ros/diamondback/ros/core/roslib/src/roslib/network.py", line 352, in read_ros_handshake_header
raise ROSHandshakeException("connection from sender terminated before handshake header received. %s bytes were received. Please check sender for additional details."%b.tell())
ROSHandshakeException: connection from sender terminated before handshake header received. 0 bytes were received. Please check sender for additional details.
[rosout][WARNING] 2012-02-20 11:25:59,957: Inbound TCP/IP connection failed: connection from sender terminated before handshake header received. 0 bytes were received. Please check sender for additional details.
[rospy.internal][ERROR] 2012-02-20 11:25:59,960: Inbound TCP/IP connection failed:
Traceback (most recent call last):
File "/opt/ros/diamondback/stacks/ros_comm/clients/rospy/src/rospy/impl/tcpros_base.py", line 298, in _tcp_server_callback
header = read_ros_handshake_header(sock, cStringIO.StringIO(), buff_size)
File "/opt/ros/diamondback/ros/core/roslib/src/roslib/network.py", line 352, in read_ros_handshake_header
raise ROSHandshakeException("connection from sender terminated before handshake header received. %s bytes were received. Please check sender for additional details."%b.tell())
ROSHandshakeException: connection from sender terminated before handshake header received. 0 bytes were received. Please check sender for additional details.
[rosout][WARNING] 2012-02-20 11:26:41,569: Inbound TCP/IP connection failed: connection from sender terminated before handshake header received. 0 bytes were received. Please check sender for additional details.
[rospy.internal][ERROR] 2012-02-20 11:26:41,591: Inbound TCP/IP connection failed:
Traceback (most recent call last):
File "/opt/ros/diamondback/stacks/ros_comm/clients/rospy/src/rospy/impl/tcpros_base.py", line 298, in _tcp_server_callback
header = read_ros_handshake_header(sock, cStringIO.StringIO(), buff_size)
File "/opt/ros/diamondback/ros/core/roslib/src/roslib/network.py", line 352, in read_ros_handshake_header
raise ROSHandshakeException("connection from sender terminated before handshake header received. %s bytes were received. Please check sender for additional details."%b.tell())
ROSHandshakeException: connection from sender terminated before handshake header received. 0 bytes were received. Please check sender for additional details.
[rosout][WARNING] 2012-02-20 11:26:43,164: Inbound TCP/IP connection failed: connection from sender terminated before handshake header received. 0 bytes were received. Please check sender for additional details.
[rospy.internal][ERROR] 2012-02-20 11:26:43,187: Inbound TCP/IP connection failed:
Traceback (most recent call last):
File "/opt/ros/diamondback/stacks/ros_comm/clients/rospy/src/rospy/impl/tcpros_base.py", line 298, in _tcp_server_callback
header = read_ros_handshake_header(sock, cStringIO.StringIO(), buff_size)
File "/opt/ros/diamondback/ros/core/roslib/src/roslib/network.py", line 352, in read_ros_handshake_header
raise ROSHandshakeException("connection from sender terminated before handshake header received. %s bytes were received. Please check sender for additional details."%b.tell())
ROSHandshakeException: connection from sender terminated before handshake header received. 0 bytes were received. Please check sender for additional details.
[rosout][WARNING] 2012-02-20 11:28:20,161: Inbound TCP/IP connection failed: connection from sender terminated before handshake header received. 0 bytes were received. Please check sender for additional details.
[rospy.internal][ERROR] 2012-02-20 11:28:20,167: Inbound TCP/IP connection failed:
Traceback (most recent call last):
File "/opt/ros/diamondback/stacks/ros_comm/clients/rospy/src/rospy/impl/tcpros_base.py", line 298, in _tcp_server_callback
header = read_ros_handshake_header(sock, cStringIO.StringIO(), buff_size)
File "/opt/ros/diamondback/ros/core/roslib/src/roslib/network.py", line 352, in read_ros_handshake_header
raise ROSHandshakeException("connection from sender terminated before handshake header received. %s bytes were received. Please check sender for additional details."%b.tell())
ROSHandshakeException: connection from sender terminated before handshake header received. 0 bytes were received. Please check sender for additional details.
[rosout][WARNING] 2012-02-20 11:30:31,384: Inbound TCP/IP connection failed: connection from sender terminated before handshake header received. 0 bytes were received. Please check sender for additional details.
[rospy.internal][ERROR] 2012-02-20 11:30:31,455: Inbound TCP/IP connection failed:
Traceback (most recent call last):
File "/opt/ros/diamondback/stacks/ros_comm/clients/rospy/src/rospy/impl/tcpros_base.py", line 298, in _tcp_server_callback
header = read_ros_handshake_header(sock, cStringIO.StringIO(), buff_size)
File "/opt/ros/diamondback/ros/core/roslib/src/roslib/network.py", line 352, in read_ros_handshake_header
raise ROSHandshakeException("connection from sender terminated before handshake header received. %s bytes were received. Please check sender for additional details."%b.tell())
ROSHandshakeException: connection from sender terminated before handshake header received. 0 bytes were received. Please check sender for additional details.
[rosout][WARNING] 2012-02-20 11:32:51,693: Inbound TCP/IP connection failed: connection from sender terminated before handshake header received. 0 bytes were received. Please check sender for additional details.
[rospy.internal][ERROR] 2012-02-20 11:32:51,807: Inbound TCP/IP connection failed:
Traceback (most recent call last):
File "/opt/ros/diamondback/stacks/ros_comm/clients/rospy/src/rospy/impl/tcpros_base.py", line 298, in _tcp_server_callback
header = read_ros_handshake_header(sock, cStringIO.StringIO(), buff_size)
File "/opt/ros/diamondback/ros/core/roslib/src/roslib/network.py", line 352, in read_ros_handshake_header
raise ROSHandshakeException("connection from sender terminated before handshake header received. %s bytes were received. Please check sender for additional details."%b.tell())
ROSHandshakeException: connection from sender terminated before handshake header received. 0 bytes were received. Please check sender for additional details.
[rosout][WARNING] 2012-02-20 11:33:01,726: Inbound TCP/IP connection failed: connection from sender terminated before handshake header received. 0 bytes were received. Please check sender for additional details.
[rospy.internal][ERROR] 2012-02-20 11:33:01,881: Inbound TCP/IP connection failed:
Traceback (most recent call last):
File "/opt/ros/diamondback/stacks/ros_comm/clients/rospy/src/rospy/impl/tcpros_base.py", line 298, in _tcp_server_callback
header = read_ros_handshake_header(sock, cStringIO.StringIO(), buff_size)
File "/opt/ros/diamondback/ros/core/roslib/src/roslib/network.py", line 352, in read_ros_handshake_header
raise ROSHandshakeException("connection from sender terminated before handshake header received. %s bytes were received. Please check sender for additional details."%b.tell())
ROSHandshakeException: connection from sender terminated before handshake header received. 0 bytes were received. Please check sender for additional details.
[rosout][WARNING] 2012-02-20 11:33:33,871: Inbound TCP/IP connection failed: connection from sender terminated before handshake header received. 0 bytes were received. Please check sender for additional details.
[rospy.internal][ERROR] 2012-02-20 11:33:33,879: Inbound TCP/IP connection failed:
Traceback (most recent call last):
File "/opt/ros/diamondback/stacks/ros_comm/clients/rospy/src/rospy/impl/tcpros_base.py", line 298, in _tcp_server_callback
header = read_ros_handshake_header(sock, cStringIO.StringIO(), buff_size)
File "/opt/ros/diamondback/ros/core/roslib/src/roslib/network.py", line 352, in read_ros_handshake_header
raise ROSHandshakeException("connection from sender terminated before handshake header received. %s bytes were received. Please check sender for additional details."%b.tell())
ROSHandshakeException: connection from sender terminated before handshake header received. 0 bytes were received. Please check sender for additional details.
[rosout][WARNING] 2012-02-20 11:33:41,618: Inbound TCP/IP connection failed: connection from sender terminated before handshake header received. 0 bytes were received. Please check sender for additional details.
[rospy.internal][ERROR] 2012-02-20 11:33:41,640: Inbound TCP/IP connection failed:
Traceback (most recent call last):
File "/opt/ros/diamondback/stacks/ros_comm/clients/rospy/src/rospy/impl/tcpros_base.py", line 298, in _tcp_server_callback
header = read_ros_handshake_header(sock, cStringIO.StringIO(), buff_size)
File "/opt/ros/diamondback/ros/core/roslib/src/roslib/network.py", line 352, in read_ros_handshake_header
raise ROSHandshakeException("connection from sender terminated before handshake header received. %s bytes were received. Please check sender for additional details."%b.tell())
ROSHandshakeException: connection from sender terminated before handshake header received. 0 bytes were received. Please check sender for additional details.
[rosout][WARNING] 2012-02-20 11:34:20,987: Inbound TCP/IP connection failed: connection from sender terminated before handshake header received. 0 bytes were received. Please check sender for additional details.
[rospy.internal][ERROR] 2012-02-20 11:34:20,993: Inbound TCP/IP connection failed:
Traceback (most recent call last):
File "/opt/ros/diamondback/stacks/ros_comm/clients/rospy/src/rospy/impl/tcpros_base.py", line 298, in _tcp_server_callback
header = read_ros_handshake_header(sock, cStringIO.StringIO(), buff_size)
File "/opt/ros/diamondback/ros/core/roslib/src/roslib/network.py", line 352, in read_ros_handshake_header
raise ROSHandshakeException("connection from sender terminated before handshake header received. %s bytes were received. Please check sender for additional details."%b.tell())
ROSHandshakeException: connection from sender terminated before handshake header received. 0 bytes were received. Please check sender for additional details.
[rosout][WARNING] 2012-02-20 11:36:49,729: Inbound TCP/IP connection failed: connection from sender terminated before handshake header received. 0 bytes were received. Please check sender for additional details.
[rospy.internal][ERROR] 2012-02-20 11:36:49,776: Inbound TCP/IP connection failed:
Traceback (most recent call last):
File "/opt/ros/diamondback/stacks/ros_comm/clients/rospy/src/rospy/impl/tcpros_base.py", line 298, in _tcp_server_callback
header = read_ros_handshake_header(sock, cStringIO.StringIO(), buff_size)
File "/opt/ros/diamondback/ros/core/roslib/src/roslib/network.py", line 352, in read_ros_handshake_header
raise ROSHandshakeException("connection from sender terminated before handshake header received. %s bytes were received. Please check sender for additional details."%b.tell())
ROSHandshakeException: connection from sender terminated before handshake header received. 0 bytes were received. Please check sender for additional details.
[rosout][WARNING] 2012-02-20 11:37:19,360: Inbound TCP/IP connection failed: connection from sender terminated before handshake header received. 0 bytes were received. Please check sender for additional details.
[rospy.internal][ERROR] 2012-02-20 11:37:19,519: Inbound TCP/IP connection failed:
Traceback (most recent call last):
File "/opt/ros/diamondback/stacks/ros_comm/clients/rospy/src/rospy/impl/tcpros_base.py", line 298, in _tcp_server_callback
header = read_ros_handshake_header(sock, cStringIO.StringIO(), buff_size)
File "/opt/ros/diamondback/ros/core/roslib/src/roslib/network.py", line 352, in read_ros_handshake_header
raise ROSHandshakeException("connection from sender terminated before handshake header received. %s bytes were received. Please check sender for additional details."%b.tell())
ROSHandshakeException: connection from sender terminated before handshake header received. 0 bytes were received. Please check sender for additional details.
[rosout][WARNING] 2012-02-20 11:37:22,980: Inbound TCP/IP connection failed: connection from sender terminated before handshake header received. 0 bytes were received. Please check sender for additional details.
[rospy.internal][ERROR] 2012-02-20 11:37:23,000: Inbound TCP/IP connection failed:
Traceback (most recent call last):
File "/opt/ros/diamondback/stacks/ros_comm/clients/rospy/src/rospy/impl/tcpros_base.py", line 298, in _tcp_server_callback
header = read_ros_handshake_header(sock, cStringIO.StringIO(), buff_size)
File "/opt/ros/diamondback/ros/core/roslib/src/roslib/network.py", line 352, in read_ros_handshake_header
raise ROSHandshakeException("connection from sender terminated before handshake header received. %s bytes were received. Please check sender for additional details."%b.tell())
ROSHandshakeException: connection from sender terminated before handshake header received. 0 bytes were received. Please check sender for additional details.
[rosout][WARNING] 2012-02-20 11:37:25,680: Inbound TCP/IP connection failed: connection from sender terminated before handshake header received. 0 bytes were received. Please check sender for additional details.
[rospy.internal][ERROR] 2012-02-20 11:37:25,683: Inbound TCP/IP connection failed:
Traceback (most recent call last):
File "/opt/ros/diamondback/stacks/ros_comm/clients/rospy/src/rospy/impl/tcpros_base.py", line 298, in _tcp_server_callback
header = read_ros_handshake_header(sock, cStringIO.StringIO(), buff_size)
File "/opt/ros/diamondback/ros/core/roslib/src/roslib/network.py", line 352, in read_ros_handshake_header
raise ROSHandshakeException("connection from sender terminated before handshake header received. %s bytes were received. Please check sender for additional details."%b.tell())
ROSHandshakeException: connection from sender terminated before handshake header received. 0 bytes were received. Please check sender for additional details.
[rosout][WARNING] 2012-02-20 11:37:31,434: Inbound TCP/IP connection failed: connection from sender terminated before handshake header received. 0 bytes were received. Please check sender for additional details.
[rospy.internal][ERROR] 2012-02-20 11:37:31,437: Inbound TCP/IP connection failed:
Traceback (most recent call last):
File "/opt/ros/diamondback/stacks/ros_comm/clients/rospy/src/rospy/impl/tcpros_base.py", line 298, in _tcp_server_callback
header = read_ros_handshake_header(sock, cStringIO.StringIO(), buff_size)
File "/opt/ros/diamondback/ros/core/roslib/src/roslib/network.py", line 352, in read_ros_handshake_header
raise ROSHandshakeException("connection from sender terminated before handshake header received. %s bytes were received. Please check sender for additional details."%b.tell())
ROSHandshakeException: connection from sender terminated before handshake header received. 0 bytes were received. Please check sender for additional details.
[rosout][WARNING] 2012-02-20 11:38:00,359: Inbound TCP/IP connection failed: connection from sender terminated before handshake header received. 0 bytes were received. Please check sender for additional details.
[rospy.internal][ERROR] 2012-02-20 11:38:00,364: Inbound TCP/IP connection failed:
Traceback (most recent call last):
File "/opt/ros/diamondback/stacks/ros_comm/clients/rospy/src/rospy/impl/tcpros_base.py", line 298, in _tcp_server_callback
header = read_ros_handshake_header(sock, cStringIO.StringIO(), buff_size)
File "/opt/ros/diamondback/ros/core/roslib/src/roslib/network.py", line 352, in read_ros_handshake_header
raise ROSHandshakeException("connection from sender terminated before handshake header received. %s bytes were received. Please check sender for additional details."%b.tell())
ROSHandshakeException: connection from sender terminated before handshake header received. 0 bytes were received. Please check sender for additional details.
[rosout][WARNING] 2012-02-20 11:39:12,910: Inbound TCP/IP connection failed: connection from sender terminated before handshake header received. 0 bytes were received. Please check sender for additional details.
[rospy.internal][ERROR] 2012-02-20 11:39:12,952: Inbound TCP/IP connection failed:
Traceback (most recent call last):
File "/opt/ros/diamondback/stacks/ros_comm/clients/rospy/src/rospy/impl/tcpros_base.py", line 298, in _tcp_server_callback
header = read_ros_handshake_header(sock, cStringIO.StringIO(), buff_size)
File "/opt/ros/diamondback/ros/core/roslib/src/roslib/network.py", line 352, in read_ros_handshake_header
raise ROSHandshakeException("connection from sender terminated before handshake header received. %s bytes were received. Please check sender for additional details."%b.tell())
ROSHandshakeException: connection from sender terminated before handshake header received. 0 bytes were received. Please check sender for additional details.
[rosout][WARNING] 2012-02-20 11:39:30,493: Inbound TCP/IP connection failed: connection from sender terminated before handshake header received. 0 bytes were received. Please check sender for additional details.
[rospy.internal][ERROR] 2012-02-20 11:39:30,587: Inbound TCP/IP connection failed:
Traceback (most recent call last):
File "/opt/ros/diamondback/stacks/ros_comm/clients/rospy/src/rospy/impl/tcpros_base.py", line 298, in _tcp_server_callback
header = read_ros_handshake_header(sock, cStringIO.StringIO(), buff_size)
File "/opt/ros/diamondback/ros/core/roslib/src/roslib/network.py", line 352, in read_ros_handshake_header
raise ROSHandshakeException("connection from sender terminated before handshake header received. %s bytes were received. Please check sender for additional details."%b.tell())
ROSHandshakeException: connection from sender terminated before handshake header received. 0 bytes were received. Please check sender for additional details.
[rosout][WARNING] 2012-02-20 11:42:30,101: Inbound TCP/IP connection failed: connection from sender terminated before handshake header received. 0 bytes were received. Please check sender for additional details.
[rospy.internal][ERROR] 2012-02-20 11:42:30,125: Inbound TCP/IP connection failed:
Traceback (most recent call last):
File "/opt/ros/diamondback/stacks/ros_comm/clients/rospy/src/rospy/impl/tcpros_base.py", line 298, in _tcp_server_callback
header = read_ros_handshake_header(sock, cStringIO.StringIO(), buff_size)
File "/opt/ros/diamondback/ros/core/roslib/src/roslib/network.py", line 352, in read_ros_handshake_header
raise ROSHandshakeException("connection from sender terminated before handshake header received. %s bytes were received. Please check sender for additional details."%b.tell())
ROSHandshakeException: connection from sender terminated before handshake header received. 0 bytes were received. Please check sender for additional details.
[rospy.internal][WARNING] 2012-02-20 12:45:29,005: Unknown error initiating TCP/IP socket to denmark:33929 (http://denmark:51220/): Traceback (most recent call last):
File "/opt/ros/diamondback/stacks/ros_comm/clients/rospy/src/rospy/impl/tcpros_base.py", line 472, in connect
File "<string>", line 1, in connect
gaierror: [Errno -5] No address associated with hostname
[xmlrpc][ERROR] 2012-02-20 12:45:29,006: Traceback (most recent call last):
File "/usr/lib/python2.6/SocketServer.py", line 558, in process_request_thread
self.finish_request(request, client_address)
File "/usr/lib/python2.6/SocketServer.py", line 320, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "/usr/lib/python2.6/SocketServer.py", line 615, in __init__
self.handle()
File "/usr/lib/python2.6/BaseHTTPServer.py", line 329, in handle
self.handle_one_request()
File "/usr/lib/python2.6/BaseHTTPServer.py", line 323, in handle_one_request
method()
File "/usr/lib/python2.6/SimpleXMLRPCServer.py", line 487, in do_POST
self.send_response(200)
File "/usr/lib/python2.6/BaseHTTPServer.py", line 382, in send_response
(self.protocol_version, code, message))
File "/usr/lib/python2.6/socket.py", line 300, in write
self.flush()
File "/usr/lib/python2.6/socket.py", line 286, in flush
self._sock.sendall(buffer)
error: [Errno 32] Broken pipe
[xmlrpc][ERROR] 2012-02-20 12:45:29,007: Traceback (most recent call last):
File "/usr/lib/python2.6/SocketServer.py", line 558, in process_request_thread
self.finish_request(request, client_address)
File "/usr/lib/python2.6/SocketServer.py", line 320, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "/usr/lib/python2.6/SocketServer.py", line 615, in __init__
self.handle()
File "/usr/lib/python2.6/BaseHTTPServer.py", line 329, in handle
self.handle_one_request()
File "/usr/lib/python2.6/BaseHTTPServer.py", line 323, in handle_one_request
method()
File "/usr/lib/python2.6/SimpleXMLRPCServer.py", line 487, in do_POST
self.send_response(200)
File "/usr/lib/python2.6/BaseHTTPServer.py", line 382, in send_response
(self.protocol_version, code, message))
File "/usr/lib/python2.6/socket.py", line 300, in write
self.flush()
File "/usr/lib/python2.6/socket.py", line 286, in flush
self._sock.sendall(buffer)
error: [Errno 32] Broken pipe
[xmlrpc][ERROR] 2012-02-20 12:45:29,008: Traceback (most recent call last):
File "/usr/lib/python2.6/SocketServer.py", line 558, in process_request_thread
self.finish_request(request, client_address)
File "/usr/lib/python2.6/SocketServer.py", line 320, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "/usr/lib/python2.6/SocketServer.py", line 615, in __init__
self.handle()
File "/usr/lib/python2.6/BaseHTTPServer.py", line 329, in handle
self.handle_one_request()
File "/usr/lib/python2.6/BaseHTTPServer.py", line 323, in handle_one_request
method()
File "/usr/lib/python2.6/SimpleXMLRPCServer.py", line 487, in do_POST
self.send_response(200)
File "/usr/lib/python2.6/BaseHTTPServer.py", line 382, in send_response
(self.protocol_version, code, message))
File "/usr/lib/python2.6/socket.py", line 300, in write
self.flush()
File "/usr/lib/python2.6/socket.py", line 286, in flush
self._sock.sendall(buffer)
error: [Errno 32] Broken pipe
[xmlrpc][ERROR] 2012-02-20 12:45:29,009: Traceback (most recent call last):
File "/usr/lib/python2.6/SocketServer.py", line 558, in process_request_thread
self.finish_request(request, client_address)
File "/usr/lib/python2.6/SocketServer.py", line 320, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "/usr/lib/python2.6/SocketServer.py", line 615, in __init__
self.handle()
File "/usr/lib/python2.6/BaseHTTPServer.py", line 329, in handle
self.handle_one_request()
File "/usr/lib/python2.6/BaseHTTPServer.py", line 323, in handle_one_request
method()
File "/usr/lib/python2.6/SimpleXMLRPCServer.py", line 487, in do_POST
self.send_response(200)
File "/usr/lib/python2.6/BaseHTTPServer.py", line 384, in send_response
self.send_header('Server', self.version_string())
File "/usr/lib/python2.6/BaseHTTPServer.py", line 390, in send_header
self.wfile.write("%s: %s\r\n" % (keyword, value))
File "/usr/lib/python2.6/socket.py", line 300, in write
self.flush()
File "/usr/lib/python2.6/socket.py", line 286, in flush
self._sock.sendall(buffer)
error: [Errno 32] Broken pipe
[xmlrpc][ERROR] 2012-02-20 12:45:29,011: Traceback (most recent call last):
File "/usr/lib/python2.6/SocketServer.py", line 558, in process_request_thread
self.finish_request(request, client_address)
File "/usr/lib/python2.6/SocketServer.py", line 320, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "/usr/lib/python2.6/SocketServer.py", line 615, in __init__
self.handle()
File "/usr/lib/python2.6/BaseHTTPServer.py", line 329, in handle
self.handle_one_request()
File "/usr/lib/python2.6/BaseHTTPServer.py", line 323, in handle_one_request
method()
File "/usr/lib/python2.6/SimpleXMLRPCServer.py", line 487, in do_POST
self.send_response(200)
File "/usr/lib/python2.6/BaseHTTPServer.py", line 384, in send_response
self.send_header('Server', self.version_string())
File "/usr/lib/python2.6/BaseHTTPServer.py", line 390, in send_header
self.wfile.write("%s: %s\r\n" % (keyword, value))
File "/usr/lib/python2.6/socket.py", line 300, in write
self.flush()
File "/usr/lib/python2.6/socket.py", line 286, in flush
self._sock.sendall(buffer)
error: [Errno 32] Broken pipe
[xmlrpc][ERROR] 2012-02-20 12:45:29,011: Traceback (most recent call last):
File "/usr/lib/python2.6/SocketServer.py", line 558, in process_request_thread
self.finish_request(request, client_address)
File "/usr/lib/python2.6/SocketServer.py", line 320, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "/usr/lib/python2.6/SocketServer.py", line 615, in __init__
self.handle()
File "/usr/lib/python2.6/BaseHTTPServer.py", line 329, in handle
self.handle_one_request()
File "/usr/lib/python2.6/BaseHTTPServer.py", line 323, in handle_one_request
method()
File "/usr/lib/python2.6/SimpleXMLRPCServer.py", line 487, in do_POST
self.send_response(200)
File "/usr/lib/python2.6/BaseHTTPServer.py", line 384, in send_response
self.send_header('Server', self.version_string())
File "/usr/lib/python2.6/BaseHTTPServer.py", line 390, in send_header
self.wfile.write("%s: %s\r\n" % (keyword, value))
File "/usr/lib/python2.6/socket.py", line 300, in write
self.flush()
File "/usr/lib/python2.6/socket.py", line 286, in flush
self._sock.sendall(buffer)
error: [Errno 32] Broken pipe
[xmlrpc][ERROR] 2012-02-20 12:45:29,073: Traceback (most recent call last):
File "/usr/lib/python2.6/SocketServer.py", line 558, in process_request_thread
self.finish_request(request, client_address)
File "/usr/lib/python2.6/SocketServer.py", line 320, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "/usr/lib/python2.6/SocketServer.py", line 615, in __init__
self.handle()
File "/usr/lib/python2.6/BaseHTTPServer.py", line 329, in handle
self.handle_one_request()
File "/usr/lib/python2.6/BaseHTTPServer.py", line 323, in handle_one_request
method()
File "/usr/lib/python2.6/SimpleXMLRPCServer.py", line 487, in do_POST
self.send_response(200)
File "/usr/lib/python2.6/BaseHTTPServer.py", line 384, in send_response
self.send_header('Server', self.version_string())
File "/usr/lib/python2.6/BaseHTTPServer.py", line 390, in send_header
self.wfile.write("%s: %s\r\n" % (keyword, value))
File "/usr/lib/python2.6/socket.py", line 300, in write
self.flush()
File "/usr/lib/python2.6/socket.py", line 286, in flush
self._sock.sendall(buffer)
error: [Errno 32] Broken pipe
[xmlrpc][ERROR] 2012-02-20 12:45:29,204: Traceback (most recent call last):
File "/usr/lib/python2.6/SocketServer.py", line 558, in process_request_thread
self.finish_request(request, client_address)
File "/usr/lib/python2.6/SocketServer.py", line 320, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "/usr/lib/python2.6/SocketServer.py", line 615, in __init__
self.handle()
File "/usr/lib/python2.6/BaseHTTPServer.py", line 329, in handle
self.handle_one_request()
File "/usr/lib/python2.6/BaseHTTPServer.py", line 323, in handle_one_request
method()
File "/usr/lib/python2.6/SimpleXMLRPCServer.py", line 487, in do_POST
self.send_response(200)
File "/usr/lib/python2.6/BaseHTTPServer.py", line 384, in send_response
self.send_header('Server', self.version_string())
File "/usr/lib/python2.6/BaseHTTPServer.py", line 390, in send_header
self.wfile.write("%s: %s\r\n" % (keyword, value))
File "/usr/lib/python2.6/socket.py", line 300, in write
self.flush()
File "/usr/lib/python2.6/socket.py", line 286, in flush
self._sock.sendall(buffer)
error: [Errno 32] Broken pipe
[rospy.core][INFO] 2012-02-20 12:45:51,777: signal_shutdown [signal-15]
[rospy.registration][WARNING] 2012-02-20 12:45:51,779: unclean shutdown
Traceback (most recent call last):
File "/opt/ros/diamondback/stacks/ros_comm/clients/rospy/src/rospy/impl/registration.py", line 359, in cleanup
File "/usr/lib/python2.6/xmlrpclib.py", line 1042, in __call__
File "/usr/lib/python2.6/xmlrpclib.py", line 1199, in __call__
File "/usr/lib/python2.6/xmlrpclib.py", line 1489, in __request
File "/usr/lib/python2.6/xmlrpclib.py", line 1235, in request
File "/usr/lib/python2.6/xmlrpclib.py", line 1349, in send_content
File "/usr/lib/python2.6/httplib.py", line 904, in endheaders
File "/usr/lib/python2.6/httplib.py", line 776, in _send_output
File "/usr/lib/python2.6/httplib.py", line 735, in send
File "/usr/lib/python2.6/httplib.py", line 716, in connect
File "/usr/lib/python2.6/socket.py", line 500, in create_connection
for res in getaddrinfo(host, port, 0, SOCK_STREAM):
gaierror: [Errno -5] No address associated with hostname
[xmlrpc][ERROR] 2012-02-20 12:45:51,786: Traceback (most recent call last):
File "/usr/lib/python2.6/SocketServer.py", line 558, in process_request_thread
self.finish_request(request, client_address)
File "/usr/lib/python2.6/SocketServer.py", line 320, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "/usr/lib/python2.6/SocketServer.py", line 615, in __init__
self.handle()
File "/usr/lib/python2.6/BaseHTTPServer.py", line 329, in handle
self.handle_one_request()
File "/usr/lib/python2.6/BaseHTTPServer.py", line 323, in handle_one_request
method()
File "/usr/lib/python2.6/SimpleXMLRPCServer.py", line 487, in do_POST
self.send_response(200)
File "/usr/lib/python2.6/BaseHTTPServer.py", line 384, in send_response
self.send_header('Server', self.version_string())
File "/usr/lib/python2.6/BaseHTTPServer.py", line 390, in send_header
self.wfile.write("%s: %s\r\n" % (keyword, value))
File "/usr/lib/python2.6/socket.py", line 300, in write
self.flush()
File "/usr/lib/python2.6/socket.py", line 286, in flush
self._sock.sendall(buffer)
error: [Errno 32] Broken pipe
[xmlrpc][ERROR] 2012-02-20 12:45:51,803: Traceback (most recent call last):
File "/usr/lib/python2.6/SocketServer.py", line 558, in process_request_thread
self.finish_request(request, client_address)
File "/usr/lib/python2.6/SocketServer.py", line 320, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "/usr/lib/python2.6/SocketServer.py", line 615, in __init__
self.handle()
File "/usr/lib/python2.6/BaseHTTPServer.py", line 329, in handle
self.handle_one_request()
File "/usr/lib/python2.6/BaseHTTPServer.py", line 323, in handle_one_request
method()
File "/usr/lib/python2.6/SimpleXMLRPCServer.py", line 487, in do_POST
self.send_response(200)
File "/usr/lib/python2.6/BaseHTTPServer.py", line 384, in send_response
self.send_header('Server', self.version_string())
File "/usr/lib/python2.6/BaseHTTPServer.py", line 390, in send_header
self.wfile.write("%s: %s\r\n" % (keyword, value))
File "/usr/lib/python2.6/socket.py", line 300, in write
self.flush()
File "/usr/lib/python2.6/socket.py", line 286, in flush
self._sock.sendall(buffer)
error: [Errno 32] Broken pipe
[xmlrpc][ERROR] 2012-02-20 12:45:51,821: Traceback (most recent call last):
File "/usr/lib/python2.6/SocketServer.py", line 558, in process_request_thread
self.finish_request(request, client_address)
File "/usr/lib/python2.6/SocketServer.py", line 320, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "/usr/lib/python2.6/SocketServer.py", line 615, in __init__
self.handle()
File "/usr/lib/python2.6/BaseHTTPServer.py", line 329, in handle
self.handle_one_request()
File "/usr/lib/python2.6/BaseHTTPServer.py", line 323, in handle_one_request
method()
File "/usr/lib/python2.6/SimpleXMLRPCServer.py", line 487, in do_POST
self.send_response(200)
File "/usr/lib/python2.6/BaseHTTPServer.py", line 384, in send_response
self.send_header('Server', self.version_string())
File "/usr/lib/python2.6/BaseHTTPServer.py", line 390, in send_header
self.wfile.write("%s: %s\r\n" % (keyword, value))
File "/usr/lib/python2.6/socket.py", line 300, in write
self.flush()
File "/usr/lib/python2.6/socket.py", line 286, in flush
self._sock.sendall(buffer)
error: [Errno 32] Broken pipe
[xmlrpc][ERROR] 2012-02-20 12:45:51,821: Traceback (most recent call last):
File "/usr/lib/python2.6/SocketServer.py", line 558, in process_request_thread
self.finish_request(request, client_address)
File "/usr/lib/python2.6/SocketServer.py", line 320, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "/usr/lib/python2.6/SocketServer.py", line 615, in __init__
self.handle()
File "/usr/lib/python2.6/BaseHTTPServer.py", line 329, in handle
self.handle_one_request()
File "/usr/lib/python2.6/BaseHTTPServer.py", line 323, in handle_one_request
method()
File "/usr/lib/python2.6/SimpleXMLRPCServer.py", line 487, in do_POST
self.send_response(200)
File "/usr/lib/python2.6/BaseHTTPServer.py", line 384, in send_response
self.send_header('Server', self.version_string())
File "/usr/lib/python2.6/BaseHTTPServer.py", line 390, in send_header
self.wfile.write("%s: %s\r\n" % (keyword, value))
File "/usr/lib/python2.6/socket.py", line 300, in write
self.flush()
File "/usr/lib/python2.6/socket.py", line 286, in flush
self._sock.sendall(buffer)
error: [Errno 32] Broken pipe
[xmlrpc][ERROR] 2012-02-20 12:45:51,827: Traceback (most recent call last):
File "/usr/lib/python2.6/SocketServer.py", line 558, in process_request_thread
self.finish_request(request, client_address)
File "/usr/lib/python2.6/SocketServer.py", line 320, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "/usr/lib/python2.6/SocketServer.py", line 615, in __init__
self.handle()
File "/usr/lib/python2.6/BaseHTTPServer.py", line 329, in handle
self.handle_one_request()
File "/usr/lib/python2.6/BaseHTTPServer.py", line 323, in handle_one_request
method()
File "/usr/lib/python2.6/SimpleXMLRPCServer.py", line 487, in do_POST
self.send_response(200)
File "/usr/lib/python2.6/BaseHTTPServer.py", line 384, in send_response
self.send_header('Server', self.version_string())
File "/usr/lib/python2.6/BaseHTTPServer.py", line 390, in send_header
self.wfile.write("%s: %s\r\n" % (keyword, value))
File "/usr/lib/python2.6/socket.py", line 300, in write
self.flush()
File "/usr/lib/python2.6/socket.py", line 286, in flush
self._sock.sendall(buffer)
error: [Errno 32] Broken pipe
[xmlrpc][ERROR] 2012-02-20 12:45:51,828: Traceback (most recent call last):
File "/usr/lib/python2.6/SocketServer.py", line 558, in process_request_thread
self.finish_request(request, client_address)
File "/usr/lib/python2.6/SocketServer.py", line 320, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "/usr/lib/python2.6/SocketServer.py", line 615, in __init__
self.handle()
File "/usr/lib/python2.6/BaseHTTPServer.py", line 329, in handle
self.handle_one_request()
File "/usr/lib/python2.6/BaseHTTPServer.py", line 323, in handle_one_request
method()
File "/usr/lib/python2.6/SimpleXMLRPCServer.py", line 487, in do_POST
self.send_response(200)
File "/usr/lib/python2.6/BaseHTTPServer.py", line 384, in send_response
self.send_header('Server', self.version_string())
File "/usr/lib/python2.6/BaseHTTPServer.py", line 390, in send_header
self.wfile.write("%s: %s\r\n" % (keyword, value))
File "/usr/lib/python2.6/socket.py", line 300, in write
self.flush()
File "/usr/lib/python2.6/socket.py", line 286, in flush
self._sock.sendall(buffer)
error: [Errno 32] Broken pipe
[rospy.impl.masterslave][INFO] 2012-02-20 12:45:51,867: signal-15
[rospy.core][INFO] 2012-02-20 12:45:52,450: signal_shutdown [atexit]
Asked by Victor Lopez on 2012-02-19 23:37:47 UTC
Comments
I'd love to be able to dig into this. Previous reports have not been able to provide reproducible test cases. Do you think you have something that is easy to reproduce? Furthermore, can you reproduce this w/o rosbridge running?
Asked by kwc on 2012-02-20 08:41:48 UTC
I'd second seeing if you can get this error sans rosbridge. Being so thread heavy and so severely abusive of rospy, it's a likely culprit. This is potentially related to the clients disconnecting without unsubscribing memory leak you spotted :). A fix for one may fix the other. I'll look into it.
Asked by tjay on 2012-02-20 10:53:39 UTC
I have not been able to determine the steps to reproduce it with or without rosbridge, but a coworker had this same issue without running rosbridge at all. He detected it because robotmonitor was using 99% CPU on top and lsof showed that it had hundreds of CLOSE-WAIT connections to localhost.
Asked by Victor Lopez on 2012-02-20 21:19:13 UTC
Can you at least post a the log files from the robot monitor node? Really can't help unless there's more to work with.
Asked by kwc on 2012-02-21 07:59:44 UTC
I've added the log output of robot_monitor. I had some errors attaching it so I had to copy it to the question.
Asked by Victor Lopez on 2012-02-29 22:34:36 UTC
Thanks, the log is helpful. Can you try the patch here: https://code.ros.org/trac/ros/changeset/16431 . It's done against Fuerte, but I believe it will work against diamondback as well. I do recommend moving to Electric, as this patch does not address the root (unknown) cause.
Asked by kwc on 2012-03-01 07:52:29 UTC
We're currently migrating to electric, but I will try this in my diamondback installation and see if I stop getting this issue. The problem is that since it's so random it might be a long time before I'm sure it is not happening.
Asked by Victor Lopez on 2012-03-01 21:59:03 UTC
I believe I've run into a similar issue which I was able to solve. It appears to be a symptom of multiple instances of rosout running and competing. They constantly restart in a race condition and thus create many rosout connections before they can be torn down.
Asked by Asomerville on 2012-12-12 14:33:57 UTC