Hi All

Our setup: Ros melodic 1.14.5 , running in ubuntu bionic 18.04.1 LTS, apache2 2.4.29, rosbridgeserver 0.11.3 (working), rosbridgeserver 0.11.4-0.11.5 (not working)

We currently implement rosbridge_server to serve a couple of websockets that we use to connect the front end of our application to the ROS backend. We use apache as a reverse proxy to route the requests from an specific outer endpoint to our locally served websockets (e.g. ProxyPass /api ws://localhost:9090).

An example of our apache2 setup is seen below

<VirtualHost *:80>
  ProxyPreserveHost On
  ProxyRequests Off
  <Location /api>
    ProxyPass ws://localhost:9090 retry=0 keepalive=On
  </Location>
</VirtualHost>

We realized that upon updating rosbridgeserver to 0.11.4, which changes the websocket backend in python to use the Autobahn library, our apache proxying stopped working and we could not get access to the web socket from the outer world. The interesting thing is that the websocket itself works correctly and we can have access to it if we connect directly, bypassing apache (e.g can connect to ws://<IPOF_MACHINE>:9090).

Our launch scripts for the rosbridge websocket can be seen below

<launch>

  <include ns="customer" file="$(find rosbridge_server)/launch/rosbridge_websocket.launch">
    <!-- use default port -->
    <arg name="port" value="9090" />
    <arg name="topics_glob" default="[*]" />
    <arg name="services_glob" default="[*]" />
    <arg name="params_glob" default="[]" />
  </include>

</launch>

We have been throwing arrows in the dark for a while trying to see what has changed. One thing we have noticed is that with rosbridgeserver 0.11.3, launching a rosbridgewebsocket node binds a python process to IPV4 and IPV6 addresses, as seen by checking with the command below

user@webvenice: : sudo ss -lptn 'sport = :9090'

State             Recv-Q             Send-Q                           Local Address:Port                           Peer Address:Port                                                             
LISTEN            0                  128                                    0.0.0.0:9090                                0.0.0.0:*                 users:(("python",pid=24777,fd=14))             
LISTEN            0                  128                                       [::]:9090                                   [::]:*                 users:(("python",pid=24777,fd=15))  

In contrast, when testing with 0.11.4 or 0.11.5 we see that only that there is a single python process binding IPV4 addresses

user@webvenice: sudo ss -lptn 'sport = :9090'

 State             Recv-Q        Send-Q                           Local Address:Port                  
Peer Address:Port                      LISTEN            0                  50                        
0.0.0.0:9090                                0.0.0.0:*                 users:(("python",pid=24777,fd=14))

We are sure that apache2 works and routes correctly the outer request but when routing to the local network it has trouble, as seen in the logs below

[Mon Apr 20 14:11:38.952259 2020] [proxy:error] [pid 601:tid 140657578448640] (111)Connection refused: AH00957: WS: attempt to connect to 127.0.0.1:9090 (localhost) failed 
[Mon Apr 20 14:11:38.952292 2020] [proxy_wstunnel:error] [pid 601:tid 140657578448640] [client ::1:62928] AH02452: failed to make connection to backend: localhost 
[Mon Apr 20 14:11:39.960145 2020] [proxy:error] [pid 601:tid 140657477801728] (111)Connection refused: AH00957: WS: attempt to connect to 127.0.0.1:9090 (localhost) failed 
[Mon Apr 20 14:11:39.960176 2020] [proxy_wstunnel:error] [pid 601:tid 140657477801728] [client ::1:62974] AH02452: failed to make connection to backend: localhost 
[Mon Apr 20 14:11:40.961435 2020] [proxy:error] [pid 602:tid 140656999646976] (111)Connection refused: AH00957: WS: attempt to connect to 127.0.0.1:9090 (localhost) failed 
[Mon Apr 20 14:11:40.961465 2020] [proxy_wstunnel:error] [pid 602:tid 140656999646976] [client ::1:63024] AH02452: failed to make connection to backend: localhost 
[Mon Apr 20 14:11:41.963662 2020] [proxy:error] [pid 601:tid 140657469409024] (111)Connection refused: AH00957: WS: attempt to connect to 127.0.0.1:9090 (localhost) failed 
[Mon Apr 20 14:11:41.963686 2020] [proxy_wstunnel:error] [pid 601:tid 140657469409024] [client ::1:63062] AH02452: failed to make connection to backend: localhost 
[Mon Apr 20 14:11:42.964781 2020] [proxy:error] [pid 601:tid 140658559936256] (111)Connection refused: AH00957: WS: attempt to connect to 127.0.0.1:9090 (localhost) failed 
[Mon Apr 20 14:11:42.964809 2020] [proxy_wstunnel:error] [pid 601:tid 140658559936256] [client ::1:63108] AH02452: failed to make connection to backend: localhost 
[Mon Apr 20 14:11:43.970101 2020] [proxy:error] [pid 602:tid 140658408933120] (111)Connection refused: AH00957: WS: attempt to connect to 127.0.0.1:9090 (localhost) failed 
[Mon Apr 20 14:11:43.970128 2020] [proxy_wstunnel:error] [pid 602:tid 140658408933120] [client ::1:63128] AH02452: failed to make connection to backend: localhost 
[Mon Apr 20 14:11:44.978513 2020] [proxy:error] [pid 601:tid 140658543150848] (111)Connection refused: AH00957: WS: attempt to connect to 127.0.0.1:9090 (localhost) failed 
[Mon Apr 20 14:11:44.978543 2020] [proxy_wstunnel:error] [pid 601:tid 140658543150848] [client ::1:63168] AH02452: failed to make connection to backend: localhost 
[Mon Apr 20 14:11:45.988754 2020] [proxy:error] [pid 602:tid 140658559936256] (111)Connection refused: AH00957: WS: attempt to connect to 127.0.0.1:9090 (localhost) failed 
[Mon Apr 20 14:11:45.988777 2020][proxy_wstunnel:error] [pid 602:tid140658559936256] [client ::1:63258] AH 02452: failed to make connection to backend: localhost

We have tried to tweak our apache2 setup with different configurations (Proxy allow all, ProxyPassReverse, instead of binding the ws socket to localhost to 0.0.0.0 or 127.0.1, etc) and also tried to tweak the laucnh script of the websockets (changing the address explicitly to 0.0.0.0 or 127.0.0.1, changing ports,etc) but nothing works with rosbridge_server 0.11.4 or 0.11.5

Does anybody have experience with this kind of setup that could give us a hand? I am also posting this in the rosbridge_server github to see if they can help

Thanks in advance