Ask Your Question

ROS2 Distributed Nodes Not Communicating

asked 2021-07-09 09:22:26 -0600

RobotDreams gravatar image

updated 2021-07-11 13:07:18 -0600


I have setup two ROS2-Foxy environments with ROS_DOMAIN_ID=0:

1) RaspberryPi 3B+ Ubuntu 20.04 64-bit Server
- - Ubuntu ufw is disabled

2) VMware Fusion running Ubuntu 20.04 64-bit Desktop
- - ROS2 Foxy
- - on Ubuntu 20.04.2 LTS 64-bit Desktop
- - in an 8 GB Ram 3-cores available to VMware Fusion virtual machine
- - on a 32GB 3.2GHz 6-Core Intel Core i7 2018 MacMini
- - running macOS Big Sur 11.0.1
- - Bridged Network Adapter (same if Bridged Ethernet or Bridged WiFi) - - Ubuntu ufw is disabled

  • Each system will successfully communicate (localhost) for
    "T1: ros2 run demo_nodes_cpp talker" and
    "T2: ros2 run demo_nodes_py listener"
    (any combination of cpp/cpp, py/cpp, cpp/py, py/py )

  • Each system is able to ping to the other

  • Both systems ifconfig report


  • Multicast test works one way - to the VM

    ROS-VM: ros2 multicast recieve
    Waiting for UDP multicast datagram...
    Received from 'Hello World!'

    ROSbot: ros2 multicast recieve does not succeed

  • Both systems have ROS_DOMAIN_ID=0

  • Both systems have ROS_LOCALHOST_ONLY=0

BUT when I run a talker on one system and listener on the other, the listener never reports receiving the "Hello World" topic.

On RaspberryPi ROSbot running the talker, "ss -tulw" reports:

pi@ROSbot:~$ ss -tulw
Netid          State            Recv-Q           Send-Q                       Local Address:Port                          Peer Address:Port          Process          
icmp6          UNCONN           0                0                         *%wlan0:ipv6-icmp                                *:*                              
udp            UNCONN           0                0                                   *                              
udp            UNCONN           0                0                                   *                              
udp            UNCONN           0                0                                   *                              
udp            UNCONN           0                0                                *                              
udp            UNCONN           0                0                                  *                              
udp            UNCONN           0                0                           *                              
udp            UNCONN           0                0                         *                              
tcp            LISTEN           0                4096                             *                              
tcp            LISTEN           0                4096                        *                              
tcp            LISTEN           0                128                                  *                              
tcp            LISTEN           0                128                         [::]:ssh                                   [::]:*                              
tcp            LISTEN           0                100                          *:8888                                     *:*

and the ROS2 virtual machine reports:

ubuntu@rosdesk:~$ ss -tulw
Netid  State   Recv-Q  Send-Q                      Local Address:Port              Peer Address:Port  Process  
icmp6  UNCONN  0       0                                       *:ipv6-icmp                    *:*              
udp    UNCONN  0       0                             *              
udp    UNCONN  0       0                               *              
udp    UNCONN  0       0                                *              
udp    UNCONN  0       0                                *              
udp    UNCONN  0       0                                *              
udp    UNCONN  0       0                        *              
udp    UNCONN  0       0                               *              
udp    UNCONN  0       0                                           0.0 ...
edit retag flag offensive close merge delete



Can you try running the multicast test as listed here:

I wonder if there is something in the VM network driver.

vinny gravatar image vinny  ( 2021-07-09 09:33:35 -0600 )edit

multicast worked one-way: to the VM only.

I performed the sudo ufw commands substituting my local net not - correct? No change in send from VM no RX on ROSbot

RobotDreams gravatar image RobotDreams  ( 2021-07-09 09:57:49 -0600 )edit
  • aha, I think multicast should be working both ways on default RMW settings in foxy.
  • That being said, try to modify your firewall for the 224* address like the guide says (there is something special about the 224* addresses
  • You may consider trying a different RMW like cyclone on both hosts (I know this helps when doing things in docker for me)
vinny gravatar image vinny  ( 2021-07-09 12:24:24 -0600 )edit

Is the VM configured with a bridged network adaptor? Or a host-only? Or a NAT one?

gvdhoorn gravatar image gvdhoorn  ( 2021-07-09 13:16:47 -0600 )edit

Wow - hope and then hopes dashed. @gvdhoom - tried both bridged ethernet and bridged WiFi - does not fix.

RobotDreams gravatar image RobotDreams  ( 2021-07-11 13:10:07 -0600 )edit

What should I check on WiFi router? I'm wondering if something there is blocking DDS - it is strange that the multicast test works only one way.

RobotDreams gravatar image RobotDreams  ( 2021-07-11 14:25:00 -0600 )edit

Thought I had it by setting optional: false in /etc/netplan/50-cloud-init.yaml - booted 7 times with ping, ssh, and DDS success. Went to celebrate. Came back and couldn't ssh in first boot. Too much to solve.

RobotDreams gravatar image RobotDreams  ( 2021-07-11 22:34:56 -0600 )edit

1 Answer

Sort by » oldest newest most voted

answered 2021-07-13 08:50:34 -0600

RobotDreams gravatar image

Currently, ROSbot Dave

has successfully booted up with full ssh, ping, arp, and ROS2 dynamic discovery Data Distribution System DDS working - 12 out of 12 attempts
- since the change of 2.4GHz band WiFi access point
- - from the xFinity Technicolor CGM4331COM XB7 router
- - to my ancient DLink DIR-825 (last firmware released was 2013).

While I can say Carl and DeskPi have had fitful connections with the xFinity router(s), neither has shown the “cannot remote in” (ssh) issue that Dave exhibited, nor did I attempt distributed data messaging across the network when Carl was trying out ROS (1).

Carl has always been a Raspberry Pi 3B based GoPiGo3, but DeskPi was in fact using the exact Raspberry Pi 3B+ (plus) board that GoPiGo3 ROSbot Dave is running, except DeskPi has always run Raspbian/PiOS.

All these tests seem to point the finger at Ubuntu 20.04.2 on RaspberryPi 3B+ (or at least this particular 3B+) having unreliable ICMP, ARP, or UDP handling during the DHCP link establishment phase with the xFinity router configured with a reserved IPv4 address for the board’s WiFi MAC address.

I don’t know enough to be able to diagnose this deeper, and having found a solution that enables me to progress with my ROS2 learning (through migrating the “Hands On ROS for Robotic Programming” exercises), I am leaving this mystery of the universe unsolved.

I do have a new WiFi 6 router on order, (that has ongoing firmware update support), to minimize the chance my network will become a crypto-bot or participant in the next distributed denial of service attack on internet democracy.

Calling this success - Onward with ROS2

image description

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Question Tools

1 follower


Asked: 2021-07-09 09:22:26 -0600

Seen: 132 times

Last updated: Jul 13