ROS Resources: Documentation | Support | Discussion Forum | Index | Service Status | ros @ Robotics Stack Exchange
Ask Your Question
1

ROS2 Distributed Nodes Not Communicating

asked 2021-07-09 09:22:26 -0500

RobotDreams gravatar image

updated 2021-07-11 13:07:18 -0500

ROS2 - FOXY

I have setup two ROS2-Foxy environments with ROS_DOMAIN_ID=0:

1) RaspberryPi 3B+ Ubuntu 20.04 64-bit Server
- - Ubuntu ufw is disabled

2) VMware Fusion running Ubuntu 20.04 64-bit Desktop
- - ROS2 Foxy
- - on Ubuntu 20.04.2 LTS 64-bit Desktop
- - in an 8 GB Ram 3-cores available to VMware Fusion virtual machine
- - on a 32GB 3.2GHz 6-Core Intel Core i7 2018 MacMini
- - running macOS Big Sur 11.0.1
- - Bridged Network Adapter (same if Bridged Ethernet or Bridged WiFi) - - Ubuntu ufw is disabled

  • Each system will successfully communicate (localhost) for
    "T1: ros2 run demo_nodes_cpp talker" and
    "T2: ros2 run demo_nodes_py listener"
    (any combination of cpp/cpp, py/cpp, cpp/py, py/py )

  • Each system is able to ping to the other

  • Both systems ifconfig report

    UP,BROADCAST,RUNNING,MULTICAST

  • Multicast test works one way - to the VM

    ROS-VM: ros2 multicast recieve
    Waiting for UDP multicast datagram...
    Received from 10.0.0.11:43044: 'Hello World!'

    ROSbot: ros2 multicast recieve does not succeed

  • Both systems have ROS_DOMAIN_ID=0

  • Both systems have ROS_LOCALHOST_ONLY=0

BUT when I run a talker on one system and listener on the other, the listener never reports receiving the "Hello World" topic.

On RaspberryPi ROSbot running the talker, "ss -tulw" reports:

pi@ROSbot:~$ ss -tulw
Netid          State            Recv-Q           Send-Q                       Local Address:Port                          Peer Address:Port          Process          
icmp6          UNCONN           0                0                         *%wlan0:ipv6-icmp                                *:*                              
udp            UNCONN           0                0                        0.0.0.0:7400                               0.0.0.0:*                              
udp            UNCONN           0                0                        0.0.0.0:7412                               0.0.0.0:*                              
udp            UNCONN           0                0                        0.0.0.0:7413                               0.0.0.0:*                              
udp            UNCONN           0                0                      127.0.0.1:58641                              0.0.0.0:*                              
udp            UNCONN           0                0                        0.0.0.0:34723                              0.0.0.0:*                              
udp            UNCONN           0                0                  127.0.0.53%lo:domain                             0.0.0.0:*                              
udp            UNCONN           0                0                10.0.0.11%wlan0:bootpc                             0.0.0.0:*                              
tcp            LISTEN           0                4096                   127.0.0.1:45585                              0.0.0.0:*                              
tcp            LISTEN           0                4096               127.0.0.53%lo:domain                             0.0.0.0:*                              
tcp            LISTEN           0                128                      0.0.0.0:ssh                                0.0.0.0:*                              
tcp            LISTEN           0                128                         [::]:ssh                                   [::]:*                              
tcp            LISTEN           0                100                          *:8888                                     *:*

and the ROS2 virtual machine reports:

ubuntu@rosdesk:~$ ss -tulw
Netid  State   Recv-Q  Send-Q                      Local Address:Port              Peer Address:Port  Process  
icmp6  UNCONN  0       0                                       *:ipv6-icmp                    *:*              
udp    UNCONN  0       0                               127.0.0.1:39592                  0.0.0.0:*              
udp    UNCONN  0       0                                 0.0.0.0:60176                  0.0.0.0:*              
udp    UNCONN  0       0                                 0.0.0.0:7400                   0.0.0.0:*              
udp    UNCONN  0       0                                 0.0.0.0:7412                   0.0.0.0:*              
udp    UNCONN  0       0                                 0.0.0.0:7413                   0.0.0.0:*              
udp    UNCONN  0       0                           127.0.0.53%lo:domain                 0.0.0.0:*              
udp    UNCONN  0       0                                 0.0.0.0:37045                  0.0.0.0:*              
udp    UNCONN  0       0                                 0.0.0.0:631                    0.0 ...
(more)
edit retag flag offensive close merge delete

Comments

2

Can you try running the multicast test as listed here: https://docs.ros.org/en/foxy/Guides/I...

I wonder if there is something in the VM network driver.

vinny gravatar image vinny  ( 2021-07-09 09:33:35 -0500 )edit

multicast worked one-way: to the VM only.

I performed the sudo ufw commands substituting my local net 10.0.0.0/4 not 224.0.0.0/4 - correct? No change in send from VM no RX on ROSbot

RobotDreams gravatar image RobotDreams  ( 2021-07-09 09:57:49 -0500 )edit
1
  • aha, I think multicast should be working both ways on default RMW settings in foxy.
  • That being said, try to modify your firewall for the 224* address like the guide says (there is something special about the 224* addresses https://superuser.com/a/441730)
  • You may consider trying a different RMW like cyclone on both hosts https://github.com/ros2/rmw_cyclonedd... (I know this helps when doing things in docker for me)
vinny gravatar image vinny  ( 2021-07-09 12:24:24 -0500 )edit
1

Is the VM configured with a bridged network adaptor? Or a host-only? Or a NAT one?

gvdhoorn gravatar image gvdhoorn  ( 2021-07-09 13:16:47 -0500 )edit

Wow - hope and then hopes dashed. @gvdhoom - tried both bridged ethernet and bridged WiFi - does not fix.

RobotDreams gravatar image RobotDreams  ( 2021-07-11 13:10:07 -0500 )edit

What should I check on WiFi router? I'm wondering if something there is blocking DDS - it is strange that the multicast test works only one way.

RobotDreams gravatar image RobotDreams  ( 2021-07-11 14:25:00 -0500 )edit

Thought I had it by setting optional: false in /etc/netplan/50-cloud-init.yaml - booted 7 times with ping, ssh, and DDS success. Went to celebrate. Came back and couldn't ssh in first boot. Too much to solve.

RobotDreams gravatar image RobotDreams  ( 2021-07-11 22:34:56 -0500 )edit

1 Answer

Sort by » oldest newest most voted
0

answered 2021-07-13 08:50:34 -0500

RobotDreams gravatar image

Currently, ROSbot Dave

has successfully booted up with full ssh, ping, arp, and ROS2 dynamic discovery Data Distribution System DDS working - 12 out of 12 attempts
- since the change of 2.4GHz band WiFi access point
- - from the xFinity Technicolor CGM4331COM XB7 router
- - to my ancient DLink DIR-825 (last firmware released was 2013).

While I can say Carl and DeskPi have had fitful connections with the xFinity router(s), neither has shown the “cannot remote in” (ssh) issue that Dave exhibited, nor did I attempt distributed data messaging across the network when Carl was trying out ROS (1).

Carl has always been a Raspberry Pi 3B based GoPiGo3, but DeskPi was in fact using the exact Raspberry Pi 3B+ (plus) board that GoPiGo3 ROSbot Dave is running, except DeskPi has always run Raspbian/PiOS.

All these tests seem to point the finger at Ubuntu 20.04.2 on RaspberryPi 3B+ (or at least this particular 3B+) having unreliable ICMP, ARP, or UDP handling during the DHCP link establishment phase with the xFinity router configured with a reserved IPv4 address for the board’s WiFi MAC address.

I don’t know enough to be able to diagnose this deeper, and having found a solution that enables me to progress with my ROS2 learning (through migrating the “Hands On ROS for Robotic Programming” exercises), I am leaving this mystery of the universe unsolved.

I do have a new WiFi 6 router on order, (that has ongoing firmware update support), to minimize the chance my network will become a crypto-bot or participant in the next distributed denial of service attack on internet democracy.

Calling this success - Onward with ROS2

image description

edit flag offensive delete link more

Question Tools

1 follower

Stats

Asked: 2021-07-09 09:22:26 -0500

Seen: 2,570 times

Last updated: Jul 13 '21