ROS Resources: Documentation | Support | Discussion Forum | Index | Service Status | ros @ Robotics Stack Exchange
Ask Your Question
3

ROS2-foxy nodes can't communicate through docker container border

asked 2021-01-29 06:48:51 -0500

wl gravatar image

updated 2021-02-01 06:31:32 -0500

I'm new to ROS2. I've got a somewhat weird setup that I'm trying to launch. My issue is that minimal publisher and subscriber from examples don't hear each other.

I've got a docker container, based on Ubuntu 18.04, with ROS2 Foxy installed from sources. This container has installed environment that I have to use, namely, CUDA 10.0 and some software using it.

This container is run on Ubuntu 20.04 host. This host has also installed ROS2 Foxy from Debian packages.

I run the container with --net host, then launch minimal subscriber from examples in it. Then I run minimal publisher on the host.

That is on host:

$ docker run --rm -it --net host container

In container:

# ros2 run examples_rclpy_minimal_subscriber subscriber_member_function

On host in another terminal:

$ source /opt/ros/foxy/setup.zsh
$ ros2 run examples_rclpy_minimal_publisher publisher_local_function

The issue is that the subscriber doesn't hear anything from the publisher. The same issue is observed, if I swap subscriber and publisher, that is run publisher in the container and subscriber on the host.

If I execute both subscriber and publisher in the container, subscriber hears it. On host:

docker exec -it container /bin/bash

In container:

ros2 run examples_rclpy_minimal_publisher publisher_local_function

And subscriber prints "I heard hello".

At the same time, multicast messages (ros2 multicast send, ros2 multicast receive) are sent and received successfully.

If I run the container without --net key, then the subscriber running in it, sees messages from the publisher. However, this limits usage of the container. It won't be seen from another PC.

I've pasted the minimal Dockerfile illustrating the issue at the bottom of this message (unfortunately, I cannot attach it).

I've also pushed image to the dockerhub, and it can be downloaded with the command docker pull wl2776/nvidia-ros:latest

What I've also checked.

  1. Running netcat both in container and on host shows that the traffic is passed through the container boundary. If I replace subsriber in the container with netcat it prints several packets. That is, on host:

    $ docker run --rm -it --net host container
    $ docker cp /bin/nc.openbsd container:/tmp
    

    In container:

    # /tmp/nc.openbsd -l -u -p 7400 | od -
    

    On host in another terminal:

    $ source /opt/ros/foxy/setup.zsh
    $ ros2 run examples_rclpy_minimal_publisher publisher_local_function
    

    And I see several dumps from nc

  2. I've also tried the experiment with minimal publisher and subscriber using docker image tagged osrf/ros:foxy-desktop from dockerhub. Everything has worked perfectly.

  3. When minimal subscriber runs in the container, its node and topic are seen on the host. On host, while subscriber in the container is running:

    $ ros2 topic list
    /parameter_events
    /rosout
    /topic
    $ ros2 node list
    /minimal_subscriber
    

    If I stop minimal subsriber in the container with Ctrl-C, then it disappears from ros2 node list

  4. Sniffing packets with tcpdump shows that subscriber in OSRF's container sends RTPS packets with version 2.3, but mine - with version 2.2. They also have different sizes (252 bytes from OSRF's ...

(more)
edit retag flag offensive close merge delete

Comments

I tried to reproduce on top of an 18.04 docker container, but the examples work for me (publishing from host and subscribing from inside the Docker container). Here's the Dockerfile I used.

jacobperron gravatar image jacobperron  ( 2021-02-04 18:36:22 -0500 )edit

I've built an image from your Dockerfile. Still the same issue. Did you use --net host? It is important.

Everything works without --net host, but fails, if network isolation is turned off.

wl gravatar image wl  ( 2021-02-05 10:10:41 -0500 )edit
1

Did you find a solution or could you isolate the problem? I have the same issue. Without --net host the communication works, but of course only isolated and not together with another computer in the network.

mschratter gravatar image mschratter  ( 2021-02-11 07:53:24 -0500 )edit

I think, the issue is in FastRTPS. If at least one of nodes (host or containerized, no matter) switches to CycloneDDS, the "hello" messages become heard. Here is the link to discussion on Github: https://github.com/eProsima/Fast-DDS/... They have deleted my comment that everything works, if I switch to CycloneDDS.

wl gravatar image wl  ( 2021-02-11 07:59:47 -0500 )edit

4 Answers

Sort by ยป oldest newest most voted
9

answered 2021-02-12 02:37:31 -0500

richiware gravatar image

The last releases of Fast-DDS come with SharedMemory transport by default. Using --net=host implies both DDS participants believe they are in the same machine and they try to communicate using SharedMemory instead of UDP. Fast-DDS team will work in to implement a mechanism to detect this kind of situation. Meanwhile, I can give you two solutions:

  • Using a XML to disable SharedMemory transport in one of the DDS participants.

XML:

<?xml version="1.0" encoding="UTF-8" ?>
<profiles xmlns="http://www.eprosima.com/XMLSchemas/fastRTPS_Profiles" >
    <transport_descriptors>
        <transport_descriptor>
            <transport_id>CustomUdpTransport</transport_id>
            <type>UDPv4</type>
        </transport_descriptor>
    </transport_descriptors>

    <participant profile_name="participant_profile" is_default_profile="true">
        <rtps>
            <userTransports>
                <transport_id>CustomUdpTransport</transport_id>
            </userTransports>

            <useBuiltinTransports>false</useBuiltinTransports>
        </rtps>
    </participant>
</profiles>
  • Enable SharedMemory between host and container. For this you should share /dev/shm:
    docker run -ti --net host -v /dev/shm:/dev/shm <DOCKER_IMAGE>
    Also, both applications should be run with the same UID. In my case, my docker container's user is root (UID=0). Then I had to run the host application as root.

I hope this helps you.

edit flag offensive delete link more

Comments

@richiware. So by using XML way can we avoid using --net=host . Because using host network for containers will compromise lot of security.

BhanuKiran.Chaluvadi gravatar image BhanuKiran.Chaluvadi  ( 2022-01-06 03:35:39 -0500 )edit

I had the same issue between two docker containers. Mounting the shared memory on both with -v /dev/shm:/dev/shm solved the problem. Thanks

Markus Bader gravatar image Markus Bader  ( 2022-09-20 09:01:23 -0500 )edit
3

answered 2021-02-12 03:07:27 -0500

wl gravatar image

As described here, "the last releases of Fast-DDS come with SharedMemory transport by default. Using --net=host implies both DDS participants believe they are in the same machine and they try to communicate using SharedMemory instead of UDP".

Can be solved with additional XML config, disabling shared memory:

  1. $ cat > fastrtps-profile.xml and paste XML from the link above
  2. $ export FASTRTPS_DEFAULT_PROFILES_FILE=$(pwd)/fastrtps-profile.xml
edit flag offensive delete link more
0

answered 2021-02-17 15:48:43 -0500

ruffsl gravatar image

If you'd like to use shared memory transport between node in separate containers, then you could adjust the PID (Process) Namespace mode for the containers as explained in this answer from a previous question:

"ROS2 connectivity across Docker containers via Host Driver"
https://answers.ros.org/question/2968...

Perhaps @richiware might know how to set a different RTPS App ID for each DDS participant to avoid the confusion of duplicate IDs from separate PID spaces on a shared host, or what advantage manually mounting the /dev/shm volume would have over using the pid and ipc options in docker.

edit flag offensive delete link more
0

answered 2022-08-10 03:14:00 -0500

maxpol gravatar image

The problem still exists on humble. Mapping /dev/shm in one of the containers resolves it. I conclude that the nodes indeed use shm transport which is actually nice in my case.

edit flag offensive delete link more

Question Tools

2 followers

Stats

Asked: 2021-01-29 06:14:09 -0500

Seen: 7,857 times

Last updated: Feb 17 '21