Remotely connect to docker network
Hi all,
I have a feasibility question about remotely connecting to ROS nodes inside of a single docker container, where that container is part of a docker network. Specifically, from my local machine I want to subscribe to nodes that are running inside of a docker network on a remote machine. The need for the docker network is that multiple developers are using the remote server simultaneously, and their ROS nodes need to be isolated (hence why we are using docker).
When I run the docker container in which I am launching the nodes from, I can map the master's port (11311) to an arbitrary port (e.g. 10023) on the remote host, and then set ROS_MASTER_URI=http://remote_ip:10023 on my local. This allows me to communicate with the master, e.g. do rostopic list. However, the problem arises when I launch a new node in the container, and it is assigned a new random port number in the docker network, one that I have not mapped, and thus isn't visible to the remote host & my local connection.
I cannot think of a way to make this work, hence me asking the community! People have looked into host <-> container communication, e.g. @ruffsl in https://answers.ros.org/question/2282..., but in those solutions each node launched needs to be configured beforehand with it's networking environment variables explicitly set. I want to launch arbitrary nodes in the same container, and be able to subscribe to them outside of the docker network.
Thanks, Rowan
Relevant part of the docker script:
docker network create ${NETWORK_NAME}
docker run --name ${DEV_NAME} \
-t \
-d \
-v "$WS_DIR":/usr/src/catkin_ws \
-p ${DEV_PORT}:8080 \
-p ${ROSBRIDGE_PORT}:9090 \
-p ${VNC_PORT}:5900 \
-p 10023:11311 \ # Map ROS master port to arbitrary host port
--network=${NETWORK_NAME} \
${ADDITIONAL_DEV_FLAGS} \
$IMAGE \
(posting this as a comment, as it's not an answer to the question, but an alternative)
Personally, I would not use Docker's NAT networking for this, but would setup a Docker network which assigns IPs to each container which get bridged into your regular network (note: this is not the regular configuration of a Docker bridged network). Those IPs can be -- with some routing -- accessed like regular hosts on your network.
After you do that, you don't have to use
-pany more (but you could still do it), and can treat the container almost like a regular remote ROS host (ie: setROS_IPetc as if it was just another computer on the network).As the NAT is then out of the way, you don't run into any of the issues with ephemeral ports being used by nodes or other such challenges.
I've done ...(more)
Yep, using port mapping with ROS is a non starter given the use of ephemeral ports by the middleware protocol. As suggested, using software defined networks would be better. If you can be sure to set the
ROS_MASTER_URIto a unique port, as not to collide/cross-talk with other containers on the same host, using thehostnetwork driver would also be an simple alternative to reaching the node remotely.Using IP aliasing on the host may be another alternative, but it still uses port mapping.
https://www.edureka.co/community/1303...
There may be a more ergonomic method of using virtual interfaces via alternative network drivers by now.
I can't find the article(s) I used last time, but you can essentially configure a custom Docker network which takes it's IPs from a pre-allocated range. Then setup static routes for the other hosts on your (real) network and you can access Docker containers like they were real hosts.
This is all doable with standard Linux networking configuration. No need for any other software, nor special NIC drivers. Docker is also not really involved, other than making it use the network config which makes this possible (ie: disable its NAT, etc).
Thanks for your advice guys! I agree that those alternatives make sense, I am a Docker newbie who was given this setup, so I'm all for changing it :). However, a couple of things that might have impact your comments:
@gvdhoorn Assigning an IP to the container running the nodes would be ideal. However the server is part of my University's network, I have no dynamic control over it. I can request IST to put static IPs for specific hardware interfaces, but I can't re-configure the network on the fly.
@ruffsl Using the host network in the containers I think would solve my problem, but it isn't possible under my requirement of providing ROS isolation for multiple users on the server, right? Is it possible to isolate multiple ROS masters running on the same host network?
If you have a willing admin, you could ask them to do this for you: https://serverfault.com/a/958374.
In a nutshell this documents what I describe in my first comment.
Note: Docker purists will probably not like this too much.
that's not how this works. In essence the Docker containers are treated like a VM almost, with their own private IP. Any 'dynamism' would be "behind" the bridge added to the machine running your Docker containers. Only hosts on the regular network with the appropriate routes would be able to route traffic to the containers.
For icing on the cake: setup a Docker aware (reverse) DNS and you can use descriptive hostnames for containers running certain nodes (and ...(more)
This is possible, but in my experience brittle: any user 'forgetting' to map their Master to a free port will cause conflicts. I've had it more often than not that "strange objects" started appearing in Gazebo simulations because someone forgot to configure their
.launchfiles with the correct ports for Gazebo and the Master.Note: I'm not saying default Docker networking with NAT and isolation can't work. What I've noticed however is that it takes a good amount of effort to get it working reliably and -- almost more importantly -- transparently enough for users to not be bothered by it.