# How to properly set up a remote roscore with multiple remote nodes?

I am very new to ros. My friend and I have a scenario where only one of us has the hardware, but we'd both like to try to get live stream data without having to replay the data. Ideally we want to use ros2, but since the networking TSG on that is not out with no ETA, we are sticking with ros1 for the time being (might do some ros2->ros1 translation in the long run) So we are trying to get a remote roscore running and connect our nodes to this. So far, I have been able to get this semi-working, but am noticing some odd behavior:

We have a remote server running our roscore. Let's pretend it's public IP is 1.2.3.4 with a local ip of 10.0.0.1 the ROS settings on the server are as follows:

ROS_MASTER_URI=http://10.0.0.1:12345
ROS_IP=10.0.0.1


Remote node setup:

ROS_MASTER_URI=http://1.2.3.4:12345
ROS_IP=0.0.0.0


With this setup I am able to run turtlesim and control it on my local computer, with roscore running on the server with an interesting side effect: They only work if the publisher is created before the subscriber (in this case, turtle_teleop_key needs to be running before turtlesim_node). Does anyone have a reason for this side effect?

Secondly: When I introduce another local computer, I am able to get the sim running the same as above (publisher must register before subscriber) purely on the local computer, however the two local computers can never talk to each other no matter which way I connect the nodes (which is our required scenario).

None of this behavior makes sense to me unless roscore is not handling all the communication and is instead communicating to nodes to open connections with other nodes. If roscore handles the communication itself - it doesn't make sense to me that I'd be able to get the connection working at all even with both publisher and subscriber on one local computer (with roscore as remote). tcpdump sort of confirms this because when I am running the two nodes on one local computer I do not seem to be seeing packets from remote server at the rate I'd expect (seems to be more of a ping or something).

If roscore is communicating to the nodes to open connections with each other, it might make more sense that there's some port forwarding issues or something like that.

Anyways, I have seen a ton of posts on this, but the way I got to a semi-working example was by combining multiple answers from multiple posts - so it'd be nice to get a full answer

I have seen that some people say they need to edit /etc/hosts -> Is this actually required? I have seen several posts say that ROS_IP should be your local IP (like 192.168.0.2 or something), others said it ...

edit retag close merge delete

Sort by » oldest newest most voted

Ideally we want to use ros2, but since the networking TSG on that is not out with no ETA,

What do you mean by this? What is a TSG?

None of this behavior makes sense to me unless roscore is not handling all the communication and is instead communicating to nodes to open connections with other nodes.

Yes, that is indeed what the function of the ROS Master is. It is not a broker, buffer or relay station, but only a coordinator (it's been likened to a DNS server sometimes).

You may want to review some documentation on its purpose and implementation: wiki/Master.

Anyways, I have seen a ton of posts on this, but the way I got to a semi-working example was by combining multiple answers from multiple posts - so it'd be nice to get a full answer

To get it out of the way (and seeing as you don't mention those pages): have you seen wiki/ROS/NetworkSetup, specifically ROS/Tutorials/MultipleMachines and the linked troubleshooting pages?

They should go into some detail about how to do all this.

I have seen that some people say they need to edit /etc/hosts -> Is this actually required? I have seen several posts say that ROS_IP should be your local IP (like 192.168.0.2 or something), others said it should be 0.0.0.0. I only got this semi-working by using 0.0.0.0. Is this the the correct setting?

0.0.0.0 is almost certainly not a valid value for ROS_IP. ROS_IP is used by ROS nodes to tell others at which IP:PORT combo they can be reached. For host0, 0.0.0.0 is a local address. For host1, 0.0.0.0 is also a local address. It should be clear why that will not work.

Set ROS_IP to the IP which is accessible to all other (remote) ROS nodes. For a simple LAN setup, that would be the IP of the NIC connected to the LAN on which the other ROS hosts are present.

Set ROS_MASTER_URI to the IP:PORT combo at which the master is reachable.

Note: if remote means "reachable over the (public) internet", and you have a typical consumer grade internet connection with something like a NAT box between you and the "remote", things will become more difficult, as you now have to deal with NAT as well.

If that's the case, I'd recommend using a (peer-to-peer) VPN to create a virtual network between you and your friend, as that will significantly reduce complexity. I've used tinc with good results, but any VPN which is not too difficult to set up should be OK. If/when you have that setup, you'll likely want to use the IP addresses in the VPN's range, instead of the ones you have configured now.

Anyways thanks for any help, really enjoying it so far, though I feel like I've been ...

more

Sorry TSG = troubleshooting guide. Yeah that's fair. Thanks for the input! That helps.. So I suppose, ROS_IP should be the public IP address of the node, though this would be very inefficient for any nodes that are on the same network. Yeah I suppose most people do single host stuff. I think I'll just write a ros2 module to relay traffic to other networks so I can still use ros2. Should be a good learning experience anyways.

( 2020-06-30 21:56:21 -0500 )edit

So I suppose, ROS_IP should be the public IP address of the node, though this would be very inefficient for any nodes that are on the same network.

I'm not sure I understand what the problem is you seem to be seeing?

I think I'll just write a ros2 module to relay traffic to other networks so I can still use ros2. Should be a good learning experience anyways.

Also not sure why you'd want to do this, but if you're happy, we're happy.

( 2020-07-01 02:39:03 -0500 )edit

If the nodes connect via public IP on the same network, the traffic needs to travel to the ISP and back instead of just across your local network. Makes sense this is not a common use case so it's not really a problem if it works.

My friend owns the 3d camera we are using for the project and he lives in a different country, If I want to process the data we need to send it across networks.

Thanks a bunch for your responses!

( 2020-07-01 10:38:29 -0500 )edit

My friend owns the 3d camera we are using for the project and he lives in a different country, If I want to process the data we need to send it across networks.

unless you absolutely need live data, I would suggest to look at using rosbag.

( 2020-07-01 11:55:14 -0500 )edit

Awesome! I'll look into that, thanks! I've already written the tool I was thinking of, but rosbag looks like it'll be really helpful as well.

The package is here if anyone cares to use it: https://github.com/tlagore/ros2relay

( 2020-07-01 17:42:03 -0500 )edit

In the end this looks like something a VPN would have solved quite nicely :)

And with ROS 2 you should be able to "just" configure the DDS implementation to join a remote network.

But you wrote you wanted to learn, so then I guess this is OK.

( 2020-07-02 02:47:51 -0500 )edit