ROS Resources: Documentation | Support | Discussion Forum | Index | Service Status | ros @ Robotics Stack Exchange
Ask Your Question

Using roslaunch to run distributed systems over ssh

asked 2011-12-22 03:00:54 -0500

tom gravatar image

updated 2011-12-27 21:02:10 -0500

I'm just trying to make my distributed ROS system run with one single command for the first time. I went through roslaunch/Architecture and roslaunch tips for larger projects which helped me a lot, but not enough.

My system is disributed over 2 machines, say Alpha and Beta, the second being localhost (authorized_keys defined, hostnames checked, communication works flawlessly). So my launchfile looks like this:

  <machine name="alpha" address="alpha" user="alpha_user" 
    ros-package-path="/very/true/package/path" default="true" />
  <machine name="beta" address="beta" user="beta_user" 
    ros-root="$(env ROS_ROOT)" 
    ros-package-path="$(env ROS_PACKAGE_PATH)" default="false" />

  <node pkg="hokuyo_node" type="hokuyo_node" name="hokuyo_node" output="screen">
    <param name="frame_id" value="/laser"/>
    <param name="port" value="/dev/ttyACM0"/>

  <node pkg="laser_scan_matcher" type="laser_scan_matcher_node"
    name="laser_scan_matcher_node" output="screen">

  <node pkg="tf" type="static_transform_publisher" name="laser_link" 
    args="0.0 0.0 0.10 0.0 0.0 0.0 /base_link /laser 40" />

  <node pkg="gmapping" type="slam_gmapping" name="gmapping" 
    output="screen" machine="beta">
    <param name="odom_frame" value="/world" />

Running this on beta yields:

started roslaunch server http://beta:59751/
remote[alpha-0] starting roslaunch
remote[alpha-0]: creating ssh connection to alpha:22, user[alpha_user]
remote[alpha-0]: ssh connection created


 * /gmapping/odom_frame
 * /rosdistro
 * /laser_scan_matcher_node/max_iterations
 * /laser_scan_matcher_node/use_odom
 * /hokuyo_node/max_ang
 * /rosversion
 * /hokuyo_node/port
 * /laser_scan_matcher_node/use_imu
 * /laser_scan_matcher_node/fixed_frame
 * /hokuyo_node/frame_id
 * /laser_scan_matcher_node/use_alpha_beta
 * /hokuyo_node/min_ang

 * beta
 * alpha

    hokuyo_node (hokuyo_node/hokuyo_node)
    laser_scan_matcher_node (laser_scan_matcher/laser_scan_matcher_node)
    laser_link (tf/static_transform_publisher)
    gmapping (gmapping/slam_gmapping)

auto-starting new master
process[master]: started with pid [3002]

setting /run_id to 3bf5a6de-307c-11e1-ba58-00216a2f5558
process[rosout-1]: started with pid [3017]
started core service [/rosout]
process[gmapping-2]: started with pid [3020]
[alpha-0]: launching nodes...
[alpha-0]: auto-starting new master
[alpha-0]: process[master]: started with pid [1433]
[alpha-0]: ROS_MASTER_URI=http://localhost:11311
[alpha-0]: setting /run_id to 3bf5a6de-307c-11e1-ba58-00216a2f5558
[alpha-0]: process[hokuyo_node-1]: started with pid [1452]
[alpha-0]: process[laser_scan_matcher_node-2]: started with pid [1453]
[alpha-0]: process[laser_link-3]: started with pid [1454]
[alpha-0]: ... done launching nodes

I think there are two roscore's being spawned which don't know about each other. I'm guessing it, because running rosgraph locally after changing:

export ROS_MASTER_URI=http://alpha:11311
export ROS_MASTER_URI=http://beta:11311

returns only nodes run on the particular computer, and because of the suspicious lines above:

[alpha-0]: launching nodes...
[alpha-0]: auto-starting new master
[alpha-0]: process[master]: started with pid [1433]
[alpha-0]: ROS_MASTER_URI=http://localhost:11311

Testing what processes are really being spawned on both machines with:

ps aux | grep ros

yields: ps_alpha ps_beta.

Where do I tell ROS to only spawn one roscore (preferably on the remote machine, i.e. beta)? Or how do I else configure the system correctly?

edit retag flag offensive close merge delete

2 Answers

Sort by ยป oldest newest most voted

answered 2011-12-30 01:23:08 -0500

tom gravatar image

updated 2011-12-30 19:35:21 -0500

@kwc confirmed two bugs which caused the above mentioned behavior, the tickets are:

The problem only occurs when ROS_MASTER_URI is set to localhost on the machine where a distributed ROS system is being launched from. It is desired to set ROS_MASTER_URI depending on hostname instead (the bug is that roslaunch doesn't catch when you fail to set this, and compounds it by starting an extra remote core).

Example: Say your local system's hostname is beta and you're using roslaunch to start a distributed ROS system on alpha (remote) and beta (local) - as in the launchfile cited in question. Per default, ROS_MASTER_URI on beta would be:


which doesn't get cought by the system and gets falsely set on alpha too, and in effect two roscore's are being started which don't know about each other. Instead, setting:


on beta before executing launchfile causes ROS_MASTER_URI on both machines to be set to http://beta:11311 which is correct.

edit flag offensive delete link more


Correction: setting your ROS_MASTER_URI to http://beta:11311 is not a temporary workaround, it's what you're always supposed to do. The bug is that roslaunch doesn't catch when you fail to set this, and compounds it by starting an extra core.
kwc gravatar image kwc  ( 2011-12-30 05:55:38 -0500 )edit
@kwc: Corrected.
tom gravatar image tom  ( 2011-12-30 19:37:38 -0500 )edit

answered 2011-12-22 05:36:40 -0500

kwc gravatar image

updated 2011-12-22 10:55:24 -0500

roslaunch cannot spawn multiple roscores, so it sounds like something else is going on.

It sounds strange that you say you are using <machine> tags inside of nodes. You cannot use a <machine tag=""> inside of a <node> tag. You must use the machine attribute.

EDIT: also should add, roslaunch has no capability to launch roscores on remote systems, so if there is one running remotely, it did not start it.

edit flag offensive delete link more


Right, edited my question. I meant an attribute, just called it wrong.
tom gravatar image tom  ( 2011-12-22 10:41:47 -0500 )edit
I edited my question and added some more details. It seems to me there are actually two masters being spawned and ROS_MASTER_URI set to localhost on both machines.
tom gravatar image tom  ( 2011-12-26 21:26:52 -0500 )edit
roslaunch cannot start a core on a remote machine. The code to do so has not been written yet. If there is a core running on the remote machine, roslaunch did not start it.
kwc gravatar image kwc  ( 2011-12-27 06:29:07 -0500 )edit
I suggest you investigate what processes are actually running on the remote machine with tools like "ps", e.g. "ps aux | grep ros"
kwc gravatar image kwc  ( 2011-12-27 06:29:41 -0500 )edit
Thanks for your time @kwc. Pasted the suggested outputs above. It still seems to me there are two roscores being spawned... - I'm on Diamondback, built from source, maybe this leaves a hint.
tom gravatar image tom  ( 2011-12-27 21:03:01 -0500 )edit
You are correct, I am wrong, there are two masters running, which is a bug, not a feature. There is an interesting bug in the ps output that you put on pastebin (server URI is invalid). I cannot find any code path that causes it. Can you "roscd log" and upload roslaunch-*.log somewhere?
kwc gravatar image kwc  ( 2011-12-28 05:27:52 -0500 )edit
Well, isn't it nice to reveal features one would think aren't already there and need to be written yet are actually provided? Even if they still need some further tuning :). You'll find my full logs on your email at wg.
tom gravatar image tom  ( 2011-12-28 21:45:39 -0500 )edit

Question Tools



Asked: 2011-12-22 03:00:54 -0500

Seen: 7,880 times

Last updated: Dec 30 '11