Robotics StackExchange | Archived questions

How to install colcon packages into system workspace

I'm working on a new snapcraft plugin for colcon (using crystal) and I've hit a bit of an issue. The way ROS snaps are built today ends up installing all packages into the same workspace, which if you ignore snaps ends up being the system workspace in /opt/ros/<rosdistro>/. This works fine with Catkin, just source /opt/ros/<rosdistro>/setup.sh, and run catkin_make_isolated --install --install-space /opt/ros/<rosdistro>. This worked because the setup.sh was the same. However, I'm trying to accomplish the same thing with Colcon and it's proving troublesome. The setup.sh and local_setup.sh is customized for the given workspace, and the one in /opt/ros/<rosdistro> is far different, and overwriting the system one with the workspace's of course doesn't work. It looks like things work with colcon build --merge-install as long as I don't install the setup.sh and local_setup.sh generated for the workspace (and instead rely on the package-level ones), although I haven't pushed that path too hard. Is there a way to request that these files aren't generated/installed? Looking through the code I thought maybe AMENT_CMAKE_ENVIRONMENT_GENERATION was what I was looking for, but it seems to have no effect on those files. Am I missing something?

Update 1 To answer some questions asked in answers:

So I assume multiple snaps installing the same file (as long as it is identical) is ok? (I am just asking because e.g. for Debian packages that is not the case.)

Yes and no. In the typical case that I'm talking about here, snaps are isolated such that if one were to install multiple ROS snaps, they install alongside each other, not on top. They end up being independent workspaces that don't know the other exists. What I'm talking about in this question is installing multiple ROS packages in a single workspace into the same snap (along with, say, ros-crystal-ros-base). In this case, any clashing file ends up being overwritten by the installation of the workspace into the snap where ros-crystal-ros-base is already located, which is why I tried to compare it to just installing with root privs on the system into the root workspace used by the OR-provided debs. It's essentially the same, it's just not garbaging up the system's ROS install at that point.

Can you please describe how they are different and if you are referring to the ones generated by colcon or by ament_cmake? (A comment line at the top of the files should clarify which template they have been expanded from.)

This may be the crux of the issue. The version installed by the OR debs is from ament, the version installed in the workspace is from colcon (this statement applies to both setup.sh and local_setup.sh). I can't pretend to be an expert here, but they don't seem compatible in that the colcon setup.sh just sources the local_setup.sh which runs _local_setup_util.py which only discovers colcon packages in <prefix>/share/colcon-core/packages/, which at least in my simple talker/listener test with ros-crystal-ros-base installed ONLY includes my talker and listener, none of the OR ROS components.

Update 2:

Ok, as far as I understand you are using two workspaces: * the underlay in /opt/ros/crystal provided by Debian packages * the overlay (in the form of a snap) somewhere else which is supposed to depend on the underlay (so it was built with colcon and the underlay being sourced before)

And the problem is that when sourcing the setup.sh from the overlay workspace that you don't get anything from the underlay. Right?

Mostly correct. The underlay is actually already IN the snap (not on the system, I was using that for comparison), and I'm trying to install the overlay into the snap as well. There are two ways to do that: install the overlay into another directory and keep it as a separate workspace and chain them in production, or install the overlay directory INTO the underlay so that in production there is only one workspace. The latter is the path we took for Catkin, and I'm attempting to do the same for Colcon. How are you handling this in the Debian packaging of Colcon components to work alongside the Ament ones?

Regarding the effect, I was about to say you were spot on, but then it started working for me like magic. I must have broken this on my own, which is pretty embarrassing, I'm sorry. However, the fact that it wasn't working made perfect sense-- these setup scripts are quite different. How is this working?

Update 3:

Ah ha, I found what I did. It was switching from a normal install to a merge install where it started finally working (teach me to change too many things at once). This still feels a bit dirty though, and I'd be surprised if it worked for everything.

Asked by kyrofa on 2019-01-28 16:17:40 UTC

Comments

Answers

This works fine with Catkin, just source /opt/ros//setup.sh, and run catkin_make_isolated --install --install-space /opt/ros/.

So I assume multiple snaps installing the same file (as long as it is identical) is ok? (I am just asking because e.g. for Debian packages that is not the case.)

The setup.sh and local_setup.sh is customized for the given workspace, and the one in/opt/ros/` is far different, and overwriting the system one with the workspace's of course doesn't work.

Can you please describe how they are different and if you are referring to the ones generated by colcon or by ament_cmake? (A comment line at the top of the files should clarify which template they have been expanded from.)

It looks like things work with colcon build --merge-install as long as I don't install the setup.sh and local_setup.sh generated for the workspace

What is the problem with these setup files? They shouldn't contain any different logic.

Is there a way to request that these files aren't generated/installed?

Not at the moment. Maybe a new command line option to make this line optional would help in that case (https://github.com/colcon/colcon-core/blob/839832f5dd079b0832bccd3f5778695163fe06f0/colcon_core/verb/build.py#L137) but I am not yet sure if that is really necessary...

Update 1:

The version installed by the OR debs is from ament, the version installed in the workspace is from colcon

Ok, as far as I understand you are using two workspaces: * the underlay in /opt/ros/crystal provided by Debian packages * the overlay (in the form of a snap) somewhere else which is supposed to depend on the underlay (so it was built with colcon and the underlay being sourced before)

And the problem is that when sourcing the setup.sh from the overlay workspace that you don't get anything from the underlay. Right?

Update 2:

There are two ways to do that: install the overlay into another directory and keep it as a separate workspace and chain them in production, or install the overlay directory INTO the underlay so that in production there is only one workspace. The latter is the path we took for Catkin, and I'm attempting to do the same for Colcon. How are you handling this in the Debian packaging of Colcon components to work alongside the Ament ones?

The short answer: we don't. The Debian packages are built just using dpkg-buildpackage which invoked CMake which uses the logic of ament_cmake to generate the setup files (only for one of the Debian packages in order to not conflict). Any workspace built ontop by the user is currently in a separate workspace. And based on this conversation I don't think it will work to install the colcon workspace into the same workspace as the not-with-colcon installed packages.

That might be something we should consider and see if we can make this work (since it did work with ROS 1 and some users might be tempted to install newer versions of self built packages into /opt).

but then it started working for me like magic.

I still think there might be a bug in our current Debian packages though. When installing them and building a workspace with colcon ontop sourcing the resulting setup.sh file in the overlay doesn't give us the environment from the underlay which it is supposed to. In comparison when building an underlay with colcon and then an overlay with colcon the resulting setup file in the overlay will make sure to also source the underlay which was sources at build time of the overlay... (and right on time the following ticket is describing this problem: https://github.com/ros2/ros2/issues/653)

Asked by Dirk Thomas on 2019-01-29 12:53:02 UTC

Comments

Excellent questions, I've updated the question with the answers. I agree that perhaps a CLI option isn't needed yet, I feel like I'm just really misunderstanding something so far.

Asked by kyrofa on 2019-01-29 14:01:32 UTC

Regarding your update: almost correct. More clarification added to the question.

Asked by kyrofa on 2019-01-29 14:44:22 UTC

Ha! I noticed that lack of proper environment too, thought maybe I just misunderstood how it was supposed to work. Okay, this is making more sense, but I agree, I'd like to make this use-case work. Is there anything I can do?

Asked by kyrofa on 2019-01-29 14:57:47 UTC

(perhaps this should be moved into discourse at this point)

Asked by kyrofa on 2019-01-29 15:04:30 UTC

I added a comment to https://github.com/ros2/ros2/issues/653 but atm I don't see an easy path to get there. This probably needs some significant (re-)design/planning. If the referenced ticket gets closed by a short term hack we might want to create a new ticket for the desired end goal...

Asked by Dirk Thomas on 2019-01-29 15:04:40 UTC