colcon: finding workspaces during build

asked 2019-12-11 02:42:30 -0600

max-krichenbauer gravatar image

updated 2019-12-13 05:23:35 -0600


I'm running into some trouble trying to find the package.xml and other files of the target package and dependency packages during build:

(1) when ament_python is used, these files are moved to the install/ folder immediately, but when ament_cmake is used then the CMAKE install command won't move the files until after the package build was completed, forcing me to search for the source folder.

(2) so I tried using the COLCON_PREFIX_PATH environment variable, assuming the default layout where src/ is right next to install. On my local environment, COLCON_PREFIX_PATH points to the install/ folder, but on the build farm, it sometimes points to /opt/ros/crystal

(3) I tried to fall-back on using AMENT_PREFIX_PATH which appears to point at install/<package>/ but it seems it's not always set when run on the build farm.

(4) I found that the current working directory (CWD) is usually the source directory for package to be build, but again this doesn't always seem to be the case.

Could someone please give me some more insight if there is a recommended standard way of searching the workspace for package.xml files?

Thank you!

Best regards, Max

[EDIT] More context on the whole issue:

I'm asking in relation to the development of ament_virtualen.

With ament_virtualenv, a packages can place place a requirements.txt file next to their package.xml file which is parsed to provide a virtual Python environment for their package.

However, since the package using ament_virtualenv may be using other packages which themselves require Python packages to run, ament_virtualenv needs to recursively check the dependencies of the package.

So the process goes something like this:

Upon building the target package, ament_virtualenv (which gets hooked into the build process) ...

(1) ... finds the currently-building package's package.xml to find all dependencies (-> with ament_python, this can be found in install/, but with ament_cmake the files don't get copied over until after the build, which poses the first problem)

(2) ... recursively searches for these dependency packages (-> these can be expected to already be fully in install/)

(3) ... for all packages in that dependency tree, combine the Python requirements into one file (-> here again, the package currently being built may not yet have copied it's requirements.txt to install/ when ament_cmake is used)

... and then use that gathered information to generate the Python venv.

In my local environment, it's not too much of a problem because the workspace has the default layout and COLCON_PREFIX_PATH environemnt variable seems to be always set. But on the build farm this fails for many reasons like COLCON_PREFIX_PATH not being set, install/ being named differently etc.

I should also mention that ament_virtualenv works with both ament_cmake and ament_python builds, but always uses the same Python script for the process.

edit retag flag offensive close merge delete


Is your package a CMake or Python package? Are you trying to find the package.xml of another package, or some files installed by that package?

sloretz gravatar image sloretz  ( 2019-12-11 08:22:24 -0600 )edit

@slorenz Both (ament_cmake and ament_python) and both (same package and those packages that dependencies).

max-krichenbauer gravatar image max-krichenbauer  ( 2019-12-11 08:52:47 -0600 )edit

I am not sure I understand what you want to achieve but a package should NOT access other packages under the src directory. It only has access to the packages it depends under the install directory. If one of your dependencies only has some files in src but not under install then those packages need to be update to actually install the files.

Dirk Thomas gravatar image Dirk Thomas  ( 2019-12-11 13:18:37 -0600 )edit

@Dirk Thomas: It does NOT rely on other package's src/ folder, but: (1) it requires it's own files, which - on ament_cmake are not moved to install/ until after the build is finished, so my fall-back is to search it's src/ folder. There is no environment variable for that and CWD isn't always the source folder. (2) it requires the files of dependency packages which are in install/, but in some environments (esp. the build farm) COLCON_PREFIX_PATH which should point there is either not set or points at /opt/ros/crystal/. (3) AMENT_PREFIX_PATH and CMAKE_PREFIX_PATH point to the install/path/to/package, which makes it difficult to find other packages from there.

max-krichenbauer gravatar image max-krichenbauer  ( 2019-12-11 23:30:23 -0600 )edit

Please describe what you want to achieve since without that context I don't understand what you are trying to do. In CMake you have the CMAKE_SOURCE_DIR variable which points to the directory of your "root" CMakeLists.txt which should in the root of your package beside the package.xml file. And when you find_package() a dependency foo you get access the CMake config file location with foo_DIR and the generated CMake config files of ament_cmake packages are always in <install>/share/<pkgname>/cmake. (Neither of the two recommendations is specific to ROS but just common CMake.)

Dirk Thomas gravatar image Dirk Thomas  ( 2019-12-11 23:47:33 -0600 )edit

Thank you @Dirk Thomas . The background is that for our package ament_virtualenv we need access to the information in whichever package is using ament_virtualenv at build time and also an additional file requirements.txt, as well all the package.xml and requirements.txt that dependency packages have already installed in the install/ folder. I did a quick test on CMAKE_SOURCE_DIR, but it is NOT set when building with ament_python and it is NOT set when building with ament_cmake either.

max-krichenbauer gravatar image max-krichenbauer  ( 2019-12-12 00:01:59 -0600 )edit

That CMake variable (CMAKE_SOURCE_DIR) is definitely available for CMake packages - it is set by CMake in every CMakeLists.txt file (

Python packages certainly don't have that since it is provide by CMake itself.

Dirk Thomas gravatar image Dirk Thomas  ( 2019-12-12 00:12:17 -0600 )edit

Please extend the question with significant more context about what the package ament_virtualenv is trying to do as well as a detailed description of your workflow (what exactly is in the workspace, how are the packages depending on each other, what command do you run, what are you tried to do). Others not familiar with what you are doing can't provide help without that kind of context.

Dirk Thomas gravatar image Dirk Thomas  ( 2019-12-12 00:14:38 -0600 )edit