Robotics StackExchange | Archived questions

colcon: finding workspaces during build

Hello!

I'm running into some trouble trying to find the package.xml and other files of the target package and dependency packages during build:

(1) when ament_python is used, these files are moved to the install/ folder immediately, but when ament_cmake is used then the CMAKE install command won't move the files until after the package build was completed, forcing me to search for the source folder.

(2) so I tried using the COLCON_PREFIX_PATH environment variable, assuming the default layout where src/ is right next to install. On my local environment, COLCON_PREFIX_PATH points to the install/ folder, but on the build farm, it sometimes points to /opt/ros/crystal

(3) I tried to fall-back on using AMENT_PREFIX_PATH which appears to point at install/<package>/ but it seems it's not always set when run on the build farm.

(4) I found that the current working directory (CWD) is usually the source directory for package to be build, but again this doesn't always seem to be the case.

Could someone please give me some more insight if there is a recommended standard way of searching the workspace for package.xml files?

Thank you!

Best regards, Max

[EDIT] More context on the whole issue:

I'm asking in relation to the development of ament_virtualen.

With ament_virtualenv, a packages can place place a requirements.txt file next to their package.xml file which is parsed to provide a virtual Python environment for their package.

However, since the package using ament_virtualenv may be using other packages which themselves require Python packages to run, ament_virtualenv needs to recursively check the dependencies of the package.

So the process goes something like this:

Upon building the target package, ament_virtualenv (which gets hooked into the build process) ...

(1) ... finds the currently-building package's package.xml to find all dependencies (-> with ament_python, this can be found in install/, but with ament_cmake the files don't get copied over until after the build, which poses the first problem)

(2) ... recursively searches for these dependency packages (-> these can be expected to already be fully in install/)

(3) ... for all packages in that dependency tree, combine the Python requirements into one file (-> here again, the package currently being built may not yet have copied it's requirements.txt to install/ when ament_cmake is used)

... and then use that gathered information to generate the Python venv.

In my local environment, it's not too much of a problem because the workspace has the default layout and COLCON_PREFIX_PATH environemnt variable seems to be always set. But on the build farm this fails for many reasons like COLCON_PREFIX_PATH not being set, install/ being named differently etc.

I should also mention that ament_virtualenv works with both ament_cmake and ament_python builds, but always uses the same Python script for the process.

Asked by max-krichenbauer on 2019-12-11 03:42:30 UTC

Comments

Is your package a CMake or Python package? Are you trying to find the package.xml of another package, or some files installed by that package?

Asked by sloretz on 2019-12-11 09:22:24 UTC

@slorenz Both (ament_cmake and ament_python) and both (same package and those packages that dependencies).

Asked by max-krichenbauer on 2019-12-11 09:52:47 UTC

I am not sure I understand what you want to achieve but a package should NOT access other packages under the src directory. It only has access to the packages it depends under the install directory. If one of your dependencies only has some files in src but not under install then those packages need to be update to actually install the files.

Asked by Dirk Thomas on 2019-12-11 14:18:37 UTC

@Dirk Thomas: It does NOT rely on other package's src/ folder, but: (1) it requires it's own files, which - on ament_cmake are not moved to install/ until after the build is finished, so my fall-back is to search it's src/ folder. There is no environment variable for that and CWD isn't always the source folder. (2) it requires the files of dependency packages which are in install/, but in some environments (esp. the build farm) COLCON_PREFIX_PATH which should point there is either not set or points at /opt/ros/crystal/. (3) AMENT_PREFIX_PATH and CMAKE_PREFIX_PATH point to the install/path/to/package, which makes it difficult to find other packages from there.

Asked by max-krichenbauer on 2019-12-12 00:30:23 UTC

Please describe what you want to achieve since without that context I don't understand what you are trying to do. In CMake you have the CMAKE_SOURCE_DIR variable which points to the directory of your "root" CMakeLists.txt which should in the root of your package beside the package.xml file. And when you find_package() a dependency foo you get access the CMake config file location with foo_DIR and the generated CMake config files of ament_cmake packages are always in <install>/share/<pkgname>/cmake. (Neither of the two recommendations is specific to ROS but just common CMake.)

Asked by Dirk Thomas on 2019-12-12 00:47:33 UTC

Thank you @Dirk Thomas . The background is that for our package ament_virtualenv we need access to the information in whichever package is using ament_virtualenv at build time and also an additional file requirements.txt, as well all the package.xml and requirements.txt that dependency packages have already installed in the install/ folder. I did a quick test on CMAKE_SOURCE_DIR, but it is NOT set when building with ament_python and it is NOT set when building with ament_cmake either.

Asked by max-krichenbauer on 2019-12-12 01:01:59 UTC

That CMake variable (CMAKE_SOURCE_DIR) is definitely available for CMake packages - it is set by CMake in every CMakeLists.txt file (https://cmake.org/cmake/help/latest/variable/CMAKE_SOURCE_DIR.html).

Python packages certainly don't have that since it is provide by CMake itself.

Asked by Dirk Thomas on 2019-12-12 01:12:17 UTC

Please extend the question with significant more context about what the package ament_virtualenv is trying to do as well as a detailed description of your workflow (what exactly is in the workspace, how are the packages depending on each other, what command do you run, what are you tried to do). Others not familiar with what you are doing can't provide help without that kind of context.

Asked by Dirk Thomas on 2019-12-12 01:14:38 UTC

Thank you @Dirk Thomas I've added more context to the question. I've tested for CMAKE_SOURCE_DIR and it is not set as an environment variable. Possibly it's a CMAKE variable only available inside CMAKE scripts?

Asked by max-krichenbauer on 2019-12-13 06:25:29 UTC

Yes, as mentioned in my previous comment CMAKE_SOURCE_DIR is a CMake variable.

Asked by Dirk Thomas on 2019-12-13 12:48:43 UTC

Since I don't know anything about how ament_virtualenv "hooks into the build process" I can't suggest anything. Maybe more event more context would help?

Asked by Dirk Thomas on 2019-12-13 12:50:28 UTC

It sounds like for Python packages you have it all working and the only problem is with CMake packages. Within CMake you should have all the information you need. So assume that is not how ament_virtualenv "hooks into the build process"?

Asked by Dirk Thomas on 2019-12-13 12:51:46 UTC

Is this the block used to determine the requirements of the current package (and its dependencies): https://github.com/esol-community/ament_virtualenv/blob/c9fd5b7a5a91ba32cb775216d458d8ed79e11fe1/ament_cmake_virtualenv/cmake/ament_generate_virtualenv.cmake#L67-L78 If yes, you can use the CMake variable mentioned above to e.g. pass the path of the requirements.txt file of the current package to the invoked scripts.

Asked by Dirk Thomas on 2019-12-15 00:56:40 UTC

Answers