colcon: finding workspaces during build
Hello!
I'm running into some trouble trying to find the package.xml and other files of the target package and dependency packages during build:
(1) when ament_python is used, these files are moved to the install/ folder immediately, but when ament_cmake is used then the CMAKE install command won't move the files until after the package build was completed, forcing me to search for the source folder.
(2) so I tried using the COLCON_PREFIX_PATH environment variable, assuming the default layout where src/ is right next to install. On my local environment, COLCON_PREFIX_PATH points to the install/ folder, but on the build farm, it sometimes points to /opt/ros/crystal
(3) I tried to fall-back on using AMENT_PREFIX_PATH which appears to point at install/<package>/ but it seems it's not always set when run on the build farm.
(4) I found that the current working directory (CWD) is usually the source directory for package to be build, but again this doesn't always seem to be the case.
Could someone please give me some more insight if there is a recommended standard way of searching the workspace for package.xml files?
Thank you!
Best regards, Max
[EDIT] More context on the whole issue:
I'm asking in relation to the development of ament_virtualen.
With ament_virtualenv, a packages can place place a requirements.txt file next to their package.xml file which is parsed to provide a virtual Python environment for their package.
However, since the package using ament_virtualenv may be using other packages which themselves require Python packages to run,
ament_virtualenv needs to recursively check the dependencies of the package.
So the process goes something like this:
Upon building the target package, ament_virtualenv (which gets hooked into the build process) ...
(1) ... finds the currently-building package's package.xml to find all dependencies
(-> with ament_python, this can be found in install/, but with ament_cmake the files don't get copied over until after the build, which poses the first problem)
(2) ... recursively searches for these dependency packages
(-> these can be expected to already be fully in install/)
(3) ... for all packages in that dependency tree, combine the Python requirements into one file
(-> here again, the package currently being built may not yet have copied it's requirements.txt to install/ when ament_cmake is used)
... and then use that gathered information to generate the Python venv.
In my local environment, it's not too much of a problem because the workspace has the default layout and COLCON_PREFIX_PATH environemnt variable seems to be always set.
But on the build farm this fails for many reasons like COLCON_PREFIX_PATH not being set, install/ being named differently etc.
I should also mention that ament_virtualenv works with both ament_cmake and ament_python builds, but always uses the same Python script for the process.
Is your package a CMake or Python package? Are you trying to find the
package.xmlof another package, or some files installed by that package?@slorenz Both (ament_cmake and ament_python) and both (same package and those packages that dependencies).
I am not sure I understand what you want to achieve but a package should NOT access other packages under the
srcdirectory. It only has access to the packages it depends under theinstalldirectory. If one of your dependencies only has some files insrcbut not underinstallthen those packages need to be update to actually install the files.@Dirk Thomas: It does NOT rely on other package's
src/folder, but: (1) it requires it's own files, which - onament_cmakeare not moved toinstall/until after the build is finished, so my fall-back is to search it'ssrc/folder. There is no environment variable for that andCWDisn't always the source folder. (2) it requires the files of dependency packages which are ininstall/, but in some environments (esp. the build farm)COLCON_PREFIX_PATHwhich should point there is either not set or points at/opt/ros/crystal/. (3)AMENT_PREFIX_PATHandCMAKE_PREFIX_PATHpoint to theinstall/path/to/package, which makes it difficult to find other packages from there.Please describe what you want to achieve since without that context I don't understand what you are trying to do. In CMake you have the
CMAKE_SOURCE_DIRvariable which points to the directory of your "root"CMakeLists.txtwhich should in the root of your package beside thepackage.xmlfile. And when youfind_package()a dependencyfooyou get access the CMake config file location withfoo_DIRand the generated CMake config files ofament_cmakepackages are always in<install>/share/<pkgname>/cmake. (Neither of the two recommendations is specific to ROS but just common CMake.)Thank you @Dirk Thomas . The background is that for our package
ament_virtualenvwe need access to the information in whichever package is usingament_virtualenvat build time and also an additional filerequirements.txt, as well all thepackage.xmlandrequirements.txtthat dependency packages have already installed in theinstall/folder. I did a quick test onCMAKE_SOURCE_DIR, but it is NOT set when building withament_pythonand it is NOT set when building withament_cmakeeither.That CMake variable (
CMAKE_SOURCE_DIR) is definitely available for CMake packages - it is set by CMake in every CMakeLists.txt file (https://cmake.org/cmake/help/latest/v...).Python packages certainly don't have that since it is provide by CMake itself.
Please extend the question with significant more context about what the package
ament_virtualenvis trying to do as well as a detailed description of your workflow (what exactly is in the workspace, how are the packages depending on each other, what command do you run, what are you tried to do). Others not familiar with what you are doing can't provide help without that kind of context.