why does catkin_make keep rebuilding my package from scratch when no changes in src?

asked 2021-04-19 06:50:27 -0500

LukeAI gravatar image

updated 2021-04-19 07:27:56 -0500

I have a "dummy" ROS package that I use to build and source a python virtual environment for my workspace ROS application. there's a shell script that installs pip and installs some packages and another for activating the environment when the workspace is sourced (via env hooks).

This all works, except that every time I run catkin_make, the whole package rebuilds from scratch - so all the pip packages are downloaded and built again - taking a lot of time. Other packages only download and build if there have been changes to the source. Why does this happen and how can I fix it?


#!/usr/bin/env bash

# install the venv package if not already present
pip2 install --user virtualenv

# OLDPWD resolves as the workspace build folder when built with catkin
virtualenv -p "/usr/bin/python2" --system-site-packages "$VENV_PATH"

# source the venv and install packages
. "$VENV_PATH/bin/activate"
python2 -m pip install --upgrade pip
pip install --ignore-installed -r requirements.txt

# if cuda is installed then install the cuda-only packages
if type nvcc; then
  pip install --ignore-installed -r requirements-cuda.txt


. $AIOS_ENV/bin/activate
# make sure our packages are imported first
export PYTHON_PATH="$AIOS_ENV/lib/python2.7/site-packages/:$PYTHON_PATH"


cmake_minimum_required(VERSION 3.10.2)


find_package(catkin REQUIRED)

# run virtualenv_build.sh at build time
  VirtualEnv ALL
  DEPENDS my-env
add_custom_command(OUTPUT my-env
                   COMMAND ${CMAKE_CURRENT_SOURCE_DIR}/virtualenv_build.sh

# run virtualenv_source.sh when workplace is sourced
catkin_add_env_hooks(50.virtualenv_source SHELLS bash DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}/env-hooks)

install(FILES virtualenv_source.sh

install(DIRECTORY my-env/
edit retag flag offensive close merge delete


Not an answer, but: did catkin_virtualenv not work for you (could be it doesn't know how to deal with the CUDA parts)?

gvdhoorn gravatar image gvdhoorn  ( 2021-04-19 10:06:07 -0500 )edit

Why does this happen and how can I fix it?

if you cd /path/to/catkin_ws/build/virtualenv_config and then make, does that also build "everything from scratch"?

if so: it's a CMake problem. If not, then it could be caused by catkin_make invoking CMake in some way, or (re)setting a bit of environment.

gvdhoorn gravatar image gvdhoorn  ( 2021-04-19 10:08:13 -0500 )edit

yes, I'm not sure how to conditionally install CUDA packages with catkin_virtualenv. If I cd to catkin_ws/build/blah/virtualenv_config and run make it does run the build script, re-downloading all the pip packages again.

LukeAI gravatar image LukeAI  ( 2021-04-20 01:53:50 -0500 )edit

Then it would appear to be a CMake problem.

It's likely CMake is unable to figure out whether your "source files" are newer than the build output, and hence decides to (re)start another build.

My guess would be there is something you need to do with the add_custom_command(..).

I'm not sure how to conditionally install CUDA packages with catkin_virtualenv.

As all catkin_virtualenv packages still have a CMakeLists.txt, can't you use that to conditionally install certain dependencies / packages?

gvdhoorn gravatar image gvdhoorn  ( 2021-04-20 02:22:57 -0500 )edit

I don't know! I guess I need to learn more about CMake

LukeAI gravatar image LukeAI  ( 2021-05-12 15:27:29 -0500 )edit