Robotics StackExchange | Archived questions

why does catkin_make keep rebuilding my package from scratch when no changes in src?

I have a "dummy" ROS package that I use to build and source a python virtual environment for my workspace ROS application. there's a shell script that installs pip and installs some packages and another for activating the environment when the workspace is sourced (via env hooks).

This all works, except that every time I run catkin_make, the whole package rebuilds from scratch - so all the pip packages are downloaded and built again - taking a lot of time. Other packages only download and build if there have been changes to the source. Why does this happen and how can I fix it?

virtualenv_build.sh

#!/usr/bin/env bash

# install the venv package if not already present
pip2 install --user virtualenv

# OLDPWD resolves as the workspace build folder when built with catkin
VENV_PATH="$OLDPWD/aios-env"
virtualenv -p "/usr/bin/python2" --system-site-packages "$VENV_PATH"

# source the venv and install packages
. "$VENV_PATH/bin/activate"
python2 -m pip install --upgrade pip
pip install --ignore-installed -r requirements.txt

# if cuda is installed then install the cuda-only packages
if type nvcc; then
  pip install --ignore-installed -r requirements-cuda.txt
fi

virtualenv_source.sh

AIOS_ENV=./build/aios-env
. $AIOS_ENV/bin/activate
# make sure our packages are imported first
export PYTHON_PATH="$AIOS_ENV/lib/python2.7/site-packages/:$PYTHON_PATH"

CMakeLists.sh

cmake_minimum_required(VERSION 3.10.2)
project(virtualenv_config)

set(CMAKE_CXX_STANDARD 14)

find_package(catkin REQUIRED)

# run virtualenv_build.sh at build time
add_custom_target(
  VirtualEnv ALL
  DEPENDS my-env
)
add_custom_command(OUTPUT my-env
                   COMMAND ${CMAKE_CURRENT_SOURCE_DIR}/virtualenv_build.sh
                   WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}/)
catkin_package()

# run virtualenv_source.sh when workplace is sourced
catkin_add_env_hooks(50.virtualenv_source SHELLS bash DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}/env-hooks)

install(FILES virtualenv_source.sh
  DESTINATION ${CATKIN_PACKAGE_SHARE_DESTINATION})

install(DIRECTORY my-env/
  DESTINATION ${CATKIN_PACKAGE_SHARE_DESTINATION}/my-env)

Asked by LukeAI on 2021-04-19 06:50:27 UTC

Comments

Not an answer, but: did catkin_virtualenv not work for you (could be it doesn't know how to deal with the CUDA parts)?

Asked by gvdhoorn on 2021-04-19 10:06:07 UTC

Why does this happen and how can I fix it?

if you cd /path/to/catkin_ws/build/virtualenv_config and then make, does that also build "everything from scratch"?

if so: it's a CMake problem. If not, then it could be caused by catkin_make invoking CMake in some way, or (re)setting a bit of environment.

Asked by gvdhoorn on 2021-04-19 10:08:13 UTC

yes, I'm not sure how to conditionally install CUDA packages with catkin_virtualenv. If I cd to catkin_ws/build/blah/virtualenv_config and run make it does run the build script, re-downloading all the pip packages again.

Asked by LukeAI on 2021-04-20 01:53:50 UTC

Then it would appear to be a CMake problem.

It's likely CMake is unable to figure out whether your "source files" are newer than the build output, and hence decides to (re)start another build.

My guess would be there is something you need to do with the add_custom_command(..).

I'm not sure how to conditionally install CUDA packages with catkin_virtualenv.

As all catkin_virtualenv packages still have a CMakeLists.txt, can't you use that to conditionally install certain dependencies / packages?

Asked by gvdhoorn on 2021-04-20 02:22:57 UTC

I don't know! I guess I need to learn more about CMake

Asked by LukeAI on 2021-05-12 15:27:29 UTC

Answers