Ask Your Question
0

Why does catkin build not consistently do the same thing?

asked 2021-05-14 11:24:27 -0500

LukeAI gravatar image

updated 2021-05-14 11:33:04 -0500

The catkin build command builds each package in a workspace’s source space in isolation in order to prevent build-time cross-talk.

I have a large workspace of ROS1 packages that builds with catkin_make. Sometimes other people with the same workspace will build and find that their build fails with some missing dependency in a CMakeLists.txt somewhere. I understand that catkin_make does non-isolated builds - so whether or not a dependency is available to a package is dependent on the order that the packages happen to built so lots of CMakeLists.txt ommissions are masked.

I have tried building my workspace with catkin build and found that it does expose a lot of build errors but it doesn't give me consistent builds. If I run it several times in succession, more and more of the packages will be successfully built. If I run catkin build && catkin clean several times, different numbers of packages will be successfully built.

If packages are being built "in isolation" I would expect the same thing to happen each time I run catkin build since the build order should be irrelevant? Clearly I am mistaken in this understanding so my question is:

What does packages being built "in isolation" mean? How is catkin build different to catkin_make ?

Is there some way to build my packages that is determinstic/repeatable? So that I can have confidence that if it builds on one machine, it'll build on another?

even if I run catkin build --force-cmake && catkin clean I get inconsistent results.

edit retag flag offensive close merge delete

Comments

1

Please refrain from asking multiple questions in a single post.

gvdhoorn gravatar image gvdhoorn  ( 2021-05-14 12:26:08 -0500 )edit

1 Answer

Sort by » oldest newest most voted
2

answered 2021-05-14 13:51:19 -0500

gvdhoorn gravatar image

updated 2021-05-15 09:29:39 -0500

I'm only going to answer this part of your question:

Is there some way to build my packages that is determinstic/repeatable?

tl;dr: if you don't state your dependencies correctly, Catkin is free to start building pkgs, which may in fact depend on each other, in parallel, leading to compiler errors and ultimately failed and aborted builds.

As the exact point in time where builds get aborted is (almost) stochastic (due to scheduling, CPU and IO performance, etc), and dependents typically will not need everything from dependencies, partial builds of dependencies may result in successful builds of (some) dependents.

This leads to successive builds appearing to make progress, as more-and-more dependents will find their dependencies where they expect them, leading to those builds succeeding, leading to their dependents being able to start their build. Etc, etc.

With enough iterations, the whole graph of dependencies gets built, until finally all dependencies are built and there are no more failures.

As to the question: always running catkin_make -j1 could help make things fail at almost the same place, however it's no guarantee, as other factors may still cause slight ordering differences in operations.


Long version: the Catkin side to this is already largely deterministic I believe. It's the other parts which may not be.

Catkin performs two main tasks before it asks CMake to build everything:

  1. discover all packages in your src space and construct a graph where the vertices are the packages and the edges encode dependencies (ie: if pkg A depends on B, there will be an edge between the vertices representing A and B expressing that dependency)
  2. run a topological sort over that graph, to determine the order in which everything should be built which satisfies those dependencies

Your packages state their dependencies using build_depends in their manifests, which Catkin parses and then uses to build the graph. After the sort, it asks CMake to start the build by building the packages which:

  1. have no dependencies (rare),
  2. for which all dependencies are already built,
  3. depend only on packages present in an underlay (or its underlay, ad infinitum)
  4. do not depend on each other

for all of those, the order in which they are built does not really matter, as pkgs in category 1 don't depend on anything, in category 2 and 3 have all their dependencies ready and in category 4 cannot fail when built at the same time as they don't need anything from each other.

After the first layer of packages has been built, Catkin moves up to the next, which will be packages which depend on (some, or all) of the packages it just built. This process then repeats, until the "top" package(s) is (are) built.

Note: only the build_depend entries in your package.xml are taken into account, not the setup.py or CMakeLists.txt (actually, there's also build_export_depend et al., but I'll ignore those here, they don't significantly change the explanation).

Now ... (more)

edit flag offensive delete link more

Comments

1

Refer to #q217475 for why we have separate (or duplicated) places to store dependency information btw.

gvdhoorn gravatar image gvdhoorn  ( 2021-05-14 14:00:39 -0500 )edit

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Question Tools

1 follower

Stats

Asked: 2021-05-14 11:24:27 -0500

Seen: 44 times

Last updated: May 15