Ask Your Question

Slow build time

asked 2018-09-12 19:29:41 -0500

baronep gravatar image

updated 2018-09-14 13:35:14 -0500

I am experiencing very slow builds (and I have been for a while) using catkin_make building the ROS navigation stack. I have tried -DCMAKE_BUILD_TYPE=Release, --use-ninga, ccache, etc but the problem still persists. Specifically, clean builds sometimes take me over an hour on a modern machine (i7 w/ 8GB RAM).

The build process proceeds rather quickly until it seems to stall out at 100%. A quick htop reveals that the culprit is a couple of linking threads that are taking 100% CPU and often are costmap_2d or trajectory planner related.

Has anyone seen this before or have any insight on how to fix this?

Additional Information

OS: Ubuntu 16.04.5 LTS  
gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.10)  
cmake version 3.5.1  
ROS Version: Kinetic 

ROS Install: ROS installed from .debs (desktop-full) and upgraded w/ apt

Steps to reproduce:

mkdir -p ~/test/src  
cd ~/test/src  
git clone  
git checkout kinetic-devel  
cd ..
... wait ...

Total Build Time: 22.5 min

The build got to 100% after <5min with the last message printed ... `

[100%] Built target move_base_node 
Then 15 min later ...
[100%] Built target costmap_2d_cloud
[100%] Built target costmap_2d_markers
Base path: /home/patrick/test
Source space: /home/patrick/test/src
Build space: /home/patrick/test/build
Devel space: /home/patrick/test/devel
Install space: /home/patrick/test/install
#### Running command: "cmake /home/patrick/test/src -DCATKIN_DEVEL_PREFIX=/home/patrick/test/devel -DCMAKE_INSTALL_PREFIX=/home/patrick/test/install -G Unix Makefiles" in "/home/patrick/test/build"
#### Running command: "make -j6 -l6" in "/home/patrick/test/build"

The full output of catkin_make is available here ...

Additional Information - Second Round

Memory consumption during the first 5 minutes of the build ...

patrick@patrick:~/test$ free -mh
              total        used        free      shared  buff/cache   available
Mem:           7.2G        4.1G        222M        1.1G        2.9G        1.5G
Swap:          7.4G        3.2G        4.3G

After reaching the 100% ...

patrick@patrick:~/test$ free -mh
              total        used        free      shared  buff/cache   available
Mem:           7.2G        4.6G        180M        1.2G        2.5G        1.0G
Swap:          7.4G        3.2G        4.2G

It looks like there should be plenty of free memory right?

Also, here are the processes that are "stalling" out in the second 15min of the build ...

Specifically, the /usr/bin/ld threads (there are 2, costmap_2d_cloud and costmap_2d_markers) which are taking 100% CPU

edit retag flag offensive close merge delete


Please add some information about your system: which OS is this, which version, how did you install ROS (.debs or completely from source), which and what version of compiler do you have installed, CMake version and what exactly you are building (ie: pkgs).

gvdhoorn gravatar imagegvdhoorn ( 2018-09-13 01:12:42 -0500 )edit

Added additional information in the post above

baronep gravatar imagebaronep ( 2018-09-13 10:03:36 -0500 )edit

What does your memory usage look like during that time?

David Lu gravatar imageDavid Lu ( 2018-09-13 10:07:05 -0500 )edit

Added additional information above

baronep gravatar imagebaronep ( 2018-09-13 11:07:20 -0500 )edit

Just an observation, if I comment out the costmap_2d_cloud and costmap_2d_markers targets in the CMakelists.txt, my build reduces from >20min to ~3min. What about these 2 targets would make them have such a dramatic impact? voxel_grid?

baronep gravatar imagebaronep ( 2018-09-14 13:12:46 -0500 )edit

2 Answers

Sort by ยป oldest newest most voted

answered 2018-09-19 16:27:07 -0500

David Lu gravatar image

I've replicated your problem. It's hanging on the linking step, but you can fix it by adding ${catkin_LIBRARIES} to the target_link_libraries command for costmap-2d_cloud and markers. Not sure why yet, but I'm going to investigate and make a PR.

edit flag offensive delete link more



answered 2018-09-17 11:10:36 -0500

baronep gravatar image

I was able to reduce my build time of the navigation stack by 75% by simply disabling the following targets in the costmap_2d CMakelists.txt

#add_executable(costmap_2d_markers src/costmap_2d_markers.cpp)
#add_dependencies(costmap_2d_markers ${${PROJECT_NAME}_EXPORTED_TARGETS} ${catkin_EXPORTED_TARGETS})
#    costmap_2d
#    )

#add_executable(costmap_2d_cloud src/costmap_2d_cloud.cpp)
#add_dependencies(costmap_2d_cloud ${${PROJECT_NAME}_EXPORTED_TARGETS} ${catkin_EXPORTED_TARGETS})
#    costmap_2d
#    )


install( TARGETS
#    costmap_2d_markers
#    costmap_2d_cloud
edit flag offensive delete link more


You should also test a newer gcc version, 5.4 is quite old now!

VictorLamoine gravatar imageVictorLamoine ( 2018-10-08 06:25:57 -0500 )edit

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Question Tools



Asked: 2018-09-12 19:29:41 -0500

Seen: 368 times

Last updated: Sep 19 '18