Ask Your Question
0

Slow build time

asked 2018-09-12 19:29:41 -0500

baronep gravatar image

updated 2018-09-14 13:35:14 -0500

I am experiencing very slow builds (and I have been for a while) using catkin_make building the ROS navigation stack. I have tried -DCMAKE_BUILD_TYPE=Release, --use-ninga, ccache, etc but the problem still persists. Specifically, clean builds sometimes take me over an hour on a modern machine (i7 w/ 8GB RAM).

The build process proceeds rather quickly until it seems to stall out at 100%. A quick htop reveals that the culprit is a couple of linking threads that are taking 100% CPU and often are costmap_2d or trajectory planner related.

Has anyone seen this before or have any insight on how to fix this?


Additional Information

OS: Ubuntu 16.04.5 LTS  
gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.10)  
cmake version 3.5.1  
ROS Version: Kinetic 

ROS Install: ROS installed from .debs (desktop-full) and upgraded w/ apt

Steps to reproduce:

mkdir -p ~/test/src  
cd ~/test/src  
catkin_init_workspace  
git clone https://github.com/ros-planning/navigation.git  
git checkout kinetic-devel  
cd ..
catkin_make  
... wait ...

Total Build Time: 22.5 min

The build got to 100% after <5min with the last message printed ... `

[100%] Built target move_base_node 
Then 15 min later ...
[100%] Built target costmap_2d_cloud
[100%] Built target costmap_2d_markers
Base path: /home/patrick/test
Source space: /home/patrick/test/src
Build space: /home/patrick/test/build
Devel space: /home/patrick/test/devel
Install space: /home/patrick/test/install
####
#### Running command: "cmake /home/patrick/test/src -DCATKIN_DEVEL_PREFIX=/home/patrick/test/devel -DCMAKE_INSTALL_PREFIX=/home/patrick/test/install -G Unix Makefiles" in "/home/patrick/test/build"
####
####
#### Running command: "make -j6 -l6" in "/home/patrick/test/build"
####

The full output of catkin_make is available here ... https://pastebin.com/ZkFUqU2s


Additional Information - Second Round

Memory consumption during the first 5 minutes of the build ...

patrick@patrick:~/test$ free -mh
              total        used        free      shared  buff/cache   available
Mem:           7.2G        4.1G        222M        1.1G        2.9G        1.5G
Swap:          7.4G        3.2G        4.3G

After reaching the 100% ...

patrick@patrick:~/test$ free -mh
              total        used        free      shared  buff/cache   available
Mem:           7.2G        4.6G        180M        1.2G        2.5G        1.0G
Swap:          7.4G        3.2G        4.2G

It looks like there should be plenty of free memory right?

Also, here are the processes that are "stalling" out in the second 15min of the build ... https://pastebin.com/CDb2qLQs

Specifically, the /usr/bin/ld threads (there are 2, costmap_2d_cloud and costmap_2d_markers) which are taking 100% CPU

edit retag flag offensive close merge delete

Comments

Please add some information about your system: which OS is this, which version, how did you install ROS (.debs or completely from source), which and what version of compiler do you have installed, CMake version and what exactly you are building (ie: pkgs).

gvdhoorn gravatar imagegvdhoorn ( 2018-09-13 01:12:42 -0500 )edit

Added additional information in the post above

baronep gravatar imagebaronep ( 2018-09-13 10:03:36 -0500 )edit

What does your memory usage look like during that time?

David Lu gravatar imageDavid Lu ( 2018-09-13 10:07:05 -0500 )edit

Added additional information above

baronep gravatar imagebaronep ( 2018-09-13 11:07:20 -0500 )edit

Just an observation, if I comment out the costmap_2d_cloud and costmap_2d_markers targets in the CMakelists.txt, my build reduces from >20min to ~3min. What about these 2 targets would make them have such a dramatic impact? voxel_grid?

baronep gravatar imagebaronep ( 2018-09-14 13:12:46 -0500 )edit

2 Answers

Sort by ยป oldest newest most voted
0

answered 2018-09-19 16:27:07 -0500

David Lu gravatar image

I've replicated your problem. It's hanging on the linking step, but you can fix it by adding ${catkin_LIBRARIES} to the target_link_libraries command for costmap-2d_cloud and markers. Not sure why yet, but I'm going to investigate and make a PR.

edit flag offensive delete link more

Comments

0

answered 2018-09-17 11:10:36 -0500

baronep gravatar image

I was able to reduce my build time of the navigation stack by 75% by simply disabling the following targets in the costmap_2d CMakelists.txt

#add_executable(costmap_2d_markers src/costmap_2d_markers.cpp)
#add_dependencies(costmap_2d_markers ${${PROJECT_NAME}_EXPORTED_TARGETS} ${catkin_EXPORTED_TARGETS})
#target_link_libraries(costmap_2d_markers
#    costmap_2d
#    )

#add_executable(costmap_2d_cloud src/costmap_2d_cloud.cpp)
#add_dependencies(costmap_2d_cloud ${${PROJECT_NAME}_EXPORTED_TARGETS} ${catkin_EXPORTED_TARGETS})
#target_link_libraries(costmap_2d_cloud
#    costmap_2d
#    )

...

install( TARGETS
#    costmap_2d_markers
#    costmap_2d_cloud
    costmap_2d_node
    DESTINATION ${CATKIN_PACKAGE_BIN_DESTINATION}
)
edit flag offensive delete link more

Comments

You should also test a newer gcc version, 5.4 is quite old now!

VictorLamoine gravatar imageVictorLamoine ( 2018-10-08 06:25:57 -0500 )edit

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Question Tools

4 followers

Stats

Asked: 2018-09-12 19:29:41 -0500

Seen: 368 times

Last updated: Sep 19 '18