Slow build time

asked 2018-09-12 19:29:41 -0500

baronep
106 ●5 ●10 ●13

updated 2018-09-14 13:35:14 -0500

I am experiencing very slow builds (and I have been for a while) using catkin_make building the ROS navigation stack. I have tried -DCMAKE_BUILD_TYPE=Release, --use-ninga, ccache, etc but the problem still persists. Specifically, clean builds sometimes take me over an hour on a modern machine (i7 w/ 8GB RAM).

The build process proceeds rather quickly until it seems to stall out at 100%. A quick htop reveals that the culprit is a couple of linking threads that are taking 100% CPU and often are costmap_2d or trajectory planner related.

Has anyone seen this before or have any insight on how to fix this?

Additional Information

OS: Ubuntu 16.04.5 LTS  
gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.10)  
cmake version 3.5.1  
ROS Version: Kinetic

ROS Install: ROS installed from .debs (desktop-full) and upgraded w/ apt

Steps to reproduce:

mkdir -p ~/test/src  
cd ~/test/src  
catkin_init_workspace  
git clone https://github.com/ros-planning/navigation.git  
git checkout kinetic-devel  
cd ..
catkin_make  
... wait ...

Total Build Time: 22.5 min

The build got to 100% after <5min with the last message printed ... `

[100%] Built target move_base_node

Then 15 min later ...

[100%] Built target costmap_2d_cloud
[100%] Built target costmap_2d_markers
Base path: /home/patrick/test
Source space: /home/patrick/test/src
Build space: /home/patrick/test/build
Devel space: /home/patrick/test/devel
Install space: /home/patrick/test/install
####
#### Running command: "cmake /home/patrick/test/src -DCATKIN_DEVEL_PREFIX=/home/patrick/test/devel -DCMAKE_INSTALL_PREFIX=/home/patrick/test/install -G Unix Makefiles" in "/home/patrick/test/build"
####
####
#### Running command: "make -j6 -l6" in "/home/patrick/test/build"
####

The full output of catkin_make is available here ... https://pastebin.com/ZkFUqU2s

Additional Information - Second Round

Memory consumption during the first 5 minutes of the build ...

patrick@patrick:~/test$ free -mh
              total        used        free      shared  buff/cache   available
Mem:           7.2G        4.1G        222M        1.1G        2.9G        1.5G
Swap:          7.4G        3.2G        4.3G

After reaching the 100% ...

patrick@patrick:~/test$ free -mh
              total        used        free      shared  buff/cache   available
Mem:           7.2G        4.6G        180M        1.2G        2.5G        1.0G
Swap:          7.4G        3.2G        4.2G

It looks like there should be plenty of free memory right?

Also, here are the processes that are "stalling" out in the second 15min of the build ... https://pastebin.com/CDb2qLQs

Specifically, the /usr/bin/ld threads (there are 2, costmap_2d_cloud and costmap_2d_markers) which are taking 100% CPU

edit retag flag offensive close merge delete

Comments

Please add some information about your system: which OS is this, which version, how did you install ROS (.debs or completely from source), which and what version of compiler do you have installed, CMake version and what exactly you are building (ie: pkgs).

gvdhoorn ( 2018-09-13 01:12:42 -0500 )edit

Added additional information in the post above

baronep ( 2018-09-13 10:03:36 -0500 )edit

What does your memory usage look like during that time?

David Lu ( 2018-09-13 10:07:05 -0500 )edit

Added additional information above

baronep ( 2018-09-13 11:07:20 -0500 )edit

Just an observation, if I comment out the costmap_2d_cloud and costmap_2d_markers targets in the CMakelists.txt, my build reduces from >20min to ~3min. What about these 2 targets would make them have such a dramatic impact? voxel_grid?

baronep ( 2018-09-14 13:12:46 -0500 )edit

add a comment

0

answered 2018-09-19 16:27:07 -0500

David Lu

10932 ●123 ●219 ●250 http://MetroRobots.com/

I've replicated your problem. It's hanging on the linking step, but you can fix it by adding ${catkin_LIBRARIES} to the target_link_libraries command for costmap-2d_cloud and markers. Not sure why yet, but I'm going to investigate and make a PR.

edit flag offensive delete link

Comments

https://github.com/ros-planning/navig...

David Lu ( 2018-10-12 08:10:30 -0500 )edit

add a comment

0

answered 2018-09-17 11:10:36 -0500

baronep
106 ●5 ●10 ●13

I was able to reduce my build time of the navigation stack by 75% by simply disabling the following targets in the costmap_2d CMakelists.txt

#add_executable(costmap_2d_markers src/costmap_2d_markers.cpp)
#add_dependencies(costmap_2d_markers ${${PROJECT_NAME}_EXPORTED_TARGETS} ${catkin_EXPORTED_TARGETS})
#target_link_libraries(costmap_2d_markers
#    costmap_2d
#    )

#add_executable(costmap_2d_cloud src/costmap_2d_cloud.cpp)
#add_dependencies(costmap_2d_cloud ${${PROJECT_NAME}_EXPORTED_TARGETS} ${catkin_EXPORTED_TARGETS})
#target_link_libraries(costmap_2d_cloud
#    costmap_2d
#    )

...

install( TARGETS
#    costmap_2d_markers
#    costmap_2d_cloud
    costmap_2d_node
    DESTINATION ${CATKIN_PACKAGE_BIN_DESTINATION}
)

edit flag offensive delete link

Comments

You should also test a newer gcc version, 5.4 is quite old now!

VictorLamoine ( 2018-10-08 06:25:57 -0500 )edit

add a comment

Slow build time

Additional Information

Additional Information - Second Round

Comments

2 Answers

Comments

Comments

Question Tools

Stats

Related questions

Slow build time edit

Additional Information

Additional Information - Second Round

Comments

2 Answers

Comments

Comments

Question Tools

Stats

Related questions

Slow build time