Kinetic Kame build very slow

I recently upgrade my machine from Ubuntu 14 to Ubuntu 16 and upgrade from ROS Jade to ROS Kinetic as well. I pulled my repository and built and every thing built just fine, tests seem to pass, however, the build is now about 5 times slower! Has anyone experienced this? Any idea what could be causing this and how to fix it?

Asked by JakeBaldwin on 2016-10-14 13:12:47 UTC

Comments

Are you referring to the actual build (ie: compilation, linking), or the output of the build (ie: binaries)? If the former, see if deleting your build and devel folders clears things up. If the latter, have you set the same CMAKE_BUILD_TYPE?

Asked by gvdhoorn on 2016-10-14 13:28:04 UTC

It is the former, the compilation and linking. Deleting the build/ and devel/ directories makes no difference. My build went from about 5 or 6 min to 30 min!

Asked by JakeBaldwin on 2016-10-14 13:46:10 UTC

That is strange indeed. Any particular pkgs that you notice it to have become much slower?

Asked by gvdhoorn on 2016-10-15 06:11:34 UTC

I've just noticed the same thing! My first guess is that there are a lot of new warnings when building against ROS Kinetic...

Asked by jonfink on 2016-10-26 12:40:07 UTC

After some further testing, I should add that I am comparing the performance of (Ubuntu 16.04 + ROS Kinetic + Clang-3.8) with (Ubuntu 14.04 + ROS Jade + Clang-3.4) and seeing an individual package build time go from 40 seconds with Jade/etc to 65 seconds with Kinetic/etc.

Asked by jonfink on 2016-10-26 14:57:17 UTC

Is this with catkin_tools or catkin_make?

Asked by gvdhoorn on 2016-10-27 04:42:37 UTC

Catkin tools for me. Same version of catkin tools on both systems. Going to try with 'catkin_make' today.

Asked by jonfink on 2016-10-27 05:59:09 UTC

Just curious if there would be a difference between the two.

My first guess is that there are a lot of new warnings when building against ROS Kinetic...

That is a possibility. Running catkin in verbose mode on both distributions should allow us to see whether that is the case.

Asked by gvdhoorn on 2016-10-27 06:41:12 UTC

Trying to standardize my tests, costmap_2d from source (+ map_server, voxel_grid)

Trusty + Jade (catkin-tools v0.4.2, catkin v0.6.18)
Xenial + Kinetic (catkin-tools v0.4.2, catkin v0.7.4)

1 + catkin build = 33s 2 + catkin build = 2m57s 1 + catkin_make = 37s 2 + catkin_make = (seems to stall)

Asked by jonfink on 2016-10-27 08:21:59 UTC

Same FS? Same HW?

Asked by gvdhoorn on 2016-10-27 08:33:16 UTC

Yes, sorry (finding the character limit here quite burdensome...)

Builds are happening in docker containers on the same system, same source code. I should note that above results were with default gcc. Switching to clang, there is a consistent factor of 2 slow down from Jade->Kinetic.

Asked by jonfink on 2016-10-27 08:50:53 UTC

Interesting. Perhaps we should move this to discourse.ros.org: no char limit and more exposure.

Asked by gvdhoorn on 2016-10-27 09:12:15 UTC

That might be a good idea - I'm a little worried this isn't a ROS issue, per se, but a compiler issue (of sorts). Clang goes from version 3.4 to 3.8 (Trusty->Xenial) and GCC goes from 4.8 to 5..

Though I am surprised more people haven't noticed similar effects...

Asked by jonfink on 2016-10-27 09:15:34 UTC

I'd put it in the Uncategorized category.

I'm a little worried this isn't a ROS issue, per se, but a compiler issue (of sorts).

Yeah, thinking that as well. Always good to get some more input on this.

If only to give @JakeBaldwin a proper answer.

Asked by gvdhoorn on 2016-10-27 09:17:27 UTC