Crash running moveit move_group node
Hi All. I'm running a UR5 simulation, using ROS Kinetic with MoveIt! and gazebo, but the moveitrosmove_group node crashes on start-up.
A backtrace shows an illegal instruction, seemingly issued by libmoveitcollisiondetection_fcl.so:
Thread 1 "move_group" received signal SIGILL, Illegal instruction.
0x00007fffeb76b430 in fcl::AABB::AABB() () from /usr/lib/x86_64-linux-gnu/libfcl.so.0.5
(gdb) bt
#0 0x00007fffeb76b430 in fcl::AABB::AABB() () from /usr/lib/x86_64-linux-gnu/libfcl.so.0.5
#1 0x00007fffef50f139 in fcl::BVHModel<fcl::OBBRSS>::BVHModel() () from /opt/ros/kinetic/lib/libmoveit_collision_detection_fcl.so.0.9.1
I followed the instructions at http://wiki.ros.org/ur_gazebo initially and the above error comes when the movegroup node is called from within the ur5moveitplanningexecution.launch file. Running 'rosrun moveitrosmovegroup movegroup' directly gives the same error and stack trace.
I've tried using MoveIt from within the Kinetic distro and tried updating from the shadow-fixed server, and also tried building MoveIt from source, but the behaviour is the same.
I'm running Linux Ubuntu, version details: 4.4.0-43-generic (buildd@lgw01-22) (gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.2)
Any advice on what to try next would be greatly appreciated!
System details:
lsb_release -a
LSB Version: core-9.20160110ubuntu0.2-amd64:core-9.20160110ubuntu0.2-noarch:security-9.20160110ubuntu0.2-amd64:security-9.20160110ubuntu0.2-noarch
Distributor ID: Ubuntu
Description: Ubuntu 16.04.1 LTS
Release: 16.04
Codename: xenial
Cpu:
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Core(TM) i7 CPU 950 @ 3.07GHz
stepping : 5
microcode : 0xf
cpu MHz : 1867.000
cache size : 8192 KB
physical id : 0
siblings : 8
core id : 0
cpu cores : 4
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm sse4_1 sse4_2 popcnt lahf_lm tpr_shadow vnmi flexpriority ept vpid dtherm ida
bugs :
bogomips : 6141.46
clflush size : 64
cache_alignment : 64
address sizes : 36 bits physical, 48 bits virtual
power management:
EDIT - more info:
installing debug files, gdb shows the problem location:
#0 fcl::AABB::AABB (this=0x92fad8) at /var/lib/jenkins/workspace/fcl0.5-pkg_builder-master-generic/repo/src/BV/AABB.cpp:47
gdb dissassemly output gives the problem as the command "vmovsd %xmm2,(%rdx,%rax,8) ", which suggests the library has been compiled on an architecture with an 'avx' flag on the cpu, which my system doesn't have
Asked by Nic_G on 2016-11-09 14:29:21 UTC
Answers
The binary version of Flexible Collision Library, libfcl.so.0.5, was incompatible with my processors; we think it requires a processor with an 'avx' flag for vector processing.
The fix was to rebuild libfcl from source. I cloned and built from https://github.com/flexible-collision-library/fcl.git, using branch origin/fcl-0.5 (master branch builds to 0.6.0, which fails with the moveit binaries). Summary of steps:
fetch source code
git clone https://github.com/flexible-collision-library/fcl.git
git checkout origin/fcl-0.5
Build as described at https://github.com/flexible-collision-library/fcl
Re-point symlink:
sudo ln -f -s [local fcl repo location]/build/lib/libfcl.so.0.5.0 /usr/lib/x86_64-linux-gnu/libfcl.so.0.5
Asked by Nic_G on 2016-11-10 09:16:46 UTC
Comments
To avoid symlinking, I've also written instructions for building FCL for MoveIt! within your catkin workspace, which I think is pretty handy: http://moveit.ros.org/install/source/dependencies/
Asked by Dave Coleman on 2016-11-10 13:57:50 UTC
Comments
This is interesting. Can you tell us your cpu arch (x86, x64, something else) and OS version (
lsb_release -a
)?Note that the
universal_robot
packages are not necessarily Kinetic compatible, but I don't think that's what causing you issues here.Asked by gvdhoorn on 2016-11-09 15:32:43 UTC
Perhaps this is related to FCL's use of SSE instructions? Just a thought.
Asked by rbbg on 2016-11-10 02:35:27 UTC
Could be, that's why I asked more info on cpu arch, etc.
Asked by gvdhoorn on 2016-11-10 02:46:36 UTC
Thanks for the reply, lsb_release -a output added to the question
Asked by Nic_G on 2016-11-10 04:53:43 UTC
And the cpu type? can you post the output of
cat /proc/cpuinfo
? (just a single processor is fine)Asked by rbbg on 2016-11-10 05:38:53 UTC
processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 26 model name : Intel(R) Core(TM) i7 CPU 950 @ 3.07GHz stepping : 5 microcode : 0xf cpu MHz : 1867.000 cache size : 8192 KB physical id : 0 siblings : 8 core id
Asked by Nic_G on 2016-11-10 05:56:04 UTC
You're last update suggests something might be wrong with the binary distribution of
libfcl
. Perhaps it'd be good to report this over at flexible-collision-library/fcl/issues. The MoveIt issue tracker might also be good to notify.Asked by gvdhoorn on 2016-11-10 07:33:30 UTC
Provided this really is an
SIGILL
due to miscompilation (and not a code corruption issue), this is serious and needs to be addressed. An i7-950 should definitely be capable of running MoveIt/FCL.Asked by gvdhoorn on 2016-11-10 07:34:39 UTC
Good point gvdhoorn, I'll post the issue to both so people are aware
Asked by Nic_G on 2016-11-10 09:04:37 UTC
Posted to https://github.com/flexible-collision-library/fcl/issues/187 and https://github.com/ros-planning/moveit/issues/345
Asked by Nic_G on 2016-11-10 09:36:11 UTC
This is really surprising since I've using an Intel i7 with the latest Kinetic for months. my system: https://gist.github.com/davetcoleman/5458a59bd28ff143b2f164002c93457d
Asked by Dave Coleman on 2016-11-10 13:59:45 UTC
@Dave Coleman: the
flags
key in yourcpuinfo
output contains theAVX
flag. An i7-950 does not support those instructions.Asked by gvdhoorn on 2016-11-10 14:09:01 UTC
That makes sense now, thanks
Asked by Dave Coleman on 2016-11-10 17:45:10 UTC