Robotics StackExchange | Archived questions

Crash running moveit move_group node

Hi All. I'm running a UR5 simulation, using ROS Kinetic with MoveIt! and gazebo, but the moveitrosmove_group node crashes on start-up.

A backtrace shows an illegal instruction, seemingly issued by libmoveitcollisiondetection_fcl.so:

 Thread 1 "move_group" received signal SIGILL, Illegal instruction.
 0x00007fffeb76b430 in fcl::AABB::AABB() () from /usr/lib/x86_64-linux-gnu/libfcl.so.0.5
 (gdb) bt
 #0  0x00007fffeb76b430 in fcl::AABB::AABB() () from /usr/lib/x86_64-linux-gnu/libfcl.so.0.5
 #1  0x00007fffef50f139 in fcl::BVHModel<fcl::OBBRSS>::BVHModel() () from /opt/ros/kinetic/lib/libmoveit_collision_detection_fcl.so.0.9.1

I followed the instructions at http://wiki.ros.org/ur_gazebo initially and the above error comes when the movegroup node is called from within the ur5moveitplanningexecution.launch file. Running 'rosrun moveitrosmovegroup movegroup' directly gives the same error and stack trace.

I've tried using MoveIt from within the Kinetic distro and tried updating from the shadow-fixed server, and also tried building MoveIt from source, but the behaviour is the same.

I'm running Linux Ubuntu, version details: 4.4.0-43-generic (buildd@lgw01-22) (gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.2)

Any advice on what to try next would be greatly appreciated!

System details:

lsb_release -a          
LSB Version:    core-9.20160110ubuntu0.2-amd64:core-9.20160110ubuntu0.2-noarch:security-9.20160110ubuntu0.2-amd64:security-9.20160110ubuntu0.2-noarch
Distributor ID: Ubuntu
Description:    Ubuntu 16.04.1 LTS
Release:        16.04
Codename:       xenial

Cpu:

processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 26
model name      : Intel(R) Core(TM) i7 CPU         950  @ 3.07GHz
stepping        : 5
microcode       : 0xf
cpu MHz         : 1867.000
cache size      : 8192 KB
physical id     : 0
siblings        : 8
core id         : 0
cpu cores       : 4
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 11
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm sse4_1 sse4_2 popcnt lahf_lm tpr_shadow vnmi flexpriority ept vpid dtherm ida
bugs            :
bogomips        : 6141.46
clflush size    : 64
cache_alignment : 64
address sizes   : 36 bits physical, 48 bits virtual
power management:

EDIT - more info:

installing debug files, gdb shows the problem location:

#0  fcl::AABB::AABB (this=0x92fad8) at /var/lib/jenkins/workspace/fcl0.5-pkg_builder-master-generic/repo/src/BV/AABB.cpp:47

gdb dissassemly output gives the problem as the command "vmovsd %xmm2,(%rdx,%rax,8) ", which suggests the library has been compiled on an architecture with an 'avx' flag on the cpu, which my system doesn't have

Asked by Nic_G on 2016-11-09 14:29:21 UTC

Comments

This is interesting. Can you tell us your cpu arch (x86, x64, something else) and OS version (lsb_release -a)?

Note that the universal_robot packages are not necessarily Kinetic compatible, but I don't think that's what causing you issues here.

Asked by gvdhoorn on 2016-11-09 15:32:43 UTC

Perhaps this is related to FCL's use of SSE instructions? Just a thought.

Asked by rbbg on 2016-11-10 02:35:27 UTC

Could be, that's why I asked more info on cpu arch, etc.

Asked by gvdhoorn on 2016-11-10 02:46:36 UTC

Thanks for the reply, lsb_release -a output added to the question

Asked by Nic_G on 2016-11-10 04:53:43 UTC

And the cpu type? can you post the output of cat /proc/cpuinfo? (just a single processor is fine)

Asked by rbbg on 2016-11-10 05:38:53 UTC

processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 26 model name : Intel(R) Core(TM) i7 CPU 950 @ 3.07GHz stepping : 5 microcode : 0xf cpu MHz : 1867.000 cache size : 8192 KB physical id : 0 siblings : 8 core id

Asked by Nic_G on 2016-11-10 05:56:04 UTC

You're last update suggests something might be wrong with the binary distribution of libfcl. Perhaps it'd be good to report this over at flexible-collision-library/fcl/issues. The MoveIt issue tracker might also be good to notify.

Asked by gvdhoorn on 2016-11-10 07:33:30 UTC

Provided this really is an SIGILL due to miscompilation (and not a code corruption issue), this is serious and needs to be addressed. An i7-950 should definitely be capable of running MoveIt/FCL.

Asked by gvdhoorn on 2016-11-10 07:34:39 UTC

Good point gvdhoorn, I'll post the issue to both so people are aware

Asked by Nic_G on 2016-11-10 09:04:37 UTC

This is really surprising since I've using an Intel i7 with the latest Kinetic for months. my system: https://gist.github.com/davetcoleman/5458a59bd28ff143b2f164002c93457d

Asked by Dave Coleman on 2016-11-10 13:59:45 UTC

@Dave Coleman: the flags key in your cpuinfo output contains the AVX flag. An i7-950 does not support those instructions.

Asked by gvdhoorn on 2016-11-10 14:09:01 UTC

That makes sense now, thanks

Asked by Dave Coleman on 2016-11-10 17:45:10 UTC

Answers

The binary version of Flexible Collision Library, libfcl.so.0.5, was incompatible with my processors; we think it requires a processor with an 'avx' flag for vector processing.

The fix was to rebuild libfcl from source. I cloned and built from https://github.com/flexible-collision-library/fcl.git, using branch origin/fcl-0.5 (master branch builds to 0.6.0, which fails with the moveit binaries). Summary of steps:

fetch source code

git clone https://github.com/flexible-collision-library/fcl.git
git checkout origin/fcl-0.5

Build as described at https://github.com/flexible-collision-library/fcl

Re-point symlink:

sudo ln -f -s [local fcl repo location]/build/lib/libfcl.so.0.5.0 /usr/lib/x86_64-linux-gnu/libfcl.so.0.5

Asked by Nic_G on 2016-11-10 09:16:46 UTC

Comments

To avoid symlinking, I've also written instructions for building FCL for MoveIt! within your catkin workspace, which I think is pretty handy: http://moveit.ros.org/install/source/dependencies/

Asked by Dave Coleman on 2016-11-10 13:57:50 UTC