Robotics StackExchange | Archived questions

ntd_gpu autoware build from source failed

Hello,

I tried to install Autoware build from source https://github.com/Autoware-AI/autoware.ai/wiki/Source-Build , version 1.12.0 but failed.
System information: Ubuntu 16.04
Memory 15.5 GiB
Processor Intel® Core™ i7-7700HQ CPU @ 2.80GHz × 8
Graphics GeForce GTX 1060/PCIe/SSE2
OS type 64-bit
Disk 83.0 GB

ROS version
mb@mb-Alienware-15-R3:~/autoware.ai$ env | grep ROS
ROSROOT=/opt/ros/kinetic/share/ros
ROS
PACKAGEPATH=/opt/ros/kinetic/share
ROS
MASTERURI=http://localhost:11311
ROS
PYTHONVERSION=2
ROS
VERSION=1
ROSLISPPACKAGEDIRECTORIES=
ROSDISTRO=kinetic
ROS
ETC_DIR=/opt/ros/kinetic/etc/ros

nvidia-smi

Wed Jan 27 18:28:07 2021
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.27.04 Driver Version: 460.27.04 CUDA Version: 11.2 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 GeForce GTX 1060 Off | 00000000:01:00.0 On | N/A |
| N/A 43C P5 11W / N/A | 425MiB / 6073MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 1086 G /usr/lib/xorg/Xorg 324MiB |
| 0 N/A N/A 2003 G compiz 90MiB |
| 0 N/A N/A 2567 G /usr/lib/firefox/firefox 1MiB |
| 0 N/A N/A 2686 G /usr/lib/firefox/firefox 1MiB |
| 0 N/A N/A 8084 G unity-control-center 1MiB |
| 0 N/A N/A 10189 G /usr/lib/firefox/firefox 1MiB |
+-----------------------------------------------------------------------------+

NVCC -V

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2015 NVIDIA Corporation
Built on TueAug1114:27:32CDT_2015
Cuda compilation tools, release 7.5, V7.5.17

Qt version:
Using Qt version 5.5.1 in /usr/lib/x86_64-linux-gnu

Issue: it generates ntd_gpu issue but every time it generates different packages processed (sometimes 99 packages not processed, sometimes 97, sometimes 9, etc,...)

mb@mb-Alienware-15-R3:~/autoware.ai$ AUTOWARECOMPILEWITHCUDA=1 colcon build --cmake-args -DCMAKEBUILDTYPE=Release Starting >>> autowarebuildflags Starting >>> autowaremsgs
Starting >>> vectormapmsgs Starting >>> autowareconfigmsgs
Starting >>> tabletsocketmsgs Starting >>> autowaresystemmsgs Starting >>> autowarecanmsgs Starting >>> gnss Finished <<< autowarebuildflags [17.3s]
Starting >>> sickldmrsdescription Finished <<< gnss [15.5s]
Starting >>> sickldmrsmsgs Finished <<< tabletsocketmsgs [15.9s]
Finished <<< autowaresystemmsgs [16.7s]
Starting >>> autowarehealthchecker Starting >>> ds4msgs
Finished <<< autoware
canmsgs [16.7s] Starting >>> custommsgs Finished <<< autowareconfigmsgs [17.7s]
Starting >>> ds4driver Finished <<< autowaremsgs [19.4s]
Starting >>> amathutilslib Finished <<< vectormapmsgs [20.2s]
Starting >>> vector
map Finished <<< sickldmrsdescription [5.35s]
Starting >>> astarsearch Finished <<< sickldmrsmsgs [6.33s]
Finished <<< ds4
msgs [5.20s] Starting >>> sickldmrsdriver Starting >>> gazebocameradescription Finished <<< ds4driver [5.87s]
Starting >>> gazebo
imudescription Finished <<< custommsgs [7.04s]
Starting >>> ndtcpu Finished <<< vectormap [6.62s]
Starting >>> vectormapserver Finished <<< gazebocameradescription [5.32s]
Starting >>> mapfile
Finished <<< gazebo
imudescription [3.78s] Finished <<< amathutilslib [8.07s]
Starting >>> ndttku Starting >>> pclompregistration Finished <<< autowarehealthchecker [12.3s]
Starting >>> waypoint
follower Finished <<< ndtcpu [7.82s]
Starting >>> fastvirtualscan Finished <<< pcl
ompregistration [7.78s]
Starting >>> kitti
boxpublisher Finished <<< ndttku [8.57s]
Starting >>> kittiplayer Finished <<< fastvirtualscan [7.50s]
Starting >>> ndt
gpu Finished <<< sickldmrsdriver [17.3s] Starting >>> rosinterface Finished <<< astarsearch [18.5s]
Starting >>> runtime
manager Finished <<< mapfile [24.3s]
Starting >>> state
machinelib Finished <<< kittiboxpublisher [20.7s]
Starting >>> imm
ukfpdatrack Finished <<< rosinterface [23.0s]
Finished <<< runtimemanager [23.1s]
Starting >>> libvectormap Starting >>> object
map
Finished <<< statemachinelib [11.7s]
Starting >>> sickldmrstools Finished <<< waypointfollower [34.6s]
Finished <<< vector
mapserver [36.8s]
Starting >>> op
utility Starting >>> laneplanner
Finished <<< kitti
player [52.1s]
Starting >>> waypointplanner --- stderr: ndtgpu
ptxas info : 0 bytes gmem /usr/include/string.h: In function ‘void* mempcpyinline(void, const void, sizet)’: /usr/include/string.h:652:42: error: ‘memcpy’ was not declared in this scope return (char *) memcpy (dest, _src, _n) + _n; ^ CMake Error at ndtgpugeneratedSymmetricEigenSolver.cu.o.cmake:266 (message): Error generating file /home/mb/autoware.ai/build/ndtgpu/CMakeFiles/ndtgpu.dir/src/./ndtgpugenerated_SymmetricEigenSolver.cu.o

make[2]: *** [CMakeFiles/ndtgpu.dir/src/ndtgpugeneratedSymmetricEigenSolver.cu.o] Error 1 make[1]: *** [CMakeFiles/ndt_gpu.dir/all] Error 2

make: *** [all] Error 2

Failed <<< ndtgpu [1min 8s, exited with code 2] Aborted <<< sickldmrstools [43.4s] Aborted <<< immukfpdatrack [50.8s] Aborted <<< libvectormap [44.4s] Aborted <<< laneplanner [43.2s]
Aborted <<< object
map [44.5s] Aborted <<< oputility [43.3s]
Aborted <<< waypoint
planner [23.2s]
Summary: 32 packages finished [2min 1s] 1 package failed: ndtgpu
7 packages aborted: imm
ukfpdatrack laneplanner libvectormap objectmap oputility sickldmrstools waypointplanner
1 package had stderr output: ndt_gpu
99 packages not processed

Asked by Mai Bui on 2021-01-27 19:54:49 UTC

Comments

This looks like a CUDA problem. What version of CUDA do you have installed?

Asked by Josh Whitley on 2021-01-30 16:41:01 UTC

It’s 11.2 version of CUDA

Asked by Mai Bui on 2021-01-30 16:43:19 UTC

According to nvcc -v command, it seems that your cuda is at 7.5. Based on requirements from https://gitlab.com/autowarefoundation/autoware.ai/autoware/-/wikis/Source-Build, for Ubuntu 16.04, version 9.0 is required.

For number of package differs at each compilation, this is not the issue, at least based on my experience. As multiple threads is used for compilation, there are some package that bound to be completed early (really depends on the system scheduler).

Asked by kurogane1031 on 2021-02-02 00:29:26 UTC

Answers