ROS Resources: Documentation | Support | Discussion Forum | Index | Service Status | ros @ Robotics Stack Exchange
Ask Your Question
0

ndt_gpu failed autoware!!

asked 2019-08-11 12:40:18 -0500

AM97 gravatar image

updated 2019-08-14 18:03:12 -0500

jayess gravatar image

I am unable to build autoware with GPU support. I just downgraded CUDA to 10.0 and tried to built by using CUDA Support.

Operating System: Ubuntu 18.04 ROS: Melodic GCC version: GCC (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0 CMake version : 3.10.2 ROS distribution: melodic ROS installation type: sudo apt-get install ros-melodic-desktop-full GPU MODEL: GeForce GTX 1060M CUDA Version: 10 CUDNN Version: 7.5 Autoware Installation type : Built from source (https://gitlab.com/autowarefoundation...) Autoware Version : 1.12.0 EIgen Version : 3.3.7

:~$ nvidia-smi
Sun Aug 11 19:09:22 2019       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 415.27       Driver Version: 415.27       CUDA Version: 10.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 1060    Off  | 00000000:01:00.0 Off |                  N/A |
| N/A   68C    P0    28W /  N/A |    558MiB /  6078MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0       505      G   ...uest-channel-token=17405615527381595598   112MiB |
|    0     22094      G   /usr/lib/xorg/Xorg                           309MiB |
|    0     22229      G   /usr/bin/gnome-shell                         133MiB |
+-----------------------------------------------------------------------------+
~$ pkg-config --modversion eigen3
3.3.7

I am getting following error when I am building using cuda support.

:~/autoware.ai$ AUTOWARE_COMPILE_WITH_CUDA=1 colcon build --cmake-args -DCMAKE_BUILD_TYPE=Release
Starting >>> autoware_build_flags
Starting >>> autoware_msgs
Starting >>> vector_map_msgs
Starting >>> autoware_config_msgs
Starting >>> tablet_socket_msgs
Starting >>> autoware_system_msgs
Starting >>> autoware_can_msgs
Starting >>> gnss
Finished <<< autoware_build_flags [2.01s]                                                                                                                   
Starting >>> sick_ldmrs_description
Finished <<< gnss [2.43s]                                                                                                                           
Starting >>> sick_ldmrs_msgs
Finished <<< sick_ldmrs_description [0.75s]                                                                                                                     
Starting >>> ds4_msgs
Finished <<< autoware_can_msgs [3.27s]                                                                                                                                 
Starting >>> custom_msgs
Finished <<< tablet_socket_msgs [3.57s]                                                                                                               
Starting >>> ds4_driver
Finished <<< autoware_system_msgs [3.81s]                                                                                                             
Starting >>> autoware_health_checker
Finished <<< ds4_msgs [1.57s]                                                                                                                         
Starting >>> gazebo_camera_description
Finished <<< ds4_driver [0.86s]                                                                                                                       
Starting >>> gazebo_imu_description
Finished <<< sick_ldmrs_msgs [2.09s]                                                                                                                  
Starting >>> sick_ldmrs_driver
Finished <<< gazebo_camera_description [1.11s]                                                                                                        
Starting >>> ndt_cpu
Finished <<< gazebo_imu_description [1.11s]                                                                                                            
Starting >>> ndt_tku
Finished <<< ndt_cpu [0.67s]                                                                                                                           
Starting >>> pcl_omp_registration
Finished <<< ndt_tku [0.64s]                                                                                                                           
Starting >>> fastvirtualscan
Finished <<< autoware_config_msgs [6.91s]                                                                                                                
Starting >>> kitti_player
Finished <<< pcl_omp_registration [0.83s]
Starting >>> ndt_gpu
Finished <<< custom_msgs [3.70s]                                                                                                                
Starting >>> rosinterface
Finished <<< fastvirtualscan [0.78s]
Starting >>> autoware_bag_tools
Finished <<< rosinterface [1.31s]                                                                                                                         
Starting >>> autoware_external_msgs
--- stderr: ndt_gpu                                                                                                                                       
** WARNING ** io features related to ensenso will be disabled
** WARNING ** io features related to davidSDK will be disabled
** WARNING ** io features related to dssdk will be disabled
** WARNING ** io features related to pcap will be disabled
** WARNING ** io features related to png will be disabled
** WARNING ** io features related to libusb-1.0 will be disabled
** WARNING ** visualization features related to ensenso will be disabled
** WARNING ** visualization features related to davidSDK will be disabled
** WARNING ** visualization features related to dssdk will be disabled
** WARNING ** visualization features related to rssdk will be disabled
CMake Error at /home/sai/autoware.ai/install/autoware_build_flags/share/autoware_build_flags/cmake/autoware_build_flags-extras.cmake:32 (message):
  GPU support on Melodic requires Eigen version>= 3.3.7
Call Stack (most recent ...
(more)
edit retag flag offensive close merge delete

Comments

I thought we might start to run into CUDA compatability problems soon. Can you please run the following commands and provide the output? 

./opt/cuda/bin/nvcc --version
./usr/local/cuda/bin/nvcc --version
uname -r
lsb_release -a
apt search cuda | grep installed
Josh Whitley gravatar image Josh Whitley  ( 2019-08-12 22:33:31 -0500 )edit

@Maximus5684 uname -r 4.15.0-55-generic lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 18.04.3 LTS Release: 18.04 Codename: bionic /usr/local/cuda/bin$ nvcc --version nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2017 NVIDIA Corporation Built on Fri_Nov__3_21:07:56_CDT_2017 Cuda compilation tools, release 9.1, V9.1.85

AM97 gravatar image AM97  ( 2019-08-13 01:41:33 -0500 )edit

@Maximus5684 apt search cuda | grep installed

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

cuda-command-line-tools-10-0/unknown,now 10.0.130-1 amd64 [installed,auto-removable] cuda-compiler-10-0/unknown,now 10.0.130-1 amd64 [installed,auto-removable] cuda-cublas-10-0/unknown,now 10.0.130-1 amd64 [installed,auto-removable] cuda-cublas-dev-10-0/unknown,now 10.0.130-1 amd64 [installed,auto-removable] cuda-cudart-10-0/unknown,now 10.0.130-1 amd64 [installed,auto-removable] cuda-cudart-dev-10-0/unknown,now 10.0.130-1 amd64 [installed,auto-removable] cuda-cufft-10-0/unknown,now 10.0.130-1 amd64 [installed,auto-removable] cuda-cufft-dev-10-0/unknown,now 10.0.130-1 amd64 [installed,auto-removable] cuda-cuobjdump-10-0/unknown,now 10.0.130-1 amd64 [installed,auto-removable] cuda-cupti-10-0/unknown,now 10.0.130-1 amd64 [installed,auto-removable] cuda-curand-10-0/unknown,now 10.0.130-1 amd64 [installed,auto-removable] cuda-curand-dev-10-0/unknown,now 10.0.130-1

AM97 gravatar image AM97  ( 2019-08-13 01:41:59 -0500 )edit

@Maximus5684 I changed so that no mismatches are there nvcc --version nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2018 NVIDIA Corporation Built on Sat_Aug_25_21:08:01_CDT_2018 Cuda compilation tools, release 10.0, V10.0.130

but still I am getting same problem!!!!!!

AM97 gravatar image AM97  ( 2019-08-13 13:09:47 -0500 )edit

@AM97 - Please also give me the output of uname -r. I need to know which kernel version you are running. Also, make sure that there are not multiple versions of CUDA installed. You can check to see if there are multiple NVCCs by doing:

sudo updatedb
sudo locate nvcc
Josh Whitley gravatar image Josh Whitley  ( 2019-08-14 13:02:41 -0500 )edit

@Maximus5684 uname -r 4.15.0-55-generic and the output of sudo locate nvcc is in this gistfile (https://gist.github.com/sandeepmaddul...)

AM97 gravatar image AM97  ( 2019-08-14 13:11:05 -0500 )edit

The version in /usr/bin/nvcc is a problem - this file belongs to the Ubuntu package nvidia-cuda-toolkit which is an old version of CUDA (9.1). Please run the following command:

sudo apt purge nvidia-cuda-*
sudo apt autoremove

and then clean (delete build/, install/, and log/) your Autoware workspace, and re-build using

AUTOWARE_COMPILE_WITH_CUDA=1 colcon build --cmake-args -DCMAKE_BUILD_TYPE=Debug
Josh Whitley gravatar image Josh Whitley  ( 2019-08-14 13:26:03 -0500 )edit
add a comment

1 Answer

Sort by ยป oldest newest most voted
1

answered 2019-08-14 14:36:40 -0500

Josh Whitley gravatar image

updated 2019-08-14 14:38:20 -0500

@AM97 You are currently using the version of CUDA that is provided by Ubuntu. On Ubuntu 18.04, you MUST install version 10.0 directly from Nvidia. To make sure that you do not have the version from Ubuntu, do the following:

sudo apt purge nvidia-cuda*
sudo apt autoremove

Then, to install 10.0, carefullyfollow the instructions at https://docs.nvidia.com/cuda/archive/.... Do not install the nvidia-cuda-toolkit or nvidia-cuda-dev packages from the Ubuntu repositories - these install CUDA 9.1, which is not compatible with Autoware on Ubuntu 18.04.

In addition, on Ubuntu 18.04, you must upgrade the version of Eigen that is provided with Ubuntu by following the instructions at https://gitlab.com/autowarefoundation....

After the above is done, you should be able to build with AUTOWARE_COMPILE_WITH_CUDA=1 to enable CUDA capabilities.

edit flag offensive delete link more

Comments

@Maximus5684 Thanks for the help. After some time I found it now I installed directly from NVidia. Now it was built successfully. Thanks a lot for your quick help.

AM97 gravatar image AM97  ( 2019-08-14 14:38:52 -0500 )edit

It is just a matter of time until NVidia breaks everything again xD https://www.youtube.com/watch?v=_36yN...

Martin Peris gravatar image Martin Peris  ( 2019-08-15 01:47:04 -0500 )edit
add a comment

Question Tools

2 followers

subscribe to rss feed

Stats

Asked: 2019-08-11 12:40:18 -0500

Seen: 4,217 times

Last updated: Aug 14 '19

Related questions

Sample rosbag data for Traffic Light Recognition module

Convert ctrl_cmd to pedal position

SLAM in AUTOWARE using 3 VLP16

AttributeError's when running Python scripts

ROS Melodic OpenCV xfeatures2d

ROS TCP Connector C# Lost Publishers & Freezing

Controlling a quadcopter in Gazebo

error getMD5Sum in customized message type

Navigation without lidar

URDF model using the a variable parameter

ROS Answers is licensed under Creative Commons Attribution 3.0 Content on this site is licensed under a Creative Commons Attribution Share Alike 3.0 license.
about | faq | help | privacy policy | terms of service
Powered by Askbot version 0.10.2