Autoware: crosscompile with CUDA support
Hi everyone! I was working with Autoware in my jetson tx2 board and everything is fine when I compile natively. Now, I want to be able to crosscompile Autoware within my desktop computer. I followed all the steps that I found in Autoware gitlab and forums and I was able to crosscompile Autoware 1.12 with the docker image. However, I could see that CUDA is not enabled and I really need it.
Is there any way to add CUDA support to docker crosscompilation? I'm pretty new with docker so it's hard for me to detect wheter some dependency is missing or if I have to add some environment variable. When I add AUTOWARECOMPILEWITH_CUDA=1 to crosscompile script I have the following error:
~/autoware.ai/ros$ ./colcon_release_cross kinetic generic-aarch64
Starting >>> autoware_build_flags
Starting >>> autoware_msgs
Starting >>> vector_map_msgs
Starting >>> autoware_config_msgs
Starting >>> tablet_socket_msgs
Starting >>> autoware_system_msgs
Starting >>> autoware_can_msgs
Starting >>> gnss
Finished <<< autoware_build_flags [16.3s]
Finished <<< gnss [16.4s]
Starting >>> ds4_msgs
Starting >>> custom_msgs
Finished <<< autoware_can_msgs [24.4s]
Finished <<< autoware_system_msgs [24.4s]
Finished <<< tablet_socket_msgs [24.4s]
Starting >>> ds4_driver
Starting >>> autoware_health_checker
Starting >>> gazebo_camera_description
Finished <<< autoware_config_msgs [31.0s]
Starting >>> gazebo_imu_description
Finished <<< autoware_msgs [31.0s]
Starting >>> amathutils_lib
Finished <<< ds4_msgs [16.9s]
Starting >>> astar_search
Finished <<< vector_map_msgs [35.8s]
Starting >>> vector_map
Finished <<< gazebo_camera_description [11.4s]
Finished <<< ds4_driver [11.4s]
Starting >>> ndt_cpu
Starting >>> ndt_tku
Finished <<< gazebo_imu_description [16.4s]
Finished <<< custom_msgs [30.8s]
Starting >>> pcl_omp_registration
Starting >>> fastvirtualscan
Finished <<< amathutils_lib [45.3s]
Starting >>> kitti_box_publisher
Finished <<< ndt_tku [47.3s]
Finished <<< vector_map [47.3s]
Starting >>> vector_map_server
Starting >>> map_file
--- stderr: astar_search
/home/autoware/Autoware/src/autoware/core_planning/astar_search/test/src/test_class.cpp: In constructor 'TestClass::TestClass()':
/home/autoware/Autoware/src/autoware/core_planning/astar_search/test/src/test_class.cpp:54:25: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
for (int idx = 0; idx < costmap_.info.width*costmap_.info.height; ++idx)
^
/home/autoware/Autoware/src/autoware/core_planning/astar_search/test/src/test_class.cpp:60:25: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
for (int row = 0; row < costmap_.info.height; ++row)
^
/home/autoware/Autoware/src/autoware/core_planning/astar_search/test/src/test_class.cpp:62:27: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
for (int col = 0; col < costmap_.info.width; ++col)
^
/home/autoware/Autoware/src/autoware/core_planning/astar_search/test/src/test_astar_search.cpp: In member function 'virtual void TestSuite_checkSetPath_Test::TestBody()':
/home/autoware/Autoware/src/autoware/core_planning/astar_search/test/src/test_astar_search.cpp:143:25: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
for (int row = 0; row < test_obj_.costmap_.info.height; ++row)
^
/home/autoware/Autoware/src/autoware/core_planning/astar_search/test/src/test_astar_search.cpp:145:27: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
for (int col = 0; col < test_obj_.costmap_.info.width; ++col)
^
---
Finished <<< astar_search [57.9s]
--- stderr: ndt_cpu
** WARNING ** io features related to openni2 will be disabled
** WARNING ** io features related to pcap will be disabled
** WARNING ** io features related to png will be disabled
** WARNING ** visualization features related to openni2 will be disabled
---
Finished <<< ndt_cpu [55.5s]
Finished <<< fastvirtualscan [43.9s]
Starting >>> kitti_player
Starting >>> ndt_gpu
Starting >>> rosinterface
--- stderr: ndt_gpu
** WARNING ** io features related to openni2 will be disabled
** WARNING ** io features related to pcap will be disabled
** WARNING ** io features related to png will be disabled
** WARNING ** visualization features related to openni2 will be disabled
CMake Error at /usr/share/cmake-3.5/Modules/FindCUDA.cmake:617 (message):
Specify CUDA_TOOLKIT_ROOT_DIR
Call Stack (most recent call first):
/home/autoware/Autoware/ros/install-generic-aarch64/share/autoware_build_flags/cmake/autoware_build_flags-extras.cmake:18 (find_package)
CMakeLists.txt:24 (AW_CHECK_CUDA)
---
Failed <<< ndt_gpu [ Exited with code 1 ]
Aborted <<< kitti_box_publisher
Aborted <<< autoware_health_checker
Aborted <<< vector_map_server
--- stderr: kitti_player
** WARNING ** io features related to openni2 will be disabled
** WARNING ** io features related to pcap will be disabled
** WARNING ** io features related to png will be disabled
** WARNING ** visualization features related to openni2 will be disabled
---
Aborted <<< kitti_player
--- stderr: map_file
** WARNING ** io features related to openni2 will be disabled
** WARNING ** io features related to pcap will be disabled
** WARNING ** io features related to png will be disabled
---
Aborted <<< map_file
--- stderr: pcl_omp_registration
** WARNING ** io features related to openni2 will be disabled
** WARNING ** io features related to pcap will be disabled
** WARNING ** io features related to png will be disabled
** WARNING ** visualization features related to openni2 will be disabled
---
Aborted <<< pcl_omp_registration
Aborted <<< rosinterface
Summary: 19 packages finished [1min 41s]
1 package failed: ndt_gpu
7 packages aborted: autoware_health_checker kitti_box_publisher kitti_player map_file pcl_omp_registration rosinterface vector_map_server
6 packages had stderr output: astar_search kitti_player map_file ndt_cpu ndt_gpu pcl_omp_registration
107 packages not processed
I also added this line to my docker image (I found in image for driveworks):
COPY crossbuild/files/FindCUDA.cmake /usr/share/cmake-3.5/Modules/FindCUDA.cmake
The error seems to be related to CUDA libraries that are not found. I check that there is no folder in /usr/local/cuda, so declaring CUDATOOLKITROOT_DIR doesn't fix it.
Autoware 1.12 docker: kinetic generic-aarch64
Jetson TX2 Ubuntu 16.04 CUDA 9.0
Asked by Fernnando33 on 2020-01-21 08:32:55 UTC
Answers
The current Docker containers do not support CUDA for cross-compilation. Doing so would require a pretty significant effort as the cross-compilation images are somewhat complex. We use an alpine image to bootstrap the creation of a "native" (using binfmt) arm64 image where all of the dependencies are installed. We then build an amd64 image on Ubuntu and copy all necessary environment from the intermediate arm64 image to /sysroot/generic-aarch64 in the amd64 environment. So, basically, you would have to do the following:
- Add another Dockerfile which uses the same process as the existing cross-compilation setup.
- Have that new Dockerfile install CUDA in the intermediate
arm64environment. - Copy all of the CUDA components from the
arm64environment into/sysroot/generic-aarch64/in theamd64environment. - Modify any scripts or CMake files which look for CUDA (such as the FindCuda.cmake file you mention above) to look in
/sysroot/generic-aarch64/usr/local/cudainstead of/usr/local/cuda.
Unfortunately, the community hasn't shown enough interest thus far in a cross-compilation-supporting-CUDA image for anyone to make the effort. However, if you're willing to give it a shot, I'm happy to put in a bit of time to help.
Asked by Josh Whitley on 2020-01-26 22:18:45 UTC
Comments
I don't see any cuda tags for autoware arm64v8 repo. Although the (implicit) amd64 repo seems to have cuda tags. It seems that the amd64 images install cuda by inheriting FROM nvidia's opengl parent image. Looks like there may be beta support for arm64v8 for nvidia-docker plugin though, so see if you could swap the parent image appropriately.
Asked by ruffsl on 2020-01-21 14:01:25 UTC