cross-compile with research ISA

asked 2020-04-02 11:23:15 -0500

broomstick
111 ●15 ●20 ●23

I would like to cross-compiling ROS2 onto a research platform with a new ABI and an extension to the RISC-V ISA. I need some help either manually building or making use of the existing cross-compile tools.

My test applications are fairly simple, and I was able to cross-compile ROS[1] applications manually (i.e., avoiding catkin) by converting them to rosserial applications. I eventually built an entire TurtleBot3 for my research platform this way (including teleop).

Now I'm looking to ROS2 for some of its real-time possibilities.

My normal cross-compile invocation for a rosserial application is something like this:

$HOME/cheri/output/sdk/bin/cheri-unknown-freebsd-clang++ \
        $SRC1 $SRC2 -o $BIN \
        -g \
        -I$HOME/ros_lib \
        -mabi=purecap \
        --sysroot=$HOME/cheri/output/sdk/sysroot128 \

I'm not sure I can get this to translate to the docker/QEMU-based cross-compile tool (https://github.com/ros-tooling/cross_...).

Is there a simpler way for me to use colcon or a recommendation for calling cmake directly?

I'm only trying to build a simple talker/listener demo at this point, so we're talking one package, two source files, and one message type.

Comments

Cross-compilation with CMake essentially comes down to providing it a suitable toolchain file. Is that something you already have for your toolchain?

I'm not claiming this will solve everything (especially sysroot stuff can be tricky to get right), but it would be a big step towards cross-compiling "arbitrary" projects which based their build scripts on CMake.

gvdhoorn ( 2020-04-02 11:31:00 -0500 )edit

Yes, I have a CMake toolchain file.

broomstick ( 2020-04-02 12:05:18 -0500 )edit

Seeing as ros-tooling/cross_compile is just about the most convenient way to cross-compile an entire ROS workspace without jumping through all sorts of hoops manually, it may pay off to post an issue on its issue tracker. ros-tooling/cross_compile#69 seems like it would be going in the direction of what you'd need.

gvdhoorn ( 2020-04-02 12:13:34 -0500 )edit

I'll follow your advice and post there, but I've also been playing with using my toolchain file with some success, but it chokes on finding PythonLibs.

Based on an invocation such as:

colcon build --ament-cmake-args -DCMAKE_TOOLCHAIN_FILE=/home/broomstick/ros2_cmake_test/CrossToolchain.cmake

I get errors like the following:

Starting >>> rcutils
--- stderr: rcutils                                                                                                            
CMake Error at /usr/share/cmake-3.10/Modules/FindPackageHandleStandardArgs.cmake:137 (message):
  Could NOT find PythonLibs (missing: PYTHON_LIBRARIES PYTHON_INCLUDE_DIRS)
  (Required is at least version "3.5")
Call Stack (most recent call first):
  /usr/share/cmake-3.10/Modules/FindPackageHandleStandardArgs.cmake:378 (_FPHSA_FAILURE_MESSAGE)
...
  CMakeLists.txt:120 (ament_lint_auto_find_test_dependencies)

broomstick ( 2020-04-03 15:28:00 -0500 )edit

My toolchain file doesn't specify a location for Python and I assumed it would default to my system version. This is probably dumb, but I'm not sure why.

broomstick ( 2020-04-03 15:32:03 -0500 )edit

[update] I've added this to the ros-tooling/cross_compileissue tracker.

broomstick ( 2020-04-03 16:20:45 -0500 )edit

Regarding the "PythonLibs not found issue", I've been facing the same with a native build and I believe this is due to rosdep behavior https://github.com/ros-infrastructure...

marguedas ( 2020-04-06 03:46:20 -0500 )edit

add a comment

answered 2020-04-08 07:21:59 -0500

broomstick
111 ●15 ●20 ●23

To close this question out:

It's challenging, as @gvdhoorn states, but I'm making progress using a cmake toolchain file. You can find my current toolchain file here.

My toolchain file doesn't (currently) set CMAKE_PREFIX_PATH, so I pass that to cmake explicitly, as discussed here.

I was able to fix (or bypass...) my PythonLibs problems by passing -DBUILD_TESTING=NO, as discussed here. This is not ideal, but is getting me over this immediate hurdle.

I also needed to pass -DTHIRDPARTY=ON to ensure my cross-compiled asio libraries are found.

Overall, my colcon invocation looks something like this:

colcon build \
    --cmake-args \
        -DCMAKE_TOOLCHAIN_FILE=$HOME/ws/CrossToolchain.cmake \
        -DBUILD_TESTING=NO \
        -DTHIRDPARTY=ON \
        -DCMAKE_PREFIX_PATH=$SYSROOT/usr/local/cheri;$SYSROOT/usr/local/cheri/libcheri/cmake \
    --no-warn-unused-cli

My problems at this point all appear to be related to cross-compile incompatibility problems (normal stuff) rather than problems specific to colcon or ROS, so I'm content.

edit flag offensive delete link

Comments

Nice. Thanks for posting your approach here.

gvdhoorn ( 2020-04-08 07:48:42 -0500 )edit

Looks like it's this architecture, correct?

From your toolchain file, it appears your OS is FreeBSD(-based). Is that correct? Are there already any papers or other publications available detailing your work?

Did you have to patch anything in ROS 2 significantly to get it to build?

gvdhoorn ( 2020-04-08 08:06:04 -0500 )edit

Yes, that's correct. There is a CHERI-aware version of FreeBSD, called CheriBSD.

No publications yet for the ROS-based CHERI work. This is really our first foray into the cyber-physical domain. ROS seemed like a good target, as it's open-source, lives in a fairly homogeneous ecosystem, and is well supported (including ROS Answers!).

As a proof of concept, I squashed the TurtleBot3 RPI and OpenCR code into a single rosserial-based executable and cross-compiled that successfully. CHERI simplifies compartmentalisation, so I used it to split the network-facing communications functions from the physical control functions. CHERI also lets you create certificate-like access tokens, so I was able to use that to control access to the cmd_vel topic.

The proof-of-concept worked well enough to start a more comprehensive port of ROS2. I want to make use of some of it's potential in real-time systems targeting our CHERI-aware FreeRTOS and RTEMS.

broomstick ( 2020-04-10 08:06:34 -0500 )edit

I'm only using a subset of ROS2 libraries at the moment, using this repo as a starting point (based on your earlier recommendation to talk to the ros-tooling/cross_compile folks.)

The required patches have been minimal.

broomstick ( 2020-04-10 08:15:52 -0500 )edit

We have had to make some changes to SQLite to fix pointer alignment issues, so anything using sqlite3.c needs a patch. This will eventually get upstreamed.

poco also selects some functions using preprocessor directives that check the size of longs and pointers. CHERI confuses these conditionals because a capability/pointer is 128-bits in memory, though its still representing a 64-bit pointer. This required a small patch. I'm looking for a way to 'improve' the code so the maintainers are willing to accept a PR.

rcutils needed __FreeBSD__ added to nearly every preprocessor directive that already included __APPLE__. I submitted a PR, which was merged.

rmw_cyclonedds has an align_ function that had some crazy casting and didn't work with our capabilities. We cleaned the functions up a bit and submitted a PR, which is approved and waiting merge.

broomstick ( 2020-04-10 08:19:17 -0500 )edit

To follow up on an earlier question from gvdhoorn, our first publication involving CHERI and ROS is now public.

broomstick ( 2020-10-22 03:57:23 -0500 )edit

add a comment

cross-compile with research ISA

Comments

1 Answer

Comments

Question Tools

Stats

Related questions

cross-compile with research ISA edit

Comments

1 Answer

Comments

Question Tools

Stats

Related questions

cross-compile with research ISA