ROS Resources: Documentation | Support | Discussion Forum | Index | Service Status | ros @ Robotics Stack Exchange
Ask Your Question
0

Build of ROS2 package: Out of memory

asked 2022-11-08 10:20:55 -0500

holunder gravatar image

Problem

I try to build a ros2 package and run out of memory so that the system gets unresponsive or the process is terminated.

I'm using ros2 foxy on ubuntu 20.04

Hardware

  • CPU: 4 x ARM A53
  • RAM: 4 GB
  • Swap: 0 Byte

Package

It's a single node without any functionality. I've created about 65 custom ros messages as _idl_ files.

As soon as I create subscriber for each message with empty callback functions, the build process ends up in consuming whole memory.

Example Code:

// Subscriber
  cust1_gen_Sub = this->create_subscription<my_pkg::msg::Cust1>("cust1_gen", 10, std::bind(&Nd::topic_callback_Cust1_gen, this, _1));

// Callback
void Nd::topic_callback_Cust1_gen(const my_pkg::msg::Cust1::SharedPtr msg) const
{
// empty
}

My CMakeLists.txt is this

cmake_minimum_required(VERSION 3.5)
project(my_pkg)

# Default to C99
if(NOT CMAKE_C_STANDARD)
  set(CMAKE_C_STANDARD 99)
endif()

# Default to C++14
if(NOT CMAKE_CXX_STANDARD)
  set(CMAKE_CXX_STANDARD 14)
endif()

if(CMAKE_COMPILER_IS_GNUCXX OR CMAKE_CXX_COMPILER_ID MATCHES "Clang")
  add_compile_options(-Wall -Wextra -Wpedantic)
endif()

# find dependencies
find_package(ament_cmake REQUIRED)
find_package(rclcpp REQUIRED)
find_package(std_msgs REQUIRED)
find_package(builtin_interfaces REQUIRED)

find_package(rosidl_default_generators REQUIRED)
rosidl_generate_interfaces(${PROJECT_NAME}
  "msg/Cust1.idl"
  ...
  "msg/Cust65.idl"
  DEPENDENCIES builtin_interfaces
 )
ament_export_dependencies(rosidl_default_runtime)

include_directories(include/${PROJECT_NAME})

add_executable(Nd xs/nd.cpp)
ament_target_dependencies(Nd rclcpp std_msgs)
rosidl_target_interfaces(Nd
  ${PROJECT_NAME} "rosidl_typesupport_cpp")


ament_package()

Observation

Observed memory consumption during build by means of _htop_. While building the message file objects, it's up to 800 MB Mem usage

Sometimes it simply stucks for hours, sometimes the process gets aborted after 10 minuts.

Process which consumes whole memory as shown by ps aux

/usr/lib/gcc/aarch64-linux-gnu/9/cc1plus -quiet -I .../my_pkg -I build/my_pkg/rosidl_generator_cpp -imultiarch aarch64-linux-gnu -D_GNU_SOURCE -D DEFAULT_RMW_IMPLEMENTATION=rmw_fastrtps_cpp -D RCUTILS_ENABLE_FAULT_INJECTION -D ROS_PACKAGE_NAME="my_pkg" -D SPDLOG_COMPILED_LIB -isystem /opt/ros/foxy/include my_pkg/Nd.cpp -quiet -dumpbase Nd.cpp -mlittle-endian -mabi=lp64 -auxbase-strip CMakeFiles/Nd.dir/os/Nd/Nd.cpp.o -Wall -Wextra -Wpedantic -std=gnu++14 -fasynchronous-unwind-tables -fstack-protector-strong -Wformat-security -fstack-clash-protection -o /tmp/ccd6zsmm.s

What I've already tried

Building with

$ MAKEFLAGS="-j1 -l1"
$ colcon build --executor sequential

Questions

Is there anything I could try to solve the issue and speed up the build on my device? I was used to build my project on target in ROS1. Is this not suitable for devices of this performance?

edit retag flag offensive close merge delete

2 Answers

Sort by ยป oldest newest most voted
2

answered 2022-11-09 02:27:18 -0500

gvdhoorn gravatar image

updated 2022-11-09 02:53:34 -0500

Edit:

What I've already tried

Building with

$ MAKEFLAGS="-j1 -l1"
$ colcon build --executor sequential

apologies, as I only now noticed you'd already configured Colcon to reduce/disable build parallelism.

I'm a bit uncertain whether MAKEFLAGS="-j1 -l1" on its own line like that would work though. I would at least expect you to have to run export MAKEFLAGS="-j1 -l1", or otherwise MAKEFLAGS="-j1 -l1" colcon build --executor sequential.

If you're still running into this problem, it's likely there isn't anything at the Colcon/CMake level you can do about this.

Note that an A53 has about 1/100th of the performance of a typical desktop processor. So things taking a little longer would not be unexpected I believe.


Original answer:

It's very likely this is caused by C++ template instantiation.

rclcpp is template heavy, and the type support code is as well.

By default, Colcon will ask CMake to use all CPU cores available for a build. On embedded systems, especially without any swap, this will typically lead to memory becoming the bottleneck (ie: there is not enough memory to let the compiler 'comfortably' use all the cores).

You have various ways to reduce parallelism (and thereby most likely also work around the limited available memory) with Colcon. See #q368249 and #q304300 for previous discussions.

Adding some swap also typically helps, but you may not have that possibility.

edit flag offensive delete link more

Comments

I've added 4 GB of swap to the machine and found that build eventually completes now. Thanks a lot for guiding me in this direction.

Unfortunately, it still needs quite long (30 minutes) to complete.

holunder gravatar image holunder  ( 2022-11-22 07:19:39 -0500 )edit
0

answered 2022-11-08 12:02:51 -0500

duck-development gravatar image

updated 2022-11-08 12:07:16 -0500

How much free ram do you have before start of the build. If you have a UI may use only console mode. Do you may try to build the ROS1 to ros2 bridge because it need ram.

Did you try

     colcon build --packages-select

Use a swap file.

edit flag offensive delete link more

Comments

I've 3.1 GiB of RAM before starting. It's a headless system. Concerning colcon build --packages-select I'd add my single package which seems to be not different to running it without this command.

I really wonder why compilation of code as given in my question does need so much memory.

holunder gravatar image holunder  ( 2022-11-08 23:16:26 -0500 )edit

As mentioned above, I considered your recommendation concerning swap. Thanks.

holunder gravatar image holunder  ( 2022-11-22 07:20:03 -0500 )edit

Question Tools

1 follower

Stats

Asked: 2022-11-08 10:20:55 -0500

Seen: 1,470 times

Last updated: Nov 09 '22