Question

I'd like have certain callbacks have thread affinity and others get whatever they are handed from the pool. There is some related affinity discussion here from a few years ago: https://github.com/ros2/rclcpp/issues/108

In my current case it is a timer update that needs to run on the same thread id, while service and topic callbacks ought to run concurrently on any thread available (subject to lock guards where needed within them).

I'd like to do something expedient in the short term but also make progress towards a better and longer term solution (if in discussion only).

Some ideas come to mind:

Very quick and very dirty solution: Have the timer callback check the thread id and the time since it last complete update, return immediately if on wrong thread, otherwise do a regular update. Set the callback rate to the number of threads times an extra margin factor times the original desired update rate. The callback could sleep until the next proper update time, and this would allow that large update rate to be smaller, combined with the MutuallyExclusive callback group type later scheduled same callbacks that can't be run just get dropped instead of wasting resources getting scheduled over and over on the wrong threads?
One callback that is only run once and then has a while loop with a sleep doesn't sound too bad. Is there a ros2 rate? Does that mess with the Executor code to never return, at the least it'll never assign anything else to that one thread even if it is sleeping most of the time.
Nest a single threaded node within a multi-threaded node (or the other way around), the outer node has a timer update which calls spinonce or spinssome on the inner node, and with proper mutexes sharing can be done between the two. I'll likely try this first because the first option is so ugly.
Inside the callback, somehow spawn a new job ('future'?) to only be run on a specific thread. Would it have to interact with the Executor machinery, or could it be done with pure std::thread calls? Haven't looked into this much.
Figure out how to run current code with constantly changing thread id- the current issue is with OpenGL in combination with SDL- SDLGLCreateContext. Maybe the context can be updated to the new thread id, maybe there is a large cost involved in doing that. It wouldn't be hard to switch to glfw if that can to it more easily. Off topic for this website.
Create a hacked new executor type based off of https://github.com/ros2/rclcpp/blob/master/rclcpp/src/rclcpp/executors/multi_threaded_executor.cpp, just make so all timer callbacks use just one thread id, don't worry about multi timers initially. Digging into this better paves the way to a more proper solution even if the first thing that works is ugly. But it may not work very easily, likelihood of breaking it and having trouble debugging it is high.
Create a proper Executor type, or add features to MultiThreadedExecutor to support it- and this would also require a new callback group type also? Try to pr this eventually if it can be done well.

Asked by lucasw on 2019-01-25 19:28:48 UTC

Answer 1

In one case I had a couple of nodes, one which needed a single thread and the other which doesn't care about thread affinity but blocks for a long time (it needs to do something other than select() and block).

Below I spin the one node in a regular std::thread and the other runs separate for that (maybe for symmetry it ought to be in another std thread also...):

void run_usb_cam(FooType foo)
{
  rclcpp::executors::SingleThreadedExecutor executor;
  auto usb_cam = std::make_shared<usb_cam::UsbCam>(foo);
  executor.add_node(usb_cam);
  executor.spin();
}
...

  rclcpp::executors::SingleThreadedExecutor single_executor;
  // imgui_ros has to be single threaded for now to avoid context switches with opengl
  auto imgui_ros = std::make_shared<imgui_ros::ImguiRos>(foo);
  single_executor.add_node(imgui_ros);

  std::thread cam_thread(std::bind(run_usb_cam, foo));
  single_executor.spin();
  cam_thread.join();

https://github.com/lucasw/imgui_ros/blob/crystal/imgui_ros/src/nodes/usb_cam_viewer.cpp

I initially tried a while loop with spin_some on both executors, but even if one of the executors is multi threaded spin_some doesn't return until all the jobs it was running (however concurrently) are complete, so it blocks the other executor.

Asked by lucasw on 2019-02-04 16:10:35 UTC

MultiThreadedExecutor thread affinity for certain callbacks?

Comments

Answers

Comments