Working in high frequencies (1000 [hz])

asked 2021-04-29 11:26:45 -0600

Iftahnaf gravatar image

updated 2022-03-05 18:21:49 -0600

lucasw gravatar image

Hi all,

I'm trying to run a software-in-the-loop simulation of a PX4 drone that flies at a really high speed (approx. 200 [m/s]). In order to guide and monitor its movement, I need to run some low-level nodes at really high frequencies - around 1000 [hz], and some high-level nodes a bit slower - around 150 [hz].

I have a powerful workstation: i9-10900k 3.7Ghz with 10 cores, 32GB of DDR4, m.2 500GB SSD, and RTX3070 (for the Gazebo), Running on ubuntu 18.04.

I generate C++ nodes from Simulink but they struggle to run at the desired high frequencies and can't maintain a constant frequency of 1000 [hz], after few seconds it starts to drop.

I tried to implement the nodes in python but got the same results.

Any ideas on how to solve this issue? maybe migrate to ROS2?

Thanks in advance,


edit retag flag offensive close merge delete



I'd suggest seeing whether enabling tcpNoDelay helps. See #q360038 for a question discussing this. Whether you can do this with code generated by Matlab I wouldn't know.

It's also possible to compile the entirety of roscpp with tcpNoDelay enabled, but that would require some more work.

And nodelets would already help significantly, but both Matlab and Gazebo make that complex/impossible.

In the end though, ROS 1 wasn't really made for kHz level pub-sub. It can be done, but it requires some special care. I'm not sure the ppl at Mathworks had those use-cases in mind when they worked on the code-generation.

gvdhoorn gravatar image gvdhoorn  ( 2021-04-29 13:25:01 -0600 )edit

Another approach could be to not generate ROS nodes from Simulink models, but ros_control controllers. Those would be running as plugins in gazebo_ros_control, time-synced with the simulation and absolutely deterministic (as in: lock-step with the simulation clock). Lock-step is possible between Matlab & Gazebo, but it's currently using topics, so I'm not sure the 1 kHz rate would be achievable (although with time suspended inside Gazebo, it may not be such a problem).

That's not a feature currently available, but I've presented work in that direction at the ROS-Industrial Americas Consortium meeting two weeks ago. See the "Automated Code Generation of Simulink Models as ros_control Controllers" presentation slot (video will become available at a later date).

If enabling tcpNoDelay is not enough, post a comment here and perhaps we can arrange something.

gvdhoorn gravatar image gvdhoorn  ( 2021-04-29 13:28:52 -0600 )edit

I'm trying to run a software-in-the-loop simulation of a PX4 drone [..]

I'm going to assume you're still using Gazebo in this case?

gvdhoorn gravatar image gvdhoorn  ( 2021-04-29 13:35:27 -0600 )edit

In all honesty, ROS2 might be a better fit for that exact situation. It's meant to work better at higher frequencies and in control settings.

stevemacenski gravatar image stevemacenski  ( 2021-04-29 13:37:08 -0600 )edit

It's up to the OP. Going to ROS 2 would require quite some changes.

Trying the two things I mentioned requires almost no changes.

And I'm aware of the work by Jackie and the RTWG, but do we have any known-good examples of 1 kHz+ systems which use ROS 2?

I only know of PAL, but I don't believe they've been able to use ROS 2 yet for their RT controller infrastructure, and are still using OROCOS.

gvdhoorn gravatar image gvdhoorn  ( 2021-04-29 13:48:34 -0600 )edit

You seem to have #q350321 still open as well ;)

gvdhoorn gravatar image gvdhoorn  ( 2021-04-29 14:03:10 -0600 )edit

Thank you both for the help.

I'll need to read about tcpNoDelay and ros_control because I'm not familiar with them. @gvdhoorn I'll start by working with the tcpNoDelay as you suggested, and I'll update.

Thanks again!

Iftahnaf gravatar image Iftahnaf  ( 2021-04-30 08:06:00 -0600 )edit