Generic process manager instead of roslaunch?
I was following these discussions https://discourse.ros.org/t/why-does-... and https://discourse.ros.org/t/how-do-yo... and was wondering if there are modern well-maintained non-robotics/non-ros tools for launching a bunch of processes and monitoring them, that separate the operator from unix process management details (and perhaps can be run in a non-unix environment but I'm not too concerned with that), has a nice terse yaml file config or similar. Probably this question belongs on a different stack site but with ros references mostly removed, but I thought I'd start here.
What I've found is PM2https://pm2.keymetrics.io/docs/usage/..., which is meant for node.js applications but apparently can run python or any process https://pm2.io/blog/2018/09/19/Manage... & https://github.com/Unitech/pm2/issues...), but if it really isn't meant for it that makes it less desirable (like having to install npm, but if the tool is good enough then that's okay). A large community isn't that much of a plus if they can't help with obscure ros/C++ binary/python-particular problems.
A web frontend would be nice but otherwise console text is fine (need to see if PM2 has that, the examples look console oriented which is a basic requirement- it shouldn't be web ui only).
Probably launch-on-boot is a basic feature most of these have (another recent discourse discussion https://discourse.ros.org/t/automatic...).
Maybe roslaunch would still be used, but mostly at a different granularity, though having to learn two different systems with a lot of overlap and an unclear delineation produces a lot of friction (on top of for example commingling Docker into the system, then docker-compose on top of that, and running some things outside of docker some times, but some very large percentage of robotics is system integration and not AI and motion control and path planning schemes after all...).
I think everything could be shoe-horned into docker-compose and it has a lot of nice management features, but I'd want to be able to run outside of docker also.
Being able to extend it with an api to make use of ros-only features (e.g. namespaces, lifecycle management) would be a stretch goal.
It would maybe be like going from rqt console to elasticsearch or another web oriented log management tool- there are a lot of problems that aren't that robotics particular that have been solved or are actively being addressed with a lot more resources than the ros-only equivalent.
Possibly Formant or Foxglove or other commercial vendors have something in this space, maybe that isn't locked exclusively into their cloud services?
I've had similar questions/ideas. See my comment on ros2/design#297. It didn't get much response though.
Kubernetes and the many alternatives/competitors come to mind, although that will probably mean everything runs in containers. If that gets natively supported, that would not be too bad, as it would nicely isolate (groups of) processes (ie: nodes), which can have all sorts of benefits.
This has been explored by Sony. See Robotics Distributed System based on Kubernetes and by others: ROS 2 on Kubernetes.
Kubernetes isn't tied to containers although it doesn't support anything else at the moment AFAIK.
Its support for distribution, scaling, monitoring and orchestration do seem to come close to what you're describing.
I remember a presentation at ROSCon by Formant along the lines of "let's reuse as much as possible from cloud orchestration/deployment and other functionality instead of reinventing the wheel in robotics". I'm not sure whether anything concrete happened after that though.
Tying it to cloud deployment would indeed not be the way to go I believe.
Personally, I believe the main challenge is not using some existing framework / tool to manage ROS nodes, but to make it usable by the ROS community. Solutions would ideally scale from single-node-on-single-machine to multiple-nodes-on-many-machines, and that's non-trivial. And they should also do that while remaining user friendly, with many users having no experience at all with these concepts.
As I was curious myself, I searched around a bit. Kubernetes can be used with Singularity as well (any CRI compliant runtime actually): Integrating with Kubernetes.
Singularity, as opposed to Docker, runs all processes as a regular user (ie: your
$USER), and allows full access to devices,$HOME, network, etc. This puts it on the opposite side of the spectrum compared to Docker when it comes to isolation. Clearly not ideal in all situations, but for use with something like ROS, not having to punch holes in each and every container to allow access to devices, or to enable communication with nodes running in other containers (or on other hosts) is a clear advantage (and could be one of those "robotics/ROS specific aspects" you mention).You'd still deploy using images, but there would be less configuration, making it less complex.
Edit: just noticed the required projects are archived ...(more)
After watching their talk from roscon 2019 I immediately tried out elasticsearch and opentracing (didn't end up doing too much with tracing after integrating it, but the log search features of elasticsearch are fantastic- though eat up some disk and cpu vs. rqt console) https://formant.io/news-and-blog/2019...
Yes, the tracing example was really nice. ROS 2 now has ros2_tracing which has similar goals I believe -- although it doesn't come with the nice reporting side to it.