Limit on concurrent connections between nodes?
Edit:
In the original question I observed a perceived limit on action client-server connections. It turns out that this limit affected any type of connections (topic, service, action) between nodes.
Original question:
I have a network of several hundred of nodes (let's call them server nodes), each implementing an action server. Then there are a handful of client nodes, where each node can potentially open up an action client to communicate with each of the server nodes.
In tests I have observed that there appears to be a specific limit on concurrent connections between action client and action server, which appears to be at 196 such connections, as the 197th one consistently fails. In roscpp I can initialize the 197th action client, but waitForServer() will block indefinitely. When sending a goal anyway, the server does not receive it. This occurs with both the ActionServer and SimpleActionServer.
I am interested in the limits to ROS' scalability (in terms of number of nodes and connections). Why is it that action connections appear to be hard-capped at 196 and what other such limits exist?
Would you not be running into a limit of (the) Linux (network stack)? On 'consumer' Linux distributions (or those configured with a consumer 'profile'), settings like max nr of sockets, file descriptors, etc are typically set relatively conservatively.
You don't tell us which OS you're using, but it would probably be a good idea to check those.
I'm not claiming there are no further limits caused by how ROS is designed and architected, but the OS is a good source of (artificial) limits that you should first investigate.
Finally:
for your sort of question I would suggest to be specific. How many exactly?
Thank you for your response. You seem to be correct in that I'm dealing with some OS side limitations rather than something directly related to ROS. Actions, topics and services all seem to use up the same 'resource' (whatever that may be, I have to investigate yet). I'm running Kubuntu Bionic.
To answer your questions regarding the number of nodes: It depends on the scenario, but let's say 800 server nodes. With 800 nodes, the following combinations of connections reached the limit:
I would first try increasing maximum nr of file descriptors for your user (or the user you're running these experiments with).
ulimit -ashould show you the current limits.Thank you, that seems to indeed have been the problem. The original limit was set to 1024 user-side; I knocked it up quite a bit and am now unable to reproduce the connection limit.