how to best run a service that many clients connect to every few seconds?

asked 2011-08-11 08:07:44 -0500

This post is a wiki. Anyone with karma >75 is welcome to improve it.

I'm running a simulator backend that allows clients to connect to it using the rosservice calls. Everything works fine with a few clients connected (<20 or so) but if I try a large-scale test with 64 clients, it will fail to connect on occasion. I'm not sure exactly what the issue is. I worked around it by implementing retries and only after trying a bunch of times failing.

Is the problem with the callback queue getting overloaded? Is there a buffer of service requests that is hidden away somewhere? (I had trouble with small buffers on my data topics already, but that was a simple fix!) Maybe it's a threading thing, although the backend is currently single-threaded, and the clients are multi-threaded but just to run a spinner on a completely disjoint messaging queue for client-client communications. If the global callback queue is involved then it is only getting touched by one thread ever per process.

Thanks for any help/pointers!

edit retag flag offensive close merge delete

Comments

Are the client - service connections persistent? That might help. I think they are non-persistent by default.

dornhege ( 2011-08-11 08:50:42 -0500 )edit

Yeah this was my first thought but they design someone else chose made that hard. If they are persistent, they actually keep the TCP connection live? That would probably make the "collision" on reconnect go away though. I just had hoped not to rewrite that much code of other people.

Devin Grady ( 2011-08-12 01:04:18 -0500 )edit

add a comment

answered 2011-08-11 18:45:03 -0500

This post is a wiki. Anyone with karma >75 is welcome to improve it.

Any specific service callback will only be called one at a time, even if there are several requests, they will be queued up and possibly overflow if you're calling it quickly.

If you do require each request to be sequential @dornhege 's suggestion of persistent connections will help.

However this sounds like you might want to consider a different interface design than a single service call. I suggest that you look into actionlib You could setup your simulator to be an action server which queues all the incoming requests and does whatever locking is necessary for thread saftey to interact with the simulator. With this technique you can allow it to interact with multiple clients at once and with concurrent requests in flight you can get better throughput in each cycle.

Comments

sounds like that'll just delay the issue, rather than solve it. I assume that multiple connections made all at the same time would still hit some sort of internal buffer of connections, and the problem is that they are dropped before they can interact at all. I do not require them to be sequential

Devin Grady ( 2011-08-12 01:01:49 -0500 )edit

add a comment

how to best run a service that many clients connect to every few seconds?

Comments

1 Answer

Comments

Question Tools

Stats

Related questions

how to best run a service that many clients connect to every few seconds? edit

Comments

1 Answer

Comments

Question Tools

Stats

Related questions

how to best run a service that many clients connect to every few seconds?