Ask Your Question
1

Throttle publish rate based on topic queue size and fill level

asked 2013-06-06 04:09:48 -0600

mcnichjm gravatar image

I've seen questions asked about how to change the subscriber's message receive rate, and filters seem to be the solution to this problem. However, this assumes that it is ok if the subscriber does not receive every published message.

My question is similar: is it possible for a publishing node to query the fill level of the topic queue to determine if there is space left? Generally speaking, is there a way to throttle an upstream publisher node's publish rate based consumption rates further down the pipeline? The key here is that I cannot afford to have any messages dropped.

The scenario: an upstream node publishes messages at a high rate and a downstream node needs to process EVERY message published. I know I could simply increase the pub/sub queue sizes, but this solution is not ideal for extended tests.

Ideally, I would like to check how much space is left in the queue before publishing, but as far as I can tell, there's no way to do that.

Any suggestions would be greatly appreciate!

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted
0

answered 2013-07-08 12:14:29 -0600

lindzey gravatar image

updated 2013-07-08 12:14:56 -0600

The functionality that you're describing doesn't really work with the publish/subscribe model, where the publishers by definition just push out data and don't care about how many subscribers they have, or what those subscribers are doing with the published data.

It would probably be a better fit for the service model, where the subscriber (or client) requests data from the publisher (or server) when it is ready to process it.

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Question Tools

1 follower

Stats

Asked: 2013-06-06 04:09:48 -0600

Seen: 226 times

Last updated: Jul 08 '13