Best way to manage multiple subscriptions.

[xbarra]

Hi.
We want to subscribe to several exchanges and some of them with more a hundred symbols.
We are launching several processes with multiprocessing.Process, sending the information we want to a mp.manager.queue.

Then several processes consume the queue.

However, even decreasing the number of exchanges and symbols (5 symbols, 2 exchanges, subscribing to ticker and trades), we don't seem able to keep up with the queue.

We try to accumulate a fair enough quantity of updates per symbol before sending them to postgres.

Any help would be welcome. Not necessarily in fixing mine, but it would be great to know how others addressed this issue.
Thanks in advance.

[bmoscon]

Hard to say what the issue is, but a single process should easily be able to handle 5 symbols worth of trade/ticker data. Your queueing system may be the issue - I'm not sure how performant those are/arent for processing lots of small messages. You probably should only use something that supports asyncio, so while the data is being put somewhere for your other process to handle it, the I/O is not blocking the rest of the system

[xbarra]

Thanks for the fast reply.

We are using asyncio all around.

I believe that the queues and managed dictionaries perform quite badly.

As in the edited question, I am quite interested in how other people addressed handling and saving into a db lots of data. (We want to be able to keep adding symbols).