Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement message bus based connector #378

Closed
wants to merge 1 commit into from

Conversation

adamruzicka
Copy link
Contributor

@adamruzicka adamruzicka commented Jan 22, 2021

Historically Dynflow used database for communication across processes. The
sending side would write a record into dynflow_envelopes table, receiving side
would poll this table and if it found a record addressed to it, it would receive
it and remove it from the db. Both polling and insert-read-delete cycles were
putting unnecessary load on the db. Using a message bus for this should be the
way to go.

Here I chose MQTT because it is lightweight and I worked with it recently and
still have it in fresh memory. However, this could be adjusted to almost
anything with minimal changes.

#373

Historically Dynflow used database for communication across processes. The
sending side would write a record into dynflow_envelopes table, receiving side
would poll this table and if it found a record addressed to it, it would receive
it and remove it from the db. Both polling and insert-read-delete cycles were
putting unnecessary load on the db. Using a message bus for this should be the
way to go.

Here I chose MQTT because it is lightweight and I worked with it recently and
still have it in fresh memory. However, this could be adjusted to almost
anything with minimal changes.
@adamruzicka
Copy link
Contributor Author

I also considered using Redis since we kind of have it already, but there we would either duplicate connections per-process if running with Sidekiq or have slightly different behavior when running with and without Sidekiq. Another gotcha is pub-sub in redis is not scoped to databases. It might not be an issue, but definitely isn't a plus

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant