Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Abort prometheus worker on farmer exit. #2104

Merged
merged 6 commits into from
Oct 17, 2023
Merged

Conversation

shamil-gadelshin
Copy link
Contributor

This PR adds abortion for an optional Prometheus task serving farmer's metrics on the application exit.

Code contributor checklist:

@shamil-gadelshin shamil-gadelshin added the networking Subspace networking (DSN) label Oct 13, 2023
@shamil-gadelshin shamil-gadelshin self-assigned this Oct 13, 2023
Copy link
Member

@nazar-pc nazar-pc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will not have the desired effect. It will work on Ctrl+C, but the idea was that it should be stopped on drop. Future returned by fn farm doesn't have to complete, it could hypothetically be simply dropped any time.

@shamil-gadelshin
Copy link
Contributor Author

This will not have the desired effect. It will work on Ctrl+C, but the idea was that it should be stopped on drop. Future returned by fn farm doesn't have to complete, it could hypothetically be simply dropped any time.

Do you mean something like AsyncAbortOnDrop similar to https://github.com/subspace/subspace/blob/51f39b9ccd45616ed9b6c1cb51911fb08c7cfaf2/crates/subspace-farmer/src/utils.rs#L24

@nazar-pc
Copy link
Member

nazar-pc commented Oct 13, 2023

Yes, AsyncJoinOnDrop should work

@nazar-pc
Copy link
Member

Well, not really. AsyncJoinOnDrop will wait for future to exit, but this one never will. You need to do something extra for that to happen, we have custom Drop implementations in various places for this to work properly. For example SingleDiskPlot ensures shutdown of various things is triggered first, such that on join they do actually exit.

@shamil-gadelshin
Copy link
Contributor Author

Well, not really. AsyncJoinOnDrop will wait for future to exit, but this one never will. You need to do something extra for that to happen, we have custom Drop implementations in various places for this to work properly. For example SingleDiskPlot ensures shutdown of various things is triggered first, such that on join they do actually exit.

Did you comment on AsyncJoinOnDrop or AsyncAbortOnDrop?

@nazar-pc
Copy link
Member

I see, we don't have AsyncAbortOnDrop yet

@nazar-pc nazar-pc removed the networking Subspace networking (DSN) label Oct 13, 2023
Copy link
Member

@nazar-pc nazar-pc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So there is a wrapper now that is sending abort to the task on drop, which is correct.

The part with Future would have also worked under different circumstances, but in this case it will not because Drop means no one will ever be able to poll the future again, so effectively the wrapper can be simplified to just self.task.abort() in Drop and it will behave exactly the same.

Since you trigger internal abort already is, all you need to do is to additionaly wrap the task in AsyncJoinOnDrop to ensure that it is both aborted on drop and Drop::drop only exits once background task has actually finished.

And I'd call it in more generic way like AsyncAbortOnDrop you suggested before.

# Conflicts:
#	crates/subspace-farmer/src/bin/subspace-farmer/commands/farm.rs
Copy link
Member

@nazar-pc nazar-pc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since AsyncJoinOnDrop wraps JoinHandle and not arbitrary future, we don't actually need Fuse, we can call JoinHandle::is_finished instead of is_terminated, in which case we can update AsyncJoinOnDrop::new() with additional argument abort_on_drop: bool and just call abort beore task::block_in_place. It'll be a simpler abstraction here.

Copy link
Member

@nazar-pc nazar-pc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is fairly clean, I'll squash-merge once CI passes

@nazar-pc nazar-pc merged commit 487ae10 into main Oct 17, 2023
10 checks passed
@nazar-pc nazar-pc deleted the abort-prometheus-task branch October 17, 2023 09:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Development

Successfully merging this pull request may close these issues.

2 participants