Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Silence strong migration warning #1037

Merged
merged 5 commits into from
Dec 1, 2023
Merged

Conversation

ngetahun
Copy link
Contributor

@ngetahun ngetahun commented Oct 5, 2023

Description

Strong migration shows lock timeout limit exceeding with migrations. This PR silences such warnings.

Change Type

Please select the correct option.

  • Bug Fix (a non-breaking change which fixes an issue)
  • New Feature (a non-breaking change which adds new functionality)
  • Documentation Update (a change which only updates documentation)

Checklist

Please check off each item if the requirement is met.

  • I have verified that my code follows RMT's coding standards with rubocop.
  • I have reviewed my own code and believe that it's ready for an external review.
  • I have provided comments for any hard-to-understand code.
  • I have documented the MANUAL.md file with any changes to the user experience.
  • RMT's test coverage remains at 100%.
  • If my changes are non-trivial, I have added a changelog entry to notify users at package/obs/rmt-server.changes.

Other Notes

Please use this space to provide notes or thoughts to the team, such as tips on how to review/demo your changes.

@SUSE SUSE deleted a comment from suse-tests-pass Oct 9, 2023
@SUSE SUSE deleted a comment from suse-tests-pass Oct 9, 2023
@SUSE SUSE deleted a comment from suse-tests-pass Oct 11, 2023
@SUSE SUSE deleted a comment from suse-tests-pass Oct 26, 2023
@SUSE SUSE deleted a comment from suse-tests-pass Oct 26, 2023
@SUSE SUSE deleted a comment from suse-tests-pass Oct 26, 2023
@SUSE SUSE deleted a comment from suse-tests-pass Oct 26, 2023
@SUSE SUSE deleted a comment from suse-tests-pass Oct 26, 2023
@SUSE SUSE deleted a comment from suse-tests-pass Nov 22, 2023
@SUSE SUSE deleted a comment from suse-tests-pass Nov 28, 2023
@suse-tests-pass
Copy link
Collaborator

Well Done! Your tests are still passing.
https://ci.suse.de/job/scc-RMT-integration-tests/345442/console
If the given link has expired,you can force a Prophet rerun by just deleting this comment. (Merged c15cb41 into b633992)

@@ -1,6 +1,6 @@
# rubocop:disable Style/NumericLiterals
unless Rails.env.production?
StrongMigrations.start_after = 20200205123840
StrongMigrations.lock_timeout_limit = 86400.seconds
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a timeout of 24h, do we expect a migration to take longer?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The problem with this check is, that it not actually checks how long a transaction takes but rather what mariadb is configured to (aka when to cancel a transaction).

See: https://dev.mysql.com/doc/refman/8.0/en/server-system-variables.html#sysvar_lock_wait_timeout which is fetched by the check here: https://github.com/ankane/strong_migrations/blob/master/lib/strong_migrations/adapters/mysql_adapter.rb#L35

To make this sane, we would need to ship our own mariadb configuration since the default is 31536000 and us setting 86400 will trigger this check and fail always with default configuration.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alright, I see. Why not remove that line entirely then instead of setting to 0?

In glue we're setting timeouts for migrations, which I guess was the goal here too when we added that line:

StrongMigrations.lock_timeout = 20.seconds
StrongMigrations.statement_timeout = 2.minutes

Would it make sense to set some limits for RMT too, so that a migration cannot block the db endlessly?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree we should set the lock_timeout and statement_timeout, then also the lock_timeout_limit works. But without you will run into: https://github.com/ankane/strong_migrations/blob/master/lib/strong_migrations/checker.rb#L162 which then is setting by default: https://github.com/ankane/strong_migrations/blob/master/lib/strong_migrations.rb#L59

I'm not sure if setting a statement_timeout is really wanted, since a migration can take a long time which is completely fine.

Copy link
Member

@digitaltom digitaltom Nov 30, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, ok that's unexpected that strong migrations sets it's own default and fails then. Maybe a comment explaining this would help.

Regarding the statement_timeout, it's possible to change it to a higher value for specific migrations that are known to take long.

Anyway, strong_migrations isn't used in production anyway it seems in RMT.

@felixsch felixsch merged commit 017d3be into master Dec 1, 2023
2 checks passed
@felixsch felixsch deleted the silence_strong_migration_warning branch December 1, 2023 17:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants