Migrations failing when upgrading to 1.12.1 #5244
Labels
Comments
|
this issue has also been reproduced on my end. |
Rolling back to v1.12.0 is not a problem, but this problem will occur on v1.12.1. |
|
can also confirm this is happening on a fresh installation using the given docker-compose files. |
|
Same issue on migrating from oncall |
|
Same issue installing helm chart v1.12.1, I use the v1.12.0 and the chart deployed all successfully |
|
Same issue on migrating from oncall 1.11.5 to 1.12.1 Let me add: after unsuccessful migration and rollback of the deployment to version 1.11.5, routes in integration in the web stopped showing. Through api query routes are visible. |
|
Just started a development instance and ran into this on both postgres and sqlite. I'm not sure if this is correct - but this is what I did to work around the issue for getting a dev instance to start up. (please take care if using this patch set against a prod instance) - also not sure how performant these migrations would be in the real world. diff --git a/engine/apps/alerts/migrations/0063_migrate_channelfilter_slack_channel_id.py b/engine/apps/alerts/migrations/0063_migrate_channelfilter_slack_channel_id.py
index dab5a459..9b589024 100644
--- a/engine/apps/alerts/migrations/0063_migrate_channelfilter_slack_channel_id.py
+++ b/engine/apps/alerts/migrations/0063_migrate_channelfilter_slack_channel_id.py
@@ -15,14 +15,20 @@ def populate_slack_channel(apps, schema_editor):
logger.info("Starting migration to populate slack_channel field.")
sql = f"""
- UPDATE {ChannelFilter._meta.db_table} AS cf
- JOIN {AlertReceiveChannel._meta.db_table} AS arc ON arc.id = cf.alert_receive_channel_id
- JOIN {Organization._meta.db_table} AS org ON org.id = arc.organization_id
- JOIN {SlackChannel._meta.db_table} AS sc ON sc.slack_id = cf._slack_channel_id
- AND sc.slack_team_identity_id = org.slack_team_identity_id
- SET cf.slack_channel_id = sc.id
- WHERE cf._slack_channel_id IS NOT NULL
- AND org.slack_team_identity_id IS NOT NULL;
+
+ with temp as (
+ SELECT cf_s.slack_channel_id as slack_channel_id, sc.id as id
+ FROM {ChannelFilter._meta.db_table} AS cf_s
+ JOIN {AlertReceiveChannel._meta.db_table} AS arc ON arc.id = cf_s.alert_receive_channel_id
+ JOIN {Organization._meta.db_table} AS org ON org.id = arc.organization_id
+ JOIN {SlackChannel._meta.db_table} AS sc ON sc.slack_id = cf_s._slack_channel_id
+ AND sc.slack_team_identity_id = org.slack_team_identity_id
+ WHERE org.slack_team_identity_id IS NOT NULL and cf_s._slack_channel_id IS NOT NULL
+ )
+ UPDATE {ChannelFilter._meta.db_table} as update_cf
+ SET slack_channel_id = temp.id
+ FROM temp
+ where update_cf.slack_channel_id = temp.slack_channel_id
"""
with schema_editor.connection.cursor() as cursor:
diff --git a/engine/apps/alerts/migrations/0064_migrate_resolutionnoteslackmessage_slack_channel_id.py b/engine/apps/alerts/migrations/0064_migrate_resolutionnoteslackmessage_slack_channel_id.py
index 4f492e31..a59254ca 100644
--- a/engine/apps/alerts/migrations/0064_migrate_resolutionnoteslackmessage_slack_channel_id.py
+++ b/engine/apps/alerts/migrations/0064_migrate_resolutionnoteslackmessage_slack_channel_id.py
@@ -17,15 +17,24 @@ def populate_slack_channel(apps, schema_editor):
logger.info("Starting migration to populate slack_channel field.")
sql = f"""
- UPDATE {ResolutionNoteSlackMessage._meta.db_table} AS rsm
- JOIN {AlertGroup._meta.db_table} AS ag ON ag.id = rsm.alert_group_id
+
+ with temp as (
+ SELECT rsm_t._slack_channel_id as _slack_channel_id, sc.id as id
+ FROM {ResolutionNoteSlackMessage._meta.db_table} AS rsm_t
+ JOIN {AlertGroup._meta.db_table} AS ag ON ag.id = rsm_t.alert_group_id
JOIN {AlertReceiveChannel._meta.db_table} AS arc ON arc.id = ag.channel_id
JOIN {Organization._meta.db_table} AS org ON org.id = arc.organization_id
- JOIN {SlackChannel._meta.db_table} AS sc ON sc.slack_id = rsm._slack_channel_id
+ JOIN {SlackChannel._meta.db_table} AS sc ON sc.slack_id = rsm_t._slack_channel_id
AND sc.slack_team_identity_id = org.slack_team_identity_id
- SET rsm.slack_channel_id = sc.id
- WHERE rsm._slack_channel_id IS NOT NULL
- AND org.slack_team_identity_id IS NOT NULL;
+ WHERE rsm_t._slack_channel_id IS NOT NULL
+ AND org.slack_team_identity_id IS NOT NULL
+ )
+
+
+ UPDATE {ResolutionNoteSlackMessage._meta.db_table} AS rsm
+ SET slack_channel_id = temp.id
+ FROM temp
+ WHERE rsm._slack_channel_id = temp._slack_channel_id
"""
with schema_editor.connection.cursor() as cursor:
diff --git a/engine/apps/schedules/migrations/0019_auto_20241021_1735.py b/engine/apps/schedules/migrations/0019_auto_20241021_1735.py
index edc89366..518f16ed 100644
--- a/engine/apps/schedules/migrations/0019_auto_20241021_1735.py
+++ b/engine/apps/schedules/migrations/0019_auto_20241021_1735.py
@@ -14,13 +14,21 @@ def populate_slack_channel(apps, schema_editor):
logger.info("Starting migration to populate slack_channel field.")
sql = f"""
+ with temp as (
+ SELECT ocs_t.slack_channel_id as slack_channel_id, sc.id as id
+ FROM {OnCallSchedule._meta.db_table} AS ocs_t
+ JOIN {Organization._meta.db_table} AS org ON org.id = ocs_t.organization_id
+ JOIN {SlackChannel._meta.db_table} AS sc ON sc.slack_id = ocs_t.channel
+ AND sc.slack_team_identity_id = org.slack_team_identity_id
+ WHERE ocs_t.channel IS NOT NULL
+ AND org.slack_team_identity_id IS NOT NULL
+ )
+
UPDATE {OnCallSchedule._meta.db_table} AS ocs
- JOIN {Organization._meta.db_table} AS org ON org.id = ocs.organization_id
- JOIN {SlackChannel._meta.db_table} AS sc ON sc.slack_id = ocs.channel
- AND sc.slack_team_identity_id = org.slack_team_identity_id
- SET ocs.slack_channel_id = sc.id
- WHERE ocs.channel IS NOT NULL
- AND org.slack_team_identity_id IS NOT NULL;
+ SET slack_channel_id = temp.id
+ FROM temp
+ WHERE ocs.slack_channel_id = temp.slack_channel_id
+
"""
with schema_editor.connection.cursor() as cursor:
diff --git a/engine/apps/user_management/migrations/0026_auto_20241017_1919.py b/engine/apps/user_management/migrations/0026_auto_20241017_1919.py
index df28b026..9006d0f8 100644
--- a/engine/apps/user_management/migrations/0026_auto_20241017_1919.py
+++ b/engine/apps/user_management/migrations/0026_auto_20241017_1919.py
@@ -14,12 +14,20 @@ def populate_default_slack_channel(apps, schema_editor):
logger.info("Starting migration to populate default_slack_channel field.")
sql = f"""
+ with temp as (
+ SELECT org_t.default_slack_channel_id as default_slack_channel_id, sc.id as id
+ FROM {Organization._meta.db_table} AS org_t
+ JOIN {SlackChannel._meta.db_table} AS sc ON sc.slack_id = org_t.general_log_channel_id
+ AND sc.slack_team_identity_id = org_t.slack_team_identity_id
+ WHERE org_t.general_log_channel_id IS NOT NULL
+ AND org_t.slack_team_identity_id IS NOT NULL
+ )
+
+
UPDATE {Organization._meta.db_table} AS org
- JOIN {SlackChannel._meta.db_table} AS sc ON sc.slack_id = org.general_log_channel_id
- AND sc.slack_team_identity_id = org.slack_team_identity_id
- SET org.default_slack_channel_id = sc.id
- WHERE org.general_log_channel_id IS NOT NULL
- AND org.slack_team_identity_id IS NOT NULL;
+ SET default_slack_channel_id = temp.id
+ FROM temp
+ WHERE org.default_slack_channel_id = temp.default_slack_channel_id
"""
with schema_editor.connection.cursor() as cursor: |
|
Migration from 1.12.0 to 1.13.1 worked fine. I guess it is best to skip 1.12.1. EDIT: Also from 1.13.1 to 1.13.2. Probably upgrading directly to 1.13.2 will work as well. |
Open
|
I wasn't able to go past 1.12.1, instead, I had to step through the individual migrations and |
This was the same experience for me. |
|
Check https://stackoverflow.com/a/7869611, the syntax in the migrations is mysql-only. |
3 tasks
|
hello! Yes, raw SQL was needed here 🙁 (as for reason's stated here, this sort of migration is not possible in a performant way using the ORM). The quickest way around this is by using I see @mwheeler-ep already has a starting point for postgres/sqlite compliant SQL, if someone wants to open a PR to patch these migration files to work on those databases as well (you can do something very similar to this upcoming migration file), we can definitely take a look at that! |
3 tasks
|
One option: keep the mysql-query and fall back to the slow mode in case of an error? Slow is still better than failing? |
3 tasks
|
@bpedersen2 thanks for the idea 👍 I'm planning on merging #5297 today and will cut a new release. For non-MySQL folks, once you upgrade to this new version, if you haven't already skipped the failing data migration files, they'll run successfully this time around, using these patch migration files. Sorry about the hassle ❤ |
joeyorlando
added a commit
that referenced
this issue
Nov 26, 2024
# What this PR does ## Which issue(s) this PR closes Fixes #5244 (comment) ## Checklist - [x] Unit, integration, and e2e (if applicable) tests updated - [x] Documentation added (or `pr:no public docs` PR label added if not required) - [x] Added the relevant release notes label (see labels prefixed w/ `release:`). These labels dictate how your PR will show up in the autogenerated release notes.
|
This has been patched in v1.13.4, please upgrade to this version and retry |
|
I just tried a new install today with v1.13.4 and I'm getting an error with the Full log: I'm using docker compose with sqlite3 as backend |
|
@exu-g what does the following look like in your db? SELECT * FROM django_migrations WHERE app = 'alerts'The previously listed migrations that were failing, |
|
I'm getting this list I'll look into skipping it, but again, I'm having this issue on a completely fresh install with a fresh database. |
|
I was upgrading from 1.11.1 directly to 1.13.4 and now Slack notifications are not working. In logs now I have following problems: |
|
For now I tried to fix the problem with following raw sql: It looks like I have no more errors in logs, but I have lot's of messages and Slack is ratelimiting oncall... Hopefully theese are tasks which was generated since problem occured.... |
|
@kvaster I think that should only happen for older alert group slack messages. Is it happening persistently for you? (it may just be a transient issue you're seeing; newer "alert group slack messages" will have the proper foreign key relationship in place for You can also try manually running the migration inside engine/apps/slack/0007_migrate_slackmessage_channel_id.py (note that this is explicitly not added to the Alternatively, you can try running this $ python manage.py batch_migrate_slack_message_channelWith that said, as you can probably notice, we're in the process of doing some clean-up/refactoring around Slack messages, sorry for the turbulence ❤ |
|
I've examined the code and I've managed to fix my problem in two steps. First one with raw sql message above (I'm using postgresql). And second one can be done in sql or in UI - I had to change forward/back default slack channel for the integrations, cause in other way oncall was not able to find slack channel (refers to sql table alerts_channelfilter - slack_channel_id was undefined here). Also I have two clusters (i.e. dev and prd) and first problem was only on dev cluster, but problem with channelfilter table was on both clusters. |
3 tasks
joeyorlando
added a commit
that referenced
this issue
Nov 28, 2024
## Which issue(s) this PR closes Fixes #5306 (and related to #5244 (comment)) + add CI check to avoid this from happening in the future ## Checklist - [x] Unit, integration, and e2e (if applicable) tests updated - [x] Documentation added (or `pr:no public docs` PR label added if not required) - [x] Added the relevant release notes label (see labels prefixed w/ `release:`). These labels dictate how your PR will show up in the autogenerated release notes.
|
For those hitting issues specifically on SQLite w/ migration We've also added some tests on CI to start running the migration files against MySQL, Postgres, and SQLite to ensure compatibility going forward. Please upgrade and retry. Should things persist, please open a new issue ✌ |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
What went wrong?
What happened:
What did you expect to happen:
How do we reproduce it?
Grafana OnCall Version
v1.12.1
Product Area
Helm/Kubernetes/Docker
Grafana OnCall Platform?
Kubernetes
User's Browser?
No response
Anything else to add?
No response
The text was updated successfully, but these errors were encountered: