From ae03b08c250fd471f3cfb85625b48840c766d78e Mon Sep 17 00:00:00 2001 From: Jacob Valdez Date: Tue, 9 Dec 2025 16:12:49 -0600 Subject: [PATCH] Docs: Creating upgrade guide content for annotation table bloat (#114883) --- .../upgrade-guide/upgrade-v12.0/index.md | 50 ++++++++++++++++++ .../upgrade-guide/upgrade-v12.1/index.md | 50 ++++++++++++++++++ .../upgrade-guide/upgrade-v12.2/index.md | 52 +++++++++++++++++++ .../upgrade-guide/upgrade-v12.3/index.md | 52 +++++++++++++++++++ 4 files changed, 204 insertions(+) diff --git a/docs/sources/upgrade-guide/upgrade-v12.0/index.md b/docs/sources/upgrade-guide/upgrade-v12.0/index.md index 779950c0fd4..feb1061dceb 100644 --- a/docs/sources/upgrade-guide/upgrade-v12.0/index.md +++ b/docs/sources/upgrade-guide/upgrade-v12.0/index.md @@ -80,3 +80,53 @@ Since Grafana 10.2, the endpoint to check compatible versions when installing a #### What if I want to ignore the compatibility check? We _do not_ recommend installing plugins declared as incompatible. However, if you need to force install a plugin despite it being declared as incompatible, refer to the [Installing a plugin from a ZIP](https://grafana.com/docs/grafana/latest/administration/plugin-management/#install-a-plugin-from-a-zip-file) guidance. + +### PostgreSQL annotation table migration + +**Plan for increased disk usage when upgrading from Grafana v11.x** + +Upgrading from Grafana v11.x to Grafana v12.x triggers a full-table rewrite of the PostgreSQL `annotation` table. The migration populates the new `dashboard_uid` column, which causes PostgreSQL to rewrite the entire table and rebuild its indexes. + +Environments with large annotation datasets can experience significant temporary disk usage increase, which may lead to: + +- Rapid disk consumption on the PostgreSQL data volume +- Database migration failures (for example, "could not extend file: No space left on device") +- Grafana startup failures +- Extended downtime during the upgrade process + +#### How do I know if I'm affected? + +You're affected if you're upgrading from Grafana v11.x to v12.x and you have a large `annotation` table in your PostgreSQL database. + +To check your annotation table size, connect to your PostgreSQL database and run the following query: + +```sql +SELECT + pg_size_pretty(pg_relation_size('annotation')) AS table_size, + pg_size_pretty(pg_indexes_size('annotation')) AS indexes_size, + pg_size_pretty(pg_total_relation_size('annotation')) AS total_size; +``` + +If your total size is several gigabytes or more, you should plan accordingly before upgrading. + +#### What should I do before upgrading? + +Before you upgrade, take the following steps: + +1. **Verify available disk space**: Ensure you have at least 2-3 times the current `annotation` table size available as free disk space on your PostgreSQL data volume. + +2. **Review your annotation data**: Consider whether you need to retain all historical annotations. + +3. **Clean up old annotations (optional)**: If you have annotations you don't need, remove them before upgrading. + +4. **Back up your database**: Always back up your Grafana database before performing an upgrade. For more information, refer to [Back up Grafana](#back-up-grafana). + +#### What should I do after upgrading? + +After successfully upgrading to Grafana v12.x, you can reclaim disk space by running a `VACUUM FULL` operation on the `annotation` table during a maintenance window: + +```sql +VACUUM FULL annotation; +``` + +This operation requires a lock on the table and may take significant time depending on the table size. Plan to run this during a low-traffic period. diff --git a/docs/sources/upgrade-guide/upgrade-v12.1/index.md b/docs/sources/upgrade-guide/upgrade-v12.1/index.md index 74095298c90..7dbcb11ece9 100644 --- a/docs/sources/upgrade-guide/upgrade-v12.1/index.md +++ b/docs/sources/upgrade-guide/upgrade-v12.1/index.md @@ -20,3 +20,53 @@ weight: 499 {{< docs/shared lookup="upgrade/upgrade-common-tasks.md" source="grafana" version="" >}} ## Technical notes + +### PostgreSQL annotation table migration + +**Plan for increased disk usage when upgrading from Grafana v11.x** + +Upgrading from Grafana v11.x to Grafana v12.x triggers a full-table rewrite of the PostgreSQL `annotation` table. The migration populates the new `dashboard_uid` column, which causes PostgreSQL to rewrite the entire table and rebuild its indexes. + +Environments with large annotation datasets can experience significant temporary disk usage increase, which may lead to: + +- Rapid disk consumption on the PostgreSQL data volume +- Database migration failures (for example, "could not extend file: No space left on device") +- Grafana startup failures +- Extended downtime during the upgrade process + +#### How do I know if I'm affected? + +You're affected if you're upgrading from Grafana v11.x to v12.x and you have a large `annotation` table in your PostgreSQL database. + +To check your annotation table size, connect to your PostgreSQL database and run the following query: + +```sql +SELECT + pg_size_pretty(pg_relation_size('annotation')) AS table_size, + pg_size_pretty(pg_indexes_size('annotation')) AS indexes_size, + pg_size_pretty(pg_total_relation_size('annotation')) AS total_size; +``` + +If your total size is several gigabytes or more, you should plan accordingly before upgrading. + +#### What should I do before upgrading? + +Before you upgrade, take the following steps: + +1. **Verify available disk space**: Ensure you have at least 2-3 times the current `annotation` table size available as free disk space on your PostgreSQL data volume. + +2. **Review your annotation data**: Consider whether you need to retain all historical annotations. + +3. **Clean up old annotations (optional)**: If you have annotations you don't need, remove them before upgrading. + +4. **Back up your database**: Always back up your Grafana database before performing an upgrade. For more information, refer to [Back up Grafana](#back-up-grafana). + +#### What should I do after upgrading? + +After successfully upgrading to Grafana v12.x, you can reclaim disk space by running a `VACUUM FULL` operation on the `annotation` table during a maintenance window: + +```sql +VACUUM FULL annotation; +``` + +This operation requires a lock on the table and may take significant time depending on the table size. Plan to run this during a low-traffic period. diff --git a/docs/sources/upgrade-guide/upgrade-v12.2/index.md b/docs/sources/upgrade-guide/upgrade-v12.2/index.md index d2605fc51d1..1e900d65793 100644 --- a/docs/sources/upgrade-guide/upgrade-v12.2/index.md +++ b/docs/sources/upgrade-guide/upgrade-v12.2/index.md @@ -18,3 +18,55 @@ weight: 498 {{< docs/shared lookup="back-up/back-up-grafana.md" source="grafana" version="" leveloffset="+1" >}} {{< docs/shared lookup="upgrade/upgrade-common-tasks.md" source="grafana" version="" >}} + +## Technical notes + +### PostgreSQL annotation table migration + +**Plan for increased disk usage when upgrading from Grafana v11.x** + +Upgrading from Grafana v11.x to Grafana v12.x triggers a full-table rewrite of the PostgreSQL `annotation` table. The migration populates the new `dashboard_uid` column, which causes PostgreSQL to rewrite the entire table and rebuild its indexes. + +Environments with large annotation datasets can experience significant temporary disk usage increase, which may lead to: + +- Rapid disk consumption on the PostgreSQL data volume +- Database migration failures (for example, "could not extend file: No space left on device") +- Grafana startup failures +- Extended downtime during the upgrade process + +#### How do I know if I'm affected? + +You're affected if you're upgrading from Grafana v11.x to v12.x and you have a large `annotation` table in your PostgreSQL database. + +To check your annotation table size, connect to your PostgreSQL database and run the following query: + +```sql +SELECT + pg_size_pretty(pg_relation_size('annotation')) AS table_size, + pg_size_pretty(pg_indexes_size('annotation')) AS indexes_size, + pg_size_pretty(pg_total_relation_size('annotation')) AS total_size; +``` + +If your total size is several gigabytes or more, you should plan accordingly before upgrading. + +#### What should I do before upgrading? + +Before you upgrade, take the following steps: + +1. **Verify available disk space**: Ensure you have at least 2-3 times the current `annotation` table size available as free disk space on your PostgreSQL data volume. + +2. **Review your annotation data**: Consider whether you need to retain all historical annotations. + +3. **Clean up old annotations (optional)**: If you have annotations you don't need, remove them before upgrading. + +4. **Back up your database**: Always back up your Grafana database before performing an upgrade. For more information, refer to [Back up Grafana](#back-up-grafana). + +#### What should I do after upgrading? + +After successfully upgrading to Grafana v12.x, you can reclaim disk space by running a `VACUUM FULL` operation on the `annotation` table during a maintenance window: + +```sql +VACUUM FULL annotation; +``` + +This operation requires a lock on the table and may take significant time depending on the table size. Plan to run this during a low-traffic period. diff --git a/docs/sources/upgrade-guide/upgrade-v12.3/index.md b/docs/sources/upgrade-guide/upgrade-v12.3/index.md index 8d3dacbf396..dc8b1b8e398 100644 --- a/docs/sources/upgrade-guide/upgrade-v12.3/index.md +++ b/docs/sources/upgrade-guide/upgrade-v12.3/index.md @@ -18,3 +18,55 @@ weight: 497 {{< docs/shared lookup="back-up/back-up-grafana.md" source="grafana" version="" leveloffset="+1" >}} {{< docs/shared lookup="upgrade/upgrade-common-tasks.md" source="grafana" version="" >}} + +## Technical notes + +### PostgreSQL annotation table migration + +**Plan for increased disk usage when upgrading from Grafana v11.x** + +Upgrading from Grafana v11.x to Grafana v12.x triggers a full-table rewrite of the PostgreSQL `annotation` table. The migration populates the new `dashboard_uid` column, which causes PostgreSQL to rewrite the entire table and rebuild its indexes. + +Environments with large annotation datasets can experience significant temporary disk usage increase, which may lead to: + +- Rapid disk consumption on the PostgreSQL data volume +- Database migration failures (for example, "could not extend file: No space left on device") +- Grafana startup failures +- Extended downtime during the upgrade process + +#### How do I know if I'm affected? + +You're affected if you're upgrading from Grafana v11.x to v12.x and you have a large `annotation` table in your PostgreSQL database. + +To check your annotation table size, connect to your PostgreSQL database and run the following query: + +```sql +SELECT + pg_size_pretty(pg_relation_size('annotation')) AS table_size, + pg_size_pretty(pg_indexes_size('annotation')) AS indexes_size, + pg_size_pretty(pg_total_relation_size('annotation')) AS total_size; +``` + +If your total size is several gigabytes or more, you should plan accordingly before upgrading. + +#### What should I do before upgrading? + +Before you upgrade, take the following steps: + +1. **Verify available disk space**: Ensure you have at least 2-3 times the current `annotation` table size available as free disk space on your PostgreSQL data volume. + +2. **Review your annotation data**: Consider whether you need to retain all historical annotations. + +3. **Clean up old annotations (optional)**: If you have annotations you don't need, remove them before upgrading. + +4. **Back up your database**: Always back up your Grafana database before performing an upgrade. For more information, refer to [Back up Grafana](#back-up-grafana). + +#### What should I do after upgrading? + +After successfully upgrading to Grafana v12.x, you can reclaim disk space by running a `VACUUM FULL` operation on the `annotation` table during a maintenance window: + +```sql +VACUUM FULL annotation; +``` + +This operation requires a lock on the table and may take significant time depending on the table size. Plan to run this during a low-traffic period.