From 8172fe542e887cc8edb5bd986532d038b2f3dae9 Mon Sep 17 00:00:00 2001 From: Jennifer Travinski Date: Tue, 16 Nov 2021 17:28:35 -0500 Subject: [PATCH 1/8] Added note to redirect to troubleshooting section --- content/rancher/v2.5/en/backups/back-up-rancher/_index.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/content/rancher/v2.5/en/backups/back-up-rancher/_index.md b/content/rancher/v2.5/en/backups/back-up-rancher/_index.md index 51cdf22dfc1..61376329171 100644 --- a/content/rancher/v2.5/en/backups/back-up-rancher/_index.md +++ b/content/rancher/v2.5/en/backups/back-up-rancher/_index.md @@ -28,6 +28,8 @@ Backups are created as .tar.gz files. These files can be pushed to S3 or Minio, 1. Click **Rancher Backups.** 1. Configure the default storage location. For help, refer to the [storage configuration section.](../configuration/storage-config) +>**NOTE:** There is a [known Fleet issue](https://github.com/rancher/backup-restore-operator/issues/164) in Rancher v2.5 that occurs after performing a restoration using the backup-restore-operator. Fleet agents in both the local and downstream Rancher clusters are unable to connect. A temporary workaround may be found [here]({{}}rancher/v2.5/en/deploy-across-clusters/fleet/#troubleshooting). + ### 2. Perform a Backup To perform a backup, a custom resource of type Backup must be created. From e29cd7226f3c198bf080a9605de9aed1fa30fab7 Mon Sep 17 00:00:00 2001 From: Jennifer Travinski Date: Tue, 16 Nov 2021 17:28:59 -0500 Subject: [PATCH 2/8] Added temp workaround for Fleet issue in Rancher --- .../v2.5/en/deploy-across-clusters/fleet/_index.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/content/rancher/v2.5/en/deploy-across-clusters/fleet/_index.md b/content/rancher/v2.5/en/deploy-across-clusters/fleet/_index.md index abea9bad31e..07652cea919 100644 --- a/content/rancher/v2.5/en/deploy-across-clusters/fleet/_index.md +++ b/content/rancher/v2.5/en/deploy-across-clusters/fleet/_index.md @@ -16,6 +16,7 @@ Fleet is a separate project from Rancher, and can be installed on any Kubernetes - [Windows Support](#windows-support) - [GitHub Repository](#github-repository) - [Using Fleet Behind a Proxy](#using-fleet-behind-a-proxy) +- [Troubleshooting](#troubleshooting) - [Documentation](#documentation) # Architecture @@ -44,6 +45,16 @@ _Available as of v2.5.8_ For details on using Fleet behind a proxy, see [this page.](./proxy) +# Troubleshooting + +- **Known Issue:** Fleet in Rancher v2.5 becomes inoperable after a restore using the [backup-restore operator](({{}}rancher/v2.5/en/backups/back-up-rancher/#install-the-rancher-backup-operator)). We will update the community once a permanent solution is in place. +- **Temporary Workaround:**
+ 1. Find the two service account tokens listed in the fleet-controller and the fleet-controller-bootstrap service accounts. These are under the fleet-system Namespace of the local cluster.
+ 2. Remove the non-existent token secret. Doing so allows for only one entry to be present for the service account token secret that actually exists.
+ 3. Delete the fleet-controller Pod in the fleet-system Namespace to reschedule.
+ 4. After the service account token issue is resolved, you can force redeployment of the fleet-agents. In the Rancher UI, go to **☰ > Cluster Management**, click on **Clusters** page, then click **Force Update**.
+ 5. If the fleet-agent bundles remain in a `Modified` state after Step 4, use the command `spec.forceSyncGeneration` for fleet-agent bundle to force re-creation. + # Documentation The Fleet documentation is at [https://fleet.rancher.io/.](https://fleet.rancher.io/) From 9d2e90f5bf44c2515fb0e14ce4df963de735eb98 Mon Sep 17 00:00:00 2001 From: Jennifer Travinski Date: Wed, 17 Nov 2021 11:27:09 -0500 Subject: [PATCH 3/8] Updated ts comments --- .../en/deploy-across-clusters/fleet/_index.md | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-) diff --git a/content/rancher/v2.5/en/deploy-across-clusters/fleet/_index.md b/content/rancher/v2.5/en/deploy-across-clusters/fleet/_index.md index 07652cea919..e8fc3a2d78c 100644 --- a/content/rancher/v2.5/en/deploy-across-clusters/fleet/_index.md +++ b/content/rancher/v2.5/en/deploy-across-clusters/fleet/_index.md @@ -47,14 +47,18 @@ For details on using Fleet behind a proxy, see [this page.](./proxy) # Troubleshooting -- **Known Issue:** Fleet in Rancher v2.5 becomes inoperable after a restore using the [backup-restore operator](({{}}rancher/v2.5/en/backups/back-up-rancher/#install-the-rancher-backup-operator)). We will update the community once a permanent solution is in place. -- **Temporary Workaround:**
- 1. Find the two service account tokens listed in the fleet-controller and the fleet-controller-bootstrap service accounts. These are under the fleet-system Namespace of the local cluster.
- 2. Remove the non-existent token secret. Doing so allows for only one entry to be present for the service account token secret that actually exists.
- 3. Delete the fleet-controller Pod in the fleet-system Namespace to reschedule.
- 4. After the service account token issue is resolved, you can force redeployment of the fleet-agents. In the Rancher UI, go to **☰ > Cluster Management**, click on **Clusters** page, then click **Force Update**.
- 5. If the fleet-agent bundles remain in a `Modified` state after Step 4, use the command `spec.forceSyncGeneration` for fleet-agent bundle to force re-creation. +**Known Issue:** +Fleet in Rancher v2.5 becomes inoperable after a restore using the [backup-restore operator]({{}}rancher/v2.5/en/backups/back-up-rancher/#1-install-the-rancher-backup-operator). We will update the community once a permanent solution is in place. + +**Temporary Workaround**: + +1. Find the two service account tokens listed in the fleet-controller and the fleet-controller-bootstrap service accounts. These are under the fleet-system Namespace of the local cluster.
+2. Remove the non-existent token secret. Doing so allows for only one entry to be present for the service account token secret that actually exists.
+3. Delete the fleet-controller Pod in the fleet-system Namespace to reschedule.
+4. After the service account token issue is resolved, you can force redeployment of the fleet-agents. In the Rancher UI, go to **☰ > Cluster Management**, click on **Clusters** page, then click **Force Update**.
+5. If the fleet-agent bundles remain in a `Modified` state after Step 4, use the command `spec.forceSyncGeneration` for fleet-agent bundle to force re-creation. + # Documentation The Fleet documentation is at [https://fleet.rancher.io/.](https://fleet.rancher.io/) From 514edd8d67f07c467ffbfd66b44345da9590a66e Mon Sep 17 00:00:00 2001 From: Jennifer Travinski Date: Wed, 17 Nov 2021 15:02:14 -0500 Subject: [PATCH 4/8] Added 2nd known issue to TS section for Fleet --- .../v2.5/en/backups/back-up-rancher/_index.md | 2 +- .../en/deploy-across-clusters/fleet/_index.md | 21 ++++++++++++------- 2 files changed, 14 insertions(+), 9 deletions(-) diff --git a/content/rancher/v2.5/en/backups/back-up-rancher/_index.md b/content/rancher/v2.5/en/backups/back-up-rancher/_index.md index 61376329171..aea4c343489 100644 --- a/content/rancher/v2.5/en/backups/back-up-rancher/_index.md +++ b/content/rancher/v2.5/en/backups/back-up-rancher/_index.md @@ -28,7 +28,7 @@ Backups are created as .tar.gz files. These files can be pushed to S3 or Minio, 1. Click **Rancher Backups.** 1. Configure the default storage location. For help, refer to the [storage configuration section.](../configuration/storage-config) ->**NOTE:** There is a [known Fleet issue](https://github.com/rancher/backup-restore-operator/issues/164) in Rancher v2.5 that occurs after performing a restoration using the backup-restore-operator. Fleet agents in both the local and downstream Rancher clusters are unable to connect. A temporary workaround may be found [here]({{}}rancher/v2.5/en/deploy-across-clusters/fleet/#troubleshooting). +>**NOTE:** There are two known issues in Fleet that occur after performing a restoration using the backup-restore-operator: Fleet agents are inoperable and clientSecretName and helmSecretName are not included in Fleet gitrepos. Refer [here]({{}}rancher/v2.5/en/deploy-across-clusters/fleet/#troubleshooting) for workarounds. ### 2. Perform a Backup diff --git a/content/rancher/v2.5/en/deploy-across-clusters/fleet/_index.md b/content/rancher/v2.5/en/deploy-across-clusters/fleet/_index.md index e8fc3a2d78c..96cb5d0611a 100644 --- a/content/rancher/v2.5/en/deploy-across-clusters/fleet/_index.md +++ b/content/rancher/v2.5/en/deploy-across-clusters/fleet/_index.md @@ -46,18 +46,23 @@ _Available as of v2.5.8_ For details on using Fleet behind a proxy, see [this page.](./proxy) # Troubleshooting +--- +* **Known Issue:** Fleet becomes inoperable after a restore using the [backup-restore-operator]({{}}rancher/v2.5/en/backups/back-up-rancher/#1-install-the-rancher-backup-operator). We will update the community once a permanent solution is in place. -**Known Issue:** +* **Temporary Workaround:**
+ 1. Find the two service account tokens listed in the fleet-controller and the fleet-controller-bootstrap service accounts. These are under the fleet-system Namespace of the local cluster.
+ 2. Remove the non-existent token secret. Doing so allows for only one entry to be present for the service account token secret that actually exists.
+ 3. Delete the fleet-controller Pod in the fleet-system Namespace to reschedule.
+ 4. After the service account token issue is resolved, you can force redeployment of the fleet-agents. In the Rancher UI, go to **☰ > Cluster Management**, click on **Clusters** page, then click **Force Update**.
+ 5. If the fleet-agent bundles remain in a `Modified` state after Step 4, use the command `spec.forceSyncGeneration` for the fleet-agent bundle to force re-creation. -Fleet in Rancher v2.5 becomes inoperable after a restore using the [backup-restore operator]({{}}rancher/v2.5/en/backups/back-up-rancher/#1-install-the-rancher-backup-operator). We will update the community once a permanent solution is in place. +--- +* **Known Issue:** clientSecretName and helmSecretName secrets for Fleet gitrepos are not included in the backup nor restore created by the [backup-restore-operator]({{}}rancher/v2.5/en/backups/back-up-rancher/#1-install-the-rancher-backup-operator). We will update the community once a permanent solution is in place. -**Temporary Workaround**: +* **Temporary Workaround:**
+By default, user-defined secrets are not backed up in Fleet. It is necessary to recreate secrets if performing a disaster recovery restore or migration of Rancher into a fresh cluster. To modify resourceSet to include extra resources you want to backup, refer to docs [here](https://github.com/rancher/backup-restore-operator#user-flow). -1. Find the two service account tokens listed in the fleet-controller and the fleet-controller-bootstrap service accounts. These are under the fleet-system Namespace of the local cluster.
-2. Remove the non-existent token secret. Doing so allows for only one entry to be present for the service account token secret that actually exists.
-3. Delete the fleet-controller Pod in the fleet-system Namespace to reschedule.
-4. After the service account token issue is resolved, you can force redeployment of the fleet-agents. In the Rancher UI, go to **☰ > Cluster Management**, click on **Clusters** page, then click **Force Update**.
-5. If the fleet-agent bundles remain in a `Modified` state after Step 4, use the command `spec.forceSyncGeneration` for fleet-agent bundle to force re-creation. +--- # Documentation From bbb14278a7e9f964c8f1af8bfa829757ab990e7a Mon Sep 17 00:00:00 2001 From: Jennifer Travinski Date: Wed, 17 Nov 2021 15:26:15 -0500 Subject: [PATCH 5/8] Adding notes for workarounds in 2.5, 2.6 --- content/rancher/v2.5/en/backups/_index.md | 2 ++ content/rancher/v2.6/en/backups/_index.md | 2 ++ .../rancher/v2.6/en/backups/back-up-rancher/_index.md | 2 ++ .../v2.6/en/deploy-across-clusters/fleet/_index.md | 11 +++++++++++ 4 files changed, 17 insertions(+) diff --git a/content/rancher/v2.5/en/backups/_index.md b/content/rancher/v2.5/en/backups/_index.md index 0f765d2d84d..01b07a0db84 100644 --- a/content/rancher/v2.5/en/backups/_index.md +++ b/content/rancher/v2.5/en/backups/_index.md @@ -63,6 +63,8 @@ The Backup and Restore custom resources can be created in the Rancher UI, or by The `rancher-backup` operator can be installed from the Rancher UI, or with the Helm CLI. In both cases, the `rancher-backup` Helm chart is installed on the Kubernetes cluster running the Rancher server. It is a cluster-admin only feature and available only for the **local** cluster. (*If you do not see `rancher-backup` in the Rancher UI, you may have selected the wrong cluster.*) +>**NOTE:** There are two known issues in Fleet that occur after performing a restoration using the backup-restore-operator: Fleet agents are inoperable and clientSecretName and helmSecretName are not included in Fleet gitrepos. Refer [here]({{}}rancher/v2.5/en/deploy-across-clusters/fleet/#troubleshooting) for workarounds. + ### Installing rancher-backup with the Rancher UI 1. In the Rancher UI's Cluster Manager, choose the cluster named **local** diff --git a/content/rancher/v2.6/en/backups/_index.md b/content/rancher/v2.6/en/backups/_index.md index b090f2972b2..9aec8aae77f 100644 --- a/content/rancher/v2.6/en/backups/_index.md +++ b/content/rancher/v2.6/en/backups/_index.md @@ -47,6 +47,8 @@ The Backup and Restore custom resources can be created in the Rancher UI, or by The `rancher-backup` operator can be installed from the Rancher UI, or with the Helm CLI. In both cases, the `rancher-backup` Helm chart is installed on the Kubernetes cluster running the Rancher server. It is a cluster-admin only feature and available only for the **local** cluster. (*If you do not see `rancher-backup` in the Rancher UI, you may have selected the wrong cluster.*) +>**NOTE:** There is a known issue in Fleet that occurs after performing a restoration using the backup-restore-operator: Secrets used for clientSecretName and helmSecretName are not included in Fleet gitrepos. Refer [here]({{}}rancher/v2.6/en/deploy-across-clusters/fleet/#troubleshooting) for a workaround. + ### Installing rancher-backup with the Rancher UI 1. In the upper left corner, click **☰ > Cluster Management**. diff --git a/content/rancher/v2.6/en/backups/back-up-rancher/_index.md b/content/rancher/v2.6/en/backups/back-up-rancher/_index.md index acfe23fc3fb..5364e06c2ba 100644 --- a/content/rancher/v2.6/en/backups/back-up-rancher/_index.md +++ b/content/rancher/v2.6/en/backups/back-up-rancher/_index.md @@ -25,6 +25,8 @@ Backups are created as .tar.gz files. These files can be pushed to S3 or Minio, 1. Configure the default storage location. For help, refer to the [storage configuration section.](../configuration/storage-config) 1. Click **Install**. +>**NOTE:** There is a known issue in Fleet that occurs after performing a restoration using the backup-restore-operator: Secrets used for clientSecretName and helmSecretName are not included in Fleet gitrepos. Refer [here]({{}}rancher/v2.6/en/deploy-across-clusters/fleet/#troubleshooting) for a workaround. + ### 2. Perform a Backup To perform a backup, a custom resource of type Backup must be created. diff --git a/content/rancher/v2.6/en/deploy-across-clusters/fleet/_index.md b/content/rancher/v2.6/en/deploy-across-clusters/fleet/_index.md index f9ef3cc818c..c59ee4dbbd1 100644 --- a/content/rancher/v2.6/en/deploy-across-clusters/fleet/_index.md +++ b/content/rancher/v2.6/en/deploy-across-clusters/fleet/_index.md @@ -12,6 +12,7 @@ Fleet is a separate project from Rancher, and can be installed on any Kubernetes - [Windows Support](#windows-support) - [GitHub Repository](#github-repository) - [Using Fleet Behind a Proxy](#using-fleet-behind-a-proxy) +- [Troubleshooting](#troubleshooting) - [Documentation](#documentation) # Architecture @@ -36,6 +37,16 @@ The Fleet Helm charts are available [here.](https://github.com/rancher/fleet/rel For details on using Fleet behind a proxy, see [this page.](./proxy) +# Troubleshooting + +--- +* **Known Issue:** clientSecretName and helmSecretName secrets for Fleet gitrepos are not included in the backup nor restore created by the [backup-restore-operator]({{}}rancher/v2.6/en/backups/back-up-rancher/#1-install-the-rancher-backups-operator). We will update the community once a permanent solution is in place. + +* **Temporary Workaround:**
+By default, user-defined secrets are not backed up in Fleet. It is necessary to recreate secrets if performing a disaster recovery restore or migration of Rancher into a fresh cluster. To modify resourceSet to include extra resources you want to backup, refer to docs [here](https://github.com/rancher/backup-restore-operator#user-flow). + +--- + # Documentation The Fleet documentation is at [https://fleet.rancher.io/.](https://fleet.rancher.io/) From 94ab27fcb778ade49a90aa20467b412d03f616ab Mon Sep 17 00:00:00 2001 From: Jennifer Travinski Date: Tue, 23 Nov 2021 12:42:10 -0500 Subject: [PATCH 6/8] Updated verbiage per feedback --- content/rancher/v2.5/en/deploy-across-clusters/fleet/_index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/rancher/v2.5/en/deploy-across-clusters/fleet/_index.md b/content/rancher/v2.5/en/deploy-across-clusters/fleet/_index.md index 96cb5d0611a..46850edc2ab 100644 --- a/content/rancher/v2.5/en/deploy-across-clusters/fleet/_index.md +++ b/content/rancher/v2.5/en/deploy-across-clusters/fleet/_index.md @@ -54,7 +54,7 @@ For details on using Fleet behind a proxy, see [this page.](./proxy) 2. Remove the non-existent token secret. Doing so allows for only one entry to be present for the service account token secret that actually exists.
3. Delete the fleet-controller Pod in the fleet-system Namespace to reschedule.
4. After the service account token issue is resolved, you can force redeployment of the fleet-agents. In the Rancher UI, go to **☰ > Cluster Management**, click on **Clusters** page, then click **Force Update**.
- 5. If the fleet-agent bundles remain in a `Modified` state after Step 4, use the command `spec.forceSyncGeneration` for the fleet-agent bundle to force re-creation. + 5. If the fleet-agent bundles remain in a `Modified` state after Step 4, update the field `spec.forceSyncGeneration` for the fleet-agent bundle to force re-creation. --- * **Known Issue:** clientSecretName and helmSecretName secrets for Fleet gitrepos are not included in the backup nor restore created by the [backup-restore-operator]({{}}rancher/v2.5/en/backups/back-up-rancher/#1-install-the-rancher-backup-operator). We will update the community once a permanent solution is in place. From f40f06ab425eba0280d2fefa166e9b0616bc2d4a Mon Sep 17 00:00:00 2001 From: Jen Travinski Date: Wed, 24 Nov 2021 14:38:22 -0500 Subject: [PATCH 7/8] Update content/rancher/v2.5/en/deploy-across-clusters/fleet/_index.md Co-authored-by: Billy Tat --- content/rancher/v2.5/en/deploy-across-clusters/fleet/_index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/rancher/v2.5/en/deploy-across-clusters/fleet/_index.md b/content/rancher/v2.5/en/deploy-across-clusters/fleet/_index.md index 46850edc2ab..61de588ea67 100644 --- a/content/rancher/v2.5/en/deploy-across-clusters/fleet/_index.md +++ b/content/rancher/v2.5/en/deploy-across-clusters/fleet/_index.md @@ -50,7 +50,7 @@ For details on using Fleet behind a proxy, see [this page.](./proxy) * **Known Issue:** Fleet becomes inoperable after a restore using the [backup-restore-operator]({{}}rancher/v2.5/en/backups/back-up-rancher/#1-install-the-rancher-backup-operator). We will update the community once a permanent solution is in place. * **Temporary Workaround:**
- 1. Find the two service account tokens listed in the fleet-controller and the fleet-controller-bootstrap service accounts. These are under the fleet-system Namespace of the local cluster.
+ 1. Find the two service account tokens listed in the fleet-controller and the fleet-controller-bootstrap service accounts. These are under the fleet-system namespace of the local cluster.
2. Remove the non-existent token secret. Doing so allows for only one entry to be present for the service account token secret that actually exists.
3. Delete the fleet-controller Pod in the fleet-system Namespace to reschedule.
4. After the service account token issue is resolved, you can force redeployment of the fleet-agents. In the Rancher UI, go to **☰ > Cluster Management**, click on **Clusters** page, then click **Force Update**.
From 2631858face9a66ee1bfbef34451c56c8d28be97 Mon Sep 17 00:00:00 2001 From: Jen Travinski Date: Wed, 24 Nov 2021 14:39:14 -0500 Subject: [PATCH 8/8] Update content/rancher/v2.5/en/deploy-across-clusters/fleet/_index.md Co-authored-by: Billy Tat --- content/rancher/v2.5/en/deploy-across-clusters/fleet/_index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/rancher/v2.5/en/deploy-across-clusters/fleet/_index.md b/content/rancher/v2.5/en/deploy-across-clusters/fleet/_index.md index 61de588ea67..340c2de633b 100644 --- a/content/rancher/v2.5/en/deploy-across-clusters/fleet/_index.md +++ b/content/rancher/v2.5/en/deploy-across-clusters/fleet/_index.md @@ -52,7 +52,7 @@ For details on using Fleet behind a proxy, see [this page.](./proxy) * **Temporary Workaround:**
1. Find the two service account tokens listed in the fleet-controller and the fleet-controller-bootstrap service accounts. These are under the fleet-system namespace of the local cluster.
2. Remove the non-existent token secret. Doing so allows for only one entry to be present for the service account token secret that actually exists.
- 3. Delete the fleet-controller Pod in the fleet-system Namespace to reschedule.
+ 3. Delete the fleet-controller Pod in the fleet-system namespace to reschedule.
4. After the service account token issue is resolved, you can force redeployment of the fleet-agents. In the Rancher UI, go to **☰ > Cluster Management**, click on **Clusters** page, then click **Force Update**.
5. If the fleet-agent bundles remain in a `Modified` state after Step 4, update the field `spec.forceSyncGeneration` for the fleet-agent bundle to force re-creation.