Merge pull request #3690 from jtravee/Backup-Operator-Fleet

Documented workarounds for Rancher backup-operator Fleet issues in 2.5, 2.6
This commit is contained in:
Billy Tat
2021-11-24 11:42:57 -08:00
committed by GitHub
6 changed files with 39 additions and 0 deletions
@@ -63,6 +63,8 @@ The Backup and Restore custom resources can be created in the Rancher UI, or by
The `rancher-backup` operator can be installed from the Rancher UI, or with the Helm CLI. In both cases, the `rancher-backup` Helm chart is installed on the Kubernetes cluster running the Rancher server. It is a cluster-admin only feature and available only for the **local** cluster. (*If you do not see `rancher-backup` in the Rancher UI, you may have selected the wrong cluster.*)
>**NOTE:** There are two known issues in Fleet that occur after performing a restoration using the backup-restore-operator: Fleet agents are inoperable and clientSecretName and helmSecretName are not included in Fleet gitrepos. Refer [here]({{<baseurl>}}rancher/v2.5/en/deploy-across-clusters/fleet/#troubleshooting) for workarounds.
### Installing rancher-backup with the Rancher UI
1. In the Rancher UI's Cluster Manager, choose the cluster named **local**
@@ -28,6 +28,8 @@ Backups are created as .tar.gz files. These files can be pushed to S3 or Minio,
1. Click **Rancher Backups.**
1. Configure the default storage location. For help, refer to the [storage configuration section.](../configuration/storage-config)
>**NOTE:** There are two known issues in Fleet that occur after performing a restoration using the backup-restore-operator: Fleet agents are inoperable and clientSecretName and helmSecretName are not included in Fleet gitrepos. Refer [here]({{<baseurl>}}rancher/v2.5/en/deploy-across-clusters/fleet/#troubleshooting) for workarounds.
### 2. Perform a Backup
To perform a backup, a custom resource of type Backup must be created.
@@ -16,6 +16,7 @@ Fleet is a separate project from Rancher, and can be installed on any Kubernetes
- [Windows Support](#windows-support)
- [GitHub Repository](#github-repository)
- [Using Fleet Behind a Proxy](#using-fleet-behind-a-proxy)
- [Troubleshooting](#troubleshooting)
- [Documentation](#documentation)
# Architecture
@@ -44,6 +45,25 @@ _Available as of v2.5.8_
For details on using Fleet behind a proxy, see [this page.](./proxy)
# Troubleshooting
---
* **Known Issue:** Fleet becomes inoperable after a restore using the [backup-restore-operator]({{<baseurl>}}rancher/v2.5/en/backups/back-up-rancher/#1-install-the-rancher-backup-operator). We will update the community once a permanent solution is in place.
* **Temporary Workaround:** </br>
1. Find the two service account tokens listed in the fleet-controller and the fleet-controller-bootstrap service accounts. These are under the fleet-system namespace of the local cluster. </br>
2. Remove the non-existent token secret. Doing so allows for only one entry to be present for the service account token secret that actually exists. </br>
3. Delete the fleet-controller Pod in the fleet-system namespace to reschedule. </br>
4. After the service account token issue is resolved, you can force redeployment of the fleet-agents. In the Rancher UI, go to **☰ > Cluster Management**, click on **Clusters** page, then click **Force Update**. </br>
5. If the fleet-agent bundles remain in a `Modified` state after Step 4, update the field `spec.forceSyncGeneration` for the fleet-agent bundle to force re-creation.
---
* **Known Issue:** clientSecretName and helmSecretName secrets for Fleet gitrepos are not included in the backup nor restore created by the [backup-restore-operator]({{<baseurl>}}rancher/v2.5/en/backups/back-up-rancher/#1-install-the-rancher-backup-operator). We will update the community once a permanent solution is in place.
* **Temporary Workaround:** </br>
By default, user-defined secrets are not backed up in Fleet. It is necessary to recreate secrets if performing a disaster recovery restore or migration of Rancher into a fresh cluster. To modify resourceSet to include extra resources you want to backup, refer to docs [here](https://github.com/rancher/backup-restore-operator#user-flow).
---
# Documentation
The Fleet documentation is at [https://fleet.rancher.io/.](https://fleet.rancher.io/)
@@ -47,6 +47,8 @@ The Backup and Restore custom resources can be created in the Rancher UI, or by
The `rancher-backup` operator can be installed from the Rancher UI, or with the Helm CLI. In both cases, the `rancher-backup` Helm chart is installed on the Kubernetes cluster running the Rancher server. It is a cluster-admin only feature and available only for the **local** cluster. (*If you do not see `rancher-backup` in the Rancher UI, you may have selected the wrong cluster.*)
>**NOTE:** There is a known issue in Fleet that occurs after performing a restoration using the backup-restore-operator: Secrets used for clientSecretName and helmSecretName are not included in Fleet gitrepos. Refer [here]({{<baseurl>}}rancher/v2.6/en/deploy-across-clusters/fleet/#troubleshooting) for a workaround.
### Installing rancher-backup with the Rancher UI
1. In the upper left corner, click **☰ > Cluster Management**.
@@ -25,6 +25,8 @@ Backups are created as .tar.gz files. These files can be pushed to S3 or Minio,
1. Configure the default storage location. For help, refer to the [storage configuration section.](../configuration/storage-config)
1. Click **Install**.
>**NOTE:** There is a known issue in Fleet that occurs after performing a restoration using the backup-restore-operator: Secrets used for clientSecretName and helmSecretName are not included in Fleet gitrepos. Refer [here]({{<baseurl>}}rancher/v2.6/en/deploy-across-clusters/fleet/#troubleshooting) for a workaround.
### 2. Perform a Backup
To perform a backup, a custom resource of type Backup must be created.
@@ -12,6 +12,7 @@ Fleet is a separate project from Rancher, and can be installed on any Kubernetes
- [Windows Support](#windows-support)
- [GitHub Repository](#github-repository)
- [Using Fleet Behind a Proxy](#using-fleet-behind-a-proxy)
- [Troubleshooting](#troubleshooting)
- [Documentation](#documentation)
# Architecture
@@ -36,6 +37,16 @@ The Fleet Helm charts are available [here.](https://github.com/rancher/fleet/rel
For details on using Fleet behind a proxy, see [this page.](./proxy)
# Troubleshooting
---
* **Known Issue:** clientSecretName and helmSecretName secrets for Fleet gitrepos are not included in the backup nor restore created by the [backup-restore-operator]({{<baseurl>}}rancher/v2.6/en/backups/back-up-rancher/#1-install-the-rancher-backups-operator). We will update the community once a permanent solution is in place.
* **Temporary Workaround:** </br>
By default, user-defined secrets are not backed up in Fleet. It is necessary to recreate secrets if performing a disaster recovery restore or migration of Rancher into a fresh cluster. To modify resourceSet to include extra resources you want to backup, refer to docs [here](https://github.com/rancher/backup-restore-operator#user-flow).
---
# Documentation
The Fleet documentation is at [https://fleet.rancher.io/.](https://fleet.rancher.io/)