public/rancher-docs
1
0
Fork 0
mirror of https://github.com/rancher/rancher-docs.git synced 2026-09-25 04:28:15 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'master' into audit-17

Browse Source
This commit is contained in:
Robert Parker
2019-09-08 13:09:37 -07:00
committed by GitHub
parent df9ac5d8d1 cbd327a952
commit 335e46d515
30 changed files with 1142 additions and 495 deletions
Download Patch File Download Diff File Expand all files Collapse all files
content/os/v1.x/en/installation/configuration/switching-docker-versions/_index.md
+1 -1
View File
@@ -83,7 +83,7 @@ FROM scratch
COPY engine /engine
```
Once the image is built a [system service]({{< baseurl >}}/os/v1.x/en/installation/system-services/adding-system-services/) configuration file must be created. An [example file](https://github.com/rancher/os-services/blob/master/d/docker-1.12.6.yml) can be found in the rancher/os-services repo. Change the `image` field to point to the Docker engine image you've built.
Once the image is built a [system service]({{< baseurl >}}/os/v1.x/en/installation/system-services/adding-system-services/) configuration file must be created. An [example file](https://github.com/rancher/os-services/blob/master/d/docker-18.06.3-ce.yml) can be found in the rancher/os-services repo. Change the `image` field to point to the Docker engine image you've built.
All of the previously mentioned methods of switching Docker engines are now available. For example, if your service file is located at `https://myservicefile` then the following cloud-config file could be used to use your custom Docker engine.
content/os/v1.x/en/installation/running-rancheros/_index.md
+6
View File
@@ -24,6 +24,12 @@ RancherOS runs on virtualization platforms, cloud providers and bare metal serve
[Azure]({{< baseurl >}}/os/v1.x/en/installation/running-rancheros/cloud/azure)
[OpenStack]({{< baseurl >}}/os/v1.x/en/installation/running-rancheros/cloud/openstack)
[VMware ESXi]({{< baseurl >}}/os/v1.x/en/installation/running-rancheros/cloud/vmware-esxi)
[Aliyun]({{< baseurl >}}/os/v1.x/en/installation/running-rancheros/cloud/aliyun)
#### Bare Metal & Virtual Servers
[PXE]({{< baseurl >}}/os/v1.x/en/installation/running-rancheros/server/pxe)
content/os/v1.x/en/installation/running-rancheros/cloud/aliyun/_index.md
+34
View File
@@ -0,0 +1,34 @@
---
title: Aliyun
weight: 111
---
# Adding the RancherOS Image into Aliyun
RancherOS is available as an image in Aliyun, and can be easily run in Elastic Compute Service (ECS). Let’s walk through how to upload the ECS image.
1. Download the most recent RancherOS image. The image `rancheros-aliyun.vhd` can be found in the [release artifacts](https://github.com/rancher/os/releases).
2. Follow Aliyun's instructions on how to [upload the image](https://help.aliyun.com/document_detail/127285.html). Before the image can be added, it must be uploaded into an OSS bucket.
3. Once the image is added to your ECS, we can start creating new instances!
Example:
![RancherOS on Aliyun 1]({{< baseurl >}}/img/os/RancherOS_aliyun1.jpg)
## Options
| Option | Description |
| --- | --- |
| Root disk size | The size must be greater than 10GB. Note: When booting the instance, the value must be kept the same. |
| Platform | Select `Others Linux` |
| Image Format | Select `VHD` |
### Launching RancherOS using Aliyun Console
After the image is uploaded, we can use the `Aliyun Console` to start a new instance. Currently, RancherOS on Aliyun only supports SSH key access, so it can only be deployed through the UI.
Since the image is private, we need to use the `Custom Images`.
![RancherOS on Aliyun 2]({{< baseurl >}}/img/os/RancherOS_aliyun2.jpg)
After the instance is successfully started, we can login with the `rancher` user via SSH.
content/os/v1.x/en/installation/running-rancheros/server/install-to-disk/_index.md
+16
View File
@@ -82,6 +82,22 @@ rancher/os:v0.5.0 remote
Alternatively, you can set the installer image to any image in System Docker to install RancherOS. This is particularly useful for machines that will not have direct access to the internet.
#### Caching Images
_Available as of v1.5.3_
Some configurations included in `cloud-config` require images to be downloaded from Docker to start. After installation, these images are downloaded automatically by RancherOS when booting. An example of these configurations are:
- rancher.services_include
- rancher.console
- rancher.docker
If you want to download and save these images to disk during installation, they will be cached and not need to be downloaded again upon each boot. You can cache these images by adding `-s` when using `ros install`:
```
$ ros install -d <disk> -c <cloud-config.yaml> -s
```
### SSH into RancherOS
After installing RancherOS, you can ssh into RancherOS using your private key and the **rancher** user.
content/rancher/v2.x/en/admin-settings/authentication/_index.md
+24
View File
@@ -40,6 +40,30 @@ Rancher relies on users and groups to determine who is allowed to log in to Ranc
For more information, see [Users and Groups]({{< baseurl >}}/rancher/v2.x/en/admin-settings/authentication/user-groups/)
## Scope of Rancher Authorization
After you configure Rancher to allow sign on using an external authentication service, you should configure who should be allowed to log in and use Rancher. The following options are available:
| Access Level | Description |
|----------------------------------------------|-------------|
| Allow any valid Users | _Any_ user in the authorization service can access Rancher. We generally discourage use of this setting! |
| Allow members of Clusters, Projects, plus Authorized Users and Organizations | Any user in the authorization service and any group added as a **Cluster Member** or **Project Member** can log in to Rancher. Additionally, any user in the authentication service or group you add to the **Authorized Users and Organizations** list may log in to Rancher. |
| Restrict access to only Authorized Users and Organizations | Only users in the authentication service or groups added to the Authorized Users and Organizations can log in to Rancher. |
To set the Rancher access level for users in the authorization service, follow these steps:
1. From the **Global** view, click **Security > Authentication.**
1. Use the **Site Access** options to configure the scope of user authorization. The table above explains the access level for each option.
1. Optional: If you choose an option other than **Allow any valid Users,** you can add users to the list of authorized users and organizations by searching for them in the text field that appears.
1. Click **Save.**
**Result:** The Rancher access configuration settings are applied.
{{< saml_caveats >}}
## External Authentication Configuration and Principal Users
Configuration of external authentication requires:
content/rancher/v2.x/en/admin-settings/authentication/local/_index.md
+1 -1
View File
@@ -5,7 +5,7 @@ aliases:
- /rancher/v2.x/en/tasks/global-configuration/authentication/local-authentication/
---
Local authentication is the default until you configure an external authentication provider. Local authentication is where Rancher stores the user information, i.e. names and passwords, of who can log in to Ranchehr. By default, the `admin` user that logs in to Rancher for the first time is a local user.
Local authentication is the default until you configure an external authentication provider. Local authentication is where Rancher stores the user information, i.e. names and passwords, of who can log in to Rancher. By default, the `admin` user that logs in to Rancher for the first time is a local user.
## Adding Local Users
content/rancher/v2.x/en/backups/restorations/ha-restoration/_index.md
+1 -1
View File
@@ -229,6 +229,6 @@ rke up --config ./rancher-cluster-restore.yml
#### Finishing Up
Rancher should now be running and available to manage your Kubernetes clusters. Swap your Rancher DNS or Load Balancer endpoints to target the new cluster. Once this is done the agents on your managed clusters should automatically reconnect. This may take 10-15 minutes due to reconnect back off timeouts.
Rancher should now be running and available to manage your Kubernetes clusters. Review the [recommended architecture]({{< baseurl >}}/rancher/v2.x/en/installation/ha/#recommended-architecture) for HA installations and update the endpoints for Rancher DNS or the Load Balancer that you built during Step 1 of the HA install ([1. Create Nodes and Load Balancer]({{< baseurl >}}/rancher/v2.x/en/installation/ha/create-nodes-lb/#load-balancer)) to target the new cluster. Once the endpoints are updated, the agents on your managed clusters should automatically reconnect. This may take 10-15 minutes due to reconnect back off timeouts.
> **IMPORTANT:** Remember to save your new RKE config (`rancher-cluster-restore.yml`) and `kubectl` credentials (`kube_config_rancher-cluster-restore.yml`) files in a safe place for future maintenance.
content/rancher/v2.x/en/cluster-admin/cloning-clusters/_index.md
+27 -83
View File
@@ -7,22 +7,18 @@ aliases:
If you have a cluster in Rancher that you want to use as a template for creating similar clusters, you can use Rancher CLI to clone the cluster's configuration, edit it, and then use it to quickly launch the cloned cluster.
## Caveats
You can clone clusters only if the nodes in the cluster are hosted by an infrastructure provider, such as EC2, Azure, or DigitalOcean.
- Only [cluster types]({{< baseurl >}}/rancher/v2.x/en/cluster-provisioning/#cluster-creation-options) that interact with cloud hosts over API can be cloned. Duplication of imported clusters and custom clusters provisioned using Docker machine is not supported.
Duplication of imported clusters, clusters in hosted Kubernetes providers, and custom clusters provisioned using Docker machine is not supported.
| Cluster Type | Cloneable? |
| -------------------------------- | ------------- |
| [Hosted Kubernetes Providers][1] | ✓ |
| [Nodes Hosted by Infrastructure Provider][2] | ✓ |
| [Custom Cluster][3] | |
| [Imported Cluster][4] | |
- During the process of duplicating a cluster, you will edit a config file full of cluster settings. However, we recommend editing only values explicitly listed in this document, as cluster duplication is designed for simple cluster copying, _not_ wide scale configuration changes. Editing other values may invalidate the config file, which will lead to cluster deployment failure.
| Cluster Type | Cloneable? |
|----------------------------------|---------------|
| [Nodes Hosted by Infrastructure Provider]({{< baseurl >}}/rancher/v2.x/en/cluster-provisioning/rke-clusters/node-pools/) | ✓ |
| [Hosted Kubernetes Providers]({{< baseurl >}}/rancher/v2.x/en/cluster-provisioning/hosted-kubernetes-clusters/) | |
| [Custom Cluster]({{< baseurl >}}/rancher/v2.x/en/cluster-provisioning/custom-clusters/) | |
| [Imported Cluster]({{< baseurl >}}/rancher/v2.x/en/cluster-provisioning/imported-clusters/) | |
[1]: {{< baseurl >}}/rancher/v2.x/en/cluster-provisioning/hosted-kubernetes-clusters/
[2]: {{< baseurl >}}/rancher/v2.x/en/cluster-provisioning/rke-clusters/node-pools/
[3]: {{< baseurl >}}/rancher/v2.x/en/cluster-provisioning/custom-clusters/
[4]: {{< baseurl >}}/rancher/v2.x/en/cluster-provisioning/imported-clusters/
> **Warning:** During the process of duplicating a cluster, you will edit a config file full of cluster settings. However, we recommend editing only values explicitly listed in this document, as cluster duplication is designed for simple cluster copying, _not_ wide scale configuration changes. Editing other values may invalidate the config file, which will lead to cluster deployment failure.
## Prerequisites
@@ -62,77 +58,25 @@ Use your favorite text editor to modify the cluster configuration in `cluster-te
>**Warning:** Only edit the cluster config values explicitly called out below. Many of the values listed in this file are used to provision your cloned cluster, and editing their values may break the provisioning process.
1. As depicted in one of the examples below, at the `<CLUSTER_NAME>` placeholder, replace your original cluster's name with a unique name (`<CLUSTER_NAME>`). If your cloned cluster has a duplicate name, the cluster will not provision successfully.
{{% accordion id="gke" label="GKE" %}}
```yml
Version: v3
clusters:
<CLUSTER_NAME>: # ENTER UNIQUE NAME
dockerRootDir: /var/lib/docker
enableNetworkPolicy: false
googleKubernetesEngineConfig:
credential: |-
{
"type": "service_account",
"project_id": "gke-cluster-221300",
"private_key_id": "1d210afae352bc298bde1b3e680ec0c8b22cdd61"
```
{{% /accordion %}}
{{% accordion id="eks" label="EKS" %}}
```yml
Version: v3
clusters:
<CLUSTER_NAME>: # ENTER UNIQUE NAME
amazonElasticContainerServiceConfig:
accessKey: 00000000000000000000
associateWorkerNodePublicIp: true
instanceType: t2.medium
maximumNodes: 3
minimumNodes: 1
region: us-west-2
secretKey: 0000000000000000000000000000000000000000
dockerRootDir: /var/lib/docker
enableNetworkPolicy: false
```
{{% /accordion %}}
{{% accordion id="aks" label="AKS" %}}
```yml
Version: v3
clusters:
<CLUSTER_NAME>: # ENTER UNIQUE NAME
azureKubernetesServiceConfig:
adminUsername: azureuser
agentPoolName: rancher
agentVmSize: Standard_D5_v2
clientId: 00000000-0000-0000-0000-000000000000
clientSecret: 00000000000000000000000000000000000000000000
count: 3
kubernetesVersion: 1.11.2
location: westus
osDiskSizeGb: 100
resourceGroup: docker-machine
sshPublicKeyContents: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJc2kDExgRaDLD
```
{{% /accordion %}}
{{% accordion id="ec2" label="Nodes Hosted by Infrastructure Provider (EC2, Azure, or DigitalOcean )" %}}
```yml
Version: v3
clusters:
<CLUSTER_NAME>: # ENTER UNIQUE NAME
dockerRootDir: /var/lib/docker
enableNetworkPolicy: false
rancherKubernetesEngineConfig:
addonJobTimeout: 30
authentication:
strategy: x509
authorization: {}
bastionHost: {}
cloudProvider: {}
ignoreDockerVersion: true
```
{{% /accordion %}}
1. As depicted in the example below, at the `<CLUSTER_NAME>` placeholder, replace your original cluster's name with a unique name (`<CLUSTER_NAME>`). If your cloned cluster has a duplicate name, the cluster will not provision successfully.
1. **Nodes Hosted by Infrastructure Provider Only:** For each `nodePools` section, replace the original nodepool name with a unique name at the `<NODEPOOL_NAME>` placeholder. If your cloned cluster has a duplicate nodepool name, the cluster will not provision successfully.
```yml
Version: v3
clusters:
<CLUSTER_NAME>: # ENTER UNIQUE NAME
dockerRootDir: /var/lib/docker
enableNetworkPolicy: false
rancherKubernetesEngineConfig:
addonJobTimeout: 30
authentication:
strategy: x509
authorization: {}
bastionHost: {}
cloudProvider: {}
ignoreDockerVersion: true
```
1. For each `nodePools` section, replace the original nodepool name with a unique name at the `<NODEPOOL_NAME>` placeholder. If your cloned cluster has a duplicate nodepool name, the cluster will not provision successfully.
```yml
nodePools:
content/rancher/v2.x/en/cluster-admin/tools/alerts/_index.md
+1 -1
View File
@@ -154,7 +154,7 @@ This alert type monitors for the overload from Prometheus expression querying, i
- [**Node**](https://github.com/prometheus/node_exporter)
- [**Container**](https://github.com/google/cadvisor)
- [**ETCD**](https://github.com/etcd-io/etcd/blob/master/Documentation/op-guide/monitoring.md)
- [**ETCD**](https://etcd.io/docs/v3.3.12/op-guide/monitoring/)
- [**Kubernetes Components**](https://github.com/kubernetes/metrics)
- [**Kubernetes Resources**](https://github.com/kubernetes/kube-state-metrics)
- [**Fluentd**](https://docs.fluentd.org/v1.0/articles/monitoring-prometheus) (supported by [Logging]({{< baseurl >}}/rancher/v2.x/en/tools/logging))
content/rancher/v2.x/en/installation/air-gap-high-availability/install-rancher/_index.md
+50 -9
View File
@@ -52,23 +52,48 @@ Based on the choice your made in [B. Choose your SSL Configuration](#b-optional-
In this section you will configure your cert manager and private registry in the Rancher template.
{{% accordion id="self-signed" label="Option A: Default Self-Signed Certificate" %}}
By default, Rancher generates a CA and uses cert manager to issue the certificate for access to the Rancher server interface.
By default, Rancher generates a CA and uses cert-manager to issue the certificate for access to the Rancher server interface.
1. From a system connected to the internet, fetch the latest cert-manager chart available from the [official Helm chart repository](https://github.com/helm/charts/tree/master/stable).
> **Note:**
> Recent changes to cert-manager require an upgrade. If you are upgrading Rancher and using a version of cert-manager older than v0.9.1, please see our [upgrade documentation]({{< baseurl >}}/rancher/v2.x/en/installation/options/upgrading-cert-manager/).
1. From a system connected to the internet, add the cert-manager repo to Helm.
```plain
helm fetch stable/cert-manager --version 0.5.2
helm repo add jetstack https://charts.jetstack.io
helm repo update
```
1. Fetch the latest cert-manager chart available from the [Helm chart repository](https://hub.helm.sh/charts/jetstack/cert-manager).
```plain
helm fetch jetstack/cert-manager --version v0.9.1
```
1. Render the cert manager template with the options you would like to use to install the chart. Remember to set the `image.repository` option to pull the image from your private registry. This will create a `cert-manager` directory with the Kubernetes manifest files.
```plain
helm template ./cert-manager-v0.5.2.tgz --output-dir . \
--name cert-manager --namespace kube-system \
helm template ./cert-manager-v0.9.1.tgz --output-dir . \
--name cert-manager --namespace cert-manager \
--set image.repository=<REGISTRY.YOURDOMAIN.COM:PORT>/quay.io/jetstack/cert-manager-controller
--set webhook.image.repository=<REGISTRY.YOURDOMAIN.COM:PORT>/quay.io/jetstack/cert-manager-webhook
--set cainjector.image.repository=<REGISTRY.YOURDOMAIN.COM:PORT>/quay.io/jetstack/cert-manager-cainjector
```
1. Download the required CRD file for cert-manager
```plain
curl -L -o cert-manager/cert-manager-crd.yaml https://raw.githubusercontent.com/jetstack/cert-manager/release-0.9/deploy/manifests/00-crds.yaml
```
1. Render the Rancher template, declaring your chosen options. Use the reference table below to replace each placeholder. Rancher needs to be configured to use the private registry in order to provision any Rancher launched Kubernetes clusters or Rancher tools. To configure Rancher to use your private registry when starting the `rancher/rancher` container, use the `CATTLE_SYSTEM_DEFAULT_REGISTRY` variable. You can set the the extra environment variable `extraEnv` to use the same `name` and `value` keys as the container manifest definitions. Remember to quote the values:
Placeholder | Description
------------|-------------
`<VERSION>` | The version number of the output tarball.
`<RANCHER.YOURDOMAIN.COM>` | The DNS name you pointed at your load balancer.
`<REGISTRY.YOURDOMAIN.COM:PORT>` | The DNS name for your private registry.).
```plain
helm template ./rancher-<VERSION>.tgz --output-dir . \
@@ -126,9 +151,25 @@ Use `kubectl` to create namespaces and apply the rendered manifests.
If you are using self-signed certificates, install cert-manager:
```plain
kubectl -n kube-system apply -R -f ./cert-manager
```
1. Create the namespace for cert-manager.
```plain
kubectl create namespace cert-manager
```
1. Label the cert-manager namespace to disable resource validation.
```plain
kubectl label namespace cert-manager certmanager.k8s.io/disable-validation=true
```
1. Create the cert-manager CustomResourceDefinitions (CRDs).
```plain
kubectl apply -f cert-manager/cert-manager-crd.yaml
```
1. Launch cert-manager.
```plain
kubectl apply -R -f ./cert-manager
```
Install rancher:
@@ -145,4 +186,4 @@ These resources could be helpful when you install Rancher:
- [Adding TLS secrets]({{<baseurl>}}/rancher/v2.x/en/installation/ha/helm-rancher/tls-secrets/)
- [Troubleshooting Rancher HA installations]({{<baseurl>}}/rancher/v2.x/en/installation/ha/helm-rancher/troubleshooting/)
### [Next: Configure Rancher System Charts]({{< baseurl >}}/rancher/v2.x/en/installation/air-gap-high-availability/config-rancher-system-charts/)
### [Next: Configure Rancher System Charts]({{< baseurl >}}/rancher/v2.x/en/installation/air-gap-high-availability/config-rancher-system-charts/)
content/rancher/v2.x/en/installation/air-gap-high-availability/prepare-private-registry/_index.md
+5 -1
View File
@@ -49,8 +49,12 @@ Start by collecting all the images needed to install Rancher in an air gap envir
1. Fetch the latest `cert-manager` Helm chart and parse the template for image details.
> **Note:** Recent changes to cert-manager require an upgrade. If you are upgrading Rancher and using a version of cert-manager older than v0.9.1, please see our [upgrade documentation]({{< baseurl >}}/rancher/v2.x/en/installation/options/upgrading-cert-manager/).
```plain
helm fetch stable/cert-manager --version 0.5.2
helm repo add jetstack https://charts.jetstack.io
helm repo update
helm fetch jetstack/cert-manager --version v0.9.1
helm template ./cert-manager-<version>.tgz | grep -oP '(?<=image: ").*(?=")' >> ./rancher-images.txt
```
content/rancher/v2.x/en/installation/ha/create-nodes-lb/nlb/_index.md
+1
View File
@@ -18,6 +18,7 @@ Configuring an Amazon NLB is a multistage process. We've broken it down into mul
Use Amazon's Wizard to create an Network Load Balancer. As part of this process, you'll add the target groups you created in **1. Create Target Groups**.
> **Note:** Rancher only supports using the Amazon NLB when terminating traffic in `tcp` mode for port 443 rather than `tls` mode. This is due to the fact that the NLB does not inject the correct headers into requests when terminated at the NLB. This means that if you want to use certificates managed by the Amazon Certificate Manager (ACM), you should use an ELB or ALB.
## Create Target Groups
content/rancher/v2.x/en/installation/ha/helm-rancher/_index.md
+51 -15
View File
@@ -33,29 +33,65 @@ There are three recommended options for the source of the certificate.
### Optional: Install cert-manager
> **Note:** cert-manager is only required for certificates issued by Rancher's generated CA (`ingress.tls.source=rancher`) and Let's Encrypt issued certificates (`ingress.tls.source=letsEncrypt`). You should skip this step if you are using your own certificate files (option `ingress.tls.source=secret`) or if you use [TLS termination on an External Load Balancer]({{< baseurl >}}/rancher/v2.x/en/installation/ha/helm-rancher/chart-options/#external-tls-termination).
**Note:** cert-manager is only required for certificates issued by Rancher's generated CA (`ingress.tls.source=rancher`) and Let's Encrypt issued certificates (`ingress.tls.source=letsEncrypt`). You should skip this step if you are using your own certificate files (option `ingress.tls.source=secret`) or if you use [TLS termination on an External Load Balancer]({{< baseurl >}}/rancher/v2.x/en/installation/ha/helm-rancher/chart-options/#external-tls-termination).
> **Important:** Due to an issue with Helm v2.12.0 and cert-manager, please use Helm v2.12.1 or higher.
> **Important:**
Rancher relies on [cert-manager](https://github.com/kubernetes/charts/tree/master/stable/cert-manager) version v0.5.2 from the official Kubernetes Helm chart repository to issue certificates from Rancher's own generated CA or to request Let's Encrypt certificates.
> Due to an issue with Helm v2.12.0 and cert-manager, please use Helm v2.12.1 or higher.
> Recent changes to cert-manager require an upgrade. If you are upgrading Rancher and using a version of cert-manager older than v0.9.1, please see our [upgrade documentation]({{< baseurl >}}/rancher/v2.x/en/installation/options/upgrading-cert-manager/).
Rancher relies on [cert-manager](https://github.com/jetstack/cert-manager) to issue certificates from Rancher's own generated CA or to request Let's Encrypt certificates.
These instructions are adapted from the [official cert-manager documentation](https://docs.cert-manager.io/en/latest/getting-started/install/kubernetes.html#installing-with-helm).
Install `cert-manager` from Kubernetes Helm chart repository.
1. Install the CustomResourceDefinition resources separately
```plain
kubectl apply -f https://raw.githubusercontent.com/jetstack/cert-manager/release-0.9/deploy/manifests/00-crds.yaml
```
1. Create the namespace for cert-manager
```plain
kubectl create namespace cert-manager
```
1. Label the cert-manager namespace to disable resource validation
```plain
kubectl label namespace cert-manager certmanager.k8s.io/disable-validation=true
```
1. Add the Jetstack Helm repository
```plain
helm repo add jetstack https://charts.jetstack.io
```
1. Update your local Helm chart repository cache
```plain
helm repo update
```
1. Install the cert-manager Helm chart
```plain
helm install \
--name cert-manager \
--namespace cert-manager \
--version v0.9.1 \
jetstack/cert-manager
```
Once you’ve installed cert-manager, you can verify it is deployed correctly by checking the cert-manager namespace for running pods:
```
helm install stable/cert-manager \
--name cert-manager \
--namespace kube-system \
--version v0.5.2
kubectl get pods --namespace cert-manager
NAME READY STATUS RESTARTS AGE
cert-manager-7cbdc48784-rpgnt 1/1 Running 0 3m
cert-manager-webhook-5b5dd6999-kst4x 1/1 Running 0 3m
cert-manager-cainjector-3ba5cd2bcd-de332x 1/1 Running 0 3m
```
Wait for `cert-manager` to be rolled out:
```
kubectl -n kube-system rollout status deploy/cert-manager
Waiting for deployment "cert-manager" rollout to finish: 0 of 1 updated replicas are available...
deployment "cert-manager" successfully rolled out
```
If the ‘webhook’ pod (2nd line) is in a ContainerCreating state, it may still be waiting for the Secret to be mounted into the pod. Wait a couple of minutes for this to happen but if you experience problems, please check the [troubleshooting](https://docs.cert-manager.io/en/latest/getting-started/troubleshooting.html) guide.
<br/>
content/rancher/v2.x/en/installation/ha/rke-add-on/layer-7-lb/nginx/_index.md
+1 -52
View File
@@ -19,58 +19,7 @@ For help installing NGINX, refer to their [install documentation](https://www.ng
## Create NGINX Configuration
After installing NGINX, you need to create the NGINX config file, `/etc/nginx/conf.d/rancher.conf`, with the IP addresses for your Linux nodes, chosen FQDN and location of the certificate file and certificate key file.
>**Note:** The example configuration below does not include all available Nginx options and may not be suitable for your production environment. For full configuration documentation, see [NGINX Load Balancing - HTTP Load Balancer](https://docs.nginx.com/nginx/admin-guide/load-balancer/http-load-balancer/).
1. Copy and paste the code sample below into your favorite text editor. Save it as `/etc/nginx/conf.d/rancher.conf`.
**Example NGINX config:**
```
upstream rancher {
server IP_NODE_1:80;
server IP_NODE_2:80;
server IP_NODE_3:80;
}
map $http_upgrade $connection_upgrade {
default Upgrade;
'' close;
}
server {
listen 443 ssl http2;
server_name FQDN;
ssl_certificate /certs/fullchain.pem;
ssl_certificate_key /certs/privkey.pem;
location / {
proxy_set_header Host $host;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_pass http://rancher;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
# This allows the ability for the execute shell window to remain open for up to 15 minutes. Without this parameter, the default is 1 minute and will automatically close.
proxy_read_timeout 900s;
}
}
server {
listen 80;
server_name FQDN;
return 301 https://$server_name$request_uri;
}
```
2. In `/etc/nginx/conf.d/rancher.conf`, replace `IP_NODE_1`, `IP_NODE_2`, and `IP_NODE_3` with the IPs of your Linux hosts.
3. In `/etc/nginx/conf.d/rancher.conf`, replace `FQDN` with the FQDN you chose for your Rancher installation.
4. In `/etc/nginx/conf.d/rancher.conf`, replace `/certs/fullchain.pem` with the path to your certificate. If there are intermediates required for you certificate, they should be included in this file.
5. In `/etc/nginx/conf.d/rancher.conf`, replace `/certs/privkey.pem` with the path to your certificate key.
See [Example NGINX config]({{< baseurl >}}/rancher/v2.x/en/installation/ha/helm-rancher/chart-options/#example-nginx-config).
## Run NGINX
content/rancher/v2.x/en/installation/options/upgrading-cert-manager/_index.md
+151
View File
@@ -0,0 +1,151 @@
---
title: Upgrading Cert-Manager
weight: 2040
---
Rancher uses cert-manager to automatically generate and renew TLS certificates for HA deployments of Rancher. As of Fall 2019, two important changes to cert-manager are set to occur that you need to take action on if you have an HA deployment of Rancher:
1. [Let's Encrypt will be blocking cert-manager instances older than 0.8.0 starting November 1st 2019.](https://community.letsencrypt.org/t/blocking-old-cert-manager-versions/98753)
1. [Cert-manager is deprecating and replacing the certificate.spec.acme.solvers field](https://docs.cert-manager.io/en/latest/tasks/upgrading/upgrading-0.7-0.8.html#upgrading-from-v0-7-to-v0-8). This change has no exact deadline.
To address these changes, this guide will do two things:
1. Document the procedure for upgrading cert-manager
1. Explain the cert-manager API changes and link to cert-manager's offficial documentation for migrating your data
## Performing the upgrade
>**Note:** The namespace used in these instructions depends on the namespace cert-manager is currently installed in. If it is in kube-system use that in the instructions below. You can verify by running `kubectl get pods --all-namespaces` and checking which namespace the cert-manager-\* pods are listed in. Do not change the namespace cert-manager is running in or this can cause issues.
In order to upgrade cert-manager, follow these instructions:
{{% accordion id="normal" label="Upgrading cert-manager with Internet access" %}}
1. Back up existing resources as a precaution
```plain
kubectl get -o yaml --all-namespaces issuer,clusterissuer,certificates > cert-manager-backup.yaml
```
1. Delete the existing deployment
```plain
helm delete --purge cert-manager
```
1. Install the CustomResourceDefinition resources separately
```plain
kubectl apply -f https://raw.githubusercontent.com/jetstack/cert-manager/release-0.9/deploy/manifests/00-crds.yaml
```
1. Label the kube-system namespace to disable resource validation
```plain
kubectl label namespace kube-system certmanager.k8s.io/disable-validation=true
```
1. Add the Jetstack Helm repository
```plain
helm repo add jetstack https://charts.jetstack.io
```
1. Update your local Helm chart repository cache
```plain
helm repo update
```
1. Install the new version of cert-manager
```plain
helm install --version 0.9.1 --name cert-manager --namespace kube-system jetstack/cert-manager
```
{{% /accordion %}}
{{% accordion id="airgap" label="Upgrading cert-manager in an airgapped environment" %}}
### Prerequisites
Before you can perform the upgrade, you must prepare your air gapped environment by adding the necessary container images to your private registry and downloading or rendering the required Kubernetes manifest files.
1. Follow the guide to [Prepare your Private Registry]({{< baseurl >}}/rancher/v2.x/en/installation/air-gap-installation/prepare-private-reg/) with the images needed for the upgrade.
1. From a system connected to the internet, add the cert-manager repo to Helm
```plain
helm repo add jetstack https://charts.jetstack.io
helm repo update
```
1. Fetch the latest cert-manager chart available from the [Helm chart repository](https://hub.helm.sh/charts/jetstack/cert-manager).
```plain
helm fetch jetstack/cert-manager --version v0.9.1
```
1. Render the cert manager template with the options you would like to use to install the chart. Remember to set the `image.repository` option to pull the image from your private registry. This will create a `cert-manager` directory with the Kubernetes manifest files.
```plain
helm template ./cert-manager-v0.9.1.tgz --output-dir . \
--name cert-manager --namespace kube-system \
--set image.repository=<REGISTRY.YOURDOMAIN.COM:PORT>/quay.io/jetstack/cert-manager-controller
--set webhook.image.repository=<REGISTRY.YOURDOMAIN.COM:PORT>/quay.io/jetstack/cert-manager-webhook
--set cainjector.image.repository=<REGISTRY.YOURDOMAIN.COM:PORT>/quay.io/jetstack/cert-manager-cainjector
```
1. Download the required CRD file for cert-manager
```plain
curl -L -o cert-manager/cert-manager-crd.yaml https://raw.githubusercontent.com/jetstack/cert-manager/release-0.9/deploy/manifests/00-crds.yaml
```
### Install cert-manager
1. Back up existing resources as a precaution
```plain
kubectl get -o yaml --all-namespaces issuer,clusterissuer,certificates > cert-manager-backup.yaml
```
1. Delete the existing deployment
```plain
helm delete --purge cert-manager
```
1. Install the CustomResourceDefinition resources separately
```plain
kubectl apply -f cert-manager/cert-manager-crd.yaml
```
1. Label the kube-system namespace to disable resource validation
```plain
kubectl label namespace kube-system certmanager.k8s.io/disable-validation=true
```
1. Install cert-manager
```plain
kubectl -n kube-system apply -R -f ./cert-manager
```
{{% /accordion %}}
Once you’ve installed cert-manager, you can verify it is deployed correctly by checking the kube-system namespace for running pods:
```
kubectl get pods --namespace kube-system
NAME READY STATUS RESTARTS AGE
cert-manager-7cbdc48784-rpgnt 1/1 Running 0 3m
cert-manager-webhook-5b5dd6999-kst4x 1/1 Running 0 3m
cert-manager-cainjector-3ba5cd2bcd-de332x 1/1 Running 0 3m
```
If the ‘webhook’ pod (2nd line) is in a ContainerCreating state, it may still be waiting for the Secret to be mounted into the pod. Wait a couple of minutes for this to happen but if you experience problems, please check cert-manager's [troubleshooting](https://docs.cert-manager.io/en/latest/getting-started/troubleshooting.html) guide.
> **Note:** The above instructions ask you to add the disable-validation label to the kube-system namespace. Here are additional resources that explain why this is necessary:
>
> - [Information on the disable-validation label](https://docs.cert-manager.io/en/latest/tasks/upgrading/upgrading-0.4-0.5.html?highlight=certmanager.k8s.io%2Fdisable-validation#disabling-resource-validation-on-the-cert-manager-namespace)
> - [Information on webhook validation for certificates](https://docs.cert-manager.io/en/latest/getting-started/webhook.html)
## Cert-Manager API change and data migration
Cert-manager has deprecated the use of the `certificate.spec.acme.solvers` field and will drop support for it completely in an upcoming release.
Per the cert-manager documentation, a new format for configuring ACME certificate resources was introduced in v0.8. Specifically, the challenge solver configuration field was moved. Both the old format and new are supported as of v0.9, but support for the old format will be dropped in an upcoming release of cert-manager. The cert-manager documentation strongly recommends that after upgrading you update your ACME Issuer and Certificate resources to the new format.
Details about the change and migration instructions can be found in the [cert-manager v0.7 to v0.8 upgrade instructions](https://docs.cert-manager.io/en/latest/tasks/upgrading/upgrading-0.7-0.8.html).
content/rancher/v2.x/en/installation/references/_index.md
+2 -2
View File
@@ -112,6 +112,6 @@ When using the [AWS EC2 node driver]({{< baseurl >}}/rancher/v2.x/en/cluster-pro
| Custom UDP Rule | UDP | 8472 | sg-xxx (rancher-nodes) | Inbound |
| Custom TCP Rule | TCP | 10250-10252 | sg-xxx (rancher-nodes) | Inbound |
| Custom TCP Rule | TCP | 10256 | sg-xxx (rancher-nodes) | Inbound |
| Custom TCP Rule | TCP | 30000-32767 | 30000-32767 | Inbound |
| Custom UDP Rule | UDP | 30000-32767 | 30000-32767 | Inbound |
| Custom TCP Rule | TCP | 30000-32767 | 0.0.0.0/0 | Inbound |
| Custom UDP Rule | UDP | 30000-32767 | 0.0.0.0/0 | Inbound |
| All traffic | All | All | 0.0.0.0/0 | Outbound |
content/rancher/v2.x/en/troubleshooting/dns/_index.md
+46 -4
View File
@@ -7,7 +7,7 @@ The commands/steps listed on this page can be used to check name resolution issu
Make sure you configured the correct kubeconfig (for example, `export KUBECONFIG=$PWD/kube_config_rancher-cluster.yml` for Rancher HA) or are using the embedded kubectl via the UI.
Before running the DNS checks, make sure that [the overlay network is functioning correctly]({{< baseurl >}}/rancher/v2.x/en/troubleshooting/networking/#check-if-overlay-network-is-functioning-correctly) as this can also be the reason why DNS resolution (partly) fails.
Before running the DNS checks, check the [default DNS provider]({{< baseurl >}}/rancher/v2.x/en/cluster-provisioning/rke-clusters/options/#default-dns-provider) for your cluster and make sure that [the overlay network is functioning correctly]({{< baseurl >}}/rancher/v2.x/en/troubleshooting/networking/#check-if-overlay-network-is-functioning-correctly) as this can also be the reason why DNS resolution (partly) fails.
### Check if DNS pods are running
@@ -15,7 +15,13 @@ Before running the DNS checks, make sure that [the overlay network is functionin
kubectl -n kube-system get pods -l k8s-app=kube-dns
```
Example output:
Example output when using CoreDNS:
```
NAME READY STATUS RESTARTS AGE
coredns-799dffd9c4-6jhlz 1/1 Running 0 76m
```
Example output when using kube-dns:
```
NAME READY STATUS RESTARTS AGE
kube-dns-5fd74c7488-h6f7n 3/3 Running 0 4m13s
@@ -123,9 +129,45 @@ command terminated with exit code 1
Cleanup the alpine DaemonSet by running `kubectl delete ds/dnstest`.
### Check upstream nameservers in kubedns container
### CoreDNS specific
By default, the configured nameservers on the host (in `/etc/resolv.conf`) will be used as upstream nameservers for `kube-dns`. Sometimes the host will run a local caching DNS nameserver, which means the address in `/etc/resolv.conf` will point to an address in the loopback range (`127.0.0.0/8`) which will be unreachable by the container. In case of Ubuntu 18.04, this is done by `systemd-resolved`. Since Rancher v2.0.7, we detect if `systemd-resolved` is running, and will automatically use the `/etc/resolv.conf` file with the correct upstream nameservers (which is located at `/run/systemd/resolve/resolv.conf`).
#### Check CoreDNS logging
```
kubectl -n kube-system logs -l k8s-app=kube-dns
```
#### Check configuration
CoreDNS configuration is stored in the configmap `coredns` in the `kube-system` namespace.
```
kubectl -n kube-system get configmap coredns -o go-template={{.data.Corefile}}
```
#### Check upstream nameservers in resolv.conf
By default, the configured nameservers on the host (in `/etc/resolv.conf`) will be used as upstream nameservers for CoreDNS. You can check this file on the host or run the following Pod with `dnsPolicy` set to `Default`, which will inherit the `/etc/resolv.conf` from the host it is running on.
```
kubectl run -i --restart=Never --rm test-${RANDOM} --image=ubuntu --overrides='{"kind":"Pod", "apiVersion":"v1", "spec": {"dnsPolicy":"Default"}}' -- sh -c 'cat /etc/resolv.conf'
```
#### Enable query logging
Enabling query logging can be done by enabling the [log plugin](https://coredns.io/plugins/log/) in the Corefile configuration in the configmap `coredns`. You can do so by using `kubectl -n kube-system edit configmap coredns` or use the command below to replace the configuration in place:
```
kubectl get configmap -n kube-system coredns -o json | kubectl get configmap -n kube-system coredns -o json | sed -e 's_loadbalance_log\\n loadbalance_g' | kubectl apply -f -
```
All queries will now be logged and can be checked using the command in [Check CoreDNS logging](#check-coredns-logging).
### kube-dns specific
#### Check upstream nameservers in kubedns container
By default, the configured nameservers on the host (in `/etc/resolv.conf`) will be used as upstream nameservers for kube-dns. Sometimes the host will run a local caching DNS nameserver, which means the address in `/etc/resolv.conf` will point to an address in the loopback range (`127.0.0.0/8`) which will be unreachable by the container. In case of Ubuntu 18.04, this is done by `systemd-resolved`. Since Rancher v2.0.7, we detect if `systemd-resolved` is running, and will automatically use the `/etc/resolv.conf` file with the correct upstream nameservers (which is located at `/run/systemd/resolve/resolv.conf`).
Use the following command to check the upstream nameservers used by the kubedns container:
content/rancher/v2.x/en/troubleshooting/kubernetes-components/_index.md
+75 -37
View File
@@ -64,12 +64,12 @@ The address where etcd is listening depends on the address configuration of the
Output should contain all the nodes with the `etcd` role and the output should be identical on all nodes.
Command when no internal address is configured on the host:
Command:
```
docker exec etcd etcdctl member list
```
Command when internal address is configured on the host:
Command when using etcd version lower than 3.3.x (Kubernetes 1.13.x and lower) and `--internal-address` was specified when adding the node:
```
docker exec etcd sh -c "etcdctl --endpoints=\$ETCDCTL_ENDPOINT member list"
```
@@ -85,12 +85,12 @@ xxx, started, etcd-xxx, https://IP:2380, https://IP:2379,https://IP:4001
The values for `RAFT TERM` should be equal and `RAFT INDEX` should be not be too far apart from each other.
Command when no internal address is configured on the host:
Command:
```
docker exec etcd etcdctl endpoint status --endpoints=$(docker exec etcd /bin/sh -c "etcdctl member list | cut -d, -f5 | sed -e 's/ //g' | paste -sd ','") --write-out table
```
Command when internal address is configured on the host:
Command when using etcd version lower than 3.3.x (Kubernetes 1.13.x and lower) and `--internal-address` was specified when adding the node:
```
docker exec etcd etcdctl endpoint status --endpoints=$(docker exec etcd /bin/sh -c "etcdctl --endpoints=\$ETCDCTL_ENDPOINT member list | cut -d, -f5 | sed -e 's/ //g' | paste -sd ','") --write-out table
```
@@ -108,12 +108,12 @@ Example output:
* Check endpoint health
Command when no internal address is configured on the host:
Command:
```
docker exec etcd etcdctl endpoint health --endpoints=$(docker exec etcd /bin/sh -c "etcdctl member list | cut -d, -f5 | sed -e 's/ //g' | paste -sd ','")
```
Command when internal address is configured on the host:
Command when using etcd version lower than 3.3.x (Kubernetes 1.13.x and lower) and `--internal-address` was specified when adding the node:
```
docker exec etcd etcdctl endpoint health --endpoints=$(docker exec etcd /bin/sh -c "etcdctl --endpoints=\$ETCDCTL_ENDPOINT member list | cut -d, -f5 | sed -e 's/ //g' | paste -sd ','")
```
@@ -127,7 +127,9 @@ https://IP:2379 is healthy: successfully committed proposal: took = 2.451201ms
* Check connectivity on port TCP/2379
Command when no internal address is configured on the host:
Requires the `curl` binary on the node.
Command:
```
for endpoint in $(docker exec etcd /bin/sh -c "etcdctl member list | cut -d, -f5"); do
echo "Validating connection to ${endpoint}/health";
@@ -135,7 +137,7 @@ for endpoint in $(docker exec etcd /bin/sh -c "etcdctl member list | cut -d, -f5
done
```
Command when internal address is configured on the host:
Command when using etcd version lower than 3.3.x (Kubernetes 1.13.x and lower) and `--internal-address` was specified when adding the node:
```
for endpoint in $(docker exec etcd /bin/sh -c "etcdctl --endpoints=\$ETCDCTL_ENDPOINT member list | cut -d, -f5"); do
echo "Validating connection to ${endpoint}/health";
@@ -145,19 +147,19 @@ done
If you are running on an operating system without `curl` (for example, RancherOS), you can use the following command which uses a Docker container to run the `curl` command.
Command when no internal address is configured on the host:
Command:
```
for endpoint in $(docker exec etcd /bin/sh -c "etcdctl member list | cut -d, -f5"); do
echo "Validating connection to ${endpoint}/health";
docker run --net=host -v /opt/rke/etc/kubernetes/ssl:/etc/kubernetes/ssl:ro appropriate/curl -s -w "\n" --cacert $(docker exec etcd printenv ETCDCTL_CACERT) --cert $(docker exec etcd printenv ETCDCTL_CERT) --key $(docker exec etcd printenv ETCDCTL_KEY) "${endpoint}/health"
echo "Validating connection to ${endpoint}/health"
docker run --net=host -v $(docker inspect kubelet --format '{{ range .Mounts }}{{ if eq .Destination "/etc/kubernetes" }}{{ .Source }}{{ end }}{{ end }}')/ssl:/etc/kubernetes/ssl:ro appropriate/curl -s -w "\n" --cacert $(docker exec etcd printenv ETCDCTL_CACERT) --cert $(docker exec etcd printenv ETCDCTL_CERT) --key $(docker exec etcd printenv ETCDCTL_KEY) "${endpoint}/health"
done
```
Command when internal address is configured on the host:
Command when using etcd version lower than 3.3.x (Kubernetes 1.13.x and lower) and `--internal-address` was specified when adding the node:
```
for endpoint in $(docker exec etcd /bin/sh -c "etcdctl --endpoints=\$ETCDCTL_ENDPOINT member list | cut -d, -f5"); do
echo "Validating connection to ${endpoint}/health";
docker run --net=host -v /opt/rke/etc/kubernetes/ssl:/etc/kubernetes/ssl:ro appropriate/curl -s -w "\n" --cacert $(docker exec etcd printenv ETCDCTL_CACERT) --cert $(docker exec etcd printenv ETCDCTL_CERT) --key $(docker exec etcd printenv ETCDCTL_KEY) "${endpoint}/health"
docker run --net=host -v $(docker inspect kubelet --format '{{ range .Mounts }}{{ if eq .Destination "/etc/kubernetes" }}{{ .Source }}{{ end }}{{ end }}')/ssl:/etc/kubernetes/ssl:ro appropriate/curl -s -w "\n" --cacert $(docker exec etcd printenv ETCDCTL_CACERT) --cert $(docker exec etcd printenv ETCDCTL_CERT) --key $(docker exec etcd printenv ETCDCTL_KEY) "${endpoint}/health"
done
```
@@ -173,37 +175,37 @@ Validating connection to https://IP:2379/health
* Check connectivity on port TCP/2380
Command when no internal address is configured on the host:
Command:
```
for endpoint in $(docker exec etcd /bin/sh -c "etcdctl member list | cut -d, -f4"); do
echo "Validating connection to ${endpoint}/version";
curl -w "\n" --cacert $(docker exec etcd printenv ETCDCTL_CACERT) --cert $(docker exec etcd printenv ETCDCTL_CERT) --key $(docker exec etcd printenv ETCDCTL_KEY) "${endpoint}/version";
curl --http1.1 -w "\n" --cacert $(docker exec etcd printenv ETCDCTL_CACERT) --cert $(docker exec etcd printenv ETCDCTL_CERT) --key $(docker exec etcd printenv ETCDCTL_KEY) "${endpoint}/version";
done
```
Command when internal address is configured on the host:
Command when using etcd version lower than 3.3.x (Kubernetes 1.13.x and lower) and `--internal-address` was specified when adding the node:
```
for endpoint in $(docker exec etcd /bin/sh -c "etcdctl --endpoints=\$ETCDCTL_ENDPOINT member list | cut -d, -f4"); do
echo "Validating connection to ${endpoint}/version";
curl -w "\n" --cacert $(docker exec etcd printenv ETCDCTL_CACERT) --cert $(docker exec etcd printenv ETCDCTL_CERT) --key $(docker exec etcd printenv ETCDCTL_KEY) "${endpoint}/version";
curl --http1.1 -w "\n" --cacert $(docker exec etcd printenv ETCDCTL_CACERT) --cert $(docker exec etcd printenv ETCDCTL_CERT) --key $(docker exec etcd printenv ETCDCTL_KEY) "${endpoint}/version";
done
```
If you are running on an operating system without `curl` (for example, RancherOS), you can use the following command which uses a Docker container to run the `curl` command.
Command when no internal address is configured on the host:
Command:
```
for endpoint in $(docker exec etcd /bin/sh -c "etcdctl member list | cut -d, -f4"); do
echo "Validating connection to ${endpoint}/version";
docker run --net=host -v /opt/rke/etc/kubernetes/ssl:/etc/kubernetes/ssl:ro appropriate/curl -s -w "\n" --cacert $(docker exec etcd printenv ETCDCTL_CACERT) --cert $(docker exec etcd printenv ETCDCTL_CERT) --key $(docker exec etcd printenv ETCDCTL_KEY) "${endpoint}/version"
docker run --net=host -v $(docker inspect kubelet --format '{{ range .Mounts }}{{ if eq .Destination "/etc/kubernetes" }}{{ .Source }}{{ end }}{{ end }}')/ssl:/etc/kubernetes/ssl:ro appropriate/curl --http1.1 -s -w "\n" --cacert $(docker exec etcd printenv ETCDCTL_CACERT) --cert $(docker exec etcd printenv ETCDCTL_CERT) --key $(docker exec etcd printenv ETCDCTL_KEY) "${endpoint}/version"
done
```
Command when internal address is configured on the host:
Command when using etcd version lower than 3.3.x (Kubernetes 1.13.x and lower) and `--internal-address` was specified when adding the node:
```
for endpoint in $(docker exec etcd /bin/sh -c "etcdctl --endpoints=\$ETCDCTL_ENDPOINT member list | cut -d, -f4"); do
echo "Validating connection to ${endpoint}/version";
docker run --net=host -v /opt/rke/etc/kubernetes/ssl:/etc/kubernetes/ssl:ro appropriate/curl -s -w "\n" --cacert $(docker exec etcd printenv ETCDCTL_CACERT) --cert $(docker exec etcd printenv ETCDCTL_CERT) --key $(docker exec etcd printenv ETCDCTL_KEY) "${endpoint}/version"
docker run --net=host -v $(docker inspect kubelet --format '{{ range .Mounts }}{{ if eq .Destination "/etc/kubernetes" }}{{ .Source }}{{ end }}{{ end }}')/ssl:/etc/kubernetes/ssl:ro appropriate/curl --http1.1 -s -w "\n" --cacert $(docker exec etcd printenv ETCDCTL_CACERT) --cert $(docker exec etcd printenv ETCDCTL_CERT) --key $(docker exec etcd printenv ETCDCTL_KEY) "${endpoint}/version"
done
```
@@ -221,12 +223,12 @@ Validating connection to https://IP:2380/version
etcd will trigger alarms, for instance when it runs out of space.
Command when no internal address is configured on the host:
Command:
```
docker exec etcd etcdctl alarm list
```
Command when internal address is configured on the host:
Command when using etcd version lower than 3.3.x (Kubernetes 1.13.x and lower) and `--internal-address` was specified when adding the node:
```
docker exec etcd sh -c "etcdctl --endpoints=\$ETCDCTL_ENDPOINT alarm list"
```
@@ -246,13 +248,13 @@ Resolution:
* Compact the keyspace
Command when no internal address is configured on the host:
Command:
```
rev=$(docker exec etcd etcdctl endpoint status --write-out json | egrep -o '"revision":[0-9]*' | egrep -o '[0-9]*')
docker exec etcd etcdctl compact "$rev"
```
Command when internal address is configured on the host:
Command when using etcd version lower than 3.3.x (Kubernetes 1.13.x and lower) and `--internal-address` was specified when adding the node:
```
rev=$(docker exec etcd sh -c "etcdctl --endpoints=\$ETCDCTL_ENDPOINT endpoint status --write-out json | egrep -o '\"revision\":[0-9]*' | egrep -o '[0-9]*'")
docker exec etcd sh -c "etcdctl --endpoints=\$ETCDCTL_ENDPOINT compact \"$rev\""
@@ -265,12 +267,12 @@ compacted revision xxx
* Defrag all etcd members
Command when no internal address is configured on the host:
Command:
```
docker exec etcd etcdctl defrag --endpoints=$(docker exec etcd /bin/sh -c "etcdctl member list | cut -d, -f5 | sed -e 's/ //g' | paste -sd ','")
```
Command when internal address is configured on the host:
Command when using etcd version lower than 3.3.x (Kubernetes 1.13.x and lower) and `--internal-address` was specified when adding the node:
```
docker exec etcd sh -c "etcdctl defrag --endpoints=$(docker exec etcd /bin/sh -c "etcdctl --endpoints=\$ETCDCTL_ENDPOINT member list | cut -d, -f5 | sed -e 's/ //g' | paste -sd ','")"
```
@@ -284,12 +286,12 @@ Finished defragmenting etcd member[https://IP:2379]
* Check endpoint status
Command when no internal address is configured on the host:
Command:
```
docker exec etcd etcdctl endpoint status --endpoints=$(docker exec etcd /bin/sh -c "etcdctl member list | cut -d, -f5 | sed -e 's/ //g' | paste -sd ','") --write-out table
```
Command when internal address is configured on the host:
Command when using etcd version lower than 3.3.x (Kubernetes 1.13.x and lower) and `--internal-address` was specified when adding the node:
```
docker exec etcd sh -c "etcdctl endpoint status --endpoints=$(docker exec etcd /bin/sh -c "etcdctl --endpoints=\$ETCDCTL_ENDPOINT member list | cut -d, -f5 | sed -e 's/ //g' | paste -sd ','") --write-out table"
```
@@ -309,14 +311,14 @@ Example output:
After verifying that the DB size went down after compaction and defragmenting, the alarm needs to be disarmed for etcd to allow writes again.
Command when no internal address is configured on the host:
Command:
```
docker exec etcd etcdctl alarm list
docker exec etcd etcdctl alarm disarm
docker exec etcd etcdctl alarm list
```
Command when internal address is configured on the host:
Command when using etcd version lower than 3.3.x (Kubernetes 1.13.x and lower) and `--internal-address` was specified when adding the node:
```
docker exec etcd sh -c "etcdctl --endpoints=\$ETCDCTL_ENDPOINT alarm list"
docker exec etcd sh -c "etcdctl --endpoints=\$ETCDCTL_ENDPOINT alarm disarm"
@@ -337,28 +339,64 @@ docker exec etcd etcdctl alarm list
The log level of etcd can be changed dynamically via the API. You can configure debug logging using the commands below.
Command when no internal address is configured on the host:
Command:
```
curl -XPUT -d '{"Level":"DEBUG"}' --cacert $(docker exec etcd printenv ETCDCTL_CACERT) --cert $(docker exec etcd printenv ETCDCTL_CERT) --key $(docker exec etcd printenv ETCDCTL_KEY) https://localhost:2379/config/local/log
curl -XPUT -d '{"Level":"DEBUG"}' --cacert $(docker exec etcd printenv ETCDCTL_CACERT) --cert $(docker exec etcd printenv ETCDCTL_CERT) --key $(docker exec etcd printenv ETCDCTL_KEY) $(docker exec etcd printenv ETCDCTL_ENDPOINTS)/config/local/log
```
Command when internal address is configured on the host:
Command when using etcd version lower than 3.3.x (Kubernetes 1.13.x and lower) and `--internal-address` was specified when adding the node:
```
curl -XPUT -d '{"Level":"DEBUG"}' --cacert $(docker exec etcd printenv ETCDCTL_CACERT) --cert $(docker exec etcd printenv ETCDCTL_CERT) --key $(docker exec etcd printenv ETCDCTL_KEY) $(docker exec etcd printenv $ETCDCTL_ENDPOINT)/config/local/log
```
To reset the log level back to the default (`INFO`), you can use the following command.
Command when no internal address is configured on the host:
Command:
```
curl -XPUT -d '{"Level":"INFO"}' --cacert $(docker exec etcd printenv ETCDCTL_CACERT) --cert $(docker exec etcd printenv ETCDCTL_CERT) --key $(docker exec etcd printenv ETCDCTL_KEY) https://localhost:2379/config/local/log
curl -XPUT -d '{"Level":"INFO"}' --cacert $(docker exec etcd printenv ETCDCTL_CACERT) --cert $(docker exec etcd printenv ETCDCTL_CERT) --key $(docker exec etcd printenv ETCDCTL_KEY) $(docker exec etcd printenv ETCDCTL_ENDPOINTS)/config/local/log
```
Command when internal address is configured on the host:
Command when using etcd version lower than 3.3.x (Kubernetes 1.13.x and lower) and `--internal-address` was specified when adding the node:
```
curl -XPUT -d '{"Level":"INFO"}' --cacert $(docker exec etcd printenv ETCDCTL_CACERT) --cert $(docker exec etcd printenv ETCDCTL_CERT) --key $(docker exec etcd printenv ETCDCTL_KEY) $(docker exec etcd printenv $ETCDCTL_ENDPOINT)/config/local/log
```
### etcd content
If you want to investigate the contents of your etcd, you can either watch streaming events or you can query etcd directly, see below for examples.
* Watch streaming events
Command:
```
docker exec etcd etcdctl watch --prefix /registry
```
Command when using etcd version lower than 3.3.x (Kubernetes 1.13.x and lower) and `--internal-address` was specified when adding the node:
```
docker exec etcd etcdctl --endpoints=\$ETCDCTL_ENDPOINT watch --prefix /registry
```
If you only want to see the affected keys (and not the binary data), you can append `| grep -a ^/registry` to the command to filter for keys only.
* Query etcd directly
Command:
```
docker exec etcd etcdctl get /registry --prefix=true --keys-only
```
Command when using etcd version lower than 3.3.x (Kubernetes 1.13.x and lower) and `--internal-address` was specified when adding the node:
```
docker exec etcd etcdctl --endpoints=\$ETCDCTL_ENDPOINT get /registry --prefix=true --keys-only
```
You can process the data to get a summary of count per key, using the command below:
```
docker exec etcd etcdctl get /registry --prefix=true --keys-only | grep -v ^$ | awk -F'/' '{ if ($3 ~ /cattle.io/) {h[$3"/"$4]++} else { h[$3]++ }} END { for(k in h) print h[k], k }' | sort -nr
```
## controlplane
This section applies to nodes with the `controlplane` role.
content/rancher/v2.x/en/upgrades/upgrades/ha-server-upgrade-helm-airgap/_index.md
+2
View File
@@ -5,6 +5,8 @@ weight: 1021
The following instructions will guide you through upgrading a high-availability Rancher Server installed in an air gap environment.
>**Note:** [Let's Encrypt will be blocking cert-manager instances older than 0.8.0 starting November 1st 2019.](https://community.letsencrypt.org/t/blocking-old-cert-manager-versions/98753) Upgrade cert-manager to the latest version by following [these instructions.]({{< baseurl >}}/rancher/v2.x/en/cluster-admin/upgrade-cert-manager-airgap)
## Prerequisites
- **Populate Images**
content/rancher/v2.x/en/upgrades/upgrades/ha-server-upgrade-helm/_index.md
+6 -2
View File
@@ -7,10 +7,14 @@ The following instructions will guide you through upgrading a high-availability
>**Note:** If you installed Rancher using the RKE Add-on yaml, see the following documents to migrate or upgrade.
>
>* [Migrating from RKE Add-On Install]({{< baseurl >}}/rancher/v2.x/en/upgrades/upgrades/migrating-from-rke-add-on)
>- [Migrating from RKE Add-On Install]({{< baseurl >}}/rancher/v2.x/en/upgrades/upgrades/migrating-from-rke-add-on)
>
> As of release v2.0.8, Rancher supports installation and upgrade by Helm chart, although RKE installs/upgrades are still supported as well. If you want to change upgrade method from RKE Add-on to Helm chart, follow this procedure.
---
>**Note:** [Let's Encrypt will be blocking cert-manager instances older than 0.8.0 starting November 1st 2019.](https://community.letsencrypt.org/t/blocking-old-cert-manager-versions/98753) Upgrade cert-manager to the latest version by following [these instructions.]({{< baseurl >}}/rancher/v2.x/en/cluster-admin/upgrade-cert-manager)
## Prerequisites
@@ -42,7 +46,7 @@ Upgrades _to_ or _from_ any chart in the [rancher-alpha repository]({{< baseurl
## Upgrade Rancher
> **Note:** For Air Gap installs see [Upgrading HA Rancher - Air Gap]({{< baseurl >}}/rancher/v2.x/en/installation/air-gap-installation/install-rancher/#upgrading-rancher)
> **Note:** For Air Gap installs see [Upgrading HA Rancher - Air Gap]({{< baseurl >}}/rancher/v2.x/en/upgrades/upgrades/ha-server-upgrade-helm-airgap/)
1. Update your local helm repo cache.
content/rke/latest/en/config-options/services/_index.md
+1 -1
View File
@@ -18,7 +18,7 @@ To deploy Kubernetes, RKE deploys several core components or services in Docker
## etcd
Kubernetes uses [etcd](https://github.com/coreos/etcd/blob/master/Documentation/docs.md) as a store for cluster state and data. Etcd is a reliable, consistent and distributed key-value store.
Kubernetes uses [etcd](https://etcd.io/) as a store for cluster state and data. Etcd is a reliable, consistent and distributed key-value store.
RKE supports running etcd in a single node mode or in HA cluster mode. It also supports adding and removing etcd nodes to the cluster.
content/rke/latest/en/etcd-snapshots/_index.md
+9 -282
View File
@@ -11,295 +11,22 @@ RKE clusters can be configured to automatically take snapshots of etcd. In a dis
_Available as of v0.2.0_
RKE can also upload your snapshots to a S3 compatible backend. Additionally, the **pki.bundle.tar.gz** file usage is no longer required as v0.2.0 has changed how the [Kubernetes cluster state is stored]({{< baseurl >}}/rke/latest/en/installation/#kubernetes-cluster-state).
RKE can upload your snapshots to a S3 compatible backend.
## One-Time Snapshots
**Note:** As of RKE v0.2.0, the `pki.bundle.tar.gz` file is no longer required because of a change in how the [Kubernetes cluster state is stored]({{< baseurl >}}/rke/latest/en/installation/#kubernetes-cluster-state).
The `rke etcd snapshot-save` command will save a snapshot of etcd from each etcd node in the cluster config file. The snapshot is saved in `/opt/rke/etcd-snapshots`. When running the command, an additional container is created to take the snapshot. When the snapshot is completed, the container is automatically removed.
# Backing Up a Cluster
Prior to v0.2.0, along with the individual snapshot, RKE saves a backup of the certificates, i.e. a file named `pki.bundle.tar.gz`, in the same location. The snapshot and pki bundle file are required for the restore process in versions prior to v0.2.0.
You can create [one-time snapshots]({{<baseurl>}}/rke/latest/en/etcd-snapshots/one-time-snapshots) to back up your cluster, and you can also configure [recurring snapshots]({{<baseurl>}}/rke/latest/en/etcd-snapshots/recurring-snapshots).
As of v0.2.0, the one-time snapshot can be uploaded to a S3 compatible backend by using the additional options to specify the S3 backend.
# Restoring a Cluster from Backup
### Options for `rke etcd snapshot-save`
You can use RKE to [restore your cluster from backup]({{<baseurl>}}/rke/latest/en/etcd-snapshots/restoring-from-backup).
| Option | Description | S3 Specific |
| --- | --- | --- |
| `--name` value | Specify snapshot name | |
| `--config` value | Specify an alternate cluster YAML file (default: "cluster.yml") [$RKE_CONFIG] | |
| `--s3` | Enabled backup to s3 | * |
| `--s3-endpoint` value | Specify s3 endpoint url (default: "s3.amazonaws.com") | * |
| `--access-key` value | Specify s3 accessKey | * |
| `--secret-key` value | Specify s3 secretKey | * |
| `--bucket-name` value | Specify s3 bucket name | * |
| `--region` value | Specify the s3 bucket location (optional) | * |
| `--ssh-agent-auth` | [Use SSH Agent Auth defined by SSH_AUTH_SOCK]({{< baseurl >}}/rke/latest/en/config-options/#ssh-agent) | |
| `--ignore-docker-version` | [Disable Docker version check]({{< baseurl >}}/rke/latest/en/config-options/#supported-docker-versions) |
# Example Scenarios
### IAM Support for Storing Snapshots in S3
In addition to API access keys, RKE supports using IAM roles for S3 authentication. The cluster etcd nodes must be assigned an IAM role that has read/write access to the designated backup bucket on S3. Also, the nodes must have network access to the S3 endpoint specified.
To give an application access to S3, refer to the AWS documentation on [Using an IAM Role to Grant Permissions to Applications Running on Amazon EC2 Instances.](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html)
### Local One-Time Snapshot Example
```
$ rke etcd snapshot-save --config cluster.yml --name snapshot-name
```
The snapshot is saved in `/opt/rke/etcd-snapshots`
### One-Time Snapshots uploaded to S3 Example
_Available as of v0.2.0_
```
$ rke etcd snapshot-save --config cluster.yml --name snapshot-name \
--s3 --access-key S3_ACCESS_KEY --secret-key S3_SECRET_KEY \
--bucket-name s3-bucket-name --s3-endpoint s3.amazonaws.com
```
The snapshot is saved in `/opt/rke/etcd-snapshots` as well as uploaded to the S3 backend.
## Recurring Snapshots
To schedule automatic recurring etcd snapshots, you can enable the `etcd-snapshot` service with [extra configuration options the etcd service](#options-for-the-etcd-snapshot-service). `etcd-snapshot` runs in a service container alongside the `etcd` container. By default, the `etcd-snapshot` service takes a snapshot for every node that has the `etcd` role and stores them to local disk in `/opt/rke/etcd-snapshots`. If you set up the [options for S3](#options-for-the-etcd-snapshot-service), the snapshot will also be uploaded to the S3 backend.
Prior to v0.2.0, along with the snapshots, RKE saves a backup of the certificates, i.e. a file named `pki.bundle.tar.gz`, in the same location. The snapshot and pki bundle file are required for the restore process in versions prior to v0.2.0.
When a cluster is launched with the `etcd-snapshot` service enabled, you can view the `etcd-rolling-snapshots` logs to confirm backups are being created automatically.
```
$ docker logs etcd-rolling-snapshots
time="2018-05-04T18:39:16Z" level=info msg="Initializing Rolling Backups" creation=1m0s retention=24h0m0s
time="2018-05-04T18:40:16Z" level=info msg="Created backup" name="2018-05-04T18:40:16Z_etcd" runtime=108.332814ms
time="2018-05-04T18:41:16Z" level=info msg="Created backup" name="2018-05-04T18:41:16Z_etcd" runtime=92.880112ms
time="2018-05-04T18:42:16Z" level=info msg="Created backup" name="2018-05-04T18:42:16Z_etcd" runtime=83.67642ms
time="2018-05-04T18:43:16Z" level=info msg="Created backup" name="2018-05-04T18:43:16Z_etcd" runtime=86.298499ms
```
### Options for the `Etcd-Snapshot` Service
Depending on your version of RKE, the options used to configure recurring snapshots may be different.
_Available as of v0.2.0_
|Option|Description| S3 Specific |
|---|---| --- |
|**interval_hours**| The duration in hours between recurring backups. This supercedes the `creation` option and will override it if both are specified.| |
|**retention**| The number of snapshots to retain before rotation. This supercedes the `retention` option and will override it if both are specified.| |
|**bucket_name**| S3 bucket name where backups will be stored| * |
|**access_key**| S3 access key with permission to access the backup bucket.| * |
|**secret_key** |S3 secret key with permission to access the backup bucket.| * |
|**region** |S3 region for the backup bucket. This is optional.| * |
|**endpoint** |S3 regions endpoint for the backup bucket.| * |
<br>
```yaml
services:
etcd:
backup_config:
interval_hours: 12
retention: 6
s3backupconfig:
access_key: S3_ACCESS_KEY
secret_key: S3_SECRET_KEY
bucket_name: s3-bucket-name
region: ""
endpoint: s3.amazonaws.com
```
#### Prior to v0.2.0
|Option|Description|
|---|---|
|**Snapshot**|By default, the recurring snapshot service is disabled. To enable the service, you need to define it as part of `etcd` and set it to `true`.|
|**Creation**|By default, the snapshot service will take snapshots every 5 minutes (`5m0s`). You can change the time between snapshots as part of the `creation` directive for the `etcd` service.|
|**Retention**|By default, all snapshots are saved for 24 hours (`24h`) before being deleted and purged. You can change how long to store a snapshot as part of the `retention` directive for the `etcd` service.|
```yaml
services:
etcd:
snapshot: true
creation: 5m0s
retention: 24h
```
## Etcd Disaster Recovery
If there is a disaster with your Kubernetes cluster, you can use `rke etcd snapshot-restore` to recover your etcd. This command reverts etcd to a specific snapshot. RKE also removes the old `etcd` container before creating a new `etcd` cluster using the snapshot that you have chosen.
>**Warning:** Restoring an etcd snapshot deletes your current etcd cluster and replaces it with a new one. Before you run the `rke etcd snapshot-restore` command, you should back up any important data in your cluster.
The snapshot used to restore your etcd cluster can either be stored locally in `/opt/rke/etcd-snapshots` or from a S3 compatible backend. The S3 backend option is available as of v0.2.0.
### Options for `rke etcd snapshot-restore`
| Option | Description | S3 Specific |
| --- | --- | ---|
| `--name` value | Specify snapshot name | |
| `--config` value | Specify an alternate cluster YAML file (default: "cluster.yml") [$RKE_CONFIG] | |
| `--s3` | Enabled backup to s3 |* |
| `--s3-endpoint` value | Specify s3 endpoint url (default: "s3.amazonaws.com") | * |
| `--access-key` value | Specify s3 accessKey | *|
| `--secret-key` value | Specify s3 secretKey | *|
| `--bucket-name` value | Specify s3 bucket name | *|
| `--region` value | Specify the s3 bucket location (optional) | *|
| `--ssh-agent-auth` | [Use SSH Agent Auth defined by SSH_AUTH_SOCK]({{< baseurl >}}/rke/latest/en/config-options/#ssh-agent) | |
| `--ignore-docker-version` | [Disable Docker version check]({{< baseurl >}}/rke/latest/en/config-options/#supported-docker-versions) |
### Example of Restoring from a Local Snapshot
When restoring etcd from a local snapshot, the snapshot is assumed to be located in `/opt/rke/etcd-snapshots`. In versions prior to v0.2.0, the `pki.bundle.tar.gz` file is also expected to be in the same location. As of v0.2.0, this file is no longer needed as v0.2.0 has changed how the [Kubernetes cluster state is stored]({{< baseurl >}}/rke/latest/en/installation/#kubernetes-cluster-state).
```
$ rke etcd snapshot-restore --config cluster.yml --name mysnapshot
```
### Example of Restoring from a Snapshot in S3
_Available as of v0.2.0_
> **Note:** Ensure your `cluster.rkestate` is present before starting the restore, as this contains your certificate data for the cluster
When restoring etcd from a snapshot located in S3, the command needs the S3 information in order to connect to the S3 backend and retrieve the snapshot.
```shell
$ rke etcd snapshot-restore --config cluster.yml --name snapshot-name \
--s3 --access-key S3_ACCESS_KEY --secret-key S3_SECRET_KEY \
--bucket-name s3-bucket-name --s3-endpoint s3.amazonaws.com
```
> **Note:** if you were restoring a cluster that had rancher installed the UI should start-up after a few minutes; you don't need to re-run helm.
### Example Scenario of restoring from a Local Snapshot
In this example, the Kubernetes cluster was deployed on two AWS nodes.
| Name | IP | Role |
|:-----:|:--------:|:----------------------:|
| node1 | 10.0.0.1 | [controlplane, worker] |
| node2 | 10.0.0.2 | [etcd] |
### Back up the `etcd` cluster
Take a local snapshot of the Kubernetes cluster. As of v0.2.0, you can also upload this snapshot directly to a S3 backend with the [S3 options](#options-for-rke-etcd-snapshot-save).
```
$ rke etcd snapshot-save --name snapshot.db --config cluster.yml
```
![etcd snapshot]({{< baseurl >}}/img/rke/rke-etcd-backup.png)
### Store the Snapshot Externally in S3
As of v0.2.0, this step is no longer required, as RKE can upload and download snapshots automatically from S3 by adding in [S3 options](#options-for-rke-etcd-snapshot-save) when running the `rke etcd snapshot-save` command.
After taking the etcd snapshot on `node2`, we recommend saving this backup in a persistence place. One of the options is to save the backup and `pki.bundle.tar.gz` file on a S3 bucket or tape backup.
> **Note:** As of v0.2.0, the file **pki.bundle.tar.gz** is no longer required for the restore process.
```
# If you're using an AWS host and have the ability to connect to S3
root@node2:~# s3cmd mb s3://rke-etcd-backup
root@node2:~# s3cmd /opt/rke/etcd-snapshots/snapshot.db /opt/rke/etcd-snapshots/pki.bundle.tar.gz s3://rke-etcd-backup/
```
### Place the backup on a new node
To simulate the failure, let's power down `node2`.
```
root@node2:~# poweroff
```
| Name | IP | Role |
|:-----:|:--------:|:----------------------:|
| node1 | 10.0.0.1 | [controlplane, worker] |
| ~~node2~~ | ~~10.0.0.2~~ | ~~[etcd]~~ |
| node3 | 10.0.0.3 | [etcd] |
| | | |
Before restoring etcd and running `rke up`, we need to retrieve the backup saved on S3 to a new node, e.g. `node3`. As of v0.2.0, you can directly retrieve the snapshot from S3 when running the restore command, so this step is for users who stored the snapshot externally without using the integrated S3 options.
```
# Make a Directory
root@node3:~# mkdir -p /opt/rke/etcdbackup
# Get the Backup from S3
root@node3:~# s3cmd get s3://rke-etcd-backup/snapshot.db /opt/rke/etcd-snapshots/snapshot.db
# Get the pki bundle from S3, only needed prior to v0.2.0
root@node3:~# s3cmd get s3://rke-etcd-backup/pki.bundle.tar.gz /opt/rke/etcd-snapshots/pki.bundle.tar.gz
```
### Restore `etcd` on the new node from the backup
Before updating and restoring etcd, you will need to add the new node into the Kubernetes cluster with the `etcd` role. In the `cluster.yml`, comment out the old node and add in the new node. `
```yaml
nodes:
- address: 10.0.0.1
hostname_override: node1
user: ubuntu
role:
- controlplane
- worker
# - address: 10.0.0.2
# hostname_override: node2
# user: ubuntu
# role:
# - etcd
- address: 10.0.0.3
hostname_override: node3
user: ubuntu
role:
- etcd
```
After the new node is added to the `cluster.yml`, run `rke etcd snapshot-restore` to launch `etcd` from the backup. The snapshot and `pki.bundle.tar.gz` file are expected to be saved at `/opt/rke/etcd-snapshots`.
As of v0.2.0, if you want to directly retrieve the snapshot from S3, add in the [S3 options](#options-for-rke-etcd-snapshot-restore).
> **Note:** As of v0.2.0, the file **pki.bundle.tar.gz** is no longer required for the restore process as the certificates required to restore are preserved within the `cluster.rkestate`
```
$ rke etcd snapshot-restore --name snapshot.db --config cluster.yml
```
Finally, we need to restore the operations on the cluster by making the Kubernetes API point to the new `etcd` by running `rke up` again using the new `cluster.yml`.
```
$ rke up --config cluster.yml
```
Confirm that your Kubernetes cluster is functional by checking the pods on your cluster.
```
> kubectl get pods
NAME READY STATUS RESTARTS AGE
nginx-65899c769f-kcdpr 1/1 Running 0 17s
nginx-65899c769f-pc45c 1/1 Running 0 17s
nginx-65899c769f-qkhml 1/1 Running 0 17s
```
These [example scenarios]({{<baseurl>}}/rke/latest/en/etcd-snapshots/example-scenarios) for backup and restore are different based on your version of RKE.
## Troubleshooting
As of **v0.1.9**, the **rke-bundle-cert** container is removed on both success and failure of a restore. To debug any issues, you will need to look at the **logs** generated from rke.
As of **v0.1.8** and below, the **rke-bundle-cert** container is left over from a failed etcd restore. If you are having an issue with restoring an **etcd snapshot** then you can do the following on each etcd nodes before attempting to do another restore:
```
docker container rm --force rke-bundle-cert
```
The rke-bundle-cert container is usually removed when a backup or restore of **etcd** succeeds. Whenever something goes wrong, the **rke-bundle-cert** container will be left over. You can look
at the logs or inspect the container to see what the issue is.
```
docker container logs --follow rke-bundle-cert
docker container inspect rke-bundle-cert
```
The important thing to note is the mounts of the container and location of the **pki.bundle.tar.gz**.
If you have trouble restoring your cluster, you can refer to the [troubleshooting]({{<baseurl>}}/rke/latest/en/etcd-snapshots/troubleshooting) page.
content/rke/latest/en/etcd-snapshots/example-scenarios/_index.md
+253
View File
@@ -0,0 +1,253 @@
---
title: Example Scenarios
weight: 4
---
These example scenarios for backup and restore are different based on your version of RKE.
{{% tabs %}}
{{% tab "RKE v0.2.0+" %}}
This walkthrough will demonstrate how to restore an etcd cluster from a local snapshot with the following steps:
1. [Back up the cluster](#1-back-up-the-cluster)
1. [Simulate a node failure](#2-simulate-a-node-failure)
1. [Add a new etcd node to the cluster](#3-add-a-new-etcd-node-to-the-kubernetes-cluster)
1. [Restore etcd on the new node from the backup](#4-restore-etcd-on-the-new-node-from-the-backup)
1. [Confirm that cluster operations are restored](#5-confirm-that-cluster-operations-are-restored)
In this example, the Kubernetes cluster was deployed on two AWS nodes.
| Name | IP | Role |
|:-----:|:--------:|:----------------------:|
| node1 | 10.0.0.1 | [controlplane, worker] |
| node2 | 10.0.0.2 | [etcd] |
### 1. Back Up the Cluster
Take a local snapshot of the Kubernetes cluster.
You can upload this snapshot directly to an S3 backend with the [S3 options]({{<baseurl>}}/rke/latest/en/etcd-snapshots/one-time-snapshots/#options-for-rke-etcd-snapshot-save).
```
$ rke etcd snapshot-save --name snapshot.db --config cluster.yml
```
![etcd snapshot]({{< baseurl >}}/img/rke/rke-etcd-backup.png)
### 2. Simulate a Node Failure
To simulate the failure, let's power down `node2`.
```
root@node2:~# poweroff
```
| Name | IP | Role |
|:-----:|:--------:|:----------------------:|
| node1 | 10.0.0.1 | [controlplane, worker] |
| ~~node2~~ | ~~10.0.0.2~~ | ~~[etcd]~~ |
### 3. Add a New etcd Node to the Kubernetes Cluster
Before updating and restoring etcd, you will need to add the new node into the Kubernetes cluster with the `etcd` role. In the `cluster.yml`, comment out the old node and add in the new node.
```yaml
nodes:
- address: 10.0.0.1
hostname_override: node1
user: ubuntu
role:
- controlplane
- worker
# - address: 10.0.0.2
# hostname_override: node2
# user: ubuntu
# role:
# - etcd
- address: 10.0.0.3
hostname_override: node3
user: ubuntu
role:
- etcd
```
### 4. Restore etcd on the New Node from the Backup
> **Prerequisite:** Ensure your `cluster.rkestate` is present before starting the restore, because this contains your certificate data for the cluster.
After the new node is added to the `cluster.yml`, run the `rke etcd snapshot-restore` to launch `etcd` from the backup:
```
$ rke etcd snapshot-restore --name snapshot.db --config cluster.yml
```
The snapshot is expected to be saved at `/opt/rke/etcd-snapshots`.
If you want to directly retrieve the snapshot from S3, add in the [S3 options](#options-for-rke-etcd-snapshot-restore).
> **Note:** As of v0.2.0, the file `pki.bundle.tar.gz` is no longer required for the restore process because the certificates required to restore are preserved within the `cluster.rkestate`.
### 5. Confirm that Cluster Operations are Restored
The `rke etcd snapshot-restore` command triggers `rke up` using the new `cluster.yml`. Confirm that your Kubernetes cluster is functional by checking the pods on your cluster.
```
> kubectl get pods
NAME READY STATUS RESTARTS AGE
nginx-65899c769f-kcdpr 1/1 Running 0 17s
nginx-65899c769f-pc45c 1/1 Running 0 17s
nginx-65899c769f-qkhml 1/1 Running 0 17s
```
{{% /tab %}}
{{% tab "RKE prior to v0.2.0" %}}
This walkthrough will demonstrate how to restore an etcd cluster from a local snapshot with the following steps:
1. [Take a local snapshot of the cluster](#take-a-local-snapshot-of-the-cluster-rke-prior-to-v0.2.0)
1. [Store the snapshot externally](#store-the-snapshot-externally-rke-prior-to-v0.2.0)
1. [Simulate a node failure](#simulate-a-node-failure-rke-prior-to-v0.2.0)
1. [Remove the Kubernetes cluster and clean the nodes](#remove-the-kubernetes-cluster-and-clean-the-nodes-rke-prior-to-v0.2.0)
1. [Retrieve the backup and place it on a new node](#retrieve-the-backup-and-place-it-on-a-new-node-rke-prior-to-v0.2.0)
1. [Add a new etcd node to the Kubernetes cluster](#add-a-new-etcd-node-to-the-kubernetes-cluster-rke-prior-to-v0.2.0)
1. [Restore etcd on the new node from the backup](#restore-etcd-on-the-new-node-from-the-backup-rke-prior-to-v0.2.0)
1. [Restore Operations on the Cluster](#restore-operations-on-the-cluster-rke-prior-to-v0.2.0)
### Example Scenario of restoring from a Local Snapshot
In this example, the Kubernetes cluster was deployed on two AWS nodes.
| Name | IP | Role |
|:-----:|:--------:|:----------------------:|
| node1 | 10.0.0.1 | [controlplane, worker] |
| node2 | 10.0.0.2 | [etcd] |
<a id="take-a-local-snapshot-of-the-cluster-rke-prior-to-v0.2.0"></a>
### 1. Take a Local Snapshot of the Cluster
Back up the Kubernetes cluster by taking a local snapshot:
```
$ rke etcd snapshot-save --name snapshot.db --config cluster.yml
```
![etcd snapshot]({{< baseurl >}}/img/rke/rke-etcd-backup.png)
<a id="store-the-snapshot-externally-rke-prior-to-v0.2.0"></a>
### 2. Store the Snapshot Externally
After taking the etcd snapshot on `node2`, we recommend saving this backup in a persistent place. One of the options is to save the backup and `pki.bundle.tar.gz` file on an S3 bucket or tape backup.
```
# If you're using an AWS host and have the ability to connect to S3
root@node2:~# s3cmd mb s3://rke-etcd-backup
root@node2:~# s3cmd \
/opt/rke/etcd-snapshots/snapshot.db \
/opt/rke/etcd-snapshots/pki.bundle.tar.gz \
s3://rke-etcd-backup/
```
<a id="simulate-a-node-failure-rke-prior-to-v0.2.0"></a>
### 3. Simulate a Node Failure
To simulate the failure, let's power down `node2`.
```
root@node2:~# poweroff
```
| Name | IP | Role |
|:-----:|:--------:|:----------------------:|
| node1 | 10.0.0.1 | [controlplane, worker] |
| ~~node2~~ | ~~10.0.0.2~~ | ~~[etcd]~~ |
<a id="remove-the-kubernetes-cluster-and-clean-the-nodes-rke-prior-to-v0.2.0"></a>
### 4. Remove the Kubernetes Cluster and Clean the Nodes
The following command removes your cluster and cleans the nodes so that the cluster can be restored without any conflicts:
```
rke remove --config rancher-cluster.yml
```
<a id="retrieve-the-backup-and-place-it-on-a-new-node-rke-prior-to-v0.2.0"></a>
### 5. Retrieve the Backup and Place it On a New Node
Before restoring etcd and running `rke up`, we need to retrieve the backup saved on S3 to a new node, e.g. `node3`.
```
# Make a Directory
root@node3:~# mkdir -p /opt/rke/etcdbackup
# Get the Backup from S3
root@node3:~# s3cmd get \
s3://rke-etcd-backup/snapshot.db \
/opt/rke/etcd-snapshots/snapshot.db
# Get the pki bundle from S3
root@node3:~# s3cmd get \
s3://rke-etcd-backup/pki.bundle.tar.gz \
/opt/rke/etcd-snapshots/pki.bundle.tar.gz
```
> **Note:** If you had multiple etcd nodes, you would have to manually sync the snapshot and `pki.bundle.tar.gz` across all of the etcd nodes in the cluster.
<a id="add-a-new-etcd-node-to-the-kubernetes-cluster-rke-prior-to-v0.2.0"></a>
### 6. Add a New etcd Node to the Kubernetes Cluster
Before updating and restoring etcd, you will need to add the new node into the Kubernetes cluster with the `etcd` role. In the `cluster.yml`, comment out the old node and add in the new node. `
```yaml
nodes:
- address: 10.0.0.1
hostname_override: node1
user: ubuntu
role:
- controlplane
- worker
# - address: 10.0.0.2
# hostname_override: node2
# user: ubuntu
# role:
# - etcd
- address: 10.0.0.3
hostname_override: node3
user: ubuntu
role:
- etcd
```
<a id="restore-etcd-on-the-new-node-from-the-backup-rke-prior-to-v0.2.0"></a>
### 7. Restore etcd on the New Node from the Backup
After the new node is added to the `cluster.yml`, run the `rke etcd snapshot-restore` command to launch `etcd` from the backup:
```
$ rke etcd snapshot-restore --name snapshot.db --config cluster.yml
```
The snapshot and `pki.bundle.tar.gz` file are expected to be saved at `/opt/rke/etcd-snapshots` on each etcd node.
<a id="restore-operations-on-the-cluster-rke-prior-to-v0.2.0"></a>
### 8. Restore Operations on the Cluster
Finally, we need to restore the operations on the cluster. We will make the Kubernetes API point to the new `etcd` by running `rke up` again using the new `cluster.yml`.
```
$ rke up --config cluster.yml
```
Confirm that your Kubernetes cluster is functional by checking the pods on your cluster.
```
> kubectl get pods
NAME READY STATUS RESTARTS AGE
nginx-65899c769f-kcdpr 1/1 Running 0 17s
nginx-65899c769f-pc45c 1/1 Running 0 17s
nginx-65899c769f-qkhml 1/1 Running 0 17s
```
{{% /tab %}}
{{% /tabs %}}
content/rke/latest/en/etcd-snapshots/one-time-snapshots/_index.md
+114
View File
@@ -0,0 +1,114 @@
---
title: One-time Snapshots
weight: 1
---
One-time snapshots are handled differently depending on your version of RKE.
{{% tabs %}}
{{% tab "RKE v0.2.0+" %}}
To save a snapshot of etcd from each etcd node in the cluster config file, run the `rke etcd snapshot-save` command.
The snapshot is saved in `/opt/rke/etcd-snapshots`.
When running the command, an additional container is created to take the snapshot. When the snapshot is completed, the container is automatically removed.
The one-time snapshot can be uploaded to a S3 compatible backend by using the additional options to specify the S3 backend.
To create a local one-time snapshot, run:
```
$ rke etcd snapshot-save --config cluster.yml --name snapshot-name
```
**Result:** The snapshot is saved in `/opt/rke/etcd-snapshots`.
To save a one-time snapshot to S3, run:
```
$ rke etcd snapshot-save \
--config cluster.yml \
--name snapshot-name \
--s3 \
--access-key S3_ACCESS_KEY \
--secret-key S3_SECRET_KEY \
--bucket-name s3-bucket-name \
--s3-endpoint s3.amazonaws.com
```
**Result:** The snapshot is saved in `/opt/rke/etcd-snapshots` as well as uploaded to the S3 backend.
### Options for `rke etcd snapshot-save`
| Option | Description | S3 Specific |
| --- | --- | --- |
| `--name` value | Specify snapshot name | |
| `--config` value | Specify an alternate cluster YAML file (default: `cluster.yml`) [$RKE_CONFIG] | |
| `--s3` | Enabled backup to s3 | * |
| `--s3-endpoint` value | Specify s3 endpoint url (default: "s3.amazonaws.com") | * |
| `--access-key` value | Specify s3 accessKey | * |
| `--secret-key` value | Specify s3 secretKey | * |
| `--bucket-name` value | Specify s3 bucket name | * |
| `--region` value | Specify the s3 bucket location (optional) | * |
| `--ssh-agent-auth` | [Use SSH Agent Auth defined by SSH_AUTH_SOCK]({{< baseurl >}}/rke/latest/en/config-options/#ssh-agent) | |
| `--ignore-docker-version` | [Disable Docker version check]({{< baseurl >}}/rke/latest/en/config-options/#supported-docker-versions) |
The `--access-key` and `--secret-key` options are not required if the `etcd` nodes are AWS EC2 instances that have been configured with a suitable IAM instance profile.
### IAM Support for Storing Snapshots in S3
In addition to API access keys, RKE supports using IAM roles for S3 authentication. The cluster etcd nodes must be assigned an IAM role that has read/write access to the designated backup bucket on S3. Also, the nodes must have network access to the S3 endpoint specified.
Below is an [example IAM policy](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_examples_s3_rw-bucket.html) that would allow nodes to store and retrieve backups from S3:
```
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "ListObjectsInBucket",
"Effect": "Allow",
"Action": ["s3:ListBucket"],
"Resource": ["arn:aws:s3:::bucket-name"]
},
{
"Sid": "AllObjectActions",
"Effect": "Allow",
"Action": "s3:*Object",
"Resource": ["arn:aws:s3:::bucket-name/*"]
}
]
}
```
For details on giving an application access to S3, refer to the AWS documentation on [Using an IAM Role to Grant Permissions to Applications Running on Amazon EC2 Instances.](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html)
{{% /tab %}}
{{% tab "RKE prior to v0.2.0" %}}
To save a snapshot of etcd from each etcd node in the cluster config file, run the `rke etcd snapshot-save` command.
When running the command, an additional container is created to take the snapshot. When the snapshot is completed, the container is automatically removed.
RKE saves a backup of the certificates, i.e. a file named `pki.bundle.tar.gz`, in the same location. The snapshot and pki bundle file are required for the restore process.
To create a local one-time snapshot, run:
```
$ rke etcd snapshot-save --config cluster.yml --name snapshot-name
```
**Result:** The snapshot is saved in `/opt/rke/etcd-snapshots`.
### Options for `rke etcd snapshot-save`
| Option | Description |
| --- | --- |
| `--name` value | Specify snapshot name |
| `--config` value | Specify an alternate cluster YAML file (default: `cluster.yml`) [$RKE_CONFIG] |
| `--ssh-agent-auth` | [Use SSH Agent Auth defined by SSH_AUTH_SOCK]({{< baseurl >}}/rke/latest/en/config-options/#ssh-agent) |
| `--ignore-docker-version` | [Disable Docker version check]({{< baseurl >}}/rke/latest/en/config-options/#supported-docker-versions) |
{{% /tab %}}
{{% /tabs %}}
content/rke/latest/en/etcd-snapshots/recurring-snapshots/_index.md
+127
View File
@@ -0,0 +1,127 @@
---
title: Recurring Snapshots
weight: 2
---
Recurring snapshots are handled differently based on your version of RKE.
{{% tabs %}}
{{% tab "RKE v0.2.0+"%}}
To schedule automatic recurring etcd snapshots, you can enable the `etcd-snapshot` service with [extra configuration options](#options-for-the-etcd-snapshot-service). `etcd-snapshot` runs in a service container alongside the `etcd` container. By default, the `etcd-snapshot` service takes a snapshot for every node that has the `etcd` role and stores them to local disk in `/opt/rke/etcd-snapshots`.
If you set up the [options for S3](#options-for-the-etcd-snapshot-service), the snapshot will also be uploaded to the S3 backend.
### Snapshot Service Logging
When a cluster is launched with the `etcd-snapshot` service enabled, you can view the `etcd-rolling-snapshots` logs to confirm backups are being created automatically.
```
$ docker logs etcd-rolling-snapshots
time="2018-05-04T18:39:16Z" level=info msg="Initializing Rolling Backups" creation=1m0s retention=24h0m0s
time="2018-05-04T18:40:16Z" level=info msg="Created backup" name="2018-05-04T18:40:16Z_etcd" runtime=108.332814ms
time="2018-05-04T18:41:16Z" level=info msg="Created backup" name="2018-05-04T18:41:16Z_etcd" runtime=92.880112ms
time="2018-05-04T18:42:16Z" level=info msg="Created backup" name="2018-05-04T18:42:16Z_etcd" runtime=83.67642ms
time="2018-05-04T18:43:16Z" level=info msg="Created backup" name="2018-05-04T18:43:16Z_etcd" runtime=86.298499ms
```
### Options for the `Etcd-Snapshot` Service
|Option|Description| S3 Specific |
|---|---| --- |
|**interval_hours**| The duration in hours between recurring backups. This supercedes the `creation` option (which was used in RKE prior to v0.2.0) and will override it if both are specified.| |
|**retention**| The number of snapshots to retain before rotation. This supercedes the `retention` option and will override it if both are specified.| |
|**bucket_name**| S3 bucket name where backups will be stored| * |
|**access_key**| S3 access key with permission to access the backup bucket.| * |
|**secret_key** |S3 secret key with permission to access the backup bucket.| * |
|**region** |S3 region for the backup bucket. This is optional.| * |
|**endpoint** |S3 regions endpoint for the backup bucket.| * |
The `--access-key` and `--secret-key` options are not required if the `etcd` nodes are AWS EC2 instances that have been configured with a suitable IAM instance profile.
### IAM Support for Storing Snapshots in S3
In addition to API access keys, RKE supports using IAM roles for S3 authentication. The cluster etcd nodes must be assigned an IAM role that has read/write access to the designated backup bucket on S3. Also, the nodes must have network access to the S3 endpoint specified.
Below is an [example IAM policy](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_examples_s3_rw-bucket.html) that would allow nodes to store and retrieve backups from S3:
```
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "ListObjectsInBucket",
"Effect": "Allow",
"Action": ["s3:ListBucket"],
"Resource": ["arn:aws:s3:::bucket-name"]
},
{
"Sid": "AllObjectActions",
"Effect": "Allow",
"Action": "s3:*Object",
"Resource": ["arn:aws:s3:::bucket-name/*"]
}
]
}
```
For details on giving an application access to S3, refer to the AWS documentation on [Using an IAM Role to Grant Permissions to Applications Running on Amazon EC2 Instances.](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html)
### Configuring the Snapshot Service in YAML
```yaml
services:
etcd:
backup_config:
interval_hours: 12
retention: 6
s3backupconfig:
access_key: S3_ACCESS_KEY
secret_key: S3_SECRET_KEY
bucket_name: s3-bucket-name
region: ""
endpoint: s3.amazonaws.com
```
{{% /tab %}}
{{% tab "RKE prior to v0.2.0"%}}
To schedule automatic recurring etcd snapshots, you can enable the `etcd-snapshot` service with [extra configuration options](#options-for-the-local-etcd-snapshot-service). `etcd-snapshot` runs in a service container alongside the `etcd` container. By default, the `etcd-snapshot` service takes a snapshot for every node that has the `etcd` role and stores them to local disk in `/opt/rke/etcd-snapshots`.
RKE saves a backup of the certificates, i.e. a file named `pki.bundle.tar.gz`, in the same location. The snapshot and pki bundle file are required for the restore process in versions prior to v0.2.0.
### Snapshot Service Logging
When a cluster is launched with the `etcd-snapshot` service enabled, you can view the `etcd-rolling-snapshots` logs to confirm backups are being created automatically.
```
$ docker logs etcd-rolling-snapshots
time="2018-05-04T18:39:16Z" level=info msg="Initializing Rolling Backups" creation=1m0s retention=24h0m0s
time="2018-05-04T18:40:16Z" level=info msg="Created backup" name="2018-05-04T18:40:16Z_etcd" runtime=108.332814ms
time="2018-05-04T18:41:16Z" level=info msg="Created backup" name="2018-05-04T18:41:16Z_etcd" runtime=92.880112ms
time="2018-05-04T18:42:16Z" level=info msg="Created backup" name="2018-05-04T18:42:16Z_etcd" runtime=83.67642ms
time="2018-05-04T18:43:16Z" level=info msg="Created backup" name="2018-05-04T18:43:16Z_etcd" runtime=86.298499ms
```
### Options for the Local `Etcd-Snapshot` Service
|Option|Description|
|---|---|
|**Snapshot**|By default, the recurring snapshot service is disabled. To enable the service, you need to define it as part of `etcd` and set it to `true`.|
|**Creation**|By default, the snapshot service will take snapshots every 5 minutes (`5m0s`). You can change the time between snapshots as part of the `creation` directive for the `etcd` service.|
|**Retention**|By default, all snapshots are saved for 24 hours (`24h`) before being deleted and purged. You can change how long to store a snapshot as part of the `retention` directive for the `etcd` service.|
### Configuring the Snapshot Service in YAML
```yaml
services:
etcd:
snapshot: true
creation: 5m0s
retention: 24h
```
{{% /tab %}}
{{% /tabs %}}
content/rke/latest/en/etcd-snapshots/restoring-from-backup/_index.md
+112
View File
@@ -0,0 +1,112 @@
---
title: Restoring from Backup
weight: 3
---
The details of restoring your cluster from backup are different depending on your version of RKE.
{{% tabs %}}
{{% tab "RKE v0.2.0+"%}}
If there is a disaster with your Kubernetes cluster, you can use `rke etcd snapshot-restore` to recover your etcd. This command reverts the etcd to a specific snapshot. The following actions are included in the command:
- Syncs the snapshot or downloads the snapshot from S3, if necessary.
- Checks snapshot checksum across etcd nodes to make sure they are identical.
- Deletes your current cluster and cleans old data by running `rke remove`. This removes the entire Kubernetes cluster, not just the etcd cluster.
- Rebuilds the etcd cluster from the chosen snapshot.
- Creates a new cluster by running `rke up`.
- Restarts cluster system pods.
>**Warning:** You should back up any important data in your cluster before running `rke etcd snapshot-restore` because the command deletes your current Kubernetes cluster and replaces it with a new one.
The snapshot used to restore your etcd cluster can either be stored locally in `/opt/rke/etcd-snapshots` or from a S3 compatible backend.
### Example of Restoring from a Local Snapshot
To restore etcd from a local snapshot, run:
```
$ rke etcd snapshot-restore --config cluster.yml --name mysnapshot
```
The snapshot is assumed to be located in `/opt/rke/etcd-snapshots`.
**Note:** The `pki.bundle.tar.gz` file is not needed because RKE v0.2.0 changed how the [Kubernetes cluster state is stored]({{< baseurl >}}/rke/latest/en/installation/#kubernetes-cluster-state).
### Example of Restoring from a Snapshot in S3
> **Prerequisite:** Ensure your `cluster.rkestate` is present before starting the restore, because this contains your certificate data for the cluster.
When restoring etcd from a snapshot located in S3, the command needs the S3 information in order to connect to the S3 backend and retrieve the snapshot.
```shell
$ rke etcd snapshot-restore \
--config cluster.yml \
--name snapshot-name \
--s3 \
--access-key S3_ACCESS_KEY \
--secret-key S3_SECRET_KEY \
--bucket-name s3-bucket-name \
--s3-endpoint s3.amazonaws.com
```
**Note:** if you were restoring a cluster that had Rancher installed, the Rancher UI should start up after a few minutes; you don't need to re-run Helm.
### Options for `rke etcd snapshot-restore`
| Option | Description | S3 Specific |
| --- | --- | ---|
| `--name` value | Specify snapshot name | |
| `--config` value | Specify an alternate cluster YAML file (default: `cluster.yml`) [$RKE_CONFIG] | |
| `--s3` | Enabled backup to s3 |* |
| `--s3-endpoint` value | Specify s3 endpoint url (default: "s3.amazonaws.com") | * |
| `--access-key` value | Specify s3 accessKey | *|
| `--secret-key` value | Specify s3 secretKey | *|
| `--bucket-name` value | Specify s3 bucket name | *|
| `--region` value | Specify the s3 bucket location (optional) | *|
| `--ssh-agent-auth` | [Use SSH Agent Auth defined by SSH_AUTH_SOCK]({{< baseurl >}}/rke/latest/en/config-options/#ssh-agent) | |
| `--ignore-docker-version` | [Disable Docker version check]({{< baseurl >}}/rke/latest/en/config-options/#supported-docker-versions) |
{{% /tab %}}
{{% tab "RKE prior to v0.2.0"%}}
If there is a disaster with your Kubernetes cluster, you can use `rke etcd snapshot-restore` to recover your etcd. This command reverts etcd to a specific snapshot.
The following actions are included in `rke etcd snapshot-restore`:
- Removes the old etcd cluster
- Rebuilds the etcd cluster using the local snapshot
Before you run this command, you must:
- Run `rke remove` to remove your Kubernetes cluster and clean the nodes
- Download your etcd snapshot from S3, if applicable. Place the etcd snapshot and the `pki.bundle.tar.gz` file in `/opt/rke/etcd-snapshots`. Manually sync the snapshot across all `etcd` nodes.
After the restore, you must rebuild your Kubernetes cluster with `rke up`.
>**Warning:** You should back up any important data in your cluster before running `rke etcd snapshot-restore` because the command deletes your current etcd cluster and replaces it with a new one.
### Example of Restoring from a Local Snapshot
To restore etcd from a local snapshot, run:
```
$ rke etcd snapshot-restore --config cluster.yml --name mysnapshot
```
The snapshot is assumed to be located in `/opt/rke/etcd-snapshots`.
The snapshot must be manually synched across all `etcd` nodes.
The `pki.bundle.tar.gz` file is also expected to be in the same location.
### Options for `rke etcd snapshot-restore`
| Option | Description |
| --- | --- |
| `--name` value | Specify snapshot name |
| `--config` value | Specify an alternate cluster YAML file (default: `cluster.yml`) [$RKE_CONFIG] |
| `--ssh-agent-auth` | [Use SSH Agent Auth defined by SSH_AUTH_SOCK]({{< baseurl >}}/rke/latest/en/config-options/#ssh-agent) |
| `--ignore-docker-version` | [Disable Docker version check]({{< baseurl >}}/rke/latest/en/config-options/#supported-docker-versions) |
{{% /tab %}}
{{% /tabs %}}
content/rke/latest/en/etcd-snapshots/troubleshooting/_index.md
+22
View File
@@ -0,0 +1,22 @@
---
title: Troubleshooting
weight: 5
---
As of **v0.1.9**, the **rke-bundle-cert** container is removed on both success and failure of a restore. To debug any issues, you will need to look at the **logs** generated from rke.
As of **v0.1.8** and below, the **rke-bundle-cert** container is left over from a failed etcd restore. If you are having an issue with restoring an **etcd snapshot** then you can do the following on each etcd nodes before attempting to do another restore:
```
docker container rm --force rke-bundle-cert
```
The rke-bundle-cert container is usually removed when a backup or restore of **etcd** succeeds. Whenever something goes wrong, the **rke-bundle-cert** container will be left over. You can look
at the logs or inspect the container to see what the issue is.
```
docker container logs --follow rke-bundle-cert
docker container inspect rke-bundle-cert
```
The important thing to note is the mounts of the container and location of the `pki.bundle.tar.gz`.
content/rke/latest/en/os/_index.md
+3 -3
View File
@@ -50,9 +50,9 @@ RKE runs on almost any Linux OS with Docker installed. Most of the development a
for module in br_netfilter ip6_udp_tunnel ip_set ip_set_hash_ip ip_set_hash_net iptable_filter iptable_nat iptable_mangle iptable_raw nf_conntrack_netlink nf_conntrack nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat nf_nat_ipv4 nf_nat_masquerade_ipv4 nfnetlink udp_tunnel veth vxlan x_tables xt_addrtype xt_conntrack xt_comment xt_mark xt_multiport xt_nat xt_recent xt_set xt_statistic xt_tcpudp;
do
if ! lsmod | grep -q $module; then
echo "module $module is not present"
fi
done;
echo "module $module is not present";
fi;
done
```
Module name |
static/img/os/RancherOS_aliyun1.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 89 KiB

static/img/os/RancherOS_aliyun2.jpg
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 82 KiB