From d4e3eb0874a1069ed6966743c48debab6aab7db7 Mon Sep 17 00:00:00 2001 From: catherineluse Date: Fri, 16 Oct 2020 17:19:39 -0700 Subject: [PATCH 01/49] Document RancherD installation --- .../install-rancher-on-linux/_index.md | 208 ++++++++++++ .../rancherd-configuration/_index.md | 316 ++++++++++++++++++ .../upgrades/_index.md | 70 ++++ .../air-gap/_index.md | 4 +- .../en/installation/requirements/_index.md | 10 + .../v2.x/en/installation/resources/_index.md | 2 +- 6 files changed, 607 insertions(+), 3 deletions(-) create mode 100644 content/rancher/v2.x/en/installation/install-rancher-on-linux/_index.md create mode 100644 content/rancher/v2.x/en/installation/install-rancher-on-linux/rancherd-configuration/_index.md create mode 100644 content/rancher/v2.x/en/installation/install-rancher-on-linux/upgrades/_index.md diff --git a/content/rancher/v2.x/en/installation/install-rancher-on-linux/_index.md b/content/rancher/v2.x/en/installation/install-rancher-on-linux/_index.md new file mode 100644 index 00000000000..63e85edb220 --- /dev/null +++ b/content/rancher/v2.x/en/installation/install-rancher-on-linux/_index.md @@ -0,0 +1,208 @@ +--- +title: Install Rancher on a Linux OS +weight: 2 +--- + +_Available as of Rancher v2.5_ + +As part of Rancher 2.5, we are excited to introduce a new, simpler way to install Rancher called RancherD. + +RancherD is a single binary that first launches an RKE2 Kubernetes cluster, then installs the Rancher server Helm chart on the cluster. + +- [About RancherD Installs](#about-rancherd-installs) +- [Prerequisites](#prerequisites) +- [Part I: Installing Rancher](#part-i-installing-rancher) +- [Part II: High Availability](#part-ii-high-availability) +- [Upgrades](#upgrades) +- [Configuration](#configuration) +- [RKE2 Documentation](#rke2-documentation) + +# About RancherD Installs + +When RancherD is launched on a host, it first installs an RKE2 Kubernetes cluster, then deploys Rancher on the cluster as a Kubernetes daemonset. + +In both the RancherD install and the Helm CLI install, Rancher is installed as a Helm chart on a Kubernetes cluster. A highly available RancherD installation is equally suitable for production compared to a highly available installation using the Helm CLI. + +Configuration and upgrading are also simplified with RancherD. When you upgrade the RancherD binary, both the Kubernetes cluster and the Rancher Helm chart are upgraded. + +In Part I of these instructions, you'll learn how to launch RancherD on a single node. The result of following the steps in Part I is a single-node [RKE2](https://docs.rke2.io/) Kubernetes cluster with the Rancher server installed. This cluster can easily become high availability later. If Rancher only needs to manage the local Kubernetes cluster, the installation is complete. + +Part II explains how to convert the single-node Rancher installation into a high-availability installation. If the Rancher server will manage downstream Kubernetes clusters, it is important to follow these steps. A discussion of recommended architecture for highly available Rancher deployments can be found in our [Best Practices Guide.]({{}}/rancher/v2.x/en/best-practices/v2.5/rancher-server) + +# Prerequisites + +### Node Requirements + +RancherD must be launched on a Linux OS. At this time, only OSes that leverage systemd are supported. + +The Linux node needs to fulfill the [installation requirements]({{}}/rancher/v2.x/en/installation/requirements) for hardware and networking. + +### Root Access + +Before running the installation commands, you will need to log in as root: + +``` +sudo -s +``` + +### Fixed Registration Address + +A fixed registration address is recommended for single-node installs and required for high-availability installs with RancherD. + +The fixed registration address is an endpoint that is used for two purposes: + +- To access the Kubernetes API. So you can, for example, modify your [kubeconfig](https://kubernetes.io/docs/concepts/configuration/organize-cluster-access-kubeconfig/) file to point to it instead of a specific node. +- To add new nodes to the Kubernetes cluster. To add nodes to the cluster later, you will run a command on the node that will specify the fixed registration address of the cluster. + +If you are installing Rancher on a single node, the fixed registration address makes it possible to add more nodes to the cluster so that you can convert the single-node install to a high-availability install without causing downtime to the cluster. If you don't set up this address when installing the single-node Kubernetes cluster, you would need to re-run the installation script with a fixed registration address in order to add new nodes to the cluster. + +The fixed registration can be the IP or hostname of any of the server nodes, but in many cases those may change over time as nodes are created and destroyed. Therefore, you should have a stable endpoint in front of the server nodes. + +This endpoint can be set up using any number of approaches, such as: + +* A layer 4 (TCP) load balancer +* Round-robin DNS +* Virtual or elastic IP addresses + +Note that the RancherD server process listens on port 9345 for new nodes to register. The Kubernetes API is served on port 6443, as normal. Configure your load balancer accordingly. + +# Part I: Installing Rancher + +### 1. Set up the config.yaml + +To avoid certificate errors with the fixed registration address, you should launch the server with the `tls-san` parameter set. This parameter should refer to your fixed registration address. + +This option adds an additional hostname or IP as a Subject Alternative Name in the server's TLS cert, and it can be specified as a list if you would like to access the Kubernetes cluster via both the IP and the hostname. + +Create the RancherD config file at `/etc/rancher/rancherd/config.yaml`: + +```yaml +token: my-shared-secret +tls-san: + - my-fixed-registration-address.com + - another-kubernetes-domain.com +``` + +The first server node establishes the secret token that other nodes would register with if they are added to the cluster. + +If you do not specify a pre-shared secret, RancherD will generate one and place it at `/var/lib/rancher/rancherd/server/node-token`. + +To specify your own pre-shared secret as the token, set the `token` argument on startup. + +### 2. Launch the first server node + +Run the RancherD installer: + +``` +curl -sfL https://get.rancher.io | sh - +``` + +Once installed, the `rancherd` binary will be on your PATH. You can check out its help text like this: + +``` +rancherd --help +NAME: + rancherd - Rancher Kubernetes Engine 2 +... +``` + +Next, launch RancherD: + +``` +systemctl enable rancherd-server.service +systemctl start rancherd-server.service +``` + +When RancherD launches, it installs an RKE2 Kubernetes cluster. Use the following command to see the logs of the Kubernetes cluster as it comes up: + +``` +journalctl -eu rancherd-server -f +``` + +### 3. Set up the kubeconfig file with kubectl + +Once the Kubernetes cluster is up, set up RancherD’s kubeconfig file and `kubectl`: + +``` +export KUBECONFIG=/etc/rancher/rke2/rke2.yaml PATH=$PATH:/var/lib/rancher/rke2/bin +``` + +### 4. Verify that Rancher is installed on the Kubernetes cluster + +Now, you can start issuing `kubectl` commands. Use the following commands to verify that Rancher is deployed as a daemonset on the cluster: + +``` +kubectl get daemonset rancher -n cattle-system + +kubectl get pod -n cattle-system +``` + +### 6. Set the initial Rancher password + +Once the `rancher` pod is up and running, run the following: + +``` +rancherd reset-admin +``` + +This will give you the URL, username and password needed to log into Rancher. Follow that URL, plug in the credentials, and you’re up and running with Rancher! + +If Rancher will only manage the local Kubernetes cluster, the installation is complete. + +# Part II: High Availability + +If you plan to use the Rancher server to manage downstream Kubernetes clusters, Rancher needs to be highly available. In these steps, you will add more nodes to achieve a high-availability cluster. Since Rancher is running as a daemonset, it will automatically launch on the nodes you add. + +An odd number of nodes is required because the etcd cluster, which contains the cluster data, needs a majority of live nodes to avoid losing quorum. A loss of quorum could require the cluster to be restored from backup. Therefore, we recommend using three nodes. + +When following these steps, you should still be logged in as root. + +### 1. Configure the fixed registration address on a new node + +Additional server nodes are launched much like the first, except that you must specify the `server` and `token` parameters so that they can successfully connect to the initial server node. + +Here is an example of what the RancherD config file would look like for additional server nodes. By default, this config file is expected to be located at `/etc/rancher/rancherd/config.yaml`. + +```yaml +server: https://my-fixed-registration-address.com:9345 +token: my-shared-secret +tls-san: + - my-fixed-registration-address.com + - another-kubernetes-domain.com +``` + +### 2. Launch an additional server node + +Run the installer on the new node: + +``` +curl -sfL https://get.rancher.io | sh - +``` + +This will download RancherD and install it as a systemd unit on your host. + + +Next, launch RancherD: + +``` +systemctl enable rancherd-server.service +systemctl start rancherd-server.service +``` + +### 3. Repeat + +Repeat steps one and two for another Linux node, bringing the number of nodes in the cluster to three. + +**Result:** Rancher is highly available and the installation is complete. + +# Upgrades + +For information on upgrades and rollbacks, refer to [this page.](./upgrades) + +# Configuration + +For information on how to configure certificates, node taints, Rancher Helm chart options, or RancherD CLI options, refer to the [configuration reference.](./rancherd-configuration) + +# RKE2 Documentation + +For more information on RKE2, the Kubernetes distribution used to provision the underlying cluster, refer to the documentation [here.](https://docs.rke2.io/) \ No newline at end of file diff --git a/content/rancher/v2.x/en/installation/install-rancher-on-linux/rancherd-configuration/_index.md b/content/rancher/v2.x/en/installation/install-rancher-on-linux/rancherd-configuration/_index.md new file mode 100644 index 00000000000..0e497eaa7cc --- /dev/null +++ b/content/rancher/v2.x/en/installation/install-rancher-on-linux/rancherd-configuration/_index.md @@ -0,0 +1,316 @@ +--- +title: RancherD Configuration Reference +weight: 1 +--- + +In RancherD, a server node is defined as a machine (bare-metal or virtual) running the `rancherd server` command. The server runs the Kubernetes API as well as Kubernetes workloads. + +An agent node is defined as a machine running the `rancherd agent` command. They don't run the Kubernetes API. To add nodes designated to run your apps and services, join agent nodes to your cluster. + +In the RancherD installation instructions, we recommend running three server nodes in the Rancher server cluster. Agent nodes are not required. + +- [Certificates for the Rancher Server](#certificates-for-the-rancher-server) +- [Node Taints](#node-taints) +- [Customizing the RancherD Helm Chart](#customizing-the-rancherd-helm-chart) +- [RancherD Server CLI Options](#rancherd-server-cli-options) +- [RancherD Agent CLI Options](#rancherd-agent-cli-options) + +# Certificates for the Rancher Server + +Rancherd does not use cert-manger to provision certs. Instead RancherD allows you to bring your own self-signed or trusted certs by storing the .pem files in `/etc/rancher/ssl/`. When doing this you should also set the `publicCA` parameter to `true` in your HelmChartConfig. + +Private key: `/etc/rancher/ssl/key.pem` + +Certificate: `/etc/rancher/ssl/cert.pem` + +CA Certificate(self-signed): `/etc/rancher/ssl/cacerts.pem` + +Additional CA Certificate: `/etc/ssl/certs/ca-additional.pem` + +# Node Taints + +By default, server nodes will be schedulable and thus your workloads can get launched on them. If you wish to have a dedicated control plane where no user workloads will run, you can use taints. The node-taint parameter will allow you to configure nodes with taints. Here is an example of adding a node taint to the `config.yaml`: + +``` +node-taint: + - "CriticalAddonsOnly=true:NoExecute" +``` +# Customizing the RancherD Helm Chart + +Rancher is launched as a [Helm](https://helm.sh/) chart using the cluster’s [Helm integration.](https://docs.rke2.io/helm/) This means that you can easily customize the application through a manifest file describing your custom parameters. + +The RancherD chart provisions Rancher in a daemonset. It exposes hostport `8080/8443` down to the container port (`80/443`), and uses hostpath to mount certs if needed. + +RancherD uses `helm-controller` to bootstrap the RancherD chart. To provide a customized `values.yaml` file, the configuration options must be passed in through the `helm-controller` custom resource definition. + +Here is an example of the manifest: + +```yaml +apiVersion: helm.cattle.io/v1 +kind: HelmChartConfig +metadata: + name: rancher + namespace: kube-system +spec: + valuesContent: | + publicCA: true +``` + +Put this manifest on your host in `/var/lib/rancher/rancherd/server/manifests` before running RancherD. + +### Common Options + +| Parameter | Default Value | Description | +| ------------------------------ | ----------------------------------------------------- | -------------------------------------------- | +| `addLocal` | "auto" | ***string*** - Have Rancher detect and import the “local” Rancher server cluster [Import "local Cluster"](https://rancher.com/docs/rancher/v2.x/en/installation/options/chart-options/#import-local-cluster) | +| `auditLog.destination` | "sidecar" | ***string*** - Stream to sidecar container console or hostPath volume - *"sidecar, hostPath"* | +| `auditLog.hostPath` | "/var/log/rancher/audit" | ***string*** - log file destination on host (only applies when **auditLog.destination** is set to **hostPath**) | +| `auditLog.level` | 0 | ***int*** - set the [API Audit Log level](https://rancher.com/docs/rancher/v2.x/en/installation/api-auditing). 0 is off. [0-3] | +| `auditLog.maxAge` | 1 | ***int*** - maximum number of days to retain old audit log files (only applies when **auditLog.destination** is set to **hostPath**) | +| `auditLog.maxBackups` | 1 | int - maximum number of audit log files to retain (only applies when **auditLog.destination** is set to **hostPath**) | +| `auditLog.maxSize` | 100 | ***int*** - maximum size in megabytes of the audit log file before it gets rotated (only applies when **auditLog.destination** is set to **hostPath**) | +| `debug` | false | ***bool*** - set debug flag on rancher server | +| `extraEnv` | [] | ***list*** - set additional environment variables for Rancher Note: *Available as of v2.2.0* | +| `imagePullSecrets` | [] | ***list*** - list of names of Secret resource containing private registry credentials | +| `proxy` | " " | ***string** - HTTP[S] proxy server for Rancher | +| `noProxy` | "127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16" | ***string*** - comma separated list of hostnames or ip address not to use the proxy | +| `resources` | {} | ***map*** - rancher pod resource requests & limits | +| `rancherImage` | "rancher/rancher" | ***string*** - rancher image source | +| `rancherImageTag` | same as chart version | ***string*** - rancher/rancher image tag | +| `rancherImagePullPolicy` | "IfNotPresent" | ***string*** - Override imagePullPolicy for rancher server images - *"Always", "Never", "IfNotPresent"* | +| `systemDefaultRegistry` | "" | ***string*** - private registry to be used for all system Docker images, e.g., [http://registry.example.com/] *Available as of v2.3.0* | +| `useBundledSystemChart` | false | ***bool*** - select to use the system-charts packaged with Rancher server. This option is used for air gapped installations. *Available as of v2.3.0* | +| `publicCA` | false | ***bool*** - Set to true if your cert is signed by a public CA | + +# RancherD Server CLI Options + +The command to run the Rancher management server is: + +``` +rancherd server [OPTIONS] +``` + +It can be run with the following options: + +### Config + +| Option | Description | +|--------|-------------| +| `--config FILE, -c FILE` | Load configuration from FILE (default: "/etc/rancher/rancherd/config.yaml") | + +### Logging + +| Option | Description | +|--------|-------------| +| `--debug` | Turn on debug logs | + +### Listener + +| Option | Description | +|--------|-------------| +| `--bind-address value` | RancherD bind address (default: 0.0.0.0) | +| `--advertise-address value` | IP address that apiserver uses to advertise to members of the cluster (default: node-external-ip/node-ip) | +| `--tls-san value` | Add additional hostname or IP as a Subject Alternative Name in the TLS cert | + +### Data + +| Option | Description | +|--------|-------------| +| `--data-dir value, -d value` | Folder to hold state (default: "/var/lib/rancher/rancherd") | + +### Networking + +| Option | Description | +|--------|-------------| +| `--cluster-cidr value` | Network CIDR to use for pod IPs (default: "10.42.0.0/16") | +| `--service-cidr value` | Network CIDR to use for services IPs (default: "10.43.0.0/16") | +| `--cluster-dns value` | Cluster IP for coredns service. Should be in your service-cidr range (default: 10.43.0.10) | +| `--cluster-domain value` | Cluster Domain (default: "cluster.local") | + +### Cluster + +| Option | Description | +|--------|-------------| +| `--token value, -t value` | Shared secret used to join a server or agent to a cluster | +| `--token-file value` | File containing the cluster-secret/token | + +### Client + +| Option | Description | +|--------|-------------| +| `--write-kubeconfig value, -o value` | Write kubeconfig for admin client to this file | +| `--write-kubeconfig-mode value` | Write kubeconfig with this mode | + +### Flags + +| Option | Description | +|--------|-------------| +| `--kube-apiserver-arg value` | Customized flag for kube-apiserver process | +| `--kube-scheduler-arg value` | Customized flag for kube-scheduler process | +| `--kube-controller-manager-arg value` | Customized flag for kube-controller-manager process | + +### Database + +| Option | Description | +|--------|-------------| +| `--etcd-disable-snapshots` | Disable automatic etcd snapshots | +| `--etcd-snapshot-schedule-cron value` | Snapshot interval time in cron spec. eg. every 5 hours '* */5 * * *' (default: "0 */12 * * *") | +| `--etcd-snapshot-retention value` | Number of snapshots to retain (default: 5) | +| `--etcd-snapshot-dir value` | Directory to save db snapshots. (Default location: ${data-dir}/db/snapshots) | +| `--cluster-reset-restore-path value` | Path to snapshot file to be restored | + +### System Images Registry + +| Option | Description | +|--------|-------------| +| `--system-default-registry value` | Private registry to be used for all system Docker images | + +### Components + +| Option | Description | +|--------|-------------| +| `--disable value` | Do not deploy packaged components and delete any deployed components (valid items: rancherd-canal, rancherd-coredns, rancherd-ingress, rancherd-kube-proxy, rancherd-metrics-server) | + +### Cloud Provider + +| Option | Description | +|--------|-------------| +| `--cloud-provider-name value` | Cloud provider name | +| `--cloud-provider-config value` | Cloud provider configuration file path | + +### Security + +| Option | Description | +|--------|-------------| +| `--profile value` | Validate system configuration against the selected benchmark (valid items: cis-1.5) | + +### Agent Node + +| Option | Description | +|--------|-------------| +| `--node-name value` | Node name | +| `--node-label value` | Registering and starting kubelet with set of labels | +| `--node-taint value` | Registering kubelet with set of taints | +| `--protect-kernel-defaults` | Kernel tuning behavior. If set, error if kernel tunables are different than kubelet defaults. | +| `--selinux` | Enable SELinux in containerd | + +### Agent Runtime + +| Option | Description | +|--------|-------------| +| `--container-runtime-endpoint value` | Disable embedded containerd and use alternative CRI implementation | +| `--snapshotter value` | Override default containerd snapshotter (default: "overlayfs") | +| `--private-registry value` | Private registry configuration file (default: "/etc/rancher/rancherd/registries.yaml") | + +### Agent Networking + +| Option | Description | +|--------|-------------| +| `--node-ip value, -i value` | IP address to advertise for node | +| `--resolv-conf value` | Kubelet resolv.conf file | + +### Agent Flags + +| Option | Description | +|--------|-------------| +| `--kubelet-arg value` | Customized flag for kubelet process | +| `--kube-proxy-arg value` | Customized flag for kube-proxy process | + +### Experimental + +| Option | Description | +|--------|-------------| +| `--agent-token value` | Shared secret used to join agents to the cluster, but not servers | +| `--agent-token-file value` | File containing the agent secret | +| `--server value, -s value` | Server to connect to, used to join a cluster | +| `--cluster-reset` | Forget all peers and become sole member of a new cluster | +| `--secrets-encryption` | Enable Secret encryption at rest | + + + +# RancherD Agent CLI Options + +The following command is used to run the RancherD agent: + +``` +rancherd agent [OPTIONS] +``` + +The following options are available. + +### Config + +| Option | Description | +|--------|-------------| +| `--config FILE, -c FILE` | Load configuration from FILE (default: "/etc/rancher/rancherd/config.yaml") | + +### Data + +| Option | Description | +|--------|-------------| +| `--data-dir value, -d value` | Folder to hold state (default: "/var/lib/rancher/rancherd") | + +### Logging + +| Option | Description | +|--------|-------------| +| `--debug` | Turn on debug logs | + +### Cluster + +| Option | Description | +|--------|-------------| +| `--token value, -t value` | Token to use for authentication | +| `--token-file value` | Token file to use for authentication | +| `--server value, -s value` | Server to connect to | + +### Agent Node + +| Option | Description | +|--------|-------------| +| `--node-name value` | Node name | +| `--node-label value` | Registering and starting kubelet with set of labels | +| `--node-taint value` | Registering kubelet with set of taints | +| `--selinux` | Enable SELinux in containerd | +| `--protect-kernel-defaults` | Kernel tuning behavior. If set, error if kernel tunables are different than kubelet defaults. | + +### Agent Runtime + +| Option | Description | +|--------|-------------| +| `--container-runtime-endpoint value` | Disable embedded containerd and use alternative CRI implementation | +| `--snapshotter value` | Override default containerd snapshotter (default: "overlayfs") | +| `--private-registry value` | Private registry configuration file (default: "/etc/rancher/rancherd/registries.yaml") | + +### Agent Networking + +| Option | Description | +|--------|-------------| +| `--node-ip value, -i value` | IP address to advertise for node | +| `--resolv-conf value` | Kubelet resolv.conf file | + +### Agent Flags + +| Option | Description | +|--------|-------------| +| `--kubelet-arg value` | Customized flag for kubelet process | +| `--kube-proxy-arg value` | Customized flag for kube-proxy process | + +### System Images Registry + +| Option | Description | +|--------|-------------| +| `--system-default-registry value` | Private registry to be used for all system Docker images | + +### Cloud Provider + +| Option | Description | +|--------|-------------| +| `--cloud-provider-name value` | Cloud provider name | +| `--cloud-provider-config value` | Cloud provider configuration file path | + +### Security + +| Option | Description | +|--------|-------------| +| `--profile value` | Validate system configuration against the selected benchmark (valid items: cis-1.5) | \ No newline at end of file diff --git a/content/rancher/v2.x/en/installation/install-rancher-on-linux/upgrades/_index.md b/content/rancher/v2.x/en/installation/install-rancher-on-linux/upgrades/_index.md new file mode 100644 index 00000000000..ec55413d585 --- /dev/null +++ b/content/rancher/v2.x/en/installation/install-rancher-on-linux/upgrades/_index.md @@ -0,0 +1,70 @@ +--- +title: Upgrades and Rollbacks +weight: +--- + +When RancherD is upgraded, the Rancher Helm controller and the Fleet pods are upgraded. + +During a RancherD upgrade, there is very little downtime, but it is possible that RKE2 may be down for a minute, during which you could lose access to Rancher. + +When Rancher is installed with RancherD, the underlying Kubernetes cluster can't be upgraded from the Rancher UI. It needs to be upgraded using the RancherD CLI. + +### Upgrading the Rancher Helm Chart without Upgrading the Underlying Cluster + +To upgrade Rancher without upgrading the underlying Kubernetes cluster, follow these steps. + +> Before upgrading, we recommend that you should: +> +> - Create a backup of the Rancher server using the [backup application.]({{}}/rancher/v2.x/en/backups/v2.5/back-up-rancher/) +> - Review the known issues for the Rancher version you are upgrading to. The known issues are listed in the release notes on [GitHub](https://github.com/rancher/rancher/releases) and on the [Rancher forums.](https://forums.rancher.com/c/announcements/12) + +1. Uninstall the chart with Helm: + + ``` + helm uninstall rancher + ``` + +2. Reinstall the Rancher chart with Helm. To install a specific Rancher version, use the `--version` flag. For example: + + ``` + helm install rancher rancher-latest/rancher \ + --namespace cattle-system \ + --set hostname=rancher.my.org \ + --version 2.5.1 + ``` + +**Result:** Rancher is upgraded to the new version. + +If necessary, restore Rancher from backup by following [these steps.]({{}}/rancher/v2.x/en/backups/v2.5/restoring-rancher/) + +To roll back Rancher to a previous version, uninstall the Helm chart and reinstall the Helm chart for the previous version. + +### Upgrading Both Rancher and the Underlying Cluster + +Upgrade both RancherD and the underlying Kubernetes cluster by re-running the RancherD installation script. + +> Before upgrading, we recommend that you should: +> +> - Create a backup of the Rancher server using the [backup application.]({{}}/rancher/v2.x/en/backups/v2.5/back-up-rancher/) +> - Review the known issues for the Rancher version you are upgrading to. The known issues are listed in the release notes on [GitHub](https://github.com/rancher/rancher/releases) and on the [Rancher forums.](https://forums.rancher.com/c/announcements/12) + +``` +sudo curl -sfL https://get.rancher.io | sudo sh - +``` + +To specify a specific version to upgrade to, use `INSTALL_RANCHERD_VERSION` environment variable: + +``` +curl -sfL https://get.rancher.io | INSTALL_RANCHERD_VERSION=v2.5.1 sh - +``` + +Then launch the server: + +``` +systemctl enable rancherd-server +systemctl start rancherd-server +``` + +The upgrade can also be performed by manually installing the binary of the desired version. + +To roll back Rancher to a previous version, re-run the installation script with the previous version specified in the `INSTALL_RANCHERD_VERSION` environment variable. diff --git a/content/rancher/v2.x/en/installation/other-installation-methods/air-gap/_index.md b/content/rancher/v2.x/en/installation/other-installation-methods/air-gap/_index.md index 2c4d94963b6..24c413fff8e 100644 --- a/content/rancher/v2.x/en/installation/other-installation-methods/air-gap/_index.md +++ b/content/rancher/v2.x/en/installation/other-installation-methods/air-gap/_index.md @@ -1,5 +1,5 @@ --- -title: Installing Rancher in an Air Gapped Environment +title: Air Gapped Helm CLI Install weight: 1 aliases: - /rancher/v2.x/en/installation/air-gap-installation/ @@ -7,7 +7,7 @@ aliases: - /rancher/v2.x/en/installation/air-gap-single-node/ --- -This section is about installations of Rancher server in an air gapped environment. An air gapped environment could be where Rancher server will be installed offline, behind a firewall, or behind a proxy. +This section is about using the Helm CLI to install the Rancher server in an air gapped environment. An air gapped environment could be where Rancher server will be installed offline, behind a firewall, or behind a proxy. The installation steps differ depending on whether Rancher is installed on an RKE Kubernetes cluster, a K3s Kubernetes cluster, or a single Docker container. diff --git a/content/rancher/v2.x/en/installation/requirements/_index.md b/content/rancher/v2.x/en/installation/requirements/_index.md index 673ca944376..c439940fc42 100644 --- a/content/rancher/v2.x/en/installation/requirements/_index.md +++ b/content/rancher/v2.x/en/installation/requirements/_index.md @@ -27,10 +27,18 @@ The Rancher UI works best in Firefox or Chrome. Rancher should work with any modern Linux distribution. +### Requirements for Installing Rancher with RancherD + +At this time, only Linux OSes that leverage systemd are supported. + +### Requirements for Installing Rancher on an RKE Kubernetes Cluster + For the container runtime, RKE should work with any modern Docker version, while K3s should work with any modern version of Docker or containerd. Rancher and RKE have been tested and are supported on Ubuntu, CentOS, Oracle Linux, RancherOS, and RedHat Enterprise Linux. +### Requirements for Installing Rancher on a K3s Kubernetes Cluster + K3s should run on just about any flavor of Linux. However, K3s is tested on the following operating systems and their subsequent non-major releases: - Ubuntu 16.04 (amd64) @@ -41,6 +49,8 @@ If you are installing Rancher on a K3s cluster with **Raspbian Buster**, follow If you are installing Rancher on a K3s cluster with Alpine Linux, follow [these steps]({{}}/k3s/latest/en/advanced/#additional-preparation-for-alpine-linux-setup) for additional setup. +### General Linux Requirements + For details on which OS and Docker versions were tested with each Rancher version, refer to the [support maintenance terms.](https://rancher.com/support-maintenance-terms/) All supported operating systems are 64-bit x86. diff --git a/content/rancher/v2.x/en/installation/resources/_index.md b/content/rancher/v2.x/en/installation/resources/_index.md index 2a29b27f342..4e3ebe85860 100644 --- a/content/rancher/v2.x/en/installation/resources/_index.md +++ b/content/rancher/v2.x/en/installation/resources/_index.md @@ -1,6 +1,6 @@ --- title: Resources -weight: 4 +weight: 5 aliases: - /rancher/v2.x/en/installation/options --- From 595eacabc1edc902d950b2036c89d578668e3f98 Mon Sep 17 00:00:00 2001 From: cluse Date: Fri, 6 Nov 2020 16:07:00 -0700 Subject: [PATCH 02/49] Add internal link to RancherD Helm chart config --- .../install-rancher-on-linux/rancherd-configuration/_index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/rancher/v2.x/en/installation/install-rancher-on-linux/rancherd-configuration/_index.md b/content/rancher/v2.x/en/installation/install-rancher-on-linux/rancherd-configuration/_index.md index 0e497eaa7cc..578d542954d 100644 --- a/content/rancher/v2.x/en/installation/install-rancher-on-linux/rancherd-configuration/_index.md +++ b/content/rancher/v2.x/en/installation/install-rancher-on-linux/rancherd-configuration/_index.md @@ -17,7 +17,7 @@ In the RancherD installation instructions, we recommend running three server nod # Certificates for the Rancher Server -Rancherd does not use cert-manger to provision certs. Instead RancherD allows you to bring your own self-signed or trusted certs by storing the .pem files in `/etc/rancher/ssl/`. When doing this you should also set the `publicCA` parameter to `true` in your HelmChartConfig. +Rancherd does not use cert-manger to provision certs. Instead RancherD allows you to bring your own self-signed or trusted certs by storing the .pem files in `/etc/rancher/ssl/`. When doing this you should also set the `publicCA` parameter to `true` in your HelmChartConfig. For more information on the HelmChartConfig, refer to the section about [customizing the RancherD Helm chart.](#customizing-the-rancherd-helm-chart) Private key: `/etc/rancher/ssl/key.pem` From 407302347468c21acb2322977470159f3f345ef3 Mon Sep 17 00:00:00 2001 From: cluse Date: Sun, 8 Nov 2020 17:29:37 -0700 Subject: [PATCH 03/49] Edit RancherD docs --- .../install-rancher-on-linux/_index.md | 18 +++++++++++++----- .../rancherd-configuration/_index.md | 8 ++++---- 2 files changed, 17 insertions(+), 9 deletions(-) diff --git a/content/rancher/v2.x/en/installation/install-rancher-on-linux/_index.md b/content/rancher/v2.x/en/installation/install-rancher-on-linux/_index.md index 63e85edb220..0ceaf5bb836 100644 --- a/content/rancher/v2.x/en/installation/install-rancher-on-linux/_index.md +++ b/content/rancher/v2.x/en/installation/install-rancher-on-linux/_index.md @@ -15,6 +15,7 @@ RancherD is a single binary that first launches an RKE2 Kubernetes cluster, then - [Part II: High Availability](#part-ii-high-availability) - [Upgrades](#upgrades) - [Configuration](#configuration) +- [Uninstall](#uninstall) - [RKE2 Documentation](#rke2-documentation) # About RancherD Installs @@ -74,12 +75,12 @@ To avoid certificate errors with the fixed registration address, you should laun This option adds an additional hostname or IP as a Subject Alternative Name in the server's TLS cert, and it can be specified as a list if you would like to access the Kubernetes cluster via both the IP and the hostname. -Create the RancherD config file at `/etc/rancher/rancherd/config.yaml`: +Create the RancherD config file at `/etc/rancher/rke2/config.yaml`: ```yaml token: my-shared-secret tls-san: - - my-fixed-registration-address.com + - https://my-fixed-registration-address.com - another-kubernetes-domain.com ``` @@ -133,11 +134,10 @@ Now, you can start issuing `kubectl` commands. Use the following commands to ver ``` kubectl get daemonset rancher -n cattle-system - kubectl get pod -n cattle-system ``` -### 6. Set the initial Rancher password +### 5. Set the initial Rancher password Once the `rancher` pod is up and running, run the following: @@ -161,7 +161,7 @@ When following these steps, you should still be logged in as root. Additional server nodes are launched much like the first, except that you must specify the `server` and `token` parameters so that they can successfully connect to the initial server node. -Here is an example of what the RancherD config file would look like for additional server nodes. By default, this config file is expected to be located at `/etc/rancher/rancherd/config.yaml`. +Here is an example of what the RancherD config file would look like for additional server nodes. By default, this config file is expected to be located at `/etc/rancher/rke2/config.yaml`. ```yaml server: https://my-fixed-registration-address.com:9345 @@ -203,6 +203,14 @@ For information on upgrades and rollbacks, refer to [this page.](./upgrades) For information on how to configure certificates, node taints, Rancher Helm chart options, or RancherD CLI options, refer to the [configuration reference.](./rancherd-configuration) +# Uninstall + +To uninstall RancherD from your system, run the command below. This will shut down the process, remove the RancherD binary, and clean up files used by RancherD. + +``` +rancherd-uninstall.sh +``` + # RKE2 Documentation For more information on RKE2, the Kubernetes distribution used to provision the underlying cluster, refer to the documentation [here.](https://docs.rke2.io/) \ No newline at end of file diff --git a/content/rancher/v2.x/en/installation/install-rancher-on-linux/rancherd-configuration/_index.md b/content/rancher/v2.x/en/installation/install-rancher-on-linux/rancherd-configuration/_index.md index 578d542954d..e8570e52777 100644 --- a/content/rancher/v2.x/en/installation/install-rancher-on-linux/rancherd-configuration/_index.md +++ b/content/rancher/v2.x/en/installation/install-rancher-on-linux/rancherd-configuration/_index.md @@ -96,7 +96,7 @@ It can be run with the following options: | Option | Description | |--------|-------------| -| `--config FILE, -c FILE` | Load configuration from FILE (default: "/etc/rancher/rancherd/config.yaml") | +| `--config FILE, -c FILE` | Load configuration from FILE (default: "/etc/rancher/rke2/config.yaml") | ### Logging @@ -200,7 +200,7 @@ It can be run with the following options: |--------|-------------| | `--container-runtime-endpoint value` | Disable embedded containerd and use alternative CRI implementation | | `--snapshotter value` | Override default containerd snapshotter (default: "overlayfs") | -| `--private-registry value` | Private registry configuration file (default: "/etc/rancher/rancherd/registries.yaml") | +| `--private-registry value` | Private registry configuration file (default: "/etc/rancher/rke2/registries.yaml") | ### Agent Networking @@ -242,7 +242,7 @@ The following options are available. | Option | Description | |--------|-------------| -| `--config FILE, -c FILE` | Load configuration from FILE (default: "/etc/rancher/rancherd/config.yaml") | +| `--config FILE, -c FILE` | Load configuration from FILE (default: "/etc/rancher/rke2/config.yaml") | ### Data @@ -280,7 +280,7 @@ The following options are available. |--------|-------------| | `--container-runtime-endpoint value` | Disable embedded containerd and use alternative CRI implementation | | `--snapshotter value` | Override default containerd snapshotter (default: "overlayfs") | -| `--private-registry value` | Private registry configuration file (default: "/etc/rancher/rancherd/registries.yaml") | +| `--private-registry value` | Private registry configuration file (default: "/etc/rancher/rke2/registries.yaml") | ### Agent Networking From 1f56d06020f7562c0b02a7432d6c8e298c68c0a9 Mon Sep 17 00:00:00 2001 From: Bastian Hofmann Date: Thu, 13 Aug 2020 16:07:00 +0200 Subject: [PATCH 04/49] Cleanup Rancher port requirements Port requirements were documented in 3 places in the Rancher docs. This consolidates the documentation in one place and correctly distinguishes between the Rancher management cluster and downstream clusters. Signed-off-by: Bastian Hofmann --- .../node-requirements/_index.md | 119 +----------- .../en/installation/requirements/_index.md | 174 +----------------- .../installation/requirements/ports/_index.md | 86 +++++++-- .../ports/common-ports-table/index.md | 20 ++ .../resources/advanced/firewall/_index.md | 2 +- layouts/shortcodes/ports-custom-nodes.html | 47 ++--- layouts/shortcodes/ports-iaas-nodes.html | 41 +---- layouts/shortcodes/ports-imported-hosted.html | 17 +- static/img/rancher/port-communications.svg | 2 +- 9 files changed, 116 insertions(+), 392 deletions(-) create mode 100644 content/rancher/v2.x/en/installation/requirements/ports/common-ports-table/index.md diff --git a/content/rancher/v2.x/en/cluster-provisioning/node-requirements/_index.md b/content/rancher/v2.x/en/cluster-provisioning/node-requirements/_index.md index 5521e62800c..cffa90400c6 100644 --- a/content/rancher/v2.x/en/cluster-provisioning/node-requirements/_index.md +++ b/content/rancher/v2.x/en/cluster-provisioning/node-requirements/_index.md @@ -58,127 +58,10 @@ The ports required to be open are different depending on how the user cluster is For a breakdown of the port requirements for etcd nodes, controlplane nodes, and worker nodes in a Kubernetes cluster, refer to the [port requirements for the Rancher Kubernetes Engine.]({{}}/rke/latest/en/os/#ports) -Details on which ports are used in each situation are found in the following sections: - -- [Commonly used ports](#commonly-used-ports) -- [Port requirements for custom clusters](#port-requirements-for-custom-clusters) -- [Port requirements for clusters hosted by an infrastructure provider](#port-requirements-for-clusters-hosted-by-an-infrastructure-provider) - - [Security group for nodes on AWS EC2](#security-group-for-nodes-on-aws-ec2) -- [Port requirements for clusters hosted by a Kubernetes provider](#port-requirements-for-clusters-hosted-by-a-kubernetes-provider) -- [Port requirements for imported clusters](#port-requirements-for-imported-clusters) -- [Port requirements for local traffic](#port-requirements-for-local-traffic) - -### Commonly Used Ports - -If security isn't a large concern and you're okay with opening a few additional ports, you can use this table as your port reference instead of the comprehensive tables in the following sections. - -These ports are typically opened on your Kubernetes nodes, regardless of what type of cluster it is. - -
Commonly Used Ports Reference
- -| Protocol | Port | Description | -|:--------: |:----------------: |------------------------------------------------- | -| TCP | 22 | Node driver SSH provisioning | -| TCP | 2376 | Node driver Docker daemon TLS port | -| TCP | 2379 | etcd client requests | -| TCP | 2380 | etcd peer communication | -| UDP | 8472 | Canal/Flannel VXLAN overlay networking | -| UDP | 4789 | Flannel VXLAN overlay networking on Windows cluster | -| TCP | 9099 | Canal/Flannel livenessProbe/readinessProbe | -| TCP | 6783 | Weave Port | -| UDP | 6783-6784 | Weave UDP Ports | -| TCP | 10250 | kubelet API | -| TCP | 10254 | Ingress controller livenessProbe/readinessProbe | -| TCP/UDP | 30000-
32767 | NodePort port range | - -### Port Requirements for Custom Clusters - -If you are launching a Kubernetes cluster on your existing infrastructure, refer to these port requirements. - -The following table depicts the port requirements for [Rancher Launched Kubernetes]({{}}/rancher/v2.x/en/cluster-provisioning/rke-clusters/) with [custom nodes]({{}}/rancher/v2.x/en/cluster-provisioning/rke-clusters/custom-nodes/). - -{{< ports-custom-nodes >}} - -### Port Requirements for Clusters Hosted by an Infrastructure Provider - -If you are launching a Kubernetes cluster on nodes that are in an infrastructure provider such as Amazon EC2, Google Container Engine, DigitalOcean, Azure, or vSphere, these port requirements apply. - -These required ports are automatically opened by Rancher during creation of clusters using cloud providers. - -The following table depicts the port requirements for [Rancher Launched Kubernetes]({{}}/rancher/v2.x/en/cluster-provisioning/rke-clusters/) with nodes created in an [Infrastructure Provider]({{}}/rancher/v2.x/en/cluster-provisioning/rke-clusters/node-pools/). - ->**Note:** ->The required ports are automatically opened by Rancher during creation of clusters in cloud providers like Amazon EC2 or DigitalOcean. - -{{< ports-iaas-nodes >}} - -#### Security Group for Nodes on AWS EC2 - -When using the [AWS EC2 node driver]({{}}/rancher/v2.x/en/cluster-provisioning/rke-clusters/node-pools/ec2/) to provision cluster nodes in Rancher, you can choose to let Rancher create a security group called `rancher-nodes`. The following rules are automatically added to this security group. - -| Type | Protocol | Port Range | Source/Destination | Rule Type | -|-----------------|:--------:|:-----------:|------------------------|:---------:| -| SSH | TCP | 22 | 0.0.0.0/0 | Inbound | -| HTTP | TCP | 80 | 0.0.0.0/0 | Inbound | -| Custom TCP Rule | TCP | 443 | 0.0.0.0/0 | Inbound | -| Custom TCP Rule | TCP | 2376 | 0.0.0.0/0 | Inbound | -| Custom TCP Rule | TCP | 2379-2380 | sg-xxx (rancher-nodes) | Inbound | -| Custom UDP Rule | UDP | 4789 | sg-xxx (rancher-nodes) | Inbound | -| Custom TCP Rule | TCP | 6443 | 0.0.0.0/0 | Inbound | -| Custom UDP Rule | UDP | 8472 | sg-xxx (rancher-nodes) | Inbound | -| Custom TCP Rule | TCP | 10250-10252 | sg-xxx (rancher-nodes) | Inbound | -| Custom TCP Rule | TCP | 10256 | sg-xxx (rancher-nodes) | Inbound | -| Custom TCP Rule | TCP | 30000-32767 | 0.0.0.0/0 | Inbound | -| Custom UDP Rule | UDP | 30000-32767 | 0.0.0.0/0 | Inbound | -| All traffic | All | All | 0.0.0.0/0 | Outbound | - -### Port Requirements for Clusters Hosted by a Kubernetes Provider - -If you are launching a cluster with a [hosted Kubernetes provider]({{}}/rancher/v2.x/en/cluster-provisioning/hosted-kubernetes-clusters). such as Google Kubernetes Engine, Amazon EKS, or Azure Kubernetes Service, refer to these port requirements. - -{{< ports-imported-hosted >}} - - -### Port Requirements for Imported Clusters - -The following table depicts the port requirements for [imported clusters]({{}}/rancher/v2.x/en/cluster-provisioning/imported-clusters/). - -{{< ports-imported-hosted >}} - - -### Port Requirements for Local Traffic - -Ports marked as `local traffic` (i.e., `9099 TCP`) in the port requirements are used for Kubernetes healthchecks (`livenessProbe` and`readinessProbe`). -These healthchecks are executed on the node itself. In most cloud environments, this local traffic is allowed by default. - -However, this traffic may be blocked when: - -- You have applied strict host firewall policies on the node. -- You are using nodes that have multiple interfaces (multihomed). - -In these cases, you have to explicitly allow this traffic in your host firewall, or in case of public/private cloud hosted machines (i.e. AWS or OpenStack), in your security group configuration. Keep in mind that when using a security group as source or destination in your security group, explicitly opening ports only applies to the private interface of the nodes/instances. +Details on which ports are used in each situation are found under [Downstream Cluster Port Requirements](({{}}/rancher/v2.x/en/installation/requirements/ports#downstream-kubernetes-cluster-nodes)). # Optional: Security Considerations If you want to provision a Kubernetes cluster that is compliant with the CIS (Center for Internet Security) Kubernetes Benchmark, we recommend to following our hardening guide to configure your nodes before installing Kubernetes. For more information on the hardening guide and details on which version of the guide corresponds to your Rancher and Kubernetes versions, refer to the [security section.]({{}}/rancher/v2.x/en/security/#rancher-hardening-guide) - -# Opening SUSE Linux Ports - -SUSE Linux may have a firewall that blocks all ports by default. To open the ports needed for adding the host to a custom cluster, - -1. SSH into the instance. -1. Edit /`etc/sysconfig/SuSEfirewall2` and open the required ports. In this example, ports 9796 and 10250 are also opened for monitoring: - ``` - FW_SERVICES_EXT_TCP="22 80 443 2376 2379 2380 6443 9099 9796 10250 10254 30000:32767" - FW_SERVICES_EXT_UDP="8472 30000:32767" - FW_ROUTE=yes - ``` -1. Restart the firewall with the new ports: - ``` - SuSEfirewall2 - ``` - -**Result:** The node has the open ports required to be added to a custom cluster. - diff --git a/content/rancher/v2.x/en/installation/requirements/_index.md b/content/rancher/v2.x/en/installation/requirements/_index.md index 673ca944376..816b0418a6a 100644 --- a/content/rancher/v2.x/en/installation/requirements/_index.md +++ b/content/rancher/v2.x/en/installation/requirements/_index.md @@ -137,176 +137,4 @@ Each node used should have a static IP configured, regardless of whether you are ### Port Requirements -This section describes the port requirements for nodes running the `rancher/rancher` container. - -The port requirements are different depending on whether you are installing Rancher on a K3s cluster, on an RKE cluster, or in a single Docker container. - -{{% tabs %}} -{{% tab "K3s" %}} -### Ports for Communication with Downstream Clusters - -To communicate with downstream clusters, Rancher requires different ports to be open depending on the infrastructure you are using. - -For example, if you are deploying Rancher on nodes hosted by an infrastructure provider, port `22` must be open for SSH. - -The following diagram depicts the ports that are opened for each [cluster type]({{}}/rancher/v2.x/en/cluster-provisioning). - -
Port Requirements for the Rancher Management Plane
- -![Basic Port Requirements]({{}}/img/rancher/port-communications.svg) - -The following tables break down the port requirements for inbound and outbound traffic: - -
Inbound Rules for Rancher Nodes
- -| Protocol | Port | Source | Description | -| -------- | ---- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------- | -| TCP | 80 | Load balancer/proxy that does external SSL termination | Rancher UI/API when external SSL termination is used | -| TCP | 443 |
  • server nodes
  • agent nodes
  • hosted/imported Kubernetes
  • any source that needs to be able to use the Rancher UI or API
| Rancher agent, Rancher UI/API, kubectl | - -
Outbound Rules for Rancher Nodes
- -| Protocol | Port | Destination | Description | -| -------- | ---- | -------------------------------------------------------- | --------------------------------------------- | -| TCP | 22 | Any node IP from a node created using Node Driver | SSH provisioning of nodes using Node Driver | -| TCP | 443 | `35.160.43.145/32`, `35.167.242.46/32`, `52.33.59.17/32` | git.rancher.io (catalogs) | -| TCP | 2376 | Any node IP from a node created using Node driver | Docker daemon TLS port used by Docker Machine | -| TCP | 6443 | Hosted/Imported Kubernetes API | Kubernetes API server | - -**Note** Rancher nodes may also require additional outbound access for any external [authentication provider]({{}}/rancher/v2.x/en/admin-settings/authentication/) which is configured (LDAP for example). - -### Additional Port Requirements for Nodes in a K3s Kubernetes Cluster - -You will need to open additional ports to launch the Kubernetes cluster that is required for a high-availability installation of Rancher. - -The K3s server needs port 6443 to be accessible by the nodes. - -The nodes need to be able to reach other nodes over UDP port 8472 when Flannel VXLAN is used. The node should not listen on any other port. K3s uses reverse tunneling such that the nodes make outbound connections to the server and all kubelet traffic runs through that tunnel. However, if you do not use Flannel and provide your own custom CNI, then port 8472 is not needed by K3s. - -If you wish to utilize the metrics server, you will need to open port 10250 on each node. - -> **Important:** The VXLAN port on nodes should not be exposed to the world as it opens up your cluster network to be accessed by anyone. Run your nodes behind a firewall/security group that disables access to port 8472. - -
Inbound Rules for Rancher Server Nodes
- -| Protocol | Port | Source | Description -|-----|-----|----------------|---| -| TCP | 6443 | K3s server nodes | Kubernetes API -| UDP | 8472 | K3s server and agent nodes | Required only for Flannel VXLAN -| TCP | 10250 | K3s server and agent nodes | kubelet - -Typically all outbound traffic is allowed. -{{% /tab %}} -{{% tab "RKE" %}} -### Ports for Communication with Downstream Clusters - -To communicate with downstream clusters, Rancher requires different ports to be open depending on the infrastructure you are using. - -For example, if you are deploying Rancher on nodes hosted by an infrastructure provider, port `22` must be open for SSH. - -The following diagram depicts the ports that are opened for each [cluster type]({{}}/rancher/v2.x/en/cluster-provisioning). - -
Port Requirements for the Rancher Management Plane
- -![Basic Port Requirements]({{}}/img/rancher/port-communications.svg) - -The following tables break down the port requirements for inbound and outbound traffic: - -
Inbound Rules for Rancher Nodes
- -| Protocol | Port | Source | Description | -| -------- | ---- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------- | -| TCP | 80 | Load balancer/proxy that does external SSL termination | Rancher UI/API when external SSL termination is used | -| TCP | 443 |
  • etcd nodes
  • controlplane nodes
  • worker nodes
  • hosted/imported Kubernetes
  • any source that needs to be able to use the Rancher UI or API
| Rancher agent, Rancher UI/API, kubectl | - -
Outbound Rules for Rancher Nodes
- -| Protocol | Port | Destination | Description | -| -------- | ---- | -------------------------------------------------------- | --------------------------------------------- | -| TCP | 22 | Any node IP from a node created using Node Driver | SSH provisioning of nodes using Node Driver | -| TCP | 443 | `35.160.43.145/32`, `35.167.242.46/32`, `52.33.59.17/32` | git.rancher.io (catalogs) | -| TCP | 2376 | Any node IP from a node created using Node driver | Docker daemon TLS port used by Docker Machine | -| TCP | 6443 | Hosted/Imported Kubernetes API | Kubernetes API server | - -**Note** Rancher nodes may also require additional outbound access for any external [authentication provider]({{}}/rancher/v2.x/en/admin-settings/authentication/) which is configured (LDAP for example). - -### Additional Port Requirements for Nodes in an RKE Kubernetes Cluster - -You will need to open additional ports to launch the Kubernetes cluster that is required for a high-availability installation of Rancher. - -If you follow the Rancher installation documentation for setting up a Kubernetes cluster using RKE, you will set up a cluster in which all three nodes have all three roles: etcd, controlplane, and worker. In that case, you can refer to this list of requirements for each node with all three roles. - -If you installed Rancher on a Kubernetes cluster that doesn't have all three roles on each node, refer to the [port requirements for the Rancher Kubernetes Engine (RKE).]({{}}/rke/latest/en/os/#ports) The RKE docs show a breakdown of the port requirements for each role. - -
Inbound Rules for Nodes with All Three Roles: etcd, Controlplane, and Worker
- -Protocol | Port | Source | Description ------------|------|----------|-------------- -TCP | 22 | Linux worker nodes only, and any network that you want to be able to remotely access this node from. | Remote access over SSH -TCP | 80 | Any source that consumes Ingress services | Ingress controller (HTTP) -TCP | 443 | Any source that consumes Ingress services | Ingress controller (HTTPS) -TCP | 2376 | Rancher nodes | Docker daemon TLS port used by Docker Machine (only needed when using Node Driver/Templates) -TCP | 2379 | etcd nodes and controlplane nodes | etcd client requests -TCP | 2380 | etcd nodes and controlplane nodes | etcd peer communication -TCP | 3389 | Windows worker nodes only, and any network that you want to be able to remotely access this node from. | Remote access over RDP -TCP | 6443 | etcd nodes, controlplane nodes, and worker nodes | Kubernetes apiserver -UDP | 8472 | etcd nodes, controlplane nodes, and worker nodes | Canal/Flannel VXLAN overlay networking -TCP | 9099 | the node itself (local traffic, not across nodes) | Canal/Flannel livenessProbe/readinessProbe -TCP | 10250 | controlplane nodes | kubelet -TCP | 10254 | the node itself (local traffic, not across nodes) | Ingress controller livenessProbe/readinessProbe -TCP/UDP | 30000-32767 | Any source that consumes NodePort services | NodePort port range - -
Outbound Rules for Nodes with All Three Roles: etcd, Controlplane, and Worker
- -Protocol | Port | Source | Destination | Description ------------|------|----------|---------------|-------------- -TCP | 22 | RKE node | Any node configured in Cluster Configuration File | SSH provisioning of node by RKE -TCP | 443 | Rancher nodes | Rancher agent | -TCP | 2379 | etcd nodes | etcd client requests | -TCP | 2380 | etcd nodes | etcd peer communication | -TCP | 6443 | RKE node | controlplane nodes | Kubernetes API server -TCP | 6443 | controlplane nodes | Kubernetes API server | -UDP | 8472 | etcd nodes, controlplane nodes, and worker nodes | Canal/Flannel VXLAN overlay networking | -TCP | 9099 | the node itself (local traffic, not across nodes) | Canal/Flannel livenessProbe/readinessProbe | -TCP | 10250 | etcd nodes, controlplane nodes, and worker nodes | kubelet | -TCP | 10254 | the node itself (local traffic, not across nodes) | Ingress controller livenessProbe/readinessProbe - -{{% /tab %}} -{{% tab "Docker" %}} -### Ports for Communication with Downstream Clusters - -For a Docker installation, you only need to open the ports required to enable Rancher to communicate with downstream user clusters. - -The port requirements depend on the infrastructure you are using. For example, if you are deploying Rancher on nodes hosted by an infrastructure provider, port `22` must be open for SSH. - -The following diagram depicts the ports that are opened for each [cluster type]({{}}/rancher/v2.x/en/cluster-provisioning). - -
Port Requirements for the Rancher Management Plane
- -![Basic Port Requirements]({{}}/img/rancher/port-communications.svg) - -The following tables break down the port requirements for Rancher nodes, for inbound and outbound traffic: - -**Note** Rancher nodes may also require additional outbound access for any external [authentication provider]({{}}/rancher/v2.x/en/admin-settings/authentication/) which is configured (LDAP for example). - - -
Inbound Rules
- -| Protocol | Port | Source | Description | -| -------- | ---- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------- | -| TCP | 80 | Load balancer/proxy that does external SSL termination | Rancher UI/API when external SSL termination is used | -| TCP | 443 |
  • etcd nodes
  • controlplane nodes
  • worker nodes
  • hosted/imported Kubernetes
  • any source that needs to be able to use the Rancher UI or API
| Rancher agent, Rancher UI/API, kubectl | - - -
Outbound Rules
- -| Protocol | Port | Source | Description | -| -------- | ---- | -------------------------------------------------------- | --------------------------------------------- | -| TCP | 22 | Any node IP from a node created using Node Driver | SSH provisioning of nodes using Node Driver | -| TCP | 443 | `35.160.43.145/32`, `35.167.242.46/32`, `52.33.59.17/32` | git.rancher.io (catalogs) | -| TCP | 2376 | Any node IP from a node created using Node driver | Docker daemon TLS port used by Docker Machine | -| TCP | 6443 | Hosted/Imported Kubernetes API | Kubernetes API server | - -**Note** Rancher nodes may also require additional outbound access for any external [authentication provider]({{}}/rancher/v2.x/en/admin-settings/authentication/) which is configured (LDAP for example). -{{% /tab %}} -{{% /tabs %}} +To operate properly, Rancher requires a number of ports to be open on Rancher nodes and on downstream Kubernetes cluster nodes. [Port Requirements]({{}}/rancher/v2.x/en/installation/requirements/ports) lists all the necessary ports for Rancher and Downstream Clusters for the different cluster types. diff --git a/content/rancher/v2.x/en/installation/requirements/ports/_index.md b/content/rancher/v2.x/en/installation/requirements/ports/_index.md index c6cb826bb90..196a1504aaf 100644 --- a/content/rancher/v2.x/en/installation/requirements/ports/_index.md +++ b/content/rancher/v2.x/en/installation/requirements/ports/_index.md @@ -23,24 +23,57 @@ If you wish to utilize the metrics server, you will need to open port 10250 on e > **Important:** The VXLAN port on nodes should not be exposed to the world as it opens up your cluster network to be accessed by anyone. Run your nodes behind a firewall/security group that disables access to port 8472. +The following tables break down the port requirements for inbound and outbound traffic: +
Inbound Rules for Rancher Server Nodes
| Protocol | Port | Source | Description |-----|-----|----------------|---| +| TCP | 80 | Load balancer/proxy that does external SSL termination | Rancher UI/API when external SSL termination is used | +| TCP | 443 |
  • server nodes
  • agent nodes
  • hosted/imported Kubernetes
  • any source that needs to be able to use the Rancher UI or API
| Rancher agent, Rancher UI/API, kubectl | | TCP | 6443 | K3s server nodes | Kubernetes API | UDP | 8472 | K3s server and agent nodes | Required only for Flannel VXLAN. | TCP | 10250 | K3s server and agent nodes | kubelet -Typically all outbound traffic is allowed. +
Outbound Rules for Rancher Nodes
+ +| Protocol | Port | Destination | Description | +| -------- | ---- | -------------------------------------------------------- | --------------------------------------------- | +| TCP | 22 | Any node IP from a node created using Node Driver | SSH provisioning of nodes using Node Driver | +| TCP | 443 | `35.160.43.145/32`, `35.167.242.46/32`, `52.33.59.17/32` | git.rancher.io (catalogs) | +| TCP | 2376 | Any node IP from a node created using Node driver | Docker daemon TLS port used by Docker Machine | +| TCP | 6443 | Hosted/Imported Kubernetes API | Kubernetes API server | {{% /tab %}} {{% tab "RKE" %}} + +Typically Rancher is installed on three RKE nodes that all have the etcd, control plane and worker roles. + +The following tables break down the port requirements for traffic between the Rancher nodes: + +
Rules for traffic between Rancher nodes
+ +| Protocol | Port | Description | +|-----|-----|----------------| +| TCP | 443 | Rancher agents | +| TCP | 2379 | etcd client requests | +| TCP | 2380 | etcd peer communication | +| TCP | 6443 | Kubernetes apiserver | +| UDP | 8472 | Canal/Flannel VXLAN overlay networking | +| TCP | 9099 | Canal/Flannel livenessProbe/readinessProbe | +| TCP | 10250 | kubelet | +| TCP | 10254 | Ingress controller livenessProbe/readinessProbe | + +The following tables break down the port requirements for inbound and outbound traffic: +
Inbound Rules for Rancher Nodes
| Protocol | Port | Source | Description | |-----|-----|----------------|---| +| TCP | 22 | RKE CLI | SSH provisioning of node by RKE | | TCP | 80 | Load Balancer/Reverse Proxy | HTTP traffic to Rancher UI/API | | TCP | 443 |
  • Load Balancer/Reverse Proxy
  • IPs of all cluster nodes and other API/UI clients
| HTTPS traffic to Rancher UI/API | +| TCP | 6443 | Kubernetes API clients | HTTPS traffic to Kubernetes API |
Outbound Rules for Rancher Nodes
@@ -49,11 +82,14 @@ Typically all outbound traffic is allowed. | TCP | 443 | `35.160.43.145`,`35.167.242.46`,`52.33.59.17` | Rancher catalog (git.rancher.io) | | TCP | 22 | Any node created using a node driver | SSH provisioning of node by node driver | | TCP | 2376 | Any node created using a node driver | Docker daemon TLS port used by node driver | +| TCP | 6443 | Hosted/Imported Kubernetes API | Kubernetes API server | | TCP | Provider dependent | Port of the Kubernetes API endpoint in hosted cluster | Kubernetes API | {{% /tab %}} {{% tab "Docker" %}} +The following tables break down the port requirements for Rancher nodes, for inbound and outbound traffic: +
Inbound Rules for Rancher Node
| Protocol | Port | Source | Description @@ -85,6 +121,12 @@ Downstream Kubernetes clusters run your apps and services. This section describe The port requirements differ depending on how the downstream cluster was launched. Each of the tabs below list the ports that need to be opened for different [cluster types]({{}}/rancher/v2.x/en/cluster-provisioning/#cluster-creation-options). +The following diagram depicts the ports that are opened for each [cluster type]({{}}/rancher/v2.x/en/cluster-provisioning). + +
Port Requirements for the Rancher Management Plane
+ +![Basic Port Requirements]({{}}/img/rancher/port-communications.svg) + >**Tip:** > >If security isn't a large concern and you're okay with opening a few additional ports, you can use the table in [Commonly Used Ports](#commonly-used-ports) as your port reference instead of the comprehensive tables below. @@ -135,21 +177,7 @@ The following table depicts the port requirements for [imported clusters]({{32767 | NodePort port range | +{{% include file="/rancher/v2.x/en/installation/requirements/ports/common-ports-table" %}} ---- @@ -181,6 +209,28 @@ When using the [AWS EC2 node driver]({{}}/rancher/v2.x/en/cluster-provi | Custom UDP Rule | UDP | 8472 | sg-xxx (rancher-nodes) | Inbound | | Custom TCP Rule | TCP | 10250-10252 | sg-xxx (rancher-nodes) | Inbound | | Custom TCP Rule | TCP | 10256 | sg-xxx (rancher-nodes) | Inbound | -| Custom TCP Rule | TCP | 30000-32767 | 0.0.0.0/0 | Inbound | -| Custom UDP Rule | UDP | 30000-32767 | 0.0.0.0/0 | Inbound | +| Custom TCP Rule | TCP | 30000-32767 | 0.0.0.0/0 | Inbound | +| Custom UDP Rule | UDP | 30000-32767 | 0.0.0.0/0 | Inbound | | All traffic | All | All | 0.0.0.0/0 | Outbound | + +### Opening Ports with firewalld + +[Opening Ports with firewalld]({{}}/rancher/v2.x/en/installation/options/firewall) describes how to use firewalld to apply the above rules. + +### Opening SUSE Linux Ports + +SUSE Linux may have a firewall that blocks all ports by default. To open the ports needed for adding the host to a custom cluster, + +1. SSH into the instance. +1. Edit /`etc/sysconfig/SuSEfirewall2` and open the required ports. In this example, ports 9796 and 10250 are also opened for monitoring: + ``` + FW_SERVICES_EXT_TCP="22 80 443 2376 2379 2380 6443 9099 9796 10250 10254 30000:32767" + FW_SERVICES_EXT_UDP="8472 30000:32767" + FW_ROUTE=yes + ``` +1. Restart the firewall with the new ports: + ``` + SuSEfirewall2 + ``` + +**Result:** The node has the open ports required to be added to a custom cluster. diff --git a/content/rancher/v2.x/en/installation/requirements/ports/common-ports-table/index.md b/content/rancher/v2.x/en/installation/requirements/ports/common-ports-table/index.md new file mode 100644 index 00000000000..12498ab1b81 --- /dev/null +++ b/content/rancher/v2.x/en/installation/requirements/ports/common-ports-table/index.md @@ -0,0 +1,20 @@ +--- +headless: true +--- +| Protocol | Port | Description | +|:--------: |:----------------: |---------------------------------------------------------------------------------- | +| TCP | 22 | Node driver SSH provisioning | +| TCP | 179 | Calico BGP Port | +| TCP | 2376 | Node driver Docker daemon TLS port | +| TCP | 2379 | etcd client requests | +| TCP | 2380 | etcd peer communication | +| UDP | 8472 | Canal/Flannel VXLAN overlay networking | +| UDP | 4789 | Flannel VXLAN overlay networking on Windows cluster | +| TCP | 9099 | Canal/Flannel livenessProbe/readinessProbe | +| TCP | 9100 | Default port required by Monitoring to scrape metrics from Linux node-exporters | +| TCP | 9796 | Default port required by Monitoring to scrape metrics from Windows node-exporters | +| TCP | 6783 | Weave Port | +| UDP | 6783-6784 | Weave UDP Ports | +| TCP | 10250 | kubelet API | +| TCP | 10254 | Ingress controller livenessProbe/readinessProbe | +| TCP/UDP | 30000-
32767 | NodePort port range | diff --git a/content/rancher/v2.x/en/installation/resources/advanced/firewall/_index.md b/content/rancher/v2.x/en/installation/resources/advanced/firewall/_index.md index 69cb99eeff4..42d2379f09b 100644 --- a/content/rancher/v2.x/en/installation/resources/advanced/firewall/_index.md +++ b/content/rancher/v2.x/en/installation/resources/advanced/firewall/_index.md @@ -105,4 +105,4 @@ After the `firewall-cmd` commands have been run on a node, use the following com firewall-cmd --reload ``` -**Result:** The firewall is updated so that Helm can communicate with the Rancher server nodes. \ No newline at end of file +**Result:** The firewall is updated so that Helm can communicate with the Rancher server nodes. diff --git a/layouts/shortcodes/ports-custom-nodes.html b/layouts/shortcodes/ports-custom-nodes.html index 4edf822d2f6..45af1975f8f 100644 --- a/layouts/shortcodes/ports-custom-nodes.html +++ b/layouts/shortcodes/ports-custom-nodes.html @@ -6,7 +6,7 @@ etcd Plane Nodes Control Plane Nodes Worker Plane Nodes - External Load Balancer + External Rancher Load Balancer Internet @@ -46,7 +46,7 @@ - 4789 UDP (7) + 4789 UDP (6) @@ -81,7 +81,7 @@ - 4789 UDP (7) + 4789 UDP (6) @@ -114,7 +114,7 @@ - 4789 UDP (7) + 4789 UDP (6) @@ -130,41 +130,16 @@ - External Load Balancer (5) - 80 TCP + Kubernetes API Clients + 6443 TCP (5) - 443 TCP (6) - - - - - - - - API / UI Clients - 80 TCP (3) - - - - 80 TCP
- - - - 443 TCP (3) - - - - 443 TCP - - - - Workload Clients + Workload Clients or Load Balancer @@ -175,19 +150,19 @@ - 80 TCP (Ingress) + 80 TCP (Ingress) - 443 TCP (Ingress) + 443 TCP (Ingress) - Notes:

1. Nodes running standalone server or Rancher HA deployment.
2. Required to fetch Rancher chart library.
3. Only without external load balancer.
4. Local traffic to the node itself (not across nodes).
5. Load balancer / proxy that handles tragging to the Rancher UI / API.
6. Only if SSL is not terminated at external load balancer.
7. Only if using Overlay mode on Windows cluster. + Notes:

1. Nodes running standalone server or Rancher HA deployment.
2. Required to fetch Rancher chart library.
3. Only without external load balancer in front of Rancher.
4. Local traffic to the node itself (not across nodes).
5. Only if Authorized Cluster Endpoints are activated.
6. Only if using Overlay mode on Windows cluster. - \ No newline at end of file + diff --git a/layouts/shortcodes/ports-iaas-nodes.html b/layouts/shortcodes/ports-iaas-nodes.html index 853c7a0e626..3079e9bdb21 100644 --- a/layouts/shortcodes/ports-iaas-nodes.html +++ b/layouts/shortcodes/ports-iaas-nodes.html @@ -6,7 +6,7 @@ etcd Plane Nodes Control Plane Nodes Worker Plane Nodes - External Load Balancer + External Rancher Load Balancer Internet @@ -121,41 +121,16 @@ - External Load Balancer (5) - 80 TCP + Kubernetes API Clients + 6443 TCP (5) - 443 TCP (6) - - - - - - - - API / UI Clients - 80 TCP (3) - - - - 80 TCP
- - - - 443 TCP (3) - - - - 443 TCP - - - - Workload Clients + Workload Clients or Load Balancer @@ -166,19 +141,19 @@ - 80 TCP (Ingress) + 80 TCP (Ingress) - 443 TCP (Ingress) + 443 TCP (Ingress) - Notes:

1. Nodes running standalone server or Rancher HA deployment.
2. Required to fetch Rancher chart library.
3. Only without external load balancer.
4. Local traffic to the node itself (not across nodes).
5. Load balancer / proxy that handles tragging to the Rancher UI / API.
6. Only if SSL is not terminated at external load balancer. + Notes:

1. Nodes running standalone server or Rancher HA deployment.
2. Required to fetch Rancher chart library.
3. Only without external load balancer in front of Rancher.
4. Local traffic to the node itself (not across nodes).
5. Only if Authorized Cluster Endpoints are activated. - \ No newline at end of file + diff --git a/layouts/shortcodes/ports-imported-hosted.html b/layouts/shortcodes/ports-imported-hosted.html index 96eef6bec3c..676551c4416 100644 --- a/layouts/shortcodes/ports-imported-hosted.html +++ b/layouts/shortcodes/ports-imported-hosted.html @@ -4,7 +4,7 @@ From / To Rancher Nodes Hosted / Imported Cluster - External Load Balancer + External Rancher Load Balancer Internet @@ -24,20 +24,13 @@ - External Load Balancer (5) - 80 TCP
443 TCP (6) + Kubernetes API Clients + Cluster / Provider Specific (6) - API / UI Clients - 80 TCP (4)
443 TCP (4) - - 80 TCP
443 TCP - - - Workload Client Cluster / Provider Specific (7) @@ -45,7 +38,7 @@ - Notes:

1. Nodes running standalone server or Rancher HA deployment.
2. Only for hosted clusters.
3. Required to fetch Rancher chart library.
4. Only without external load balancer.
5. From worker nodes.
6. Only if SSL is not terminated at external load balancer.
7. Usually Ingress backed by infrastructure load balancer and/or nodeport. + Notes:

1. Nodes running standalone server or Rancher HA deployment.
2. Only for hosted clusters.
3. Required to fetch Rancher chart library.
4. Only without external load balancer.
5. From worker nodes.
6. For direct access to the Kubernetes API without Rancher.
7. Usually Ingress backed by infrastructure load balancer and/or nodeport. - \ No newline at end of file + diff --git a/static/img/rancher/port-communications.svg b/static/img/rancher/port-communications.svg index 183d2adcc5d..f57ca9ac8be 100644 --- a/static/img/rancher/port-communications.svg +++ b/static/img/rancher/port-communications.svg @@ -1,2 +1,2 @@ -

Rancher Management Plane
[Not supported by viewer]
Text
Text
Rancher Launched Kubernetes

 
  Cluster with custom nodes.
[Not supported by viewer]
Imported Clusters
  • kops
  • Tectonic
  • RKE CLI
  • non-Rancher provisioned cloud cluster
[Not supported by viewer]
Hosted Kubernetes Provider

 Provisioned by Rancher.
  • Google GKE
  • Amazon EKS
  • Microsoft AKS
[Not supported by viewer]
 ▲ Rancher Server TLS: 443
[Not supported by viewer]
▲ Rancher Server TLS: 443
[Not supported by viewer]
▼ SSH: 22
▼ Docker Daemon TLS: 2376
[Not supported by viewer]
 ▲ Rancher Server TLS: 443
[Not supported by viewer]
 ▼ Kubernetes API: 6443
[Not supported by viewer]
 ▲ Rancher Server TLS: 443
[Not supported by viewer]
Rancher Launched Kubernetes

 Nodes hosted by an IaaS. 
  • Amazon EC2
  • Digital Ocean
  • Azure
  • Vsphere
[Not supported by viewer] \ No newline at end of file +

Rancher Management Plane
[Not supported by viewer]
Text
Text
Rancher Launched Kubernetes

 
  Cluster with custom nodes.
[Not supported by viewer]
Imported Clusters
  • kops
  • Tectonic
  • RKE CLI
  • non-Rancher provisioned cloud cluster
[Not supported by viewer]
Hosted Kubernetes Provider

 Provisioned by Rancher.
  • Google GKE
  • Amazon EKS
  • Microsoft AKS
[Not supported by viewer]
 ▲ Rancher Server TLS: 443
[Not supported by viewer]
▲ Rancher Server TLS: 443
[Not supported by viewer]
▼ SSH: 22
▼ Docker Daemon TLS: 2376
[Not supported by viewer]
 ▲ Rancher Server TLS: 443
[Not supported by viewer]
 ▼ Kubernetes API: 6443
[Not supported by viewer]
 ▲ Rancher Server TLS: 443
[Not supported by viewer]
Rancher Launched Kubernetes

 Nodes hosted by an IaaS. 
  • Amazon EC2
  • Digital Ocean
  • Azure
  • Vsphere
[Not supported by viewer] From e842604f7cd471bedf23eb660ca3d94db36dc67a Mon Sep 17 00:00:00 2001 From: Bastian Hofmann Date: Tue, 17 Nov 2020 15:05:24 +0100 Subject: [PATCH 05/49] Add new 2.5 webhook port requirements to imported clusters https://github.com/rancher/rancher/issues/30101 Signed-off-by: Bastian Hofmann --- .../requirements/ports/common-ports-table/index.md | 2 ++ layouts/shortcodes/ports-imported-hosted.html | 14 ++++++++++++-- 2 files changed, 14 insertions(+), 2 deletions(-) diff --git a/content/rancher/v2.x/en/installation/requirements/ports/common-ports-table/index.md b/content/rancher/v2.x/en/installation/requirements/ports/common-ports-table/index.md index 12498ab1b81..86bb7177bbe 100644 --- a/content/rancher/v2.x/en/installation/requirements/ports/common-ports-table/index.md +++ b/content/rancher/v2.x/en/installation/requirements/ports/common-ports-table/index.md @@ -10,8 +10,10 @@ headless: true | TCP | 2380 | etcd peer communication | | UDP | 8472 | Canal/Flannel VXLAN overlay networking | | UDP | 4789 | Flannel VXLAN overlay networking on Windows cluster | +| TCP | 8443 | Rancher webhook | | TCP | 9099 | Canal/Flannel livenessProbe/readinessProbe | | TCP | 9100 | Default port required by Monitoring to scrape metrics from Linux node-exporters | +| TCP | 9443 | Rancher webhook | | TCP | 9796 | Default port required by Monitoring to scrape metrics from Windows node-exporters | | TCP | 6783 | Weave Port | | UDP | 6783-6784 | Weave UDP Ports | diff --git a/layouts/shortcodes/ports-imported-hosted.html b/layouts/shortcodes/ports-imported-hosted.html index 676551c4416..ea9cf448bad 100644 --- a/layouts/shortcodes/ports-imported-hosted.html +++ b/layouts/shortcodes/ports-imported-hosted.html @@ -10,11 +10,21 @@ - Rancher Nodes (1) + Rancher Nodes (1) Kubernetes API
Endpoint Port (2) - git.rancher.io (3):
35.160.43.145:32
35.167.242.46:32
52.33.59.17:32 + git.rancher.io (3):
35.160.43.145:32
35.167.242.46:32
52.33.59.17:32 + + + + 8443 TCP + + + + + 9443 TCP + Hosted / Imported Cluster From a731fed051c4b928c86179e9479563ba8e95fcd4 Mon Sep 17 00:00:00 2001 From: Catherine Luse Date: Thu, 3 Dec 2020 16:57:09 -0700 Subject: [PATCH 06/49] Document configuration for routes, receivers, and rules in Rancher UI --- .../v2.5/configuration/_index.md | 56 +-- .../v2.5/configuration/alertmanager/_index.md | 202 ++++++++ .../v2.5/configuration/expression/_index.md | 432 ++++++++++++++++++ .../configuration/prometheusrules/_index.md | 93 ++++ 4 files changed, 734 insertions(+), 49 deletions(-) create mode 100644 content/rancher/v2.x/en/monitoring-alerting/v2.5/configuration/alertmanager/_index.md create mode 100644 content/rancher/v2.x/en/monitoring-alerting/v2.5/configuration/expression/_index.md create mode 100644 content/rancher/v2.x/en/monitoring-alerting/v2.5/configuration/prometheusrules/_index.md diff --git a/content/rancher/v2.x/en/monitoring-alerting/v2.5/configuration/_index.md b/content/rancher/v2.x/en/monitoring-alerting/v2.5/configuration/_index.md index 4ad287a383f..c8ab7edbac1 100644 --- a/content/rancher/v2.x/en/monitoring-alerting/v2.5/configuration/_index.md +++ b/content/rancher/v2.x/en/monitoring-alerting/v2.5/configuration/_index.md @@ -13,8 +13,8 @@ For information on configuring custom scrape targets and rules for Prometheus, p - [Configuring Targets with ServiceMonitors and PodMonitors](#configuring-targets-with-servicemonitors-and-podmonitors) - [ServiceMonitors](#servicemonitors) - [PodMonitors](#podmonitors) - - [PrometheusRules](#prometheusrules) - - [Alertmanager Config](#alertmanager-config) +- [PrometheusRules](#prometheusrules) +- [Alertmanager Config](#alertmanager-config) - [Trusted CA for Notifiers](#trusted-ca-for-notifiers) - [Additional Scrape Configurations](#additional-scrape-configurations) - [Examples](#examples) @@ -45,40 +45,15 @@ For more information about how ServiceMonitors work, refer to the [Prometheus Op This CRD declaratively specifies how group of pods should be monitored. Any Pods in your cluster that match the labels located within the PodMonitor `selector` field will be monitored based on the `podMetricsEndpoints` specified on the PodMonitor. For more information on what fields can be specified, please look at the [spec](https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/api.md#podmonitorspec) provided by Prometheus Operator. -### PrometheusRules +# PrometheusRules This CRD defines a group of Prometheus alerting and/or recording rules. -To add a group of alerting / recording rules, you should create a PrometheusRule CR the defines a RuleGroup with your desired rules, each specifying: +For information on configuring Prometheus rules, refer to [this page.](./prometheusrules) -- The name of the new alert / record -- A PromQL expression for the new alert / record -- Labels that should be attached to the alert / record that identify it (e.g. cluster name or severity) -- Annotations that encode any additional important pieces of information that need to be displayed on the notification for an alert (e.g. summary, description, message, runbook URL, etc.). This field is not required for recording rules. +# Alertmanager Config -For more information on what fields can be specified, please look at the [Prometheus Operator spec.](https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/api.md#prometheusrulespec) - -### Alertmanager Config - -The [Alertmanager Config](https://prometheus.io/docs/alerting/latest/configuration/#configuration-file) Secret contains the configuration of an Alertmanager instance that sends out notifications based on alerts it receives from Prometheus. - -By default, Rancher Monitoring deploys a single Alertmanager onto a cluster that uses a default Alertmanager Config Secret. As part of the chart deployment options, you can opt to increase the number of replicas of the Alertmanager deployed onto your cluster that can all be managed using the same underlying Alertmanager Config Secret. - -This Secret should be updated or modified any time you want to: - -- Add in new notifiers or receivers -- Change the alerts that should be sent to specific notifiers or receivers -- Change the group of alerts that are sent out - -> By default, you can either choose to supply an existing Alertmanager Config Secret (i.e. any Secret in the `cattle-monitoring-system` namespace) or allow Rancher Monitoring to deploy a default Alertmanager Config Secret onto your cluster. By default, the Alertmanager Config Secret created by Rancher will never be modified / deleted on an upgrade / uninstall of the `rancher-monitoring` chart to prevent users from losing or overwriting their alerting configuration when executing operations on the chart. - -For more information on what fields can be specified in this secret, please look at the [Prometheus Alertmanager docs](https://prometheus.io/docs/alerting/latest/alertmanager/) - -The full spec for the Alertmanager configuration file and what it takes in can be found [here.](https://prometheus.io/docs/alerting/latest/configuration/#configuration-file) - -The notification integrations are configured with the `receiver`, which is documented [here.](https://prometheus.io/docs/alerting/latest/configuration/#receiver) - -For more information, refer to the [official Prometheus documentation about configuring routes.](https://www.prometheus.io/docs/alerting/latest/configuration/#route) +For information on configuring the Alertmanager, refer to [this page.](./alertmanager) # Trusted CA for Notifiers @@ -114,21 +89,4 @@ Prometheus rule files are held in PrometheusRule custom resources. Use the label ### Alertmanager Config -To set up notifications via Slack, the following Alertmanager Config YAML should be placed into the `alertmanager.yaml` key of the Alertmanager Config Secret, where the `api_url` should be updated to use your Webhook URL from Slack: - -```yaml -route: - group_by: ['job'] - group_wait: 30s - group_interval: 5m - repeat_interval: 3h - receiver: 'slack-notifications' -receivers: -- name: 'slack-notifications' - slack_configs: - - send_resolved: true - text: '{{ template "slack.rancher.text" . }}' - api_url: -templates: -- /etc/alertmanager/config/*.tmpl -``` \ No newline at end of file +For an example configuration, refer to [this section.](./alertmanager/#example-alertmanager-config) \ No newline at end of file diff --git a/content/rancher/v2.x/en/monitoring-alerting/v2.5/configuration/alertmanager/_index.md b/content/rancher/v2.x/en/monitoring-alerting/v2.5/configuration/alertmanager/_index.md new file mode 100644 index 00000000000..29cd61bae69 --- /dev/null +++ b/content/rancher/v2.x/en/monitoring-alerting/v2.5/configuration/alertmanager/_index.md @@ -0,0 +1,202 @@ +--- +title: Alertmanager +weight: 1 +--- + +The [Alertmanager Config](https://prometheus.io/docs/alerting/latest/configuration/#configuration-file) Secret contains the configuration of an Alertmanager instance that sends out notifications based on alerts it receives from Prometheus. + +- [Overview](#overview) +- [Creating Receivers in the Rancher UI](#creating-receivers-in-the-rancher-ui) +- [Receiver Configuration](#receiver-configuration) + - [Slack](#slack) + - [Email](#email) + - [PagerDuty](#pagerduty) + - [Opsgenie](#opsgenie) + - [Webhook](#webhook) + - [Custom](#custom) +- [Route Configuration](#route-configuration) + - [Receiver](#receiver) + - [Grouping](#grouping) + - [Matching](#matching) +- [Example Alertmanager YAML](#example-alertmanager-yaml) + +# Overview + +By default, Rancher Monitoring deploys a single Alertmanager onto a cluster that uses a default Alertmanager Config Secret. As part of the chart deployment options, you can opt to increase the number of replicas of the Alertmanager deployed onto your cluster that can all be managed using the same underlying Alertmanager Config Secret. + +This Secret should be updated or modified any time you want to: + +- Add in new notifiers or receivers +- Change the alerts that should be sent to specific notifiers or receivers +- Change the group of alerts that are sent out + +> By default, you can either choose to supply an existing Alertmanager Config Secret (i.e. any Secret in the `cattle-monitoring-system` namespace) or allow Rancher Monitoring to deploy a default Alertmanager Config Secret onto your cluster. By default, the Alertmanager Config Secret created by Rancher will never be modified / deleted on an upgrade / uninstall of the `rancher-monitoring` chart to prevent users from losing or overwriting their alerting configuration when executing operations on the chart. + +For more information on what fields can be specified in this secret, please look at the [Prometheus Alertmanager docs.](https://prometheus.io/docs/alerting/latest/alertmanager/) + +The full spec for the Alertmanager configuration file and what it takes in can be found [here.](https://prometheus.io/docs/alerting/latest/configuration/#configuration-file) + +For more information, refer to the [official Prometheus documentation about configuring routes.](https://www.prometheus.io/docs/alerting/latest/configuration/#route) + +# Creating Receivers in the Rancher UI +_Available as of v2.5.4_ + +> **Prerequisite:** The monitoring application needs to be installed. + +To create notification receivers in the Rancher UI, + +1. Click **Cluster Explorer > Monitoring** and click **Receiver.** +2. Enter a name for the receiver. +3. Configure one or more providers for the receiver. For help filling out the forms, refer to the configuration options below. +4. Click **Create.** + +**Result:** Alerts can be configured to send notifications to the receiver(s). + +# Receiver Configuration + +The notification integrations are configured with the `receiver`, which is explained in the [Prometheus documentation.](https://prometheus.io/docs/alerting/latest/configuration/#receiver) + +Rancher v2.5.4 introduced the capability to configure reducers by filling out forms in the Rancher UI. + +{{% tabs %}} +{{% tab "Rancher v2.5.4+" %}} + +The following types of receivers can be configured in the Rancher UI: + +- Slack +- Email +- PagerDuty +- Opsgenie +- Webhook +- Custom + +The custom receiver option can be used to configure any receiver in YAML that cannot be configured by filling out the other forms in the Rancher UI. + +### Slack + +| Field | Type | Description | +|------|--------------|------| +| URL | String | Enter your Slack webhook URL. For instructions to create a Slack webhook, see the [Slack documentation.](https://get.slack.help/hc/en-us/articles/115005265063-Incoming-WebHooks-for-Slack) | +| Default Channel | String | Enter the name of the channel that you want to send alert notifications in the following format: `#` | +| Proxy URL | String | Proxy for the webhook notifications. | +| Enable send resolved alerts | Bool | When true, you will receive alerts through the notifier even if the alert condition is no longer true. For example, if an alert is triggered because your CPU is too high, you will still receive the alert after CPU goes back to normal levels. | + +### Email + +| Field | Type | Description | +|------|--------------|------| +| Default Recipient Address | String | The email address that will receive notifications. | +| Enable send resolved alerts | Bool | When true, you will receive alerts through the notifier even if the alert condition is no longer true. For example, if an alert is triggered because your CPU is too high, you will still receive the alert after CPU goes back to normal levels. | + +SMTP options: + +| Field | Type | Description | +|------|--------------|------| +| Sender | String | Enter an email address available on your SMTP mail server that you want to send the notification from. | +| Host | String | Enter the IP address or hostname for your SMTP server. Example: `smtp.email.com` | +| Use TLS | Bool | Use TLS for encryption. | +| Username | String | Enter a username to authenticate with the SMTP server. | +| Password | String | Enter a password to authenticate with the SMTP server. | + +### PagerDuty + +| Field | Type | Description | +|------|------|-------| +| Integration Type | String | Events API v2 or Prometheus. | +| Default Integration Key | String | For instructions to get an integration key, see the [PagerDuty documentation.](https://www.pagerduty.com/docs/guides/prometheus-integration-guide/) | +| Proxy URL | String | Proxy for the PagerDuty notifications. | +| Enable send resolved alerts | Bool | When true, you will receive alerts through the notifier even if the alert condition is no longer true. For example, if an alert is triggered because your CPU is too high, you will still receive the alert after CPU goes back to normal levels. | + +### Opsgenie + +| Field | Description | +|------|-------------| +| API Key | For instructions to get an API key, refer to the [Opsgenie documentation.](https://docs.opsgenie.com/docs/api-key-management) | +| Proxy URL | Proxy for the Opsgenie notifications. | +| Enable send resolved alerts | When true, you will receive alerts through the notifier even if the alert condition is no longer true. For example, if an alert is triggered because your CPU is too high, you will still receive the alert after CPU goes back to normal levels. | + +Opsgenie Responders: + +| Field | Type | Description | +|-------|------|--------| +| Type | String | Schedule, Team, User, or Escalation. For more information on alert responders, refer to the [Opsgenie documentation.](https://docs.opsgenie.com/docs/alert-recipients-and-teams) | +| Send To | String | Id, Name, or Username of the Opsgenie recipient. | + +### Webhook + +| Field | Description | +|-------|--------------| +| URL | Webhook URL for the app of your choice. | +| Proxy URL | Proxy for the webhook notification | +| Enable send resolved alerts | When true, you will receive alerts through the notifier even if the alert condition is no longer true. For example, if an alert is triggered because your CPU is too high, you will still receive the alert after CPU goes back to normal levels. | + +### Custom + +The YAML provided here will be directly appended to your receiver within the Alertmanager Config Secret. + +{{% /tab %}} +{{% tab "Rancher v2.5.0-2.5.3" %}} +The Alertmanager must be configured in YAML, as shown in this [example.](#example-alertmanager-config) +{{% /tab %}} +{{% /tabs %}} + + +# Route Configuration + +{{% tabs %}} +{{% tab "Rancher v2.5.4+" %}} + +### Receiver +The route needs to refer to a [receiver](#receiver-configuration) that has already been configured. + +### Grouping + +| Field | Default | Description | +|-------|--------------|---------| +| Group By | N/a | The labels by which incoming alerts are grouped together. For example, `[ group_by: '[' , ... ']' ]` Multiple alerts coming in for labels such as `cluster=A` and `alertname=LatencyHigh` can be batched into a single group. To aggregate by all possible labels, use the special value `'...'` as the sole label name, for example: `group_by: ['...']` Grouping by `...` effectively disables aggregation entirely, passing through all alerts as-is. This is unlikely to be what you want, unless you have a very low alert volume or your upstream notification system performs its own grouping. +| Group Wait | 30s | How long to wait to buffer alerts of the same group before sending initially. | +| Group Interval | 5m | How long to wait before sending an alert that has been added to a group of alerts for which an initial notification has already been sent. | +| Repeat Interval | 4h | How long to wait before re-sending a given alert that has already been sent. | + +### Matching + +The **Match** field refers to a set of equality matchers an alert has to fulfill to match the node. When you add key-value pairs to the Rancher UI, they correspond to the YAML in this format: + +```yaml +match: + [ : , ... ] +``` + +The **Match Regex** field refers to a set of regex-matchers an alert has to fulfill to match the node. When you add key-value pairs in the Rancher UI, they correspond to the YAML in this format: + +```yaml +match_re: + [ : , ... ] +``` + +{{% /tab %}} +{{% tab "Rancher v2.5.0-2.5.3" %}} +The Alertmanager must be configured in YAML, as shown in this [example.](#example-alertmanager-config) +{{% /tab %}} +{{% /tabs %}} + +# Example Alertmanager Config + +To set up notifications via Slack, the following Alertmanager Config YAML can be placed into the `alertmanager.yaml` key of the Alertmanager Config Secret, where the `api_url` should be updated to use your Webhook URL from Slack: + +```yaml +route: + group_by: ['job'] + group_wait: 30s + group_interval: 5m + repeat_interval: 3h + receiver: 'slack-notifications' +receivers: +- name: 'slack-notifications' + slack_configs: + - send_resolved: true + text: '{{ template "slack.rancher.text" . }}' + api_url: +templates: +- /etc/alertmanager/config/*.tmpl +``` \ No newline at end of file diff --git a/content/rancher/v2.x/en/monitoring-alerting/v2.5/configuration/expression/_index.md b/content/rancher/v2.x/en/monitoring-alerting/v2.5/configuration/expression/_index.md new file mode 100644 index 00000000000..9203aea75e0 --- /dev/null +++ b/content/rancher/v2.x/en/monitoring-alerting/v2.5/configuration/expression/_index.md @@ -0,0 +1,432 @@ +--- +title: Prometheus Expressions +weight: 4 +aliases: + - /rancher/v2.x/en/project-admin/tools/monitoring/expression + - /rancher/v2.x/en/cluster-admin/tools/monitoring/expression + - /rancher/v2.x/en/monitoring-alerting/legacy/monitoring/cluster-monitoring/expression +--- + +The PromQL expressions in this doc can be used to configure alerts. + +For more information about querying Prometheus, refer to the official [Prometheus documentation.](https://prometheus.io/docs/prometheus/latest/querying/basics/) + + + +- [Cluster Metrics](#cluster-metrics) + - [Cluster CPU Utilization](#cluster-cpu-utilization) + - [Cluster Load Average](#cluster-load-average) + - [Cluster Memory Utilization](#cluster-memory-utilization) + - [Cluster Disk Utilization](#cluster-disk-utilization) + - [Cluster Disk I/O](#cluster-disk-i-o) + - [Cluster Network Packets](#cluster-network-packets) + - [Cluster Network I/O](#cluster-network-i-o) +- [Node Metrics](#node-metrics) + - [Node CPU Utilization](#node-cpu-utilization) + - [Node Load Average](#node-load-average) + - [Node Memory Utilization](#node-memory-utilization) + - [Node Disk Utilization](#node-disk-utilization) + - [Node Disk I/O](#node-disk-i-o) + - [Node Network Packets](#node-network-packets) + - [Node Network I/O](#node-network-i-o) +- [Etcd Metrics](#etcd-metrics) + - [Etcd Has a Leader](#etcd-has-a-leader) + - [Number of Times the Leader Changes](#number-of-times-the-leader-changes) + - [Number of Failed Proposals](#number-of-failed-proposals) + - [GRPC Client Traffic](#grpc-client-traffic) + - [Peer Traffic](#peer-traffic) + - [DB Size](#db-size) + - [Active Streams](#active-streams) + - [Raft Proposals](#raft-proposals) + - [RPC Rate](#rpc-rate) + - [Disk Operations](#disk-operations) + - [Disk Sync Duration](#disk-sync-duration) +- [Kubernetes Components Metrics](#kubernetes-components-metrics) + - [API Server Request Latency](#api-server-request-latency) + - [API Server Request Rate](#api-server-request-rate) + - [Scheduling Failed Pods](#scheduling-failed-pods) + - [Controller Manager Queue Depth](#controller-manager-queue-depth) + - [Scheduler E2E Scheduling Latency](#scheduler-e2e-scheduling-latency) + - [Scheduler Preemption Attempts](#scheduler-preemption-attempts) + - [Ingress Controller Connections](#ingress-controller-connections) + - [Ingress Controller Request Process Time](#ingress-controller-request-process-time) +- [Rancher Logging Metrics](#rancher-logging-metrics) + - [Fluentd Buffer Queue Rate](#fluentd-buffer-queue-rate) + - [Fluentd Input Rate](#fluentd-input-rate) + - [Fluentd Output Errors Rate](#fluentd-output-errors-rate) + - [Fluentd Output Rate](#fluentd-output-rate) +- [Workload Metrics](#workload-metrics) + - [Workload CPU Utilization](#workload-cpu-utilization) + - [Workload Memory Utilization](#workload-memory-utilization) + - [Workload Network Packets](#workload-network-packets) + - [Workload Network I/O](#workload-network-i-o) + - [Workload Disk I/O](#workload-disk-i-o) +- [Pod Metrics](#pod-metrics) + - [Pod CPU Utilization](#pod-cpu-utilization) + - [Pod Memory Utilization](#pod-memory-utilization) + - [Pod Network Packets](#pod-network-packets) + - [Pod Network I/O](#pod-network-i-o) + - [Pod Disk I/O](#pod-disk-i-o) +- [Container Metrics](#container-metrics) + - [Container CPU Utilization](#container-cpu-utilization) + - [Container Memory Utilization](#container-memory-utilization) + - [Container Disk I/O](#container-disk-i-o) + + + +# Cluster Metrics + +### Cluster CPU Utilization + +| Catalog | Expression | +| --- | --- | +| Detail | `1 - (avg(irate(node_cpu_seconds_total{mode="idle"}[5m])) by (instance))` | +| Summary | `1 - (avg(irate(node_cpu_seconds_total{mode="idle"}[5m])))` | + +### Cluster Load Average + +| Catalog | Expression | +| --- | --- | +| Detail |
load1`sum(node_load1) by (instance) / count(node_cpu_seconds_total{mode="system"}) by (instance)`
load5`sum(node_load5) by (instance) / count(node_cpu_seconds_total{mode="system"}) by (instance)`
load15`sum(node_load15) by (instance) / count(node_cpu_seconds_total{mode="system"}) by (instance)`
| +| Summary |
load1`sum(node_load1) by (instance) / count(node_cpu_seconds_total{mode="system"})`
load5`sum(node_load5) by (instance) / count(node_cpu_seconds_total{mode="system"})`
load15`sum(node_load15) by (instance) / count(node_cpu_seconds_total{mode="system"})`
| + +### Cluster Memory Utilization + +| Catalog | Expression | +| --- | --- | +| Detail | `1 - sum(node_memory_MemAvailable_bytes) by (instance) / sum(node_memory_MemTotal_bytes) by (instance)` | +| Summary | `1 - sum(node_memory_MemAvailable_bytes) / sum(node_memory_MemTotal_bytes)` | + +### Cluster Disk Utilization + +| Catalog | Expression | +| --- | --- | +| Detail | `(sum(node_filesystem_size_bytes{device!="rootfs"}) by (instance) - sum(node_filesystem_free_bytes{device!="rootfs"}) by (instance)) / sum(node_filesystem_size_bytes{device!="rootfs"}) by (instance)` | +| Summary | `(sum(node_filesystem_size_bytes{device!="rootfs"}) - sum(node_filesystem_free_bytes{device!="rootfs"})) / sum(node_filesystem_size_bytes{device!="rootfs"})` | + +### Cluster Disk I/O + +| Catalog | Expression | +| --- | --- | +| Detail |
read`sum(rate(node_disk_read_bytes_total[5m])) by (instance)`
written`sum(rate(node_disk_written_bytes_total[5m])) by (instance)`
| +| Summary |
read`sum(rate(node_disk_read_bytes_total[5m]))`
written`sum(rate(node_disk_written_bytes_total[5m]))`
| + +### Cluster Network Packets + +| Catalog | Expression | +| --- | --- | +| Detail |
receive-droppedsum(rate(node_network_receive_drop_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*"}[5m])) by (instance)
receive-errssum(rate(node_network_receive_errs_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*"}[5m])) by (instance)
receive-packetssum(rate(node_network_receive_packets_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*"}[5m])) by (instance)
transmit-droppedsum(rate(node_network_transmit_drop_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*"}[5m])) by (instance)
transmit-errssum(rate(node_network_transmit_errs_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*"}[5m])) by (instance)
transmit-packetssum(rate(node_network_transmit_packets_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*"}[5m])) by (instance)
| +| Summary |
receive-droppedsum(rate(node_network_receive_drop_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*"}[5m]))
receive-errssum(rate(node_network_receive_errs_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*"}[5m]))
receive-packetssum(rate(node_network_receive_packets_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*"}[5m]))
transmit-droppedsum(rate(node_network_transmit_drop_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*"}[5m]))
transmit-errssum(rate(node_network_transmit_errs_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*"}[5m]))
transmit-packetssum(rate(node_network_transmit_packets_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*"}[5m]))
| + +### Cluster Network I/O + +| Catalog | Expression | +| --- | --- | +| Detail |
receivesum(rate(node_network_receive_bytes_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*"}[5m])) by (instance)
transmitsum(rate(node_network_transmit_bytes_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*"}[5m])) by (instance)
| +| Summary |
receivesum(rate(node_network_receive_bytes_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*"}[5m]))
transmitsum(rate(node_network_transmit_bytes_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*"}[5m]))
| + +# Node Metrics + +### Node CPU Utilization + +| Catalog | Expression | +| --- | --- | +| Detail | `avg(irate(node_cpu_seconds_total{mode!="idle", instance=~"$instance"}[5m])) by (mode)` | +| Summary | `1 - (avg(irate(node_cpu_seconds_total{mode="idle", instance=~"$instance"}[5m])))` | + +### Node Load Average + +| Catalog | Expression | +| --- | --- | +| Detail |
load1`sum(node_load1{instance=~"$instance"}) / count(node_cpu_seconds_total{mode="system",instance=~"$instance"})`
load5`sum(node_load5{instance=~"$instance"}) / count(node_cpu_seconds_total{mode="system",instance=~"$instance"})`
load15`sum(node_load15{instance=~"$instance"}) / count(node_cpu_seconds_total{mode="system",instance=~"$instance"})`
| +| Summary |
load1`sum(node_load1{instance=~"$instance"}) / count(node_cpu_seconds_total{mode="system",instance=~"$instance"})`
load5`sum(node_load5{instance=~"$instance"}) / count(node_cpu_seconds_total{mode="system",instance=~"$instance"})`
load15`sum(node_load15{instance=~"$instance"}) / count(node_cpu_seconds_total{mode="system",instance=~"$instance"})`
| + +### Node Memory Utilization + +| Catalog | Expression | +| --- | --- | +| Detail | `1 - sum(node_memory_MemAvailable_bytes{instance=~"$instance"}) / sum(node_memory_MemTotal_bytes{instance=~"$instance"})` | +| Summary | `1 - sum(node_memory_MemAvailable_bytes{instance=~"$instance"}) / sum(node_memory_MemTotal_bytes{instance=~"$instance"}) ` | + +### Node Disk Utilization + +| Catalog | Expression | +| --- | --- | +| Detail | `(sum(node_filesystem_size_bytes{device!="rootfs",instance=~"$instance"}) by (device) - sum(node_filesystem_free_bytes{device!="rootfs",instance=~"$instance"}) by (device)) / sum(node_filesystem_size_bytes{device!="rootfs",instance=~"$instance"}) by (device)` | +| Summary | `(sum(node_filesystem_size_bytes{device!="rootfs",instance=~"$instance"}) - sum(node_filesystem_free_bytes{device!="rootfs",instance=~"$instance"})) / sum(node_filesystem_size_bytes{device!="rootfs",instance=~"$instance"})` | + +### Node Disk I/O + +| Catalog | Expression | +| --- | --- | +| Detail |
read`sum(rate(node_disk_read_bytes_total{instance=~"$instance"}[5m]))`
written`sum(rate(node_disk_written_bytes_total{instance=~"$instance"}[5m]))`
| +| Summary |
read`sum(rate(node_disk_read_bytes_total{instance=~"$instance"}[5m]))`
written`sum(rate(node_disk_written_bytes_total{instance=~"$instance"}[5m]))`
| + +### Node Network Packets + +| Catalog | Expression | +| --- | --- | +| Detail |
receive-droppedsum(rate(node_network_receive_drop_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*",instance=~"$instance"}[5m])) by (device)
receive-errssum(rate(node_network_receive_errs_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*",instance=~"$instance"}[5m])) by (device)
receive-packetssum(rate(node_network_receive_packets_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*",instance=~"$instance"}[5m])) by (device)
transmit-droppedsum(rate(node_network_transmit_drop_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*",instance=~"$instance"}[5m])) by (device)
transmit-errssum(rate(node_network_transmit_errs_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*",instance=~"$instance"}[5m])) by (device)
transmit-packetssum(rate(node_network_transmit_packets_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*",instance=~"$instance"}[5m])) by (device)
| +| Summary |
receive-droppedsum(rate(node_network_receive_drop_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*",instance=~"$instance"}[5m]))
receive-errssum(rate(node_network_receive_errs_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*",instance=~"$instance"}[5m]))
receive-packetssum(rate(node_network_receive_packets_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*",instance=~"$instance"}[5m]))
transmit-droppedsum(rate(node_network_transmit_drop_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*",instance=~"$instance"}[5m]))
transmit-errssum(rate(node_network_transmit_errs_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*",instance=~"$instance"}[5m]))
transmit-packetssum(rate(node_network_transmit_packets_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*",instance=~"$instance"}[5m]))
| + +### Node Network I/O + +| Catalog | Expression | +| --- | --- | +| Detail |
receivesum(rate(node_network_receive_bytes_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*",instance=~"$instance"}[5m])) by (device)
transmitsum(rate(node_network_transmit_bytes_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*",instance=~"$instance"}[5m])) by (device)
| +| Summary |
receivesum(rate(node_network_receive_bytes_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*",instance=~"$instance"}[5m]))
transmitsum(rate(node_network_transmit_bytes_total{device!~"lo | veth.* | docker.* | flannel.* | cali.* | cbr.*",instance=~"$instance"}[5m]))
| + +# Etcd Metrics + +### Etcd Has a Leader + +`max(etcd_server_has_leader)` + +### Number of Times the Leader Changes + +`max(etcd_server_leader_changes_seen_total)` + +### Number of Failed Proposals + +`sum(etcd_server_proposals_failed_total)` + +### GRPC Client Traffic + +| Catalog | Expression | +| --- | --- | +| Detail |
in`sum(rate(etcd_network_client_grpc_received_bytes_total[5m])) by (instance)`
out`sum(rate(etcd_network_client_grpc_sent_bytes_total[5m])) by (instance)`
| +| Summary |
in`sum(rate(etcd_network_client_grpc_received_bytes_total[5m]))`
out`sum(rate(etcd_network_client_grpc_sent_bytes_total[5m]))`
| + +### Peer Traffic + +| Catalog | Expression | +| --- | --- | +| Detail |
in`sum(rate(etcd_network_peer_received_bytes_total[5m])) by (instance)`
out`sum(rate(etcd_network_peer_sent_bytes_total[5m])) by (instance)`
| +| Summary |
in`sum(rate(etcd_network_peer_received_bytes_total[5m]))`
out`sum(rate(etcd_network_peer_sent_bytes_total[5m]))`
| + +### DB Size + +| Catalog | Expression | +| --- | --- | +| Detail | `sum(etcd_debugging_mvcc_db_total_size_in_bytes) by (instance)` | +| Summary | `sum(etcd_debugging_mvcc_db_total_size_in_bytes)` | + +### Active Streams + +| Catalog | Expression | +| --- | --- | +| Detail |
lease-watch`sum(grpc_server_started_total{grpc_service="etcdserverpb.Lease",grpc_type="bidi_stream"}) by (instance) - sum(grpc_server_handled_total{grpc_service="etcdserverpb.Lease",grpc_type="bidi_stream"}) by (instance)`
watch`sum(grpc_server_started_total{grpc_service="etcdserverpb.Watch",grpc_type="bidi_stream"}) by (instance) - sum(grpc_server_handled_total{grpc_service="etcdserverpb.Watch",grpc_type="bidi_stream"}) by (instance)`
| +| Summary |
lease-watch`sum(grpc_server_started_total{grpc_service="etcdserverpb.Lease",grpc_type="bidi_stream"}) - sum(grpc_server_handled_total{grpc_service="etcdserverpb.Lease",grpc_type="bidi_stream"})`
watch`sum(grpc_server_started_total{grpc_service="etcdserverpb.Watch",grpc_type="bidi_stream"}) - sum(grpc_server_handled_total{grpc_service="etcdserverpb.Watch",grpc_type="bidi_stream"})`
| + +### Raft Proposals + +| Catalog | Expression | +| --- | --- | +| Detail |
applied`sum(increase(etcd_server_proposals_applied_total[5m])) by (instance)`
committed`sum(increase(etcd_server_proposals_committed_total[5m])) by (instance)`
pending`sum(increase(etcd_server_proposals_pending[5m])) by (instance)`
failed`sum(increase(etcd_server_proposals_failed_total[5m])) by (instance)`
| +| Summary |
applied`sum(increase(etcd_server_proposals_applied_total[5m]))`
committed`sum(increase(etcd_server_proposals_committed_total[5m]))`
pending`sum(increase(etcd_server_proposals_pending[5m]))`
failed`sum(increase(etcd_server_proposals_failed_total[5m]))`
| + +### RPC Rate + +| Catalog | Expression | +| --- | --- | +| Detail |
total`sum(rate(grpc_server_started_total{grpc_type="unary"}[5m])) by (instance)`
fail`sum(rate(grpc_server_handled_total{grpc_type="unary",grpc_code!="OK"}[5m])) by (instance)`
| +| Summary |
total`sum(rate(grpc_server_started_total{grpc_type="unary"}[5m]))`
fail`sum(rate(grpc_server_handled_total{grpc_type="unary",grpc_code!="OK"}[5m]))`
| + +### Disk Operations + +| Catalog | Expression | +| --- | --- | +| Detail |
commit-called-by-backend`sum(rate(etcd_disk_backend_commit_duration_seconds_sum[1m])) by (instance)`
fsync-called-by-wal`sum(rate(etcd_disk_wal_fsync_duration_seconds_sum[1m])) by (instance)`
| +| Summary |
commit-called-by-backend`sum(rate(etcd_disk_backend_commit_duration_seconds_sum[1m]))`
fsync-called-by-wal`sum(rate(etcd_disk_wal_fsync_duration_seconds_sum[1m]))`
| + +### Disk Sync Duration + +| Catalog | Expression | +| --- | --- | +| Detail |
wal`histogram_quantile(0.99, sum(rate(etcd_disk_wal_fsync_duration_seconds_bucket[5m])) by (instance, le))`
db`histogram_quantile(0.99, sum(rate(etcd_disk_backend_commit_duration_seconds_bucket[5m])) by (instance, le))`
| +| Summary |
wal`sum(histogram_quantile(0.99, sum(rate(etcd_disk_wal_fsync_duration_seconds_bucket[5m])) by (instance, le)))`
db`sum(histogram_quantile(0.99, sum(rate(etcd_disk_backend_commit_duration_seconds_bucket[5m])) by (instance, le)))`
| + +# Kubernetes Components Metrics + +### API Server Request Latency + +| Catalog | Expression | +| --- | --- | +| Detail | `avg(apiserver_request_latencies_sum / apiserver_request_latencies_count) by (instance, verb) /1e+06` | +| Summary | `avg(apiserver_request_latencies_sum / apiserver_request_latencies_count) by (instance) /1e+06` | + +### API Server Request Rate + +| Catalog | Expression | +| --- | --- | +| Detail | `sum(rate(apiserver_request_count[5m])) by (instance, code)` | +| Summary | `sum(rate(apiserver_request_count[5m])) by (instance)` | + +### Scheduling Failed Pods + +| Catalog | Expression | +| --- | --- | +| Detail | `sum(kube_pod_status_scheduled{condition="false"})` | +| Summary | `sum(kube_pod_status_scheduled{condition="false"})` | + +### Controller Manager Queue Depth + +| Catalog | Expression | +| --- | --- | +| Detail |
volumes`sum(volumes_depth) by instance`
deployment`sum(deployment_depth) by instance`
replicaset`sum(replicaset_depth) by instance`
service`sum(service_depth) by instance`
serviceaccount`sum(serviceaccount_depth) by instance`
endpoint`sum(endpoint_depth) by instance`
daemonset`sum(daemonset_depth) by instance`
statefulset`sum(statefulset_depth) by instance`
replicationmanager`sum(replicationmanager_depth) by instance`
| +| Summary |
volumes`sum(volumes_depth)`
deployment`sum(deployment_depth)`
replicaset`sum(replicaset_depth)`
service`sum(service_depth)`
serviceaccount`sum(serviceaccount_depth)`
endpoint`sum(endpoint_depth)`
daemonset`sum(daemonset_depth)`
statefulset`sum(statefulset_depth)`
replicationmanager`sum(replicationmanager_depth)`
| + +### Scheduler E2E Scheduling Latency + +| Catalog | Expression | +| --- | --- | +| Detail | `histogram_quantile(0.99, sum(scheduler_e2e_scheduling_latency_microseconds_bucket) by (le, instance)) / 1e+06` | +| Summary | `sum(histogram_quantile(0.99, sum(scheduler_e2e_scheduling_latency_microseconds_bucket) by (le, instance)) / 1e+06)` | + +### Scheduler Preemption Attempts + +| Catalog | Expression | +| --- | --- | +| Detail | `sum(rate(scheduler_total_preemption_attempts[5m])) by (instance)` | +| Summary | `sum(rate(scheduler_total_preemption_attempts[5m]))` | + +### Ingress Controller Connections + +| Catalog | Expression | +| --- | --- | +| Detail |
reading`sum(nginx_ingress_controller_nginx_process_connections{state="reading"}) by (instance)`
waiting`sum(nginx_ingress_controller_nginx_process_connections{state="waiting"}) by (instance)`
writing`sum(nginx_ingress_controller_nginx_process_connections{state="writing"}) by (instance)`
accepted`sum(ceil(increase(nginx_ingress_controller_nginx_process_connections_total{state="accepted"}[5m]))) by (instance)`
active`sum(ceil(increase(nginx_ingress_controller_nginx_process_connections_total{state="active"}[5m]))) by (instance)`
handled`sum(ceil(increase(nginx_ingress_controller_nginx_process_connections_total{state="handled"}[5m]))) by (instance)`
| +| Summary |
reading`sum(nginx_ingress_controller_nginx_process_connections{state="reading"})`
waiting`sum(nginx_ingress_controller_nginx_process_connections{state="waiting"})`
writing`sum(nginx_ingress_controller_nginx_process_connections{state="writing"})`
accepted`sum(ceil(increase(nginx_ingress_controller_nginx_process_connections_total{state="accepted"}[5m])))`
active`sum(ceil(increase(nginx_ingress_controller_nginx_process_connections_total{state="active"}[5m])))`
handled`sum(ceil(increase(nginx_ingress_controller_nginx_process_connections_total{state="handled"}[5m])))`
| + +### Ingress Controller Request Process Time + +| Catalog | Expression | +| --- | --- | +| Detail | `topk(10, histogram_quantile(0.95,sum by (le, host, path)(rate(nginx_ingress_controller_request_duration_seconds_bucket{host!="_"}[5m]))))` | +| Summary | `topk(10, histogram_quantile(0.95,sum by (le, host)(rate(nginx_ingress_controller_request_duration_seconds_bucket{host!="_"}[5m]))))` | + +# Rancher Logging Metrics + + +### Fluentd Buffer Queue Rate + +| Catalog | Expression | +| --- | --- | +| Detail | `sum(rate(fluentd_output_status_buffer_queue_length[5m])) by (instance)` | +| Summary | `sum(rate(fluentd_output_status_buffer_queue_length[5m]))` | + +### Fluentd Input Rate + +| Catalog | Expression | +| --- | --- | +| Detail | `sum(rate(fluentd_input_status_num_records_total[5m])) by (instance)` | +| Summary | `sum(rate(fluentd_input_status_num_records_total[5m]))` | + +### Fluentd Output Errors Rate + +| Catalog | Expression | +| --- | --- | +| Detail | `sum(rate(fluentd_output_status_num_errors[5m])) by (type)` | +| Summary | `sum(rate(fluentd_output_status_num_errors[5m]))` | + +### Fluentd Output Rate + +| Catalog | Expression | +| --- | --- | +| Detail | `sum(rate(fluentd_output_status_num_records_total[5m])) by (instance)` | +| Summary | `sum(rate(fluentd_output_status_num_records_total[5m]))` | + +# Workload Metrics + +### Workload CPU Utilization + +| Catalog | Expression | +| --- | --- | +| Detail |
cfs throttled seconds`sum(rate(container_cpu_cfs_throttled_seconds_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`
user seconds`sum(rate(container_cpu_user_seconds_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`
system seconds`sum(rate(container_cpu_system_seconds_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`
usage seconds`sum(rate(container_cpu_usage_seconds_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`
| +| Summary |
cfs throttled seconds`sum(rate(container_cpu_cfs_throttled_seconds_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`
user seconds`sum(rate(container_cpu_user_seconds_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`
system seconds`sum(rate(container_cpu_system_seconds_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`
usage seconds`sum(rate(container_cpu_usage_seconds_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`
| + +### Workload Memory Utilization + +| Catalog | Expression | +| --- | --- | +| Detail | `sum(container_memory_working_set_bytes{namespace="$namespace",pod_name=~"$podName", container_name!=""}) by (pod_name)` | +| Summary | `sum(container_memory_working_set_bytes{namespace="$namespace",pod_name=~"$podName", container_name!=""})` | + +### Workload Network Packets + +| Catalog | Expression | +| --- | --- | +| Detail |
receive-packets`sum(rate(container_network_receive_packets_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`
receive-dropped`sum(rate(container_network_receive_packets_dropped_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`
receive-errors`sum(rate(container_network_receive_errors_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`
transmit-packets`sum(rate(container_network_transmit_packets_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`
transmit-dropped`sum(rate(container_network_transmit_packets_dropped_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`
transmit-errors`sum(rate(container_network_transmit_errors_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`
| +| Summary |
receive-packets`sum(rate(container_network_receive_packets_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`
receive-dropped`sum(rate(container_network_receive_packets_dropped_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`
receive-errors`sum(rate(container_network_receive_errors_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`
transmit-packets`sum(rate(container_network_transmit_packets_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`
transmit-dropped`sum(rate(container_network_transmit_packets_dropped_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`
transmit-errors`sum(rate(container_network_transmit_errors_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`
| + +### Workload Network I/O + +| Catalog | Expression | +| --- | --- | +| Detail |
receive`sum(rate(container_network_receive_bytes_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`
transmit`sum(rate(container_network_transmit_bytes_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`
| +| Summary |
receive`sum(rate(container_network_receive_bytes_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`
transmit`sum(rate(container_network_transmit_bytes_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`
| + +### Workload Disk I/O + +| Catalog | Expression | +| --- | --- | +| Detail |
read`sum(rate(container_fs_reads_bytes_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`
write`sum(rate(container_fs_writes_bytes_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m])) by (pod_name)`
| +| Summary |
read`sum(rate(container_fs_reads_bytes_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`
write`sum(rate(container_fs_writes_bytes_total{namespace="$namespace",pod_name=~"$podName",container_name!=""}[5m]))`
| + +# Pod Metrics + +### Pod CPU Utilization + +| Catalog | Expression | +| --- | --- | +| Detail |
cfs throttled seconds`sum(rate(container_cpu_cfs_throttled_seconds_total{container_name!="POD",namespace="$namespace",pod_name="$podName", container_name!=""}[5m])) by (container_name)`
usage seconds`sum(rate(container_cpu_usage_seconds_total{container_name!="POD",namespace="$namespace",pod_name="$podName", container_name!=""}[5m])) by (container_name)`
system seconds`sum(rate(container_cpu_system_seconds_total{container_name!="POD",namespace="$namespace",pod_name="$podName", container_name!=""}[5m])) by (container_name)`
user seconds`sum(rate(container_cpu_user_seconds_total{container_name!="POD",namespace="$namespace",pod_name="$podName", container_name!=""}[5m])) by (container_name)`
| +| Summary |
cfs throttled seconds`sum(rate(container_cpu_cfs_throttled_seconds_total{container_name!="POD",namespace="$namespace",pod_name="$podName", container_name!=""}[5m]))`
usage seconds`sum(rate(container_cpu_usage_seconds_total{container_name!="POD",namespace="$namespace",pod_name="$podName", container_name!=""}[5m]))`
system seconds`sum(rate(container_cpu_system_seconds_total{container_name!="POD",namespace="$namespace",pod_name="$podName", container_name!=""}[5m]))`
user seconds`sum(rate(container_cpu_user_seconds_total{container_name!="POD",namespace="$namespace",pod_name="$podName", container_name!=""}[5m]))`
| + +### Pod Memory Utilization + +| Catalog | Expression | +| --- | --- | +| Detail | `sum(container_memory_working_set_bytes{container_name!="POD",namespace="$namespace",pod_name="$podName",container_name!=""}) by (container_name)` | +| Summary | `sum(container_memory_working_set_bytes{container_name!="POD",namespace="$namespace",pod_name="$podName",container_name!=""})` | + +### Pod Network Packets + +| Catalog | Expression | +| --- | --- | +| Detail |
receive-packets`sum(rate(container_network_receive_packets_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`
receive-dropped`sum(rate(container_network_receive_packets_dropped_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`
receive-errors`sum(rate(container_network_receive_errors_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`
transmit-packets`sum(rate(container_network_transmit_packets_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`
transmit-dropped`sum(rate(container_network_transmit_packets_dropped_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`
transmit-errors`sum(rate(container_network_transmit_errors_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`
| +| Summary |
receive-packets`sum(rate(container_network_receive_packets_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`
receive-dropped`sum(rate(container_network_receive_packets_dropped_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`
receive-errors`sum(rate(container_network_receive_errors_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`
transmit-packets`sum(rate(container_network_transmit_packets_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`
transmit-dropped`sum(rate(container_network_transmit_packets_dropped_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`
transmit-errors`sum(rate(container_network_transmit_errors_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`
| + +### Pod Network I/O + +| Catalog | Expression | +| --- | --- | +| Detail |
receive`sum(rate(container_network_receive_bytes_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`
transmit`sum(rate(container_network_transmit_bytes_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`
| +| Summary |
receive`sum(rate(container_network_receive_bytes_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`
transmit`sum(rate(container_network_transmit_bytes_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`
| + +### Pod Disk I/O + +| Catalog | Expression | +| --- | --- | +| Detail |
read`sum(rate(container_fs_reads_bytes_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m])) by (container_name)`
write`sum(rate(container_fs_writes_bytes_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m])) by (container_name)`
| +| Summary |
read`sum(rate(container_fs_reads_bytes_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`
write`sum(rate(container_fs_writes_bytes_total{namespace="$namespace",pod_name="$podName",container_name!=""}[5m]))`
| + +# Container Metrics + +### Container CPU Utilization + +| Catalog | Expression | +| --- | --- | +| cfs throttled seconds | `sum(rate(container_cpu_cfs_throttled_seconds_total{namespace="$namespace",pod_name="$podName",container_name="$containerName"}[5m]))` | +| usage seconds | `sum(rate(container_cpu_usage_seconds_total{namespace="$namespace",pod_name="$podName",container_name="$containerName"}[5m]))` | +| system seconds | `sum(rate(container_cpu_system_seconds_total{namespace="$namespace",pod_name="$podName",container_name="$containerName"}[5m]))` | +| user seconds | `sum(rate(container_cpu_user_seconds_total{namespace="$namespace",pod_name="$podName",container_name="$containerName"}[5m]))` | + +### Container Memory Utilization + +`sum(container_memory_working_set_bytes{namespace="$namespace",pod_name="$podName",container_name="$containerName"})` + +### Container Disk I/O + +| Catalog | Expression | +| --- | --- | +| read | `sum(rate(container_fs_reads_bytes_total{namespace="$namespace",pod_name="$podName",container_name="$containerName"}[5m]))` | +| write | `sum(rate(container_fs_writes_bytes_total{namespace="$namespace",pod_name="$podName",container_name="$containerName"}[5m]))` | diff --git a/content/rancher/v2.x/en/monitoring-alerting/v2.5/configuration/prometheusrules/_index.md b/content/rancher/v2.x/en/monitoring-alerting/v2.5/configuration/prometheusrules/_index.md new file mode 100644 index 00000000000..33077d5b72c --- /dev/null +++ b/content/rancher/v2.x/en/monitoring-alerting/v2.5/configuration/prometheusrules/_index.md @@ -0,0 +1,93 @@ +--- +title: PrometheusRules +weight: 2 +--- + +The PrometheusRules CRD defines a group of Prometheus alerting and/or recording rules. + +- [About PrometheusRule Custom Resources](#about-prometheusrule-custom-resources) +- [Creating PrometheusRules in the Rancher UI](#creating-prometheusrules-in-the-rancher-ui) +- [Configuration](#configuration) + - [Rule Group](#rule-group) + - [Alerting Rules](#alerting-rules) + - [Recording Rules](#recording-rules) + +# About PrometheusRule Custom Resources + +Prometheus rule files are held in PrometheusRule custom resources. + +The PrometheusRule custom resource defines a RuleGroup with your desired rules. Each specifies the following: + +- The name of the new alert or record +- A PromQL (Prometheus query language) expression for the new alert or record +- Labels that should be attached to the alert or record that identify it (e.g. cluster name or severity) +- Annotations that encode any additional important pieces of information that need to be displayed on the notification for an alert (e.g. summary, description, message, runbook URL, etc.). This field is not required for recording rules. + +Alerting rules define alert conditions based on PromQL queries, and recording rules precompute frequently needed or computationally expensive queries at defined intervals. + +For more information on what fields can be specified, please look at the [Prometheus Operator spec.](https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/api.md#prometheusrulespec) + +Use the label selector field `ruleSelector` in the Prometheus object to define the rule files that you want to be mounted into Prometheus. + +For examples, refer to the Prometheus documentation on [recording rules](https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/) and [alerting rules.](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/) + + +# Creating PrometheusRules in the Rancher UI + +_Available as of v2.5.4_ + +> **Prerequisite:** The monitoring application needs to be installed. + +To create rule groups in the Rancher UI, + +1. Click **Cluster Explorer > Monitoring** and click **Prometheus Rules.** +1. Click **Create.** +1. Enter a **Group Name.** +1. Configure the rules. A rule group may contain either alert rules or recording rules, but not both. For help filling out the forms, refer to the configuration options below. +1. Click **Create.** + +**Result:** Alerts can be configured to send notifications to the receiver(s). + +# Configuration + +{{% tabs %}} +{{% tab "Rancher v2.5.4" %}} +Rancher v2.5.4 introduced the capability to configure reducers by filling out forms in the Rancher UI. + + +### Rule Group + +| Field | Description | +|-------|----------------| +| Group Name | The name of the group. Must be unique within a rules file. | +| Override Group Interval | Duration in seconds for how often rules in the group are evaluated. | + + +### Alerting Rules + +[Alerting rules](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/) allow you to define alert conditions based on PromQL (Prometheus expression language) expressions and to send notifications about firing alerts to an external service. + +| Field | Description | +|-------|----------------| +| Alert Name | The name of the alert. Must be a valid label value. | +| Wait to fire for | Duration in seconds. Alerts are considered firing once they have been returned for this long. Alerts which have not yet fired for long enough are considered pending. | +| PromQL Expression | The PromQL expression to evaluate. Every evaluation cycle this is evaluated at the current time, and all resultant time series become pending/firing alerts. For more information, refer to the [Prometheus documentation](https://prometheus.io/docs/prometheus/latest/querying/basics/) or our [example PromQL expressions.](../expression) | +| Labels | Labels to add or overwrite for each alert. | +| Severity | When enabled, labels are attached to the alert or record that identify it by the severity level. | +| Severity Label Value | Critical, warning, or none | + +### Recording Rules + +[Recording rules](https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/#recording-rules) allow you to precompute frequently needed or computationally expensive PromQL (Prometheus expression language) expressions and save their result as a new set of time series. + +| Field | Description | +|-------|----------------| +| Time Series Name | The name of the time series to output to. Must be a valid metric name. | +| PromQL Expression | The PromQL expression to evaluate. Every evaluation cycle this is evaluated at the current time, and the result recorded as a new set of time series with the metric name as given by 'record'. For more information about expressions, refer to the [Prometheus documentation](https://prometheus.io/docs/prometheus/latest/querying/basics/) or our [example PromQL expressions.](../expression) | +| Labels | Labels to add or overwrite before storing the result. | + +{{% /tab %}} +{{% tab "Rancher v2.5.0-v2.5.3" %}} +For Rancher v2.5.0-v2.5.3, PrometheusRules must be configured in YAML. For examples, refer to the Prometheus documentation on [recording rules](https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/) and [alerting rules.](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/) +{{% /tab %}} +{{% /tabs %}} \ No newline at end of file From 89b762608a36b5594d59246700ab39a7fc4a1cec Mon Sep 17 00:00:00 2001 From: Donnie Adams Date: Fri, 4 Dec 2020 10:58:27 -0700 Subject: [PATCH 07/49] Add description for clusters with private-only API Support for clusters with private-only API endpoint has been improved, but there is an extra step for users to get aware of. Therefore, we are adding a short description to the docs. --- .../hosted-kubernetes-clusters/eks/_index.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/content/rancher/v2.x/en/cluster-provisioning/hosted-kubernetes-clusters/eks/_index.md b/content/rancher/v2.x/en/cluster-provisioning/hosted-kubernetes-clusters/eks/_index.md index 80bd0052162..7aec2359fb1 100644 --- a/content/rancher/v2.x/en/cluster-provisioning/hosted-kubernetes-clusters/eks/_index.md +++ b/content/rancher/v2.x/en/cluster-provisioning/hosted-kubernetes-clusters/eks/_index.md @@ -135,6 +135,12 @@ Optional: To encrypt secrets, select or enter a key created in [AWS Key Manageme Configuring Public/Private API access is an advanced use case. For details, refer to the EKS cluster endpoint access control [documentation.](https://docs.aws.amazon.com/eks/latest/userguide/cluster-endpoint.html) +### Private-only API Endpoints + +If you enable private and disable public API endpoint access when creating a cluster, then there is an extra step you must take in order for Rancher to connect to the cluster successfully. In this case, a pop-up will be displayed with instructions on how to complete the setup. Once the cluster is provisioned, you can run the displayed command anywhere you can connect to the cluster's Kubernetes API. + +Note: you can avoid this extra step by enabling both private and public API endpoint access on cluster creation. You can disable public access after the cluster is created and in an active state and Rancher will continue to communicate with the EKS cluster. + ### Public Access Endpoints @@ -564,7 +570,7 @@ UpstreamSpec represents the cluster as it is in EKS and is refreshed on an inter The effective desired state can be thought of as the UpstreamSpec + all non-nil fields in the EKSConfig. This is what is displayed in the UI. -If Rancher and another source attempt to update an EKS cluster at the same time or within the 5 minute refresh window of an update finishing, then it is likely any “managed” fields can be caught in a race condition. For example, a cluster may have PrivateAccess as a managed field. If PrivateAccess is false and then enabled in EKS console, then finishes at 11:01, and then tags are updated from Rancher before 11:05 the value will likely be overwritten. This would also occur if tags were updated while the cluster was processing the update. If the cluster was registered and the PrivateAccess fields was nil then this issue should not occur in the aforementioend case. +If Rancher and another source attempt to update an EKS cluster at the same time or within the 5 minute refresh window of an update finishing, then it is likely any “managed” fields can be caught in a race condition. For example, a cluster may have PrivateAccess as a managed field. If PrivateAccess is false and then enabled in EKS console, then finishes at 11:01, and then tags are updated from Rancher before 11:05 the value will likely be overwritten. This would also occur if tags were updated while the cluster was processing the update. If the cluster was registered and the PrivateAccess fields was nil then this issue should not occur in the aforementioned case. ### Configuring the Refresh Interval From 975af8e99f4d1b7ea36eb2af0dd14feec786d3b6 Mon Sep 17 00:00:00 2001 From: Nick Gerace Date: Mon, 7 Dec 2020 18:23:19 -0500 Subject: [PATCH 08/49] Add default backend section to ingress controllers Add default backend section to ingress controllers. This section provides a guide and brief explanation on the default backend service. --- .../add-ons/ingress-controllers/_index.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/content/rke/latest/en/config-options/add-ons/ingress-controllers/_index.md b/content/rke/latest/en/config-options/add-ons/ingress-controllers/_index.md index 4e32fb33858..ea60f92abe5 100644 --- a/content/rke/latest/en/config-options/add-ons/ingress-controllers/_index.md +++ b/content/rke/latest/en/config-options/add-ons/ingress-controllers/_index.md @@ -52,6 +52,17 @@ ingress: enable-ssl-passthrough: "" ``` +### Disabling NGINX Ingress Default Backend + +As of v0.20.0, you can disable the [default backend service](https://kubernetes.github.io/ingress-nginx/user-guide/default-backend/) for the ingress controller. This is possible because `ingress-nginx` will fall back to a local 404 page, and does not require a backend service. The service can be enabled/disabled with a boolean value. + +```yaml +ingress: + default_backend: false +``` + +> **What happens if the field is omitted?** The value of `default_backend` will default to `true`. This maintains behavior with older versions of `rke`. However, a future version of `rke` will change the default value to `false`. + ## Configuring an NGINX Default Certificate When configuring an ingress object with TLS termination, you must provide it with a certificate used for encryption/decryption. Instead of explicitly defining a certificate each time you configure an ingress, you can set up a custom certificate that's used by default. From 976fe98c43b523af24b2728048f76e8112958029 Mon Sep 17 00:00:00 2001 From: Catherine Luse Date: Wed, 9 Dec 2020 22:29:38 -0700 Subject: [PATCH 09/49] Explain Prometheus alert rule annotations --- .../v2.5/configuration/prometheusrules/_index.md | 1 + 1 file changed, 1 insertion(+) diff --git a/content/rancher/v2.x/en/monitoring-alerting/v2.5/configuration/prometheusrules/_index.md b/content/rancher/v2.x/en/monitoring-alerting/v2.5/configuration/prometheusrules/_index.md index 33077d5b72c..7a479b31c68 100644 --- a/content/rancher/v2.x/en/monitoring-alerting/v2.5/configuration/prometheusrules/_index.md +++ b/content/rancher/v2.x/en/monitoring-alerting/v2.5/configuration/prometheusrules/_index.md @@ -75,6 +75,7 @@ Rancher v2.5.4 introduced the capability to configure reducers by filling out fo | Labels | Labels to add or overwrite for each alert. | | Severity | When enabled, labels are attached to the alert or record that identify it by the severity level. | | Severity Label Value | Critical, warning, or none | +| Annotations | Annotations are a set of informational labels that can be used to store longer additional information, such as alert descriptions or runbook links. A [runbook](https://docs.gitlab.com/ee/user/project/clusters/runbooks/) is a set of documentation about how to handle alerts. The annotation values can be [templated.](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/#templating) | ### Recording Rules From 3ff77a07143357e7773b362b7a6c009df5f87a8e Mon Sep 17 00:00:00 2001 From: Catherine Luse Date: Wed, 9 Dec 2020 23:05:57 -0700 Subject: [PATCH 10/49] Document requirements for RKE cluster with Flatcar Linux nodes --- .../node-requirements/_index.md | 44 ++++++++++++++++--- .../installation/requirements/ports/_index.md | 4 -- 2 files changed, 37 insertions(+), 11 deletions(-) diff --git a/content/rancher/v2.x/en/cluster-provisioning/node-requirements/_index.md b/content/rancher/v2.x/en/cluster-provisioning/node-requirements/_index.md index cffa90400c6..fc9553cfd35 100644 --- a/content/rancher/v2.x/en/cluster-provisioning/node-requirements/_index.md +++ b/content/rancher/v2.x/en/cluster-provisioning/node-requirements/_index.md @@ -1,13 +1,11 @@ --- -title: Node Requirements for User Clusters +title: Node Requirements for Rancher Managed Clusters weight: 1 --- -This page describes the requirements for the nodes where your apps and services will be installed. +This page describes the requirements for the Rancher managed Kubernetes clusters where your apps and services will be installed. These downstream clusters should be separate from the cluster (or single node) running Rancher. -In this section, "user cluster" refers to a cluster running your apps, which should be separate from the cluster (or single node) running Rancher. - -> If Rancher is installed on a high-availability Kubernetes cluster, the Rancher server cluster and user clusters have different requirements. For Rancher installation requirements, refer to the node requirements in the [installation section.]({{}}/rancher/v2.x/en/installation/requirements/) +> If Rancher is installed on a high-availability Kubernetes cluster, the Rancher server cluster and downstream clusters have different requirements. For Rancher installation requirements, refer to the node requirements in the [installation section.]({{}}/rancher/v2.x/en/installation/requirements/) Make sure the nodes for the Rancher server fulfill the following requirements: @@ -28,11 +26,15 @@ If you plan to use ARM64, see [Running on ARM64 (Experimental).]({{}}/r For information on how to install Docker, refer to the official [Docker documentation.](https://docs.docker.com/) +### Oracle Linux and RHEL Derived Linux Nodes + Some distributions of Linux derived from RHEL, including Oracle Linux, may have default firewall rules that block communication with Helm. We recommend disabling firewalld. For Kubernetes 1.19, firewalld must be turned off. -SUSE Linux may have a firewall that blocks all ports by default. In that situation, follow [these steps](#opening-suse-linux-ports) to open the ports needed for adding a host to a custom cluster. +### SUSE Linux Nodes -### Requirements for Windows Nodes +SUSE Linux may have a firewall that blocks all ports by default. In that situation, follow [these steps]({{}}/rancher/v2.x/en/installation/requirements/ports/#opening-suse-linux-ports) to open the ports needed for adding a host to a custom cluster. + +### Windows Nodes _Windows worker nodes can be used as of Rancher v2.3.0_ @@ -40,6 +42,34 @@ Nodes with Windows Server must run Docker Enterprise Edition. Windows nodes can be used for worker nodes only. See [Configuring Custom Clusters for Windows]({{}}/rancher/v2.x/en/cluster-provisioning/rke-clusters/windows-clusters/) +### Flatcar Linux Nodes + +To deploy an RKE Kubernetes cluster using Flatcar Linux (flatcar-linux-stable-2605.6.0) nodes, we recommend the following configuration in the `rancher-cluster.yml`: + +{{% accordion label="click to expand" %}} +```yaml +nodes: + - address: + internal_address: + user: core + role: [etcd, controlplane, worker] + ssh_key_path: + +network: + plugin: calico + options: + calico_flex_volume_plugin_dir: /opt/kubernetes/kubelet-plugins/volume/exec/nodeagent~uds + flannel_backend_type: vxlan + +services: + kube-controller: + extra_args: + flex-volume-plugin-dir: /opt/kubernetes/kubelet-plugins/volume/exec/ +``` +{{% /accordion %}} + + + # Hardware Requirements The hardware requirements for nodes with the `worker` role mostly depend on your workloads. The minimum to run the Kubernetes node components is 1 CPU (core) and 1GB of memory. diff --git a/content/rancher/v2.x/en/installation/requirements/ports/_index.md b/content/rancher/v2.x/en/installation/requirements/ports/_index.md index 196a1504aaf..4a4e74269c9 100644 --- a/content/rancher/v2.x/en/installation/requirements/ports/_index.md +++ b/content/rancher/v2.x/en/installation/requirements/ports/_index.md @@ -213,10 +213,6 @@ When using the [AWS EC2 node driver]({{}}/rancher/v2.x/en/cluster-provi | Custom UDP Rule | UDP | 30000-32767 | 0.0.0.0/0 | Inbound | | All traffic | All | All | 0.0.0.0/0 | Outbound | -### Opening Ports with firewalld - -[Opening Ports with firewalld]({{}}/rancher/v2.x/en/installation/options/firewall) describes how to use firewalld to apply the above rules. - ### Opening SUSE Linux Ports SUSE Linux may have a firewall that blocks all ports by default. To open the ports needed for adding the host to a custom cluster, From 98f2a72a49be8123c7333779a6d35b4594114416 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Steenis Date: Thu, 10 Dec 2020 22:04:51 +0100 Subject: [PATCH 11/49] Update Flatcar requirements --- .../node-requirements/_index.md | 76 ++++++++++++------- content/rke/latest/en/os/_index.md | 47 ++++++++++++ 2 files changed, 95 insertions(+), 28 deletions(-) diff --git a/content/rancher/v2.x/en/cluster-provisioning/node-requirements/_index.md b/content/rancher/v2.x/en/cluster-provisioning/node-requirements/_index.md index fc9553cfd35..2046ce6ed4b 100644 --- a/content/rancher/v2.x/en/cluster-provisioning/node-requirements/_index.md +++ b/content/rancher/v2.x/en/cluster-provisioning/node-requirements/_index.md @@ -34,6 +34,54 @@ Some distributions of Linux derived from RHEL, including Oracle Linux, may have SUSE Linux may have a firewall that blocks all ports by default. In that situation, follow [these steps]({{}}/rancher/v2.x/en/installation/requirements/ports/#opening-suse-linux-ports) to open the ports needed for adding a host to a custom cluster. +### Flatcar Container Linux Nodes + +When [Launching Kubernetes with Rancher]({{}}/rancher/v2.x/en/cluster-provisioning/rke-clusters/) using Flatcar Container Linux nodes, it is required to use the following configuration in the [Cluster Config File]({{}}/rancher/v2.x/en/cluster-provisioning/rke-clusters/options/#cluster-config-file) + +{{% tabs %}} +{{% tab "Canal"%}} + +```yaml +rancher_kubernetes_engine_config: + network: + plugin: canal + options: + canal_flex_volume_plugin_dir: /opt/kubernetes/kubelet-plugins/volume/exec/nodeagent~uds + flannel_backend_type: vxlan + + services: + kube-controller: + extra_args: + flex-volume-plugin-dir: /opt/kubernetes/kubelet-plugins/volume/exec/ +``` +{{% /tab %}} + +{{% tab "Calico"%}} + +```yaml +rancher_kubernetes_engine_config: + network: + plugin: calico + options: + calico_flex_volume_plugin_dir: /opt/kubernetes/kubelet-plugins/volume/exec/nodeagent~uds + flannel_backend_type: vxlan + + services: + kube-controller: + extra_args: + flex-volume-plugin-dir: /opt/kubernetes/kubelet-plugins/volume/exec/ +``` +{{% /tab %}} +{{% /tabs %}} + +It is also required to enable the Docker service, you can enable the Docker service using the following command: + +``` +systemctl enable docker.service +``` + +The Docker service is enabled automatically when using [Node Drivers]({{}}/rancher/v2.x/en/admin-settings/drivers/#node-drivers). + ### Windows Nodes _Windows worker nodes can be used as of Rancher v2.3.0_ @@ -42,34 +90,6 @@ Nodes with Windows Server must run Docker Enterprise Edition. Windows nodes can be used for worker nodes only. See [Configuring Custom Clusters for Windows]({{}}/rancher/v2.x/en/cluster-provisioning/rke-clusters/windows-clusters/) -### Flatcar Linux Nodes - -To deploy an RKE Kubernetes cluster using Flatcar Linux (flatcar-linux-stable-2605.6.0) nodes, we recommend the following configuration in the `rancher-cluster.yml`: - -{{% accordion label="click to expand" %}} -```yaml -nodes: - - address: - internal_address: - user: core - role: [etcd, controlplane, worker] - ssh_key_path: - -network: - plugin: calico - options: - calico_flex_volume_plugin_dir: /opt/kubernetes/kubelet-plugins/volume/exec/nodeagent~uds - flannel_backend_type: vxlan - -services: - kube-controller: - extra_args: - flex-volume-plugin-dir: /opt/kubernetes/kubelet-plugins/volume/exec/ -``` -{{% /accordion %}} - - - # Hardware Requirements The hardware requirements for nodes with the `worker` role mostly depend on your workloads. The minimum to run the Kubernetes node components is 1 CPU (core) and 1GB of memory. diff --git a/content/rke/latest/en/os/_index.md b/content/rke/latest/en/os/_index.md index 7a60fce88ff..2297c05b579 100644 --- a/content/rke/latest/en/os/_index.md +++ b/content/rke/latest/en/os/_index.md @@ -16,6 +16,7 @@ weight: 5 - [OpenSSH version](#openssh-version) - [Creating a Docker Group](#creating-a-docker-group) + - [Flatcar Container Linux](#flatcar-container-linux) - [Software](#software) - [Ports](#ports) @@ -162,6 +163,52 @@ By default, Atomic hosts do not come with a Docker group. You can update the own # chown /var/run/docker.sock ``` +### Flatcar Container Linux + +When using Flatcar Container Linux nodes, it is required to use the following configuration in the cluster configuration file: + +{{% tabs %}} +{{% tab "Canal"%}} + +```yaml +rancher_kubernetes_engine_config: + network: + plugin: canal + options: + canal_flex_volume_plugin_dir: /opt/kubernetes/kubelet-plugins/volume/exec/nodeagent~uds + flannel_backend_type: vxlan + + services: + kube-controller: + extra_args: + flex-volume-plugin-dir: /opt/kubernetes/kubelet-plugins/volume/exec/ +``` +{{% /tab %}} + +{{% tab "Calico"%}} + +```yaml +rancher_kubernetes_engine_config: + network: + plugin: calico + options: + calico_flex_volume_plugin_dir: /opt/kubernetes/kubelet-plugins/volume/exec/nodeagent~uds + flannel_backend_type: vxlan + + services: + kube-controller: + extra_args: + flex-volume-plugin-dir: /opt/kubernetes/kubelet-plugins/volume/exec/ +``` +{{% /tab %}} +{{% /tabs %}} + +It is also required to enable the Docker service, you can enable the Docker service using the following command: + +``` +systemctl enable docker.service +``` + ## Software This section describes the requirements for Docker, Kubernetes, and SSH. From 44e281491eaa7df4a08bce4b4a37f03d1f19ee3f Mon Sep 17 00:00:00 2001 From: Catherine Luse Date: Thu, 10 Dec 2020 15:33:04 -0700 Subject: [PATCH 12/49] Reorganize upgrade docs because RancherD docs were added --- .../backup/docker-backups/_index.md | 18 +- .../backup/rke-backups/_index.md | 48 +-- .../restore/rke-restore/_index.md | 2 + .../chart-options/_index.md | 9 +- .../rollbacks/_index.md | 21 +- .../install-rancher-on-k8s/upgrades/_index.md | 288 ++++++++++++++++++ .../upgrades}/helm2/_index.md | 3 + .../migrating-from-rke-add-on/_index.md | 2 + .../upgrades/namespace-migration/_index.md | 2 + .../rollbacks/_index.md | 6 + .../upgrades/_index.md | 8 +- .../custom-ca-root-certificate/_index.md | 5 +- .../resources/encryption/_index.md | 6 - .../{encryption => }/tls-secrets/_index.md | 1 + .../{encryption => }/tls-settings/_index.md | 11 +- .../upgrading-cert-manager/_index.md | 1 + .../helm-2-instructions/_index.md | 1 + .../installation/upgrades-rollbacks/_index.md | 19 -- .../rollbacks/ha-server-rollbacks/_index.md | 17 -- .../upgrades-rollbacks/upgrades/_index.md | 33 -- .../upgrades-rollbacks/upgrades/ha/_index.md | 231 -------------- layouts/shortcodes/requirements_rollback.html | 15 - 22 files changed, 382 insertions(+), 365 deletions(-) rename content/rancher/v2.x/en/installation/{resources => install-rancher-on-k8s}/chart-options/_index.md (97%) rename content/rancher/v2.x/en/installation/{upgrades-rollbacks => install-rancher-on-k8s}/rollbacks/_index.md (74%) create mode 100644 content/rancher/v2.x/en/installation/install-rancher-on-k8s/upgrades/_index.md rename content/rancher/v2.x/en/installation/{upgrades-rollbacks/upgrades/ha => install-rancher-on-k8s/upgrades}/helm2/_index.md (97%) rename content/rancher/v2.x/en/installation/{upgrades-rollbacks => install-rancher-on-k8s}/upgrades/migrating-from-rke-add-on/_index.md (95%) rename content/rancher/v2.x/en/installation/{upgrades-rollbacks => install-rancher-on-k8s}/upgrades/namespace-migration/_index.md (98%) create mode 100644 content/rancher/v2.x/en/installation/install-rancher-on-linux/rollbacks/_index.md rename content/rancher/v2.x/en/installation/resources/{encryption => }/custom-ca-root-certificate/_index.md (75%) delete mode 100644 content/rancher/v2.x/en/installation/resources/encryption/_index.md rename content/rancher/v2.x/en/installation/resources/{encryption => }/tls-secrets/_index.md (96%) rename content/rancher/v2.x/en/installation/resources/{encryption => }/tls-settings/_index.md (82%) rename content/rancher/v2.x/en/installation/resources/{encryption => }/upgrading-cert-manager/_index.md (99%) rename content/rancher/v2.x/en/installation/resources/{encryption => }/upgrading-cert-manager/helm-2-instructions/_index.md (98%) delete mode 100644 content/rancher/v2.x/en/installation/upgrades-rollbacks/_index.md delete mode 100644 content/rancher/v2.x/en/installation/upgrades-rollbacks/rollbacks/ha-server-rollbacks/_index.md delete mode 100644 content/rancher/v2.x/en/installation/upgrades-rollbacks/upgrades/_index.md delete mode 100644 content/rancher/v2.x/en/installation/upgrades-rollbacks/upgrades/ha/_index.md delete mode 100644 layouts/shortcodes/requirements_rollback.html diff --git a/content/rancher/v2.x/en/backups/v2.0.x-v2.4.x/backup/docker-backups/_index.md b/content/rancher/v2.x/en/backups/v2.0.x-v2.4.x/backup/docker-backups/_index.md index f3707b64917..4f176c7d470 100644 --- a/content/rancher/v2.x/en/backups/v2.0.x-v2.4.x/backup/docker-backups/_index.md +++ b/content/rancher/v2.x/en/backups/v2.0.x-v2.4.x/backup/docker-backups/_index.md @@ -12,17 +12,27 @@ aliases: After completing your Docker installation of Rancher, we recommend creating backups of it on a regular basis. Having a recent backup will let you recover quickly from an unexpected disaster. -## Before You Start +### How to Read Placeholders During the creation of your backup, you'll enter a series of commands, replacing placeholders with data from your environment. These placeholders are denoted with angled brackets and all capital letters (``). Here's an example of a command with a placeholder: ``` -docker run --volumes-from rancher-data- -v $PWD:/backup busybox tar pzcvf /backup/rancher-data-backup--.tar.gz /var/lib/rancher +docker run \ + --volumes-from rancher-data- \ + -v $PWD:/backup busybox tar pzcvf /backup/rancher-data-backup--.tar.gz /var/lib/rancher ``` In this command, `` is a placeholder for the date that the data container and backup were created. `9-27-18` for example. -Cross reference the image and reference table below to learn how to obtain this placeholder data. Write down or copy this information before starting the [procedure below](#creating-a-backup). +### Obtaining Placeholder Data + +Get the placeholder data by running: + +``` +docker ps +``` + +Write down or copy this information before starting the [procedure below](#creating-a-backup). Terminal `docker ps` Command, Displaying Where to Find `` and `` ![Placeholder Reference]({{}}/img/rancher/placeholder-ref.png) @@ -37,7 +47,7 @@ Cross reference the image and reference table below to learn how to obtain this You can obtain `` and `` by logging into your Rancher Server by remote connection and entering the command to view the containers that are running: `docker ps`. You can also view containers that are stopped with `docker ps -a`. Use these commands for help anytime while creating backups. -## Creating a Backup +### Creating a Backup This procedure creates a backup that you can restore if Rancher encounters a disaster scenario. diff --git a/content/rancher/v2.x/en/backups/v2.0.x-v2.4.x/backup/rke-backups/_index.md b/content/rancher/v2.x/en/backups/v2.0.x-v2.4.x/backup/rke-backups/_index.md index f14318ea1d3..7a660a3e2bf 100644 --- a/content/rancher/v2.x/en/backups/v2.0.x-v2.4.x/backup/rke-backups/_index.md +++ b/content/rancher/v2.x/en/backups/v2.0.x-v2.4.x/backup/rke-backups/_index.md @@ -13,16 +13,24 @@ aliases: --- This section describes how to create backups of your high-availability Rancher install. ->**Prerequisites:** {{< requirements_rollback >}} - -## RKE Kubernetes Cluster Data - In an RKE installation, the cluster data is replicated on each of three etcd nodes in the cluster, providing redundancy and data duplication in case one of the nodes fails. -
Architecture of an RKE Kubernetes Cluster Running the Rancher Management Server
+
Cluster Data within an RKE Kubernetes Cluster Running the Rancher Management Server
![Architecture of an RKE Kubernetes cluster running the Rancher management server]({{}}/img/rancher/rke-server-storage.svg) -## Backup Outline +# Requirements + +### RKE Version + +The commands for taking `etcd` snapshots are only available in RKE v0.1.7 and later. + +### RKE Config File + +You'll need the RKE config file that you used for Rancher install, `rancher-cluster.yml`. You created this file during your initial install. Place this file in same directory as the RKE binary. + + +# Backup Outline + Backing up your high-availability Rancher cluster is process that involves completing multiple tasks. @@ -30,13 +38,12 @@ Backing up your high-availability Rancher cluster is process that involves compl Take snapshots of your current `etcd` database using Rancher Kubernetes Engine (RKE). -1. [Store Snapshot(s) Externally](#2-backup-snapshots-to-a-safe-location) +1. [Store Snapshot(s) Externally](#2-back-up-local-snapshots-to-a-safe-location) After taking your snapshots, export them to a safe location that won't be affected if your cluster encounters issues. -
-### 1. Take Snapshots of the `etcd` Database +# 1. Take Snapshots of the `etcd` Database Take snapshots of your `etcd` database. You can use these snapshots later to recover from a disaster scenario. There are two ways to take snapshots: recurringly, or as a one-off. Each option is better suited to a specific use case. Read the short description below each link to know when to use each option. @@ -48,7 +55,7 @@ Take snapshots of your `etcd` database. You can use these snapshots later to rec We advise taking one-time snapshots before events like upgrades or restoration of another snapshot. -#### Option A: Recurring Snapshots +### Option A: Recurring Snapshots For all high-availability Rancher installs, we recommend taking recurring snapshots so that you always have a safe restoration point available. @@ -103,7 +110,7 @@ To take recurring snapshots, enable the `etcd-snapshot` service, which is a serv **Result:** RKE is configured to take recurring snapshots of `etcd` on all nodes running the `etcd` role. Snapshots are saved locally to the following directory: `/opt/rke/etcd-snapshots/`. If configured, the snapshots are also uploaded to your S3 compatible backend. -#### Option B: One-Time Snapshots +### Option B: One-Time Snapshots When you're about to upgrade Rancher or restore it to a previous snapshot, you should snapshot your live image so that you have a backup of `etcd` in its last known state. @@ -114,7 +121,9 @@ When you're about to upgrade Rancher or restore it to a previous snapshot, you s 2. Enter the following command. Replace `` with any name that you want to use for the snapshot (e.g. `upgrade.db`). ``` - rke etcd snapshot-save --name --config rancher-cluster.yml + rke etcd snapshot-save \ + --name \ + --config rancher-cluster.yml ``` **Result:** RKE takes a snapshot of `etcd` running on each `etcd` node. The file is saved to `/opt/rke/etcd-snapshots`. @@ -128,15 +137,20 @@ _Available as of RKE v0.2.0_ 2. Enter the following command. Replace `` with any name that you want to use for the snapshot (e.g. `upgrade.db`). ```shell - rke etcd snapshot-save --config rancher-cluster.yml --name snapshot-name \ - --s3 --access-key S3_ACCESS_KEY --secret-key S3_SECRET_KEY \ - --bucket-name s3-bucket-name --s3-endpoint s3.amazonaws.com \ - --folder folder-name # Available as of v2.3.0 + rke etcd snapshot-save \ + --config rancher-cluster.yml \ + --name snapshot-name \ + --s3 \ + --access-key S3_ACCESS_KEY \ + --secret-key S3_SECRET_KEY \ + --bucket-name s3-bucket-name \ + --s3-endpoint s3.amazonaws.com \ + --folder folder-name # Available as of v2.3.0 ``` **Result:** RKE takes a snapshot of `etcd` running on each `etcd` node. The file is saved to `/opt/rke/etcd-snapshots`. It is also uploaded to the S3 compatible backend. -### 2. Backup Local Snapshots to a Safe Location +# 2. Back up Local Snapshots to a Safe Location > **Note:** If you are using RKE v0.2.0, you can enable saving the backups to a S3 compatible backend directly and skip this step. diff --git a/content/rancher/v2.x/en/backups/v2.0.x-v2.4.x/restore/rke-restore/_index.md b/content/rancher/v2.x/en/backups/v2.0.x-v2.4.x/restore/rke-restore/_index.md index 122006cfef6..65d9d9e07fc 100644 --- a/content/rancher/v2.x/en/backups/v2.0.x-v2.4.x/restore/rke-restore/_index.md +++ b/content/rancher/v2.x/en/backups/v2.0.x-v2.4.x/restore/rke-restore/_index.md @@ -34,6 +34,8 @@ Prepare by creating 3 new nodes to be the target for the restored Rancher instan Alternatively you can re-use the existing nodes after clearing Kubernetes and Rancher configurations. This will destroy the data on these nodes. See [Node Cleanup]({{}}/rancher/v2.x/en/faq/cleaning-cluster-nodes/) for the procedure. +You must restore each of your etcd nodes to the same snapshot. Copy the snapshot you're using from one of your nodes to the others before running the `etcd snapshot-restore` command. + > **IMPORTANT:** Before starting the restore make sure all the Kubernetes services on the old cluster nodes are stopped. We recommend powering off the nodes to be sure. ### 2. Place Snapshot diff --git a/content/rancher/v2.x/en/installation/resources/chart-options/_index.md b/content/rancher/v2.x/en/installation/install-rancher-on-k8s/chart-options/_index.md similarity index 97% rename from content/rancher/v2.x/en/installation/resources/chart-options/_index.md rename to content/rancher/v2.x/en/installation/install-rancher-on-k8s/chart-options/_index.md index 41c18e66db6..09464b6fe66 100644 --- a/content/rancher/v2.x/en/installation/resources/chart-options/_index.md +++ b/content/rancher/v2.x/en/installation/install-rancher-on-k8s/chart-options/_index.md @@ -1,12 +1,19 @@ --- title: Helm Chart Options -weight: 2 +weight: 1 aliases: - /rancher/v2.x/en/installation/options/ - /rancher/v2.x/en/installation/options/chart-options/ - /rancher/v2.x/en/installation/options/helm2/helm-rancher/chart-options/ + - /rancher/v2.x/en/installation/resources/chart-options --- +This page is a configuration reference for the Rancher Helm chart. + +For help choosing a Helm chart version, refer to [this page.]({{}}/rancher/v2.x/en/installation/resources/choosing-version/) + +For information on enabling experimental features, refer to [this page.]({{}}/rancher/v2.x/en/installation/resources/feature-flags/) + - [Common Options](#common-options) - [Advanced Options](#advanced-options) - [API Audit Log](#api-audit-log) diff --git a/content/rancher/v2.x/en/installation/upgrades-rollbacks/rollbacks/_index.md b/content/rancher/v2.x/en/installation/install-rancher-on-k8s/rollbacks/_index.md similarity index 74% rename from content/rancher/v2.x/en/installation/upgrades-rollbacks/rollbacks/_index.md rename to content/rancher/v2.x/en/installation/install-rancher-on-k8s/rollbacks/_index.md index 4af727bf04b..8464b15f5e4 100644 --- a/content/rancher/v2.x/en/installation/upgrades-rollbacks/rollbacks/_index.md +++ b/content/rancher/v2.x/en/installation/install-rancher-on-k8s/rollbacks/_index.md @@ -1,22 +1,24 @@ --- title: Rollbacks -weight: 1010 +weight: 3 aliases: - /rancher/v2.x/en/upgrades/rollbacks + - /rancher/v2.x/en/installation/upgrades-rollbacks/rollbacks + - /rancher/v2.x/en/upgrades/ha-server-rollbacks + - /rancher/v2.x/en/upgrades/rollbacks/ha-server-rollbacks + - /rancher/v2.x/en/installation/upgrades-rollbacks/rollbacks/ha-server-rollbacks + - /rancher/v2.x/en/installation/install-rancher-on-k8s/upgrades-rollbacks/rollbacks --- -This section contains information about how to roll back your Rancher server to a previous version. +To roll back to Rancher v2.5+, use the `rancher-backup` application and restore Rancher from backup according to [this section.]({{}}/rancher/v2.x/en/backups/restoring-rancher/) -If you upgrade Rancher and the upgrade does not complete successfully, you may need to [restore Rancher from backup.](../../backups/restores) +To roll back to Rancher prior to v2.5, follow the procedure detailed here: [Restoring Backups — Kubernetes installs]({{}}/rancher/v2.x/en/backups/restorations/ha-restoration) Restoring a snapshot of the Rancher Server cluster will revert Rancher to the version and state at the time of the snapshot. -Restoring a snapshot of the Rancher Server cluster will revert Rancher to the version and state at the time of the snapshot. +For information on how to roll back Rancher installed with Docker, refer to [this page.]({{}}/rancher/v2.x/en/installation/other-installation-methods/single-node-docker/single-node-rollbacks) ->**Note:** Managed clusters are authoritative for their state. This means restoring the rancher server will not revert workload deployments or changes made on managed clusters after the snapshot was taken. +> Managed clusters are authoritative for their state. This means restoring the rancher server will not revert workload deployments or changes made on managed clusters after the snapshot was taken. -- [Rolling back Rancher installed with Docker]({{}}/rancher/v2.x/en/installation/other-installation-methods/single-node-docker/single-node-rollbacks) -- [Rolling back Rancher installed on a Kubernetes cluster]({{}}/rancher/v2.x/en/upgrades/rollbacks/ha-server-rollbacks/) - -### Special Scenarios regarding Rollbacks +### Rolling back to v2.0.0-v2.1.5 If you are rolling back to versions in either of these scenarios, you must follow some extra instructions in order to get your clusters working. @@ -27,7 +29,6 @@ Because of the changes necessary to address [CVE-2018-20321](https://cve.mitre.o 1. Record the `serviceAccountToken` for each cluster. To do this, save the following script on a machine with `kubectl` access to the Rancher management plane and execute it. You will need to run these commands on the machine where the rancher container is running. Ensure JQ is installed before running the command. The commands will vary depending on how you installed Rancher. - **Rancher Installed with Docker** ``` docker exec kubectl get clusters -o json | jq '[.items[] | select(any(.status.conditions[]; .type == "ServiceAccountMigrated")) | {name: .metadata.name, token: .status.serviceAccountToken}]' > tokens.json diff --git a/content/rancher/v2.x/en/installation/install-rancher-on-k8s/upgrades/_index.md b/content/rancher/v2.x/en/installation/install-rancher-on-k8s/upgrades/_index.md new file mode 100644 index 00000000000..f32a0fccbad --- /dev/null +++ b/content/rancher/v2.x/en/installation/install-rancher-on-k8s/upgrades/_index.md @@ -0,0 +1,288 @@ +--- +title: Upgrades +weight: 2 +aliases: + - /rancher/v2.x/en/upgrades/upgrades + - /rancher/v2.x/en/installation/upgrades-rollbacks/upgrades + - /rancher/v2.x/en/upgrades/upgrades/ha-server-upgrade-helm-airgap + - /rancher/v2.x/en/upgrades/air-gap-upgrade/ + - /rancher/v2.x/en/upgrades/upgrades/ha + - /rancher/v2.x/en/installation/install-rancher-on-k8s/upgrades/upgrades/ha + - /rancher/v2.x/en/installation/upgrades-rollbacks/upgrades/ + - /rancher/v2.x/en/upgrades/upgrades/ha-server-upgrade-helm/ + - /rancher/v2.x/en/installation/upgrades-rollbacks/upgrades/ha + - /rancher/v2.x/en/installation/install-rancher-on-k8s/upgrades-rollbacks/upgrades +--- +The following instructions will guide you through upgrading a Rancher server that was installed on a Kubernetes cluster with Helm. These steps also apply to air gap installs with Helm. + +For the instructions to upgrade Rancher installed on Kubernetes with RancherD, refer to [this page.]({{}}/rancher/v2.x/en/installation/install-rancher-on-linux/upgrades) + +For the instructions to upgrade Rancher installed with Docker, refer to [ths page.]({{}}/rancher/v2.x/en/installation/other-installation-methods/single-node-docker/single-node-upgrades) + +To upgrade the components in your Kubernetes cluster, or the definition of the [Kubernetes services]({{}}/rke/latest/en/config-options/services/) or [add-ons]({{}}/rke/latest/en/config-options/add-ons/), refer to the [upgrade documentation for RKE]({{}}/rke/latest/en/upgrades/), the Rancher Kubernetes Engine. + +If you installed Rancher using the RKE Add-on yaml, follow the directions to [migrate or upgrade]({{}}/rancher/v2.x/en/upgrades/upgrades/migrating-from-rke-add-on). + +- [Prerequisites](#prerequisites) +- [Upgrade Outline](#upgrade-outline) +- [Known Upgrade Issues](#known-upgrade-issues) +- [RKE Add-on Installs](#rke-add-on-installs) + +# Prerequisites + +### Access to kubeconfig + +Helm should be run from the same location as your kubeconfig file, or the same location where you run your kubectl commands from. + +If you installed Kubernetes with RKE, the config will have been created in the directory you ran `rke up` in. + +The kubeconfig can also be manually targeted for the intended cluster with the `--kubeconfig` tag (see: https://helm.sh/docs/helm/helm/) + +### Review Known Issues + +Review the [known upgrade issues](#known-upgrade-issues) in the Rancher documentation for the most noteworthy issues to consider when upgrading Rancher. + +A more complete list of known issues for each Rancher version can be found in the release notes on [GitHub](https://github.com/rancher/rancher/releases) and on the [Rancher forums.](https://forums.rancher.com/c/announcements/12) + +Note that upgrades _to_ or _from_ any chart in the [rancher-alpha repository]({{}}/rancher/v2.x/en/installation/resources/chart-options/#helm-chart-repositories/) aren't supported. + +### Helm Version + +The upgrade instructions assume you are using Helm 3. + +For migration of installs started with Helm 2, refer to the official [Helm 2 to 3 migration docs.](https://helm.sh/blog/migrate-from-helm-v2-to-helm-v3/) The [Helm 2 upgrade page here]({{}}/rancher/v2.x/en/installation/upgrades-rollbacks/upgrades/ha/helm2)provides a copy of the older upgrade instructions that used Helm 2, and it is intended to be used if upgrading to Helm 3 is not feasible. + +### For air gap installs: Populate private registry + +-For [air gap installs only,]({{}}/rancher/v2.x/en/installation/other-installation-methods/air-gap) collect and populate images for the new Rancher server version. Follow the guide to [populate your private registry]({{}}/rancher/v2.x/en/installation/other-installation-methods/air-gap/populate-private-registry/) with the images for the Rancher version that you want to upgrade to. + +### For upgrades from a Rancher server with a hidden local cluster + +If you are upgrading to Rancher v2.5 from a Rancher server that was started with the Helm chart option `--add-local=false`, you will need to drop that flag when upgrading. Otherwise, the Rancher server will not start. The `restricted-admin` role can be used to continue restricting access to the local cluster. For more information, see [this section.]({{}}/rancher/v2.x/en/admin-settings/rbac/global-permissions/#upgrading-from-rancher-with-a-hidden-local-cluster) + +### For upgrades from v2.0-v2.2 with external TLS termination + +If you are upgrading Rancher from v2.x to v2.3+, and you are using external TLS termination, you will need to edit the cluster.yml to [enable using forwarded host headers.]({{}}/rancher/v2.x/en/installation/resources/chart-options/#configuring-ingress-for-external-tls-when-using-nginx-v0-25) + +### For upgrades with cert-manager older than 0.8.0 + +[Let's Encrypt will be blocking cert-manager instances older than 0.8.0 starting November 1st 2019.](https://community.letsencrypt.org/t/blocking-old-cert-manager-versions/98753) Upgrade cert-manager to the latest version by following [these instructions.]({{}}/rancher/v2.x/en/installation/options/upgrading-cert-manager) + +# Upgrade Outline + +Follow the steps to upgrade Rancher server: + +- [1. Back up your Kubernetes cluster that is running Rancher server](#1-back-up-your-kubernetes-cluster-that-is-running-rancher-server) +- [2. Update the Helm chart repository](#2-update-the-helm-chart-repository) +- [3. Upgrade Rancher](#3-upgrade-rancher) +- [4. Verify the Upgrade](#4-verify-the-upgrade) + +# 1. Back up Your Kubernetes Cluster that is Running Rancher Server + +For Rancher v2.5+, use the [backup application]({{}}/rancher/v2.x/en/backups/v2.5/back-up-rancher) to back up Rancher. + +For Rancher v2.0-v2.4, [take a one-time snapshot]({{}}/rancher/v2.x/en/backups/backups/ha-backups/#option-b-one-time-snapshots) +of your Kubernetes cluster running Rancher server. + +You'll use the backup as a restoration point if something goes wrong during upgrade. + +# 2. Update the Helm chart repository + +1. Update your local helm repo cache. + + ``` + helm repo update + ``` + +1. Get the repository name that you used to install Rancher. + + For information about the repos and their differences, see [Helm Chart Repositories]({{}}/rancher/v2.x/en/installation/resources/chart-options/#helm-chart-repositories). + + {{< release-channel >}} + + ``` + helm repo list + + NAME URL + stable https://kubernetes-charts.storage.googleapis.com + rancher- https://releases.rancher.com/server-charts/ + ``` + + > **Note:** If you want to switch to a different Helm chart repository, please follow the [steps on how to switch repositories]({{}}/rancher/v2.x/en/installation/resources/chart-options/#switching-to-a-different-helm-chart-repository). If you switch repositories, make sure to list the repositories again before continuing onto Step 3 to ensure you have the correct one added. + + +1. Fetch the latest chart to install Rancher from the Helm chart repository. + + This command will pull down the latest charts and save it in the current directory as a `.tgz` file. + + ```plain + helm fetch rancher-/rancher + ``` + You can fetch the chart for the specific version you are upgrading to by adding in the `--version=` tag. For example: + + ```plain + helm fetch rancher-/rancher --version=v2.4.11 + ``` + +# 3. Upgrade Rancher + +This section describes how to upgrade normal (Internet-connected) or air gap installations of Rancher with Helm. + +{{% tabs %}} +{{% tab "Kubernetes Upgrade" %}} + +Get the values, which were passed with `--set`, from the current Rancher Helm chart that is installed. + +``` +helm get values rancher -n cattle-system + +hostname: rancher.my.org +``` + +> **Note:** There will be more values that are listed with this command. This is just an example of one of the values. + +If you are also upgrading cert-manager to the latest version from a version older than 0.11.0, follow [Option B: Reinstalling Rancher and cert-manager.](#option-b-reinstalling-rancher-and-cert-manager) + +Otherwise, follow [Option A: Upgrading Rancher.](#option-a-upgrading-rancher) + +### Option A: Upgrading Rancher + +Upgrade Rancher to the latest version with all your settings. + +Take all the values from the previous step and append them to the command using `--set key=value`: + +``` +helm upgrade rancher rancher-/rancher \ + --namespace cattle-system \ + --set hostname=rancher.my.org +``` + +> **Note:** The above is an example, there may be more values from the previous step that need to be appended. + +Alternatively, it's possible to export the current values to a file and reference that file during upgrade. For example, to only change the Rancher version: + +``` +helm get values rancher -n cattle-system -o yaml > values.yaml + +helm upgrade rancher rancher-/rancher \ + --namespace cattle-system \ + -f values.yaml \ + --version=2.4.5 +``` + +### Option B: Reinstalling Rancher and cert-manager + +If you are currently running the cert-manger whose version is older than v0.11, and want to upgrade both Rancher and cert-manager to a newer version, then you need to reinstall both Rancher and cert-manger due to the API change in cert-manger v0.11. + +1. Uninstall Rancher + + ``` + helm delete rancher -n cattle-system + ``` + +2. Uninstall and reinstall `cert-manager` according to the instructions on the [Upgrading Cert-Manager]({{}}/rancher/v2.x/en/installation/options/upgrading-cert-manager) page. + +3. Reinstall Rancher to the latest version with all your settings. Take all the values from the step 1 and append them to the command using `--set key=value`. Note: There will be many more options from the step 1 that need to be appended. + + ``` + helm install rancher rancher-/rancher \ + --namespace cattle-system \ + --set hostname=rancher.my.org + ``` + +{{% /tab %}} + +{{% tab "Kubernetes Air Gap Upgrade" %}} + +Render the Rancher template using the same chosen options that were used when installing Rancher. Use the reference table below to replace each placeholder. Rancher needs to be configured to use the private registry in order to provision any Rancher launched Kubernetes clusters or Rancher tools. + +Based on the choice you made during installation, complete one of the procedures below. + +Placeholder | Description +------------|------------- +`` | The version number of the output tarball. +`` | The DNS name you pointed at your load balancer. +`` | The DNS name for your private registry. +`` | Cert-manager version running on k8s cluster. + + +### Option A: Default Self-signed Certificate + + ```plain +helm template ./rancher-.tgz --output-dir . \ + --name rancher \ + --namespace cattle-system \ + --set hostname= \ + --set certmanager.version= \ + --set rancherImage=/rancher/rancher \ + --set systemDefaultRegistry= \ # Available as of v2.2.0, set a default private registry to be used in Rancher + --set useBundledSystemChart=true # Available as of v2.3.0, use the packaged Rancher system charts +``` + +### Option B: Certificates from Files using Kubernetes Secrets + +```plain +helm template ./rancher-.tgz --output-dir . \ +--name rancher \ +--namespace cattle-system \ +--set hostname= \ +--set rancherImage=/rancher/rancher \ +--set ingress.tls.source=secret \ +--set systemDefaultRegistry= \ # Available as of v2.2.0, set a default private registry to be used in Rancher +--set useBundledSystemChart=true # Available as of v2.3.0, use the packaged Rancher system charts +``` + +If you are using a Private CA signed cert, add `--set privateCA=true` following `--set ingress.tls.source=secret`: + +```plain +helm template ./rancher-.tgz --output-dir . \ +--name rancher \ +--namespace cattle-system \ +--set hostname= \ +--set rancherImage=/rancher/rancher \ +--set ingress.tls.source=secret \ +--set privateCA=true \ +--set systemDefaultRegistry= \ # Available as of v2.2.0, set a default private registry to be used in Rancher +--set useBundledSystemChart=true # Available as of v2.3.0, use the packaged Rancher system charts +``` + +### Apply the Rendered Templates + +Copy the rendered manifest directories to a system with access to the Rancher server cluster and apply the rendered templates. + +Use `kubectl` to apply the rendered manifests. + +```plain +kubectl -n cattle-system apply -R -f ./rancher +``` + +{{% /tab %}} +{{% /tabs %}} + +# 4. Verify the Upgrade + +Log into Rancher to confirm that the upgrade succeeded. + +>**Having network issues following upgrade?** +> +> See [Restoring Cluster Networking]({{}}/rancher/v2.x/en/installation/install-rancher-on-k8s/upgrades/namespace-migration/#restoring-cluster-networking). + +# Known Upgrade Issues + +The following table lists some of the most noteworthy issues to be considered when upgrading Rancher. A more complete list of known issues for each Rancher version can be found in the release notes on [GitHub](https://github.com/rancher/rancher/releases) and on the [Rancher forums.](https://forums.rancher.com/c/announcements/12) + +Upgrade Scenario | Issue +---|--- +Upgrading to v2.4.6 or v2.4.7 | These Rancher versions had an issue where the `kms:ListKeys` permission was required to create, edit, or clone Amazon EC2 node templates. This requirement was removed in v2.4.8. +Upgrading to v2.3.0+ | Any user provisioned cluster will be automatically updated upon any edit as tolerations were added to the images used for Kubernetes provisioning. +Upgrading to v2.2.0-v2.2.x | Rancher introduced the [system charts](https://github.com/rancher/system-charts) repository which contains all the catalog items required for features such as monitoring, logging, alerting and global DNS. To be able to use these features in an air gap install, you will need to mirror the `system-charts` repository locally and configure Rancher to use that repository. Please follow the instructions to [configure Rancher system charts]({{}}/rancher/v2.x/en/installation/resources/local-system-charts/#setting-up-system-charts-for-rancher-prior-to-v2-3-0). +Upgrading from v2.0.13 or earlier | If your cluster's certificates have expired, you will need to perform [additional steps]({{}}/rancher/v2.x/en/cluster-admin/certificate-rotation/#rotating-expired-certificates-after-upgrading-older-rancher-versions) to rotate the certificates. +Upgrading from v2.0.7 or earlier | Rancher introduced the `system` project, which is a project that's automatically created to store important namespaces that Kubernetes needs to operate. During upgrade to v2.0.7+, Rancher expects these namespaces to be unassigned from all projects. Before beginning upgrade, check your system namespaces to make sure that they're unassigned to [prevent cluster networking issues]({{}}/rancher/v2.x/en/upgrades/upgrades/namespace-migration/#preventing-cluster-networking-issues). + +# RKE Add-on Installs + +**Important: RKE add-on install is only supported up to Rancher v2.0.8** + +Please use the Rancher helm chart to install Rancher on a Kubernetes cluster. For details, see the [Kubernetes Install - Installation Outline]({{}}/rancher/v2.x/en/installation/install-rancher-on-k8s/#installation-outline). + +If you are currently using the RKE add-on install method, see [Migrating from a RKE add-on install]({{}}/rancher/v2.x/en/upgrades/upgrades/migrating-from-rke-add-on/) for details on how to move to using the helm chart. diff --git a/content/rancher/v2.x/en/installation/upgrades-rollbacks/upgrades/ha/helm2/_index.md b/content/rancher/v2.x/en/installation/install-rancher-on-k8s/upgrades/helm2/_index.md similarity index 97% rename from content/rancher/v2.x/en/installation/upgrades-rollbacks/upgrades/ha/helm2/_index.md rename to content/rancher/v2.x/en/installation/install-rancher-on-k8s/upgrades/helm2/_index.md index 32d387a69e3..fcf2b346c4b 100644 --- a/content/rancher/v2.x/en/installation/upgrades-rollbacks/upgrades/ha/helm2/_index.md +++ b/content/rancher/v2.x/en/installation/install-rancher-on-k8s/upgrades/helm2/_index.md @@ -4,6 +4,9 @@ weight: 1050 aliases: - /rancher/v2.x/en/upgrades/upgrades/ha/helm2 - /rancher/v2.x/en/upgrades/helm2 + - /rancher/v2.x/en/installation/upgrades-rollbacks/upgrades/ha/helm2 + - /rancher/v2.x/en/installation/install-rancher-on-k8s/upgrades-rollbacks/upgrades/ha/helm2 + - /rancher/v2.x/en/installation/install-rancher-on-k8s/upgrades-rollbacks/upgrades/helm2 --- > Helm 3 has been released. If you are using Helm 2, we recommend [migrating to Helm 3](https://helm.sh/blog/migrate-from-helm-v2-to-helm-v3/) because it is simpler to use and more secure than Helm 2. diff --git a/content/rancher/v2.x/en/installation/upgrades-rollbacks/upgrades/migrating-from-rke-add-on/_index.md b/content/rancher/v2.x/en/installation/install-rancher-on-k8s/upgrades/migrating-from-rke-add-on/_index.md similarity index 95% rename from content/rancher/v2.x/en/installation/upgrades-rollbacks/upgrades/migrating-from-rke-add-on/_index.md rename to content/rancher/v2.x/en/installation/install-rancher-on-k8s/upgrades/migrating-from-rke-add-on/_index.md index c48914cb4da..cb8f88198d4 100644 --- a/content/rancher/v2.x/en/installation/upgrades-rollbacks/upgrades/migrating-from-rke-add-on/_index.md +++ b/content/rancher/v2.x/en/installation/install-rancher-on-k8s/upgrades/migrating-from-rke-add-on/_index.md @@ -5,6 +5,8 @@ aliases: - /rancher/v2.x/en/upgrades/ha-server-upgrade/ - /rancher/v2.x/en/upgrades/upgrades/ha-server-upgrade/ - /rancher/v2.x/en/upgrades/upgrades/migrating-from-rke-add-on + - /rancher/v2.x/en/installation/upgrades-rollbacks/upgrades/migrating-from-rke-add-on + - /rancher/v2.x/en/installation/install-rancher-on-k8s/upgrades-rollbacks/upgrades/migrating-from-rke-add-on --- > **Important: RKE add-on install is only supported up to Rancher v2.0.8** diff --git a/content/rancher/v2.x/en/installation/upgrades-rollbacks/upgrades/namespace-migration/_index.md b/content/rancher/v2.x/en/installation/install-rancher-on-k8s/upgrades/namespace-migration/_index.md similarity index 98% rename from content/rancher/v2.x/en/installation/upgrades-rollbacks/upgrades/namespace-migration/_index.md rename to content/rancher/v2.x/en/installation/install-rancher-on-k8s/upgrades/namespace-migration/_index.md index 01dff5d23b3..1e4f7ad3fbd 100644 --- a/content/rancher/v2.x/en/installation/upgrades-rollbacks/upgrades/namespace-migration/_index.md +++ b/content/rancher/v2.x/en/installation/install-rancher-on-k8s/upgrades/namespace-migration/_index.md @@ -3,6 +3,8 @@ title: Upgrading to v2.0.7+ — Namespace Migration weight: 1040 aliases: - /rancher/v2.x/en/upgrades/upgrades/namespace-migration + - /rancher/v2.x/en/installation/upgrades-rollbacks/upgrades/namespace-migration + - /rancher/v2.x/en/installation/install-rancher-on-k8s/upgrades-rollbacks/upgrades/namespace-migration --- >This section applies only to Rancher upgrades from v2.0.6 or earlier to v2.0.7 or later. Upgrades from v2.0.7 to later version are unaffected. diff --git a/content/rancher/v2.x/en/installation/install-rancher-on-linux/rollbacks/_index.md b/content/rancher/v2.x/en/installation/install-rancher-on-linux/rollbacks/_index.md new file mode 100644 index 00000000000..5b3ed82e35c --- /dev/null +++ b/content/rancher/v2.x/en/installation/install-rancher-on-linux/rollbacks/_index.md @@ -0,0 +1,6 @@ +--- +title: Rollbacks +weight: 3 +--- + +To roll back Rancher to a previous version, re-run the installation script with the previous version specified in the `INSTALL_RANCHERD_VERSION` environment variable. \ No newline at end of file diff --git a/content/rancher/v2.x/en/installation/install-rancher-on-linux/upgrades/_index.md b/content/rancher/v2.x/en/installation/install-rancher-on-linux/upgrades/_index.md index ec55413d585..6244161d137 100644 --- a/content/rancher/v2.x/en/installation/install-rancher-on-linux/upgrades/_index.md +++ b/content/rancher/v2.x/en/installation/install-rancher-on-linux/upgrades/_index.md @@ -1,6 +1,6 @@ --- -title: Upgrades and Rollbacks -weight: +title: Upgrades +weight: 2 --- When RancherD is upgraded, the Rancher Helm controller and the Fleet pods are upgraded. @@ -37,8 +37,6 @@ To upgrade Rancher without upgrading the underlying Kubernetes cluster, follow t If necessary, restore Rancher from backup by following [these steps.]({{}}/rancher/v2.x/en/backups/v2.5/restoring-rancher/) -To roll back Rancher to a previous version, uninstall the Helm chart and reinstall the Helm chart for the previous version. - ### Upgrading Both Rancher and the Underlying Cluster Upgrade both RancherD and the underlying Kubernetes cluster by re-running the RancherD installation script. @@ -67,4 +65,4 @@ systemctl start rancherd-server The upgrade can also be performed by manually installing the binary of the desired version. -To roll back Rancher to a previous version, re-run the installation script with the previous version specified in the `INSTALL_RANCHERD_VERSION` environment variable. + diff --git a/content/rancher/v2.x/en/installation/resources/encryption/custom-ca-root-certificate/_index.md b/content/rancher/v2.x/en/installation/resources/custom-ca-root-certificate/_index.md similarity index 75% rename from content/rancher/v2.x/en/installation/resources/encryption/custom-ca-root-certificate/_index.md rename to content/rancher/v2.x/en/installation/resources/custom-ca-root-certificate/_index.md index c7c10c9e947..fddbc22de66 100644 --- a/content/rancher/v2.x/en/installation/resources/encryption/custom-ca-root-certificate/_index.md +++ b/content/rancher/v2.x/en/installation/resources/custom-ca-root-certificate/_index.md @@ -3,6 +3,7 @@ title: About Custom CA Root Certificates weight: 1 aliases: - /rancher/v2.x/en/installation/options/custom-ca-root-certificate/ + - /rancher/v2.x/en/installation/resources/choosing-version/encryption/custom-ca-root-certificate --- If you're using Rancher in an internal production environment where you aren't exposing apps publicly, use a certificate from a private certificate authority (CA). @@ -21,7 +22,7 @@ Examples of services that Rancher can access: For details on starting a Rancher container with your private CA certificates mounted, refer to the installation docs: -- [Docker Install]({{}}/rancher/v2.x/en/installation/other-installation-methods/single-node-docker/advanced/#custom-ca-certificate) +- [Docker install Custom CA certificate options]({{}}/rancher/v2.x/en/installation/other-installation-methods/single-node-docker/advanced/#custom-ca-certificate) -- [Kubernetes Install]({{}}/rancher/v2.x/en/installation/resources/chart-options/#additional-trusted-cas) +- [Kubernetes install options for Additional Trusted CAs]({{}}/rancher/v2.x/en/installation/install-rancher-on-k8s/chart-options/#additional-trusted-cas) diff --git a/content/rancher/v2.x/en/installation/resources/encryption/_index.md b/content/rancher/v2.x/en/installation/resources/encryption/_index.md deleted file mode 100644 index 345f6b72818..00000000000 --- a/content/rancher/v2.x/en/installation/resources/encryption/_index.md +++ /dev/null @@ -1,6 +0,0 @@ ---- -title: Encryption -weight: 3 ---- - -The documents in this section contain information about certificate configuration and `cert-manager`. \ No newline at end of file diff --git a/content/rancher/v2.x/en/installation/resources/encryption/tls-secrets/_index.md b/content/rancher/v2.x/en/installation/resources/tls-secrets/_index.md similarity index 96% rename from content/rancher/v2.x/en/installation/resources/encryption/tls-secrets/_index.md rename to content/rancher/v2.x/en/installation/resources/tls-secrets/_index.md index 019727c55be..3c339f16c1a 100644 --- a/content/rancher/v2.x/en/installation/resources/encryption/tls-secrets/_index.md +++ b/content/rancher/v2.x/en/installation/resources/tls-secrets/_index.md @@ -3,6 +3,7 @@ title: Adding TLS Secrets weight: 2 aliases: - /rancher/v2.x/en/installation/options/tls-secrets/ + - /rancher/v2.x/en/installation/resources/encryption/tls-secrets --- Kubernetes will create all the objects and services for Rancher, but it will not become available until we populate the `tls-rancher-ingress` secret in the `cattle-system` namespace with the certificate and key. diff --git a/content/rancher/v2.x/en/installation/resources/encryption/tls-settings/_index.md b/content/rancher/v2.x/en/installation/resources/tls-settings/_index.md similarity index 82% rename from content/rancher/v2.x/en/installation/resources/encryption/tls-settings/_index.md rename to content/rancher/v2.x/en/installation/resources/tls-settings/_index.md index 340f5da17d2..6c2ae09d236 100644 --- a/content/rancher/v2.x/en/installation/resources/encryption/tls-settings/_index.md +++ b/content/rancher/v2.x/en/installation/resources/tls-settings/_index.md @@ -4,19 +4,20 @@ weight: 3 aliases: - /rancher/v2.x/en/installation/options/tls-settings/ - /rancher/v2.x/en/admin-settings/tls-settings + - /rancher/v2.x/en/installation/resources/encryption/tls-settings --- In Rancher v2.1.7, the default TLS configuration changed to only accept TLS 1.2 and secure TLS cipher suites. TLS 1.3 and TLS 1.3 exclusive cipher suites are not supported. -## Configuring TLS settings +# Configuring TLS settings The Audit Log is enabled and configured by passing environment variables to the Rancher server container. See the following to enable on your installation. -- [Installing Rancher on a single node with Docker]({{}}/rancher/v2.x/en/installation/other-installation-methods/single-node-docker/advanced/#tls-settings) +- [TLS settings in Docker options]({{}}/v2.x/en/installation/other-installation-methods/single-node-docker/advanced/#tls-settings) -- [Installing Rancher on Kubernetes]({{}}/rancher/v2.x/en/installation/resources/chart-options/#tls-settings) +- [TLS settings in Helm chart options]({{}}/rancher/v2.x/en/installation/install-rancher-on-k8s/chart-options/#tls-settings) -## TLS settings +# TLS Environment Variables | Parameter | Description | Default | Available options | |-----|-----|-----|-----| @@ -24,7 +25,7 @@ The Audit Log is enabled and configured by passing environment variables to the | `CATTLE_TLS_CIPHERS` | Allowed TLS cipher suites | `TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,`
`TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,`
`TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,`
`TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,`
`TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,`
`TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305` | See [Golang tls constants](https://golang.org/pkg/crypto/tls/#pkg-constants) | -## Legacy configuration +# Legacy configuration If you need to configure TLS the same way as it was before Rancher v2.1.7, please use the following settings: diff --git a/content/rancher/v2.x/en/installation/resources/encryption/upgrading-cert-manager/_index.md b/content/rancher/v2.x/en/installation/resources/upgrading-cert-manager/_index.md similarity index 99% rename from content/rancher/v2.x/en/installation/resources/encryption/upgrading-cert-manager/_index.md rename to content/rancher/v2.x/en/installation/resources/upgrading-cert-manager/_index.md index 0fe87b4103c..00b29200ab3 100644 --- a/content/rancher/v2.x/en/installation/resources/encryption/upgrading-cert-manager/_index.md +++ b/content/rancher/v2.x/en/installation/resources/upgrading-cert-manager/_index.md @@ -4,6 +4,7 @@ weight: 4 aliases: - /rancher/v2.x/en/installation/options/upgrading-cert-manager - /rancher/v2.x/en/installation/options/upgrading-cert-manager/helm-2-instructions + - /rancher/v2.x/en/installation/resources/encryption/upgrading-cert-manager --- Rancher uses cert-manager to automatically generate and renew TLS certificates for HA deployments of Rancher. As of Fall 2019, three important changes to cert-manager are set to occur that you need to take action on if you have an HA deployment of Rancher: diff --git a/content/rancher/v2.x/en/installation/resources/encryption/upgrading-cert-manager/helm-2-instructions/_index.md b/content/rancher/v2.x/en/installation/resources/upgrading-cert-manager/helm-2-instructions/_index.md similarity index 98% rename from content/rancher/v2.x/en/installation/resources/encryption/upgrading-cert-manager/helm-2-instructions/_index.md rename to content/rancher/v2.x/en/installation/resources/upgrading-cert-manager/helm-2-instructions/_index.md index 5d93d0f2656..7dbcbeee2b7 100644 --- a/content/rancher/v2.x/en/installation/resources/encryption/upgrading-cert-manager/helm-2-instructions/_index.md +++ b/content/rancher/v2.x/en/installation/resources/upgrading-cert-manager/helm-2-instructions/_index.md @@ -3,6 +3,7 @@ title: Upgrading Cert-Manager with Helm 2 weight: 2040 aliases: - /rancher/v2.x/en/installation/options/upgrading-cert-manager/helm-2-instructions + - /rancher/v2.x/en/installation/resources/choosing-version/encryption/upgrading-cert-manager/helm-2-instructions --- Rancher uses cert-manager to automatically generate and renew TLS certificates for HA deployments of Rancher. As of Fall 2019, three important changes to cert-manager are set to occur that you need to take action on if you have an HA deployment of Rancher: diff --git a/content/rancher/v2.x/en/installation/upgrades-rollbacks/_index.md b/content/rancher/v2.x/en/installation/upgrades-rollbacks/_index.md deleted file mode 100644 index d235ebcea30..00000000000 --- a/content/rancher/v2.x/en/installation/upgrades-rollbacks/_index.md +++ /dev/null @@ -1,19 +0,0 @@ ---- -title: Upgrades and Rollbacks -weight: 150 -aliases: - - /rancher/v2.x/en/upgrades ---- - -### Upgrading Rancher - -To upgrade Rancher, refer to [these instructions.](./upgrades/) - -### Rolling Back Unsuccessful Upgrades - -In the event that your Rancher Server does not upgrade successfully, you can rollback to your installation prior to upgrade using these instructions: [Rollbacks for Rancher installed on a Kubernetes cluster](./rollbacks/ha-server-rollbacks) - -> **Note:** If you are rolling back to versions in either of these scenarios, you must follow some extra [instructions]({{}}/rancher/v2.x/en/upgrades/rollbacks/) in order to get your clusters working. -> ->- Rolling back from v2.1.6+ to any version between v2.1.0 - v2.1.5 or v2.0.0 - v2.0.10. ->- Rolling back from v2.0.11+ to any version between v2.0.0 - v2.0.10. diff --git a/content/rancher/v2.x/en/installation/upgrades-rollbacks/rollbacks/ha-server-rollbacks/_index.md b/content/rancher/v2.x/en/installation/upgrades-rollbacks/rollbacks/ha-server-rollbacks/_index.md deleted file mode 100644 index 955a0938879..00000000000 --- a/content/rancher/v2.x/en/installation/upgrades-rollbacks/rollbacks/ha-server-rollbacks/_index.md +++ /dev/null @@ -1,17 +0,0 @@ ---- -title: Kubernetes Rollback -weight: 1025 -aliases: - - /rancher/v2.x/en/upgrades/ha-server-rollbacks - - /rancher/v2.x/en/upgrades/rollbacks/ha-server-rollbacks ---- - -If you upgrade Rancher and the upgrade does not complete successfully, you may need to rollback your Rancher Server to its last healthy state. - -To restore Rancher prior to v2.5, follow the procedure detailed here: [Restoring Backups — Kubernetes installs]({{}}/rancher/v2.x/en/backups/restorations/ha-restoration) - -To restore Rancher v2.5, you can use the `rancher-backup` application and restore Rancher from backup according to [this section.]({{}}/rancher/v2.x/en/backups/restoring-rancher/) - -Restoring a snapshot of the Rancher Server cluster will revert Rancher to the version and state at the time of the snapshot. - ->**Note:** Managed cluster are authoritative for their state. This means restoring the rancher server will not revert workload deployments or changes made on managed clusters after the snapshot was taken. diff --git a/content/rancher/v2.x/en/installation/upgrades-rollbacks/upgrades/_index.md b/content/rancher/v2.x/en/installation/upgrades-rollbacks/upgrades/_index.md deleted file mode 100644 index 4429dbd6a77..00000000000 --- a/content/rancher/v2.x/en/installation/upgrades-rollbacks/upgrades/_index.md +++ /dev/null @@ -1,33 +0,0 @@ ---- -title: Upgrades -weight: 1005 -aliases: - - /rancher/v2.x/en/upgrades/upgrades ---- -This section contains information about how to upgrade your Rancher server to a newer version. Regardless if you installed in an air gap environment or not, the upgrade steps mainly depend on whether you have a single node or high-availability installation of Rancher. Select from the following options: - -- [Upgrading Rancher installed with Docker]({{}}/rancher/v2.x/en/installation/other-installation-methods/single-node-docker/single-node-upgrades) -- [Upgrading Rancher installed on a Kubernetes cluster]({{}}/rancher/v2.x/en/installation/upgrades-rollbacks/upgrades/ha/) - -### Known Upgrade Issues - -The following table lists some of the most noteworthy issues to be considered when upgrading Rancher. A more complete list of known issues for each Rancher version can be found in the release notes on [GitHub](https://github.com/rancher/rancher/releases) and on the [Rancher forums.](https://forums.rancher.com/c/announcements/12) - -Upgrade Scenario | Issue ----|--- -Upgrading to v2.4.6 or v2.4.7 | These Rancher versions had an issue where the `kms:ListKeys` permission was required to create, edit, or clone Amazon EC2 node templates. This requirement was removed in v2.4.8. -Upgrading to v2.3.0+ | Any user provisioned cluster will be automatically updated upon any edit as tolerations were added to the images used for Kubernetes provisioning. -Upgrading to v2.2.0-v2.2.x | Rancher introduced the [system charts](https://github.com/rancher/system-charts) repository which contains all the catalog items required for features such as monitoring, logging, alerting and global DNS. To be able to use these features in an air gap install, you will need to mirror the `system-charts` repository locally and configure Rancher to use that repository. Please follow the instructions to [configure Rancher system charts]({{}}/rancher/v2.x/en/installation/resources/local-system-charts/#setting-up-system-charts-for-rancher-prior-to-v2-3-0). -Upgrading from v2.0.13 or earlier | If your cluster's certificates have expired, you will need to perform [additional steps]({{}}/rancher/v2.x/en/cluster-admin/certificate-rotation/#rotating-expired-certificates-after-upgrading-older-rancher-versions) to rotate the certificates. -Upgrading from v2.0.7 or earlier | Rancher introduced the `system` project, which is a project that's automatically created to store important namespaces that Kubernetes needs to operate. During upgrade to v2.0.7+, Rancher expects these namespaces to be unassigned from all projects. Before beginning upgrade, check your system namespaces to make sure that they're unassigned to [prevent cluster networking issues]({{}}/rancher/v2.x/en/upgrades/upgrades/namespace-migration/#preventing-cluster-networking-issues). - -### Caveats -Upgrades _to_ or _from_ any chart in the [rancher-alpha repository]({{}}/rancher/v2.x/en/installation/resources/chart-options/#helm-chart-repositories/) aren't supported. - -### RKE Add-on Installs - -**Important: RKE add-on install is only supported up to Rancher v2.0.8** - -Please use the Rancher helm chart to install Rancher on a Kubernetes cluster. For details, see the [Kubernetes Install - Installation Outline]({{}}/rancher/v2.x/en/installation/install-rancher-on-k8s/#installation-outline). - -If you are currently using the RKE add-on install method, see [Migrating from a RKE add-on install]({{}}/rancher/v2.x/en/upgrades/upgrades/migrating-from-rke-add-on/) for details on how to move to using the helm chart. diff --git a/content/rancher/v2.x/en/installation/upgrades-rollbacks/upgrades/ha/_index.md b/content/rancher/v2.x/en/installation/upgrades-rollbacks/upgrades/ha/_index.md deleted file mode 100644 index b415647193b..00000000000 --- a/content/rancher/v2.x/en/installation/upgrades-rollbacks/upgrades/ha/_index.md +++ /dev/null @@ -1,231 +0,0 @@ ---- -title: Upgrading Rancher Installed on Kubernetes -weight: 1020 -aliases: - - /rancher/v2.x/en/upgrades/upgrades/ha-server-upgrade-helm-airgap - - /rancher/v2.x/en/upgrades/air-gap-upgrade/ - - /rancher/v2.x/en/upgrades/upgrades/ha - - /rancher/v2.x/en/installation/install-rancher-on-k8s/upgrades/upgrades/ha - - /rancher/v2.x/en/installation/upgrades-rollbacks/upgrades/ - - /rancher/v2.x/en/upgrades/upgrades/ha-server-upgrade-helm/ ---- - -The following instructions will guide you through using Helm to upgrade a Rancher server that was installed on a Kubernetes cluster. - -To upgrade the components in your Kubernetes cluster, or the definition of the [Kubernetes services]({{}}/rke/latest/en/config-options/services/) or [add-ons]({{}}/rke/latest/en/config-options/add-ons/), refer to the [upgrade documentation for RKE]({{}}/rke/latest/en/upgrades/), the Rancher Kubernetes Engine. - -If you installed Rancher using the RKE Add-on yaml, follow the directions to [migrate or upgrade]({{}}/rancher/v2.x/en/upgrades/upgrades/migrating-from-rke-add-on). - ->**Notes:** -> -> - If you are upgrading to Rancher v2.5 from a Rancher server that was started with the Helm chart option `--add-local=false`, you will need to drop that flag when upgrading. Otherwise, the Rancher server will not start. The `restricted-admin` role can be used to continue restricting access to the local cluster. For more information, see [this section.]({{}}/rancher/v2.x/en/admin-settings/rbac/global-permissions/#upgrading-from-rancher-with-a-hidden-local-cluster) -> - [Let's Encrypt will be blocking cert-manager instances older than 0.8.0 starting November 1st 2019.](https://community.letsencrypt.org/t/blocking-old-cert-manager-versions/98753) Upgrade cert-manager to the latest version by following [these instructions.]({{}}/rancher/v2.x/en/installation/options/upgrading-cert-manager) -> - Helm should be run from the same location as your kubeconfig file (where you run your kubectl commands from. If you installed K8s with RKE, the config will have been created in the directory you ran `rke up` in) or should manually target the kubeconfig for the intended cluster with the `--kubeconfig` tag (see: https://helm.sh/docs/helm/helm/) -> - The upgrade instructions assume you are using Helm 3. For migration of installs started with Helm 2, refer to the official [Helm 2 to 3 migration docs.](https://helm.sh/blog/migrate-from-helm-v2-to-helm-v3/) The [Helm 2 upgrade page here]({{}}/rancher/v2.x/en/installation/upgrades-rollbacks/upgrades/ha/helm2)provides a copy of the older upgrade instructions that used Helm 2, and it is intended to be used if upgrading to Helm 3 is not feasible. -> - If you are upgrading Rancher from v2.x to v2.3+, and you are using external TLS termination, you will need to edit the cluster.yml to [enable using forwarded host headers.]({{}}/rancher/v2.x/en/installation/resources/chart-options/#configuring-ingress-for-external-tls-when-using-nginx-v0-25) - -# Prerequisites - -- **Review the [known upgrade issues]({{}}/rancher/v2.x/en/upgrades/upgrades/#known-upgrade-issues) and [caveats]({{}}/rancher/v2.x/en/upgrades/upgrades/#caveats)** in the Rancher documentation for the most noteworthy issues to consider when upgrading Rancher. A more complete list of known issues for each Rancher version can be found in the release notes on [GitHub](https://github.com/rancher/rancher/releases) and on the [Rancher forums.](https://forums.rancher.com/c/announcements/12) -- **For [air gap installs only,]({{}}/rancher/v2.x/en/installation/other-installation-methods/air-gap) collect and populate images for the new Rancher server version.** Follow the guide to [populate your private registry]({{}}/rancher/v2.x/en/installation/other-installation-methods/air-gap/populate-private-registry/) with the images for the Rancher version that you want to upgrade to. - -# Upgrade Outline - -Follow the steps to upgrade Rancher server: - -- [A. Back up your Kubernetes cluster that is running Rancher server](#a-back-up-your-kubernetes-cluster-that-is-running-rancher-server) -- [B. Update the Helm chart repository](#b-update-the-helm-chart-repository) -- [C. Upgrade Rancher](#c-upgrade-rancher) -- [D. Verify the Upgrade](#d-verify-the-upgrade) - -### A. Back up Your Kubernetes Cluster that is Running Rancher Server - -[Take a one-time snapshot]({{}}/rancher/v2.x/en/backups/backups/ha-backups/#option-b-one-time-snapshots) -of your Kubernetes cluster running Rancher server. You'll use the snapshot as a restoration point if something goes wrong during upgrade. - -### B. Update the Helm chart repository - -1. Update your local helm repo cache. - - ``` - helm repo update - ``` - -1. Get the repository name that you used to install Rancher. - - For information about the repos and their differences, see [Helm Chart Repositories]({{}}/rancher/v2.x/en/installation/resources/chart-options/#helm-chart-repositories). - - {{< release-channel >}} - - ``` - helm repo list - - NAME URL - stable https://kubernetes-charts.storage.googleapis.com - rancher- https://releases.rancher.com/server-charts/ - ``` - - > **Note:** If you want to switch to a different Helm chart repository, please follow the [steps on how to switch repositories]({{}}/rancher/v2.x/en/installation/resources/chart-options/#switching-to-a-different-helm-chart-repository). If you switch repositories, make sure to list the repositories again before continuing onto Step 3 to ensure you have the correct one added. - - -1. Fetch the latest chart to install Rancher from the Helm chart repository. - - This command will pull down the latest charts and save it in the current directory as a `.tgz` file. - - ```plain - helm fetch rancher-/rancher - ``` - You can fetch the chart for the specific version you are upgrading to by adding in the `--version=` tag. For example: - - ```plain - helm fetch rancher-/rancher --version=v2.4.11 - ``` - -### C. Upgrade Rancher - -This section describes how to upgrade normal (Internet-connected) or air gap installations of Rancher with Helm. - -{{% tabs %}} -{{% tab "Kubernetes Upgrade" %}} - -Get the values, which were passed with `--set`, from the current Rancher Helm chart that is installed. - -``` -helm get values rancher -n cattle-system - -hostname: rancher.my.org -``` - -> **Note:** There will be more values that are listed with this command. This is just an example of one of the values. - -If you are also upgrading cert-manager to the latest version from a version older than 0.11.0, follow `Option B: Reinstalling Rancher and cert-manager`. Otherwise, follow `Option A: Upgrading Rancher`. - -{{% accordion label="Option A: Upgrading Rancher" %}} - -Upgrade Rancher to the latest version with all your settings. - -Take all the values from the previous step and append them to the command using `--set key=value`: - -``` -helm upgrade rancher rancher-/rancher \ - --namespace cattle-system \ - --set hostname=rancher.my.org -``` - -> **Note:** The above is an example, there may be more values from the previous step that need to be appended. - -Alternatively, it's possible to export the current values to a file and reference that file during upgrade. For example, to only change the Rancher version: - -``` -helm get values rancher -n cattle-system -o yaml > values.yaml - -helm upgrade rancher rancher-/rancher \ - --namespace cattle-system \ - -f values.yaml \ - --version=2.4.5 -``` - -{{% /accordion %}} - -{{% accordion label="Option B: Reinstalling Rancher and cert-manager" %}} - -If you are currently running the cert-manger whose version is older than v0.11, and want to upgrade both Rancher and cert-manager to a newer version, then you need to reinstall both Rancher and cert-manger due to the API change in cert-manger v0.11. - -1. Uninstall Rancher - - ``` - helm delete rancher -n cattle-system - ``` - -2. Uninstall and reinstall `cert-manager` according to the instructions on the [Upgrading Cert-Manager]({{}}/rancher/v2.x/en/installation/options/upgrading-cert-manager) page. - -3. Reinstall Rancher to the latest version with all your settings. Take all the values from the step 1 and append them to the command using `--set key=value`. Note: There will be many more options from the step 1 that need to be appended. - - ``` - helm install rancher rancher-/rancher \ - --namespace cattle-system \ - --set hostname=rancher.my.org - ``` - -{{% /accordion %}} - -{{% /tab %}} - -{{% tab "Kubernetes Air Gap Upgrade" %}} - -1. Render the Rancher template using the same chosen options that were used when installing Rancher. Use the reference table below to replace each placeholder. Rancher needs to be configured to use the private registry in order to provision any Rancher launched Kubernetes clusters or Rancher tools. - - Based on the choice you made during installation, complete one of the procedures below. - - Placeholder | Description - ------------|------------- - `` | The version number of the output tarball. - `` | The DNS name you pointed at your load balancer. - `` | The DNS name for your private registry. - `` | Cert-manager version running on k8s cluster. - -{{% accordion id="self-signed" label="Option A-Default Self-Signed Certificate" %}} - - ```plain -helm template ./rancher-.tgz --output-dir . \ - --name rancher \ - --namespace cattle-system \ - --set hostname= \ - --set certmanager.version= \ - --set rancherImage=/rancher/rancher \ - --set systemDefaultRegistry= \ # Available as of v2.2.0, set a default private registry to be used in Rancher - --set useBundledSystemChart=true # Available as of v2.3.0, use the packaged Rancher system charts -``` - -{{% /accordion %}} -{{% accordion id="secret" label="Option B: Certificates From Files using Kubernetes Secrets" %}} - -```plain -helm template ./rancher-.tgz --output-dir . \ ---name rancher \ ---namespace cattle-system \ ---set hostname= \ ---set rancherImage=/rancher/rancher \ ---set ingress.tls.source=secret \ ---set systemDefaultRegistry= \ # Available as of v2.2.0, set a default private registry to be used in Rancher ---set useBundledSystemChart=true # Available as of v2.3.0, use the packaged Rancher system charts -``` - -If you are using a Private CA signed cert, add `--set privateCA=true` following `--set ingress.tls.source=secret`: - -```plain -helm template ./rancher-.tgz --output-dir . \ ---name rancher \ ---namespace cattle-system \ ---set hostname= \ ---set rancherImage=/rancher/rancher \ ---set ingress.tls.source=secret \ ---set privateCA=true \ ---set systemDefaultRegistry= \ # Available as of v2.2.0, set a default private registry to be used in Rancher ---set useBundledSystemChart=true # Available as of v2.3.0, use the packaged Rancher system charts -``` - -{{% /accordion %}} - -2. Copy the rendered manifest directories to a system with access to the Rancher server cluster and apply the rendered templates. - - Use `kubectl` to apply the rendered manifests. - - ```plain - kubectl -n cattle-system apply -R -f ./rancher - ``` - -{{% /tab %}} -{{% /tabs %}} - -### D. Verify the Upgrade - -Log into Rancher to confirm that the upgrade succeeded. - ->**Having network issues following upgrade?** -> -> See [Restoring Cluster Networking]({{}}/rancher/v2.x/en/upgrades/upgrades/namespace-migration/#restoring-cluster-networking). - -## Rolling Back - -Should something go wrong, follow the [roll back]({{}}/rancher/v2.x/en/upgrades/rollbacks/ha-server-rollbacks/) instructions to restore the snapshot you took before you preformed the upgrade. diff --git a/layouts/shortcodes/requirements_rollback.html b/layouts/shortcodes/requirements_rollback.html deleted file mode 100644 index ccf2c8d316c..00000000000 --- a/layouts/shortcodes/requirements_rollback.html +++ /dev/null @@ -1,15 +0,0 @@ -
-
    -
  • -

    Rancher Kubernetes Engine v0.1.7 or later

    -

    The commands for taking etcd snapshots are only available in RKE v0.1.7 and later.

    -
    • -
  • -

    rancher-cluster.yml

    -

    You'll need the RKE config file that you used for Rancher install, rancher-cluster.yml. You created this file during your initial install. Place this file in same directory as the RKE binary.

    -
    • -
  • - You must restore each of your etcd nodes to the same snapshot. Copy the snapshot you're using from one of your nodes to the others before running the etcd snapshot-restore command. -
    • -
-
\ No newline at end of file From 46d790c25db376239bb514ccbc4422574e841d0c Mon Sep 17 00:00:00 2001 From: Lars Kerick Date: Wed, 2 Dec 2020 09:25:15 +0100 Subject: [PATCH 13/49] Added cgroup documentation for raspbian --- content/k3s/latest/en/advanced/_index.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/content/k3s/latest/en/advanced/_index.md b/content/k3s/latest/en/advanced/_index.md index 7edd6c620dc..633399b6614 100644 --- a/content/k3s/latest/en/advanced/_index.md +++ b/content/k3s/latest/en/advanced/_index.md @@ -19,6 +19,7 @@ This section contains advanced information describing the different ways you can - [Additional preparation for Alpine Linux setup](#additional-preparation-for-alpine-linux-setup) - [Running K3d (K3s in Docker) and docker-compose](#running-k3d-k3s-in-docker-and-docker-compose) - [Enabling legacy iptables on Raspbian Buster](#enabling-legacy-iptables-on-raspbian-buster) +- [Enabling cgroups for Raspbian Buster](#enabling-cgroups-for-raspbian-buster) - [SELinux Support](#selinux-support) # Certificate Rotation @@ -304,6 +305,15 @@ sudo update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy sudo reboot ``` +# Enabling cgroups for Raspbian Buster + +Standard Raspbian Buster installations do not start with `cgroups` enabled. **K3S** needs `cgroups` to start the systemd service. `cgroups`can be enabled by appending `cgroup_memory=1 cgroup_enable=memory` to `/boot/cmdline.txt`. + +## example of /boot/cmdline.txt +``` +console=serial0,115200 console=tty1 root=PARTUUID=58b06195-02 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait cgroup_memory=1 cgroup_enable=memory +``` + # SELinux Support _Supported as of v1.19.4+k3s1. Experimental as of v1.17.4+k3s1._ From d0b659358053afcc0457e2c50453f72b1f9af553 Mon Sep 17 00:00:00 2001 From: Tejeev Date: Fri, 4 Dec 2020 11:11:45 +0000 Subject: [PATCH 14/49] Add note on how to fetch specific version --- .../air-gap/install-rancher/_index.md | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/content/rancher/v2.x/en/installation/other-installation-methods/air-gap/install-rancher/_index.md b/content/rancher/v2.x/en/installation/other-installation-methods/air-gap/install-rancher/_index.md index c27acbcca55..58a9e26b8de 100644 --- a/content/rancher/v2.x/en/installation/other-installation-methods/air-gap/install-rancher/_index.md +++ b/content/rancher/v2.x/en/installation/other-installation-methods/air-gap/install-rancher/_index.md @@ -40,9 +40,14 @@ From a system that has access to the internet, fetch the latest Helm chart and c ``` 3. Fetch the latest Rancher chart. This will pull down the chart and save it in the current directory as a `.tgz` file. -```plain -helm fetch rancher-/rancher -``` + ```plain + helm fetch rancher-/rancher + ``` + + If you require a specific version of Rancher, you can fetch this with the Helm `--version` parameter like in the following example: + ```plain + helm fetch rancher-stable/rancher --version=v2.4.8 + ``` ### B. Choose your SSL Configuration From 5c856ffce30f86490e8ae281d077e7dfaa8f4c08 Mon Sep 17 00:00:00 2001 From: Tejeev Date: Fri, 4 Dec 2020 12:24:02 +0000 Subject: [PATCH 15/49] actually explained how to choose a version this instruction was conspicuously missing from the doc of the same name --- .../installation/resources/choosing-version/_index.md | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/content/rancher/v2.x/en/installation/resources/choosing-version/_index.md b/content/rancher/v2.x/en/installation/resources/choosing-version/_index.md index 2d6af682c01..328f9066724 100644 --- a/content/rancher/v2.x/en/installation/resources/choosing-version/_index.md +++ b/content/rancher/v2.x/en/installation/resources/choosing-version/_index.md @@ -35,9 +35,16 @@ Instructions on when to select these repos are available below in [Switching to ### Helm Chart Versions -Rancher Helm chart versions match the Rancher version (i.e `appVersion`). +Rancher Helm chart versions match the Rancher version (i.e `appVersion`). Once you've added the repo you can search it to show available versions with the following command:
+    `helm search repo --versions` -For the Rancher v2.1.x versions, there were some Helm charts, that were using a version that was a build number, i.e. `yyyy.mm.`. These charts have been replaced with the equivalent Rancher version and are no longer available. +If you have several repos you can specify the repo name, ie. `helm search repo rancher-stable/rancher --versions`
+For more information, see https://helm.sh/docs/helm/helm_search_repo/ + +To fetch a specific version of your chosen repo, define the `--version` parameter like in the following example:
+    `helm fetch rancher-stable/rancher --version=2.4.8` + +For the Rancher v2.1.x versions, there were some Helm charts where the version was a build number, i.e. `yyyy.mm.`. These charts have been replaced with the equivalent Rancher version and are no longer available. ### Switching to a Different Helm Chart Repository From 208ca04dedb25122209442ce04b96dbfeafe994b Mon Sep 17 00:00:00 2001 From: Nick Gerace Date: Fri, 4 Dec 2020 11:56:17 -0500 Subject: [PATCH 16/49] Update Windows and RBAC sections in Logging Update Windows section to remove fluentd configcheck warning. Refactor RBAC section to be more streamlined. Add syslog note. Perform general edits on grammar and style. --- .../rancher/v2.x/en/logging/v2.5/_index.md | 99 +++++++++---------- 1 file changed, 49 insertions(+), 50 deletions(-) diff --git a/content/rancher/v2.x/en/logging/v2.5/_index.md b/content/rancher/v2.x/en/logging/v2.5/_index.md index cc8b7a446c6..7e0be50fbc5 100644 --- a/content/rancher/v2.x/en/logging/v2.5/_index.md +++ b/content/rancher/v2.x/en/logging/v2.5/_index.md @@ -11,19 +11,17 @@ weight: 1 - [Configuring the Logging Application](#configuring-the-logging-application) - [Working with Taints and Tolerations](#working-with-taints-and-tolerations) - # Changes in Rancher v2.5 The following changes were introduced to logging in Rancher v2.5: -- The [Banzai Cloud Logging operator](https://banzaicloud.com/docs/one-eye/logging-operator/) now powers Rancher's logging in place of the former, in-house logging solution. -- [Fluent Bit](https://fluentbit.io/) is now used to aggregate the logs. [Fluentd](https://www.fluentd.org/) is used for filtering the messages and routing them to the outputs. Previously, only Fluentd was used. -- Logging can be configured with a Kubernetes manifest, because now the logging uses a Kubernetes operator with Custom Resource Definitions. +- The [Banzai Cloud Logging operator](https://banzaicloud.com/docs/one-eye/logging-operator/) now powers Rancher's logging solution in place of the former, in-house solution. +- [Fluent Bit](https://fluentbit.io/) is now used to aggregate the logs, and [Fluentd](https://www.fluentd.org/) is used for filtering the messages and routing them to the outputs. Previously, only Fluentd was used. +- Logging can be configured with a Kubernetes manifest, because logging now uses a Kubernetes operator with Custom Resource Definitions. - We now support filtering logs. - We now support writing logs to multiple outputs. - We now always collect Control Plane and etcd logs. - The following figure from the [Banzai documentation](https://banzaicloud.com/docs/one-eye/logging-operator/#architecture) shows the new logging architecture:
How the Banzai Cloud Logging Operator Works with Fluentd and Fluent Bit
@@ -34,20 +32,20 @@ The following figure from the [Banzai documentation](https://banzaicloud.com/doc You can enable the logging for a Rancher managed cluster by going to the Apps page and installing the logging app. -1. In the Rancher UI, go to the cluster where you want to install logging and click **Cluster Explorer.** -1. Click **Apps.** +1. In the Rancher UI, go to the cluster where you want to install logging and click **Cluster Explorer**. +1. Click **Apps**. 1. Click the `rancher-logging` app. -1. Scroll to the bottom of the Helm chart README and click **Install.** +1. Scroll to the bottom of the Helm chart README and click **Install**. **Result:** The logging app is deployed in the `cattle-logging-system` namespace. # Uninstall Logging -1. From the **Cluster Explorer,** click **Apps & Marketplace.** -1. Click **Installed Apps.** +1. From the **Cluster Explorer**, click **Apps & Marketplace**. +1. Click **Installed Apps**. 1. Go to the `cattle-logging-system` namespace and check the boxes for `rancher-logging` and `rancher-logging-crd`. -1. Click **Delete.** -1. Confirm **Delete.** +1. Click **Delete**. +1. Confirm **Delete**. **Result** `rancher-logging` is uninstalled. @@ -55,21 +53,23 @@ You can enable the logging for a Rancher managed cluster by going to the Apps pa Rancher logging has two roles, `logging-admin` and `logging-view`. -`logging-admin` allows users full access to namespaced flows and outputs. +- `logging-admin` gives users full access to namespaced flows and outputs +- `logging-view` allows users to *view* namespaced flows and outputs, and cluster flows and outputs -The `logging-view` role allows users to view namespaced flows and outputs, and cluster flows and outputs. +> **Why choose one role over the other?** Edit access to cluster flow and cluster output resources is powerful. Any user with it has edit access for all logs in the cluster. -Edit access to the cluster flow and cluster output resources is powerful as it allows any user with edit access control of all logs in the cluster. +In Rancher, the cluster administrator role is the only role with full access to all `rancher-logging` resources. Cluster members are not able to edit or read any logging resources. Project owners and members have the following privileges: -In Rancher, the cluster administrator role is the only role with full access to all rancher-logging resources. +Project Owners | Project Members +--- | --- +able to create namespaced flows and outputs in their projects' namespaces | only able to view the flows and outputs in projects' namespaces +can collect logs from anything in their projects' namespaces | cannot collect any logs in their projects' namespaces -Cluster members are not able to edit or read any logging resources. - -Project owners are able to create namespaced flows and outputs in the namespaces under their projects. This means that project owners can collect logs from anything in their project namespaces. Project members are able to view the flows and outputs in the namespaces under their projects. Project owners and project members require at least 1 namespace in their project to use logging. If they do not have at least one namespace in their project they may not see the logging button in the top nav dropdown. +Both project owners and project members require at least *one* namespace in their project to use logging. If they do not, then they may not see the logging button in the top nav dropdown. # Configuring the Logging Application -To configure the logging application, go to the **Cluster Explorer** in the Rancher UI. In the upper left corner, click **Cluster Explorer > Logging.** +To configure the logging application, go to the **Cluster Explorer** in the Rancher UI. In the upper left corner, click **Cluster Explorer > Logging**. ### Overview of Logging Custom Resources @@ -84,9 +84,8 @@ According to the [Banzai Cloud documentation,](https://banzaicloud.com/docs/one- ### Examples -Let's say you wanted to send all logs in your cluster to an elasticsearch cluster. +**Cluster Output to ElasticSearch:** Let's say you wanted to send all logs in your cluster to an `elasticsearch` cluster. First, we create a cluster output. -First lets create our cluster output: ```yaml apiVersion: logging.banzaicloud.io/v1beta1 kind: ClusterOutput @@ -100,9 +99,9 @@ spec: scheme: http ``` -We have created a cluster output, without elasticsearch configuration, in the same namespace as our operator `cattle-logging-system.`. Any time we create a cluster flow or cluster output we have to put it in the `cattle-logging-system` namespace. +We have created this cluster output, without elasticsearch configuration, in the same namespace as our operator: `cattle-logging-system.`. Any time we create a cluster flow or cluster output, we have to put it in the `cattle-logging-system` namespace. -Now we have configured where we want the logs to go, lets configure all logs to go to that output. +Now that we have configured where we want the logs to go, let's configure all logs to go to that output. ```yaml apiVersion: logging.banzaicloud.io/v1beta1 @@ -117,9 +116,9 @@ spec: We should now see our configured index with logs in it. -What if we have an application team who only wants logs from a specific namespaces sent to a splunk server? For this case can use namespaced outputs and flows. +**Output to Splunk:** What if we have an application team who only wants logs from a specific namespaces sent to a `splunk` server? For this case, we can use namespaced outputs and flows. -Before we start lets set up a scenario. +Before we start, let's set up that team's application: `coolapp`. ```yaml apiVersion: v1 @@ -149,7 +148,7 @@ spec: image: paynejacob/loggenerator:latest ``` -like before we start with an output, unlike cluster outputs we create our output in our application's namespace: +With `coolapp` running, we will follow a similar path as when we created a cluster output. However, unlike cluster outputs, we create our output in our application's namespace. ```yaml apiVersion: logging.banzaicloud.io/v1beta1 @@ -164,7 +163,7 @@ spec: protocol: http ``` -Once again, lets give our output some logs: +Once again, let's feed our output some logs. ```yaml apiVersion: logging.banzaicloud.io/v1beta1 @@ -177,7 +176,7 @@ spec: - "devteam-splunk" ``` -For the final example we create an output to write logs to a destination that is not supported out of the box (e.g. syslog): +**Unsupported Ouput:** For the final example, we create an output to write logs to a destination that is not supported out of the box (e.g. syslog): ```yaml apiVersion: v1 @@ -265,29 +264,31 @@ spec: ignore_network_errors_at_startup: false ``` -if we break down what is happening, first we create a deployment of a container that has the additional syslog plugin and accepts logs forwarded from another fluentd. Next we create an output configured as a forwarder to our deployment. The deployment fluentd will then forward all logs to the configured syslog destination. +Let's break down what is happening here. First, we create a deployment of a container that has the additional `syslog` plugin and accepts logs forwarded from another `fluentd`. Next we create an output configured as a forwarder to our deployment. The deployment `fluentd` will then forward all logs to the configured `syslog` destination. + +> **Note on syslog** The example provides an overview on using unsupported plugins, but `syslog` support is coming in a future Rancher v2.x release. # Working with Taints and Tolerations "Tainting" a Kubernetes node causes pods to repel running on that node. -Unless the pods have a ```toleration``` for that node's taint, they will run on other nodes in the cluster. -[Taints and tolerations](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/) can work in conjunction with the ```nodeSelector``` [field](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#nodeselector) within the ```PodSpec```, which enables the *opposite* effect of a taint. -Using ```nodeSelector``` gives pods an affinity towards certain nodes. +Unless the pods have a `toleration` for that node's taint, they will run on other nodes in the cluster. +[Taints and tolerations](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/) can work in conjunction with the `nodeSelector` [field](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#nodeselector) within the `PodSpec`, which enables the *opposite* effect of a taint. +Using `nodeSelector` gives pods an affinity towards certain nodes. Both provide choice for the what node(s) the pod will run on. ### Default Implementation in Rancher's Logging Stack -By default, Rancher taints all Linux nodes with ```cattle.io/os=linux```, and does not taint Windows nodes. -The logging stack pods have ```tolerations``` for this taint, which enables them to run on Linux nodes. -Moreover, we can populate the ```nodeSelector``` to ensure that our pods *only* run on Linux nodes. +By default, Rancher taints all Linux nodes with `cattle.io/os=linux`, and does not taint Windows nodes. +The logging stack pods have `tolerations` for this taint, which enables them to run on Linux nodes. +Moreover, we can populate the `nodeSelector` to ensure that our pods *only* run on Linux nodes. Let's look at an example pod YAML file with these settings... ```yaml apiVersion: v1 kind: Pod -# metadata: +# metadata... spec: - # containers: + # containers... tolerations: - key: cattle.io/os operator: "Equal" @@ -297,33 +298,31 @@ spec: kubernetes.io/os: linux ``` -In the above example, we ensure that our pod only runs on Linux nodes, and we add a ```toleration``` for the taint we have on all of our Linux nodes. +In the above example, we ensure that our pod only runs on Linux nodes, and we add a `toleration` for the taint we have on all of our Linux nodes. You can do the same with Rancher's existing taints, or with your own custom ones. ### Windows Support -Clusters with Windows worker support logging with some small caveats: - -1. Windows node logs are currently unable to be exported. -2. ```fluentd-configcheck``` pod(s) will fail due to an [upstream issue](https://github.com/banzaicloud/logging-operator/issues/592), where ```tolerations``` and ```nodeSelector``` settings are not inherited from the ```logging-operator```. +Clusters with Windows workers support exporting logs from Linux nodes, but Windows node logs are currently unable to be exported. +Only Linux node logs are able to be exported. ### Adding NodeSelector Settings and Tolerations for Custom Taints -If you would like to add your own ```nodeSelector``` settings, or if you would like to add ```tolerations``` for additional taints, you can pass the following to the chart's values. +If you would like to add your own `nodeSelector` settings, or if you would like to add `tolerations` for additional taints, you can pass the following to the chart's values. ```yaml tolerations: - # insert tolerations list + # insert tolerations... nodeSelector: - # insert nodeSelector settings + # insert nodeSelector... ``` -These values will add both settings to the ```fluentd```, ```fluentbit```, and ```logging-operator``` containers. +These values will add both settings to the `fluentd`, `fluentbit`, and `logging-operator` containers. Essentially, these are global settings for all pods in the logging stack. -However, if you would like to add tolerations for *only* the ```fluentbit``` container, you can add the following to the chart's values. +However, if you would like to add tolerations for *only* the `fluentbit` container, you can add the following to the chart's values. ```yaml fluentbit_tolerations: - # insert tolerations list for fluentbit containers only -``` \ No newline at end of file + # insert tolerations list for fluentbit containers only... +``` From e8a0443bc02a5a17aef7bb90a16ded377bbd040b Mon Sep 17 00:00:00 2001 From: JenTing Hsiao Date: Tue, 8 Dec 2020 12:34:36 +0800 Subject: [PATCH 17/49] Add anchor for Nodes Without a Hostname Signed-off-by: JenTing Hsiao --- content/k3s/latest/en/networking/_index.md | 1 + 1 file changed, 1 insertion(+) diff --git a/content/k3s/latest/en/networking/_index.md b/content/k3s/latest/en/networking/_index.md index a98c5c0af9e..549096266eb 100644 --- a/content/k3s/latest/en/networking/_index.md +++ b/content/k3s/latest/en/networking/_index.md @@ -16,6 +16,7 @@ For information on which ports need to be opened for K3s, refer to the [Installa - [Usage](#usage) - [Excluding the Service LB from Nodes](#excluding-the-service-lb-from-nodes) - [Disabling the Service LB](#disabling-the-service-lb) +- [Nodes Without a Hostname](#nodes-without-a-hostname) # CoreDNS From 328a54b2bf05852b153a3e5f90b976d54144c011 Mon Sep 17 00:00:00 2001 From: Catherine Luse Date: Tue, 8 Dec 2020 18:29:05 -0700 Subject: [PATCH 18/49] Say that you can migrate from Docker install with backup operator --- .../rancher/v2.x/en/installation/_index.md | 32 ++++++++++++++----- .../other-installation-methods/_index.md | 6 +++- .../air-gap/launch-kubernetes/_index.md | 6 ++-- .../air-gap/prepare-nodes/_index.md | 6 ++-- .../single-node-docker/_index.md | 8 +++-- .../v2.x/en/overview/architecture/_index.md | 8 ++++- 6 files changed, 48 insertions(+), 18 deletions(-) diff --git a/content/rancher/v2.x/en/installation/_index.md b/content/rancher/v2.x/en/installation/_index.md index 62364ea83ec..92f72123065 100644 --- a/content/rancher/v2.x/en/installation/_index.md +++ b/content/rancher/v2.x/en/installation/_index.md @@ -8,7 +8,7 @@ aliases: This section provides an overview of the architecture options of installing Rancher, describing advantages of each option. -### Terminology +# Terminology In this section, @@ -16,7 +16,7 @@ In this section, - **RKE (Rancher Kubernetes Engine)** is a certified Kubernetes distribution and CLI/library which creates and manages a Kubernetes cluster. - **K3s (Lightweight Kubernetes)** is also a fully compliant Kubernetes distribution. It is newer than RKE, easier to use, and more lightweight, with a binary size of less than 100 MB. As of Rancher v2.4, Rancher can be installed on a K3s cluster. -### Changes to Installation in Rancher v2.5 +# Changes to Installation in Rancher v2.5 In Rancher v2.5, the Rancher management server can be installed on any Kubernetes cluster, including hosted clusters, such as Amazon EKS clusters. @@ -24,13 +24,29 @@ For Docker installations, a local Kubernetes cluster is installed in the single The `restrictedAdmin` Helm chart option was added. When this option is set to true, the initial Rancher user has restricted access to the local Kubernetes cluster to prevent privilege escalation. For more information, see the section about the [restricted-admin role.]({{}}/rancher/v2.x/en/admin-settings/rbac/global-permissions/#restricted-admin) -### Overview of Installation Options +# Overview of Installation Options Rancher can be installed on these main architectures: -- **High-availability Kubernetes Install:** We recommend using Helm, a Kubernetes package manager, to install Rancher on multiple nodes on a dedicated Kubernetes cluster. For RKE clusters, three nodes are required to achieve a high-availability cluster. For K3s clusters, only two nodes are required. -- **Single-node Kubernetes Install:** Another option is to install Rancher with Helm on a Kubernetes cluster, but to only use a single node in the cluster. In this case, the Rancher server doesn't have high availability, which is important for running Rancher in production. However, this option is useful if you want to save resources by using a single node in the short term, while preserving a high-availability migration path. In the future, you can add nodes to the cluster to get a high-availability Rancher server. -- **Docker Install:** For test and demonstration purposes, Rancher can be installed with Docker on a single node. This installation works out-of-the-box, but there is no migration path from a Docker installation to a high-availability installation. Therefore, you may want to use a Kubernetes installation from the start. +### High-availability Kubernetes Install + +We recommend using Helm, a Kubernetes package manager, to install Rancher on multiple nodes on a dedicated Kubernetes cluster. For RKE clusters, three nodes are required to achieve a high-availability cluster. For K3s clusters, only two nodes are required. + +### Single-node Kubernetes Install + +Another option is to install Rancher with Helm on a Kubernetes cluster, but to only use a single node in the cluster. In this case, the Rancher server doesn't have high availability, which is important for running Rancher in production. + +However, this option is useful if you want to save resources by using a single node in the short term, while preserving a high-availability migration path. In the future, you can add nodes to the cluster to get a high-availability Rancher server. + +### Docker Install + +For test and demonstration purposes, Rancher can be installed with Docker on a single node. + +For Rancher v2.0-v2.4, there is no migration path from a Docker installation to a high-availability installation. Therefore, you may want to use a Kubernetes installation from the start. + +For Rancher v2.5+, the Rancher backup operator can be used to migrate Rancher from the single Docker container install to an installation on a high-availability Kubernetes cluster. For details, refer to the documentation on [migrating Rancher to a new cluster.]({{}}/rancher/v2.x/en/backups/v2.5/migrating-rancher/) + +### Other Options There are also separate instructions for installing Rancher in an air gap environment or behind an HTTP proxy: @@ -58,10 +74,10 @@ When the nodes in your Kubernetes cluster are running and fulfill the [node requ For a longer discussion of Rancher architecture, refer to the [architecture overview,]({{}}/rancher/v2.x/en/overview/architecture) [recommendations for production-grade architecture,]({{}}/rancher/v2.x/en/overview/architecture-recommendations) or our [best practices guide.]({{}}/rancher/v2.x/en/best-practices/deployment-types) -### Prerequisites +# Prerequisites Before installing Rancher, make sure that your nodes fulfill all of the [installation requirements.]({{}}/rancher/v2.x/en/installation/requirements/) -### Architecture Tip +# Architecture Tip For the best performance and greater security, we recommend a separate, dedicated Kubernetes cluster for the Rancher management server. Running user workloads on this cluster is not advised. After deploying Rancher, you can [create or import clusters]({{}}/rancher/v2.x/en/cluster-provisioning/#cluster-creation-in-rancher) for running your workloads. diff --git a/content/rancher/v2.x/en/installation/other-installation-methods/_index.md b/content/rancher/v2.x/en/installation/other-installation-methods/_index.md index b9e2266abf8..0eab7f2d18c 100644 --- a/content/rancher/v2.x/en/installation/other-installation-methods/_index.md +++ b/content/rancher/v2.x/en/installation/other-installation-methods/_index.md @@ -17,4 +17,8 @@ The Docker installation is for development and testing environments only. Since there is only one node and a single Docker container, if the node goes down, there is no copy of the etcd data available on other nodes and you will lose all the data of your Rancher server. -When the Rancher server is installed with Docker, it cannot be migrated to a Kubernetes cluster for a production environment. +The ability to migrate Rancher to a high-availability cluster depends on the Rancher version: + +- For Rancher v2.0-v2.4, there was no migration path from a Docker installation to a high-availability installation. Therefore, if you are using Rancher prior to v2.5, you may want to use a Kubernetes installation from the start. + +- For Rancher v2.5+, the Rancher backup operator can be used to migrate Rancher from the single Docker container install to an installation on a high-availability Kubernetes cluster. For details, refer to the documentation on [migrating Rancher to a new cluster.]({{}}/rancher/v2.x/en/backups/v2.5/migrating-rancher/) \ No newline at end of file diff --git a/content/rancher/v2.x/en/installation/other-installation-methods/air-gap/launch-kubernetes/_index.md b/content/rancher/v2.x/en/installation/other-installation-methods/air-gap/launch-kubernetes/_index.md index ad3e85da78d..0e0390de878 100644 --- a/content/rancher/v2.x/en/installation/other-installation-methods/air-gap/launch-kubernetes/_index.md +++ b/content/rancher/v2.x/en/installation/other-installation-methods/air-gap/launch-kubernetes/_index.md @@ -11,11 +11,11 @@ This section describes how to install a Kubernetes cluster according to our [bes For Rancher prior to v2.4, Rancher should be installed on an [RKE]({{}}/rke/latest/en/) (Rancher Kubernetes Engine) Kubernetes cluster. RKE is a CNCF-certified Kubernetes distribution that runs entirely within Docker containers. -As of Rancher v2.4, the Rancher management server can be installed on either an RKE cluster or a K3s Kubernetes cluster. K3s is also a fully certified Kubernetes distribution released by Rancher, but is newer than RKE. We recommend installing Rancher on K3s because K3s is easier to use, and more lightweight, with a binary size of less than 100 MB. Note: After Rancher is installed on an RKE cluster, there is no migration path to a K3s setup at this time. +In Rancher v2.4, the Rancher management server can be installed on either an RKE cluster or a K3s Kubernetes cluster. K3s is also a fully certified Kubernetes distribution released by Rancher, but is newer than RKE. We recommend installing Rancher on K3s because K3s is easier to use, and more lightweight, with a binary size of less than 100 MB. The Rancher management server can only be run on a Kubernetes cluster in an infrastructure provider where Kubernetes is installed using RKE or K3s. Use of Rancher on hosted Kubernetes providers, such as EKS, is not supported. Note: After Rancher is installed on an RKE cluster, there is no migration path to a K3s setup at this time. -The Rancher management server can only be run on Kubernetes cluster in an infrastructure provider where Kubernetes is installed using RKE or K3s. Use of Rancher on hosted Kubernetes providers, such as EKS, is not supported. +As of Rancher v2.5, Rancher can be installed on any Kubernetes cluster, including hosted Kubernetes providers. -The steps to set up an air-gapped Kubernetes cluster depend on whether RKE or K3s is used to install Kubernetes. +The steps to set up an air-gapped Kubernetes cluster on RKE or K3s are shown below. {{% tabs %}} {{% tab "K3s" %}} diff --git a/content/rancher/v2.x/en/installation/other-installation-methods/air-gap/prepare-nodes/_index.md b/content/rancher/v2.x/en/installation/other-installation-methods/air-gap/prepare-nodes/_index.md index 0c6d0980e56..7465fe5a2cd 100644 --- a/content/rancher/v2.x/en/installation/other-installation-methods/air-gap/prepare-nodes/_index.md +++ b/content/rancher/v2.x/en/installation/other-installation-methods/air-gap/prepare-nodes/_index.md @@ -150,9 +150,9 @@ If you need help with creating a private registry, please refer to the [official {{% tab "Docker" %}} > The Docker installation is for Rancher users that are wanting to test out Rancher. Since there is only one node and a single Docker container, if the node goes down, you will lose all the data of your Rancher server. > -> For running Rancher in production, we recommend installing Rancher on a high-availability Kubernetes cluster. There is no upgrade path to transition your Docker installation to a Kubernetes Installation. -> -> If you want to save resources by using a single node in the short term, while preserving a migration path to a high-availability installation, we recommend installing Rancher on a single-node Kubernetes cluster. +> For Rancher v2.0-v2.4, there is no migration path from a Docker installation to a high-availability installation. Therefore, you may want to use a Kubernetes installation from the start. + +> For Rancher v2.5+, the Rancher backup operator can be used to migrate Rancher from the single Docker container install to an installation on a high-availability Kubernetes cluster. For details, refer to the documentation on [migrating Rancher to a new cluster.]({{}}/rancher/v2.x/en/backups/v2.5/migrating-rancher/) ### 1. Set up a Linux Node diff --git a/content/rancher/v2.x/en/installation/other-installation-methods/single-node-docker/_index.md b/content/rancher/v2.x/en/installation/other-installation-methods/single-node-docker/_index.md index 723d7819ff4..80ea46df0ec 100644 --- a/content/rancher/v2.x/en/installation/other-installation-methods/single-node-docker/_index.md +++ b/content/rancher/v2.x/en/installation/other-installation-methods/single-node-docker/_index.md @@ -8,8 +8,6 @@ aliases: - /rancher/v2.x/en/installation/other-installation-methods/single-node --- -> The Docker installation is for development and testing environments only. When the Rancher server is installed with Docker, it cannot be migrated to a Kubernetes cluster for a production environment. - Rancher can be installed by running a single Docker container. In this installation scenario, you'll install Docker on a single Linux host, and then deploy Rancher on your host using a single Docker container. @@ -17,6 +15,12 @@ In this installation scenario, you'll install Docker on a single Linux host, and > **Want to use an external load balancer?** > See [Docker Install with an External Load Balancer]({{}}/rancher/v2.x/en/installation/options/single-node-install-external-lb) instead. +A Docker installation of Rancher is recommended only for development and testing purposes. The ability to migrate Rancher to a high-availability cluster depends on the Rancher version: + +- For Rancher v2.0-v2.4, there was no migration path from a Docker installation to a high-availability installation. Therefore, if you are using Rancher prior to v2.5, you may want to use a Kubernetes installation from the start. + +- For Rancher v2.5+, the Rancher backup operator can be used to migrate Rancher from the single Docker container install to an installation on a high-availability Kubernetes cluster. For details, refer to the documentation on [migrating Rancher to a new cluster.]({{}}/rancher/v2.x/en/backups/v2.5/migrating-rancher/) + ### Privileged Access for Rancher v2.5+ When the Rancher server is deployed in the Docker container, a local Kubernetes cluster is installed within the container for Rancher to use. Because many features of Rancher run as deployments, and privileged mode is required to run containers within containers, you will need to install Rancher with the `--privileged` option. diff --git a/content/rancher/v2.x/en/overview/architecture/_index.md b/content/rancher/v2.x/en/overview/architecture/_index.md index 8c05752602d..20c2547ddea 100644 --- a/content/rancher/v2.x/en/overview/architecture/_index.md +++ b/content/rancher/v2.x/en/overview/architecture/_index.md @@ -41,7 +41,13 @@ The diagram below shows how users can manipulate both [Rancher-launched Kubernet You can install Rancher on a single node, or on a high-availability Kubernetes cluster. -A high-availability Kubernetes installation is recommended for production. A Docker installation may be used for development and testing purposes, but there is no migration path from a single-node to a high-availability installation. Therefore, you may want to use a Kubernetes installation from the start. +A high-availability Kubernetes installation is recommended for production. + +A Docker installation of Rancher is recommended only for development and testing purposes. The ability to migrate Rancher to a high-availability cluster depends on the Rancher version: + +- For Rancher v2.0-v2.4, there was no migration path from a Docker installation to a high-availability installation. Therefore, if you are using Rancher prior to v2.5, you may want to use a Kubernetes installation from the start. + +- For Rancher v2.5+, the Rancher backup operator can be used to migrate Rancher from the single Docker container install to an installation on a high-availability Kubernetes cluster. For details, refer to the documentation on [migrating Rancher to a new cluster.]({{}}/rancher/v2.x/en/backups/v2.5/migrating-rancher/) The Rancher server, regardless of the installation method, should always run on nodes that are separate from the downstream user clusters that it manages. If Rancher is installed on a high-availability Kubernetes cluster, it should run on a separate cluster from the cluster(s) it manages. From 78b6998bec2f066fc5ae4012f96f0367975cc195 Mon Sep 17 00:00:00 2001 From: Bastian Hofmann Date: Wed, 9 Dec 2020 15:28:16 +0100 Subject: [PATCH 19/49] Add link to S3 credentials example on backup storage configuration page Signed-off-by: Bastian Hofmann --- .../en/backups/v2.5/configuration/storage-config/_index.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/rancher/v2.x/en/backups/v2.5/configuration/storage-config/_index.md b/content/rancher/v2.x/en/backups/v2.5/configuration/storage-config/_index.md index 77a7516d946..3f08a943c93 100644 --- a/content/rancher/v2.x/en/backups/v2.5/configuration/storage-config/_index.md +++ b/content/rancher/v2.x/en/backups/v2.5/configuration/storage-config/_index.md @@ -28,7 +28,7 @@ You can choose to not have any operator-level storage location configured. If yo | Parameter | Description | | -------------- | -------------- | -| Credential Secret | Choose the credentials for S3 from your secrets in Rancher. | +| Credential Secret | Choose the credentials for S3 from your secrets in Rancher. [Example]({{}}/rancher/v2.x/en/backups/v2.5/examples/#example-credential-secret-for-storing-backups-in-s3). | | Bucket Name | Enter the name of the [S3 bucket](https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html) where the backups will be stored. Default: `rancherbackups`. | | Region | The [AWS region](https://aws.amazon.com/about-aws/global-infrastructure/regions_az/) where the S3 bucket is located. | | Folder | The [folder in the S3 bucket](https://docs.aws.amazon.com/AmazonS3/latest/user-guide/using-folders.html) where the backups will be stored. | @@ -111,4 +111,4 @@ nodeSelector: {} tolerations: [] affinity: {} -``` \ No newline at end of file +``` From 58790a4312a68ee8fe97fb67b2a0fbe1e18a26fc Mon Sep 17 00:00:00 2001 From: Catherine Luse Date: Wed, 9 Dec 2020 13:12:18 -0700 Subject: [PATCH 20/49] Update notifier page for Rancher v2.0-2.4 --- .../v2.0.x-v2.4.x/notifiers/_index.md | 170 ++++++++++++------ 1 file changed, 119 insertions(+), 51 deletions(-) diff --git a/content/rancher/v2.x/en/monitoring-alerting/v2.0.x-v2.4.x/notifiers/_index.md b/content/rancher/v2.x/en/monitoring-alerting/v2.0.x-v2.4.x/notifiers/_index.md index a4bb73060ce..b7d4e45e75a 100644 --- a/content/rancher/v2.x/en/monitoring-alerting/v2.0.x-v2.4.x/notifiers/_index.md +++ b/content/rancher/v2.x/en/monitoring-alerting/v2.0.x-v2.4.x/notifiers/_index.md @@ -17,76 +17,144 @@ Rancher integrates with a variety of popular IT services, including: - **Email**: Choose email recipients for alert notifications. - **PagerDuty**: Route notifications to staff by phone, SMS, or personal email. - **WebHooks**: Update a webpage with alert notifications. -- **WeChat**: Send alert notifications to your Enterprise WeChat contacts. +- **WeChat**: (Available as of v2.2.0) Send alert notifications to your Enterprise WeChat contacts. +- **DingTalk**: (Available as of v2.4.6) Send alert notifications to DingChat using a webhook. +- **Microsoft Teams**: (Available as of v2.4.6) Send alert notifications to Teams using a webhook. This section covers the following topics: - [Roles-based access control for notifiers](#roles-based-access-control-for-notifiers) - [Adding notifiers](#adding-notifiers) +- [Configuration](#configuration) - [Managing notifiers](#managing-notifiers) - [Example payload for a webhook alert notifier](#example-payload-for-a-webhook-alert-notifier) -### Roles-based Access Control for Notifiers +# Roles-based Access Control for Notifiers Notifiers are configured at the cluster level. This model ensures that only cluster owners need to configure notifiers, leaving project owners to simply configure alerts in the scope of their projects. You don't need to dispense privileges like SMTP server access or cloud account access. -### Adding Notifiers +# Adding Notifiers Set up a notifier so that you can begin configuring and sending alerts. 1. From the **Global View**, open the cluster that you want to add a notifier. - 1. From the main menu, select **Tools > Notifiers**. Then click **Add Notifier**. - -1. Select the service you want to use as your notifier, and then fill out the form. -{{% accordion id="slack" label="Slack" %}} -1. Enter a **Name** for the notifier. -1. From Slack, create a webhook. For instructions, see the [Slack Documentation](https://get.slack.help/hc/en-us/articles/115005265063-Incoming-WebHooks-for-Slack). -1. From Rancher, enter your Slack webhook **URL**. -1. Enter the name of the channel that you want to send alert notifications in the following format: `#`. - - Both public and private channels are supported. -1. Click **Test**. If the test is successful, the Slack channel you're configuring for the notifier outputs `Slack setting validated`. -{{% /accordion %}} -{{% accordion id="email" label="Email" %}} -1. Enter a **Name** for the notifier. -1. In the **Sender** field, enter an email address available on your mail server that you want to send the notification. -1. In the **Host** field, enter the IP address or hostname for your SMTP server. Example: `smtp.email.com` -1. In the **Port** field, enter the port used for email. Typically, TLS uses `587` and SSL uses `465`. If you're using TLS, make sure **Use TLS** is selected. -1. Enter a **Username** and **Password** that authenticate with the SMTP server. -1. In the **Default Recipient** field, enter the email address that you want to receive the notification. -1. Click **Test**. If the test is successful, Rancher prints `settings validated` and you receive a test notification email. -{{% /accordion %}} -{{% accordion id="pagerduty" label="PagerDuty" %}} -1. Enter a **Name** for the notifier. -1. From PagerDuty, create a Prometheus integration. For instructions, see the [PagerDuty Documentation](https://www.pagerduty.com/docs/guides/prometheus-integration-guide/). -1. From PagerDuty, copy the integration's **Integration Key**. -1. From Rancher, enter the key in the **Service Key** field. -1. Click **Test**. If the test is successful, your PagerDuty endpoint outputs `PagerDuty setting validated`. -{{% /accordion %}} -{{% accordion id="webhook" label="WebHook" %}} -1. Enter a **Name** for the notifier. -1. Using the app of your choice, create a webhook URL. -1. Enter your webhook **URL**. -1. Click **Test**. If the test is successful, the URL you're configuring as a notifier outputs `Webhook setting validated`. -{{% /accordion %}} -{{% accordion id="WeChat" label="WeChat" %}} - -_Available as of v2.2.0_ - -1. Enter a **Name** for the notifier. -1. In the **Corporation ID** field, enter the "EnterpriseID" of your corporation, you could get it from [Profile page](https://work.weixin.qq.com/wework_admin/frame#profile). -1. From Enterprise WeChat, create an application in the [Application page](https://work.weixin.qq.com/wework_admin/frame#apps), and then enter the "AgentId" and "Secret" of this application to the **Application Agent ID** and **Application Secret** fields. -1. Select the **Recipient Type** and then enter a corresponding id to **Default Recipient** field, for example, the party id, tag id or user account that you want to receive the notification. You could get contact information from [Contacts page](https://work.weixin.qq.com/wework_admin/frame#contacts). -{{% /accordion %}} - -1. _Available as of v2.3.0_ - Select **Enable** for **Send Resolved Alerts** if you wish to notify about resolved alerts. +1. Select the service you want to use as your notifier, and then fill out the form. For help filling out the form, refer to the configuration section below. +1. Click **Test.** You should receive a notification confirming that the notifier is configured correctly. 1. Click **Add** to complete adding the notifier. **Result:** Your notifier is added to Rancher. +# Configuration -### Managing Notifiers +- [Slack](#slack) +- [Email](#email) +- [PagerDuty](#pagerduty) +- [Webhook](#webhook) +- [WeChat](#wechat) +- [DingTalk](#dingtalk) +- [MicrosoftTeams](#microsoftteams) + +### Slack + +| Field | Explanation | +|----------|----------------------| +| Name | Enter a **Name** for the notifier. | +| URL | From Slack, create a webhook. For instructions, see the [Slack Documentation](https://get.slack.help/hc/en-us/articles/115005265063-Incoming-WebHooks-for-Slack). Then enter the Slack webhook URL. | +| Default Channel | Enter the name of the channel that you want to send alert notifications in the following format: `#`. Both public and private channels are supported. | +| Proxy URL | Proxy for the Slack webhook. | +| Send Resolved Alerts | _Available as of v2.3.0_ When enabled, you will receive resolved alerts, such as an alert about high CPU usage after the CPU has returned to normal levels. | + +**Validation:** Click **Test**. If the test is successful, the Slack channel you're configuring for the notifier outputs **Slack setting validated.** + +### Email + +| Field | Explanation | +|----------|----------------------| +| Name | Enter a **Name** for the notifier. | +| Default Recipient Address | Enter the email address that you want to receive the notification. | +| Send Resolved Alerts | _Available as of v2.3.0_ When enabled, you will receive resolved alerts, such as an alert about high CPU usage after the CPU has returned to normal levels. | + +SMTP Server Configuration: + +| Field | Explanation | +|----------|----------------------| +| Sender | Enter an email address available on your mail server that you want to send the notification. | +| Host | Enter the IP address or hostname for your SMTP server. Example: `smtp.email.com` | +| Port | In the **Port** field, enter the port used for email. Typically, TLS uses `587` and SSL uses `465`. | +| Use TLS | If you're using TLS, make sure **Use TLS** is selected. | +| Username | Username to authenticate with the SMTP server. | +| Password | Password to authenticate with the SMTP server. | + +**Validation:** Click **Test**. If the test is successful, Rancher prints **settings validated** and you receive a test notification email. + +### PagerDuty + +| Field | Explanation | +|----------|----------------------| +| Name | Enter a **Name** for the notifier. | +| Default Integration Key | From PagerDuty, create a Prometheus integration. For instructions, see the [PagerDuty Documentation](https://www.pagerduty.com/docs/guides/prometheus-integration-guide/). Then enter the integration key. +| Service Key | The same as the integration key. For instructions on creating a Prometheus integration, see the [PagerDuty Documentation](https://www.pagerduty.com/docs/guides/prometheus-integration-guide/). Then enter the integration key. | +| Send Resolved Alerts | _Available as of v2.3.0_ When enabled, you will receive resolved alerts, such as an alert about high CPU usage after the CPU has returned to normal levels. | + +**Validation:** Click **Test**. If the test is successful, your PagerDuty endpoint outputs **PagerDuty setting validated.** + +### Webhook + +| Field | Explanation | +|----------|----------------------| +| Name | Enter a **Name** for the notifier. | +| URL | Using the app of your choice, create a webhook URL. | +| Proxy URL | Proxy for the webhook. | +| Send Resolved Alerts | _Available as of v2.3.0_ When enabled, you will receive resolved alerts, such as an alert about high CPU usage after the CPU has returned to normal levels. | + +**Validation:** Click **Test**. If the test is successful, the URL you're configuring as a notifier outputs **Webhook setting validated.** + +### WeChat + +_Available as of v2.2.0_ + +| Field | Explanation | +|----------|----------------------| +| Name | Enter a **Name** for the notifier. | +| Corporation ID | Enter the "EnterpriseID" of your corporation. You can get it fro the [Profile page](https://work.weixin.qq.com/wework_admin/frame#profile). | +| Application Agent ID | From Enterprise WeChat, create an application in the [Application page](https://work.weixin.qq.com/wework_admin/frame#apps), and then enter the "AgentId" of this application. You will also need to enter the application secret. | +| Application Secret | The secret that corresponds to the Application Agent ID. | +| Recipient Type | Party, tag, or user. | +| Default Recipient | The default recipient ID should correspond to the recipient type. It should be the party ID, tag ID or user account that you want to receive the notification. You could get contact information from [Contacts page](https://work.weixin.qq.com/wework_admin/frame#contacts). | +| Proxy URL | If you are using a proxy, enter the proxy URL. | +| Send Resolved Alerts | _Available as of v2.3.0_ When enabled, you will receive resolved alerts, such as an alert about high CPU usage after the CPU has returned to normal levels. | + +**Validation:** Click **Test.** If the test is successful, you should receive an alert message. + +### DingTalk + +_Available as of v2.4.6_ + +| Field | Explanation | +|----------|----------------------| +| Name | Enter a **Name** for the notifier. | +| Webhook URL | Enter the DingTalk webhook URL. For help setting up the webhook, refer to the [DingTalk documentation.](https://www.alibabacloud.com/help/doc-detail/52872.htm) | +| Secret | Optional: Enter a secret for the DingTalk webhook. | +| Proxy URL | Optional: Enter a proxy for the DingTalk webhook. | +| Send Resolved Alerts | When enabled, you will receive resolved alerts, such as an alert about high CPU usage after the CPU has returned to normal levels. | + +**Validation:** Click **Test.** If the test is successful, the DingTalk notifier output is **DingTalk setting validated.** + +### Microsoft Teams + +_Available as of v2.4.6_ + +| Field | Explanation | +|----------|----------------------| +| Name | Enter a **Name** for the notifier. | +| Webhook URL | Enter the Microsoft Teams webhook URL. For help setting up the webhook, refer to the [Teams Documentation.](https://docs.microsoft.com/en-us/microsoftteams/platform/webhooks-and-connectors/how-to/add-incoming-webhook) | +| Proxy URL | Optional: Enter a proxy for the Teams webhook. | +| Send Resolved Alerts | When enabled, you will receive resolved alerts, such as an alert about high CPU usage after the CPU has returned to normal levels. | + +**Validation:** Click **Test.** If the test is successful, the Teams notifier output is **MicrosoftTeams setting validated.** + +# Managing Notifiers After you set up notifiers, you can manage them. From the **Global** view, open the cluster that you want to manage your notifiers. Select **Tools > Notifiers**. You can: @@ -94,7 +162,7 @@ After you set up notifiers, you can manage them. From the **Global** view, open - **Clone** them, to quickly setup slightly different notifiers. - **Delete** them when they're no longer necessary. -### Example Payload for a Webhook Alert Notifier +# Example Payload for a Webhook Alert Notifier ```json { @@ -130,7 +198,7 @@ After you set up notifiers, you can manage them. From the **Global** view, open } } ``` -### What's Next? +# What's Next? After creating a notifier, set up alerts to receive notifications of Rancher system events. From dcf7422043557d9b214f0edf1a7b3e2846c67cfc Mon Sep 17 00:00:00 2001 From: Catherine Luse Date: Wed, 9 Dec 2020 13:14:39 -0700 Subject: [PATCH 21/49] Fix typo --- .../en/monitoring-alerting/v2.0.x-v2.4.x/notifiers/_index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/rancher/v2.x/en/monitoring-alerting/v2.0.x-v2.4.x/notifiers/_index.md b/content/rancher/v2.x/en/monitoring-alerting/v2.0.x-v2.4.x/notifiers/_index.md index b7d4e45e75a..64598db73ff 100644 --- a/content/rancher/v2.x/en/monitoring-alerting/v2.0.x-v2.4.x/notifiers/_index.md +++ b/content/rancher/v2.x/en/monitoring-alerting/v2.0.x-v2.4.x/notifiers/_index.md @@ -18,7 +18,7 @@ Rancher integrates with a variety of popular IT services, including: - **PagerDuty**: Route notifications to staff by phone, SMS, or personal email. - **WebHooks**: Update a webpage with alert notifications. - **WeChat**: (Available as of v2.2.0) Send alert notifications to your Enterprise WeChat contacts. -- **DingTalk**: (Available as of v2.4.6) Send alert notifications to DingChat using a webhook. +- **DingTalk**: (Available as of v2.4.6) Send alert notifications to DingTalk using a webhook. - **Microsoft Teams**: (Available as of v2.4.6) Send alert notifications to Teams using a webhook. This section covers the following topics: From cc9339114121d811839301e5c5754065447fb76d Mon Sep 17 00:00:00 2001 From: Catherine Luse Date: Wed, 9 Dec 2020 14:30:48 -0700 Subject: [PATCH 22/49] Fix internal link --- .../en/monitoring-alerting/v2.0.x-v2.4.x/notifiers/_index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/rancher/v2.x/en/monitoring-alerting/v2.0.x-v2.4.x/notifiers/_index.md b/content/rancher/v2.x/en/monitoring-alerting/v2.0.x-v2.4.x/notifiers/_index.md index 64598db73ff..a9d4d80ad39 100644 --- a/content/rancher/v2.x/en/monitoring-alerting/v2.0.x-v2.4.x/notifiers/_index.md +++ b/content/rancher/v2.x/en/monitoring-alerting/v2.0.x-v2.4.x/notifiers/_index.md @@ -53,7 +53,7 @@ Set up a notifier so that you can begin configuring and sending alerts. - [Webhook](#webhook) - [WeChat](#wechat) - [DingTalk](#dingtalk) -- [MicrosoftTeams](#microsoftteams) +- [Microsoft Teams](#microsoft-teams) ### Slack From ff6870d72af7c9e5885cbb54e54ad0d0675fc43c Mon Sep 17 00:00:00 2001 From: Catherine Luse Date: Wed, 9 Dec 2020 21:17:40 -0700 Subject: [PATCH 23/49] Document advanced config options for Rancher v2.5 monitoring --- .../cluster-monitoring/prometheus/_index.md | 53 +++++++++++++++++-- 1 file changed, 48 insertions(+), 5 deletions(-) diff --git a/content/rancher/v2.x/en/monitoring-alerting/v2.0.x-v2.4.x/cluster-monitoring/prometheus/_index.md b/content/rancher/v2.x/en/monitoring-alerting/v2.0.x-v2.4.x/cluster-monitoring/prometheus/_index.md index f817e8603ca..b265606bc01 100644 --- a/content/rancher/v2.x/en/monitoring-alerting/v2.0.x-v2.4.x/cluster-monitoring/prometheus/_index.md +++ b/content/rancher/v2.x/en/monitoring-alerting/v2.0.x-v2.4.x/cluster-monitoring/prometheus/_index.md @@ -9,9 +9,16 @@ aliases: _Available as of v2.2.0_ - While configuring monitoring at either the [cluster level]({{}}/rancher/v2.x/en/monitoring-alerting/legacy/monitoring/cluster-monitoring/#enabling-cluster-monitoring) or [project level]({{}}/rancher/v2.x/en/project-admin/tools/monitoring/#enabling-project-monitoring), there are multiple options that can be configured. +- [Basic Configuration](#basic-configuration) +- [Advanced Options](#advanced-options) +- [Node Exporter](#node-exporter) +- [Persistent Storage](#persistent-storage) +- [Remote Storage](#remote-storage) + +# Basic Configuration + Option | Description -------|------------- Data Retention | How long your Prometheus instance retains monitoring data scraped from Rancher objects before it's purged. @@ -24,9 +31,45 @@ Prometheus [CPU Reservation](https://kubernetes.io/docs/concepts/configuration/m Prometheus [Memory Limit](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#meaning-of-memory) | Memory resource limit for the Prometheus pod. Prometheus [Memory Reservation](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#meaning-of-memory) | Memory resource requests for the Prometheus pod. Selector | Ability to select the nodes in which Prometheus and Grafana pods are deployed to. To use this option, the nodes must have labels. -Advanced Options | Since monitoring is an [application](https://github.com/rancher/system-charts/tree/dev/charts/rancher-monitoring) from the [Rancher catalog]({{}}/rancher/v2.x/en/catalog/), it can be [configured like any other catalog application]({{}}/rancher/v2.x/en/catalog/catalog-config/). _Warning: Any modification to the application without understanding the entire application can lead to catastrophic errors._ -## Node Exporter +# Advanced Options + +Since monitoring is an [application](https://github.com/rancher/system-charts/tree/dev/charts/rancher-monitoring) from the [Rancher catalog]({{}}/rancher/v2.x/en/catalog/), it can be configured like any other catalog application, by passing in values to Helm. + +> **Warning:** Any modification to the application without understanding the entire application can lead to catastrophic errors. + +### Prometheus RemoteRead and RemoteWrite + +_Available as of v2.4.0_ + +Prometheus RemoteRead and RemoteWrite can be configured as custom answers in the **Advanced Options** section. + +For more information on remote endpoints and storage, refer to the [Prometheus documentation.](https://prometheus.io/docs/operating/integrations/#remote-endpoints-and-storage) + +The Prometheus operator documentation contains the full [RemoteReadSpec](https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/api.md#remotereadspec) and [RemoteWriteSpec.](https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/api.md#remotewritespec) + +An example configuration would be: + +| Variable | Value | +|--------------|------------| +| `prometheus.remoteWrite[0].url` | `http://mytarget.com` | + +### LivenessProbe and ReadinessProbe + +_Available as of v2.4.0_ + +Prometheus LivenessProbe and ReadinessProbe can be configured as custom answers in the **Advanced Options** section. + +The Kubernetes probe spec is [here.](https://v1-17.docs.kubernetes.io/docs/reference/generated/kubernetes-api/v1.17/#probe-v1-core) + +Some example key-value pairs are: + +| Variable | Value | +|--------------|------------| +| `prometheus.livenessProbe.timeoutSeconds` | 60 | +| `prometheus.readinessProbe.timeoutSeconds` | 60 | + +# Node Exporter The [node exporter](https://github.com/prometheus/node_exporter/blob/master/README.md) is a popular open source exporter, which exposes the metrics for hardware and \*NIX kernels OS. It is designed to monitor the host system. However, there are still issues with namespaces when running it in a container, mostly around filesystem mount spaces. In order to monitor actual network metrics for the container network, the node exporter must be deployed with the `hostNetwork` mode. @@ -34,7 +77,7 @@ When configuring Prometheus and enabling the node exporter, enter a host port in >**Warning:** In order for Prometheus to collect the metrics of the node exporter, after enabling cluster monitoring, you must open the Node Exporter Host Port in the host firewall rules to allow intranet access. By default, `9796` is used as that host port. -## Persistent Storage +# Persistent Storage >**Prerequisite:** Configure one or more [storage class]({{}}/rancher/v2.x/en/cluster-admin/volumes-and-storage/#adding-storage-classes) to use as [persistent storage]({{}}/rancher/v2.x/en/cluster-admin/volumes-and-storage/) for your Prometheus or Grafana pod. @@ -42,7 +85,7 @@ By default, when you enable Prometheus for either a cluster or project, all moni When enabling persistent storage for Prometheus or Grafana, specify the size of the persistent volume and select the [storage class]({{}}/rancher/v2.x/en/cluster-admin/volumes-and-storage/#storage-classes). -## Remote Storage +# Remote Storage >**Prerequisite:** Need a remote storage endpoint to be available. The possible list of integrations is available [here](https://prometheus.io/docs/operating/integrations/) From 6c27c8f38444bf3eee97e725135713daeee1252b Mon Sep 17 00:00:00 2001 From: Catherine Luse Date: Thu, 10 Dec 2020 12:45:33 -0700 Subject: [PATCH 24/49] Correction to RKE docs about vSphere cloud provider --- .../en/config-options/cloud-providers/vsphere/_index.md | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/content/rke/latest/en/config-options/cloud-providers/vsphere/_index.md b/content/rke/latest/en/config-options/cloud-providers/vsphere/_index.md index d73d2169d74..dea75f8121c 100644 --- a/content/rke/latest/en/config-options/cloud-providers/vsphere/_index.md +++ b/content/rke/latest/en/config-options/cloud-providers/vsphere/_index.md @@ -3,13 +3,11 @@ title: vSphere Cloud Provider weight: 254 --- -In order to provision Kubernetes clusters in vSphere with the RKE CLI, you must enable the vSphere cloud provider. - -The vSphere cloud provider must also be enabled in order to provision clusters with Rancher, which uses RKE as a library when provisioning [RKE clusters.]({{}}/rancher/v2.x/en/cluster-provisioning/rke-clusters/) +This section describes how to enable the vSphere cloud provider. You will need to use the `cloud_provider` directive in the cluster YAML file. The [vSphere Cloud Provider](https://vmware.github.io/vsphere-storage-for-kubernetes/documentation/) interacts with VMware infrastructure (vCenter or standalone ESXi server) to provision and manage storage for persistent volumes in a Kubernetes cluster. -This section describes how to enable the vSphere cloud provider. You will need to use the `cloud_provider` directive in the cluster YAML file. +When provisioning Kubernetes using RKE CLI or using [RKE clusters]({{< baseurl >}}/rancher/v2.x/en/cluster-provisioning/rke-clusters/) in Rancher, the vSphere Cloud Provider can be enabled by configuring the `cloud_provider` directive in the cluster YAML file. ### Related Links From c99e2d72151cad9405e2e72aed5b9dccfec597bb Mon Sep 17 00:00:00 2001 From: Catherine Luse Date: Thu, 10 Dec 2020 22:54:33 -0700 Subject: [PATCH 25/49] Document port requirements for RancherD install --- .../installation/requirements/ports/_index.md | 119 +++++++++++++----- 1 file changed, 89 insertions(+), 30 deletions(-) diff --git a/content/rancher/v2.x/en/installation/requirements/ports/_index.md b/content/rancher/v2.x/en/installation/requirements/ports/_index.md index 4a4e74269c9..68a3cef0ef4 100644 --- a/content/rancher/v2.x/en/installation/requirements/ports/_index.md +++ b/content/rancher/v2.x/en/installation/requirements/ports/_index.md @@ -6,14 +6,39 @@ weight: 300 To operate properly, Rancher requires a number of ports to be open on Rancher nodes and on downstream Kubernetes cluster nodes. -## Rancher Nodes +- [Rancher Nodes](#rancher-nodes) + - [Ports for Rancher Server Nodes on K3s](#ports-for-rancher-server-nodes-on-k3s) + - [Ports for Rancher Server Nodes on RKE](#ports-for-rancher-server-nodes-on-rke) + - [Ports for Rancher Server Nodes on RancherD or RKE2](#ports-for-rancher-server-nodes-on-rancherd-or-rke2) + - [Ports for Rancher Server in Docker](#ports-for-rancher-server-in-docker) +- [Downstream Kubernetes Cluster Nodes](#downstream-kubernetes-cluster-nodes) + - [Ports for Rancher Launched Kubernetes Clusters using Node Pools](#ports-for-rancher-launched-kubernetes-clusters-using-node-pools) + - [Ports for Rancher Launched Kubernetes Clusters using Custom Nodes](#ports-for-rancher-launched-kubernetes-clusters-using-custom-nodes) + - [Ports for Hosted Kubernetes Clusters](#ports-for-hosted-kubernetes-clusters) + - [Ports for Registered Clusters](#ports-for-registered-clusters) +- [Other Port Considerations](#other-port-considerations) + - [Commonly Used Ports](#commonly-used-ports) + - [Local Node Traffic](#local-node-traffic) + - [Rancher AWS EC2 Security Group](#rancher-aws-ec2-security-group) + - [Opening SUSE Linux Ports](#opening-suse-linux-ports) + +# Rancher Nodes The following table lists the ports that need to be open to and from nodes that are running the Rancher server. -The port requirements differ based on whether Rancher is installed in a K3s Kubernetes cluster, an RKE Kubernetes cluster, or a single Docker container. +The port requirements differ based on the Rancher server architecture. -{{% tabs %}} -{{% tab "K3s" %}} +As of Rancher v2.5, Rancher can be installed on any Kubernetes cluster. For Rancher installs on a K3s, RKE, or RKE2 Kubernetes cluster, refer to the tabs below. For other Kubernetes distributions, refer to the distribution's documentation for the port requirements for cluster nodes. + +> **Notes:** +> +> - Rancher nodes may also require additional outbound access for any external authentication provider which is configured (LDAP for example). +> - Kubernetes recommends TCP 30000-32767 for node port services. +> - For firewalls, traffic may need to be enabled within the cluster and pod CIDR. + +### Ports for Rancher Server Nodes on K3s + +{{% accordion label="Click to expand" %}} The K3s server needs port 6443 to be accessible by the nodes. @@ -44,8 +69,11 @@ The following tables break down the port requirements for inbound and outbound t | TCP | 2376 | Any node IP from a node created using Node driver | Docker daemon TLS port used by Docker Machine | | TCP | 6443 | Hosted/Imported Kubernetes API | Kubernetes API server | -{{% /tab %}} -{{% tab "RKE" %}} +{{% /accordion %}} + +### Ports for Rancher Server Nodes on RKE + +{{% accordion label="Click to expand" %}} Typically Rancher is installed on three RKE nodes that all have the etcd, control plane and worker roles. @@ -85,8 +113,40 @@ The following tables break down the port requirements for inbound and outbound t | TCP | 6443 | Hosted/Imported Kubernetes API | Kubernetes API server | | TCP | Provider dependent | Port of the Kubernetes API endpoint in hosted cluster | Kubernetes API | -{{% /tab %}} -{{% tab "Docker" %}} +{{% /accordion %}} + +### Ports for Rancher Server Nodes on RancherD or RKE2 + +{{% accordion label="Click to expand" %}} + +The RancherD (or RKE2) server needs port 6443 and 9345 to be accessible by other nodes in the cluster. + +All nodes need to be able to reach other nodes over UDP port 8472 when Flannel VXLAN is used. + +If you wish to utilize the metrics server, you will need to open port 10250 on each node. + +**Important:** The VXLAN port on nodes should not be exposed to the world as it opens up your cluster network to be accessed by anyone. Run your nodes behind a firewall/security group that disables access to port 8472. + +
Inbound Rules for RancherD or RKE2 Server Nodes
+ +| Protocol | Port | Source | Description +|-----|-----|----------------|---| +| TCP | 9345 | RancherD/RKE2 agent nodes | Kubernetes API +| TCP | 6443 | RancherD/RKE2 agent nodes | Kubernetes API +| UDP | 8472 | RancherD/RKE2 server and agent nodes | Required only for Flannel VXLAN +| TCP | 10250 | RancherD/RKE2 server and agent nodes | kubelet +| TCP | 2379 | RancherD/RKE2 server nodes | etcd client port +| TCP | 2380 | RancherD/RKE2 server nodes | etcd peer port +| TCP | 30000-32767 | RancherD/RKE2 server and agent nodes | NodePort port range +| HTTP | 8080 | Load balancer/proxy that does external SSL termination | Rancher UI/API when external SSL termination is used | +| HTTPS | 8443 |
  • hosted/imported Kubernetes
  • any source that needs to be able to use the Rancher UI or API
| Rancher agent, Rancher UI/API, kubectl. Not needed if you have LB doing TLS termination. | + +Typically all outbound traffic is allowed. +{{% /accordion %}} + +### Ports for Rancher Server in Docker + +{{% accordion label="Click to expand" %}} The following tables break down the port requirements for Rancher nodes, for inbound and outbound traffic: @@ -106,16 +166,9 @@ The following tables break down the port requirements for Rancher nodes, for inb | TCP | 2376 | Any node IP from a node created using a node driver | Docker daemon TLS port used by Docker Machine | | TCP | 6443 | Hosted/Imported Kubernetes API | Kubernetes API server | -{{% /tab %}} -{{% /tabs %}} +{{% /accordion %}} -> **Notes:** -> -> - Rancher nodes may also require additional outbound access for any external authentication provider which is configured (LDAP for example). -> - Kubernetes recommends TCP 30000-32767 for node port services. -> - For firewalls, traffic may need to be enabled within the cluster and pod CIDR. - -## Downstream Kubernetes Cluster Nodes +# Downstream Kubernetes Cluster Nodes Downstream Kubernetes clusters run your apps and services. This section describes what ports need to be opened on the nodes in downstream clusters so that Rancher can communicate with them. @@ -131,9 +184,9 @@ The following diagram depicts the ports that are opened for each [cluster type]( > >If security isn't a large concern and you're okay with opening a few additional ports, you can use the table in [Commonly Used Ports](#commonly-used-ports) as your port reference instead of the comprehensive tables below. -{{% tabs %}} +### Ports for Rancher Launched Kubernetes Clusters using Node Pools -{{% tab "Node Pools" %}} +{{% accordion label="Click to expand" %}} The following table depicts the port requirements for [Rancher Launched Kubernetes]({{}}/rancher/v2.x/en/cluster-provisioning/rke-clusters/) with nodes created in an [Infrastructure Provider]({{}}/rancher/v2.x/en/cluster-provisioning/rke-clusters/node-pools/). @@ -142,36 +195,42 @@ The following table depicts the port requirements for [Rancher Launched Kubernet {{< ports-iaas-nodes >}} -{{% /tab %}} +{{% /accordion %}} -{{% tab "Custom Nodes" %}} +### Ports for Rancher Launched Kubernetes Clusters using Custom Nodes + +{{% accordion label="Click to expand" %}} The following table depicts the port requirements for [Rancher Launched Kubernetes]({{}}/rancher/v2.x/en/cluster-provisioning/rke-clusters/) with [Custom Nodes]({{}}/rancher/v2.x/en/cluster-provisioning/rke-clusters/custom-nodes/). {{< ports-custom-nodes >}} -{{% /tab %}} +{{% /accordion %}} -{{% tab "Hosted Clusters" %}} +### Ports for Hosted Kubernetes Clusters + +{{% accordion label="Click to expand" %}} The following table depicts the port requirements for [hosted clusters]({{}}/rancher/v2.x/en/cluster-provisioning/hosted-kubernetes-clusters). {{< ports-imported-hosted >}} -{{% /tab %}} +{{% /accordion %}} -{{% tab "Imported Clusters" %}} +### Ports for Registered Clusters + +Note: Registered clusters were called imported clusters prior to Rancher v2.5. + +{{% accordion label="Click to expand" %}} The following table depicts the port requirements for [imported clusters]({{}}/rancher/v2.x/en/cluster-provisioning/imported-clusters/). {{< ports-imported-hosted >}} -{{% /tab %}} - -{{% /tabs %}} +{{% /accordion %}} -## Other Port Considerations +# Other Port Considerations ### Commonly Used Ports @@ -193,7 +252,7 @@ However, this traffic may be blocked when: In these cases, you have to explicitly allow this traffic in your host firewall, or in case of public/private cloud hosted machines (i.e. AWS or OpenStack), in your security group configuration. Keep in mind that when using a security group as source or destination in your security group, explicitly opening ports only applies to the private interface of the nodes / instances. -### Rancher AWS EC2 security group +### Rancher AWS EC2 Security Group When using the [AWS EC2 node driver]({{}}/rancher/v2.x/en/cluster-provisioning/rke-clusters/node-pools/ec2/) to provision cluster nodes in Rancher, you can choose to let Rancher create a security group called `rancher-nodes`. The following rules are automatically added to this security group. From 6b3e9d3be57b7d9d3d0d904a2474071b076da042 Mon Sep 17 00:00:00 2001 From: Catherine Luse Date: Mon, 14 Dec 2020 10:36:59 -0700 Subject: [PATCH 26/49] Add 'Enable Istio with Pod Security Policies' page to 2.5 Istio docs --- .../enable-istio-with-psp/_index.md | 51 +++++++++++++++++++ 1 file changed, 51 insertions(+) create mode 100644 content/rancher/v2.x/en/istio/v2.5/setup/enable-istio-in-cluster/enable-istio-with-psp/_index.md diff --git a/content/rancher/v2.x/en/istio/v2.5/setup/enable-istio-in-cluster/enable-istio-with-psp/_index.md b/content/rancher/v2.x/en/istio/v2.5/setup/enable-istio-in-cluster/enable-istio-with-psp/_index.md new file mode 100644 index 00000000000..39bb3dd572c --- /dev/null +++ b/content/rancher/v2.x/en/istio/v2.5/setup/enable-istio-in-cluster/enable-istio-with-psp/_index.md @@ -0,0 +1,51 @@ +--- +title: Enable Istio with Pod Security Policies +aliases: + - /rancher/v2.x/en/cluster-admin/tools/istio/setup/enable-istio-in-cluster/enable-istio-with-psp + - /rancher/v2.x/en/istio/legacy/setup/enable-istio-in-cluster/enable-istio-with-psp +--- + + >**Note:** The following guide is only for RKE provisioned clusters. + +If you have restrictive Pod Security Policies enabled, then Istio may not be able to function correctly, because it needs certain permissions in order to install itself and manage pod infrastructure. In this section, we will configure a cluster with PSPs enabled for an Istio install, and also set up the Istio CNI plugin. + +The Istio CNI plugin removes the need for each application pod to have a privileged `NET_ADMIN` container. For further information, see the [Istio CNI Plugin docs](https://istio.io/docs/setup/additional-setup/cni). Please note that the [Istio CNI Plugin is in alpha](https://istio.io/about/feature-stages/). + +- 1. [Configure the System Project Policy to allow Istio install.](#2-configure-the-system-project-policy-to-allow-istio-install) +- 2. [Install the CNI plugin in the System project.](#3-install-the-cni-plugin-in-the-system-project) +- 3. [Install Istio.](#4-install-istio) + +### 1. Configure the System Project Policy to allow Istio install + +1. From the main menu of the **Dashboard**, select **Projects/Namespaces**. +1. Find the **Project: System** project and select the **⋮ > Edit**. +1. Change the Pod Security Policy option to be unrestricted, then click Save. + + +### 2. Install the CNI Plugin in the System Project + +1. From the main menu of the **Dashboard**, select **Projects/Namespaces**. +1. Select the **Project: System** project. +1. Choose **Tools > Catalogs** in the navigation bar. +1. Add a catalog with the following: + 1. Name: istio-cni + 1. Catalog URL: https://github.com/istio/cni + 1. Branch: The branch that matches your current release, for example: `release-1.4`. +1. From the main menu select **Apps** +1. Click Launch and select istio-cni +1. Update the namespace to be "kube-system" +1. In the answers section, click "Edit as YAML" and paste in the following, then click launch: + +``` +--- + logLevel: "info" + excludeNamespaces: + - "istio-system" + - "kube-system" +``` + +### 3. Install Istio + +Follow the [primary instructions]({{}}/rancher/v2.x/en/cluster-admin/tools/istio/setup/enable-istio-in-cluster/), adding a custom answer: `istio_cni.enabled: true`. + +After Istio has finished installing, the Apps page in System Projects should show both istio and `istio-cni` applications deployed successfully. Sidecar injection will now be functional. From 00b882647e62bd43919cfb715964fc991763c63e Mon Sep 17 00:00:00 2001 From: Catherine Luse Date: Thu, 10 Dec 2020 23:36:39 -0700 Subject: [PATCH 27/49] Document that all-in-one Jaeger is bundled with Istio --- content/rancher/v2.x/en/istio/v2.5/_index.md | 25 ++++++++++++++++---- 1 file changed, 21 insertions(+), 4 deletions(-) diff --git a/content/rancher/v2.x/en/istio/v2.5/_index.md b/content/rancher/v2.x/en/istio/v2.5/_index.md index a41e62e1e4b..8c3777276c3 100644 --- a/content/rancher/v2.x/en/istio/v2.5/_index.md +++ b/content/rancher/v2.x/en/istio/v2.5/_index.md @@ -18,13 +18,10 @@ This core service mesh provides features that include but are not limited to the After [setting up istio]({{}}/rancher/v2.x/en/cluster-admin/tools/istio/setup) you can leverage Istio's control plane functionality through the Cluster Explorer, `kubectl`, or `istioctl`. -Rancher's Istio integration comes with a comprehensive visualization aid: - -- **Get the full picture of your microservice architecture with Kiali.** [Kiali](https://www.kiali.io/) provides a diagram that shows the services within a service mesh and how they are connected, including the traffic rates and latencies between them. You can check the health of the service mesh, or drill down to see the incoming and outgoing requests to a single component. - Istio needs to be set up by a `cluster-admin` before it can be used in a project. - [What's New in Rancher v2.5](#what-s-new-in-rancher-v2-5) +- [Tools Bundled with Istio](#tools-bundled-with-istio) - [Prerequisites](#prerequisites) - [Setup Guide](#setup-guide) - [Remove Istio](#remove-istio) @@ -44,6 +41,26 @@ Istio has migrated away from Helm as a way to install Istio and now provides ins This Helm chart will be available via the Apps and Marketplace in the UI. A user that has access to the Rancher Chart's catalog will need to set up Istio before it can be used in the project. +# Tools Bundled with Istio + +Our [Istio](https://istio.io/) installer wraps the istioctl binary commands in a handy Helm chart, including an overlay file option to allow complex customization. + +It also includes the following: + +### Kiali + +Kiali is a comprehensive visualization aid used for graphing traffic flow throughout the service mesh. It allows you to see how they are connected, including the traffic rates and latencies between them. + +You can check the health of the service mesh, or drill down to see the incoming and outgoing requests to a single component. + +### Jaeger + +_Bundled as of v2.5.4_ + +Our Istio installer includes a quick-start, all-in-one installation of [Jaeger,](https://www.jaegertracing.io/) a tool used for tracing distributed systems. + +Note that this is not a production-qualified deployment of Jaeger. This deployment uses an in-memory storage component, while a persistent storage component is recommended for production. For more information on which deployment strategy you may need, refer to the [Jaeger documentation.](https://www.jaegertracing.io/docs/latest/operator/#production-strategy) + # Prerequisites Before enabling Istio, we recommend that you confirm that your Rancher worker nodes have enough [CPU and memory]({{}}/rancher/v2.x/en/cluster-admin/tools/istio/resources) to run all of the components of Istio. From cd262cee1b2a5869094494b4c64e294c7fb44dc2 Mon Sep 17 00:00:00 2001 From: Caleb Bron Date: Mon, 14 Dec 2020 15:28:49 -0700 Subject: [PATCH 28/49] Adds custom values to global DNS --- .../v2.x/en/helm-charts/legacy-catalogs/globaldns/_index.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/content/rancher/v2.x/en/helm-charts/legacy-catalogs/globaldns/_index.md b/content/rancher/v2.x/en/helm-charts/legacy-catalogs/globaldns/_index.md index 53945664571..a30aa567414 100644 --- a/content/rancher/v2.x/en/helm-charts/legacy-catalogs/globaldns/_index.md +++ b/content/rancher/v2.x/en/helm-charts/legacy-catalogs/globaldns/_index.md @@ -38,6 +38,7 @@ By default, only [global administrators]({{}}/rancher/v2.x/en/admin-set 1. From the **Global View**, select **Tools > Global DNS Providers**. 1. To add a provider, choose from the available provider options and configure the Global DNS Provider with necessary credentials and an optional domain. 1. (Optional) Add additional users so they could use the provider when creating Global DNS entries as well as manage the Global DNS provider. +1. (Optional) Pass any custom values in the Additional Options section. {{% accordion id="route53" label="Route53" %}} 1. Enter a **Name** for the provider. @@ -96,6 +97,7 @@ The [global administrators]({{}}/rancher/v2.x/en/admin-settings/rbac/gl - Root Domain - Access Key & Secret Key - Members +- Custom values 1. From the **Global View**, select **Tools > Global DNS Providers**. From b55af3d4a764e310b069d3db34161ecb7663d76d Mon Sep 17 00:00:00 2001 From: Sebastiaan van Steenis Date: Tue, 15 Dec 2020 15:17:59 +0100 Subject: [PATCH 29/49] Add information on configuring add-on tolerations --- .../en/config-options/add-ons/_index.md | 6 + .../en/config-options/add-ons/dns/_index.md | 111 +++++++++++++----- .../add-ons/ingress-controllers/_index.md | 55 ++++++--- .../add-ons/metrics-server/_index.md | 27 ++++- .../add-ons/network-plugins/_index.md | 85 +++++++++++--- 5 files changed, 223 insertions(+), 61 deletions(-) diff --git a/content/rke/latest/en/config-options/add-ons/_index.md b/content/rke/latest/en/config-options/add-ons/_index.md index f2cb7765e3b..89695c786c3 100644 --- a/content/rke/latest/en/config-options/add-ons/_index.md +++ b/content/rke/latest/en/config-options/add-ons/_index.md @@ -49,3 +49,9 @@ _Applies to v0.2.3 and higher_ | kube-dns | - `beta.kubernetes.io/os:NotIn:windows`
- `node-role.kubernetes.io/worker` `Exists` | none | - `NoSchedule:Exists`
- `NoExecute:Exists`
- `CriticalAddonsOnly:Exists` | | nginx-ingress | - `beta.kubernetes.io/os:NotIn:windows`
- `node-role.kubernetes.io/worker` `Exists` | none | - `NoSchedule:Exists`
- `NoExecute:Exists` | | metrics-server | - `beta.kubernetes.io/os:NotIn:windows`
- `node-role.kubernetes.io/worker` `Exists` | none | - `NoSchedule:Exists`
- `NoExecute:Exists` | + +## Tolerations + +_Available as of v1.2.4_ + +Tolerations can be configured per add-on and apply to Deployment resources. The configured tolerations will replace the existing tolerations so make sure you configure all the tolerations you need. See the specific add-on doc pages for more information. diff --git a/content/rke/latest/en/config-options/add-ons/dns/_index.md b/content/rke/latest/en/config-options/add-ons/dns/_index.md index 2e63a5c25be..1904441038e 100644 --- a/content/rke/latest/en/config-options/add-ons/dns/_index.md +++ b/content/rke/latest/en/config-options/add-ons/dns/_index.md @@ -34,16 +34,16 @@ If you only want the CoreDNS pod to be deployed on specific nodes, you can set a ```yaml nodes: - - address: 1.1.1.1 - role: [controlplane,worker,etcd] - user: root - labels: - app: dns +- address: 1.1.1.1 + role: [controlplane,worker,etcd] + user: root + labels: + app: dns dns: - provider: coredns - node_selector: - app: dns + provider: coredns + node_selector: + app: dns ``` ## Configuring CoreDNS @@ -56,10 +56,37 @@ When you set `upstreamnameservers`, the `provider` also needs to be set. ```yaml dns: - provider: coredns - upstreamnameservers: - - 1.1.1.1 - - 8.8.4.4 + provider: coredns + upstreamnameservers: + - 1.1.1.1 + - 8.8.4.4 +``` + +### Tolerations + +_Available as of v1.2.4_ + +The configured tolerations apply to the `coredns` and the `coredns-autoscaler` Deployment. + +``` +dns: + provider: coredns + tolerations: + - key: "node.kubernetes.io/unreachable" + operator: "Exists" + effect: "NoExecute" + tolerationseconds: 300 + - key: "node.kubernetes.io/not-ready" + operator: "Exists" + effect: "NoExecute" + tolerationseconds: 300 +``` + +To check for applied tolerations on the `coredns` and `coredns-autoscaler` Deployment, use the following commands: + +``` +kubectl -n kube-system get deploy coredns -o jsonpath='{.spec.template.spec.tolerations}' +kubectl -n kube-system get deploy coredns-autoscaler -o jsonpath='{.spec.template.spec.tolerations}' ``` # kube-dns @@ -76,16 +103,16 @@ If you only want the kube-dns pod to be deployed on specific nodes, you can set ```yaml nodes: - - address: 1.1.1.1 - role: [controlplane,worker,etcd] - user: root - labels: - app: dns +- address: 1.1.1.1 + role: [controlplane,worker,etcd] + user: root + labels: + app: dns dns: - provider: kube-dns - node_selector: - app: dns + provider: kube-dns + node_selector: + app: dns ``` ## Configuring kube-dns @@ -100,10 +127,38 @@ When you set `upstreamnameservers`, the `provider` also needs to be set. ```yaml dns: - provider: kube-dns - upstreamnameservers: - - 1.1.1.1 - - 8.8.4.4 + provider: kube-dns + upstreamnameservers: + - 1.1.1.1 + - 8.8.4.4 +``` + +### Tolerations + +_Available as of v1.2.4_ + +The configured tolerations apply to the `kube-dns` and the `kube-dns-autoscaler` Deployment. + +``` +dns: + provider: kube-dns + tolerations: + - key: "node.kubernetes.io/unreachable" + operator: "Exists" + effect: "NoExecute" + tolerationseconds: 300 + - key: "node.kubernetes.io/not-ready" + operator: "Exists" + effect: "NoExecute" + tolerationseconds: 300 + +``` + +To check for applied tolerations on the `coredns` and `coredns-autoscaler` Deployment, use the following commands: + +``` +kubectl get deploy kube-dns -n kube-system -o jsonpath='{.spec.template.spec.tolerations}' +kubectl get deploy kube-dns-autoscaler -n kube-system -o jsonpath='{.spec.template.spec.tolerations}' ``` # Disabling deployment of a DNS provider @@ -114,7 +169,7 @@ You can disable the default DNS provider by specifying `none` to the dns `provi ```yaml dns: - provider: none + provider: none ``` # NodeLocal DNS @@ -137,9 +192,9 @@ The `ip_address` parameter is used to configure what link-local IP address will ```yaml dns: - provider: coredns - nodelocal: - ip_address: "169.254.20.10" + provider: coredns + nodelocal: + ip_address: "169.254.20.10" ``` > **Note:** When enabling NodeLocal DNS on an existing cluster, pods that are currently running will not be modified, the updated `/etc/resolv.conf` configuration will take effect only for pods started after enabling NodeLocal DNS. diff --git a/content/rke/latest/en/config-options/add-ons/ingress-controllers/_index.md b/content/rke/latest/en/config-options/add-ons/ingress-controllers/_index.md index ea60f92abe5..ad70ea165a4 100644 --- a/content/rke/latest/en/config-options/add-ons/ingress-controllers/_index.md +++ b/content/rke/latest/en/config-options/add-ons/ingress-controllers/_index.md @@ -18,16 +18,41 @@ If you only wanted ingress controllers to be deployed on specific nodes, you can ```yaml nodes: - - address: 1.1.1.1 - role: [controlplane,worker,etcd] - user: root - labels: - app: ingress +- address: 1.1.1.1 + role: [controlplane,worker,etcd] + user: root + labels: + app: ingress ingress: - provider: nginx - node_selector: - app: ingress + provider: nginx + node_selector: + app: ingress +``` + +## Tolerations + +_Available as of v1.2.4_ + +The configured tolerations apply to the `default-http-backend` Deployment. + +``` +ingress: + tolerations: + - key: "node.kubernetes.io/unreachable" + operator: "Exists" + effect: "NoExecute" + tolerationseconds: 300 + - key: "node.kubernetes.io/not-ready" + operator: "Exists" + effect: "NoExecute" + tolerationseconds: 300 +``` + +To check for applied tolerations `default-http-backend` Deployment, use the following commands: + +``` +kubectl -n ingress-nginx get deploy default-http-backend -o jsonpath='{.spec.template.spec.tolerations}' ``` ## Disabling the Default Ingress Controller @@ -44,12 +69,12 @@ For the configuration of NGINX, there are configuration options available in Kub ```yaml ingress: - provider: nginx - options: - map-hash-bucket-size: "128" - ssl-protocols: SSLv2 - extra_args: - enable-ssl-passthrough: "" + provider: nginx + options: + map-hash-bucket-size: "128" + ssl-protocols: SSLv2 + extra_args: + enable-ssl-passthrough: "" ``` ### Disabling NGINX Ingress Default Backend @@ -58,7 +83,7 @@ As of v0.20.0, you can disable the [default backend service](https://kubernetes. ```yaml ingress: - default_backend: false + default_backend: false ``` > **What happens if the field is omitted?** The value of `default_backend` will default to `true`. This maintains behavior with older versions of `rke`. However, a future version of `rke` will change the default value to `false`. diff --git a/content/rke/latest/en/config-options/add-ons/metrics-server/_index.md b/content/rke/latest/en/config-options/add-ons/metrics-server/_index.md index 61f0d303601..21c541dc344 100644 --- a/content/rke/latest/en/config-options/add-ons/metrics-server/_index.md +++ b/content/rke/latest/en/config-options/add-ons/metrics-server/_index.md @@ -9,6 +9,31 @@ RKE will deploy Metrics Server as a Deployment. The image used for Metrics Server is under the [`system_images` directive]({{}}/rke/latest/en/config-options/system-images/). For each Kubernetes version, there is a default image associated with the Metrics Server, but these can be overridden by changing the image tag in `system_images`. +## Tolerations + +_Available as of v1.2.4_ + +The configured tolerations apply to the `metrics-server` Deployment. + +``` +monitoring: + tolerations: + - key: "node.kubernetes.io/unreachable" + operator: "Exists" + effect: "NoExecute" + tolerationseconds: 300 + - key: "node.kubernetes.io/not-ready" + operator: "Exists" + effect: "NoExecute" + tolerationseconds: 300 +``` + +To check for applied tolerations on the `metrics-server` Deployment, use the following commands: + +``` +kubectl -n kube-system get deploy metrics-server -o jsonpath='{.spec.template.spec.tolerations}' +``` + ## Disabling the Metrics Server _Available as of v0.2.0_ @@ -17,5 +42,5 @@ You can disable the default controller by specifying `none` to the monitoring `p ```yaml monitoring: - provider: none + provider: none ``` diff --git a/content/rke/latest/en/config-options/add-ons/network-plugins/_index.md b/content/rke/latest/en/config-options/add-ons/network-plugins/_index.md index 7da2af08643..88828943652 100644 --- a/content/rke/latest/en/config-options/add-ons/network-plugins/_index.md +++ b/content/rke/latest/en/config-options/add-ons/network-plugins/_index.md @@ -17,7 +17,7 @@ By default, the network plug-in is `canal`. If you want to use another network p ```yaml # Setting the flannel network plug-in network: - plugin: flannel + plugin: flannel ``` The images used for network plug-ins are under the [`system_images` directive]({{}}/rke/latest/en/config-options/system-images/). For each Kubernetes version, there are default images associated with each network plug-in, but these can be overridden by changing the image tag in `system_images`. @@ -28,7 +28,7 @@ You can disable deploying a network plug-in by specifying `none` to the network ```yaml network: - plugin: none + plugin: none ``` # Network Plug-in Options @@ -39,10 +39,10 @@ Besides the different images that could be used to deploy network plug-ins, cert ```yaml network: - plugin: canal - options: - canal_iface: eth1 - canal_flannel_backend_type: vxlan + plugin: canal + options: + canal_iface: eth1 + canal_flannel_backend_type: vxlan ``` #### Canal Interface @@ -50,14 +50,40 @@ network: By setting the `canal_iface`, you can configure the interface to use for inter-host communication. The `canal_flannel_backend_type` option allows you to specify the type of [flannel backend](https://github.com/coreos/flannel/blob/master/Documentation/backends.md) to use. By default the `vxlan` backend is used. +## Canal Network Plug-in Tolerations + +_Available as of v1.2.4_ + +The configured tolerations apply to the `calico-kube-controllers` Deployment. + +``` +network: + plugin: canal + tolerations: + - key: "node.kubernetes.io/unreachable" + operator: "Exists" + effect: "NoExecute" + tolerationseconds: 300 + - key: "node.kubernetes.io/not-ready" + operator: "Exists" + effect: "NoExecute" + tolerationseconds: 300 +``` + +To check for applied tolerations on the `calico-kube-controllers` Deployment, use the following command: + +``` +kubectl -n kube-system get deploy calico-kube-controllers -o jsonpath='{.spec.template.spec.tolerations}' +``` + ## Flannel Network Plug-in Options ```yaml network: - plugin: flannel - options: - flannel_iface: eth1 - flannel_backend_type: vxlan + plugin: flannel + options: + flannel_iface: eth1 + flannel_backend_type: vxlan ``` #### Flannel Interface @@ -69,9 +95,9 @@ The `flannel_backend_type` option allows you to specify the type of [flannel bac ```yaml network: - plugin: calico - options: - calico_cloud_provider: aws + plugin: calico + options: + calico_cloud_provider: aws ``` #### Calico Cloud Provider @@ -82,20 +108,45 @@ Calico currently only supports 2 cloud providers, AWS or GCE, which can be set u - `aws` - `gce` +## Calico Network Plug-in Tolerations + +_Available as of v1.2.4_ + +The configured tolerations apply to the `calico-kube-controllers` Deployment. + +``` +network: + plugin: calico + tolerations: + - key: "node.kubernetes.io/unreachable" + operator: "Exists" + effect: "NoExecute" + tolerationseconds: 300 + - key: "node.kubernetes.io/not-ready" + operator: "Exists" + effect: "NoExecute" + tolerationseconds: 300 +``` + +To check for applied tolerations on the `calico-kube-controllers` Deployment, use the following command: + +``` +kubectl -n kube-system get deploy calico-kube-controllers -o jsonpath='{.spec.template.spec.tolerations}' +``` + ## Weave Network Plug-in Options ```yaml network: - plugin: weave - weave_network_provider: - password: "Q]SZOQ5wp@n$oijz" + plugin: weave + weave_network_provider: + password: "Q]SZOQ5wp@n$oijz" ``` #### Weave encryption Weave encryption can be enabled by passing a string password to the network provider config. - ## Custom Network Plug-ins It is possible to add a custom network plug-in by using the [user-defined add-on functionality]({{}}/rke/latest/en/config-options/add-ons/user-defined-add-ons/) of RKE. In the `addons` field, you can add the add-on manifest of a cluster that has the network plugin-that you want, as shown in [this example.]({{}}/rke/latest/en/config-options/add-ons/network-plugins/custom-network-plugin-example) From e6653ace3bac19761e6278e731f2b61d21c6ae18 Mon Sep 17 00:00:00 2001 From: Nick Gerace Date: Tue, 15 Dec 2020 17:46:18 -0500 Subject: [PATCH 30/49] Add release semver to syslog support in logging Add release semver 2.5.4 to syslog support in logging. Change syslog example headers to markdown headers rather than bolded text. --- content/rancher/v2.x/en/logging/v2.5/_index.md | 18 +++++++++++++----- 1 file changed, 13 insertions(+), 5 deletions(-) diff --git a/content/rancher/v2.x/en/logging/v2.5/_index.md b/content/rancher/v2.x/en/logging/v2.5/_index.md index 7e0be50fbc5..01771387ab4 100644 --- a/content/rancher/v2.x/en/logging/v2.5/_index.md +++ b/content/rancher/v2.x/en/logging/v2.5/_index.md @@ -82,9 +82,13 @@ According to the [Banzai Cloud documentation,](https://banzaicloud.com/docs/one- > You can define `outputs` (destinations where you want to send your log messages, for example, Elasticsearch, or and Amazon S3 bucket), and `flows` that use filters and selectors to route log messages to the appropriate outputs. You can also define cluster-wide outputs and flows, for example, to use a centralized output that namespaced users cannot modify. -### Examples +# Examples -**Cluster Output to ElasticSearch:** Let's say you wanted to send all logs in your cluster to an `elasticsearch` cluster. First, we create a cluster output. +Once logging is installed, you can use these examples to help craft your own logging pipeline. + +### Cluster Output to ElasticSearch + +Let's say you wanted to send all logs in your cluster to an `elasticsearch` cluster. First, we create a cluster output. ```yaml apiVersion: logging.banzaicloud.io/v1beta1 @@ -116,7 +120,9 @@ spec: We should now see our configured index with logs in it. -**Output to Splunk:** What if we have an application team who only wants logs from a specific namespaces sent to a `splunk` server? For this case, we can use namespaced outputs and flows. +### Output to Splunk + +What if we have an application team who only wants logs from a specific namespaces sent to a `splunk` server? For this case, we can use namespaced outputs and flows. Before we start, let's set up that team's application: `coolapp`. @@ -176,7 +182,9 @@ spec: - "devteam-splunk" ``` -**Unsupported Ouput:** For the final example, we create an output to write logs to a destination that is not supported out of the box (e.g. syslog): +### Unsupported Output + +For the final example, we create an output to write logs to a destination that is not supported out of the box (e.g. syslog): ```yaml apiVersion: v1 @@ -266,7 +274,7 @@ spec: Let's break down what is happening here. First, we create a deployment of a container that has the additional `syslog` plugin and accepts logs forwarded from another `fluentd`. Next we create an output configured as a forwarder to our deployment. The deployment `fluentd` will then forward all logs to the configured `syslog` destination. -> **Note on syslog** The example provides an overview on using unsupported plugins, but `syslog` support is coming in a future Rancher v2.x release. +> **Note on syslog** Official `syslog` support is coming in Rancher v2.5.4. However, this example still provides an overview on using unsupported plugins. # Working with Taints and Tolerations From b4c94c706e84500464e0bb235ab18d331b4b740b Mon Sep 17 00:00:00 2001 From: Donnie Adams Date: Fri, 11 Dec 2020 07:27:42 -0700 Subject: [PATCH 31/49] Include description for security groups and private API The advanced user can use security groups to enable private-only API access, but still allow Rancher to communicate with the cluster. A note about this configuration is added with a link to the AWS documentation. --- .../hosted-kubernetes-clusters/eks/_index.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/content/rancher/v2.x/en/cluster-provisioning/hosted-kubernetes-clusters/eks/_index.md b/content/rancher/v2.x/en/cluster-provisioning/hosted-kubernetes-clusters/eks/_index.md index 7aec2359fb1..63b763b38ef 100644 --- a/content/rancher/v2.x/en/cluster-provisioning/hosted-kubernetes-clusters/eks/_index.md +++ b/content/rancher/v2.x/en/cluster-provisioning/hosted-kubernetes-clusters/eks/_index.md @@ -137,9 +137,11 @@ Configuring Public/Private API access is an advanced use case. For details, refe ### Private-only API Endpoints -If you enable private and disable public API endpoint access when creating a cluster, then there is an extra step you must take in order for Rancher to connect to the cluster successfully. In this case, a pop-up will be displayed with instructions on how to complete the setup. Once the cluster is provisioned, you can run the displayed command anywhere you can connect to the cluster's Kubernetes API. +If you enable private and disable public API endpoint access when creating a cluster, then there is an extra step you must take in order for Rancher to connect to the cluster successfully. In this case, a pop-up will be displayed with a command that you will run on the cluster to register it with Rancher. Once the cluster is provisioned, you can run the displayed command anywhere you can connect to the cluster's Kubernetes API. -Note: you can avoid this extra step by enabling both private and public API endpoint access on cluster creation. You can disable public access after the cluster is created and in an active state and Rancher will continue to communicate with the EKS cluster. +There are two ways to avoid this extra manual step: +* You can create the cluster with both private and public API endpoint access on cluster creation. You can disable public access after the cluster is created and in an active state and Rancher will continue to communicate with the EKS cluster. +* You can ensure that Rancher shares a subnet with the EKS cluster. Then security groups can be used to enable Rancher to communicate with the cluster's API endpoint. In this case, the command to register the cluster is not needed, and Rancher will be able to communicate with your cluster. For more information on configuring security groups, refer to the [security groups documentation](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_SecurityGroups.html). ### Public Access Endpoints From cab6c790aa0481265f548e367b2c6be4c3ba2a69 Mon Sep 17 00:00:00 2001 From: Catherine Luse Date: Tue, 15 Dec 2020 23:09:33 -0700 Subject: [PATCH 32/49] Edit docs on routes, receivers and PrometheusRules --- .../v2.0.x-v2.4.x/notifiers/_index.md | 14 ++++---- .../en/monitoring-alerting/v2.5/_index.md | 4 +++ .../v2.5/configuration/_index.md | 4 ++- .../v2.5/configuration/alertmanager/_index.md | 36 +++++++++++-------- .../configuration/prometheusrules/_index.md | 26 ++++++++------ 5 files changed, 51 insertions(+), 33 deletions(-) diff --git a/content/rancher/v2.x/en/monitoring-alerting/v2.0.x-v2.4.x/notifiers/_index.md b/content/rancher/v2.x/en/monitoring-alerting/v2.0.x-v2.4.x/notifiers/_index.md index a9d4d80ad39..3b64aedfa6c 100644 --- a/content/rancher/v2.x/en/monitoring-alerting/v2.0.x-v2.4.x/notifiers/_index.md +++ b/content/rancher/v2.x/en/monitoring-alerting/v2.0.x-v2.4.x/notifiers/_index.md @@ -63,7 +63,7 @@ Set up a notifier so that you can begin configuring and sending alerts. | URL | From Slack, create a webhook. For instructions, see the [Slack Documentation](https://get.slack.help/hc/en-us/articles/115005265063-Incoming-WebHooks-for-Slack). Then enter the Slack webhook URL. | | Default Channel | Enter the name of the channel that you want to send alert notifications in the following format: `#`. Both public and private channels are supported. | | Proxy URL | Proxy for the Slack webhook. | -| Send Resolved Alerts | _Available as of v2.3.0_ When enabled, you will receive resolved alerts, such as an alert about high CPU usage after the CPU has returned to normal levels. | +| Send Resolved Alerts | _Available as of v2.3.0_ Whether to send a follow-up notification if an alert has been resolved (e.g. [Resolved] High CPU Usage) | **Validation:** Click **Test**. If the test is successful, the Slack channel you're configuring for the notifier outputs **Slack setting validated.** @@ -73,7 +73,7 @@ Set up a notifier so that you can begin configuring and sending alerts. |----------|----------------------| | Name | Enter a **Name** for the notifier. | | Default Recipient Address | Enter the email address that you want to receive the notification. | -| Send Resolved Alerts | _Available as of v2.3.0_ When enabled, you will receive resolved alerts, such as an alert about high CPU usage after the CPU has returned to normal levels. | +| Send Resolved Alerts | _Available as of v2.3.0_ Whether to send a follow-up notification if an alert has been resolved (e.g. [Resolved] High CPU Usage) | SMTP Server Configuration: @@ -95,7 +95,7 @@ SMTP Server Configuration: | Name | Enter a **Name** for the notifier. | | Default Integration Key | From PagerDuty, create a Prometheus integration. For instructions, see the [PagerDuty Documentation](https://www.pagerduty.com/docs/guides/prometheus-integration-guide/). Then enter the integration key. | Service Key | The same as the integration key. For instructions on creating a Prometheus integration, see the [PagerDuty Documentation](https://www.pagerduty.com/docs/guides/prometheus-integration-guide/). Then enter the integration key. | -| Send Resolved Alerts | _Available as of v2.3.0_ When enabled, you will receive resolved alerts, such as an alert about high CPU usage after the CPU has returned to normal levels. | +| Send Resolved Alerts | _Available as of v2.3.0_ Whether to send a follow-up notification if an alert has been resolved (e.g. [Resolved] High CPU Usage) | **Validation:** Click **Test**. If the test is successful, your PagerDuty endpoint outputs **PagerDuty setting validated.** @@ -106,7 +106,7 @@ SMTP Server Configuration: | Name | Enter a **Name** for the notifier. | | URL | Using the app of your choice, create a webhook URL. | | Proxy URL | Proxy for the webhook. | -| Send Resolved Alerts | _Available as of v2.3.0_ When enabled, you will receive resolved alerts, such as an alert about high CPU usage after the CPU has returned to normal levels. | +| Send Resolved Alerts | _Available as of v2.3.0_ Whether to send a follow-up notification if an alert has been resolved (e.g. [Resolved] High CPU Usage) | **Validation:** Click **Test**. If the test is successful, the URL you're configuring as a notifier outputs **Webhook setting validated.** @@ -123,7 +123,7 @@ _Available as of v2.2.0_ | Recipient Type | Party, tag, or user. | | Default Recipient | The default recipient ID should correspond to the recipient type. It should be the party ID, tag ID or user account that you want to receive the notification. You could get contact information from [Contacts page](https://work.weixin.qq.com/wework_admin/frame#contacts). | | Proxy URL | If you are using a proxy, enter the proxy URL. | -| Send Resolved Alerts | _Available as of v2.3.0_ When enabled, you will receive resolved alerts, such as an alert about high CPU usage after the CPU has returned to normal levels. | +| Send Resolved Alerts | _Available as of v2.3.0_ Whether to send a follow-up notification if an alert has been resolved (e.g. [Resolved] High CPU Usage) | **Validation:** Click **Test.** If the test is successful, you should receive an alert message. @@ -137,7 +137,7 @@ _Available as of v2.4.6_ | Webhook URL | Enter the DingTalk webhook URL. For help setting up the webhook, refer to the [DingTalk documentation.](https://www.alibabacloud.com/help/doc-detail/52872.htm) | | Secret | Optional: Enter a secret for the DingTalk webhook. | | Proxy URL | Optional: Enter a proxy for the DingTalk webhook. | -| Send Resolved Alerts | When enabled, you will receive resolved alerts, such as an alert about high CPU usage after the CPU has returned to normal levels. | +| Send Resolved Alerts | Whether to send a follow-up notification if an alert has been resolved (e.g. [Resolved] High CPU Usage) | **Validation:** Click **Test.** If the test is successful, the DingTalk notifier output is **DingTalk setting validated.** @@ -150,7 +150,7 @@ _Available as of v2.4.6_ | Name | Enter a **Name** for the notifier. | | Webhook URL | Enter the Microsoft Teams webhook URL. For help setting up the webhook, refer to the [Teams Documentation.](https://docs.microsoft.com/en-us/microsoftteams/platform/webhooks-and-connectors/how-to/add-incoming-webhook) | | Proxy URL | Optional: Enter a proxy for the Teams webhook. | -| Send Resolved Alerts | When enabled, you will receive resolved alerts, such as an alert about high CPU usage after the CPU has returned to normal levels. | +| Send Resolved Alerts | Whether to send a follow-up notification if an alert has been resolved (e.g. [Resolved] High CPU Usage) | **Validation:** Click **Test.** If the test is successful, the Teams notifier output is **MicrosoftTeams setting validated.** diff --git a/content/rancher/v2.x/en/monitoring-alerting/v2.5/_index.md b/content/rancher/v2.x/en/monitoring-alerting/v2.5/_index.md index 8130b139c28..3abcea08ed0 100644 --- a/content/rancher/v2.x/en/monitoring-alerting/v2.5/_index.md +++ b/content/rancher/v2.x/en/monitoring-alerting/v2.5/_index.md @@ -136,6 +136,8 @@ To see the Prometheus Rules, install `rancher-monitoring`. Then go to the **Clus
Rules in the Prometheus UI
![Prometheus Rules UI]({{}}/img/rancher/prometheus-rules-ui.png) +For more information on Prometheus Rules in Rancher, see [this page.](./configuration/prometheusrules) + ### Viewing Active Alerts in Alertmanager When `rancher-monitoring` is installed, the Prometheus Alertmanager UI is deployed. @@ -148,6 +150,8 @@ To see the Prometheus Rules, install `rancher-monitoring`. Then go to the **Clus **Result:** The Alertmanager UI opens in a new tab. For help with configuration, refer to the [official Alertmanager documentation.](https://prometheus.io/docs/alerting/latest/alertmanager/) +For more information on configuring Alertmanager in Rancher, see [this page.](./configuration/alertmanager) +
The Alertmanager UI
![Alertmanager UI]({{}}/img/rancher/alertmanager-ui.png) diff --git a/content/rancher/v2.x/en/monitoring-alerting/v2.5/configuration/_index.md b/content/rancher/v2.x/en/monitoring-alerting/v2.5/configuration/_index.md index c8ab7edbac1..2479ff2314c 100644 --- a/content/rancher/v2.x/en/monitoring-alerting/v2.5/configuration/_index.md +++ b/content/rancher/v2.x/en/monitoring-alerting/v2.5/configuration/_index.md @@ -85,7 +85,9 @@ An example PodMonitor can be found [here.](https://github.com/prometheus-operato ### PrometheusRule -Prometheus rule files are held in PrometheusRule custom resources. Use the label selector field ruleSelector in the Prometheus object to define the rule files that you want to be mounted into Prometheus. An example PrometheusRule is on [this page.](https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/user-guides/alerting.md) +Prometheus rule files are held in PrometheusRule custom resources. For users who are familiar with Prometheus, a PrometheusRule contains the alerting and recording rules that you would normally place in a [Prometheus rule file](https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/). + +Use the label selector field ruleSelector in the Prometheus object to define the rule files that you want to be mounted into Prometheus. An example PrometheusRule is on [this page.](https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/user-guides/alerting.md) ### Alertmanager Config diff --git a/content/rancher/v2.x/en/monitoring-alerting/v2.5/configuration/alertmanager/_index.md b/content/rancher/v2.x/en/monitoring-alerting/v2.5/configuration/alertmanager/_index.md index 29cd61bae69..ad986dc02f4 100644 --- a/content/rancher/v2.x/en/monitoring-alerting/v2.5/configuration/alertmanager/_index.md +++ b/content/rancher/v2.x/en/monitoring-alerting/v2.5/configuration/alertmanager/_index.md @@ -6,6 +6,7 @@ weight: 1 The [Alertmanager Config](https://prometheus.io/docs/alerting/latest/configuration/#configuration-file) Secret contains the configuration of an Alertmanager instance that sends out notifications based on alerts it receives from Prometheus. - [Overview](#overview) + - [Connecting Routes and PrometheusRules](#connecting-routes-and-prometheusrules) - [Creating Receivers in the Rancher UI](#creating-receivers-in-the-rancher-ui) - [Receiver Configuration](#receiver-configuration) - [Slack](#slack) @@ -38,10 +39,17 @@ The full spec for the Alertmanager configuration file and what it takes in can b For more information, refer to the [official Prometheus documentation about configuring routes.](https://www.prometheus.io/docs/alerting/latest/configuration/#route) +### Connecting Routes and PrometheusRules + +When you define a Rule within a RuleGroup of a PrometheusRule, the spec of the Rule itself contains labels that are used by Prometheus to figure out which Route should receive this Alert. For example, an Alert with the label `team: front-end` will be sent to all Routes that match on that label. + # Creating Receivers in the Rancher UI _Available as of v2.5.4_ -> **Prerequisite:** The monitoring application needs to be installed. +> **Prerequisites:** +> +>- The monitoring application needs to be installed. +>- If you configured monitoring with an existing Alertmanager Secret, it must have a format that is supported by Rancher's UI. Otherwise you will only be able to make changes based on modifying the Alertmanager Secret directly. Note: We are continuing to make enhancements to what kinds of Alertmanager Configurations we can support using the Routes and Receivers UI, so please [file an issue](https://github.com/rancher/rancher/issues/new) if you have a request for a feature enhancement. To create notification receivers in the Rancher UI, @@ -56,7 +64,7 @@ To create notification receivers in the Rancher UI, The notification integrations are configured with the `receiver`, which is explained in the [Prometheus documentation.](https://prometheus.io/docs/alerting/latest/configuration/#receiver) -Rancher v2.5.4 introduced the capability to configure reducers by filling out forms in the Rancher UI. +Rancher v2.5.4 introduced the capability to configure receivers by filling out forms in the Rancher UI. {{% tabs %}} {{% tab "Rancher v2.5.4+" %}} @@ -77,23 +85,23 @@ The custom receiver option can be used to configure any receiver in YAML that ca | Field | Type | Description | |------|--------------|------| | URL | String | Enter your Slack webhook URL. For instructions to create a Slack webhook, see the [Slack documentation.](https://get.slack.help/hc/en-us/articles/115005265063-Incoming-WebHooks-for-Slack) | -| Default Channel | String | Enter the name of the channel that you want to send alert notifications in the following format: `#` | +| Default Channel | String | Enter the name of the channel that you want to send alert notifications in the following format: `#`. | | Proxy URL | String | Proxy for the webhook notifications. | -| Enable send resolved alerts | Bool | When true, you will receive alerts through the notifier even if the alert condition is no longer true. For example, if an alert is triggered because your CPU is too high, you will still receive the alert after CPU goes back to normal levels. | +| Enable Send Resolved Alerts | Bool | Whether to send a follow-up notification if an alert has been resolved (e.g. [Resolved] High CPU Usage). | ### Email | Field | Type | Description | |------|--------------|------| | Default Recipient Address | String | The email address that will receive notifications. | -| Enable send resolved alerts | Bool | When true, you will receive alerts through the notifier even if the alert condition is no longer true. For example, if an alert is triggered because your CPU is too high, you will still receive the alert after CPU goes back to normal levels. | +| Enable Send Resolved Alerts | Bool | Whether to send a follow-up notification if an alert has been resolved (e.g. [Resolved] High CPU Usage). | SMTP options: | Field | Type | Description | |------|--------------|------| | Sender | String | Enter an email address available on your SMTP mail server that you want to send the notification from. | -| Host | String | Enter the IP address or hostname for your SMTP server. Example: `smtp.email.com` | +| Host | String | Enter the IP address or hostname for your SMTP server. Example: `smtp.email.com`. | | Use TLS | Bool | Use TLS for encryption. | | Username | String | Enter a username to authenticate with the SMTP server. | | Password | String | Enter a password to authenticate with the SMTP server. | @@ -102,10 +110,10 @@ SMTP options: | Field | Type | Description | |------|------|-------| -| Integration Type | String | Events API v2 or Prometheus. | +| Integration Type | String | `Events API v2` or `Prometheus`. | | Default Integration Key | String | For instructions to get an integration key, see the [PagerDuty documentation.](https://www.pagerduty.com/docs/guides/prometheus-integration-guide/) | | Proxy URL | String | Proxy for the PagerDuty notifications. | -| Enable send resolved alerts | Bool | When true, you will receive alerts through the notifier even if the alert condition is no longer true. For example, if an alert is triggered because your CPU is too high, you will still receive the alert after CPU goes back to normal levels. | +| Enable Send Resolved Alerts | Bool | Whether to send a follow-up notification if an alert has been resolved (e.g. [Resolved] High CPU Usage). | ### Opsgenie @@ -113,7 +121,7 @@ SMTP options: |------|-------------| | API Key | For instructions to get an API key, refer to the [Opsgenie documentation.](https://docs.opsgenie.com/docs/api-key-management) | | Proxy URL | Proxy for the Opsgenie notifications. | -| Enable send resolved alerts | When true, you will receive alerts through the notifier even if the alert condition is no longer true. For example, if an alert is triggered because your CPU is too high, you will still receive the alert after CPU goes back to normal levels. | +| Enable Send Resolved Alerts | Whether to send a follow-up notification if an alert has been resolved (e.g. [Resolved] High CPU Usage). | Opsgenie Responders: @@ -127,8 +135,8 @@ Opsgenie Responders: | Field | Description | |-------|--------------| | URL | Webhook URL for the app of your choice. | -| Proxy URL | Proxy for the webhook notification | -| Enable send resolved alerts | When true, you will receive alerts through the notifier even if the alert condition is no longer true. For example, if an alert is triggered because your CPU is too high, you will still receive the alert after CPU goes back to normal levels. | +| Proxy URL | Proxy for the webhook notification. | +| Enable Send Resolved Alerts | Whether to send a follow-up notification if an alert has been resolved (e.g. [Resolved] High CPU Usage). | ### Custom @@ -153,21 +161,21 @@ The route needs to refer to a [receiver](#receiver-configuration) that has alrea | Field | Default | Description | |-------|--------------|---------| -| Group By | N/a | The labels by which incoming alerts are grouped together. For example, `[ group_by: '[' , ... ']' ]` Multiple alerts coming in for labels such as `cluster=A` and `alertname=LatencyHigh` can be batched into a single group. To aggregate by all possible labels, use the special value `'...'` as the sole label name, for example: `group_by: ['...']` Grouping by `...` effectively disables aggregation entirely, passing through all alerts as-is. This is unlikely to be what you want, unless you have a very low alert volume or your upstream notification system performs its own grouping. +| Group By | N/a | The labels by which incoming alerts are grouped together. For example, `[ group_by: '[' , ... ']' ]` Multiple alerts coming in for labels such as `cluster=A` and `alertname=LatencyHigh` can be batched into a single group. To aggregate by all possible labels, use the special value `'...'` as the sole label name, for example: `group_by: ['...']` Grouping by `...` effectively disables aggregation entirely, passing through all alerts as-is. This is unlikely to be what you want, unless you have a very low alert volume or your upstream notification system performs its own grouping. | | Group Wait | 30s | How long to wait to buffer alerts of the same group before sending initially. | | Group Interval | 5m | How long to wait before sending an alert that has been added to a group of alerts for which an initial notification has already been sent. | | Repeat Interval | 4h | How long to wait before re-sending a given alert that has already been sent. | ### Matching -The **Match** field refers to a set of equality matchers an alert has to fulfill to match the node. When you add key-value pairs to the Rancher UI, they correspond to the YAML in this format: +The **Match** field refers to a set of equality matchers used to identify which alerts to send to a given Route based on labels defined on that alert. When you add key-value pairs to the Rancher UI, they correspond to the YAML in this format: ```yaml match: [ : , ... ] ``` -The **Match Regex** field refers to a set of regex-matchers an alert has to fulfill to match the node. When you add key-value pairs in the Rancher UI, they correspond to the YAML in this format: +The **Match Regex** field refers to a set of regex-matchers used to identify which alerts to send to a given Route based on labels defined on that alert. When you add key-value pairs in the Rancher UI, they correspond to the YAML in this format: ```yaml match_re: diff --git a/content/rancher/v2.x/en/monitoring-alerting/v2.5/configuration/prometheusrules/_index.md b/content/rancher/v2.x/en/monitoring-alerting/v2.5/configuration/prometheusrules/_index.md index 7a479b31c68..5ba169f87f3 100644 --- a/content/rancher/v2.x/en/monitoring-alerting/v2.5/configuration/prometheusrules/_index.md +++ b/content/rancher/v2.x/en/monitoring-alerting/v2.5/configuration/prometheusrules/_index.md @@ -3,27 +3,28 @@ title: PrometheusRules weight: 2 --- -The PrometheusRules CRD defines a group of Prometheus alerting and/or recording rules. +A PrometheusRule defines a group of Prometheus alerting and/or recording rules. - [About PrometheusRule Custom Resources](#about-prometheusrule-custom-resources) +- [Connecting Routes and PrometheusRules](#connecting-routes-and-prometheusrules) - [Creating PrometheusRules in the Rancher UI](#creating-prometheusrules-in-the-rancher-ui) - [Configuration](#configuration) - [Rule Group](#rule-group) - [Alerting Rules](#alerting-rules) - [Recording Rules](#recording-rules) -# About PrometheusRule Custom Resources +### About PrometheusRule Custom Resources Prometheus rule files are held in PrometheusRule custom resources. -The PrometheusRule custom resource defines a RuleGroup with your desired rules. Each specifies the following: +A PrometheusRule allows you to define one or more RuleGroups. Each RuleGroup consists of a set of alerting or recording rules with the following fields: - The name of the new alert or record - A PromQL (Prometheus query language) expression for the new alert or record - Labels that should be attached to the alert or record that identify it (e.g. cluster name or severity) - Annotations that encode any additional important pieces of information that need to be displayed on the notification for an alert (e.g. summary, description, message, runbook URL, etc.). This field is not required for recording rules. -Alerting rules define alert conditions based on PromQL queries, and recording rules precompute frequently needed or computationally expensive queries at defined intervals. +Alerting rules define alert conditions based on PromQL queries. Recording rules precompute frequently needed or computationally expensive queries at defined intervals. For more information on what fields can be specified, please look at the [Prometheus Operator spec.](https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/api.md#prometheusrulespec) @@ -31,8 +32,11 @@ Use the label selector field `ruleSelector` in the Prometheus object to define t For examples, refer to the Prometheus documentation on [recording rules](https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/) and [alerting rules.](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/) +### Connecting Routes and PrometheusRules -# Creating PrometheusRules in the Rancher UI +When you define a Rule within a RuleGroup of a PrometheusRule, the spec of the Rule itself contains labels that are used by Prometheus to figure out which Route should receive this Alert. For example, an Alert with the label `team: front-end` will be sent to all Routes that match on that label. + +### Creating PrometheusRules in the Rancher UI _Available as of v2.5.4_ @@ -43,7 +47,7 @@ To create rule groups in the Rancher UI, 1. Click **Cluster Explorer > Monitoring** and click **Prometheus Rules.** 1. Click **Create.** 1. Enter a **Group Name.** -1. Configure the rules. A rule group may contain either alert rules or recording rules, but not both. For help filling out the forms, refer to the configuration options below. +1. Configure the rules. In Rancher's UI, we expect a rule group to contain either alert rules or recording rules, but not both. For help filling out the forms, refer to the configuration options below. 1. Click **Create.** **Result:** Alerts can be configured to send notifications to the receiver(s). @@ -52,7 +56,7 @@ To create rule groups in the Rancher UI, {{% tabs %}} {{% tab "Rancher v2.5.4" %}} -Rancher v2.5.4 introduced the capability to configure reducers by filling out forms in the Rancher UI. +Rancher v2.5.4 introduced the capability to configure PrometheusRules by filling out forms in the Rancher UI. ### Rule Group @@ -70,12 +74,12 @@ Rancher v2.5.4 introduced the capability to configure reducers by filling out fo | Field | Description | |-------|----------------| | Alert Name | The name of the alert. Must be a valid label value. | -| Wait to fire for | Duration in seconds. Alerts are considered firing once they have been returned for this long. Alerts which have not yet fired for long enough are considered pending. | -| PromQL Expression | The PromQL expression to evaluate. Every evaluation cycle this is evaluated at the current time, and all resultant time series become pending/firing alerts. For more information, refer to the [Prometheus documentation](https://prometheus.io/docs/prometheus/latest/querying/basics/) or our [example PromQL expressions.](../expression) | +| Wait To Fire For | Duration in seconds. Alerts are considered firing once they have been returned for this long. Alerts which have not yet fired for long enough are considered pending. | +| PromQL Expression | The PromQL expression to evaluate. Prometheus will evaluate the current value of this PromQL expression on every evaluation cycle and all resultant time series become pending/firing alerts. For more information, refer to the [Prometheus documentation](https://prometheus.io/docs/prometheus/latest/querying/basics/) or our [example PromQL expressions.](../expression) | | Labels | Labels to add or overwrite for each alert. | | Severity | When enabled, labels are attached to the alert or record that identify it by the severity level. | | Severity Label Value | Critical, warning, or none | -| Annotations | Annotations are a set of informational labels that can be used to store longer additional information, such as alert descriptions or runbook links. A [runbook](https://docs.gitlab.com/ee/user/project/clusters/runbooks/) is a set of documentation about how to handle alerts. The annotation values can be [templated.](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/#templating) | +| Annotations | Annotations are a set of informational labels that can be used to store longer additional information, such as alert descriptions or runbook links. A [runbook](https://en.wikipedia.org/wiki/Runbook) is a set of documentation about how to handle alerts. The annotation values can be [templated.](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/#templating) | ### Recording Rules @@ -84,7 +88,7 @@ Rancher v2.5.4 introduced the capability to configure reducers by filling out fo | Field | Description | |-------|----------------| | Time Series Name | The name of the time series to output to. Must be a valid metric name. | -| PromQL Expression | The PromQL expression to evaluate. Every evaluation cycle this is evaluated at the current time, and the result recorded as a new set of time series with the metric name as given by 'record'. For more information about expressions, refer to the [Prometheus documentation](https://prometheus.io/docs/prometheus/latest/querying/basics/) or our [example PromQL expressions.](../expression) | +| PromQL Expression | The PromQL expression to evaluate. Prometheus will evaluate the current value of this PromQL expression on every evaluation cycle and the result recorded as a new set of time series with the metric name as given by 'record'. For more information about expressions, refer to the [Prometheus documentation](https://prometheus.io/docs/prometheus/latest/querying/basics/) or our [example PromQL expressions.](../expression) | | Labels | Labels to add or overwrite before storing the result. | {{% /tab %}} From 1c1bc35e929fb8c1dff96fc53da82dbcff42c423 Mon Sep 17 00:00:00 2001 From: Catherine Luse Date: Mon, 14 Dec 2020 14:15:03 -0700 Subject: [PATCH 33/49] Update CNI docs for Istio in Rancher v2.5.4+ #2877 --- .../setup/enable-istio-in-cluster/_index.md | 7 +- .../enable-istio-with-psp/_index.md | 72 ++++++++++++++++--- 2 files changed, 69 insertions(+), 10 deletions(-) diff --git a/content/rancher/v2.x/en/istio/v2.5/setup/enable-istio-in-cluster/_index.md b/content/rancher/v2.x/en/istio/v2.5/setup/enable-istio-in-cluster/_index.md index 52da36b2041..0270e0fb722 100644 --- a/content/rancher/v2.x/en/istio/v2.5/setup/enable-istio-in-cluster/_index.md +++ b/content/rancher/v2.x/en/istio/v2.5/setup/enable-istio-in-cluster/_index.md @@ -6,8 +6,11 @@ aliases: - /rancher/v2.x/en/istio/setup/enable-istio-in-cluster --- - ->**Prerequisite:** Only a user with the `cluster-admin` [Kubernetes default role](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#user-facing-roles) assigned can configure and install Istio in a Kubernetes cluster. +>**Prerequisites:** +> +>- Only a user with the `cluster-admin` [Kubernetes default role](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#user-facing-roles) assigned can configure and install Istio in a Kubernetes cluster. +>- If you have pod security policies, you will need to install Istio with the CNI enabled. For details, see [this section.](./enable-istio-with-psp) +>- To install Istio on an RKE2 cluster, additional steps are enabled. For details, see [this section.](./rke2) 1. From the **Cluster Explorer**, navigate to available **Charts** in **Apps & Marketplace** 1. Select the Istio chart from the rancher provided charts diff --git a/content/rancher/v2.x/en/istio/v2.5/setup/enable-istio-in-cluster/enable-istio-with-psp/_index.md b/content/rancher/v2.x/en/istio/v2.5/setup/enable-istio-in-cluster/enable-istio-with-psp/_index.md index 39bb3dd572c..6dd0b28f60c 100644 --- a/content/rancher/v2.x/en/istio/v2.5/setup/enable-istio-in-cluster/enable-istio-with-psp/_index.md +++ b/content/rancher/v2.x/en/istio/v2.5/setup/enable-istio-in-cluster/enable-istio-with-psp/_index.md @@ -5,23 +5,76 @@ aliases: - /rancher/v2.x/en/istio/legacy/setup/enable-istio-in-cluster/enable-istio-with-psp --- - >**Note:** The following guide is only for RKE provisioned clusters. - If you have restrictive Pod Security Policies enabled, then Istio may not be able to function correctly, because it needs certain permissions in order to install itself and manage pod infrastructure. In this section, we will configure a cluster with PSPs enabled for an Istio install, and also set up the Istio CNI plugin. The Istio CNI plugin removes the need for each application pod to have a privileged `NET_ADMIN` container. For further information, see the [Istio CNI Plugin docs](https://istio.io/docs/setup/additional-setup/cni). Please note that the [Istio CNI Plugin is in alpha](https://istio.io/about/feature-stages/). -- 1. [Configure the System Project Policy to allow Istio install.](#2-configure-the-system-project-policy-to-allow-istio-install) -- 2. [Install the CNI plugin in the System project.](#3-install-the-cni-plugin-in-the-system-project) -- 3. [Install Istio.](#4-install-istio) +The steps differ based on the Rancher version. + +{{% tabs %}} +{{% tab "v2.5.4+" %}} + +> **Prerequisites:** +> +> - The cluster must be an RKE Kubernetes cluster. +> - The cluster must have been created with a default PodSecurityPolicy. +> +> To enable pod security policy support when creating a Kubernetes cluster in the Rancher UI, go to Advanced Options. In the Pod Security Policy Support section, click Enabled. Then select a default pod security policy. + +1. [Set the PodSecurityPolicy to unrestricted](#1-set-the-podsecuritypolicy-to-unrestricted) +2. [Enable the CNI](#2-enable-the-cni) +3. [Verify that the CNI is working.](#3-verify-that-the-cni-is-working) + +### 1. Set the PodSecurityPolicy to unrestricted + +An unrestricted PSP allows Istio to be installed. + +Set the PSP to `unrestricted` in the project where is Istio is installed, or the project where you plan to install Istio. + +1. From the cluster view of the **Cluster Manager,** select **Projects/Namespaces.** +1. Find the **Project: System** and select the **⋮ > Edit**. +1. Change the Pod Security Policy option to be unrestricted, then click **Save.** + +### 2. Enable the CNI + +When installing or upgrading Istio through **Apps & Marketplace,** + +1. Click **Components.** +2. Check the box next to **Enabled CNI.** +3. Finish installing or upgrading Istio. + +The CNI can also be enabled by editing the `values.yaml`: + +``` +istio_cni.enabled: true +``` + +Istio should install successfully with the CNI enabled in the cluster. + +### 3. Verify that the CNI is working + +Verify that the CNI is working by deploying a [sample application](https://istio.io/latest/docs/examples/bookinfo/) or deploying one of your own applications. + +{{% /tab %}} +{{% tab "v2.5.0-v2.5.3" %}} + +> **Prerequisites:** +> +> - The cluster must be an RKE Kubernetes cluster. +> - The cluster must have been created with a default PodSecurityPolicy. +> +> To enable pod security policy support when creating a Kubernetes cluster in the Rancher UI, go to Advanced Options. In the Pod Security Policy Support section, click Enabled. Then select a default pod security policy. + +1. [Configure the System Project Policy to allow Istio install.](#1-configure-the-system-project-policy-to-allow-istio-install) +2. [Install the CNI plugin in the System project.](#2-install-the-cni-plugin-in-the-system-project) +3. [Install Istio.](#3-install-istio) ### 1. Configure the System Project Policy to allow Istio install -1. From the main menu of the **Dashboard**, select **Projects/Namespaces**. -1. Find the **Project: System** project and select the **⋮ > Edit**. +1. From the cluster view of the **Cluster Manager,** select **Projects/Namespaces.** +1. Find the **Project: System** and select the **⋮ > Edit**. 1. Change the Pod Security Policy option to be unrestricted, then click Save. - ### 2. Install the CNI Plugin in the System Project 1. From the main menu of the **Dashboard**, select **Projects/Namespaces**. @@ -49,3 +102,6 @@ The Istio CNI plugin removes the need for each application pod to have a privile Follow the [primary instructions]({{}}/rancher/v2.x/en/cluster-admin/tools/istio/setup/enable-istio-in-cluster/), adding a custom answer: `istio_cni.enabled: true`. After Istio has finished installing, the Apps page in System Projects should show both istio and `istio-cni` applications deployed successfully. Sidecar injection will now be functional. + +{{% /tab %}} +{{% /tabs %}} \ No newline at end of file From 71b7e1ac642e593ebbf22a4029d363bf263d41f2 Mon Sep 17 00:00:00 2001 From: Catherine Luse Date: Wed, 16 Dec 2020 00:00:38 -0700 Subject: [PATCH 34/49] Document installing Istio on an RKE2 cluster #2877 https://github.com/rancher/docs/issues/2877#issuecomment-739077980 --- content/rancher/v2.x/en/istio/v2.5/_index.md | 36 ++++++++++++++++++++ 1 file changed, 36 insertions(+) diff --git a/content/rancher/v2.x/en/istio/v2.5/_index.md b/content/rancher/v2.x/en/istio/v2.5/_index.md index 8c3777276c3..a3151365d15 100644 --- a/content/rancher/v2.x/en/istio/v2.5/_index.md +++ b/content/rancher/v2.x/en/istio/v2.5/_index.md @@ -28,6 +28,7 @@ Istio needs to be set up by a `cluster-admin` before it can be used in a project - [Migrate from Previous Istio Version](#migrate-from-previous-istio-version) - [Accessing Visualizations](#accessing-visualizations) - [Architecture](#architecture) +- [Additional steps for installing Istio on an RKE2 cluster](#additional-steps-for-installing-istio-on-an-rke2-cluster) # What's New in Rancher v2.5 @@ -65,6 +66,8 @@ Note that this is not a production-qualified deployment of Jaeger. This deployme Before enabling Istio, we recommend that you confirm that your Rancher worker nodes have enough [CPU and memory]({{}}/rancher/v2.x/en/cluster-admin/tools/istio/resources) to run all of the components of Istio. +If you are installing Istio on RKE2 cluster, some additional steps are required. For details, see [this section.](#additional-steps-for-installing-istio-on-an-rke2-cluster) + # Setup Guide Refer to the [setup guide]({{}}/rancher/v2.x/en/cluster-admin/tools/istio/setup) for instructions on how to set up Istio and use it in a project. @@ -116,3 +119,36 @@ By default, each Rancher-provisioned cluster has one NGINX ingress controller al ### Egress Support By default the Egress gateway is disabled, but can be enabled on install or upgrade through the values.yaml or via the [overlay file]({{}}/rancher/v2.x/en/istio/setup/enable-istio-in-cluster/#overlay-file). + +# Additional Steps for Installing Istio on an RKE2 Cluster + +Through the **Cluster Explorer,** when installing or upgrading Istio through **Apps & Marketplace,** + +1. Click **Components.** +1. Check the box next to **Enabled CNI.** +1. Add a custom overlay file specifying `cniBinDir` and `cniConfDir`. For more information on these options, refer to the [Istio documentation.](https://istio.io/latest/docs/setup/additional-setup/cni/#helm-chart-parameters) An example is below: + + ```yaml + apiVersion: install.istio.io/v1alpha1 + kind: IstioOperator + spec: + components: + cni: + enabled: true + values: + cni: + image: rancher/istio-install-cni:1.7.3 + excludeNamespaces: + - istio-system + - kube-system + logLevel: info + cniBinDir: /opt/cni/bin + cniConfDir: /etc/cni/net.d + ``` +1. After installing Istio, you'll notice the cni-node pods in the istio-system namespace in a CrashLoopBackoff error. Manually edit the `istio-cni-node` daemonset to include the following on the `install-cni` container: + ```yaml + securityContext: + privileged: true + ``` + +**Result:** Now you should be able to utilize Istio as desired, including sidecar injection and monitoring via Kiali. From 2b43b90bcbdb722cf3d2bb1f0fb3bbee9bdaa866 Mon Sep 17 00:00:00 2001 From: Catherine Luse Date: Wed, 16 Dec 2020 00:04:26 -0700 Subject: [PATCH 35/49] Fix YAML front matter in Istio doc --- .../enable-istio-in-cluster/enable-istio-with-psp/_index.md | 1 + 1 file changed, 1 insertion(+) diff --git a/content/rancher/v2.x/en/istio/v2.5/setup/enable-istio-in-cluster/enable-istio-with-psp/_index.md b/content/rancher/v2.x/en/istio/v2.5/setup/enable-istio-in-cluster/enable-istio-with-psp/_index.md index 6dd0b28f60c..7c5f1df618a 100644 --- a/content/rancher/v2.x/en/istio/v2.5/setup/enable-istio-in-cluster/enable-istio-with-psp/_index.md +++ b/content/rancher/v2.x/en/istio/v2.5/setup/enable-istio-in-cluster/enable-istio-with-psp/_index.md @@ -1,5 +1,6 @@ --- title: Enable Istio with Pod Security Policies +weight: 1 aliases: - /rancher/v2.x/en/cluster-admin/tools/istio/setup/enable-istio-in-cluster/enable-istio-with-psp - /rancher/v2.x/en/istio/legacy/setup/enable-istio-in-cluster/enable-istio-with-psp From e11e6af4763035af73ac762cbc48122a8ac74d9c Mon Sep 17 00:00:00 2001 From: Sebastiaan van Steenis Date: Wed, 16 Dec 2020 14:07:43 +0100 Subject: [PATCH 36/49] Explain new cattle-cluster-agent behavior --- .../rke-clusters/rancher-agents/_index.md | 28 +++++++++++++++++-- 1 file changed, 25 insertions(+), 3 deletions(-) diff --git a/content/rancher/v2.x/en/cluster-provisioning/rke-clusters/rancher-agents/_index.md b/content/rancher/v2.x/en/cluster-provisioning/rke-clusters/rancher-agents/_index.md index 0c5c967613c..88eae431f57 100644 --- a/content/rancher/v2.x/en/cluster-provisioning/rke-clusters/rancher-agents/_index.md +++ b/content/rancher/v2.x/en/cluster-provisioning/rke-clusters/rancher-agents/_index.md @@ -22,16 +22,38 @@ The `cattle-node-agent` is used to interact with nodes in a [Rancher Launched Ku ### Scheduling rules -_Applies to v2.3.0 and higher_ +_Applies to v2.5.4 and higher_ + +Starting with Rancher v2.5.4, the tolerations for the `cattle-cluster-agent` changed from `operator:Exists` (allowing all taints) to a fixed set of tolerations (listed below, if no controlplane nodes are visible in the cluster) or dynamically added tolerations based on taints applied to the controlplane nodes. This change was made to allow [Taint based Evictions](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/#taint-based-evictions) to work properly for `cattle-cluster-agent`. The default tolerations are described below. If controlplane nodes are present the cluster, the tolerations will be replaced with tolerations matching the taints on the controlplane nodes. + +| Component | nodeAffinity nodeSelectorTerms | nodeSelector | Tolerations | +| ---------------------- | ------------------------------------------ | ------------ | ------------------------------------------------------------------------------ | +| `cattle-cluster-agent` | `beta.kubernetes.io/os:NotIn:windows` | none | **Note:** These are the default tolerations, and will be replaced by tolerations matching taints applied to controlplane nodes.

`effect:NoSchedule`
`key:node-role.kubernetes.io/controlplane`
`value:true`

`effect:NoSchedule`
`key:node-role.kubernetes.io/control-plane`
`operator:Exists`

`effect:NoSchedule`
`key:node-role.kubernetes.io/master`
`operator:Exists` | +| `cattle-node-agent` | `beta.kubernetes.io/os:NotIn:windows` | none | `operator:Exists` | + +The `cattle-cluster-agent` Deployment has preferred scheduling rules using `preferredDuringSchedulingIgnoredDuringExecution`, favoring to be scheduled on nodes with the `controlplane` node. When there are no controlplane nodes visible in the cluster (this is usually the case when using [Clusters from Hosted Kubernetes Providers]({{}}/rancher/v2.x/en/cluster-provisioning/hosted-kubernetes-clusters/)), you can add the label `cattle.io/cluster-agent=true` on a node to prefer scheduling the `cattle-cluster-agent` pod to that node. + +See [Kubernetes: Assigning Pods to Nodes](https://kubernetes.io/docs/concepts/configuration/assign-pod-node/) to find more information about scheduling rules. + +The `preferredDuringSchedulingIgnoredDuringExecution` configuration is shown in the table below: + +| Weight | Expression | +| ------ | ------------------------------------------------ | +| 100 | `node-role.kubernetes.io/controlplane:In:"true"` | +| 100 | `node-role.kubernetes.io/control-plane:In:"true"` | +| 100 | `node-role.kubernetes.io/master:In:"true"` | +| 1 | `cattle.io/cluster-agent:In:"true"` | + +_Applies to v2.3.0 up to v2.5.3_ | Component | nodeAffinity nodeSelectorTerms | nodeSelector | Tolerations | | ---------------------- | ------------------------------------------ | ------------ | ------------------------------------------------------------------------------ | | `cattle-cluster-agent` | `beta.kubernetes.io/os:NotIn:windows` | none | `operator:Exists` | | `cattle-node-agent` | `beta.kubernetes.io/os:NotIn:windows` | none | `operator:Exists` | -The `cattle-cluster-agent` Deployment has preferred scheduling rules using `requiredDuringSchedulingIgnoredDuringExecution`, favoring to be scheduled on nodes with the `controlplane` node. See [Kubernetes: Assigning Pods to Nodes](https://kubernetes.io/docs/concepts/configuration/assign-pod-node/) to find more information about scheduling rules. +The `cattle-cluster-agent` Deployment has preferred scheduling rules using `preferredDuringSchedulingIgnoredDuringExecution`, favoring to be scheduled on nodes with the `controlplane` node. See [Kubernetes: Assigning Pods to Nodes](https://kubernetes.io/docs/concepts/configuration/assign-pod-node/) to find more information about scheduling rules. -The `requiredDuringSchedulingIgnoredDuringExecution` configuration is shown in the table below: +The `preferredDuringSchedulingIgnoredDuringExecution` configuration is shown in the table below: | Weight | Expression | | ------ | ------------------------------------------------ | From 6657a4ee34f396f4fc9e0173f6c3d269bec5c3a8 Mon Sep 17 00:00:00 2001 From: Prachi Damle Date: Wed, 16 Dec 2020 17:17:46 -0800 Subject: [PATCH 37/49] CIS v2 scheduling, alerting and custom Benchmark Version docs --- .../rancher/v2.x/en/cis-scans/v2.5/_index.md | 151 +++++++++++++++++- 1 file changed, 143 insertions(+), 8 deletions(-) diff --git a/content/rancher/v2.x/en/cis-scans/v2.5/_index.md b/content/rancher/v2.x/en/cis-scans/v2.5/_index.md index d878f359f5b..2381182116c 100644 --- a/content/rancher/v2.x/en/cis-scans/v2.5/_index.md +++ b/content/rancher/v2.x/en/cis-scans/v2.5/_index.md @@ -28,25 +28,31 @@ We now support running CIS scans on any Kubernetes cluster, including hosted Kub In Rancher v2.4, the CIS scan tool was available from the **cluster manager** in the Rancher UI. Now it is available in the **Cluster Explorer** and it can be enabled and deployed using a Helm chart. It can be installed from the Rancher UI, but it can also be installed independently of Rancher. It deploys a CIS scan operator for the cluster, and deploys Kubernetes custom resources for cluster scans. The custom resources can be managed directly from the **Cluster Explorer.** -In v1 of the CIS scan tool, which was available in Rancher v2.4 through the cluster manager, recurring scans could be scheduled. The ability to schedule recurring scans is not yet available in Rancher v2.5. +In v1 of the CIS scan tool, which was available in Rancher v2.4 through the cluster manager, recurring scans could be scheduled. The ability to schedule recurring scans is now also available for CIS v2 from Rancher v2.5.4. -Support for alerting for the cluster scan results is not available for Rancher v2.5 yet. +Support for alerting for the cluster scan results is now also available from Rancher v2.5.4. -More test profiles were added. In Rancher v2.4, permissive and hardened profiles were included. In Rancher v2.5, the following profiles are available: +More test profiles were added. In Rancher v2.4, permissive and hardened profiles were included. From Rancher v2.5.4 onwards, the following profiles are available: - Generic CIS 1.5 -- RKE permissive -- RKE hardened +- Generic CIS 1.6 +- RKE permissive 1.5 +- RKE hardened 1.5 +- RKE permissive 1.6 +- RKE hardened 1.6 - EKS - GKE +- RKE2 permissive 1.5 +- RKE2 permissive 1.5 The default profile depends on the type of cluster that will be scanned: -- For RKE Kubernetes clusters, the RKE permissive profile is the default. +- For RKE Kubernetes clusters, the RKE Permissive 1.6 profile is the default. - EKS and GKE have their own CIS Benchmarks published by `kube-bench`. The corresponding test profiles are used by default for those clusters. -- For cluster types other than RKE, EKS and GKE, the Generic CIS 1.5 profile will be used by default. +- For RKE2 Kubernetes clusters, the RKE2 Permissive 1.5 profile is the default. +- For cluster types other than RKE, RKE2, EKS and GKE, the Generic CIS 1.5 profile will be used by default. -The `rancher-cis-benchmark` currently supports the CIS 1.5 Benchmark version. +The `rancher-cis-benchmark` now supports the CIS 1.6 Benchmark version since the release 2.5.4 > **Note:** CIS v1 cannot run on a cluster when CIS v2 is deployed. In other words, after `rancher-cis-benchmark` is installed, you can't run scans by going to the Cluster Manager view in the Rancher UI and clicking **Tools > CIS Scans.** @@ -200,3 +206,132 @@ For information about permissions, refer to }}/rancher/v2.x/en/cis-scans/configuration" target="_blank">this page. + +# Running a Scan periodically on a schedule + +Since Rancher v2.5.4, it is possible to run a ClusterScan on a schedule. + +To run a scan on a schedule, + +1. Go to the **Cluster Explorer** in the Rancher UI. In the top left dropdown menu, click **Cluster Explorer > CIS Benchmark.** +1. In the **Scans** section, click **Create.** +1. Choose a cluster scan profile. The profile determines which CIS Benchmark version will be used and which tests will be performed. If you choose the Default profile, then the CIS Operator will choose a profile applicable to the type of Kubernetes cluster it is installed on. +1. Choose the option "Run scan on a schedule" +1. Enter a valid cron schedule expression in the field "Schedule" +1. Choose a "Retention" count which indicates the number of reports maintained for this recurring scan. By default this count is 3. When this retention limit is reached, older reports will get purged. +1. Click **Create.** + +**Result:** The scan runs and reschedules to run according to the cron schedule provided. The "Next Scan" value indicates the next time this scan will run again. +A report is generated with the scan results every time the scan runs. To see the latest results, click the name of the scan that appears. +You can also see the previous reports by choosing the report from the "Reports" dropdown on the scan detail page. + +# Enabling Alerting for rancher-cis-benchmark + +Alerts can be configured to be sent out for a scan that runs on a schedule. + +To configure alerts, + +1. While installing or upgrading the `rancher-cis-benchmark` application, set the following flag to `true` in `Values YAML` + + ```yaml + alerts: + enabled: true + ``` +1. Before enabling alerts for `rancher-cis-benchmark`, make sure to install the `rancher-monitoring` application and configure the Receivers and Routes. + Please check [this section.]({{}}/rancher/v2.x/en/monitoring-alerting/v2.5/configuration/#alertmanager-config) +1. While configuring the routes for `rancher-cis-benchmark` alerts, you can specify the matching using the (key: value) pair (job: rancher-cis-scan) + +# Configuring Alerts for a periodic Scan on a schedule + +From Rancher v2.5.4, it is possible to run a ClusterScan on a schedule and also specify if you should receive alerts when the scan completes. +Alerts are supported only for a scan that runs on a schedule. + +The `rancher-cis-benchmark` application supports two types of alerts: +1. Alert on scan completion - this alert is sent out when the scan run finishes. The alert includes details including the ClusterScan's name, ClusterScanProfile name. +1. Alert on scan failure - this alert is sent out if there are some test failures in the scan run or if the scan is in a `Fail` state. + +To configure alerts for a scan that runs on a schedule, + +1. Please enable alerts on the `rancher-cis-benchmark` application (#enabling-alerting-for-rancher-cis-benchmark) +1. Go to the **Cluster Explorer** in the Rancher UI. In the top left dropdown menu, click **Cluster Explorer > CIS Benchmark.** +1. In the **Scans** section, click **Create.** +1. Choose a cluster scan profile. The profile determines which CIS Benchmark version will be used and which tests will be performed. If you choose the Default profile, then the CIS Operator will choose a profile applicable to the type of Kubernetes cluster it is installed on. +1. Choose the option "Run scan on a schedule" +1. Enter a valid cron schedule expression in the field "Schedule" +1. Check the boxes next to the Alert types under `Alerting` +1. Please ensure that Rancher's Monitoring and Alerting app is installed and the Receivers and Routes are configured to send out alerts. +1. Optionally Choose a "Retention" count which indicates the number of reports maintained for this recurring scan. By default this count is 3. When this retention limit is reached, older reports will get purged. +1. Click **Create.** + +**Result:** The scan runs and reschedules to run according to the cron schedule provided. Alerts are sent out when the scan finishes if routes and receiver are configured under `rancher-monitoring` application. +A report is generated with the scan results every time the scan runs. To see the latest results, click the name of the scan that appears. + +# Creating a custom Benchmark Version for running a cluster scan + +When a cluster scan is run, you need to select a Profile which points to a specific Benchmark Version. +Each Benchmark Version defines a set of test configuration files that define the CIS tests to be run by the kube-bench tool. +The `rancher-cis-benchmark` application installs a few default Benchmark Versions which are listed under CIS Benchmark application menu. + +But there could be some kubernetes cluster setups that require custom configurations of the Benchmark tests. +For example, path to the kubernetes config files or certs might be different than the standard location where the upstream +CIS benchmarks look for. + +With Rancher v2.5.4 it is now possible to create a custom Benchmark Version for running a cluster scan using the `rancher-cis-benchmark` application. + +Follow all the steps below to add a custom Benchmark Version and run a scan using it. + +## Preparing the custom Benchmark Version ConfigMap +To create a custom Benchmark Version first you need to create a ConfigMap containing the benchmark version's config files and upload it to your kubernetes cluster where you want to run the scan. + +To prepare a custom Benchmark Version ConfigMap, +1. Suppose we want to add a custom Benchmark Version named `foo` +1. Create a directory named `foo` and place all the config yaml files that kube-bench tool looks for, inside this directory. +1. For example, here are the config yaml files for a Generic CIS 1.5 Benchmark Version https://github.com/aquasecurity/kube-bench/tree/master/cfg/cis-1.5 +1. Place the complete config.yaml file which includes all the components that should be tested. +1. To the config.yaml file add the Benchmark version name to the `target_mapping` section. + ```yaml + target_mapping: + "foo": + - "master" + - "node" + - "controlplane" + - "etcd" + - "policies" + ``` +1. Now upload this directory to your Kubernetes Cluster by creating a Configmap: + ```yaml + kubectl create configmap -n foo --from-file= + ``` +## Adding the custom Benchmark Version to your Cluster + +1. Once the configmap has been created in your cluster, navigate to the **Cluster Explorer** in the Rancher UI. +1. In the top left dropdown menu, click **Cluster Explorer > CIS Benchmark.** +1. In the **Benchmark Versions** section, click **Create.** +1. Enter the Name as `foo` and a description for your custom Benchmark. +1. Choose the cluster provider that your Benchmark Version applies to. +1. Choose the ConfigMap you have uploaded from the dropdown. +1. Add the minimum and maximum Kubernetes version limits applicable if any. +1. Click **Create.** + +## Create a new Profile for your custom Benchmark Version +To run a scan using your custom Benchmark Version, you need to add a new Profile pointing to this Benchmark Version. + +1. Once the custom Benchmark Version has been created in your cluster, navigate to the **Cluster Explorer** in the Rancher UI. +1. In the top left dropdown menu, click **Cluster Explorer > CIS Benchmark.** +1. In the **Profiles** section, click **Create.** +1. Provide a name say `foo-profile` and description +1. Choose the Benchmark Version `foo` from the dropdown +1. Click **Create.** + +## Running a scan using your custom Benchmark Version + +Once the Profile pointing to your custom Benchmark Version `foo` has been created, you can create a new Scan to run the custom test configs in the Benchmark Version. +To run a scan, + +1. Go to the **Cluster Explorer** in the Rancher UI. In the top left dropdown menu, click **Cluster Explorer > CIS Benchmark.** +1. In the **Scans** section, click **Create.** +1. Choose the new cluster scan profile `foo-profile`. +1. Click **Create.** + +**Result:** A report is generated with the scan results. To see the results, click the name of the scan that appears. + From a719692617ba63b050f7372d55cf8a97ee83ab9a Mon Sep 17 00:00:00 2001 From: Catherine Luse Date: Fri, 18 Dec 2020 18:50:31 -0700 Subject: [PATCH 38/49] Edit CIS scan docs for v2.5.4 --- .../rancher/v2.x/en/cis-scans/v2.5/_index.md | 371 +++++++++--------- .../cis-scans/v2.5/custom-benchmark/_index.md | 82 ++++ .../v2.x/en/cis-scans/v2.5/rbac/_index.md | 1 + 3 files changed, 272 insertions(+), 182 deletions(-) create mode 100644 content/rancher/v2.x/en/cis-scans/v2.5/custom-benchmark/_index.md diff --git a/content/rancher/v2.x/en/cis-scans/v2.5/_index.md b/content/rancher/v2.x/en/cis-scans/v2.5/_index.md index 2381182116c..3340e563ec9 100644 --- a/content/rancher/v2.x/en/cis-scans/v2.5/_index.md +++ b/content/rancher/v2.x/en/cis-scans/v2.5/_index.md @@ -4,25 +4,29 @@ shortTitle: Rancher v2.5 weight: 1 --- - Rancher can run a security scan to check whether Kubernetes is deployed according to security best practices as defined in the CIS Kubernetes Benchmark. The `rancher-cis-benchmark` app leverages kube-bench, an open-source tool from Aqua Security, to check clusters for CIS Kubernetes Benchmark compliance. Also, to generate a cluster-wide report, the application utilizes Sonobuoy for report aggregation. - [Changes in Rancher v2.5](#changes-in-rancher-v2-5) - [About the CIS Benchmark](#about-the-cis-benchmark) -- [Installing rancher-cis-benchmark](#installing-rancher-cis-benchmark) -- [Uninstalling rancher-cis-benchmark](#uninstalling-rancher-cis-benchmark) -- [Running a Scan](#running-a-scan) -- [Skipping Tests](#skipping-tests) -- [Viewing Reports](#viewing-reports) -- [About the generated report](#about-the-generated-report) +- [About the Generated Report](#about-the-generated-report) - [Test Profiles](#test-profiles) - [About Skipped and Not Applicable Tests](#about-skipped-and-not-applicable-tests) -- [Roles-based access control](./rbac) +- [Roles-based Access Control](./rbac) - [Configuration](./configuration) +- [How-to Guides](#how-to-guides) + - [Installing rancher-cis-benchmark](#installing-rancher-cis-benchmark) + - [Uninstalling rancher-cis-benchmark](#uninstalling-rancher-cis-benchmark) + - [Running a Scan](#running-a-scan) + - [Running a Scan Periodically on a Schedule](#running-a-scan-periodically-on-a-schedule) + - [Skipping Tests](#skipping-tests) + - [Viewing Reports](#viewing-reports) + - [Enabling Alerting for rancher-cis-benchmark](#enabling-alerting-for-rancher-cis-benchmark) + - [Configuring Alerts for a Periodic Scan on a Schedule](#configuring-alerts-for-a-periodic-scan-on-a-schedule) + - [Creating a Custom Benchmark Version for Running a Cluster Scan](#creating-a-custom-benchmark-version-for-running-a-cluster-scan) -### Changes in Rancher v2.5 +# Changes in Rancher v2.5 We now support running CIS scans on any Kubernetes cluster, including hosted Kubernetes providers such as EKS, AKS, and GKE. Previously it was only supported to run CIS scans on RKE Kubernetes clusters. @@ -32,8 +36,10 @@ In v1 of the CIS scan tool, which was available in Rancher v2.4 through the clus Support for alerting for the cluster scan results is now also available from Rancher v2.5.4. -More test profiles were added. In Rancher v2.4, permissive and hardened profiles were included. From Rancher v2.5.4 onwards, the following profiles are available: +In Rancher v2.4, permissive and hardened profiles were included. In Rancher v2.5.0 and in v2.5.4, more profiles were included. +{{% tabs %}} +{{% tab "Profiles in v2.5.4" %}} - Generic CIS 1.5 - Generic CIS 1.6 - RKE permissive 1.5 @@ -44,17 +50,39 @@ More test profiles were added. In Rancher v2.4, permissive and hardened profiles - GKE - RKE2 permissive 1.5 - RKE2 permissive 1.5 +{{% /tab %}} +{{% tab "Profiles in v2.5.0" %}} +- Generic CIS 1.5 +- RKE permissive +- RKE hardened +- EKS +- GKE +{{% /tab %}} +{{% /tabs %}} +
-The default profile depends on the type of cluster that will be scanned: +The default profile and the supported CIS benchmark version depends on the type of cluster that will be scanned and the Rancher version: + +{{% tabs %}} +{{% tab "v2.5.4" %}} - For RKE Kubernetes clusters, the RKE Permissive 1.6 profile is the default. - EKS and GKE have their own CIS Benchmarks published by `kube-bench`. The corresponding test profiles are used by default for those clusters. - For RKE2 Kubernetes clusters, the RKE2 Permissive 1.5 profile is the default. - For cluster types other than RKE, RKE2, EKS and GKE, the Generic CIS 1.5 profile will be used by default. -The `rancher-cis-benchmark` now supports the CIS 1.6 Benchmark version since the release 2.5.4 +The `rancher-cis-benchmark` supports the CIS 1.6 Benchmark version. +{{% /tab %}} +{{% tab "v2.5.0" %}} +- For RKE Kubernetes clusters, the RKE permissive profile is the default. +- EKS and GKE have their own CIS Benchmarks published by `kube-bench`. The corresponding test profiles are used by default for those clusters. +- For cluster types other than RKE, EKS and GKE, the Generic CIS 1.5 profile will be used by default. -> **Note:** CIS v1 cannot run on a cluster when CIS v2 is deployed. In other words, after `rancher-cis-benchmark` is installed, you can't run scans by going to the Cluster Manager view in the Rancher UI and clicking **Tools > CIS Scans.** +The `rancher-cis-benchmark` supports the CIS 1.5 Benchmark version. +{{% /tab %}} +{{% /tabs %}} + +> **Note:** CIS v1 cannot run on a cluster when CIS v2 is deployed. In other words, after `rancher-cis-benchmark` is installed, you can't run scans by going to the Cluster Manager view in the Rancher UI and clicking Tools > CIS Scans. # About the CIS Benchmark @@ -65,80 +93,6 @@ CIS Benchmarks are best practices for the secure configuration of a target syste The official Benchmark documents are available through the CIS website. The sign-up form to access the documents is here. -# Installing rancher-cis-benchmark - -1. In the Rancher UI, go to the **Cluster Explorer.** -1. Click **Apps.** -1. Click `rancher-cis-benchmark`. -1. Click **Install.** - -**Result:** The CIS scan application is deployed on the Kubernetes cluster. - -# Uninstalling rancher-cis-benchmark - -1. From the **Cluster Explorer,** go to the top left dropdown menu and click **Apps & Marketplace.** -1. Click **Installed Apps.** -1. Go to the `cis-operator-system` namespace and check the boxes next to `rancher-cis-benchmark-crd` and `rancher-cis-benchmark`. -1. Click **Delete** and confirm **Delete.** - -**Result:** The `rancher-cis-benchmark` application is uninstalled. - -# Running a Scan - -When a ClusterScan custom resource is created, it launches a new CIS scan on the cluster for the chosen ClusterScanProfile. - -Note: There is currently a limitation of running only one CIS scan at a time for a cluster. If you create multiple ClusterScan custom resources, they will be run one after the other by the operator, and until one scan finishes, the rest of the ClusterScan custom resources will be in the "Pending" state. - -To run a scan, - -1. Go to the **Cluster Explorer** in the Rancher UI. In the top left dropdown menu, click **Cluster Explorer > CIS Benchmark.** -1. In the **Scans** section, click **Create.** -1. Choose a cluster scan profile. The profile determines which CIS Benchmark version will be used and which tests will be performed. If you choose the Default profile, then the CIS Operator will choose a profile applicable to the type of Kubernetes cluster it is installed on. -1. Click **Create.** - -**Result:** A report is generated with the scan results. To see the results, click the name of the scan that appears. - -# Skipping Tests - -CIS scans can be run using test profiles with user-defined skips. - -To skip tests, you will create a custom CIS scan profile. A profile contains the configuration for the CIS scan, which includes the benchmark versions to use and any specific tests to skip in that benchmark. - -1. In the **Cluster Explorer,** go to the top-left dropdown menu and click **CIS Benchmark.** -1. Click **Profiles.** -1. From here, you can create a profile in multiple ways. To make a new profile, click **Create** and fill out the form in the UI. To make a new profile based on an existing profile, go to the existing profile, click the three vertical dots, and click **Clone as YAML.** If you are filling out the form, add the tests to skip using the test IDs, using the relevant CIS Benchmark as a reference. If you are creating the new test profile as YAML, you will add the IDs of the tests to skip in the `skipTests` directive. You will also give the profile a name: - - ```yaml - apiVersion: cis.cattle.io/v1 - kind: ClusterScanProfile - metadata: - annotations: - meta.helm.sh/release-name: clusterscan-operator - meta.helm.sh/release-namespace: cis-operator-system - labels: - app.kubernetes.io/managed-by: Helm - name: "" - spec: - benchmarkVersion: cis-1.5 - skipTests: - - "1.1.20" - - "1.1.21" - ``` -1. Click **Create.** - -**Result:** A new CIS scan profile is created. - -When you [run a scan](#running-a-scan) that uses this profile, the defined tests will be skipped during the scan. The skipped tests will be marked in the generated report as `Skip`. - -# Viewing Reports - -To view the generated CIS scan reports, - -1. In the **Cluster Explorer,** go to the top left dropdown menu and click **Cluster Explorer > CIS Benchmark.** -1. The **Scans** page will show the generated reports. To see a detailed report, go to a scan report and click the name. - -One can download the report from the Scans list or from the scan detail page. - # About the Generated Report Each scan generates a report can be viewed in the Rancher UI and can be downloaded in CSV format. @@ -172,11 +126,27 @@ Refer to the t The following profiles are available: -- Generic CIS 1.5 (default) +{{% tabs %}} +{{% tab "Profiles in v2.5.4" %}} +- Generic CIS 1.5 +- Generic CIS 1.6 +- RKE permissive 1.5 +- RKE hardened 1.5 +- RKE permissive 1.6 +- RKE hardened 1.6 +- EKS +- GKE +- RKE2 permissive 1.5 +- RKE2 permissive 1.5 +{{% /tab %}} +{{% tab "Profiles in v2.5.0" %}} +- Generic CIS 1.5 - RKE permissive - RKE hardened - EKS - GKE +{{% /tab %}} +{{% /tabs %}} You also have the ability to customize a profile by saving a set of tests to skip. @@ -207,48 +177,146 @@ For information about permissions, refer to }}/rancher/v2.x/en/cis-scans/configuration" target="_blank">this page. -# Running a Scan periodically on a schedule +# How-to Guides -Since Rancher v2.5.4, it is possible to run a ClusterScan on a schedule. +- [Installing rancher-cis-benchmark](#installing-rancher-cis-benchmark) +- [Uninstalling rancher-cis-benchmark](#uninstalling-rancher-cis-benchmark) +- [Running a Scan](#running-a-scan) +- [Running a Scan Periodically on a Schedule](#running-a-scan-periodically-on-a-schedule) +- [Skipping Tests](#skipping-tests) +- [Viewing Reports](#viewing-reports) +- [Enabling Alerting for rancher-cis-benchmark](#enabling-alerting-for-rancher-cis-benchmark) +- [Configuring Alerts for a Periodic Scan on a Schedule](#configuring-alerts-for-a-periodic-scan-on-a-schedule) +- [Creating a Custom Benchmark Version for Running a Cluster Scan](#creating-a-custom-benchmark-version-for-running-a-cluster-scan) +### Installing rancher-cis-benchmark -To run a scan on a schedule, +1. In the Rancher UI, go to the **Cluster Explorer.** +1. Click **Apps.** +1. Click `rancher-cis-benchmark`. +1. Click **Install.** + +**Result:** The CIS scan application is deployed on the Kubernetes cluster. + +### Uninstalling rancher-cis-benchmark + +1. From the **Cluster Explorer,** go to the top left dropdown menu and click **Apps & Marketplace.** +1. Click **Installed Apps.** +1. Go to the `cis-operator-system` namespace and check the boxes next to `rancher-cis-benchmark-crd` and `rancher-cis-benchmark`. +1. Click **Delete** and confirm **Delete.** + +**Result:** The `rancher-cis-benchmark` application is uninstalled. + +### Running a Scan + +When a ClusterScan custom resource is created, it launches a new CIS scan on the cluster for the chosen ClusterScanProfile. + +Note: There is currently a limitation of running only one CIS scan at a time for a cluster. If you create multiple ClusterScan custom resources, they will be run one after the other by the operator, and until one scan finishes, the rest of the ClusterScan custom resources will be in the "Pending" state. + +To run a scan, 1. Go to the **Cluster Explorer** in the Rancher UI. In the top left dropdown menu, click **Cluster Explorer > CIS Benchmark.** 1. In the **Scans** section, click **Create.** 1. Choose a cluster scan profile. The profile determines which CIS Benchmark version will be used and which tests will be performed. If you choose the Default profile, then the CIS Operator will choose a profile applicable to the type of Kubernetes cluster it is installed on. -1. Choose the option "Run scan on a schedule" -1. Enter a valid cron schedule expression in the field "Schedule" -1. Choose a "Retention" count which indicates the number of reports maintained for this recurring scan. By default this count is 3. When this retention limit is reached, older reports will get purged. 1. Click **Create.** -**Result:** The scan runs and reschedules to run according to the cron schedule provided. The "Next Scan" value indicates the next time this scan will run again. -A report is generated with the scan results every time the scan runs. To see the latest results, click the name of the scan that appears. -You can also see the previous reports by choosing the report from the "Reports" dropdown on the scan detail page. +**Result:** A report is generated with the scan results. To see the results, click the name of the scan that appears. +### Running a Scan Periodically on a Schedule +_Available as of v2.5.4_ -# Enabling Alerting for rancher-cis-benchmark +To run a ClusterScan on a schedule, + +1. Go to the **Cluster Explorer** in the Rancher UI. In the top left dropdown menu, click **Cluster Explorer > CIS Benchmark.** +1. In the **Scans** section, click **Create.** +1. Choose a cluster scan profile. The profile determines which CIS Benchmark version will be used and which tests will be performed. If you choose the Default profile, then the CIS Operator will choose a profile applicable to the type of Kubernetes cluster it is installed on. +1. Choose the option **Run scan on a schedule.** +1. Enter a valid [cron schedule expression](https://en.wikipedia.org/wiki/Cron#CRON_expression) in the field **Schedule.** +1. Choose a **Retention** count, which indicates the number of reports maintained for this recurring scan. By default this count is 3. When this retention limit is reached, older reports will get purged. +1. Click **Create.** + +**Result:** The scan runs and reschedules to run according to the cron schedule provided. The **Next Scan** value indicates the next time this scan will run again. + +A report is generated with the scan results every time the scan runs. To see the latest results, click the name of the scan that appears. + +You can also see the previous reports by choosing the report from the **Reports** dropdown on the scan detail page. + +### Skipping Tests + +CIS scans can be run using test profiles with user-defined skips. + +To skip tests, you will create a custom CIS scan profile. A profile contains the configuration for the CIS scan, which includes the benchmark versions to use and any specific tests to skip in that benchmark. + +1. In the **Cluster Explorer,** go to the top-left dropdown menu and click **CIS Benchmark.** +1. Click **Profiles.** +1. From here, you can create a profile in multiple ways. To make a new profile, click **Create** and fill out the form in the UI. To make a new profile based on an existing profile, go to the existing profile, click the three vertical dots, and click **Clone as YAML.** If you are filling out the form, add the tests to skip using the test IDs, using the relevant CIS Benchmark as a reference. If you are creating the new test profile as YAML, you will add the IDs of the tests to skip in the `skipTests` directive. You will also give the profile a name: + + ```yaml + apiVersion: cis.cattle.io/v1 + kind: ClusterScanProfile + metadata: + annotations: + meta.helm.sh/release-name: clusterscan-operator + meta.helm.sh/release-namespace: cis-operator-system + labels: + app.kubernetes.io/managed-by: Helm + name: "" + spec: + benchmarkVersion: cis-1.5 + skipTests: + - "1.1.20" + - "1.1.21" + ``` +1. Click **Create.** + +**Result:** A new CIS scan profile is created. + +When you [run a scan](#running-a-scan) that uses this profile, the defined tests will be skipped during the scan. The skipped tests will be marked in the generated report as `Skip`. + +### Viewing Reports + +To view the generated CIS scan reports, + +1. In the **Cluster Explorer,** go to the top left dropdown menu and click **Cluster Explorer > CIS Benchmark.** +1. The **Scans** page will show the generated reports. To see a detailed report, go to a scan report and click the name. + +One can download the report from the Scans list or from the scan detail page. + +### Enabling Alerting for rancher-cis-benchmark +_Available as of v2.5.4_ Alerts can be configured to be sent out for a scan that runs on a schedule. -To configure alerts, +> **Prerequisite:** +> +> Before enabling alerts for `rancher-cis-benchmark`, make sure to install the `rancher-monitoring` application and configure the Receivers and Routes. Please check [this section.]({{}}/rancher/v2.x/en/monitoring-alerting/v2.5/configuration/alertmanager/) +> +> While configuring the routes for `rancher-cis-benchmark` alerts, you can specify the matching using the key-value pair `job: rancher-cis-scan`. -1. While installing or upgrading the `rancher-cis-benchmark` application, set the following flag to `true` in `Values YAML` +While installing or upgrading the `rancher-cis-benchmark` application, set the following flag to `true` in the `values.yaml`: - ```yaml - alerts: - enabled: true - ``` -1. Before enabling alerts for `rancher-cis-benchmark`, make sure to install the `rancher-monitoring` application and configure the Receivers and Routes. - Please check [this section.]({{}}/rancher/v2.x/en/monitoring-alerting/v2.5/configuration/#alertmanager-config) -1. While configuring the routes for `rancher-cis-benchmark` alerts, you can specify the matching using the (key: value) pair (job: rancher-cis-scan) +```yaml +alerts: + enabled: true +``` -# Configuring Alerts for a periodic Scan on a schedule +### Configuring Alerts for a Periodic Scan on a Schedule +_Available as of v2.5.4_ + +From Rancher v2.5.4, it is possible to run a ClusterScan on a schedule. + +A scheduled scan can also specify if you should receive alerts when the scan completes. -From Rancher v2.5.4, it is possible to run a ClusterScan on a schedule and also specify if you should receive alerts when the scan completes. Alerts are supported only for a scan that runs on a schedule. The `rancher-cis-benchmark` application supports two types of alerts: -1. Alert on scan completion - this alert is sent out when the scan run finishes. The alert includes details including the ClusterScan's name, ClusterScanProfile name. -1. Alert on scan failure - this alert is sent out if there are some test failures in the scan run or if the scan is in a `Fail` state. + +- Alert on scan completion: This alert is sent out when the scan run finishes. The alert includes details including the ClusterScan's name and the ClusterScanProfile name. +- Alert on scan failure: This alert is sent out if there are some test failures in the scan run or if the scan is in a `Fail` state. + +> **Prerequisites:** +> +> Please ensure that Rancher's Monitoring and Alerting app is installed and the Receivers and Routes are configured to send out alerts. Please check [this section.]({{}}/rancher/v2.x/en/monitoring-alerting/v2.5/configuration/alertmanager/) +> +> While configuring the routes for `rancher-cis-benchmark` alerts, you can specify the matching using the key-value pair `job: rancher-cis-scan`. To configure alerts for a scan that runs on a schedule, @@ -256,82 +324,21 @@ To configure alerts for a scan that runs on a schedule, 1. Go to the **Cluster Explorer** in the Rancher UI. In the top left dropdown menu, click **Cluster Explorer > CIS Benchmark.** 1. In the **Scans** section, click **Create.** 1. Choose a cluster scan profile. The profile determines which CIS Benchmark version will be used and which tests will be performed. If you choose the Default profile, then the CIS Operator will choose a profile applicable to the type of Kubernetes cluster it is installed on. -1. Choose the option "Run scan on a schedule" -1. Enter a valid cron schedule expression in the field "Schedule" -1. Check the boxes next to the Alert types under `Alerting` -1. Please ensure that Rancher's Monitoring and Alerting app is installed and the Receivers and Routes are configured to send out alerts. -1. Optionally Choose a "Retention" count which indicates the number of reports maintained for this recurring scan. By default this count is 3. When this retention limit is reached, older reports will get purged. +1. Choose the option **Run scan on a schedule.** +1. Enter a valid [cron schedule expression](https://en.wikipedia.org/wiki/Cron#CRON_expression) in the field **Schedule.** +1. Check the boxes next to the Alert types under **Alerting.** +1. Optional: Choose a **Retention** count, which indicates the number of reports maintained for this recurring scan. By default this count is 3. When this retention limit is reached, older reports will get purged. 1. Click **Create.** **Result:** The scan runs and reschedules to run according to the cron schedule provided. Alerts are sent out when the scan finishes if routes and receiver are configured under `rancher-monitoring` application. + A report is generated with the scan results every time the scan runs. To see the latest results, click the name of the scan that appears. -# Creating a custom Benchmark Version for running a cluster scan - -When a cluster scan is run, you need to select a Profile which points to a specific Benchmark Version. -Each Benchmark Version defines a set of test configuration files that define the CIS tests to be run by the kube-bench tool. -The `rancher-cis-benchmark` application installs a few default Benchmark Versions which are listed under CIS Benchmark application menu. +### Creating a Custom Benchmark Version for Running a Cluster Scan +_Available as of v2.5.4_ -But there could be some kubernetes cluster setups that require custom configurations of the Benchmark tests. -For example, path to the kubernetes config files or certs might be different than the standard location where the upstream -CIS benchmarks look for. +There could be some Kubernetes cluster setups that require custom configurations of the Benchmark tests. For example, the path to the Kubernetes config files or certs might be different than the standard location where the upstream CIS Benchmarks look for them. -With Rancher v2.5.4 it is now possible to create a custom Benchmark Version for running a cluster scan using the `rancher-cis-benchmark` application. +It is now possible to create a custom Benchmark Version for running a cluster scan using the `rancher-cis-benchmark` application. -Follow all the steps below to add a custom Benchmark Version and run a scan using it. - -## Preparing the custom Benchmark Version ConfigMap -To create a custom Benchmark Version first you need to create a ConfigMap containing the benchmark version's config files and upload it to your kubernetes cluster where you want to run the scan. - -To prepare a custom Benchmark Version ConfigMap, -1. Suppose we want to add a custom Benchmark Version named `foo` -1. Create a directory named `foo` and place all the config yaml files that kube-bench tool looks for, inside this directory. -1. For example, here are the config yaml files for a Generic CIS 1.5 Benchmark Version https://github.com/aquasecurity/kube-bench/tree/master/cfg/cis-1.5 -1. Place the complete config.yaml file which includes all the components that should be tested. -1. To the config.yaml file add the Benchmark version name to the `target_mapping` section. - ```yaml - target_mapping: - "foo": - - "master" - - "node" - - "controlplane" - - "etcd" - - "policies" - ``` -1. Now upload this directory to your Kubernetes Cluster by creating a Configmap: - ```yaml - kubectl create configmap -n foo --from-file= - ``` -## Adding the custom Benchmark Version to your Cluster - -1. Once the configmap has been created in your cluster, navigate to the **Cluster Explorer** in the Rancher UI. -1. In the top left dropdown menu, click **Cluster Explorer > CIS Benchmark.** -1. In the **Benchmark Versions** section, click **Create.** -1. Enter the Name as `foo` and a description for your custom Benchmark. -1. Choose the cluster provider that your Benchmark Version applies to. -1. Choose the ConfigMap you have uploaded from the dropdown. -1. Add the minimum and maximum Kubernetes version limits applicable if any. -1. Click **Create.** - -## Create a new Profile for your custom Benchmark Version -To run a scan using your custom Benchmark Version, you need to add a new Profile pointing to this Benchmark Version. - -1. Once the custom Benchmark Version has been created in your cluster, navigate to the **Cluster Explorer** in the Rancher UI. -1. In the top left dropdown menu, click **Cluster Explorer > CIS Benchmark.** -1. In the **Profiles** section, click **Create.** -1. Provide a name say `foo-profile` and description -1. Choose the Benchmark Version `foo` from the dropdown -1. Click **Create.** - -## Running a scan using your custom Benchmark Version - -Once the Profile pointing to your custom Benchmark Version `foo` has been created, you can create a new Scan to run the custom test configs in the Benchmark Version. -To run a scan, - -1. Go to the **Cluster Explorer** in the Rancher UI. In the top left dropdown menu, click **Cluster Explorer > CIS Benchmark.** -1. In the **Scans** section, click **Create.** -1. Choose the new cluster scan profile `foo-profile`. -1. Click **Create.** - -**Result:** A report is generated with the scan results. To see the results, click the name of the scan that appears. - +For details, see [this page.](./custom-benchmark) \ No newline at end of file diff --git a/content/rancher/v2.x/en/cis-scans/v2.5/custom-benchmark/_index.md b/content/rancher/v2.x/en/cis-scans/v2.5/custom-benchmark/_index.md new file mode 100644 index 00000000000..f55e25478ae --- /dev/null +++ b/content/rancher/v2.x/en/cis-scans/v2.5/custom-benchmark/_index.md @@ -0,0 +1,82 @@ +--- +title: Creating a Custom Benchmark Version for Running a Cluster Scan +weight: 4 +--- + +_Available as of v2.5.4_ + +Each Benchmark Version defines a set of test configuration files that define the CIS tests to be run by the kube-bench tool. +The `rancher-cis-benchmark` application installs a few default Benchmark Versions which are listed under CIS Benchmark application menu. + +But there could be some Kubernetes cluster setups that require custom configurations of the Benchmark tests. For example, the path to the Kubernetes config files or certs might be different than the standard location where the upstream CIS Benchmarks look for them. + +It is now possible to create a custom Benchmark Version for running a cluster scan using the `rancher-cis-benchmark` application. + +When a cluster scan is run, you need to select a Profile which points to a specific Benchmark Version. + +Follow all the steps below to add a custom Benchmark Version and run a scan using it. + +1. [Prepare the Custom Benchmark Version ConfigMap](#1-prepare-the-custom-benchmark-version-configmap) +2. [Add a Custom Benchmark Version to a Cluster](#2-add-a-custom-benchmark-version-to-a-cluster) +3. [Create a New Profile for the Custom Benchmark Version](#3-create-a-new-profile-for-the-custom-benchmark-version) +4. [Run a Scan Using the Custom Benchmark Version](#4-run-a-scan-using-the-custom-benchmark-version) + +### 1. Prepare the Custom Benchmark Version ConfigMap + +To create a custom benchmark version, first you need to create a ConfigMap containing the benchmark version's config files and upload it to your Kubernetes cluster where you want to run the scan. + +To prepare a custom benchmark version ConfigMap, suppose we want to add a custom Benchmark Version named `foo`. + +1. Create a directory named `foo` and inside this directory, place all the config YAML files that the kube-bench tool looks for. For example, here are the config YAML files for a Generic CIS 1.5 Benchmark Version https://github.com/aquasecurity/kube-bench/tree/master/cfg/cis-1.5 +1. Place the complete `config.yaml` file, which includes all the components that should be tested. +1. Add the Benchmark version name to the `target_mapping` section of the `config.yaml`: + + ```yaml + target_mapping: + "foo": + - "master" + - "node" + - "controlplane" + - "etcd" + - "policies" + ``` +1. Upload this directory to your Kubernetes Cluster by creating a ConfigMap: + + ```yaml + kubectl create configmap -n foo --from-file= + ``` + +### 2. Add a Custom Benchmark Version to a Cluster + +1. Once the ConfigMap has been created in your cluster, navigate to the **Cluster Explorer** in the Rancher UI. +1. In the top left dropdown menu, click **Cluster Explorer > CIS Benchmark.** +1. In the **Benchmark Versions** section, click **Create.** +1. Enter the **Name** and a description for your custom benchmark version. +1. Choose the cluster provider that your benchmark version applies to. +1. Choose the ConfigMap you have uploaded from the dropdown. +1. Add the minimum and maximum Kubernetes version limits applicable, if any. +1. Click **Create.** + +### 3. Create a New Profile for the Custom Benchmark Version + +To run a scan using your custom benchmark version, you need to add a new Profile pointing to this benchmark version. + +1. Once the custom benchmark version has been created in your cluster, navigate to the **Cluster Explorer** in the Rancher UI. +1. In the top left dropdown menu, click **Cluster Explorer > CIS Benchmark.** +1. In the **Profiles** section, click **Create.** +1. Provide a **Name** and description. In this example, we name it `foo-profile`. +1. Choose the Benchmark Version `foo` from the dropdown. +1. Click **Create.** + +### 4. Run a Scan Using the Custom Benchmark Version + +Once the Profile pointing to your custom benchmark version `foo` has been created, you can create a new Scan to run the custom test configs in the Benchmark Version. + +To run a scan, + +1. Go to the **Cluster Explorer** in the Rancher UI. In the top left dropdown menu, click **Cluster Explorer > CIS Benchmark.** +1. In the **Scans** section, click **Create.** +1. Choose the new cluster scan profile `foo-profile`. +1. Click **Create.** + +**Result:** A report is generated with the scan results. To see the results, click the name of the scan that appears. \ No newline at end of file diff --git a/content/rancher/v2.x/en/cis-scans/v2.5/rbac/_index.md b/content/rancher/v2.x/en/cis-scans/v2.5/rbac/_index.md index 078c859a6f2..a3424ba562c 100644 --- a/content/rancher/v2.x/en/cis-scans/v2.5/rbac/_index.md +++ b/content/rancher/v2.x/en/cis-scans/v2.5/rbac/_index.md @@ -11,6 +11,7 @@ This section describes the permissions required to use the rancher-cis-benchmark The rancher-cis-benchmark is a cluster-admin only feature by default. However, the `rancher-cis-benchmark` chart installs these two default `ClusterRoles`: + - cis-admin - cis-view From 6fcb833430028206d9a5d57087cf7336511c18f0 Mon Sep 17 00:00:00 2001 From: cluse Date: Sat, 19 Dec 2020 18:09:57 -0700 Subject: [PATCH 39/49] Document iptables and container-selinux config for RancherD install on SELinux --- .../installation/install-rancher-on-linux/_index.md | 1 + .../v2.x/en/installation/requirements/_index.md | 11 +++++++++++ 2 files changed, 12 insertions(+) diff --git a/content/rancher/v2.x/en/installation/install-rancher-on-linux/_index.md b/content/rancher/v2.x/en/installation/install-rancher-on-linux/_index.md index 0ceaf5bb836..cb4278538de 100644 --- a/content/rancher/v2.x/en/installation/install-rancher-on-linux/_index.md +++ b/content/rancher/v2.x/en/installation/install-rancher-on-linux/_index.md @@ -38,6 +38,7 @@ RancherD must be launched on a Linux OS. At this time, only OSes that leverage s The Linux node needs to fulfill the [installation requirements]({{}}/rancher/v2.x/en/installation/requirements) for hardware and networking. +To install RancherD on SELinux Enforcing CentOS 8 nodes, some [additional steps]({{}}/rancher/v2.x/en/installation/requirements/#rancherd-on-selinux-enforcing-centos-8-nodes) are required. ### Root Access Before running the installation commands, you will need to log in as root: diff --git a/content/rancher/v2.x/en/installation/requirements/_index.md b/content/rancher/v2.x/en/installation/requirements/_index.md index 45b6ef00f1d..9a199bf4514 100644 --- a/content/rancher/v2.x/en/installation/requirements/_index.md +++ b/content/rancher/v2.x/en/installation/requirements/_index.md @@ -31,6 +31,8 @@ Rancher should work with any modern Linux distribution. At this time, only Linux OSes that leverage systemd are supported. +To install RancherD on SELinux Enforcing CentOS 8 nodes, some [additional steps](#rancherd-on-selinux-enforcing-centos-8-nodes) are required. + ### Requirements for Installing Rancher on an RKE Kubernetes Cluster For the container runtime, RKE should work with any modern Docker version, while K3s should work with any modern version of Docker or containerd. @@ -148,3 +150,12 @@ Each node used should have a static IP configured, regardless of whether you are ### Port Requirements To operate properly, Rancher requires a number of ports to be open on Rancher nodes and on downstream Kubernetes cluster nodes. [Port Requirements]({{}}/rancher/v2.x/en/installation/requirements/ports) lists all the necessary ports for Rancher and Downstream Clusters for the different cluster types. + +# RancherD on SELinux Enforcing CentOS 8 Nodes + +Before installing Rancher on SELinux Enforcing CentOS 8 nodes, you must install `container-selinux` and `iptables`: + +``` +sudo yum install iptables +sudo yum install container-selinux +``` \ No newline at end of file From 142c2066ab22836ad3375e1d0647af48899afc9f Mon Sep 17 00:00:00 2001 From: Catherine Luse Date: Mon, 21 Dec 2020 11:25:04 -0700 Subject: [PATCH 40/49] Call out RHEL 8 nodes as needing extra config for RancherD --- .../v2.x/en/installation/install-rancher-on-linux/_index.md | 2 +- content/rancher/v2.x/en/installation/requirements/_index.md | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/content/rancher/v2.x/en/installation/install-rancher-on-linux/_index.md b/content/rancher/v2.x/en/installation/install-rancher-on-linux/_index.md index cb4278538de..30c69a01a0f 100644 --- a/content/rancher/v2.x/en/installation/install-rancher-on-linux/_index.md +++ b/content/rancher/v2.x/en/installation/install-rancher-on-linux/_index.md @@ -38,7 +38,7 @@ RancherD must be launched on a Linux OS. At this time, only OSes that leverage s The Linux node needs to fulfill the [installation requirements]({{}}/rancher/v2.x/en/installation/requirements) for hardware and networking. -To install RancherD on SELinux Enforcing CentOS 8 nodes, some [additional steps]({{}}/rancher/v2.x/en/installation/requirements/#rancherd-on-selinux-enforcing-centos-8-nodes) are required. +To install RancherD on SELinux Enforcing CentOS 8 nodes or RHEL 8 nodes, some [additional steps]({{}}/rancher/v2.x/en/installation/requirements/#rancherd-on-selinux-enforcing-centos-8-nodes) are required. ### Root Access Before running the installation commands, you will need to log in as root: diff --git a/content/rancher/v2.x/en/installation/requirements/_index.md b/content/rancher/v2.x/en/installation/requirements/_index.md index 9a199bf4514..446fb435e7e 100644 --- a/content/rancher/v2.x/en/installation/requirements/_index.md +++ b/content/rancher/v2.x/en/installation/requirements/_index.md @@ -31,7 +31,7 @@ Rancher should work with any modern Linux distribution. At this time, only Linux OSes that leverage systemd are supported. -To install RancherD on SELinux Enforcing CentOS 8 nodes, some [additional steps](#rancherd-on-selinux-enforcing-centos-8-nodes) are required. +To install RancherD on SELinux Enforcing CentOS 8 or RHEL 8 nodes, some [additional steps](#rancherd-on-selinux-enforcing-centos-8-or-rhel-8-nodes) are required. ### Requirements for Installing Rancher on an RKE Kubernetes Cluster @@ -151,9 +151,9 @@ Each node used should have a static IP configured, regardless of whether you are To operate properly, Rancher requires a number of ports to be open on Rancher nodes and on downstream Kubernetes cluster nodes. [Port Requirements]({{}}/rancher/v2.x/en/installation/requirements/ports) lists all the necessary ports for Rancher and Downstream Clusters for the different cluster types. -# RancherD on SELinux Enforcing CentOS 8 Nodes +# RancherD on SELinux Enforcing CentOS 8 or RHEL 8 Nodes -Before installing Rancher on SELinux Enforcing CentOS 8 nodes, you must install `container-selinux` and `iptables`: +Before installing Rancher on SELinux Enforcing CentOS 8 nodes or RHEL 8 nodes, you must install `container-selinux` and `iptables`: ``` sudo yum install iptables From ac128aac27dff6512915f850fd5a3e925dbe1897 Mon Sep 17 00:00:00 2001 From: Catherine Luse Date: Mon, 21 Dec 2020 13:28:29 -0700 Subject: [PATCH 41/49] Add example for CIS alerts in Alertmanager doc --- .../rancher/v2.x/en/cis-scans/v2.5/_index.md | 14 +++++----- .../v2.5/configuration/alertmanager/_index.md | 27 ++++++++++++++++++- 2 files changed, 33 insertions(+), 8 deletions(-) diff --git a/content/rancher/v2.x/en/cis-scans/v2.5/_index.md b/content/rancher/v2.x/en/cis-scans/v2.5/_index.md index 3340e563ec9..64dc4ed02cd 100644 --- a/content/rancher/v2.x/en/cis-scans/v2.5/_index.md +++ b/content/rancher/v2.x/en/cis-scans/v2.5/_index.md @@ -152,7 +152,7 @@ You also have the ability to customize a profile by saving a set of tests to ski All profiles will have a set of not applicable tests that will be skipped during the CIS scan. These tests are not applicable based on how a RKE cluster manages Kubernetes. -There are 2 types of RKE cluster scan profiles: +There are two types of RKE cluster scan profiles: - **Permissive:** This profile has a set of tests that have been will be skipped as these tests will fail on a default RKE Kubernetes cluster. Besides the list of skipped tests, the profile will also not run the not applicable tests. - **Hardened:** This profile will not skip any tests, except for the non-applicable tests. @@ -287,9 +287,9 @@ Alerts can be configured to be sent out for a scan that runs on a schedule. > **Prerequisite:** > -> Before enabling alerts for `rancher-cis-benchmark`, make sure to install the `rancher-monitoring` application and configure the Receivers and Routes. Please check [this section.]({{}}/rancher/v2.x/en/monitoring-alerting/v2.5/configuration/alertmanager/) +> Before enabling alerts for `rancher-cis-benchmark`, make sure to install the `rancher-monitoring` application and configure the Receivers and Routes. For more information, see [this section.]({{}}/rancher/v2.x/en/monitoring-alerting/v2.5/configuration/alertmanager/) > -> While configuring the routes for `rancher-cis-benchmark` alerts, you can specify the matching using the key-value pair `job: rancher-cis-scan`. +> While configuring the routes for `rancher-cis-benchmark` alerts, you can specify the matching using the key-value pair `job: rancher-cis-scan`. An example route configuration is [here.]({{}}/rancher/v2.x/en/monitoring-alerting/v2.5/configuration/alertmanager/#example-route-config-for-cis-scan-alerts) While installing or upgrading the `rancher-cis-benchmark` application, set the following flag to `true` in the `values.yaml`: @@ -312,11 +312,11 @@ The `rancher-cis-benchmark` application supports two types of alerts: - Alert on scan completion: This alert is sent out when the scan run finishes. The alert includes details including the ClusterScan's name and the ClusterScanProfile name. - Alert on scan failure: This alert is sent out if there are some test failures in the scan run or if the scan is in a `Fail` state. -> **Prerequisites:** -> -> Please ensure that Rancher's Monitoring and Alerting app is installed and the Receivers and Routes are configured to send out alerts. Please check [this section.]({{}}/rancher/v2.x/en/monitoring-alerting/v2.5/configuration/alertmanager/) +> **Prerequisite:** > -> While configuring the routes for `rancher-cis-benchmark` alerts, you can specify the matching using the key-value pair `job: rancher-cis-scan`. +> Before enabling alerts for `rancher-cis-benchmark`, make sure to install the `rancher-monitoring` application and configure the Receivers and Routes. For more information, see [this section.]({{}}/rancher/v2.x/en/monitoring-alerting/v2.5/configuration/alertmanager/) +> +> While configuring the routes for `rancher-cis-benchmark` alerts, you can specify the matching using the key-value pair `job: rancher-cis-scan`. An example route configuration is [here.]({{}}/rancher/v2.x/en/monitoring-alerting/v2.5/configuration/alertmanager/#example-route-config-for-cis-scan-alerts) To configure alerts for a scan that runs on a schedule, diff --git a/content/rancher/v2.x/en/monitoring-alerting/v2.5/configuration/alertmanager/_index.md b/content/rancher/v2.x/en/monitoring-alerting/v2.5/configuration/alertmanager/_index.md index ad986dc02f4..f306902e888 100644 --- a/content/rancher/v2.x/en/monitoring-alerting/v2.5/configuration/alertmanager/_index.md +++ b/content/rancher/v2.x/en/monitoring-alerting/v2.5/configuration/alertmanager/_index.md @@ -20,6 +20,7 @@ The [Alertmanager Config](https://prometheus.io/docs/alerting/latest/configurati - [Grouping](#grouping) - [Matching](#matching) - [Example Alertmanager YAML](#example-alertmanager-yaml) +- [Example Route Config for CIS Scan Alerts](#example-route-config-for-cis-scan-alerts) # Overview @@ -207,4 +208,28 @@ receivers: api_url: templates: - /etc/alertmanager/config/*.tmpl -``` \ No newline at end of file +``` + +# Example Route Config for CIS Scan Alerts + +While configuring the routes for `rancher-cis-benchmark` alerts, you can specify the matching using the key-value pair `job: rancher-cis-scan`. + +For example, the following example route configuration could be used with a Slack receiver named `test-cis`: + +```yaml +spec: + receiver: test-cis + group_by: +# - string + group_wait: 30s + group_interval: 30s + repeat_interval: 30s + match: + job: rancher-cis-scan +# key: string + match_re: + {} +# key: string +``` + +For more information on enabling alerting for `rancher-cis-benchmark`, see [this section.]({{}}/rancher/v2.x/en/cis-scans/v2.5/#enabling-alerting-for-rancher-cis-benchmark) \ No newline at end of file From ccceb987d49e651bce1760caf5d838e2c4dde7f1 Mon Sep 17 00:00:00 2001 From: Catherine Luse Date: Mon, 21 Dec 2020 13:57:16 -0700 Subject: [PATCH 42/49] Respond to feedback on CIS scan alerting doc --- .../rancher/v2.x/en/cis-scans/v2.5/_index.md | 20 ++++++++++++------- 1 file changed, 13 insertions(+), 7 deletions(-) diff --git a/content/rancher/v2.x/en/cis-scans/v2.5/_index.md b/content/rancher/v2.x/en/cis-scans/v2.5/_index.md index 64dc4ed02cd..affe72c2015 100644 --- a/content/rancher/v2.x/en/cis-scans/v2.5/_index.md +++ b/content/rancher/v2.x/en/cis-scans/v2.5/_index.md @@ -51,7 +51,7 @@ In Rancher v2.4, permissive and hardened profiles were included. In Rancher v2.5 - RKE2 permissive 1.5 - RKE2 permissive 1.5 {{% /tab %}} -{{% tab "Profiles in v2.5.0" %}} +{{% tab "Profiles in v2.5.0-v2.5.3" %}} - Generic CIS 1.5 - RKE permissive - RKE hardened @@ -66,19 +66,23 @@ The default profile and the supported CIS benchmark version depends on the type {{% tabs %}} {{% tab "v2.5.4" %}} + +The `rancher-cis-benchmark` supports the CIS 1.6 Benchmark version. + - For RKE Kubernetes clusters, the RKE Permissive 1.6 profile is the default. - EKS and GKE have their own CIS Benchmarks published by `kube-bench`. The corresponding test profiles are used by default for those clusters. - For RKE2 Kubernetes clusters, the RKE2 Permissive 1.5 profile is the default. - For cluster types other than RKE, RKE2, EKS and GKE, the Generic CIS 1.5 profile will be used by default. -The `rancher-cis-benchmark` supports the CIS 1.6 Benchmark version. {{% /tab %}} -{{% tab "v2.5.0" %}} +{{% tab "v2.5.0-v2.5.3" %}} + +The `rancher-cis-benchmark` supports the CIS 1.5 Benchmark version. + - For RKE Kubernetes clusters, the RKE permissive profile is the default. - EKS and GKE have their own CIS Benchmarks published by `kube-bench`. The corresponding test profiles are used by default for those clusters. - For cluster types other than RKE, EKS and GKE, the Generic CIS 1.5 profile will be used by default. -The `rancher-cis-benchmark` supports the CIS 1.5 Benchmark version. {{% /tab %}} {{% /tabs %}} @@ -97,7 +101,9 @@ The official Benchmark documents are available through the CIS website. The sign Each scan generates a report can be viewed in the Rancher UI and can be downloaded in CSV format. -In Rancher v2.5, the scan will use the CIS Benchmark v1.5. The Benchmark version is included in the generated report. +From Rancher v2.5.4, the scan uses the CIS Benchmark v1.6 by default. In Rancher v2.5.0-2.5.3, the CIS Benchmark v1.5. is used. + +The Benchmark version is included in the generated report. The Benchmark provides recommendations of two types: Scored and Not Scored. Recommendations marked as Not Scored in the Benchmark are not included in the generated report. @@ -139,7 +145,7 @@ The following profiles are available: - RKE2 permissive 1.5 - RKE2 permissive 1.5 {{% /tab %}} -{{% tab "Profiles in v2.5.0" %}} +{{% tab "Profiles in v2.5.0-v2.5.3" %}} - Generic CIS 1.5 - RKE permissive - RKE hardened @@ -229,7 +235,7 @@ To run a ClusterScan on a schedule, 1. In the **Scans** section, click **Create.** 1. Choose a cluster scan profile. The profile determines which CIS Benchmark version will be used and which tests will be performed. If you choose the Default profile, then the CIS Operator will choose a profile applicable to the type of Kubernetes cluster it is installed on. 1. Choose the option **Run scan on a schedule.** -1. Enter a valid [cron schedule expression](https://en.wikipedia.org/wiki/Cron#CRON_expression) in the field **Schedule.** +1. Enter a valid cron schedule expression in the field **Schedule.** 1. Choose a **Retention** count, which indicates the number of reports maintained for this recurring scan. By default this count is 3. When this retention limit is reached, older reports will get purged. 1. Click **Create.** From 21f2e0f107dd1c8024d227f817eaa3aaf6c7a26f Mon Sep 17 00:00:00 2001 From: Catherine Luse Date: Mon, 21 Dec 2020 17:06:56 -0700 Subject: [PATCH 43/49] Edit docs on Alertmanager configuration in Rancher UI --- .../v2.x/en/monitoring-alerting/v2.5/_index.md | 12 ++++++------ .../monitoring-alerting/v2.5/configuration/_index.md | 8 +++++--- .../v2.5/configuration/alertmanager/_index.md | 2 +- .../v2.5/configuration/prometheusrules/_index.md | 8 ++++---- 4 files changed, 16 insertions(+), 14 deletions(-) diff --git a/content/rancher/v2.x/en/monitoring-alerting/v2.5/_index.md b/content/rancher/v2.x/en/monitoring-alerting/v2.5/_index.md index 3abcea08ed0..da627e8e719 100644 --- a/content/rancher/v2.x/en/monitoring-alerting/v2.5/_index.md +++ b/content/rancher/v2.x/en/monitoring-alerting/v2.5/_index.md @@ -38,7 +38,7 @@ For more information about upgrading the Monitoring app in Rancher 2.5, please r - [Grafana UI](#grafana-ui) - [Prometheus UI](#prometheus-ui) - [Viewing the Prometheus Targets](#viewing-the-prometheus-targets) - - [Viewing the Prometheus Rules](#viewing-the-prometheus-rules) + - [Viewing the PrometheusRules](#viewing-the-prometheus-rules) - [Viewing Active Alerts in Alertmanager](#viewing-active-alerts-in-alertmanager) - [Uninstall Monitoring](#uninstall-monitoring) - [Setting Resource Limits and Requests](#setting-resource-limits-and-requests) @@ -129,14 +129,14 @@ To see the Prometheus Targets, install `rancher-monitoring`. Then go to the **Cl
Targets in the Prometheus UI
![Prometheus Targets UI]({{}}/img/rancher/prometheus-targets-ui.png) -### Viewing the Prometheus Rules +### Viewing the PrometheusRules -To see the Prometheus Rules, install `rancher-monitoring`. Then go to the **Cluster Explorer.** In the top left corner, click **Cluster Explorer > Monitoring.** Then click **Prometheus Rules.** +To see the PrometheusRules, install `rancher-monitoring`. Then go to the **Cluster Explorer.** In the top left corner, click **Cluster Explorer > Monitoring.** Then click **Prometheus Rules.**
Rules in the Prometheus UI
-![Prometheus Rules UI]({{}}/img/rancher/prometheus-rules-ui.png) +![PrometheusRules UI]({{}}/img/rancher/prometheus-rules-ui.png) -For more information on Prometheus Rules in Rancher, see [this page.](./configuration/prometheusrules) +For more information on PrometheusRules in Rancher, see [this page.](./configuration/prometheusrules) ### Viewing Active Alerts in Alertmanager @@ -146,7 +146,7 @@ The Alertmanager handles alerts sent by client applications such as the Promethe In the Alertmanager UI, you can view your alerts and the current Alertmanager configuration. -To see the Prometheus Rules, install `rancher-monitoring`. Then go to the **Cluster Explorer.** In the top left corner, click **Cluster Explorer > Monitoring.** Then click **Alertmanager.** +To see the PrometheusRules, install `rancher-monitoring`. Then go to the **Cluster Explorer.** In the top left corner, click **Cluster Explorer > Monitoring.** Then click **Alertmanager.** **Result:** The Alertmanager UI opens in a new tab. For help with configuration, refer to the [official Alertmanager documentation.](https://prometheus.io/docs/alerting/latest/alertmanager/) diff --git a/content/rancher/v2.x/en/monitoring-alerting/v2.5/configuration/_index.md b/content/rancher/v2.x/en/monitoring-alerting/v2.5/configuration/_index.md index 2479ff2314c..8f7ae53ca05 100644 --- a/content/rancher/v2.x/en/monitoring-alerting/v2.5/configuration/_index.md +++ b/content/rancher/v2.x/en/monitoring-alerting/v2.5/configuration/_index.md @@ -49,7 +49,7 @@ This CRD declaratively specifies how group of pods should be monitored. Any Pods This CRD defines a group of Prometheus alerting and/or recording rules. -For information on configuring Prometheus rules, refer to [this page.](./prometheusrules) +For information on configuring PrometheusRules, refer to [this page.](./prometheusrules) # Alertmanager Config @@ -85,9 +85,11 @@ An example PodMonitor can be found [here.](https://github.com/prometheus-operato ### PrometheusRule -Prometheus rule files are held in PrometheusRule custom resources. For users who are familiar with Prometheus, a PrometheusRule contains the alerting and recording rules that you would normally place in a [Prometheus rule file](https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/). +For users who are familiar with Prometheus, a PrometheusRule contains the alerting and recording rules that you would normally place in a [Prometheus rule file](https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/). -Use the label selector field ruleSelector in the Prometheus object to define the rule files that you want to be mounted into Prometheus. An example PrometheusRule is on [this page.](https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/user-guides/alerting.md) +For a more fine-grained application of PrometheusRules within your cluster, the ruleSelector field on a Prometheus resource allows you to select which PrometheusRules should be loaded onto Prometheus based on the labels attached to the PrometheusRules resources. + +An example PrometheusRule is on [this page.](https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/user-guides/alerting.md) ### Alertmanager Config diff --git a/content/rancher/v2.x/en/monitoring-alerting/v2.5/configuration/alertmanager/_index.md b/content/rancher/v2.x/en/monitoring-alerting/v2.5/configuration/alertmanager/_index.md index f306902e888..1862398d06e 100644 --- a/content/rancher/v2.x/en/monitoring-alerting/v2.5/configuration/alertmanager/_index.md +++ b/content/rancher/v2.x/en/monitoring-alerting/v2.5/configuration/alertmanager/_index.md @@ -42,7 +42,7 @@ For more information, refer to the [official Prometheus documentation about conf ### Connecting Routes and PrometheusRules -When you define a Rule within a RuleGroup of a PrometheusRule, the spec of the Rule itself contains labels that are used by Prometheus to figure out which Route should receive this Alert. For example, an Alert with the label `team: front-end` will be sent to all Routes that match on that label. +When you define a Rule (which is declared within a RuleGroup in a PrometheusRule resource), the [spec of the Rule itself](https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/api.md#rule) contains labels that are used by Prometheus to figure out which Route should receive this Alert. For example, an Alert with the label `team: front-end` will be sent to all Routes that match on that label. # Creating Receivers in the Rancher UI _Available as of v2.5.4_ diff --git a/content/rancher/v2.x/en/monitoring-alerting/v2.5/configuration/prometheusrules/_index.md b/content/rancher/v2.x/en/monitoring-alerting/v2.5/configuration/prometheusrules/_index.md index 5ba169f87f3..6e249101c48 100644 --- a/content/rancher/v2.x/en/monitoring-alerting/v2.5/configuration/prometheusrules/_index.md +++ b/content/rancher/v2.x/en/monitoring-alerting/v2.5/configuration/prometheusrules/_index.md @@ -17,7 +17,7 @@ A PrometheusRule defines a group of Prometheus alerting and/or recording rules. Prometheus rule files are held in PrometheusRule custom resources. -A PrometheusRule allows you to define one or more RuleGroups. Each RuleGroup consists of a set of alerting or recording rules with the following fields: +A PrometheusRule allows you to define one or more RuleGroups. Each RuleGroup consists of a set of Rule objects that can each represent either an alerting or a recording rule with the following fields: - The name of the new alert or record - A PromQL (Prometheus query language) expression for the new alert or record @@ -34,7 +34,7 @@ For examples, refer to the Prometheus documentation on [recording rules](https:/ ### Connecting Routes and PrometheusRules -When you define a Rule within a RuleGroup of a PrometheusRule, the spec of the Rule itself contains labels that are used by Prometheus to figure out which Route should receive this Alert. For example, an Alert with the label `team: front-end` will be sent to all Routes that match on that label. +When you define a Rule (which is declared within a RuleGroup in a PrometheusRule resource), the [spec of the Rule itself](https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/api.md#rule) contains labels that are used by Prometheus to figure out which Route should receive this Alert. For example, an Alert with the label `team: front-end` will be sent to all Routes that match on that label. ### Creating PrometheusRules in the Rancher UI @@ -75,7 +75,7 @@ Rancher v2.5.4 introduced the capability to configure PrometheusRules by filling |-------|----------------| | Alert Name | The name of the alert. Must be a valid label value. | | Wait To Fire For | Duration in seconds. Alerts are considered firing once they have been returned for this long. Alerts which have not yet fired for long enough are considered pending. | -| PromQL Expression | The PromQL expression to evaluate. Prometheus will evaluate the current value of this PromQL expression on every evaluation cycle and all resultant time series become pending/firing alerts. For more information, refer to the [Prometheus documentation](https://prometheus.io/docs/prometheus/latest/querying/basics/) or our [example PromQL expressions.](../expression) | +| PromQL Expression | The PromQL expression to evaluate. Prometheus will evaluate the current value of this PromQL expression on every evaluation cycle and all resultant time series will become pending/firing alerts. For more information, refer to the [Prometheus documentation](https://prometheus.io/docs/prometheus/latest/querying/basics/) or our [example PromQL expressions.](../expression) | | Labels | Labels to add or overwrite for each alert. | | Severity | When enabled, labels are attached to the alert or record that identify it by the severity level. | | Severity Label Value | Critical, warning, or none | @@ -88,7 +88,7 @@ Rancher v2.5.4 introduced the capability to configure PrometheusRules by filling | Field | Description | |-------|----------------| | Time Series Name | The name of the time series to output to. Must be a valid metric name. | -| PromQL Expression | The PromQL expression to evaluate. Prometheus will evaluate the current value of this PromQL expression on every evaluation cycle and the result recorded as a new set of time series with the metric name as given by 'record'. For more information about expressions, refer to the [Prometheus documentation](https://prometheus.io/docs/prometheus/latest/querying/basics/) or our [example PromQL expressions.](../expression) | +| PromQL Expression | The PromQL expression to evaluate. Prometheus will evaluate the current value of this PromQL expression on every evaluation cycle and the result will be recorded as a new set of time series with the metric name as given by 'record'. For more information about expressions, refer to the [Prometheus documentation](https://prometheus.io/docs/prometheus/latest/querying/basics/) or our [example PromQL expressions.](../expression) | | Labels | Labels to add or overwrite before storing the result. | {{% /tab %}} From abd33fa9dd3b9ecb4080ac859b75b9af311d90b9 Mon Sep 17 00:00:00 2001 From: cluse Date: Wed, 23 Dec 2020 12:35:06 -0700 Subject: [PATCH 44/49] Fix link to CentOS/RHEL 8 prereqs for RancherD --- .../v2.x/en/installation/install-rancher-on-linux/_index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/rancher/v2.x/en/installation/install-rancher-on-linux/_index.md b/content/rancher/v2.x/en/installation/install-rancher-on-linux/_index.md index 30c69a01a0f..09403de4231 100644 --- a/content/rancher/v2.x/en/installation/install-rancher-on-linux/_index.md +++ b/content/rancher/v2.x/en/installation/install-rancher-on-linux/_index.md @@ -38,7 +38,7 @@ RancherD must be launched on a Linux OS. At this time, only OSes that leverage s The Linux node needs to fulfill the [installation requirements]({{}}/rancher/v2.x/en/installation/requirements) for hardware and networking. -To install RancherD on SELinux Enforcing CentOS 8 nodes or RHEL 8 nodes, some [additional steps]({{}}/rancher/v2.x/en/installation/requirements/#rancherd-on-selinux-enforcing-centos-8-nodes) are required. +To install RancherD on SELinux Enforcing CentOS 8 nodes or RHEL 8 nodes, some [additional steps]({{}}/rancher/v2.x/en/installation/requirements/#rancherd-on-selinux-enforcing-centos-8-or-rhel-8-nodes) are required. ### Root Access Before running the installation commands, you will need to log in as root: From 5355b3625acb1f92c90b033452b1490e4a512fed Mon Sep 17 00:00:00 2001 From: Catherine Luse Date: Tue, 5 Jan 2021 13:16:56 -0700 Subject: [PATCH 45/49] Address feedback on RancherD docs #2935 --- .../install-rancher-on-linux/_index.md | 28 ++++++++++++++++--- .../rancherd-configuration/_index.md | 2 +- .../en/installation/requirements/_index.md | 5 +++- 3 files changed, 29 insertions(+), 6 deletions(-) diff --git a/content/rancher/v2.x/en/installation/install-rancher-on-linux/_index.md b/content/rancher/v2.x/en/installation/install-rancher-on-linux/_index.md index 09403de4231..4f66f2fa523 100644 --- a/content/rancher/v2.x/en/installation/install-rancher-on-linux/_index.md +++ b/content/rancher/v2.x/en/installation/install-rancher-on-linux/_index.md @@ -36,7 +36,7 @@ Part II explains how to convert the single-node Rancher installation into a high RancherD must be launched on a Linux OS. At this time, only OSes that leverage systemd are supported. -The Linux node needs to fulfill the [installation requirements]({{}}/rancher/v2.x/en/installation/requirements) for hardware and networking. +The Linux node needs to fulfill the [installation requirements]({{}}/rancher/v2.x/en/installation/requirements) for hardware and networking. Docker is not required for RancherD installs. To install RancherD on SELinux Enforcing CentOS 8 nodes or RHEL 8 nodes, some [additional steps]({{}}/rancher/v2.x/en/installation/requirements/#rancherd-on-selinux-enforcing-centos-8-or-rhel-8-nodes) are required. ### Root Access @@ -66,11 +66,15 @@ This endpoint can be set up using any number of approaches, such as: * Round-robin DNS * Virtual or elastic IP addresses -Note that the RancherD server process listens on port 9345 for new nodes to register. The Kubernetes API is served on port 6443, as normal. Configure your load balancer accordingly. +The following should be taken into consideration when configuring the load balancer or other endpoint: + +- The RancherD server process listens on port 9345 for new nodes to register. +- The Kubernetes API is served on port 6443, as normal. +- In RancherD installs, the Rancher UI is served on port 8443 by default. (This is different from Helm chart installs, where port 443 is used by default.) # Part I: Installing Rancher -### 1. Set up the config.yaml +### 1. Set up Configurations To avoid certificate errors with the fixed registration address, you should launch the server with the `tls-san` parameter set. This parameter should refer to your fixed registration address. @@ -87,10 +91,14 @@ tls-san: The first server node establishes the secret token that other nodes would register with if they are added to the cluster. -If you do not specify a pre-shared secret, RancherD will generate one and place it at `/var/lib/rancher/rancherd/server/node-token`. +If you do not specify a pre-shared secret, RancherD will generate one and place it at `/var/lib/rancher/rke2/server/node-token`. To specify your own pre-shared secret as the token, set the `token` argument on startup. +Installing Rancher this way will use Rancher-generated certificates. To use your own self-signed or trusted certificates, refer to the [configuration guide.]({{}}/rancher/v2.x/en/installation/install-rancher-on-linux/rancherd-configuration/#certificates-for-the-rancher-server) + +For information on customizing the RancherD Helm chart values.yaml, refer to [this section.]({{}}/rancher/v2.x/en/installation/install-rancher-on-linux/rancherd-configuration/#customizing-the-rancherd-helm-chart) + ### 2. Launch the first server node Run the RancherD installer: @@ -99,6 +107,12 @@ Run the RancherD installer: curl -sfL https://get.rancher.io | sh - ``` +The RancherD version can be specified using the `INSTALL_RANCHERD_VERSION` environment variable: + +``` +curl -sfL https://get.rancher.io | INSTALL_RANCHERD_VERSION=v2.5.4-rc6 sh - +``` + Once installed, the `rancherd` binary will be on your PATH. You can check out its help text like this: ``` @@ -138,6 +152,12 @@ kubectl get daemonset rancher -n cattle-system kubectl get pod -n cattle-system ``` +If you watch the pods, you will see the following pods installed: + +- `helm-operation` pods in the `cattle-system` namespace +- a `rancher` pod and `rancher-webhook` pod in the `cattle-system` namespace +- a `fleet-agent`, `fleet-controller`, and `gitjob` pod in the `fleet-system` namespace +- a `rancher-operator` pod in the `rancher-operator-system` namespace ### 5. Set the initial Rancher password Once the `rancher` pod is up and running, run the following: diff --git a/content/rancher/v2.x/en/installation/install-rancher-on-linux/rancherd-configuration/_index.md b/content/rancher/v2.x/en/installation/install-rancher-on-linux/rancherd-configuration/_index.md index e8570e52777..271f4851a84 100644 --- a/content/rancher/v2.x/en/installation/install-rancher-on-linux/rancherd-configuration/_index.md +++ b/content/rancher/v2.x/en/installation/install-rancher-on-linux/rancherd-configuration/_index.md @@ -56,7 +56,7 @@ spec: publicCA: true ``` -Put this manifest on your host in `/var/lib/rancher/rancherd/server/manifests` before running RancherD. +Put this manifest on your host in `/var/lib/rancher/rke2/server/manifests` before running RancherD. ### Common Options diff --git a/content/rancher/v2.x/en/installation/requirements/_index.md b/content/rancher/v2.x/en/installation/requirements/_index.md index 446fb435e7e..8bbd84c728c 100644 --- a/content/rancher/v2.x/en/installation/requirements/_index.md +++ b/content/rancher/v2.x/en/installation/requirements/_index.md @@ -33,6 +33,8 @@ At this time, only Linux OSes that leverage systemd are supported. To install RancherD on SELinux Enforcing CentOS 8 or RHEL 8 nodes, some [additional steps](#rancherd-on-selinux-enforcing-centos-8-or-rhel-8-nodes) are required. +Docker is not required for RancherD installs. + ### Requirements for Installing Rancher on an RKE Kubernetes Cluster For the container runtime, RKE should work with any modern Docker version, while K3s should work with any modern version of Docker or containerd. @@ -65,8 +67,9 @@ If you plan to run Rancher on ARM64, see [Running on ARM64 (Experimental).]({{}}/rancher/v2.x/en/installation/requirements/installing-docker) to install Docker with one command. +Docker is required for Helm chart installs, and it can be installed by following the steps in the official [Docker documentation.](https://docs.docker.com/) Rancher also provides [scripts]({{}}/rancher/v2.x/en/installation/requirements/installing-docker) to install Docker with one command. +Docker is not required for RancherD installs. # Hardware Requirements This section describes the CPU, memory, and disk requirements for the nodes where the Rancher server is installed. From 07cd31ac9d1c6b41a60af37fe40f117aa5a5fdc1 Mon Sep 17 00:00:00 2001 From: Caleb Bron Date: Tue, 5 Jan 2021 13:29:45 -0700 Subject: [PATCH 46/49] Recommend not auto-upgrading GKE nodes --- .../hosted-kubernetes-clusters/gke/_index.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/content/rancher/v2.x/en/cluster-provisioning/hosted-kubernetes-clusters/gke/_index.md b/content/rancher/v2.x/en/cluster-provisioning/hosted-kubernetes-clusters/gke/_index.md index f08a196861b..f26c83b8a57 100644 --- a/content/rancher/v2.x/en/cluster-provisioning/hosted-kubernetes-clusters/gke/_index.md +++ b/content/rancher/v2.x/en/cluster-provisioning/hosted-kubernetes-clusters/gke/_index.md @@ -38,6 +38,10 @@ Use {{< product >}} to set up and configure your Kubernetes cluster. >**Note:** After submitting your private key, you may have to enable the Google Kubernetes Engine API. If prompted, browse to the URL displayed in the Rancher UI to enable the API. -6. Select your **Cluster Options**, customize your **Nodes** and customize the **Security** for the GKE cluster. Review your options to confirm they're correct. Then click **Create**. +6. Select your **Cluster Options** +7. Customize your **Node Options** + * Enabling the Auto Upgrade feature for Nodes is not recommended. +8. Select your **Security Options** +9. Review your options to confirm they're correct. Then click **Create**. {{< result_create-cluster >}} From 297508d221ed674fe58b2cc87b9621ce483700b1 Mon Sep 17 00:00:00 2001 From: Catherine Luse Date: Tue, 5 Jan 2021 20:43:47 -0700 Subject: [PATCH 47/49] Say that RancherD install is experimental --- content/rancher/v2.x/en/installation/_index.md | 16 ++++++++++++++-- .../install-rancher-on-linux/_index.md | 6 +++--- .../v2.x/en/installation/requirements/_index.md | 6 ++++-- 3 files changed, 21 insertions(+), 7 deletions(-) diff --git a/content/rancher/v2.x/en/installation/_index.md b/content/rancher/v2.x/en/installation/_index.md index 92f72123065..3ff51deceba 100644 --- a/content/rancher/v2.x/en/installation/_index.md +++ b/content/rancher/v2.x/en/installation/_index.md @@ -15,6 +15,8 @@ In this section, - **The Rancher server** manages and provisions Kubernetes clusters. You can interact with downstream Kubernetes clusters through the Rancher server's user interface. - **RKE (Rancher Kubernetes Engine)** is a certified Kubernetes distribution and CLI/library which creates and manages a Kubernetes cluster. - **K3s (Lightweight Kubernetes)** is also a fully compliant Kubernetes distribution. It is newer than RKE, easier to use, and more lightweight, with a binary size of less than 100 MB. As of Rancher v2.4, Rancher can be installed on a K3s cluster. +- **RKE2** is a fully conformant Kubernetes distribution that focuses on security and compliance within the U.S. Federal Government sector. +- **RancherD** is a new tool for installing Rancher, which is experimental as of Rancher v2.5.4. RancherD is a single binary that first launches an RKE2 Kubernetes cluster, then installs the Rancher server Helm chart on the cluster. # Changes to Installation in Rancher v2.5 @@ -28,13 +30,23 @@ The `restrictedAdmin` Helm chart option was added. When this option is set to tr Rancher can be installed on these main architectures: -### High-availability Kubernetes Install +### High-availability Kubernetes Install with the Helm CLI We recommend using Helm, a Kubernetes package manager, to install Rancher on multiple nodes on a dedicated Kubernetes cluster. For RKE clusters, three nodes are required to achieve a high-availability cluster. For K3s clusters, only two nodes are required. +### High-availability Kubernetes Install with RancherD + +_Experimental as of v2.5.4_ + +RancherD is a single binary that first launches an RKE2 Kubernetes cluster, then installs the Rancher server Helm chart on the cluster. + +In both the RancherD install and the Helm CLI install, Rancher is installed as a Helm chart on a Kubernetes cluster. + +Configuration and upgrading are also simplified with RancherD. When you upgrade the RancherD binary, both the Kubernetes cluster and the Rancher Helm chart are upgraded. + ### Single-node Kubernetes Install -Another option is to install Rancher with Helm on a Kubernetes cluster, but to only use a single node in the cluster. In this case, the Rancher server doesn't have high availability, which is important for running Rancher in production. +Rancher can be installed on a single-node Kubernetes cluster. In this case, the Rancher server doesn't have high availability, which is important for running Rancher in production. However, this option is useful if you want to save resources by using a single node in the short term, while preserving a high-availability migration path. In the future, you can add nodes to the cluster to get a high-availability Rancher server. diff --git a/content/rancher/v2.x/en/installation/install-rancher-on-linux/_index.md b/content/rancher/v2.x/en/installation/install-rancher-on-linux/_index.md index 4f66f2fa523..3ffa34f53da 100644 --- a/content/rancher/v2.x/en/installation/install-rancher-on-linux/_index.md +++ b/content/rancher/v2.x/en/installation/install-rancher-on-linux/_index.md @@ -3,9 +3,9 @@ title: Install Rancher on a Linux OS weight: 2 --- -_Available as of Rancher v2.5_ +_Experimental as of Rancher v2.5.4_ -As part of Rancher 2.5, we are excited to introduce a new, simpler way to install Rancher called RancherD. +We are excited to introduce a new, simpler way to install Rancher called RancherD. RancherD is a single binary that first launches an RKE2 Kubernetes cluster, then installs the Rancher server Helm chart on the cluster. @@ -22,7 +22,7 @@ RancherD is a single binary that first launches an RKE2 Kubernetes cluster, then When RancherD is launched on a host, it first installs an RKE2 Kubernetes cluster, then deploys Rancher on the cluster as a Kubernetes daemonset. -In both the RancherD install and the Helm CLI install, Rancher is installed as a Helm chart on a Kubernetes cluster. A highly available RancherD installation is equally suitable for production compared to a highly available installation using the Helm CLI. +In both the RancherD install and the Helm CLI install, Rancher is installed as a Helm chart on a Kubernetes cluster. Configuration and upgrading are also simplified with RancherD. When you upgrade the RancherD binary, both the Kubernetes cluster and the Rancher Helm chart are upgraded. diff --git a/content/rancher/v2.x/en/installation/requirements/_index.md b/content/rancher/v2.x/en/installation/requirements/_index.md index 8bbd84c728c..1573fca8762 100644 --- a/content/rancher/v2.x/en/installation/requirements/_index.md +++ b/content/rancher/v2.x/en/installation/requirements/_index.md @@ -29,15 +29,17 @@ Rancher should work with any modern Linux distribution. ### Requirements for Installing Rancher with RancherD +_RancherD installs are experimental as of v2.5.4_ + At this time, only Linux OSes that leverage systemd are supported. To install RancherD on SELinux Enforcing CentOS 8 or RHEL 8 nodes, some [additional steps](#rancherd-on-selinux-enforcing-centos-8-or-rhel-8-nodes) are required. Docker is not required for RancherD installs. -### Requirements for Installing Rancher on an RKE Kubernetes Cluster +### Requirements for Installing Rancher on an RKE Kubernetes Cluster with the Helm CLI -For the container runtime, RKE should work with any modern Docker version, while K3s should work with any modern version of Docker or containerd. +RKE should work with any modern Docker version, while K3s should work with any modern version of Docker or containerd. Rancher and RKE have been tested and are supported on Ubuntu, CentOS, Oracle Linux, RancherOS, and RedHat Enterprise Linux. From 0bd2eacde3c69c88a470c772befcaafbbf78c134 Mon Sep 17 00:00:00 2001 From: Catherine Luse Date: Wed, 6 Jan 2021 10:30:37 -0700 Subject: [PATCH 48/49] Edit language around RancherD as an experimental feature --- content/rancher/v2.x/en/installation/_index.md | 6 ++++-- .../install-rancher-on-linux/_index.md | 4 +++- .../v2.x/en/installation/requirements/_index.md | 14 +++++++------- 3 files changed, 14 insertions(+), 10 deletions(-) diff --git a/content/rancher/v2.x/en/installation/_index.md b/content/rancher/v2.x/en/installation/_index.md index 3ff51deceba..d7d27ea890b 100644 --- a/content/rancher/v2.x/en/installation/_index.md +++ b/content/rancher/v2.x/en/installation/_index.md @@ -16,7 +16,7 @@ In this section, - **RKE (Rancher Kubernetes Engine)** is a certified Kubernetes distribution and CLI/library which creates and manages a Kubernetes cluster. - **K3s (Lightweight Kubernetes)** is also a fully compliant Kubernetes distribution. It is newer than RKE, easier to use, and more lightweight, with a binary size of less than 100 MB. As of Rancher v2.4, Rancher can be installed on a K3s cluster. - **RKE2** is a fully conformant Kubernetes distribution that focuses on security and compliance within the U.S. Federal Government sector. -- **RancherD** is a new tool for installing Rancher, which is experimental as of Rancher v2.5.4. RancherD is a single binary that first launches an RKE2 Kubernetes cluster, then installs the Rancher server Helm chart on the cluster. +- **RancherD** is a new tool for installing Rancher, which is available as of Rancher v2.5.4. It is an experimental feature. RancherD is a single binary that first launches an RKE2 Kubernetes cluster, then installs the Rancher server Helm chart on the cluster. # Changes to Installation in Rancher v2.5 @@ -36,7 +36,9 @@ We recommend using Helm, a Kubernetes package manager, to install Rancher on mul ### High-availability Kubernetes Install with RancherD -_Experimental as of v2.5.4_ +_Available as of v2.5.4_ + +> This is an experimental feature. RancherD is a single binary that first launches an RKE2 Kubernetes cluster, then installs the Rancher server Helm chart on the cluster. diff --git a/content/rancher/v2.x/en/installation/install-rancher-on-linux/_index.md b/content/rancher/v2.x/en/installation/install-rancher-on-linux/_index.md index 3ffa34f53da..567ca98ae14 100644 --- a/content/rancher/v2.x/en/installation/install-rancher-on-linux/_index.md +++ b/content/rancher/v2.x/en/installation/install-rancher-on-linux/_index.md @@ -3,7 +3,9 @@ title: Install Rancher on a Linux OS weight: 2 --- -_Experimental as of Rancher v2.5.4_ +_Available as of Rancher v2.5.4_ + +> This is an experimental feature. We are excited to introduce a new, simpler way to install Rancher called RancherD. diff --git a/content/rancher/v2.x/en/installation/requirements/_index.md b/content/rancher/v2.x/en/installation/requirements/_index.md index 1573fca8762..4376f180bb2 100644 --- a/content/rancher/v2.x/en/installation/requirements/_index.md +++ b/content/rancher/v2.x/en/installation/requirements/_index.md @@ -27,9 +27,15 @@ The Rancher UI works best in Firefox or Chrome. Rancher should work with any modern Linux distribution. +### Requirements for Installing Rancher on an RKE Kubernetes Cluster with the Helm CLI + +RKE should work with any modern Docker version, while K3s should work with any modern version of Docker or containerd. + +Rancher and RKE have been tested and are supported on Ubuntu, CentOS, Oracle Linux, RancherOS, and RedHat Enterprise Linux. + ### Requirements for Installing Rancher with RancherD -_RancherD installs are experimental as of v2.5.4_ +_The RancherD install is available as of v2.5.4. It is an experimental feature._ At this time, only Linux OSes that leverage systemd are supported. @@ -37,12 +43,6 @@ To install RancherD on SELinux Enforcing CentOS 8 or RHEL 8 nodes, some [additio Docker is not required for RancherD installs. -### Requirements for Installing Rancher on an RKE Kubernetes Cluster with the Helm CLI - -RKE should work with any modern Docker version, while K3s should work with any modern version of Docker or containerd. - -Rancher and RKE have been tested and are supported on Ubuntu, CentOS, Oracle Linux, RancherOS, and RedHat Enterprise Linux. - ### Requirements for Installing Rancher on a K3s Kubernetes Cluster K3s should run on just about any flavor of Linux. However, K3s is tested on the following operating systems and their subsequent non-major releases: From 86671f4d2bde2ae15d17f84f70fa2da35c806a1a Mon Sep 17 00:00:00 2001 From: Catherine Luse Date: Thu, 7 Jan 2021 09:28:04 -0700 Subject: [PATCH 49/49] Update node requirements --- .../v2.x/en/installation/requirements/_index.md | 16 +++++++--------- 1 file changed, 7 insertions(+), 9 deletions(-) diff --git a/content/rancher/v2.x/en/installation/requirements/_index.md b/content/rancher/v2.x/en/installation/requirements/_index.md index 4376f180bb2..ed008e58e0c 100644 --- a/content/rancher/v2.x/en/installation/requirements/_index.md +++ b/content/rancher/v2.x/en/installation/requirements/_index.md @@ -25,13 +25,15 @@ The Rancher UI works best in Firefox or Chrome. # Operating Systems and Container Runtime Requirements -Rancher should work with any modern Linux distribution. +The node requirements depend on how Rancher is installed. ### Requirements for Installing Rancher on an RKE Kubernetes Cluster with the Helm CLI -RKE should work with any modern Docker version, while K3s should work with any modern version of Docker or containerd. +Rancher should work with any modern Linux distribution. -Rancher and RKE have been tested and are supported on Ubuntu, CentOS, Oracle Linux, RancherOS, and RedHat Enterprise Linux. +For details on which OS and Docker versions were tested with each Rancher version, refer to the [support maintenance terms.](https://rancher.com/support-maintenance-terms/) + +RKE should work with any modern Docker version, while K3s should work with any modern version of Docker or containerd. ### Requirements for Installing Rancher with RancherD @@ -45,11 +47,9 @@ Docker is not required for RancherD installs. ### Requirements for Installing Rancher on a K3s Kubernetes Cluster -K3s should run on just about any flavor of Linux. However, K3s is tested on the following operating systems and their subsequent non-major releases: +K3s should run on just about any flavor of Linux. -- Ubuntu 16.04 (amd64) -- Ubuntu 18.04 (amd64) -- Raspbian Buster (armhf) +For details on which OS and Docker versions were tested with each Rancher version, refer to the [support maintenance terms.](https://rancher.com/support-maintenance-terms/) If you are installing Rancher on a K3s cluster with **Raspbian Buster**, follow [these steps]({{}}/k3s/latest/en/advanced/#enabling-legacy-iptables-on-raspbian-buster) to switch to legacy iptables. @@ -57,8 +57,6 @@ If you are installing Rancher on a K3s cluster with Alpine Linux, follow [these ### General Linux Requirements -For details on which OS and Docker versions were tested with each Rancher version, refer to the [support maintenance terms.](https://rancher.com/support-maintenance-terms/) - All supported operating systems are 64-bit x86. The `ntp` (Network Time Protocol) package should be installed. This prevents errors with certificate validation that can occur when the time is not synchronized between the client and server.